程序员子悠 · 好记性不如烂笔头

Azkaband 搭建

获取压缩包tar.gz

1. 拉取相应版本代码
   https://github.com/azkaban/azkaban/releases
   git clone https://github.com/azkaban/azkaban.git
2. 在项目根目录下一次执行
   • 清理一下./gradlew clean
   • 压缩成jar包./gradlew distTar
3. 生成azkaban-web-server，azkaban-exec-server，azkaban-db等tar.gz包
   ~/zakaban/azkaban-web-server/build/distributions/azkaban-web-server-3.49.0-4-g7efe67b.tar.gz
~/azkaban/azkaban-exec-server/build/distributions/azkaban-exec-server-3.49.0-4-g7efe67b.tar.gz
~/azkaban/azkaban-db/build/distributions/azkaban-db-3.49.0-4-g7efe67b.tar.gz
4. 复制上面三个tar.gz到/opt/azkaban/目录，并解压重命名

初始化azkaban数据库

1. 创建数据库azkaban
   create database azkaban;
2. 创建用户和密码
   CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban123456';
3. 给用户赋予数据库权限
   GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;
4. 执行sql脚本初始化数据库
   source /opt/azkaban/azkaban-db-3.49.0-4/create-all-sql-3.49.0-4-g7efe67b.sql
5. 配置jetty ssl
   密码：123456
   

配置azkaban-web-server

1. 在/opt/azkaban/azkaban-web-server-3.49.0-4目录下创建conf目录(如果目录下有conf目录则忽略，没有则从azkaban-solo-server目录下面拷贝过来，并拷贝log4j.properties 放到conf目录下)
2. 编辑azkaban.properties文件，其中mysql数据库，用户密码根据上一步配置的进行调整，此处没有配置jetty ssl，jetty.use.ssl=false

silence-pro:conf silence$ cat azkaban.properties
# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai

# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml

# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
database.type=mysql
#h2.path=./h2
#h2.create.tables=true
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban123456
mysql.numconnections=100

# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081
jetty.ssl.port=8443
jetty.keystore=keystore
jetty.password=123456
jetty.keypassword=123456
jetty.truststore=keystore
jetty.trustpassword=123456
jetty.excludeCipherSuites=SSL_RSA_WITH_DES_CBC_SHA,SSL_DHE_RSA_WITH_DES_CBC_SHA,SSL_DHE_DSS_WITH_DES_CBC_SHA,SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA,SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA

# Azkaban Executor settings
executor.port=12321
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
#azkaban.jobtype.plugin.dir=plugins/jobtypes

3. 配置azkaban-users.xml文件，测试使用默认设置

silence-pro:conf silence$ cat azkaban-users.xml
<azkaban-users>
  <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
  <user password="metrics" roles="metrics" username="metrics"/>

  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>

4. 根据需要配置log4j.properties文件，测试使用默认

silence-pro:conf silence$ cat log4j.properties
log4j.rootLogger=INFO, Console
log4j.logger.azkaban=INFO, server
log4j.appender.server=org.apache.log4j.RollingFileAppender
log4j.appender.server.layout=org.apache.log4j.PatternLayout
log4j.appender.server.File=local/azkaban-webserver.log
log4j.appender.server.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n
log4j.appender.server.MaxFileSize=102400MB
log4j.appender.server.MaxBackupIndex=2
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
log4j.appender.Console.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n
silence-pro:conf silence$

5. 启动azkaban-web-server

./bin/start-web.sh，关闭使用./bin/shutdown-web.sh

6. 查看日志文件

tail -f ./local/azkaban-webserver.log

7. 访问http://localhost:8081/index 初始密码为azkaban-user.xml里面配置的账号密码，默认为azkaban，azkaban

配置azkaban-exec-server

1. 复制azkaban-web-server下的conf目录到azkaban-exec-server的conf目录下

mkdir conf
cp ../azkaban-web-server-3.49.0-4/conf/* ./conf/

2. 编辑azkaban.properties

silence-pro:conf silence$ cat azkaban.properties
# Azkaban Personalization Settings
default.timezone.id=Asia/Shanghai
# Azkaban JobTypes Plugins
azkaban.jobtype.plugin.dir=plugins/jobtypes
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=azkaban
mysql.password=azkaban123456
mysql.numconnections=100

# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true

2. 编辑log4j.propertis文件，根据需要调整，这里更换了一个日志文件名称azkaban-execserver.log

silence-pro:conf silence$ cat log4j.properties
log4j.rootLogger=INFO, Console
log4j.logger.azkaban=INFO, server
log4j.appender.server=org.apache.log4j.RollingFileAppender
log4j.appender.server.layout=org.apache.log4j.PatternLayout
log4j.appender.server.File=local/azkaban-execserver.log
log4j.appender.server.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n
log4j.appender.server.MaxFileSize=102400MB
log4j.appender.server.MaxBackupIndex=2
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
log4j.appender.Console.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS Z} %p [%c{1}] [Azkaban] %m%n

3. 启动azkaban-exec-server
   ./bin/start-exec.sh，关闭使用./bin/shutdown-exec.sh
4. 查看日志
   tail -f azkaban-execserver.log

测试任务执行

1. 创建简单任务脚本

silence-pro:usercase silence$ cat command.job
#command.job
type=command
command=echo 'hello azkaban'
silence-pro:usercase silence$

2. 压缩成zip包

silence-pro:usercase silence$ zip command.job.zip command.job
  adding: command.job (deflated 16%)
silence-pro:usercase silence$ ll
total 16
-rw-r--r--  1 silence  wheel   55  6 28 17:25 command.job
-rw-r--r--  1 silence  wheel  218  6 30 16:43 command.job.zip
silence-pro:usercase silence$

3. 在Azkaban系统创建测试project，并上传zip包进行测试
   ![Alt text]https://zxsilence.cn(/images/posts/articles/2018-06-30/Azkaban1.png)
   
   

搭建过程遇到的坑

1. Missing required property 'azkaban.native.lib'
   解决方案：设置execute.as.user=false

silence-pro:jobtypes silence$ pwd
/opt/azkaban/azkaban-exec-server-3.49.0-4/plugins/jobtypes
silence-pro:jobtypes silence$ cat commonprivate.properties
# set execute-as-user
execute.as.user=false
#execute.as.user=true
#azkaban.native.lib=/opt/azkaban/azkaban-web-server-3.49.0-4/bin
silence-pro:jobtypes silence$

安装gradle

https://docs.gradle.org/current/userguide/installation.html
配置环境变量

vim ~/.bash_profile
export Gradle_HOME=/Users/silence/Work/gradle/gradle-4.8.1
export PATH=$PATH:$Gradle_HOME/bin
gradle -v