Downloaded 30 times
Uploaded bywyukawa
Azkaban
This document summarizes Azkaban, an open source workflow scheduler that was created at LinkedIn to manage Hadoop jobs and their dependencies. Key features of Azkaban include defining job dependencies in a simple interface, retry functionality, scheduling, and viewing logs and execution details in the web UI. The document also discusses how the author uses Azkaban to manage Python batch jobs at their company, including writing job files in YAML format and using the Azkaban API. In conclusion, the author finds Azkaban simple to use and sees no reason to replace it, though hopes for more active development.
Azkaban
- 1.
- 2. Azkaban • Implemented at LinkedIn to solve the problem of Hadoop job dependencies • Written in Java –Not modern Java(raw servlet, velocity,...)
- 3. Azkaban feature • Simple Job Management Tool – Define job dependency –Retry – Scheduling – Web UI • See dependency/execution time/log • Store log to db as blob – SPOF – Not register holiday – Not triggered by file creation event • Mail notification only – HTTP Job Callback • No binary – Need to build source • Not so active development • Mailing List doesn’t function very well
- 4.
- 5.
- 6.
- 7. Failure Options • Finish Current Running – finishes only the currently running job. It will not start any new jobs. •Cancel All – immediately kills all jobs and fails the flow. • Finish All Possible – will keep executing jobs as long as its dependencies are met.
- 8.
- 9.
- 10.
- 11. Concurrent Execution Options • Skip Execution – Do not run flow if it is already running. •Run Concurrently – Run the flow anyway. Previous execution is unaffected. • Pipeline
- 12.
- 13.
- 14.
- 15. My use case • Use Azkaban to manage hadoop job –Write batch in python • Use Azkaban API – I created client https://github.com/wyukawa/eboshi – Commit scheduling information to GHE • Painful to write job file – I created generation tool https://github.com/wyukawa/ayd – generate 1 flow from 1 yaml file
- 16. Python batch example def validate_before(self): hive.exists("access_log", "yyyymmdd='%s'" % (...)) def process(self): insert_query= """ INSERT OVERWRITE TABLE aggregate PARTITION(yyyymmdd='%s') SELECT ... FROM access_log WHERE ... GROUP BY ... """ % (...) hiveCli.query(insert_query) def validate_after(self): hive.exists("aggregate", "yyyymmdd='%s'" % (...))
- 17.
- 18.
- 19.
- 20. My usage situation • More than 120 Azkaban flows • Many daily batches, a few hourly, weekly, monthly batches •Most flows are related to hive • There is the Azkaban in batch server • I prepare the template Azkaban flows to reaggregate past data – Set job name and date to parameter – Set Run Concurrently • I don’t use SLA but I may use in the future – https://github.com/azkaban/azkaban/pull/911 • I don’t use HTTP Job Callback – use hipchat in python ETL
- 21. My feeling • Simple • Easy to use •Web UI is convenient • API is useful • There is no reason to replace Azkaban • I hope development become active
- 22.