Підсумує по огляду, тобто, що це, які вони є, юзкейси і чому не треба придумувати власний, а чому треба.

1. Что нам стоіт таск манагер построіт

2. • > 7 years in software development
• involved in scientific research since 2008
• hate front-end development
• called ”data engineer” last 3 years
• but was doing it much longer time
• who always look for solutions at stackoverflow
• have a lot of stories to tell you

3. And if you wonder it’s not my leg

4. Agenda
• Task managers. Do we need them?
• Review of most popular of them
• Components of task managers. How to
built it

5. My first day

6. My first month

7. My first year

8. One time I’ve
discovered…

9. Manager
comes and…

10. Try to run cron
every 72min

11. • Smallest resolution 1 minute
• Error handling
• Logging
• Cron pull you out of the application
Couple more
problems

12. And someone
says

13. After couple
years your life…

14. And your life

15. Why?

16. Random
images

17. Serious part

18. Common components

19. common
components
Client
Worker
Scheduler
Queue

20. Airflow

21. Brief about
• Developed at AirBnB
• Open-sourced June 2015
•~5,347 commits, 95 releases, 569 contributors
•In Apache Incubator starting 2016/05
•https://github.com/apache/incubator-airflow

22. DAG’s!

23. DAG’s!
• Workflow is (python) code
• Specify tasks & dependencies
programmatically
• Manages workflow metadata
• Nice GUI

24. Gimme code!

25. Architecture

26. Basic concepts
• DAG: graph of operator usages (=tasks)
• Operator: "Transformation" step
• Sensor: Operator which polls with
frequency / timeout (e.g. LocalFileSensor)
• Executor: Trigger operation (e.g.
HiveOperator, BashOperator, PigOperator)
• Task: Usage of Operator in DAG
• Task Instance: run of a Task at a point in time

27. More concepts
• XCom: send „messages“ between tasks
• Trigger Rules: specify handling for
multiple upstream dependencies
(e.g. all_success, one_success, ..)
• Variables: define key/value mappings
in airflow metadata DB (value can be
nested JSON as well)
• Branching: define python function
to choose which downstream path
to follow
• SubDAGs: encapsulate repeating
functionality

28. Luigi

29. Brief about
• Developed at Spotify
• Open-sourced 2014
•~3,677 commits, 40 releases, 366 contributors
•https://github.com/spotify/luigi

30. Also DAG’s
(dependency a…)

31. DAG’s! • Boring, the same as in Airflow

32. Gimme code!

33. Architecture

34. Basic concepts
• Workflow: graph of task and target usages
• Task: "Transformation" step
• Target: I/O step for task (tells task what
storage to read and how)
• Parameter: The same as task, but mostly
used for parametrization
• Dependencies: specific code style of
expressing dependencies between tasks
(enums, recursion, date algebra etc)

35. Different then
airflow

36. How to scale?

37. Tanks!