Tao Feng gave a presentation on Airflow at Lyft. Some key points: 1) Lyft uses Apache Airflow for ETL workflows with over 600 DAGs and 800 DAG runs daily across three AWS Auto Scaling Groups of worker nodes. 2) Lyft has customized Airflow with additional UI links, DAG dependency graphs, and integration with internal tools. 3) Lyft is working to improve the backfill experience, support DAG-level access controls, and explore running Airflow with Kubernetes executors. 4) Tao discussed challenges like daylight saving time issues and long-running tasks occupying slots, and thanked other Lyft engineers contributing to Airflow.

1. April 10th 2019
Tao Feng | @feng-tao | Software Engineer, Lyft
Airflow @ Lyft

2. 2
Who
● Tao Feng
● Engineer at Lyft Data Platform
● Apache Airflow PMC and Committer
● Working on different data products (Airflow,
Amundsen, etc)

3. Agenda
• Airflow in general
• Airflow @ Lyft
• Upstream @ Lyft
• Next Step
• Summary
3

4. Airflow in general
4

5. Airflow in general
5
• Airflow just became an Apache top level project(TLP) recently.
‒ Total 20 PMCs / committers
• Most recent release 1.10, 1.10.1, 1.10.2 (1.10.3 is coming).
‒ New Features: Airflow RBAC, Airflow K8S integration, etc
• New Process in Airflow for proposing architecture change.
‒ Airflow Improvements Proposals (currently 19+ proposals)
• Recent community conducted Airflow user survey (link).
11k+
github
stars
740+
contributors
250+
Companies
using

6. Airflow @ Lyft
6

7. 7
Core Infra high level architecture @ Lyft

8. Airflow Architecture @ Lyft
8

9. Airflow Architecture @ Lyft
• WebUI: the portal for users to view the related status of the DAGs.
• Metadata DB: the Airflow metastore for storing various job status.
• Scheduler: a multi-process which parses the DAG bag, creates a DAG object and
triggers executor to execute those dependency met tasks.
• Executor: A message queuing process that orchestrates worker processes to execute
tasks. We uses CeleryExecutor at Lyft.
• TARS: Airflow development / backfill environment, which provides access to production
data. 9

10. Airflow Architecture @ Lyft
10
• Main Cluster Config: Apache Airflow 1.8.2 with cherry-picks and numerous in-
house Lyft patches.
• Scale: Three set of ASGs for workers.
‒ ASG #1: 15 worker nodes each of which is the r5.4xlarge (16vcpu, 128g mem) type. This fleet of
workers is for processing low-priority memory intensive tasks.
‒ ASG #2: 3 worker nodes each of which is the m4.4xlarge (16vcpu, 64g mem) )type. This fleet of
workers is dedicated for those DAGs with a strict SLA.
‒ ASG #3: 1 worker node with m4.10xlarge (40vcpu, 160g mem) type. The single node is used to
process the compute-intensive workloads from a critical team’s DAGs.
‒ Backfill Box(TARS): 1 node with m4.16xlarge (64vcpu, 256g mem) )type. This box is used for fast
DAG prototyping and backfill.

11. Airflow daily stats @ Lyft
11
600+
DAGs
800+
DagRuns
25k+
TIs

12. Airflow Monitoring @
Lyft
12

13. Airflow Availability
• Scheduler and worker health check
‒ Use Canary monitoring DAG.
‒ No task has been scheduled for 10 mins is considered downtime.
• UI health check
‒ Leverage Envoy membership health check.
• Total system Uptime pct
‒ Airflow is down if either scheduler, workers, or web server is down.
13

14. Schedule Delay
• scheduler delay = TI.start_time - TI.execution_date
14

15. DAG last run time
• The time that have elapsed since the DAG file was last processed.
• If the time becomes too long, it means scheduler has issues processing the
DAG files.
‒ E.g could due to parser threads occupied by malicious DAG files.
15

16. Executor parallelism
• Parallelism: control the #. concurrent running tasks.
‒ Please monitor your worker nodes’ cpu utilization before increasing the value.
16

17. Airflow monitoring @ Lyft
17
Stats Name Meaning
dag_processing.last_run.seconds_ago.<d
ag_file>
Seconds since <dag_file> was last
processed
executor.open_slots Number of of open slots on executor
(parallelism - # running task)
executor.queued_tasks Number of queued tasks on executor
executor.running_tasks Number of running tasks on executor
pool.starving_tasks.<pool_name> Number of starving tasks in the pool.
Check how many tasks are starving due to
pool count limitation.
…...

18. Airflow Customization
@ Lyft
18

19. Airflow customization @ Lyft
• UI auditing
• Extra link for task instance UI panel (AIRFLOW-161)
19

20. Airflow customization @ Lyft
• DAG dependency graph
20

21. Improve Airflow
Reliability @ Lyft
21

22. Improving Airflow Performance @ Lyft
• Reduce Airflow UI page load time
‒ Change default_dag_run_display_number to 5.
• Tunables that impacts tasks execution parallelisms
‒ Parallelism
‒ Concurrency
‒ Max_active_runs
‒ Pool
22

23. Improving Airflow Reliability at Lyft
• Source Control For Pools
‒ All Airflow pools are defined in a source controlled github source file.
‒ Airflow pools are configured based on the file in runtime.
• Integration tests for DAG to enforce best practice and improve reliability
‒ All the DAGs should be loadable within time threshold.
‒ All the DAGs should have valid pools associated.
‒ External task sensors should be valid((dag_id, task_id) exists).
‒ Each pool is used by at least by one DAG.
‒ The sensor has a reasonable timeout.
‒ Each DAG has a non dynamic start date.
• Secure UI access
23

24. Production Debug @
Lyft
24

25. Production Debug @ Lyft
• We document every production issue investigation in the doc.
• Couples of methodologies:
‒ View the centralized Airflow dashboard.
‒ Identify whether it is UI or Airflow scheduler(backend) issues.
‒ View the webserver log or scheduler log.
∙ If the log is not available in machine, check the log in kibana.
∙ To further identify issues, we sometimes even look at logs in S3
‒ Use different tools for further investigation
∙ If exceptions is thrown, understand which part of Airflow code throws the exception.
∙ If CPU / memory alarm, use top to identify which DAG causes the issue.
∙ If failure related to celery, login to celery flower UI to further investigate.
∙ ...
25

26. Airflow Gotchas @ Lyft
26

27. Airflow Gotchas at Lyft
• DST
‒ UI doesn’t have timezone support even in upstream.
‒ Scheduler internal version has no timezone support.
• DAGs with dynamic start date.
‒ Hard to predict when the DAG is scheduled
• Long running external task sensors that don’t have valid external tasks.
• HivePartitionSensor doesn’t work for partial partition
‒ It only checks whether data exists, not check whether data fully loaded.
• Backfill experience
‒ We use local executor to backfill.
• Long running sensor occupies task slot of the pool
• User confused with DAG level argument vs Task level argument
‒ E.g Put max_active_run in default task argument
• Legacy high abstraction framework over Airflow
‒ Hard to debug for the user and us. 27

28. Upstream @ Lyft
28

29. Improve backfill
experience
29

30. Improve backfill experience
30
• New options for backfill
‒ --reset_dagruns: if used, Airflow will first check if there are any existing dag_runs /
task_instances associated with the backfill date range. If yes, it will prompt user whether the
user wants to clear those task_instances first. (AIRFLOW-2718)
‒ --rerun_failed_tasks: if used, Airflow automatically try to rerun those failed tasks again
without requiring any user intervention. (AIRFLOW-2566)
• Backfill respects pool for isolation (AIRFLOW-1557)

31. Improve backfill experience
Support batch backfill
• Use {{ prev_ds }} and {{ ds }} in SQL
‒ Prev_ds equals to ds -
schedule_interval
‒ User could change the
schedule_interval in the DAG file
during backfill.
• Use could override dag param with -c
options during backfill.
31
INSERT OVERWRITE TABLE {{
dest_db(default.superhero_data) }}
SELECT supe.superhero_name AS superhero_name,
pop.popularity AS popularity
FROM
{{ source_table(events.superheroes) }} supe
WHERE {{ prev_ds }} >= ds AND ds < {{ ds }}
airflow backfill superheroes -s 2018-05-01 -e
2018-05-08 -c {‘hive_cluster’:
‘backfill_cluster’}

32. Airflow DAG level
access
33

33. Airflow DAG level access @ Lyft
34
• DAG access control has always been a real need at Lyft
‒ HR data, Financial data, etc
‒ The workaround is to build an isolated dedicated cluster for each use case.
• Airflow introduces the RBAC feature in 1.10
‒ Airflow new webserver is based on Flask-Appbuilder.
‒ Ships with 5 static roles(Admin, User, Op, Viewer, Public).
‒ ...
• Airflow DAG level access (AIRFLOW-2267)
‒ Provides additional granular access control on DAG level.

34. Airflow DAG level access @ Lyft
• New Airflow UI migrates from Flask-Admin to Flask-Appbuilder(FAB).
• FAB’s security model.
35

35. Airflow DAG level access @ Lyft
• Which Airflow includes the change?
‒ 1.10.2 includes initial implementation
‒ 1.10.3(upcoming) includes the enhancements
• How it works
‒ Two new perms: can_dag_read (read), can_dag_edit (write).
‒ DAG level role could be created through cli / UI by Admin (doc).
‒ DAG level role could only see the viewable DAGs.
‒ User could declare permissions in DAG file (AIRFLOW-2694).
36

36. Airflow DAG level access @ Lyft
37
• We build a new cluster based on
Airflow master branch and
onboard couples of new sensitive
data use cases.
‒ Each use case has its own repo.
‒ User role relationship source
controlled in a YAML file.
• DAG owners specify the access
control info in the DAG files.
• Gotchas
‒ New user onboarding
‒ Integration between FAB and
google authentication(OAUTH)
‒ Integration with internal ACL
service
‒ ...
User registration flow

37. Next Step
38

38. Next Step
• Support Airflow DAG level access feature in beta internally.
• Integrate Airflow RBAC / DAG level feature with internal ACL service(FAB issue).
• Migrate all the existing DAGs to this new cluster.
• Explore running Airflow with k8s executor internally.
39

39. Summary
40

40. Summary
41
• Airflow community has been growing a lot!
• We share our experience on operating Airflow at Lyft.
• We share some of our upstream work
‒ Improve Airflow backfill experience
‒ Support Airflow DAG level Access

41. Acknowledgement
42
• Members who maintain Airflow at Lyft
‒ Alagappan Sethuraman
‒ Andrew Stahlman
‒ Chao-han Tsai
‒ Jinhyuk Chang
‒ Junda Yang
‒ Max Payton
‒ Tao Feng
• Special thanks to Maxime Beauchemin who provides numerous suggestions for
us.

42. Tao Feng | @feng-tao
Slides at TBD
Blog at go.lyft.com/airflowblog
Icons under Creative Commons License from https://thenounproject.com/ 43

43. Backup
44