Apache Airflow is an open-source workflow management platform that was created at Airbnb in 2014 to author, schedule, and monitor complex workflows. It allows users to define workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler then executes the tasks on workers based on dependencies. Airflow is commonly used for ETL pipelines, data processing, machine learning workflows, and automating devops tasks like monitoring cron jobs. Companies like Robinhood and Google use Airflow for complex data workflows and as a managed service on Google Cloud.

1. Airflow 4 Manager
Alex Pongpech

2. What is Airflow?
● Apache Airflow is an open-source workflow
management platform. It started at Airbnb in October
2014 as a solution to manage the company's
increasingly complex workflows
● Airflow is a platform to programmatically author,
schedule and monitor workflows.
● Use airflow to author workflows as directed acyclic
graphs (DAGs) of tasks. The airflow scheduler executes
your tasks on an array of workers while following the
specified dependencies.
2

3. Why use Airflow?
● Have you ever managed messy data pipeline? In
real life, data pipeline can be pretty messy
● Using CLI interface
● Had you ever have to adjust your workflow? Surely
you want to be able to scale your workflow up and
down quickly and effectively
3

4. When not to use Airflow
A sampling of examples that Airflow can not satisfy in a first-class
way includes:
● DAGs which need to be run off-schedule or with no schedule
at all
● DAGs that run concurrently with the same start time
● DAGs with complicated branching logic
● DAGs with many fast tasks
● DAGs which rely on the exchange of data
● Parametrized DAGs
● Dynamic DAGs
4

5. What Airflow is used for in general?
● Monitoring Cron jobs
● transferring data from one place to other.
● Automating your DevOps operations.
● Periodically fetching data from websites and
update the database for your awesome price
comparison system.
● Data processing for recommendation based
systems.
● Machine Learning Pipelines.
5

6. So how others have used Airflow
6

7. Robinhood
● Managing dependencies between jobs was difficult.
With cron we would use worst-case expected durations
for upstream jobs to schedule downstream jobs.
● Failure handling and alerting had to be managed by the
job. We would have to rely on the job, or the on-call
engineer to handle retries and upstream failures in the
case of dependent jobs.
● Retrospection was difficult. We would need to sift
through logs or alerts to check how a job may have
performed on a certain day in the past.
7

8. Google
● In May 2018 Google announced Google Cloud
Composer, a managed Apache Airflow service that is
fully integrated in the Google Cloud platform and has
thus become one of the cornerstones for
orchestrating managed services in Google Cloud.
●
8

9. References
1. https://medium.com/analytics-and-data/10-benefits-to-using-airflow-33d312537bae
2. https://medium.com/analytics-and-data/on-the-evolution-of-data-engineering-c5e5
6d273e37
3. https://towardsdatascience.com/getting-started-with-apache-airflow-df1aa77d7b1b
4. https://en.paradigmadigital.com/dev/apache-airflow/
5. https://robinhood.engineering/why-robinhood-uses-airflow-aed13a9a90c8
6. https://towardsdatascience.com/building-a-production-level-etl-pipeline-platform-us
ing-apache-airflow-a4cf34203fbd
7. https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4
9