Apache Airflow is a platform for authoring, scheduling, and monitoring workflows or directed acyclic graphs (DAGs). It allows defining and monitoring cron jobs, automating DevOps tasks, moving data periodically, and building machine learning pipelines. Many large companies use Airflow for tasks like data ingestion, analytics automation, and machine learning workflows. The author proposes using Airflow to manage data movement and automate tasks for their organization to benefit business units. Instructions are provided on installing Airflow using pip, Docker, or Helm along with developing sample DAGs connecting to Azure services like Blob Storage, Cosmos DB, and Databricks.

1. Apache Airflow
Liangjun Jiang
06/07/2019

2. Outlines
What’s Apache airflow &
what they can do
Install Apache airflow
Working Process Sample DAG with Azure

3. What’s Apache Airflow
A platform to programmatically author, scheduler and
monitor workflows, has gained its popularity in the
world of data engineering and machine learning. Its
use cases can be
• Define and monitor Cron jobs
• Automate certain DevOps operation
• Move data periodically
• Machine learning pipeline

4. Who is using
• Airbnb
• Lyft
• Bloomberg
• Jets.com
• Sam’s Club
• a long list ...

5. Why I think it is useful for us?
• Need a way to manage data movement in and out
• Reliable and consistent way to schedule and monitor moving data
tasks
• Data Ingestion automation for hubble
• Machine learning pipelines automation
• For the benefit of other business units, or small groups by developing
and deploying as an part of infrastructure

6. Install Airflow
• 1. `pip install airflow`
• 2. User *unofficial* airflow Docker Image
docker pull puckel/docker-airflow
• 3. Use Helm/stable/airflow
helm install --namespace "airflow" --name "airflow" stable/airflow

7. Install Airflow (cont’d)
I create an fork of puckel/docker-airflow and helm/stable/airlfow with the following
features:
1. Role based authentication
2. Third party python packages to support Azure, GCP and Kubernetes Operators
3. Mount DAGs with Azure File for App Service and AKS deployment
4. Ingress Controller for TLS termination for AKS deployment
5. Sample Airflow Connection with Azure Blob Storage, Cosmos Db, Azure
Databricks
6. Sample DAGs with Azure Cosmos db , blob storage and Cosmos, etc
See this link for the details

8. Working Process
• Use your local Docker image for DAG
development
docker-compose -f docker-compose-
CeleryExecutor.yml up –d
• Pull request for production DAG
• This dags folder is git managed

9. Sample DAG
with Azure

10. Sample DAG
with Azure

11. Sample DAG with Azure – web UI
• http://localhost:8080

12. Final Thoughts
• Need to work with cloud team to deploy this as secure and trusted
app service