This document discusses Apache Airflow and Google Cloud Composer. It begins by providing background on Apache Airflow, including that it is an open source workflow engine contributed by Airbnb. It then discusses how Codementor uses Airflow for ETL pipelines and machine learning workflows. The document mainly focuses on comparing self-hosting Airflow versus using Google Cloud Composer. Cloud Composer reduces efforts around hosting, permissions management, and monitoring. However, it has some limitations like occasional zombie tasks and higher costs. Overall, Cloud Composer allows teams to focus more on data logic and performance versus infrastructure maintenance.

1. From Airflow to
Google Cloud Composer
Bruce Kuo

2. • Know the efforts on hosting Apache Airflow
• Know how Google Cloud Composer reduces hosting
efforts
• Share and discuss what we care about when using Cloud
Composer
KEY TAKEAWAYS
1
0
0 1
1 0

3. • Apache Incubator Project
• Contribute by Airbnb
• An open source workflow
engine
APACHE AIRFLOW
1
0
0 1
1 0

4. Dynamic
Workflow as code
Extensible
Customize your code easily
Elegant
Parameter your scripts is
built into the core of Airflow
with Jinja
Scalable
Easy to scale your machines
APACHE AIRFLOW
1
0
0 1
1 0

8. • ETL Pipeline
• Batch Data Transformation between DB and BigQuery
• Machine Learning Pipeline
• Training pipeline
• Deploy model pipeline
• Model predictions: community quality score, applicant quality score
AIRFLOW IN CODEMENTOR
1
0
0 1
1 0

9. Web Servers
WorkersScheduler
Task QueuesAirflow Database
AIRFLOW ARCHITECTURE
1
0
0 1
1 0

10. Web Servers
(AWS EC2 * 1)
Workers
(AWS EC2 * 1)
Scheduler
(AWS EC2 * 1)
Task Queues
(AWS EC2 * 1)
Airflow Database
(AWS EC2 * 1)
ENVIRONMENT IN CODEMENTOR
1
0
0 1
1 0

11. • Machine Managements
• Web Server
• Scheduler
• Worker
• DB
• Task Queue - Celery on Redis
HOST AIRFLOW
1
0
0 1
1 0

12. • Deployment - deploy to all machines when code changed
• DAG changes - Hot reload
• Airflow configuration changes (restart service)
• Package dependencies changes (restart service)
HOST AIRFLOW
1
0
0 1
1 0

13. • Permission management
• Use command airflow create_user
• Monitor Airflow
• DAG
• Environments
HOST AIRFLOW
1
0
0 1
1 0

14. • Announce on May, 2018
• Airflow as a Service
• Core: Airflow 1.9
• Support other versions in the future
• Native integration with GCP components
CLOUD COMPOSER
1
0
0 1
1 0

18. • How to deploy DAG
• Put DAGS on GCS folder: gs://project-bucket/dags
• How to deploy pypi packages
• Setting on GUI
• Execute command: gcloud beta composer environments update cm-airflow-prod
--update-pypi-packages-from-file gcp_requirements.txt --location us-east1
• How to deploy private packages
• Put packages on GCS folder: gs://project-bucket/plugins
DEPLOY ON COMPOSER
1
0
0 1
1 0

19. • Permission management
• Just setting IAM Roles on GCP
• Monitor cloud composer
• Webserver: uptime robot
• Scheduler & Worker:
• add a DAG to send a message to CloudWatch
• https://cloud.google.com/composer/docs/how-to/managing/monitoring-
environments
• No machine efforts!
PERMISSION & MONITOR
1
0
0 1
1 0

20. Self-host Airflow Google Cloud Composer
Machine
efforts
Manage cluster
status / scaling
No
Permission
Management
Setting by our selves in different
environments
(web / scheduler / worker)
Only setting IAM roles and
worker permission are closed
Monitoring
Uptime Robots (Web)
Monitor DAG (scheduler+worker)
Uptime Robots (Web)
Monitor DAG (scheduler+worker)
Deploy Configs
Deploy configs to all machines and
restart service
Setting on GUI or
Executing Command
COMPARISON
1
0
0 1
1 0

21. Self-host Airflow Google Cloud Composer
DAG Update Deploy DAGs to all machines Put DAGs to GCS
Deploy Python
Package
Deploy packages to all machines
and restart service
Setting on GUI or
Executing Command
Deploy Private
Package
Same as deploy python package Put private packages to GCS
Deploy Configs
Deploy configs to all machines and
restart service
Setting on GUI or
Executing Command
COMPARISON
1
0
0 1
1 0

22. Notes from Practice
Notes From Practice

23. • Use local path between workers
• Cloud composer sync gs://project-bucket/data on
local path /home/airflow/gcs
• Sync bidirectionally
• Details: https://cloud.google.com/composer/docs/
concepts/cloud-storage
TRICKS
1
0
0 1
1 0

24. • To make whole environment stable, here is our airflow
settings after tuning
TRICKS
1
0
0 1
1 0

25. 1. See DAG log on web console
INCIDENT SOP
1
0
0 1
1 0

26. 2. Stackdriver
INCIDENT SOP
1
0
0 1
1 0

27. 3. Trace Kubernetes workloads
INCIDENT SOP
1
0
0 1
1 0

28. • Rerun is hard, following is the SOP
1. Clear task instances
2. Clear failed jobs
• DB may be locked unfortunately if you don’t follow rerun
SOP QQ.
RERUN SOP
1
0
0 1
1 0

29. • Expensive: about 10-20 usd / day in our usage
• Sometimes there are zombie tasks
• The zombie task is caused by worker crash
CONCERNS
1
0
0 1
1 0

30. • More focus on
• DAG development
• Data logics
• Performance tuning
• Reduce many efforts on maintaining the service
CONCLUSION
1
0
0 1
1 0

31. • https://www.youtube.com/watch?v=gFQVmsRRY_A
• Discussion:
• https://groups.google.com/forum/#!forum/cloud-
composer-discuss
• https://stackoverflow.com/questions/tagged/google-
cloud-composer
REFERENCE
1
0
0 1
1 0

32. Notes from Practice
Q & A

33. codementor.io
WE’RE HIRING
CHECK OUT OUR CAREER PAGE
www.codementor.io/careers