Downloaded 353 times
![About Me
2
Work [ed | s] @
Maintainer on
Reports to
Co-Chair for](https://image.slidesharecdn.com/datadayseattle2016v4-160723233038/85/Introduction-to-Apache-Airflow-Data-Day-Seattle-2016-2-320.jpg)
Uploaded bySid Anand
Introduction to Apache Airflow - Data Day Seattle 2016
Apache Airflow is a platform for authoring, scheduling, and monitoring workflows or directed acyclic graphs (DAGs) of tasks. It includes a DAG scheduler, web UI, and CLI. Airflow allows users to author DAGs in Python without needing to bundle many XML files. The UI provides tree and Gantt chart views to monitor DAG runs over time. Airflow was accepted into the Apache Incubator in 2016 and has over 300 users from 40+ companies. Agari uses Airflow to orchestrate message scoring pipelines across AWS services like S3, Spark, SQS, and databases to enforce SLAs on correctness and timeliness. Areas for further improvement include security, APIs, execution scaling, and on
More Related Content
Similar to Introduction to Apache Airflow - Data Day Seattle 2016
Introduction to Apache Airflow - Data Day Seattle 2016
- 1. Introducing Apache Airflow (Incubating) SidAnand (@r39132) Data Day Seattle 2016 1
- 2. About Me 2 Work [ed| s] @ Maintainer on Reports to Co-Chair for
- 3.
- 4. 4 Apache Airflow :What is it? Airflow is a platform to programmatically author, schedule and monitor workflows (a.k.a. DAGs)
- 5. 5 Apache Airflow :What is it? Airflow is a platform to programmatically author, schedule and monitor workflows (a.k.a. DAGs) It ships with • DAG Scheduler • Web application (UI) • Powerful CLI
- 6. 6 Apache Airflow :What is it?
- 7. Airflow - AuthoringDAGs 7 Airflow: Visualizing a DAG
- 8. 8 Airflow: Author DAGsin Python! No need to bundle many XML files! Airflow - Authoring DAGs
- 9. 9 Airflow: The TreeView offers a view of DAG Runs over time! Airflow - Authoring DAGs
- 10. Airflow - PerformanceInsights 10 Airflow: Gantt charts reveal the slowest tasks for a run!
- 11. 11 Airflow: …And wecan easily see performance trends over time Airflow - Performance Insights
- 12. 12 Apache Airflow :What is it? When would you use a Workflow Scheduler like Airflow? • ETL Pipelines • Machine Learning Pipelines • Predictive Data Pipelines • Fraud Detection, Scoring/Ranking, Classification, Recommender System, etc… • General Job Scheduling (e.g. Cron) • DB Back-ups, Scheduled code/config deployment
- 13. 13 Apache Airflow :What is it? What should a Workflow Scheduler do well? • Schedule a graph of dependencies • where Workflow = A DAG of Tasks • Handle task failures • Report / Alert on failures • Monitor performance of tasks over time • Enforce SLAs • E.g. Alerting if time or correctness SLAs are not met • Scale
- 14. 14 Apache Airflow :What is it? What Does Apache Airflow Add? • Configuration-as-code • Usability - Stunning UI / UX • Centralized configuration • Resource Pooling • Extensibility
- 15.
- 16. 16 Apache Airflow :Incubating Timeline • Airflow was created @ Airbnb in 2015 by Maxime Beauchemin • Max launched it @ Hadoop Summit in Summer 2015 • On 3/31/2016, Airflow —> Apache Incubator Today • 166+ Contributors • 300+ Users • 40+ companies officially using it! • 9 Committers/Maintainers <— We’re growing here
- 17.
- 18. Agari : WhatWe Do 18
- 19. 19 Agari : WhatWe Do
- 20. 20 Agari : WhatWe Do
- 21. 21 Agari : WhatWe Do
- 22. 22 Agari : WhatWe Do
- 23. 23 Enterprise Customers email metadata apply trust models email md + trust score Agari’sCurrent Product Agari : What We Do
- 24. 24 email metadata apply trust models email md + trust score Agari’sFuture ProductEnterprise Customers Agari : What We Do
- 25. Apache Airflow @Agari How Do We Use It? 25
- 26. Classes of Orchestration 26 applytrust models (message scoring) build trust models cron++ (general job scheduler) New Product (Enterprise Protect) Operational Automation BI / ETL N / A
- 27. Classes of Orchestration 27 applytrust models (message scoring) build trust models cron++ (general job scheduler) New Product (Enterprise Protect) Operational Automation BI / ETL N / A This Talk
- 28. Use-Case : Message Scoring BatchPipeline Architecture 28
- 29. Use-Case : MessageScoring 29 enterprise A enterprise B enterprise C S3 S3 uploads every 15 minutes
- 30. Use-Case : MessageScoring 30 enterprise A enterprise B enterprise C S3 Airflow kicks of a Spark message scoring job every hour
- 31. Use-Case : MessageScoring 31 enterprise A enterprise B enterprise C S3 Spark job writes scored messages and stats to another S3 bucket S3
- 32. Use-Case : MessageScoring 32 enterprise A enterprise B enterprise C S3 This triggers SNS/SQS messages events S3 SNS SQS
- 33. Use-Case : MessageScoring 33 enterprise A enterprise B enterprise C S3 An Autoscale Group (ASG) of Importers spins up when it detects SQS messages S3 SNS SQS Importers ASG
- 34. 34 enterprise A enterprise B enterpriseC S3 The importers rapidly ingest scored messages and aggregate statistics into the DB S3 SNS SQS Importers ASG DB Use-Case : Message Scoring
- 35. 35 enterprise A enterprise B enterpriseC S3 Users receive alerts of untrusted emails & can review them in the web app S3 SNS SQS Importers ASG DB Use-Case : Message Scoring
- 36. 36 enterprise A enterprise B enterpriseC S3 S3 SNS SQS Importers ASG DB Airflow manages the entire process Use-Case : Message Scoring
- 37.
- 38.
- 39. 39 5 minute waitfor S3 eventual consistency Airflow DAG
- 40. 40 1 hour aday, we also build new models Airflow DAG
- 41.
- 42. 42 dummy needed for branchoperator Airflow DAG
- 43.
- 44.
- 45. 45 • Run Spark •Verify a record is written to the DB • Wait for the SQS queue to empty Airflow DAG
- 46. 46 • Compute discrepancies • Sendemail report • Update monitoring graphs • Raise SLA (correctness) alerts Airflow DAG
- 47.
- 48. 48 Desirable Qualities ofa Resilient Data Pipeline OperabilityCorrectness Timeliness Scalable/Available • Data Integrity (no loss, etc…) • Expected data distributions • All output within time-bound SLAs (e.g. 1 hour) • Fine-grained Monitoring & Alerting of Correctness & Timeliness SLAs • Quick Recoverability • ASGs, SQS, SNS, S3
- 49. 49 Desirable Qualities ofa Resilient Data Pipeline OperabilityCorrectness Timeliness • Data Integrity (no loss, etc…) • Expected data distributions • All output within time-bound SLAs (e.g. 1 hour) • Fine-grained Monitoring & Alerting of Correctness & Timeliness SLAs • Quick Recoverability SLA SLA • ASGs, SQS, SNS, S3 Scalable/Available
- 50. 50 Correctness : EmailReporting orgs
- 51. 51 Correctness : EmailReporting For each org, we check for duplicate or missing data as a count & percentage orgs
- 52. 52 Correctness : EmailReporting These are the 3 stages of the pipeline. We can detect where a discrepancy is coming from - often related to a code push! orgs
- 53.
- 54. 54 Airflow: …And easyto integrate with Ops tools! Correctness & Timeliness : Alerting
- 55. 55 Airflow: …And easyto integrate with Ops tools! Correctness & Timeliness : Alerting Timeliness SLA miss
- 56. 56 Airflow: …And easyto integrate with Ops tools! Correctness & Timeliness : Alerting Timeliness SLA miss dag = DAG(DAG_NAME, schedule_interval='@hourly', default_args=default_args, sla_miss_callback=sla_alert_func)
- 57. 57 Airflow: …And easyto integrate with Ops tools! Correctness & Timeliness : Alerting Timeliness SLA miss Correctness SLA miss
- 58. 58 Airflow: …And easyto integrate with Ops tools! Correctness & Timeliness : Alerting Timeliness & Correctness SLA misses sent to PagerDuty/VictorOps
- 59. Use-Case : ModelBuilding v2 For Both Batch & Near Realtime Scoring Pipelines 59
- 60.
- 61. 61 Model Building DAG Launchan EMR cluster
- 62. 62 Run model building as EMRsteps Model Building DAG
- 63. 63 Validate models Send email notification if testsfail Model Building DAG
- 64.
- 65. Apache Airflow NextSteps 65 Areas for Improvement
- 66. 66 Apache Airflow NextSteps Improvement Areas • Security • API (though we do have a CLI) • Deployment / Versioning • Execution Scale Out • On-demand Execution
- 67. Acknowledgments 67 • Vidur Apparao •Stephen Cattaneo • Jon Chase • Andrew Flury • William Forrester • Chris Haag • Mike Jones • Scot Kennedy • Thede Loder • Paul Lorence • Kevin Mandich • Gabriel Ortiz • Jacob Rideout • Josh Yang • Julian Mehnle None of this work would be possible without the contributions of the strong team below
- 68.