Download as PDF, PPTX
Uploaded byCloudera, Inc.
Building Robust Pipelines with Airflow | Wrangle Conference 2017
The document discusses the use of Airflow at Zymergen to build robust experimental pipelines for improving microbial strains through genetic engineering. It highlights challenges such as process failures and batch effects, and emphasizes the importance of a flexible and modular ETL pipeline for handling heterogeneous data sources and complex processing tasks. Additionally, it outlines how Airflow facilitates the construction of machine learning workflows and the monitoring of experimental bias.
More Related Content
Similar to Building Robust Pipelines with Airflow | Wrangle Conference 2017
More from Cloudera, Inc.
Building Robust Pipelines with Airflow | Wrangle Conference 2017
- 1. @erinshellman Wrangle Conf July 20th,2017 Building Robust Pipelines with Airflow
- 2. Zymology: is thescience of fermentation and it’s applied to make materials and molecules ! " # $ Beer Insulin Food additives Plastics
- 4. Zymergen provides a platformfor rapid improvement of microbial strains through genetic engineering.
- 5. Robotic automation Our experimentation isincreasingly orchestrated with robotics and machine learning.
- 6. Learning how toefficiently navigate the genome is the mission of data science at Zymergen
- 7. Blocker: process failure Orchestratingcomplex experiments with robots is hard, and there are process failures. These failures often cause sporadic, extreme measurement values.
- 8. Blocker: batch effects Wesee temporal effects based on when experiments were performed
- 9. Blocker: different interpretations ofresults We’re building a platform that can support any microbe and any molecule. Sometimes that results in a proliferation of solutions with disagreement on which is best.
- 10. Processing pipeline 1.Identify processfailures 2.Quantify and remove process- related bias 3.Identify strains that show improvement using consistent criteria Clean model inputs Outlier detection Normalization Hit detection
- 11. Rolling our ownETL pipeline There are many ways to measure the concentration of a molecule. Any microbe, any molecule… any experiment, many data formats.
- 12. Describing complex processing dependencies is hard. Rollingour own ETL pipeline
- 13. Airflow https://airflow.incubator.apache.org/ “Airflow is aplatform to programmatically author, schedule and monitor workflows.” Airflow gives us flexibility to apply a common set of processing steps to variable data inputs, schedule complex processing workflows, and has become a delivery mechanism for our products.
- 14.
- 15. e.g. Normalization Airflow workflowsare described as directed acyclic graphs (DAGs). Each task node in the DAG is an operator.
- 16. The anatomy ofa DAG Custom operators Ordering Instantiate DAG
- 17.
- 18. Airflow + PyStan WithBayesian hierarchical models we estimate (and monitor) the distribution of batch effects. Experimental bias
- 19. DropBox • Scientists atZymergen work with data using many different tools including JMP, SQL, and Excel. • We use a custom DropBox hook to make quick data ingestion pipelines.
- 20.
- 21. 3rd-party hooks &operators
- 22.
- 23. Pairs well withSuperset! “Apache Superset is a modern, enterprise-ready business intelligence web application” https://github.com/apache/incubator-superset
- 24.
- 25. Fairflow: Functional Airflow •The core of Fairflow is an abstract base class foperator that takes care of instantiating your Airflow operators and setting their dependencies. • In Fairflow, DAGs are constructed from foperators that create the upstream operators when the final foperator is called.
- 26. Configuring complex ML workflows…functionally
- 27. Defining ML workflows Inthe DAG definition, create an instance of the task. Then, instantiate a DAG like usual and call the compare task on the DAG.
- 28. Defining ML workflows Thedesign allows for simple creation of complicated experimental workflows with arbitrary sets of models, parameters, and evaluation metrics.
- 29. Is Airflow foryou? Do you have heterogeneous data sources? Do you have complex dependencies between processing tasks? Do you have data with different velocities? Do you have constraints on your time? Probably!
- 30.