Presentation given at Coolblue B.V. demonstrating Apache Airflow (incubating), what we learned from the underlying design principles and how an implementation of these principles reduce the amount of ETL effort. Why choose Airflow? Because it makes your engineering life easier, more people can contribute to how data flows through the organization, so that you can spend more time applying your brain to more difficult problems like Machine Learning, Deep Learning and higher level analysis.

1. Gerard Toonstra | Sr. Data Platform Engineer | g.toonstra@coolblue.nl | 08-10-2017

3. “Seek more understanding
through data analytics
about business processes.”

5. Where is time spent?

6. Follow important ETL principles
● Processes are idempotent

7. Follow important ETL principles
● Processes are idempotent
● Reusable, parameterisable components

8. Follow important ETL principles
● Processes are idempotent
● Reusable, parameterisable components
● Data at rest between operations
s3://[bucket]/[dag-id]/[task-id]/2017/10/13

9. Follow important ETL principles
● Processes are idempotent
● Reusable, parameterisable components
● Data at rest between operations
s3://[my-bucket-name]/[dag-id]/[task-id]/2017/10/13
● Testing strategy

10. “Airflow is a platform to
programmatically author,
schedule and monitor
workflows.”

11. What is a scheduler?

12. What is a DAG?

13. Conceptual Architecture

14. Operator?

15. Apache Airflow (incubating)

17. Architecture on AWS

18. Architecture on AWS

19. Architecture on AWS

20. Architecture on AWS

27. extract_customer = PostgresToPostgresOperator(
src_postgres_conn_id='postgres_oltp',
dest_postgress_conn_id='postgres_dwh',
sql='select_customer.sql',
pg_table='staging.customer',
parameters={"window_start_date":"{{ ds }}",
"window_end_date":"{{ tomorrow_ds }}"},
pool='postgres_dwh')

28. extract_customer = PostgresToPostgresOperator(
src_postgres_conn_id='postgres_oltp',
dest_postgress_conn_id='postgres_dwh',
sql='select_customer.sql',
pg_table='staging.customer',
parameters={"window_start_date":"{{ ds }}",
"window_end_date":"{{ tomorrow_ds }}"},
pool='postgres_dwh')

29. extract_customer = PostgresToPostgresOperator(
src_postgres_conn_id='postgres_oltp',
dest_postgress_conn_id='postgres_dwh',
sql='select_customer.sql',
pg_table='staging.customer',
parameters={"window_start_date":"{{ ds }}",
"window_end_date":"{{ tomorrow_ds }}"},
pool='postgres_dwh')

30. extract_customer = PostgresToPostgresOperator(
src_postgres_conn_id='postgres_oltp',
dest_postgress_conn_id='postgres_dwh',
sql='select_customer.sql',
pg_table='staging.customer',
parameters={"window_start_date":"{{ ds }}",
"window_end_date":"{{ tomorrow_ds }}"},
pool='postgres_dwh')

31. Where is time spent?

32. Follow important ETL principles
● Idempotency
● Reusable, parameterizable components
● Data from rest to rest
● Testing strategy

34. Gerard Toonstra | Sr. Data Platform Engineer | g.toonstra@coolblue.nl