Gerard Toonstra, a Senior Data Platform Engineer, emphasizes the importance of understanding business processes through data analytics and adhering to key ETL principles such as idempotency and reusable components. He discusses the use of Apache Airflow for managing workflows, including specific SQL operations to extract customer data from a Postgres database. The document outlines various architectural considerations for implementing these strategies on AWS.

1. Gerard Toonstra | Sr. Data Platform Engineer | g.toonstra@coolblue.nl | 08-10-2017

3. “Seek more understanding
through data analytics
about business processes.”

5. Where is time spent?

6. Follow important ETL principles
● Processes are idempotent

7. Follow important ETL principles
● Processes are idempotent
● Reusable, parameterisable components

8. Follow important ETL principles
● Processes are idempotent
● Reusable, parameterisable components
● Data at rest between operations
s3://[bucket]/[dag-id]/[task-id]/2017/10/13

9. Follow important ETL principles
● Processes are idempotent
● Reusable, parameterisable components
● Data at rest between operations
s3://[my-bucket-name]/[dag-id]/[task-id]/2017/10/13
● Testing strategy

10. “Airflow is a platform to
programmatically author,
schedule and monitor
workflows.”

11. What is a scheduler?

12. What is a DAG?

13. Conceptual Architecture

14. Operator?

15. Apache Airflow (incubating)

17. Architecture on AWS

18. Architecture on AWS

19. Architecture on AWS

20. Architecture on AWS

27. extract_customer = PostgresToPostgresOperator(
src_postgres_conn_id='postgres_oltp',
dest_postgress_conn_id='postgres_dwh',
sql='select_customer.sql',
pg_table='staging.customer',
parameters={"window_start_date":"{{ ds }}",
"window_end_date":"{{ tomorrow_ds }}"},
pool='postgres_dwh')

28. extract_customer = PostgresToPostgresOperator(
src_postgres_conn_id='postgres_oltp',
dest_postgress_conn_id='postgres_dwh',
sql='select_customer.sql',
pg_table='staging.customer',
parameters={"window_start_date":"{{ ds }}",
"window_end_date":"{{ tomorrow_ds }}"},
pool='postgres_dwh')

29. extract_customer = PostgresToPostgresOperator(
src_postgres_conn_id='postgres_oltp',
dest_postgress_conn_id='postgres_dwh',
sql='select_customer.sql',
pg_table='staging.customer',
parameters={"window_start_date":"{{ ds }}",
"window_end_date":"{{ tomorrow_ds }}"},
pool='postgres_dwh')

30. extract_customer = PostgresToPostgresOperator(
src_postgres_conn_id='postgres_oltp',
dest_postgress_conn_id='postgres_dwh',
sql='select_customer.sql',
pg_table='staging.customer',
parameters={"window_start_date":"{{ ds }}",
"window_end_date":"{{ tomorrow_ds }}"},
pool='postgres_dwh')

31. Where is time spent?

32. Follow important ETL principles
● Idempotency
● Reusable, parameterizable components
● Data from rest to rest
● Testing strategy

34. Gerard Toonstra | Sr. Data Platform Engineer | g.toonstra@coolblue.nl