From AWS Data Pipeline to Airflow - managing data pipelines in Nielsen Marketing Cloud

Copyright © 2017 The Nielsen Company (US), LLC. Conﬁdential and proprietary. Do not distribute.
The Big
Web
Theory
The Big
Web
Theory

2
Copyright©2018TheNielsenCompany(US),LLC.Conﬁdentialandproprietary.Donotdistribute.
AIRFLOWAIRFLOW

3
● Tal Sharon - Software Architect
● Aviel Buskila - DevOps Tech Lead
● Max Peres - Data Engineer
Who we are?

4
What will you learn today ?
• Airﬂow and how it solved our problems
• How you can deploy Airﬂow to production
• Best practices for annoying data problems

5
Nielsen Marketing Cloud
● eXelate - acquired by Nielsen in 2015
● Marketing data cloud service
● Creating targeting proﬁles
● VERY BIG DATA
● Machine learning

6
Nielsen Marketing Cloud - main challenge
How many unique users of a certain profile can we
reach?
e.g. campaign for young women who love tech

7
What we had...

8
Data pipeline UI
Multiple workﬂow parameters
are missing - duration,
successes & failures, ..

9
And this is how we had to configure it ...
Not all
conﬁgurations
are visible
The Workﬂow
is hidden

10
We wanted something else..
● Configuration and workflow visibility
● Monitoring and statistics
● Share common configuration/code between our
workflows
● Ability to have only 1 concurrent execution of a
workflow

11
The Alternatives
Community Main Purpose Flow
Deﬁnition
UI Auto
scheduling
Smart
Scheduling
Airﬂow
(Apache)
Very Active General Purpose Python Rich V V
Luigi
(Spotify)
Active General Purpose Python Limited X X
Oozie
(Apache)
Active Hadoop Job
Scheduling
XML Limited V X
Azkaban
(LinkedIn)
Not very active Hadoop Job
Scheduling
Custom DSL Rich V X

12
What is Airflow ?
‘A platform to programmatically author, schedule, and monitor workﬂows’
Each workﬂow is described by a DAG(Directed Acyclic Graph) which is constructed by -
● Operators: determine what actually gets done
● Sensors: monitor the job and report success/failure

13
Airflow overview
Webserver
Accepts HTTP requests and allows the
user to interact with it. It provides the
ability to act on the DAG status (pause,
unpause, trigger).
Scheduler
Monitors the DAGs and periodically
inspects tasks to see if they can be
triggered.
Worker
Daemons that actually execute
the logic of tasks.

14
Airflow UI

15
Taking it to Production

16
Challenges
1. Code dependency management
2. Automated deployment from dev to prod
3. Long running tasks infrastructure
4. Scaling airﬂow workers
5. Deploy to kubernetes

17
Code Dependency management - DAG Structure
DAG Dependency
DAG Dependency
DAG Helpers
DAG File

18
Code Dependency management - DAG Build

19
Automated deployment from dev to prod
Package artifact with
a version and push
Deploy a DAG file
with a specific
version

20
Function as a service (FaaS)
FaaS is a category of cloud computing services that
provides a platform allowing customers to develop, run,
and manage application functionalities without the
complexity of building and maintaining the infrastructure
typically associated with developing and launching an app.
Source:
https://en.wikipedia.org/wiki/Function_as_a_service

21
OpenFaaS - Long running tasks infrastructure
OpenFaaS (Functions as a Service) is a framework for building Serverless
functions with Docker and Kubernetes which has ﬁrst-class support for
metrics. Any process can be packaged as a function enabling you to consume
a range of web events without repetitive boiler-plate coding.

22
Airflow and OpenFaaS
+ =

23
Open source donations

24
Scaling airflow workers
Airﬂow has 3 types of executors:
1. Sequential Executor - One task & one worker
2. Local Executor - Parallel tasks & one worker
3. Celery Executor - Parallel tasks & multiple workers

25
Celery executor setup

26
Deployment to kubernetes

27
So what did we achieve?
✓ Dependency management
✓ Automated deployment from dev to prod
✓ Long running tasks infrastructure
✓ Scalable airﬂow workers
✓ Deploy to kubernetes

28
Annoying
Problems
Annoying
Problems

29
Annoying Data Maintenance Problems
✓ Troubleshooting broken pipelines
✓ Rerunning parts of a pipeline

30
Classical Cron Scheduling

31
Troubleshooting broken pipelines

32
Understanding a Pipeline (DAG)

33
Sub Pipeline (sub-DAG)
Create_EMR_Cluster Create_EMR_Step Step_Sensor

34
Rerunning a Task
Create_EMR_Cluster Create_EMR_Step Step_Sensor
Failed Task

35
Cool Features

36
Scaling Out

37
Scaling Out

38
Message Exchange (Xcom)
Allows sharing data between tasks
(example: conﬁgs)

39
Sample DAG

40
Sample DAG

41

42
Summary
● Maintenance becomes a BRE E Z E …...
● Recovery is a NO BRAINER !
● Lots of cool & convenient features

43
● Amazing tool for data pipelines
● Open source community
● Cool UI & API
AIRFLOWAIRFLOW
Make you life EASIER !!!Make you life EASIER !!!

44
Questions?

45
Come and join us!
https://www.comeet.co/jobs/nielsen/33.000

46
Thank you!

47
BACKUP SLIDES

48
{
"Data-In-Dpu_us": {
"bucket": "xl8emr-assets",
"file": "data-in/config/dpu_airflow.json"
},
"Data-In-Dpu_eu": {
"bucket": "xl8emr-assets-eu",
},
"Data-In-Dpu_ap": {
"bucket": "xl8emr-assets-eu",
}
}

From AWS Data Pipeline to Airflow - managing data pipelines in Nielsen Marketing Cloud

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to From AWS Data Pipeline to Airflow - managing data pipelines in Nielsen Marketing Cloud

Similar to From AWS Data Pipeline to Airflow - managing data pipelines in Nielsen Marketing Cloud (20)

More from Itai Yaffe

More from Itai Yaffe (20)

Recently uploaded

Recently uploaded (20)

From AWS Data Pipeline to Airflow - managing data pipelines in Nielsen Marketing Cloud