Certaines fonctionnalités au cœur du produit de Datadog reposent sur des pipelines de données construits avec Spark qui traitent des milliers de milliards de points chaque jour. Dans cette présentation, nous verrons les grands principes que nous appliquons chez Datadog pour assurer que nos pipelines restent fiables malgré la croissance exponentielle du volume de données, les pannes matérielles, les données corrompues et les erreurs humaines. Paris Data Eng' Meetup du 26 février 2019 @Datadog

1. Building highly reliable data
pipelines @ Datadog
Quentin FRANCOIS
Team Lead, Data Engineering
1
Data Engineers Meetup Paris
26 Février 2019

2. 2

3. 3

4. 4

5. 5

6. Building highly reliable data
pipelines @ Datadog
6
Quentin FRANCOIS
Team Lead, Data Engineering
Data Engineers Meetup Paris
26 Février 2019

7. Reliability is the probability that a system will
produce correct outputs up to some given time t.
Source: E.J. McClusky & S. Mitra (2004). "Fault Tolerance" in Computer Science Handbook 2ed. ed. A.B. Tucker. CRC Press.
7

8. 1. Architecture
Highly reliable data pipelines
8

9. 1. Architecture
2. Monitoring
Highly reliable data pipelines
9

10. 1. Architecture
2. Monitoring
3. Failures handling
Highly reliable data pipelines
10

11. Historical metric queries
Time series data
metric system.load.1
timestamp 1526382440
value 0.92
tags host:i-xyz,env:dev,...
11

12. Historical metric queries
12
1 point/second

13. Historical metric queries
13
1 point/day

14. Historical metric queries
14
High
resolution
data
Low
resolution
data
1pt /min
1pt /hour
1pt /day
1pt /sec
AWS S3
• Runs once a day.
• Dozens of TBs of input data.
• Trillions of points processed.
Rollups
pipeline

15. 1. Architecture
2. Monitoring
3. Failures handling
Highly reliable data pipelines
15

16. Our big data platform architecture
CLUSTERS
DATA
WORKERS
USERS
Luigi Spark
Datadog
monitoring
S3
EMR
Web SchedulerCLI
16
EMR EMR EMR

17. Our big data platform architecture
CLUSTERS
DATA
WORKERS
USERS
Luigi Spark
Datadog
monitoring
S3
EMR
Web SchedulerCLI
17
EMR EMR EMR

18. Many ephemeral clusters
• New cluster for every pipeline.
• Dozens of clusters at a time.
• Median lifetime of ~3 hours.
18

19. Total isolation
We know what is happening and why.
19

20. Pick the best hardware for each job
For CPU-bound jobs
c3
For memory-bound jobs
r3
20

21. Scale up/down clusters
• If we are behind.
• Scale as we grow.
• No more waiting on loaded clusters.
21

22. Safer upgrades of EMR/Hadoop/Spark
5.13 5.12
5.125.125.12
22

23. Spot-instance clusters
Ridiculous savings
(up to 80% off the on-demand price)
Nodes can die at any time
23
-
+

24. Spot-instance clusters
Ridiculous savings
(up to 80% off the on-demand price)
Nodes can die at any time
24
-
+

25. How can we build highly reliable data pipelines
with instances killed randomly all the time?
25
?

26. No long running jobs
• The longer the job, the more work you lose on average.
• The longer the job, the longer it takes to recover.
26

27. No long running jobs
0 9
27
Pipeline A
Pipeline B
Time
(hours)

28. No long running jobs
Pipeline A
Pipeline B
0
Time
(hours)97 10 16
Job failure
28

29. Break down jobs into smaller pieces
Vertically - persist intermediate data between transformations.
Horizontally - partition the input data.
29

30. Example
30
Input
data
Output
data
Rollups pipeline
Aggregated time
series data
(custom file format)
Raw time series
data

31. Example
31
Input
data
Output
data
Rollups pipeline Raw time series
data
Aggregated time
series data
(custom file format)
1. Aggregate high
resolution data.
2. Store the
aggregated data
in our custom file
format.
1
2

32. Example
32
Output
data
Checkpoint
data
Input
data
Vertical split
Aggregated time
series data
(Parquet format)
Raw time series
data
Aggregated time
series data
(custom file format)
1
2
1. Aggregate high
resolution data.
2. Store the
aggregated data
in our custom file
format.
1
2

33. Example
33
Output
data
Checkpoint
data
Horizontal split
Aggregated time
series data
(Parquet format)
Raw time series
data
Aggregated time
series data
(custom file format)
1
2
A B
C D
1. Aggregate high
resolution data.
2. Store the
aggregated data
in our custom file
format.
1
2

34. Example
34
A
A
Horizontal split
B
B
C
C
D
D
Aggregated time
series data
(Parquet format)
Raw time series
data
Aggregated time
series data
(custom file format)
1
2
1. Aggregate high
resolution data.
2. Store the
aggregated data
in our custom file
format.
1
2

35. Break down jobs into smaller pieces
35
Fault tolerance
Performance

36. Lessons
• Many clusters for better isolation.
• Break down jobs into smaller pieces (no longer than ~3 hours).
• Trade-off between performance and fault tolerance.
36

37. 1. Architecture
2. Monitoring
3. Failures handling
Highly reliable data pipelines
37

38. Reliability is the probability that a system will
produce correct outputs up to some given time t.
38

39. Reliability is the probability that a system will
produce correct outputs up to some given time t.
39

40. Monitoring data pipelines
40
We monitor actively 3 types of metrics:
• Data lags metrics.
• Cluster health metrics.
• Job health metrics.
1. Is the data pipeline going to finish before the deadline?

41. Monitoring data pipelines
41
We monitor actively 3 types of metrics:
• Data lags metrics.
• Cluster health metrics.
• Job health metrics.
1. Is the data pipeline going to finish before the deadline?

42. Monitoring data pipelines
42
1. Is the data pipeline going to finish before the deadline?

43. Monitoring data pipelines
43
We monitor actively 3 types of metrics:
• Data lags metrics.
• Cluster health metrics.
• Job health metrics.
1. Is the data pipeline going to finish before the deadline?

44. Monitoring data pipelines
44
1. Is the data pipeline going to finish before the deadline?

45. Monitoring data pipelines
45
1. Is the data pipeline going to finish before the deadline?

46. Monitoring data pipelines
46
We monitor actively 3 types of metrics:
• Data lags metrics.
• Cluster health metrics.
• Job health metrics.
1. Is the data pipeline going to finish before the deadline?

47. Monitoring data pipelines
47
1. Is the data pipeline going to finish before the deadline?

48. Monitoring data pipelines
48
1. Is the data pipeline going to finish before the deadline?

49. Monitoring data pipelines
49
1. Is the data pipeline going to finish before the deadline?

50. Monitoring data pipelines
50
● Add custom counters throughout the pipelines.
○ Count records.
○ Count duplicates.
○ Count records that can’t join.
● Ad hoc checks on the output data.
2. Is the data produced correct?

51. Lessons
• Monitoring = will we finish before t? + is the data correct?
• Measure, measure and measure!
• Alert on meaningful and actionable metrics.
51

52. 1. Architecture
2. Monitoring
3. Failures handling
Highly reliable data pipelines
52

53. 53

54. Data pipelines will break
Hardware
failures
Bad code
changes
54
Upstream
delays
Increasing
volume of
data

55. Data pipelines will break
1. Recover fast
We want to fix the issues ASAP.
2. Degrade gracefully
We want to limit the customer-facing impact.
55

56. Recover fast
• No long running job.
• Switch from spot to on-demand clusters.
• Increase cluster size.
• Easy ways to rerun jobs (not always trivial!).
56

57. Example: rerun the rollups pipeline
2018-01
2018-02
2018-03
2018-04
2018-05
s3://bucket/
57

58. Example: rerun the rollups pipeline
as-of_2018-05-01
as-of_2018-05-02
...
as-of_2018-05-21
s3://bucket/2018-05/
58

59. Example: rerun the rollups pipeline
as-of_2018-05-01
as-of_2018-05-02
...
as-of_2018-05-21
s3://bucket/2018-05/
Active location
59

60. Example: rerun the rollups pipeline
as-of_2018-05-01
as-of_2018-05-02
...
as-of_2018-05-21
as-of_2018-05-22
s3://bucket/2018-05/
Active location
60

61. Example: rerun the rollups pipeline
as-of_2018-05-01
as-of_2018-05-02
...
as-of_2018-05-21
as-of_2018-05-22
s3://bucket/2018-05/
Active location
61

62. Example: rerun the rollups pipeline
as-of_2018-05-01
as-of_2018-05-02
...
as-of_2018-05-21
as-of_2018-05-22
s3://bucket/2018-05/
Active location
62

63. Example: rerun the rollups pipeline
as-of_2018-05-01
as-of_2018-05-02
...
as-of_2018-05-21
as-of_2018-05-22
as-of_2018-05-22_run-2
s3://bucket/2018-05/
Active location
63

64. Degrade
64
A
A
B
B
C
C
D
D
gracefully
• Isolate issues to a limited
number of customers thanks
to horizontal sharding.
• Keep the functionalities
operational at the cost of
performance/accuracy.

65. Lessons
• Think about potential issues ahead of time.
• Have knobs ready to recover fast.
• Have knobs ready to limit the customer facing impact.
65

66. Conclusion
66
Building highly reliable data pipelines

67. Conclusion
• Know your time constraints.
67
Building highly reliable data pipelines

68. Conclusion
• Know your time constraints.
• Break down jobs into small survivable pieces.
68
Building highly reliable data pipelines

69. Conclusion
• Know your time constraints.
• Break down jobs into small survivable pieces.
• Monitor cluster metrics, job metrics and data lags.
69
Building highly reliable data pipelines

70. Conclusion
• Know your time constraints.
• Break down jobs into small survivable pieces.
• Monitor cluster metrics, job metrics and data lags.
• Think about failures ahead of time and get prepared.
70
Building highly reliable data pipelines

71. Thanks!
We’re hiring!
qf@datadoghq.com
https://jobs.datadoghq.com
71