Resilient Predictive Data Pipelines (GOTO Chicago 2016)
•
2 likes•1,484 views
Sid Anand presented on building resilient predictive data pipelines. The key challenges discussed were the "blast radius" problem where bugs can affect downstream jobs and data, and ensuring timeliness as pipelines and algorithms change over time. Design goals for resilient pipelines include operability, correctness, timeliness and cost. AWS services like SQS, SNS, EMR and auto scaling groups were presented as ways to address these goals by enabling quick recoverability, pay as you go costs, and auto scaling to improve timeliness. Apache Airflow was also discussed as a way to provide workflow automation, scheduling, performance insights and integration with monitoring tools to improve operability and correctness.
![About Me
2
Work [ed | s] @
Committer &
PPMC on
Report to
Co-Chair for](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (9)
Similar to Resilient Predictive Data Pipelines (GOTO Chicago 2016)
Similar to Resilient Predictive Data Pipelines (GOTO Chicago 2016) (20)
More from Sid Anand
More from Sid Anand (18)
Recently uploaded
Recently uploaded (20)
Resilient Predictive Data Pipelines (GOTO Chicago 2016)
- 1. Resilient Predictive Data Pipelines Sid Anand (@r39132) GOTO Chicago 2016 1
- 2. About Me 2 Work [ed | s] @ Committer & PPMC on Report to Co-Chair for
- 3. Motivation Why is a Data Pipeline talk in this Always Available Track? 3
- 4. Motivation 4 Always On work has traditionally focused on the availability of Serving Systems : • Synchronous or Semi-Synchronous • Often Transactional • Latency-sensitive
- 5. HA Goals of Serving Systems 5 Outages are Big News Items!
- 6. And sometimes your failures become your brand! 6
- 7. Motivation 7 Always On work has traditionally focused on the availability of Serving Systems : • Synchronous or Semi-Synchronous • Often Transactional • Latency-sensitive
- 8. Motivation 8 Arguably, the more valuable parts of online services are driven by Data Flow Systems (a.k.a. Data Pipelines): • Asynchronous • Throughput-sensitive
- 10. Serving + Data Pipelines 10 A business’s viability is based on its ability to • keep the site up (Always-On Serving Architectures) & • maintain engagement (views & clicks) with customers (Always- On Data Pipelines) This talk is about Always On Data Pipelines!
- 11. Serving + Data Pipelines 11 Serving Data Pipeline Data Integration Layer Web Servers Microservice Layer Data Layer (DB, Search, Caching, Graph DB, Object Store) FE Load Balancers DAGs + Scheduler + Distributed Computation Engine
- 13. Data Pipeline Challenges Problem 1 : The Blast Radius Problem 13
- 14. The Blast Radius Problem 14 • A developer introduces a bug in Data Pipeline Job 1 • Data Pipeline Job 1 reads Data A & writes Data B
- 15. The Blast Radius Problem 15 • A developer introduces a bug in Data Pipeline Job 1 • Data Pipeline Job 1 reads Data A & writes Data B • Data Pipeline Job 2 reads Data B & writes Data C
- 16. The Blast Radius Problem 16 • A developer introduces a bug in Data Pipeline Job 1 • Data Pipeline Job 1 reads Data A & writes Data B • Data Pipeline Job 2 reads Data B & writes Data C • Data Pipeline Job 3 reads Data C & writes Data D to a Serving System DB
- 17. The Blast Radius Problem 17 • A developer introduces a bug in Data Pipeline Job 1 • Data Pipeline Job 1 reads Data A & writes Data B • Data Pipeline Job 2 reads Data B & writes Data C • Data Pipeline Job 3 reads Data C & writes Data D to a Serving System DB • Serving System 4 reads Data D, where the bug is discovered!
- 18. The Blast Radius Problem 18 •The previous diagram only shows one path of a tree •The reality is much worse •For each data set produced, there are multiple consuming jobs and hence multiple bad downstream outputs
- 19. The Blast Radius Problem An acute pain point 19
- 20. The Blast Radius Problem 20 Detect Bug Job 1 Job 2 Job 3 Serving System
- 21. The Blast Radius Problem 21 Detect Bug Identify Cause
- 23. The Blast Radius Problem 23 Detect Bug Identify Cause
- 24. The Blast Radius Problem 24 Detect Bug Identify Cause Deploy a Fix
- 25. Rollout a Fix & Rerun all Downstream Jobs in the Affected Time Window 25
- 26. The Blast Radius Problem 26 Detect Bug Identify Cause Re_Run All Jobs over a Time Window
- 27. Take Aways? 27 • The cost in people, time, and morale for a Data Pipeline bug is high and they can occur frequently. • In most areas of software, testing is invaluable, less so in data pipelines • Data Pipeline bugs can be due to a logic problem or bad input data! • Best Option : Detect & Rollback/Fix Forward
- 28. The Blast Radius Solution 28 Detect Bug Identify Cause Re_Run 1 Job
- 30. Timeliness Job 1 Job 2 Job 3 Definition : job = workflow = DAG of tasks Job 3’s output is pushed to a serving system
- 31. Timeliness Run 1 : Monday Run 2 : Tuesday Run 3 : Wednesday Consider the Daily Run Schedule below:
- 32. 32 Timeliness Within Time SLA OUT Run 4 Run 5 R u n 6
- 33. Timeliness Why do jobs get slower? 33
- 34. 34 Timeliness
- 40. Take Aways? 40 • Data Science & Engineering work is a virtuous cycle of adding features (and the like) + tuning performance • Latency does matter (a bit)
- 41. Design Goals Desirable Qualities of a Resilient Data Pipeline 41
- 42. 42 Desirable Qualities of a Resilient Data Pipeline OperabilityCorrectness Timeliness Cost
- 43. 43 Desirable Qualities of a Resilient Data Pipeline OperabilityCorrectness Timeliness Cost • Data Integrity (no loss, etc…) • Expected data distributions • All output within time-bound SLAs • Fine-grained Monitoring & Alerting of Correctness & Timeliness SLAs • Quick Recoverability • Pay-as-you-go
- 44. Quickly Recoverable 44 • Bugs happen! • Bugs in Predictive Data Pipelines have a large blast radius • Optimize for MTTR
- 45. Implementation Using AWS to meet Design Goals 45
- 47. SQS - Overview 47 AWS’s low-latency, highly scalable, highly available message queue Infinitely Scalable Queue (though not FIFO) Low End-to-end latency (generally sub-second) Pull-based
- 48. visibility timer SQS - Typical Operation Flow 48 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 1: A consumer reads a message from SQS. This starts a visibility timer!
- 49. visibility timer SQS - Typical Operation Flow 49 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 2: Consumer persists message contents to DB
- 50. visibility timer SQS - Typical Operation Flow 50 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 3: Consumer ACKs message in SQS
- 51. visibility timer SQS - Time Out Example 51 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 1: A consumer reads a message from SQS
- 52. visibility timer SQS - Time Out Example 52 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 2: Consumer attempts persists message contents to DB
- 53. visibility time out SQS - Time Out Example 53 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 3: A Visibility Timeout occurs & the message becomes visible again.
- 54. visibility timer SQS - Time Out Example 54 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 m1 SQS Step 4: Another consumer reads and persists the same message
- 55. visibility timer SQS - Time Out Example 55 Producer Producer Producer m1m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Step 5: Consumer ACKs message in SQS
- 56. SQS - Dead Letter Queue 56 SQS - DLQ visibility timer Producer Producer Producer m2m3m4m5 Consumer Consumer Consumer DB m1 SQS Redrive rule : 2x m1
- 58. SNS - Overview 58 Highly Scalable, Highly Available, Push-based Topic Service Whereas SQS is pull-based, SNS is push-based There is no message retention & there is a finite retry count No Reliable Message Delivery Whereas SQS ensures each message is seen by at least 1 consumer SNS ensures that each message is seen by every consumer Reliable Multi-Push Can we work around this limitation while getting Reliable Multi-push?
- 59. SNS + SQS Design Pattern 59 m1m2 m1m2 m1m2 SQS Q1 SQS Q2 SNS T1 Reliable Multi Push Reliable Message Delivery
- 60. SNS + SQS 60 Producer Producer Producer m1m2 Consumer Consumer Consumer DB m1 m1m2 m1m2 SQS Q1 SQS Q2 SNS T1 Consumer Consumer Consumer ES m1
- 61. Batch Pipeline Architecture Putting the Pieces Together 61
- 62. But First …. 62
- 63. What Does Agari Do? 63
- 64. What Does Agari Do? 64 Customers email metadata apply trust models email + trust score Agari’s Current Product
- 65. What Does Agari Do? 65 Enterprise Customers email metadata apply trust models email md + trust score Agari’s Future Product
- 66. Batch Pipeline Architecture Putting the Pieces Together 66
- 67. Batch Architecture 67 •S3 to hold all source & computed data (Avro) •EMR Spark for scoring + summarization •Apache Airflow for hourly job scheduling •SNS+SQS for messaging •ASG Importer to import •WebApp in Ruby-on- Rails
- 68. Tackling Cost & Timeliness Leveraging the AWS Cloud 68
- 69. Tackling Cost 69 Between Hourly Runs During Hourly Runs
- 70. Tackling Timeliness Auto Scaling Group (ASG) 70
- 71. ASG - Overview 71 What is it? A means to automatically scale out/in clusters to handle variable load/traffic A means to keep a cluster/service of a fixed size always up
- 72. ASG - Data Pipeline 72 importer importer importer importer Importer ASG scaleout/in SQS DB
- 73. 73 Sent CPU ACKd/Recvd CPU-based auto-scaling is good at scaling in/out to keep the average CPU constant ASG : CPU-based
- 74. ASG : CPU-based 74 Sent CPU Recv Premature Scale-in Premature Scale-in: • The CPU drops to noise-levels before all messages are consumed • This causes scale in to occur while the last few messages are still being committed
- 75. 75 Scale-out: When Visible Messages > 0 (a.k.a. when queue depth > 0) Scale-in: When Invisible Messages = 0 (a.k.a. when the last in-flight message is ACK’d) This causes the ASG to grow This causes the ASG to shrink ASG : Queue-based
- 76. 76 Desirable Qualities of a Resilient Data Pipeline OperabilityCorrectness Timeliness Cost • ASG • EMR Spark • ASG • EMR Spark
- 77. Tackling Operability & Correctness Leveraging Tooling 77
- 78. 78 A simple way to author and manage workflows Provides visual insight into the state & performance of workflow runs Integrates with our alerting and monitoring tools Tackling Operability : Requirements
- 79. Apache Airflow Workflow Automation & Scheduling 79
- 80. 80 Airflow: Author DAGs in Python! No need to bundle many config files! Apache Airflow - Authoring DAGs
- 81. 81 Airflow: Visualizing a DAG Apache Airflow - Authoring DAGs
- 82. 82 Airflow: It’s easy to manage multiple DAGs Apache Airflow - Managing DAGs
- 83. Apache Airflow - Perf. Insights 83 Airflow: Gantt chart view reveals the slowest tasks for a run!
- 84. 84 Apache Airflow - Perf. Insights Airflow: Task Duration chart view show task completion time trends!
- 85. 85 Airflow: …And easy to integrate with Ops tools! Apache Airflow - Alerting
- 86. 86 Apache Airflow - Correctness
- 87. 87 Desirable Qualities of a Resilient Data Pipeline OperabilityCorrectness Timeliness Cost
- 88. Near-Real Time Data Pipelines Stream Processing @ Agari 88
- 91. 91 The Architecture is composed of repeated patterns of : ASG-based compute Kinesis streams (i.e. AWS’ managed “Kafka”) Lambda-based Avro Schema Registry NRT Architecture
- 92. Avro Schema Registry Avro Schema Storage 92
- 93. 93 { "namespace":"com.agari.ep.collector.model", "type":"record", "doc":"This Schema describes the server-side configuration of Agari's Enterprise-Protect Collector", "name":"collector_config", "fields":[ {"name": "email_log_enabled", "type": "boolean"}, {"name": "email_log_interval_seconds", "type": ["int", "null"]}, {"name": "email_log_bucket_name", "type": "string"}, {"name": "phone_home_interval_seconds", "type": "int"}, {"name": "phone_home_sns_topic_ARN", "type": "string"}, {"name": "config_pull_interval_seconds", "type": "int"}, {"name": "receiver_netblocks", "type": ["null", {"type": "array", "items": "string"}]}, { "name": "connecting_ip", "type": [ "null", { "type": "record", "name": "connecting_ip_record", "fields": [ { "name": "received_header_index", "type": "int" }, ] …… A self-describing (schema’d) serialization format What is Avro?
- 94. What is Avro? 94 Typically, the schema is stored in the same file as the data it represents In HDFS, where files are typically large, the schema overhead is negligible
- 95. Schema Registry 95 P P C C C Kinesis Stream In streaming, where each record may be sent individually, the schema will be the majority of the data transmitted! This is a fat message Can we be smarter?
- 97. What is AWS Lambda? 97 AWS-hosted code execution environment (Python, Node, Java, Ruby) You upload some code & specify a simple memory and CPU profile (e.g. medium CPU, 256 GB memory) The code will get a new version (e.g. v2) Code Rollback as easy as setting $LATEST alias to a previous version (e.g. $LATEST=v1)
- 98. Elastic Stream Processing How Do We Handle Increasing Traffic? 98
- 100. Elastic Stream Processing 100 P P C C C Kinesis Stream
- 101. Elastic Stream Processing 101 C C C Kinesis Stream C C C P P Agari Scaling Utils
- 102. Open Source Plans 102 In late Q2/early Q3, we plan to open-source our cloud tools for : • Avro Schema Registry & • Agari (Kinesis+ASG) scaling tools To be notified, follow @AgariEng & @r39132
- 103. Acknowledgments 103 • Vidur Apparao • Stephen Cattaneo • Jon Chase • Andrew Flury • William Forrester • Chris Haag • Mike Jones • Scot Kennedy • Thede Loder • Paul Lorence • Kevin Mandich • Gabriel Ortiz • Jacob Rideout • Josh Yang None of this work would be possible behind the contributions of the strong team below