Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Flink Forward Berlin 2018: Paris Carbone - "Stream Loops on Flink: Reinventing the wheel for the streaming era"

400 views

Published on

You have probably heard that stream processing subsumes batch workloads, a valid but not yet fully implemented claim. Our lab research aims to fulfil this dream and delve further into the deep world of iterative processes, a fundamental building block for graph and machine learning algorithms. Yet, a building block that is missing from your stream pipelines today. In this talk, we will investigate why bulk and stale synchronous iterative models are nothing more than a special case of out-of-order stream processing, the paradigm behind your ultra fast watermark-based window aggregations on Flink and Beam. Next, we will examine how watermarks can be extended to incorporate more metrics for tracking iterative progress as well as the necessary structured graph modifications (spoiler alert: loops) that can make our lives easier. Finally, we will demonstrate how on top of these primitives we can execute scalable multi-pass window aggregations with purgeable and persistent managed state as well as robust flow control and several domain specific applications such as Vertex-centric graph aggregations and Stochastic Gradient Descent on stream windows.

Published in: Technology
  • Be the first to comment

Flink Forward Berlin 2018: Paris Carbone - "Stream Loops on Flink: Reinventing the wheel for the streaming era"

  1. 1. Stream Loops on Flink Reinventing the wheel for the streaming era Paris Carbone @FF2018-Berlin Systems Researcher@KTH/SICS <parisc@kth.se> Committer@Flink <senorcarbone@apache.org> 1
  2. 2. Stream Compute Landmarks A Timeline Sliding Window Semantics Task Parallel Execution Arbitrary Computation Nesting Out-of-order Processing (watermarks) State Management -Fault Tolerance -Reconfiguration 2
  3. 3. Has Streaming Subsumed Batch? short answer: NO long answer: not YET • Staged Processing • Allows Bulk/Delta Iterative Computation multi-pass Flink DataSet • Continuous Processing • Allows correct Acyclic Computation Flink DataStream single-pass ? 3
  4. 4. Why Iterations Matter • Iterations are fundamental building blocks for: • Graph Analysis (PageRank, Conn.Comp., SSSP etc.) • Machine Learning (Gradient Descent, PCA etc.) • Transactions (e.g., optimistic concurrency control) Iterative Processing Primitives are currently not present in todays’ production-grade stream processing systems. 4
  5. 5. Current Challenges • Data is born and evolves continuously as a data stream (e.g., user interactions, sensor events, server logs) 5
  6. 6. Current Challenges Day 1 Day 2 Day 3 Day 4 • To run iterative analysis users have manually do the… • bucketing • laundry (schedule jobs) • cleaning (integration) 6
  7. 7. Current Challenges • To run iterative analysis users have manually do the… • bucketing • laundry (iterative job scheduling) • cleaning (integration) Day 1 Day 2 Day 3 Day 4 7
  8. 8. Current Challenges • To run iterative analysis users have manually do the… • bucketing • laundry (iterative job scheduling) • sorting (model integration) Day 1 Day 2 Day 3 Day 4 8
  9. 9. Current Challenges • To run iterative analysis users have manually do the… • bucketing • laundry (iterative job scheduling) • sorting (model integration) Day 1 Day 2 Day 3 Day 4 9
  10. 10. A Partially Solved Problem • Most challenges mentioned are solved and automated for single-pass aggregation via stream windowing. How would a distributed iterative computation look like as a special type of window aggregation? window computation windows are a natural fit for declaring iterative computation 10
  11. 11. Anatomy of Iterative Computation Termination (fixpoint/fixed) Loop Step Function (computation) Final Result-Solution Solution (finite state) 11
  12. 12. Programming Model Extensions Distributed Dataflow Execution DataStream Model StreamML Stream Libraries Core Stream API Runtime Graph CEP SQL Table • Multi-Pass Window Aggregations • Synchrony and Termination • Pregel on Windows Proposed Additions based on Flink 1.6 12
  13. 13. Window Iterations An extension of existing window aggregation to multi-pass aggregates 13
  14. 14. entry step finalizewindow iterate OUTIN R Loop Context 14
  15. 15. PageRank Example entry step finalize window iterate OUT IN 15
  16. 16. PageRank Example entry step finalize window iterate OUT IN 16
  17. 17. PageRank Example entry step finalize window iterate OUT IN 17
  18. 18. PageRank Example entry step finalize window iterate OUT IN 18
  19. 19. PageRank Example entry step finalize window iterate OUT IN 19
  20. 20. Higher-Level Abstractions • Vertex-Centric Model (Part of experimental Gelly-Streams lib) Example: single source shortest paths 20
  21. 21. Enough about what • We need to examine “how” to implement iterations executed in parallel and out-of-order using event-time progress window computation 21
  22. 22. How Event-Time Progress Works T t progress metric record • Every record carries a timestamp t • t ≥ T • T can only increment (monotonicity) T: indication of a low event-time boundstream task 22
  23. 23. How Event-Time Progress Works • low watermarks propagate progress metrics along the computational graph • progress metric = minimum watermark across channels • works correctly as long as monotonicity is maintained 23
  24. 24. Intuition Iteration Superstep progress bears similarities to Event-Time progress 1. Can be used to run computation for out-of-order streams. 2. Need for a decentralised mechanism (no coordinator/scheduler). 3. Only relevant to respective parts of the graph (i.e., sources cannot make use of sink progress) 24
  25. 25. Watermarks and Unstructured Loops t2 t4t1 W=n … …t3 • Low watermarking is a very powerful mechanism to measure and propagate progress of a single progress metric. • However it breaks when we have arbitrary cycles (current Flink iterations) old watermark 25 which superstep? which cycle?
  26. 26. Proposed Extensions Distributed Dataflow Execution DataStream Model StreamML Stream Libraries Core Stream API Runtime Graph CEP SQL Table • Structured Loops • Progress Timestamps • Cyclic Flow Control runtime Additions based on Flink 1.6 26
  27. 27. Structured Loops IN OUT structured loop global scope • Structure loops are low level stream graph primitives. • They can be seen as ‘Iterative Operators’ • Introduce the notion of “structured programming” for data streaming. • Each structured loop has a scope. • Each scope operates on its own progress metric. • Global scope operates on event-time progress (no changes made). R 27
  28. 28. Structured Loops Composition R IN OUT loop scope global scope R’ nested loop scope 28 • Encourages compositionally (arbitrary nesting)
  29. 29. Using Progress Timestamps • Nests progress timestamps • (e.g., last superstep pT per window Pctx) • How do we guarantee monotonic progress metrics? 29
  30. 30. Structured Loop Embeddings R IN OUT LIN LOUT LH LT nest unnest incr [32] [0,32] [32] [0,32] [1,32] [1,32] [31] term [ø,32] User-Defined Loop DAG [ø,32][ø,32] • Initializes Scope’s Progress Metric for both records and watermarks (nest) • Triggers final output of an iterative operator • Removes inner progress metric (unnest) • Increments Progress Metric (incr) • Forwards Records and Watermarks • Evaluates and Disseminates Termination (ø) 30
  31. 31. Example: Window Iteration R IN OUT LIN LOUT LH LT entry step fin window split Managed State 31
  32. 32. Further Challenges • Cyclic Flow Control • avoid deadlocks • encourage iteration completion • State Management Integration • one more level of indirection (mapvalue per key) what keeps us busy 32
  33. 33. Cyclic Flow Control R IN OUT LIN LOUT LH LT union entry step fin window split Managed State flow control R IN • (Consumed Buffers < Threshold): Round-Robin between IN and R • (Consumed Buffers >= Threshold): Exclusive Ingestion of R 33
  34. 34. FLIP-15 https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66853132 • Introduces Scoping and Nesting (for Structured Loops) • Addresses Flow Control Alternatives • Discusses Computation-agnostic Loop Termination 34
  35. 35. Research Team and Sponsors • Marius Melzer (TUDresden) • Asterios Katsifodimos (TUDelft) • Vasiliki Kalavri (ETH) • Seif Haridi (KTH) • Pramod Bhatotia (Un.Edinburgh) 35
  36. 36. References • Scalable and Reliable Data Stream Processing - Paris Carbone (2018) • Inflight Progress Tracking for Stream Processing Systems with Cyclic Dataflows - Marius Melzer (2017) • Naiad: a timely dataflow system - Murray, McSherry et al. (2013) • The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing - Tyler Akidau et al. (2015) • Out-of-order processing: a new architecture for high- performance stream systems - Li, Maier et al. (2008) • Flexible time management in data stream systems - Srivastava, Widom (2004) 36
  37. 37. Stream Loops on Flink Reinventing the wheel for the streaming era Paris Carbone @FF2018-Berlin Systems Researcher@KTH/SICS <parisc@kth.se> Committer@Flink <senorcarbone@apache.org> 37

×