Uploaded byFlink Forward
Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview
The document discusses the concepts of stateful big data processing using Apache Beam, highlighting its architecture, features, and use cases. It covers the integration of state and timers within the processing model, emphasizing their role in enabling new functionalities and optimizing performance. Additionally, it provides links to further resources, community discussions, and avenues for contributing to the project.
More Related Content
Similar to Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview
Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview
- 1. Portable stateful bigdata processing in Apache Beam Kenneth Knowles Apache Beam PMC Software Engineer @ Google klk@google.com / @KennKnowles https://s.apache.org/ffsf-2017-beam-state Flink Forward San Francisco 2017
- 2. Agenda 1. What isApache Beam? 2. State 3. Timers 4. Example & Little Demo
- 3.
- 4. TL;DR 4 (Flink draws itmore like this)
- 5. 20142004 2006 20082010 2012 20162005 2007 2009 2013 20152011 MapReduce (paper) Apache Hadoop Dataflow Model (paper) MillWheel (paper) Heron Apache Spark Apache Storm Apache Gearpump (incubating) Apache Apex Apache Flink Cloud Dataflow FlumeJava (paper) Apache Beam DAGs, DAGs, DAGs Apache Samza
- 6. The Beam Vision SumPer Key 6 input.apply( Sum.integersPerKey()) Java input | Sum.PerKey() Python ⋮ Apache Flink local, on-prem, cloud Apache Spark local, on-prem, cloud Cloud Dataflow: fully managed ⋮ Apache Apex local, on-prem, cloud Apache Gearpump (incubating)
- 7. The Beam Vision KafkaIO 7 input| KakaIO.read() Python ⋮ class KafkaIO extends UnboundedSource { … } Java Apache Flink local, on-prem, cloud Apache Spark local, on-prem, cloud Cloud Dataflow: fully managed ⋮ Apache Apex local, on-prem, cloud Apache Gearpump (incubating)
- 8.
- 9. The Beam Model Whatare you computing? (read, map, reduce) Where in event time? (event time windowing) 9 When in processing time are results produced? (triggers) How do refinements relate? (accumulation mode)
- 10. What are youcomputing? 10 Read Parallel connectors to external systems ParDo Per element "Map" Grouping Group by key, Combine per key, "Reduce" Composite Encapsulated subgraph
- 11.
- 12. What are youcomputing? 12 Read Parallel connectors to external systems ParDo Per element "Map" Grouping Group by key, Combine per key, "Reduce" Composite Encapsulated subgraph
- 13.
- 14. "Example" 14 Per Key Quantiles ParDo.of(newDoFn<...>() { // declare some state @ProcessElement public void process(...) { // update quantiles // output if needed } })
- 15.
- 16. Parallelism! 16 Quantiles DoFn Quantiles DoFn Quantiles DoFn new DoFn<Foo, Quantiles<Foo>>(){ @StateId("quantiles") private final StateSpec<...> quantilesState = StateSpecs.combining(); … }
- 17. Windowed 17 Quantiles DoFn Quantiles DoFn Quantiles DoFn Window into Fixedwindows of one hour Expected result: Quantiles for each hour
- 18. Windowed 18 Quantiles DoFn Quantiles DoFn Quantiles DoFn Window into windowsof 30 min sliding by 10 min Expected result: Quantiles for 30 minutes sliding by 10 min
- 19. <k, w>1 <k, w>2 <k,w>3 ... "x" 3 7 15 "y" "fizz" "7" "fizzbuzz" ... State is per key and window Bonus: automatically garbage collected when a window expires (vs manual clearing of keyed state)
- 20. Windowed 20 Quantiles DoFn Quantiles DoFn Quantiles DoFn Window into Fixedwindows of one hour Expected result: Quantiles for each hour
- 21. What about Combine? 21 Quantiles Combiner Windowinto Fixed windows of one hour Expected result: Quantiles for each hour Quantiles Combiner Quantiles Combiner
- 22. Combine vs State(naive conceptualization) 22 Window hourly Expected result: Quantiles for each hour Quantiles Combiner Window hourly Quantiles DoFn Expected result: Quantiles for each hour
- 23. Combine vs State(likely execution plan) Quantiles Combiner Quantiles Combiner Quantiles Combiner Shuffle Accumulators Quantiles DoFn Shuffle Elements associative commutative single-output enables optimizations (engine is in control) non-associative* non-commutative* multi-output side outputs (user is in control)
- 24. Combine vs State Quantiles Combiner Quantiles DoFn outputgoverned by trigger (data/computation unaware) "output only when there's an interesting change" (data/computation aware)
- 25. Example: Arbitrary indices D,0 A Assignindices B C D E C,1 A,2 B,3 E,4 non-associative non-commutative non-deterministic and totally fine!
- 26. Kinds of state ●Value - just a mutable cell for a value ● Bag - supports "blind writes" ● Combining - has a CombineFn built in; can support blind writes and lazy accumulation ● Set - membership checking ● Map - lookups and partial writes
- 27.
- 28. Timers for ParDo 28 StatefulParDo new DoFn<...>() { @TimerId("timeout") private final TimerSpec timeoutTimer = TimerSpecs.timer(TimeDomain.PROCESSING_TIME); @OnTimer("timout") public void timeout(...) { // access state, set new timers, output } … }
- 29. Timers in processingtime "call me back in 5 seconds" output based only on incoming element output return value of batched RPC buffer request batched RPC On Timer On Element
- 30. Timers in eventtime "call me back when the watermark hits the end of the window" output speculative result immediately output replacement value only if changed store speculative result On Timer On Element
- 31. ● Per-key arbitrarynumbering ● Output only when result changes ● Tighter "side input" management for slowly changing dimension ● Streaming join-matrix / join-biclique ● Fine-grained combine aggregation and output control ● Per-key "workflows" like user sign up flow w/ expiration ● Low-latency deduplication (let the first through, squash the rest) More example uses for state & timers
- 32. Performance considerations (cross-runner) 32 ●Shuffle to colocate keys ● Linear processing of elements for key+window ● Window merging ● Storage of state and timers ● GC of state
- 33.
- 34. Summary State and Timersin Beam... ● … unlock new uses cases ● … they "just work" with event time windowing ● … are portable across runners (implementation ongoing)
- 35. Thank you forlistening! This talk: ● Me - @KennKnowles ● These Slides - https://s.apache.org/ffsf-2017-beam-state Go Deeper ● Design doc - https://s.apache.org/beam-state ● Blog post - https://beam.apache.org/blog/2017/02/13/stateful-processing.html Join the Beam community: ● User discussions - user@beam.apache.org ● Development discussions - dev@beam.apache.org ● Follow @ApacheBeam on Twitter You can contribute to Beam + Flink ● New types of state ● Easy launch of Beam job on Flink-on-YARN ● Integration tests at scale ● Fit and finish: polish, polish, polish! ● … and lots more! https://beam.apache.org