Downloaded 511 times
![How do refinements relate?
• How should multiple outputs per window accumulate?
• Appropriate choice depends on consumer.
Firing Elements
Speculative [3]
Watermark [5, 1]
Late [2]
Last Observed
Total Observed
Discarding
3
6
2
2
11
Accumulating
3
9
11
11
23
Acc. & Retracting
3
9, -3
11, -9
11
11
(Accumulating & Retracting not yet implemented.)](https://image.slidesharecdn.com/june30410googlebonaci-160711213841/85/Apache-Beam-A-unified-model-for-batch-and-stream-processing-data-27-320.jpg)
![Solution: Dynamic work rebalancing
class MyReader
extends BoundedReader<T> {
[...]
getFractionConsumed() { }
splitAtFraction(...) { }
}
class MySource
extends BoundedSource<T> {
[...]
splitIntoBundles() { }
getEstimatedSizeBytes() { }
}](https://image.slidesharecdn.com/june30410googlebonaci-160711213841/85/Apache-Beam-A-unified-model-for-batch-and-stream-processing-data-51-320.jpg)
Uploaded byDataWorks Summit/Hadoop Summit
Apache Beam: A unified model for batch and stream processing data
Apache Beam is a unified programming model for batch and streaming data processing. It defines concepts for describing what computations to perform (the transformations), where the data is located in time (windowing), when to emit results (triggering), and how to accumulate results over time (accumulation mode). Beam aims to provide portable pipelines across multiple execution engines, including Apache Flink, Apache Spark, and Google Cloud Dataflow. The talk will cover the key concepts of the Beam model and how it provides unified, efficient, and portable data processing pipelines.
More Related Content
Similar to Apache Beam: A unified model for batch and stream processing data
More from DataWorks Summit/Hadoop Summit
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
Apache Beam: A unified model for batch and stream processing data
- 1. Unbounded, unordered, globalscale datasets are increasingly common in day-today business, and consumers of these datasets have detailed requirements for latency, cost, and completeness. Apache Beam (incubating) defines a new data processing programming model that evolved from more than a decade of experience building Big Data infrastructure within Google, including MapReduce, FlumeJava, Millwheel, and Cloud Dataflow. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtime environments, both open source (e.g., Apache Flink, Apache Spark, et al.), and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, touch on its evolution, and describe main concepts in the programming model. During the talk, we’ll argue why Beam is unified, efficient and portable. Abstract
- 2. Davor Bonaci Apache BeamPPMC Software Engineer, Google Inc. Apache Beam: A Unified Model for Batch and Streaming Data Processing Hadoop Summit, June 28-30, 2016, San Jose, CA
- 3. Apache Beam is aunified programming model designed to provide efficient and portable data processing pipelines
- 4. 1. The BeamModel: What / Where / When / How 2. SDKs for writing Beam pipelines: • Java • Python 3. Runners for Existing Distributed Processing Backends • Apache Flink • Apache Spark • Google Cloud Dataflow • Local runner for testing What is Apache Beam? Beam Model: Fn Runners Apache Flink Apache Spark Beam Model: Pipeline Construction Other LanguagesBeam Java Beam Python Execution Execution Google Cloud Dataflow Execution
- 5. The Evolution ofBeam MapReduce Google Cloud Dataflow Apache Beam BigTable DremelColossus FlumeMegastoreSpanner PubSub Millwheel
- 6. Apache Beam is aunified programming model designed to provide efficient and portable data processing pipelines
- 7.
- 8.
- 9.
- 10. … maybe infinitelybig... 9:008:00 14:0013:0012:0011:0010:00
- 11. … with unknowndelays. 9:008:00 14:0013:0012:0011:0010:00 8:00 8:008:00
- 12. Reality Formalizing Event-Time Skew ProcessingTime EventTime Ideal Skew
- 13. Formalizing Event-Time Skew Watermarksdescribe event time progress. "No timestamp earlier than the watermark will be seen" ProcessingTime Event Time ~Watermark Ideal Skew Often heuristic-based. Too Slow? Results are delayed. Too Fast? Some data is late.
- 14. What are youcomputing? Where in event time? When in processing time? How do refinements relate?
- 15. What are youcomputing? Element-Wise Aggregating Composite
- 16. What: Computing IntegerSums // Collection of raw log lines PCollection<String> raw = IO.read(...); // Element-wise transformation into team/score pairs PCollection<KV<String, Integer>> input = raw.apply(ParDo.of(new ParseFn()); // Composite transformation containing an aggregation PCollection<KV<String, Integer>> scores = input.apply(Sum.integersPerKey());
- 17.
- 18.
- 19. Windowing divides datainto event-time-based finite chunks. Often required when doing aggregations over unbounded data. Where in event time? Fixed Sliding 1 2 3 54 Sessions 2 431 Key 2 Key 1 Key 3 Time 2 3 4
- 20. Where: Fixed 2-minuteWindows PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Minutes(2))) .apply(Sum.integersPerKey());
- 21. Where: Fixed 2-minuteWindows
- 22. When in processingtime? • Triggers control when results are emitted. • Triggers are often relative to the watermark. ProcessingTime Event Time ~Watermark Ideal Skew
- 23. When: Triggering atthe Watermark PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Minutes(2)) .triggering(AtWatermark())) .apply(Sum.integersPerKey());
- 24. When: Triggering atthe Watermark
- 25. When: Early andLate Firings PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Minutes(2)) .triggering(AtWatermark() .withEarlyFirings(AtPeriod(Minutes(1))) .withLateFirings(AtCount(1)))) .apply(Sum.integersPerKey());
- 26. When: Early andLate Firings
- 27. How do refinementsrelate? • How should multiple outputs per window accumulate? • Appropriate choice depends on consumer. Firing Elements Speculative [3] Watermark [5, 1] Late [2] Last Observed Total Observed Discarding 3 6 2 2 11 Accumulating 3 9 11 11 23 Acc. & Retracting 3 9, -3 11, -9 11 11 (Accumulating & Retracting not yet implemented.)
- 28. How: Add Newest,Remove Previous PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Minutes(2)) .triggering(AtWatermark() .withEarlyFirings(AtPeriod(Minutes(1))) .withLateFirings(AtCount(1))) .accumulatingAndRetractingFiredPanes()) .apply(Sum.integersPerKey());
- 29. How: Add Newest,Remove Previous
- 30.
- 31.
- 32. Distributed Systems areDistributed
- 33. Event Time Resultsare Stable
- 34.
- 35. Sessions PCollection<KV<String, Integer>> scores= input .apply(Window.into(Sessions.withGapDuration(Minutes(1)) .triggering(AtWatermark() .withEarlyFirings(AtPeriod(Minutes(1))) .withLateFirings(AtCount(1))) .accumulatingAndRetractingFiredPanes()) .apply(Sum.integersPerKey());
- 36. Identifying Bursts ofUser Activity
- 37.
- 38. 1.Classic Batch 2.Batch with Fixed Windows 3. Streaming 5. Streaming With Retractions 4. Streaming with Speculative + Late Data 6. Sessions
- 39.
- 40. PCollection<KV<String, Integer>> scores= input .apply(Sum.integersPerKey()); PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Minutes(2)) .triggering(AtWatermark() .withEarlyFirings(AtPeriod(Minutes(1))) .withLateFirings(AtCount(1))) .accumulatingAndRetractingFiredPanes()) .apply(Sum.integersPerKey()); PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Minutes(2)) .triggering(AtWatermark() .withEarlyFirings(AtPeriod(Minutes(1))) .withLateFirings(AtCount(1))) .apply(Sum.integersPerKey()); PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Minutes(2)) .triggering(AtWatermark())) .apply(Sum.integersPerKey()); PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Minutes(2))) .apply(Sum.integersPerKey()); PCollection<KV<String, Integer>> scores = input .apply(Window.into(Sessions.withGapDuration(Minutes(2)) .triggering(AtWatermark() .withEarlyFirings(AtPeriod(Minutes(1))) .withLateFirings(AtCount(1))) .accumulatingAndRetractingFiredPanes()) .apply(Sum.integersPerKey()); 1.Classic Batch 2. Batch with Fixed Windows 3. Streaming 5. Streaming With Retractions 4. Streaming with Speculative + Late Data 6. Sessions
- 41.
- 42. Apache Beam is aunified programming model designed to provide efficient and portable data processing pipelines
- 43. Workload vary inpipelines over timeWorkload Time Batch pipelines go through stagesStreaming pipeline’s input varies
- 44. Perils of fixeddecisionsWorkload Time Under-provisioned / average caseOver-provisioned / worst case Workload Time
- 45.
- 46. Solution: bundles class MyDoFnextends DoFn<String, String> { void startBundle(...) { } void processElement(...) { } void finishBundle(...) { } } • User code operates on bundles of elements. • Easy parallelization. • Dynamic sizing. • Parallelism decisions in the runner’s hands.
- 47. The Straggler Problem •Work is unevenly distributed across tasks. • Reasons: • Underlying data. • Processing. • Effects multiplied per stage. Worker Time
- 48. “Standard” workarounds forstragglers • Split files into equal sizes? • Pre-emptively over-split? • Detect slow workers and re- execute? • Sample extensively and then split? Worker Time
- 49. No amount ofupfront heuristic tuning (be it manual or automatic) is enough to guarantee good performance: the system will always hit unpredictable situations at run-time. A system that's able to dynamically adapt and get out of a bad situation is much more powerful than one that heuristically hopes to avoid getting into it.
- 50. Solution: Dynamic WorkRebalancing Done work Active work Predicted completion Split Now Average completion Time Now Average completion Time
- 51. Solution: Dynamic workrebalancing class MyReader extends BoundedReader<T> { [...] getFractionConsumed() { } splitAtFraction(...) { } } class MySource extends BoundedSource<T> { [...] splitIntoBundles() { } getEstimatedSizeBytes() { } }
- 52.
- 53. Dynamic bundles +work re-balancing + autoscaling
- 54. Apache Beam is aunified programming model designed to provide efficient and portable data processing pipelines
- 55. 1. Write: Choosean SDK to write your pipeline in. 2. Execute: Choose any runner at execution time. Apache Beam Architecture Beam Model: Fn Runners Apache Flink Apache Spark Beam Model: Pipeline Construction Other LanguagesBeam Java Beam Python Execution Execution Google Cloud Dataflow Execution
- 56.
- 57. 1. End users:who want to write pipelines or transform libraries in a language that’s familiar. 2. SDK writers: who want to make Beam concepts available in new languages. 3. Runner writers: who have a distributed processing environment and want to support Beam pipelines Multiple categories of users Beam Model: Fn Runners Apache Flink Apache Spark Beam Model: Pipeline Construction Other LanguagesBeam Java Beam Python Execution Execution Google Cloud Dataflow Execution
- 58. • If youhave Big Data APIs, write a Beam SDK or DSL or library of transformations. • If you have a distributed processing backend, write a Beam runner! • If you have a data storage or messaging system, write a Beam IO connector! Growing the Open Source Community
- 59. Apache Beam is aunified programming model designed to provide efficient and portable data processing pipelines
- 60. Visions are aJourney 02/01/2016 Enter Apache Incubator End 2016 Beam pipelines run on many runners in production uses Early 2016 Design for use cases, begin refactoring Mid 2016 Additional refactoring, non-production uses Late 2016 Multiple runners execute Beam pipelines 02/25/2016 1st commit to ASF repository 06/14/2016 1st incubating release June 2016 Python SDK moves to Beam
- 61. Learn More! Apache Beam(incubating) http://beam.incubator.apache.org The World Beyond Batch 101 & 102 https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102 Join the Beam mailing lists! user-subscribe@beam.incubator.apache.org dev-subscribe@beam.incubator.apache.org Follow @ApacheBeam on Twitter
- 62.
- 63.
- 64. Element-wise transformations 13:00 14:008:009:00 10:00 11:00 12:00 Processing Time
- 65. Aggregating via Processing-TimeWindows 13:00 14:008:00 9:00 10:00 11:00 12:00 Processing Time
- 66. Aggregating via Event-TimeWindows Event Time Processing Time 11:0010:00 15:0014:0013:0012:00 11:0010:00 15:0014:0013:0012:00 Input Output
- 67. Processing Time ResultsDiffer
- 68. Identifying Bursts ofUser Activity
- 69.
- 70.
- 72. Calculating the AverageSession Length .apply(Window.into(FixedWindows.of(Minutes(2))) .trigger(AtWatermark()) .withEarlyFirings(AtPeriod(Minutes(1))) .accumulatingFiredPanes()) .apply(Mean.globally()); input .apply(Window.into(Sessions.withGapDuration(Minutes(1))) .trigger(AtWatermark()) .discardingFiredPanes()) .apply(CalculateWindowLength()));
Editor's Notes
- #6 Google published the original paper on MapReduce in 2004 -- fundamentally change the way we do distributed processing. <animate> Inside Google, kept innovating, but just published papers <animate> Externally the open source community created Hadoop. Entire ecosystem flourished, partially influenced by those Google papers. <animate> In 2014, Google Cloud Dataflow -- included both a new programming model and fully managed service share this model more broadly -- both because it is awesome and because users benefit from a larger ecosystem and portability across multiple runtimes. So Google, along with a handful of partners donated this programming model to the Apache Software Foundation, as the incubating project Apache Beam...
- #8 here’s gaming logs each square represents an event where a user scored some points for their team
- #9 game gets popular
- #10 start organizing it into a repeated structure
- #11 repetitive structure just a cheap way of representing an infinite data source. game logs are continuous distributed systems can cause ambiguity...
- #12 Lets look at some points that were scored at 8am <animate> red score 8am, received quickly <animate> yellow score also happened at 8am, received at 8:30 due to network congestion <animate> green element was hours late. this was someone playing in airplane mode on the plane. had to wait for it to land. so now we’ve got an unordered, infinite data set, how do we process it...
- #13 Blue axis is event, Green is processing. Ideally no delay -- elements processed when they occurred <animate> Reality looks more like that red squiggly line, where processing time is slightly delayed off event time. <animate> The variable distance between reality and ideal is called skew. need to track in order to reason about correctness.
- #14 red line the watermark -- no event times earlier than this point are expected to appear in the future. often heuristic based too slow → unnecessary latency. too fast → some data comes in late, after we thought we were done for a given time period. how do we reason about these types of infinite, out-of-order datasets...
- #15 not too hard if you know what kinds of questions to ask! What results are calculated? sums, joins, histograms, machine learning models? Where in event time are results calculated? Does the time each event originally occurred affect results? Are results aggregated for all time, in fixed windows, or as user activity sessions? When in processing time are results materialized? Does the time each element arrives in the system affect results? How do we know when to emit a result? What do we do about data that comes in late from those pesky users playing on transatlantic flights? And finally, how do refinements relate? If we choose to emit results multiple times, is each result independent and distinct, do they build upon one another? Let’s dive into how each question contributes when we build a pipeline...
- #16 The first thing to figure out is what you are actually computing. transform each element independently, similar to the Map function in MapReduce, easy to parallelize Other transformations, like Grouping and Combining, require inspecting multiple elements at a time Some operations are really just subgraphs of other more primitive operations. Now let’s see a code snippet for our gaming example...
- #17 Psuedo Java for compactness/clarity! start by reading a collection of raw events transform it into a more structured collection containing key value pairs with a team name and the number of points scored during the event. use a composite operation to sum up all the points per team. Let’s see how this code excutes...
- #18 Looking at points scored for a given team blue axis, green axis <animate> ideal <animate> This score of 3 from just before 12:07 arrives almost immediately. <animate> 7 minutes delayed. elevator or subway. graph not big enough to show offline mode from transatlantic flight
- #19 time is thick white line. accumulate sum into the intermediate state produces output represented by the blue rectangle. all the data available, rectangle covers all events, no matter when in time they occurred. single final result emitted when it’s all complete pretty standard batch processing -> let’s see what happens if we tweak the other questions
- #20 windowing lets us create individual results for different slices of event time. divides data into finite chunks based on the event time of each element. common patterns include fixed time (like hourly, daily, monthly), sliding windows (like the last 24 hours worth of data, every hour) -- single element may be in multiple overlapping windows session based windows that capture bursts of user activity -- unaligned per key very common when trying to aggregations on infinite data also actually common pattern in batch, though historically done using composite keys.
- #21 fixed windows that are 2 minutes long
- #22 independent answer for every two minute period of event time. still waiting until the entire computation completes to emit any results. won’t work for infinite data! want to reduce latency...
- #23 trigger define when in processing time to emit results often relative to the watermark, which is that heuristic about event time progress.
- #24 request that results are emitted when we think we’ve roughly seen all the elements for a given window. actually default -- just written it for clarity.
- #25 left graph shows a perfect watermark -- tracks when all the data for a given event time has arrived emit the result from each window as soon as the watermark passes. <animate> watermark is usually just a heuristic, so look more like graph on the right. now 9 is missed and if the watermark is delayed, like in the first graph, need to wait a long time for anything. would like speculative. lets use a more advanced trigger...
- #26 ask for early, speculative firings every minute get updates every time a late element comes in.
- #27 in all cases, able to get speculative results before the watermark. now get results when watermark passes, but still handle late value 9 even with heuristic watermark in this case, we accumulate across the multiple results per window In the final window, we see and emit 3 but then still include that 3 in the next update of 12. but this behavior around multiple firings is configurable...
- #28 fire three times for a window -- a speculative firing with 3, watermark with two more values 5 and 1, and finally a late value 2. one option is emit new elements that have come in since the last result. requires consumer to be able to do final sum could produce the running sum every time. consumer may overcount produce both the new running sum and retract the old one.
- #29 use accumulating and retracting.
- #30 speculative results, on time results, and retractions. now the final window emits 3, then retracts the 3 when emitting 12. So those are the four questions...
- #31 those are the four key questions are they the right questions? here are 5 reasons...
- #32 the results we get are correct this is not something we’ve historically gotten with streaming systems.
- #33 distributed systems are … distributed. if the winds had been blowing from the east instead of the west, elements might have arrived in a slightly different order.
- #34 aggregating based on event time may have different intermediate results but the final results are identical across the two arrival scenarios.
- #35 next, the abstractions can represent powerful and complex algorithms.
- #36 earlier mentioned session windows -- burst of user activity simple code change...
- #37 want to identify two groupings of points in other words, Tyler was playing the game, got distracted by a squirrel, and then resumed his play.
- #38 ok… flexibility for covering all sorts of uses cases
- #39 By tuning our what/where/when/how knobs, we’ve covered everything from classic batch… to sessions
- #40 And not only that, we do so with lovely modular code
- #41 all these uses cases -- and we never changed our core algorithm just integer summing here, but the same would apply with much more complex algorithms too
- #42 so there you go -- 5 reasons that these 4 questions are awesome
- #48 Data 1 file per task & files of different sizes Bigtable key range partitioned lexicographically, assuming uniform Processing Hot shuffle key ranges Data-dependent computation
- #49 Pre-job stage: chunk files into equal sizes Choice of constant? Does not handle runtime asymmetry Pre-emptively over-split How much is enough? How much is too much? Per-task overheads can dominate Detect slow workers and re-execute Does not handle processing asymmetry Sample (maybe extensively) and then split Overhead Still does not handle runtime asymmetry
- #53 400 workers Read GCS → Parse → GroupByKey → Write
- #57 The Beam model is attempting to generalize semantics -- will not align perfectly with all possible runtimes. Started categorizing the features in the model and the various levels of runner support. This will help users understand mismatches like using event time processing in Spark or exactly once processing with Samza.
- #58 fully support three different categories of users End users who want to write data processing pipelines Includes adding value like additional connectors -- we’ve got Kafka! Additionally, support community-sourced SDKs and runners Each community has very different sets of goals and needs. having a vision and reaching it are two different things...
- #59 And one of the things we’re most excited about is the collaboration opportunities that Beam enables. Been doing this stuff for a while at Google -- very hermetic environment. Looking forward to incorporating new perspectives -- to build a truly generalizable solution. Growing the Beam development community over the next few months, whether they are looking to write transform libraries for end users, new SDKs, or provide new runners.
- #61 Beam entered incubation in early February. Quickly did the code donations and began bootstrapping the infrastructure. initial focus is on stabilizing internal APIs and integrating the additional runners. Part of that is understanding what different runners can do...
- #63 Credits?
- #64 Credits?
- #65 Element wise transformations work on individual elements parsing, translating or filtering applied as elements flow past but other transforms like counting or joining require combining multiple elements together ...
- #66 when doing aggregations, need to divide the infinite stream of elements into finite sized chunks that can be processed independently. simplest way using arrival time in fixed time periods can mean elements are being processed out of order, late elements may be aggregated with unrelated elements that arrived about the same time...
- #67 reorganize data base on when they occurred, not when they arrived red element arrived relatively on time and stays in the noon window. green that arrived at 12:30, was actually created about 11:30, so it moves up to the 11am window. requires formalizing the difference between processing time and event time
- #68 if we were aggregating based on processing time, this would result in different results for the two orderings.
- #69 now you can see the sessions being built over time at first we see multiple components in the first session not until late element 9 comes in that we realize it’s one big session
- #70 next -- we’ve seen what the four questions can do. what if we ask the questions twice?
- #71 code to calculate the length of a user session
- #72 Remember that these graphs are always shown per key here’s the graph calculating session legths for Frances and the ones for Tyler
- #73 now lets take those session lengths per user ask the questions again this time using fixed windows to take the mean across the entire collection...
- #74 Now calculating the average length of all sessions that ended in a given time period if we rolled out an update to our game, this would let us quickly understand if that resulted a change in user behavior if the change made the game less fun, we could see a sudden drop in how long users play