Download to read offline
![Discarding Accumulation Mode
pcollection | WindowInto(
FixedWindows(1 * 60),
trigger=Repeating(AfterCount(3)),
accumulation_mode=AccumulationMode.DISCARDING)
[5, 8, 3, 1, 2, 6, 9, 7]](https://image.slidesharecdn.com/streamingdatapipelineswithapachebeamato-221115162527-5ab557f5/85/Streaming-Data-Pipelines-With-Apache-Beam-60-320.jpg)
![Discarding Accumulation Mode
pcollection | WindowInto(
FixedWindows(1 * 60),
trigger=Repeating(AfterCount(3)),
accumulation_mode=AccumulationMode.DISCARDING)
[5, 8, 3, 1, 2, 6, 9, 7] -> [5, 8, 3]](https://image.slidesharecdn.com/streamingdatapipelineswithapachebeamato-221115162527-5ab557f5/85/Streaming-Data-Pipelines-With-Apache-Beam-61-320.jpg)
![Discarding Accumulation Mode
pcollection | WindowInto(
FixedWindows(1 * 60),
trigger=Repeating(AfterCount(3)),
accumulation_mode=AccumulationMode.DISCARDING)
[5, 8, 3, 1, 2, 6, 9, 7] -> [5, 8, 3]
[1, 2, 6]](https://image.slidesharecdn.com/streamingdatapipelineswithapachebeamato-221115162527-5ab557f5/85/Streaming-Data-Pipelines-With-Apache-Beam-62-320.jpg)
![Discarding Accumulation Mode
pcollection | WindowInto(
FixedWindows(1 * 60),
trigger=Repeating(AfterCount(3)),
accumulation_mode=AccumulationMode.DISCARDING)
[5, 8, 3, 1, 2, 6, 9, 7] -> [5, 8, 3]
[1, 2, 6]
[9, 7]](https://image.slidesharecdn.com/streamingdatapipelineswithapachebeamato-221115162527-5ab557f5/85/Streaming-Data-Pipelines-With-Apache-Beam-63-320.jpg)
![Discarding Accumulation Mode
pcollection | WindowInto(
FixedWindows(1 * 60),
trigger=Repeating(AfterCount(3)),
accumulation_mode=AccumulationMode.Accumulating)
[5, 8, 3, 1, 2, 6, 9, 7]](https://image.slidesharecdn.com/streamingdatapipelineswithapachebeamato-221115162527-5ab557f5/85/Streaming-Data-Pipelines-With-Apache-Beam-64-320.jpg)
![Discarding Accumulation Mode
pcollection | WindowInto(
FixedWindows(1 * 60),
trigger=Repeating(AfterCount(3)),
accumulation_mode=AccumulationMode.Accumulating)
[5, 8, 3, 1, 2, 6, 9, 7] -> [5, 8, 3]](https://image.slidesharecdn.com/streamingdatapipelineswithapachebeamato-221115162527-5ab557f5/85/Streaming-Data-Pipelines-With-Apache-Beam-65-320.jpg)
![Discarding Accumulation Mode
pcollection | WindowInto(
FixedWindows(1 * 60),
trigger=Repeating(AfterCount(3)),
accumulation_mode=AccumulationMode.Accumulating)
[5, 8, 3, 1, 2, 6, 9, 7] -> [5, 8, 3]
[5, 8, 3, 1, 2, 6]](https://image.slidesharecdn.com/streamingdatapipelineswithapachebeamato-221115162527-5ab557f5/85/Streaming-Data-Pipelines-With-Apache-Beam-66-320.jpg)
![Discarding Accumulation Mode
pcollection | WindowInto(
FixedWindows(1 * 60),
trigger=Repeating(AfterCount(3)),
accumulation_mode=AccumulationMode.Accumulating)
[5, 8, 3, 1, 2, 6, 9, 7] -> [5, 8, 3]
[5, 8, 3, 1, 2, 6]
[5, 8, 3, 1, 2, 6, 9, 7]](https://image.slidesharecdn.com/streamingdatapipelineswithapachebeamato-221115162527-5ab557f5/85/Streaming-Data-Pipelines-With-Apache-Beam-67-320.jpg)
Uploaded byAll Things Open
Streaming Data Pipelines With Apache Beam
The document presents an overview of Apache Beam, a unified model for processing both batch and streaming data. It covers key components such as PCollections, transforms, pipelines, and methods for handling trade-offs between cost, completeness, and latency in data processing. Additionally, it discusses advanced topics like windowing, watermarks, and triggers that are essential for managing real-time data streams.
More Related Content
Similar to Streaming Data Pipelines With Apache Beam
Streaming Data Pipelines With Apache Beam
- 1. Streaming Data Pipelineswith Apache Beam Danny McCormick
- 2. Agenda ● Who amI ● What is Apache Beam ● Beam Basics ● Processing streaming data ● Demo
- 3.
- 4.
- 5. What is ApacheBeam
- 6. In the beginning,there was MapReduce Datastore Map Map Map Map Map Map Reduce Reduce Reduce Reduce Reduce Reduce Shuffle Datastore
- 7. In the beginning,there was MapReduce
- 8. Then came Flume(and Spark, Flink, and many more) Datastore Map Map Datastore Map Group by Key (Reduce) Combine Map Map Combine Datastore Datastore Datastore
- 9. From Flume cameBeam Datastore Map Map Datastore Map Group by Key (Reduce) Combine Map Map Combine Datastore Datastore Datastore
- 10. Unified Model forBatch and Streaming ● Batch processing is a special case of stream processing ● Batch + Stream = Beam
- 11. Build your pipelinein whatever language(s) you want… Group by Key
- 12. … with whateverexecution engine you want Cloud Dataflow Apache Spark Apache Flink Apache Apex Gearpump Apache Samza Apache Nemo (incubating) IBM Streams Group by Key
- 13.
- 14. Terms ● PCollection -distributed multi-element dataset ● Transform - operation that takes N PCollections and produces M PCollections ● Pipeline - directed acyclic graph of Transforms and PCollections
- 15.
- 16. Basic Beam Pipeline defadd_one(element): return element + 1 import apache_beam as beam with beam.Pipeline() as pipeline: pipeline | beam.io.ReadFromText('gs://some/inputData.txt') | beam.Map(add_one) | beam.io.WriteToText('gs://some/outputData') Read Text file Map Transform Write to text file
- 17. How to useBeam to process huge amounts of streaming data
- 18. We want togo from this:
- 19.
- 20.
- 21. Streaming data mightbe: ● Late ● Incomplete ● Rate limited ● Infinite
- 22. You will needto make tradeoffs between: ● Cost ● Completeness ● Low Latency
- 23. Example 1: BillingPipeline Completeness Low Latency Low Cost Important Not Important
- 24. Example 2: BillingEstimator Completeness Low Latency Low Cost ` Important Not Important
- 25. Example 3: FraudDetection Completeness Low Latency Low Cost ` Important Not Important
- 26. Windows 9:00 8:00 14:00 13:00 12:00 11:00 10:00 Aggregate oroutput Aggregate or output Aggregate or output Aggregate or outpu output
- 27. Fixed Windows 9:00 8:00 14:00 13:00 12:00 11:00 10:00 Aggregate oroutput Aggregate or output Aggregate or output Aggregate or output Aggregate or output Aggregate or output Aggregate or output ggregate r output
- 28. Sliding Windows Aggregate oroutput Aggregate or output Aggregate or output Aggregate or output Aggregate or output Aggregate or output output Agg Aggregate or output
- 29. Sliding Windows 9:00 8:00 14:00 13:00 12:00 11:00 10:00 Aggregateor output Aggregate or output ate or output Aggregate or o Aggregate or output Aggregate or output Aggregate or output Aggregate or output
- 30. Session Windows 9:00 8:00 14:00 13:00 12:00 11:00 10:00 Aggregateor output Aggregate or output Aggregate or output Aggregate or output A
- 31.
- 32. Code ● items |beam.WindowInto(window.FixedWindows(60)) # 60s fixed windows ● items | beam.WindowInto(window.SlidingWindows(30, 5)) # 30s sliding window every 5s ● items | beam.WindowInto(window.Sessions(10 * 60)) # window breaks after 10 empty min ● items | beam.WindowInto(window.GlobalWindows()) # single global window
- 33. Real Time vsEvent Time - Expectation Event Time Processing Time
- 34. Real Time vsEvent Time - Reality Event Time Processing Time
- 35. How do weknow its safe to finish a window’s work? Event Time Processing Time
- 36.
- 37. Processing Time? Lotsof late data won’t be counted Event Time Processing Time
- 38. Beam’s Solution -Watermarks! Event Time Processing Time
- 39. Watermarks ● Beam’s notionof when data is complete ● When a watermark passes the end of a window, additional data is late ● Beam has several built in watermark estimators
- 40. Example: Timestamp observingestimation Event Time Processing Time
- 41. Example: Timestamp observingestimation Event Time Processing Time
- 42. Example: Timestamp observingestimation Event Time Processing Time
- 43. Example: Timestamp observingestimation Event Time Processing Time
- 44. Example: Timestamp observingestimation Event Time Processing Time
- 45. Example: Timestamp observingestimation Event Time Processing Time
- 46. Example: Timestamp observingestimation Event Time Processing Time
- 47. Example: Timestamp observingestimation Event Time Processing Time
- 48. Example: Timestamp observingestimation Event Time Processing Time
- 49. Example: Timestamp observingestimation Event Time Processing Time
- 50. Example: Timestamp observingestimation Event Time Processing Time
- 51. Example: Timestamp observingestimation Event Time Processing Time
- 52. Example: Timestamp observingestimation Event Time Processing Time Late Data*
- 53. Watermarks ● Handled atthe source I/O level ● Most pipelines don’t need to implement estimation, but do need to be aware of it
- 54. Recall Tradeoffs Completeness LowLatency Low Cost ` Important Not Important
- 55. Triggers ● Beam’s mechanismfor controlling tradeoffs ● Describe when to emit aggregated results of a single window ● Allow emitting early results or results including late data
- 56. Types of Triggers ●Event Time Triggers ● Processing Time Triggers ● Data-Driven Triggers ● Composite Triggers
- 57. Set on windows pcollection| WindowInto( FixedWindows(1 * 60), trigger=AfterProcessingTime(1 * 60), accumulation_mode=AccumulationMode.DISCARDING)
- 58. Example Triggers ● AfterProcessingTime(delay=1* 60) ● AfterCount(1) ● AfterWatermark( early=AfterProcessingTime(delay=1 * 60), late=AfterCount(1)) ● AfterAny(AfterCount(1), AfterProcessingTime(delay=1 * 60))
- 59. Accumulation Mode ● Describeshow to handle data that has already been emitted ● 2 types: Accumulating and Discarding
- 60. Discarding Accumulation Mode pcollection| WindowInto( FixedWindows(1 * 60), trigger=Repeating(AfterCount(3)), accumulation_mode=AccumulationMode.DISCARDING) [5, 8, 3, 1, 2, 6, 9, 7]
- 61. Discarding Accumulation Mode pcollection| WindowInto( FixedWindows(1 * 60), trigger=Repeating(AfterCount(3)), accumulation_mode=AccumulationMode.DISCARDING) [5, 8, 3, 1, 2, 6, 9, 7] -> [5, 8, 3]
- 62. Discarding Accumulation Mode pcollection| WindowInto( FixedWindows(1 * 60), trigger=Repeating(AfterCount(3)), accumulation_mode=AccumulationMode.DISCARDING) [5, 8, 3, 1, 2, 6, 9, 7] -> [5, 8, 3] [1, 2, 6]
- 63. Discarding Accumulation Mode pcollection| WindowInto( FixedWindows(1 * 60), trigger=Repeating(AfterCount(3)), accumulation_mode=AccumulationMode.DISCARDING) [5, 8, 3, 1, 2, 6, 9, 7] -> [5, 8, 3] [1, 2, 6] [9, 7]
- 64. Discarding Accumulation Mode pcollection| WindowInto( FixedWindows(1 * 60), trigger=Repeating(AfterCount(3)), accumulation_mode=AccumulationMode.Accumulating) [5, 8, 3, 1, 2, 6, 9, 7]
- 65. Discarding Accumulation Mode pcollection| WindowInto( FixedWindows(1 * 60), trigger=Repeating(AfterCount(3)), accumulation_mode=AccumulationMode.Accumulating) [5, 8, 3, 1, 2, 6, 9, 7] -> [5, 8, 3]
- 66. Discarding Accumulation Mode pcollection| WindowInto( FixedWindows(1 * 60), trigger=Repeating(AfterCount(3)), accumulation_mode=AccumulationMode.Accumulating) [5, 8, 3, 1, 2, 6, 9, 7] -> [5, 8, 3] [5, 8, 3, 1, 2, 6]
- 67. Discarding Accumulation Mode pcollection| WindowInto( FixedWindows(1 * 60), trigger=Repeating(AfterCount(3)), accumulation_mode=AccumulationMode.Accumulating) [5, 8, 3, 1, 2, 6, 9, 7] -> [5, 8, 3] [5, 8, 3, 1, 2, 6] [5, 8, 3, 1, 2, 6, 9, 7]
- 68. More! ● Pipeline State ●Timers ● Runner initiated splits ● Self checkpointing ● Bundle finalization
- 69.
- 70. Come join ourcommunity!
- 71.
Editor's Notes
- #5 My path: Studied at Vanderbilt, got Bachelors + Masters Joined Microsoft + worked on Azure DevOps - first started to fall in love with OSS here. Particularly shaped by experiences w/ big OSS repos (GulpJs, Prettier) - maintainers matter! Got to work on GitHub Actions, helped v2 GA, authored most first party actions (setup-node, toolkit) Joined Google to work on Apache Beam and Google’s execution engine, Dataflow. Currently Apache committer and the technical lead of Google’s Beam and Dataflow Machine Learning team. Neat to be part of a bigger community driven project, where decisions are made on the distribution list, not in company meetings. Full circle; I hope to be like those initial OSS maintainers who welcomed me into open source.
- #24 Slide adapted from https://docs.google.com/presentation/d/1SHie3nwe-pqmjGum_QDznPr-B_zXCjJ2VBDGdafZme8/edit#slide=id.g12846a6162_0_2098
- #25 Slide adapted from https://docs.google.com/presentation/d/1SHie3nwe-pqmjGum_QDznPr-B_zXCjJ2VBDGdafZme8/edit#slide=id.g12846a6162_0_2098
- #26 Slide adapted from https://docs.google.com/presentation/d/1SHie3nwe-pqmjGum_QDznPr-B_zXCjJ2VBDGdafZme8/edit#slide=id.g12846a6162_0_2098
- #32 Usually not used in streaming scenarios, unless you’re using specific triggering setups
- #33 Call out its easy to change your aggregation strategy
- #38 Lots of data will be considered late
- #55 Slide adapted from https://docs.google.com/presentation/d/1SHie3nwe-pqmjGum_QDznPr-B_zXCjJ2VBDGdafZme8/edit#slide=id.g12846a6162_0_2098
- #56 Set when you window
- #57 Examples: Event time - afterwatermark Processing time - AfterProcessingTime (early firing) AfterCount
- #71 Highlight areas of growth (ML, x-lang, performance, new SDKs)