http://flink-forward.org/kb_sessions/apache-beam-a-unified-model-for-batch-and-streaming-data-processing/ Unbounded, unordered, global-scale datasets are increasingly common in day-to-day business, and consumers of these datasets have detailed requirements for latency, cost, and completeness. Apache Beam (incubating) defines a new data processing programming model that evolved from more than a decade of experience within Google, including MapReduce, FlumeJava, MillWheel, and Cloud Dataflow. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtimes, both open-source (e.g., Apache Flink, Apache Spark, et al.) and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, touch on its evolution, describe main concepts in the programming model, and compare with similar systems. We’ll go from a simple scenario to a relatively complex data processing pipeline, and finally demonstrate execution of that pipeline on multiple runtimes.

1. Apache Beam (incubating)
Kenneth Knowles
Apache Beam (incubating) PPMC
Software Engineer @ Google
klk@google.com / @KennKnowles Flink Forward 2016
https://goo.gl/jzlvD9
A Unified Model for Batch and Streaming Data Processing

2. What is Apache Beam?
Apache Beam is
a unified programming model
for expressing
efficient and portable
data processing pipelines.

3. Big Data: Infinite & Out of Order
The Beam Model
Beam Project / Technical Vision
Agenda
1
2
3
3

4. 4
Big Data:
Infinite & Out of Order
1

5. https://commons.wikimedia.org/wiki/File:Globe_centered_in_the_Atlantic_Ocean_(green_and_grey_globe_scheme).svg
5

6. 6
Unbounded, delayed, out of order
9:008:00 14:0013:0012:0011:0010:002:001:00 7:006:005:004:003:00
6
8:00
8:008:00

7. Incoming!
Score per
user?
7

8. Organizing the stream
8
8:00
8:00
8:00

9. Completeness Latency Cost
$$$
Data Processing Tradeoffs
9

10. What is important for your application?
Completeness Low Latency Low Cost
Important
Not Important
$$$
10

11. Monthly Billing
Completeness Low Latency Low Cost
Important
Not Important
$$$
11

12. Billing estimate
Completeness Low Latency Low Cost
Important
Not Important
$$$
12

13. Abuse Detection
Completeness Low Latency Low Cost
Important
Not Important
$$$
13

14. 20142004 2006 2008 2010 2012 20162005 2007 2009 2013 20152011
MapReduce
(paper)
Apache
Hadoop
Dataflow Model
(paper)
See also: Tyler Akidau's Evolution of Massive-Scale Data Processing (goo.gl/VlVAEp)
MillWheel
(paper)
Heron
Apache
Spark
Apache
Storm
Apache
Gearpump
(incubating)
Apache
Apex
Apache
Flink
Cloud
Dataflow
FlumeJava
(paper)
Apache Beam
(incubating)
Choices abound
Apache
Samza

15. 15
The Beam Model2

16. The Beam Model
Pipeline
16
PTransform
PCollection
(bounded or
unbounded)

17. The Beam Vision (for users)
Sum Per Key
17
input.apply(
Sum.integersPerKey())
Java
input | Sum.PerKey()
Python
Apache Flink
Apache Spark
Cloud
Dataflow
⋮ ⋮
Apache
Gearpump
(incubating)
Apache Apex

18. Pipeline p = Pipeline.create(options);
p.apply(TextIO.Read.from("gs://dataflow-samples/shakespeare/*"))
.apply(FlatMapElements.via(line → Arrays.asList(line.split("[^a-zA-Z']+"))))
.apply(Filter.byPredicate(word → !word.isEmpty()))
.apply(Count.perElement())
.apply(MapElements.via(count → count.getKey() + ": " + count.getValue())
.apply(TextIO.Write.to("gs://..."));
p.run();
What your (Java) Code Looks Like
18

19. The Beam Model: Asking the Right Questions
What are you computing?
Where in event time?
When in processing time are results produced?
How do refinements relate?
19

20. The Beam Model: Asking the Right Questions
What are you computing?
Where in event time?
When in processing time are results produced?
How do refinements relate?
20
Aggregations,
transformations,
...

21. The Beam Model: What are you computing?
Sum Per
User
21

22. The Beam Model: What are you computing?
Sum Per Key
22
input.apply(Sum.integersPerKey())
.apply(BigQueryIO.Write.to(...));
Java
input | Sum.PerKey()
| Write(BigQuerySink(...))
Python

23. Per element
(ParDo)
Grouping
(Group/Combine Per Key)
Composite
The Beam Model: What are you computing?

24. The Beam Model: What are you computing?
Sum Per Key
24
input.apply(Sum.integersPerKey())
.apply(BigQueryIO.Write.to(...));
Java
input | Sum.PerKey()
| Write(BigQuerySink(...))
Python

25. The Beam Model: Asking the Right Questions
What are you computing?
Where in event time?
When in processing time are results produced?
How do refinements relate?
25
Event time
windowing

26. 26
The Beam Model: Where in Event Time?
8:00
8:00
8:00

27. Processing Time vs Event Time
Event Time = Processing Time ??
27

28. Processing Time vs Event Time
28
ProcessingTime
Event Time

29. ProcessingTime
Processing Time vs Event Time
Realtime
29
This is not possible
Event Time

30. Processing Time vs Event Time
30
Processing Delay
ProcessingTime

31. Processing Time vs Event Time
Very delayed
31
ProcessingTime
Event Time

32. Processing Time windows
(probably are not what you want)
ProcessingTime
Event Time 32

33. Event Time Windows
33
ProcessingTime
Event Time

34. ProcessingTime
Event Time
Event Time Windows
34
(implementing processing time windows)
Just throw away
your data's
timestamps and
replace them with
"now()"

35. input | WindowInto(FixedWindows(3600)
| Sum.PerKey()
| Write(BigQuerySink(...))
Python
The Beam Model: Where in Event Time?
Sum Per Key
Window Into
35
input.apply(
Window.into(
FixedWindows.of(
Duration.standardHours(1)))
.apply(Sum.integersPerKey())
.apply(BigQueryIO.Write.to(...))
Java

36. Fixed Windows
(also called Tumbling)
Sliding Windows
User Sessions
The Beam Model: Where in Event Time?
1. Assign each timestamped
event to one or more
windows
2. Merge those windows
according to custom logic

37. So that's what and where...
37

38. The Beam Model: Asking the Right Questions
What are you computing?
Where in event time?
When in processing time are results produced?
How do refinements relate?
38
Watermarks
& Triggers

39. Event time windows
ProcessingTime
39
Event Time

40. Fixed cutoff (we can do better)
ProcessingTime
Event Time
40
Allowed
delay
Concurrent windows

41. Perfect watermark
ProcessingTime
41
Event Time
Check out Slava's
slides from Strata
London 2016 talk on
watermarks:
https://goo.gl/K4FnqQ

42. Heuristic Watermark
ProcessingTime
42
Event Time

43. Heuristic Watermark
ProcessingTime
43
Current processing time
Event Time

44. Heuristic Watermark
ProcessingTime
44
Current processing time
Event Time

45. Heuristic Watermark
ProcessingTime
45
Current processing time
Late data
Event Time

46. Watermarks measure completeness
46
$$$
$$$
$$
? Running Total
✔ Monthly billing
? Abuse Detection

47. The Beam Model: When in Processing Time?
Sum Per Key
Window Into
47
input
.apply(Window.into(FixedWindows.of(...))
.triggering(
AfterWatermark.pastEndOfWindow()))
.apply(Sum.integersPerKey())
.apply(BigQueryIO.Write.to(...))
Java
input | WindowInto(FixedWindows(3600),
trigger=AfterWatermark())
| Sum.PerKey()
| Write(BigQuerySink(...))
Python
Trigger after end
of window

48. ProcessingTime
Event Time
AfterWatermark.pastEndOfWindow()
48

49. Current processing time
ProcessingTime
Event Time
49
AfterWatermark.pastEndOfWindow()

50. ProcessingTime
Event Time
Late data
50
Current processing time
AfterWatermark.pastEndOfWindow()

51. ProcessingTime
Event Time
51
High completeness
Potentially high latency
Low cost
AfterWatermark.pastEndOfWindow()
$$$

52. ProcessingTime
Event Time
Repeatedly.forever(
AfterPane.elementCountAtLeast(2))
52

53. ProcessingTime
Event Time
53
Current processing time
Repeatedly.forever(
AfterPane.elementCountAtLeast(2))

54. Current processing time
ProcessingTime
Event Time
54
Repeatedly.forever(
AfterPane.elementCountAtLeast(2))

55. ProcessingTime
Event Time
55
Current processing time
Repeatedly.forever(
AfterPane.elementCountAtLeast(2))

56. ProcessingTime
Event Time
56
Repeatedly.forever(
AfterPane.elementCountAtLeast(2))
Low completeness
Low latency
Cost driven by input$$$

57. Build a finely tuned trigger for your use case
AfterWatermark.pastEndOfWindow()
.withEarlyFirings(
AfterProcessingTime
.pastFirstElementInPane()
.plusDuration(Duration.standardMinutes(1))
.withLateFirings(AfterPane.elementCountAtLeast(1))
57
Bill at end of month
Near real-time estimates
Immediate corrections

58. ProcessingTime
Event Time
58
.withEarlyFirings(after 1 minute)
.withLateFirings(ASAP after each element)

59. ProcessingTime
Event Time
59
Current processing time
.withEarlyFirings(after 1 minute)
.withLateFirings(ASAP after each element)

60. ProcessingTime
Event Time
60
Current processing time
Low completeness
Low latency
Low cost, driven by time$$$
.withEarlyFirings(after 1 minute)
.withLateFirings(ASAP after each element)

61. Current processing time
ProcessingTime
Event Time
61
.withEarlyFirings(after 1 minute)
.withLateFirings(ASAP after each element)

62. Current processing time
ProcessingTime
Event Time
Late output
62
.withEarlyFirings(after 1 minute)
.withLateFirings(ASAP after each element)

63. ProcessingTime
Event Time
Late output
63
.withEarlyFirings(after 1 minute)
.withLateFirings(ASAP after each element)

64. Trigger Catalogue
Composite TriggersBasic Triggers
64
AfterEndOfWindow()
AfterCount(n)
AfterProcessingTimeDelay(Δ)
AfterEndOfWindow()
.withEarlyFirings(A)
.withLateFirings(B)
AfterAny(A, B)
AfterAll(A, B)
Repeat(A)
Sequence(A, B)

65. The Beam Model: Asking the Right Questions
What are you computing?
Where in event time?
When in processing time are results produced?
How do refinements relate?
65
Accumulation
Mode

66. 66
The Beam Model: How do refinements relate?
2
5 7 14 25
Window.into(...)
.triggering(...)
.discardingFiredPanes()
5
Window.into(...)
.triggering(...)
.accumulatingFiredPanes()
7
11

67. The Beam Model: Asking the Right Questions
What are you computing?
Where in event time?
When in processing time are results produced?
How do refinements relate?
67

68. 68
Beam Project / Technical Vision3

69. Dataflow → Beam
GoogleCloudPlatform/DataflowJavaSDK
cloudera/spark-dataflow
dataArtisans/flink-dataflow
apache/incubator-beam
Contributors [with GitHub badges] from:
Google, Data Artisans, Cloudera, Talend, Paypal, Spotify, Intel, Twitter,
Capital One, DataTorrent, …, <your org here>

70. End users - who want to write pipelines
in a language that’s familiar.
SDK authors - who want to make Beam
concepts available in new languages.
Runner authors - who have a distributed
processing environment and want to run
Beam pipelines Beam Fn API: Invoke user-definable functions
Apache
Flink
Apache
Spark
Beam Runner API: Build and submit a pipeline
Other
LanguagesBeam Java
Beam
Python
Execution Execution
Cloud
Dataflow
Execution
The Beam Vision
Apache
Apex
Apache
Gearpump
(incubating)

71. Outlook
Dataflow Java 1.x
Apache Beam Java 0.x
Apache Beam Java 2.x
Bug Fix
Feature
Breaking Change
We
are
here
Feb
2016

72. Capability Matrix
http://beam.apache.org/learn/runners/capability-matrix/

73. Unified - One model handles batch and streaming
use cases.
Portable - Pipelines can be executed on multiple
execution environments, avoiding lock-in.
Extensible - Supports user and community driven
SDKs, Runners, transformation libraries, and IO
connectors.
Why Apache Beam?

74. Why Apache Beam?
http://data-artisans.com/why-apache-beam/
"We firmly believe that the Beam model is the
correct programming model for streaming and
batch data processing."
- Kostas Tzoumas (Data Artisans)
https://cloud.google.com/blog/big-data/2016/0
5/why-apache-beam-a-google-perspective
"We hope it will lead to a healthy ecosystem of
sophisticated runners that compete by making
users happy, not [via] API lock in."
- Tyler Akidau (Google)

75. Beam here at Flink Forward 2016
75
I hope you saw:
Beaming Flink to the Cloud @ Netflix
Monal Daxini - Netflix
And stay in this room for:
Flink and Beam: Current State & Roadmap
Maximilian Michels - data Artisans
No shard left behind: Dynamic work rebalancing in Apache Beam
Malo Denielou - Google

76. http://beam.incubator.apache.org/
Join the community!
User discussions - user-subscribe@beam.incubator.apache.org
Development discussions - dev-subscribe@beam.incubator.apache.org
Follow @ApacheBeam on Twitter
Good Reads
Why Apache Beam? (from Data Artisans)
Why Apache Beam? (from Google)
Streaming 101
Streaming 102
The Dataflow Beam Model
More Beam!
76

77. END
77