Fundamentals of Stream Processing with Apache Beam, Tyler Akidau, F...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Frances Perry & Tyler Akidau
@francesjperry, @ takidau
Apache Beam Committers & Google Engineers
Fundamentals of Stream Pr...

NOTE:
These slides are not being actively maintained.
For up to date presentations on Apache Beam, please see:
beam.incuba...

Infinite, Out-of-Order Data Sets
What, Where, When, How
Reasons This is Awesome
Agenda
Apache Beam (incubating)
2
4
1
3

Infinite, Out-of-Order Data Sets1

Data...

...can be big...

...really, really big...
Tuesday
Wednesday
Thursday

… maybe infinitely big...
9:008:00 14:0013:0012:0011:0010:002:001:00 7:006:005:004:003:00

… with unknown delays.
9:008:00 14:0013:0012:0011:0010:00
8:00
8:008:00

Element-wise transformations
13:00 14:008:00 9:00 10:00 11:00 12:00
Processing
Time

Aggregating via Processing-Time Windows
13:00 14:008:00 9:00 10:00 11:00 12:00
Processing
Time

Aggregating via Event-Time Windows
Event Time
Processing
Time 11:0010:00 15:0014:0013:0012:00
11:0010:00 15:0014:0013:0012...

Reality
Formalizing Event-Time Skew
ProcessingTime
Event Time
Ideal
Skew

Formalizing Event-Time Skew
Watermarks describe event
time progress.
"No timestamp earlier than the
watermark will be seen...

What, Where, When, How2

What are you computing?
Where in event time?
When in processing time?
How do refinements relate?

What are you computing?
Element-Wise Aggregating Composite

What: Computing Integer Sums
// Collection of raw log lines
PCollection<String> raw = IO.read(...);
// Element-wise transf...

What: Computing Integer Sums

What: Computing Integer Sums

Windowing divides data into event-time-based finite chunks.
Often required when doing aggregations over unbounded data.
Wh...

Where: Fixed 2-minute Windows
PCollection<KV<String, Integer>> scores = input
.apply(Window.into(FixedWindows.of(Minutes(2...

Where: Fixed 2-minute Windows

When in processing time?
• Triggers control
when results are
emitted.
• Triggers are often
relative to the
watermark.
Proc...

When: Triggering at the Watermark
PCollection<KV<String, Integer>> scores = input
.apply(Window.into(FixedWindows.of(Minut...

When: Triggering at the Watermark

When: Early and Late Firings
PCollection<KV<String, Integer>> scores = input
.apply(Window.into(FixedWindows.of(Minutes(2)...

When: Early and Late Firings

How do refinements relate?
• How should multiple outputs per window accumulate?
• Appropriate choice depends on consumer.
...

How: Add Newest, Remove Previous
PCollection<KV<String, Integer>> scores = input
.apply(Window.into(FixedWindows.of(Minute...

How: Add Newest, Remove Previous

Reasons This is Awesome3

Correctness
Power
Composability
Flexibility
Modularity
What / Where / When / How

Correctness
Power
Composability
Flexibility
Modularity
What / Where / When / How

Distributed Systems are Distributed

Processing Time Results Differ

Event Time Results are Stable

Correctness
Power
Composability
Flexibility
Modularity
What / Where / When / How

Sessions
PCollection<KV<String, Integer>> scores = input
.apply(Window.into(Sessions.withGapDuration(Minutes(1))
.triggeri...

Identifying Bursts of User Activity

Identifying Bursts of User Activity

Correctness
Power
Composability
Flexibility
Modularity
What / Where / When / How

Calculating Session Lengths
input
.apply(Window.into(Sessions.withGapDuration(Minutes(1)))
.trigger(AtWatermark())
.discar...

Calculating the Average Session Length
.apply(Window.into(FixedWindows.of(Minutes(2)))
.trigger(AtWatermark())
.withEarlyF...

Correctness
Power
Composability
Flexibility
Modularity
What / Where / When / How

1.Classic Batch 2. Batch with Fixed
Windows
3. Streaming
5. Streaming With
Retractions
4. Streaming with
Speculative + Lat...

Correctness
Power
Composability
Flexibility
Modularity
What / Where / When / How

PCollection<KV<String, Integer>> scores = input
.apply(Sum.integersPerKey());
PCollection<KV<String, Integer>> scores = in...

Correctness
Power
Composability
Flexibility
Modularity
What / Where / When / How

Apache Beam (incubating)4

The Evolution of Beam
MapReduce
Google Cloud
Dataflow
Apache
Beam
BigTable DremelColossus
FlumeMegastoreSpanner
PubSub
Mil...

1. The Beam Model: What / Where / When / How
2. SDKs for writing Beam pipelines -- starting with Java
3. Runners for Exist...

1. End users: who want to write
pipelines or transform libraries in
a language that’s familiar.
2. SDK writers: who want t...

Visions are a Journey
02/01/2016
Enter Apache
Incubator
Early 2016
Internal API redesign
Slight Chaos
Mid 2016
API Stabili...

Categorizing Runner Capabilities
http://beam.incubator.apache.org/capability-matrix/

Collaborate - Beam is becoming a community-
driven effort with participation from many
organizations and contributors
Grow...

Learn More!
Apache Beam (incubating)
http://beam.incubator.apache.org
The World Beyond Batch 101 & 102
https://www.oreilly...

Thank you!

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Building a Modern Data Architecture... by Slim Baltagi 9169 views
Hadoop or Spark: is it an either-or... by Slim Baltagi 17968 views
Aljoscha Krettek - Portable statefu... by Ververica 1531 views
Apache Beam: A unified model for ba... by DataWorks Summit/... 22789 views
Analysis-of-Major-Trends-in-big-dat... by Slim Baltagi 8409 views
Kafka Streams for Java enthusiasts by Slim Baltagi 5250 views

View on Slideshare

1 of 60 Ad

View on Slideshare

1 of 60 Ad