DataWorks Summit, profile picture
Uploaded byDataWorks Summit
PPTX, PDF621 views

Foundations of streaming SQL: stream & table theory

The document provides an overview of streaming SQL and time-varying relations. It discusses: 1) How relations evolve over time in streaming SQL, with data divided into time intervals. This allows querying the relation at any point in time. 2) The closure properties of relational algebra still apply to time-varying relations. Operations like filtering and grouping can be performed on intervals of the relation. 3) Streaming SQL extends classic SQL to handle continuous queries over streaming data, represented as time-varying relations divided into time-based intervals.

Embed presentation

Downloaded 26 times
1 Foundations of streaming SQL or: stream & table theory Anton Kedin Apache Beam Contributor Software Engineer at Google Original Presentation by Tyler Akidau, @takidau Covering ideas from across the Apache Beam, Apache Calcite, Apache Kafka, and Apache Flink communities, with thoughts and contributions from Tyler Akidau, Julian Hyde, Fabian Hueske, Shaoxuan Wang, Kenn Knowles, Ben Chambers, Reuven Lax, Mingmin Xu, James Xu, Martin Kleppmann, Jay Kreps and many more, not to mention that whole database community thing... Dataworks San Jose 2018
2 Table of Contents 01 02 Stream & Table Theory A Basics B The Beam Model Streaming SQL A Time-varying relations B SQL language extensions Chapter 7 Chapter 9
3 01 Stream & Table Theory TFW you realize everything you do was invented by the database community decades ago... A Basics B The Beam Model
4 Stream & table basics https://www.confluent.io/blog/making-sense-of-stream-processing/ https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/
5 Special theory of stream & table relativity streams → tables: tables → streams: The aggregation of a stream of updates over time yields a table. The observation of changes to a table over time yields a stream.
6 Non-relativistic stream & table definitions Tables are data at rest. Streams are data in motion.
7 01 Stream & Table Theory TFW you realize everything you do was invented by the database community decades ago... A Basics B The Beam Model
8 The Beam Model What results are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
9 Reconciling streams & tables w/ the Beam Model ● How does batch processing fit into all of this? ● What is the relationship of streams to bounded and unbounded datasets? ● How do the four what, where, when, how questions map onto a streams/tables world?
10 ReduceMap MapReduce input output
11 MapReduce input output MapRead Map MapWrite ReduceRead Reduce ReduceWrite
12 MapReduce MapRead Map MapWrite ReduceRead Reduce ReduceWrite ? ? ? ? ? ??
13 MapReduce MapRead Map MapWrite ReduceRead Reduce ReduceWrite ? ? ? ? ? table table
14 Map phase MapRead Map MapWrite table ? ? ?
15 Map phase API void map(K1 key, V1 value, Emit<K2, V2>);
16 Map phase API void map(K1 key, V1 value, Emit<K2, V2>);
17 Map phase MapRead Map MapWrite table stream ? ?
18 Map phase API void map(K1 key, V1 value, Emit<K2, V2>);
19 Map phase MapRead Map MapWrite table stream stream ?
20 Map phase API void map(K1 key, V1 value, Emit<K2, V2>); void reduce(K2 key, Iterable<V2> value, Emit<V3>);
21 Map phase MapRead Map MapWrite table stream stream table
22 MapReduce MapRead Map MapWrite ReduceRead Reduce ReduceWrite table stream stream table ? ? table
23 Map phase API void map(K1 key, V1 value, Emit<K2, V2>); void reduce(K2 key, Iterable<V2> value, Emit<V3>);
24 Map phase API void map(K1 key, V1 value, Emit<K2, V2>); void reduce(K2 key, Iterable<V2> value, Emit<V3>);
25 table MapReduce MapRead Map MapWrite ReduceRead Reduce ReduceWrite table stream stream table stream stream
26 Reconciling streams & tables w/ the Beam Model ● How does batch processing fit into all of this? ● What is the relationship of streams to bounded and unbounded datasets? ● How do the four what, where, when, how questions map onto a streams/tables world? 1. Tables are read into streams. 2. Streams are processed into new streams until a grouping operation is hit. 3. Grouping turns the stream into a table. 4. Repeat steps 1-3 until you run out of operations.
27 Reconciling streams & tables w/ the Beam Model ● How does batch processing fit into all of this? ● What is the relationship of streams to bounded and unbounded datasets? ● How do the four what, where, when, how questions map onto a streams/tables world? Streams are the in-motion form of data both bounded and unbounded.
28 Reconciling streams & tables w/ the Beam Model ● How does batch processing fit into all of this? ● What is the relationship of streams to bounded and unbounded datasets? ● How do the four what, where, when, how questions map onto a streams/tables world?
29 The Beam Model What results are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
30 The Beam Model What results are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
31 Example data: individual user scores
32 What is calculated? PCollection<KV<Team, Score>> input = IO.read(...) .apply(ParDo.of(new ParseFn()); .apply(Sum.integersPerKey());
What is calculated?
34 The Beam Model What results are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
35 Windowing divides data into event-time-based finite chunks. Often required when doing aggregations over unbounded data. Fixed Sliding 1 2 3 54 Sessions 2 431 Key 2 Key 1 Key 3 Time 2 3 4 Where in event time?
36 Where in event time? PCollection<KV<User, Score>> input = IO.read(...) .apply(ParDo.of(new ParseFn()); .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2))) .apply(Sum.integersPerKey());
Where in event time?
38 The Beam Model What results are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
39 • Triggers control when results are emitted. • Triggers are often relative to the watermark. ProcessingTime Event Time ~Watermark Ideal Skew When in processing time?
40 When in processing time? PCollection<KV<User, Score>> input = IO.read(...) .apply(ParDo.of(new ParseFn()); .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2)) .triggering(AtWatermark()) .apply(Sum.integersPerKey());
When in processing time?
42 The Beam Model: asking the right questions What results are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
43 How do refinements relate? PCollection<KV<User, Score>> input = IO.read(...) .apply(ParDo.of(new ParseFn()); .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2)) .triggering(AtWatermark().withLateFirings(AtCount(1))) .accumulatingFiredPanes()) .apply(Sum.integersPerKey());
How do refinements relate?
45 What/Where/When/How Summary 3. Streaming 4. Streaming + Late Data Handling 1. Classic Batch 2. Windowed Batch
46 Reconciling streams & tables w/ the Beam Model ● How does batch processing fit into all of this? ● What is the relationship of streams to bounded and unbounded datasets? ● How do the four what, where, when, how questions map onto a streams/tables world?
47 General theory of stream & table relativity Pipelines : tables + streams + operations Tables : data at rest Streams : data in motion Operations : (stream | table) → (stream | table) transformations ● stream → stream: Non-grouping (element-wise) operations Leaves stream data in motion, yielding another stream. ● stream → table: Grouping operations Brings stream data to rest, yielding a table. Windowing adds the dimension of time to grouping. ● table → stream: Ungrouping (triggering) operations Puts table data into motion, yielding a stream. Accumulation dictates the nature of the stream (deltas, values, retractions). ● table → table: (none) Impossible to go from rest and back to rest without being put into motion.
48 02 Streaming SQL Contorting relational algebra for fun and profit A Time-varying relations B SQL language extensions
49 Relational algebra User Score Time Julie 7 12:01 Frank 3 12:03 Julie 1 12:03 Julie 4 12:07 Score Time 7 12:01 3 12:03 1 12:03 4 12:07 πScore,Time(UserScores)πUserScoresπ SELECT Score, Time FROM UserScores; ----------------- | Score | Time | ----------------- | 7 | 12:01 | | 3 | 12:03 | | 1 | 12:03 | | 4 | 12:07 | ----------------- Relational algebra SQLRelation
50 Relations evolve over time 12:07> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 1 | 12:03 | | Julie | 4 | 12:07 | ------------------------- 12:03> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 1 | 12:03 | ------------------------- 12:00> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- ------------------------- 12:01> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | -------------------------
51 Classic SQL vs Streaming SQL Classic SQL classic relations single point in time:: :: Streaming SQL time-varying relations every point in time:: ::
52 Classic SQL vs Streaming SQL Classic SQL classic relations single point in time:: :: Streaming SQL time-varying relations every point in time:: ::
53 Classic relations 12:07> SELECT * FROM UserScores; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | | | | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:07> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 1 | 12:03 | | Julie | 4 | 12:07 | ------------------------- 12:03> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 1 | 12:03 | ------------------------- 12:00> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- ------------------------- 12:01> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | ------------------------- Time-varying relation
54 Closure property of relational algebra remains intact with time-varying relations.
55 Time-varying relations: variations 12:07> SELECT * FROM UserScores; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | | | | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | -----------------------------------------------------------------------------------------------------------------
56 Time-varying relations: filtering 12:07> SELECT * FROM UserScores; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | | | | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:07> SELECT * FROM UserScores WHERE Name = “Julie”; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | | | | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | -----------------------------------------------------------------------------------------------------------------
57 Time-varying relations: grouping 12:07> SELECT * FROM UserScores; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | | | | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | -----------------------------------------------------------------------------------------------------------------
58 How does this relate to streams & tables?
59 Time-varying relations: tables 12:07> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 12 | 12:07 | | Frank | 3 | 12:03 | ------------------------- 12:03> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 8 | 12:03 | | Frank | 3 | 12:03 | ------------------------- 12:01> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | ------------------------- 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- -------------------------
60 Time-varying relations: tables 12:01> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | ------------------------- 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:07> SELECT TABLE Name, SUM(Score), MAX(Time) AS OF SYSTEM TIME ‘12:01’ FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | -------------------------
61 Time-varying relations: tables 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- ... 12:00 12:00> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- -------------------------
62 Time-varying relations: streams 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | ... 12:01 12:00> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- -------------------------
63 Time-varying relations: streams 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | ... 12:01 12:01> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | -------------------------
64 Time-varying relations: streams 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 8 | 12:03 | ... 12:03 12:01> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | -------------------------
65 Time-varying relations: streams 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 8 | 12:03 | ... 12:03 12:03> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 8 | 12:03 | | Frank | 3 | 12:03 | -------------------------
66 Time-varying relations: streams 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 8 | 12:03 | | Julie | 12 | 12:07 | ... 12:07 12:03> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 8 | 12:03 | | Frank | 3 | 12:03 | -------------------------
67 Time-varying relations: streams 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 8 | 12:03 | | Julie | 12 | 12:07 | ... 12:07 12:07> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 12 | 12:07 | | Frank | 3 | 12:03 | -------------------------
68 How does this relate to streams & tables? capture a point-in-time snapshot of a time-varying relation. capture the evolution of a time- varying relation over time. Tables Streams
69 02 Streaming SQL Contorting relational algebra for fun and profit A Time-varying relations B SQL language extensions
70 When do you need SQL extensions for streaming? As a table: As a stream: SQL extensions rarely needed. SQL extensions sometimes needed. How is output consumed? good defaults = often not needed
71 When do you need SQL extensions for streaming?* Explicit table / stream selection ● SELECT TABLE * from X; ● SELECT STREAM * from X; Timestamps and windowing ● Event-time columns ● Windowing. E.g., SELECT * FROM X GROUP BY SESSION(<COLUMN> INTERVAL '5' MINUTE); ○ Grouping by timestamp ○ Complex multi-row transactions inexpressible in declarative SQL (e.g., session windows) Sane default table / stream selection ● If all inputs are tables, output is a table ● If any inputs are streams, output is a stream Simple triggers ● Implicitly defined by characteristics of the sink ● Optionally be configured outside of query. ● Per-query, e.g.: SELECT * from X EMIT <WHEN>; ● Focused set of use cases: ○ Repeated updates ... EMIT AFTER <TIMEDELTA> ○ Completeness ... EMIT WHEN WATERMARK PAST <COLUMN> ○ Repeated updates + completeness (e.g., early/on-time/late pattern) ... EMIT AFTER <TIMEDELTA> AND WHEN WATERMARK PAST <COLUMN> * Most of these extensions are theoretical at this point; very few have concrete implementations.
72 Summary streams ⇄ tables streams & tables : Beam Model time-varying relations SQL language extensions
73 Thank you! In early release now streamingsystems.net Streaming SQL spec (WIP: Apex, Beam, Calcite, Flink): http://s.apache.org/streaming-sql-spec Streaming in Calcite (Julian Hyde): https://calcite.apache.org/docs/stream.html Streams, joins & temporal tables (Julian Hyde): http://s.apache.org/streams-joins-and-temporal-tables Streaming 101: http://oreilly.com/ideas/the-world-beyond-batch-streaming-101 Streaming 102: http://oreilly.com/ideas/the-world-beyond-batch-streaming-102 Animations: http://github.com/takidau/animations Apache Beam: http://beam.apache.org Apache Calcite: http://calcite.apache.org Apache Flink: http://flink.apache.org

More Related Content

PPTX
Unified Batch & Stream Processing with Apache Samza
byDataWorks Summit
 
PPTX
Apache Hadoop YARN 3.x in Alibaba
byDataWorks Summit
 
PPTX
Ozone: scaling HDFS to trillions of objects
byDataWorks Summit
 
PDF
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
byDatabricks
 
PPTX
Practice of large Hadoop cluster in China Mobile
byDataWorks Summit
 
PPTX
Running secured Spark job in Kubernetes compute cluster and integrating with ...
byDataWorks Summit
 
PDF
Serverless Data Platform
byShu-Jeng Hsieh
 
PDF
IEEE International Conference on Data Engineering 2015
byYousun Jeong
 
Unified Batch & Stream Processing with Apache Samza
byDataWorks Summit
 
Apache Hadoop YARN 3.x in Alibaba
byDataWorks Summit
 
Ozone: scaling HDFS to trillions of objects
byDataWorks Summit
 
Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...
byDatabricks
 
Practice of large Hadoop cluster in China Mobile
byDataWorks Summit
 
Running secured Spark job in Kubernetes compute cluster and integrating with ...
byDataWorks Summit
 
Serverless Data Platform
byShu-Jeng Hsieh
 
IEEE International Conference on Data Engineering 2015
byYousun Jeong
 

What's hot

PDF
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
byDataWorks Summit
 
PPTX
Flexible and Real-Time Stream Processing with Apache Flink
byDataWorks Summit
 
PDF
Building Event Streaming Architectures on Scylla and Kafka
byScyllaDB
 
PDF
Performance Troubleshooting Using Apache Spark Metrics
byDatabricks
 
PDF
Scaling Data Analytics Workloads on Databricks
byDatabricks
 
PPTX
What's the Hadoop-la about Kubernetes?
byDataWorks Summit
 
PDF
Deep Dive into GPU Support in Apache Spark 3.x
byDatabricks
 
PPTX
How to overcome mysterious problems caused by large and multi-tenancy Hadoop ...
byDataWorks Summit/Hadoop Summit
 
PDF
SOS: Optimizing Shuffle I/O with Brian Cho and Ergin Seyfe
byDatabricks
 
PDF
Optimizing Performance and Computing Resource Efficiency of In-Memory Big Dat...
byDatabricks
 
PPTX
Rich placement constraints: Who said YARN cannot schedule services?
byDataWorks Summit
 
PDF
Parallelization of Structured Streaming Jobs Using Delta Lake
byDatabricks
 
PPTX
Building Value Within the Heavy Vehicle Industry Using Big Data and Streaming...
byDataWorks Summit
 
PPTX
Realtime olap architecture in apache kylin 3.0
byShi Shao Feng
 
PDF
Why is My Stream Processing Job Slow? with Xavier Leaute
byDatabricks
 
PPTX
Bridging the gap of Relational to Hadoop using Sqoop @ Expedia
byDataWorks Summit/Hadoop Summit
 
PPTX
Exploiting machine learning to keep Hadoop clusters healthy
byDataWorks Summit
 
PDF
Develop Scalable Applications with DataStax Drivers (Alex Popescu, Bulat Shak...
byDataStax
 
PPTX
Hadoop engineering bo_f_final
byRamya Sunil
 
PPTX
Enterprise Grade Streaming under 2ms on Hadoop
byDataWorks Summit/Hadoop Summit
 
Apache Druid Auto Scale-out/in for Streaming Data Ingestion on Kubernetes
byDataWorks Summit
 
Flexible and Real-Time Stream Processing with Apache Flink
byDataWorks Summit
 
Building Event Streaming Architectures on Scylla and Kafka
byScyllaDB
 
Performance Troubleshooting Using Apache Spark Metrics
byDatabricks
 
Scaling Data Analytics Workloads on Databricks
byDatabricks
 
What's the Hadoop-la about Kubernetes?
byDataWorks Summit
 
Deep Dive into GPU Support in Apache Spark 3.x
byDatabricks
 
How to overcome mysterious problems caused by large and multi-tenancy Hadoop ...
byDataWorks Summit/Hadoop Summit
 
SOS: Optimizing Shuffle I/O with Brian Cho and Ergin Seyfe
byDatabricks
 
Optimizing Performance and Computing Resource Efficiency of In-Memory Big Dat...
byDatabricks
 
Rich placement constraints: Who said YARN cannot schedule services?
byDataWorks Summit
 
Parallelization of Structured Streaming Jobs Using Delta Lake
byDatabricks
 
Building Value Within the Heavy Vehicle Industry Using Big Data and Streaming...
byDataWorks Summit
 
Realtime olap architecture in apache kylin 3.0
byShi Shao Feng
 
Why is My Stream Processing Job Slow? with Xavier Leaute
byDatabricks
 
Bridging the gap of Relational to Hadoop using Sqoop @ Expedia
byDataWorks Summit/Hadoop Summit
 
Exploiting machine learning to keep Hadoop clusters healthy
byDataWorks Summit
 
Develop Scalable Applications with DataStax Drivers (Alex Popescu, Bulat Shak...
byDataStax
 
Hadoop engineering bo_f_final
byRamya Sunil
 
Enterprise Grade Streaming under 2ms on Hadoop
byDataWorks Summit/Hadoop Summit
 

Similar to Foundations of streaming SQL: stream & table theory

PPTX
Flink Forward SF 2017: Timo Walther - Table & SQL API – unified APIs for bat...
byFlink Forward
 
PDF
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
byJulian Hyde
 
PPTX
Why and how to leverage the power and simplicity of SQL on Apache Flink
byFabian Hueske
 
PDF
Timo Walther - Table & SQL API - unified APIs for batch and stream processing
byVerverica
 
PDF
Apache Flink's Table & SQL API - unified APIs for batch and stream processing
byTimo Walther
 
PDF
Streaming SQL
byJulian Hyde
 
PDF
SQL Extensions to Support Streaming Data With Fabian Hueske | Current 2022
byHostedbyConfluent
 
PDF
Streaming SQL
byJulian Hyde
 
PDF
Streaming SQL
byDataWorks Summit/Hadoop Summit
 
PDF
Streaming SQL
byJulian Hyde
 
PDF
Streams and Tables: Two Sides of the Same Coin (BIRTE 2018)
byconfluent
 
PDF
Julian Hyde - Streaming SQL
byFlink Forward
 
PDF
Streaming SQL (at FlinkForward, Berlin, 2016/09/12)
byJulian Hyde
 
PPT
Big Data Unit- 4 Powerpoint Presentation
bysureshkumartt1
 
PDF
Streaming SQL Foundations: Why I ❤ Streams+Tables
byC4Media
 
PPTX
Fabian Hueske - Stream Analytics with SQL on Apache Flink
byVerverica
 
PDF
Streaming SQL
byJulian Hyde
 
PDF
A Brief History of Stream Processing
byAleksandr Kuboskin, CFA
 
PDF
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
byDataWorks Summit/Hadoop Summit
 
PDF
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
byJulian Hyde
 
Flink Forward SF 2017: Timo Walther - Table & SQL API – unified APIs for bat...
byFlink Forward
 
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
byJulian Hyde
 
Why and how to leverage the power and simplicity of SQL on Apache Flink
byFabian Hueske
 
Timo Walther - Table & SQL API - unified APIs for batch and stream processing
byVerverica
 
Apache Flink's Table & SQL API - unified APIs for batch and stream processing
byTimo Walther
 
Streaming SQL
byJulian Hyde
 
SQL Extensions to Support Streaming Data With Fabian Hueske | Current 2022
byHostedbyConfluent
 
Streaming SQL
byJulian Hyde
 
Streaming SQL
byDataWorks Summit/Hadoop Summit
 
Streaming SQL
byJulian Hyde
 
Streams and Tables: Two Sides of the Same Coin (BIRTE 2018)
byconfluent
 
Julian Hyde - Streaming SQL
byFlink Forward
 
Streaming SQL (at FlinkForward, Berlin, 2016/09/12)
byJulian Hyde
 
Big Data Unit- 4 Powerpoint Presentation
bysureshkumartt1
 
Streaming SQL Foundations: Why I ❤ Streams+Tables
byC4Media
 
Fabian Hueske - Stream Analytics with SQL on Apache Flink
byVerverica
 
Streaming SQL
byJulian Hyde
 
A Brief History of Stream Processing
byAleksandr Kuboskin, CFA
 
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
byDataWorks Summit/Hadoop Summit
 
Querying the Internet of Things: Streaming SQL on Kafka/Samza and Storm/Trident
byJulian Hyde
 

More from DataWorks Summit

PPTX
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
byDataWorks Summit
 
PPTX
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
byDataWorks Summit
 
PPTX
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
byDataWorks Summit
 
PDF
Presto: Optimizing Performance of SQL-on-Anything Engine
byDataWorks Summit
 
PPTX
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
byDataWorks Summit
 
PPTX
Extending Twitter's Data Platform to Google Cloud
byDataWorks Summit
 
PPTX
Supporting Apache HBase : Troubleshooting and Supportability Improvements
byDataWorks Summit
 
PPTX
Floating on a RAFT: HBase Durability with Apache Ratis
byDataWorks Summit
 
PPTX
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
byDataWorks Summit
 
PPTX
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
byDataWorks Summit
 
PPTX
HBase Global Indexing to support large-scale data ingestion at Uber
byDataWorks Summit
 
PPTX
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
byDataWorks Summit
 
PPTX
Practical NoSQL: Accumulo's dirlist Example
byDataWorks Summit
 
PPTX
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
byDataWorks Summit
 
PPTX
Data Science Crash Course
byDataWorks Summit
 
PPTX
Managing the Dewey Decimal System
byDataWorks Summit
 
PPTX
Security Framework for Multitenant Architecture
byDataWorks Summit
 
PPTX
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
byDataWorks Summit
 
PDF
HBase Tales From the Trenches - Short stories about most common HBase operati...
byDataWorks Summit
 
PDF
Computer Vision: Coming to a Store Near You
byDataWorks Summit
 
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
byDataWorks Summit
 
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
byDataWorks Summit
 
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
byDataWorks Summit
 
Presto: Optimizing Performance of SQL-on-Anything Engine
byDataWorks Summit
 
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
byDataWorks Summit
 
Extending Twitter's Data Platform to Google Cloud
byDataWorks Summit
 
Supporting Apache HBase : Troubleshooting and Supportability Improvements
byDataWorks Summit
 
Floating on a RAFT: HBase Durability with Apache Ratis
byDataWorks Summit
 
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
byDataWorks Summit
 
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
byDataWorks Summit
 
HBase Global Indexing to support large-scale data ingestion at Uber
byDataWorks Summit
 
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
byDataWorks Summit
 
Practical NoSQL: Accumulo's dirlist Example
byDataWorks Summit
 
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
byDataWorks Summit
 
Data Science Crash Course
byDataWorks Summit
 
Managing the Dewey Decimal System
byDataWorks Summit
 
Security Framework for Multitenant Architecture
byDataWorks Summit
 
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
byDataWorks Summit
 
HBase Tales From the Trenches - Short stories about most common HBase operati...
byDataWorks Summit
 
Computer Vision: Coming to a Store Near You
byDataWorks Summit
 

Recently uploaded

PDF
One String, Many Prompts: AI Code Generation
byGeoffrey Goetz
 
PPT
Database Management Systems: Basics, History, Data Models, etc.
byParithi Thamizh
 
PDF
UiPath Coded Agents - A Developer Driven Agentic Automation Era
byUiPathCommunity
 
PPTX
apidays Australia 2025 | Building AI RAG Applications with No Code.pptx
byapidays
 
PDF
Safer’s Picks: Recent FME Enhancements with Big Everyday Impact
bySafe Software
 
PPTX
AI Meets DBA Transforming Database Administration with Intelligence
byGeoPITS Global Pvt Ltd
 
PDF
Teaching Robots how to Read 1/2: AI Center & Classic Document Understanding (...
byanabulhac
 
PDF
Pervasive Encryption - Keeping Data Secure.pdf
byPrecisely
 
PDF
Adding Copilot+ PCs with Snapdragon to your HP fleet won’t require IT deploym...
byPrincipled Technologies
 
PPTX
Microsoft Azure News - January 2026 - BAUG
byDaniel Toomey
 
PDF
MINDCTI Revenue Release Quarter 3 2025 - Financial Presentation
byMIND CTI
 
PPTX
Spacewalk 2026 - Presented at the IMAX in Raleigh, NC
byAll Things Open
 
PPTX
The Abomination of AI – keynote at ICoSCI 2026
byAlan Dix
 
PDF
Computer Science Conferences 2026 | Research by SAI Conference
byHealth Conference 2026
 
PPTX
Robot Vacuums are Cleaning Up. The global robot vacuum segment has high marke...
byRobert March
 
PDF
Transcript: What ONIX can do: Leveraging metadata to support the discoverabil...
byBookNet Canada
 
PPT
Telecommunication system introduction to wireless communication
by143irha
 
PDF
TopMate ES11 3-Wheel Electric Scooter: Comfortable, Stable Mobility for Every...
byTopmate
 
PDF
Special Reeling Cable Specification _ Anhui Feichun Special Cable Co., Ltd_.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
PPTX
computer science class 11 ( phishing and fraud mails) .pptx
bybalajichess314
 
One String, Many Prompts: AI Code Generation
byGeoffrey Goetz
 
Database Management Systems: Basics, History, Data Models, etc.
byParithi Thamizh
 
UiPath Coded Agents - A Developer Driven Agentic Automation Era
byUiPathCommunity
 
apidays Australia 2025 | Building AI RAG Applications with No Code.pptx
byapidays
 
Safer’s Picks: Recent FME Enhancements with Big Everyday Impact
bySafe Software
 
AI Meets DBA Transforming Database Administration with Intelligence
byGeoPITS Global Pvt Ltd
 
Teaching Robots how to Read 1/2: AI Center & Classic Document Understanding (...
byanabulhac
 
Pervasive Encryption - Keeping Data Secure.pdf
byPrecisely
 
Adding Copilot+ PCs with Snapdragon to your HP fleet won’t require IT deploym...
byPrincipled Technologies
 
Microsoft Azure News - January 2026 - BAUG
byDaniel Toomey
 
MINDCTI Revenue Release Quarter 3 2025 - Financial Presentation
byMIND CTI
 
Spacewalk 2026 - Presented at the IMAX in Raleigh, NC
byAll Things Open
 
The Abomination of AI – keynote at ICoSCI 2026
byAlan Dix
 
Computer Science Conferences 2026 | Research by SAI Conference
byHealth Conference 2026
 
Robot Vacuums are Cleaning Up. The global robot vacuum segment has high marke...
byRobert March
 
Transcript: What ONIX can do: Leveraging metadata to support the discoverabil...
byBookNet Canada
 
Telecommunication system introduction to wireless communication
by143irha
 
TopMate ES11 3-Wheel Electric Scooter: Comfortable, Stable Mobility for Every...
byTopmate
 
Special Reeling Cable Specification _ Anhui Feichun Special Cable Co., Ltd_.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
computer science class 11 ( phishing and fraud mails) .pptx
bybalajichess314
 

Foundations of streaming SQL: stream & table theory

  • 1.
    1 Foundations of streamingSQL or: stream & table theory Anton Kedin Apache Beam Contributor Software Engineer at Google Original Presentation by Tyler Akidau, @takidau Covering ideas from across the Apache Beam, Apache Calcite, Apache Kafka, and Apache Flink communities, with thoughts and contributions from Tyler Akidau, Julian Hyde, Fabian Hueske, Shaoxuan Wang, Kenn Knowles, Ben Chambers, Reuven Lax, Mingmin Xu, James Xu, Martin Kleppmann, Jay Kreps and many more, not to mention that whole database community thing... Dataworks San Jose 2018
  • 2.
    2 Table of Contents 01 02 Stream& Table Theory A Basics B The Beam Model Streaming SQL A Time-varying relations B SQL language extensions Chapter 7 Chapter 9
  • 3.
    3 01 Stream &Table Theory TFW you realize everything you do was invented by the database community decades ago... A Basics B The Beam Model
  • 4.
    4 Stream & tablebasics https://www.confluent.io/blog/making-sense-of-stream-processing/ https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/
  • 5.
    5 Special theory ofstream & table relativity streams → tables: tables → streams: The aggregation of a stream of updates over time yields a table. The observation of changes to a table over time yields a stream.
  • 6.
    6 Non-relativistic stream &table definitions Tables are data at rest. Streams are data in motion.
  • 7.
    7 01 Stream &Table Theory TFW you realize everything you do was invented by the database community decades ago... A Basics B The Beam Model
  • 8.
    8 The Beam Model Whatresults are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
  • 9.
    9 Reconciling streams &tables w/ the Beam Model ● How does batch processing fit into all of this? ● What is the relationship of streams to bounded and unbounded datasets? ● How do the four what, where, when, how questions map onto a streams/tables world?
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
    15 Map phase API voidmap(K1 key, V1 value, Emit<K2, V2>);
  • 16.
    16 Map phase API voidmap(K1 key, V1 value, Emit<K2, V2>);
  • 17.
  • 18.
    18 Map phase API voidmap(K1 key, V1 value, Emit<K2, V2>);
  • 19.
  • 20.
    20 Map phase API voidmap(K1 key, V1 value, Emit<K2, V2>); void reduce(K2 key, Iterable<V2> value, Emit<V3>);
  • 21.
  • 22.
  • 23.
    23 Map phase API voidmap(K1 key, V1 value, Emit<K2, V2>); void reduce(K2 key, Iterable<V2> value, Emit<V3>);
  • 24.
    24 Map phase API voidmap(K1 key, V1 value, Emit<K2, V2>); void reduce(K2 key, Iterable<V2> value, Emit<V3>);
  • 25.
  • 26.
    26 Reconciling streams &tables w/ the Beam Model ● How does batch processing fit into all of this? ● What is the relationship of streams to bounded and unbounded datasets? ● How do the four what, where, when, how questions map onto a streams/tables world? 1. Tables are read into streams. 2. Streams are processed into new streams until a grouping operation is hit. 3. Grouping turns the stream into a table. 4. Repeat steps 1-3 until you run out of operations.
  • 27.
    27 Reconciling streams &tables w/ the Beam Model ● How does batch processing fit into all of this? ● What is the relationship of streams to bounded and unbounded datasets? ● How do the four what, where, when, how questions map onto a streams/tables world? Streams are the in-motion form of data both bounded and unbounded.
  • 28.
    28 Reconciling streams &tables w/ the Beam Model ● How does batch processing fit into all of this? ● What is the relationship of streams to bounded and unbounded datasets? ● How do the four what, where, when, how questions map onto a streams/tables world?
  • 29.
    29 The Beam Model Whatresults are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
  • 30.
    30 The Beam Model Whatresults are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
  • 31.
  • 32.
    32 What is calculated? PCollection<KV<Team,Score>> input = IO.read(...) .apply(ParDo.of(new ParseFn()); .apply(Sum.integersPerKey());
  • 33.
  • 34.
    34 The Beam Model Whatresults are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
  • 35.
    35 Windowing divides datainto event-time-based finite chunks. Often required when doing aggregations over unbounded data. Fixed Sliding 1 2 3 54 Sessions 2 431 Key 2 Key 1 Key 3 Time 2 3 4 Where in event time?
  • 36.
    36 Where in eventtime? PCollection<KV<User, Score>> input = IO.read(...) .apply(ParDo.of(new ParseFn()); .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2))) .apply(Sum.integersPerKey());
  • 37.
  • 38.
    38 The Beam Model Whatresults are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
  • 39.
    39 • Triggers control whenresults are emitted. • Triggers are often relative to the watermark. ProcessingTime Event Time ~Watermark Ideal Skew When in processing time?
  • 40.
    40 When in processingtime? PCollection<KV<User, Score>> input = IO.read(...) .apply(ParDo.of(new ParseFn()); .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2)) .triggering(AtWatermark()) .apply(Sum.integersPerKey());
  • 41.
  • 42.
    42 The Beam Model:asking the right questions What results are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
  • 43.
    43 How do refinementsrelate? PCollection<KV<User, Score>> input = IO.read(...) .apply(ParDo.of(new ParseFn()); .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2)) .triggering(AtWatermark().withLateFirings(AtCount(1))) .accumulatingFiredPanes()) .apply(Sum.integersPerKey());
  • 44.
  • 45.
    45 What/Where/When/How Summary 3. Streaming4. Streaming + Late Data Handling 1. Classic Batch 2. Windowed Batch
  • 46.
    46 Reconciling streams &tables w/ the Beam Model ● How does batch processing fit into all of this? ● What is the relationship of streams to bounded and unbounded datasets? ● How do the four what, where, when, how questions map onto a streams/tables world?
  • 47.
    47 General theory ofstream & table relativity Pipelines : tables + streams + operations Tables : data at rest Streams : data in motion Operations : (stream | table) → (stream | table) transformations ● stream → stream: Non-grouping (element-wise) operations Leaves stream data in motion, yielding another stream. ● stream → table: Grouping operations Brings stream data to rest, yielding a table. Windowing adds the dimension of time to grouping. ● table → stream: Ungrouping (triggering) operations Puts table data into motion, yielding a stream. Accumulation dictates the nature of the stream (deltas, values, retractions). ● table → table: (none) Impossible to go from rest and back to rest without being put into motion.
  • 48.
    48 02 Streaming SQL Contortingrelational algebra for fun and profit A Time-varying relations B SQL language extensions
  • 49.
    49 Relational algebra User ScoreTime Julie 7 12:01 Frank 3 12:03 Julie 1 12:03 Julie 4 12:07 Score Time 7 12:01 3 12:03 1 12:03 4 12:07 πScore,Time(UserScores)πUserScoresπ SELECT Score, Time FROM UserScores; ----------------- | Score | Time | ----------------- | 7 | 12:01 | | 3 | 12:03 | | 1 | 12:03 | | 4 | 12:07 | ----------------- Relational algebra SQLRelation
  • 50.
    50 Relations evolve overtime 12:07> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 1 | 12:03 | | Julie | 4 | 12:07 | ------------------------- 12:03> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 1 | 12:03 | ------------------------- 12:00> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- ------------------------- 12:01> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | -------------------------
  • 51.
    51 Classic SQL vsStreaming SQL Classic SQL classic relations single point in time:: :: Streaming SQL time-varying relations every point in time:: ::
  • 52.
    52 Classic SQL vsStreaming SQL Classic SQL classic relations single point in time:: :: Streaming SQL time-varying relations every point in time:: ::
  • 53.
    53 Classic relations 12:07> SELECT* FROM UserScores; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | | | | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:07> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 1 | 12:03 | | Julie | 4 | 12:07 | ------------------------- 12:03> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 1 | 12:03 | ------------------------- 12:00> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- ------------------------- 12:01> SELECT * FROM UserScores; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | ------------------------- Time-varying relation
  • 54.
    54 Closure property ofrelational algebra remains intact with time-varying relations.
  • 55.
    55 Time-varying relations: variations 12:07>SELECT * FROM UserScores; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | | | | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | -----------------------------------------------------------------------------------------------------------------
  • 56.
    56 Time-varying relations: filtering 12:07>SELECT * FROM UserScores; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | | | | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:07> SELECT * FROM UserScores WHERE Name = “Julie”; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | | | | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | -----------------------------------------------------------------------------------------------------------------
  • 57.
    57 Time-varying relations: grouping 12:07>SELECT * FROM UserScores; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | | | | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | -----------------------------------------------------------------------------------------------------------------
  • 58.
    58 How does thisrelate to streams & tables?
  • 59.
    59 Time-varying relations: tables 12:07>SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 12 | 12:07 | | Frank | 3 | 12:03 | ------------------------- 12:03> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 8 | 12:03 | | Frank | 3 | 12:03 | ------------------------- 12:01> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | ------------------------- 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- -------------------------
  • 60.
    60 Time-varying relations: tables 12:01>SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | ------------------------- 12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:07> SELECT TABLE Name, SUM(Score), MAX(Time) AS OF SYSTEM TIME ‘12:01’ FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | -------------------------
  • 61.
    61 Time-varying relations: tables 12:07>SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- ... 12:00 12:00> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- -------------------------
  • 62.
    62 Time-varying relations: streams 12:07>SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | ... 12:01 12:00> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- -------------------------
  • 63.
    63 Time-varying relations: streams 12:07>SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | ... 12:01 12:01> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | -------------------------
  • 64.
    64 Time-varying relations: streams 12:07>SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 8 | 12:03 | ... 12:03 12:01> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | -------------------------
  • 65.
    65 Time-varying relations: streams 12:07>SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 8 | 12:03 | ... 12:03 12:03> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 8 | 12:03 | | Frank | 3 | 12:03 | -------------------------
  • 66.
    66 Time-varying relations: streams 12:07>SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 8 | 12:03 | | Julie | 12 | 12:07 | ... 12:07 12:03> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 8 | 12:03 | | Frank | 3 | 12:03 | -------------------------
  • 67.
    67 Time-varying relations: streams 12:07>SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ----------------------------------------------------------------------------------------------------------------- | [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | | | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | | | | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | | | ------------------------- | ------------------------- | ------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- 12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 7 | 12:01 | | Frank | 3 | 12:03 | | Julie | 8 | 12:03 | | Julie | 12 | 12:07 | ... 12:07 12:07> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name; ------------------------- | Name | Score | Time | ------------------------- | Julie | 12 | 12:07 | | Frank | 3 | 12:03 | -------------------------
  • 68.
    68 How does thisrelate to streams & tables? capture a point-in-time snapshot of a time-varying relation. capture the evolution of a time- varying relation over time. Tables Streams
  • 69.
    69 02 Streaming SQL Contortingrelational algebra for fun and profit A Time-varying relations B SQL language extensions
  • 70.
    70 When do youneed SQL extensions for streaming? As a table: As a stream: SQL extensions rarely needed. SQL extensions sometimes needed. How is output consumed? good defaults = often not needed
  • 71.
    71 When do youneed SQL extensions for streaming?* Explicit table / stream selection ● SELECT TABLE * from X; ● SELECT STREAM * from X; Timestamps and windowing ● Event-time columns ● Windowing. E.g., SELECT * FROM X GROUP BY SESSION(<COLUMN> INTERVAL '5' MINUTE); ○ Grouping by timestamp ○ Complex multi-row transactions inexpressible in declarative SQL (e.g., session windows) Sane default table / stream selection ● If all inputs are tables, output is a table ● If any inputs are streams, output is a stream Simple triggers ● Implicitly defined by characteristics of the sink ● Optionally be configured outside of query. ● Per-query, e.g.: SELECT * from X EMIT <WHEN>; ● Focused set of use cases: ○ Repeated updates ... EMIT AFTER <TIMEDELTA> ○ Completeness ... EMIT WHEN WATERMARK PAST <COLUMN> ○ Repeated updates + completeness (e.g., early/on-time/late pattern) ... EMIT AFTER <TIMEDELTA> AND WHEN WATERMARK PAST <COLUMN> * Most of these extensions are theoretical at this point; very few have concrete implementations.
  • 72.
    72 Summary streams ⇄ tables streams& tables : Beam Model time-varying relations SQL language extensions
  • 73.
    73 Thank you! In earlyrelease now streamingsystems.net Streaming SQL spec (WIP: Apex, Beam, Calcite, Flink): http://s.apache.org/streaming-sql-spec Streaming in Calcite (Julian Hyde): https://calcite.apache.org/docs/stream.html Streams, joins & temporal tables (Julian Hyde): http://s.apache.org/streams-joins-and-temporal-tables Streaming 101: http://oreilly.com/ideas/the-world-beyond-batch-streaming-101 Streaming 102: http://oreilly.com/ideas/the-world-beyond-batch-streaming-102 Animations: http://github.com/takidau/animations Apache Beam: http://beam.apache.org Apache Calcite: http://calcite.apache.org Apache Flink: http://flink.apache.org