code.talks 2019 - Scotty: Efficient Window Aggregation for your Stream Processing System

Philipp M. Grulich (TU Berlin) & Jonas Traub (TU Berlin)
Scotty: Efficient Window Aggregation
for your Stream Processing System
Big Data Track

Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Scotty: Efficient Window Aggregation for your Stream Processing System
About Us
Philipp M. Grulich
Research Associate (TU Berlin)
grulich@tu-berlin.de
Jonas Traub
jonas.traub@tu-berlin.de
2

About Us
Philipp M. Grulich
grulich@tu-berlin.de
Jonas Traub
jonas.traub@tu-berlin.de
Database Systems and Information Management Research Group at TU Berlin
www.dima.tu-berlin.de
2

Stream Processing Systems
Souce: Rajaraman, A., & Ullman, J. D. (2012). Mining of massive datasets (Vol. 77). Cambridge University Press. Chapter 4, www.mmds.org
3

Aggregations in Stream Processing Pipelines
A stream processing pipeline is a series of concurrently running operators.
4

Window
Aggregation
4
53

Window
Aggregation
53
4

Window
Aggregation
8
4

Window
Aggregation
Arithmetic Operations
Sum, min, max etc.
Statistics / Analysis
Reservoir Sampling
ML Model Updates
Concept Drift Detection
Aggregation Examples:
4

Apache Flink - Stateful Stream Processing
“Apache Flink is a framework and distributed processing engine for stateful computations
over unbounded and bounded data streams. Flink has been designed to run in all common cluster
environments, perform computations at in-memory speed and at any scale.” (flink.apache.org)
5

Motivation
6

Cutty: Aggregate Sharing for User-Defined Windows
P. Carbone, J. Traub, A. Katsifodimos, S. Haridi, V. Markl
ACM International on Conference on Information and Knowledge Management (CIKM 2016)
Scotty: Efficient Window Aggregation for out-of-order Stream Processing
J. Traub, P. M. Grulich, A. R. Cuéllar, S. Breß, A. Katsifodimos, T. Rabl, V. Markl
IEEE International Conference on Data Engineering (ICDE 2018)
Efficient Window Aggregation with General Stream Slicing
J. Traub, P. M. Grulich, AR. Cuéllar, S. Breß, A. Katsifodimos, T. Rabl, V. Markl
International Conference on Extending Database Technology (EDBT 2019; Best Paper Award)
Scotty Window Processor:
Efficent Window Aggregations for Flink, Beam, and Storm
https://github.com/TU-Berlin-DIMA/scotty-window-processor
7
Research Background

Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Scotty: Efficient Window Aggregation for your Stream Processing System 8
Stream Slicing Example

The number of slices depends on the workload.
9

We store partial aggregates instead of all tuples. => Small memory footprint.

We assign each tuple to exactly one slice. => O(1) per-tuple complexity.

We require just a few computation steps to calculate final aggregates. => Low latency.
16

Workload
Characteristics
Window
Types
Context Free
Forward Context Free
Forward Context Aware
Stream
Order
in-order
out-of-order
Window
Measures
time
tuple count
arbitrary
Aggregation
Functions
distributive
algebraic
holistic
associativity
cummutativity
invertibility
17
Workload Characteristics

Workload
Characteristics
Window
Types
Context Free
Forward Context Free
Forward Context Aware
Stream
Order
in-order
out-of-order
Window
Measures
time
tuple count
arbitrary
Aggregation
Functions
distributive
algebraic
holistic
associativity
cummutativity
invertibility
General Stream Slicing combines generality and efficiency in a single solution.
17
Workload Characteristics

Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
Count-based tumbling window
with a length of 5 tuples.
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
11 13 12
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
11 13 12
What if the stream is out-of-order?
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
5
49
Out-of-order Tuple
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
5
49
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
5
49
13 12
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
5
49
13 12
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
5
49
13 12
5
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
5
49
13 125 + - 3
5
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
5
49
13 123 1+ -5 + - 3
5
18

1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
5
49
13 123 1+ -5 + - 3
5
What if the aggregation function is not invertible?
18

Scotty Window Processor:
Efficent Window Aggregations
for Flink, Beam, and Storm
https://github.com/TU-Berlin-DIMA/scotty-window-processor
19

Key-Facts
Features:
● One window operator for many systems.
● High performance window aggregations with stream slicing.
● Scales to thousands of concurrent windows.
● Aggregate sharing among multiple window queries.
● Adapts to workload characteristics:
○ Window Types
○ Aggregation Functions
○ Window Measures
○ Stream Order
20

Key-Facts
Features:
● High performance window aggregations with stream slicing.
● Adapts to workload characteristics:
○ Window Types
○ Aggregation Functions
○ Window Measures
○ Stream Order
Connectors:
…more coming soon…
20

Scotty Core
21

Scotty Core
Scotty adapts to work load characteristics
and combines generality and efficiency in a single solution.
21

Benchmark
Concurrent Windows with Built-in Window Operator:
● Flink performs well
with a single window
(no overlap; one
bucket at a time)
0
500.000
1.000.000
1.500.000
2.000.000
2.500.000
1 10 20 50 100 500 1000
Flink Storm Flink on Beam
Throughput(Tuples/sec.)
Number of Councurrent Windows
22

Benchmark
Concurrent Windows with Built-in Window Operator:
● Flink performs well
with a single window
(no overlap; one
bucket at a time)
0
500.000
1.000.000
1.500.000
2.000.000
2.500.000
1 10 20 50 100 500 1000
Flink Storm Flink on Beam
● With overlapping
concurrent windows,
the throughput drops
drastically.
22

0
500.000
1.000.000
1.500.000
2.000.000
2.500.000
1 10 20 50 100 500 1000
Flink+Scotty Storm+Scotty Beam+Flink+Scotty
Benchmark
Concurrent Windows with Scotty:
● With Scotty, the throughput is
independent of the number of
concurrent windows.
23

Using Scotty on Flink
1. Clone Scotty and install to maven
24

1. Clone Scotty and install to maven
2. Add Scotty to your Flink Project:
24

1. Initialize Scotty Window Operator
25

2. Add Window Definitions
25

3. Add Scotty to your Flink Job
2. Add Window Definitions
25

Implement your own Aggregations Functions
26

Example:
• Average -> Sum/Count
27

8|4
6 7 1 2 5
3|3
Input Stream: Output Stream:
PartialState
Example:
28

8|4
6 7 1 2
1. lift
5
5|1
3|3
Example:
28

8|4
6 7 1 2
1. lift
5
5|1
2. combine
3|38|4
Example:
28

8|4
6 7 1 2
1. lift
5
5|1
2. combine
2
2
3. lower
3|38|4
Example:
28

Upcoming Research Projects
29

The NebulaStream Platform:
Data and Application Management
for the Internet of Things
https://arxiv.org/pdf/1910.07867.pdf
29

Agora:
An Open Ecosystem for Democratizing Data
Science & Artificial Intelligence
https://arxiv.org/pdf/1910.07867.pdf https://arxiv.org/pdf/1909.03026.pdf
29

Agora:
An Open Ecosystem for Democratizing Data
Science & Artificial Intelligence
https://arxiv.org/pdf/1910.07867.pdf https://arxiv.org/pdf/1909.03026.pdf
We are hiring!
Research Associates / PhD Students & Post Docs (m/w/d)
Catch us after the talks or send a mail to
jobs@dima.tu-berlin.de
29

Acknowledgements: This talk is supported by the Berlin Big Data Center (01IS14013A), the Berlin Center for Machine Learning (01IS18037A), and Software Campus (1-3000473-18TP).
Scotty Features:
● High performance with stream slicing.
● Adapts to workload characteristics
tu-berlin-dima.github.io/
scotty-window-processor
Open Source Repository:
30
Scotty Window Processor

code.talks 2019 - Scotty: Efficient Window Aggregation for your Stream Processing System

More Related Content

Similar to code.talks 2019 - Scotty: Efficient Window Aggregation for your Stream Processing System

More from Jonas Traub

Recently uploaded

code.talks 2019 - Scotty: Efficient Window Aggregation for your Stream Processing System