Aggregate Sharing for User-Define Data Stream Windows

@CIKM16
Cutty: Aggregate Sharing for
User-Deﬁned Windows
Paris Carbone <parisc@kth.se, senorcarbone@apache.org>
Jonas Traub <jonas.traub@tu-berlin.de>
Asterios Katsifodimos <asterios.katsifodimos@tu-berlin.de>
Seif Haridi <haridi@kth.se>
Volker Markl <volker.markl@tu-berlin.de>
1
Presentation : Paris Carbone
PhD Candidate @ KTH Sweden
Committer @ Apache Flink

@CIKM16
4 Reasons
Not to check your email during this talk
1. Windows are the backbone of data stream analysis.
2. We generalise the concept of data stream windows.
3. Our technique makes aggregations on general stream
windows more efﬁcient than ever.
4. We can multiplex and share aggregations of diverse
types of sliding windows that run simultaneously.
2

@CIKM16
Outline
• Partial Sliding Window Aggregation
• Fundamental Limitations of Existing Approaches
• Introducing User-Defined Windows(UDWs)
• Multi-Query Aggregation of UDWs with Cutty
• Performance Comparison
3

@CIKM16
4
Window Aggregation
1 2 3 4 5 6 7 8 9 10 11 12 13 14 …
1 2 3 4 5
3 4 5 6 7
5 6 7 8 9
Stream
Discretization
fd

@CIKM16
Window Aggregation
5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 …
1 2 3 4 5
3 4 5 6 7
5 6 7 8 9
A1
A2
A3
fa

@CIKM16
lift
record —> (val, count)
combine
(val1 + val2, count1 + count2)
lower
(val, count) —> val / count
Partial Aggregation
6
1 2 3 4 5
1. lift
3. lower
A1
M (2,1)(1,1)
2. combine
M
M
M
M
M
3
record type
partial type
aggr type
?
Example - AVG
(3,2)
(1,1)
(3,1)
(6,3) (4,1)
(10,4) (5,1)
(15,5)

@CIKM16
Partial Aggregation
7
•#Invocations <—> Computational Complexity
•Commutativity & Associativity are typically assumed.
2. combine
M

@CIKM16
Redundancy in Sliding
Window Aggregation
8
1 2 3 4 5 6 7 8 9 10 11 12 13 14 …
1 2 3 4 5
3 4 5 6 7
5 6 7 8 9
… …
overlapping means
redundant combine calls
we need to optimise…

@CIKM16
tumbling
single-type
periodic Punctuation
Snapshot
FCF/CF
Lower-Bound
Session
multi-type
ADWIN
Delta-based
FCA
slicing
Optimise…which windows?
pre-compute
non-overlapping partials
Periodic

@CIKM16
Slicing
10
1 2 3 4 5 6 7 8 9 10 11
12
13 14 15 16 17 18 19
Example - Count Window
range: 10, slide:3
1 2 3 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19
If a sliding window can be defined in terms of a fixed range and slide
the system can pre-aggregate consecutive, non-overlapping slices.
Panes1
gcd(range,slide)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Pairs2
p2: range%slide
p1: slide-p2
12
periodic windows
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19Cutty
(preview)
1. No Pane, No Gain: Efficient Evaluation of Sliding-Window Aggregates over Data Streams
SIGMOD 2005
2. On-the-Fly Sharing for Streamed Aggregation - SIGMOD 2006

@CIKM16
Slicing - Observations
• Computational Complexity —> upd: O(1) ,merge: O(#partials)
• Space Complexity (#stored sliced partials):
• Similar space requirements when range is a multiple of slice
• Pairs has been extended for multi-query aggregation sharing
11
d
range
gcd(range, slide)
e
d
2 ⇥ range
slide
e
Memory (#partials)
Panes
Pairs
—> 10 partials
—> 7 partials
…from previous example

@CIKM16
tumbling
single-type
Snapshot
FCF/CF
Lower-Bound
Session
multi-type
ADWIN
Delta-based
FCA
slicing
Optimise…which windows?
pre-compute
non-overlapping partials
pre-compute overlapping partials for
arbitrary aggregation lookups
eager pre-aggregation
Non-Periodic

@CIKM16
Eager Pre-Aggregation
13
When windowing cannot be expressed simply by a range and slide : eagerly
pre-compute partial aggregates and update a binary tree, bottom-up.
1 2 3 4 5 6 7 8
3 7 11 15
10 26
36
9 10
19
30
21
…
arbitrary window lookups
logn}
}n pre-computed
partials
n leaves ~ records}
}2n

@CIKM16
Eager Pre-Aggregation
Observations
• Implementations: FlatFAT1, B-Int2
• High Space Complexity (#raw records…twice)
• Most pre-aggregates are never used
• Update+Aggregation complexity :
• Generic and Suitable for Ad-Hoc Queries
• Potential for Multi-Query Window Pre-Aggregation
14
log(leaves)
1.General incremental sliding-window aggregation - VLDB 15
2.Resource sharing in continuous sliding-window aggregates - VLDB 04

@CIKM16
tumbling
single-type
Snapshot
FCF/CF
Lower-Bound
Session
multi-type
ADWIN
Delta-based
FCA
efﬁcient
slicing
generic, high-cost
pre-aggregation
Non-Periodic
Periodic

@CIKM16
tumbling
single-type
Snapshot
FCF/CF
Lower-Bound
Session
multi-type
ADWIN
Delta-based
FCA
efﬁcient
slicing
generic, high-cost
pre-aggregation
Non-Deterministic
Deterministic

@CIKM16
Deterministic Windows: Intuition
17
Slices
Higher
order
partials
price
[in USD]
time
[in min.]
0
0
5 10 15 20 25 31 35
10
Window
Window Begin
Threshold
Pre-Aggregate

@CIKM16
18
Slices
Higher
order
partials
price
[in USD]
time
[in min.]
0
0
5 10 15 20 25 31 35
10
Window
Window Begin
Threshold
Pre-Aggregate
only need to determine when new windows start

@CIKM16
19
Slices
Higher
order
partials
price
[in USD]
time
[in min.]
0
0
5 10 15 20 25 31 35
10
Window
Window Begin
Threshold
Pre-Aggregate
only need to determine when new windows start

@CIKM16
User-Deﬁned Windows
Deterministic: Expressed as a UDF that assigns each
record to number of new or complete windows.
20
Trivial templating of existing window types
Non-Deterministic: Expressed as a UDF that assigns
a record to complete windows and a reference to their
beginning.

@CIKM16
Cutty Concept
21
Slices
Higher
order
partials
price
[in USD]
time
[in min.]
0
0
5 10 15 20 25 31 35
10
Window
Window Begin
Threshold
Pre-Aggregate
1
Slicing
Eager
Pre-Aggregation

@CIKM16
Cutty Overview
Exploits Deterministic Windows for the most efﬁcient yet
aggregation slicing.
Utilises eager pre-aggregation at a low memory cost over
optimally sliced partials.
Supports both single and multi-query multiplexed
execution out-of-the-box for efﬁcient operator sharing.
Non-Deterministic Windows can still utilise eager pre-
aggregation.
22

@CIKM16
Cutty - Demo
24
1 2 3 4 5 6 7 8 9 10
-
Active
Partial
-
-
Stored
Partials
-
- - - -
Records
Windows

@CIKM16
Cutty - Demo
25
1 2 3 4 5 6 7 8 9 10
1
Active
Partial
-
-
Stored
Partials
-
- - - -
Records
Windows

@CIKM16
Cutty - Demo
26
1 2 3 4 5 6 7 8 9 10
Active
Partial
-
-
Stored
Partials
-
- - - -
Records
Windows
3

@CIKM16
Cutty - Demo
27
1 2 3 4 5 6 7 8 9 10
3
Active
Partial
3
3
Stored
Partials
-
3 - - -
Records
Windows

@CIKM16
Cutty - Demo
28
1 2 3 4 5 6 7 8 9 10
Active
Partial
3
3
Stored
Partials
-
3 - - -
Records
Windows
3

@CIKM16
Cutty - Demo
29
1 2 3 4 5 6 7 8 9 10
Active
Partial
3
3
Stored
Partials
-
3 - - -
Records
15
Windows
12

@CIKM16
Cutty - Demo
30
1 2 3 4 5 6 7 8 9 10
Active
Partial
15
15
Stored
Partials
-
3 12 - -
Records
15
Windows 21
6

@CIKM16
Cutty - Demo
31
1 2 3 4 5 6 7 8 9 10
Active
Partial
15
15
Stored
Partials
-
3 12 - -
Records
15
Windows 21
13

@CIKM16
Cutty - Demo
32
1 2 3 4 5 6 7 8 9 10
Active
Partial
15
28
Stored
Partials
13
3 12 13 -
Records
15
Windows 21
33
8

@CIKM16
Implementation
33
• Apache Flink
•UDW API (Contributed to Apache Flink - 0.9)
•Shared Aggregation Operator (experimental)
•Optimiser collocates parallel windows in operators
• Aggregate Store
•Adaptation of FlatFAT1
•Circular Resizable Buffer Strategies
•Non-Eager Strategy Supported for Experiments
1.General incremental sliding-window aggregation - VLDB 15

@CIKM16
Performance Analysis
Periodic Window Aggregation (DEBS12 dataset)
34
20 40 60 80 100
Number of Queries
0.0
0.5
1.0
1.5
2.0
2.5
3.0
NumberofPartials
⇥105
Cutty
Pairs/Pairs+
COUNT-RANGES COUNT-SLIDES
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
NumberofRecords
20 40 60 80 100
Number of Queries
0k
500k
1000k
1500k
2000k
2500k
3000k
3500k
4000k
4500k
Throughput(records/sec) Cutty
Pairs+
RA
1 10 20 30 40 50 60 70 80 90 100
Number of Queries
104
105
106
107
108
109
1010
1011
TotalReduceCalls
Cutty (eager)
Pairs+
Cutty (lazy)
Pairs
RA
Naive

@CIKM16
Performance Analysis
Session Window Aggregation (DEBS12 dataset)
35
SESSION LENGTHS
0
5000
10000
15000
20000
25000
30000
35000
NumberofRecords
1 10 20 30 40 50 60 70 80 90 100
Number of Queries
103
104
105
106
107
108
109
TotalReduceCalls
Cutty (UPD)
Cutty (MERGE)
RA (UPD)
RA (MERGE)
1 10 20 30 40 50 60 70 80 90 100
Number of Queries
100
101
102
103
104
105
106
MaxAllocation(#partials)

@CIKM16
No limits in multiplexing
36
distance [in km]
time
[in min.]
0
0
6 12 18 24
5
10
15
20
Slice Window Window Begin Record
1

@CIKM16
Summary
• UDWs extend the potential of pre-aggregation in
window classes beyond ﬁxed periodic windows.
• Cutty takes slicing a step further in terms of
computational efﬁciency which combines
seamlessly with eager aggregation.
• First work that addresses multi-query aggregation
across diverse window types.
37

@CIKM16
Thank you!
38
@SenorCarbone
https://ﬂink.apache.org/
https://github.com/apache/ﬂink

Aggregate Sharing for User-Define Data Stream Windows

More Related Content

What's hot

Viewers also liked

Similar to Aggregate Sharing for User-Define Data Stream Windows

More from Paris Carbone

Recently uploaded

Aggregate Sharing for User-Define Data Stream Windows