Jonas Traub, profile picture
Uploaded byJonas Traub
PDF, PPTX879 views

Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)

The document discusses efficient window aggregation with general stream slicing in stream processing pipelines, highlighting its advantages in memory efficiency, low latency, and reduced redundancy. It details how stream slicing adapts to workload characteristics and performs well across various aggregation functions, even in scenarios of out-of-order data. Key operations such as merging, splitting, and updating slices are examined to illustrate the technique's adaptability and effectiveness.

Embed presentation

Download as PDF, PPTX
Jonas Traub Philipp M. Grulich Alejandro Rodríguez Cuéllar Sebastian Breß Asterios Katsifodimos Tilmann Rabl Volker Markl Efficient Window Aggregation with General Stream Slicing 22nd International Conference on Extending Database Technology March 26-29, 2019, Lisbon, Portugal
Stream Processing Pipelines 27.03.2019 Efficient Window Aggregation with General Stream Slicing 2 A stream processing pipeline is a series of concurrently running operators.
Stream Processing Pipelines 27.03.2019 Efficient Window Aggregation with General Stream Slicing 2 A stream processing pipeline is a series of concurrently running operators. Window Aggregation
Stream Processing Pipelines 27.03.2019 Efficient Window Aggregation with General Stream Slicing 2 A stream processing pipeline is a series of concurrently running operators. Window Aggregation 53
Stream Processing Pipelines 27.03.2019 Efficient Window Aggregation with General Stream Slicing 2 A stream processing pipeline is a series of concurrently running operators. Window Aggregation 8
Motivation 27.03.2019 Efficient Window Aggregation with General Stream Slicing 3
Motivation 27.03.2019 Efficient Window Aggregation with General Stream Slicing 3
Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 4
Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 5
The number of slices depends on the workload. Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 5
Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 6
Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 7
Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 8
Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 9
We store partial aggregates instead of all tuples.  Small memory footprint. Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 9
Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 10
We assign each tuple to exactly one slice.  O(1) per-tuple complexity. Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 10
Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 11
We require just a few computation steps to calculate final aggregates.  Low latency. Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 11
Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 12
We share partial aggregations among all users and queries.  Efficiency by preventing redundancy. Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 12
General Stream Slicing 27.03.2019 Efficient Window Aggregation with General Stream Slicing 13
General Stream Slicing Workload Characteristics 27.03.2019 Efficient Window Aggregation with General Stream Slicing 13
General Stream Slicing Workload Characteristics Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 27.03.2019 Efficient Window Aggregation with General Stream Slicing 13
General Stream Slicing Workload Characteristics Window Types Context Free Forward Context Free Forward Context Aware Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 27.03.2019 Efficient Window Aggregation with General Stream Slicing 13
General Stream Slicing Workload Characteristics Window Types Context Free Forward Context Free Forward Context Aware Window Measures time tuple count arbitrary Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 27.03.2019 Efficient Window Aggregation with General Stream Slicing 13
General Stream Slicing Workload Characteristics Window Types Context Free Forward Context Free Forward Context Aware Stream Order in-order out-of-order Window Measures time tuple count arbitrary Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 27.03.2019 Efficient Window Aggregation with General Stream Slicing 13
General Stream Slicing Workload Characteristics Window Types Context Free Forward Context Free Forward Context Aware Stream Order in-order out-of-order Window Measures time tuple count arbitrary Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 27.03.2019 Efficient Window Aggregation with General Stream Slicing 13 General Stream Slicing combines generality and efficiency in a single solution.
Window Aggregation Concepts 27.03.2019 Efficient Window Aggregation with General Stream Slicing 14 Variations of Stream SlicingNon-Slicing Techniques
General Slicing Core 27.03.2019 Efficient Window Aggregation with General Stream Slicing 15
General Slicing Core The General Slicing Core adapts to work load characteristics and provides extension point for user-defined window types and aggregation functions. 27.03.2019 Efficient Window Aggregation with General Stream Slicing 15
General Stream Slicing Internals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16
General Stream Slicing Internals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Part 1: Three Fundamental Operations on Slices
General Stream Slicing Internals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Part 1: Three Fundamental Operations on Slices
General Stream Slicing Internals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Part 1: Three Fundamental Operations on Slices
General Stream Slicing Internals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices
General Stream Slicing Internals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices Part 2: Adapt to Workload Characteristics:
General Stream Slicing Internals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices Part 2: Adapt to Workload Characteristics: Do we need to store original tuples?
General Stream Slicing Internals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices Part 2: Adapt to Workload Characteristics: Do we need to store original tuples? Do we potentially need to split slices?
General Stream Slicing Internals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices Part 2: Adapt to Workload Characteristics: Do we need to store original tuples? Do we potentially need to split slices? Do we potentially need to remove tuples from slices?
General Stream Slicing Internals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices Part 2: Adapt to Workload Characteristics: Do we need to store original tuples? Do we potentially need to split slices? Do we potentially need to remove tuples from slices?
General Stream Slicing Internals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices Part 2: Adapt to Workload Characteristics: Do we need to store original tuples? Do we potentially need to split slices? Do we potentially need to remove tuples from slices? General Stream Slicing adapts to current workload characteristics.
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 Count-based tumbling window with a length of 5 tuples.
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Count-based tumbling window with a length of 5 tuples.
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Count-based tumbling window with a length of 5 tuples. 11 13 12
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 11 13 12 What if the stream is out-of-order?
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order?
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 5 49 Out-of-order Tuple
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 5 49 Out-of-order Tuple
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 5 49
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 5 49 13 12
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 12
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 12 5
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 125 + - 3 5
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 123 1+ -5 + - 3 5
Impact of Workload Characteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 123 1+ -5 + - 3 5 What if the aggregation function is not invertible?
In-order Processing with Context Free Windows 27.03.2019 Efficient Window Aggregation with General Stream Slicing 18
In-order Processing with Context Free Windows 27.03.2019 Efficient Window Aggregation with General Stream Slicing 18 Slicing techniques scale to large numbers of concurrent windows.
Impact of Stream Order 27.03.2019 Efficient Window Aggregation with General Stream Slicing 19
Impact of Stream Order 27.03.2019 Efficient Window Aggregation with General Stream Slicing 19 Slicing techniques are robust against out-of-order tuples.
Impact of Aggregation Functions (20% out-of-order) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 20
Impact of Aggregation Functions (20% out-of-order) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 20 Stream Slicing performs well on many different kinds of aggregation functions.
Efficient Window Aggregation with General Stream Slicing 27.03.2019 Efficient Window Aggregation with General Stream Slicing 21
Efficient Window Aggregation with General Stream Slicing • We identify workload characteristics which impact applicability and performance of window aggregation techniques. 27.03.2019 Efficient Window Aggregation with General Stream Slicing 21
Efficient Window Aggregation with General Stream Slicing • We identify workload characteristics which impact applicability and performance of window aggregation techniques. • We present a generally applicable and highly efficient solution for streaming window aggregation. 27.03.2019 Efficient Window Aggregation with General Stream Slicing 21
Efficient Window Aggregation with General Stream Slicing • We identify workload characteristics which impact applicability and performance of window aggregation techniques. • We present a generally applicable and highly efficient solution for streaming window aggregation. • We show that general stream slicing is generally applicable and offers better performance than alternative approaches. 27.03.2019 Efficient Window Aggregation with General Stream Slicing 21
Efficient Window Aggregation with General Stream Slicing • We identify workload characteristics which impact applicability and performance of window aggregation techniques. • We present a generally applicable and highly efficient solution for streaming window aggregation. • We show that general stream slicing is generally applicable and offers better performance than alternative approaches. 27.03.2019 Efficient Window Aggregation with General Stream Slicing 21 tu-berlin-dima.github.io/scotty-window-processor Open Source Repository:

More Related Content

PDF
Aggregate Sharing for User-Define Data Stream Windows
byParis Carbone
 
PDF
Scotty: Efficient Window Aggregation with General Stream Slicing - jonas Trau...
byFlink Forward
 
PDF
FlinkForward Berlin 2019 - Scotty: Efficient Window Aggregation with General ...
byJonas Traub
 
PDF
code.talks 2019 - Scotty: Efficient Window Aggregation for your Stream Proces...
byJonas Traub
 
PDF
Scotty: Efficient Window Aggregation for Out-of-Order Stream Processing
byJonas Traub
 
PDF
Flink Forward 2018: Efficient Window Aggregation with Stream Slicing
byJonas Traub
 
PDF
Flink Forward Berlin 2018: Jonas Traub & Philipp Grulich - "Efficient Window ...
byFlink Forward
 
PPTX
Fabian Hueske - Stream Analytics with SQL on Apache Flink
byVerverica
 
Aggregate Sharing for User-Define Data Stream Windows
byParis Carbone
 
Scotty: Efficient Window Aggregation with General Stream Slicing - jonas Trau...
byFlink Forward
 
FlinkForward Berlin 2019 - Scotty: Efficient Window Aggregation with General ...
byJonas Traub
 
code.talks 2019 - Scotty: Efficient Window Aggregation for your Stream Proces...
byJonas Traub
 
Scotty: Efficient Window Aggregation for Out-of-Order Stream Processing
byJonas Traub
 
Flink Forward 2018: Efficient Window Aggregation with Stream Slicing
byJonas Traub
 
Flink Forward Berlin 2018: Jonas Traub & Philipp Grulich - "Efficient Window ...
byFlink Forward
 
Fabian Hueske - Stream Analytics with SQL on Apache Flink
byVerverica
 

Similar to Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)

PDF
Parallel Patterns for Window-based Stateful Operators on Data Streams: an Alg...
byTiziano De Matteis
 
PPTX
Flink SQL & TableAPI in Large Scale Production at Alibaba
byDataWorks Summit
 
PPTX
Flink Forward SF 2017: Shaoxuan Wang_Xiaowei Jiang - Blinks Improvements to F...
byFlink Forward
 
PPTX
Stream processing - Apache flink
byRenato Guimaraes
 
PPTX
Flink-window-function-basic
byPreetdeep Kumar
 
PDF
Streaming SQL
byDataWorks Summit/Hadoop Summit
 
PDF
Julian Hyde - Streaming SQL
byFlink Forward
 
PDF
Streaming SQL
byJulian Hyde
 
PDF
Streaming SQL (at FlinkForward, Berlin, 2016/09/12)
byJulian Hyde
 
PPTX
Apache Flink Training: DataStream API Part 2 Advanced
byFlink Forward
 
PDF
Windowing in apex
byYogi Devendra Vyavahare
 
PDF
Windowing in Apache Apex
byApache Apex
 
PDF
SobhanBadiozamanyPhD
bySobhan Badiozamany
 
PPTX
Data Streaming (in a Nutshell) ... and Spark's window operations
byVincenzo Gulisano
 
PDF
Development of a Distributed Stream Processing System
byMaycon Viana Bordin
 
PDF
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
byData Con LA
 
PDF
Introduction to Data streaming - 05/12/2014
byRaja Chiky
 
PPTX
Stream Analytics with SQL on Apache Flink
byFabian Hueske
 
PPTX
Have your cake and eat it too, further dispelling the myths of the lambda arc...
byDimos Raptis
 
PDF
Streaming SQL
byJulian Hyde
 
Parallel Patterns for Window-based Stateful Operators on Data Streams: an Alg...
byTiziano De Matteis
 
Flink SQL & TableAPI in Large Scale Production at Alibaba
byDataWorks Summit
 
Flink Forward SF 2017: Shaoxuan Wang_Xiaowei Jiang - Blinks Improvements to F...
byFlink Forward
 
Stream processing - Apache flink
byRenato Guimaraes
 
Flink-window-function-basic
byPreetdeep Kumar
 
Streaming SQL
byDataWorks Summit/Hadoop Summit
 
Julian Hyde - Streaming SQL
byFlink Forward
 
Streaming SQL
byJulian Hyde
 
Streaming SQL (at FlinkForward, Berlin, 2016/09/12)
byJulian Hyde
 
Apache Flink Training: DataStream API Part 2 Advanced
byFlink Forward
 
Windowing in apex
byYogi Devendra Vyavahare
 
Windowing in Apache Apex
byApache Apex
 
SobhanBadiozamanyPhD
bySobhan Badiozamany
 
Data Streaming (in a Nutshell) ... and Spark's window operations
byVincenzo Gulisano
 
Development of a Distributed Stream Processing System
byMaycon Viana Bordin
 
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
byData Con LA
 
Introduction to Data streaming - 05/12/2014
byRaja Chiky
 
Stream Analytics with SQL on Apache Flink
byFabian Hueske
 
Have your cake and eat it too, further dispelling the myths of the lambda arc...
byDimos Raptis
 
Streaming SQL
byJulian Hyde
 

More from Jonas Traub

PDF
Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentat...
byJonas Traub
 
PDF
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
byJonas Traub
 
PDF
I²: Interactive Real-Time Visualization for Streaming Data
byJonas Traub
 
PDF
Scalable Detection of Concept Drifts on Data Streams with Parallel Adaptive W...
byJonas Traub
 
PDF
Efficient SIMD Vectorization for Hashing in OpenCL
byJonas Traub
 
PDF
Database Research at TU Berlin DIMA and DFKI IAM - USA Excursion Slides 2019
byJonas Traub
 
PDF
Efficient Data Stream Processing in the Internet of Things - SoftwareCampus A...
byJonas Traub
 
PDF
I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
byJonas Traub
 
PDF
UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...
byJonas Traub
 
PDF
Resense: Transparent Record and Replay of Sensor Data in the Internet of Thin...
byJonas Traub
 
PDF
JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...
byJonas Traub
 
PDF
LWA 2015: The Apache Flink Platform (Poster)
byJonas Traub
 
PDF
Definitely not Java! A Hands-on Introduction to Efficient Functional Programm...
byJonas Traub
 
Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentat...
byJonas Traub
 
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
byJonas Traub
 
I²: Interactive Real-Time Visualization for Streaming Data
byJonas Traub
 
Scalable Detection of Concept Drifts on Data Streams with Parallel Adaptive W...
byJonas Traub
 
Efficient SIMD Vectorization for Hashing in OpenCL
byJonas Traub
 
Database Research at TU Berlin DIMA and DFKI IAM - USA Excursion Slides 2019
byJonas Traub
 
Efficient Data Stream Processing in the Internet of Things - SoftwareCampus A...
byJonas Traub
 
I²: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
byJonas Traub
 
UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...
byJonas Traub
 
Resense: Transparent Record and Replay of Sensor Data in the Internet of Thin...
byJonas Traub
 
JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...
byJonas Traub
 
LWA 2015: The Apache Flink Platform (Poster)
byJonas Traub
 
Definitely not Java! A Hands-on Introduction to Efficient Functional Programm...
byJonas Traub
 

Recently uploaded

PDF
Oracle Data Breach_ Causes, Impact, and Security Lessons.pdf
bydexpose Lastdexpose
 
PPTX
Fundamental of cyber security - Chapter seven
bySenaitDesalegn2
 
PPTX
Data-Mining-in-Data-Analytics-for-Cyber-Security.pptx
bySenaitDesalegn2
 
PPTX
[DSC Europe 25] Behzad Hosseini - AI Agents in the Wild: Deploying Models tha...
byDataScienceConferenc1
 
PPTX
Integrated matching and rebalancing for ride-hailing services.pptx
byWenjiaShen1
 
PPTX
[DSC Europe 25] Nikolay Burlutskiy - Best Practices for Building Enterprise M...
byDataScienceConferenc1
 
PDF
High-Performance Loading of Financial Market Data Streams to the Snowflake Pl...
byDataFuze
 
PPTX
[DSC Europe 25] Tatevik Maytesyan - How to actually use AI in marketing: gett...
byDataScienceConferenc1
 
PDF
DiaryofaWebSDKImplemenation-LaunchDebugging.pdf
bygmuir1066
 
PDF
Using Data & Technology To Drive Revenue - Fitness KPI 12.25 Webinar
byBryan K. O'Rourke
 
PPTX
[DSC Europe 25] Dunja Adzic Jovanovic - AI and Cybersecurity: Defending Data ...
byDataScienceConferenc1
 
PPTX
TCP IP and Routing Protocols, Information systems
byryanjamescampos789
 
PDF
Tutorial 1 Introduction to Environmental Economics.pdf
bychakmajoykumar416
 
PPTX
[DSC Europe 25] Hans Kleinsman - The Compliance Gearbox: How Tax Tech Mediate...
byDataScienceConferenc1
 
PDF
ICT Prep2 Lesson1-2 SM_251002_074210.pdf
bymhalexeg82
 
PPTX
Edison Research's Top 10 Findings of 2025
bymlazovick
 
PPTX
[DSC Europe 25] Branko Urosevic -Rethinking Financial Talent: Integrating Cod...
byDataScienceConferenc1
 
PPTX
Presentation on Hypothesis testing In statistics
bysilversteps03
 
PPTX
[DSC Europe 25] Bassam Maharmeh - Artificial Intelligence: Opportunities and ...
byDataScienceConferenc1
 
PDF
McCrindle 2026 Trends of the Year
byMark McCrindle
 
Oracle Data Breach_ Causes, Impact, and Security Lessons.pdf
bydexpose Lastdexpose
 
Fundamental of cyber security - Chapter seven
bySenaitDesalegn2
 
Data-Mining-in-Data-Analytics-for-Cyber-Security.pptx
bySenaitDesalegn2
 
[DSC Europe 25] Behzad Hosseini - AI Agents in the Wild: Deploying Models tha...
byDataScienceConferenc1
 
Integrated matching and rebalancing for ride-hailing services.pptx
byWenjiaShen1
 
[DSC Europe 25] Nikolay Burlutskiy - Best Practices for Building Enterprise M...
byDataScienceConferenc1
 
High-Performance Loading of Financial Market Data Streams to the Snowflake Pl...
byDataFuze
 
[DSC Europe 25] Tatevik Maytesyan - How to actually use AI in marketing: gett...
byDataScienceConferenc1
 
DiaryofaWebSDKImplemenation-LaunchDebugging.pdf
bygmuir1066
 
Using Data & Technology To Drive Revenue - Fitness KPI 12.25 Webinar
byBryan K. O'Rourke
 
[DSC Europe 25] Dunja Adzic Jovanovic - AI and Cybersecurity: Defending Data ...
byDataScienceConferenc1
 
TCP IP and Routing Protocols, Information systems
byryanjamescampos789
 
Tutorial 1 Introduction to Environmental Economics.pdf
bychakmajoykumar416
 
[DSC Europe 25] Hans Kleinsman - The Compliance Gearbox: How Tax Tech Mediate...
byDataScienceConferenc1
 
ICT Prep2 Lesson1-2 SM_251002_074210.pdf
bymhalexeg82
 
Edison Research's Top 10 Findings of 2025
bymlazovick
 
[DSC Europe 25] Branko Urosevic -Rethinking Financial Talent: Integrating Cod...
byDataScienceConferenc1
 
Presentation on Hypothesis testing In statistics
bysilversteps03
 
[DSC Europe 25] Bassam Maharmeh - Artificial Intelligence: Opportunities and ...
byDataScienceConferenc1
 
McCrindle 2026 Trends of the Year
byMark McCrindle
 

Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)

  • 1.
    Jonas Traub PhilippM. Grulich Alejandro Rodríguez Cuéllar Sebastian Breß Asterios Katsifodimos Tilmann Rabl Volker Markl Efficient Window Aggregation with General Stream Slicing 22nd International Conference on Extending Database Technology March 26-29, 2019, Lisbon, Portugal
  • 2.
    Stream Processing Pipelines 27.03.2019Efficient Window Aggregation with General Stream Slicing 2 A stream processing pipeline is a series of concurrently running operators.
  • 3.
    Stream Processing Pipelines 27.03.2019Efficient Window Aggregation with General Stream Slicing 2 A stream processing pipeline is a series of concurrently running operators. Window Aggregation
  • 4.
    Stream Processing Pipelines 27.03.2019Efficient Window Aggregation with General Stream Slicing 2 A stream processing pipeline is a series of concurrently running operators. Window Aggregation 53
  • 5.
    Stream Processing Pipelines 27.03.2019Efficient Window Aggregation with General Stream Slicing 2 A stream processing pipeline is a series of concurrently running operators. Window Aggregation 8
  • 6.
    Motivation 27.03.2019 Efficient WindowAggregation with General Stream Slicing 3
  • 7.
    Motivation 27.03.2019 Efficient WindowAggregation with General Stream Slicing 3
  • 8.
    Stream Slicing Example 27.03.2019Efficient Window Aggregation with General Stream Slicing 4
  • 9.
    Stream Slicing Example 27.03.2019Efficient Window Aggregation with General Stream Slicing 5
  • 10.
    The number ofslices depends on the workload. Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 5
  • 11.
    Stream Slicing Example 27.03.2019Efficient Window Aggregation with General Stream Slicing 6
  • 12.
    Stream Slicing Example 27.03.2019Efficient Window Aggregation with General Stream Slicing 7
  • 13.
    Stream Slicing Example 27.03.2019Efficient Window Aggregation with General Stream Slicing 8
  • 14.
    Stream Slicing Example 27.03.2019Efficient Window Aggregation with General Stream Slicing 9
  • 15.
    We store partialaggregates instead of all tuples.  Small memory footprint. Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 9
  • 16.
    Stream Slicing Example 27.03.2019Efficient Window Aggregation with General Stream Slicing 10
  • 17.
    We assign eachtuple to exactly one slice.  O(1) per-tuple complexity. Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 10
  • 18.
    Stream Slicing Example 27.03.2019Efficient Window Aggregation with General Stream Slicing 11
  • 19.
    We require justa few computation steps to calculate final aggregates.  Low latency. Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 11
  • 20.
    Stream Slicing Example 27.03.2019Efficient Window Aggregation with General Stream Slicing 12
  • 21.
    We share partialaggregations among all users and queries.  Efficiency by preventing redundancy. Stream Slicing Example 27.03.2019 Efficient Window Aggregation with General Stream Slicing 12
  • 22.
    General Stream Slicing 27.03.2019Efficient Window Aggregation with General Stream Slicing 13
  • 23.
    General Stream Slicing Workload Characteristics 27.03.2019Efficient Window Aggregation with General Stream Slicing 13
  • 24.
  • 25.
    General Stream Slicing Workload Characteristics Window Types ContextFree Forward Context Free Forward Context Aware Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 27.03.2019 Efficient Window Aggregation with General Stream Slicing 13
  • 26.
    General Stream Slicing Workload Characteristics Window Types ContextFree Forward Context Free Forward Context Aware Window Measures time tuple count arbitrary Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 27.03.2019 Efficient Window Aggregation with General Stream Slicing 13
  • 27.
    General Stream Slicing Workload Characteristics Window Types ContextFree Forward Context Free Forward Context Aware Stream Order in-order out-of-order Window Measures time tuple count arbitrary Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 27.03.2019 Efficient Window Aggregation with General Stream Slicing 13
  • 28.
    General Stream Slicing Workload Characteristics Window Types ContextFree Forward Context Free Forward Context Aware Stream Order in-order out-of-order Window Measures time tuple count arbitrary Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 27.03.2019 Efficient Window Aggregation with General Stream Slicing 13 General Stream Slicing combines generality and efficiency in a single solution.
  • 29.
    Window Aggregation Concepts 27.03.2019Efficient Window Aggregation with General Stream Slicing 14 Variations of Stream SlicingNon-Slicing Techniques
  • 30.
    General Slicing Core 27.03.2019Efficient Window Aggregation with General Stream Slicing 15
  • 31.
    General Slicing Core TheGeneral Slicing Core adapts to work load characteristics and provides extension point for user-defined window types and aggregation functions. 27.03.2019 Efficient Window Aggregation with General Stream Slicing 15
  • 32.
    General Stream SlicingInternals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16
  • 33.
    General Stream SlicingInternals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Part 1: Three Fundamental Operations on Slices
  • 34.
    General Stream SlicingInternals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Part 1: Three Fundamental Operations on Slices
  • 35.
    General Stream SlicingInternals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Part 1: Three Fundamental Operations on Slices
  • 36.
    General Stream SlicingInternals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices
  • 37.
    General Stream SlicingInternals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices Part 2: Adapt to Workload Characteristics:
  • 38.
    General Stream SlicingInternals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices Part 2: Adapt to Workload Characteristics: Do we need to store original tuples?
  • 39.
    General Stream SlicingInternals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices Part 2: Adapt to Workload Characteristics: Do we need to store original tuples? Do we potentially need to split slices?
  • 40.
    General Stream SlicingInternals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices Part 2: Adapt to Workload Characteristics: Do we need to store original tuples? Do we potentially need to split slices? Do we potentially need to remove tuples from slices?
  • 41.
    General Stream SlicingInternals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices Part 2: Adapt to Workload Characteristics: Do we need to store original tuples? Do we potentially need to split slices? Do we potentially need to remove tuples from slices?
  • 42.
    General Stream SlicingInternals 27.03.2019 Efficient Window Aggregation with General Stream Slicing 16 Merge Slices Split Slices Update Slices Part 1: Three Fundamental Operations on Slices Part 2: Adapt to Workload Characteristics: Do we need to store original tuples? Do we potentially need to split slices? Do we potentially need to remove tuples from slices? General Stream Slicing adapts to current workload characteristics.
  • 43.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17
  • 44.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
  • 45.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 Count-based tumbling window with a length of 5 tuples.
  • 46.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Count-based tumbling window with a length of 5 tuples.
  • 47.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Count-based tumbling window with a length of 5 tuples. 11 13 12
  • 48.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 11 13 12 What if the stream is out-of-order?
  • 49.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order?
  • 50.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 5 49 Out-of-order Tuple
  • 51.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 5 49 Out-of-order Tuple
  • 52.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 5 49
  • 53.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 5 49 13 12
  • 54.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 12
  • 55.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 12 5
  • 56.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 125 + - 3 5
  • 57.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 123 1+ -5 + - 3 5
  • 58.
    Impact of WorkloadCharacteristics (Example) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 17 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 123 1+ -5 + - 3 5 What if the aggregation function is not invertible?
  • 59.
    In-order Processing withContext Free Windows 27.03.2019 Efficient Window Aggregation with General Stream Slicing 18
  • 60.
    In-order Processing withContext Free Windows 27.03.2019 Efficient Window Aggregation with General Stream Slicing 18 Slicing techniques scale to large numbers of concurrent windows.
  • 61.
    Impact of StreamOrder 27.03.2019 Efficient Window Aggregation with General Stream Slicing 19
  • 62.
    Impact of StreamOrder 27.03.2019 Efficient Window Aggregation with General Stream Slicing 19 Slicing techniques are robust against out-of-order tuples.
  • 63.
    Impact of AggregationFunctions (20% out-of-order) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 20
  • 64.
    Impact of AggregationFunctions (20% out-of-order) 27.03.2019 Efficient Window Aggregation with General Stream Slicing 20 Stream Slicing performs well on many different kinds of aggregation functions.
  • 65.
    Efficient Window Aggregationwith General Stream Slicing 27.03.2019 Efficient Window Aggregation with General Stream Slicing 21
  • 66.
    Efficient Window Aggregationwith General Stream Slicing • We identify workload characteristics which impact applicability and performance of window aggregation techniques. 27.03.2019 Efficient Window Aggregation with General Stream Slicing 21
  • 67.
    Efficient Window Aggregationwith General Stream Slicing • We identify workload characteristics which impact applicability and performance of window aggregation techniques. • We present a generally applicable and highly efficient solution for streaming window aggregation. 27.03.2019 Efficient Window Aggregation with General Stream Slicing 21
  • 68.
    Efficient Window Aggregationwith General Stream Slicing • We identify workload characteristics which impact applicability and performance of window aggregation techniques. • We present a generally applicable and highly efficient solution for streaming window aggregation. • We show that general stream slicing is generally applicable and offers better performance than alternative approaches. 27.03.2019 Efficient Window Aggregation with General Stream Slicing 21
  • 69.
    Efficient Window Aggregationwith General Stream Slicing • We identify workload characteristics which impact applicability and performance of window aggregation techniques. • We present a generally applicable and highly efficient solution for streaming window aggregation. • We show that general stream slicing is generally applicable and offers better performance than alternative approaches. 27.03.2019 Efficient Window Aggregation with General Stream Slicing 21 tu-berlin-dima.github.io/scotty-window-processor Open Source Repository: