In the last decade, many distributed stream processing engines (SPEs) were developed to perform continuous queries on massive online data. The central design principle of these engines is to handle queries that potentially run forever on data streams with a query-at-a-time model, i.e., each query is optimized and executed separately. In many real applications, streams are not only processed with long-running queries, but also thousands of short-running ad-hoc queries. To support this efficiently, it is essential to share resources and computation for stream ad-hoc queries in a multi-user environment. The goal of this talk is to bridge the gap between stream processing and ad-hoc queries in SPEs by sharing computation and resources. We define three main requirements for ad-hoc shared stream processing: (1) Integration: Ad-hoc query processing should be a composable layer which can extend stream operators, such as join, aggregation, and window operators; (2) Consistency: Ad-hoc query creation and deletion must be performed in a consistent manner and ensure exactly-once semantics and correctness; (3) Performance: In contrast to state-of-the-art SPEs, ad-hoc SPE should not only maximize data throughput but also query throughout via incremental computation and resource sharing. Based on these requirements, we have developed AStream, an ad-hoc, shared computation stream processing framework. To the best of our knowledge, AStream is the first system that supports distributed ad-hoc stream processing. AStream is built on top of Apache Flink. Our experiments show that AStream shows comparable results to Flink for single query deployments and outperforms it in orders of magnitude with multiple queries.

1. AStream: Ad-hoc Shared Stream
Processing
Jeyhun Karimov (DFKI), Tilmann Rabl (HPI), Volker Markl (TU Berlin, DFKI)
14.10.2019 FlinkForward 2019, Berlin 1

2. Use-case
• Online gaming scenarios
• Ads stream
• Purchases stream
• Many users, high query
and data throughput
• Ad-hoc stream queries
Source: https://www.spiegel.de/netzwelt/games/pubg-global-invitational-berlin-im-e-
sport-fieber-oder-auch-nicht-a-1220440.html
14.10.2019 FlinkForward 2019, Berlin 2

3. Use-case
• Q1 – Financial analysis
• Q2 – Behavior analysis
• Q3 – Loyalty analysis
• Problems:
• Ad-hoc query creation and deletion
• Different selection predicates
• Consistency
• Performance
14.10.2019 FlinkForward 2019, Berlin 3

4. AStream and Requirements for Ad-hoc Stream
Processing
• Integration
• Easily integrate with modern stream processing engines (SPEs)
• Benefit from built-in features of SPEs
• Consistency
• Consistently add and remove user queries
• Ensure exactly-once semantics and correctness of results
• Performance
• Modern SPEs maximize data throughput and minimize latency
• Ad-hoc SPE should maximize data + query throughput and minimize latency
14.10.2019 FlinkForward 2019, Berlin 4

5. AStream System Overview
14.10.2019 FlinkForward 2019, Berlin 5

6. Data Model – Query-set
• Query-set is a bitmap
appended to each tuple
Time T1: Create Q1 and Q2 Time T2: Delete Q2, Create Q3
• Naïve approach
• Leads to a sparse
query-set
• AStream approach
14.10.2019 FlinkForward 2019, Berlin 6

7. Computation Model
• Window operator
• Non-overlapping slices
• Shared join
• Incremental computing
14.10.2019 FlinkForward 2019, Berlin 7
Remove slice

8. Implementation Details – Joining Slices
• Consistent join between slices
• Filter redundant computation
14.10.2019 FlinkForward 2019, Berlin 8
Slice data
structure

9. Implementation details
• Optimizations
• Incremental query processing
• Redundant data copy and shuffling
elimination
• Dynamic slice data structure
• Exactly-once semantics
• AStream inherits Flink’s fault-
tolerance semantics
• All AStream components are
deterministic
• Aggregation query sharing
• Similar semantics with join query sharing
• One input stream operator
• Integration
• Designed as a composable layer over an SPE
• Can also be integrated with Apache Spark,
Storm, and etc.
14.10.2019 FlinkForward 2019, Berlin 9

10. Experiments
• 4- and 8-node cluster setups
• Apache Flink v1.5.2 vs AStream
• Two workload scenarios (SC1, SC2)
SC1 SC2
14.10.2019 FlinkForward 2019, Berlin 10

11. Workload Scenario 1
• Overall data throughput
• Flink cannot sustain the workload
0.9M
1.06M
2.5M
3.9M
1.4M
1.5M
3.5M
5.8M
1.3M
1.6M
3.5M
4.5M
1.8M
2.15M
4.5M
6.1M
Deployment latency for 1q/s 20qp
14.10.2019 FlinkForward 2019, Berlin 11
FlinkAStream

12. Workload Scenario 2
• More dynamic workload
• Fewer queries
• Better event-time latency
• Short-running queries
• Smaller query-set and changelog size
14.10.2019 FlinkForward 2019, Berlin 12

13. Conclusion
• AStream – The First Distributed Ad-hoc SPE
• State-of-the-art SPEs are not designed to perform ad-hoc queries
• Not only data latency and throughput but also query deployment latency and
query throughput are bottlenecks
• AStream supports majority of stream use-cases
• Windowed joins, aggregations, and combinations of them
• AStream ensures (1) ease of integration, (2) correctness and
consistency, and (3) high performance
14.10.2019 FlinkForward 2019, Berlin 13

14. Backup slides: Sharing Overhead
14.10.2019 FlinkForward 2019, Berlin 14

15. Data Model - Changelog-set
Ad-hoc queries
Changelogs w.r.t. adjacent time slot
Changelog w.r.t. older time slot
14.10.2019 FlinkForward 2019, Berlin 15

16. Workload Scenario 1
• Event-time latency increases with more parallel queries
• Sustainable latency – no prolonged backpressure
• Changelog batch-size impacts the deployment latency
14.10.2019 FlinkForward 2019, Berlin 16