ksqlDB is a streaming database that uses Kafka Streams to execute queries against data in Apache Kafka®. Historically, each query was compiled into its own Kafka Streams program to be executed inside the ksqlDB servers. As ksqlDB moved to support broader and more complex use cases, this query execution strategy became the bottleneck for scaling up the number of persistent queries. This talk will examine the problems faced and how we addressed them. Using too many Kafka Streams instances requires too many resources in both threads and consumers. One way to avoid this is using Modular Topologies, which are coming to Kafka Streams in KIP-809. Modular Topologies allow us to dynamically change the workload of a Kafka Streams application while it’s running and share resources such as consumer/producer clients and processing threads. This makes it possible to use a single Kafka Streams runtime for multiple topologies that share consumers and threads across them. We will see in detail how this makes it possible for ksqlDB to consolidate queries into a shared Kafka Streams runtime. Kafka Streams developers will take away from this talk an understanding of how to utilize ModularTopologies, and dynamically upgrade their Kafka Streams workload effectively.

1. Using Modular Topologies in Kafka Streams
to scale KSQL’s persistent queries
A. Sophie Blee-Goldman
Walker Carlson
@ableegoldman | sophie@confluent.io
wcarlson@confluent.io
1

2. Overview
• Confluent Cloud limits to 20 persistent queries per cluster.
• We re-architectured KSQL, improving the scalability
2

3. Road map
1. What we improved in KSQL
a. Current World
b. Old World
c. Issues
d. New World
2. How we made it possible
a. Why it’s hard
b. Why it’s possible
c. Why it’s worth it
d. What is next?
3

4. Terminology
• KSQL - A structured language that uses Kafka Streams to execute
• Streams Runtime - single streams client that executes a topology. A group
of these with the same applicationID makes a streams application
• Topology - A Kafka Streams program graph that is a collection of tasks
• Tasks - A unit of work in the topology. Each topic partition has a task.
4

5. Persistent query
Create Stream foo As Select * From bar;
* Limit of 20 on the number of these per-cluster in Confluent Cloud.
Input topic Output topic
Persistent query
~ Topology
Current World
5

6. Streams
Kafka Streams
Metadata
Stream Thread 1
Stream Thread 2
Stream Thread 3
Current World
6
Runtime == One Node in an Streams Application

7. Old architecture
KSQL
Query 1 Query 2 Query 3
Streams Streams
Streams
Persistent Query == Topology
Old World
7

8. Issues of scale
• Threads scale with number of Queries
• Each Threads has its own Consumer
• Assignment logic only accounts for each query
• Long start up times for Streams Runtimes
• Other various overhead
Results in a limited number of persistent queries.
8

9. New Architecture
Query 2 Query 3
Streams Runtime
KSQL
Persistent Query == Topology
Threads, Consumers, ect…
Query 1
New World
9

10. Scaling Improvements
• Queries can share Streams Runtimes
Threads
Metadata
Consumer Group
• Assignment Logic is shared
• Runtimes are already started
• Possibility for topology or state optimizations across queries
10

11. KSQL
Not all topologies can fit into a single runtime yet!
Pre Execution - Runtime Assignment
Runtime
Topology - A
Topology - B
New Runtime
A and B both use the
same source topic
Topology == Persistent Query
New World
11

12. Pre Execution - Query Validation
KSQL
Runtime 1
Runtime 2
Runtime 3
Runtime 4
Validation Runtime 3
New World
Copy of
Potential
Query’s
Topology
Sandbox
copy
Runtime == Kafka Streams Client 12

13. Pre Execution - Multi Cluster Assignment
Needs to go into the same runtime
New Query
Assignment:
Runtime 1
KSQL Server 1
Runtime 1
KSQL Server 2
Runtime 1
Runtime == Kafka Streams Client
Command Topic -
internal topic to keep all
KSQL servers in sync
New World
13

14. Why it’s hard
14

15. 15
Otter monitoring shells == persistent query
Stream processing with Ottla

16. 16
Otter monitoring shells == persistent query
Stream processing
+1 green
1 green shell

17. 17
Otter monitoring shells == persistent query
Stream processing
+1 orange
1 green shell
1 orange shell

18. Persistent Queries
Why it’s hard
Ksql persistent query == Kafka Streams topology
What are they really?
Create Stream foo As Select * From bar;
Input topic Output topic
Persistent query
~ Topology
18

19. Topologies
Why it’s hard
terms_and_ conditions_events
Processor Topology
19
Input Topic
things_you_care_about
Output Topic
source
sink
filter

20. Topologies
Why it’s hard
20
source
sink
filter
Ksql persistent query == Kafka Streams topology
source
sink
count
shell_
count
state store
source
B
sink
join
source
A

21. Why it’s hard
21
Subtopology: subset of processor nodes connected by repartition topics
A
B
to sink…
Repartition topic
subtopology 1
subtopology 2

22. Why it’s hard
22
Kafka Streams runtime == Java application executing a processor topology
X
count
fish
Disjoint Topologies
Streams Runtime
info
X
my_store
shells

23. Why it’s hard
23
Disjoint topologies == subset of nodes with no edges =/= isolation
X
count
fish
Disjoint Topologies
Streams Runtime
info
OUT
my_store
shells
X
count
crabs

24. Why it’s hard
24
Runtimes are static & topologies are not always forwards/backwards compatible
X
count
fish
Static runtime
Streams Runtime
X
count2
crabs
?
+
Query C
info
OUT
my_store
shells
Subtopology 1 Subtopology 2
Query B
Query A

25. Why it’s hard
25
Runtimes are static & topologies are not always forwards/backwards compatible
X
count
fish
Static topology
Streams Runtime
X
count2
crabs
info
OUT
my_store
shells
Subtopology 1 Subtopology 3
Subtopology 2
Query A
Query C Query B

26. Why it’s possible
26
Modular topologies == disjoint topologies that can be dynamically added & removed
X
count
fish
Modular Topologies
Streams Runtime
info
OUT
my_store
shells
X
count
crabs
Topology C
Topology B
Topology A

27. Why it’s possible
27
Remember: modular topologies can’t share any source topics within a runtime!
Modular Topologies
Streams Runtime
Topic A
Topic B
Topic C
Topic D
Topic E

28. Why it’s possible
28
Remember: modular topologies can’t share any source topics within a runtime!
Modular Topologies
Streams Runtime
Topic A
Topic B
Topic C
Topic D
Topic E
0
1 A
A B
?

29. 29
Stream processing
Each otter monitoring shells == 1 persistent query
NOT COOL

30. 30
Stream processing
Each otter monitoring shells == 1 persistent query
Stream processing
Each otter monitoring shells == 1 persistent query

31. Why it’s worth it
Performance
Recap: old engine
● Serious issues past 20 query cliff
● Endless rebalancing*
● Throughput scales poorly
Results: New engine
● No degradation in throughput past 20 & beyond!
● No rebalancing loops
31
*Dropping out of the group – cooperative rebalancing won’t help here!

32. Why it’s worth it
Performance
Stateless query benchmarks (records consumed/s)
32
avg: 80474.66
min: 80438.10
max: 80511.21
(2 runs)
20 queries 100 queries
20 input topics 1 input topic 100 input topics
Non-shared
(old engine) avg: 68,025.3
min: 66741.96
max: 68977.11
(4 runs)
avg: 72,557.94
min: 66,940.67
max: 78,349.66
(4 runs)
avg: 48,553.75
min: 47,992.30
max: 49,005.56
(4 runs)
Shared
(new engine)
avg: 80,474.66
min: 80,438.10
max: 80,511.21
(2 runs)
avg: 62,037.23
min: 61,254.18
max: 62,820.27
(2 runs)
avg: 65,867.26
min: 65,654.27
max: 66,080.25
(2 run)

33. Why it’s worth it
Performance
Stateful query benchmarks (records consumed/s)
33
avg: 80474.66
min: 80438.10
max: 80511.21
(2 runs)
20 queries 100 queries
20 input topics 1 input topic 100 input topics
Non-shared
(old engine) avg: 13,938.41
min: 13,933.81
max: 13,943.01
(2 runs)
avg: 13,754.95
min: 13,519.31
max: 13,990.58
(2 runs)
avg: 12,669.4
min: 12660.73
max: 12678.06
(2 runs)
Shared
(new engine)
avg: 12,897.32
min: 12,419.81
max: 13,374.84
(2 runs)
avg: 10,337.91
min: 10,207.7
max: 10,468.11
(2 runs)
avg: 15,411.09
min: 14,827.35
max: 15,994.83
(2 runs)

34. Why it’s still going
34
Worst case: number of runtimes == number of queries!
Where next?
34
Runtime #1
Topic A
Topic B
Topic C
Topic D
Topic E
Runtime #2

35. 35
Actual processing of shell == persistent query
More natural engine Why it’s worth it

36. Coming to an application near you
36
Kafka Streams users:
Want to participate in the KIP discussion?
Join the kafka dev mailing list by sending an email to
dev-subscribe@kafka.apache.org
Look out for the subject line [DISCUSS] KIP-809 in the next few days!

37. Coming to an application near you
37
KSQL users:
This feature is currently in EA
Contact us if you’re interested in trialing the early product
The feature will become generally available in Cloud
coming soon!

38. Your Apache Kafka
journey begins here
developer.confluent.io