Bill Bejeck, Software Engineer, Confluent

1. 1
Kick Your Database to the Curb
Using Kafka Streams Interactive Queries to Enable
Powerful MicroServices

2. 2
Brief Introduction
• Worked at Confluent (Streams Team) 2 years
• Apache Kafka Committer
• Author Kafka Streams in Action
Special thanks to @gamussa!

3. 3
Agenda
• What is State
• Kafka Streams Overview
• Describe Interactive Queries
• Live Demo!

4. 4
Stateful Stream Processing
What is State?

5. 5
Stateful Stream Processing
What is State?
Information your application needs to remember
beyond the scope of a single record

6. GroupBy Example
public static void main(String[] args) {
int counter = 0;
int sendInterval = 15;
Map<String, Integer> groupByCounts = new HashMap<>();
try(..consumer = new KafkaConsumer<>(consumerProperties());
..producer = new KafkaProducer<>(producerProperties())){
consumer.subscribe(Arrays.asList(”A”,”B”));

7. GroupBy Example
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}

8. GroupBy Example
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}

9. GroupBy Example
if(counter++ % sendInterval == 0) {
for(Entry<String, Integer> groupedEntry:groupByCounts.entrySet()){
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord<>("group-by-counts",
groupedEntry.getKey(),
groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}

10. GroupBy Example
if(counter++ % sendInterval == 0) {
for(Entry<String, Integer> groupedEntry:groupByCounts.entrySet()){
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord<>("group-by-counts",
groupedEntry.getKey(),
groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}

11. 11
Streams GroupBy
...
stream = streamBuilder.stream(Arrays.asList(“A”, “B”))
stream.groupByKey()
.count()
.toStream()
.to(“output-topic”,
Produced.with(Serdes.String(), Serdes.Long()))

12. 12
Streams GroupBy
...
stream = streamBuilder.stream(Arrays.asList(“A”, “B”))
stream.groupByKey()
.count()
.toStream()
.to(“output-topic”,
Produced.with(Serdes.String(), Serdes.Long()))

13. 13
Streams GroupBy
...
stream = streamBuilder.stream(Arrays.asList(“A”, “B”))
stream.groupByKey()
.count()
.toStream()
.to(“output-topic”,
Produced.with(Serdes.String(), Serdes.Long()))

14. 14
Stateful Stream Processing
Streams Stateful Operations
• Joins
• Windowing operations
• Aggregation/Reduce
Using any of these operations, Streams creates a state
store

15. 15
Making Streams Results Queryable
Kafka Streams Application
KAFKA
External Application /
REST Service

16. 16
Making Streams Results Queryable
Kafka Streams Application
KAFKA
External Application /
REST Service
Database

17. 17
Making Streams Results Queryable
Kafka Streams Application
KAFKA
External Application /
REST Service
Database

18. 18
Making Streams Queryable
stream.groupByKey()
.count()
.toStream()
.to(“output-topic”,
Produced.with(Serdes.String(), Serdes.Long()))
..
consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
someService.save(record.key(), record.value())
..
}
..

19. 19
Making Streams State Directly Queryable
...
stream = streamBuilder.stream(Arrays.asList(“A”, “B”))
stream.groupByKey()
.count(Materialized.as(“count-store”))
.toStream()
.to(“output-topic”,
Produced.with(Serdes.String(), Serdes.Long()))

20. 20
Making Streams State Directly Queryable
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG,
"ks-interactive-stock-analysis-appid");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
"localhost:9092");
properties.put(StreamsConfig.APPLICATION_SERVER_CONFIG,
host+":"+port);
...

21. 21
Making Streams State Directly Queryable
Kafka Streams Application
KAFKA
Embedded RPC

22. 22
What’s with the APPLICATION_SERVER_ID
• A single Streams instance doesn’t contain all keys
• Streams will query other instances for store misses
• A single Streams instance can be proxy for all instances

23. 24
Making Streams Results Queryable
Streams app “A”
Host = hostA:4567
KAFKA
Streams app “B”
Host = hostB:4568

24. 25
Making Streams Results Queryable
Streams app “A”
Host = hostA:4567
Metadata -> hostB:4568
KAFKA
Streams app “B”
Host = hostB:4568
Metadata -> hostA:4567

25. 26
Topic Partitions and Streams Tasks
Streams app “A”
Host = hostA:4567
Streams app “B”
Host = hostB:4568
State Store State Store
Topic with four partitions
Four partitions are converted to 4 tasks so
each streams application is assigned 2
partitions/tasks

26. 27
Making Streams Results Queryable
Streams app “A”
Host = hostA:4567
Streams app “B”
Host = hostB:4568
State Store State Store
{“ENERGY”:”10000”} written to
partition 0 assigned to App A
{“FINANCE”:”11000”} written to
partition 1 assigned to App B

27. 28
Making Streams Results Queryable
Streams app “A”
Host = hostA:4567
Streams app “B”
Host = hostB:4568
State Store State Store
{“ENERGY”:”10000”} written to
partition 0 assigned to App A
{“FINANCE”:”11000”} written to
partition 1 assigned to App B
http://hostA:4567?key=FINANCE

28. 29
Example of a Streams RPC
Kafka Streams Application
KAFKA
{ JS }
Demo Time!

29. 30
Embedding the Web Server
KafkaStreams kafkaStreams =
new KafkaStreams(builder.build(), streamsConfig);
InteractiveQueryServer queryServer =
new InteractiveQueryServer(kafkaStreams, hostInfo);

30. 31
Embedding the Web Server
.
kafkaStreams.setStateListener(((newState, oldState) -> {
if (newState == KafkaStreams.State.RUNNING
&& oldState == KafkaStreams.State.REBALANCING) {
queryServer.setReady(true);
} else if (newState != KafkaStreams.State.RUNNING) {
queryServer.setReady(false);
}
}))

31. 32
Embedding the Web Server
.
kafkaStreams.setStateListener(((newState, oldState) -> {
if (newState == KafkaStreams.State.RUNNING
&& oldState == KafkaStreams.State.REBALANCING) {
queryServer.setReady(true);
} else if (newState != KafkaStreams.State.RUNNING) {
queryServer.setReady(false);
}
}))

32. 33
Embedding the Web Server
public void init() {
get("/window/:store/:key/:from/:to", (req, res) -> ready ?
fetchFromWindowStore(req.params()) : STORES_NOT_ACCESSIBLE);
get("/window/:store/:key", (req, res) -> ready ?
fetchFromWindowStore(req.params()) : STORES_NOT_ACCESSIBLE);
get("/kv/:store", (req, res) -> ready ? fetchAllFromKeyValueStore(req.params()) :
STORES_NOT_ACCESSIBLE);
get("/kv/:store/:local", (req, res) -> ready ?
fetchAllFromLocalKeyValueStore(req.params()) : STORES_NOT_ACCESSIBLE);
get("/session/:store/:key",
(req, res) -> ready ?
fetchFromSessionStore(req.params()) : STORES_NOT_ACCESSIBLE);
get("/iq",
(req, res) -> {
res.redirect("interactiveQueriesApplication.html");
return "";
});

33. 34
Embedding the Web Server
fetchFromSessionStore(Map<String, String> params) {
String store = params.get(STORE_PARAM);
String key = params.get(KEY_PARAM);
HostInfo storeHostInfo = getHostInfo(store, key);
if (storeHostInfo.host().equals("unknown")) {
return STORES_NOT_ACCESSIBLE;
}
if (dataNotLocal(storeHostInfo)) {
return fetchRemote(storeHostInfo, "session", params);
}
ReadOnlySessionStore<String, CustomerTransactions> readOnlySessionStore = kafkaStreams.store(store,
QueryableStoreTypes.sessionStore());

34. 35
Embedding the Web Server
getHostInfo(String storeName, String key) {
StreamsMetadata metadata =
kafkaStreams.metadataForKey(storeName, key, stringSerializer);
return metadata.hostInfo();
}

35. 36
Embedding the Web Server
fetchFromSessionStore(Map<String, String> params) {
String store = params.get(STORE_PARAM);
String key = params.get(KEY_PARAM);
HostInfo storeHostInfo = getHostInfo(store, key);
if (storeHostInfo.host().equals("unknown")) {
return STORES_NOT_ACCESSIBLE;
}
if (dataNotLocal(storeHostInfo)) {
return fetchRemote(storeHostInfo, "session", params);
}
ReadOnlySessionStore<String, CustomerTransactions> readOnlySessionStore = kafkaStreams.store(store,
QueryableStoreTypes.sessionStore());

36. 37
Embedding the Web Server
fetchFromSessionStore(Map<String, String> params) {
String store = params.get(STORE_PARAM);
String key = params.get(KEY_PARAM);
HostInfo storeHostInfo = getHostInfo(store, key);
if (storeHostInfo.host().equals("unknown")) {
return STORES_NOT_ACCESSIBLE;
}
if (dataNotLocal(storeHostInfo)) {
return fetchRemote(storeHostInfo, "session", params);
}
ReadOnlySessionStore<String, CustomerTransactions>
readOnlySessionStore = kafkaStreams.store(
store,
QueryableStoreTypes.sessionStore());

37. 38
Embedding the Web Server
fetchFromSessionStore(Map<String, String> params) {
String store = params.get(STORE_PARAM);
String key = params.get(KEY_PARAM);
HostInfo storeHostInfo = getHostInfo(store, key);
if (storeHostInfo.host().equals("unknown")) {
return STORES_NOT_ACCESSIBLE;
}
if (dataNotLocal(storeHostInfo)) {
return fetchRemote(storeHostInfo, "session", params);
}
// Iterate over readOnlySessionStore and
// store results in a list sessionResults
return gson.toJson(sessionResults);

38. 39
Client View Development
<body>
<h2>Kafka Streams Equities Dashboard Application</h2>
<!–- Other div elements left out for clarity -->
<div id="sessionDiv">
<h3 id="sessionHeader">Customer Session Equity Activity Table</h3>
<table id="sessionTable">
<tr>
<th>Customer Id</th>
<th>Average Equity Transaction Spent Per Session</th>
</tr>
</table>
</div>
</body>

39. 40
Client View Development
<script>
function loadIqTables() {
$.getJSON("/kv/TransactionsBySector", function (response) {
updateTable(response, $('#txnsTable'))
$('#txnsHeader').animate({color:'red'},500).animate({color:'#CCCCCC'}, 500)
})
updateTableWithList("/window/NumberSharesPerPeriod/", symbols,
$('#stockTable'), $('#stockHeader'));
updateTableWithList("/session/CustomerPurchaseSessions/", customers,
$('#sessionTable'), $('#sessionHeader'))
}
setInterval(loadIqTables, 7000);
</script>

40. 41
Client View Development
<script>
function loadIqTables() {
$.getJSON("/kv/TransactionsBySector", function (response) {
updateTable(response, $('#txnsTable'))
$('#txnsHeader').animate({color:'red'},500).animate({color:'#CCCCCC'}, 500)
})
updateTableWithList("/window/NumberSharesPerPeriod/", symbols,
$('#stockTable'), $('#stockHeader'));
updateTableWithList("/session/CustomerPurchaseSessions/", customers,
$('#sessionTable'), $('#sessionHeader'))
}
setInterval(loadIqTables, 7000);
</script>

41. 42
Security
Kafka Streams Application
KAFKA https://

42. 43
Security
ReadOnlySessionStore<String, CustomerTransactions>
readOnlySessionStore = kafkaStreams.store(
store,
QueryableStoreTypes.sessionStore());
try (KeyValueIterator<Windowed<String>, CustomerTransactions>
iterator = readOnlySessionStore.fetch(key)) {
while (iterator.hasNext()) {
//Transform or mask data here and return sanitized
//data
}
}
return gson.toJson(sanitizedRecords);

43. 44
Summary
Interactive Queries is a powerful abstraction that
simplifies stateful stream processing
There are still cases for which external database/storage
might be a better

44. 45
Summary
Kafka Streams in Action Examples: https://github.com/bbejeck/kafka-streams-in-
action/blob/master/src/main/java/bbejeck/webserver/InteractiveQueryServer.java
Music example: https://github.com/confluentinc/examples/blob/master/kafka-
streams/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/
KafkaMusicExample.java
Streaming Movie Ratings: https://github.com/confluentinc/demo-
scene/tree/master/streams-movie-demo

45. 46
Thanks!
Stay in Touch!
• https://slackpass.io/confluentcommunity
• https://www.confluent.io/blog/
• Twitter @bbejeck
• We are hiring! https://www.confluent.io/careers/

46. KS19Meetup.
CONFLUENT COMMUNITY DISCOUNT CODE
25% OFF*
*Standard Priced Conference pass

47. NOMINATE YOURSELF OR A PEER AT
CONFLUENT.IO/NOMINATE