"Kafka Streams is a powerful stream-processing library that offers stateful operations. Along with the stateful operations is a unique feature of Kafka Streams, Interactive Queries, or IQ. IQ allows you to leverage the application's state from the outside by directly querying the state stores. IQ has always been a part of Kafka Streams, but the original version lacked some key features. Namely, the ability to query the state of a custom state store. With the addition of IQ v2, that has now changed. Additionally, IQ v2 brings other benefits like pushing queries down to the store level making querying more efficient. In this talk, I will cover: * Building an Interactive Query service including routing queries between app instances * Creating custom queries using Interactive Query v2 * Testing your IQ Application Attending this talk will teach you how to use IQ v2 to create queries more aligned with your unique business needs."

1. Building Interactive Queries
In Kafka Streams (part deux)
Bill Bejeck
@bbejeck

2. @bbejeck
Nice to meet you!
• Member of the DevX team
• Prior to DevX ~3 years as engineer on Kafka Streams team
• Apache Kafka® Committer and PMC member
• Author of “Kafka Streams in Action” - 2nd edition underway!
2

3. @bbejeck
Agenda
• Kafka Streams Background
• Interactive Queries
• Internal Communication
• Custom Queries
• Performance Considerations
3

4. Kafka Streams Background

5. @bbejeck
Kafka Streams – DSL API
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String > stockStr = builder.stream(”stocks”);
stockStr.groupByKey()
.aggregate(()-> new TradeStats(),
(k, v, tradeStats) -> tradeStats.add(v),
Materialized.as(“trade-stats-store”))
.toStream().mapValues(..)
.to(“output”);
5

6. @bbejeck
Processor Topology
builder.stream(..)
stockStr.groupByKey()
.aggregate()
mapValues()
to(..)
6

7. @bbejeck
Tasks
7

8. @bbejeck
Stream Threads
8

9. Interactive Queries

10. @bbejeck
Interactive Queries
12

11. @bbejeck
Interactive Queries
13

12. @bbejeck
Interactive Queries
Naming the store makes it available
for Interactive Queries
Materialized.as(“trade-stats-store”))
14

13. @bbejeck
Interactive Queries
Set the application.server configuration
15

14. @bbejeck
Interactive Queries
server.port=7076
application.server=http://somehost:${server.port}
16

15. @bbejeck
Executing Queries
Two step process
1. Find the correct host *
2. Query the current instance or forward to the correct one
17

16. @bbejeck
Executing Queries
KeyQueryMetadata =
KafkaStreams.queryMetadataForKey(StoreName,
<Key>,
<Key Serializer>)
18

17. @bbejeck
Executing Queries
if (keyMetadata.activeHost().equals(thisHostInfo)) {
//query current Kafka Streams instance
}
19

18. @bbejeck
Executing Queries
} else {
return doRemoteQuery(...)
}
20

19. @bbejeck
Executing Queries
StateQueryResult<R> result =
kafkaStreams.query(
StateQueryRequest.inStore(storeName)
.withQuery(Query<R> instance)
.enableExecutionInfo()
);
21

20. @bbejeck
Executing Queries – Query Types
KeyQuery
RangeQuery *
WindowKeyQuery
WindowRangeQuery *
22

21. @bbejeck
Executing Queries – Query Types
23

22. Building The Query Service

23. @bbejeck
Building The Query Service
26

24. @bbejeck
Building The Query Service
Spring Boot
1. Provides the Spring container
2. Embeds a web-server
3. Quickly set up a stand-alone application
27

25. @bbejeck
KafkaStreams Instance
@Component
public class KafkaStreamsService {
@Bean
public KafkaStreams kafkaStreams() {
return kafkaStreams;
}
}
28

26. @bbejeck
Start Kafka Streams
@PostConstruct
public void init() {
kafkaStreams = new KafkaStreams(topology(), properties);
kafkaStreams.cleanUp(); //Optional – not a production setting
kafkaStreams.start();
}
29

27. @bbejeck
Stop Kafka Streams
@PreDestroy
public void tearDown(){
kafkaStreams.close(Duration.ofSeconds(10));
}
30

28. @bbejeck
Expose REST endpoint
@RestController
@RequestMapping("/streams-iq”)
public class StockController {
private final KafkaStreams kafkaStreams;
private final RestTemplate restTemplate
}
31

29. @bbejeck
Handle Incoming Requests
private static final String IQ_URL = "http://{host}:{port}/streams-iq{path}";
@GetMapping(value = "/range")
public QueryResponse<List<V>> getAggregationRange(
@RequestParam(required = false) String lower,
@RequestParam(required = false) String upper,
@RequestParam(required = false) String predicate) {
//Execute the query
32

30. IQ Internal Communication

31. @bbejeck
Internal Communication
34

32. @bbejeck
Internal Communication
35

33. @bbejeck
Internal Communication
36

34. @bbejeck
Internal Communication
37

35. @bbejeck
Internal Communication
38

36. @bbejeck
Internal Communication
39

37. @bbejeck
Internal Communication
40

38. @bbejeck
Internal Communication
41

39. @bbejeck
Why gRPC?
42
•High performance RPC framework
•Simple and quick service definition with a proto file
•Uses Protocol Buffers for serialization
•More performant than REST*

40. @bbejeck
Why gRPC?
43
•Easy integration via Spring Boot
•Build support with gradle/maven plugins

41. @bbejeck
gRPC Proto File Definitions
44
message StockTransactionAggregationProto {
string symbol = 1;
double buys = 2;
double sells = 3;
...
}

42. @bbejeck
gRPC - Proto File Definitions
45
service InternalQuery {
rpc KeyQueryService (KeyQueryRequestProto) returns (QueryResponseProto);
rpc MultiKeyQueryService (MultKeyQueryRequestProto) returns (QueryResponseProto);
rpc RangeQueryService (RangeQueryRequestProto) returns (QueryResponseProto);
}

43. @bbejeck
gRPC – Define the service
46
@GRpcService
@Component
public class InternalQueryService extends
InternalQueryGrpc.InternalQueryImplBase {
}

44. @bbejeck
gRPC – Define the service
47
@GRpcService
@Component
public class InternalQueryService extends
InternalQueryGrpc.InternalQueryImplBase {
private final KafkaStreams kafkaStreams;
@Autowired
public InternalQueryService(KafkaStreams kafkaStreams) {
this.kafkaStreams = kafkaStreams;
}
}

45. @bbejeck
gRPC – Define the service
48
@Override
public void keyQueryService(
final KeyQueryRequestProto request,
final StreamObserver<QueryResponseProto> responseObserver) {
}

46. @bbejeck
gRPC – Define the service
49
@Override
public void keyQueryService(
final KeyQueryRequestProto request,
final StreamObserver<QueryResponseProto> responseObserver) {
....
queryResult = kafkaStreams.query(...)
Builder responseBuilder = QueryResponseProto.newBuilder();
aggregation = queryResult.getResult();
....
responseBuilder.addAggregations(aggregation);
responseObserver.onNext(repsonseBuilder.build());
responseObserver.onCompleted();
}

47. @bbejeck
gRPC – Internal communication
50

48. @bbejeck
gRPC Tradeoffs
51
• Use Protobuf in the Kafka Streams app?
• Need to have Protobuf equivalents for query related classes

49. @bbejeck
gRPC Tradeoffs
52
HostInfo {
private String host;
private int port;
...
}
message HostInfoProto {
string host = 1;
int32 port = 2;
}

50. @bbejeck
gRPC Tradeoffs
53
@Bean
public Jackson2ObjectMapperBuilderCustomizer
jackson2ObjectMapperBuilderCustomizer() {
JsonFormat.TypeRegistry typeRegistry =...
return objectMapperBuilder ->
objectMapperBuilder.serializerByType(....) {
...
jsonGenerator.writeRawValue(printer.print(message));
}
}

51. Custom Queries

52. @bbejeck
Custom Queries- Why?
55

53. @bbejeck
Custom Queries- Why?
56

54. @bbejeck
Custom Queries- Steps
1. Extend the Query interface
2. Create wrapper delegate class for state stores * and supplier
3. Override the KeyValueStore.query method
* KIP proposal in the works for registering a custom handler
57

55. @bbejeck
Custom Queries- Extend Query interface
public interface CustomQuery<R> extends Query<R> {
enum Type {
MULTI_KEY,
FILTERED_RANGE
}
Type type();
}
58

56. @bbejeck
Custom Queries- Wrapper store class
public abstract class StoreDelegate implements KeyValueStore<Bytes, byte[]> {
private final KeyValueStore<Bytes, byte[]> delegate;
StoreDelegate(KeyValueStore<Bytes, byte[]> delegate) {
this.delegate = delegate;
}
@Override
public void put(Bytes key, byte[] value) {
delegate.put(key, value);
}
59

57. @bbejeck
Custom Queries- Store wrapper
public class CustomQueryStore extends StoreDelegate {
@Override
public <R> QueryResult<R> query(final Query<R> query,
final PositionBound positionBound,
final QueryConfig config) {
if (!(query instanceof CustomQuery)) {
return super.query(query, positionBound, config);
}
}
60

58. @bbejeck
Custom Queries- Store wrapper
else {
CustomQuery.Type queryType = ((CustomQuery<?>) query).type();
return switch (queryType) {
case MULTI_KEY -> handleMultiKeyQuery(query, positionBound, config);
case FILTERED_RANGE -> handleFilteredRangeQuery(query, positionBound, config);
default -> QueryResult.forUnknownQueryType(query, this);
};
}
61

59. @bbejeck
Custom Queries- Extend Query interface
public class FilteredRangeQuery<K, V>
implements CustomQuery<KeyValueIterator<K, V>> {
private final String predicate;
private final Optional<K> lowerBound;
private final Optional<K> upperBound;
....
}
62

60. @bbejeck
Custom Queries- Filtered Range
63
serialize(aggregation) -> { price:9, value:400 }

61. @bbejeck
Custom Queries- Filtered Range
64
JsonPath - https://github.com/json-path/JsonPath
$.[ ( @.price >= 9) ]

62. @bbejeck
Custom Queries- Filtered Range
65
try (KeyValueIterator<Bytes, byte[]> iterator = range(....)) {
// build up JSON results
});
// From client pushed to all Kafka Streams clients
predicate = “@.numberShares > 10 && @.buys > 50 “
filteredJsonResults = JsonPath.using(jsonPathConfig)
.parse(json)
.read("$.[?(" + predicate + ")]", protoTypeRef)

63. @bbejeck
Plugging into Kafka Streams
66
KeyValueBytesStoreSupplier persistentSupplier =
CustomQueryStores.customPersistentStoreSupplier(...);
Materialized<......> materialized = Materialized.as(persistentSupplier);
.....
inputStream.groupByKey()
.aggregate(initializer,
(k, v, agg) ->...., materialized)
.....

64. Demo

65. Performance Considerations

66. @bbejeck
Performance Considerations
69

67. @bbejeck
Performance Considerations
70
1. Key space of 200
2. Produce 8K records per second
3. Run Kafka Streams for 10 minutes
4. Record process rate from Thread metrics
5. Run IQ for 10 minutes then record process rate again

68. @bbejeck
Performance Considerations
71
while true
do
curl -X GET http://localhost:7076/streams-iq/range > /dev/null
done
~ 55 / sec
~ 3300 / minute

69. @bbejeck
Performance Considerations
Process rate without running IQ = 7337
Process rate running IQ = 6797
Performance penalty of 7.35%
72

70. @bbejeck
Summary
• Interactive Queries allow to you view the state of a running Kafka Streams
application
• By leveraging Spring Boot and creating an @RestController quickly get a
Kafka Streams application with IQ enabled
• gRPC provides a good mechanism for internal communication
• IQ V2 allows for querying a custom store.
73

71. @bbejeck
Resources
• Source Code - https://github.com/bbejeck/KafkaStreamsInteractiveQueries
• Kafka Tutorials - https://kafka-tutorials.confluent.io/
• Confluent Developer - developer.confluent.io
• Spring Boot - https://spring.io/projects/spring-boot/
• Spring Kafka - https://spring.io/projects/spring-kafka
• Documentation - https://docs.confluent.io/current/streams
74