Building an Interactive Query Service in Kafka Streams With Bill Bejeck | Current 2022

Building Interactive Queries
In Kafka Streams
Bill Bejeck
@bbejeck

@bbejeck
Nice to meet you!
• Member of the DevX team
• Prior to DevX ~3 years as engineer on Kafka Streams team
• Apache Kafka® Committer and PMC member
• Author of “Kafka Streams in Action” - 2nd edition underway!
2

@bbejeck
Agenda
• Kafka Streams Background
• Interactive Queries
• Interactive Query Requirements
• Building the Query Service
• Executing Queries
3

@bbejeck
Kafka Streams – DSL API
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String > stockStr = builder.stream(”stocks”);
stockStr.groupByKey()
.windowedBy(TimeWindows.of(Duration.ofMinutes(5))
.aggregate(()-> new TradeStats(),
(k, v, tradeStats) -> tradeStats.add(v),
Materialized.as(“trade-stats-store”))
.toStream().mapValues(..)
.to(“output”);
6

@bbejeck
Processor Topology
builder.stream(..)
stockStr.groupByKey()
.windowedBy()
.aggregate()
mapValues()
to(..)
7

@bbejeck
State Management & Task Assignment
13

@bbejeck
State Management & Task Assignment
14

@bbejeck
Fault Tolerance with State
15

@bbejeck
Solution: Standby Task
16

@bbejeck
Failover with Standby Tasks
17

@bbejeck
Standby Frictional Lag
18

@bbejeck
Traditional Querying
20

@bbejeck
Interactive Queries
21

@bbejeck
Interactive Queries
22

Interactive Query Requirements

@bbejeck
Naming the store makes it available
for Interactive Queries
Materialized.as(“trade-stats-store”))
27

@bbejeck
Set the application.server configuration
28

@bbejeck
server.port=7076
application.server=http://somehost:${server.port}
29

Implementing The Query Service

@bbejeck
1. Inversion of Control container
2. Uses Dependency Injection to implement IoC
3. Inject dependencies without implementation specifics
4. Autowiring achieved with annotations
31

@bbejeck
Spring Boot
1. Provides the Spring container
2. Embeds a web-server
3. Quickly set up a stand-alone application
32

@bbejeck
1. Build the Topology
2. Handle Kafka Streams Lifecycle
3. Expose REST endpoint / Handle Queries
4. A view for rendering query results
33

@bbejeck
Build the Topology
@Component
public class AggregationStream {
private final KafkaStreamsAppConfiguration streamsConfiguration;
@Autowired
public AggregationStream(KafkaStreamsAppConfiguration streamsConfiguration) {
this.streamsConfiguration = streamsConfiguration;
}
34

@bbejeck
Build the Topology
public Topology topology() {
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> input =
builder.stream(streamsConfiguration.inputTopic(),
Consumed.with(stringSerde, stringSerde)
//Details omitted
return builder.build()
}
35

@bbejeck
KafkaStreams Instance
@Component
public class KafkaStreamsContainer {
private final KafkaStreamsAppConfiguration appConfiguration;
private final AggregationStream aggregationStream;
private KafkaStreams kafkaStreams
@Autowired
public KafkaStreamsContainer(final AggregationStream aggregationStream,
final KafkaStreamsAppConfiguration appConfiguration) {
}
//Details omitted
36

@bbejeck
KafkaStreams Instance
@Component
public class KafkaStreamsContainer {
@Bean
public KafkaStreams kafkaStreams() {
return kafkaStreams;
}
}
37

@bbejeck
Start Kafka Streams
@PostConstruct
public void init() {
Properties properties = appConfiguration.streamsConfigs();
Topology topology = aggregationStream.topology();
kafkaStreams = new KafkaStreams(topology, properties);
kafkaStreams.cleanUp(); //Optional – not a production setting
kafkaStreams.start();
}
38

@bbejeck
Stop Kafka Streams
@PreDestroy
public void tearDown(){
kafkaStreams.close(Duration.ofSeconds(10));
}
39

@bbejeck
Expose REST endpoint
@RestController
@RequestMapping("/streams-iq”)
public class StockController {
private final KafkaStreams kafkaStreams;
private final RestTemplate restTemplate
}
40

@bbejeck
Inject KafkaStreams Object
@Autowired
public StockController(final KafkaStreams kafkaStreams,
final RestTemplateBuilder restTemplateBuilder) {
this.kafkaStreams = kafkaStreams;
this.restTemplate = restTemplateBuilder.build()
}
41

@bbejeck
Set “this” HostInfo
@Value("${application.server}")
private String applicationServer;
. . . .
@PostConstruct
public void init() {
String[] parts = applicationServer.split(":");
thisHostInfo = new HostInfo(parts[0], Integer.parseInt(parts[1]));
}
42

@bbejeck
Set “this” HostInfo
43

@bbejeck
Handle Incoming Requests
private static final String IQ_URL = "http://{host}:{port}/streams-iq{path}";
@GetMapping(value = "/keyquery/{symbol}")
public QueryResponse<V> getAggregation(@PathVariable String symbol) {
//Execute the query
}
44

@bbejeck
Handle Incoming Requests
private static final String IQ_URL = "http://{host}:{port}/streams-iq{path}";
@GetMapping(value = "/range")
public QueryResponse<List<V>> getAggregationRange(
@RequestParam(required = false) String lower,
@RequestParam(required = false) String upper) {
//Execute the query
45

@bbejeck
Executing Queries
Two step process
1. Find the correct host *
2. Query the current instance or forward to the correct one
47

@bbejeck
Executing Queries
KeyQueryMetadata =
KafkaStreams.queryMetadataForKey(StoreName,
<Key>,
<Key Serializer>)
48

@bbejeck
Executing Queries
HostInfo = KeyQueryMetadata.activeHost()
Set<HostInfo> = KeyQueryMetadata.standByHosts()
int partition = KeyQueryMetadata.partition();
49

@bbejeck
Executing Queries
if (keyMetadata.activeHost().equals(thisHostInfo)) {
//query current Kafka Streams instance
}
50

@bbejeck
Executing Queries
} else {
String host = keyMetadata.activeHost().host();
int port = keyMetadata.activeHost().port();
String path = "/keyquery/” + symbol;
51

@bbejeck
Executing Queries
} else {
String host = keyMetadata.activeHost().host();
int port = keyMetadata.activeHost().port();
String path = String.format("/keyquery/" + symbol;
try {
resp = restTemplate.getForObject(IQ_URL, QueryResponse.class, host, port, path);
} catch (RestClientException exception) {
//Handle error
}
52

@bbejeck
Executing Queries – Query All Hosts*
53

@bbejeck
Collection<StreamsMetadata> storeMetadata =
KafkaStreams.allMetadataForStore(“store-name”);
Collection<StreamsMetadata> allStreamsMetadata =
KafkaStreams.metadataForAllStreamsClients();*
54

@bbejeck
@GetMapping(value = "/range")
public QueryResponse<R> getAggregationRange(
@RequestParam(required = false) String upper) {
Collection<StreamsMetadata> streamsMetadata =
kafkaStreams.streamsMetadataForStore(storeName);
streamsMetadata.forEach(streamsClient -> {
// query each client
}
55

@bbejeck
@GetMapping(value = "/internal/range")
public QueryResponse<R> getAggregationRangeInternal(
@RequestParam(required = false) String upper,
@RequestParam(required = false) List<> partitions
return doRangeQuery(thisHostInfo, partitions, lower, upper);
}
56

@bbejeck
Executing Queries
StateQueryResult<R> result =
KafkaStreams.query(
StateQueryRequest.inStore(StoreName)
.withQuery(Query<R> instance)
);
58

@bbejeck
Executing Queries – Query Types
KeyQuery
RangeQuery *
WindowKeyQuery
WindowRangeQuery *
59

@bbejeck
Executing Queries – Options
// Default is active or standby
StateQueryRequest.requireActive();
//Default is all partitions
StateQueryRequest.withPartitions(Set<Integer>partitions);
60

@bbejeck
Executing Queries – Options
StateQueryRequest.withPositionBound(PositionBound pb);
StateQueryRequest.enableExecutionInfo();
61

@bbejeck
Executing Queries – Results
Map<Integer QueryResult<R>> result =
StateQueryResult.getPartitionResults();
//other methods
StateQueryResult.getGlobalResult();
StateQueryResult.getOnlyPartitionResult();
62

@bbejeck
Index Page Location
src
main
resources
public
index.html
js and css files
64

@bbejeck
Single Page Application
<script type="text/javascript">
function runIqTables() {
getRangeTableData("http://localhost:7076/streams-iq/range")
updateKeyTableWithList("http://localhost:7076/streams-iq/keyquery/");
}
function getRangeTableData(path) {
$.getJSON(path, function (response) {
. . .
});
}
....
setInterval(runIqTables, 5000);
</script>
65

@bbejeck
Summary
• Interactive Queries allow to you view the state of a running Kafka Streams
application
• You can leverage standbys for higher availability with IQ
• Use Interactive Queries v2
• By leveraging Spring Boot and creating an @RestController quickly get a
Kafka Streams application with IQ enabled
66

@bbejeck
Thank You!
67
@bbejeck
bill@confluent.io

@bbejeck
Resources
• Source Code - https://github.com/bbejeck/KafkaStreamsInteractiveQueries
• Kafka Tutorials - https://kafka-tutorials.confluent.io/
• Confluent Developer - developer.confluent.io
• Spring Boot - https://spring.io/projects/spring-boot/
• Spring Kafka - https://spring.io/projects/spring-kafka
• Documentation - https://docs.confluent.io/current/streams
68

Your Apache Kafka®
journey begins here
developer.confluent.io
69

Building an Interactive Query Service in Kafka Streams With Bill Bejeck | Current 2022

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building an Interactive Query Service in Kafka Streams With Bill Bejeck | Current 2022

Similar to Building an Interactive Query Service in Kafka Streams With Bill Bejeck | Current 2022 (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Building an Interactive Query Service in Kafka Streams With Bill Bejeck | Current 2022