Think of ksqlDB before using KafkaStreams, Patrick Neff, Data Scientist, BAADER Meetup link: https://www.meetup.com/Hamburg-Kafka/events/284618386/

1. Think of ksqlDB before
using Kafka Streams

2. Patrick Neff Think of ksqlDB before using Kafka Streams
Who am I
2 / 9
Patrick Neff
Data Scientist /
Software Developer
patrick-neff@web.de
patrick-neff

3. Patrick Neff Think of ksqlDB before using Kafka Streams
Content
3 / 9
● Introduction
● Kafka Streams
● ksqlDB
● Comparison

4. Patrick Neff Think of ksqlDB before using Kafka Streams
In more detail
4 / 9
● Set up Kafka locally
○ docker-compose
● Kafka Streams
○ Stateless and stateful transformations
● ksqlDB
○ Scalar and aggregate functions
● Comparison
Avro Schema

5. Patrick Neff Think of ksqlDB before using Kafka Streams 5 / 9
{
"sensorId": "sensor_1",
"value": [
{
"type": "temperature",
"value": 66.5,
"unit": "Fahrenheit"
},
{
"type": "humidity",
"value": 71.7,
"unit": "Percent"
}
],
"timestamp": "2022-03-25T10:55:47.070Z"
}
{
"key": {
"sensorId": "sensor_1",
"type": "temperature"
},
"value": {
"valueAvg": 19.7,
"count": 5,
"unit": "Celsius",
"timestamp": "2022-03-25T10:56:59.332Z"
}
}
{
"key": {
"sensorId": "sensor_1",
"type": "humidity"
},
"value": {
"valueAvg": 72.5,
"count": 5,
"unit": "Percent",
"timestamp": "2022-03-25T10:56:59.332Z"
}
}
IoT sensor
From raw to structured data
2 sec
10 sec

6. Patrick Neff Think of ksqlDB before using Kafka Streams
Data wrangling
6 / 9
● Transform:
○ Fahrenheit -> Celsius
● Split:
○ one event with all types -> one event per type
● Aggregate over time:
○ group by aggregation by hopping window

7. Patrick Neff Think of ksqlDB before using Kafka Streams 7 / 9
Let’s dig into it
Code is available under:
https://github.com/pneff93/Think-of-ksqlDB-before-using-K
afkaStreams

8. Patrick Neff Think of ksqlDB before using Kafka Streams 8 / 9
Comparison
Kafka Streams ksqlDB
● You need to deploy a service ● You are very fast up and
running with less and intuitive
code
● You can test the topology
(TopologyTestDriver)
● You can test single statements
with a select statement
● You can define your own
aggregation functions
● You can implement UDF (not for
Confluent Cloud so far)

9. Patrick Neff Think of ksqlDB before using Kafka Streams 9 / 9
What’s next?
Hope to see you there … :)