Tim Berglund, Confluent, Senior Director of Developer Relations You’ve got streams of data that you want to process and store? You’ve got events from which you’d like to derive state or build aggregates? And you want to do all of this in a scalable and fault-tolerant manner? It’s just as well that Kafka and ksqlDB exist! This talk will be built around a live demonstration of the concepts and capabilities of ksqlDB. We’ll see how you can apply transformations to a stream of events from one Kafka topic to another. We’ll use ksqlDB connectors to bring in data from other systems and use this to join and enrich streams—and we’ll serve the results up directly to an application, without even needing an external data store. https://www.meetup.com/Mexico-Kafka/events/271111978/

1. ksqlDB
@tlberglund
Stream Processing with

2. Kafka Architecture
consumer
producer
consumer consumer
broker
broker
broker
broker
producer

3. Logs and Pub/Sub
consumer A
producer
consumer B
8764321 5
first record
latest record

4. Scalable Consumption
consumer
group A
producer
consumer
group A
consumer
group B
consumer
group B
…
…
…
partition 1
partition 2
partition 3
Partitioned Topic

5. Streaming Platform
Applications
Databases
Offline SystemsDWH
hdfs/
spark
Stream
Processors
Real-time
analytics Streaming Platform

6. ksqlDBis the
Database
for
Apache Kafka
Streaming

7. CREATE STREAM clickstream (
time BIGINT,
url VARCHAR,
status INTEGER,
bytes INTEGER,
userid VARCHAR,
agent VARCHAR)
WITH (
value_format = ‘JSON’,
kafka_topic=‘my_clickstream_topic’
);
ksqlDB DDL: streams

8. CREATE TABLE users (
user_id INTEGER,
registered_at LONG,
username VARCHAR,
name VARCHAR,
city VARCHAR,
level VARCHAR)
WITH (
key = ‘user_id',
kafka_topic=‘clickstream_users’,
value_format=‘JSON');
ksqlDB DDL: tables

9. Your first ksqlDB Query
SELECT status, bytes
FROM clickstream
WHERE user_agent =
‘Mozilla/5.0 (compatible; MSIE 6.0)’
EMIT CHANGES;

10. Joining a Stream and a Table
CREATE STREAM vip_actions AS
SELECT userid, page, action FROM clickstream c
LEFT JOIN users u ON c.userid = u.user_id
WHERE u.level = 'Platinum';

11. Aggregates and Windows
CREATE TABLE possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 SECONDS)
GROUP BY card_number
HAVING count(*) > 3;

12. KSQL for Data Transformation
CREATE STREAM views_by_userid
WITH (PARTITIONS=6,
VALUE_FORMAT=‘JSON’,
TIMESTAMP=‘view_time’) AS
SELECT * FROM clickstream PARTITION BY user_id;
Make simple derivations of existing topics from the command line

13. How to run KSQL
JVM
KSQL Server
KSQL CLI
JVM
KSQL Server
JVM
KSQL Server
Kafka Cluster
#2 CLIENT-SERVER

14. How to run KSQL
Kafka Cluster
JVM
KSQL Server
JVM
KSQL Server
JVM
KSQL Server
#3 AS A STANDALONE APPLICATION