Speaker: Matthias J. Sax, Software Engineer, Confluent KSQL is the Streaming SQL engine for Apache Kafka that allows for continuous data stream processing. While KSQL looks very similar to SQL, it provides quite different semantics. First, KSQL queries can be defined over data streams, not just tables. Second, queries over tables are no snapshot queries, but run forever. And third, time is a core concept in KSQL and data stream processing in general. In this talk, we explore the nature of Streaming SQL and its temporal semantics that apply to both streams and tables. We will explain continuous queries semantics, the relationship between streams and tables, and demystify the temporal nature of KSQL tables. Furthermore, we dig into filter, aggregation, and join operations over stream and tables as well as stream specific operators like windowing. At the end, you will be equipped to query streams and tables using KSQL and understand their temporal relationship to each other.

1. 1
1
Streams, Tables, and
Time in KSQL
Matthias J. Sax | Software Engineer | @MatthiasJSax

2. 2
KSQL – Streaming SQL Engine for Apache Kafka
• Enable Stream Processing for Non-Engineers
• Everybody knows SQL
• Look Ma, no code!
• Declarative stream processing language
• Fast prototyping
• Streaming SQL engine for Apache Kafka
• Streaming ETL
• Ad-hoc topic inspection

3. 3
Apache Kafka and KSQL
@MatthiasJSax
Kafka Cluster
KSQL Cluster
KSQL CLI
Deploy Queries
Read Data
Write Data

4. 4
Topics
@MatthiasJSax

5. 5
Streams from Topics
@MatthiasJSax
CREATE STREAM clickstream (
time BIGINT,
url VARCHAR,
status INTEGER,
bytes INTEGER,
user_id VARCHAR,
agent VARCHAR)
WITH (
value_format = 'JSON',
kafka_topic = 'my_clickstream_topic'
);

6. 6
Querying a Stream
@MatthiasJSax
CREATE STREAM user_clicks AS
SELECT *
FROM clickstream
WHERE user_id = 'mjsax';

7. 7
Joining Streams
@MatthiasJSax
CREATE STREAM addclicks AS
SELECT *
FROM clickstream c JOIN addstream a
WITHIN 1 MINUTES
ON c.url = a.url;

8. 8
Joining Streams
@MatthiasJSax
window size
stream1
stream2
result stream
Event-time based!

9. 9
Know your Time
@MatthiasJSax
• Kafka topics:
• Messages are key-value pairs plus a timestamp
• KSQL streams:
• Records have a value-schema plus a ROWTIME attribute
• ROWTIME == timestamp (default)
• ROWTIME == schema-attribute (user specified)
• Join window is defined over ROWTIME
• Independent of processing time (ie, when the query is executed)
• Deterministic result if old data is (re-)processed

10. 10@MatthiasJSax

11. 11
Tables and Topics
@MatthiasJSax
CREATE TABLE users (
user_id INTEGER,
registered_at LONG,
username VARCHAR,
name VARCHAR,
city VARCHAR,
level VARCHAR)
WITH (
key = 'user_id',
kafka_topic = 'clickstream_users',
value_format = 'AVRO');

12. 12
Tables and Topics
@MatthiasJSax

13. 13
Querying Tables
@MatthiasJSax
CREATE TABLE berlin_user AS
SELECT *
FROM users
WHERE city = 'Berlin';

14. Do you think that’s a table you are querying ?

15. 15
Querying Tables
@MatthiasJSax
Query

16. 16
Joining Tables
@MatthiasJSax
CREATE TABLE user_profile AS
SELECT *
FROM users u JOIN user_details d
ON c.user_id = d.user_id;
primary key join

17. Do you think that’s a table you are querying ?

18. 18
Temporal Table Semantics
@MatthiasJSax
0 1 2 3 4 5 6
<A,1> <B,2> <A,3>
A 1 A 1 A 3
B 2 B 2
v1 v4 v6
A 1 A 1 A 1
B 2
v2 v3 v5

19. 19
Temporal Table-Table Join Semantics
@MatthiasJSax
v1 v5 v6
v2 v6
v2
users
details
profile v5 v6

20. 20
Stream Aggregation
@MatthiasJSax
CREATE TABLE clicks AS
SELECT user_id, COUNT(url)
FROM clickstream
WINDOW TUMBLING (size 30 seconds)
GROUP BY user_id
HAVING COUNT(url) > 20
WHERE bytes > 1024;

21. 21
Stream Aggregation
@MatthiasJSax

22. 22
Tumbling Window
@MatthiasJSax

23. 23
Hopping Window
@MatthiasJSax

24. 24
Temporal Stream-Table Join
@MatthiasJSax
CREATE STREAM vip_actions AS
SELECT c.user_id, fullname, url, status
FROM clickstream c
LEFT JOIN users u ON c.user_id = u.user_id
WHERE u.level = 'Platinum';

25. 25
Temporal Stream-Table Join
@MatthiasJSax

26. 26
Temporal Stream-Table Join
@MatthiasJSax
v1users
clickstream
vip_action
v5 v6

27. 27
Summary
@MatthiasJSax
• KSQL
• Streaming SQL engine for Apache Kafka
• Streams and Tables
• Temporal semantics
• https://github.com/confluentinc/ksql
• https://docs.confluent.io/current/ksql/docs/index.html
• https://kafka-summit.org/
• New York (April 2)
• London (May 13-14)

28. 28
28
We are hiring!