This document discusses ksqlDB, a streaming SQL engine for Apache Kafka. It allows users to write streaming applications using familiar SQL queries against Kafka topic data. Some key points made include: - ksqlDB allows users to create, select, and join streaming data in Kafka topics using SQL queries without the need for Java or other code - It provides a simpler way to build streaming applications compared to Kafka Streams by using SQL - Examples show how ksqlDB can be used for real-time monitoring, anomaly detection, streaming ETL, and data transformations.

1. 1C O N F I D E N T I A L
Building Consciousness
on Real Time Events: ksqlDB Recipes

2. 2C O N F I D E N T I A L
Gnanaguru(Guru) Sattanathan | guru@confluent.io | @avoguru

3. 3C O N F I D E N T I A L
காஃ$காKAFKA

4. 4C O N F I D E N T I A L
A company is built on
DATA FLOWS
but
All we have is
DATASTORES

5. 5
App App App App
search
HadoopDWH
monitoring security
MQ MQ
cache
cache
A bit of a mess…

6. 6
Kafka is a Streaming Platform
KAFKA
DWH Hadoop
App
App App App App
App
App
App
request-response
messaging
OR
stream
processing
streaming data pipelines
changelogs

7. 7
Event Streaming Platform
• Storage
• Pub / Sub
• Processing

8. 8
The log is a simple idea
Messages are added at the end of the
log
Old New

9. 9
Shard data to get scalability
Messages are sent to different partitions
Producer
(1)
Producer
(2)
Producer
(3)
Cluster of
machines
Partitions live on
different machines
Messages are sent to
different partitions

10. 10
Linearly Scalable Architecture
Single topic:
- Many producers machines
- Many consumer machines
- Many Broker machines
No Bottleneck!
Producers
Consumers

11. 11C O N F I D E N T I A L

12. 12
Streaming
is the toolset
for dealing with
events
as they move!

13. 13C O N F I D E N T I A L
KSQL
The streaming SQL engine for Apache Kafka®
to write real-time applications in SQL

14. 14C O N F I D E N T I A L
KSQL
CREATE STREAM fraudulent_payments AS
SELECT * FROM payments
WHERE fraudProbability > 0.8;
streams
Lowering the
bar: KSQL vs.
Kafka
Streams
Lower the bar to enter the world of streaming
vs.

15. 15C O N F I D E N T I A L
KSQL
● You write only SQL.
No Java, Python, or
other boilerplate to
wrap around it!
● Create KSQL user
defined functions in
Java when needed.
CREATE STREAM fraudulent_payments AS
SELECT * FROM payments
WHERE fraudProbability > 0.8;

16. 16C O N F I D E N T I A L
All you need is Kafka and KSQL
1.Build & package
2. Submit job
required for
fault-tolerance
ksql> SELECT * FROM myStream
Without KSQL With KSQL
processing
storage

17. 17C O N F I D E N T I A L
Something to remember !
KSQL is a process.*
*But what was announced at #kafkasummit is slightly different

18. 18C O N F I D E N T I A L
Data exploration
KSQL example use cases
Data enrichment Streaming ETL
Filter, cleanse, mask Real-time monitoring Anomaly detection

19. 19C O N F I D E N T I A L
Example: CDC from DB via Kafka to Elastic
KSQL processes table
changes in real-time
Kafka Connect
streams data in
Kafka Connect
streams data out

20. 20C O N F I D E N T I A L
Example: Retail
KSQL joins the two
streams in real-time
Stream of shipments
that arrive
Stream of purchases from
online and physical stores

21. 21C O N F I D E N T I A L
Example: IoT, Automotive, Connected Cars
KSQL joins the two
streams in real-time
Kafka Connect
streams data in
Cars send telemetry data
via Kafka API
Kafka Streams application
to notify customers

22. 22C O N F I D E N T I A L
KSQL for Real-Time Monitoring
● Log data monitoring
● Tracking and alerting
● Syslog data
● Sensor / IoT data
● Application metrics
CREATE STREAM syslog_invalid_users AS
SELECT host, message
FROM syslog
WHERE message LIKE '%Invalid user%';
http://cnfl.io/syslogs-filtering / http://cnfl.io/syslog-alerting

23. 23C O N F I D E N T I A L
KSQL for Anomaly Detection
● Identify patterns or
anomalies in real-
time data, surfaced
in milliseconds
CREATE TABLE possible_fraud AS
SELECT card_number, COUNT(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 SECONDS)
GROUP BY card_number
HAVING COUNT(*) > 3;

24. 24C O N F I D E N T I A L
KSQL for Streaming ETL
● Joining, filtering, and
aggregating streams
of event data
CREATE STREAM vip_actions AS
SELECT user_id, page, action
FROM clickstream c
LEFT JOIN users u
ON c.user_id = u.user_id
WHERE u.level = 'Platinum';

25. 25C O N F I D E N T I A L
KSQL for Data Transformation
● Easily make
derivations of
existing topics
CREATE STREAM pageviews_avro
WITH (PARTITIONS=6,
VALUE_FORMAT='AVRO') AS
SELECT * FROM pageviews_json
PARTITION BY user_id;

26. 26C O N F I D E N T I A L
Updates from Kafka Summit San Fran 2019
Connectors to work
closely with KSQL
Lookups made simpler
Overall KSQL is a Process
& also a database
https://bit.ly/33V17X8

27. 27C O N F I D E N T I A L
ksqldb.io

28. 28C O N F I D E N T I A L
https://ksqldb.io/

29. 29C O N F I D E N T I A L
Demo
https://bit.ly/32UsGQ4
https://bit.ly/32I0qAr
https://kafka-tutorials.confluent.io

30. 30C O N F I D E N T I A L