Join us as we build a complete streaming application with KSQL. There will be plenty of hands-on action, plus a description of our thought process and design choices along the way. Look out for advice on best practices and handy tips and tricks as we go. This is part 2 out of 3 in the Empowering Streams through KSQL series. The code from the talk is available here: https://gist.github.com/rmoff/7efa882dfd808dbab4eb7b8e6f9eda16.

1. KSQLApplication
Live Coding a
Robin Moffatt
Partner Technology Evangelist, EMEA
@rmoff

2. 2
Robin, who is also an Oracle ACE Director, earlier
this month wrote an informative blog post titled
“KSQL in Action: Real-Time Streaming ETL from
Oracle Transactional Data.” His career has always
involved data, from the old worlds of COBOL and
DB2, through the worlds of Oracle and Hadoop and
into the current world with Kafka and KSQL. His
particular interests are analytics, systems
architecture, performance testing and optimization.
Robin Moffatt
Partner Technology Evangelist, EMEA, Confluent

3. KSQLis the
Streaming
SQL Engine
for
Apache Kafka

4. KSQL Concepts
• Streams are first-class citizens
• Tables are first-class citizens
• All queries run until terminated
• Some queries are persistent

5. 5
What is it for ?
● Streaming ETL
○ Kafka is popular for data pipelines.
○ KSQL enables easy transformations of data within the pipe
CREATE STREAM vip_actions AS
SELECT userid, page, action FROM
clickstream c LEFT JOIN users u ON c.userid
= u.user_id
WHERE u.level = 'Platinum';

6. 6
What is it for ?
● Anomaly Detection
○ Identifying patterns or anomalies in real-time data, surfaced in milliseconds
CREATE TABLE possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 SECONDS)
GROUP BY card_number
HAVING count(*) > 3;

7. 7
What is it for ?
● Real Time Monitoring
○ Log data monitoring, tracking and alerting
○ Sensor / IoT data
CREATE TABLE error_counts AS
SELECT error_code, count(*)
FROM monitoring_stream
WINDOW TUMBLING (SIZE 1 MINUTE)
WHERE type = 'ERROR'
GROUP BY error_code;

8. 8
Batch = Slow
Ecommerce Site HadoopEvent Stream Support

9. 9
Kafka & KSQL = Realtime
Ecommerce Site Event Stream Support

10. 1
0
Standing On the Shoulders of (Streaming) Giants
• Native, 100%-compatible Kafka integration
• Secure stream processing using Kafka’s security features
• Elastic and highly scalable
• Fault-tolerant
• Stateful and stateless computations
• Interactive queries
• Time model
• Supports late-arriving and out-of-order data
• Windowing
• Millisecond processing latency, no micro-batching
• At-least-once and exactly-once processing guarantees

11. 1
1
Stream Processing with Kafka and KSQL
Ratings
Stream
User
data
ratings
users BAD_RATINGS
RATINGS_WITH_USER_DETAILS
RATINGS_PER_MINUTE

12. Demo!
Code available online at https://gist.github.com/rmoff/7efa882dfd808dbab4eb7b8e6f9eda16

13. Questions?
Check out the #ksql channel on:
https://slackpass.io/confluentcommunity