KSQL is an open source, Apache 2.0 licensed streaming SQL engine that enables stream processing against Apache Kafka. KSQL makes it easy to read, write, and process streaming data in real-time, at scale, using SQL-like semantics.

1. KSQL
Streaming Sql for Kafka
Neha Bhardwaj
Software Consultant

2. Agenda
1. Kafka - Introduction
2. Stream Processing
3. KSQL
4. Benefits
5. Use Cases
6. KSQL Installation: Docker/ Non-Docker
7. Demo

3. Kafka - Introduction
1. Publish subscribe messaging system.
2. Data transportation, with any necessary transformation
happening in the target datastore.

4. Stream Processing

5. What Solution is KSQL providing?
1. Simple, SQL interface over Kafka.
2. Need not park your data, just query on the fly.
3. Transformations are done continuously as new data arrives
in the Kafka topic as compared to one off transformations.
4. Powerful stream processing operations including
aggregations, joins, windowing, sessionization, and much
more.

6. KSQL
1. Open source, Apache 2.0 Licensed.
2. Enables reading, transforming, converting data formats in
real-time.
3. Provides a simple and completely interactive SQL interface
for processing data in Kafka.
4. Distributed, scalable, reliable, and real-time.
5. Currently available as a developer preview.

7. Benefits
1. Various streams and tables coming from different sources
can be joined directly.
2. Each stream or table created in KSQL will be stored in a
separate topic.
3. KSQL can work both in standalone and client-server mode.
4. Simplifies deployment- No jars, artifacts, binaries; Just SQL

8. Use Cases
● Real-time monitoring meets real-time analytics
● Security and anomaly detection
● Online data integration
● Application Development

9. Core Abstractions
Streams
CREATE STREAM pageviews (viewtime BIGINT,
userid VARCHAR, pageid VARCHAR)
WITH (kafka_topic='pageviews',
value_format=’JSON’);
Tables
CREATE TABLE users (registertime BIGINT, gender
VARCHAR, regionid VARCHAR, userid VARCHAR)
WITH (kafka_topic='users',
value_format='DELIMITED');

10. Supported Commands, Functions and
Datatypes
CREATE, SELECT, FROM, WHERE, LIKE, GROUP BY,
HAVING, DROP, AS, OR, LIMIT
TIMESTAMPTOSTRING(), EXTRACTJSONFIELD(), LCASE()
BOOLEAN, INTEGER, BIGINT, DOUBLE, VARCHAR,
ARRAY<ArrayType>and MAP<VARCHAR, ValueType>

11. Components
KSQL CLI
The KSQL CLI allows you to interactively write KSQL queries.
Its interface should be familiar to users of MySQL, Postgres,
Oracle, Hive, Presto, etc.
The KSQL CLI acts as a client to the KSQL server.
KSQL Server
The KSQL server runs the engine that executes KSQL queries,
which includes the data processing as well as reading data from
and writing data to the target Kafka cluster.

12. Installation

13. Non-Docker Setup for KSQL
1. $ git clone git@github.com:confluentinc/ksql.git
2. $ cd ksql
3. $ mvn clean compile install -DskipTests
4. $ ./bin/ksql-cli local

14. DEMO
Usethefollowing connector configuration:

16. References
https://www.confluent.io/blog/
https://www.confluent.io/online-talks/
https://www.rittmanmead.com/blog/2017/10/ksql-streaming-sql-for-apache-kafka/
https://www.confluent.io/blog/using-ksql-to-analyse-query-and-transform-data-in-kafka
https://www.confluent.io/wp-content/uploads/tweet_kafka-1024x617.png