This document provides an overview of deploying and operating KSQL. It introduces Nick Dearden and Hojjat Jafarpour who work on KSQL at Confluent. The agenda includes discussing deployment, configuration, scaling, and monitoring of KSQL. Specific topics covered are getting started with KSQL, starting the KSQL server, connecting to Kafka and Schema Registry, using the KSQL CLI, deployment patterns and log files. The presentation also demonstrates viewing metrics for KSQL servers, queries, input and output topics through JMX.

1. 1
ONLINE TALK
Deploying and Operating
KSQL

2. 2
Nick is one of the masterminds and co-creators of KSQL. He is also a technology
and product leader at Confluent, where he enjoys leveraging many years of
experience in the world of data and analytic systems to help design and explain
the power of a streaming platform for every business. Prior to Confluent, he led
the data platform group for a leading online real-estate seller and was chief
architect for a cloud-based financial analytics platform. His early career stretches
all the way back through multiple data warehouse and business intelligence
adventures to the green-screen days of mainframe banking systems.
Nick Dearden
Director of Engineering, Confluent
Hojjat started the KSQL project at Confluent and is the lead engineer for
the project. Before joining Confluent he has worked at NEC Labs,
Informatica, Quantcast and Tidemark on various big data management
projects. He has a PhD in computer science from UC Irvine where he
worked on scalable stream processing and publish/subscribe systems.
Hojjat Jafarpour
KSQL Project Lead, Confluent

3. 3
Housekeeping Items
● This session will last about an hour.
● This session will be recorded.
● You can submit your questions by entering them into the GoToWebinar panel.
● The last 10-15 minutes will consist of Q&A.
● The slides and recording will be available after the talk.

4. Agenda
• Deployment
• Configuration
• Scaling
• Monitoring

5. 5
First things first
• Getting KSQL binaries is easy
Download: https://www.confluent.io/download/
Confluent Open Source 4.1+ (free)
Confluent Enterprise 4.1+ (30-day trial)
• Links to downloads, docs, news, examples, etc.
https://confluent.io/ksql
• KSQL code is open source (Apache license)
https://github.com/confluentinc/ksql

6. 6
Deploying KSQL – Getting Started
• For development, e.g. on your laptop, use the Confluent CLI:
$ confluent start
• Starts up a full set of services:
Zookeeper & Kafka Broker
Schema Registry
KSQL Server
REST Proxy
Kafka Connect
Control Center

7. 7
Deploying KSQL – Starting KSQL Server
• KSQL Server acts as a Kafka client
Run it on nodes separate from the Kafka Brokers
• Provide a configuration file of settings
• From your installation directory:
$ bin/ksql-server-start config/ksql-server.properties

8. 8
KSQL Server Configuration
• The configuration file has only a few mandatory options:
bootstrap.servers – where to find the Kafka Broker(s)
listeners – ports on which to listen for connections from the KSQL CLI
• Optional:
ksql.service.id – a name to group together a pool of KSQL Servers
• Optionally, add any property the embedded Kafka consumers and producers or Kafka Streams API would understand
e.g. security configurations
• Example:
bootstrap.servers=broker1:9092
listeners=http://localhost:8088
ksql.streams.commit.interval.ms=1000
producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor

9. 9
Connecting to a Secured Kafka cluster
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=
org.apache.kafka.common.security.plain.PlainLoginModule required
username="<name of the user KSQL should use>"
password="<the password>";
Exact settings you will need will vary depending on what SASL mechanism your
Kafka cluster is using and how your SSL certificates are signed. For full details,
please refer to the Security section of the Kafka documentation
<http://kafka.apache.org/documentation.html#security>

10. 10
Connecting to a Schema Registry (Optional)
• Add Schema Registry address in the same configuration file
• If your Schema Registry is secured, you will also need to set a KSQL_OPTS environment variable when starting KSQL
Server to specify the connection credentials

11. 11
Starting KSQL CLI
• KSQL CLI interfaces with a KSQL Server over HTTP
• Start by specifying the address of the target KSQL Server

12. 12
The Bigger Picture

13. 13
Patterns & Best Practices
• KSQL Server pools
– per team / project / use-case
• “headless” vs. “interactive”
$ bin/ksql-server-start config/my.properties - -queries-file /path/to/foo.sql

14. 14
Log Files
• See config/log4j.properties or config/log4j-rolling.properties

15. 15
Scaling
• KSQL Servers are Kafka clients
• Queries act as Consumer Groups
• Partitions are the limit of scale-out

16. 16
Example Application
Ecommerce Site HadoopEvent Stream Support

17. 17
Example Application

18. 18
Show & Explain Queries

19. 19
KSQL-Server JMX metrics
$ export JMX_PORT=1099 && bin/ksql-server-start config/nicks-ksqlserver.properties
• Attach JConsole or tool of choice

20. 20
Viewing JMX Metrics

21. 21
KSQL Server Metrics

22. 22
KSQL Metrics

23. 23
Input Topic Metrics

24. 24
Output Topic Metrics

25. 25
Transient Query Metrics

26. 26
Transient vs Persistent Queries

27. Questions
?

28. 28
Thank you for joining us!