Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

1C O N F I D E N T I A L
Stream Processing with Confluent
Kafka Streams and KSQL
Kai Waehner
Technology Evangelist
kontakt...

2C O N F I D E N T I A L

3C O N F I D E N T I A L

4C O N F I D E N T I A L
Ubiquitous connectivity
Globally scalable platform for all
event producers and consumers
Immediat...

5C O N F I D E N T I A L

6C O N F I D E N T I A L
The beginning of a new Era
https://engineering.linkedin.com/distributed-systems/log-what-every-so...

7C O N F I D E N T I A L
● Global-scale
● Real-time
● Persistent Storage
● Stream Processing
Apache Kafka: The De-facto St...

8C O N F I D E N T I A L
Apache Kafka at Scale at Tech Giants
> 4.5 trillion messages / day > 6 Petabytes / day
“You name ...

9C O N F I D E N T I A L
Confluents Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make mo...

10C O N F I D E N T I A L
A Modern, Distributed Platform for
Data Streams.
Messaging + Storage + Processing!

11C O N F I D E N T I A L
Stream Processing
processing-time
event-time
windowing
alice
bob
dave

12C O N F I D E N T I A L
Confluent Delivers a Mission-Critical Event Streaming Platform
Apache Kafka®
Core | Connect API ...

13C O N F I D E N T I A L
What We Cover Today:
streams
The streaming SQL engine
for Apache Kafka® to write
real-time appli...

14C O N F I D E N T I A L
Lower the bar to enter the world of streaming
User Population
CodingSophistication
Core develope...

15C O N F I D E N T I A L
CREATE STREAM fraudulent_payments AS
SELECT * FROM payments
WHERE fraudProbability > 0.8
Lower t...

16C O N F I D E N T I A L
Confluent
● Kafka Streams and
KSQL for stream
processing
● Lower-level Kafka
Producer and Kafka
...

17C O N F I D E N T I A L
Microservices
Example use cases
Data enrichment Streaming ETL
Filter, cleanse, mask Real-time mo...

18C O N F I D E N T I A L
Similarities and Differences
of KSQL and Kafka Streams

19C O N F I D E N T I A L
Similarities
Enterprise
support
All you need is
Kafka
Run everywhere Elastic, scalable,
fault-to...

20C O N F I D E N T I A L
Similarities
KSQL
(processing)
Kafka
(data)
JVM application
with Kafka Streams (processing)
Does...

21C O N F I D E N T I A L
Differences
Consumer,
Producer
KSQL
Kafka Streams
Flexibilit
y
Ease of Use
CREATE STREAM ...
CRE...

22C O N F I D E N T I A L
Differences
You write…
UI included for human interaction
CLI included for human interaction
Data...

23C O N F I D E N T I A L
Guidance
streams
KSQL
• New to streaming and Kafka
• Prefer SQL to writing code in Java, Scala
•...

24C O N F I D E N T I A L
KSQL and Kafka Streams
A closer look

25C O N F I D E N T I A L
Next: KSQL in more detail
streams
The streaming SQL engine
for Apache Kafka® to write
real-time ...

26C O N F I D E N T I A L
KSQL
● You write only SQL.
No Java, Python, or
other boilerplate to
wrap around it!
● Create KSQ...

27C O N F I D E N T I A L
New user experience: interactive stream processing

28C O N F I D E N T I A L
KSQL can be used interactively + programmatically
ksql>
1 UI
POST /query
2 CLI 3 REST 4 Headless

29C O N F I D E N T I A L
KSQL REST API
POST /query HTTP/1.1
{
"ksql": "SELECT * FROM users WHERE name LIKE 'a%';"
"stream...

30C O N F I D E N T I A L
All you need is Kafka & KSQL
1.Build & package
2. Submit job
ksql> SELECT * FROM myStream
Withou...

31C O N F I D E N T I A L
KSQL is a stream processing technology
As such it is not yet a great fit for:
Ad-hoc queries
● N...

32C O N F I D E N T I A L
Data exploration
KSQL example use cases
Data enrichment Streaming ETL
Filter, cleanse, mask Real...

33C O N F I D E N T I A L
Example: CDC from DB via Kafka to Elastic
KSQL processes table changes in
real-time to continuou...

34C O N F I D E N T I A L
Example: Retail
KSQL joins the two
streams in real-time
Stream of shipments
that arrive
Stream o...

35C O N F I D E N T I A L
Example: IoT, Automotive, Connected Cars
KSQL joins the stream
and table in real-time, and
spots...

36C O N F I D E N T I A L
KSQL for Streaming ETL
● Joining, filtering,
and aggregating
streams of event
data
CREATE STREAM...

37C O N F I D E N T I A L
KSQL for Data Transformation
● Easily make
derivations of
existing topics
● Change data format
●...

38C O N F I D E N T I A L
KSQL for Real-Time Monitoring
● Filtering, tracking,
and alerting
● Log data monitoring
● Syslog...

39C O N F I D E N T I A L
KSQL for Anomaly Detection
● Identify patterns or
anomalies in real-
time data, surfaced
in mill...

40C O N F I D E N T I A L
Streams and Tables
Important because most use cases need both

41C O N F I D E N T I A L
§
Do you think that’s a table
you are querying?

42C O N F I D E N T I A L
The Stream-Table Duality
aggregation
changelog “materialized view”
of the stream
(like SUM, COUN...

43C O N F I D E N T I A L
The Stream-Table Duality
CREATE TABLE num_visited_locations_per_user AS
SELECT username, COUNT(*...

45C O N F I D E N T I A L
Scalability, Elasticity,
Fault-Tolerance

46C O N F I D E N T I A L
Fault-tolerance, powered by Kafka
Server A:
“I do stateful stream
processing, like tables,
joins...

47C O N F I D E N T I A L
Fault-tolerance, powered by Kafka
Processing fails over automatically, without data loss or misc...

48C O N F I D E N T I A L
Elasticity and Scalability, powered by Kafka
You can add, remove, restart servers during live op...

49C O N F I D E N T I A L
Deploying KSQL

50C O N F I D E N T I A L
Deploying KSQL
KSQL Server
(JVM process)
…and many more…
DEB, RPM, ZIP, TAR downloads
http://con...

51C O N F I D E N T I A L
Deploying KSQL
#1 Interactive KSQL, for development & testing
ksql>
POST /query
...
KSQL
(proces...

52C O N F I D E N T I A L
Deploying KSQL
#2 Headless KSQL, for production
servers started
with same
.sql file ...
interact...

53C O N F I D E N T I A L
Deploying KSQL
read,
write
…
BookingsTeam
…
FraudTeam
…
MobileTeam
KSQL
(processing)
Kafka
(data...

54C O N F I D E N T I A L
Monitoring KSQL
https://www.confluent.io/blog/troubleshooting-ksql-part-2
Confluent Control Cent...

56C O N F I D E N T I A L
Next: Kafka Streams in more detail
streams
The streaming SQL engine
for Apache Kafka® to write
r...

57C O N F I D E N T I A L
Kafka Streams
● You write standard
Java or Scala
applications to
process your data
● The Kafka S...

58C O N F I D E N T I A L
All you need is Kafka & Kafka Streams
Without Kafka Streams With Kafka Streams
JVM application
1...

59C O N F I D E N T I A L
DB (key-value store) included, and queryable
Location-tracking
Application:
“I continuously trac...

60C O N F I D E N T I A L
Microservices
Kafka Streams example use cases
Data enrichment Streaming ETL
Filter, cleanse, mas...

61C O N F I D E N T I A L
Writing Kafka Streams applications
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>...

62C O N F I D E N T I A L
Writing Kafka Streams applications
DSL Processor API
And of course, you can combine the DSL and ...

63C O N F I D E N T I A L
Deploying Kafka Streams applications
JVM application
with Kafka Streams (processing)
Develop you...

64C O N F I D E N T I A L
Elasticity and Scalability, powered by Kafka
You can add, remove, restart instances of your appl...

65C O N F I D E N T I A L
Deploying Kafka Streams applications
read,
write
App
(processing)
Kafka
(data)
More Apps
Booking...

66C O N F I D E N T I A L
Confluent Platform as
Central Nervous System

67C O N F I D E N T I A L
Confluent’s Streaming Maturity Model - where are you?
Value
Maturity (Investment & time)
2
Enter...

69C O N F I D E N T I A L
Kafka Connect
Kafka Cluster
CRM
Integration
Domain-Driven Design for your Event Steaming Platfor...

70C O N F I D E N T I A L
Confluent Schema Registry for Message Validation
Input Data
Schema
Registry
App 1
• “Kafka Benef...

71C O N F I D E N T I A L
Resources and Next Steps
confluentinc/kafka-streams-examples
https://docs.confluent.io/current/s...

72C O N F I D E N T I A L
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
@KaiWaehner
www.confluent.io
www.kai-wa...

73C O N F I D E N T I A L

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1038569 views
Storytelling For The Web: Integrate... by Chiara Aliotta 950715 views
Artificial Intelligence, Data and C... by OECD Directorate ... 873965 views
How to Leverage AI to Boost Employe... by SocialHRCamp 367437 views
2024 State of Marketing Report – by... by Marius Sescu 208765 views
Everything You Need To Know About C... by Expeed Software 228715 views

View on Slideshare

1 of 70 Ad

View on Slideshare

1 of 70 Ad