The document discusses stream processing with Apache Kafka and KSQL. It begins with an overview of Kafka Streams and KSQL, describing them as Java APIs and a SQL-like language for building real-time applications. It then highlights several key features of Kafka Streams, including its ability to elastically scale applications across instances and deploy them anywhere, as well as its support for exactly-once processing, event-time processing, and powerful stream operations. The document concludes by discussing how KSQL lowers the barrier to entry for stream processing and enables different users like developers and analysts to interact with streaming data.

1. Crossing the Streams:
Rethinking Stream
Processing
with
Kafka Streams and KSQL
DEVNEXUS 2019 | @tlberglund | @gamussa

2. @
@gamussa | @tlberglund | #DEVnexus

3. https://cnfl.io/streams-movie-demo

4. Raffle, yeah 🚀
Follow @gamussa @tlberglund
📸 🖼 👬
Tag @gamussa @tlberglund
With #devnexus

5. Preface

6. @gamussa | @tlberglund | #DEVnexus
Streaming
is the toolset for
dealing with
events
as they move!

7. @
@gamussa | @tlberglund | #DEVnexus
Serving Layer
(Microservices,
Elastic, etc.)
Java Apps with
Kafka Streams or
KSQL
Continuous
Computation
High Throughput
Streaming platform
API based
clustering

8. Apache Kafka
Event Streaming
Platform 101

9. @@gamussa | @tlberglund | #DEVnexus
Event Streaming Platform
Architecture
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka Streams
Kafka Brokers
KSQL
Kafka Streams

10. @gamussa | @tlberglund | #DEVnexus
The log is a simple idea
Messages are added at the end of the log
Old New

11. @gamussa | @tlberglund | #DEVnexus
The log is a simple idea
Messages are added at the end of the log
Old New

12. @gamussa | @tlberglund | #DEVnexus
Consumers have a position all of their own
Old New
Robin
is here
Scan
Viktor
is here
Scan
Ricardo
is here
Scan

13. @gamussa | @tlberglund | #DEVnexus
Consumers have a position all of their own
Old New
Robin
is here
Scan
Viktor
is here
Scan
Ricardo
is here
Scan

14. @gamussa | @tlberglund | #DEVnexus
Consumers have a position all of their own
Old New
Robin
is here
Scan
Viktor
is here
Scan
Ricardo
is here
Scan

15. @gamussa | @tlberglund | #DEVnexus
Only Sequential Access
Old New
Read to offset & scan

16. @gamussa | @tlberglund | #DEVnexus
Shard data to get scalability
Producer (1) Producer (2) Producer (3)
Cluster of
machines
Partitions live on
different machines
Messages are sent to
different partitions

17. CONSUMER GROUP
COORDINATORCONSUMERS
CONSUMER GROUP

18. @gamussa | @tlberglund | #DEVnexus
Linearly Scalable Architecture
Single topic:
- Many producers machines
- Many consumer machines
- Many Broker machines
No Bottleneck!!
Producers
Consumers

19. @
@gamussa | @tlberglund | #DEVnexus
// in-memory store, not persistent
Map<String, Integer> groupByCounts = new HashMap <>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer <>(consumerProperties());
KafkaProducer<String, Integer> producer = new KafkaProducer <>(producerProperties())) {
consumer.subscribe(Arrays.asList("A", "B"));
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

20. @
@gamussa | @tlberglund | #DEVnexus
// in-memory store, not persistent
Map<String, Integer> groupByCounts = new HashMap <>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer <>(consumerProperties());
KafkaProducer<String, Integer> producer = new KafkaProducer <>(producerProperties())) {
consumer.subscribe(Arrays.asList("A", "B"));
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

21. @
@gamussa | @tlberglund | #DEVnexus
// in-memory store, not persistent
Map<String, Integer> groupByCounts = new HashMap <>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer <>(consumerProperties());
KafkaProducer<String, Integer> producer = new KafkaProducer <>(producerProperties())) {
consumer.subscribe(Arrays.asList("A", "B"));
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

22. @
@gamussa | @tlberglund | #DEVnexus
// in-memory store, not persistent
Map<String, Integer> groupByCounts = new HashMap <>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer <>(consumerProperties());
KafkaProducer<String, Integer> producer = new KafkaProducer <>(producerProperties())) {
consumer.subscribe(Arrays.asList("A", "B"));
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

23. @
@gamussa | @tlberglund | #DEVnexus
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

24. @
@gamussa | @tlberglund | #DEVnexus
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1; // actually doing something useful
groupByCounts.put(key, count);
}
}

25. @
@gamussa | @tlberglund | #DEVnexus
if (counter ++ % sendInterval == 0) {
for (Map.Entry<String, Integer> groupedEntry : groupByCounts.entrySet()) {
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord <>("group-by-counts", groupedEntry.getKey(), groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}
}
}

26. @
@gamussa | @tlberglund | #DEVnexus
if (counter ++ % sendInterval == 0) {
for (Map.Entry<String, Integer> groupedEntry : groupByCounts.entrySet()) {
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord <>("group-by-counts", groupedEntry.getKey(), groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}
}
}

27. @
@gamussa | @tlberglund | #DEVnexus
if (counter ++ % sendInterval == 0) {
for (Map.Entry<String, Integer> groupedEntry : groupByCounts.entrySet()) {
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord <>("group-by-counts", groupedEntry.getKey(), groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}
}
}

28. https://twitter.com/monitoring_king/status/1048264580743479296

29. @
@gamussa | @tlberglund | #DEVnexus
LET’S TALK ABOUT THIS FRAMEWORK
OF YOURS.
I THINK ITS GOOD, EXCEPT IT SUCKS

30. @
@gamussa | @tlberglund | #DEVnexus
SO LET ME SHOW KAFKA STREAMS
THAT WAY IT MIGHT BE REALLY GOOD

31. Talk is cheap!
Show me code!

32. @gamussa | @tlberglund | #DEVnexus
Every framework Wants to be when
it grows up
Scalable Elastic Fault-tolerant
Stateful Distributed

33. @
@gamussa | @tlberglund | #DEVnexus

34. @

35. @gamussa | @tlberglund | #DEVnexus
the KAFKA STREAMS API is a
JAVA API to
BUILD REAL-TIME APPLICATIONS

36. @gamussa | @tlberglund | #DEVnexus
App
Streams
API
Not running
inside brokers!

37. @gamussa | @tlberglund | #DEVnexus
Brokers?
Nope!
App
Streams
API
App
Streams
API
App
Streams
API
Same app, many instances

38. @gamussa | @tlberglund | #DEVnexus
Before
DashboardProcessing Cluster
Your Job
Shared Database

39. @gamussa | @tlberglund | #DEVnexus
After
Dashboard
APP
Streams
API

40. @gamussa | @tlberglund | #DEVnexus
this means you can
DEPLOY your app ANYWHERE using
WHATEVER TECHNOLOGY YOU
WANT

41. @gamussa | @tlberglund | #DEVnexus
So many places to run you app!
...and many more...

42. @gamussa | @tlberglund | #DEVnexus
Things Kafka Stream Does
Open Source Elastic, Scalable,
Fault-tolerant
Supports Streams
and Tables
Runs Everywhere
Exactly-Once
Processing
Event-Time
Processing
Kafka Security
Integration
Powerful Processing incl.
Filters, Transforms, Joins,
Aggregations, Windowing
Enterprise Support

43. Talk is cheap!
Show me code!

44. Talk is cheap!
Show me code!

46. Do you think that’s a table
you are querying ?

47. @gamussa | @tlberglund | #DEVnexus
The Stream-Table Duality
Alice + €50
time
Alice €50
Bob + €18
Alice €50 Alice €50
Bob €18
Alice €50
Bob €18
Alice €75
Bob €18
Alice €75
Bob €18
Alice €15
Bob €18
Alice + €25 Alice – €60Stream
(payments)
Table
(balance)

48. Talk is cheap!
Show me code!

49. What’s next?

50. https://twitter.com/IDispose/status/1048602857191170054

51. @gamussa | @tlberglund | #DEVnexus
Lower the bar to enter the world of
streaming
User Population
CodingSophistication
Core developers who use Java/Scala
Core developers who don’t use Java/Scala
Data engineers, architects, DevOps/SRE
BI analysts
streams

52. @gamussa | @tlberglund | #DEVnexus
KSQL #FTW
4 Headless1 UI 2 CLI
ksql>
3 REST
POST /query

53. @gamussa | @tlberglund | #DEVnexus
Interaction with Kafka
Kafka
(data)
KSQL
(processing)
JVM application
with Kafka Streams (processing)
Does not run on
Kafka brokers
Does not run on
Kafka brokers

54. @gamussa | @tlberglund | #DEVnexus
Standing on the shoulders of Streaming
Giants
Producer,
Consumer APIs
Kafka Streams
KSQL
Ease of use
Flexibility
KSQL
Powered by
Powered by

55. One last thing…

56. @
@gamussa | @tlberglund | #DEVnexus
https://kafka-summit.org
Gamov30

57. @
@gamussa | @tlberglund | #DEVnexus
Thanks!
@gamussa viktor@confluent.io
@tlberglund
tim@confluent.io
We are hiring!
https://www.confluent.io/careers/