Crossing the Streams: Event Streaming with Kafka Streams, Viktor Gamov, Developer Advocate, Confluent

1. @gAmUssA | #confluentvug | @confluentinc
Crossing the Streams:
Event Streaming with Kafka
Streams
June 3rd / Online

2. @gamussa | #confluentvug | @confluentinc

3. Preface

4. @gamussa | #confluentvug | @confluentinc
Stream Processing
is the toolset for
dealing with
events
as they move!

5. @gamussa | #confluentvug | @confluentinc

6. @gamussa | #confluentvug | @confluentinc
Event Streaming
platform

7. @gamussa | #confluentvug | @confluentinc
Java Apps
with Kafka Streams
or KSQL
Continuous
Computation
Event Streaming
platform
API based
clustering

8. @gamussa | #confluentvug | @confluentinc
Serving Layer
(Microservices,
Elastic, etc.)
Java Apps
with Kafka Streams
or KSQL
Continuous
Computation
Event Streaming
platform
API based
clustering

9. Apache Kafka®
Event Streaming
Platform 101

10. @
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture

11. @
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Kafka Brokers

12. @
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Zookeeper NodesKafka Brokers

13. @
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Zookeeper Nodes
Application
Native Client
library
Kafka Brokers

14. @
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Zookeeper Nodes
Schema Registry
Application
Native Client
library
Kafka Brokers

15. @
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Zookeeper Nodes
Schema Registry
Application
Native Client
library
Application
Kafka
Streams
Kafka Brokers

16. @
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Zookeeper Nodes
Schema Registry
Application
Native Client
library
Application
Kafka
Streams
Kafka Brokers
KSQL
Kafka
Streams

17. @
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Kafka Connect
Zookeeper Nodes
Schema Registry
Application
Native Client
library
Application
Kafka
Streams
Kafka Brokers
KSQL
Kafka
Streams

18. @
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka
Streams
Kafka Brokers
KSQL
Kafka
Streams

19. @
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka
Streams
Kafka Brokers
KSQL
Kafka
Streams

20. @gamussa | #confluentvug | @confluentinc
The log is a simple idea
Messages are added at the end of the log
Old New

21. @gamussa | #confluentvug | @confluentinc
The log is a simple idea
Messages are added at the end of the log
Old New

22. @gamussa | #confluentvug | @confluentinc
Consumers have a position all of their own
Old New
Robin
is here
Scan
Viktor
is here
Scan
Ricardo
is here
Scan

23. @gamussa | #confluentvug | @confluentinc
Consumers have a position all of their own
Old New
Robin
is here
Scan
Viktor
is here
Scan
Ricardo
is here
Scan

24. @gamussa | #confluentvug | @confluentinc
Consumers have a position all of their own
Old New
Robin
is here
Scan
Viktor
is here
Scan
Ricardo
is here
Scan

25. @gamussa | #confluentvug | @confluentinc
Only Sequential Access
Old New
Read to offset & scan

26. @gamussa | #confluentvug | @confluentinc
Shard data to get scalability

27. @gamussa | #confluentvug | @confluentinc
Shard data to get scalability
Cluster
of
machines

28. @gamussa | #confluentvug | @confluentinc
Shard data to get scalability
Producer (1) Producer (2) Producer (3)
Cluster
of
machines
Partitions live on
different machines
Messages are sent
to different
partitions

29. @gamussa | #confluentvug | @confluentinc
// in-memory store, not persistent
Map<String, Integer> groupByCounts = new HashMap<>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProperties());
KafkaProducer<String, Integer> producer = new KafkaProducer<>(producerProperties())) {
consumer.subscribe(Arrays.asList("A", "B"));
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

30. @gamussa | #confluentvug | @confluentinc
// in-memory store, not persistent
Map<String, Integer> groupByCounts = new HashMap<>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProperties());
KafkaProducer<String, Integer> producer = new KafkaProducer<>(producerProperties())) {
consumer.subscribe(Arrays.asList("A", "B"));
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

31. @gamussa | #confluentvug | @confluentinc
// in-memory store, not persistent
Map<String, Integer> groupByCounts = new HashMap<>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProperties());
KafkaProducer<String, Integer> producer = new KafkaProducer<>(producerProperties())) {
consumer.subscribe(Arrays.asList("A", "B"));
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

32. @gamussa | #confluentvug | @confluentinc
// in-memory store, not persistent
Map<String, Integer> groupByCounts = new HashMap<>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProperties());
KafkaProducer<String, Integer> producer = new KafkaProducer<>(producerProperties())) {
consumer.subscribe(Arrays.asList("A", "B"));
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

33. @gamussa | #confluentvug | @confluentinc
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

34. @gamussa | #confluentvug | @confluentinc
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1; // actually doing something useful
groupByCounts.put(key, count);
}
}

35. @gamussa | #confluentvug | @confluentinc
if (counter++ % sendInterval == 0) {
for (Map.Entry<String, Integer> groupedEntry : groupByCounts.entrySet()) {
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord<>("group-by-counts", groupedEntry.getKey(), groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}
}
}

36. @gamussa | #confluentvug | @confluentinc
if (counter++ % sendInterval == 0) {
for (Map.Entry<String, Integer> groupedEntry : groupByCounts.entrySet()) {
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord<>("group-by-counts", groupedEntry.getKey(), groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}
}
}

37. @gamussa | #confluentvug | @confluentinc
if (counter++ % sendInterval == 0) {
for (Map.Entry<String, Integer> groupedEntry : groupByCounts.entrySet()) {
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord<>("group-by-counts", groupedEntry.getKey(), groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}
}
}

38. https://twitter.com/monitoring_king/status/1048264580743479296

39. @gamussa | #confluentvug | @confluentinc
LET’S TALK ABOUT THIS FRAMEWORK OF YOURS.
I THINK ITS GOOD, EXCEPT IT SUCKS

40. @gamussa | #confluentvug | @confluentinc
SO LET ME SHOW KAFKA STREAMS
THAT WAY IT MIGHT BE REALLY GOOD

41. Talk is cheap!
Show me code!

42. @gamussa | #confluentvug | @confluentinc
final StreamsBuilder streamsBuilder = new StreamsBuilder();
final KStream<String, Long> stream = streamsBuilder.stream(Arrays.asList("A", "B"));
stream.groupByKey()
.count()
.toStream()
.to("group-by-counts",
Produced.with(Serdes.String(), Serdes.Long()));
final Topology topology = streamsBuilder.build();
final KafkaStreams kafkaStreams = new KafkaStreams(topology, streamsProperties());
kafkaStreams.start();

43. @gamussa | #confluentvug | @confluentinc
final StreamsBuilder streamsBuilder = new StreamsBuilder();
final KStream<String, Long> stream = streamsBuilder.stream(Arrays.asList("A", "B"));
// actual work
stream.groupByKey()
.count()
.toStream()
.to("group-by-counts",
Produced.with(Serdes.String(), Serdes.Long()));
final Topology topology = streamsBuilder.build();
final KafkaStreams kafkaStreams = new KafkaStreams(topology, streamsProperties());
kafkaStreams.start();

44. @gamussa | #confluentvug | @confluentinc
final StreamsBuilder streamsBuilder = new StreamsBuilder();
final KStream<String, Long> stream = streamsBuilder.stream(Arrays.asList("A", "B"));
// actual work
stream.groupByKey()
.count()
.toStream()
.to("group-by-counts",
Produced.with(Serdes.String(), Serdes.Long()));
final Topology topology = streamsBuilder.build();
final KafkaStreams kafkaStreams = new KafkaStreams(topology, streamsProperties());
kafkaStreams.start();

45. @gamussa | #confluentvug | @confluentinc
final StreamsBuilder streamsBuilder = new StreamsBuilder();
final KStream<String, Long> stream = streamsBuilder.stream(Arrays.asList("A", "B"));
// actual work
stream.groupByKey()
.count()
.toStream()
.to("group-by-counts",
Produced.with(Serdes.String(), Serdes.Long()));
final Topology topology = streamsBuilder.build();
final KafkaStreams kafkaStreams = new KafkaStreams(topology, streamsProperties());
kafkaStreams.start();

46. @gamussa | #confluentvug | @confluentinc

47. @gamussa | #confluentvug | @confluentinc

48. @gamussa | #confluentvug | @confluentinc
Every framework Wants to be when it grows up

49. @gamussa | #confluentvug | @confluentinc
Every framework Wants to be when it grows up
Scalable

50. @gamussa | #confluentvug | @confluentinc
Every framework Wants to be when it grows up
Scalable Elastic

51. @gamussa | #confluentvug | @confluentinc
Every framework Wants to be when it grows up
Scalable Elastic Fault-tolerant

52. @gamussa | #confluentvug | @confluentinc
Every framework Wants to be when it grows up
Scalable Elastic Fault-tolerant
Stateful

53. @gamussa | #confluentvug | @confluentinc
Every framework Wants to be when it grows up
Scalable Elastic Fault-tolerant
Stateful Distributed

54. @gAmUssA | #confluentvug | @confluentinc
Where do I put my compute?

55. @gAmUssA | #confluentvug | @confluentinc
Where do I put my state?

56. @gAmUssA | #confluentvug | @confluentinc
The actual question is
Where is my code?

58. @gamussa | #confluentvug | @confluentinc
the KAFKA STREAMS API is a
JAVA API to
BUILD REAL-TIME APPLICATIONS

59. @gamussa | #confluentvug | @confluentinc
App
Streams
API

60. @gamussa | #confluentvug | @confluentinc
App
Streams
API
Not running
inside brokers!

61. @gamussa | #confluentvug | @confluentinc
Brokers?
Nope!
App
Streams
API
App
Streams
API
App
Streams
API
Same app, many instances

62. @gamussa | #confluentvug | @confluentinc
Brokers?
Nope!
App
Streams
API
App
Streams
API
App
Streams
API
Same app, many instances

63. @gamussa | #confluentvug | @confluentinc
Before
DashboardProcessing Cluster
Your Job
Shared Database

64. @gamussa | #confluentvug | @confluentinc
After
Dashboard
APP
Streams
API

65. @gamussa | #confluentvug | @confluentinc
this means you can
DEPLOYyour app ANYWHERE using
WHATEVER TECHNOLOGY YOU
WANT

66. @gamussa | #confluentvug | @confluentinc
So many places to run you app!
...and many more...

67. @gamussa | #confluentvug | @confluentinc
Things Kafka Stream Does
Open Source Elastic, Scalable,
Fault-tolerant
Supports Streams
and Tables
Runs Everywhere
Exactly-Once
Processing
Event-Time
Processing
Kafka Security
Integration
Powerful Processing incl.
Filters, Transforms, Joins,
Aggregations, Windowing
Enterprise Support

68. Talk is cheap!
Show me code!

69. @gamussa | #confluentvug | @confluentinc
Want to learn more?

70. developer.confluent.io
Learn Kafka.
Start building with
Apache Kafka at
Confluent Developer.

71. @gAmUssA | #confluentvug | @confluentinc
ALL UPCOMING MEETUPS
NEW EVENT EMAIL ALERTS
THE CONFLUENT MEETUP HUB
CNFL.IO/MEETUP-HUB
VIDEOS OF PAST
MEETUPS
SLIDES FROM THE
TALKS

72. @gamussa | #confluentvug | @confluentinc
Confluent Community
Slack
A vibrant community of over 16,000
members
Come along and discuss Apache Kafka and
Confluent Platform on dedicated channels
including #ksqlDB, #connect, #clients, and
more
http://cnfl.io/slack

73. @gamussa | #confluentvug | @confluentinc
Free eBooks Designing Event-Driven Systems
Ben Stopford
Kafka: The Definitive Guide
Neha Narkhede, Gwen Shapira, Todd Palino
Making Sense of Stream Processing
Martin Kleppmann
I ❤ Logs
Jay Kreps
http://cnfl.io/book-bundle