1.
@gAmUssA | #confluentvug | @confluentinc
Crossing the Streams:
Event Streaming with Kafka
Streams
June 3rd / Online

2.
@gamussa | #confluentvug | @confluentinc

3.
Preface

4.
@gamussa | #confluentvug | @confluentinc
Stream Processing
is the toolset for
dealing with
events
as they move!

5.
@gamussa | #confluentvug | @confluentinc

6.
@gamussa | #confluentvug | @confluentinc
Event Streaming
platform

7.
@gamussa | #confluentvug | @confluentinc
Java Apps
with Kafka Streams
or KSQL
Continuous
Computation
Event Streaming
platform
API based
clustering

8.
@gamussa | #confluentvug | @confluentinc
Serving Layer
(Microservices,
Elastic, etc.)
Java Apps
with Kafka Streams
or KSQL
Continuous
Computation
Event Streaming
platform
API based
clustering

9.
Apache Kafka®
Event Streaming
Platform 101

10.
@
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture

11.
@
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Kafka Brokers

12.
@
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Zookeeper NodesKafka Brokers

13.
@
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Zookeeper Nodes
Application
Native Client
library
Kafka Brokers

14.
@
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Zookeeper Nodes
Schema Registry
Application
Native Client
library
Kafka Brokers

15.
@
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Zookeeper Nodes
Schema Registry
Application
Native Client
library
Application
Kafka
Streams
Kafka Brokers

16.
@
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Zookeeper Nodes
Schema Registry
Application
Native Client
library
Application
Kafka
Streams
Kafka Brokers
KSQL
Kafka
Streams

17.
@
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Kafka Connect
Zookeeper Nodes
Schema Registry
Application
Native Client
library
Application
Kafka
Streams
Kafka Brokers
KSQL
Kafka
Streams

18.
@
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka
Streams
Kafka Brokers
KSQL
Kafka
Streams

19.
@
@gamussa | #confluentvug | @confluentinc
Event Streaming Platform Architecture
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka
Streams
Kafka Brokers
KSQL
Kafka
Streams

20.
@gamussa | #confluentvug | @confluentinc
The log is a simple idea
Messages are added at the end of the log
Old New

21.
@gamussa | #confluentvug | @confluentinc
The log is a simple idea
Messages are added at the end of the log
Old New

22.
@gamussa | #confluentvug | @confluentinc
Consumers have a position all of their own
Old New
Robin
is here
Scan
Viktor
is here
Scan
Ricardo
is here
Scan

23.
@gamussa | #confluentvug | @confluentinc
Consumers have a position all of their own
Old New
Robin
is here
Scan
Viktor
is here
Scan
Ricardo
is here
Scan

24.
@gamussa | #confluentvug | @confluentinc
Consumers have a position all of their own
Old New
Robin
is here
Scan
Viktor
is here
Scan
Ricardo
is here
Scan

25.
@gamussa | #confluentvug | @confluentinc
Only Sequential Access
Old New
Read to offset & scan

26.
@gamussa | #confluentvug | @confluentinc
Shard data to get scalability

27.
@gamussa | #confluentvug | @confluentinc
Shard data to get scalability
Cluster
of
machines

28.
@gamussa | #confluentvug | @confluentinc
Shard data to get scalability
Producer (1) Producer (2) Producer (3)
Cluster
of
machines
Partitions live on
different machines
Messages are sent
to different
partitions

29.
@gamussa | #confluentvug | @confluentinc
// in-memory store, not persistent
Map<String, Integer> groupByCounts = new HashMap<>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProperties());
KafkaProducer<String, Integer> producer = new KafkaProducer<>(producerProperties())) {
consumer.subscribe(Arrays.asList("A", "B"));
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

30.
@gamussa | #confluentvug | @confluentinc
// in-memory store, not persistent
Map<String, Integer> groupByCounts = new HashMap<>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProperties());
KafkaProducer<String, Integer> producer = new KafkaProducer<>(producerProperties())) {
consumer.subscribe(Arrays.asList("A", "B"));
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

31.
@gamussa | #confluentvug | @confluentinc
// in-memory store, not persistent
Map<String, Integer> groupByCounts = new HashMap<>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProperties());
KafkaProducer<String, Integer> producer = new KafkaProducer<>(producerProperties())) {
consumer.subscribe(Arrays.asList("A", "B"));
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

32.
@gamussa | #confluentvug | @confluentinc
// in-memory store, not persistent
Map<String, Integer> groupByCounts = new HashMap<>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProperties());
KafkaProducer<String, Integer> producer = new KafkaProducer<>(producerProperties())) {
consumer.subscribe(Arrays.asList("A", "B"));
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

33.
@gamussa | #confluentvug | @confluentinc
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1;
groupByCounts.put(key, count);
}
}

34.
@gamussa | #confluentvug | @confluentinc
while (true) { // consumer poll loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(5));
for (ConsumerRecord<String, String> record : records) {
String key = record.key();
Integer count = groupByCounts.get(key);
if (count == null) {
count = 0;
}
count += 1; // actually doing something useful
groupByCounts.put(key, count);
}
}

35.
@gamussa | #confluentvug | @confluentinc
if (counter++ % sendInterval == 0) {
for (Map.Entry<String, Integer> groupedEntry : groupByCounts.entrySet()) {
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord<>("group-by-counts", groupedEntry.getKey(), groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}
}
}

36.
@gamussa | #confluentvug | @confluentinc
if (counter++ % sendInterval == 0) {
for (Map.Entry<String, Integer> groupedEntry : groupByCounts.entrySet()) {
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord<>("group-by-counts", groupedEntry.getKey(), groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}
}
}

37.
@gamussa | #confluentvug | @confluentinc
if (counter++ % sendInterval == 0) {
for (Map.Entry<String, Integer> groupedEntry : groupByCounts.entrySet()) {
ProducerRecord<String, Integer> producerRecord =
new ProducerRecord<>("group-by-counts", groupedEntry.getKey(), groupedEntry.getValue());
producer.send(producerRecord);
}
consumer.commitSync();
}
}
}

38.
https://twitter.com/monitoring_king/status/1048264580743479296

39.
@gamussa | #confluentvug | @confluentinc
LET’S TALK ABOUT THIS FRAMEWORK OF YOURS.
I THINK ITS GOOD, EXCEPT IT SUCKS

40.
@gamussa | #confluentvug | @confluentinc
SO LET ME SHOW KAFKA STREAMS
THAT WAY IT MIGHT BE REALLY GOOD

41.
Talk is cheap!
Show me code!

42.
@gamussa | #confluentvug | @confluentinc
final StreamsBuilder streamsBuilder = new StreamsBuilder();
final KStream<String, Long> stream = streamsBuilder.stream(Arrays.asList("A", "B"));
stream.groupByKey()
.count()
.toStream()
.to("group-by-counts",
Produced.with(Serdes.String(), Serdes.Long()));
final Topology topology = streamsBuilder.build();
final KafkaStreams kafkaStreams = new KafkaStreams(topology, streamsProperties());
kafkaStreams.start();

43.
@gamussa | #confluentvug | @confluentinc
final StreamsBuilder streamsBuilder = new StreamsBuilder();
final KStream<String, Long> stream = streamsBuilder.stream(Arrays.asList("A", "B"));
// actual work
stream.groupByKey()
.count()
.toStream()
.to("group-by-counts",
Produced.with(Serdes.String(), Serdes.Long()));
final Topology topology = streamsBuilder.build();
final KafkaStreams kafkaStreams = new KafkaStreams(topology, streamsProperties());
kafkaStreams.start();

44.
@gamussa | #confluentvug | @confluentinc
final StreamsBuilder streamsBuilder = new StreamsBuilder();
final KStream<String, Long> stream = streamsBuilder.stream(Arrays.asList("A", "B"));
// actual work
stream.groupByKey()
.count()
.toStream()
.to("group-by-counts",
Produced.with(Serdes.String(), Serdes.Long()));
final Topology topology = streamsBuilder.build();
final KafkaStreams kafkaStreams = new KafkaStreams(topology, streamsProperties());
kafkaStreams.start();

45.
@gamussa | #confluentvug | @confluentinc
final StreamsBuilder streamsBuilder = new StreamsBuilder();
final KStream<String, Long> stream = streamsBuilder.stream(Arrays.asList("A", "B"));
// actual work
stream.groupByKey()
.count()
.toStream()
.to("group-by-counts",
Produced.with(Serdes.String(), Serdes.Long()));
final Topology topology = streamsBuilder.build();
final KafkaStreams kafkaStreams = new KafkaStreams(topology, streamsProperties());
kafkaStreams.start();

46.
@gamussa | #confluentvug | @confluentinc

47.
@gamussa | #confluentvug | @confluentinc

48.
@gamussa | #confluentvug | @confluentinc
Every framework Wants to be when it grows up

49.
@gamussa | #confluentvug | @confluentinc
Every framework Wants to be when it grows up
Scalable

50.
@gamussa | #confluentvug | @confluentinc
Every framework Wants to be when it grows up
Scalable Elastic

51.
@gamussa | #confluentvug | @confluentinc
Every framework Wants to be when it grows up
Scalable Elastic Fault-tolerant

52.
@gamussa | #confluentvug | @confluentinc
Every framework Wants to be when it grows up
Scalable Elastic Fault-tolerant
Stateful

53.
@gamussa | #confluentvug | @confluentinc
Every framework Wants to be when it grows up
Scalable Elastic Fault-tolerant
Stateful Distributed

54.
@gAmUssA | #confluentvug | @confluentinc
Where do I put my compute?

55.
@gAmUssA | #confluentvug | @confluentinc
Where do I put my state?

56.
@gAmUssA | #confluentvug | @confluentinc
The actual question is
Where is my code?

57.
@gamussa | #confluentvug | @confluentinc
the KAFKA STREAMS API is a
JAVA API to
BUILD REAL-TIME APPLICATIONS

58.
@gamussa | #confluentvug | @confluentinc
App
Streams
API

59.
@gamussa | #confluentvug | @confluentinc
App
Streams
API
Not running
inside brokers!

60.
@gamussa | #confluentvug | @confluentinc
Brokers?
Nope!
App
Streams
API
App
Streams
API
App
Streams
API
Same app, many instances

61.
@gamussa | #confluentvug | @confluentinc
Brokers?
Nope!
App
Streams
API
App
Streams
API
App
Streams
API
Same app, many instances

62.
@gamussa | #confluentvug | @confluentinc
Before
DashboardProcessing Cluster
Your Job
Shared Database

63.
@gamussa | #confluentvug | @confluentinc
After
Dashboard
APP
Streams
API

64.
@gamussa | #confluentvug | @confluentinc
this means you can
DEPLOYyour app ANYWHERE using
WHATEVER TECHNOLOGY YOU
WANT

65.
@gamussa | #confluentvug | @confluentinc
So many places to run you app!
...and many more...

66.
@gamussa | #confluentvug | @confluentinc
Things Kafka Stream Does
Open Source Elastic, Scalable,
Fault-tolerant
Supports Streams
and Tables
Runs Everywhere
Exactly-Once
Processing
Event-Time
Processing
Kafka Security
Integration
Powerful Processing incl.
Filters, Transforms, Joins,
Aggregations, Windowing
Enterprise Support

67.
Talk is cheap!
Show me code!

68.
@gamussa | #confluentvug | @confluentinc
Want to learn more?

69.
developer.confluent.io
Learn Kafka.
Start building with
Apache Kafka at
Confluent Developer.

70.
@gAmUssA | #confluentvug | @confluentinc
ALL UPCOMING MEETUPS
NEW EVENT EMAIL ALERTS
THE CONFLUENT MEETUP HUB
CNFL.IO/MEETUP-HUB
VIDEOS OF PAST
MEETUPS
SLIDES FROM THE
TALKS

71.
@gamussa | #confluentvug | @confluentinc
Confluent Community
Slack
A vibrant community of over 16,000
members
Come along and discuss Apache Kafka and
Confluent Platform on dedicated channels
including #ksqlDB, #connect, #clients, and
more
http://cnfl.io/slack

72.
@gamussa | #confluentvug | @confluentinc
Free eBooks Designing Event-Driven Systems
Ben Stopford
Kafka: The Definitive Guide
Neha Narkhede, Gwen Shapira, Todd Palino
Making Sense of Stream Processing
Martin Kleppmann
I ❤ Logs
Jay Kreps
http://cnfl.io/book-bundle