https://www.meetup.com/de-DE/Microservices-Meetup-Munich/events/256626373/ html-version of the slides with links available at https://github.com/andreas-schroeder/demo-segmenter/releases/download/v1.0/microservices-meetup-munich-slides.zip In microservices architectures, Kafka is a popular choice for asynchronous message passing. Still, there are a couple other ways that an existing Kafka infrastructure can be used. In particular, it allows to use stream-processing libraries like Kafka Streams. This talk will cover two ways in which Kafka and Kafka Streams can be used in the scope of microservice architectures: 1) Transforming and distribute data in order to realise graceful degradation, i.e. improve overall system behaviour under failure (also known as change data capture). 2) Implementing event processors in event-driven architectures. These use cases will be illustrated with examples and will present pitfalls and successful patterns based on experiences made while building systems with Kafka Streams. Among other things, this talk will discuss how the unusual event sourcing approach followed by Kafka Streams works, and when to switch to other technologies than Kafka Streams.

1. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 1/32
BUILDING MICROSERVICES WITH KAFKA STREAMSBUILDING MICROSERVICES WITH KAFKA STREAMS
Beyond Kafka-for-Messaging
Andreas Schroeder

2. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 2/32
WHYWHY
Kafka
is a thing
can be used in interesting ways beyond messaging
Kafka Streams - not just for data analytics
both have surprises and pitfalls

3. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 3/32
ABOUT MEABOUT ME
Dr. Andreas Schroeder
@AndSchroeder
andreas-schroeder
Kafka user & experimenter since 2016
Pet topics: distributed systems, architecutre, ops
Currently @ BMW
Akka Alpakka contributor
All views presented here are my own

4. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 4/32
KAFKA CLUSTER-SIDEKAFKA CLUSTER-SIDE

5. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 5/32
PARTITIONS & OFFSETSPARTITIONS & OFFSETS
0
0 1
1
2
2 3
3 4
4 5
5 6
6
0 1 2 3 4 5 6 7
7
7
8
8
8
9
9
9 10
10
11 12
Records
offset
value
key
timestamp
p0
p1
p2

6. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 6/32
PRODUCERS & CONSUMERSPRODUCERS & CONSUMERS
Producer
Consumer
read read
append
Consumer
consumergroup
Consumer
read

7. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 7/32
DATA RETENTIONDATA RETENTION
time / size-based
7 days
key-based log compaction
superseded by update
Records
current
state of
keyed
entities

8. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 8/32
PITFALLS: KAFKA DEFAULT CONFIGSPITFALLS: KAFKA DEFAULT CONFIGS
availability + throughput > consistency + durability
not so great for event sourcing or CDC
producer acks: 1 vs. all
producer retries (Before 2.1): 0 vs. 2147483647
producer delivery timeout (Since 2.1): 2 mins vs. 1 day
topic min.insync.replicas: 1 vs. 2

9. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 9/32
EVENT SOURCINGEVENT SOURCING

10. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 10/32
CQRS EVENT SOURCING BIG PICTURECQRS EVENT SOURCING BIG PICTURE
events reducers state
service

11. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 11/32
GETTING KAFKA STREAMS EVENT SOURCINGGETTING KAFKA STREAMS EVENT SOURCING
1. Key abstractions: KStream and KTable
KStream: unbounded stream of data
KTable: continuously updated key-value store
Stream-Table duality
2. KStream with groupBy & aggregate produces KTable
aggregating events = event sourcing
3. KTable - backed by distributed & fault-tolerant store

12. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 12/32
STATE MANAGEMENTSTATE MANAGEMENT
in
State
outapp
in-memory or
RocksDB
changelog
stateful
transformation
for backup &
standby replicas

13. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 13/32
DEMODEMO
events segmenter
sess.
device 1
device 2
device 3
device 4
device 5
driver
segmenter-sessions-
changelog
session-
events
reader
DeviceWokeUp
DataReceived
AllDataReceived
device-id → event dev-id + sess-id → event
Data emitted in sessions with start / end markers
Task: generate and maintain session ids
https://github.com/andreas-schroeder/demo-segmenter

14. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 14/32
EVENT SOURCING COMPAREDEVENT SOURCING COMPARED
events
snapshots
app
app'
ephemeral derivation
persistent source of truth
events app
state
changelog
persistent
ephemeral
limited retention
system input
events - source of truth vs. audit log
state - derivation vs. persistent
fixes - reprocess vs. compensation events

15. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 15/32
RETAINING EVENTSRETAINING EVENTS
Events can also be kept indefinitely
Store events in log-compacted topic with unique ids
as keys
Use groupBy(entity key) & aggregate
Needs explicit deletion of events

16. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 16/32
NEW LANGUAGENEW LANGUAGE
http request → record
key-value put → groupByKey, aggregate
key-value lookup → stream + table join

17. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 17/32
SIDE-EFFECTING OUT OF KAFKASIDE-EFFECTING OUT OF KAFKA
Kafka Streams is only for
CPU-bound data transformation on Kafka
No HTTP calls
No I/O other than Kafka
Consequence: side-effects must go through topics
Use Kafka connect or Akka/Alpakka
for acting on the world

18. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 18/32
PITFALLSPITFALLS
Joins and aggregate are difficult to fully grasp
Read the docs on - really
Losing my religion consumer offsets
Protect yourself with timeouts
i.e. stream.filter(recent)
stateful transformations

19. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 19/32
EVENT SOURCING TAKE AWAYEVENT SOURCING TAKE AWAY
Kafka Streams
can be used to build event reducers
works only on Kafka topics - no external I/O

20. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 20/32
CHANGE DATA CAPTURECHANGE DATA CAPTURE

21. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 21/32
HIGH FANOUT SERVICESHIGH FANOUT SERVICES
query / read
endpoint
High query volume
Tight availability requirements

22. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 22/32
SYNC COUPLING AND CONSISTENCYSYNC COUPLING AND CONSISTENCY
A B C
API
sync
async
avAPI
tAPI
= a ⋅ a ⋅ avA vB vC
= + +tA tB tC
avAPI
tAPI
= avA
= tA

23. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 23/32
TRADING CONSISTENCY FOR AVAILABILITYTRADING CONSISTENCY FOR AVAILABILITY
Source
Source
Connector
Sink
Connector
Client
full stock
Write
API
push all
updates
store full
records
follow & apply
updates
retention policy:
log compaction
Failures of individual services are partial system failures only

24. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 24/32
BLUE/GREEN DATA MIGRATIONBLUE/GREEN DATA MIGRATION
Ablue
full stock
Agreen
Read stock topic from the start

25. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 25/32
DATA TRANSFORMATION PIPELINESDATA TRANSFORMATION PIPELINES
Transform
stock A
Sink
Connector
aggregatedstock B
stock C
Combining multiple stocks is a stateful transformation

26. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 26/32
CDC TECHNOLOGIESCDC TECHNOLOGIES
Source
Connector
Sink
Connector
Transform
Relational DBs
Kafka Connect
Debezium
Other sources
Kafka Connect
Alpakka / Akka Stream
Kafka Connect
Alpakka / Akka Stream
Kafka Streams
Flink

27. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 27/32
CDC PITFALLSCDC PITFALLS
Handrolling DB source connectors
Source connectors are complex
initial load and deltas
at-least-once delivery

28. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 28/32
CDC TAKE AWAYCDC TAKE AWAY
Asynchronous decoupling allows trading
consistency for resilience
CDC is not trivial to get started with
Rich ecosystem of technoligies exists
Combine multiple sources with Kafka Streams

29. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 29/32
WRAP-UPWRAP-UP

30. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 30/32
SUMMARYSUMMARY
Kafka Streams
CQRS event reducers
Change Data Capture
New Language
Connectors needed get back out of Kafka

31. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 31/32
O'Reilly Ideas
RECOMMENDED READSRECOMMENDED READS
Martin Kleppmann | Kafka Summit SFMartin Kleppmann | Kafka Summit SFMartin Kleppmann | Kafka Summit SF ……… Jay Kreps | Kafka Summit SF 2018 KeyJay Kreps | Kafka Summit SF 2018 KeyJay Kreps | Kafka Summit SF 2018 Key………
Streaming 101
Streaming 102

32. 10.1.2019 Building Microservices with Kafka Streams
http://localhost:8000/?print-pdf#/ 32/32
THANK YOUTHANK YOU