You have been building your applications with stateless microservices. You might even be a rockstar using Kafka for inter service communication. Everything works wonderfully but you feel you could do something more. You want your microservices to have a state. Developing stateful microservices can be hard. I will share my experience with building stateful applications with Kafka and Spring Cloud Stream libraries. Kafka Streams State Stores and Interactive Queries are the main building blocks. They are used by stream processing applications to store and query data. They can scale and be fault tolerant together with your application instances in your container platform. But there are some limitations and we need to know how to monitor their performance. This session is targeted for developers who are interested in learning event streaming practices. Demo application code will be available to participants.

1. Stateful Microservices with
Apache Kafka and Spring Cloud
Stream
Jan Svoboda, Solution Engineer
jsvoboda@confluent.io

2. Agenda
2
Basics Recap
Motivation for Event Driven Microservices
Event Types
EDM Components
Demo Application
Use Cases
Application Architecture
Stateful Microservices Concepts
Event Sourcing using State Store
Microservices decoupling (CQRS)
State Store Scaling
Smart Load Balancing

3. Jan Svoboda
https://www.linkedin.com/in/svobodajan/
• Confluent, Solutions Engineer 2020 - now
• Event Streaming, Event-Driven Microservices
• Apache Kafka, Kafka Streams
• Pivotal, Platform Architect, 2016 - 2020
• PaaS, Microservices
• Cloud Foundry, Spring, Rabbit MQ
• IBM, Solutions Architect, 2011 - 2016
• DevOps, Agile
• Bluemix, Rational

4. Basics Recap
Motivation for Kafka and Microservices

5. What is your Microservices Journey?
5
James Watters (Pivotal SVP), Kafka Summit 2019 Keynote, https://youtu.be/9I3CDfHKfNY?t=582
● Stage 1 -> Strangling the Monolith
○ DB based integration
● Stage 2-> Provide Independent Data
Sources
○ REST based integration
● Stage 3 -> Build Event Driven
Microservices
○ Kafka based integration

6. What is in the Event Message?
6
Martin Fowler (Thoughtworks), 7th Feb 2017, https://martinfowler.com/articles/201701-event-driven.html
● Event Notification
○ simple notification of change in its domain
○ no response required
● Event Carried State Transfer
○ contains details of the changed data
○ repeated data
● Event Sourcing
○ contains change to the state of a system
○ rebuild state by reprocessing changes
● Command Query Responsibility Segregation
○ separate structures for reading and writing
○ combined with previous message patterns
Neil Avery,, 31st Jan 2019,
https://www.confluent.io/blog/journey-to-event-driven-part-1-why-e
vent-first-thinking-changes-everything/

7. Event Driven Microservices Components
7
Apache Kafka
Broker for EDM events
Kafka Streams
Stream Processing API
RocksDB
Persistence layer for Kafka
Streams
Spring Cloud Stream
Stream abstraction layer
Kafka Streams Binder
SCS binding for Kafka
topics

8. Demo Application
Use Cases and Architecture

9. Demo Application - Use Cases
9
Simple Bookmarks Manager to demonstrate
Event Driven Microservices principles.
As a user I can:
● save a bookmark
● read list of my saved bookmarks
● change bookmark URL
● delete bookmark
Code available at
https://github.com/griga23/bookmarksDemo

10. Demo Application - Architecture and Dataflow
• Spring Boot (2 jars)
• Apache Kafka (2 topics)
• Spring Cloud Streams + Kafka
Streams Binder (RocksDB as a
state store)
Jan:Google,http://google.com
Jan:Google,http://google.com
JanGoogle: Google, http://google.com
JanGoogle,http://google.com

11. Stateful Microservices Concepts
Getting hands dirty...

12. 1. Event Sourcing using State Store
My Data Liberation
● From Data sourcing to Event sourcing
● From synchronous Request/Reply to asynchronous Publish/Subscribe
● Store key/values in Kafka Streams State Store
● Kafka Streams Interactive Queries for data lookup
https://docs.confluent.io/platform/current/streams/developer-guide/interactive-queries.html
Spring Cloud Stream components
➔ StreamBridge class
➔ Java Lambda Functions for aggregating data to the
Materialized State Store
➔ InteractiveQueryService class

13. 2. Microservices Decoupling (CQRS)
Command Query Responsibility
Segregation
● Pattern separates read and
update operations for a data
store.
● Implementing CQRS can
maximize performance,
scalability, and security.
● Allows a system to better
evolve over time and prevents
update commands from
causing merge conflicts at the
domain level.
https://docs.microsoft.com/en-us/azure/
architecture/patterns/cqrs
Kafka Streams State Store
➔ Change-log compacted topic
➔ RocksDB state.dir, applicationId, checkpoint
➔ Standby Replica

14. 3. Scaling – Stateful Microservices
• Multiple State Stores
with different
partitions
• HTTP REST calls to
provide State Stores
inter communication
Jan:Google,http://google.com
Jan:Yahoo,http://yahoo.com
JanGoogle:Google,http://google.com
JanYahoo:Yahoo,http://yahoo.com
JanGoogle:Google,http://google.com
JanYahoo:Yahoo,http://yahooo.com
https://docs.confluent.io/platform/current/streams/developer-guide/interactive-queries.html#querying-remote-state-stores-for-the-entire-app
Kafka Streams State Store
➔ configuration application.server
➔ methods getAllHostsInfo, KeyQueryMetadata, getHostInfo

15. 4. Smart Load Balancing – Routing to the Right
State Store
https://medium.com/bakdata/queryable-kafka-topics-with-kafka-streams-8d2cca9de33f
• Spring Cloud Gateway for
incoming HTTP traffic to redirect
to proper State Store
• State Store Host Awareness
Jan:Google,http://google.com
Jan:Google
JanYahoo:Yahoo,http://yahoo.com
Kafka Streams State Store
➔ getHostInfo

16. Questions/Comments
Nice ones only ;)

17. #CONFLUENTSKO2021
Thank you!
jsvoboda@confliuent.io