This document discusses different strategies for upgrading Kafka Streams applications: rolling upgrades, blue-green deployments, and full restarts. It describes the basic Kafka Streams architecture including topics, state stores, and sub-topologies. It then explains the benefits, considerations, and when to use each upgrade strategy. Rolling upgrades avoid downtime but require careful planning. Blue-green deployments minimize downtime through parallel workloads but are more expensive. Full restarts incur downtime but are needed when upgrading all legacy applications.

1. Moving Towards Better Upgrades in Kafka Streams
Russ Katz, Customer Success Technical Architect
Whitney Steward, Customer Success Technical Architect

2. 2
Whitney Steward
Whitney loves UK post punk, is a coffee snob,
and outside of work loves to spend time with
her husband and her two dogs. She is
passionate about helping her customers be
successful, and educating customers new to
kafka with a creative approach.
Russ Katz
Russ Katz's career spans over 20 years
as a developer, operator and architect
including a decade of distributed
systems experience. Outside of working
with large scale systems Russ has a
passion for growing (and eating) the
hottest chili peppers he can find.

3. Agenda
● Introduction
● Simple Kafka Streams Architecture
● KStreams Architecture
● Three Upgrade Strategies
○ Rolling
○ Blue Green
○ Full Restart

4. Architecture

5. Topology Processor
● Is a node in the processor topology
● Represent a processing step
○ Receives an input record and could produce one
or multiple output records
Sub-Topology
● Set of processors
● Connected as parent/child or via state store
● Different sub-topologies exchange data via
topics
File System
● StreamConfig.STATE_DIR_CONFIG ( state.dir )
● Sub-topology + partition

6. Stateless vs Stateful

7. KStreams And State Stores
7
TOPOLOGY
└─ users-application-v2
├─ 0_0
├─ 1_0
│ └─ rocksdb
│ └─ KSTREAM-AGGREGATE-STATE-STORE-00000005
└─ kafka-streams-process-metadata
TOPICS
# kafka-topics –bootstrap-server node1:9092 –list | grep users
users
users-application-v1-KSTREAM-AGGREGATE-STATE-STORE-0000000005-repartition
users-application-v1-KSTREAM-AGGREGATE-STATE-STORE-0000000005-changelog
users-region

8. KStreams Custom Naming
8
TOPOLOGY
└─ users-application-v2
├─ 0_0
├─ 1_0
│ └─ rocksdb
│ └─ users-by-region-store
└─ kafka-streams-process-metadata
TOPICS
# kafka-topics –bootstrap-server node1:9092 –list | grep users
users
users-application-v2-grouping-users-by-region-repartition
users-application-v2-users-by-regions-store-changelog
users-region

9. Upgrade Strategies

10. Rolling Upgrades
Benefits:
Does not incur downtime - except when
re-balancing the partitions amongst the threads of
the Kafka Streams application
Cons:
Meticulous/requires planning to execute
Considerations:
State store migration ~ requires
extra-consideration to guarantee state consistency
across the upgrade and no data loss.
When to use?
When you do not have breaking changes in
topology

11. Blue-Green Deployment
Considerations:
● When can you move traffic over?
● Do you have something in place to validate that
newest version is effective?
● When can you clean up?
Benefits:
Minimal downtime
When to use?
When you want parallel workloads
Cons:
Expensive

12. Full Restart Upgrade
Benefits:
Upgrade to the latest version to take advantage of
new features available in kstreams or there’s a
performance issue.
Cons:
Incurs downtime
When to use?
When you need to upgrade all your legacy
applications and can afford the downtime.
Pre Upgrade Dance: Waltz

13. Full Restart Upgrade
Benefits:
Upgrade to the latest version to take advantage of
new features available in kstreams or there’s a
performance issue.
Cons:
Incurs downtime
When to use?
When you need to upgrade all your legacy
applications and can afford the downtime.
Post Upgrade Dance: Tango