1
Wax on Wax off:
The Learnings of Karate kid
applied to Apache Kafka
Pere Urbon-Bayes (@purbon)
Solutions Architect

2
@purbon
Topics for today
● Common questions when building an Streaming Platform with Apache Kafka?
○ Topic Management
○ Schemas Management
● Building self sufficient teams with the Kafka Topology Builder
● Database migrations for Apache Kafka, the Kafka Migration tool

33
Common management questions for
KafkaOps teams

4
@purbon
Common
questions for
KafkaOps

5
@purbon
Topics with structure and automation!

6
@purbon
Automate all the things!

7
@purbon
A declarative method

8
@purbon
A CI/CD pipeline for Kafka Management

9
@purbon
The topology descriptor

10
@purbon
The topology descriptor

11
@purbon
An imperative method

12
@purbon
An imperative CI/CD pipeline

13
@purbon
But not only for CI/CD

14
@purbon

15
@purbon

16
@purbon

17
@purbon
But how do we organise this
How do you replicate state between environments:
● Upgrade changes across environments.
○ Merge changes across descriptors.
○ Apply migrations across environments.
● Graduate state from lower environments.
○ Useful for schema registry using the version/latest Resource
○ Very hard for Apache Kafka (pulling latest state is very hard)
● State replication needs CI/CD and automation.

18
@purbon
As teams using Apache Kafka
As a user of an Apache Kafka based system, I want to:
● Be self sufficient
● Care about my area of work / project
● Get test and validation for all the changes requested

19
@purbon
Approaches for making self sufficient teams
There are different approaches to make self sufficient teams:
● Via CI/CD and github.
● Using a web interface (ala Control Center, Kafkawize or KafkaTool)
● Using service brokers / operators (ala Containers)

2020
Questions? Thanks!
Pere Urbon Bayes
Solution Architect
pere@confluent.io