1.
1
Wax on Wax off:
The Learnings of Karate kid
applied to Apache Kafka
Pere Urbon-Bayes (@purbon)
Solutions Architect

2.
2
@purbon
Topics for today
● Common questions when building an Streaming Platform with Apache Kafka?
○ Topic Management
○ Schemas Management
● Building self sufficient teams with the Kafka Topology Builder
● Database migrations for Apache Kafka, the Kafka Migration tool

3.
33
Common management questions for
KafkaOps teams

4.
4
@purbon
Common
questions for
KafkaOps

5.
5
@purbon
Topics with structure and automation!

6.
6
@purbon
Automate all the things!

7.
7
@purbon
A declarative method

8.
8
@purbon
A CI/CD pipeline for Kafka Management

9.
9
@purbon
The topology descriptor

10.
10
@purbon
The topology descriptor

11.
11
@purbon
An imperative method

12.
12
@purbon
An imperative CI/CD pipeline

13.
13
@purbon
But not only for CI/CD

14.
14
@purbon

15.
15
@purbon

16.
16
@purbon

17.
17
@purbon
But how do we organise this
How do you replicate state between environments:
● Upgrade changes across environments.
○ Merge changes across descriptors.
○ Apply migrations across environments.
● Graduate state from lower environments.
○ Useful for schema registry using the version/latest Resource
○ Very hard for Apache Kafka (pulling latest state is very hard)
● State replication needs CI/CD and automation.

18.
18
@purbon
As teams using Apache Kafka
As a user of an Apache Kafka based system, I want to:
● Be self sufficient
● Care about my area of work / project
● Get test and validation for all the changes requested

19.
19
@purbon
Approaches for making self sufficient teams
There are different approaches to make self sufficient teams:
● Via CI/CD and github.
● Using a web interface (ala Control Center, Kafkawize or KafkaTool)
● Using service brokers / operators (ala Containers)

20.
2020
Questions? Thanks!
Pere Urbon Bayes
Solution Architect
pere@confluent.io