Wax on Wax off:
The Learnings of Karate kid
applied to Apache Kafka
Pere Urbon-Bayes (@purbon)
Topics for today
● Common questions when building an Streaming Platform with Apache Kafka?
○ Topic Management
○ Schemas Management
● Building self suﬃcient teams with the Kafka Topology Builder
● Database migrations for Apache Kafka, the Kafka Migration tool
But how do we organise this
How do you replicate state between environments:
● Upgrade changes across environments.
○ Merge changes across descriptors.
○ Apply migrations across environments.
● Graduate state from lower environments.
○ Useful for schema registry using the version/latest Resource
○ Very hard for Apache Kafka (pulling latest state is very hard)
● State replication needs CI/CD and automation.
As teams using Apache Kafka
As a user of an Apache Kafka based system, I want to:
● Be self suﬃcient
● Care about my area of work / project
● Get test and validation for all the changes requested
Approaches for making self suﬃcient teams
There are different approaches to make self suﬃcient teams:
● Via CI/CD and github.
● Using a web interface (ala Control Center, Kafkawize or KafkaTool)
● Using service brokers / operators (ala Containers)