Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Topic and schema management-meetupberlin

858 views

Published on

Pere Urbon-Bayes, Solutions Architect, Confluent
Wax on Wax off: The Learnings of Karate Kid applied to Apache Kafka® https://www.meetup.com/Berlin-Apache-Kafka-Meetup-by-Confluent/events/266567425/

Published in: Technology
  • Login to see the comments

  • Be the first to like this

Topic and schema management-meetupberlin

  1. 1. 1 Wax on Wax off: The Learnings of Karate kid applied to Apache Kafka Pere Urbon-Bayes (@purbon) Solutions Architect
  2. 2. 2 @purbon Topics for today ● Common questions when building an Streaming Platform with Apache Kafka? ○ Topic Management ○ Schemas Management ● Building self sufficient teams with the Kafka Topology Builder ● Database migrations for Apache Kafka, the Kafka Migration tool
  3. 3. 33 Common management questions for KafkaOps teams
  4. 4. 4 @purbon Common questions for KafkaOps
  5. 5. 5 @purbon Topics with structure and automation!
  6. 6. 6 @purbon Automate all the things!
  7. 7. 7 @purbon A declarative method
  8. 8. 8 @purbon A CI/CD pipeline for Kafka Management
  9. 9. 9 @purbon The topology descriptor
  10. 10. 10 @purbon The topology descriptor
  11. 11. 11 @purbon An imperative method
  12. 12. 12 @purbon An imperative CI/CD pipeline
  13. 13. 13 @purbon But not only for CI/CD
  14. 14. 14 @purbon
  15. 15. 15 @purbon
  16. 16. 16 @purbon
  17. 17. 17 @purbon But how do we organise this How do you replicate state between environments: ● Upgrade changes across environments. ○ Merge changes across descriptors. ○ Apply migrations across environments. ● Graduate state from lower environments. ○ Useful for schema registry using the version/latest Resource ○ Very hard for Apache Kafka (pulling latest state is very hard) ● State replication needs CI/CD and automation.
  18. 18. 18 @purbon As teams using Apache Kafka As a user of an Apache Kafka based system, I want to: ● Be self sufficient ● Care about my area of work / project ● Get test and validation for all the changes requested
  19. 19. 19 @purbon Approaches for making self sufficient teams There are different approaches to make self sufficient teams: ● Via CI/CD and github. ● Using a web interface (ala Control Center, Kafkawize or KafkaTool) ● Using service brokers / operators (ala Containers)
  20. 20. 2020 Questions? Thanks! Pere Urbon Bayes Solution Architect pere@confluent.io

×