More Related Content

Slideshows for you(20)

Similar to Topic and schema management-meetupberlin(20)


More from confluent(20)


Topic and schema management-meetupberlin

  1. 1 Wax on Wax off: The Learnings of Karate kid applied to Apache Kafka Pere Urbon-Bayes (@purbon) Solutions Architect
  2. 2 @purbon Topics for today ● Common questions when building an Streaming Platform with Apache Kafka? ○ Topic Management ○ Schemas Management ● Building self sufficient teams with the Kafka Topology Builder ● Database migrations for Apache Kafka, the Kafka Migration tool
  3. 33 Common management questions for KafkaOps teams
  4. 4 @purbon Common questions for KafkaOps
  5. 5 @purbon Topics with structure and automation!
  6. 6 @purbon Automate all the things!
  7. 7 @purbon A declarative method
  8. 8 @purbon A CI/CD pipeline for Kafka Management
  9. 9 @purbon The topology descriptor
  10. 10 @purbon The topology descriptor
  11. 11 @purbon An imperative method
  12. 12 @purbon An imperative CI/CD pipeline
  13. 13 @purbon But not only for CI/CD
  14. 14 @purbon
  15. 15 @purbon
  16. 16 @purbon
  17. 17 @purbon But how do we organise this How do you replicate state between environments: ● Upgrade changes across environments. ○ Merge changes across descriptors. ○ Apply migrations across environments. ● Graduate state from lower environments. ○ Useful for schema registry using the version/latest Resource ○ Very hard for Apache Kafka (pulling latest state is very hard) ● State replication needs CI/CD and automation.
  18. 18 @purbon As teams using Apache Kafka As a user of an Apache Kafka based system, I want to: ● Be self sufficient ● Care about my area of work / project ● Get test and validation for all the changes requested
  19. 19 @purbon Approaches for making self sufficient teams There are different approaches to make self sufficient teams: ● Via CI/CD and github. ● Using a web interface (ala Control Center, Kafkawize or KafkaTool) ● Using service brokers / operators (ala Containers)
  20. 2020 Questions? Thanks! Pere Urbon Bayes Solution Architect