Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kafka on Kubernetes

30 views

Published on

Victor Gamov from Confluent presented 'Streams must fFlow: Developing fault tolerant stream processing application with Kafka Streams and Kubernetes’ at Montreal's very first Cloud Native Day, which took place on June 11, 2019.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Kafka on Kubernetes

  1. 1. @gamussa | #ImCloudNative | @ConfluentINc Kafka on Kubernetes: Does it really have to be «The Hard Way»? June, 2019 / Montreal, CA, 2019 @gamussa | #ImCloudNative | @ConfluentINc
  2. 2. @gamussa | #ImCloudNative | @ConfluentINc 2
  3. 3. @gamussa | #ImCloudNative | @ConfluentINc 3 I build highly scalable Hello World apps
  4. 4. Raffle, yeah 🚀 Follow @gamussa @confluentinc 📸 🖼 👬 Tag @gamussa With #ImCloudNative
  5. 5. @gamussa | #ImCloudNative | @ConfluentINc 5 A company is build on DATA FLOWS but All we have is DATA STORES
  6. 6. @gamussa | #ImCloudNative | @ConfluentINc 6 Pre-Streaming
  7. 7. @gamussa | #ImCloudNative | @ConfluentINc 7
  8. 8. @gamussa | #ImCloudNative | @ConfluentINc 8 New World Streaming first • DB/DWH + Many more distributed data systems • Monolith -> Microservices • Batch -> Real-time
  9. 9. @gamussa | #ImCloudNative | @ConfluentINc 9 Evolution of #devkafkaops Shell scripts ansible/chef Docker Kubernetes
  10. 10. @gamussa | #ImCloudNative | @ConfluentINc 10 https://twitter.com/claesmogren/status/1108456688175079424
  11. 11. @gamussa | #ImCloudNative | @ConfluentINc 11 🙋
  12. 12. @gamussa | #ImCloudNative | @ConfluentINc 12 Who run stateless workloads in Kubernetes? Who thinks it’s a good idea? Who run stateful workloads in Kubernetes? Who thinks it’s a good idea? 🙋
  13. 13. @gamussa | #ImCloudNative | @ConfluentINc 13 kafkaesque world of Kafka on Kubernetes
  14. 14. @gamussa | #ImCloudNative | @ConfluentINc 14 Well, it’s tricky © Translating an existing architecture to Kubernetes External access to brokers and other components Persistent Storage options on prem and clouds Security Configuration and Upgrades #devkafkaops
  15. 15. @gamussa | #ImCloudNative | @ConfluentINc 15 But I just want to deploy kafka
  16. 16. @gamussa | #ImCloudNative | @ConfluentINc 16 Workloads Deployment
  17. 17. @gamussa | #ImCloudNative | @ConfluentINc 17 Helm Charts
  18. 18. @gamussa | #ImCloudNative | @ConfluentINc 18 Helm Charts
  19. 19. @gamussa | #ImCloudNative | @ConfluentINc 19 Helm Charts
  20. 20. @gamussa | #ImCloudNative | @ConfluentINc 20 https://cnfl.io/helm_video
  21. 21. @gamussa | #ImCloudNative | @ConfluentINc 21 Helm Charts is just a GO Templates. How Charts help with rolling update?
  22. 22. @gamussa | #ImCloudNative | @ConfluentINc 22 Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under- replicated partitions 5. Upgrade the next broker
  23. 23. @gamussa | #ImCloudNative | @ConfluentINc 23 We will use StatefulSets with OrderedReady
  24. 24. @gamussa | #ImCloudNative | @ConfluentINc 24 We need SRE / Operator knowledge to manage the platform. You need Operator!
  25. 25. @gamussa | #ImCloudNative | @ConfluentINc 25
  26. 26. @gamussa | #ImCloudNative | @ConfluentINc 26 Show me your Operator
  27. 27. @gamussa | #ImCloudNative | @ConfluentINc 27 Rider Driver 2 Web Apps powered by a microservice each Apps are a ride sharing driver and rider location tracking Rider hails a ride and the driver drives towards the rider Real time location events powered by Kafka on Kubernetes KLyfft - Kafka app on Kubernetes
  28. 28. @gamussa | #ImCloudNative | @ConfluentINc 28 Demo
  29. 29. @gamussa | #ImCloudNative | @ConfluentINc 29 DO KAFKA ON KUBERNETES DEMO AND EVERYONE LOOSES THEIR MIND
  30. 30. @gamussa | #ImCloudNative | @ConfluentINc 30 Confluent Operator - Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components
  31. 31. @gamussa | #ImCloudNative | @ConfluentINc 31 Be like Justin!
  32. 32. @gamussa | #ImCloudNative | @ConfluentINc 32 Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under- replicated partitions 5. Upgrade the next broker
  33. 33. @gamussa | #ImCloudNative | @ConfluentINc 33 Want to see advanced deployment? vs.
  34. 34. @gamussa | #ImCloudNative | @ConfluentINc 34 http://gamov.dev/ksummit_nyc19_k8s
  35. 35. @gamussa | #ImCloudNative | @ConfluentINc 35 GA Plans● We are in private Preview Release is over! ● 24 customers testing the Operator in Preview: ● Global customers ● Banks, Fin Tech, Retailers, Consumer Tech ● We are in the final stages of Preview and about to launch soon
  36. 36. @@gamussa | #ImCloudNative | @ConfluentINc Thanks! @gamussa viktor@confluent.io https://slackpass.io/confluentcommunity #kubernetes
  37. 37. 37

×