Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Gluecon - Kafka and the service mesh

3,269 views

Published on

Exploring the problem of Microservices communication and how both Kafka and Service Mesh solutions address it. We then look at some approaches for combining both.

Published in: Technology
  • Be the first to comment

Gluecon - Kafka and the service mesh

  1. 1. 1 Kafka and the Service Mesh Gwen Shapira, Principal Data Architect @gwenshap
  2. 2. You Have Microservices 2
  3. 3. They need to communicate 3
  4. 4. 4 Orders Returns ? Invento ry Fulfill Order Validat e Order I know! I’ll use REST APIs
  5. 5. Synchronous request-response communication Leads to Tight point-to-point coupling 5
  6. 6. 6 Making Changes is Risky Change
  7. 7. 7 Adding Services Requires Explicit Calls Mistak e Handle r
  8. 8. All these lines of communication require: Logging, monitoring, error handling and more 8
  9. 9. The most difficult thing about microservices is data. Especially joining data. 9
  10. 10. We need a way to communicate that is: Decoupled Flexible Operationally Transparent Data Aware 10
  11. 11. Kafka? 11
  12. 12. 15 From This:
  13. 13. 16 To This:
  14. 14. 17 And This: New Servi ce
  15. 15. Kafka: - Immutable, ordered, stream (log) of events - Highly Available - Scalable - Load balancing consumers - HA consumers - (Potentially) infinite storage
  16. 16. Kafka + Schema Registry 20
  17. 17. APIs between services are Contracts In Stream Processing World – Event Schemas ARE the API …except they stick around for a lot longer
  18. 18. We need a way to communicate that is: Decoupled Flexible Operationally Transparent Data Aware 27
  19. 19. Service Mesh? 28
  20. 20. It is a proxy.
  21. 21. Sidecar: Components of the application, deployed in a separate container to provide isolation and encapsulation. This pattern allows applications to be composed of heterogeneous components.
  22. 22. 31
  23. 23. Proxy as sidecar: Proxy Proxy Proxy Proxy Proxy
  24. 24. Proxy I have a new IP now. YOLO! Who cares? I magically know all about it!
  25. 25. Proxy I can recover from errors without drowning Error? No worries! Lets retry every millisecond forever LOL. I’m dropping 99% of the retries.
  26. 26. Cool Things proxies do! - One stable uri for each service - Service discovery - Log requests - Monitor request latency - Routing - A/B testing, green/blue deployments - Circuit breaking - Protocol translation - SSL Termination
  27. 27. They don’t do it alone
  28. 28. We need a way to communicate that is: Decoupled Flexible Operationally Transparent Data Aware 37
  29. 29. How do we choose? 38
  30. 30. 40 Two different paradigms Request-Response Events Streams - Very Low latency - Often sync - Point-to-point - “Bespoke API” - Continuous processing - Often async - Event driven - General-purpose events
  31. 31. Vision #1: Using Service Mesh to hide Kafka Introduce
  32. 32. Proxy I am somehow getting events from Kafka I’m using REST to talk to a service I’m really re- directing events to Kafka
  33. 33. REST Proxy does bits of that. HTTP on one-side, Kafka on the other. We are adding more protocols: - JMS - MQTT - gRPC
  34. 34. Vision #2: Kafka as part of control plane
  35. 35. Proxy I am using REST too! Kafka? Never heard of her.I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka
  36. 36. Vision #3: Kafka as a Service in a Mesh
  37. 37. 47 Like This: Proxy Proxy Proxy Proxy Proxy
  38. 38. Vision #4: Front Kafka
  39. 39. 52 Like This: P R O X Y P R O X Y
  40. 40. Lets Sum Things Up 53
  41. 41. - REST API point-to-point communication hits the wall - Service Mesh adds discovery and operationalibility - Kafka adds data, compatibility, immutable storage - You can have both. In many different tastes. - Coming soon to a proxy near you
  42. 42. Resources and Next Steps https://github.com/confluentinc/cp-demo https://www.confluent.io/download/ https://slackpass.io/confluentcommunity https://www.confluent.io/blog https://www.confluent.io/confluent-operator/ https://github.com/envoyproxy/envoy/issues/2852

×