Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd

1. 1Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Event Streaming Platform and Service Mesh Cloud-Native Apache Kafka with Kubernetes, Envoy and Istio Kai Waehner Technology Evangelist contact@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de

2. 2Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Key Takeaways • Apache Kafka decouples services, including event streams and request-response • Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem • Service Mesh helps with security and observability at ecosystem / organization scale • Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses +

3. 3Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Agenda • Motivation, Challenges, Requirements of Microservices • Apache Kafka - The Event Streaming Platform for Microservices • Kubernetes for Cloud-Native Microservices • Service Mesh • Service Proxy (aka Data Plane) • Control Plane • Kafka and Service Mesh • Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio

5. 5Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Business Digitalization Trends are Driving the Need to Process Events at a whole new Scale, Speed and Efficiency Mobile Cloud Microservices Internet of Things Machine Learning The world has changed!

6. 6Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Microservices to the rescue? • Significant Operations Overhead • Substantial DevOps Skills Required • Implicit Interfaces • Duplication Of Effort http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html • Distributed System Complexity • Asynchronicity Is Difficult • Testability Challenges

7. 7Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Key Requirements for Microservices Decoupled Flexible Operationally Transparent Data Aware Elastic

9. 9Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner The Log ConnectorsConnectors Producer Consumer Streaming Engine Apache Kafka—The Rise of an Event Streaming Platform

10. 10Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Apache Kafka at Scale at Tech Giants > 4.5 trillion messages / day > 6 Petabytes / day “You name it” * Kafka Is not just used by tech giants ** Kafka is not just used for big data

11. 11Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Confluent - Business Value per Use Case Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $ Example Case Studies (of many)

12. 12Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Apache Kafka - A Distributed Commit Log Writers Kafka cluster Readers

13. 13Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafka Topics my-topic my-topic-partition-0 my-topic-partition-1 my-topic-partition-2 broker-1 broker-2 broker-3

14. 14Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner P Producing to Kafka Time C2 C3C1

15. 15Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Partition Leadership and Replication Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4

16. 16Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Schema are about how teams work together

17. 17Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner A quick change of the timestamp format…

18. 18Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner … breaks things!

19. 19Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner APIs between services are Contracts In Stream Processing World – Event Schemas ARE the API Governance in a Streaming Architecture

20. 20Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Confluent Schema Registry

21. 21Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafka Connect Kafka Cluster CRM Integration Domain-Driven Design (DDD) for your Event Steaming Platform Legacy Integration Custom Application ESB Connector Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform CRM Domain Legacy Domain Payment Domain è Independent and loosely coupled, but scalable, highly available and reliable!

24. 24Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Cloud-Native Platforms in last five years

25. 25Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kubernetes won the battle!

26. 26Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Cloud-Native Deployment leveraging Kubernetes

27. 27Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Evolution of Kafka DevOps Shell scripts Ansible/Chef Docker Kubernetes

28. 28Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafkaesque world of Kafka on Kubernetes

29. 29 Kafka on Kubernetes – It’s tricky L • Translating an existing architecture to Kubernetes • Failover handling and data balancing • Communication between ZooKeeper, Kafka Brokers, Clients (Java, REST, Connect, KSQL), Schema Registry, etc. • External access from / to outside Kubernetes cluster • Persistent storage options on prem and in the cloud • Security configuration • Rolling upgrades • Etc.

30. 30Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafka Operator for Kubernetes The Operator pattern for Kubernetes aims to capture the key aim of a human operator who is managing a service or set of services. Human operators who look after specific applications and services have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems. People who run workloads on Kubernetes often like to use automation to take care of repeatable tasks. The Operator pattern captures how you can write code to automate a task beyond what Kubernetes itself provides. Some Kafka Operators:

31. 31Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Confluent’s Kubernetes Journey building “Confluent Cloud” 05/2017 Confluent Cloud Early Access 2016 Confluent Cloud Development 11/2017 Confluent Cloud GA (AWS) 07/2019 Confluent Operator GA (Confluent Platform) 2019 Confluent Cloud GA on AWS, GCP, Azure

32. 32Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Confluent Operator Deployment and management automation for Confluent Platform on Kubernetes Including Apache Kafka, Zookeeper, Schema Registry, Connect, Control Center, Replicator, KSQL For organizations standardized on Kubernetes as platform runtime Operationalizes years of experience running Kafka on Kubernetes on premises or the leading public clouds Confluent Platform Confluent Operator Kubernetes AWS Azure GCP RH OpenShift Mesosphere Pivotal On-Premises Cloud Docker Images Automate Deployment of Confluent Platform on Kubernetes on Any Platform at Any Scale

33. 33Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Confluent Operator enables you to: Automate provisioning of Kafka pods and security configuration in minutes Monitor SLAs through Confluent Control Center or Prometheus Scale Kafka elastically & Automate rolling updates Built on our first hand knowledge of running Confluent at scale Cloud-Native Deployment of Kafka and Confluent Platform

34. 34Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kubernetes Cluster K8 NodeK8 NodeK8 Node Replicator Pod C3 Pod SR Pod K8 NodeOperator Kafka Pod ZK Pod Persistent Volumes (AWS EBS, GCE Persistent Disk, Local Persistent Volume, etc.) External Access Load Balancers Configurations ConfigMapsKSQL Pod REST Proxy Pod Confluent Operator Deployment

37. 37Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Excursus: Kubernetes Pod “pod == small herd of aquatic mammals, esp. of whales or dolphins” https://geekdudes.wordpress.com/2019/07/14/kubernetes-creating-pods-on-windows-10/

38. 38Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Sidecar Pattern 38 Components of the application, deployed in a separate container to provide isolation and encapsulation. This pattern allows applications to be composed of heterogeneous components.

39. 39Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Service Mesh A microservice pattern to move visibility, reliability, and security primitives for service-to-service communication into the infrastructure layer, out of the application layer. https://www.infoq.com/articles/linkerd-v2-production-adoption/ Data Plane Touches every packet/request in the system. Responsible for service discovery, health checking, routing, load balancing, authentication/authorization, and observability. Control Plane Provides policy and configuration for all of the running data planes in the mesh. Does not touch any packets/requests in the system. The control plane turns all of the data planes into a distributed system.

40. 40Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Service Mesh Out of process architecture • Self contained process • Run alongside every application server • Application sends and receives messages to and from localhost and is unaware of the network topology Benefits Compared to “fat client proxy libraries” like Finagle (Twitter), Hystrix (Netflix), Stubby (Google): • Works with any application language (Java, C++, Go, PHP, Python, etc.) • Can be deployed and upgraded quickly across an entire infrastructure transparently https://www.infoq.com/articles/linkerd-v2-production-adoption/

42. 42Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Excursus: Load Balancing and Proxy at L3/L4 vs. L7 of OSI Model https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236 L3/L4 vs. L7 is not always the right abstraction level!

43. 43Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Excursus: Load Balancing and Proxy at L3/L4 vs. L7 https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236 Example: Envoy Proxy Features L3/L4 filter architecture HTTP L7 filter architecture HTTP L7 routing gRPC support MongoDB L7 support DynamoDB L7 support Kafka L7 support (Pull request merged in May 2019) Service discovery and dynamic configuration Health checking Advanced load balancing Front / edge proxy support Observability (stats, tracing)

44. 44Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Proxy Proxy Proxy Proxy Proxy Service Proxy as Sidecar

45. 45Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Proxy I have a new IP now. Who cares? I magically know all about it! Example – Service Proxy as Inbound Sidecar

46. 46Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Proxy I can recover from errors without drowning Error? No worries! Lets retry every millisecond forever LOL. I’m dropping 99% of the retries. Example - Service Proxy as Outbound Sidecar

47. 47Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Service Proxy Features • Metrics without instrumenting apps • Trace flow of requests across services • One stable URI for each service • Service discovery • Monitor request latency • Routing - A/B testing, green/blue deployments • Circuit breaking • Protocol translation (HTTP, gRPC, Kafka Protocol, etc.) • Mutual TLS (mTLS) • SSL Termination • Integrate with 3rd party tools like Prometheus, Grafana, Zipkin, etc. • Much more… Observability “is by far the most important thing that a Proxy and the Service Mesh provide in a distributed Microservice architecture!” Matt Klein

48. 48Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Why Lyft built “envoy” Proxy https://www.youtube.com/watch?v=55yi4MMVBi4 Matt Klein at QCon NY 2018 Developers should be able to spend their time on writing business applications

49. 49Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Lyft today with “envoy” Proxy 100% (!!!) communication coverage - Everything talks through Envoy Proxies à Make monitoring, debugging, firefighting as consistent as possible https://www.youtube.com/watch?v=55yi4MMVBi4 Matt Klein at QCon NY 2018 Service Mesh to the rescue: • Abstract network from application developers • Get operational transparency and more flexibility

50. 50Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Example: Advanced Load Balancing with Linkerd https://linkerd.io/2016/03/16/beyond-round-robin-load-balancing-for-latency/ Since latency and failure are often tied together in distributed systems via timeouts, we can also express the results in terms of failure. If the caller of our system used a timeout of 1 second, its success rate would be approximately 95% with round robin, 99% with least loaded, and 99.9% with peak EWMA (exponentially-weighted moving average) - a significant difference.

52. 52Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Control Plane + Proxy as Sidecar = Service Mesh (Human Control Plane) https://blog.envoyproxy.io/service-mesh-data-plane-vs-control-plane-2774e720f7fc

53. 53Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Control Plane + Proxy as Sidecar = Service Mesh (Advanced Service Mesh Control Plane) https://blog.envoyproxy.io/service-mesh-data-plane-vs-control-plane-2774e720f7fc “Ultimately, the goal of a control plane is to set policy that will eventually be enacted by the data plane.”

54. 54Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Example for Control Plane - Istio Architecture Pilot: Service discovery and configuration of Envoy sidecar proxies Mixer (Istio-Policy and Istio-Telemetry): Enforcement of usage policies and gathering of telemetry data Ingress / Egress Gateway: Points for traffic to ingress or exit from outside the cluster Citadel: Automation of key and certificate management Galley: Configuration management services

55. 55Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Support for the Three Pillars of Observability

57. 57Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Service Mesh Interface (SMI) https://www.infoq.com/presentations/service-mesh-interface

59. 59Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Service Mesh and Event Streaming Platform 59 Request-Response Events Streams - Low latency - Typically sync - Point-to-point - “Bespoke API” - e.g. HTTP, gRPC - Continuous processing - Often async - Event driven - General-purpose events - e.g. Apache Kafka Traditionally, these are two different paradigms: Please… No REST vs. Streaming FUD! Most architectures need request-response and event streams!

60. 60Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Why not use Service Mesh and Event Streaming Platform together? Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy

61. 61Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafka Connect Kafka Cluster CRM Integration Clients and Servers are Independent (including their Ops Teams) Legacy Integration Custom Application ESB Connector Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform CRM Domain Legacy Domain Payment Domain Proxy Proxy Proxy Proxy Proxy Proxy Control Plane

62. 62Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Proxy I am somehow getting events from Kafka I’m using REST to talk to a service I’m really re- directing events to Kafka Introduce Vision #1: Using Service Mesh to Hide Kafka Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh https://www.youtube.com/watch?v=Fi292CqOm8A

63. 63Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Proxy I am using REST too! Kafka? Never heard of her.I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka Vision #2: Kafka as Part of Control Plane Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh https://www.youtube.com/watch?v=Fi292CqOm8A

64. 64Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Vision #3: Kafka as a Service in a Mesh Proxy Proxy Proxy Proxy Proxy Kafka Protocol (TCP) Kafka Protocol (TCP) Kafka Protocol (TCP) Kafka Protocol (TCP) Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh https://www.youtube.com/watch?v=Fi292CqOm8A

65. 65Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Vision #4: Front Kafka (-as-a-Service) P R O X Y P R O X Y Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh https://www.youtube.com/watch?v=Fi292CqOm8A

66. 66Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner (Potential) Features for Kafka + Service Mesh Implementation Protocol conversion from HTTP / gRPC to Kafka • Tap feature to dump to a Kafka stream • Protocol parsing for observability (stats, logging, and trace linking with HTTP RPCs) • Shadow requests to a Kafka stream instead of HTTP / gRPC shadow • Integrate with Kafka Connect and its whole ecosystem of connectors Validation of Events • Serialization format (JSON, Avro, Protobuf, etc.) • Message schema • Headers, attributes, etc. Security • SSL Termination • Mutual TLS (mTLS) • Authorization Proxy features • Dynamic Routing • Rate limiting at both the L4 connection and L7 message level • Filter, add compression, … • Automatic topic name conversion (e.g. for canary release or blue/green deployment) Monitoring and Tracing • Request logs and stats • Data lineage / audit log • Audit log by taking request logs and enriching them with the user info. • Client specific metrics (Byte rate per client id / per consumer groups, versions of the client libraries, consumer lag monitoring for the entire data center)

67. 67Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Put logic outside Kafka vs. make deployment as simple as possible Server-side Schema Validation on Kafka Broker Goal: Tiered Storage and Autoscaling

69. 69Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Service Mesh Implementation Various options for a Service Mesh implementation; examples à Some examples with Kafka, Kubernetes*, Envoy**, Istio: • L4: Filter on Kafka Client side (rate limiting, mTLS, etc.) • L4: Filter on Kafka Broker side (rate limiting, mTLS, etc.) • L7: Confluent REST Proxy on Server side • L7: Envoy’s Kafka Protocol Filter • L7 Filter + Routing • L7 Observability • Many more Kafka-specific features possible • L7: Custom proxy implementation • Example: https://github.com/travisjeffery/kafka-proxy * Kubernetes is assumed as de facto standard ** Envoy has best Kafka integration (in September 2019)

70. 70Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner L4 Example: Kafka + Istio @ Banzai Cloud https://banzaicloud.com/blog/kafka-on-istio-performance/

71. 71Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner L4 Example: Kafka + Istio @ Banzai Cloud https://banzaicloud.com/blog/kafka-on-istio-performance/

72. 72Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner L7 Example: Kafka + Confluent REST Proxy Envoy Proxy I am using REST too! Kafka? Never heard of her.I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka Confluent REST Proxy I support only TCP! HTTP HTTP

73. 73Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafka Support in Envoy (Pull Request Merged in May 2019) https://github.com/envoyproxy/envoy/issues/2852 https://github.com/envoyproxy/envoy/pull/4950

74. 74Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafka Support in Istio? (August 2019) • Before PR #4950, Envoy treats Kafka as TCP, so that Istio-TCP-rules will apply (already). • With PR #4950, Envoy can do some more fancy things and get stuff like number-of-messages in telemetry from Kafka semantics. • Now the 2nd part here is Istio, which needs a new vocabulary to be able to configure Envoy. Think of ‘VirtualService’ and ‘DestinationRule’ for Kafka (or messaging in a more global sense). • TLDR: L4 works in Istio; L7 needs some new PRs in Istio project, too! https://istio.io/docs/tasks/traffic-management/tcp-traffic-shifting/

75. 75Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner L7 Example: Kafka + Envoy Kafka Protocol Filter Envoy Proxy I am using REST too! Kafka? Never heard of her.I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka HTTP TCP (Kafka Protocol)

77. 77Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Event Streaming Platform and Service Mesh A Match Made In Heaven + =

78. 78Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Key Takeaways • Apache Kafka decouples services, including event streams and request-response • Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem • Service Mesh helps with security and observability at ecosystem / organization scale • Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses +

79. 79Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kai Waehner Technology Evangelist contact@kai-waehner.de @KaiWaehner www.kai-waehner.de www.confluent.io LinkedIn Questions? Feedback? Let’s connect!

Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd

Kai Wähner

Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd