Microservice architectures are not free lunch! Microservices need to be decoupled, flexible, operationally transparent, data aware and elastic. Most material from last years only discusses point-to-point architectures with inflexible and non-scalable technologies like REST / HTTP. This video takes a look at cutting edge technologies like Apache Kafka, Kubernetes, Envoy, Linkerd and Istio to implement a cloud-native service mesh to solve these challenges and bring microservices to the next level of scale, speed and efficiency.
Key takeaways:
- Apache Kafka decouples services, including event streams and request-response
- Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem
- Service Mesh helps with security and observability at ecosystem / organization scale
- Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses
Blog post: http://www.kai-waehner.de/blog/2019/09/24/cloud-native-apache-kafka-kubernetes-envoy-istio-linkerd-service-mesh
Video recording of this slide deck: https://youtu.be/Us_C4RFOUrA
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
1. 1Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Event Streaming Platform and Service Mesh
Cloud-Native Apache Kafka with Kubernetes, Envoy and Istio
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
2. 2Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Takeaways
• Apache Kafka decouples services, including event streams and request-response
• Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem
• Service Mesh helps with security and observability at ecosystem / organization scale
• Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses
+
3. 3Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
4. 4Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
5. 5Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Business Digitalization Trends are Driving the Need to Process
Events at a whole new Scale, Speed and Efficiency
Mobile Cloud Microservices Internet of Things Machine Learning
The world has changed!
6. 6Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Microservices to the rescue?
• Significant Operations Overhead
• Substantial DevOps Skills Required
• Implicit Interfaces
• Duplication Of Effort
http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html
• Distributed System Complexity
• Asynchronicity Is Difficult
• Testability Challenges
7. 7Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Requirements for Microservices
Decoupled
Flexible
Operationally Transparent
Data Aware
Elastic
8. 8Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
9. 9Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
Apache Kafka—The Rise of an Event Streaming Platform
10. 10Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Apache Kafka at Scale at Tech Giants
> 4.5 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka Is not just used by tech giants
** Kafka is not just used for big data
11. 11Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Confluent - Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log
aggregation, Splunk
replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)
12. 12Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Apache Kafka - A Distributed Commit Log
Writers
Kafka
cluster
Readers
13. 13Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafka Topics
my-topic
my-topic-partition-0
my-topic-partition-1
my-topic-partition-2
broker-1
broker-2
broker-3
14. 14Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
P
Producing to Kafka
Time
C2 C3C1
16. 16Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Schema are about how teams work together
17. 17Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
A quick change of the timestamp format…
18. 18Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
… breaks things!
19. 19Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
APIs between services are Contracts
In Stream Processing World – Event Schemas ARE the API
Governance in a Streaming Architecture
20. 20Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Confluent Schema Registry
21. 21Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafka Connect
Kafka Cluster
CRM Integration
Domain-Driven Design (DDD) for your Event Steaming Platform
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
è Independent and loosely coupled, but scalable, highly available and reliable!
22. 22Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Requirements for Microservices
Decoupled
Flexible
Operationally Transparent
Data Aware
Elastic
23. 23Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
24. 24Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Cloud-Native Platforms in last five years
25. 25Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kubernetes won the battle!
26. 26Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Cloud-Native Deployment leveraging Kubernetes
27. 27Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Evolution of Kafka DevOps
Shell scripts
Ansible/Chef Docker
Kubernetes
28. 28Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafkaesque world of Kafka on Kubernetes
29. 29
Kafka on Kubernetes –
It’s tricky L
• Translating an existing architecture to Kubernetes
• Failover handling and data balancing
• Communication between ZooKeeper, Kafka Brokers, Clients (Java,
REST, Connect, KSQL), Schema Registry, etc.
• External access from / to outside Kubernetes cluster
• Persistent storage options on prem and in the cloud
• Security configuration
• Rolling upgrades
• Etc.
30. 30Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafka Operator for Kubernetes
The Operator pattern for Kubernetes aims to capture the key aim of a human operator
who is managing a service or set of services. Human operators who look after specific
applications and services have deep knowledge of how the system ought to behave,
how to deploy it, and how to react if there are problems.
People who run workloads on Kubernetes often like to use automation to take care of
repeatable tasks. The Operator pattern captures how you can write code to automate a
task beyond what Kubernetes itself provides.
Some Kafka Operators:
31. 31Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Confluent’s Kubernetes Journey building “Confluent Cloud”
05/2017
Confluent Cloud
Early Access
2016
Confluent Cloud
Development
11/2017
Confluent Cloud
GA (AWS)
07/2019
Confluent Operator GA
(Confluent Platform)
2019
Confluent Cloud GA
on AWS, GCP, Azure
32. 32Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Confluent Operator
Deployment and management automation for
Confluent Platform on Kubernetes
Including Apache Kafka, Zookeeper, Schema Registry,
Connect, Control Center, Replicator, KSQL
For organizations standardized on Kubernetes as
platform runtime
Operationalizes years of experience running Kafka on
Kubernetes on premises or the leading public clouds
Confluent Platform
Confluent Operator
Kubernetes
AWS Azure GCP
RH OpenShift Mesosphere Pivotal
On-Premises Cloud
Docker Images
Automate Deployment of Confluent Platform on Kubernetes on Any Platform at Any Scale
33. 33Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Confluent Operator enables you to:
Automate provisioning of
Kafka pods and security
configuration in minutes
Monitor SLAs through Confluent
Control Center or Prometheus
Scale Kafka elastically &
Automate rolling updates
Built on our first hand knowledge
of running Confluent at scale
Cloud-Native Deployment of Kafka and Confluent Platform
34. 34Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kubernetes
Cluster K8 NodeK8 NodeK8 Node
Replicator
Pod
C3 Pod SR Pod
K8 NodeOperator
Kafka
Pod
ZK Pod
Persistent Volumes
(AWS EBS, GCE Persistent Disk, Local Persistent Volume, etc.)
External
Access
Load
Balancers
Configurations
ConfigMapsKSQL Pod
REST Proxy
Pod
Confluent Operator Deployment
35. 35Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Requirements for Microservices
Decoupled
Flexible
Operationally Transparent
Data Aware
Elastic
36. 36Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
37. 37Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Excursus: Kubernetes Pod
“pod == small herd of aquatic mammals, esp. of whales or dolphins”
https://geekdudes.wordpress.com/2019/07/14/kubernetes-creating-pods-on-windows-10/
38. 38Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Sidecar Pattern
38
Components of the application, deployed in a separate container to provide
isolation and encapsulation. This pattern allows applications to be composed of
heterogeneous components.
39. 39Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Service Mesh
A microservice pattern to move visibility, reliability, and
security primitives for service-to-service communication
into the infrastructure layer, out of the application layer.
https://www.infoq.com/articles/linkerd-v2-production-adoption/
Data Plane
Touches every packet/request in the system.
Responsible for service discovery, health
checking, routing, load balancing,
authentication/authorization, and observability.
Control Plane
Provides policy and configuration for all of the
running data planes in the mesh. Does not touch
any packets/requests in the system. The control
plane turns all of the data planes into a distributed
system.
40. 40Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Service Mesh
Out of process architecture
• Self contained process
• Run alongside every application server
• Application sends and receives messages to
and from localhost and is unaware of the
network topology
Benefits
Compared to “fat client proxy libraries” like Finagle
(Twitter), Hystrix (Netflix), Stubby (Google):
• Works with any application language (Java,
C++, Go, PHP, Python, etc.)
• Can be deployed and upgraded quickly across
an entire infrastructure transparently
https://www.infoq.com/articles/linkerd-v2-production-adoption/
41. 41Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
42. 42Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Excursus: Load Balancing and Proxy at L3/L4 vs. L7 of OSI Model
https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy
https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236
L3/L4 vs. L7 is
not always
the right
abstraction level!
43. 43Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Excursus: Load Balancing and Proxy at L3/L4 vs. L7
https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy
https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236
Example: Envoy Proxy Features
L3/L4 filter architecture
HTTP L7 filter architecture
HTTP L7 routing
gRPC support
MongoDB L7 support
DynamoDB L7 support
Kafka L7 support (Pull request merged in May 2019)
Service discovery and dynamic configuration
Health checking
Advanced load balancing
Front / edge proxy support
Observability (stats, tracing)
44. 44Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Proxy
Proxy
Proxy
Proxy
Proxy
Service Proxy as Sidecar
45. 45Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Proxy
I have a new IP
now.
Who
cares?
I magically
know all about
it!
Example – Service Proxy as Inbound Sidecar
46. 46Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Proxy
I can recover from
errors without
drowning
Error?
No worries!
Lets retry every
millisecond forever
LOL. I’m
dropping 99%
of the retries.
Example - Service Proxy as Outbound Sidecar
47. 47Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Service Proxy Features
• Metrics without instrumenting apps
• Trace flow of requests across services
• One stable URI for each service
• Service discovery
• Monitor request latency
• Routing - A/B testing, green/blue deployments
• Circuit breaking
• Protocol translation (HTTP, gRPC, Kafka Protocol, etc.)
• Mutual TLS (mTLS)
• SSL Termination
• Integrate with 3rd party tools like Prometheus, Grafana, Zipkin, etc.
• Much more…
Observability
“is by far the most important thing that a Proxy and the Service Mesh
provide in a distributed Microservice architecture!” Matt Klein
48. 48Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Why Lyft built “envoy” Proxy
https://www.youtube.com/watch?v=55yi4MMVBi4
Matt Klein at QCon NY 2018
Developers should be able
to spend their time on
writing business applications
49. 49Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Lyft today with “envoy” Proxy
100% (!!!) communication coverage - Everything talks through Envoy Proxies
à Make monitoring, debugging, firefighting as consistent as possible
https://www.youtube.com/watch?v=55yi4MMVBi4
Matt Klein at QCon NY 2018
Service Mesh to the rescue:
• Abstract network from
application developers
• Get operational transparency
and more flexibility
50. 50Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Example: Advanced Load Balancing with Linkerd
https://linkerd.io/2016/03/16/beyond-round-robin-load-balancing-for-latency/
Since latency and failure are often
tied together in distributed systems
via timeouts, we can also express the
results in terms of failure.
If the caller of our system used a
timeout of 1 second, its success rate
would be approximately 95% with
round robin, 99% with least loaded,
and 99.9% with peak EWMA
(exponentially-weighted moving
average) - a significant difference.
51. 51Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
52. 52Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Control Plane + Proxy as Sidecar = Service Mesh
(Human Control Plane)
https://blog.envoyproxy.io/service-mesh-data-plane-vs-control-plane-2774e720f7fc
53. 53Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Control Plane + Proxy as Sidecar = Service Mesh
(Advanced Service Mesh Control Plane)
https://blog.envoyproxy.io/service-mesh-data-plane-vs-control-plane-2774e720f7fc
“Ultimately, the goal of a control plane
is to set policy that will eventually be
enacted by the data plane.”
54. 54Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Example for Control Plane - Istio Architecture
Pilot: Service discovery and
configuration of Envoy sidecar proxies
Mixer (Istio-Policy and Istio-Telemetry):
Enforcement of usage policies and
gathering of telemetry data
Ingress / Egress Gateway: Points for
traffic to ingress or exit from outside
the cluster
Citadel: Automation of key and
certificate management
Galley: Configuration management
services
55. 55Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Support for the Three Pillars of Observability
56. 56Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Requirements for Microservices
Decoupled
Flexible
Operationally Transparent
Data Aware
Elastic
57. 57Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Service Mesh Interface (SMI)
https://www.infoq.com/presentations/service-mesh-interface
58. 58Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
59. 59Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Service Mesh and Event Streaming Platform
59
Request-Response Events Streams
- Low latency
- Typically sync
- Point-to-point
- “Bespoke API”
- e.g. HTTP, gRPC
- Continuous processing
- Often async
- Event driven
- General-purpose events
- e.g. Apache Kafka
Traditionally, these are two different paradigms:
Please…
No REST vs.
Streaming FUD!
Most architectures need request-response and event streams!
60. 60Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Why not use Service Mesh and Event Streaming Platform together?
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
61. 61Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafka Connect
Kafka Cluster
CRM Integration
Clients and Servers are Independent (including their Ops Teams)
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
Proxy Proxy Proxy
Proxy Proxy Proxy
Control
Plane
62. 62Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Proxy
I am somehow
getting events
from Kafka
I’m using REST to
talk to a service
I’m really re-
directing
events to Kafka
Introduce
Vision #1: Using Service Mesh to Hide Kafka
Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh
https://www.youtube.com/watch?v=Fi292CqOm8A
63. 63Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Proxy
I am using REST too!
Kafka? Never heard of
her.I’m using REST to
talk to a service
I’m proxying
REST.
And also logging
stuff to Kafka
Vision #2: Kafka as Part of Control Plane
Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh
https://www.youtube.com/watch?v=Fi292CqOm8A
64. 64Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Vision #3: Kafka as a Service in a Mesh
Proxy
Proxy
Proxy
Proxy
Proxy
Kafka
Protocol
(TCP)
Kafka
Protocol
(TCP)
Kafka
Protocol
(TCP)
Kafka
Protocol
(TCP)
Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh
https://www.youtube.com/watch?v=Fi292CqOm8A
65. 65Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Vision #4: Front Kafka (-as-a-Service)
P
R
O
X
Y
P
R
O
X
Y
Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh
https://www.youtube.com/watch?v=Fi292CqOm8A
66. 66Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
(Potential) Features for Kafka + Service Mesh Implementation
Protocol conversion from HTTP /
gRPC to Kafka
• Tap feature to dump to a Kafka stream
• Protocol parsing for observability (stats,
logging, and trace linking with HTTP
RPCs)
• Shadow requests to a Kafka stream
instead of HTTP / gRPC shadow
• Integrate with Kafka Connect and its
whole ecosystem of connectors
Validation of Events
• Serialization format (JSON,
Avro, Protobuf, etc.)
• Message schema
• Headers, attributes, etc.
Security
• SSL Termination
• Mutual TLS (mTLS)
• Authorization
Proxy features
• Dynamic Routing
• Rate limiting at both the L4 connection
and L7 message level
• Filter, add compression, …
• Automatic topic name conversion (e.g.
for canary release or blue/green
deployment)
Monitoring and Tracing
• Request logs and stats
• Data lineage / audit log
• Audit log by taking request logs and
enriching them with the user info.
• Client specific metrics (Byte rate per
client id / per consumer groups,
versions of the client libraries,
consumer lag monitoring for the
entire data center)
67. 67Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Put logic outside Kafka vs. make deployment as simple as possible
Server-side Schema Validation on Kafka Broker
Goal: Tiered Storage and Autoscaling
68. 68Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
69. 69Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Service Mesh Implementation
Various options for a Service Mesh implementation; examples à
Some examples with Kafka, Kubernetes*, Envoy**, Istio:
• L4: Filter on Kafka Client side (rate limiting, mTLS, etc.)
• L4: Filter on Kafka Broker side (rate limiting, mTLS, etc.)
• L7: Confluent REST Proxy on Server side
• L7: Envoy’s Kafka Protocol Filter
• L7 Filter + Routing
• L7 Observability
• Many more Kafka-specific features possible
• L7: Custom proxy implementation
• Example: https://github.com/travisjeffery/kafka-proxy
* Kubernetes is assumed as de facto standard
** Envoy has best Kafka integration (in September 2019)
70. 70Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
L4 Example: Kafka + Istio @ Banzai Cloud
https://banzaicloud.com/blog/kafka-on-istio-performance/
71. 71Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
L4 Example: Kafka + Istio @ Banzai Cloud
https://banzaicloud.com/blog/kafka-on-istio-performance/
72. 72Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
L7 Example: Kafka + Confluent REST Proxy
Envoy
Proxy
I am using REST too!
Kafka? Never heard of
her.I’m using REST to
talk to a service
I’m proxying
REST.
And also logging
stuff to Kafka
Confluent
REST Proxy
I support only TCP!
HTTP
HTTP
73. 73Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafka Support in Envoy (Pull Request Merged in May 2019)
https://github.com/envoyproxy/envoy/issues/2852
https://github.com/envoyproxy/envoy/pull/4950
74. 74Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafka Support in Istio? (August 2019)
• Before PR #4950, Envoy treats Kafka as TCP, so that Istio-TCP-rules will apply (already).
• With PR #4950, Envoy can do some more fancy things and get stuff like number-of-messages in
telemetry from Kafka semantics.
• Now the 2nd part here is Istio, which needs a new vocabulary to be able to configure Envoy.
Think of ‘VirtualService’ and ‘DestinationRule’ for Kafka (or messaging in a more global sense).
• TLDR: L4 works in Istio; L7 needs some new PRs in Istio project, too!
https://istio.io/docs/tasks/traffic-management/tcp-traffic-shifting/
75. 75Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
L7 Example: Kafka + Envoy Kafka Protocol Filter
Envoy
Proxy
I am using REST too!
Kafka? Never heard of
her.I’m using REST to
talk to a service
I’m proxying
REST.
And also logging
stuff to Kafka
HTTP
TCP
(Kafka Protocol)
76. 76Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Requirements for Microservices
Decoupled
Flexible
Operationally Transparent
Data Aware
Elastic
77. 77Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Event Streaming Platform and Service Mesh
A Match Made In Heaven
+ =
78. 78Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Takeaways
• Apache Kafka decouples services, including event streams and request-response
• Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem
• Service Mesh helps with security and observability at ecosystem / organization scale
• Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses
+
79. 79Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!