SlideShare a Scribd company logo
1 of 79
1Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Event Streaming Platform and Service Mesh
Cloud-Native Apache Kafka with Kubernetes, Envoy and Istio
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
2Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Takeaways
• Apache Kafka decouples services, including event streams and request-response
• Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem
• Service Mesh helps with security and observability at ecosystem / organization scale
• Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses
+
3Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
4Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
5Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Business Digitalization Trends are Driving the Need to Process
Events at a whole new Scale, Speed and Efficiency
Mobile Cloud Microservices Internet of Things Machine Learning
The world has changed!
6Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Microservices to the rescue?
• Significant Operations Overhead
• Substantial DevOps Skills Required
• Implicit Interfaces
• Duplication Of Effort
http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html
• Distributed System Complexity
• Asynchronicity Is Difficult
• Testability Challenges
7Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Requirements for Microservices
Decoupled
Flexible
Operationally Transparent
Data Aware
Elastic
8Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
9Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
Apache Kafka—The Rise of an Event Streaming Platform
10Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Apache Kafka at Scale at Tech Giants
> 4.5 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka Is not just used by tech giants
** Kafka is not just used for big data
11Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Confluent - Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log
aggregation, Splunk
replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)
12Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Apache Kafka - A Distributed Commit Log
Writers
Kafka
cluster
Readers
13Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafka Topics
my-topic
my-topic-partition-0
my-topic-partition-1
my-topic-partition-2
broker-1
broker-2
broker-3
14Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
P
Producing to Kafka
Time
C2 C3C1
15Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Partition Leadership and Replication
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4
16Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Schema are about how teams work together
17Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
A quick change of the timestamp format…
18Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
… breaks things!
19Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
APIs between services are Contracts
In Stream Processing World – Event Schemas ARE the API
Governance in a Streaming Architecture
20Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Confluent Schema Registry
21Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafka Connect
Kafka Cluster
CRM Integration
Domain-Driven Design (DDD) for your Event Steaming Platform
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
è Independent and loosely coupled, but scalable, highly available and reliable!
22Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Requirements for Microservices
Decoupled
Flexible
Operationally Transparent
Data Aware
Elastic
23Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
24Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Cloud-Native Platforms in last five years
25Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kubernetes won the battle!
26Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Cloud-Native Deployment leveraging Kubernetes
27Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Evolution of Kafka DevOps
Shell scripts
Ansible/Chef Docker
Kubernetes
28Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafkaesque world of Kafka on Kubernetes
29
Kafka on Kubernetes –
It’s tricky L
• Translating an existing architecture to Kubernetes
• Failover handling and data balancing
• Communication between ZooKeeper, Kafka Brokers, Clients (Java,
REST, Connect, KSQL), Schema Registry, etc.
• External access from / to outside Kubernetes cluster
• Persistent storage options on prem and in the cloud
• Security configuration
• Rolling upgrades
• Etc.
30Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafka Operator for Kubernetes
The Operator pattern for Kubernetes aims to capture the key aim of a human operator
who is managing a service or set of services. Human operators who look after specific
applications and services have deep knowledge of how the system ought to behave,
how to deploy it, and how to react if there are problems.
People who run workloads on Kubernetes often like to use automation to take care of
repeatable tasks. The Operator pattern captures how you can write code to automate a
task beyond what Kubernetes itself provides.
Some Kafka Operators:
31Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Confluent’s Kubernetes Journey building “Confluent Cloud”
05/2017
Confluent Cloud
Early Access
2016
Confluent Cloud
Development
11/2017
Confluent Cloud
GA (AWS)
07/2019
Confluent Operator GA
(Confluent Platform)
2019
Confluent Cloud GA
on AWS, GCP, Azure
32Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Confluent Operator
Deployment and management automation for
Confluent Platform on Kubernetes
Including Apache Kafka, Zookeeper, Schema Registry,
Connect, Control Center, Replicator, KSQL
For organizations standardized on Kubernetes as
platform runtime
Operationalizes years of experience running Kafka on
Kubernetes on premises or the leading public clouds
Confluent Platform
Confluent Operator
Kubernetes
AWS Azure GCP
RH OpenShift Mesosphere Pivotal
On-Premises Cloud
Docker Images
Automate Deployment of Confluent Platform on Kubernetes on Any Platform at Any Scale
33Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Confluent Operator enables you to:
Automate provisioning of
Kafka pods and security
configuration in minutes
Monitor SLAs through Confluent
Control Center or Prometheus
Scale Kafka elastically &
Automate rolling updates
Built on our first hand knowledge
of running Confluent at scale
Cloud-Native Deployment of Kafka and Confluent Platform
34Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kubernetes
Cluster K8 NodeK8 NodeK8 Node
Replicator
Pod
C3 Pod SR Pod
K8 NodeOperator
Kafka
Pod
ZK Pod
Persistent Volumes
(AWS EBS, GCE Persistent Disk, Local Persistent Volume, etc.)
External
Access
Load
Balancers
Configurations
ConfigMapsKSQL Pod
REST Proxy
Pod
Confluent Operator Deployment
35Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Requirements for Microservices
Decoupled
Flexible
Operationally Transparent
Data Aware
Elastic
36Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
37Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Excursus: Kubernetes Pod
“pod == small herd of aquatic mammals, esp. of whales or dolphins”
https://geekdudes.wordpress.com/2019/07/14/kubernetes-creating-pods-on-windows-10/
38Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Sidecar Pattern
38
Components of the application, deployed in a separate container to provide
isolation and encapsulation. This pattern allows applications to be composed of
heterogeneous components.
39Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Service Mesh
A microservice pattern to move visibility, reliability, and
security primitives for service-to-service communication
into the infrastructure layer, out of the application layer.
https://www.infoq.com/articles/linkerd-v2-production-adoption/
Data Plane
Touches every packet/request in the system.
Responsible for service discovery, health
checking, routing, load balancing,
authentication/authorization, and observability.
Control Plane
Provides policy and configuration for all of the
running data planes in the mesh. Does not touch
any packets/requests in the system. The control
plane turns all of the data planes into a distributed
system.
40Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Service Mesh
Out of process architecture
• Self contained process
• Run alongside every application server
• Application sends and receives messages to
and from localhost and is unaware of the
network topology
Benefits
Compared to “fat client proxy libraries” like Finagle
(Twitter), Hystrix (Netflix), Stubby (Google):
• Works with any application language (Java,
C++, Go, PHP, Python, etc.)
• Can be deployed and upgraded quickly across
an entire infrastructure transparently
https://www.infoq.com/articles/linkerd-v2-production-adoption/
41Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
42Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Excursus: Load Balancing and Proxy at L3/L4 vs. L7 of OSI Model
https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy
https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236
L3/L4 vs. L7 is
not always
the right
abstraction level!
43Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Excursus: Load Balancing and Proxy at L3/L4 vs. L7
https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy
https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236
Example: Envoy Proxy Features
L3/L4 filter architecture
HTTP L7 filter architecture
HTTP L7 routing
gRPC support
MongoDB L7 support
DynamoDB L7 support
Kafka L7 support (Pull request merged in May 2019)
Service discovery and dynamic configuration
Health checking
Advanced load balancing
Front / edge proxy support
Observability (stats, tracing)
44Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Proxy
Proxy
Proxy
Proxy
Proxy
Service Proxy as Sidecar
45Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Proxy
I have a new IP
now.
Who
cares?
I magically
know all about
it!
Example – Service Proxy as Inbound Sidecar
46Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Proxy
I can recover from
errors without
drowning
Error?
No worries!
Lets retry every
millisecond forever
LOL. I’m
dropping 99%
of the retries.
Example - Service Proxy as Outbound Sidecar
47Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Service Proxy Features
• Metrics without instrumenting apps
• Trace flow of requests across services
• One stable URI for each service
• Service discovery
• Monitor request latency
• Routing - A/B testing, green/blue deployments
• Circuit breaking
• Protocol translation (HTTP, gRPC, Kafka Protocol, etc.)
• Mutual TLS (mTLS)
• SSL Termination
• Integrate with 3rd party tools like Prometheus, Grafana, Zipkin, etc.
• Much more…
Observability
“is by far the most important thing that a Proxy and the Service Mesh
provide in a distributed Microservice architecture!” Matt Klein
48Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Why Lyft built “envoy” Proxy
https://www.youtube.com/watch?v=55yi4MMVBi4
Matt Klein at QCon NY 2018
Developers should be able
to spend their time on
writing business applications
49Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Lyft today with “envoy” Proxy
100% (!!!) communication coverage - Everything talks through Envoy Proxies
à Make monitoring, debugging, firefighting as consistent as possible
https://www.youtube.com/watch?v=55yi4MMVBi4
Matt Klein at QCon NY 2018
Service Mesh to the rescue:
• Abstract network from
application developers
• Get operational transparency
and more flexibility
50Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Example: Advanced Load Balancing with Linkerd
https://linkerd.io/2016/03/16/beyond-round-robin-load-balancing-for-latency/
Since latency and failure are often
tied together in distributed systems
via timeouts, we can also express the
results in terms of failure.
If the caller of our system used a
timeout of 1 second, its success rate
would be approximately 95% with
round robin, 99% with least loaded,
and 99.9% with peak EWMA
(exponentially-weighted moving
average) - a significant difference.
51Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
52Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Control Plane + Proxy as Sidecar = Service Mesh
(Human Control Plane)
https://blog.envoyproxy.io/service-mesh-data-plane-vs-control-plane-2774e720f7fc
53Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Control Plane + Proxy as Sidecar = Service Mesh
(Advanced Service Mesh Control Plane)
https://blog.envoyproxy.io/service-mesh-data-plane-vs-control-plane-2774e720f7fc
“Ultimately, the goal of a control plane
is to set policy that will eventually be
enacted by the data plane.”
54Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Example for Control Plane - Istio Architecture
Pilot: Service discovery and
configuration of Envoy sidecar proxies
Mixer (Istio-Policy and Istio-Telemetry):
Enforcement of usage policies and
gathering of telemetry data
Ingress / Egress Gateway: Points for
traffic to ingress or exit from outside
the cluster
Citadel: Automation of key and
certificate management
Galley: Configuration management
services
55Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Support for the Three Pillars of Observability
56Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Requirements for Microservices
Decoupled
Flexible
Operationally Transparent
Data Aware
Elastic
57Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Service Mesh Interface (SMI)
https://www.infoq.com/presentations/service-mesh-interface
58Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
59Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Service Mesh and Event Streaming Platform
59
Request-Response Events Streams
- Low latency
- Typically sync
- Point-to-point
- “Bespoke API”
- e.g. HTTP, gRPC
- Continuous processing
- Often async
- Event driven
- General-purpose events
- e.g. Apache Kafka
Traditionally, these are two different paradigms:
Please…
No REST vs.
Streaming FUD!
Most architectures need request-response and event streams!
60Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Why not use Service Mesh and Event Streaming Platform together?
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
61Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafka Connect
Kafka Cluster
CRM Integration
Clients and Servers are Independent (including their Ops Teams)
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
Proxy Proxy Proxy
Proxy Proxy Proxy
Control
Plane
62Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Proxy
I am somehow
getting events
from Kafka
I’m using REST to
talk to a service
I’m really re-
directing
events to Kafka
Introduce
Vision #1: Using Service Mesh to Hide Kafka
Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh
https://www.youtube.com/watch?v=Fi292CqOm8A
63Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Proxy
I am using REST too!
Kafka? Never heard of
her.I’m using REST to
talk to a service
I’m proxying
REST.
And also logging
stuff to Kafka
Vision #2: Kafka as Part of Control Plane
Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh
https://www.youtube.com/watch?v=Fi292CqOm8A
64Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Vision #3: Kafka as a Service in a Mesh
Proxy
Proxy
Proxy
Proxy
Proxy
Kafka
Protocol
(TCP)
Kafka
Protocol
(TCP)
Kafka
Protocol
(TCP)
Kafka
Protocol
(TCP)
Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh
https://www.youtube.com/watch?v=Fi292CqOm8A
65Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Vision #4: Front Kafka (-as-a-Service)
P
R
O
X
Y
P
R
O
X
Y
Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh
https://www.youtube.com/watch?v=Fi292CqOm8A
66Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
(Potential) Features for Kafka + Service Mesh Implementation
Protocol conversion from HTTP /
gRPC to Kafka
• Tap feature to dump to a Kafka stream
• Protocol parsing for observability (stats,
logging, and trace linking with HTTP
RPCs)
• Shadow requests to a Kafka stream
instead of HTTP / gRPC shadow
• Integrate with Kafka Connect and its
whole ecosystem of connectors
Validation of Events
• Serialization format (JSON,
Avro, Protobuf, etc.)
• Message schema
• Headers, attributes, etc.
Security
• SSL Termination
• Mutual TLS (mTLS)
• Authorization
Proxy features
• Dynamic Routing
• Rate limiting at both the L4 connection
and L7 message level
• Filter, add compression, …
• Automatic topic name conversion (e.g.
for canary release or blue/green
deployment)
Monitoring and Tracing
• Request logs and stats
• Data lineage / audit log
• Audit log by taking request logs and
enriching them with the user info.
• Client specific metrics (Byte rate per
client id / per consumer groups,
versions of the client libraries,
consumer lag monitoring for the
entire data center)
67Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Put logic outside Kafka vs. make deployment as simple as possible
Server-side Schema Validation on Kafka Broker
Goal: Tiered Storage and Autoscaling
68Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
69Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Service Mesh Implementation
Various options for a Service Mesh implementation; examples à
Some examples with Kafka, Kubernetes*, Envoy**, Istio:
• L4: Filter on Kafka Client side (rate limiting, mTLS, etc.)
• L4: Filter on Kafka Broker side (rate limiting, mTLS, etc.)
• L7: Confluent REST Proxy on Server side
• L7: Envoy’s Kafka Protocol Filter
• L7 Filter + Routing
• L7 Observability
• Many more Kafka-specific features possible
• L7: Custom proxy implementation
• Example: https://github.com/travisjeffery/kafka-proxy
* Kubernetes is assumed as de facto standard
** Envoy has best Kafka integration (in September 2019)
70Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
L4 Example: Kafka + Istio @ Banzai Cloud
https://banzaicloud.com/blog/kafka-on-istio-performance/
71Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
L4 Example: Kafka + Istio @ Banzai Cloud
https://banzaicloud.com/blog/kafka-on-istio-performance/
72Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
L7 Example: Kafka + Confluent REST Proxy
Envoy
Proxy
I am using REST too!
Kafka? Never heard of
her.I’m using REST to
talk to a service
I’m proxying
REST.
And also logging
stuff to Kafka
Confluent
REST Proxy
I support only TCP!
HTTP
HTTP
73Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafka Support in Envoy (Pull Request Merged in May 2019)
https://github.com/envoyproxy/envoy/issues/2852
https://github.com/envoyproxy/envoy/pull/4950
74Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kafka Support in Istio? (August 2019)
• Before PR #4950, Envoy treats Kafka as TCP, so that Istio-TCP-rules will apply (already).
• With PR #4950, Envoy can do some more fancy things and get stuff like number-of-messages in
telemetry from Kafka semantics.
• Now the 2nd part here is Istio, which needs a new vocabulary to be able to configure Envoy.
Think of ‘VirtualService’ and ‘DestinationRule’ for Kafka (or messaging in a more global sense).
• TLDR: L4 works in Istio; L7 needs some new PRs in Istio project, too!
https://istio.io/docs/tasks/traffic-management/tcp-traffic-shifting/
75Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
L7 Example: Kafka + Envoy Kafka Protocol Filter
Envoy
Proxy
I am using REST too!
Kafka? Never heard of
her.I’m using REST to
talk to a service
I’m proxying
REST.
And also logging
stuff to Kafka
HTTP
TCP
(Kafka Protocol)
76Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Requirements for Microservices
Decoupled
Flexible
Operationally Transparent
Data Aware
Elastic
77Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Event Streaming Platform and Service Mesh
A Match Made In Heaven
+ =
78Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Key Takeaways
• Apache Kafka decouples services, including event streams and request-response
• Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem
• Service Mesh helps with security and observability at ecosystem / organization scale
• Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses
+
79Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

More Related Content

What's hot

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...confluent
 
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요Jo Hoon
 
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...HostedbyConfluent
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignMichael Noll
 
카프카, 산전수전 노하우
카프카, 산전수전 노하우카프카, 산전수전 노하우
카프카, 산전수전 노하우if kakao
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache KafkaJeff Holoman
 
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?confluent
 
Application modernization patterns with apache kafka, debezium, and kubernete...
Application modernization patterns with apache kafka, debezium, and kubernete...Application modernization patterns with apache kafka, debezium, and kubernete...
Application modernization patterns with apache kafka, debezium, and kubernete...Bilgin Ibryam
 
Spring Boot+Kafka: the New Enterprise Platform
Spring Boot+Kafka: the New Enterprise PlatformSpring Boot+Kafka: the New Enterprise Platform
Spring Boot+Kafka: the New Enterprise PlatformVMware Tanzu
 
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaReal-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaKai Wähner
 
Kafka and Avro with Confluent Schema Registry
Kafka and Avro with Confluent Schema RegistryKafka and Avro with Confluent Schema Registry
Kafka and Avro with Confluent Schema RegistryJean-Paul Azar
 
Apache Flink and what it is used for
Apache Flink and what it is used forApache Flink and what it is used for
Apache Flink and what it is used forAljoscha Krettek
 
Best Practices for Middleware and Integration Architecture Modernization with...
Best Practices for Middleware and Integration Architecture Modernization with...Best Practices for Middleware and Integration Architecture Modernization with...
Best Practices for Middleware and Integration Architecture Modernization with...Claus Ibsen
 
Exactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka StreamsExactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka StreamsGuozhang Wang
 
Why kubernetes matters
Why kubernetes mattersWhy kubernetes matters
Why kubernetes mattersPlatform9
 
Micro services Architecture
Micro services ArchitectureMicro services Architecture
Micro services ArchitectureAraf Karsh Hamid
 
Service Mesh - Observability
Service Mesh - ObservabilityService Mesh - Observability
Service Mesh - ObservabilityAraf Karsh Hamid
 
MSA 전략 2: 마이크로서비스, 어떻게 구현할 것인가?
MSA 전략 2: 마이크로서비스, 어떻게 구현할 것인가?MSA 전략 2: 마이크로서비스, 어떻게 구현할 것인가?
MSA 전략 2: 마이크로서비스, 어떻게 구현할 것인가?VMware Tanzu Korea
 
Kubernetes Introduction
Kubernetes IntroductionKubernetes Introduction
Kubernetes IntroductionPeng Xiao
 

What's hot (20)

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
 
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
 
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - Verisign
 
카프카, 산전수전 노하우
카프카, 산전수전 노하우카프카, 산전수전 노하우
카프카, 산전수전 노하우
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?
 
Application modernization patterns with apache kafka, debezium, and kubernete...
Application modernization patterns with apache kafka, debezium, and kubernete...Application modernization patterns with apache kafka, debezium, and kubernete...
Application modernization patterns with apache kafka, debezium, and kubernete...
 
Spring Boot+Kafka: the New Enterprise Platform
Spring Boot+Kafka: the New Enterprise PlatformSpring Boot+Kafka: the New Enterprise Platform
Spring Boot+Kafka: the New Enterprise Platform
 
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaReal-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
 
Kafka and Avro with Confluent Schema Registry
Kafka and Avro with Confluent Schema RegistryKafka and Avro with Confluent Schema Registry
Kafka and Avro with Confluent Schema Registry
 
Apache Flink and what it is used for
Apache Flink and what it is used forApache Flink and what it is used for
Apache Flink and what it is used for
 
Best Practices for Middleware and Integration Architecture Modernization with...
Best Practices for Middleware and Integration Architecture Modernization with...Best Practices for Middleware and Integration Architecture Modernization with...
Best Practices for Middleware and Integration Architecture Modernization with...
 
Kafka 101
Kafka 101Kafka 101
Kafka 101
 
Exactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka StreamsExactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka Streams
 
Why kubernetes matters
Why kubernetes mattersWhy kubernetes matters
Why kubernetes matters
 
Micro services Architecture
Micro services ArchitectureMicro services Architecture
Micro services Architecture
 
Service Mesh - Observability
Service Mesh - ObservabilityService Mesh - Observability
Service Mesh - Observability
 
MSA 전략 2: 마이크로서비스, 어떻게 구현할 것인가?
MSA 전략 2: 마이크로서비스, 어떻게 구현할 것인가?MSA 전략 2: 마이크로서비스, 어떻게 구현할 것인가?
MSA 전략 2: 마이크로서비스, 어떻게 구현할 것인가?
 
Kubernetes Introduction
Kubernetes IntroductionKubernetes Introduction
Kubernetes Introduction
 

Similar to Apache Kafka, Kubernetes and Service Mesh

Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Sou...
Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Sou...Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Sou...
Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Sou...Kai Wähner
 
How to Leverage the Apache Kafka Ecosystem to Productionize Machine Learning ...
How to Leverage the Apache Kafka Ecosystem to Productionize Machine Learning ...How to Leverage the Apache Kafka Ecosystem to Productionize Machine Learning ...
How to Leverage the Apache Kafka Ecosystem to Productionize Machine Learning ...Codemotion
 
Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...
Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...
Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...Kai Wähner
 
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...Kai Wähner
 
Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...
Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...
Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...confluent
 
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...Kai Wähner
 
Unleashing Apache Kafka and TensorFlow in Hybrid Cloud Architectures
Unleashing Apache Kafka and TensorFlow in Hybrid Cloud ArchitecturesUnleashing Apache Kafka and TensorFlow in Hybrid Cloud Architectures
Unleashing Apache Kafka and TensorFlow in Hybrid Cloud ArchitecturesKai Wähner
 
Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...
Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...
Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...Red Hat Developers
 
Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (Apa...
Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (Apa...Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (Apa...
Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (Apa...Kai Wähner
 
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Kai Wähner
 
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Kai Wähner
 
Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL
Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQLRethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL
Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQLKai Wähner
 
Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent
Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, ConfluentCan Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent
Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, ConfluentHostedbyConfluent
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...HostedbyConfluent
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Kai Wähner
 
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...HostedbyConfluent
 
Top 5 Event Streaming Use Cases for 2021 with Apache Kafka
Top 5 Event Streaming Use Cases for 2021 with Apache KafkaTop 5 Event Streaming Use Cases for 2021 with Apache Kafka
Top 5 Event Streaming Use Cases for 2021 with Apache KafkaKai Wähner
 
The Top 5 Event Streaming Use Cases & Architectures in 2021
The Top 5 Event Streaming Use Cases & Architectures in 2021The Top 5 Event Streaming Use Cases & Architectures in 2021
The Top 5 Event Streaming Use Cases & Architectures in 2021confluent
 
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...confluent
 
Supply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaSupply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaKai Wähner
 

Similar to Apache Kafka, Kubernetes and Service Mesh (20)

Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Sou...
Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Sou...Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Sou...
Deep Learning at Extreme Scale (in the Cloud) 
with the Apache Kafka Open Sou...
 
How to Leverage the Apache Kafka Ecosystem to Productionize Machine Learning ...
How to Leverage the Apache Kafka Ecosystem to Productionize Machine Learning ...How to Leverage the Apache Kafka Ecosystem to Productionize Machine Learning ...
How to Leverage the Apache Kafka Ecosystem to Productionize Machine Learning ...
 
Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...
Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...
Deep Learning Streaming Platform with Kafka Streams, TensorFlow, DeepLearning...
 
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
 
Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...
Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...
Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...
 
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
 
Unleashing Apache Kafka and TensorFlow in Hybrid Cloud Architectures
Unleashing Apache Kafka and TensorFlow in Hybrid Cloud ArchitecturesUnleashing Apache Kafka and TensorFlow in Hybrid Cloud Architectures
Unleashing Apache Kafka and TensorFlow in Hybrid Cloud Architectures
 
Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...
Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...
Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...
 
Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (Apa...
Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (Apa...Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (Apa...
Apache Kafka Open Source Ecosystem for Machine Learning at Extreme Scale (Apa...
 
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
 
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
 
Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL
Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQLRethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL
Rethinking Stream Processing with Apache Kafka, Kafka Streams and KSQL
 
Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent
Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, ConfluentCan Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent
Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
 
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
 
Top 5 Event Streaming Use Cases for 2021 with Apache Kafka
Top 5 Event Streaming Use Cases for 2021 with Apache KafkaTop 5 Event Streaming Use Cases for 2021 with Apache Kafka
Top 5 Event Streaming Use Cases for 2021 with Apache Kafka
 
The Top 5 Event Streaming Use Cases & Architectures in 2021
The Top 5 Event Streaming Use Cases & Architectures in 2021The Top 5 Event Streaming Use Cases & Architectures in 2021
The Top 5 Event Streaming Use Cases & Architectures in 2021
 
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
 
Supply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaSupply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache Kafka
 

More from Kai Wähner

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
 
When NOT to use Apache Kafka?
When NOT to use Apache Kafka?When NOT to use Apache Kafka?
When NOT to use Apache Kafka?Kai Wähner
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKai Wähner
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaKai Wähner
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareKai Wähner
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Kai Wähner
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
 
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Kai Wähner
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail IndustryKai Wähner
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKai Wähner
 
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Kai Wähner
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingKai Wähner
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKai Wähner
 
The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Kai Wähner
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Kai Wähner
 

More from Kai Wähner (20)

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
 
When NOT to use Apache Kafka?
When NOT to use Apache Kafka?When NOT to use Apache Kafka?
When NOT to use Apache Kafka?
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
 
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid Cloud
 
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and Manufacturing
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology Comparison
 
The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
 

Recently uploaded

Professional Resume Template for Software Developers
Professional Resume Template for Software DevelopersProfessional Resume Template for Software Developers
Professional Resume Template for Software DevelopersVinodh Ram
 
Optimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTVOptimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTVshikhaohhpro
 
Advancing Engineering with AI through the Next Generation of Strategic Projec...
Advancing Engineering with AI through the Next Generation of Strategic Projec...Advancing Engineering with AI through the Next Generation of Strategic Projec...
Advancing Engineering with AI through the Next Generation of Strategic Projec...OnePlan Solutions
 
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...OnePlan Solutions
 
Software Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsSoftware Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsArshad QA
 
why an Opensea Clone Script might be your perfect match.pdf
why an Opensea Clone Script might be your perfect match.pdfwhy an Opensea Clone Script might be your perfect match.pdf
why an Opensea Clone Script might be your perfect match.pdfjoe51371421
 
HR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.comHR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.comFatema Valibhai
 
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...gurkirankumar98700
 
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfThe Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfkalichargn70th171
 
5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdfWave PLM
 
Hand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptxHand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptxbodapatigopi8531
 
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsUnveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsAlberto González Trastoy
 
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AISyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AIABDERRAOUF MEHENNI
 
Active Directory Penetration Testing, cionsystems.com.pdf
Active Directory Penetration Testing, cionsystems.com.pdfActive Directory Penetration Testing, cionsystems.com.pdf
Active Directory Penetration Testing, cionsystems.com.pdfCionsystems
 
Unlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language ModelsUnlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language Modelsaagamshah0812
 
How To Use Server-Side Rendering with Nuxt.js
How To Use Server-Side Rendering with Nuxt.jsHow To Use Server-Side Rendering with Nuxt.js
How To Use Server-Side Rendering with Nuxt.jsAndolasoft Inc
 

Recently uploaded (20)

Professional Resume Template for Software Developers
Professional Resume Template for Software DevelopersProfessional Resume Template for Software Developers
Professional Resume Template for Software Developers
 
Optimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTVOptimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTV
 
Advancing Engineering with AI through the Next Generation of Strategic Projec...
Advancing Engineering with AI through the Next Generation of Strategic Projec...Advancing Engineering with AI through the Next Generation of Strategic Projec...
Advancing Engineering with AI through the Next Generation of Strategic Projec...
 
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
 
Software Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsSoftware Quality Assurance Interview Questions
Software Quality Assurance Interview Questions
 
why an Opensea Clone Script might be your perfect match.pdf
why an Opensea Clone Script might be your perfect match.pdfwhy an Opensea Clone Script might be your perfect match.pdf
why an Opensea Clone Script might be your perfect match.pdf
 
HR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.comHR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.com
 
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
 
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfThe Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
 
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdfMicrosoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
 
5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf
 
Hand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptxHand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptx
 
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time ApplicationsUnveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
 
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
 
Exploring iOS App Development: Simplifying the Process
Exploring iOS App Development: Simplifying the ProcessExploring iOS App Development: Simplifying the Process
Exploring iOS App Development: Simplifying the Process
 
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AISyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
 
Active Directory Penetration Testing, cionsystems.com.pdf
Active Directory Penetration Testing, cionsystems.com.pdfActive Directory Penetration Testing, cionsystems.com.pdf
Active Directory Penetration Testing, cionsystems.com.pdf
 
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS LiveVip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
 
Unlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language ModelsUnlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language Models
 
How To Use Server-Side Rendering with Nuxt.js
How To Use Server-Side Rendering with Nuxt.jsHow To Use Server-Side Rendering with Nuxt.js
How To Use Server-Side Rendering with Nuxt.js
 

Apache Kafka, Kubernetes and Service Mesh

  • 1. 1Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Event Streaming Platform and Service Mesh Cloud-Native Apache Kafka with Kubernetes, Envoy and Istio Kai Waehner Technology Evangelist contact@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
  • 2. 2Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Key Takeaways • Apache Kafka decouples services, including event streams and request-response • Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem • Service Mesh helps with security and observability at ecosystem / organization scale • Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses +
  • 3. 3Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Agenda • Motivation, Challenges, Requirements of Microservices • Apache Kafka - The Event Streaming Platform for Microservices • Kubernetes for Cloud-Native Microservices • Service Mesh • Service Proxy (aka Data Plane) • Control Plane • Kafka and Service Mesh • Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
  • 4. 4Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Agenda • Motivation, Challenges, Requirements of Microservices • Apache Kafka - The Event Streaming Platform for Microservices • Kubernetes for Cloud-Native Microservices • Service Mesh • Service Proxy (aka Data Plane) • Control Plane • Kafka and Service Mesh • Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
  • 5. 5Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Business Digitalization Trends are Driving the Need to Process Events at a whole new Scale, Speed and Efficiency Mobile Cloud Microservices Internet of Things Machine Learning The world has changed!
  • 6. 6Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Microservices to the rescue? • Significant Operations Overhead • Substantial DevOps Skills Required • Implicit Interfaces • Duplication Of Effort http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html • Distributed System Complexity • Asynchronicity Is Difficult • Testability Challenges
  • 7. 7Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Key Requirements for Microservices Decoupled Flexible Operationally Transparent Data Aware Elastic
  • 8. 8Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Agenda • Motivation, Challenges, Requirements of Microservices • Apache Kafka - The Event Streaming Platform for Microservices • Kubernetes for Cloud-Native Microservices • Service Mesh • Service Proxy (aka Data Plane) • Control Plane • Kafka and Service Mesh • Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
  • 9. 9Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner The Log ConnectorsConnectors Producer Consumer Streaming Engine Apache Kafka—The Rise of an Event Streaming Platform
  • 10. 10Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Apache Kafka at Scale at Tech Giants > 4.5 trillion messages / day > 6 Petabytes / day “You name it” * Kafka Is not just used by tech giants ** Kafka is not just used for big data
  • 11. 11Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Confluent - Business Value per Use Case Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $ Example Case Studies (of many)
  • 12. 12Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Apache Kafka - A Distributed Commit Log Writers Kafka cluster Readers
  • 13. 13Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafka Topics my-topic my-topic-partition-0 my-topic-partition-1 my-topic-partition-2 broker-1 broker-2 broker-3
  • 14. 14Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner P Producing to Kafka Time C2 C3C1
  • 15. 15Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Partition Leadership and Replication Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
  • 16. 16Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Schema are about how teams work together
  • 17. 17Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner A quick change of the timestamp format…
  • 18. 18Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner … breaks things!
  • 19. 19Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner APIs between services are Contracts In Stream Processing World – Event Schemas ARE the API Governance in a Streaming Architecture
  • 20. 20Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Confluent Schema Registry
  • 21. 21Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafka Connect Kafka Cluster CRM Integration Domain-Driven Design (DDD) for your Event Steaming Platform Legacy Integration Custom Application ESB Connector Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform CRM Domain Legacy Domain Payment Domain è Independent and loosely coupled, but scalable, highly available and reliable!
  • 22. 22Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Key Requirements for Microservices Decoupled Flexible Operationally Transparent Data Aware Elastic
  • 23. 23Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Agenda • Motivation, Challenges, Requirements of Microservices • Apache Kafka - The Event Streaming Platform for Microservices • Kubernetes for Cloud-Native Microservices • Service Mesh • Service Proxy (aka Data Plane) • Control Plane • Kafka and Service Mesh • Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
  • 24. 24Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Cloud-Native Platforms in last five years
  • 25. 25Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kubernetes won the battle!
  • 26. 26Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Cloud-Native Deployment leveraging Kubernetes
  • 27. 27Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Evolution of Kafka DevOps Shell scripts Ansible/Chef Docker Kubernetes
  • 28. 28Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafkaesque world of Kafka on Kubernetes
  • 29. 29 Kafka on Kubernetes – It’s tricky L • Translating an existing architecture to Kubernetes • Failover handling and data balancing • Communication between ZooKeeper, Kafka Brokers, Clients (Java, REST, Connect, KSQL), Schema Registry, etc. • External access from / to outside Kubernetes cluster • Persistent storage options on prem and in the cloud • Security configuration • Rolling upgrades • Etc.
  • 30. 30Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafka Operator for Kubernetes The Operator pattern for Kubernetes aims to capture the key aim of a human operator who is managing a service or set of services. Human operators who look after specific applications and services have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems. People who run workloads on Kubernetes often like to use automation to take care of repeatable tasks. The Operator pattern captures how you can write code to automate a task beyond what Kubernetes itself provides. Some Kafka Operators:
  • 31. 31Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Confluent’s Kubernetes Journey building “Confluent Cloud” 05/2017 Confluent Cloud Early Access 2016 Confluent Cloud Development 11/2017 Confluent Cloud GA (AWS) 07/2019 Confluent Operator GA (Confluent Platform) 2019 Confluent Cloud GA on AWS, GCP, Azure
  • 32. 32Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Confluent Operator Deployment and management automation for Confluent Platform on Kubernetes Including Apache Kafka, Zookeeper, Schema Registry, Connect, Control Center, Replicator, KSQL For organizations standardized on Kubernetes as platform runtime Operationalizes years of experience running Kafka on Kubernetes on premises or the leading public clouds Confluent Platform Confluent Operator Kubernetes AWS Azure GCP RH OpenShift Mesosphere Pivotal On-Premises Cloud Docker Images Automate Deployment of Confluent Platform on Kubernetes on Any Platform at Any Scale
  • 33. 33Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Confluent Operator enables you to: Automate provisioning of Kafka pods and security configuration in minutes Monitor SLAs through Confluent Control Center or Prometheus Scale Kafka elastically & Automate rolling updates Built on our first hand knowledge of running Confluent at scale Cloud-Native Deployment of Kafka and Confluent Platform
  • 34. 34Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kubernetes Cluster K8 NodeK8 NodeK8 Node Replicator Pod C3 Pod SR Pod K8 NodeOperator Kafka Pod ZK Pod Persistent Volumes (AWS EBS, GCE Persistent Disk, Local Persistent Volume, etc.) External Access Load Balancers Configurations ConfigMapsKSQL Pod REST Proxy Pod Confluent Operator Deployment
  • 35. 35Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Key Requirements for Microservices Decoupled Flexible Operationally Transparent Data Aware Elastic
  • 36. 36Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Agenda • Motivation, Challenges, Requirements of Microservices • Apache Kafka - The Event Streaming Platform for Microservices • Kubernetes for Cloud-Native Microservices • Service Mesh • Service Proxy (aka Data Plane) • Control Plane • Kafka and Service Mesh • Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
  • 37. 37Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Excursus: Kubernetes Pod “pod == small herd of aquatic mammals, esp. of whales or dolphins” https://geekdudes.wordpress.com/2019/07/14/kubernetes-creating-pods-on-windows-10/
  • 38. 38Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Sidecar Pattern 38 Components of the application, deployed in a separate container to provide isolation and encapsulation. This pattern allows applications to be composed of heterogeneous components.
  • 39. 39Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Service Mesh A microservice pattern to move visibility, reliability, and security primitives for service-to-service communication into the infrastructure layer, out of the application layer. https://www.infoq.com/articles/linkerd-v2-production-adoption/ Data Plane Touches every packet/request in the system. Responsible for service discovery, health checking, routing, load balancing, authentication/authorization, and observability. Control Plane Provides policy and configuration for all of the running data planes in the mesh. Does not touch any packets/requests in the system. The control plane turns all of the data planes into a distributed system.
  • 40. 40Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Service Mesh Out of process architecture • Self contained process • Run alongside every application server • Application sends and receives messages to and from localhost and is unaware of the network topology Benefits Compared to “fat client proxy libraries” like Finagle (Twitter), Hystrix (Netflix), Stubby (Google): • Works with any application language (Java, C++, Go, PHP, Python, etc.) • Can be deployed and upgraded quickly across an entire infrastructure transparently https://www.infoq.com/articles/linkerd-v2-production-adoption/
  • 41. 41Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Agenda • Motivation, Challenges, Requirements of Microservices • Apache Kafka - The Event Streaming Platform for Microservices • Kubernetes for Cloud-Native Microservices • Service Mesh • Service Proxy (aka Data Plane) • Control Plane • Kafka and Service Mesh • Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
  • 42. 42Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Excursus: Load Balancing and Proxy at L3/L4 vs. L7 of OSI Model https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236 L3/L4 vs. L7 is not always the right abstraction level!
  • 43. 43Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Excursus: Load Balancing and Proxy at L3/L4 vs. L7 https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236 Example: Envoy Proxy Features L3/L4 filter architecture HTTP L7 filter architecture HTTP L7 routing gRPC support MongoDB L7 support DynamoDB L7 support Kafka L7 support (Pull request merged in May 2019) Service discovery and dynamic configuration Health checking Advanced load balancing Front / edge proxy support Observability (stats, tracing)
  • 44. 44Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Proxy Proxy Proxy Proxy Proxy Service Proxy as Sidecar
  • 45. 45Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Proxy I have a new IP now. Who cares? I magically know all about it! Example – Service Proxy as Inbound Sidecar
  • 46. 46Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Proxy I can recover from errors without drowning Error? No worries! Lets retry every millisecond forever LOL. I’m dropping 99% of the retries. Example - Service Proxy as Outbound Sidecar
  • 47. 47Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Service Proxy Features • Metrics without instrumenting apps • Trace flow of requests across services • One stable URI for each service • Service discovery • Monitor request latency • Routing - A/B testing, green/blue deployments • Circuit breaking • Protocol translation (HTTP, gRPC, Kafka Protocol, etc.) • Mutual TLS (mTLS) • SSL Termination • Integrate with 3rd party tools like Prometheus, Grafana, Zipkin, etc. • Much more… Observability “is by far the most important thing that a Proxy and the Service Mesh provide in a distributed Microservice architecture!” Matt Klein
  • 48. 48Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Why Lyft built “envoy” Proxy https://www.youtube.com/watch?v=55yi4MMVBi4 Matt Klein at QCon NY 2018 Developers should be able to spend their time on writing business applications
  • 49. 49Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Lyft today with “envoy” Proxy 100% (!!!) communication coverage - Everything talks through Envoy Proxies à Make monitoring, debugging, firefighting as consistent as possible https://www.youtube.com/watch?v=55yi4MMVBi4 Matt Klein at QCon NY 2018 Service Mesh to the rescue: • Abstract network from application developers • Get operational transparency and more flexibility
  • 50. 50Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Example: Advanced Load Balancing with Linkerd https://linkerd.io/2016/03/16/beyond-round-robin-load-balancing-for-latency/ Since latency and failure are often tied together in distributed systems via timeouts, we can also express the results in terms of failure. If the caller of our system used a timeout of 1 second, its success rate would be approximately 95% with round robin, 99% with least loaded, and 99.9% with peak EWMA (exponentially-weighted moving average) - a significant difference.
  • 51. 51Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Agenda • Motivation, Challenges, Requirements of Microservices • Apache Kafka - The Event Streaming Platform for Microservices • Kubernetes for Cloud-Native Microservices • Service Mesh • Service Proxy (aka Data Plane) • Control Plane • Kafka and Service Mesh • Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
  • 52. 52Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Control Plane + Proxy as Sidecar = Service Mesh (Human Control Plane) https://blog.envoyproxy.io/service-mesh-data-plane-vs-control-plane-2774e720f7fc
  • 53. 53Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Control Plane + Proxy as Sidecar = Service Mesh (Advanced Service Mesh Control Plane) https://blog.envoyproxy.io/service-mesh-data-plane-vs-control-plane-2774e720f7fc “Ultimately, the goal of a control plane is to set policy that will eventually be enacted by the data plane.”
  • 54. 54Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Example for Control Plane - Istio Architecture Pilot: Service discovery and configuration of Envoy sidecar proxies Mixer (Istio-Policy and Istio-Telemetry): Enforcement of usage policies and gathering of telemetry data Ingress / Egress Gateway: Points for traffic to ingress or exit from outside the cluster Citadel: Automation of key and certificate management Galley: Configuration management services
  • 55. 55Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Support for the Three Pillars of Observability
  • 56. 56Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Key Requirements for Microservices Decoupled Flexible Operationally Transparent Data Aware Elastic
  • 57. 57Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Service Mesh Interface (SMI) https://www.infoq.com/presentations/service-mesh-interface
  • 58. 58Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Agenda • Motivation, Challenges, Requirements of Microservices • Apache Kafka - The Event Streaming Platform for Microservices • Kubernetes for Cloud-Native Microservices • Service Mesh • Service Proxy (aka Data Plane) • Control Plane • Kafka and Service Mesh • Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
  • 59. 59Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Service Mesh and Event Streaming Platform 59 Request-Response Events Streams - Low latency - Typically sync - Point-to-point - “Bespoke API” - e.g. HTTP, gRPC - Continuous processing - Often async - Event driven - General-purpose events - e.g. Apache Kafka Traditionally, these are two different paradigms: Please… No REST vs. Streaming FUD! Most architectures need request-response and event streams!
  • 60. 60Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Why not use Service Mesh and Event Streaming Platform together? Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy Proxy
  • 61. 61Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafka Connect Kafka Cluster CRM Integration Clients and Servers are Independent (including their Ops Teams) Legacy Integration Custom Application ESB Connector Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform CRM Domain Legacy Domain Payment Domain Proxy Proxy Proxy Proxy Proxy Proxy Control Plane
  • 62. 62Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Proxy I am somehow getting events from Kafka I’m using REST to talk to a service I’m really re- directing events to Kafka Introduce Vision #1: Using Service Mesh to Hide Kafka Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh https://www.youtube.com/watch?v=Fi292CqOm8A
  • 63. 63Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Proxy I am using REST too! Kafka? Never heard of her.I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka Vision #2: Kafka as Part of Control Plane Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh https://www.youtube.com/watch?v=Fi292CqOm8A
  • 64. 64Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Vision #3: Kafka as a Service in a Mesh Proxy Proxy Proxy Proxy Proxy Kafka Protocol (TCP) Kafka Protocol (TCP) Kafka Protocol (TCP) Kafka Protocol (TCP) Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh https://www.youtube.com/watch?v=Fi292CqOm8A
  • 65. 65Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Vision #4: Front Kafka (-as-a-Service) P R O X Y P R O X Y Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh https://www.youtube.com/watch?v=Fi292CqOm8A
  • 66. 66Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner (Potential) Features for Kafka + Service Mesh Implementation Protocol conversion from HTTP / gRPC to Kafka • Tap feature to dump to a Kafka stream • Protocol parsing for observability (stats, logging, and trace linking with HTTP RPCs) • Shadow requests to a Kafka stream instead of HTTP / gRPC shadow • Integrate with Kafka Connect and its whole ecosystem of connectors Validation of Events • Serialization format (JSON, Avro, Protobuf, etc.) • Message schema • Headers, attributes, etc. Security • SSL Termination • Mutual TLS (mTLS) • Authorization Proxy features • Dynamic Routing • Rate limiting at both the L4 connection and L7 message level • Filter, add compression, … • Automatic topic name conversion (e.g. for canary release or blue/green deployment) Monitoring and Tracing • Request logs and stats • Data lineage / audit log • Audit log by taking request logs and enriching them with the user info. • Client specific metrics (Byte rate per client id / per consumer groups, versions of the client libraries, consumer lag monitoring for the entire data center)
  • 67. 67Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Put logic outside Kafka vs. make deployment as simple as possible Server-side Schema Validation on Kafka Broker Goal: Tiered Storage and Autoscaling
  • 68. 68Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Agenda • Motivation, Challenges, Requirements of Microservices • Apache Kafka - The Event Streaming Platform for Microservices • Kubernetes for Cloud-Native Microservices • Service Mesh • Service Proxy (aka Data Plane) • Control Plane • Kafka and Service Mesh • Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio
  • 69. 69Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Service Mesh Implementation Various options for a Service Mesh implementation; examples à Some examples with Kafka, Kubernetes*, Envoy**, Istio: • L4: Filter on Kafka Client side (rate limiting, mTLS, etc.) • L4: Filter on Kafka Broker side (rate limiting, mTLS, etc.) • L7: Confluent REST Proxy on Server side • L7: Envoy’s Kafka Protocol Filter • L7 Filter + Routing • L7 Observability • Many more Kafka-specific features possible • L7: Custom proxy implementation • Example: https://github.com/travisjeffery/kafka-proxy * Kubernetes is assumed as de facto standard ** Envoy has best Kafka integration (in September 2019)
  • 70. 70Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner L4 Example: Kafka + Istio @ Banzai Cloud https://banzaicloud.com/blog/kafka-on-istio-performance/
  • 71. 71Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner L4 Example: Kafka + Istio @ Banzai Cloud https://banzaicloud.com/blog/kafka-on-istio-performance/
  • 72. 72Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner L7 Example: Kafka + Confluent REST Proxy Envoy Proxy I am using REST too! Kafka? Never heard of her.I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka Confluent REST Proxy I support only TCP! HTTP HTTP
  • 73. 73Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafka Support in Envoy (Pull Request Merged in May 2019) https://github.com/envoyproxy/envoy/issues/2852 https://github.com/envoyproxy/envoy/pull/4950
  • 74. 74Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kafka Support in Istio? (August 2019) • Before PR #4950, Envoy treats Kafka as TCP, so that Istio-TCP-rules will apply (already). • With PR #4950, Envoy can do some more fancy things and get stuff like number-of-messages in telemetry from Kafka semantics. • Now the 2nd part here is Istio, which needs a new vocabulary to be able to configure Envoy. Think of ‘VirtualService’ and ‘DestinationRule’ for Kafka (or messaging in a more global sense). • TLDR: L4 works in Istio; L7 needs some new PRs in Istio project, too! https://istio.io/docs/tasks/traffic-management/tcp-traffic-shifting/
  • 75. 75Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner L7 Example: Kafka + Envoy Kafka Protocol Filter Envoy Proxy I am using REST too! Kafka? Never heard of her.I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka HTTP TCP (Kafka Protocol)
  • 76. 76Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Key Requirements for Microservices Decoupled Flexible Operationally Transparent Data Aware Elastic
  • 77. 77Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Event Streaming Platform and Service Mesh A Match Made In Heaven + =
  • 78. 78Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Key Takeaways • Apache Kafka decouples services, including event streams and request-response • Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem • Service Mesh helps with security and observability at ecosystem / organization scale • Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses +
  • 79. 79Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner Kai Waehner Technology Evangelist contact@kai-waehner.de @KaiWaehner www.kai-waehner.de www.confluent.io LinkedIn Questions? Feedback? Let’s connect!