The document discusses the challenges of microservices communication, particularly the risks of tight coupling with REST APIs, and presents Kafka as a solution for decoupled, flexible, and data-aware messaging. It explores the integration of service meshes with Kafka to enhance operational transparency and support various communication protocols. Different visions for utilizing service meshes and Kafka are proposed, highlighting the benefits of combining both technologies for resilient and scalable microservices architectures.

1. 1
Kafka and the Service Mesh
Gwen Shapira, Principal Data Architect
@gwenshap

2. You Have Microservices
2

3. They need to communicate
3

4. 4
Orders
Returns
?
Invento
ry
Fulfill
Order
Validat
e Order
I know! I’ll use REST APIs

5. Synchronous request-response communication
Leads to
Tight point-to-point coupling
5

6. 6
Making Changes is Risky
Change

7. 7
Adding Services Requires Explicit Calls
Mistak
e
Handle
r

8. All these lines of communication require:
Logging, monitoring, error handling and more
8

9. The most difficult thing about microservices
is data.
Especially joining data.
9

10. We need a way to communicate that is:
Decoupled
Flexible
Operationally Transparent
Data Aware
10

11. Kafka?
11

15. 15
From This:

16. 16
To This:

17. 17
And This: New
Servi
ce

19. Kafka:
- Immutable, ordered, stream (log) of events
- Highly Available
- Scalable
- Load balancing consumers
- HA consumers
- (Potentially) infinite storage

20. Kafka + Schema Registry
20

24. APIs between services are Contracts
In Stream Processing World – Event Schemas ARE the API
…except they stick around for a lot longer

27. We need a way to communicate that is:
Decoupled
Flexible
Operationally Transparent
Data Aware
27

28. Service Mesh?
28

29. It is a proxy.

30. Sidecar:
Components of the application, deployed in a separate
container to provide isolation and encapsulation.
This pattern allows applications to be composed of
heterogeneous components.

31. 31

32. Proxy as sidecar:
Proxy
Proxy
Proxy
Proxy
Proxy

33. Proxy
I have a new
IP now.
YOLO!
Who
cares?
I magically
know all
about it!

34. Proxy
I can recover
from errors
without drowning
Error?
No worries!
Lets retry every
millisecond forever
LOL. I’m
dropping 99%
of the retries.

35. Cool Things proxies do!
- One stable uri for each service
- Service discovery
- Log requests
- Monitor request latency
- Routing - A/B testing, green/blue deployments
- Circuit breaking
- Protocol translation
- SSL Termination

36. They don’t do it alone

37. We need a way to communicate that is:
Decoupled
Flexible
Operationally Transparent
Data Aware
37

38. How do we choose?
38

40. 40
Two different paradigms
Request-Response Events Streams
- Very Low latency
- Often sync
- Point-to-point
- “Bespoke API”
- Continuous processing
- Often async
- Event driven
- General-purpose events

41. Vision #1:
Using Service Mesh to hide Kafka
Introduce

42. Proxy
I am somehow
getting events
from Kafka
I’m using REST to
talk to a service
I’m really re-
directing
events to
Kafka

43. REST Proxy does bits of that.
HTTP on one-side, Kafka on the other.
We are adding more protocols:
- JMS
- MQTT
- gRPC

44. Vision #2:
Kafka as part of control plane

45. Proxy
I am using REST too!
Kafka? Never heard of
her.I’m using REST to
talk to a service
I’m proxying
REST.
And also
logging stuff to
Kafka

46. Vision #3:
Kafka as a Service in a Mesh

47. 47
Like This:
Proxy
Proxy
Proxy
Proxy
Proxy

51. Vision #4:
Front Kafka

52. 52
Like This:
P
R
O
X
Y
P
R
O
X
Y

53. Lets Sum Things Up
53

54. - REST API point-to-point communication
hits the wall
- Service Mesh adds discovery and operationalibility
- Kafka adds data, compatibility, immutable storage
- You can have both. In many different tastes.
- Coming soon to a proxy near you

55. Resources and Next Steps
https://github.com/confluentinc/cp-demo
https://www.confluent.io/download/
https://slackpass.io/confluentcommunity
https://www.confluent.io/blog
https://www.confluent.io/confluent-operator/
https://github.com/envoyproxy/envoy/issues/2852