This document discusses integrating gRPC services with Kafka to enable end-to-end streaming between gRPC services. It describes how the open source Zilla library can translate between gRPC and Kafka protocols to allow gRPC requests and responses to be sent over Kafka topics. This integration enables features like load balancing, edge caching, and elastic scalability for gRPC services. Examples of using Zilla to implement request-response streaming and server streaming patterns between gRPC and Kafka are provided.

1. End-to-end Streaming
Between gRPC Services via Kafka
John Fallows

2. 2
From Northern Ireland, UK
California, USA
Now
Experience John Fallows
CTO @ Aklivity
Serial Entrepreneur
HTML5 Contributor
WebSocket Pioneer
25+ years Java
Author Pro JSF & Ajax
JMS, Kafka, gRPC, MQTT, …
grpc-kafka who am i?

3. grpc-kafka what is grpc?
❖ open source RPC (Remote Procedure Call) framework
❖ protocol buffers for service definitions
package example;
service EchoService
{
rpc EchoUnary(EchoMessage) returns (EchoMessage);
}
message EchoMessage
{
string message = 1;
}

4. grpc-kafka why grpc?
❖ strong contract for service definitions
❖ code generation for many languages
❖ protobuf over http/2 optimizes communication
❖ unified http/2 connection per client

5. grpc-kafka grpc challenges
❖ load balancing, client vs proxy, connection vs rpc
❖ limited browser support, requires proxy
❖ binary message format is not human readable
❖ lack of edge caching

6. grpc-kafka compare grpc vs kafka
service
service
service
⚙
service
⚙
service
⚙
service
❖ contract
❖ coupling
❖ communication
❖ codec

7. grpc-kafka integrate grpc + kafka (open source zilla)
Every network and application data flow is a stream.
Streams are composed to create efficient protocol translation pipelines.
shared memory
tcpin
tlsin
httpin
grpcin
kafkaout
tcpout
● Garbage free
● Wait free
● Elastic scalability
● Sliding window flow control
● Multi-core CPU friendly
● Extensible by design
⚙

8. grpc-kafka unary rpc
package example;
service EchoService
{
rpc EchoUnary(EchoMessage) returns (EchoMessage);
rpc EchoBidiStream(stream EchoMessage) returns (stream EchoMessage);
}
message EchoMessage
{
string message = 1;
}

9. server
(unary)
grpc-kafka unary rpc
grpc
kafka
kafka
grpc
zilla zilla
kafka
client
(unary)
POST POST
(correlated)

10. grpc-kafka bidi streaming rpc
package example;
service EchoService
{
rpc EchoUnary(EchoMessage) returns (EchoMessage);
rpc EchoBidiStream(stream EchoMessage) returns (stream EchoMessage);
}
message EchoMessage
{
string message = 1;
}

11. server
(bidi)
grpc-kafka bidi streaming rpc
grpc
kafka
kafka
grpc
zilla zilla
kafka
client
(bidi)
POST POST

12. demo
grpc end-to-end streaming via kafka

13. grpc-kafka integration impact (observe)
HEADERS
:method POST
:path /example.EchoService/EchoBidiStream
content-type: application/grpc
1: {"Hello, world"}
topic: echo-requests
zilla:service example.EchoService
zilla:method EchoBidiStream
zilla:reply-to echo-responses
zilla:correlation-id <UUID>
1: {"Hello, world"}
null
1: {"Hello, again"} 1: {"Hello, again"}
EOS
HEADERS
:status 200
content-type: application/grpc
1: {"Hello, world"}
1: {"Hello, again"}
topic: echo-responses
zilla:service example.EchoService
zilla:method EchoBidiStream
zilla:correlation-id <UUID>
1: {"Hello, world"}
null
1: {"Hello, again"}
TRAILERS
:grpc-status 0
HEADERS
:method POST
:path /example.EchoService/EchoBidiStream
content-type: application/grpc
1: {"Hello, world"}
1: {"Hello, again"}
EOS
HEADERS
:status 200
content-type: application/grp
1: {"Hello, world"}
1: {"Hello, again"}
TRAILERS
:grpc-status 0

14. grpc-kafka integration impact (reach)
topic: echo-requests
zilla:service example.EchoService
zilla:method EchoBidiStream
zilla:reply-to echo-responses
zilla:correlation-id <UUID>
1: {"Hello, world"}
null
1: {"Hello, again"}
HEADERS
:status 200
content-type: application/grpc
1: {"Hello, world"}
1: {"Hello, again"}
topic: echo-responses
zilla:service example.EchoService
zilla:method EchoBidiStream
zilla:correlation-id <UUID>
1: {"Hello, world"}
null
1: {"Hello, again"}
TRAILERS
:grpc-status 0
HEADERS
:method POST
:path /example.EchoService/EchoBidiStream
content-type: application/grpc
1: {"Hello, world"}
1: {"Hello, again"}
EOS
event
driven
micro
service

15. grpc-kafka integration impact (migrate)
HEADERS
:method POST
:path /example.EchoService/EchoBidiStream
content-type: application/grpc
1: {"Hello, world"}
topic: echo-requests
zilla:service example.EchoService
zilla:method EchoBidiStream
zilla:reply-to echo-responses
zilla:correlation-id <UUID>
1: {"Hello, world"}
null
1: {"Hello, again"} 1: {"Hello, again"}
EOS
topic: echo-responses
zilla:service example.EchoService
zilla:method EchoBidiStream
zilla:correlation-id <UUID>
1: {"Hello, world"}
null
1: {"Hello, again"}
HEADERS
:status 200
content-type: application/grp
1: {"Hello, world"}
1: {"Hello, again"}
TRAILERS
:grpc-status 0
event
driven
micro
service

16. grpc-kafka beyond request-response
❖ grpc server streaming from kafka (fanout)
❖ grpc server streaming from kafka (fanout, reliable)
❖ grpc client streaming to kafka (oneway)
❖ grpc client streaming to kafka (oneway, reliable)

17. grpc-kafka server streaming (fanout)
package example;
import "google/protobuf/empty.proto";
service FanoutService
{
rpc FanoutServerStream(google.protobuf.Empty) returns (stream FanoutMessage);
}
message FanoutMessage
{
string message = 1;
}

18. grpc-kafka server streaming (fanout)
grpc
kafka
zilla kafka
client
(server streaming)
POST
event
driven
micro
service

19. grpc-kafka server streaming (fanout, reliable)
HEADERS
:method POST
:path /example.FanoutService/…
content-type: application/grpc
EOS
HEADERS
:status 200
content-type: application/grpc
1: {"Hello, world"}
32767: 0xfda8c…
grpc
kafka
zilla kafka
client
1: {"Hello, again"}
32767: 0xe1ac…
✖

20. grpc-kafka server streaming (fanout, recovery)
HEADERS
:method POST
:path /example.FanoutService/…
content-type: application/grpc
last-message-id-bin: base64(0xfda8c…)
EOS
HEADERS
:status 200
content-type: application/grp
grpc
kafka
zilla kafka
client
1: {"Hello, again"}
32767: 0xe1ac…
✔

21. demo
grpc reliable server streaming via kafka

22. grpc-kafka zilla examples
❖ grpc.kafka.echo
Echoes messages sent to a Kafka topic via gRPC from a gRPC client
❖ grpc.kafka.proxy
Correlates gRPC requests and responses over separate Kafka topics
❖ grpc.kafka.fanout
Streams messages published to a Kafka topic, applying conflation
based on log compaction
❖ …
github.com/aklivity/zilla-examples

23. grpc-kafka multi-protocol

24. www.aklivity.io
github.com/aklivity/zilla ★
Q & A
pssst…we have 😎 socks at Booth S11
Stream Data Anywhere.