A Better Kafka Connect With Kubernetes, Stefan Sprenger & Hakan Lofcali | Current 2022 Kafka Connect executes multiple connectors in the same process and does not offer any mechanism for the isolation of connectors. Multiple connectors share the same resources (vCPUs, MEM, etc.), which has the unfortunate side effect that excessive resource requests of single connectors impact the health of the rest of the Kafka Connect cluster. This talk proposes a novel, cloud-native deployment model for Kafka Connect, which uses the different concepts of Kubernetes for executing, scaling, and isolating single Kafka Connect connectors. In a nutshell, we build unique container images for each Kafka Connect connector type. We run connectors as Kubernetes Deployments, which allows us to either set the number of connector instances (or tasks) manually or let Kubernetes scale the connectors elastically. We use Kubernetes’ Resource Management for declaring the resource requests and limits of single connector instances. As a consequence, we achieve fully self-contained connectors, a necessity for production deployments of Kafka Connect. In a comprehensive evaluation, we compared the presented approach with a Strimzi-based deployment of Kafka Connect. We discuss the results and highlight the benefits and disadvantages of the presented approach (there's no free lunch!). We answer questions, such as: What’s the impact of running single-connector clusters on the overall resource consumption? How well does the elastic scaling of connectors work? Can single connectors go rogue without having an impact on the rest of the cluster?

1. Unlocking a
Harder, Better, Stronger
Kafka Connect
with Kubernetes
Stefan Sprenger & Hakan Lofcali
{stefan,hakan}@datacater.io

2. Kafka Connect is a
distributed, scalable service
for managing connectors
that integrate Apache Kafka with systems.
3

3. 4
KAFKA CONNECT CLUSTER
Worker 1 Worker 2
Kafka Connect Cluster
Worker 0
Connector
Connector
Connector
Connector
Connector
Connector
Connector
Connector
Connector

4. No self-containment
If one connector goes rogue, an entire
node or cluster can go down.
5

5. 6
KAFKA CONNECT ON KUBERNETES
Cluster
Worker
Deployment
Pod
Kafka Connect Kubernetes

6. 7
EXAMPLE: CONNECTOR EXCEEDING MEMORY
Kafka Connect Cluster
CDC source connector backfilling a snapshot
S3 source connector consuming GBs of CSV files
Sink connector streaming critical ticker information

7. 8
EXAMPLE: CONNECTOR EXCEEDING MEMORY
CDC source connector backfilling a snapshot
S3 source connector consuming GBs of CSV files
Sink connector streaming critical ticker information
Kafka Connect Cluster

8. 9
EXAMPLE: CONNECTOR EXCEEDING MEMORY
CDC source connector backfilling a snapshot
S3 source connector consuming GBs of CSV files
Sink connector streaming critical ticker information
Kafka Connect Cluster

9. If you don’t want services to
interfere with each other,
separate them.
10

10. In our case, separation is a
trade-off between resource
efficiency and robustness.
11

11. 12
All connectors in same cluster
Same connector types in same cluster
Single connectors in single clusters
Robustness
Resource efficiency

12. 13
Let’s turn
Kafka Connect
inside-out

13. 14
CONNECTORS BECOME CLUSTERS
Cluster
Connector
Connector
Connector
Connector
Connector
Connector
Connector
Connector
Connector
One “cluster” per connector.

14. 15
DEPLOYMENT MODEL
Connector
Connector Connector
Connector Connector
Connector
K8s
Deployment
K8s
Deployment
K8s
Deployment

15. 16
UNIQUE CONTAINER IMAGES
Connector-specific
container images
Update connectors without
taking others offline

16. 17
RESOURCE REQUESTS & LIMITS
Connector-level resource
management
Different connector types and
instances require different
resources

17. 18
MULTI-TASK CONNECTORS
Connector
Connector
K8s
Deployment
Pod
Task Task Task
Pod
Pod
Horizontal Pod Autoscaler

18. 19
EXAMPLE: CONNECTOR EXCEEDING MEMORY
CDC source connector backfilling a snapshot
S3 source connector consuming GBs of CSV files
Sink connector streaming critical ticker information
Kafka Connect Cluster

19. 20
Perks

20. 21
TRUE SELF-CONTAINMENT
True self-containment
enables large-scale, organisation-wide
connector services

21. 22
CONNECTOR-LEVEL LOGS
Access to connector-level logs without
the deployment of additional tooling
improves developer experience

22. 23
IMPROVED CONFIG HANDLING
Sharing and encrypting connector
configs becomes convenient through
Kubernetes’ config maps and secrets.

23. 24
Downsides

24. 25
INEFFICIENT RESOURCE USAGE
Isolation of workloads requires much
more resources, especially memory
~ 0.5 - 1 GB MEM per connector (task)

25. 26
Moving
forward

26. 27
K8S OPERATOR
Develop operator and introduce
custom resource definitions

27. 28
CONSIDER ALTERNATIVES TO JVM
Can we become more resource-
efficient?
Move to a JVM alternative (e.g., GraalVM) or try
a different language (shout-out to Conduit)

28. Unlocking a
Harder, Better, Stronger
Kafka Connect
with Kubernetes
Stefan Sprenger & Hakan Lofcali
{stefan,hakan}@datacater.io