"With Kafka Connect providing the ability to connect numerous external systems without having to write custom code it can be an integral part of the pipeline for putting data in motion. However, there are some settings and practices that can lead to a better experience if you are an admin of a Kafka Connect cluster as more teams add additional connectors for other use cases or more load is added to the Kafka Connect cluster overall. This presentation will look at the best practices to configure Kafka Connect to output important data when evaluating if more resources are needed for a Kafka Connect worker or if a new node should be added to the cluster overall. Areas of interest to be discussed: * What memory areas make up a Kafka Connect process (JVM process)? * What is verbose gc and why should it be enabled? * Is Kafka Connect configured to give me the specific information for my connectors and their respective tasks? * Which connect.protocol should I be using in environment?"

1. Configuring Kafka Connect To Be
Successful At Scale
Travis Sweet
Sr. Technical Support Engineer

2. About Me
Travis Sweet
I’ve been supporting Java applications since
August 2008 through the present day, i.e. 15+
years. Over 5+ years of that time has been
supporting Confluent Platform. In 2023 my
focus has been on Kafka Connect.
If I had my own theme music for this session
it would be:
We Gon’ Be Alright by Ty Tribbett
I like fries and ice cream.*
@jtsweet

3. Agenda
1. High Level Overview of Kafka Connect
2. Purpose
3. Foundation
4. Kafka Connect Has To Do More Than Just Run
@jtsweet

4. Overview of Kafka Connect
● Kafka Connect is a wonderful piece of computer software that allows users to
connect to external systems to put data into motion:
○ Source systems (where data comes from)
■ The data is produced (written to Kafka brokers)
○ Sink systems (where the data goes to, its new home)
■ The data is consumed (written to the new destination system)
@jtsweet

5. Agenda
1. High Level Overview of Kafka Connect
2. Purpose
3. Foundation
4. Kafka Connect Has To Do More Than Just Run
@jtsweet

6. Purpose

7. Purpose
● The house represents Kafka Connect Clusters that have not been configured
to be successful at scale.
@jtsweet

8. Purpose
● The house represents Kafka Connect Clusters that have not been configured
to be successful at scale.
● Similarities between the house and Kafka Connect Clusters.
@jtsweet

9. Purpose
● The house represents Kafka Connect Clusters that have not been configured
to be successful at scale.
● Similarities between the house and Kafka Connect Clusters.
● Where, when, and why does this happen?
@jtsweet

10. Purpose
● The house represents Kafka Connect Clusters that have not been configured
to be successful at scale.
● Similarities between the house and Kafka Connect Clusters.
● When, where, and why does this happen?
● We can prevent this!
@jtsweet

11. Purpose
● The house represents Kafka Connect Clusters that have not been configured
to be successful at scale.
● Similarities between the house and Kafka Connect Clusters.
● When, where, and why does this happen?
● We can prevent this!
● Please spread the word!
@jtsweet

12. Agenda
1. High Level Overview of Kafka Connect
2. Purpose
3. Foundation
4. Kafka Connect Has To Do More Than Just Run
@jtsweet

13. Foundation
● CPUs/Cores
@jtsweet

14. Foundation
● CPUs/Cores
● Memory
@jtsweet

15. Foundation
● Memory
○ In respect to Kafka Connect (JVM or Java Virtual Machine)
@jtsweet

16. Foundation
● Memory
○ In respect to Kafka Connect (JVM)
○ There are 3 memory areas
@jtsweet

17. Foundation
● Memory
○ In respect to Kafka Connect (JVM)
○ There are 3 memory areas
■ Metaspace memory area - previously named PermGen prior to Java 8
@jtsweet

18. Foundation
● Memory
○ In respect to Kafka Connect (JVM)
○ There are 3 memory areas
■ Metaspace memory area - previously named PermGen prior to Java 8
■ Native memory area
@jtsweet

19. Foundation
● Memory
○ In respect to Kafka Connect (JVM)
○ There are 3 memory areas
■ Metaspace memory area - previously named PermGen prior to Java 8
■ Native memory area
■ Java heap
@jtsweet

20. Foundation
● Memory
○ In respect to Kafka Connect (JVM)
○ There are 3 memory areas
■ Metaspace memory area - previously named PermGen prior to Java 8
■ Native memory area
■ Java heap
○ Total memory footprint of a Java process
@jtsweet

21. Foundation
● Memory
○ In respect to Kafka Connect (JVM)
○ There are 3 memory areas
■ Metaspace memory area - previously named PermGen prior to Java 8
■ Native memory area
■ Java heap
○ Total memory footprint of a Java process
■ It is the sum of the 3 memory areas
@jtsweet

22. Foundation
● Memory
○ In respect to Kafka Connect (JVM)
○ There are 3 memory areas
■ Metaspace memory area - previously named PermGen prior to Java 8
■ Native memory area
■ Java heap
○ Total memory footprint of a Java process
■ It is the sum of the 3 memory areas
○ What does the JVM try to do in regards to garbage collection?
@jtsweet

23. Foundation
● Memory
○ In respect to Kafka Connect (JVM)
○ There are 3 memory areas
■ Metaspace memory area - previously named PermGen prior to Java 8
■ Native memory area
■ Java heap
○ Total memory footprint of a Java process
■ It is the sum of the 3 memory areas
○ What does the JVM try to do in regards to garbage collection?
■ By design the JVM will try to keep the heap utilization between 40% to 70% of the Java
heap.
@jtsweet

24. Let's Apply What We Have Learned So Far
● Our Kafka Connect worker will be running on a machine with 16 GB of RAM
@jtsweet

25. Let's Apply What We Have Learned So Far
● Our Kafka Connect worker will be running on a machine with 16 GB of RAM
○ How much of the RAM goes to Kafka Connect?
@jtsweet

26. Let's Apply What We Have Learned So Far
● Our Kafka Connect worker will be running on a machine with 16 GB of RAM
○ How much of the RAM goes to Kafka Connect?
■ On average the Operating System will use around 2 GB of RAM for its own use.
@jtsweet

27. Let's Apply What We Have Learned So Far
● Our Kafka Connect worker will be running on a machine with 16 GB of RAM
○ How much of the RAM goes to Kafka Connect?
■ On average the Operating System will use around 2 GB of RAM for its own use.
○ That leaves us 14 GB of RAM for Kafka Connect and as a best practice there are no additional
components running on the machine.
○ Now we take the 14 GB of RAM left and apply it to our 3 memory areas:
■ Java heap
■ Native memory area
■ Metaspace memory area
@jtsweet

28. Let's Apply What We Have Learned So Far
● Our Kafka Connect worker will be running on a machine with 16 GB of RAM
○ How much of the RAM goes to Kafka Connect?
■ On average the Operating System will use around 2 GB of RAM for its own use.
○ That leaves us 14 GB of RAM for Kafka Connect and as a best practice there are no additional
components running on the machine.
○ Now we take the 14 GB of RAM left and apply it to our 3 memory areas:
■ Java heap
● -Xms2G -Xmx6G
■ Native memory area
■ Metaspace memory area
@jtsweet

29. Let's Apply What We Have Learned So Far
● Our Kafka Connect worker will be running on a machine with 16 GB of RAM
○ How much of the RAM goes to Kafka Connect?
■ On average the Operating System will use around 2 GB of RAM for its own use.
○ That leaves us 14 GB of RAM for Kafka Connect and as a best practice there are no additional
components running on the machine.
○ Now we take the 14 GB of RAM left and apply it to our 3 memory areas:
■ Java heap
● -Xms2G -Xmx6G
○ This leaves us with 8GB of RAM to be used between (14GB - 6GB = 8GB)
■ Native memory area
■ Metaspace memory area
@jtsweet

30. Let's Apply What We Have Learned So Far
● Our Kafka Connect worker will be running on a machine with 16 GB of RAM
○ How much of the RAM goes to Kafka Connect?
■ On average the Operating System will use around 2 GB of RAM for its own use.
○ That leaves us 14 GB of RAM for Kafka Connect and as a best practice there are no additional
components running on the machine.
○ Now we take the 14 GB of RAM left and apply it to our 3 memory areas:
■ Java heap
● -Xms2G -Xmx6G
○ This leaves us with 8GB of RAM to be used between
■ Native memory area
■ Metaspace memory area
○ Remember the JVM’s ideal heap utilization of 40% to 70% of the Java heap
@jtsweet

31. Let's Apply What We Have Learned So Far
● Our Kafka Connect worker will be running on a machine with 16 GB of RAM
○ How much of the RAM goes to Kafka Connect?
■ On average the Operating System will use around 2 GB of RAM for its own use.
○ That leaves us 14 GB of RAM for Kafka Connect and as a best practice there are no additional
components running on the machine.
○ Now we take the 14 GB of RAM left and apply it to our 3 memory areas:
■ Java heap
● -Xms2G -Xmx6G
○ This leaves us with 8GB of RAM to be used between
■ Native memory area
■ Metaspace memory area
○ Remember the JVM’s ideal heap utilization of 40% to 70% of the Java heap
■ In regards to the 6GB maximum heap size we have an ideal range of 2.4 GB to 4.2 GB
@jtsweet

32. Agenda
1. High Level Overview of Kafka Connect
2. Purpose
3. Foundation
4. Kafka Connect Has To Do More Than Just Run
@jtsweet

33. Kafka Connect Has To Do More Than Run
● How do we investigate the Java heap utilization of our Kafka Connect worker?
@jtsweet

34. Kafka Connect Has To Do More Than Run
● How do we investigate the Java heap utilization of our Kafka Connect worker?
○ Enable verbose garbage collection (verbose gc) and heap dump generation
■ How do I enable it or turn it on?
@jtsweet

35. Kafka Connect Has To Do More Than Run
● Enabling verbose gc and heap dump generation on Java 8
@jtsweet

36. Kafka Connect Has To Do More Than Run
● Enabling verbose gc and heap dump generation on Java 11 and Java 17
-Xlog:gc*,gc+ref=debug,gc+heap=debug,gc+age=trace:file=/tmp/gc.log:time,uptim
e,level,tags:filecount=5,filesize=100m -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/heapdumpsFromJVM
Same disclaimer that the process has to be able to write the locations specified.
The location for the heap dump must be large enough for it to be written out
completely without it being truncated.
@jtsweet

37. Kafka Connect Has To Do More Than Run
● How do we investigate the Java heap utilization of our Kafka Connect worker?
○ Enable verbose garbage collection (verbose gc) and heap dump generation
○ Enable JMX metrics for Kafka Connect
@jtsweet

38. Kafka Connect Has To Do More Than Run
● How do we investigate the Java heap utilization of our Kafka Connect worker?
○ Enable verbose garbage collection (verbose gc) and heap dump generation
○ Enable JMX metrics for Kafka Connect
● Should you enable all of the above?
@jtsweet

39. Kafka Connect Has To Do More Than Run
● How do we investigate the Java heap utilization of our Kafka Connect worker?
○ Enable verbose garbage collection (verbose gc)
○ Enable JMX metrics for Kafka Connect
● Should you enable all of the above?
○ Yes, 100%
● Do your connect logs show the specific connector and specific task?
@jtsweet

40. Kafka Connect Has To Do More Than Run
● How do we investigate the Java heap utilization of our Kafka Connect worker?
○ Enable verbose garbage collection (verbose gc)
○ Enable JMX metrics for Kafka Connect
● Should you enable all of the above?
○ Yes, 100%
● Do your connect logs show the specific connector and specific task?
○ Enable Mapped Diagnostic Context (MDC) logging in your Kafka Connect environments
■ Do you have examples?
@jtsweet

41. Kafka Connect Has To Do More Than Run
Logs without Mapped Diagnostic Context enabled
INFO Using multi thread/connection supporting pooling connection manager
(io.searchbox.client.JestClientFactory)
INFO Using default GSON instance (io.searchbox.client.JestClientFactory)
INFO Node Discovery disabled... (io.searchbox.client.JestClientFactory)
INFO Idle connection reaping disabled... (io.searchbox.client.JestClientFactory)
@jtsweet

42. Kafka Connect Has To Do More Than Run
Logs with Mapped Diagnostic Context enabled
INFO [sink-elastic-orders-00|task-0] Using multi thread/connection supporting
pooling connection manager (io.searchbox.client.JestClientFactory:223)
INFO [sink-elastic-orders-00|task-0] Using default GSON instance
(io.searchbox.client.JestClientFactory:69)
INFO [sink-elastic-orders-00|task-0] Node Discovery disabled...
(io.searchbox.client.JestClientFactory:86)
@jtsweet

43. Kafka Connect Has To Do More Than Run
● How do we investigate the Java heap utilization of our Kafka Connect worker?
○ Enable verbose garbage collection (verbose gc)
○ Enable JMX metrics for Kafka Connect
● Should you enable all of the above?
○ Yes, 100%
● Do your connect logs show the specific connector and specific task?
○ Enable Mapped Diagnostic Context (MDC) logging in your Kafka Connect environments
■ Do you have examples?
● Which connect protocol should I be using in my environment?
@jtsweet

44. Kafka Connect Has To Do More Than Run
● How do we investigate the Java heap utilization of our Kafka Connect worker?
○ Enable verbose garbage collection (verbose gc)
○ Enable JMX metrics for Kafka Connect
● Should you enable all of the above?
○ Yes, 100%
● Do your connect logs show the specific connector and specific task?
○ Enable Mapped Diagnostic Context (MDC) logging in your Kafka Connect environments
■ Do you have examples?
● Which connect protocol should I be using in my environment?
○ Are you running Replicator as a connector?
@jtsweet

45. Kafka Connect Has To Do More Than Run
● How do we investigate the Java heap utilization of our Kafka Connect worker?
○ Enable verbose garbage collection (verbose gc)
○ Enable JMX metrics for Kafka Connect
● Should you enable all of the above?
○ Yes, 100%
● Do your connect logs show the specific connector and specific task?
○ Enable Mapped Diagnostic Context (MDC) logging in your Kafka Connect environments
■ Do you have examples?
● Which connect protocol should I be using in my environment?
○ Are you running Replicator as a connector?*
@jtsweet

46. Kafka Connect Has To Do More Than Run
● Java Runtime Environment (JRE) vs. Java Development Environment (JDK)
@jtsweet

47. Agenda
1. High Level Overview of Kafka Connect
2. Purpose
3. Foundation
4. Kafka Connect Has To Do More Than Just Run
@jtsweet

48. You can find me @
X (Previously Known As Twitter): RevJ1980
LinkedIn:
linkedin.com/in/james-t-sweet-01465582

49. Links:
File descriptors and mmap settings
How to enable verbose gc and heap dump Knowledge Base Article
JMX Metrics for Kafka Connect
Mapped Diagnostic Context (MDC)
Replicator
Deep Dive into Kafka Connect Protocol
Theme music - We Gon’ Be Alright by Ty Tribbet

50. Questions*