"Kafka Connect is quintessential for most production-grade Kafka data pipelines. As a core Kafka Connect team supporting more than 75 connectors at Confluent, we needed a platform that not only enables easy repeatability of tests across a breadth of connectors but also facilitates benchmarking and extensive analysis of diverse test runs. This session goes over the challenges in designing a framework to test connector performance. We will cover specific aspects of Connector testing, fine tuning parameters like connector configurations, clusters specifications for optimal performance. The framework integrates with various tools such as Jmeter, Hammerdb, ChaosMesh, Prometheus etc., scaled for Cloud-native environments. We will further go over how we optimized connector performance for maximum public record poll rates, cpu/memory usage, throughput, latencies for different data formats. Attendees learn in detail how real-world scenarios can be simulated in a test setup for a few connectors like S3 Sink Connector. We also go over some generic guidelines for performance testing of Kafka Connectors and practical considerations for implementing such test frameworks in general."

1. Connect, Test, Optimize!
Create an Ultimate Kafka Connector Benchmarking Toolkit
Suchi Amalapurapu
Principle Engineer, Confluent
Sudesh Wasnik
Senior Software Engineer II, Confluent

2. Kafka Streaming Application
A
No-Code way of connecting external systems to Kafka
Kafka
syslog
SOURCES SINKS
Kafka
Connect
Kafka
Connect
Apps
ksqlDB

3. Kafka Streaming Application
A
No-Code way of connecting external systems to Kafka
Kafka
syslog
SOURCES SINKS
Kafka
Connect
Kafka
Connect
Database Connectors File Store Connectors MQ Connectors SAAS Connectors

4. Sample Streaming Application
Kafka
Connect
External System
LetsA
MySQL Database
Kafka
MySQL Debezium Connector
Product Catalogue database
Captures change events
Stores change events

5. MySQL Stream Application
LetsA
MySQL Database
Kafka
High throughput
Slow CDC
Lag!
MySQL Debezium Connector
Kafka
Connect
External System

6. S3 Sink Streaming Application
LetsA
High throughput
Slower file flush rates
Slower ingestion
Kafka
Connect
External System
AWS S3
Kafka
Kafka Connector

7. LetsA
MySQL Database
Kafka
Very high throughput
Captures events, but slowly
Lag!
MySQL Debezium Connector
Kafka
Connect
External System
Performance Determinants
● Untuned Connector configuration
● #Tasks
● CPU, Resource crunch
● Unresponsive Sink
● Kafka consumer throttling
How to simulate test-scenarios ?
Slower file flush rates
S3 Sink Streaming Application

8. Challenges :
1. High breadth of connectors
2. End-system specifics
3. Various configuration combinations
4. Pre-existing test-scripts and systems
5. Integration with CI/CD
Test Setup

9. Test Setup
Kafka
Analyzer
Fetch Data
External System Kafka Connector
Generate
Load
Load Generator
Pluggable: Any Loadgen can be used
Sizeable : Configurable CPU/Memory
Scalable : Increase Replicas
Report Generation
Pluggable: Any Analyzer can be used
Sizeable : Configurable CPU/Memory
Custom Tests Designs
Report Generation

10. Test Setup
Kafka
Analyzer
Fetch Data
External System Kafka Connector
Generate
Load
Load Generator
Metric Analyzer
1. Scrape and replay metrics
2. Compare baselines metrics
3. Export metrics
Record Analyzer
1. Consumes destination Topic
2. Verify delivery-semantics
a. At-least once
b. Data-loss

11. Source System
Performance test pipeline
Run Test
Notification
Test Results
Analyze Results
Dashboards
Kafka Cluster
Connect
Analyzer
Load Generator
Storage
CI/CD
Input Specification
Kubernetes

12. Performance test pipeline
Run Test
Notification
Test Results
Analyze Results
Dashboards
Source System
Kafka Cluster
Connect
Analyzer
Load Generator
Kubernetes
Storage
CI/CD
Input Specification
Input Specification
Test Specification
• Each spec represents a
distinct test
• Exhaustive Spec
• Template engines like jinja2,
Django templating can be
used.

13. Performance test pipeline
Run Test
Notification
Test Results
Analyze Results
Dashboards
Source System
Kafka Cluster
Connect
Analyzer
Load Generator
Kubernetes
Storage
CI/CD
Input Specification
Source System
Source System
➔ Tuned.
➔ Pod crash monitoring
➔ Emit pod + end-system
metrics.
Source System

14. Source System
Performance test pipeline
Run Test
Notification
Test Results
Analyze Results
Dashboards
Kafka Cluster
Connect
Analyzer
Load Generator
Storage
CI/CD
Input Specification
Kubernetes
Load Generator

15. Source System
Performance test pipeline
Run Test
Notification
Test Results
Analyze Results
Dashboards
Kafka Cluster
Connect
Analyzer
Load Generator
Storage
CI/CD
Input Specification
Kubernetes
Kafka Cluster
Connect
➔ Set Kafka and Connect
base properties
➔ Scale application
➔ Emit JMX metrics
Kafka & Connect Cluster

16. Source System
Performance test pipeline
Run Test
Notification
Test Results
Analyze Results
Dashboards
Kafka Cluster
Connect
Analyzer
Load Generator
Storage
CI/CD
Input Specification
Kubernetes
Analyzer
➔ Scrape JMX / Custom
metrics
➔ Collect resource profiles
➔ Scrape custom metrics
➔ Export metrics to
metric-store (Bigquery,
Datadog)
Observability
Analysis Template
Metrics Threshold
1. CPU Usage 20%
2. Heap Memory Usage 20%
3. Source Record Poll Rate 10%
4. Producer Bytes ProduceRate 10%
➔ List Metric-name and their
properties
➔ Ex - Scrape
“kafka-consumer” metrics
only
Analysis Template

17. Framework is ready !
Let's review test scenarios. ..

18. S3-Sink Test :
Kafka
Analyzer
Fetch Data
Kafka Connector
Generate
Load
Load Generator
S3 Sink
Schema Registry

19. S3-Sink Test : Avro v/s JSON_SR
Result
➔ Test input-data-format : AVRO v/s JSON_SR
➔ Expectation :
◆ Avro messages smaller
◆ Avro should perform better
TEST
Low performance boost !

20. S3-Sink Test : File flush frequency
Result
➔ Test Flush-rate frequency
◆ Configuration-A:
● Scheduled Rotation Interval = 3 mins,
● Flush Size = 1000
◆ Configuration-B:
● Scheduled Rotation Interval = 6 mins,
● Flush Size = 3000
➔ Expectation: Configuration-B will perform
better
TEST
High performance boost !

21. JMS Source Test: v1 v/s v2 performance
➔ Test and compare : v1 v/s v2
➔ Issue-1 : Duplicates
➔ Issue-2: Pod restart during peak-load caused data-loss
➔ Fault Injection tests
Delivery semantics verified

22. Simulate Chaos in Performance tests

23. Chaos Scenarios Common to All Connectors
Network Loss (Chaos-Mesh)
Simulate a packet loss scenario
Pod Kill (Chaos-Mesh)
Delete k8s pods randomly
Noisy-Neighbor Scenario
Run Resource Intensive connector in same node
CPU Stress (Chaos-Mesh)
Simulate High CPU load.
Memory Stress (Chaos-Mesh)
Simulate high memory usage
Chaos Tests : Generic Scenarios

24. Source System
Performance test pipeline
Kafka Cluster
Connect
Analyzer
Load Generator
CI/CD
Kubernetes
A
Inject Chaos

25. Impact
1. Easy + repeatable tests
2. Uncover connector bottlenecks
3. LoadGen integrations
4. Report generation
5. Chaos DR tests on connectors
6. Performance gating of 15+ connectors

26. Takeaways
1. Importance of Kafka Connect Perf tests
2. Factors affecting connector performance
3. Test platform design guidelines
4. Configuration tuning and impact
5. Chaos tests

27. Special Thanks
● Scott Hendricks
● Anupam Aggarwal
● Vikas Balani
Thank you!