The document discusses tuning Apache Kafka connectors for Flink, focusing on the migration from test to production and addressing issues related to message size and batch processing in Kafka. It outlines steps to resolve common errors, optimize configurations, and improve performance through adjustments in topic settings, batch compression, and custom partitioning strategies. The author emphasizes the necessity of adapting Kafka sink/source configurations based on varying data loads and business requirements.

1. Tuning Apache Kafka® Connectors for Flink| Confidential | 2022-07-10
Tuning Apache Kafka®
Connectors for Flink

2. A story:
Test -> Production
migration frustration
Same: Business logic, Code.
Different: Data, Load, Clusters and database configs.

3. Job Status: FAILED
Reason:
...org.apache.kafka.common.errors.RecordTooLargeException:
The message is [X] bytes when serialized which is larger than [Y]

4. Job Status: FAILED
Step 1. Fix in Flink Sink config:
'properties.max.request.size' >=
MAX_MESSAGE_SIZE_IN_BYTES

5. Job Status: FAILED
Reason:
...org.apache.kafka.common.errors.RecordTooLargeException:
The request included a message larger than the max message
size the server will accept.

6. Job Status: RUNNING
Reason:
...org.apache.kafka.common.errors.RecordTooLargeException:
The request included a message larger than the max message
size the server will accept.
Step 2. Fix in Kafka topic config:
'max.message.bytes' >= MAX_MESSAGE_SIZE_IN_BYTES

7. logs/TOPIC_NAME_AND_PARTITION_NUMBER/
Kafka segments on a disk

8. logs/TOPIC_NAME_AND_PARTITION_NUMBER/
Messages inside segments

9. logs/TOPIC_NAME_AND_PARTITION_NUMBER/
50 times smaller segment

10. logs/TOPIC_NAME_AND_PARTITION_NUMBER/
50 times smaller segment

11. logs/TOPIC_NAME_AND_PARTITION_NUMBER/
Increased max.message.bytes

12. Job Status: FAILED
Reason:
...org.apache.kafka.common.errors.RecordBatchTooLargeException:
The request included message batch larger than the configured
segment size on the server.

13. Job Status: RUNNING
Fix in Kafka topic config:
'segment.bytes' >= MAX_MESSAGE_SIZE_IN_BYTES
Reason:
...org.apache.kafka.common.errors.RecordBatchTooLargeException:
The request included message batch larger than the configured
segment size on the server.

14. Impact on Flink Source
Performance improvement:
'properties.request.timeout.ms' > 30 Sec
Job Status: RUNNING (No progress)
Reason:
...INFO...org.apache.kafka.clients.FetchSessionHandler - Error
sending fetch request to node 5:
...org.apache.kafka.common.errors.DisconnectException: null

15. Big batches

16. Big batches

17. Big batches

18. Big batches

19. Big batches
Performance improvement:
'properties.properties.batch.size' >=
X * MAX_MESSAGE_SIZE_IN_BYTES

20. Job Status: RUNNING (No progress)
Reason:
...WARN...org.apache.kafka.clients.producer.internals.Sender - Got
error produce response in correlation id 137 on topic-partition
response-0, splitting and retrying (2147483647 attempts left).
Error: MESSAGE_TOO_LARGE

21. Job Status: RUNNING
Fix in Kafka topic config:
'max.message.bytes' >= X * MAX_MESSAGE_SIZE_IN_BYTES
Reason:
...WARN...org.apache.kafka.clients.producer.internals.Sender - Got
error produce response in correlation id 137 on topic-partition
response-0, splitting and retrying (2147483647 attempts left).
Error: MESSAGE_TOO_LARGE

22. Job Status: RUNNING (No progress)
Reason:
...WARN...org.apache.kafka.clients.producer.internals.Sender - Got
error produce response in correlation id 137 on topic-partition
response-0, splitting and retrying (2147483647 attempts left).
Error: MESSAGE_TOO_LARGE

23. Job Status: FINISHED
Fix in Flink Sink Config:
'properties.compression.type' = 'snappy' ['gzip', 'lz4', 'zstd']
Reason:
...WARN...org.apache.kafka.clients.producer.internals.Sender - Got
error produce response in correlation id 137 on topic-partition
response-0, splitting and retrying (2147483647 attempts left).
Error: MESSAGE_TOO_LARGE

24. Batch compression
Pros:
● Save disk space on Kafka
● The bigger batch, the better compression
● Fewer chances to get a timeout

25. Batch compression
Cons:
● Every consumer has to decompress messages
● A Flink Sink task runs slower
● Compression uses additional memory (~ 315 MB of JVM heap
per 5000 partitions)

26. Impact on Flink Source
Job Status: RUNNING
Performance improvement:
'properties.max.partition.fetch.bytes' >=
X * MAX_MESSAGE_SIZE_IN_BYTES

27. High load and spikes

28. High load and spikes
properties.batch.size + properties.linger.ms

29. Impact on Flink Source
properties.fetch.min.bytes + properties.fetch.max.wait.ms

30. High load
properties.max.in.flight.requests.per.connection = 1

31. High load
properties.max.in.flight.requests.per.connection = 3

32. High load
properties.max.in.flight.requests.per.connection = 3

33. High load
properties.max.in.flight.requests.per.connection = 3

34. High load
Warning
To keep records order in partition with retry > 0 (Default)
'max.in.flight.requests.per.connection' = 1 AND
‘enable.idempotence’ = False
OR
'max.in.flight.requests.per.connection' = 5(Default) AND
‘acks’ = ‘all’ AND ‘enable.idempotence’ = True

35. Custom Partitioner

36. Why custom partitioner?
org.apache.kafka.clients.producer.internals.DefaultPartitioner
Records for specific topic with null keys and no
assigned partition will be sent to the same
partition until the batch is ready to be sent.
When a new batch is created, a new partition is
chosen

37. Why custom partitioner?

38. Why custom partitioner?

39. Why custom partitioner?
org.apache.kafka.clients.producer.internals.DefaultPartitioner
If no partition is specified but a key is present,
choose a partition based on a hash of the key

40. Why custom partitioner?

41. Why custom partitioner?

42. Why custom partitioner?

43. Why custom partitioner?

44. Custom partitioner apply
// Input: sortedWeights = {'DE': 20, 'FR': 11, 'DK': 2, 'CZ': 1}, nPartitions = 3
totalSum = sum(sortedWeights.values()); accSum = 0;
keysSumWeights = sortedWeights.forEach((k, v) ->
{ accSum += v; return (k, accSum /totalSum) }
// {"DE":0.588, 'FR': 0.912, 'DK': 0.97, 'CZ': 1}
fraction = 1.0 / nPartitions; partition = 1; partitionsDistribution = {}
for k, v in keysSumWeights
partitionDistribution[k] = min(partition, nPartitions) - 1
if v >= partition * fraction
partition += 1
return partitionsDistribution

45. Custom partitioner apply

46. Custom partitioner apply
Generate: 'DE', 'GB', 'FI', 'ES', 'NO'
DE - individual
GB - individual
FI, ES, NO - shared

47. Custom partitioner apply
Generate: 'DE', 'GB', 'FI','ES', 'NO'
DE - individual
GB - individual
FI, ES, NO - shared

48. Custom partitioner apply

49. Custom partitioner apply

50. Custom partitioner apply

51. Flink partitioner vs Kafka partitioner
Flink Partitioner
Class org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner
.option("sink.partitioner", "org.myorg.quickstart.FlinkWeightedPartitioner")
Kafka Partitioner
Class org.apache.kafka.clients.producer.Partitioner
.option("properties.partitioner.class", "org.apache.kafka.clients.producer.RoundRobinPartitioner")

52. ❏ A custom partitioner could improve the performance of a Flink
Source.
❏ Add custom ConsumerPartitionAssignor in
‘properties.partition.assignment.strategy’ if necessary.
Impact on Flink Source/Kafka Consumer

53. Summary
● Enormous messages
● Group records to big batches
● Spikes, High load
● Custom partitions
Adapt Apache Kafka Sink/Source according
to business needs and data load

54. Ask me about Flink and Kafka!
Olena Babenko
Senior Software Engineer
olena@aiven.io