Flink Forward San Francisco 2022. In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you! by Olena Babenko

1. Tuning Apache Kafka® Connectors for Flink| Confidential | 2022-07-10
Tuning Apache Kafka®
Connectors for Flink

2. A story:
Test -> Production
migration frustration
Same: Business logic, Code.
Different: Data, Load, Clusters and database configs.

3. Job Status: FAILED
Reason:
...org.apache.kafka.common.errors.RecordTooLargeException:
The message is [X] bytes when serialized which is larger than [Y]

4. Job Status: FAILED
Step 1. Fix in Flink Sink config:
'properties.max.request.size' >=
MAX_MESSAGE_SIZE_IN_BYTES

5. Job Status: FAILED
Reason:
...org.apache.kafka.common.errors.RecordTooLargeException:
The request included a message larger than the max message
size the server will accept.

6. Job Status: RUNNING
Reason:
...org.apache.kafka.common.errors.RecordTooLargeException:
The request included a message larger than the max message
size the server will accept.
Step 2. Fix in Kafka topic config:
'max.message.bytes' >= MAX_MESSAGE_SIZE_IN_BYTES

7. logs/TOPIC_NAME_AND_PARTITION_NUMBER/
Kafka segments on a disk

8. logs/TOPIC_NAME_AND_PARTITION_NUMBER/
Messages inside segments

9. logs/TOPIC_NAME_AND_PARTITION_NUMBER/
50 times smaller segment

10. logs/TOPIC_NAME_AND_PARTITION_NUMBER/
50 times smaller segment

11. logs/TOPIC_NAME_AND_PARTITION_NUMBER/
Increased max.message.bytes

12. Job Status: FAILED
Reason:
...org.apache.kafka.common.errors.RecordBatchTooLargeException:
The request included message batch larger than the configured
segment size on the server.

13. Job Status: RUNNING
Fix in Kafka topic config:
'segment.bytes' >= MAX_MESSAGE_SIZE_IN_BYTES
Reason:
...org.apache.kafka.common.errors.RecordBatchTooLargeException:
The request included message batch larger than the configured
segment size on the server.

14. Impact on Flink Source
Performance improvement:
'properties.request.timeout.ms' > 30 Sec
Job Status: RUNNING (No progress)
Reason:
...INFO...org.apache.kafka.clients.FetchSessionHandler - Error
sending fetch request to node 5:
...org.apache.kafka.common.errors.DisconnectException: null

15. Big batches

16. Big batches

17. Big batches

18. Big batches

19. Big batches
Performance improvement:
'properties.properties.batch.size' >=
X * MAX_MESSAGE_SIZE_IN_BYTES

20. Job Status: RUNNING (No progress)
Reason:
...WARN...org.apache.kafka.clients.producer.internals.Sender - Got
error produce response in correlation id 137 on topic-partition
response-0, splitting and retrying (2147483647 attempts left).
Error: MESSAGE_TOO_LARGE

21. Job Status: RUNNING
Fix in Kafka topic config:
'max.message.bytes' >= X * MAX_MESSAGE_SIZE_IN_BYTES
Reason:
...WARN...org.apache.kafka.clients.producer.internals.Sender - Got
error produce response in correlation id 137 on topic-partition
response-0, splitting and retrying (2147483647 attempts left).
Error: MESSAGE_TOO_LARGE

22. Job Status: RUNNING (No progress)
Reason:
...WARN...org.apache.kafka.clients.producer.internals.Sender - Got
error produce response in correlation id 137 on topic-partition
response-0, splitting and retrying (2147483647 attempts left).
Error: MESSAGE_TOO_LARGE

23. Job Status: FINISHED
Fix in Flink Sink Config:
'properties.compression.type' = 'snappy' ['gzip', 'lz4', 'zstd']
Reason:
...WARN...org.apache.kafka.clients.producer.internals.Sender - Got
error produce response in correlation id 137 on topic-partition
response-0, splitting and retrying (2147483647 attempts left).
Error: MESSAGE_TOO_LARGE

24. Batch compression
Pros:
● Save disk space on Kafka
● The bigger batch, the better compression
● Fewer chances to get a timeout

25. Batch compression
Cons:
● Every consumer has to decompress messages
● A Flink Sink task runs slower
● Compression uses additional memory (~ 315 MB of JVM heap
per 5000 partitions)

26. Impact on Flink Source
Job Status: RUNNING
Performance improvement:
'properties.max.partition.fetch.bytes' >=
X * MAX_MESSAGE_SIZE_IN_BYTES

27. High load and spikes

28. High load and spikes
properties.batch.size + properties.linger.ms

29. Impact on Flink Source
properties.fetch.min.bytes + properties.fetch.max.wait.ms

30. High load
properties.max.in.flight.requests.per.connection = 1

31. High load
properties.max.in.flight.requests.per.connection = 3

32. High load
properties.max.in.flight.requests.per.connection = 3

33. High load
properties.max.in.flight.requests.per.connection = 3

34. High load
Warning
To keep records order in partition with retry > 0 (Default)
'max.in.flight.requests.per.connection' = 1 AND
‘enable.idempotence’ = False
OR
'max.in.flight.requests.per.connection' = 5(Default) AND
‘acks’ = ‘all’ AND ‘enable.idempotence’ = True

35. Custom Partitioner

36. Why custom partitioner?
org.apache.kafka.clients.producer.internals.DefaultPartitioner
Records for specific topic with null keys and no
assigned partition will be sent to the same
partition until the batch is ready to be sent.
When a new batch is created, a new partition is
chosen

37. Why custom partitioner?

38. Why custom partitioner?

39. Why custom partitioner?
org.apache.kafka.clients.producer.internals.DefaultPartitioner
If no partition is specified but a key is present,
choose a partition based on a hash of the key

40. Why custom partitioner?

41. Why custom partitioner?

42. Why custom partitioner?

43. Why custom partitioner?

44. Custom partitioner apply
// Input: sortedWeights = {'DE': 20, 'FR': 11, 'DK': 2, 'CZ': 1}, nPartitions = 3
totalSum = sum(sortedWeights.values()); accSum = 0;
keysSumWeights = sortedWeights.forEach((k, v) ->
{ accSum += v; return (k, accSum /totalSum) }
// {"DE":0.588, 'FR': 0.912, 'DK': 0.97, 'CZ': 1}
fraction = 1.0 / nPartitions; partition = 1; partitionsDistribution = {}
for k, v in keysSumWeights
partitionDistribution[k] = min(partition, nPartitions) - 1
if v >= partition * fraction
partition += 1
return partitionsDistribution

45. Custom partitioner apply

46. Custom partitioner apply
Generate: 'DE', 'GB', 'FI', 'ES', 'NO'
DE - individual
GB - individual
FI, ES, NO - shared

47. Custom partitioner apply
Generate: 'DE', 'GB', 'FI','ES', 'NO'
DE - individual
GB - individual
FI, ES, NO - shared

48. Custom partitioner apply

49. Custom partitioner apply

50. Custom partitioner apply

51. Flink partitioner vs Kafka partitioner
Flink Partitioner
Class org.apache.flink.streaming.connectors.kafka.partitioner.FlinkKafkaPartitioner
.option("sink.partitioner", "org.myorg.quickstart.FlinkWeightedPartitioner")
Kafka Partitioner
Class org.apache.kafka.clients.producer.Partitioner
.option("properties.partitioner.class", "org.apache.kafka.clients.producer.RoundRobinPartitioner")

52. ❏ A custom partitioner could improve the performance of a Flink
Source.
❏ Add custom ConsumerPartitionAssignor in
‘properties.partition.assignment.strategy’ if necessary.
Impact on Flink Source/Kafka Consumer

53. Summary
● Enormous messages
● Group records to big batches
● Spikes, High load
● Custom partitions
Adapt Apache Kafka Sink/Source according
to business needs and data load

54. Ask me about Flink and Kafka!
Olena Babenko
Senior Software Engineer
olena@aiven.io