When your Kafka clusters start growing so is the cost associated with them. As administrators we have to ensure that the service we support is operating in the most reliable way to satisfy the customers. However, for our business it is as important that we ensure the same service is also cost-efficient. There are two ways we can optimize the cost of service – tuning broker machines and tuning the data transfers. Minimizing data transfer is the largest return on investment since that is what accounts for the most spend. With the use of Kafka administrative tools and metrics we can find multiple ways to reduce the data transfers in the clusters. The presentation will cover various techniques administrators of Kafka service can employ to reduce the data transfers and to save the operational costs. Reducing cross-AZ traffic, optimizing batching with use of DumpLogSegment script, utilizing Kafka metrics to shut down unused data streams and more. With an objective of making our Kafka deployment as cost effective as possible, we have gained money saving tricks. And we would love to share them with the community.

1. Administrative Techniques to
Reduce Kafka Costs
Anna Kepler
Technical Product Manager

2. About me
Technical Product Manager for Data Platform
akepler
Previously
● Technical Lead and Individual contributor
for Streaming Data product
Working with data since 2014

5. Kafka Cluster Cost Breakdown
● EC2 Instances
● EBS Volumes
● Network
○ Cross AZ
○ Peering
○ Internet
● Cost of Monitoring

6. Cross-AZ traffic
Cost
1GB = $0.02
1TB = $20
1PB = $20,000
Configuration
broker.rack
Partition 1 Leader
us-west-1a Broker
Partition 1 Follower
us-west-1b Broker
us-west-1b Consumer
Partition 2 Follower
Partition 2 Leader
Free
$
0
.
2
/
G
B

7. Kafka 2.4+
KIP-392: Allow consumers to fetch from
closest replica
Producer Configurations
○ client.rack - corresponds with the
broker config 'broker.rack’
Broker Configurations
Pluggable Replica Selector with 2
implementations
○ LeaderSelector - default
○ RackAwareReplicaSelector

8. Cross-AZ traffic
Potential Cost Savings
50%
Partition 1 Leader
us-west-1a Broker
Partition 1 Follower
us-west-1b Broker
us-west-1b Consumer
Partition 2 Follower
Partition 2 Leader
Free
Free

9. Batching and
Compression
compression.type
default: none
values: none, gzip, snappy, lz4, zstd
batch.size
default: 16KB
linger.ms
default: 0

10. Tuning Spark Producers
Tool:
bin/kafka-run-class.sh kafka.tools.DumpLogSegments
--files topic_name/00000000124229902957.log --deep-
iteration
Output:
baseOffset: 124229953565 lastOffset: 124229953598
count: 34 baseSequence: -1 lastSequence: -1
producerId: -1
...
compresscodec: SNAPPY crc: 1379030279 isvalid: true
buffer.memory
max.request.size
batch.size
compression.type
linger.ms

11. Savings
~330 MB/s ~260 MB/s
22% ↓

12. Clients Presets

13. Client Metrics
Producer
● 28 Request metrics
● 8 Connection metrics
● 6 Per Broker metrics
● 9 Per Topic metrics
+ Consumer metrics
+ Broker metrics
+ Topic metrics
Benefit Cost

14. Other tips
Reduce replication
Eliminate unread streams
Introduce cost reporting mechanism

15. Summary of all techniques
Cross AZ Network Cost Reduction
Tuning Compression and Batching to Reduce Costs
Client Metrics - gather what you need
Smaller Administrative Techniques

16. Questions