In a typical deployment of Kafka with many topics and partitions, scaling the Kafka consumer efficiently is one of the important tasks in maintaining overall smooth Kafka operations. The traditional Kubernetes Horizontal Pod Scaling (HPA) that uses basic CPU and/or memory metrics is not suitable for scaling Kafka consumers. The more appropriate workload metric for Kafka consumer is the number of messages in Kafka broker queue. More specifically, the message production rate of a specific topic would be the right workload metric for a Kafka consumer. While using message production rate is a better way to decide the number of consumer replicas, this is still a reaction based auto-scaling. Using machine-learning based forecasting, it is possible for predict the upcoming increase or decrease of message production rate. With predicted workload, scaling the Kafka consumers could be achieved in a more timely manner, resulting with better performance KPI's.

1. Intelligent Autoscaling of Kafka
Consumers with Workload Prediction
Kafka Summit Americas 2021
Ming Sheu
EVP, Product
ProphetStor Data Services, Inc.

2. Agenda
Scaling Kafka Consumers
Scaling with Kubernetes Native HPA
Intelligent Autoscaling with Predictions
Conclusion

3. Producers
Producer 1
Producer 2
Kafka Cluster
Kafka Broker
Topic 1
Partition 0
Partition 1
Topic 2
Partition 0
Consumers
Consumer 1
Consumer
Group
Consumer 2
Consumer 3
ZooKeeper Service
Scaling Kafka Consumers
Copyright©2021 ProphetStor
• Long latency if not enough
Consumers
• Too many consumers result in
waste of resources
Performance vs Cost
3

4. Agenda
Scaling Kafka Consumers
Scaling with Kubernetes Native
HPA
Intelligent Autoscaling with Predictions
Conclusion

5. Kubernetes Native Horizontal Pod Autoscaling
5
Copyright©2021 ProphetStor
• Autoscaling of pods based on one or more
metrics
• Specify target metric threshold as a basis
for scaling up or down
• HPA Autoscaler calculates replica count
that best meets the target metric
threshold

6. Kafka Consumer Autoscaling Testing
6
Copyright©2021 ProphetStor
• Kubernetes Cluster: 1 Master, 3 Workers
• Brokers: 3
• ZooKeepers: 3
• Producer: 1 Topic: 1 Partitions: 40
• Consumer Scaling Range: 1 to 40
• Message Production Rate: various rates from 20K msg/min ~ 120K msg/min,
avg 85K msg/min
Kubernetes Native Horizontal Pod Autoscaling
• Scaling based on avg CPU% of Kafka consumer pods
• Scaling based on Consumer Lag

7. Native HPA with Consumer CPU Utilization
Consumer 1 Consumer 2 Consumer 3
Decreasing Replicas
Increasing Consumer Lag
Scaling Replicas Based on Target
Maintain Consumer Lag
Decreasing Replicas
Increasing Consumer Lag
7
Copyright©2021 ProphetStor

8. Native HPA with Target KPI Metrics
Copyright©2021 ProphetStor
❑ Steep increase/decrease
of consumer replicas not
matching real workloads
❑ Difficult to set right target
thresholds
❑ Many ways to tune HPA
settings for autoscaling
behaviors
Using the ratio of current and
target consumer lag is NOT
the best way to calculate
consumer replicas.
500
Consumer
Lag Target Consumer Replicas Message Production Rate
1000
2000
8

9. Agenda
Scaling Kafka Consumers
Scaling with Kubernetes Native HPA
Intelligent Autoscaling with Predictions
Conclusion

10. 10
Copyright©2021 ProphetStor
Problems with Native HPA
Resource Utilization is not always an indication of
real workload of an application.
What would be the right target threshold to
choose?
Current HPA calculation method does not apply to
all application metrics.
Reactive autoscaling: scaling based on what
currently observed.

11. Federator.ai Intelligent Autoscaling with Workload Prediction
ZooKeeper Service
Producers
Producer 1
Kafka Cluster
Kafka Broker
Topic 1
Partition 0
Partition 1
Topic 2
Partition 0
Consumers
Consumer 1
Consumer
Group
Consumer 2
Consumer 3
Producer 2
• Use message production rate as a
workload indicator
• Make predictions on the workload
• Calculate right number of
consumers based on workload
prediction and target KPI metrics
11
Autoscaling
Message Production Rate
Consumer Lag
Workload Prediction

12. Better Autoscaling Kafka Consumers with Intelligence
12
Copyright©2021 ProphetStor
Consumer Replicas
Production/Consumption Rate
❑ Number of consumer replicas matches the
message production rate
❑ 20% less average number of replicas than
Kubernetes native HPA while meeting
performance target
❑ AI-based algorithm; NO guessing on what the
right target metric threshold should be
❑ Predicted workload helps finding the right
number of replicas at the right time

13. Agenda
Scaling Kafka Consumers
Scaling with Kubernetes Native HPA
Intelligent Autoscaling with Predictions
Conclusion

14. 14
Copyright©2021 ProphetStor
Proactive scaling of consumer pods based on predicted workloads.
Using intelligent algorithm to find pod capacity in terms of workload
handling helps calculating the right number of consumer replicas.
No guessing and no experimenting on what metric threshold to set
in Kubernetes native HPA.
Result in better use of resource for desired performance.
Advantages of Prediction-based Autoscaling for Kafka Consumers

15. Federator.ai – AI-based Solution for Cloud Applications
• Timeseries Forecasting
• Reinforcement Learning
• Predictive Analytics
• Multi-layer correlation
15
• Cost effective resource
recommendations for
applications and clusters
• Prediction-based resource
recommendations ensure
application performance
• Intelligent autoscaling for
dynamic application
workloads
• Automatic resource
provisioning
Copyright©2021 ProphetStor
Dynamic Workload Metrics

16. Thank You!
16
Copyright©2021 ProphetStor