"A key operational challenge for running Kafka in production is managing Kafka Partition Lag. Kafka partitions exhibit a variety of normal trends, influenced by how consumers consume data in partitions. Kafka Lag also exhibits abnormal patterns caused by issues in the Kafka clusters or in its consumers. Administrators need to monitor Kafka Lag, distinguish between normal and abnormal trends and act when application outcomes are impacted. Lag impacts latency and accuracy of data and insights produced from a Big Data pipeline. How can we continuously monitor Kafka Lag automatically, identify normal and abnormal trends and forecast Lag issues ahead of time? In this session, we will discuss our work in this regard using machine learning. We will discuss popular lag patterns and how our ensemble forecasting system learns from the past and predicts future trends. We will also showcase some case studies and benefits of having such a system as part of a Kafka observability platform."

1. Forecasting Kafka Lag
with Machine Learning
Kumaran Ponnambalam
Principal Engineer - AI

2. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
Speaker Intro
2
• AI/ML Leader & Author
• 20+ years in Data, Analytics, AI/ML
• Current
• Principal Engineer – AI
• Outshift by Cisco
• Author
• LinkedIn Learning – AI/ML & Big Data
• Apache Kafka Essential Training – Getting Started
• Apache Kafka Essential Training – Building scalable apps
Kumaran Ponnambalam

3. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Agenda
• The Kafka consumer lag problem
• Need for lag forecasting
• Current approaches to lag forecasting
• Using Retrieval Augmented Generation
• Our approach
• Examples of real time forecasting

4. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Confidential.
Kafka consumer lag

5. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
• One topic – multiple consumer groups
• Consumers consume data independently of producers
• Lag indicates ”pending” records not yet consumed by a consumer
group
• Lag can happen by design (batch consumers) or because the
consumers are not able to keep up with the producers
• Lag directly impacts latency of data processing pipelines & hence
when outcomes are achieved
The Kafka consumer lag problem

6. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
• Pending records keeps increasing over time
• Delays in achieving desired application service levels
• Business impact
• A real-time threat alert system will delay alerts, leading to threat becoming
real
• A real-time recommendation system may not provide recommendations in
time, leading to loss of business
• Restoring service levels is involved and takes time
• Increasing capacity of brokers / consumers
• Resetting consumer offsets
Impact of unplanned lag

7. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Popular lag patterns

8. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
• Each consumer group has a steady-state consumption pattern
• Repeating steady-state patterns over time
• Keeps lag closer to zero or a steady value
• Hard failures are easy to track, but lag creep is not
• Increasing load vs limited capacity
• Soft failures across systems/ infrastructure
• When lag happens, will the consumer will eventually catch up?
• Steady state and exception patterns do exist
• Patterns vary based on individual consumer group
Trends observed in consumer lag

9. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Confidential.
Forecasting Kafka Lag patterns

10. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
• So, we know if the consumer group won’t catch up eventually
• We will know ahead of time !
• We can mitigate the problem
• We could possibly prevent it !
• Have time to implement corrective actions (add nodes etc.)
• Minimize business impact
• Mitigate before customers notice
• Avoid loss of revenue or trust
Why forecast Kafka Lag?

11. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
• A pattern represents values for lag over
time
• For forecasting, we split the pattern into a
key and a value
• Given a Query (key) can we forecast the
value ?
• In real time, at specified intervals (say every
1 min), we can get the last X data points
(key) and forecast the next Y data points (
value)
Forecasting Lag
Key Value
Forecasting

12. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Current approaches to Kafka lag forecasting
ARIMA
Moving average computation
to forecast time series
Easy to implement and
provide good forecasts on
repeating patterns
Cannot handle non-linear
relationships and sudden
shocks
ML Sequence Models
Use a deep learning model
to forecast next pattern in
time series
Can model complex patterns
and relationships, especially
exceptions
Expensive, especially when
prediction need to be done
in real time for 5 sec. or so
Knowledge Base of previous
patterns
Use a knowledge base of
previously observed patterns
and the eventual result
Easy to manage and low
cost for lookups
Need prior patterns for
consumer group; Cannot
forecast new trends

13. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Generative AI for forecasting
• Generative AI has taken the world by storm this year
• Applied to several use cases for machine learning
• Forecasting sequences of numbers is also a “generation” use case
• While Gen AI is popular with text & images, applications for numeric data also exist
• Generative models can be built to understand sequence patterns and forecast
into the future

14. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Retrieval Augmented Generation (RAG)
• Popular technique in Generative AI for creating knowledge-based applications (
e.g.: Question-Answering )
• Uses a combination of
• A Deep Learning model that can generate outputs (patterns, text, images) based on its learning
• A knowledge base that stores information about a specific domain /context.
• Using a knowledge base constrains the output to the specific domain. It also
saves significant costs than using a deep learning model as a knowledge store
• The deep learning model compliments the knowledge base to generate output

15. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Forecast
Engine
Pattern
Matching
Service
Span
Estimation
Service
Pattern
DB
Estimation
Models
Clients
Forecasting Kafka lag with RAG
Built based on
previously observed
patterns for a given
Kafka topic
consumer
Ensemble of models,
trained on previously
observed patterns
across Kafka topic
consumers
Retrieval based on similarity
between current lag
observed and patterns in DB
Choose best forecast
(generation) from
predictions of the model
ensemble
Create forecast from
retrieved patterns +
generated patterns based
on native heuristics
Send current observed
lag pattern and fetch
forecast for immediate
future

16. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Forecasting with RAG
• Pattern DB
• Built based on past observed data
• Provides steady state lag patterns (and exception patterns observed previously)
• Usually sufficient to forecast lag during steady state operations
• Cheap
• Ensemble of estimation models
• Provides general patterns, especially unknown ones for the specific consumer
• Only used when pattern DB has no good matches ( high initially, exceptions only later)
• Expensive, but helps model unknown scenarios

17. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Forecasting with RAG (contd.)
• Forecast Engine
• Smart engine that picks pattern DB vs ensemble models based on quality of predictions
• Pattern DB first
• Uses ensemble models only when pattern DB does not return high quality results
• Uses heuristics & ML to make decisions
• Auto ML for training
• Kafka lag observed for a given consumer-group and used to build pattern DB/ models
• Pattern DB updated when new patterns are observed
• Models retrained when new patterns / drift are observed

18. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Confidential.
Real time forecasting examples

19. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Example 1

20. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Example 2

21. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Example 3

22. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Using lag forecasts
• Setup thresholds based on expected consumer behavior
• Automate by querying forecast at regular intervals
• Use the predicted sequences / trend line to forecast expected lag values in the
future
• Monitor predictions over time to ensure that the trend stays consistent
• Trigger alerts / actions when trend points to threshold violations in the future

23. © 2023 Outshift by Cisco and/or its affiliates. All rights reserved. Outshift by Cisco Public.
Future work
• Add ARIMA based models too, to have a 3-dimensional forecasting engine
• Build a Kafka plugin to observe lag by consumer-group and publish to forecast
engine
• Add User interfaces for provisioning and analytics
• Triggers for exceptions

24. Thank You !