Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020
One of the key metrics to monitor when working with Apache Kafka, as a data pipeline or a streaming platform, is Consumer Groups Lag. Lag is the delta between the last produced message and the last committed message of a partition. In other words, lag indicates how far behind your application is in processing up-to-date information. For a long time, we used our own service to keep track of these metrics, collect them and visualize them. But this didn’t scale well. You had to perform many manual operations, redeploy it and to do other tedious manual tasks, but most importantly, the biggest gap for us, was that its output was represented in absolute numbers (e.g - your lag is 30K), which basically tells you nothing as a human being. We understood that we had to find a more suitable solution that will give us better visibility and will allow us to measure the lag in a time-based format that we all understand. In this talk, I’m going to go over the core concepts of Kafka offsets and lags, and explain why lag even matters and is an important KPI to measure. I’ll also talk about the kind of research we did to find the right tool, what the options in the market were at the time, and eventually why we chose Linkedin’s Burrow as the right tool for us. And finally, I’ll take a closer look at Burrow, its building blocks, how we build and deploy it, how we monitor better with it, and eventually the most important improvement - how we transformed its output from numbers to time-based metrics.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020
Similar to Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020 (20)
More from HostedbyConfluent
More from HostedbyConfluent (20)
Recently uploaded
Recently uploaded (20)
Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020
- 1. Kafka Lag Monitoring For Human Beings Elad Leev Aug 2020 (Sorry robots)
- 2. Elad Leev Platform Engineer Real Time Infrastructure Team Who am I?
- 4. AppsFlyer is a mobile attribution and analytics platform AppsFlyer in a Nutshell
- 5. 1+ Million Incoming HTTP requests/sec 100+ Billion Events per day 20+ Kafka Clusters
- 7. Kafka Is used for building real-time data pipelines and streaming applications
- 8. Offset Is a simple integer that is used by Kafka to maintain the current position of a consumer
- 9. Lag Delta between the last produced message and the last committed offset
- 10. __consumer_offsets Offsets can be stored either in Zookeeper or in a special topic called __consumer_offsets https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
- 11. Zookeeper is not built for high-write load such as offset storage https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
- 12. A consistent, fault tolerant and partitioned way of storing offsets https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management __consumer_offsets
- 13. __consumer_offsets OffsetCommitRequest Broker Broker Broker Group Coordinator Application
- 16. Why Lag Matters?
- 17. Lag is a major KPI when working with Kafka
- 18. Lag indicates how far behind your application is in processing up-to-date information Why Lag Matters? __consumer_offsets
- 19. Lag indicates how far behind your application is in processing up-to-date information Why Lag Matters?
- 20. Kafka persistence is based on retention Why Lag Matters?
- 21. We want to keep the lag of our application to be as small as possible Why Lag Matters?
- 22. How did we previously monitor it?
- 23. How did we previously monitor it? We created a Clojure service Kafka Monitor
- 24. How did we previously monitor it? Parsed the values from Kafka-consumer-group.sh And send it to our metrics stack
- 25. How did we used to monitor it? Hard to maintain Not scalable Small part of the config file
- 26. You get a PagerDuty alert on Kafka lag How did we used to monitor it?
- 27. You want to understand what’s going on in your system How did we used to monitor it?
- 28. You go to our shiny, full of knowledge Grafana dashboard How did we used to monitor it?
- 29. And then you realise that…. How did we used to monitor it?
- 31. How did we used to monitor it? It’s just a number.
- 32. How did we used to monitor it? What the hell does 40K lag mean?
- 33. What we wanted to achieve?
- 34. Automatic No need to change config file Filter consumer groups based on Regex Scalable Small footprint Easy to scale Simple, easy to use Support both ZK and __consumer_offsets topic What we wanted to achieve?
- 35. The “raw” metrics that we looked for are: > Per Partition > Per Consumer Group > Per Topic What we wanted to achieve?
- 36. What are the options?
- 37. Linkedin - Burrow > A LinkedIn Project > More than 2.5K stars > Active community > Production ready Lightbend - Kafka Lag Exporter > Smart > Time Based > Still in Beta Zalando - Remora > Inspired by Burrow > CloudWatch & DataDog integration > Wrap around Kafka CLI What are the options?
- 38. Burrow
- 39. Burrow is a monitoring solution for Apache Kafka that provides consumer lag checking as a service Burrow
- 40. Burrow has a modular design Burrow
- 41. Cluster / Consumers Kafka client that periodically update cluster information Burrow
- 43. Evaluator Calculate the status of each group Burrow
- 44. Notifier Requests status on consumer groups and notifies Burrow
- 45. HTTP Server Provides an API Interface for Burrow Burrow
- 46. Associated Projects Burrow Burrow UI Burrow Dashboard
- 47. How do we use it?
- 49. Deployment Process Clone Burrow Source Trigger Gitlab Ci Config file (.toml) Linter Check that all hosts are resolvable Build Burrow container Deploy using in-house deployment system
- 50. Burrow Dashboard
- 51. Burrow Dashboard
- 52. Burrow Dashboard Lag By Number Produce vs. Consume Producer Rate Consumer Rate
- 54. And the endgame
- 55. Burrow Dashboard Time Based Metrics
- 56. Time Lag - How did we do it? Diff ( Last_Consumed , Last_Produced ) Producer Rate
- 57. Time Lag - How did we do it? Timeline Time: 12:00AM Msg_offset: 134 Time: 12:10AM Msg_offset: 144 Time: 12:20AM Msg_offset: 154 Consumer Producer Lag
- 58. What’s next?
- 59. Smart Alerts Dynamic alerts based on lag and retention Decoupling As we grow, Burrow will be deployed per cluster Migration Migrating crucial part of the infrastructure is hard What’s next?