Download as PDF, PPTX
Uploaded byHostedbyConfluent
Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020
The document discusses Kafka lag monitoring, emphasizing the need for effective monitoring solutions due to the complexities of handling high volumes of data. It introduces Burrow, a monitoring tool for Apache Kafka that provides consumer lag checking and features a modular design. The document highlights the challenges faced in previous monitoring methods and outlines future steps including smart alerts and dynamic monitoring as systems scale.
More Related Content
Similar to Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020
Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020
- 1. Kafka Lag Monitoring For HumanBeings Elad Leev Aug 2020 (Sorry robots)
- 2. Elad Leev Platform Engineer RealTime Infrastructure Team Who am I?
- 3.
- 4. AppsFlyer is amobile attribution and analytics platform AppsFlyer in a Nutshell
- 5. 1+ Million Incoming HTTP requests/sec 100+ Billion Events perday 20+ Kafka Clusters
- 6.
- 7. Kafka Is used forbuilding real-time data pipelines and streaming applications
- 8. Offset Is a simpleinteger that is used by Kafka to maintain the current position of a consumer
- 9. Lag Delta between thelast produced message and the last committed offset
- 10. __consumer_offsets Offsets can bestored either in Zookeeper or in a special topic called __consumer_offsets https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
- 11. Zookeeper is not builtfor high-write load such as offset storage https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
- 12. A consistent, fault tolerantand partitioned way of storing offsets https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management __consumer_offsets
- 13. __consumer_offsets OffsetCommitRequest Broker Broker Broker GroupCoordinator Application
- 14.
- 15.
- 16.
- 17. Lag is a majorKPI when working with Kafka
- 18. Lag indicates how farbehind your application is in processing up-to-date information Why Lag Matters? __consumer_offsets
- 19. Lag indicates how farbehind your application is in processing up-to-date information Why Lag Matters?
- 20. Kafka persistence is basedon retention Why Lag Matters?
- 21. We want tokeep the lag of our application to be as small as possible Why Lag Matters?
- 22. How did we previouslymonitor it?
- 23. How did wepreviously monitor it? We created a Clojure service Kafka Monitor
- 24. How did wepreviously monitor it? Parsed the values from Kafka-consumer-group.sh And send it to our metrics stack
- 25. How did weused to monitor it? Hard to maintain Not scalable Small part of the config file
- 26. You get a PagerDuty alerton Kafka lag How did we used to monitor it?
- 27. You want to understand what’sgoing on in your system How did we used to monitor it?
- 28. You go toour shiny, full of knowledge Grafana dashboard How did we used to monitor it?
- 29. And then yourealise that…. How did we used to monitor it?
- 31. How did weused to monitor it? It’s just a number.
- 32. How did weused to monitor it? What the hell does 40K lag mean?
- 33. What we wanted toachieve?
- 34. Automatic No need tochange config file Filter consumer groups based on Regex Scalable Small footprint Easy to scale Simple, easy to use Support both ZK and __consumer_offsets topic What we wanted to achieve?
- 35. The “raw” metricsthat we looked for are: > Per Partition > Per Consumer Group > Per Topic What we wanted to achieve?
- 36. What are theoptions?
- 37. Linkedin - Burrow >A LinkedIn Project > More than 2.5K stars > Active community > Production ready Lightbend - Kafka Lag Exporter > Smart > Time Based > Still in Beta Zalando - Remora > Inspired by Burrow > CloudWatch & DataDog integration > Wrap around Kafka CLI What are the options?
- 38.
- 39. Burrow is a monitoringsolution for Apache Kafka that provides consumer lag checking as a service Burrow
- 40. Burrow has a modulardesign Burrow
- 41. Cluster / Consumers Kafkaclient that periodically update cluster information Burrow
- 42.
- 43. Evaluator Calculate the statusof each group Burrow
- 44. Notifier Requests status on consumergroups and notifies Burrow
- 45. HTTP Server Provides anAPI Interface for Burrow Burrow
- 46.
- 47. How do weuse it?
- 48.
- 49. Deployment Process Clone Burrow Source Trigger GitlabCi Config file (.toml) Linter Check that all hosts are resolvable Build Burrow container Deploy using in-house deployment system
- 50.
- 51.
- 52. Burrow Dashboard Lag ByNumber Produce vs. Consume Producer Rate Consumer Rate
- 53.
- 54.
- 55. Burrow Dashboard Time BasedMetrics
- 56. Time Lag -How did we do it? Diff ( Last_Consumed , Last_Produced ) Producer Rate
- 57. Time Lag -How did we do it? Timeline Time: 12:00AM Msg_offset: 134 Time: 12:10AM Msg_offset: 144 Time: 12:20AM Msg_offset: 154 Consumer Producer Lag
- 58.
- 59. Smart Alerts Dynamic alertsbased on lag and retention Decoupling As we grow, Burrow will be deployed per cluster Migration Migrating crucial part of the infrastructure is hard What’s next?
- 60.