The document discusses Kafka monitoring at LinkedIn. It provides background on Kafka including brokers, topics, partitions, producers and consumers. It then discusses the history of Kafka monitoring at LinkedIn, including the many host, JVM and broker metrics they collect. Key metrics to monitor include under replicated partitions, offline partitions, active controller count, and partitions with replication below the minimum threshold. The future of monitoring at LinkedIn will focus on service level objectives and external availability monitoring. Monitoring collects all data while alerting on very few, key metrics.

1. Apache Kafka
Monitoring vs Alerting
Ratish Ravindran
Sr. Site Reliability Engineer - LinkedIn

2. Agenda
Kafka 101
History of Kafka monitoring at LinkedIn
Key Metrics to alert on and why
Future of Kafka monitoring at LinkedIn
Q&A

3. Kafka 101

4. Brokers and
Clusters
● Single Kafka Server
● Multiple brokers

5. Topics and
Partitions
● Categorized messages
● Broken down into partitions

6. Leader and
Replica
● Broker owning partition
● Broker with partition copy

7. Producer and
Consumer

8. Controller
● Broker with more responsibilities
● Partition management
• Leader election
• Leader Switch
• New topic and partition
• New broker

9. History of Kafka Monitoring
at LinkedIn

10. Common Metrics
● Host metrics
● JVM metrics
● Broker or Application metrics

11. Host Metrics
● Load average
● Memory usage (free, used, %)
● CPU
● Inbound/Outbound network stats
● Disk usage
● RAID stats
● Inode (free, used, %)
● IO stats
● Disk states
● TCP connections (est. count, error etc)
~ 320 host metrics

12. JVM Metrics
● Minor/Full GC time
● Minor/Full GC count
● Total thread count
● Used thread count
● Total heap size available
● Total heap size used

13. Broker Metrics
● Bytes and messages in/out rate
● Fetch/produce queue size
● Request handler idle ratio
● Request pool usage
● Request latencies – at different stages
● Log flush rate
● Produce and consume latency
2100 broker metrics and 165 graphs

14. Kafka at LinkedIn

15. Kafka at LinkedIn
● 2500+ Kafka brokers
● 1 PB In per day
● 3.9 PB Out per day

16. Kafka Key Metrics

17. Kafka as a service
● Broker/Cluster Health
● Message Delivery
● Performance
● Capacity

18. Metrics you need to know
Partitions that are
not fully replicated
within the cluster
Partitions
unavailable for
produce and
consume
Should always be 1
URP Offline
Partitions
Active Controller
Count of partitions
with ISR < MinISR
Under MinISR
Count

19. Under Replicated Partitions
● Highly discussed
● Overall cluster health
● Replication is consumer and producer

20. Under Replicated Partitions

21. Under Replicated Partitions

22. Under Replicated Partitions

23. Metrics you need to know
Partitions that are
not fully replicated
within the cluster
Partitions
unavailable for
produce and
consume
Should always be 1
URP Offline
Partitions
Active Controller
Count of partitions
with ISR < MinISR
UnderMinIsrP
artitionCount

24. Offline partition count
● Partition(s) unavailable
● All brokers hosting replica down OR
● unclean.leader.election.enabled=false
● Potential data loss

25. Metrics you need to know
Partitions that are
not fully replicated
within the cluster
Partitions
unavailable for
produce and
consume
Should always be 1
URP Offline
Partitions
Active Controller
Count of partitions
with ISR < MinISR
UnderMinIsrP
artitionCount

26. Active Controller Count
● What is Kafka controller ?
● Partition management
● There should only 1 controller

27. Active Controller Count < 1

28. Active Controller Count > 1

29. Metrics you need to know
Partitions that are
not fully replicated
within the cluster
Partitions
unavailable for
produce and
consume
Should always be 1
URP Offline
Partitions
Active Controller
Count of partitions
with ISR < MinISR
UnderMinIsrP
artitionCount

30. UnderMinIsrPartitionCount
● min.insync.replicas
● in-sync replicas count < min.insync.replicas
● Per-broker metrics

31. UnderMinIsrPartitionCount
Screen Shot 2018-06-18 at 3.12.17 PMOffline partitions URP
Under MinIsr Partition count

32. Operating System
and Hardware
Metrics
● Should I be worried ?
● What application is causing it ?
● Don’t alert unless:
● 100% clear signal
● 100% actionable

33. Capacity
Planning
● Plan in advance
● Multi-factor
● Don’t alert for capacity

34. Capacity
Metrics
● Request Handler Idle Ratio
● Disk Utilization
● Partition Count
● Network Utilization

35. Future of Kafka Monitoring at
LInkedIn

36. Future of Monitoring Kafka at LinkedIn
● SLO based
● Monitor and alert on SLO metrics
• Latency – Produce and consume
• Availability – Produce and consume
• Retention

37. Availability
Monitoring
● Measured externally
● Client focused
● github.com/linkedin/kafka-monitor

38. Conclusion

39. Monitoring is not alerting
● Collect everything
● Alert on almost nothing
● SLO based monitoring

40. Getting (and Giving) Help
• Community
users@kafka.apache.org
dev@kafka.apache.org
• Bugs and Work
https://issues.apache.org/jira/projects/KAFKA
• Kafka Monitor
https://github.com/linkedin/kafka-monitor
• Burrow
https://github.com/linkedin/Burrow
• Cruise Control
https://github.com/linkedin/cruise-control
• kafka-tools
https://github.com/linkedin/kafka-tools

41. Questions