Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

1

Share

Download to read offline

Rootconf

Download to read offline

How LinkedIn is leveraging kafka for log aggregation and monitoring

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Rootconf

  1. 1. How LinkedIn is leveraging kafka for log aggregation and monitoring Akbar KM, Muzammil Rehman SRE Linkedin
  2. 2. Agenda ● What is Pub Sub? ● Kafka basics ● Kafka for monitoring ● inGraph’s ● ELK at Linkedin
  3. 3. What is pub sub ? Producer Consumer Producer Consumer Topic 1 Topic 2 Topic 3 subscribe publish(topic, msg) Publish subscribe system msg msg
  4. 4. Kafka
  5. 5. Consumer Groups
  6. 6. inGraph ● inGraphs is a visualization frontend for a linkedin’s metrics ● Agent is a library used by all java application ● Data is transported over kafka ● Collector consume and write to rrd ● Features ○ Rest based ○ Self service ○ Little over 500,000 metrics collected per minute ○ Each RRD is roughly 815k. Each RRD is written to two collectors to maintain data integrity ○ We currently have 1.4TB of SSD in production colo
  7. 7. ELK at linkedin ● ELK (Elasticsearch, Logstash, Kibana) ○ E -> Near Real-Time Search & Analysis ○ L -> Plumbing and Glue for all your data ○ K -> Line graphs, pie charts, dashboard ● Logstash kafka plugin consumes logs and end up written into elasticsearch
  8. 8. Why Kafka 1. Near real time delivery 2. Supports multiple consumers groups aka queuing 3. No overhead on clients that comes up logstash agent 4. Kafka scales horizontally 5. Supports REST out of the box 6. In house support
  9. 9. Reference http://engineering.linkedin.com/52/autometrics- self-service-metrics-collection ELK http://www.elastic.co/guide/
  • HashFyre

    Apr. 28, 2015

How LinkedIn is leveraging kafka for log aggregation and monitoring

Views

Total views

532

On Slideshare

0

From embeds

0

Number of embeds

2

Actions

Downloads

6

Shares

0

Comments

0

Likes

1

×