Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Tuning Kafka for Fun and Profit

4,960 views

Published on

This presentation was given at the ApacheCon 2015 Kafka Meetup.

These slides go into some detail on how to tune and scale Kafka clusters and the components involved. The slides themselves are bullet points, and all the detail is in the slide notes, so please download the original presentation and review those.

Published in: Data & Analytics

Tuning Kafka for Fun and Profit

  1. 1. ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. Tuning Kafka for Fun and Profit
  2. 2. ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. Zookeeper  5-node vs. 3-node Ensembles  Solid State Disks – Use good SSDs – Transaction logs only – Significant improvement in latency and outstanding requests 2
  3. 3. ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. Kafka Broker Disks  Disk Layout  JBOD vs. RAID – JBOD and RAID-0 are similar – RAID-5/6 has significant performance overhead – RAID-10 still offers the best performance and protection  Filesystem – New testing shows XFS has a clear benefit – No tuning required – Will be continuing testing with more production traffic 3
  4. 4. ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. Scaling Kafka Clusters  Disk Capacity  Network Capacity  Partition Counts – Per-Cluster – Per-Broker  Limitations – Topic list length 4
  5. 5. ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. Topic Configuration  Retention Settings  Partition Counts – Balance over consumers – Balance over brokers – Partition size on disk – Application-specific requirements 5
  6. 6. ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. Mirror Maker  Network Locality  Consumer Tuning – Number of streams – Partition assignment strategy  Producer Tuning – Number of streams – In flight requests – Linger time 6

×