• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Apache Kafka Lightning Talk
 

Apache Kafka Lightning Talk

on

  • 511 views

This short deck provides a high-level overview of how Apache Kafka works under the covers. It covers logs, topics partitions, consumer groups, and replication from a conceptual perspective.

This short deck provides a high-level overview of how Apache Kafka works under the covers. It covers logs, topics partitions, consumer groups, and replication from a conceptual perspective.

Statistics

Views

Total Views
511
Views on SlideShare
511
Embed Views
0

Actions

Likes
1
Downloads
10
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Apache Kafka Lightning Talk Apache Kafka Lightning Talk Presentation Transcript

    • Q Q Q Q Q Q Q Tuesday, December 3, 13 Q Q Q Q Q Q Q Q Q Q Q QQ Q Q Q Jeff Kunkle Nov 22, 2013 Q Q Q Q Q Q Q Q Q Apache Kafka Q Q Q Q Q Q Q Q Q Q Q Q Q Q
    • Q Q Q Q Q Q Q Tuesday, December 3, 13 Q Q Q Q Q Q Q Q Q Q Q QQ Q Q Q Q Q Q Q Q Q Q Q Q Q What is Kafka? Q Q Q Q Q Q Q Q Q Q Q Q Q Q
    • Kafka is a distributed, partitioned, replicated commit log service. Tuesday, December 3, 13
    • Tuesday, December 3, 13
    • ? Tuesday, December 3, 13
    • ... a non-JMS messaging system with a unique design. Tuesday, December 3, 13
    • Key Concepts 1 2 Consumer Groups 3 Tuesday, December 3, 13 Topics, Logs, and Partitions Replication
    • Topics and Logs Tuesday, December 3, 13
    • Topics and Logs partition 0 partition 1 partition 2 Tuesday, December 3, 13
    • Topics and Logs log partition 0 1 2 3 4 5 6 7 log partition 1 1 2 3 4 5 6 log partition 2 1 old Tuesday, December 3, 13 2 3 4 5 new
    • Topics and Logs log partition 0 1 2 3 4 5 6 7 log partition 1 1 2 3 4 5 writes 6 log partition 2 1 old Tuesday, December 3, 13 2 3 4 5 new
    • partition 0 partition 1 P2 5 2 1 3 4 partition 2 Tuesday, December 3, 13 Producers P1 2 6 5 4 3 1 7
    • partition 0 1 2 partition 1 1 P2 5 2 3 4 partition 2 Tuesday, December 3, 13 1 Producers P1 2 6 5 4 3 7
    • bulk publishing partition 0 1 2 partition 1 1 P2 5 2 3 4 partition 2 Tuesday, December 3, 13 1 Producers P1 2 6 5 4 3 7
    • bulk publishing partition 0 1 2 partition 1 1 P2 5 partition 2 Tuesday, December 3, 13 1 2 3 4 Producers P1 2 6 5 4 3 7
    • parallel publishing partition 0 1 2 partition 1 1 P2 5 partition 2 Tuesday, December 3, 13 1 2 3 4 Producers P1 2 6 5 4 3 7
    • parallel publishing partition 0 1 2 3 4 partition 1 1 2 P2 5 partition 2 Tuesday, December 3, 13 1 2 3 4 Producers P1 4 3 6 5 7
    • partition 0 1 2 3 4 5 6 7 partition 1 1 2 3 4 5 6 P2 partition 2 Tuesday, December 3, 13 1 2 3 4 5 Producers P1
    • consumer group 1 Consumers C1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 7 partition 0 C2 C3 C4 consumer group 2 Tuesday, December 3, 13 1 partition 1 partition 2
    • consumer group 1 Consumers 1 C1 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 7 partition 0 C2 C3 C4 1 consumer group 2 Tuesday, December 3, 13 2 partition 1 partition 2
    • consumer group 1 Consumers 1 C1 3 4 5 6 2 3 4 5 6 2 3 4 5 7 partition 0 C2 1 C3 C4 1 consumer group 2 Tuesday, December 3, 13 2 1 partition 1 partition 2
    • consumer group 1 Consumers 1 C1 3 4 5 6 2 3 4 5 6 2 3 4 5 7 partition 0 1 C2 1 C3 C4 1 consumer group 2 Tuesday, December 3, 13 2 partition 1 partition 2
    • consumer group 1 Consumers 1 3 C1 4 5 6 4 5 6 4 5 7 partition 0 1 2 C2 1 2 C3 C4 3 2 1 consumer group 2 Tuesday, December 3, 13 3 3 partition 1 partition 2
    • consumer group 1 Consumers 1 3 C1 3 4 5 6 4 5 6 4 5 7 partition 0 1 2 C2 1 2 C3 C4 3 2 1 consumer group 2 Tuesday, December 3, 13 bulk consumption 3 partition 1 partition 2
    • consumer group 1 bulk consumption Consumers 1 3 C1 partition 0 1 2 C2 6 4 3 5 4 1 2 C3 C4 6 5 4 3 2 1 consumer group 2 Tuesday, December 3, 13 7 3 5 4 5 6 partition 1 partition 2
    • consumer group 1 parallel consumption Consumers 1 3 C1 partition 0 1 2 C2 6 4 3 5 4 1 2 C3 C4 6 5 4 3 2 1 consumer group 2 Tuesday, December 3, 13 7 3 5 4 5 6 partition 1 partition 2
    • consumer group 1 parallel consumption Consumers 1 3 5 4 C1 partition 0 1 2 C2 6 4 3 5 6 1 2 C3 3 C4 6 5 4 3 2 1 consumer group 2 Tuesday, December 3, 13 7 4 5 partition 1 partition 2
    • consumer group 1 Consumers 1 3 5 4 C1 1 2 C2 6 4 3 5 1 2 C3 3 7 4 5 C4 7 6 5 4 3 2 1 consumer group 2 Tuesday, December 3, 13 partition 0 partition 1 partition 2
    • Replication (all reads and writes) server A A master P1 C5 A7 A6 A4 A3 A2 A1 B4 B3 B2 B1 (all reads and writes) (all reads and writes) Tuesday, December 3, 13 server C C master server B B master P2 C1 A5 C2 C3 C4 B6 B5
    • Replication (all reads and writes) A1 server A A master P1 C5 A7 A6 A4 A3 A2 B4 B3 B2 B1 (all reads and writes) (all reads and writes) Tuesday, December 3, 13 server C C master server B B master P2 C1 A5 C2 C3 C4 B6 B5
    • Replication (all reads and writes) A1 server A A master P1 C5 A7 A6 A4 A3 A2 B4 B3 B2 B1 (all reads and writes) A1 (all reads and writes) Tuesday, December 3, 13 server C C master server B B master P2 C1 A5 C2 C3 C4 B6 B5
    • Replication (all reads and writes) A1 A2 server A A master P1 C5 A7 A6 A4 A3 B4 B3 B2 B1 (all reads and writes) A1 A2 (all reads and writes) Tuesday, December 3, 13 server C C master server B B master P2 C1 A5 C2 C3 C4 B6 B5
    • Replication (all reads and writes) A1 A2 C1 server A A master P1 C5 A7 A6 A4 A3 B4 B3 B2 B1 (all reads and writes) A1 A2 (all reads and writes) Tuesday, December 3, 13 C1 server C C master server B B master P2 A5 C2 C3 C4 B6 B5
    • Replication (all reads and writes) server A A master A1 A2 A3 A4 C1 C2 C3 C4 C5 P1 A7 A6 (all reads and writes) B1 B2 B3 B4 B5 B6 A1 A2 A3 A4 (all reads and writes) C1 C2 C3 C4 C5 B1 B2 B3 B4 B5 Tuesday, December 3, 13 B6 server C C master server B B master P2 A5
    • Replication (all reads and writes) A1 A2 A3 A4 A5 A6 A7 C1 C2 C3 C4 C5 server A A master P1 (all reads and writes) B1 B2 B3 B4 B5 B6 A1 A2 A3 A4 (all reads and writes) C1 C2 C3 C4 C5 B1 B2 B3 B4 B5 Tuesday, December 3, 13 B6 server C C master server B B master P2
    • Replication (all reads and writes) A1 A2 A3 A4 A5 A6 A7 C1 C2 C3 C4 C5 server A A master P1 (all reads and writes) B1 B2 B3 B4 B5 B6 A1 A2 A3 A4 A5 A6 A7 (all reads and writes) C1 C2 C3 C4 C5 B1 B2 B3 B4 B5 Tuesday, December 3, 13 B6 server C C master server B B master P2
    • 123 Messages in a partition are assigned a sequential id number, referred to as the offset. Consumers keep track of their consumed offset in the partition. Kafka doesn’t maintain any metadata. Kafka maintains all messages for a configurable time, regardless of whether they’ve been consumed. Tuesday, December 3, 13
    • Alternatives http://queues.io Tuesday, December 3, 13
    • Questions? Tuesday, December 3, 13