Q

Q

Q

Q

Q

Q

Q

Tuesday, December 3, 13

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

QQ

Q

Q
Q

Jeff Kunkle
Nov 22, 2013
Q

Q
Q...
Q

Q

Q

Q

Q

Q

Q

Tuesday, December 3, 13

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

QQ

Q

Q
Q

Q
Q

Q
Q

Q

Q

Q
Q

Q

Q

What...
Kafka is a distributed,
partitioned, replicated
commit log service.
Tuesday, December 3, 13
Tuesday, December 3, 13
?
Tuesday, December 3, 13
... a non-JMS messaging
system with a unique design.

Tuesday, December 3, 13
Key Concepts
1
2

Consumer Groups

3

Tuesday, December 3, 13

Topics, Logs, and Partitions
Replication
Topics and Logs

Tuesday, December 3, 13
Topics and Logs
partition 0

partition 1

partition 2

Tuesday, December 3, 13
Topics and Logs
log

partition 0

1

2

3

4

5

6

7

log

partition 1

1

2

3

4

5

6

log

partition 2

1

old
Tuesda...
Topics and Logs
log

partition 0

1

2

3

4

5

6

7

log

partition 1

1

2

3

4

5

writes

6

log

partition 2

1

ol...
partition 0

partition 1
P2
5
2
1
3
4

partition 2

Tuesday, December 3, 13

Producers

P1
2
6
5
4
3
1
7
partition 0

1

2

partition 1

1
P2
5
2
3
4

partition 2

Tuesday, December 3, 13

1

Producers

P1
2
6
5
4
3
7
bulk publishing
partition 0

1

2

partition 1

1
P2
5
2
3
4

partition 2

Tuesday, December 3, 13

1

Producers

P1
2
6
5...
bulk publishing
partition 0

1

2

partition 1

1
P2
5

partition 2

Tuesday, December 3, 13

1

2

3

4

Producers

P1
2
...
parallel publishing
partition 0

1

2

partition 1

1
P2
5

partition 2

Tuesday, December 3, 13

1

2

3

4

Producers

P...
parallel publishing
partition 0

1

2

3

4

partition 1

1

2
P2
5

partition 2

Tuesday, December 3, 13

1

2

3

4

Pro...
partition 0

1

2

3

4

5

6

7

partition 1

1

2

3

4

5

6
P2

partition 2

Tuesday, December 3, 13

1

2

3

4

5

P...
consumer group 1

Consumers

C1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

7

partition 0

C2
C3

C4
consumer group ...
consumer group 1

Consumers

1
C1

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

7

partition 0

C2
C3

C4
1
consumer group...
consumer group 1

Consumers

1
C1

3

4

5

6

2

3

4

5

6

2

3

4

5

7

partition 0

C2
1
C3

C4
1
consumer group 2

...
consumer group 1

Consumers

1
C1

3

4

5

6

2

3

4

5

6

2

3

4

5

7

partition 0

1
C2
1
C3

C4
1
consumer group 2...
consumer group 1

Consumers

1
3
C1

4

5

6

4

5

6

4

5

7

partition 0

1
2
C2
1
2
C3

C4
3
2
1
consumer group 2

Tue...
consumer group 1

Consumers

1
3
C1

3

4

5

6

4

5

6

4

5

7

partition 0

1
2
C2
1
2
C3

C4
3
2
1
consumer group 2

...
consumer group 1

bulk consumption

Consumers

1
3
C1

partition 0

1
2
C2
6
4
3
5
4

1
2
C3

C4
6
5
4
3
2
1
consumer grou...
consumer group 1

parallel consumption

Consumers

1
3
C1

partition 0

1
2
C2
6
4
3
5
4

1
2
C3

C4
6
5
4
3
2
1
consumer ...
consumer group 1

parallel consumption

Consumers

1
3
5
4
C1

partition 0

1
2
C2
6
4
3
5
6

1
2
C3
3

C4
6
5
4
3
2
1
con...
consumer group 1

Consumers

1
3
5
4
C1
1
2
C2
6
4
3
5
1
2
C3
3
7
4
5

C4
7
6
5
4
3
2
1
consumer group 2

Tuesday, Decembe...
Replication
(all reads and writes)

server A

A master

P1
C5
A7
A6
A4
A3
A2
A1
B4
B3
B2
B1
(all reads and writes)

(all r...
Replication
(all reads and writes)

A1

server A

A master

P1
C5
A7
A6
A4
A3
A2
B4
B3
B2
B1
(all reads and writes)

(all ...
Replication
(all reads and writes)

A1

server A

A master

P1
C5
A7
A6
A4
A3
A2
B4
B3
B2
B1
(all reads and writes)

A1

(...
Replication
(all reads and writes)

A1 A2

server A

A master

P1
C5
A7
A6
A4
A3
B4
B3
B2
B1
(all reads and writes)

A1 A2...
Replication
(all reads and writes)

A1 A2
C1

server A

A master

P1
C5
A7
A6
A4
A3
B4
B3
B2
B1
(all reads and writes)

A1...
Replication
(all reads and writes)

server A

A master

A1 A2 A3 A4
C1 C2 C3 C4 C5

P1
A7
A6
(all reads and writes)

B1 B2...
Replication
(all reads and writes)

A1 A2 A3 A4 A5 A6 A7
C1 C2 C3 C4 C5

server A

A master

P1
(all reads and writes)

B1...
Replication
(all reads and writes)

A1 A2 A3 A4 A5 A6 A7
C1 C2 C3 C4 C5

server A

A master

P1
(all reads and writes)

B1...
123

Messages in a partition are assigned a sequential id
number, referred to as the offset.

Consumers keep track of thei...
Alternatives
http://queues.io

Tuesday, December 3, 13
Questions?

Tuesday, December 3, 13
Upcoming SlideShare
Loading in …5
×

Apache Kafka Lightning Talk

1,132 views

Published on

This short deck provides a high-level overview of how Apache Kafka works under the covers. It covers logs, topics partitions, consumer groups, and replication from a conceptual perspective.

Published in: Technology, Self Improvement
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,132
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
19
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Apache Kafka Lightning Talk

  1. 1. Q Q Q Q Q Q Q Tuesday, December 3, 13 Q Q Q Q Q Q Q Q Q Q Q QQ Q Q Q Jeff Kunkle Nov 22, 2013 Q Q Q Q Q Q Q Q Q Apache Kafka Q Q Q Q Q Q Q Q Q Q Q Q Q Q
  2. 2. Q Q Q Q Q Q Q Tuesday, December 3, 13 Q Q Q Q Q Q Q Q Q Q Q QQ Q Q Q Q Q Q Q Q Q Q Q Q Q What is Kafka? Q Q Q Q Q Q Q Q Q Q Q Q Q Q
  3. 3. Kafka is a distributed, partitioned, replicated commit log service. Tuesday, December 3, 13
  4. 4. Tuesday, December 3, 13
  5. 5. ? Tuesday, December 3, 13
  6. 6. ... a non-JMS messaging system with a unique design. Tuesday, December 3, 13
  7. 7. Key Concepts 1 2 Consumer Groups 3 Tuesday, December 3, 13 Topics, Logs, and Partitions Replication
  8. 8. Topics and Logs Tuesday, December 3, 13
  9. 9. Topics and Logs partition 0 partition 1 partition 2 Tuesday, December 3, 13
  10. 10. Topics and Logs log partition 0 1 2 3 4 5 6 7 log partition 1 1 2 3 4 5 6 log partition 2 1 old Tuesday, December 3, 13 2 3 4 5 new
  11. 11. Topics and Logs log partition 0 1 2 3 4 5 6 7 log partition 1 1 2 3 4 5 writes 6 log partition 2 1 old Tuesday, December 3, 13 2 3 4 5 new
  12. 12. partition 0 partition 1 P2 5 2 1 3 4 partition 2 Tuesday, December 3, 13 Producers P1 2 6 5 4 3 1 7
  13. 13. partition 0 1 2 partition 1 1 P2 5 2 3 4 partition 2 Tuesday, December 3, 13 1 Producers P1 2 6 5 4 3 7
  14. 14. bulk publishing partition 0 1 2 partition 1 1 P2 5 2 3 4 partition 2 Tuesday, December 3, 13 1 Producers P1 2 6 5 4 3 7
  15. 15. bulk publishing partition 0 1 2 partition 1 1 P2 5 partition 2 Tuesday, December 3, 13 1 2 3 4 Producers P1 2 6 5 4 3 7
  16. 16. parallel publishing partition 0 1 2 partition 1 1 P2 5 partition 2 Tuesday, December 3, 13 1 2 3 4 Producers P1 2 6 5 4 3 7
  17. 17. parallel publishing partition 0 1 2 3 4 partition 1 1 2 P2 5 partition 2 Tuesday, December 3, 13 1 2 3 4 Producers P1 4 3 6 5 7
  18. 18. partition 0 1 2 3 4 5 6 7 partition 1 1 2 3 4 5 6 P2 partition 2 Tuesday, December 3, 13 1 2 3 4 5 Producers P1
  19. 19. consumer group 1 Consumers C1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 7 partition 0 C2 C3 C4 consumer group 2 Tuesday, December 3, 13 1 partition 1 partition 2
  20. 20. consumer group 1 Consumers 1 C1 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 7 partition 0 C2 C3 C4 1 consumer group 2 Tuesday, December 3, 13 2 partition 1 partition 2
  21. 21. consumer group 1 Consumers 1 C1 3 4 5 6 2 3 4 5 6 2 3 4 5 7 partition 0 C2 1 C3 C4 1 consumer group 2 Tuesday, December 3, 13 2 1 partition 1 partition 2
  22. 22. consumer group 1 Consumers 1 C1 3 4 5 6 2 3 4 5 6 2 3 4 5 7 partition 0 1 C2 1 C3 C4 1 consumer group 2 Tuesday, December 3, 13 2 partition 1 partition 2
  23. 23. consumer group 1 Consumers 1 3 C1 4 5 6 4 5 6 4 5 7 partition 0 1 2 C2 1 2 C3 C4 3 2 1 consumer group 2 Tuesday, December 3, 13 3 3 partition 1 partition 2
  24. 24. consumer group 1 Consumers 1 3 C1 3 4 5 6 4 5 6 4 5 7 partition 0 1 2 C2 1 2 C3 C4 3 2 1 consumer group 2 Tuesday, December 3, 13 bulk consumption 3 partition 1 partition 2
  25. 25. consumer group 1 bulk consumption Consumers 1 3 C1 partition 0 1 2 C2 6 4 3 5 4 1 2 C3 C4 6 5 4 3 2 1 consumer group 2 Tuesday, December 3, 13 7 3 5 4 5 6 partition 1 partition 2
  26. 26. consumer group 1 parallel consumption Consumers 1 3 C1 partition 0 1 2 C2 6 4 3 5 4 1 2 C3 C4 6 5 4 3 2 1 consumer group 2 Tuesday, December 3, 13 7 3 5 4 5 6 partition 1 partition 2
  27. 27. consumer group 1 parallel consumption Consumers 1 3 5 4 C1 partition 0 1 2 C2 6 4 3 5 6 1 2 C3 3 C4 6 5 4 3 2 1 consumer group 2 Tuesday, December 3, 13 7 4 5 partition 1 partition 2
  28. 28. consumer group 1 Consumers 1 3 5 4 C1 1 2 C2 6 4 3 5 1 2 C3 3 7 4 5 C4 7 6 5 4 3 2 1 consumer group 2 Tuesday, December 3, 13 partition 0 partition 1 partition 2
  29. 29. Replication (all reads and writes) server A A master P1 C5 A7 A6 A4 A3 A2 A1 B4 B3 B2 B1 (all reads and writes) (all reads and writes) Tuesday, December 3, 13 server C C master server B B master P2 C1 A5 C2 C3 C4 B6 B5
  30. 30. Replication (all reads and writes) A1 server A A master P1 C5 A7 A6 A4 A3 A2 B4 B3 B2 B1 (all reads and writes) (all reads and writes) Tuesday, December 3, 13 server C C master server B B master P2 C1 A5 C2 C3 C4 B6 B5
  31. 31. Replication (all reads and writes) A1 server A A master P1 C5 A7 A6 A4 A3 A2 B4 B3 B2 B1 (all reads and writes) A1 (all reads and writes) Tuesday, December 3, 13 server C C master server B B master P2 C1 A5 C2 C3 C4 B6 B5
  32. 32. Replication (all reads and writes) A1 A2 server A A master P1 C5 A7 A6 A4 A3 B4 B3 B2 B1 (all reads and writes) A1 A2 (all reads and writes) Tuesday, December 3, 13 server C C master server B B master P2 C1 A5 C2 C3 C4 B6 B5
  33. 33. Replication (all reads and writes) A1 A2 C1 server A A master P1 C5 A7 A6 A4 A3 B4 B3 B2 B1 (all reads and writes) A1 A2 (all reads and writes) Tuesday, December 3, 13 C1 server C C master server B B master P2 A5 C2 C3 C4 B6 B5
  34. 34. Replication (all reads and writes) server A A master A1 A2 A3 A4 C1 C2 C3 C4 C5 P1 A7 A6 (all reads and writes) B1 B2 B3 B4 B5 B6 A1 A2 A3 A4 (all reads and writes) C1 C2 C3 C4 C5 B1 B2 B3 B4 B5 Tuesday, December 3, 13 B6 server C C master server B B master P2 A5
  35. 35. Replication (all reads and writes) A1 A2 A3 A4 A5 A6 A7 C1 C2 C3 C4 C5 server A A master P1 (all reads and writes) B1 B2 B3 B4 B5 B6 A1 A2 A3 A4 (all reads and writes) C1 C2 C3 C4 C5 B1 B2 B3 B4 B5 Tuesday, December 3, 13 B6 server C C master server B B master P2
  36. 36. Replication (all reads and writes) A1 A2 A3 A4 A5 A6 A7 C1 C2 C3 C4 C5 server A A master P1 (all reads and writes) B1 B2 B3 B4 B5 B6 A1 A2 A3 A4 A5 A6 A7 (all reads and writes) C1 C2 C3 C4 C5 B1 B2 B3 B4 B5 Tuesday, December 3, 13 B6 server C C master server B B master P2
  37. 37. 123 Messages in a partition are assigned a sequential id number, referred to as the offset. Consumers keep track of their consumed offset in the partition. Kafka doesn’t maintain any metadata. Kafka maintains all messages for a configurable time, regardless of whether they’ve been consumed. Tuesday, December 3, 13
  38. 38. Alternatives http://queues.io Tuesday, December 3, 13
  39. 39. Questions? Tuesday, December 3, 13

×