This short deck provides a high-level overview of how Apache Kafka works under the covers. It covers logs, topics partitions, consumer groups, and replication from a conceptual perspective.
29. Replication
(all reads and writes)
server A
A master
P1
C5
A7
A6
A4
A3
A2
A1
B4
B3
B2
B1
(all reads and writes)
(all reads and writes)
Tuesday, December 3, 13
server C
C master
server B
B master
P2
C1
A5
C2
C3
C4
B6
B5
30. Replication
(all reads and writes)
A1
server A
A master
P1
C5
A7
A6
A4
A3
A2
B4
B3
B2
B1
(all reads and writes)
(all reads and writes)
Tuesday, December 3, 13
server C
C master
server B
B master
P2
C1
A5
C2
C3
C4
B6
B5
31. Replication
(all reads and writes)
A1
server A
A master
P1
C5
A7
A6
A4
A3
A2
B4
B3
B2
B1
(all reads and writes)
A1
(all reads and writes)
Tuesday, December 3, 13
server C
C master
server B
B master
P2
C1
A5
C2
C3
C4
B6
B5
32. Replication
(all reads and writes)
A1 A2
server A
A master
P1
C5
A7
A6
A4
A3
B4
B3
B2
B1
(all reads and writes)
A1 A2
(all reads and writes)
Tuesday, December 3, 13
server C
C master
server B
B master
P2
C1
A5
C2
C3
C4
B6
B5
33. Replication
(all reads and writes)
A1 A2
C1
server A
A master
P1
C5
A7
A6
A4
A3
B4
B3
B2
B1
(all reads and writes)
A1 A2
(all reads and writes)
Tuesday, December 3, 13
C1
server C
C master
server B
B master
P2
A5
C2
C3
C4
B6
B5
34. Replication
(all reads and writes)
server A
A master
A1 A2 A3 A4
C1 C2 C3 C4 C5
P1
A7
A6
(all reads and writes)
B1 B2 B3 B4 B5 B6
A1 A2 A3 A4
(all reads and writes)
C1 C2 C3 C4 C5
B1 B2 B3 B4 B5
Tuesday, December 3, 13
B6
server C
C master
server B
B master
P2
A5
35. Replication
(all reads and writes)
A1 A2 A3 A4 A5 A6 A7
C1 C2 C3 C4 C5
server A
A master
P1
(all reads and writes)
B1 B2 B3 B4 B5 B6
A1 A2 A3 A4
(all reads and writes)
C1 C2 C3 C4 C5
B1 B2 B3 B4 B5
Tuesday, December 3, 13
B6
server C
C master
server B
B master
P2
36. Replication
(all reads and writes)
A1 A2 A3 A4 A5 A6 A7
C1 C2 C3 C4 C5
server A
A master
P1
(all reads and writes)
B1 B2 B3 B4 B5 B6
A1 A2 A3 A4 A5 A6 A7
(all reads and writes)
C1 C2 C3 C4 C5
B1 B2 B3 B4 B5
Tuesday, December 3, 13
B6
server C
C master
server B
B master
P2
37. 123
Messages in a partition are assigned a sequential id
number, referred to as the offset.
Consumers keep track of their consumed offset in
the partition. Kafka doesn’t maintain any metadata.
Kafka maintains all messages for a configurable time,
regardless of whether they’ve been consumed.
Tuesday, December 3, 13