This presentation introduces every aspects of the Kafka ecosystem:
- Concepts: explain all misleading concepts such as topic vs partition vs replication, producer vs consumer vs consumer group, group leader vs group coordinator, ...
- Advanced concepts: delivery semantic; idempotent producers; isolation levels; differences between the offsets such as High Watermark, Log End Offset, Last Stable Offset ...
- Kafka architecture: explain all Kafka components such as brokers, controllers, zookeeper, ...
- Overview of Kafka security: TLS/SSL, SASL, Kerberos, ...
- Overview of Kafka ecosystem: Kafka Stream, Kafka Connect, Schema Registry, monitoring tools.
- Kafka in Golang: How to use Kafka client in Golang.
- Comparison with other message queues such as RabbitMQ.
40. Consumer Group
- Consumers can join a group by using the same group.id.
- Ka#ka assigns partitions to the consumers in the same group.
- Each partition is consumed by exactly one consumer in the group.
- Each consumer within a group reads from exclusive partitions.
- One consumer can consume multiple partitions.
- It cannot have more consumers than partitions. Otherwise, some
consumers will be inactive state.
Question: Do Ka#ka allow one topic have multiple consumer groups?
Answer: Yes
- E.g: consumer groups for saving the data to database. Consumer group
for transforming the data into other systems.
50. Group Leader and Coordinator relationship
Rebalancing Event occurs when:
- A new consumer joins the consumer group.
- A consumer leaves the consumer group.
- A consumer is "disconnected" to consumer group (group coordinator
doesn't receive heartbeat event from this consumer)
- The group leader will make the assignment of each partition to each
consumer. Then the group leader sends this assignment to the group
coordinator.
- The group coordinator will send back the assignment to all consumers.
- Rebalancing event happens.
Question: Why doesn't the group coordinator (broker) take the job that
assigns partitions to consumers, but the group leader (consumer) do that?