proof of concept on kafka

1. Proof of Concept on Kafka
Presented by Dhritishman Kotaki and Sukhdev Suthar

2. What is Apache kafka?
● a software platform which is based on a distributed streaming process.
● a publish-subscribe messaging system.
● lets exchanging of data between applications, servers, and processors as well.

3. Why should we use it?
● Low Latency
● High Throughput
● Fault tolerance
● Durability
● Scalability

4. Publish-subscribe messaging system in Apache kafka
● allows a sender to send/write the message and a receiver to read that message
● a sender is known as a producer who publishes messages
● a receiver is known as a consumer who consumes that message by subscribing it.

5. Kafka model

7. Topics in Kafka
● a category or a common name used to store and publish a particular stream of data.
● in Kafka, we can create n number of topics as we want.
● it is identified by its name.
● a producer publishes data to the topics.
● consumer reads that data from the topic by subscribing it.

8. Partitions
● A topic is split into several parts which are known as the partitions.
● Each message gets stored into partitions with an incremental id known as its offset value.

10. Brokers
● a Kafka cluster is comprised of one or more servers.
● known as brokers or Kafka brokers.
● it is a container that holds several topics with their multiple partitions.
● brokers in the cluster are identified by an integer id.

12. Brokers and Topics

13. Kafka Topic Replication

14. ● Partition 0 of Topic-x is having its replicas in Broker 1 and Broker 2.
● Partition1 of Topic-x is having its replication in Broker 2 and Broker 3.
● Confusion may occur regarding which broker should serve the client request.

15. Solution to the above problem
● choose one of the broker's partition as a leader.
● rest of them becomes its followers.
● these replicas are known as ISR(in-sync-replica).
● only the leader is allowed to serve the client request.

16. Example

17. ● Partition-0 under Broker 1 is provided with the leadership.
● Partition 0 under Broker 2 will become its replica or ISR.
● Partition 1 under Broker 2 is the leader and Partition 1 under Broker 3 is its replica.
● In case, Broker 1 fails to serve, Broker 2 with Partition 0 replica will become the leader.

18. How does the producer write data to the cluster?
● Message Keys
● Acknowledgment

19. Message Keys
● send the messages in a specific order.
● In Kafka, load balancing is done when the producer writes data to the Kafka topic without specifying any
key.
● Kafka distributes little-little bit data to each partition.

20. Example

21. ● data is written without specifying the key.
● So, the data gets distributed among each partition of Topic-T under each broker

22. Example

23. ● specify a key as Prod_id.
● data of Prod_id_1(say) will always be sent to partition 0 under Broker 1.
● data of Prod_id_2 will always be in partition 1 under Broker 2.
● data will not be distributed to each partition after applying the key.

24. Acknowledgement
● the producer has another choice of acknowledgment.
● acks=0: the producer sends the data to the broker but does not wait for the acknowledgement.
● acks=1: the producer will wait for the leader's acknowledgement.
● acks=all: the acknowledgment is done by both the leader and its followers.

25. Kafka: Consumer and Consumer Groups

26. Consumer Groups

27. Another example
Here, consumer 3 is inactive since three consumers are present in one group only

28. Zookeeper
● is used to store information about the Kafka cluster and details of the consumer clients.
● manages brokers by maintaining a list of them.
● s responsible for choosing a leader for the partitions.

29. Apache Kafka Architecture