Cheat Sheet for Kafka

1. ********************************************************************************
*****************************************************
Single Broker Cluster
********************************************************************************
*****************************************************
Start Zookeeper [seperate terminal]
---------------
bin/zookeeper-server-start.sh config/zookeeper.properties
Start Kafka broker [seperate terminal]
------------------
bin/kafka-server-start.sh config/server.properties
________________________________________________________________________________
_____________________________________________________
Create Topic
------------
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1
--partitions 1 --topic testtopic
List Topic
----------
bin/kafka-topics.sh --list --zookeeper localhost:2181
Display Topic Information
-------------------------
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic testtopic
Add Partitions to a Topic
-------------------------
bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic testopic
--partitions 2
________________________________________________________________________________
_____________________________________________________
Start Producer [seperate terminal]
--------------
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testtopic
Start Consumer [seperate terminal]
--------------
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-
beginning
________________________________________________________________________________
_____________________________________________________
Note: To understand the flow refer Kafka Tutorial.mp4
********************************************************************************
*****************************************************
Multi Broker Cluster
********************************************************************************
*****************************************************
Create config file for 2 more Kafka Brokers
-------------------------------------------
cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties
Edit config file 2 Kafka Brokers created above
----------------------------------------------
config/server-1.properties:
broker.id=1
port=9093

2. log.dir=/tmp/kafka-logs-1
config/server-2.properties:
broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2
Start 2 newly created Kafka Brokers [seperate terminal]
-----------------------------------
JMX_PORT=9997 bin/kafka-server-start.sh config/server-1.properties &
JMX_PORT=9998 bin/kafka-server-start.sh config/server-2.properties &
Create Topic
------------
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3
--partitions 1 --topic my-replicated-topic
List Topic
----------
bin/kafka-topics.sh --list --zookeeper localhost:2181
Start Producer [seperate terminal]
--------------
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-
replicated-topic
Start Consumer [seperate terminal]
--------------
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning
--topic my-replicated-topic
Kill one of the Kafka Brokers
-----------------------------
pkill -9 -f server-1.properties
List Topic
----------
bin/kafka-topics.sh --list --zookeeper localhost:2181
Start Consumer [seperate terminal]
--------------
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning
--topic my-replicated-topic