45. 45
Broker 1 Broker 2 Broker N
Producer Producer Producer
Consumer Consumer
Kafka Cluster
Topic partition
REPLICATIONREPLICATION
Controller
Where did my topic partitions go?
46. 46
● Leader Election
● Replica Reassignment
● Create Topic
● Delete Topic
● Add Partitions
● Broker start and shutdown
Responsibilities of the Controller
No Controller - No Cluster
47. 47
● Controller state - topic creation, topic deletion etc
● Time taken to perform an operation
● Rate at which an admin operation is performed
● Queue sizes within the controller
Lack of Metrics
49. 49
Root Cause
Upgrades!
● Synchronous per-partition
zookeeper writes
● Sequential per-partition
controller-to-broker requests
● Complicated concurrency
semantics
● No separation of control plane
from data plane
51. 51
Zero controller downtime!
● Highly available cluster
● 10x faster leader elections
● More number of topic partitions per cluster
● Faster broker shutdown and upgrades
53. 53
Broker 1 Broker 2 Broker N
Producer Producer Producer
Consumer Consumer
Kafka Cluster
Topic partition
REPLICATIONREPLICATION
Can I get my latency?
54. 54
But I have bytes quota set
● Throttle byte rate per second on the broker
● Response delayed on exceeding threshold
● Avoids bad clients from consuming all the bandwidth
56. 56
Root Cause
● Too many small sized requests
● DDos attack from client
● Decompression on the server takes a long time
● With more consumer instances, more requests
57. 57
bin/kafka-configs --zookeeper localhost:2181 --alter --add-config
'request_percentage=50' --entity-name user1 --entity-type users
Request Quota - percentage of time a client can spend on request
handler (I/O) threads and network threads within each quota window
Predictable Latency