In the Apache Kafka world, there is such a great diversity of open source tools available (I counted over 50!) that it’s easy to get lost. Over the years I have dealt with Kafka, I have learned to particularly enjoy a few of them that save me a tremendous amount of time over performing manual tasks. I will be sharing my experience and doing live demos of my favorite Kafka tools, so that you too can hopefully increase your productivity and efficiency when managing and administering Kafka. Come learn about the latest and greatest tools for CLI, UI, Replication, Management, Security, Monitoring, and more!

1. Show Me
Kafka Tools
That Will Improve
My Productivity
Stephane Maarek - DataCumulus

2. When you get started with Kafka
The Basics
• zookeeper-server-start
• kafka-server-start
• kafka-topics
• kafka-console-consumer
• kafka-console-producer
Photo by Wes Hicks on Unsplash

3. Existential question

4. Things start to get pretty long
kafka-console-consumer --bootstrap-
server 127.0.0.1:9092 --topic test
--from-beginning --property
print.key=true --property
key.separator=,

5. When you use Kafka in production
Advanced CLI
• kafka-consumer-groups
• kafka-acls
• kafka-configs
• kafka-preferred-replica-
election
• kafka-reassign-partitions
• kafka-run-class
Photo by JESHOOTS.COM on Unsplash

6. When you don’t know how to use a CLI
…Or which CLI to use
Photo by Icons8 team on Unsplash

7. Our goal: be a conductor
Ease, grace, confidence
You
Happy users
Brokers
BrokersBrokers
Happy users
Happy boss

8. Who am I?
• I’m Stephane!
• Consultant & Solution Architect at DataCumulus
• Apache Kafka SeriesVideo Courses on Udemy (coupon KAFKASUMMIT)
• I always like to improve my productivity
• You can find me on
• GitHub: https://github.com/simplesteph
• LinkedIn: https://www.linkedin.com/in/stephanemaarek
• Medium: https://medium.com/@stephane.maarek
• Twitter: https://twitter.com/stephanemaarek
• Udemy: https://udemy.com/stephane-maarek

9. Where can I improve my productivity?
Some possible areas:
1. Developers CLI [demo]
2. Administration Tools [demo]
3. Consumer Groups Lag Monitoring [demo]
4. Kafka Web UI & Kafka Availability Audit [demo]
5. Security
6. Replication
7. Desktop [demo]
Let’s see some of them. I counted over 50 available tools

10. Developers CLI
• My pick
• ❤ kafkacat: Magnus Edenhill (Confluent) – written in C
https://github.com/edenhill/kafkacat
• Promising:
• Kaf: Infinimesh – written in Go
https://github.com/infinimesh/kaf
• Keep an eye out:
• https://github.com/fgeller/kt – Golang
• https://github.com/kdrakon/topiks - Rust

11. kafkacat - Demo
Main commands

12. Kafka Administration Tools
• Broker Rolling Restart CLI:
• ❤ https://github.com/Yelp/kafka-utils - Python
• Reassign partitions CLI:
• https://github.com/DataDog/kafka-kit - Golang
• ❤ https://github.com/linkedin/kafka-tools - Python
• https://github.com/Yelp/kafka-utils - Python
• https://github.com/airbnb/kafkat - Ruby
• Automated Healing and Balancing:
• https://github.com/pinterest/doctorkafka - Java
• ❤ https://github.com/linkedin/cruise-control Java (https://github.com/linkedin/cruise-control-ui)

13. kafka-utils (Yelp) - Demo
Broker Rolling Restart

14. Consumer Groups Lag Monitoring
HTTP options
• Remora (improvement over Burrow) - Scala
https://github.com/zalando-incubator/remora
• Burrow - Golang
https://github.com/linkedin/Burrow
❤ HTTP interface ❤ Integra/on with CloudWatch / DataDog

15. Consumer Groups Lag Monitoring
Other
• Kafka Offset Monitor - Scala
https://github.com/Morningstar/kafka-
offset-monitor
• Kafka Hawk (Consumer offset commit stats) - Java:
https://github.com/farmdawgnation/kafka-hawk
An application that records stats about consumer group
offset commits and reports them as Prometheus metrics
❤ Graph visualiza-on

16. Demo – Consumer Offsets
Remora

17. Kafka Web GUI
Free / Open Source
• Kafka Manager - Scala
https://github.com/yahoo/kafka-manager
• Kafka Eagle - Java
https://github.com/smartloli/kafka-eagle
Has metrics, consumers, topics, brokers,
zookeeper + CLI
❤ Good old UI, maintained by the community ❤ graph visualiza:ons + metrics monitoring

18. Kafka Web GUI
Free / Open Source
• Kafdrop - Java
https://github.com/HomeAdvisor/Kafdrop
• Kafka HQ – Java – Promising, integration with SR
https://github.com/tchiotludo/kafkahq
• KafkaView – Rust
(Kafka Manager-like, read only)
https://github.com/fede1024/kafka-view
❤ simple, one page view, /me metrics
❤ fast itera/ons and a lot of features are being added
❤ possibility to search across clusters

19. Other UI
• Kafka Connect UI
https://github.com/Landoop/kafka-connect-ui
• Schema Registry UI
https://github.com/Landoop/schema-registry-ui
❤ the only free open source connect UI ❤ dedicated lightweight schema registry UI

20. Kafka Web GUI
Consumer UI
Kafka Webview – Java, Has filtering
• https://github.com/SourceLabOrg/kafka-
webview
• Consumer in webview:
https://github.com/PandoraMedia/kbrowse
❤ powerful filtering and selec3on features ❤ simplicity

21. Kafka Web GUI
Paid UI / Commercially supported
• Confluent Control Center:
https://www.confluent.io/confluent
-control-center/
• Lenses:
https://lenses.io/lenses-overview/
❤ Consumer lag feature
❤ Dashboard with important metrics
❤ Data flow visualiza:on
❤ Ability to start streams from the UI

22. Kafka Health Check & Audit
h"ps://github.com/linkedin/ka6a-monitor/wiki/Design-Overview

23. Kafka Health Check & Audit
• LinkedIn Kafka Monitor
(availability of cluster):
https://github.com/linkedin/kafk
a-monitor
• Health Check Kafka Cluster
https://github.com/andreas-
schroeder/kafka-health-check
❤ graphs
❤ Single broker latency measurement
❤ Mul6 clusters latency measurement
=> Aler6ng if Mirror Maker is down
❤ CLI
❤ Golang
❤ HTTP interface

24. Demo: LinkedIn Kafka Monitor

25. Kafka Security
• ❤ Kafka Security Manager:
https://github.com/simplesteph/kafka-security-manager
• Ranger plugin for Kafka
• Atlas plugin for Kafka
• LDAP Authorizer plugin from Confluent

26. Kafka Replication
• Open Source:
• Mirror Maker
• ❤ Mirror Maker 2 (Kafka Connect) – Kafka 2.3
• Uber uReplicator: https://github.com/uber/uReplicator
• ComCast Kafka Connect MirrorMaker:
https://github.com/Comcast/MirrorTool-for-Kafka-Connect
• SalesForce Mirus: https://github.com/salesforce/mirus
• Commercially supported:
• ❤ Confluent Replicator

27. Our goal: be a conductor
Ease, grace, confidence
You
Happy users
Brokers
BrokersBrokers
Happy users
Happy boss

28. One more tool
h"ps://xkcd.com/927/

29. Final Demo:
Desktop Client for Apache Kafka

30. • Try it out, give us feedback!
• Download: https://conduktor.io
• Twitter for updates: @getconduktor
• Tons of feature in the roadmap: Schema Registry, Connect, Rebalancing /
Repartitioning, & more!
• Thanks @sderosiaux

31. Thank you!
Any questions?