1. Custom management apps for Kafka
2018/12/18
「Apache Kafka Meetup Japan #5 @LINE」
Kimura, Sotaro(@kimutansk)
https://www.flickr.com/photos/wwworks/8693876413/
2. I am...
• Kimura, Sotaro(@kimutansk)
– Data Engineer IT choreman at Dwango
• Data Analytics Infrastracture team
• On premiss~Cloud, SRE~Individual function
• Batch / Stream processing, developer / leader
– In other words, IT choreman
– Language, Scala/Java/Python/Ruby/Go etc...
– Favorite
• Stream data processing
• Kafka, Flink, Fluend, Spark, Pulsar etc...
3. Systems of our team abstract
Otherteam’s system
Explanatorynotes
Externalreq/res
Internalreq/res
Job execution
Monitor/Manage
4. Systems of our team abstract
Otherteam’s system
Explanatorynotes
Externalreq/res
Internalreq/res
Job execution
Monitor/Manage
5. Kafka monitoring/manage funcs
• Topic setting manage
• User account / ACLs manage
• Topic list documents page generate
• Resource monitoring
– CPU utilization, Memory usage, Disk usage, IO wait
– Network processor / Request handler idle rates
• Flow manage
– Auto recommend expanding partitions
6. Kafka monitoring/manage funcs
• Topic setting manage
• User account / ACLs manage
• Topic list documents page generate
• Resource monitoring
– CPU utilization, Memory usage, Disk usage, IO wait
– Network processor / Request handler idle rates
• Flow manage
– Auto recommend expanding partitions
Today’s target.
8. Topic setting manage
• Operating kafka from CLI
– Topic create
– Topic partition expand
– Topic specific setting
$ ./kafka-topics.sh --zookeeper kafka01:2181--create --topic sample_topic01
$ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --partitions 3
$ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --config cleanup.policy=compact
Too hard of human’s operation !
9. User account / ACLs manage
• Premise
– We decided to use SSL for Kafka AuthN/ACLs manage.
• Because, we investigated it at about 2017/04.
• At that time, ruby-kafka didn’t support SASL.
– In ruby-kafka v0.4.0, supported :)
– In our team, AuthN is not applied.
• AuthN is only applied to the access from other teams.
10. User account / ACLs manage
• Operating kafka from CLI
– Add consumer permission to user
– Add producer permission to user
– Remove producer permission from user
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-consumer01" --consumer --
topic sample_topic01 --group test_group01
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-producer01"--producer --
topic sample_topic01
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--remove --allow-principal"User:CN=test-producer01" --producer
--topic sample_topic01
11. User account / ACLs manage
• Operating kafka from CLI
– Add consumer permission to user
– Add producer permission to user
– Remove producer permission from user
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-consumer01" --consumer --
topic sample_topic01 --group test_group01
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-producer01"--producer --
topic sample_topic01
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--remove --allow-principal"User:CN=test-producer01" --producer
--topic sample_topic01
These commands confuse us!
12. Our needs
• Operators don’t execute raw command.
• Topic/ACLs settings are managed by file.
– To enable reviewing each other.
• So...
13. The administrative client for Kafka
• org.apache.kafka.clients.admin.AdminClient
– Through Broker API Endpoint
– Introduced in 0.11.0.0
– Manage below elements.
• Broker Configuration
• Topic
• Partition
• Topic Configuration
• ACLs
• ReplicaLogDir
14. Developed apps(Topic manage)
• KafkaTopicManageApp
– Using org.apache.kafka.clients.admin.AdminClient
– Process flow
• Load application settings, topic settings.
• Get topic information from kafka cluster.
• Create topic with specified settings.
• Update topic with specified settings.
– Applying settings is idempotent, so app updates always.