SlideShare a Scribd company logo
1 of 21
Download to read offline
Custom management apps for Kafka
2018/12/18
「Apache Kafka Meetup Japan #5 @LINE」
Kimura, Sotaro(@kimutansk)
https://www.flickr.com/photos/wwworks/8693876413/
I am...
• Kimura, Sotaro(@kimutansk)
– Data Engineer IT choreman at Dwango
• Data Analytics Infrastracture team
• On premiss~Cloud, SRE~Individual function
• Batch / Stream processing, developer / leader
– In other words, IT choreman
– Language, Scala/Java/Python/Ruby/Go etc...
– Favorite
• Stream data processing
• Kafka, Flink, Fluend, Spark, Pulsar etc...
Systems of our team abstract
Otherteam’s system
Explanatorynotes
Externalreq/res
Internalreq/res
Job execution
Monitor/Manage
Systems of our team abstract
Otherteam’s system
Explanatorynotes
Externalreq/res
Internalreq/res
Job execution
Monitor/Manage
Kafka monitoring/manage funcs
• Topic setting manage
• User account / ACLs manage
• Topic list documents page generate
• Resource monitoring
– CPU utilization, Memory usage, Disk usage, IO wait
– Network processor / Request handler idle rates
• Flow manage
– Auto recommend expanding partitions
Kafka monitoring/manage funcs
• Topic setting manage
• User account / ACLs manage
• Topic list documents page generate
• Resource monitoring
– CPU utilization, Memory usage, Disk usage, IO wait
– Network processor / Request handler idle rates
• Flow manage
– Auto recommend expanding partitions
Today’s target.
Topic setting manage
• Operating kafka from CLI
– Topic create
– Topic partition expand
– Topic specific setting
$ ./kafka-topics.sh --zookeeper kafka01:2181--create --topic sample_topic01
$ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --partitions 3
$ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --config cleanup.policy=compact
Topic setting manage
• Operating kafka from CLI
– Topic create
– Topic partition expand
– Topic specific setting
$ ./kafka-topics.sh --zookeeper kafka01:2181--create --topic sample_topic01
$ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --partitions 3
$ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --config cleanup.policy=compact
Too hard of human’s operation !
User account / ACLs manage
• Premise
– We decided to use SSL for Kafka AuthN/ACLs manage.
• Because, we investigated it at about 2017/04.
• At that time, ruby-kafka didn’t support SASL.
– In ruby-kafka v0.4.0, supported :)
– In our team, AuthN is not applied.
• AuthN is only applied to the access from other teams.
User account / ACLs manage
• Operating kafka from CLI
– Add consumer permission to user
– Add producer permission to user
– Remove producer permission from user
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-consumer01" --consumer --
topic sample_topic01 --group test_group01
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-producer01"--producer --
topic sample_topic01
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--remove --allow-principal"User:CN=test-producer01" --producer
--topic sample_topic01
User account / ACLs manage
• Operating kafka from CLI
– Add consumer permission to user
– Add producer permission to user
– Remove producer permission from user
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-consumer01" --consumer --
topic sample_topic01 --group test_group01
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-producer01"--producer --
topic sample_topic01
$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--remove --allow-principal"User:CN=test-producer01" --producer
--topic sample_topic01
These commands confuse us!
Our needs
• Operators don’t execute raw command.
• Topic/ACLs settings are managed by file.
– To enable reviewing each other.
• So...
The administrative client for Kafka
• org.apache.kafka.clients.admin.AdminClient
– Through Broker API Endpoint
– Introduced in 0.11.0.0
– Manage below elements.
• Broker Configuration
• Topic
• Partition
• Topic Configuration
• ACLs
• ReplicaLogDir
Developed apps(Topic manage)
• KafkaTopicManageApp
– Using org.apache.kafka.clients.admin.AdminClient
– Process flow
• Load application settings, topic settings.
• Get topic information from kafka cluster.
• Create topic with specified settings.
• Update topic with specified settings.
– Applying settings is idempotent, so app updates always.
Developed apps(Topic manage)
• KafkaTopicManageApp
– Application setting
bootstrapServers:
- "kafka01:9092"
- "kafka02:9092"
- "kafka03:9092"
defaultPartitions: 1
defaultReplicationFactor: 3
Developed apps(Topic manage)
• KafkaTopicManageApp
– Topic setting
topics:
- name: TestTopicName
partitionNum: 4
config:
min.insync.replicas: 2
segment.bytes: 104857600
Developed apps(ACLs manage)
• KafkaAclManageApp
– Using org.apache.kafka.clients.admin.AdminClient
– Process flow
• Load application settings, acl settings.
• Apply acls to kafka cluster.
– Applying settings is idempotent, so app applys always.
Developed apps(Topic manage)
• KafkaAclManageApp
– Acl setting
userAcls:
- name: "CN=test_user01"
authorized_consumer_topics:
- "sample_topic01"
- "sample_topic02"
specific_consumer_group:
- "test_user01_group01"
- "test_user01_group02"
Apps operation
• Settings are managed by Github.
• Update flow
– Update settings.
– PR & review
– Execute apps from jenkins.
• Check flow
– Check difference between settings and kafka cluster
periodically.
– If differences exist, notify to operators.(Slack)
Conclusion
• Kafka cluster operation commands confuse us.
• So, let’s automate by kafka manage tools or
your hand made apps with AdminClient!
Thank you for your attention!
https://www.flickr.com/photos/savannahcorps/7409364642

More Related Content

What's hot

High Performance Solr and JVM Tuning Strategies used for MapQuest’s Search Ah...
High Performance Solr and JVM Tuning Strategies used for MapQuest’s Search Ah...High Performance Solr and JVM Tuning Strategies used for MapQuest’s Search Ah...
High Performance Solr and JVM Tuning Strategies used for MapQuest’s Search Ah...Lucidworks
 
Operational Tips For Deploying Apache Spark
Operational Tips For Deploying Apache SparkOperational Tips For Deploying Apache Spark
Operational Tips For Deploying Apache SparkDatabricks
 
Introduction to Presto at Treasure Data
Introduction to Presto at Treasure DataIntroduction to Presto at Treasure Data
Introduction to Presto at Treasure DataTaro L. Saito
 
Sqoop on Spark for Data Ingestion
Sqoop on Spark for Data IngestionSqoop on Spark for Data Ingestion
Sqoop on Spark for Data IngestionDataWorks Summit
 
Data Engineering with Solr and Spark
Data Engineering with Solr and SparkData Engineering with Solr and Spark
Data Engineering with Solr and SparkLucidworks
 
Designing Payloads for Event-Driven Systems | Lorna Mitchell, Aiven
Designing Payloads for Event-Driven Systems | Lorna Mitchell, AivenDesigning Payloads for Event-Driven Systems | Lorna Mitchell, Aiven
Designing Payloads for Event-Driven Systems | Lorna Mitchell, AivenHostedbyConfluent
 
Building a unified data pipeline in Apache Spark
Building a unified data pipeline in Apache SparkBuilding a unified data pipeline in Apache Spark
Building a unified data pipeline in Apache SparkDataWorks Summit
 
Modernizing Infrastructures for Fast Data with Spark, Kafka, Cassandra, React...
Modernizing Infrastructures for Fast Data with Spark, Kafka, Cassandra, React...Modernizing Infrastructures for Fast Data with Spark, Kafka, Cassandra, React...
Modernizing Infrastructures for Fast Data with Spark, Kafka, Cassandra, React...Lightbend
 
Webinar: Deep Dive on Apache Flink State - Seth Wiesman
Webinar: Deep Dive on Apache Flink State - Seth WiesmanWebinar: Deep Dive on Apache Flink State - Seth Wiesman
Webinar: Deep Dive on Apache Flink State - Seth WiesmanVerverica
 
Spark Summit EU talk by Jakub Hava
Spark Summit EU talk by Jakub HavaSpark Summit EU talk by Jakub Hava
Spark Summit EU talk by Jakub HavaSpark Summit
 
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming by Ew...
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming by Ew...Building Realtime Data Pipelines with Kafka Connect and Spark Streaming by Ew...
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming by Ew...Spark Summit
 
Presto at Facebook - Presto Meetup @ Boston (10/6/2015)
Presto at Facebook - Presto Meetup @ Boston (10/6/2015)Presto at Facebook - Presto Meetup @ Boston (10/6/2015)
Presto at Facebook - Presto Meetup @ Boston (10/6/2015)Martin Traverso
 
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...Databricks
 
MySQL Query Optimization (Basics)
MySQL Query Optimization (Basics)MySQL Query Optimization (Basics)
MySQL Query Optimization (Basics)Karthik .P.R
 
Architectures, Frameworks and Infrastructure
Architectures, Frameworks and InfrastructureArchitectures, Frameworks and Infrastructure
Architectures, Frameworks and Infrastructureharendra_pathak
 
Presto anatomy
Presto anatomyPresto anatomy
Presto anatomyDongmin Yu
 
Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...
Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...
Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...Lightbend
 
KSQL: Streaming SQL for Kafka
KSQL: Streaming SQL for KafkaKSQL: Streaming SQL for Kafka
KSQL: Streaming SQL for Kafkaconfluent
 
Hoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on SparkHoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on SparkVinoth Chandar
 
Search Architecture at Evernote: Presented by Christian Kohlschütter, Evernote
Search Architecture at Evernote: Presented by Christian Kohlschütter, EvernoteSearch Architecture at Evernote: Presented by Christian Kohlschütter, Evernote
Search Architecture at Evernote: Presented by Christian Kohlschütter, EvernoteLucidworks
 

What's hot (20)

High Performance Solr and JVM Tuning Strategies used for MapQuest’s Search Ah...
High Performance Solr and JVM Tuning Strategies used for MapQuest’s Search Ah...High Performance Solr and JVM Tuning Strategies used for MapQuest’s Search Ah...
High Performance Solr and JVM Tuning Strategies used for MapQuest’s Search Ah...
 
Operational Tips For Deploying Apache Spark
Operational Tips For Deploying Apache SparkOperational Tips For Deploying Apache Spark
Operational Tips For Deploying Apache Spark
 
Introduction to Presto at Treasure Data
Introduction to Presto at Treasure DataIntroduction to Presto at Treasure Data
Introduction to Presto at Treasure Data
 
Sqoop on Spark for Data Ingestion
Sqoop on Spark for Data IngestionSqoop on Spark for Data Ingestion
Sqoop on Spark for Data Ingestion
 
Data Engineering with Solr and Spark
Data Engineering with Solr and SparkData Engineering with Solr and Spark
Data Engineering with Solr and Spark
 
Designing Payloads for Event-Driven Systems | Lorna Mitchell, Aiven
Designing Payloads for Event-Driven Systems | Lorna Mitchell, AivenDesigning Payloads for Event-Driven Systems | Lorna Mitchell, Aiven
Designing Payloads for Event-Driven Systems | Lorna Mitchell, Aiven
 
Building a unified data pipeline in Apache Spark
Building a unified data pipeline in Apache SparkBuilding a unified data pipeline in Apache Spark
Building a unified data pipeline in Apache Spark
 
Modernizing Infrastructures for Fast Data with Spark, Kafka, Cassandra, React...
Modernizing Infrastructures for Fast Data with Spark, Kafka, Cassandra, React...Modernizing Infrastructures for Fast Data with Spark, Kafka, Cassandra, React...
Modernizing Infrastructures for Fast Data with Spark, Kafka, Cassandra, React...
 
Webinar: Deep Dive on Apache Flink State - Seth Wiesman
Webinar: Deep Dive on Apache Flink State - Seth WiesmanWebinar: Deep Dive on Apache Flink State - Seth Wiesman
Webinar: Deep Dive on Apache Flink State - Seth Wiesman
 
Spark Summit EU talk by Jakub Hava
Spark Summit EU talk by Jakub HavaSpark Summit EU talk by Jakub Hava
Spark Summit EU talk by Jakub Hava
 
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming by Ew...
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming by Ew...Building Realtime Data Pipelines with Kafka Connect and Spark Streaming by Ew...
Building Realtime Data Pipelines with Kafka Connect and Spark Streaming by Ew...
 
Presto at Facebook - Presto Meetup @ Boston (10/6/2015)
Presto at Facebook - Presto Meetup @ Boston (10/6/2015)Presto at Facebook - Presto Meetup @ Boston (10/6/2015)
Presto at Facebook - Presto Meetup @ Boston (10/6/2015)
 
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
 
MySQL Query Optimization (Basics)
MySQL Query Optimization (Basics)MySQL Query Optimization (Basics)
MySQL Query Optimization (Basics)
 
Architectures, Frameworks and Infrastructure
Architectures, Frameworks and InfrastructureArchitectures, Frameworks and Infrastructure
Architectures, Frameworks and Infrastructure
 
Presto anatomy
Presto anatomyPresto anatomy
Presto anatomy
 
Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...
Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...
Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And...
 
KSQL: Streaming SQL for Kafka
KSQL: Streaming SQL for KafkaKSQL: Streaming SQL for Kafka
KSQL: Streaming SQL for Kafka
 
Hoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on SparkHoodie: How (And Why) We built an analytical datastore on Spark
Hoodie: How (And Why) We built an analytical datastore on Spark
 
Search Architecture at Evernote: Presented by Christian Kohlschütter, Evernote
Search Architecture at Evernote: Presented by Christian Kohlschütter, EvernoteSearch Architecture at Evernote: Presented by Christian Kohlschütter, Evernote
Search Architecture at Evernote: Presented by Christian Kohlschütter, Evernote
 

Similar to Custom management apps for Kafka

Real time Analytics with Apache Kafka and Apache Spark
Real time Analytics with Apache Kafka and Apache SparkReal time Analytics with Apache Kafka and Apache Spark
Real time Analytics with Apache Kafka and Apache SparkRahul Jain
 
Knock Knock, Who’s There? With Justin Chen and Dhruv Jauhar | Current 2022
Knock Knock, Who’s There? With Justin Chen and Dhruv Jauhar | Current 2022Knock Knock, Who’s There? With Justin Chen and Dhruv Jauhar | Current 2022
Knock Knock, Who’s There? With Justin Chen and Dhruv Jauhar | Current 2022HostedbyConfluent
 
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...HostedbyConfluent
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackRich Lee
 
Spark on Yarn
Spark on YarnSpark on Yarn
Spark on YarnQubole
 
Introduction to apache kafka
Introduction to apache kafkaIntroduction to apache kafka
Introduction to apache kafkaSamuel Kerrien
 
Streaming in Practice - Putting Apache Kafka in Production
Streaming in Practice - Putting Apache Kafka in ProductionStreaming in Practice - Putting Apache Kafka in Production
Streaming in Practice - Putting Apache Kafka in Productionconfluent
 
Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example
Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life ExampleKafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example
Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Exampleconfluent
 
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...confluent
 
Spark Summit EU talk by Luca Canali
Spark Summit EU talk by Luca CanaliSpark Summit EU talk by Luca Canali
Spark Summit EU talk by Luca CanaliSpark Summit
 
Kafka meetup JP #3 - Engineering Apache Kafka at LINE
Kafka meetup JP #3 - Engineering Apache Kafka at LINEKafka meetup JP #3 - Engineering Apache Kafka at LINE
Kafka meetup JP #3 - Engineering Apache Kafka at LINEkawamuray
 
Making MySQL Administration a Breeze - A Look Into a MySQL DBA's Toolchest
Making MySQL Administration a Breeze - A Look Into a MySQL DBA's ToolchestMaking MySQL Administration a Breeze - A Look Into a MySQL DBA's Toolchest
Making MySQL Administration a Breeze - A Look Into a MySQL DBA's ToolchestLenz Grimmer
 
Building Enterprise Search Engines using Open Source Technologies
Building Enterprise Search Engines using Open Source TechnologiesBuilding Enterprise Search Engines using Open Source Technologies
Building Enterprise Search Engines using Open Source TechnologiesRahul Singh
 
Building Enterprise Search Engines using Open Source Technologies
Building Enterprise Search Engines using Open Source TechnologiesBuilding Enterprise Search Engines using Open Source Technologies
Building Enterprise Search Engines using Open Source TechnologiesAnant Corporation
 
Akka Streams And Kafka Streams: Where Microservices Meet Fast Data
Akka Streams And Kafka Streams: Where Microservices Meet Fast DataAkka Streams And Kafka Streams: Where Microservices Meet Fast Data
Akka Streams And Kafka Streams: Where Microservices Meet Fast DataLightbend
 
ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)Mathew Beane
 
Productionizing Spark and the Spark Job Server
Productionizing Spark and the Spark Job ServerProductionizing Spark and the Spark Job Server
Productionizing Spark and the Spark Job ServerEvan Chan
 
The Need For Speed - NxtGen Cambridge
The Need For Speed - NxtGen CambridgeThe Need For Speed - NxtGen Cambridge
The Need For Speed - NxtGen CambridgePhil Pursglove
 

Similar to Custom management apps for Kafka (20)

Real time Analytics with Apache Kafka and Apache Spark
Real time Analytics with Apache Kafka and Apache SparkReal time Analytics with Apache Kafka and Apache Spark
Real time Analytics with Apache Kafka and Apache Spark
 
Knock Knock, Who’s There? With Justin Chen and Dhruv Jauhar | Current 2022
Knock Knock, Who’s There? With Justin Chen and Dhruv Jauhar | Current 2022Knock Knock, Who’s There? With Justin Chen and Dhruv Jauhar | Current 2022
Knock Knock, Who’s There? With Justin Chen and Dhruv Jauhar | Current 2022
 
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stack
 
Spark on Yarn
Spark on YarnSpark on Yarn
Spark on Yarn
 
Introduction to apache kafka
Introduction to apache kafkaIntroduction to apache kafka
Introduction to apache kafka
 
Kafka blr-meetup-presentation - Kafka internals
Kafka blr-meetup-presentation - Kafka internalsKafka blr-meetup-presentation - Kafka internals
Kafka blr-meetup-presentation - Kafka internals
 
Streaming in Practice - Putting Apache Kafka in Production
Streaming in Practice - Putting Apache Kafka in ProductionStreaming in Practice - Putting Apache Kafka in Production
Streaming in Practice - Putting Apache Kafka in Production
 
Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example
Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life ExampleKafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example
Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example
 
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
Show Me Kafka Tools That Will Increase My Productivity! (Stephane Maarek, Dat...
 
Spark Summit EU talk by Luca Canali
Spark Summit EU talk by Luca CanaliSpark Summit EU talk by Luca Canali
Spark Summit EU talk by Luca Canali
 
Kafka meetup JP #3 - Engineering Apache Kafka at LINE
Kafka meetup JP #3 - Engineering Apache Kafka at LINEKafka meetup JP #3 - Engineering Apache Kafka at LINE
Kafka meetup JP #3 - Engineering Apache Kafka at LINE
 
Making MySQL Administration a Breeze - A Look Into a MySQL DBA's Toolchest
Making MySQL Administration a Breeze - A Look Into a MySQL DBA's ToolchestMaking MySQL Administration a Breeze - A Look Into a MySQL DBA's Toolchest
Making MySQL Administration a Breeze - A Look Into a MySQL DBA's Toolchest
 
Velocity - Edge UG
Velocity - Edge UGVelocity - Edge UG
Velocity - Edge UG
 
Building Enterprise Search Engines using Open Source Technologies
Building Enterprise Search Engines using Open Source TechnologiesBuilding Enterprise Search Engines using Open Source Technologies
Building Enterprise Search Engines using Open Source Technologies
 
Building Enterprise Search Engines using Open Source Technologies
Building Enterprise Search Engines using Open Source TechnologiesBuilding Enterprise Search Engines using Open Source Technologies
Building Enterprise Search Engines using Open Source Technologies
 
Akka Streams And Kafka Streams: Where Microservices Meet Fast Data
Akka Streams And Kafka Streams: Where Microservices Meet Fast DataAkka Streams And Kafka Streams: Where Microservices Meet Fast Data
Akka Streams And Kafka Streams: Where Microservices Meet Fast Data
 
ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)
 
Productionizing Spark and the Spark Job Server
Productionizing Spark and the Spark Job ServerProductionizing Spark and the Spark Job Server
Productionizing Spark and the Spark Job Server
 
The Need For Speed - NxtGen Cambridge
The Need For Speed - NxtGen CambridgeThe Need For Speed - NxtGen Cambridge
The Need For Speed - NxtGen Cambridge
 

More from Sotaro Kimura

スキーマ 付き 分散ストリーム処理 を実行可能な FlinkSQLClient の紹介
スキーマ 付き 分散ストリーム処理 を実行可能な FlinkSQLClient の紹介スキーマ 付き 分散ストリーム処理 を実行可能な FlinkSQLClient の紹介
スキーマ 付き 分散ストリーム処理 を実行可能な FlinkSQLClient の紹介Sotaro Kimura
 
Spark Structured Streaming with Kafka
Spark Structured Streaming with KafkaSpark Structured Streaming with Kafka
Spark Structured Streaming with KafkaSotaro Kimura
 
Modern stream processing by Spark Structured Streaming
Modern stream processing by Spark Structured StreamingModern stream processing by Spark Structured Streaming
Modern stream processing by Spark Structured StreamingSotaro Kimura
 
Spark Structured StreamingでKafkaクラスタのデータをお手軽活用
Spark Structured StreamingでKafkaクラスタのデータをお手軽活用Spark Structured StreamingでKafkaクラスタのデータをお手軽活用
Spark Structured StreamingでKafkaクラスタのデータをお手軽活用Sotaro Kimura
 
Kinesis Analyticsの適用できない用途と、Kinesis Firehoseの苦労話
Kinesis Analyticsの適用できない用途と、Kinesis Firehoseの苦労話Kinesis Analyticsの適用できない用途と、Kinesis Firehoseの苦労話
Kinesis Analyticsの適用できない用途と、Kinesis Firehoseの苦労話Sotaro Kimura
 
利用者主体で行う分析のための分析基盤
利用者主体で行う分析のための分析基盤利用者主体で行う分析のための分析基盤
利用者主体で行う分析のための分析基盤Sotaro Kimura
 
最近のストリーム処理事情振り返り
最近のストリーム処理事情振り返り最近のストリーム処理事情振り返り
最近のストリーム処理事情振り返りSotaro Kimura
 
Stream dataprocessing101
Stream dataprocessing101Stream dataprocessing101
Stream dataprocessing101Sotaro Kimura
 
Apache NiFiと 他プロダクトのつなぎ方
Apache NiFiと他プロダクトのつなぎ方Apache NiFiと他プロダクトのつなぎ方
Apache NiFiと 他プロダクトのつなぎ方Sotaro Kimura
 
Kafkaを活用するためのストリーム処理の基本
Kafkaを活用するためのストリーム処理の基本Kafkaを活用するためのストリーム処理の基本
Kafkaを活用するためのストリーム処理の基本Sotaro Kimura
 
Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~
Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~
Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~Sotaro Kimura
 
JVM上でのストリーム処理エンジンの変遷
JVM上でのストリーム処理エンジンの変遷JVM上でのストリーム処理エンジンの変遷
JVM上でのストリーム処理エンジンの変遷Sotaro Kimura
 
スキーマつきストリーム データ処理基盤、 Confluent Platformとは?
スキーマつきストリーム データ処理基盤、 Confluent Platformとは?スキーマつきストリーム データ処理基盤、 Confluent Platformとは?
スキーマつきストリーム データ処理基盤、 Confluent Platformとは?Sotaro Kimura
 
Gearpump, akka based Distributed Reactive Realtime Engine
Gearpump, akka based Distributed Reactive Realtime EngineGearpump, akka based Distributed Reactive Realtime Engine
Gearpump, akka based Distributed Reactive Realtime EngineSotaro Kimura
 
リアルタイム処理エンジン Gearpumpの紹介
リアルタイム処理エンジンGearpumpの紹介リアルタイム処理エンジンGearpumpの紹介
リアルタイム処理エンジン Gearpumpの紹介Sotaro Kimura
 

More from Sotaro Kimura (15)

スキーマ 付き 分散ストリーム処理 を実行可能な FlinkSQLClient の紹介
スキーマ 付き 分散ストリーム処理 を実行可能な FlinkSQLClient の紹介スキーマ 付き 分散ストリーム処理 を実行可能な FlinkSQLClient の紹介
スキーマ 付き 分散ストリーム処理 を実行可能な FlinkSQLClient の紹介
 
Spark Structured Streaming with Kafka
Spark Structured Streaming with KafkaSpark Structured Streaming with Kafka
Spark Structured Streaming with Kafka
 
Modern stream processing by Spark Structured Streaming
Modern stream processing by Spark Structured StreamingModern stream processing by Spark Structured Streaming
Modern stream processing by Spark Structured Streaming
 
Spark Structured StreamingでKafkaクラスタのデータをお手軽活用
Spark Structured StreamingでKafkaクラスタのデータをお手軽活用Spark Structured StreamingでKafkaクラスタのデータをお手軽活用
Spark Structured StreamingでKafkaクラスタのデータをお手軽活用
 
Kinesis Analyticsの適用できない用途と、Kinesis Firehoseの苦労話
Kinesis Analyticsの適用できない用途と、Kinesis Firehoseの苦労話Kinesis Analyticsの適用できない用途と、Kinesis Firehoseの苦労話
Kinesis Analyticsの適用できない用途と、Kinesis Firehoseの苦労話
 
利用者主体で行う分析のための分析基盤
利用者主体で行う分析のための分析基盤利用者主体で行う分析のための分析基盤
利用者主体で行う分析のための分析基盤
 
最近のストリーム処理事情振り返り
最近のストリーム処理事情振り返り最近のストリーム処理事情振り返り
最近のストリーム処理事情振り返り
 
Stream dataprocessing101
Stream dataprocessing101Stream dataprocessing101
Stream dataprocessing101
 
Apache NiFiと 他プロダクトのつなぎ方
Apache NiFiと他プロダクトのつなぎ方Apache NiFiと他プロダクトのつなぎ方
Apache NiFiと 他プロダクトのつなぎ方
 
Kafkaを活用するためのストリーム処理の基本
Kafkaを活用するためのストリーム処理の基本Kafkaを活用するためのストリーム処理の基本
Kafkaを活用するためのストリーム処理の基本
 
Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~
Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~
Hadoop基盤上のETL構築実践例 ~多様なデータをどう扱う?~
 
JVM上でのストリーム処理エンジンの変遷
JVM上でのストリーム処理エンジンの変遷JVM上でのストリーム処理エンジンの変遷
JVM上でのストリーム処理エンジンの変遷
 
スキーマつきストリーム データ処理基盤、 Confluent Platformとは?
スキーマつきストリーム データ処理基盤、 Confluent Platformとは?スキーマつきストリーム データ処理基盤、 Confluent Platformとは?
スキーマつきストリーム データ処理基盤、 Confluent Platformとは?
 
Gearpump, akka based Distributed Reactive Realtime Engine
Gearpump, akka based Distributed Reactive Realtime EngineGearpump, akka based Distributed Reactive Realtime Engine
Gearpump, akka based Distributed Reactive Realtime Engine
 
リアルタイム処理エンジン Gearpumpの紹介
リアルタイム処理エンジンGearpumpの紹介リアルタイム処理エンジンGearpumpの紹介
リアルタイム処理エンジン Gearpumpの紹介
 

Recently uploaded

Introduction to Machine Learning Unit-2 Notes for II-II Mechanical Engineering
Introduction to Machine Learning Unit-2 Notes for II-II Mechanical EngineeringIntroduction to Machine Learning Unit-2 Notes for II-II Mechanical Engineering
Introduction to Machine Learning Unit-2 Notes for II-II Mechanical EngineeringC Sai Kiran
 
A brief about Jeypore Sub-station Presentation
A brief about Jeypore Sub-station PresentationA brief about Jeypore Sub-station Presentation
A brief about Jeypore Sub-station PresentationJeyporess2021
 
Caltrans District 8 Update for the CalAPA Spring Asphalt Conference 2024
Caltrans District 8 Update for the CalAPA Spring Asphalt Conference 2024Caltrans District 8 Update for the CalAPA Spring Asphalt Conference 2024
Caltrans District 8 Update for the CalAPA Spring Asphalt Conference 2024California Asphalt Pavement Association
 
PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...
PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...
PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...Luuk Brederode
 
Introduction to Data Structures .
Introduction to Data Structures        .Introduction to Data Structures        .
Introduction to Data Structures .Ashutosh Satapathy
 
First Review Group 1 PPT.pptx with slide
First Review Group 1 PPT.pptx with slideFirst Review Group 1 PPT.pptx with slide
First Review Group 1 PPT.pptx with slideMonika860882
 
Field Report on present condition of Ward 1 and Ward 2 of Pabna Municipality
Field Report on present condition of Ward 1 and Ward 2 of Pabna MunicipalityField Report on present condition of Ward 1 and Ward 2 of Pabna Municipality
Field Report on present condition of Ward 1 and Ward 2 of Pabna MunicipalityMorshed Ahmed Rahath
 
Evaluation Methods for Social XR Experiences
Evaluation Methods for Social XR ExperiencesEvaluation Methods for Social XR Experiences
Evaluation Methods for Social XR ExperiencesMark Billinghurst
 
Research paper publications: Meaning of Q1 Q2 Q3 Q4 Journal
Research paper publications: Meaning of Q1 Q2 Q3 Q4 JournalResearch paper publications: Meaning of Q1 Q2 Q3 Q4 Journal
Research paper publications: Meaning of Q1 Q2 Q3 Q4 JournalDr. Manjunatha. P
 
The Journey of Process Safety Management: Past, Present, and Future Trends
The Journey of Process Safety Management: Past, Present, and Future TrendsThe Journey of Process Safety Management: Past, Present, and Future Trends
The Journey of Process Safety Management: Past, Present, and Future Trendssoginsider
 
عناصر نباتية PDF.pdf architecture engineering
عناصر نباتية PDF.pdf architecture engineeringعناصر نباتية PDF.pdf architecture engineering
عناصر نباتية PDF.pdf architecture engineeringmennamohamed200y
 
Conventional vs Modern method (Philosophies) of Tunneling-re.pptx
Conventional vs Modern method (Philosophies) of Tunneling-re.pptxConventional vs Modern method (Philosophies) of Tunneling-re.pptx
Conventional vs Modern method (Philosophies) of Tunneling-re.pptxSAQIB KHURSHEED WANI
 
Injection Power Cycle - The most efficient power cycle
Injection Power Cycle - The most efficient power cycleInjection Power Cycle - The most efficient power cycle
Injection Power Cycle - The most efficient power cyclemarijomiljkovic1
 
Searching and Sorting Algorithms
Searching and Sorting AlgorithmsSearching and Sorting Algorithms
Searching and Sorting AlgorithmsAshutosh Satapathy
 
Chapter 2 Canal Falls at Mnnit Allahabad .pptx
Chapter 2 Canal Falls at Mnnit Allahabad .pptxChapter 2 Canal Falls at Mnnit Allahabad .pptx
Chapter 2 Canal Falls at Mnnit Allahabad .pptxButcher771
 
12. Stairs by U Nyi Hla ngae from Myanmar.pdf
12. Stairs by U Nyi Hla ngae from Myanmar.pdf12. Stairs by U Nyi Hla ngae from Myanmar.pdf
12. Stairs by U Nyi Hla ngae from Myanmar.pdftpo482247
 
autonomous_vehicle_working_paper_01072020-_508_compliant.pdf
autonomous_vehicle_working_paper_01072020-_508_compliant.pdfautonomous_vehicle_working_paper_01072020-_508_compliant.pdf
autonomous_vehicle_working_paper_01072020-_508_compliant.pdfPandurangGurakhe
 

Recently uploaded (20)

Caltrans view on recycling of in-place asphalt pavements
Caltrans view on recycling of in-place asphalt pavementsCaltrans view on recycling of in-place asphalt pavements
Caltrans view on recycling of in-place asphalt pavements
 
Introduction to Machine Learning Unit-2 Notes for II-II Mechanical Engineering
Introduction to Machine Learning Unit-2 Notes for II-II Mechanical EngineeringIntroduction to Machine Learning Unit-2 Notes for II-II Mechanical Engineering
Introduction to Machine Learning Unit-2 Notes for II-II Mechanical Engineering
 
A brief about Jeypore Sub-station Presentation
A brief about Jeypore Sub-station PresentationA brief about Jeypore Sub-station Presentation
A brief about Jeypore Sub-station Presentation
 
Caltrans District 8 Update for the CalAPA Spring Asphalt Conference 2024
Caltrans District 8 Update for the CalAPA Spring Asphalt Conference 2024Caltrans District 8 Update for the CalAPA Spring Asphalt Conference 2024
Caltrans District 8 Update for the CalAPA Spring Asphalt Conference 2024
 
PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...
PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...
PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...
 
Introduction to Data Structures .
Introduction to Data Structures        .Introduction to Data Structures        .
Introduction to Data Structures .
 
First Review Group 1 PPT.pptx with slide
First Review Group 1 PPT.pptx with slideFirst Review Group 1 PPT.pptx with slide
First Review Group 1 PPT.pptx with slide
 
Update on the latest research with regard to RAP
Update on the latest research with regard to RAPUpdate on the latest research with regard to RAP
Update on the latest research with regard to RAP
 
FOREST FIRE USING IoT-A Visual to UG students
FOREST FIRE USING IoT-A Visual to UG studentsFOREST FIRE USING IoT-A Visual to UG students
FOREST FIRE USING IoT-A Visual to UG students
 
Field Report on present condition of Ward 1 and Ward 2 of Pabna Municipality
Field Report on present condition of Ward 1 and Ward 2 of Pabna MunicipalityField Report on present condition of Ward 1 and Ward 2 of Pabna Municipality
Field Report on present condition of Ward 1 and Ward 2 of Pabna Municipality
 
Evaluation Methods for Social XR Experiences
Evaluation Methods for Social XR ExperiencesEvaluation Methods for Social XR Experiences
Evaluation Methods for Social XR Experiences
 
Research paper publications: Meaning of Q1 Q2 Q3 Q4 Journal
Research paper publications: Meaning of Q1 Q2 Q3 Q4 JournalResearch paper publications: Meaning of Q1 Q2 Q3 Q4 Journal
Research paper publications: Meaning of Q1 Q2 Q3 Q4 Journal
 
The Journey of Process Safety Management: Past, Present, and Future Trends
The Journey of Process Safety Management: Past, Present, and Future TrendsThe Journey of Process Safety Management: Past, Present, and Future Trends
The Journey of Process Safety Management: Past, Present, and Future Trends
 
عناصر نباتية PDF.pdf architecture engineering
عناصر نباتية PDF.pdf architecture engineeringعناصر نباتية PDF.pdf architecture engineering
عناصر نباتية PDF.pdf architecture engineering
 
Conventional vs Modern method (Philosophies) of Tunneling-re.pptx
Conventional vs Modern method (Philosophies) of Tunneling-re.pptxConventional vs Modern method (Philosophies) of Tunneling-re.pptx
Conventional vs Modern method (Philosophies) of Tunneling-re.pptx
 
Injection Power Cycle - The most efficient power cycle
Injection Power Cycle - The most efficient power cycleInjection Power Cycle - The most efficient power cycle
Injection Power Cycle - The most efficient power cycle
 
Searching and Sorting Algorithms
Searching and Sorting AlgorithmsSearching and Sorting Algorithms
Searching and Sorting Algorithms
 
Chapter 2 Canal Falls at Mnnit Allahabad .pptx
Chapter 2 Canal Falls at Mnnit Allahabad .pptxChapter 2 Canal Falls at Mnnit Allahabad .pptx
Chapter 2 Canal Falls at Mnnit Allahabad .pptx
 
12. Stairs by U Nyi Hla ngae from Myanmar.pdf
12. Stairs by U Nyi Hla ngae from Myanmar.pdf12. Stairs by U Nyi Hla ngae from Myanmar.pdf
12. Stairs by U Nyi Hla ngae from Myanmar.pdf
 
autonomous_vehicle_working_paper_01072020-_508_compliant.pdf
autonomous_vehicle_working_paper_01072020-_508_compliant.pdfautonomous_vehicle_working_paper_01072020-_508_compliant.pdf
autonomous_vehicle_working_paper_01072020-_508_compliant.pdf
 

Custom management apps for Kafka

  • 1. Custom management apps for Kafka 2018/12/18 「Apache Kafka Meetup Japan #5 @LINE」 Kimura, Sotaro(@kimutansk) https://www.flickr.com/photos/wwworks/8693876413/
  • 2. I am... • Kimura, Sotaro(@kimutansk) – Data Engineer IT choreman at Dwango • Data Analytics Infrastracture team • On premiss~Cloud, SRE~Individual function • Batch / Stream processing, developer / leader – In other words, IT choreman – Language, Scala/Java/Python/Ruby/Go etc... – Favorite • Stream data processing • Kafka, Flink, Fluend, Spark, Pulsar etc...
  • 3. Systems of our team abstract Otherteam’s system Explanatorynotes Externalreq/res Internalreq/res Job execution Monitor/Manage
  • 4. Systems of our team abstract Otherteam’s system Explanatorynotes Externalreq/res Internalreq/res Job execution Monitor/Manage
  • 5. Kafka monitoring/manage funcs • Topic setting manage • User account / ACLs manage • Topic list documents page generate • Resource monitoring – CPU utilization, Memory usage, Disk usage, IO wait – Network processor / Request handler idle rates • Flow manage – Auto recommend expanding partitions
  • 6. Kafka monitoring/manage funcs • Topic setting manage • User account / ACLs manage • Topic list documents page generate • Resource monitoring – CPU utilization, Memory usage, Disk usage, IO wait – Network processor / Request handler idle rates • Flow manage – Auto recommend expanding partitions Today’s target.
  • 7. Topic setting manage • Operating kafka from CLI – Topic create – Topic partition expand – Topic specific setting $ ./kafka-topics.sh --zookeeper kafka01:2181--create --topic sample_topic01 $ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --partitions 3 $ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --config cleanup.policy=compact
  • 8. Topic setting manage • Operating kafka from CLI – Topic create – Topic partition expand – Topic specific setting $ ./kafka-topics.sh --zookeeper kafka01:2181--create --topic sample_topic01 $ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --partitions 3 $ ./kafka-topics.sh --zookeeper kafka01:2181--alter --topic sample_topic01 --config cleanup.policy=compact Too hard of human’s operation !
  • 9. User account / ACLs manage • Premise – We decided to use SSL for Kafka AuthN/ACLs manage. • Because, we investigated it at about 2017/04. • At that time, ruby-kafka didn’t support SASL. – In ruby-kafka v0.4.0, supported :) – In our team, AuthN is not applied. • AuthN is only applied to the access from other teams.
  • 10. User account / ACLs manage • Operating kafka from CLI – Add consumer permission to user – Add producer permission to user – Remove producer permission from user $ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-consumer01" --consumer -- topic sample_topic01 --group test_group01 $ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-producer01"--producer -- topic sample_topic01 $ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--remove --allow-principal"User:CN=test-producer01" --producer --topic sample_topic01
  • 11. User account / ACLs manage • Operating kafka from CLI – Add consumer permission to user – Add producer permission to user – Remove producer permission from user $ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-consumer01" --consumer -- topic sample_topic01 --group test_group01 $ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--add --allow-principal "User:CN=test-producer01"--producer -- topic sample_topic01 $ ./kafka-acls.sh --authorizer-properties zookeeper.connect=kafka01:2181--remove --allow-principal"User:CN=test-producer01" --producer --topic sample_topic01 These commands confuse us!
  • 12. Our needs • Operators don’t execute raw command. • Topic/ACLs settings are managed by file. – To enable reviewing each other. • So...
  • 13. The administrative client for Kafka • org.apache.kafka.clients.admin.AdminClient – Through Broker API Endpoint – Introduced in 0.11.0.0 – Manage below elements. • Broker Configuration • Topic • Partition • Topic Configuration • ACLs • ReplicaLogDir
  • 14. Developed apps(Topic manage) • KafkaTopicManageApp – Using org.apache.kafka.clients.admin.AdminClient – Process flow • Load application settings, topic settings. • Get topic information from kafka cluster. • Create topic with specified settings. • Update topic with specified settings. – Applying settings is idempotent, so app updates always.
  • 15. Developed apps(Topic manage) • KafkaTopicManageApp – Application setting bootstrapServers: - "kafka01:9092" - "kafka02:9092" - "kafka03:9092" defaultPartitions: 1 defaultReplicationFactor: 3
  • 16. Developed apps(Topic manage) • KafkaTopicManageApp – Topic setting topics: - name: TestTopicName partitionNum: 4 config: min.insync.replicas: 2 segment.bytes: 104857600
  • 17. Developed apps(ACLs manage) • KafkaAclManageApp – Using org.apache.kafka.clients.admin.AdminClient – Process flow • Load application settings, acl settings. • Apply acls to kafka cluster. – Applying settings is idempotent, so app applys always.
  • 18. Developed apps(Topic manage) • KafkaAclManageApp – Acl setting userAcls: - name: "CN=test_user01" authorized_consumer_topics: - "sample_topic01" - "sample_topic02" specific_consumer_group: - "test_user01_group01" - "test_user01_group02"
  • 19. Apps operation • Settings are managed by Github. • Update flow – Update settings. – PR & review – Execute apps from jenkins. • Check flow – Check difference between settings and kafka cluster periodically. – If differences exist, notify to operators.(Slack)
  • 20. Conclusion • Kafka cluster operation commands confuse us. • So, let’s automate by kafka manage tools or your hand made apps with AdminClient!
  • 21. Thank you for your attention! https://www.flickr.com/photos/savannahcorps/7409364642