Your SlideShare is downloading. ×
  • Like
Secure Kafka at
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Secure Kafka at


Presented at Kafka meetup 2014

Presented at Kafka meetup 2014

Published in Engineering , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Anand, you may also want to try Joe Stein's
    Are you sure you want to
    Your message goes here
  • Hi Rajasekar. Thank you for sharing this slide deck. Do you have any plans of open sourcing the REST interface to publish data to Kafka brokers. Does the rest interface work with Kafka version 0.8.1 and above.
    Are you sure you want to
    Your message goes here
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Secure Kafka at Rajasekar Elango - Lead Developer
  • 2. What I do? Work for Monitoring and Management Team We build tools for monitoring health and performance of infrastructure. Tools are used by Site Reliability and R&D development for troubleshooting, performance analysis, etc.
  • 3. Why Kafka? We have application servers grouped into multiple clusters and distributed across multiple datacenters. Build scalable, near real time monitoring framework that collects data from all production datacenters and pushes it to secure DMZ datacenter for aggregation and reporting. Monitoring data we ship are JMX Metrics, System metrics (cpu, load, memory) from application servers, custom database metrics from database nodes.
  • 4. Architecture App Servers Cluster Prod DC App Servers App Servers Graphite DMZ Kafka Cluster Cluster MM Kafka MM Kafka Kafka Kafka MM Kafka Prod DC Prod DC
  • 5. Architecture Zookeeper x 3 Broker x 5 Rest Interfa ce Graphite Consumer Graphite JMX Metrics Producer System Metrics Producer DB Metrics Producer Mirror maker x2 Production DMZ Zookeeper x 3 Broker x 5
  • 6. Components Rest Interface for abstracting producers. AVRO for data format specification and serialization. Producers - JMX Metric producer, collectd for system metrics, database metric producers. Consumers - Graphite Consumer. MirrorMaker - for cross datacenter replication.
  • 7. Secure Kafka Implementation We wanted to secure traffic across datacenter to prevent malicious client eavesdropping data Implemented SSL/TLS MutualAuth between broker and producer/ consumer to add encryption and authentication SSL Based socket channel based on JSSE doc Secure mode can be toggled on/off by secure=true|false property in Broker registers secure property in zookeeper.
  • 8. Secure Kafka Configuration secure=true security.config.file=config/ & security.config.file=config/
  • 9. Secure Kafka Configuration want.client.auth=true need.client.auth=true # Keystore file keystore=<path to server keystore> keystorePwd=<keystore password> keyPwd=<key password> # Truststore file truststore=<path to server truststore> truststorePwd=<truststore password>
  • 10. Secure Kafka Configuration # Keystore file keystore=<path to client keystore file> keystorePwd=<keystore password> keyPwd=<key password> # Truststore file truststore=<path to client truststore file> truststorePwd=<trust store password>
  • 11. Scripts Producer bin/ --broker-list localhost:9092:true -- security.config.file config/ --topic test Consumer bin/ --topic test --zookeeper localhost: 2181 --from-beginning --security.config.file config/
  • 12. Limitations Doesn’t provide authorization. Doesn’t use secure communication with Zookeeper. We implemented secure features branched off from older snapshot version of kafka 0.8 release.
  • 13. Demo bin/ config/ bin/ config/ bin/ --broker-list relango-ltmr.home: 9092:true --topic test < messages.txt bin/ --topic test --zookeeper localhost:2181 --from-beginning