Secure Kafka at Salesforce.com
Rajasekar Elango - Lead Developer
What I do?
Work for Monitoring and Management Team
We build tools for monitoring health and performance of
Tools are used by Site Reliability and R&D development for
troubleshooting, performance analysis, etc.
We have application servers grouped into multiple clusters and
distributed across multiple datacenters.
Build scalable, near real time monitoring framework that collects data
from all production datacenters and pushes it to secure DMZ
datacenter for aggregation and reporting.
Monitoring data we ship are JMX Metrics, System metrics (cpu, load,
memory) from application servers, custom database metrics from
Prod DC Prod DC
Rest Interface for abstracting producers.
AVRO for data format specification and serialization.
Producers - JMX Metric producer, collectd for system metrics,
database metric producers.
Consumers - Graphite Consumer.
MirrorMaker - for cross datacenter replication.
Secure Kafka Implementation
We wanted to secure traffic across datacenter to prevent malicious
client eavesdropping data
Implemented SSL/TLS MutualAuth between broker and producer/
consumer to add encryption and authentication
SSL Based socket channel based on JSSE doc
Secure mode can be toggled on/off by secure=true|false property
Broker registers secure property in zookeeper.