Your SlideShare is downloading. ×
0
Secure Kafka at Salesforce.com
Rajasekar Elango - Lead Developer
What I do?
Work for Monitoring and Management Team
We build tools for monitoring health and performance of
salesforce.com ...
Why Kafka?
We have application servers grouped into multiple clusters and
distributed across multiple datacenters.
Build s...
Architecture
App
Servers
Cluster
Prod DC
App
Servers
App
Servers
Graphite
DMZ
Kafka
Cluster Cluster
MM
Kafka
MM
Kafka Kafk...
Architecture
Zookeeper
x 3
Broker
x 5
Rest
Interfa
ce
Graphite
Consumer
Graphite
JMX Metrics
Producer
System Metrics
Produ...
Components
Rest Interface for abstracting producers.
AVRO for data format specification and serialization.
Producers - JMX...
Secure Kafka Implementation
We wanted to secure traffic across datacenter to prevent malicious
client eavesdropping data
I...
Secure Kafka Configuration
server.properties
secure=true
security.config.file=config/server.security.properties
producer.p...
Secure Kafka Configuration
server.security.properties
want.client.auth=true
need.client.auth=true
# Keystore file
keystore...
Secure Kafka Configuration
client.security.properties
# Keystore file
keystore=<path to client keystore file>
keystorePwd=...
Scripts
Producer
bin/kafka-console-producer.sh --broker-list localhost:9092:true --
security.config.file config/client.sec...
Limitations
Doesn’t provide authorization.
Doesn’t use secure communication with Zookeeper.
We implemented secure features...
Demo
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
bin/kafk...
Secure Kafka at Salesforce.com
Upcoming SlideShare
Loading in...5
×

Secure Kafka at Salesforce.com

2,125

Published on

Presented at Kafka meetup 2014

Published in: Engineering, Technology
2 Comments
6 Likes
Statistics
Notes
No Downloads
Views
Total Views
2,125
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
40
Comments
2
Likes
6
Embeds 0
No embeds

No notes for slide

Transcript of "Secure Kafka at Salesforce.com"

  1. 1. Secure Kafka at Salesforce.com Rajasekar Elango - Lead Developer
  2. 2. What I do? Work for Monitoring and Management Team We build tools for monitoring health and performance of salesforce.com infrastructure. Tools are used by Site Reliability and R&D development for troubleshooting, performance analysis, etc.
  3. 3. Why Kafka? We have application servers grouped into multiple clusters and distributed across multiple datacenters. Build scalable, near real time monitoring framework that collects data from all production datacenters and pushes it to secure DMZ datacenter for aggregation and reporting. Monitoring data we ship are JMX Metrics, System metrics (cpu, load, memory) from application servers, custom database metrics from database nodes.
  4. 4. Architecture App Servers Cluster Prod DC App Servers App Servers Graphite DMZ Kafka Cluster Cluster MM Kafka MM Kafka Kafka Kafka MM Kafka Prod DC Prod DC
  5. 5. Architecture Zookeeper x 3 Broker x 5 Rest Interfa ce Graphite Consumer Graphite JMX Metrics Producer System Metrics Producer DB Metrics Producer Mirror maker x2 Production DMZ Zookeeper x 3 Broker x 5
  6. 6. Components Rest Interface for abstracting producers. AVRO for data format specification and serialization. Producers - JMX Metric producer, collectd for system metrics, database metric producers. Consumers - Graphite Consumer. MirrorMaker - for cross datacenter replication.
  7. 7. Secure Kafka Implementation We wanted to secure traffic across datacenter to prevent malicious client eavesdropping data Implemented SSL/TLS MutualAuth between broker and producer/ consumer to add encryption and authentication SSL Based socket channel based on JSSE doc Secure mode can be toggled on/off by secure=true|false property in server.properties. Broker registers secure property in zookeeper.
  8. 8. Secure Kafka Configuration server.properties secure=true security.config.file=config/server.security.properties producer.properties & consumer.properties security.config.file=config/client.security.properties
  9. 9. Secure Kafka Configuration server.security.properties want.client.auth=true need.client.auth=true # Keystore file keystore=<path to server keystore> keystorePwd=<keystore password> keyPwd=<key password> # Truststore file truststore=<path to server truststore> truststorePwd=<truststore password>
  10. 10. Secure Kafka Configuration client.security.properties # Keystore file keystore=<path to client keystore file> keystorePwd=<keystore password> keyPwd=<key password> # Truststore file truststore=<path to client truststore file> truststorePwd=<trust store password>
  11. 11. Scripts Producer bin/kafka-console-producer.sh --broker-list localhost:9092:true -- security.config.file config/client.security.properties --topic test Consumer bin/kafka-console-consumer.sh --topic test --zookeeper localhost: 2181 --from-beginning --security.config.file config/ client.security.properties
  12. 12. Limitations Doesn’t provide authorization. Doesn’t use secure communication with Zookeeper. We implemented secure features branched off from older snapshot version of kafka 0.8 release.
  13. 13. Demo bin/zookeeper-server-start.sh config/zookeeper.properties bin/kafka-server-start.sh config/server.properties bin/kafka-console-producer.sh --broker-list relango-ltmr.home: 9092:true --topic test < messages.txt bin/kafka-console-consumer.sh --topic test --zookeeper localhost:2181 --from-beginning
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×