Apache Kafka Security

© Hortonworks Inc. 2014
Apache Kafka Security
SSL, Kerberos & Authorization
Manikumar Reddy
Hortonworks
@omkreddy

Kafka Security Authors
Sriharsha Chintalapani
Apache Kafka Committer
Apache Storm Committer & PMC
Parth Brahmbhatt
Apache Kafka Contributor
Apache Storm Committer & PMC

Why Kafka Security?
• Kafka is becoming centralized data bus connecting
external data sources to Hadoop eco system.
• There are lot of requests/discussions in Kafka mailing
lists to add security

Kafka Security - Overview
• Wire encryption and Authentication via SSL
• Role Based authentication via SASL ( Kerberos,
Plaintext)
• Authorizer to add fine-grain access controls to Kafka
topics per User, per Host.

Authentication
• Brokers support listening for connections on multiple
ports
• Plain text (no wire encryption/no authentication)
• SSL (wire encryption/authentication)
• SASL (Kerberos/Plain text authentication)
• SSL + SASL ( SSL for wire encryption + SASL for
authentication)
Ex:
listeners=PLAINTEXT://host.name:port,SSL://host.name:port

Kafka Security – SSL
• Kafka SSL / SASL requirements
• No User-level API changes to clients
• Retain length-encoded Kafka protocols
• Client must authenticate before sending/receiving requests
• Kafka Channel
• Instead of using socket channel, we added KafkaChannel
which consists a TransportLayer, Authenticator.

Kafka Networking
KafkaChannel
TransportLayer
Authenticator
Kafka Server
handshake
authenticate

• Principal Builder
• By default, SSL user name will be of the form
"CN=hostname,OU=organizationunit,O=organization,L=locati
on,ST=state,C=country".
• X509Certificate has lot more information about a client
identity.
• PrincipalBuilder provides interface to plug in a custom
PrincipalBuilder that has access to X509Certificate and can
construct a user identity out of it.

• Broker Configs:
• listeners=SSL://host.name:port
• ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
• ssl.keystore.password=test1234
• ssl.key.password=test1234
• ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks
• ssl.truststore.password=test1234
• security.inter.broker.protocol=SSL
• ssl.client.auth=true

• Client Configs:
• security.protocol=SSL
• ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
• ssl.truststore.password=test1234
• ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
• ssl.keystore.password=test1234
• ssl.key.password=test1234

Kafka Security – SASL
• Simple Authentication and Security Layer, or SASL
• Provides flexibility in using mechanisms
• Challenge/Response protocols
• Mechanisms : GSSAPI/Kerberos, clear text username/password, DIGEST-
MD5
• JAAS Login
• Before client & server can handshake , they need to authenticate with
Kerberos or other Identity Provider.
• JAAS provides a pluggable way of providing user credentials. One can easily
add LDAP or other mechanism just by changing a config file.
• Kafka supports GSSAPI/Kerberos, clear text username/password

Client Broker
Connection
Mechanism list
Selected Mechanism & sasl data
Evaluate and Response
Sasl data
Client Authenticated

• Prepare JAAS Config file
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
serviceName="kafka"
keyTab="/vagrant/keytabs/kafka1.keytab"
principal="kafka/host@EXAMPLE.COM";
};
KafkaClient {
useKeyTab=true
storeKey=true
serviceName="kafka"
keyTab="/vagrant/keytabs/client1.keytab"
principal=”client/host@EXAMPLE.COM";
};
• Pass JAAS config file as jvm parameter. -Djava.security.auth.login.config
• security.inter.broker.protocol=SASL_PLAINTEXT
• security.protocol=SASL_PLAINTEXT

• Kerberos principal name
• {username}/{hostname}@{REALM}
• Ex: kafka/kafka.host1.com@{TEST.COM}
• {username} part taken as default principal
• sasl.kerberos.principal.to.local.rules – customize principal
name

Kafka Security – Resources
• SSL
• http://kafka.apache.org/documentation.html#security_ssl
• SASL
• http://kafka.apache.org/documentation.html#security_sasl
• Vagrant Setup
• SASL
• https://github.com/harshach/kafka-vagrant/tree/master/
• SSL
• https://github.com/harshach/kafka-vagrant/tree/ssl/

Authorizer
• Controls who can do what
• Pluggable
• Acl based approach

Acl
• Alice is Allowed to Read from Orders-topic from Host-1
Principal Permission Operation Resource Host
Alice Allow Read Orders Host-1

Principal
• PrincipalType:Name
• Supported types: User
• Extensible so users can add their own types
• Wild Card User:*

Operations and Resources
• Operation
• Read, Write, Create, Delete, Describe, ClusterAction, All
• Resource
• ResourceType:ResourceName
• Topic, Cluster and ConsumerGroup
• Wild card resource ResourceType:*
• Topic -> Read, Write, Describe
• ConsumerGroup -> Read
• Cluster -> Create, ClusterAction

Permissions
• Allow and Deny
• Anyone without an explicit Allow ACL is denied
• Deny works as negation
• Deny takes precedence over Allow Acls

Hosts
• Allows authorizer to provide firewall type security even in
non secure environment.
• * as Wild card.

Configuration
• Authorizer class
• Super users
• Authorizer properties
• Default behavior for resources with no ACLs
– allow.everyone.if.no.acl.found = false

SimpleAclAuthorizer
• Out of box authorizer implementation.
• Stores all of its ACLs in zookeeper.
• In built ACL cache to avoid performance penalty.
• Provides authorizer audit log.

Client Broker Authorizer Zookeeper
configure
Read ACLs
Load
Cache
Request
authorize
ACL match
Or Super User?
Allowed/Den
ied

CLI
• Add, Remove and List acls
• Convenience options:
– Producer
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
--add --allow-principal User:Bob --producer --topic Test-topic
– Consumer
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
--add --allow-principal User:Bob --consumer --topic test-topic --group Group-1

Ranger Policy

Ranger Auditing

Securing Zookeeper
• Kafka’s metadata store , ACLs
• Create , Delete directly interacts with zookeeper
• Has its own security mechanism that supports SASL and
MD5-DIGEST for establishing identity and ACL based
authorization
• Set zookeeper.set.acl = true
• ZK paths are writable by brokers and readable by all

Client JAAS
Client {
useKeyTab=true
storeKey=true
serviceName="zookeeper"
keyTab="/vagrant/keytabs/kafka.keytab"
principal="kafka/kafka@WITZEND.COM";
};

Future
• KIP-4 (Admin API): Move everything to server side, no
direct interactions with zookeeper
• Group Support
• Pluggable Auditor

Apache Kafka 0.10.0.0
• New Client Library, Kafka Streams
• New timestamp field for messages
• Balancing Replicas Across Racks
• Authentication using SASL/PLAIN.
• New Consumer configuration parameter 'max.poll.records'

Summary
• SSL for wire encryption
• SASL for authentication
• Authorization
• Secure Zookeeper
Thanks to the community for participation.

Kafka Networking

Kafka Networking
http://www.slideshare.net/jjkoshy/troubleshooting-kafkas-socket-server-from-incident-to-resolution

• SSLTransportLayer
• Before sending any application data, both client and server
needs to go though SSL handshake
• SSLTransportLayer uses SSLEngine to establish a non-
blocking handshake.
• SSLEngine provides a state machine to go through several
steps of SSLhandshake

• SSLTransportLayer
• SocketChannel read
• Returns encrypted data
• Decrypts the data and returns the length of the data from Kafka protocols
• SocketChannel Write
• Writes encrypted data onto channel
• Regular socketChannel returns length of the data written to socket.
• Incase of SSL since we encrypt the data we can’t return exact length written to
socket which will be more than actual data
• Its important to keep track length of data written to network. This signifies if we
successfully written data to the network or not and move on to next request.

Apache Kafka Security

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Apache Kafka Security

Similar to Apache Kafka Security (20)

More from DataWorks Summit/Hadoop Summit

More from DataWorks Summit/Hadoop Summit (20)

Recently uploaded

Recently uploaded (20)

Apache Kafka Security