Kafka Security

© Hortonworks Inc. 2014
Kafka Security
SSL, Kerberos & Authorization

Who Are We?
Sriharsha Chintalapani
Apache Kafka Committer
Apache Storm Committer & PMC
Parth Brahmbhatt
Apache Kafka Contributor
Apache Storm Committer & PMC

Kafka Security
• SSL ( wire encryption)
• SASL ( Kerberos )
• Authorizer (Topic/Host/User level Authorization)

SSL

Kafka Security – SSL
• Kafka networking
• A TCP server listening for incoming connections
• Uses Non-blocking network I/O
• When a client connects to a server it opens a socket channel on
server side and hands it over selector.
• Selector gets polled in a loop. It will wake up whenever there are
connections ready with data to be read or write.
• Long – living connections , once established it will be used to
read/write data until client closed or an exception occurs.

• Kafka networking

• Kafka SSL / SASL requirements
• No User-level API changes to clients
• Retain length-encoded Kafka protocols
• Client must authenticate before sending/receiving requests
• Kafka Channel
• Instead of using socket channel, we added KafkaChannel
which consists a TransportLayer, Authenticator.

• TransportLayer
• Handles network level byte transfers
• PlaintextTransportLayer
• SSLTransportLayer
• Authenticator
• A pluggable interface for authentication implementations
• SaslAuthenticator – Provides SASL handshake and
authenticated user.

KafkaChannel
TransportLayer
Authenticator
Kafka Server
handshake
authenticate

• SSL - Handshake
• Kafka Server configures with Keystore and Truststore
• Kafka Client also needs a truststore with Kafka Server
certificate added to the truststore.
• Keystore configuration on client side is optional unless user wants
client side authentication.

• KafkaChannel
• Before write or read application data , checks if the
channel.ready()
• A channel is ready if its established a connection and
authenticated. No-OP of PlaintextTransportLayer
• If a channel is not ready it goes through channel.prepare()
which internally calls transportLayer.handshake()

• Before sending any application data, both client and server
needs to go though SSL handshake
• SSLTransportLayer uses SSLEngine to establish a non-
blocking handshake.
• SSLEngine provides a state machine to go through several
steps of SSLhandshake

• SocketChannel read
• Returns encrypted data
• Decrypts the data and returns the length of the data from Kafka protocols
• SocketChannel Write
• Writes encrypted data onto channel
• Regular socketChannel returns length of the data written to socket.
• Incase of SSL since we encrypt the data we can’t return exact length written to
socket which will be more than actual data
• Its important to keep track length of data written to network. This signifies if we
successfully written data to the network or not and move on to next request.

• Principal Builder
• SSLTransportLayer gives hostname as authenticated user
• X509Certificate has lot more information about a client
identity.
• PrincipalBuilder provides interface to plug in a custom
PrincipalBuilder that has access to X509Certificate and can
construct a user string out of it.
• Authenticator can use this custom principal to add ACLs

• Performance Impact
• Decrease in throughput by 20%.
• Latency increased by 30 %
• KAFKA-2481 (Ben Stopford) has more details

• listeners=SSL://host.name:port
• ssl.keystore.location
• ssl.keystore.password
• ssl.key.password
• ssl.truststore.location
• ssl.truststore.password
• security.inter.broker.protocol (optional)

SASL/ Kerberos

Kafka Security – SASL
• Simple Authentication and Security Layer, or SASL
• Provides flexibility in using Login Mechanisms
• One can use Kerberos , LDAP or simple passwords to authenticate.
• JAAS Login
• Before client & server can handshake , they need to authenticate with
Kerberos or other Identity Provider.
• JAAS provides a pluggable way of providing user credentials. One can
easily add LDAP or other mechanism just by changing a config file.

• JAAS Config file
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
serviceName="kafka"
keyTab="/vagrant/keytabs/kafka1.keytab"
principal="kafka/host@EXAMPLE.COM";
};
KafkaConfig {
useKeyTab=true
storeKey=true
serviceName="kafka"
keyTab="/vagrant/keytabs/client1.keytab"
principal=”client/host@EXAMPLE.COM";
};

• SASL Authenticator
• Uses configured login credentials of JAAS config.
• Non-blocking handshake to establish clients identity
• Once handshake established , Kerberos principal name will be the
authenticated user.
• Can be layered with SSL for wire encryption or Plaintext incase of wire
encryption not needed.
• SASL can provide encryption but it has huge performance penalties

Client Broker
Connection
Mechanism list
Selected Mechanism & sasl data
Evaluate and Response
Sasl data
Client Authenticated

• Pass JAAS config file as jvm parameter
• -Djava.security.auth.login.config

Kafka Security – Resources
• SSL
• https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
• SASL
• https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61326390
• Vagrant Setup
• SASL
• https://github.com/harshach/kafka-vagrant/tree/master/

Authorization

Authorizer
• Controls who can do what
• Pluggable
• Acl based approach

Acl
• Alice is Allowed to Read from Orders-topic from Host-1
Principal Permission Operation Resource Host
Alice Allow Read Orders Host-1

Principal
• PrincipalType:Name
• Supported types: User and Group
• Extensible so users can add their own types
• Wild Card User:*

Operation
• Read, Write, Create, Delete, Alter, Describe,
ClusterAction, All
• Each API as an Operation VS Classification that maps to
APIs.

Resource
• ResourceType:ResourceName
• Topic, Cluster and ConsumerGroup
• Wild card resource ResourceType:*

Permissions
• Allow and Deny
• Anyone without an explicit Allow ACL is denied
• Then why do we have Deny?
• Deny works as negation
• Deny takes precedence over Allow Acls

Hosts
• Why provide this granularity?
• Allows authorizer to provide firewall type security even in
non secure environment.
• * as Wild card.

Configuration
• Authorizer class
• Super users
• Authorizer properties
• Default behavior for resources with no ACLs

SimpleAclAuthorizer
• Out of box authorizer implementation.
• Stores all of its ACLs in zookeeper.
• In built ACL cache to avoid performance penalty.
• Provides authorizer audit log.

CLI
• Add, Remove and List acls
• Convenience options:
--producer and --consumer.

Ranger Policy

Ranger Auditing

Ranger ACL management Audit

Unsecure zookeeper

Zookeeper
• Kafka’s metadata store
• Has its own security mechanism that supports SASL and
MD5-DIGEST for establishing identity and ACL based
authorization
• Create , Delete directly interacts with zookeeper

Securing zookeeper
• Acl on zk nodes: user:cdrwa
• Zookeeper.set.acl
• ZkSecurityMigrator script
• Credit where its due: Flavio Junqueira

Client JAAS
Client {
useKeyTab=true
storeKey=true
serviceName="zookeeper"
keyTab="/vagrant/keytabs/kafka.keytab"
principal="kafka/kafka@WITZEND.COM";
};

Future
• KIP-4: Move everything to server side, no direct
interactions with zookeeper
• Group Support (PR already available)
• Pluggable Auditor

Summary
• SSL for wire encryption
• Sasl for authentication
• Authorization
• Secure Zookeeper
Thanks to the community for participation.

Kafka Security

More Related Content

What's hot

Viewers also liked

Similar to Kafka Security

Recently uploaded

Kafka Security