Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Page1 © Hortonworks Inc. 2014
Apache Kafka Security
SSL, Kerberos & Authorization
Manikumar Reddy
Hortonworks
@omkreddy
Page2 © Hortonworks Inc. 2014
Kafka Security Authors
Sriharsha Chintalapani
Apache Kafka Committer
Apache Storm Committer ...
Page3 © Hortonworks Inc. 2014
Why Kafka Security?
• Kafka is becoming centralized data bus connecting
external data source...
Page4 © Hortonworks Inc. 2014
Kafka Security - Overview
• Wire encryption and Authentication via SSL
• Role Based authenti...
Page5 © Hortonworks Inc. 2014
Authentication
• Brokers support listening for connections on multiple
ports
• Plain text (n...
Page6 © Hortonworks Inc. 2014
Kafka Security – SSL
• Kafka SSL / SASL requirements
• No User-level API changes to clients
...
Page7 © Hortonworks Inc. 2014
Kafka Networking
KafkaChannel
TransportLayer
Authenticator
Kafka Server
handshake
authentica...
Page8 © Hortonworks Inc. 2014
Kafka Security – SSL
Page9 © Hortonworks Inc. 2014
Kafka Security – SSL
• Principal Builder
• By default, SSL user name will be of the form
"CN...
Page10 © Hortonworks Inc. 2014
Kafka Security – SSL
• Broker Configs:
• listeners=SSL://host.name:port
• ssl.keystore.loca...
Page11 © Hortonworks Inc. 2014
Kafka Security – SSL
• Client Configs:
• security.protocol=SSL
• ssl.truststore.location=/v...
Page12 © Hortonworks Inc. 2014
Kafka Security – SASL
• Simple Authentication and Security Layer, or SASL
• Provides flexib...
Page13 © Hortonworks Inc. 2014
Kafka Security – SASL
Client Broker
Connection
Mechanism list
Selected Mechanism & sasl dat...
Page14 © Hortonworks Inc. 2014
Kafka Security – SASL
• Prepare JAAS Config file
KafkaServer {
com.sun.security.auth.module...
Page15 © Hortonworks Inc. 2014
Kafka Security – SASL
• Kerberos principal name
• {username}/{hostname}@{REALM}
• Ex: kafka...
Page16 © Hortonworks Inc. 2014
Kafka Security – Resources
• SSL
• http://kafka.apache.org/documentation.html#security_ssl
...
Page17 © Hortonworks Inc. 2014
Authorizer
• Controls who can do what
• Pluggable
• Acl based approach
Page18 © Hortonworks Inc. 2014
Acl
• Alice is Allowed to Read from Orders-topic from Host-1
Principal Permission Operation...
Page19 © Hortonworks Inc. 2014
Principal
• PrincipalType:Name
• Supported types: User
• Extensible so users can add their ...
Page20 © Hortonworks Inc. 2014
Operations and Resources
• Operation
• Read, Write, Create, Delete, Describe, ClusterAction...
Page21 © Hortonworks Inc. 2014
Permissions
• Allow and Deny
• Anyone without an explicit Allow ACL is denied
• Deny works ...
Page22 © Hortonworks Inc. 2014
Hosts
• Allows authorizer to provide firewall type security even in
non secure environment....
Page23 © Hortonworks Inc. 2014
Configuration
• Authorizer class
• Super users
• Authorizer properties
• Default behavior f...
Page24 © Hortonworks Inc. 2014
SimpleAclAuthorizer
• Out of box authorizer implementation.
• Stores all of its ACLs in zoo...
Page25 © Hortonworks Inc. 2014
Client Broker Authorizer Zookeeper
configure
Read ACLs
Load
Cache
Request
authorize
ACL mat...
Page27 © Hortonworks Inc. 2014
CLI
• Add, Remove and List acls
• Convenience options:
– Producer
bin/kafka-acls.sh --autho...
Page28 © Hortonworks Inc. 2014
Ranger Policy
Page29 © Hortonworks Inc. 2014
Ranger Auditing
Page30 © Hortonworks Inc. 2014
Securing Zookeeper
• Kafka’s metadata store , ACLs
• Create , Delete directly interacts wit...
Page31 © Hortonworks Inc. 2014
Client JAAS
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
s...
Page32 © Hortonworks Inc. 2014
Future
• KIP-4 (Admin API): Move everything to server side, no
direct interactions with zoo...
Page33 © Hortonworks Inc. 2014
Apache Kafka 0.10.0.0
• New Client Library, Kafka Streams
• New timestamp field for message...
Page34 © Hortonworks Inc. 2014
Summary
• SSL for wire encryption
• SASL for authentication
• Authorization
• Secure Zookee...
Page35 © Hortonworks Inc. 2014 35
Page36 © Hortonworks Inc. 2014
Kafka Networking
Page37 © Hortonworks Inc. 2014
Kafka Networking
http://www.slideshare.net/jjkoshy/troubleshooting-kafkas-socket-server-fro...
Page38 © Hortonworks Inc. 2014
Kafka Networking
Page39 © Hortonworks Inc. 2014
Kafka Security – SSL
• SSLTransportLayer
• Before sending any application data, both client...
Page40 © Hortonworks Inc. 2014
Kafka Security – SSL
• SSLTransportLayer
• SocketChannel read
• Returns encrypted data
• De...
Upcoming SlideShare
Loading in …5
×

Apache Kafka Security

6,168 views

Published on

Apache Kafka Security

Published in: Technology

Apache Kafka Security

  1. 1. Page1 © Hortonworks Inc. 2014 Apache Kafka Security SSL, Kerberos & Authorization Manikumar Reddy Hortonworks @omkreddy
  2. 2. Page2 © Hortonworks Inc. 2014 Kafka Security Authors Sriharsha Chintalapani Apache Kafka Committer Apache Storm Committer & PMC Parth Brahmbhatt Apache Kafka Contributor Apache Storm Committer & PMC
  3. 3. Page3 © Hortonworks Inc. 2014 Why Kafka Security? • Kafka is becoming centralized data bus connecting external data sources to Hadoop eco system. • There are lot of requests/discussions in Kafka mailing lists to add security
  4. 4. Page4 © Hortonworks Inc. 2014 Kafka Security - Overview • Wire encryption and Authentication via SSL • Role Based authentication via SASL ( Kerberos, Plaintext) • Authorizer to add fine-grain access controls to Kafka topics per User, per Host.
  5. 5. Page5 © Hortonworks Inc. 2014 Authentication • Brokers support listening for connections on multiple ports • Plain text (no wire encryption/no authentication) • SSL (wire encryption/authentication) • SASL (Kerberos/Plain text authentication) • SSL + SASL ( SSL for wire encryption + SASL for authentication) Ex: listeners=PLAINTEXT://host.name:port,SSL://host.name:port
  6. 6. Page6 © Hortonworks Inc. 2014 Kafka Security – SSL • Kafka SSL / SASL requirements • No User-level API changes to clients • Retain length-encoded Kafka protocols • Client must authenticate before sending/receiving requests • Kafka Channel • Instead of using socket channel, we added KafkaChannel which consists a TransportLayer, Authenticator.
  7. 7. Page7 © Hortonworks Inc. 2014 Kafka Networking KafkaChannel TransportLayer Authenticator Kafka Server handshake authenticate
  8. 8. Page8 © Hortonworks Inc. 2014 Kafka Security – SSL
  9. 9. Page9 © Hortonworks Inc. 2014 Kafka Security – SSL • Principal Builder • By default, SSL user name will be of the form "CN=hostname,OU=organizationunit,O=organization,L=locati on,ST=state,C=country". • X509Certificate has lot more information about a client identity. • PrincipalBuilder provides interface to plug in a custom PrincipalBuilder that has access to X509Certificate and can construct a user identity out of it.
  10. 10. Page10 © Hortonworks Inc. 2014 Kafka Security – SSL • Broker Configs: • listeners=SSL://host.name:port • ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks • ssl.keystore.password=test1234 • ssl.key.password=test1234 • ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks • ssl.truststore.password=test1234 • security.inter.broker.protocol=SSL • ssl.client.auth=true
  11. 11. Page11 © Hortonworks Inc. 2014 Kafka Security – SSL • Client Configs: • security.protocol=SSL • ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks • ssl.truststore.password=test1234 • ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks • ssl.keystore.password=test1234 • ssl.key.password=test1234
  12. 12. Page12 © Hortonworks Inc. 2014 Kafka Security – SASL • Simple Authentication and Security Layer, or SASL • Provides flexibility in using mechanisms • Challenge/Response protocols • Mechanisms : GSSAPI/Kerberos, clear text username/password, DIGEST- MD5 • JAAS Login • Before client & server can handshake , they need to authenticate with Kerberos or other Identity Provider. • JAAS provides a pluggable way of providing user credentials. One can easily add LDAP or other mechanism just by changing a config file. • Kafka supports GSSAPI/Kerberos, clear text username/password
  13. 13. Page13 © Hortonworks Inc. 2014 Kafka Security – SASL Client Broker Connection Mechanism list Selected Mechanism & sasl data Evaluate and Response Sasl data Client Authenticated
  14. 14. Page14 © Hortonworks Inc. 2014 Kafka Security – SASL • Prepare JAAS Config file KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true serviceName="kafka" keyTab="/vagrant/keytabs/kafka1.keytab" principal="kafka/host@EXAMPLE.COM"; }; KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true serviceName="kafka" keyTab="/vagrant/keytabs/client1.keytab" principal=”client/host@EXAMPLE.COM"; }; • Pass JAAS config file as jvm parameter. -Djava.security.auth.login.config • security.inter.broker.protocol=SASL_PLAINTEXT • security.protocol=SASL_PLAINTEXT
  15. 15. Page15 © Hortonworks Inc. 2014 Kafka Security – SASL • Kerberos principal name • {username}/{hostname}@{REALM} • Ex: kafka/kafka.host1.com@{TEST.COM} • {username} part taken as default principal • sasl.kerberos.principal.to.local.rules – customize principal name
  16. 16. Page16 © Hortonworks Inc. 2014 Kafka Security – Resources • SSL • http://kafka.apache.org/documentation.html#security_ssl • SASL • http://kafka.apache.org/documentation.html#security_sasl • Vagrant Setup • SASL • https://github.com/harshach/kafka-vagrant/tree/master/ • SSL • https://github.com/harshach/kafka-vagrant/tree/ssl/
  17. 17. Page17 © Hortonworks Inc. 2014 Authorizer • Controls who can do what • Pluggable • Acl based approach
  18. 18. Page18 © Hortonworks Inc. 2014 Acl • Alice is Allowed to Read from Orders-topic from Host-1 Principal Permission Operation Resource Host Alice Allow Read Orders Host-1
  19. 19. Page19 © Hortonworks Inc. 2014 Principal • PrincipalType:Name • Supported types: User • Extensible so users can add their own types • Wild Card User:*
  20. 20. Page20 © Hortonworks Inc. 2014 Operations and Resources • Operation • Read, Write, Create, Delete, Describe, ClusterAction, All • Resource • ResourceType:ResourceName • Topic, Cluster and ConsumerGroup • Wild card resource ResourceType:* • Topic -> Read, Write, Describe • ConsumerGroup -> Read • Cluster -> Create, ClusterAction
  21. 21. Page21 © Hortonworks Inc. 2014 Permissions • Allow and Deny • Anyone without an explicit Allow ACL is denied • Deny works as negation • Deny takes precedence over Allow Acls
  22. 22. Page22 © Hortonworks Inc. 2014 Hosts • Allows authorizer to provide firewall type security even in non secure environment. • * as Wild card.
  23. 23. Page23 © Hortonworks Inc. 2014 Configuration • Authorizer class • Super users • Authorizer properties • Default behavior for resources with no ACLs – allow.everyone.if.no.acl.found = false
  24. 24. Page24 © Hortonworks Inc. 2014 SimpleAclAuthorizer • Out of box authorizer implementation. • Stores all of its ACLs in zookeeper. • In built ACL cache to avoid performance penalty. • Provides authorizer audit log.
  25. 25. Page25 © Hortonworks Inc. 2014 Client Broker Authorizer Zookeeper configure Read ACLs Load Cache Request authorize ACL match Or Super User? Allowed/Den ied
  26. 26. Page27 © Hortonworks Inc. 2014 CLI • Add, Remove and List acls • Convenience options: – Producer bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --producer --topic Test-topic – Consumer bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --consumer --topic test-topic --group Group-1
  27. 27. Page28 © Hortonworks Inc. 2014 Ranger Policy
  28. 28. Page29 © Hortonworks Inc. 2014 Ranger Auditing
  29. 29. Page30 © Hortonworks Inc. 2014 Securing Zookeeper • Kafka’s metadata store , ACLs • Create , Delete directly interacts with zookeeper • Has its own security mechanism that supports SASL and MD5-DIGEST for establishing identity and ACL based authorization • Set zookeeper.set.acl = true • ZK paths are writable by brokers and readable by all
  30. 30. Page31 © Hortonworks Inc. 2014 Client JAAS Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true serviceName="zookeeper" keyTab="/vagrant/keytabs/kafka.keytab" principal="kafka/kafka@WITZEND.COM"; };
  31. 31. Page32 © Hortonworks Inc. 2014 Future • KIP-4 (Admin API): Move everything to server side, no direct interactions with zookeeper • Group Support • Pluggable Auditor
  32. 32. Page33 © Hortonworks Inc. 2014 Apache Kafka 0.10.0.0 • New Client Library, Kafka Streams • New timestamp field for messages • Balancing Replicas Across Racks • Authentication using SASL/PLAIN. • New Consumer configuration parameter 'max.poll.records'
  33. 33. Page34 © Hortonworks Inc. 2014 Summary • SSL for wire encryption • SASL for authentication • Authorization • Secure Zookeeper Thanks to the community for participation.
  34. 34. Page35 © Hortonworks Inc. 2014 35
  35. 35. Page36 © Hortonworks Inc. 2014 Kafka Networking
  36. 36. Page37 © Hortonworks Inc. 2014 Kafka Networking http://www.slideshare.net/jjkoshy/troubleshooting-kafkas-socket-server-from-incident-to-resolution
  37. 37. Page38 © Hortonworks Inc. 2014 Kafka Networking
  38. 38. Page39 © Hortonworks Inc. 2014 Kafka Security – SSL • SSLTransportLayer • Before sending any application data, both client and server needs to go though SSL handshake • SSLTransportLayer uses SSLEngine to establish a non- blocking handshake. • SSLEngine provides a state machine to go through several steps of SSLhandshake
  39. 39. Page40 © Hortonworks Inc. 2014 Kafka Security – SSL • SSLTransportLayer • SocketChannel read • Returns encrypted data • Decrypts the data and returns the length of the data from Kafka protocols • SocketChannel Write • Writes encrypted data onto channel • Regular socketChannel returns length of the data written to socket. • Incase of SSL since we encrypt the data we can’t return exact length written to socket which will be more than actual data • Its important to keep track length of data written to network. This signifies if we successfully written data to the network or not and move on to next request.

×