This document discusses Kafka security and provides tips for implementing it. It covers the three main aspects of Kafka security: encryption, authentication, and authorization. For encryption, it explains how to set up SSL and discusses options for end-to-end encryption. Authentication details how to use SSL client authentication or SASL mechanisms like Kerberos or PLAIN. Authorization explains managing access control lists (ACLs) stored in Zookeeper to control access. The document concludes by emphasizing the challenges of securing Kafka clients and provides advice like creating standardized client wrappers and Docker images.

1. Kafka Security 101
& Real World Tips
Stephane Maarek - DataCumulus

2. My Kafka Security Journey
Stephane, implement
Kafka Security!

3. Who am I?
• I’m Stephane!
• Consultant & Solution Architect at DataCumulus
• Apache Kafka SeriesVideo Courses on Udemy
• Full productions deployments (with security)
• You can find me on
• GitHub: https://github.com/simplesteph
• LinkedIn: https://www.linkedin.com/in/stephanemaarek
• Medium: https://medium.com/@stephane.maarek
• Twitter: https://twitter.com/stephanemaarek
• Udemy: https://udemy.com/stephane-maarek

4. Who has not secured Kafka?

5. Kafka without Security is RISKY
5 disastrous scenarios
1. Read all your data
2. Write to any topic and break your consumers
3. Intercept and read plaintext network packets
4. Delete all your Kafka data in one command without SSH
5. Kafka Connect? Database Credentials are in a Kafka Topic, plaintext

6. You need Kafka Security
If you intend to make Kafka a cornerstone of your infrastructure

7. What’s Kafka Security?
Disclaimer: the source of truth is always the documentation

8. Kafka Security in three words
Encryption
Authentication
Authorization

9. Encryption in Kafka
• SSL encryption = secure communications
• Similar to HTTPS
Super secret
message
Kafka Client
(producer / consumer )
Kafka Brokers
Port 9093 - SSL
aGVsbG8gd29
ybGQgZWh…
Encrypted data
Kafka Client
(producer / consumer )
Kafka Brokers
Port 9092 - PLAINTEXT

10. SSL, Concretely?
• Create a Certificate Authority (CA)
• Generate certificates for your brokers, sign them
• Make sure your broker and clients trust the CA Root.
ssl.keystore.location=/home/ubuntu/ssl/kafka.server.keystore.jks
ssl.keystore.password=serversecret
ssl.key.password=serversecret
ssl.truststore.location=/home/ubuntu/ssl/kafka.server.truststore.jks
ssl.truststore.password=serversecret

11. SSL in the Real World
• SSL lowers the performance of your brokers
• You lose the zero-copy optimization
• Kafka heap usage increases
• CPU usage increases
• SSL only allows to encrypt data in flight
• Data at rest sits un-encrypted on Kafka Disk

12. What about end-to-end encryption?
• Closed source: Apple
• Open source:
• https://github.com/Quicksign/kafka-encryption
• https://github.com/nucypher/kafka-oss
• POC in progress at DataCumulus
Producer
Kafka
PLAINTEXT
Consumer
Encrypted
data
Encrypted
data
encrypt data decrypt data

13. Check Point
Encryption
Authentication
Authorization

14. Authentication in Kafka
• Clients need to have and prove their identity
• ~= Login (username / password or token)
Kafka Client Kafka Broker
Authentication data
Verify authentication
Client is authenticated

15. 99 Forms Of Authentication
But Easy Ain’t One
• SSL Authentication: two way client authentication
• SASL (Simple Authentication and Security Layer):
• SASL/GSSAPI (Kerberos) – v0.9.0.0+ - Enterprises (Microsoft AD)
• SASL/PLAIN – v0.10.0.0+ - Passwords hardcoded in broker
• SASL/SCRAM-SHA-256/512 – v0.10.2.0+ - Passwords in Zookeeper (secure it)
• SASL/OAUTHBEARER – v2.0+ - Leverage OAuth 2
• Write your own (contribute back!)
• Extend SASL/PLAIN and SASL/SCRAM with KIP-86 (change credentials store)
• Real world advice:
choose the authentication mechanism you already have in your enterprise

16. Take-aways from the battlefield
• SSL authentication makes it really hard to revoke authentication
• SASL (Simple Authentication and Security Layer) is not simple (YMMV)
• Kerberos is by far the hardest to setup right. Errors are cryptic
• This is the most challenging part of the Kafka security journey

17. Authentication in Kerberos, concretely?
1. Create Kerberos or use Active Directory
2. Ensure Kafka servers have correct CNAME & hostname
3. Generate credentials for the brokers
4. Generate KeyTabs for the brokers from the credentials
5. Create a JAAS file:
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/tmp/kafka.service.keytab"
principal="kafka/<<KAFKA-SERVER-INTERNAL-DNS>>@KAFKA.SECURE";
};

18. Authentication in Kerberos, concretely?
Continued…
• Start Kafka and use java options to reference JAAS file
• Add properties to Kafka:
• Start Kafka
• Pray !
advertised.listeners=SASL_SSL://<<KAFKA-SERVER-DNS>>:9094
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka

19. Real-World Tips: Authentication
• Turn on DEBUG log during setup!
• Go slow
• Ensure CNAME and PTR records are correct
• Scrape and repeat can sometimes solve issues
• Automate

20. Almost there…
Encryption
Authentication
Authorization

21. Authorisation in Kafka
• Kafka knows our client’s identity
• + Authorization rules:
• ”User alice can read topic finance”
• ”User bob cannot write topic trucks”
• = Security
• ACL (Access Control Lists) have to be maintained by administrators

22. ACLs, where are they?
• Default:ACLs are stored in Zookeeper
• Must secure Zookeeper (network rules or authentication)
• OR write your own authorizer (AD, LDAP, a database, Kafka…)

23. Managing ACLs
Producers
• Adding Permissions:
• Shortcuts for producer:
~/kafka/bin/kafka-acls.sh
--authorizer-properties zookeeper.connect=<<ZOOKEEPER-DNS>>:2181
--add --allow-principal User:myproducer --operation Write --topic mytopic
~/kafka/bin/kafka-acls.sh
--authorizer-properties zookeeper.connect=<<ZOOKEEPER-DNS>>:2181
--add --allow-principal User:myproducer --producer --topic mytopic

24. Managing ACLs
Consumers
• Adding Permissions:
• Shortcut for consumers:
~/kafka/bin/kafka-acls.sh
--authorizer-properties zookeeper.connect=<<ZOOKEEPER-DNS>>:2181
--add --allow-principal User:myconsumer --operation Read --topic mytopic
~/kafka/bin/kafka-acls.sh
--authorizer-properties zookeeper.connect=<<ZOOKEEPER-DNS>>:2181
--add --allow-principal User:writer --consumer --topic mytopic -–group mygroup
~/kafka/bin/kafka-acls.sh
--authorizer-properties zookeeper.connect=<<ZOOKEEPER-DNS>>:2181
--add --allow-principal User:myconsumer --operation Write --group mygroup

25. Managing ACLs at Scale?
• Look into Kafka Security Manager
(https://github.com/simplesteph/kafka-security-manager )

26. Real World Tips on ACLs
• Authorisation denials will be logged as INFO in the Kafka log.
• Define your broker as super users
• Careful with: allow.everyone.if.no.acl.found=true
• ACLs can be applied to:
• Topics: Create, Read, Describe,Write, etc…
• Groups: Read,Write, Describe
• Cluster: DescribeConfigs,AlterConfigs, Create
• Wildcards are supported in Kafka 2.0! (useful for Kafka Streams)

27. Cluster Security
Broker
Broker
Broker ZookeeperSASL_SSL
SASL
Clients
SASL_SSL
Kafka Cluster Zookeeper Cluster

28. Kafka Server is Secured ! Done?
Encryption
Authentication
Authorization

29. Security Journey
Continued…
Stephane, secure
Kafka Clients!

30. Broker
Security
Client Security
YOU

31. Kafka Client Security
is the real challenge
• Technical Challenge:
• Java Clients: easy
• Non Java Clients: please use a client that wraps librdkafka
• People Challenge:
• Kafka Administrator: I’m a security guru! But I don’t want to secure all the apps
• Kafka Developer: wt* is security?

32. Client Security in Java
security.protocol=SASL_SSL
sasl.kerberos.service.name=kafka
ssl.truststore.location=/home/kafka/ssl/kafka.client.truststore.jks
ssl.truststore.password=clientpass
sasl.jaas.config='com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/security/keytabs/client.service.keytab"
principal="clientusername";'
• It’s not fun
• It’s not easy
• It’s error prone

33. Kafka Clients are not tailored to your
security needs
• Observation 1: Default Kafka clients have every options for security
• Observation 2: Your enterprise will only have one security setup
• Observation 3: Every client security configuration will look the same
• Take-away: don’t use the default Kafka clients

34. Real World Advice #1
Distribute your own wrapped Kafka Clients
• Developers love nice APIs:
• Standardized applications
• No copy and paste errors
• Centralized debugging
• Reduced learning curve for devs
new MyCorpKafkaProducer(bootstrapServers, keySerializer, valueSerializer)
.withSSL(sslEnabled, pathToTrustStore)
.withAuth(authEnabled, pathToKeyTab, usernameOrPrincipal)
.withSchemaRegistry(url)
.withExtraProperties(properties)
.build()

35. Real World Advice #2
Create a Kafka Client Base Docker Image
Security at scale goes hands in hands with consistency.
• Embed modified Java Truststore
• Standard Retrieval of SSL certificates
• Standard Retrieval of Credentials fromVault / Secure Store
• Kafka environment switches, Security switches
• Bootstrap Server Discovery & Schema Registry Discovery
• Extend to Kafka Connect & Schema Registry

36. Real World Advice #3
Make a checklist before going to prod
• What’s the application username?
• Are all ACLs listed and created?
• Is the application using the MyCorp Kafka clients?
• Is the application running in the standardized Docker Container?
• Are quotas defined for this application?
• Is the application monitored?
• …Check? Release!

37. Next steps
Where to take your learning from here!

38. Okay, I want to implement security!
What’s next?
• Read the docs:
• Kafka Documentation: https://kafka.apache.org/documentation/#security
• Confluent Documentation: https://docs.confluent.io/current/security.html
• Read some blogs:
• https://medium.com/@stephane.maarek/introduction-to-apache-kafka-security-
c8951d410adf
• https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-
encryption/
• Video Course:
• ConfluentYoutube: https://www.youtube.com/watch?v=MsQo-yoVleU&t=21s
• Udemy: https://www.udemy.com/apache-kafka-security (coupon KAFKASUMMIT18)

39. Thank you!
Any questions?