The document discusses the perils of secret sprawl and clear text secrets in Apache Kafka configurations. It proposes using a key management system (KMS) to securely store secrets and introducing a configuration provider that allows replacing secrets in configuration files with references to the KMS. The Kafka Improvement Proposals (KIPs) 297 and 421 aim to address this by externalizing secrets to providers and automatically resolving secrets during configuration parsing. Key recommendations include selecting a KMS, moving secrets to it, adding the configuration provider, and replacing secrets with indirection tuples pointing to the KMS.

1. Overcoming the Perils of Kafka Secret
Sprawl
Tejal Adsul | Security Engineer | @TejalAdsul

2. Agenda

3. Agenda
• Security implications of clear text secrets and secret sprawl
• Key Management Systems
• Kafka Config Provider

4. Secrets

5. Authentication
Application/User Secret
External Service

6. Secrets
Digital Certificates
API keys
Username and Passwords

7. How do we use secrets in Apache Kafka

8. Apache Kafka Security 101
Plaintext
Plaintext
Producer
ConsumerPlaintext
B
R
O
K
E
R
B
R
O
K
E
R

9. Apache Kafka Security 101
Producer ConsumerPlaintext
B
R
O
K
E
R
B
R
O
K
E
R
Plaintext
Plaintext

10. Apache Kafka Security 101
Producer
ConsumerB
R
O
K
E
R
B
R
O
K
E
R
Network Attacks

11. Apache Kafka Security 101
Security Goal
Reduce or Minimize Potential Vulnerable Points

12. ENCRYPTION
Apache Kafka Security 101
Producer B
R
O
K
E
R

13. Authentication
Apache Kafka Security 101
Producer
B
R
O
K
E
R
Client Authentication

14. Authentication
Apache Kafka Security 101
Producer
B
R
O
K
E
R
Server Authentication

15. # list of brokers used for bootstrapping knowledge about the rest of the
cluster
# format: host1:port1,host2:port2 ...
bootstrap.servers=localhost:9092
# specify the compression codec for all data generated: none, gzip, snappy,
lz4, zstd
compression.type=none
#SASL Configuration
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required username=”kafka-producer" password=”producer-secret";
Kafka Producer Configuration File
Apache Kafka Security 101
Producer

16. Kafka Broker Configuration File
Apache Kafka Security 101
B
R
O
K
E
R
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
# The address the socket server listens on. It will get the value returned from
listeners=SASL_SSL://:9092
# SASL Configuration
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required username=“admin” password=“admin-secret”
user_kafka-producer=”producer-secret";

17. TLS Configuration
Keystore
Truststore
Keystore
Truststore
Apache Kafka Security 101
Producer
B
R
O
K
E
R

18. Kafka Broker Configuration File
Apache Kafka Security 101
B
R
O
K
E
R
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
# The address the socket server listens on. It will get the value returned from
listeners=SASL_SSL://:9092
# SASL Configuration
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required username=“admin” password=“admin-secret”
user_kafka-producer=”producer-secret";
ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=ZtFCq5EYdcV30GwL
ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks
ssl.truststore.password=y77VVeQYqC8e8ceB

19. Security Implications of Clear Text Secrets

24. Passwords on
Post-its’ Syndrome

25. Clear text secrets in
Configuration File
Syndrome

26. ConsumerB
R
O
K
E
R
B
R
O
K
E
R
Producer

27. ConsumerB
R
O
K
E
R
B
R
O
K
E
R

28. ConsumerB
R
O
K
E
R
B
R
O
K
E
R
Producer

29. Secret Sprawl

30. Producer
Producer
Consumer
Consumer
B
R
O
K
E
R
Zookeeper
Secret Sprawl
B
R
O
K
E
R
B
R
O
K
E
R

31. Producer
Producer
Consumer
Consumer
B
R
O
K
E
R
Zookeeper
Secret Sprawl
B
R
O
K
E
R
B
R
O
K
E
R

32. Producer
Producer
Consumer
Consumer
B
R
O
K
E
R
Zookeeper
Secret Sprawl
B
R
O
K
E
R
B
R
O
K
E
R
Internal
Topic

33. Producer
Producer
Consumer
Consumer
B
R
O
K
E
R
Zookeeper
Secret Sprawl
B
R
O
K
E
R
B
R
O
K
E
R
Internal
Topic
Logs

34. Producer
Producer
Consumer
Consumer
B
R
O
K
E
R
Zookeeper
Secret Sprawl
B
R
O
K
E
R
B
R
O
K
E
R
Internal
Topic
Logs
Ansible
Kubernetes
Splunk

35. Producer
Producer
Consumer
Consumer
B
R
O
K
E
R
Zookeeper
Secret Sprawl
B
R
O
K
E
R
B
R
O
K
E
R
Internal
Topic
Logs
Ansible
Kubernetes
Splunk

36. Producer
Producer
Consumer
Consumer
B
R
O
K
E
R
Zookeeper
Secret Sprawl
B
R
O
K
E
R
B
R
O
K
E
R
Internal
Topic
Logs
Ansible
Kubernetes
Splunk

37. KMS
Secrets
Free
Zone

38. Key Management Systems

39. Encryption
Audit logs
Rotations and Version Control
Access Control

40. KMS

41. KIP-297 KIP-421
KMS

42. KIP-297 Externalizing Secrets for Connect
Configuration

43. KIP-297

44. KIP-297

45. KIP-297

46. KIP-297
B
R
O
K
E
R
Config Transformer

47. KIP-297
Config Transformer

48. KIP-297
Config Transformer
${ provider [:path] : key }

49. KIP-297
Config Transformer
${ provider [:path] : key }

50. KIP-297
Config Transformer
${ provider [:path] : key }
Config Provider
KMS

51. KIP-297
Config Transformer
${ provider [:path] : key }
Config Provider
KMS
secret
key

52. Config Provider
KIP-297
databaseSecret$123
Vault Config Provider
Vault
db_password
/var/private/vault
db_password = databaseSecret$123
db_password = ${ vault : /var/private/vault : db_password }

53. ${ vault : /var/private/vault : db_password }
Config Provider
KIP-297
databaseSecret$123databaseSecret$123
Config Transformer Vault Config Provider Vault
{/var/private/vault:db_password} db_password

54. Config Provider
KIP-297
Configure(params)
Get(Path, keys)
Subscribe(ConfigChangeCallback)
key
secret
KMS

55. Configure a Configuration Provider
config.providers = vault, file
// Vault Config Provider
config.providers.vault.class = org.vault.configs.GenericVaultConfigProvider
config.providers.vault.param.token = /run/secrets/vault-token
config.providers.vault.param.uri = 168.229.49.188
// File Config Provider
config.providers.file.class = org.apache.configs.FileConfigProvider
KIP-297
B
R
O
K
E
R

56. KIP-421 Automatically resolve external
configurations.

57. KIP-421
Configuration
Config Def
Abstract Config
configuration
parsed & validated configuration
Name : SSL_KEYSTORE_PASSWORD
Type: PASSWORD
Importance: High
Default_Value: null

58. KIP-421
broker.id = 0
security.inter.broker.protocol = SSL
ssl.keystore.location = /var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password = YB3PgSf7M2t3mUo9
B
R
O
K
E
R

59. KIP-421
broker.id = 0
security.inter.broker.protocol = SSL
ssl.keystore.location = /var/private/ssl/kafka.server.keystore.jks
ssl. keystore.password = ${vault:ssl.keystore.password}
B
R
O
K
E
R

60. KIP-421
B
R
O
K
E
R
broker.id = 0
security.inter.broker.protocol = SSL
ssl.keystore.password = ${vault:ssl. keystore.password}
config.providers = vault
config.providers.vault.class = com.org.apache.configs.VaultConfigProvider
config.providers.vault.param.token = /run/secrets/vault-token
config.providers.vault.param.uri = 168.229.49.188

61. KIP-421
Abstract Config
broker.id = 0
security.inter.broker.protocol = SSL
ssl.key.password = ${vault:ssl.keystore.password}
config.providers = vault
config.providers.vault.class = com.org.apache.configs.VaultConfigProvider
config.providers.vault.param.token = /run/secrets/vault-token
config.providers.vault.param.uri = 168.229.49.188

62. KIP-421
Abstract Config
Vault Config Provider
vault.param.token = /run/secrets/vault-token
vault.param.uri = 168.229.49.188

63. KIP-421
Abstract Config
broker.id = 0
security.inter.broker.protocol = SSL
ssl.keystore.password = ${vault:ssl.keystore.password}
config.providers = vault
config.providers.vault.class = com.org.apache.configs.VaultConfigProvider
config.providers.vault.param.token = /run/secrets/vault-token
config.providers.vault.param.uri = 168.229.49.188
${}

64. KIP-421
Abstract Config
ssl.keystore.password = ${vault:ssl.keystore.password}
Config Transformer

65. KIP-421
Abstract Config
ssl.keystore.password = YB3PgSf7M2t3mUo9
Config Transformer

66. KIP-421
Abstract Config
ssl.keystore.password = YB3PgSf7M2t3mUo9
broker.id = 0
security.inter.broker.protocol = SSL
Config Def

67. KIP-421
Abstract Config
ssl.keystore.password = YB3PgSf7M2t3mUo9
broker.id = 0
security.inter.broker.protocol = SSL
Config Def

68. Securing Passwords in Log File
Config DefConfiguration
TYPE

69. Recap
1. Select Key Management System
2. Move Secrets to KMS
3. Add the Configuration provider to Kafka
4. Replace the secrets in configuration file
with indirection tuple

70. ssl.keystore.password = YB3PgSf7M2t3mUo9
ssl.keystore.password = YB3PgSf7M2t3mUo9
Config Transformer
Vault Config
Provider
Vault
ssl.keystore.password ssl.keystore.password
YB3PgSf7M2t3mUo9YB3PgSf7M2t3mUo9
ssl.keystore.password=${vault:ssl.keystore.password}
ssl.keystore.password=YB3PgSf7M2t3mUo9
1
2 3
45
6
7
8
Abstract Config
Config Def

71. B
R
O
K
E
R
Consumer
Producer
Connect
Stream

72. Demo

73. Tejal Adsul

74. Reference
• Apache Kafka Security 101
• Secret Sprawl
• KIP-297: Externalizing Secrets for Connect Configurations
• KIP-421: Automatically resolve external configurations
• Confluent Secret Protection