Learn how to secure a Pulsar cluster with Hashicorp Vault and deploy it on Kubernetes. Vault provides a secure way to generate tokens and store sensitive data and Pulsar has a pluggable architecture for authentication, authorization and secret management. This talk will walk through how to create custom plugins for Vault, integrate them with Pulsar and then deploy a Pulsar cluster on Kubernetes.

1. @cckellogg #PulsarSummit
Securing Your Pulsar Cluster
with Vault
&

2. Chris Kellogg
Software Engineer at Splunk
Contributor to Apache Pulsar and Apache Heron committer
cckellogg
You can find me on:
cckellogg

3. Agenda
• Vault Overview
• Why Pulsar and Vault
• Pulsar Authentication/Authorization Model
• Creating Custom Plugins
• Packaging Custom Plugins
• Kubernetes Integration
• Demo

4. https://www.vaultproject.io
“Vault is a tool for securely accessing secrets. A secret is anything
that you want to tightly control access to, such as API keys,
passwords, or certificates. Vault provides a unified interface to any
secret, while providing tight access control and recording a detailed
audit log.”
What is Vault?

5. Vault Features
Secret Management
Authentication and Identity
Data Encryption

6. Why Vault
• Single source to manage secrets and tokens
• Dynamic and Revokable tokens and secrets
• Audit tracking for secrets and token
• Merges identities across providers
- LDAP, Okta, Kubernetes, AWS, GCP
• Cloud friendly

7. Why Pulsar and Vault
• No more forever tokens
• Revokable tokens
• Secure secret management for functions and connectors
• Supports authenticating against many trusted sources of identity
- LDAP, Okta, Kubernetes, AWS, GCP, GitHub
• Central location for all security

8. Pulsar Security

9. Default is No Security
• Produce and consume from any topic
• Modify any tenant, namespace, topic or function
• Function/Connector secrets stored as plain text in configs
• No auditing of actions

10. Pulsar Security Features
• TLS Encryption for traffic
• Authentication - validate identity
• Authorization - can user perform an action
• Data encryption between producers and consumers

11. Pulsar Authentication
• Responsible for determining identity of clients
• Plugin System
• Built-in Plugins
- TLS
- JWT
- Authenz
- Kerberos

12. Pulsar Authorization
• Determines if a client has permission to perform an action
• Plugin System
• Built-in Plugin - Role based system backed by Zookeeper
- SuperUsers
- Tenant Admins
- Actions: produce/consume/functions

13. Developing Auth Plugin

14. Building Plugins Best Practices
• Minimize third party dependencies
• Use your own executor and threads for remote requests
• Cache responses

15. public class VaultAuthenticationProvider implements AuthenticationProvider {
void initialize(ServiceConfiguration config) throws IOException {};
String getAuthMethodName() { return "token" };
boolean authenticateHttpRequest(HttpServletRequest req, HttpServletResponse resp)
throws Exception {
throw new AuthenticationException("Not supported");
}
String authenticate(AuthenticationDataSource authData) throws AuthenticationException {
// Implement code to authenticate the client with vault
}
AuthenticationState newAuthState(AuthData authData,
SocketAddress remoteAddress, SSLSession sslSession) throws AuthenticationException {
// Implement code to authenticate the client with vault -
// Used in binary connections for challenges
}
}
Vault Authentication Plugin

16. ### --- Authentication --- ###
# Enable authentication
authenticationEnabled=true
# Autentication provider name list, which is comma separated list of class names
authenticationProviders=org.apache.pulsar.vault.authentication.VaultAuthentictionProvider
# Interval of time for checking for expired authentication credentials
authenticationRefreshCheckSeconds=60
Configuring Auth Plugin
broker.conf

17. Client
BROKER
Vault Authentication Provider
Authentication
Service
1
5 2
4
6
3
1. Client Request with Vault Token
2. Authenticate Client
3. Token pass to Vault for Authentication
4. Vault token info returned
5. Return user identity
6. Return result to client
Pulsar Vault Authentication

18. Developing Function Plugins

19. Pulsar Secret Plugins
Secrets Provider
• Run in the instance
• Provides secrets through the function context api
Secrets Configurator
• Runs on the server (Broker or Function Worker)
• Determines the Secret Provider the instance should use

20. public interface SecretsProvider {
// Initialize the SecretsProvider.
default void init(Map<String, String> config) {}
// Fetches a secret
String provideSecret(String secretName, Object pathToSecret);
}
public class MySecretFunction implements Function<String, Void> {
@Override
public Void process(String input, Context context) throws Exception {
final String password = context.getSecret("password");
context.getLogger().info("read secret password=" + password);
return null;
}
}
Example code
SecretsProvider - Client Side Plugin

21. public interface SecretsProviderConfigurator {
default void init(Map<String, String> config) {}
void configureKubernetesRuntimeSecretsProvider(V1PodSpec ps, String container,
Function.FunctionDetails details;
void configureProcessRuntimeSecretsProvider(ProcessBuilder pb,
Function.FunctionDetails detailsetails);
Type getSecretObjectType();
default void doAdmissionChecks(AppsV1Api appsV1Api, CoreV1Api coreV1Api,
String ns, Function.FunctionDetails details) {}
String getSecretsProviderClassName(Function.FunctionDetails details);
Map<String, String> getSecretsProviderConfig(Function.FunctionDetails details);
}
SecretsProviderConfigurator - Server Side Plugin
Highlighted methods are used to setup secrets plugins on the instances

22. ########################
# Secrets
########################
secretsProviderConfiguratorClassName: org.apache.pulsar.vault.secrets.VaultSecretsProviderConfigurator
secretsProviderConfiguratorConfig:
vaultAddress: http://localhost:8200
tokenPath: /etc/auth/token
Configuring Secret Plugins
Secrets Configurator
functions_worker.yml

23. Secrets Provider
public class VaultSecretsProviderConfigurator implements SecretsProviderConfigurator {
@Override
public String getSecretsProviderClassName(Function.FunctionDetails details) {
if (!isEmpty(functionDetails.getSecretsMap())) {
if (Function.FunctionDetails.Runtime.JAVA == details.getRuntime()) {
return "org.apache.pulsar.vault.secrets.VaultSecretsProvider";
} else if (Function.FunctionDetails.Runtime.PYTHON == details.getRuntime()) {
return "python_secret_provider";
}
}
return null;
}
@Override
public Map<String, String> getSecretsProviderConfig(Function.FunctionDetails details) {
final Map<String, String> secrets = new HashMap<>();
secrets.put("vaultAddress", "http://localhost:8200");
secrets.put("tokenPath", "/var/auth/token");
return secrets;
}
Configuring Secret Plugins

24. Java Function Instance
User Code
final String password =
context.getSecret("password");
Vault Secret Provider
1
2
3
4
1. Request secret from code
2. Secret request with token
3. Secret returned to plugin
4. Return secret value
Vault Secret Provider

25. Pulsar Kubernetes Plugins
Kubernetes Manifest Customizer
• Runs on the server (Broker or Function Worker)
• Enables customization to the K8s function specs
Kubernetes Function Auth Provider
• Runs on the server (Broker or Function Worker)
• Determines the auth params passed to the instances

26. public interface KubernetesManifestCustomizer extends RuntimeCustomizer {
default V1StatefulSet customizeStatefulSet(Function.FunctionDetails funcDetails,
V1StatefulSet statefulSet) {
return statefulSet;
}
default V1Service customizeService(Function.FunctionDetails funcDetails,
V1Service service) {
return service;
}
default String customizeNamespace(Function.FunctionDetails funcDetails,
String currentNamespace) {
return currentNamespace;
}
}
KubernetesManifestCustomizer - Server Side Plugin

27. public interface KubernetesFunctionAuthProvider extends FunctionAuthProvider {
public void configureAuthDataStatefulSet(V1StatefulSet sts, Optional<FunctionAuthData> o) {}
public void configureAuthenticationConfig(AuthenticationConfig config,
Optional<FunctionAuthData> o) {
** configures the client auth for the function instances
}
public Optional<FunctionAuthData> cacheAuthData(Function.FunctionDetails details,
AuthenticationDataSource s) throws Exception {
** Optional<FunctionAuthData> returned is used in configureAuthenticationConfig
}
public Optional<FunctionAuthData> updateAuthData(Function.FunctionDetails details,
Optional<FunctionAuthData> o, AuthenticationDataSource s) throws Exception {
** Optional<FunctionAuthData> returned is used in configureAuthenticationConfig
}
public void cleanUpAuthData(Function.FunctionDetails details, Optional<FunctionAuthData> o)
throws Exception {}
}
KubernetesFunctionAuthProvider - Server Side Plugin

28. Java Function Instance
User Code
final String password =
context.getSecret("password");
Vault Secret Provider
1
2
3
1. Request secret from code
2. Read secret from file
3. Return secret value
Vault Secret Provider with Vault Agent

29. Packaging Plugins
Where do my plugins go?
pulsar/
instances/
lib/
authentication.jar
secret-configurator.jar
secret-provider.jar
deps/
kubernetes-plugins.jar

30. Kubernetes Pulsar Vault

31. pulsar functions
vault
zookeeper brokers bookies
proxy

32. Pulsar Kubernetes Pod
Pulsar Process
Vault Agent
Kubernetes JWT
4
3
2
1
1. Service Account JWT passed to Vault
for Authentication
2. Vault Token auth returned
3. Write token to file
4. Pulsar process reads token from file
Pulsar Vault Kubernetes Integration

33. Function Secret Configuration
tenant: "public"
namespace: "default"
name: “secrets-printer"
className: “secrets_printer”
inputs: ["public/default/secrets-trigger"]
autoAck: true
parallelism: 1
resources:
cpu: 0.5
ram: 536870912
disk: 536870912
secrets:
username:
path: "internal/data/database/config"
key: username
password:
path: "internal/data/database/config"
key: password
customRuntimeOptions: >-
{
"serviceAccountName": "pf-secrets-printer"
}
Used by the VaultKubernetesCustomizer to
add annotations for vault token and secret
injection

34. Function Vault Annotations
vault.hashicorp.com/role: pf-secrets-printer
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/agent-inject-token: 'true'
vault.hashicorp.com/agent-inject-secret-password: secret-path
vault.hashicorp.com/agent-inject-template-password: |
'{{- with secret "secret-path"
}}{{ .Data.data.password }}{{ end }}'

35. Demo

36. Future Enhancements
• Vault for certificate management
• Pulsar Vault authorization plugin
• Vault for data encryption

37. Resources
• https://pulsar.apache.org/docs/en/security-overview/
• https://pulsar.apache.org/docs/en/security-authorization/
• https://pulsar.apache.org/docs/en/security-extending/
• https://github.com/hashicorp/vault-k8s
• https://www.vaultproject.io/docs/platform/k8s/helm
• https://www.vaultproject.io/docs/platform/k8s/injector
• https://learn.hashicorp.com/vault/kubernetes/k8s-reference-architecture
Pulsar
Vault

38. Questions

39. Thank You
cckellogg
cckellogg
#PulsarSummit
Code: https://github.com/cckellogg/pulsar-vault