This document provides an overview of securing Apache Pulsar. It discusses securing the different cluster components like Zookeeper, Bookkeeper and brokers. It describes how to enable TLS for securing communication between these components. It also covers setting up TLS, keystores and truststores for brokers and clients. The document references Pulsar and Zookeeper documentation for more details on configuring security.

1. #PulsarSummit Asia 2020
1

2. ● Senior Developer at
Nutanix responsible for all
things pulsar
● Love spending time with
data (stores, streams,
analytics etc)
● Ex-MySQL - started out
with 3 great years building
MySQL Replication
● Contributions to pulsar &
MySQL
Who am I ?
https://www.linkedin.com/in/shivjijha/
https://twitter.com/ShivjiJha
2

3. Catalogue
• Background: Apache Pulsar
• The cluster components
• Background: Security
• The secure coordination
• The secure store
• The secure serving
3

4. Background : Apache Pulsar
4

5. Background: Apache Pulsar
Pulsar: cloud-native, distributed messaging and streaming platform
5

6. Background: Apache Pulsar
Pulsar: cloud-native, distributed messaging and streaming platform
APACHE PULSAR
6

7. Background: Apache Pulsar
Pulsar: cloud-native, distributed messaging and streaming platform
Highlights:
1. Modular design
2. Horizontally scalable
3. Low latency with durability
4. Multi-tenancy
5. Geo Replication
APACHE PULSAR
7

8. The Cluster Components
8

9. The Cluster Components : zookeeper
An open-source server which
enables highly reliable distributed
coordination.
Centralized service for:
1. Configuration information
2. Distributed synchronization
3. Group Services
Use Case: Bookkeeper, broker
9

10. A scalable, fault-tolerant and
low-latency storage service
optimized for realtime workloads.
1. Stand-alone apache project
2. Overlapping committers
Use Case: Broker
The Cluster Components : bookkeeper
10

11. A stateless component that’s
primarily responsible for:
1. Dispatcher:
Async TCP server over custom
binary protocol for all data
transfers.
2. HTTP Server:
REST APIs for admin tasks.
The Cluster Components : broker
11

12. The Cluster
Geo Replication is the replication of
persistently stored data across multiple
clusters.
Messages are instantly replicated across
clusters.12

13. Background: Security
13

14. TLS: Transport Layer Security
1. Encryption : Hide data being
transferred.
2. Authentication : Parties
exchanging info are who they claim
to be.
3. Integrity : Verify data is not
tempered with.
Background : Security - TLS
14

15. 1. Certificate Authority (CA) issues digital certs that contain:
a. public key
b. identity of the owner
2. Keep private key secret. Distribute public key.
3. CA is responsible for saying:
a. yes, clients are who they say they are.
b. And we the CA certify that.
Background : Security - CA
15

16. In general, there are three files:
1. Certifying authority (CA) certificate
2. RSA key pair
a. private key
b. public key
3. X.509 is a standard format for any digital certificate.
Background : Security - Crypto Keys
16

17. 1. Enabling HTTPS on the server (one-way TLS)
2. Require the client to identify itself (two way TLS)
3. Two way TLS based on trusting the Certificate Authority
Background : Security - Crypto Keys
17

18. 1. Several commonly used filename extensions for X.509
certificate files.
2. Password-protected files that sit on the same file system as
our running application
3. We will encounter:
a. jks
b. pkcs12
c. pem
Background : Security - Crypto Keys
Jks : java key store
The default format used for these
files is JKS until Java 8.
18

19. 1. Several commonly used filename extensions for X.509
certificate files.
2. Password-protected files that sit on the same file system as
our running application
3. We will encounter:
a. jks
b. pkcs12
c. pem
Background : Security - Crypto Keys
Since Java 9, the default
keystore format is PKCS12.
JKS is a format specific to Java,
PKCS12 is language-neutral
19

20. 1. Several commonly used filename extensions for X.509
certificate files.
2. Password-protected files that sit on the same file system as
our running application
3. We will encounter:
a. jks
b. pkcs12
c. pem
Background : Security - Crypto Keys
Base64 encoded DER certificate,
enclosed between
"-----BEGIN CERTIFICATE-----"
and
"-----END CERTIFICATE-----"20

21. 1. Several commonly used filename extensions for X.509
certificate files.
2. Password-protected files that sit on the same file system as
our running application
3. We will encounter:
a. jks
b. pkcs12
c. pem
Conversion possible:
pem <==> pkcs12 <==> jks
Background : Security - Crypto Keys
21

22. 1. Can use PEM / jks with broker.
2. Can use jks with bookkeeper.
3. Can use PEM / jks with zookeeper.
Background : Security - Crypto Keys
22

23. 1. Use openssl command to look at certificate data (CA cert or
public key):
openssl x509 -noout -text -in
/path/to/your/ca-certificates/file.pem
Background : Security - Crypto Keys
23

24. Background : Security - Crypto Keys
https://sites.google.com/site/ddmwsst/digital-certificates
24

25. 1. To check if your private key is ok,
openssl rsa -in /path/to/private/keyfile.key -check
RSA key ok
writing RSA key
-----BEGIN PRIVATE KEY-----
…..
-----BEGIN PRIVATE KEY-----
Background : Security - Crypto Keys
25

26. 1. To check if your tls port is serving traffic.
openssl s_client -connect hostname:port
Background : Security - Crypto Keys
26

27. The Secure Coordination
27

28. Secure coordination : Zookeeper (ZK)
1. By Default, network communications of ZK are not
encrypted.
2. We will use the SSL feature of zookeeper.
3. ZK was initially designed over java NIO package.
4. Later Netty package added, to optionally replace NIO.
5. SSL support only added over Netty package usage.
28

29. Secure coordination : Zookeeper (ZK)
1. Enable Netty to use SSL feature.
Set Java system property:
zookeeper.clientCnxnSocket=
"org.apache.zookeeper.ClientCnxnSocketNetty"
zookeeper.serverCnxnFactory=
"org.apache.zookeeper.server.NettyServerCnxnFactory"
29

30. The Secure Store : Zookeeper (ZK)
1. Configure client-server communication to use SSL.
a. server => zookeeper cluster nodes
b. client => bookkeeper / broker server nodes
2. Configure the zk nodes to talk over SSL among
themselves ( Quorum SSL ).
30

31. The Secure Store : Zookeeper (ZK)
Set up server to accept secure connections:
( Add following to zookeeper.conf)
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.keyStore.location="/path/to/your/keystore"
ssl.keyStore.password="keystore_password"
ssl.trustStore.location="/path/to/your/truststore"
ssl.trustStore.password="truststore_password”
ssl.hostnameVerification=true
31

32. The Secure Store : Zookeeper (ZK)
On ZK servers:
Provide a secure port to listen to secure connections:
secureClientPort=2281
Also use port unification to move from non-tls to tls
portUnification = true
Once complete setup is running with tls,
portUnification = false32

33. The Secure Store : Zookeeper (ZK)
Set up client (bookkeeper and broker) to talk over secure connections
In pulsar_env.sh, append these options to extra opts:
export PULSAR_EXTRA_OPTS="
-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-Dzookeeper.ssl.keyStore.location=/path/to/keystore/file.jks
-Dzookeeper.ssl.keyStore.password=testpass
-Dzookeeper.ssl.trustStore.location=/path/to/truststore/file.jks
-Dzookeeper.ssl.trustStore.password=testpass"
-Dzookeeper.client.secure=true
33

34. The Secure Store : Zookeeper (ZK)
Set up server to use SSL cert files to accept secure connections from
peer ZK servers.
In zookeeper.conf, append these configurations:
sslQuorum=true
ssl.quorum.keyStore.location=/path/to/keystore/file.jks
ssl.quorum.keyStore.password=testpass
ssl.quorum.trustStore.location=/path/to/trustore/file,jks
ssl.quorum.trustStore.password=testpass
ssl.quorum.hostnameVerification=true
34

35. The Secure Store
35

36. The Secure Store : Bookkeeper options
# Port that bookie server listen on
bookiePort=3181
The same bookkeeper port is used for tls as well as non-tls
traffic.
36

37. The Secure Store : Bookkeeper options
######################################################################
## TLS settings
######################################################################
# TLS Provider (JDK or OpenSSL).
tlsProvider=OpenSSL
# The path to the class that provides security.
tlsProviderFactoryClass=org.apache.bookkeeper.tls.TLSContextFactory
# Type of security used by server.
tlsClientAuthentication=true
# Bookie Keystore type.
tlsKeyStoreType=JKS
37

38. The Secure Store : Bookkeeper options
# Bookie Keystore location (path).
tlsKeyStore=/path/to/keystore/file.jks
# Bookie Keystore password path, if the keystore is protected by a password.
tlsKeyStorePasswordPath=/path/to/keystore/password/file.jks
# Bookie Truststore type.
tlsTrustStoreType=/path/to/truststore/file.jks
# Bookie Truststore location (path).
tlsTrustStore=/path/to/truststore/password/file.jks
# Bookie Truststore password path, if the trust store is protected by a password.
tlsTrustStorePasswordPath=/path/to/truststore/password/file.jks
38

39. The Secure Serving
39

40. Secure Serving : Broker options
# Broker data port
brokerServicePort=6650
# Broker data port for TLS - By default TLS is disabled
brokerServicePortTls=6651
# Port to use to server HTTP request
webServicePort=8080
# Port to use to server HTTPS request - By default TLS is disabled
webServicePortTls=8443
40

41. The Secure Serving : Broker options
# Path for the TLS certificate file
tlsCertificateFilePath=/etc/pulsar/certs/pulsarcluster1-broker-node-1.bm.infra.crt
# Path for the TLS private key file
tlsKeyFilePath=/path/to/private/keyfile.pem
# Path for the trusted TLS certificate file.
# This cert is used to verify that any certs presented by connecting clients
# are signed by a certificate authority. If this verification
# fails, then the certs are untrusted and the connections are dropped.
tlsTrustCertsFilePath=/path/to/ca-certificates/file.pem
# Accept untrusted TLS certificate from client.
# tlsAllowInsecureConnection=false 41

42. The Secure Serving : Broker options
# Specify the tls protocols the broker will use to negotiate during TLS handshake
# (a comma-separated list of protocol names).
# Examples:- [TLSv1.2, TLSv1.1, TLSv1]
tlsProtocols=
# Specify the tls cipher the broker will use to negotiate during TLS Handshake
# (a comma-separated list of ciphers).
# Examples:- [TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
tlsCiphers=
42

43. The Secure Serving : Broker options
### --- KeyStore TLS config variables --- ###
# Enable TLS with KeyStore type configuration in broker.
tlsEnabledWithKeyStore=false
# TLS Provider for KeyStore type
tlsProvider=
# TLS KeyStore type configuration in broker: JKS, PKCS12
tlsKeyStoreType=JKS
# TLS KeyStore path in broker
tlsKeyStore=
# TLS KeyStore password for broker
tlsKeyStorePassword= 43

44. The Secure Serving : Broker options
### --- KeyStore TLS config variables --- ###
……
# TLS TrustStore type configuration in broker: JKS, PKCS12
tlsTrustStoreType=JKS
# TLS TrustStore path in broker
tlsTrustStore=
# TLS TrustStore password in broker
tlsTrustStorePassword=
44

45. The Secure Serving : Broker options
Authentication options in broker:
# Enable authentication
authenticationEnabled=true
# Autentication provider name list, which is comma separated list of class names
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
# Interval of time for checking for expired authentication credentials
authenticationRefreshCheckSeconds=60
# Enforce authorization
authorizationEnabled=true
…….
45

46. The Secure Serving : Broker options
Authentication options in broker:
……
# Authorization provider fully qualified class-name
authorizationProvider=org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider
# Role names that are treated as "super-user", meaning they will be able to do all admin
# operations and publish/consume from all topics
superUserRoles=admin
46

47. The Secure Serving : Broker options
Peer to peer secure connection options in broker:
# Authentication settings of the broker itself. Used when the broker connects to other
#brokers, either in same or other clusters
brokerClientTlsEnabled=true
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
brokerClientAuthenticationParameters=token:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
brokerClientTrustCertsFilePath=/usr/local/share/ca-certificates/pulsar-gov-pki-ca.pem
# Supported Athenz provider domain names(comma separated) for authentication
athenzDomainNames=
47

48. The Secure Serving : Broker options
Setting up authentication in pulsar client (client.conf)
## Authentication plugin to authenticate with servers
# e.g. for TLS
# authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
authPlugin=
# Parameters passed to authentication plugin.
# A comma separated list of key:value pairs.
# Keys depend on the configured authPlugin.
# e.g. for TLS
# authParams=tlsCertFile:/path/to/client-cert.pem,tlsKeyFile:/path/to/client-key.pem
authParams=
48

49. The Secure Serving : Broker options
Setting up TLS in pulsar client (client.conf)
# Allow TLS connections to servers whose certificate cannot be verified to have
been #signed by a trusted certificate authority.
tlsAllowInsecureConnection=false
# Whether server hostname must match the common name of the certificate the
server #is using.
tlsEnableHostnameVerification=false
tlsTrustCertsFilePath=
# Enable TLS with KeyStore type configuration in broker.
useKeyStoreTls=false
49

50. The Secure Serving : Broker options
Setting up TLS in pulsar client (client.conf)
# TLS KeyStore type configuration: JKS, PKCS12
tlsTrustStoreType=JKS
# TLS TrustStore path
tlsTrustStorePath=
# TLS TrustStore password
tlsTrustStorePassword=
50

51. References
1. Pulsar docs :: https://pulsar.apache.org/docs
2. Digital Certificates : https://sites.google.com/site/ddmwsst/digital-certificates
3. Mutual TLS : https://dzone.com/articles/hakky54mutual-tls-1
4. Broker tls http://pulsar.apache.org/docs/en/security-tls-transport/
5. BookKeeper TLS: https://bookkeeper.apache.org/docs/latest/security/tls/
6. ZooKeeper TLS:
● https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide
● https://zookeeper.apache.org/doc/r3.5.7/zookeeperAdmin.html#sc_authOptions
● https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+and+SASL
51

52. Stay Connected:
● Pulsar Mailing Lists
○ users@pulsar.apache.org
○ dev@pulsar.apache.org
● Pulsar Slack
○ https://apache-pulsar.slack.com
● You can contact me at:
○ https://twitter.com/ShivjiJha
○ https://www.linkedin.com/in/shivjijha/
Q & A
52