1.
1

2.
Who are We?
● Leading Cloud data Platform Team
● Loves distributed Systems, open Source
● Data geek (data stores, stream, analytics etc)
● Pulsar & MySql contributor
● Developer by passion
● Love Distributed Systems, Streaming Platform
● Pulsar Enthusiast
Shivji Jha
SME for Pulsar,
Nutanix
Sourabh Agrawal
Pulsar Ninja,
Nutanix
2

3.
Catalogue
● Quick Intro: Apache Pulsar
● Pulsar Isolation
○ What
○ Why
○ How
● Isolating by clusters
● Isolating brokers
● Isolating bookies
● Demo
3

4.
About Pulsar
Apache Pulsar is a cloud-native, distributed, open-source pub-sub messaging and streaming
platform. Originally developed by Yahoo and contributed to the Apache Software Foundation in
2016
Provides :
● Stateless Brokers
● Horizontally Scalable
● Isolate Read and Writes
● Multi-tenant System
● Geo-Replication
● Active Community Support
4

5.
Multi-tenancy
5

6.
What is Pulsar Isolation ?
● Scale storage and Serving layer independently.
● Can separate resources (brokers & bookies) for specific use cases.
● Scale up/down resources as per need for specific namespaces by keeping rest of the
cluster untouched.
● Localized resource allocation
6

7.
Why Pulsar Isolation ?
● Avoid Sharing resources within teams
○ Prevent unexpected consequences.
● Dedicated resources for a namespace.
● Allocate Localized resource in multi-region setup.
● Scale up/down nodes as per load on a namespace.
● More Secure (Permissions managed at namespace level)
● Easy setup
7

8.
Isolation Categories
● Hard Isolation
○ Different physical Pulsar clusters (or nodes) for your isolation units
■ Failure Domains
■ Anti affinity Groups
■ Broker Isolation using Namespace Isolation Policy
■ Bookie Isolation using Affinity Groups
● Soft Isolation
○ Same Physical clusters but restrictions on tenants / namespaces by configuration
■ Disk Quotas
■ Throttling
8

9.
Isolating by Configuration
9

10.
Soft isolation
● Limit by configurations
○ Namespace:
● encryptionRequired
● Auth_policies
● Backlog_quota
● retention
■ Topic:
● maxProducersPerTopic
● maxConsumersPerTopic
● maxSubscriptionsPerTopic
● Limits by resources (capped per broker):
■ loadBalancerBrokerMaxTopics
■ loadBalancerBrokerOverloadedThresholdPercentage
10

11.
Isolating by Cluster
11

12.
Geo Replication
eu-west-2 eu-west-3
Geo
Replication
On
12
New Hotel company starts operating in London
Expand business to Paris
Enable HA

13.
Isolating Brokers
13

14.
14
https://github.com/apache/pulsar/wiki/PIP-7:-Pulsar-Failure-domain-and-Anti-affinity-namespaces
● Failure Domain
○ Group brokers into failure domains
○ Reduces blast radius
○ Example
■ Fix a bug / patch
■ Upgrade brokers by failure domain
■ Anti affinity of namespaces
Failure Domains

15.
15
https://github.com/apache/pulsar/wiki/PIP-7:-Pulsar-Failure-domain-and-Anti-affinity-namespaces
● Anti affinity Group
○ Group of namespaces that should stay far.
○ Best effort
■ Prefer different failure domain
■ Prefer different broker
Anti affinity Groups

16.
16
What is “Namespace Isolation Policy”?

17.
Namespace Isolation Policy: Internals
min_availability=1
Primary = Broker 1,2
17

18.
Namespace Isolation Policy: Internals
min_availability=1
Primary = Broker 1,2
18

19.
Namespace Isolation Policy: Internals
min_availability=1
Primary = Broker 1,2
19

20.
Namespace Isolation Policy: Internals
min_availability=1
Secondary = Broker 5
Primary = Broker 1,2
20

21.
Namespace Isolation Policy: Setup
Configure Isolation Policy
Unload the namespace to take effect for existing topics in namespaces or restart brokers.
./pulsar-admin ns-isolation-policy set --auto-failover-policy-type min_available
--auto-failover-policy-params min_limit=?,usage_threshold=80 --namespaces <tenant/namespace>
--primary broker-node-1,broker-node2 my-cluster <policy-name> --secondary broker-node-5
./pulsar-admin namespaces unload <tenant/namespace>
./pulsar-admin topics lookup <tenant/namespace/topic>
21

22.
Isolating Bookies
22

23.
Bookie Affinity Groups
23
● Isolate storage at namespace
● Leverage data durability configs in
isolated bookies
● Leverage
ZkIsolatedBookieEnsemblePlacement
Policy
● Scale bookies for namespace
https://streamnative.io/uploads/images/blogs/isolation-3.png

24.
Bookie Isolation using
Affinity Group
Setup Bookie Rack
Bookie Affinity Group
./pulsar-admin bookies set-bookie-rack --bookie 127.0.0.1:3181 --hostname 127.0.0.1:3181
--group group1 --rack rack1
./pulsar-admin namespaces set-bookie-affinity-group <tenant/namespace> --primary-group
<anti-affinity-group-name> --secondary-group <anti-affinity-group-name>
./pulsar-admin namespaces set-bookie-affinity-group public/default --primary-group group1
24
Configure Bookie Group
Configure Bookie Group with secondary

25.
Scaling Up/Down
Broker
● When scaling up brokers, change the isolation groups and add newly added broker as
primary or secondary broker for required namespaces.
● When scaling down, make sure existing isolation groups have enough brokers.
Bookie
● When scaling up bookies, change the bookie affinity group and add newly added bookies as
primary or secondary group for required namespaces.
● When scaling down, make sure existing affinity groups have enough bookies.
25

26.
Demo : Namespace Isolation Policy
26

27.
References
Namespace Isolation: https://pulsar.apache.org/docs/en/administration-isolation/#broker-isolation
RackAware Ensemble Policy:
https://bookkeeper.apache.org/docs/latest/api/javadoc/org/apache/bookkeeper/client/EnsemblePl
acementPolicy.html
Bookie Affinity Group:
https://pulsar.apache.org/docs/en/administration-isolation/#bookie-isolation,
https://github.com/apache/pulsar/pull/3933
Crash course on pulsar :
https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works
Soft Isolation:
https://streamnative.io/en/blog/tech/2021-03-02-taking-an-in-depth-look-at-how-to-achieve-isolat
ion-in-pulsar/
27

28.
Questions?
Pulsar Slack
https://apache-pulsar.slack.com/
Pulsar Mail Group:
users@pulsar.apache.org
dev@pulsar.apache.org
LinkedIn :
https://www.linkedin.com/in/sourabhaggrawal/
https://www.linkedin.com/in/shivjijha/
28