In order to leverage the best performance characters of your stream backend, it is important to understand the nitty gritty details of how pulsar stores your data. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand. With this underlying philosophy, in this talk, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code! This session will empower you with the right background to map your data right with pulsar.

1. content page

2. ● SME for pulsar at Nutanix
● Love
○ Distributed systems
○ Open source
● Hands on developer, aspiring architect
● Love spending time with data (stores,
steam, analytics etc)
● Contributions to pulsar & MySQL
Who am I ?
https://www.linkedin.com/in/shivjijha/
https://twitter.com/ShivjiJha

3. Catalogue
• Pulsar modules
• Data stores
• Distributed storage
• Storage internals (Read / Write)
• Tying together (Metadata layer)

4. Pulsar Modules

5. Pulsar Modules
SERVING LAYER
METADATA LAYER
STORAGE LAYER
CLIENT

6. Pulsar Modules
SERVING LAYER (BROKER)
METADATA LAYER (ZOOKEEPER)
STORAGE LAYER (BOOKKEEPER)
CLIENT

7. Pulsar Data Stores

8. Pulsar Data Stores
SERVING LAYER (BROKER)
METADATA LAYER (ZOOKEEPER)
STORAGE LAYER (BOOKKEEPER)
CLIENT

9. Pulsar Data Stores
SERVING LAYER (BROKER)
METADATA LAYER (ZOOKEEPER)
BOOKKEEPER
CLIENT
OBJECT STORE

10. Pulsar Data Stores
SERVING LAYER
(BROKER)
METADATA LAYER (ZOOKEEPER)
BOOKKEEPER
CLIENT
OBJECT STORE
CACHE

11. Pulsar Data Stores : Overview
• Broker Cache
• Single broker owner for topic
• Primary Store
• Bookkeeper
• Cold Store
• Object Store
• Metadata Layer
• Zookeeper

12. Pulsar @ scale

13. scale compute
Pulsar @ scale

14. scale compute
scale storage
Pulsar @ scale

15. Pulsar Data Stores : Bookkeeper
LEDGER
Ledger Metadata
Status : Open
Last Entry Id : -1
Ensemble Size
Write Quorum Size
Read Quorum Size
Ensembles : [ [], [] ]

16. Pulsar Data Stores : Bookkeeper
LEDGER
entry 0
Ledger Metadata
Status : Open
Last Entry Id : 0
Ensemble Size
Write Quorum Size
Read Quorum Size
Ensembles : [ [], [] ]

17. Pulsar Data Stores : Bookkeeper
LEDGER
entry 0
entry 1
entry 2
Ledger Metadata
Status : Closed
Last Entry Id : 2
Ensemble Size
Write Quorum Size
Read Quorum Size
Ensembles : [ [], [] ]

18. Pulsar Data Stores : Bookkeeper
LEDGER
entry 0
entry 1
entry 2
Entry Data
Metadata:
1.LedgerId
2.EntryId
3.Last Add Confirmed
4.Digest (CRC32 / CRC32C)
Data : byte []

19. Pulsar Data Stores : Bookkeeper

20. Pulsar Data Stores : Write Path
1.Client sends write on topic
2.The owner broker (bookkeeper client)
writes to current ensemble of bookies
3.Client waits for #ackQuorum acks
4.Acknowledge to client.

21. Pulsar Data Stores : Write Path
1.Client sends write on topic
2.The owner broker (bookkeeper client)
writes to current ensemble of bookies
3.Client waits for #ackQuorum acks
4.Acknowledge to client.
New ledger registered in zookeeper
against topicName.

22. Pulsar Data Stores : Read Path
1.Client sends read on topic
2.The owner broker searches local
cache.
3.If cache miss, read from closest
bookie.
a. On failure try other bookie

23. Pulsar Data Stores : Bookkeeper
• Internally, journal for short lived write queue
• Facilitates quick writes (appends)
• Write cache for organized and flush writes
• Read cache for quick reads
• if data not in broker cache
• if data not in write cache
• Ledgers for organized data interleaving topics.
• RocksDB to index data ledger by ledger
• Topic to ledger(s) mapping in zookeeper.

24. Pulsar Data Stores : Bookkeeper
● Consistency over availability
● Durability
○ Ensemble
○ Write Quorum
○ Ack Quorum
● Configure how much / how long to store in bookkeeper
● Configure replication factor
● Choose durability vs latency / throughput

25. Pulsar Data Stores : Cold Store
• Jcloud library
• aws-s3
• google-cloud-storage
• filesystem
• Broker writes to cold store, not bookkeeper.
• bookkeeper => broker => object store
• too much bandwidth?
• Why not offload directly from bookie to object store?
• Schema not stored with data in object store.
• consumers like flink, presto etc can work with just bookkeeper

26. Distributed Storage

27. Pulsar Data Stores : Distribution
tenant
namespace
bundle1 bundle2 bundle3
T
O
P
I
C
1
T
O
P
I
C
2
T
O
P
I
C
3
T
O
P
I
C
4
T
O
P
I
C
5
T
O
P
I
C
6
• Shards (aka namespace Bundles) for load balancing

28. Pulsar Data Stores : Distribution
tenant
namespace
bundles
• Shards (aka namespace Bundles) for load balancing
topic
ledgers

29. Pulsar Data Stores : Distribution
tenant
namespace
bundles
• Shards (aka namespace Bundles) for load balancing
topic
ledgers
ledgers[], schemaLedgers[],
compactedLedgers[]

30. Pulsar Data Stores : Distribution
tenant
namespace
bundles
• Shards (aka namespace Bundles) for load balancing
topic
ledgers
ledgerId, entries range,
ledger size, offloaded?
ledgers[], schemaLedgers[],
compactedLedgers[]

31. Bookkeeper Internals

32. Bookkeeper: Client & Server
• Bookkeeper has no leader / follower.
• All bookies have same responsibilities.
• Thick bookie client implements replication, consistency etc
• Simple bookie APIs
• Resources
• Ledger
• Entry

33. Bookkeeper: Client & Server
ENTRY <L1 EO>
ENTRY <L1 E1>
ENTRY <L1 E2>
ENTRY<L2 EO>
ENTRY<L2 E1>
ENTRY <L1 E3>
ENTRY <L2
E2>
ENTRY <L3 EO>
BOOKKEEPER
CLIENT
BROKER
JOURNAL 0
DISK
BOOKKEEPER

34. Bookkeeper: Client & Server
ENTRY <L1 EO>
ENTRY <L1 E1>
ENTRY <L1 E2>
ENTRY<L2 EO>
ENTRY<L2 E1>
ENTRY <L1 E3>
ENTRY <L2
E2>
ENTRY <L3 EO>
BOOKKEEPER
CLIENT
BROKER
JOURNAL 0
DISK
BOOKKEEPER
• First step in pulsar write path
• Distributed WAL
• like databases
(MySQL, postgres etc)
• sequential writes
• no reads from journal
• write cache
• write isolation with
separate disks for journal
and ledger

35. Bookkeeper: Client & Server
ENTRY <L1 EO>
ENTRY <L1 E1>
ENTRY <L1 E2>
ENTRY<L2 EO>
ENTRY<L2 E1>
ENTRY <L1 E3>
ENTRY <L2
E2>
ENTRY <L3 EO>
BOOKKEEPER
CLIENT
BROKER
JOURNAL 0
WRITE
CACHE
DISK
MEMORY
BOOKKEEPER

36. Bookkeeper: Read Cache
write cache
read
cache
entry log
L1 index
L2 index
flush

37. Bookkeeper: Read Cache
write cache
read
cache
entry log
L1 index
L2 index
flush
entry =>
message batch

38. Pulsar Data Stores : Read Write Internals

39. Bookkeeper: Ledgers
• Sequential reads
• Still interleaved across topics
• indexed
• rocksDB
• (ledger, entry id => log file, offset)
• Read path:
• Broker çache
• no n/w trip,
• no disk access
• Bookkeeper
• Write Cache
• Read Cache
• RocksDB index
• Access from disk

40. Bookkeeper: Ledgers
• Sequential reads
• Still interleaved across topics
• indexed
• rocksDB
• (ledger, entry id => log file, offset)
• Read path:
• Broker çache
• no n/w trip,
• no disk access
• Bookkeeper
• Write Cache
• Read Cache
• RocksDB index
• Access from disk

41. Bookkeeper: LAC & LAP
• LAC : Last add confirmed
• In response to write().
• This entry and all previous written (cumulative ack).
• As a result of the sequential write.
• LAP : Last add pushed
• Readers can read until LAC

42. Bookkeeper: Fencing
• Recovery after
• bookie failure
• network partition b/w broker and bookkeeper
• New bookie
• Puts ledger state in recovery,
• Fences the ledger with consensus
• Writes to new ledger.
• Old owner can’t write to ledger anymore.
• Consistency
• No split brain

43. Metadata Layer

44. Metadata Layer
1.Pointers to data
a.Topic ledgers mapping
b.Ledger topics mapping
c.Topic schema mapping
2.Service Discovery
a.List of available bookies (read / write / both?)
b.List of available brokers
c.Which broker owns which topic
d.How much load on which topic etc
3.Distributed coordination
a.Locks
b.Leader election

45. Metadata Layer
4. System Configuration
a.Dynamic configurations for hot reload
b.feature flags
5. Provisioning Configuration
a.Metadata for tenants, namespaces etc
b.Namespace policies

46. I want to know more..

47. References:
• https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-
works
• Pulsar without Zookeeper: Introducing the Metadata Access Layer in Pulsar
• TGI Pulsar 009: Introduction of Apache BookKeeper

48. Q & A time
Drop me a hello at:
https://www.linkedin.com/in/shivjijha/
https://twitter.com/ShivjiJha
Pulsar community
https://apache-pulsar.slack.com/
users@pulsar.apache.org
dev@pulsar.apache.org