Download to read offline
Uploaded byStreamNative
TGIPulsar - EP #006: Lifecycle of a Pulsar message
1. Pulsar uses bookies to persist messages and brokers to serve clients and select bookies. ZooKeeper stores metadata. 2. When a message is produced, it is sent to a broker and written to multiple bookies. Consumers connect to brokers and receive messages from caches or by brokers reading from bookies. 3. Pulsar retains messages based on retention policies like time and size. Messages are deleted by segment once all subscriptions are caught up to avoid deleting messages still needed.
More Related Content
![[Demo session] 관리형 Kafka 서비스 - Oracle Event Hub Service](https://cdn.slidesharecdn.com/ss_thumbnails/oraclouddevmeetupeventhubpublic-180521084803-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to TGIPulsar - EP #006: Lifecycle of a Pulsar message
TGIPulsar - EP #006: Lifecycle of a Pulsar message
- 2. streamnative.io Lifecycle of a PulsarMessage #TGIPulsar EP-006
- 3. Message Lifecycle ✓ MessageFlow ✓ Message Retention
- 4.
- 5. Brokers + Bookies Bookie0 Bookie 1 Bookie 2 The processes for storing data are called bookies. They persist data for Pulsar. Broker 0 Broker 1 Broker 2 Brokers are “stateless”. They serve clients for producing and consuming events
- 6. ZooKeeper Bookie 0 Bookie1 Bookie 2 The processes for storing data are called bookies. They persist data for Pulsar. Broker 0 Broker 1 Broker 2 Brokers are “stateless”. They serve clients for producing and consuming events ZooKeeper ZooKeeper ZooKeeper ZooKeeper is used for storing the metadata for Pulsar and bookkeeper as well as for discovering brokers and bookies.
- 7. Pulsar Producer 0Producer 1 Topic Partition 0 Partition 1 Partition 2 Broker X Broker Y Broker Z Subscription A Consumer (P012)
- 8. Produce Producer 0Producer 1 Topic Partition 0 Partition 1 Partition 2 Broker 0 Broker 1 Broker 2 Bookie 0 Bookie 1 Bookie 2 1. A message is created and a partition is selected 2. The message is sent to the owner broker that serves the selected partition 3. The message is written to N bookies in parallel by the owner broker. The message is written once and stored in their entirety. 4. Once the message has been written by 2 bookies, the broker will acknowledge the message
- 9. Consume (Cached) Topic Partition 0 Partition1 Partition 2 Broker 0 Broker 1 Broker 2 Bookie 0 Bookie 1 Bookie 2 Consumer (P012) 1. The consumer subscribes to a topic. It connects to the owner brokers serving the partitions. 2. Broker sends messages for the partition coming out of its memory cache 3. Consumer acknowledges a message after processing it. Broker updates cursor once it receives acknowledgment.
- 10. Consume (BK) Topic Partition 0 Partition1 Partition 2 Broker 0 Broker 1 Broker 2 Bookie 0 Bookie 1 Bookie 2 Consumer (P012) 1. The consumer subscribes to a topic. It connects to the owner brokers serving the partitions. 2. Broker does not have the data in the memory and will read from one of the Bookies that have the data. 3. Consumer acknowledges a message after processing it. Broker updates cursor once it receives acknowledgment.
- 11. Failures Producer 0Producer 1 Topic Partition 0 Partition 1 Partition 2 Broker 0 Broker 1 Broker 2 Bookie 0 Bookie 1 Bookie 2 In flights messages will be automatically retried by Pulsar clients Brokers are stateless. Any broker process that dies that doesn’t impact data storage. Consumer (P012) When a bookie dies, all the data is still accessible and will be replicated by other replicas
- 12.
- 13. Message retention ✓ Retention ✓TTL ✓ Message backlog ✓ Storage size
- 14. Subscription & Cursor Partition(Event Stream) Subscription A (1, 1) Subscription B (2, 2) Subscription C (3, 2)
- 15. Subscription Initial Position Partition(Event Stream)
- 16.
- 17.
- 18.
- 19.
- 20. Message retention (1) Partition(Event Stream) Subscription B (2, 2) Subscription C (3, 2)
- 21. Message retention (2) Partition(Event Stream) Subscription B (2, 2) Subscription C (3, 2) NOT OK to deleteOK to delete
- 22. Message retention (3) Partition(Event Stream) Subscription B (2, 2) Subscription C (3, 2) NOT OK to deleteOK to delete Message Retention
- 23. Message retention (4) Partition(Event Stream) Subscription B (2, 2) Subscription C (3, 2) Yet to be processedOK to delete Message Retention
- 24. Message retention (5) Acked Msg1 Acked Msg 2 Acked Msg 3 Acked Msg 4 Acked Msg 5 Acked Msg 6 Acked Msg 7 Acked Msg 8 Unacked Msg 9 Unacked Msg 10 Unacked Msg 11 Deleted Retention Yet to be processed
- 25. Message expiry (1) Acked Msg5 Acked Msg 6 Acked Msg 7 Acked Msg 8 Unacked Msg 9 Unacked Msg 10 Unacked Msg 11 Deleted Retention Not within TTL (may still be processed) Unacked Msg 12 Unacked Msg 13 Unacked Msg 14 Unacked Msg 15 Within the applied TTL
- 26. Message expiry (2) Acked Msg5 Acked Msg 6 Acked Msg 7 Acked Msg 8 Acked Msg 9 Acked Msg 10 Acked Msg 11 Deleted Retention Not within TTL (may still be processed) Acked Msg 12 Unacked Msg 13 Unacked Msg 14 Unacked Msg 15
- 27. Backlog (1) Acked Msg 5 Acked Msg6 Acked Msg 7 Acked Msg 8 Unacked Msg 9 Unacked Msg 10 Unacked Msg 11 Deleted Retention Unacked Msg 12 Unacked Msg 13 Unacked Msg 14 Unacked Msg 15 Yet to be processed
- 28. Backlog (2) Acked Msg 5 Acked Msg6 Acked Msg 7 Acked Msg 8 Unacked Msg 9 Unacked Msg 10 Unacked Msg 11 Deleted Retention Unacked Msg 12 Unacked Msg 13 Unacked Msg 14 Unacked Msg 15 Yet to be processed SUB 0 SUB 2 Backlog
- 29. Message deletion (1) Acked Msg5 Acked Msg 6 Acked Msg 7 Acked Msg 8 Acked Msg 9 Acked Msg 10 Acked Msg 11 Deleted Retention Acked Msg 12 Acked Msg 13 Acked Msg 14 Acked Msg 15
- 30. Message deletion (2) Acked Msg5 Acked Msg 6 Acked Msg 7 Acked Msg 8 Acked Msg 9 Acked Msg 10 Acked Msg 11 Deleted Retention Acked Msg 12 Acked Msg 13 Acked Msg 14 Acked Msg 15
- 31. Message deletion (3) Acked Msg5 Acked Msg 6 Acked Msg 7 Acked Msg 8 Acked Msg 9 Acked Msg 10 Acked Msg 11 Deleted Retention Acked Msg 12 Acked Msg 13 Acked Msg 14 Acked Msg 15 S1 S2 S3 S4
- 32. Message deletion (4) Acked Msg5 Acked Msg 6 Acked Msg 7 Acked Msg 8 Acked Msg 9 Acked Msg 10 Acked Msg 11 Deleted Retention Acked Msg 12 Acked Msg 13 Acked Msg 14 Acked Msg 15 S1 S2 S3 S4
- 33. Message deletion (5) Acked Msg5 Acked Msg 6 Acked Msg 7 Acked Msg 8 Acked Msg 9 Acked Msg 10 Acked Msg 11 Deleted Retention Acked Msg 12 Acked Msg 13 Acked Msg 14 Acked Msg 15 S1 S2 S3 S4
- 34. Storage size ✓ Allthe storage occupied by the “undeleted” segments, in bytes
- 35. Message deletion ✓ Messagesare deleted segment by segment ✓ The disk space of a segment is reclaimed by a garbage collector thread after it is deleted ✓ The garbage collector is running periodically ○ gcWaitTime
- 36. Retention settings ✓ Retention(broker / namespace) ○ defaultRetentionTimeInMinutes ○ defaultRetentionSizeInMB ✓ TTL (broker / namespace) ○ ttlDurationDefaultInSeconds
- 37. Trigger retention ✓ LedgerRollover ○ managedLedgerMinLedgerRolloverTimeMinutes ○ managedLedgerMaxLedgerRolloverTimeMinutes ○ managedLedgerMaxEntriesPerLedger
- 38. Garbage collection settings ✓Bookie settings ○ gcWaitTime ○ majorCompactionThreshold ○ majorCompactionInterval ○ minorCompactionThreshold ○ minorCompactionInterval