Apache Pulsar is a cloud-native messaging and event streaming platform characterized by a multi-layered architecture and segment-centric storage. It supports various data processing categories, including batch, streaming, and interactive processing, and offers scalability, fault tolerance, and high throughput for both messaging and batch data reading. Pulsar's features include instant scaling, unlimited topic partition storage, and a unified view of data, making it suitable for diverse event-driven applications.

1. streamnative.io
Stream/Segment - best way to access
events in Pulsar
Neng Lu

2. Who Am I
❏ StreamNative Software Engineer
❏ Ex-Twitter
❏ Contributed to Apache Projects - Heron, Pulsar
❏ Interested in event streaming technologies

3. Pulsar 1.X

4. Apache Pulsar
“Flexible Pub/Sub Messaging Backed by Durable Log Storage”

5. Pulsar 2.X

6. Apache Pulsar
“Cloud-native Messaging and Event Streaming Platform”

7. Pulsar Use Cases
❏ Unified Event Center/Bus (Queuing + Streaming)
❏ Billing Service
❏ Push Notification
❏ Worker Queue
❏ Logging Pipeline
❏ IoT
❏ Streaming-first, unified data processing

8. Data Processing with Apache Pulsar

9. Data Processing Categories
❏ Batch
❏ The amount of data is huge
❏ Can run on a huge cluster
❏ Fine-grained fault tolerance

10. Data Processing Categories
❏ Batch
❏ The amount of data is huge
❏ Can run on a huge cluster
❏ Fine-grained fault tolerance
❏ Streaming
❏ Long running jobs
❏ Time critical
❏ scalability as well as fault tolerant

11. Data Processing Categories
❏ Interactive
❏ Time critical
❏ Medium data size
❏ Rerun on failures
❏ Batch
❏ The amount of data is huge
❏ Can run on a huge cluster
❏ Fine-grained fault tolerance
❏ Streaming
❏ Long running jobs
❏ Time critical
❏ scalability as well as fault tolerant

12. Data Processing Categories
❏ Interactive
❏ Time critical
❏ Medium data size
❏ Rerun on failures
❏ Batch
❏ The amount of data is huge
❏ Can run on a huge cluster
❏ Fine-grained fault tolerance
❏ Streaming
❏ Long running jobs
❏ Time critical
❏ scalability as well as fault tolerant
❏ Serverless
❏ Simple, light-weight processing
❏ Processing data with high
velocity

13. Apache Pulsar Layered Architecture
Stateless Serving
Durable Storage

14. Pulsar Messaging API
❏ Read data from brokers with different Subscription Modes
❏ Consume / Seek / Receive
❏ Reprocessing data by rewinding (seeking) the cursors

15. Subscription Mode
❏ Exclusive
❏ Failover
❏ Shared
❏ Key_Shared

16. Pulsar Segment API
❏ Read data from storage (bookkeeper or tiered storage)
❏ Fine-grained Parallelism
❏ Predicate pushdown (publish timestamp)

17. Segment Centric Storage
❏ Topic Partition (Managed Ledger)
❏ The storage layer for a single topic
partition
❏ Segment (Ledger)
❏ Single writer, append-only
❏ Replicated to multiple bookies

18. Tired Storage
❏ Long retention
❏ Low cost
❏ Easy to access

19. Apache Pulsar Data APIs
Bookie1 Bookie2 Bookie3 Bookie4
Producer Consumer
Broker 1 Broker 2 Broker 3
Bookie5
HADOOPGCSS3
Messaging API
Segment API

20. Pulsar - Infinite Event Stream Storage

21. Pulsar - Topic

22. Pulsar - Topic Partitions

23. Pulsar - Segments

24. Pulsar - Stream

25. Pulsar - Infinite Event Stream Storage

26. Benefits
❏ Unlimited Topic Partition Storage
❏ Instant Scaling without Data Rebalancing
❏ Broker Failure Recovery
❏ Bookie Failure Recovery
❏ Cluster Expansion
❏ Low latency reading for messaging data
❏ High throughput reading for batch data
❏ Reduced cost for whole data storage

27. Pulsar SQL Case

28. Pulsar Flink Case
Flink
Job18 7 6 5 4 3 2 1
1
2
1
1
1
0
9
Flink
Job2

29. Conclusion
❏ Apache Pulsar is a cloud-native messaging streaming system
❏ Multi layered architecture
❏ Segment centric storage
❏ Two levels of reading API: Pub/Sub + Segment
❏ Apache Pulsar provides a unified view of data

30. Community
❏ Pulsar Website: https://pulsar.apache.org
❏ Twitter: @apache_pulsar / @streamnativeio
❏ Slack: https://apache-pulsar.herokuapp.com
❏ Mailing Lists dev@pulsar.apache.org , users@pulsar.apache.org
❏ Github: https://github.com/apache/pulsar
❏ Medium: https://medium.com/streamnative

31. Thank You!

32. Reference
❏ http://pulsar.apache.org/docs/en/concepts-overview/
❏ https://www.splunk.com/en_us/blog/it/comparing-pulsar-and-kafka-how-a-segment-based-architecture-delivers-better-
performance-scalability-and-
resilience.html#:~:text=Pulsar%20Architecture%20Basics&text=An%20Apache%20Pulsar%20cluster%20is,bookies%20that%2
0durably%20store%20messages.
❏ https://pulsar.apache.org/docs/en/concepts-tiered-storage/
❏ https://flink.apache.org/2019/05/03/pulsar-flink.html