1. Unified
Messaging and
Streaming
03-Oct-2022

2. ● Apache Pulsar Committer | Author of Pulsar In Action
● Former Principal Software Engineer on Splunk’s messaging
team that is responsible for Splunk’s internal
Pulsar-as-a-Service platform.
● Former Director of Solution Architecture at Streamlio.
David
Kjerrumgaard
Developer Advocate

3. Tim Spann
Developer Advocate
Tim Spann, Developer Advocate at StreamNative
● FLiP(N) Stack = Flink, Pulsar and NiFI Stack
● Streaming Systems & Data Architecture Expert
● Experience:
○ 15+ years of experience with streaming technologies including Pulsar,
Flink, Spark, NiFi, Big Data, Cloud, MXNet, IoT, Python and more.
○ Today, he helps to grow the Pulsar community sharing rich technical
knowledge and experience at both global conferences and through
individual conversations.

4. Apache Pulsar is a Cloud-Native
Messaging and Event-Streaming Platform.

5. 101
Unified
Messaging
Platform
Guaranteed
Message
Delivery
Resiliency Infinite
Scalability

6. API Model
Pulsar’s core API design supports both messaging and
streaming use cases with a unified messaging model.
It combines the capabilities of systems like RabbitMQ or
ActiveMQ, but is built on top and exposes the capabilities of a
distributed log, like Apache Kafka or Kinesis.

7. Messaging
Ideal for work queues that do not
require tasks to be performed in a
particular order—for example,
sending one email message to many
recipients.
RabbitMQ and Amazon SQS are
examples of popular queue-based
message systems.
Pulsar: Unified Messaging + Data Streaming

8. Messaging
Ideal for work queues that do not
require tasks to be performed in a
particular order—for example,
sending one email message to many
recipients.
RabbitMQ and Amazon SQS are
examples of popular queue-based
message systems.
Pulsar: Unified Messaging + Data Streaming
.. and Streaming
Works best in situations where the
order of messages is important—for
example, data ingestion.
Kafka and Amazon Kinesis are
examples of messaging systems that
use streaming semantics for
consuming messages.

9. Messaging vs. Streaming
#1 Message
Queuing
#2 Data
Streaming
● Not built for the cloud
● Single tenant systems
● Monolithic architecture couples compute with storage
● Lack of geo replication support

10. Pulsar Subscription Modes
Different subscription modes
have different semantics:
Exclusive/Failover -
guaranteed order, single active
consumer
Shared - multiple active
consumers, no order
Key_Shared - multiple active
consumers, order for given key
Producer 1
Producer 2
Pulsar Topic
Subscription D
Consumer D-1
Consumer D-2
Key-Shared
<
K
1,
V
10
>
<
K
1,
V
11
>
<
K
1,
V
12
>
<
K
2
,V
2
0
>
<
K
2
,V
2
1>
<
K
2
,V
2
2
>
Subscription C
Consumer C-1
Consumer C-2
Shared
<
K
1,
V
10
>
<
K
2,
V
21
>
<
K
1,
V
12
>
<
K
2
,V
2
0
>
<
K
1,
V
11
>
<
K
2
,V
2
2
>
Subscription A Consumer A
Exclusive
Subscription B
Consumer B-1
Consumer B-2
In case of failure in
Consumer B-1
Failover

11. Streaming
Consumer
Consumer
Consumer
Subscription
Shared
Failover
Consumer
Consumer
Subscription
In case of failure in
Consumer B-0
Consumer
Consumer
Subscription
Exclusive
X
Consumer
Consumer
Key-Shared
Subscription
Pulsar
Topic/Partition
Messaging

12. Architecture and Scaling

13. ● “Bookies”
● Stores messages and cursors
● Messages are grouped in
segments/ledgers
● A group of bookies form an
“ensemble” to store a ledger
● “Brokers”
● Handles message routing and
connections
● Stateless, but with caches
● Automatic load-balancing
● Topics are composed of
multiple segments
●
● Stores metadata for both
Pulsar and BookKeeper
● Service discovery
Store
Messages
Metadata &
Service Discovery
Metadata &
Service Discovery
Pulsar Architecture
MetaData
Storage

14. Multi-Tiered Architecture

15. Pulsar is built for easy scale-out.
*Illustrations by Jack Vanlightly

16. Connectivity
• Functions - Lightweight Stream
Processing (Java, Python, Go)
• Connectors - Sources & Sinks (InfluxDB,
Kafka, S3, Kinesis, Lambda, …)
• Protocol Handlers - AoP (AMQP), KoP
(Kafka), MoP (MQTT)
• Processing Engines - Flink, Spark,
Presto/Trino via Pulsar SQL
• Data Offloaders - Tiered Storage - (S3)
hub.streamnative.io

17. Moving Data In and Out of Pulsar
IO/Connectors are a simple way to integrate with external systems and
move data in and out of Pulsar.
● Built on top of Pulsar Functions
● Built-in connectors - hub.streamnative.io
Source Sink

18. Aerospike <-> Apache Pulsar

19. Aerospike ← Apache Pulsar

20. Aerospike -> Apache Pulsar

21. Today’s Quick Aircraft Data Demo

22. https://medium.com/p/554124125011
https://github.com/tspannhw/FLiP-Py-ADS-B
https://github.com/tspannhw/pulsar-adsb-function
https://www.youtube.com/watch?v=0LMOglFN8ZA

27. FLiP Stack Weekly
This week in Apache Flink, Apache Pulsar, Apache
NiFi, Apache Spark and open source friends.
https://bit.ly/32dAJft

28. ● https://docs.aerospike.com/connect/pulsar/from-asdb
● https://docs.aerospike.com/connect/pulsar/to-asdb/from-pu
lsar-to-asdb-overview
● https://pulsar.apache.org/docs/io-aerospike-sink/
● https://aerospike.com/products/connect-for-pulsar/
Aerospike <-> Apache Pulsar Resources

29. Tim Spann
Developer Advocate
@PaaSDev
https://www.linkedin.com/in/timothyspann
https://github.com/tspannhw

31. StreamNative: By the Creators Of Apache
Pulsar
✓ Original creators of Apache
Pulsar & BookKeeper
✓ Operated the largest
Pulsar/BookKeeper cluster
✓ Data veterans with extensive
industry experience
CONFIDENTIAL. DO NOT SHARE.
ASF Member
Pulsar/BookKeeper PMC
Founder and CEO
Sijie Guo
ASF Member
Pulsar/BookKeeper PMC
CTO
Matteo Merli
Pulsar/BookKeeper PMC
Co-Founder
Jia Zhai

32. Pulsar ranked as a Top 5 ASF Project
> 580+
Contributors
> 10,000+
Commits
> 8,500+
Slack Members
> 1,000+
Organizations
using Pulsar

33. Pulsar has already been deployed by
thousands of companies across the globe

34. Q&A
Additional Resources
● For a first look at Pulsar benchmark report, share your email in the chat
● Join the Pulsar Slack channel - Apache-Pulsar.slack.com
● Follow @streamnativeio and @apache_pulsar on Twitter
● Contact StreamNative Sales - doug@streamnative.io