2. Timothy Spann
Fast Data On-Ramp with Apache
Pulsar on K8

3. Timothy Spann
Developer Advocate
Fast Data On-Ramp with
Apache Pulsar on K8
17 May 2022 I Valencia, Spain

4. As the Pulsar communities grows, more and more connectors will be added. To enhance the
availability of sources and sinks and to make use of the greater Apache Streaming community,
joining forces between Apache NiFi and Apache Pulsar is a perfect fit.
Apache NiFi also adds the benefits of ELT, ETL, routing, data crunching, transformation,
validation and batch data processing. Once data is ready to be an event, NiFi can launch it into
Pulsar at light speed. I will walk through how to get started, some use cases and demos and
answer questions. Benefits to the Ecosystem.

5. Apache NiFi Benefits
https://www.influxdata.com/integration/mqtt-monitoring/
https://www.influxdata.com/integration/mqtt-monitoring/
• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Supports push and pull
models
• Hundreds of processors
• Visual command and
control
• Over a 300 components
• Flow templates
• Pluggable/multi-role
security
• Designed for extension
• Clustering
• Version Control

6. Building
Microservices
Asynchronous
Communication
Building Real Time
Applications
Highly Resilient
Tiered storage
Apache Pulsar Benefits

7. ● “Bookies”
● Stores messages and cursors
● Messages are grouped in
segments/ledgers
● A group of bookies form an “ensemble” to
store a ledger
● “Brokers”
● Handles message routing and
connections
● Stateless, but with caches
● Automatic load-balancing
● Topics are composed of
multiple segments
●
● Stores metadata for both Pulsar
and BookKeeper
● Service discovery
Store
Messages
Metadata & Service
Discovery
Metadata & Service
Discovery
Metadata Store
(ZK, RocksDB, etcd, …)
Pulsar Cluster

8. Pulsar Cluster
CLUSTER
Global ZooKeeper
Configuration
Store
ZK3
ZK2
ZK1
Local
ZooKeeper
Quorum
Bookie
0
Bookie
1
Bookie
2
BookKeeper
Bookie
Ensemble
Pulsar
Broker 0
Pulsar
Broker 1
Pulsar
Broker 2

9. Apache Pulsar
Apache BookKeeper
Offloader & Tiered Storage
Offloading
Broker 0
Producer Consumer
Broker 1 Broker 2
Bookie 0 Bookie 1 Bookie 2 Bookie 3 Bookie 4
S S S S
T
1
T
2
T
3
T
4
T
0
S

10. Function Mesh
Pulsar Functions, along with Pulsar IO/Connectors, provide a powerful API for ingesting, transforming,
and outputting data.
Function Mesh, another StreamNative project, makes it easier for developers to create entire
applications built from sources, functions, and sinks all through a declarative API.

11. Accessing historical as well as real-time data
Pub/sub model enables event streams to be sent from multiple
producers, and consumed by multiple consumers
To process large amounts of data in a highly scalable way
Messaging and Streaming Usage

12. Apache NiFi Pulsar Connector
https://streamnative.io/apache-nifi-connector/
https://github.com/tspannhw/awesome-nifi-pulsar
https://t.co/TbcYhdUPVn

13. https://github.com/streamnative/pulsar-nifi-bundle
Apache NiFi Pulsar Connector

14. Apache Pulsar Kafka K8
https://docs.streamnative.io/platform/v1.3.0/quickstart

15. Why StreamNative Cloud?

16. Apache Pulsar Resources
● https://pulsar.apache.org/docs/en/kubernetes-helm/
● https://pulsar.apache.org/charts
● https://streamnative.io/streamnativecloud/
● https://medium.com/@_oleksii_/how-to-deploy-apache-puls
ar-cluster-in-kubernetes-808ecdfd87
Academy.StreamNative.io

17. Visit My Booth #S91
Stop By S91
Scan the QR Code to learn more about
StreamNative and our exclusive booth offerings.

18. Auto Scaling Pulsar FN with K8
https://streamnative.io/blog/engineering/2022-01-19-auto-scaling-pulsar-functions-in-kubernetes-using-custom-metrics/