The talk is about how Apache Pulsar can have topic backlogs of unlimited size, opening up a whole array of Big Data use-cases that are not possible with other messaging systems. We also delve into tiered storage, which can make these massive backlogs very cheap. Messaging systems are an essential part of any real-time analytics engine. A common pattern is to feed a user event stream into a processing engine, show the result to the user, capture feedback from the user, push the feedback back into the event stream, and so on. The quality of the result shown to the user is often a function of the amount of data in the event stream, so the more your event stream scales, the better you can serve your users. Messaging systems have recently started to push into the field of long-term data storage and event stores, where you cannot compromise on retention. If data is written to the system, it must stay there. Infinite retention can be challenging for a messaging system. As data grows for a single topic, you need to start storing different parts of the backlog on different sets of machines without losing consistency. In this talk, I will describe how Pulsar uses Apache BookKeeper in its segment oriented architecture. BookKeeper provides a unit of consensus called a ledger. Pulsar strings together a number of BookKeeper ledgers to build the complete topic backlog. Each ledger in the topic backlog is independent of all previous ledgers with regards to location. This allows us to scale the size of the topic backlog simply by adding more machines. When the storage node is added to a Pulsar cluster, the brokers will detect it, and gradually start writing new data to the new node. There’s no disruptive rebalancing operation necessary. Of course, adding more machines will eventually get very expensive. This is where tiered storage comes in. With tiered storage, parts of the topic backlog can be moved to cheaper storage such as Amazon S3 or Google Cloud Storage. I will also discuss the architecture of tiered storage, and how it is a natural continuation of Pulsar’s segment oriented architecture. Finally, if you start storing data for a long time in Pulsar, you may want a means to query it. I will introduce our SQL implementation, based on the Presto query engine, which allows users to easily query topic backlog data, without having to read the whole thing.

1. Infinite* Topic Backlogs
with Apache Pulsar
Ivan Kelly
@ivankelly

2. Peter Norvig - Google
We don’t have better algorithms than anyone else; we
just have more data

3. An example system
Stream
Processing
Engine
Model
User
events
Serving
System
Songs user has
listened to
Recommendation algorithm
User listening model
Recommended songs

4. Stream
Processing
Engine
Partial Model
Live User
events
Serving
System
Historical
User Events
Batch
Processing
Engine
Partial Model
Combined
Model

5. Stream
Processing
Engine
Live User
events
Serving
System
Partial Model
Historical
User Events
Batch
Processing
Engine
Partial Model
Combined
Model
Model

6. Apache Pulsar
Messaging System
Pubsub and queuing
Strong guarantees on message persistence
Unlimited topic backlog size
Tiered Storage
Lightweight compute
SQL interface available

7. Anatomy of a topic backlog
A B C
Broker
Client produces message
Broker acknowledges
message to client
Broker writes message
to topic backlog
D E F G
Topic Backlog

8. Anatomy of a topic backlog
A B C D E F G
Closed Open
Topic Backlog

9. A B C
Storage 2
Storage 3
Storage 1

10. Storage layer
Storage 1 Storage 2 Storage 3
Serving layer
Broker 1 Broker 2

11. Storage layer
Add nodes to add storage
Topic Backlog

12. Mirroring
1/n = 0.5 space efficiency
Tolerates n - 1 = 1 failures
Gets worse with more replicas
Striping with parity
1 - 1/n = 0.833 space efficiency
Tolerates 1 failure
Space efficiency increases with nParity

13. Tiered Storage
Broker
Client writes
messages
Storage nodes
Long-term storage
Client reads
messages

14. 1. Pulsar topic metadata updated with potential new location
2. All messages from segment copied to data object in long-term storage
3. Copying process tracks message offsets at defined interval
4. Offsets written to index object in long-term storage
5. Pulsar topic metadata updated with offload completed status
6. Original segment deleted from Pulsar storage after grace period

15. Available long-term storage options
Since v2.1.0
Since v2.2.0
Implemented but not merged

16. Implementing your own
public interface LedgerOffloader {
String getOffloadDriverName();
Map<String, String> getOffloadDriverMetadata();
CompletableFuture<Void> offload(ReadHandle ledger, UUID uid,
Map<String, String> extraMetadata);
CompletableFuture<ReadHandle> readOffloaded(long ledgerId, UUID uid,
Map<String, String> offloadDriverMetadata);
CompletableFuture<Void> deleteOffloaded(long ledgerId, UUID uid,
Map<String, String> offloadDriverMetadata);
}
1. Implement LedgerOffloader interface
2. Drop NAR file in <pulsar-directory>/offloaders

17. Demo
Bloom
Filter
Function
Twitter topic
Query topic
Result topic

18. Querying massive backlogs
Pulsar provides SQL interface from v2.2.0
Based on PrestoDB
Table schema from topic schema
Queries data at rest

19. class ListenEvent {
UUID userId;
String artist;
String title;
String album;
}
Schema schema = Schema.AVRO(ListenEvent.class);
String topic = “user_profiling”;
try (Producer<ListenEvent> p = client.newProducer(schema)
.topic(topic).create()) {
p.send(new ListenEvent(myId, "Metallica",
"Fade to Black", "Ride the lightning"));
p.send(new ListenEvent(myId, "Black Sabbath",
"Sweet Leaf", "Master of Reality"));
p.send(new ListenEvent(myId, "Dio", "Holy Diver", "Holy Diver"));
}
presto> SELECT * from pulsar."public/default".user_profiling WHERE artist = 'Dio';
userId | artist | title | album |
--------------------------------------+--------+------------+------------+
79a6540c-e113-11e8-a275-636165afe980 | Dio | Holy Diver | Holy Diver |

20. Long-term
storage
Broker
StorageStorageStorageStorageStorageStorage
BrokerBrokerBrokerBroker
Presto
Worker
Presto
Worker
Presto
Worker
Presto
client

21. Summary
Apache Pulsar
• Unlimited backlog size
• Offload older data to cheaper storage
• Query with an SQL interface
Other uses cases for massive backlogs
• CQRS Event sourcing
• Data marts
• Audit logs
• Security logs

22. Blog: https://streaml.io/blog
@streamlio @ivankelly