Downloaded 31 times
Uploaded byKarthik Ramasamy
Linked In Stream Processing Meetup - Apache Pulsar
Apache Pulsar is a fast, highly scalable, and flexible pub/sub messaging system. It provides guaranteed message delivery, ordering, and durability by backing messages with a replicated log storage. Pulsar's architecture allows for independent scalability of brokers and storage nodes. It supports multi-tenancy, geo-replication, and high throughput of over 1.8 million messages per second in a single partition.
More Related Content
Similar to Linked In Stream Processing Meetup - Apache Pulsar
Linked In Stream Processing Meetup - Apache Pulsar
- 1. Karthik Ramasamy Cofounder andChief Product Officer fast, durable, flexible pub/sub messaging
- 2. What is ApachePulsar? 2 “Flexible pub-sub system backed by a durable log storage”
- 3. Pulsar Highlights 3 Ordering Guaranteed ordering Multi-tenancy Asingle cluster can support many tenants and use cases High throughput Can reach 1.8 M messages/s in a single partition Durability Data replicated and synced to disk Geo-replication Out of box support for geographically distributed applications Unified messaging model Support both Topic & Queue semantic in a single model Delivery Guarantees At least once, at most once and effectively once Low Latency Low publish latency of 5ms at 99pct Highly scalable Can support millions of topics
- 4. Pulsar architecture basics 4 •Brokers — Serving nodes • Bookies (Apache BookKeeper) — Storage nodes • Each layer can be scaled independently • No data locality — Data for a single topic/partition is not tied to any particular node
- 5.
- 6. Additional Advantages • Statefulsystems can become unbalanced when traffic changes • The system needs to be designed to allow for quick reaction, distributing the load across all nodes 6
- 7. Pulsar broker • Brokeris the only point of interaction for clients • Brokers acquire ownership of group of topics and “serve” them • Broker has no durable state • There’s a service discovery mechanism for client to connect to right broker 7
- 8.
- 9. Segment centric storage •Storage for a topic is an infinite “stream” of messages • Implemented as a sequence of segments • Each segment is a replicated log — BookKeeper “ledger” • Segments are rolled over based on time, size and after crashes 9
- 10.
- 11. Broker failure recovery 11 •Topic is reassigned to an available broker based on load • Can reconstruct the previous state consistently • No data needs to be copied • Failover handled transparently by client library
- 12. Bookie failure recovery 12 •After a write failure, BookKeeper will immediately switch write to a new bookie, within the same segment. • As long as we have any 3 bookies in the cluster, we can continue to write
- 13. Bookie failure recovery 13 •In background, starts a many-to-many recovery process to regain the configured replication factor
- 14.
- 15. Why should youcare? “Segment centric” vs “Partition centric” 15
- 16. Partition Vs SegmentDistribution
- 17. Partition Based Systems:Disadvantages • In Kafka, partitions are assigned to brokers “permanently” • A single partition is stored entirely in a single node • Retention is limited by a single node storage capacity • Failure recovery and capacity expansion require “rebalancing” • Rebalancing has a big impact over the system, affecting regular traffic 17
- 18. Apache BookKeeper • Areplicated distributed log storage • Low-latency durable writes • Simple repeatable read consistency • Highly available • Store many segments per node • I/O Isolation 18
- 19. BookKeeper - Storage •A single bookie can serve and store hundreds of thousands of ledgers • Write and read paths are separated: • Avoid read activity to impact write latency • Writes are added to in- memory write-cache and committed to journal • Write cache is flushed in background to separated device • Entries are sorted to allow for mostly sequential reads 19
- 20.
- 21. Multi-Tenancy • Authentication /Authorization / Namespaces / Admin APIs • I/O Isolations between writes and reads • Provided by BookKeeper - Ensure readers draining backlog won’t affect publishers • Soft isolation • Storage quotas — flow-control — back-pressure — rate limiting • Hardware isolation • Constrain some tenants on a subset of brokers or bookies 21
- 22. Geo-Replication • Scalable asynchronous replication • Integratedin the broker message flow • Simple configuration to add/remove regions 22 Topic (T1) Topic (T1) Topic (T1) Subscrip/on (S1) Subscrip/on (S1) Producer (P1) Consumer (C1) Producer (P3) Producer (P2) Consumer (C2) Data Center A Data Center B Data Center C
- 23. Pulsar client library •Java — C++ — Python — WebSocket APIs • Partitioned topics • Apache Kafka compatibility wrapper API • Transparent batching of messages • Compression • TLS encryption and authentication • End-to-end encryption 23
- 24. Pulsar Producer 24 PulsarClient client= PulsarClient.create( “http://broker.usw.example.com:8080”); Producer producer = client.createProducer( “persistent://my-property/us-west/my-namespace/my-topic”); // handles retries in case of failure producer.send("my-message".getBytes()); // Async version: producer.sendAsync("my-message".getBytes()).thenRun(() -> { // Message was persisted });
- 25. Pulsar Consumer 25 PulsarClient client= PulsarClient.create( "http://broker.usw.example.com:8080"); Consumer consumer = client.subscribe( "persistent://my-property/us-west/my-namespace/my-topic", "my-subscription-name"); while (true) { // Wait for a message Message msg = consumer.receive(); System.out.println("Received message: " + msg.getData()); // Acknowledge the message so that it can be deleted by broker consumer.acknowledge(msg); }
- 26. Pulsar Functions 26 f(x) Incoming Messages Output Messages ETL Reac/ve Services Classifica/on Real-/me Aggrega/on Event Rou/ng Microservices defprocess(input): return input + '!' SDK Less for Java and Python
- 27. Benchmark • Testing goals •Throughput & latency under different conditions • Min 2 guaranteed copies • Running on 3 EC2 VMs with local SSDs 27
- 28. Max throughput 1 Topic 1Partition 1 Producer 1 Consumer 1Kb msg 28
- 29. Latency at fixedrate - 50K msg/s 29 1 Topic 1 Partition 1 Producer 1 Consumer 1Kb msg
- 30. Pulsar Use Cases- Message Queue 30 • Async processing of time intensive background tasks • Publishes and complete HTTP request • Examples: Image / Video transcoding, bulk operations (large folder deletions)
- 31. Use cases -Microservices 31 • Delegate the processing to multiple compute jobs which interact with micro-services • Wait until the response is pushed back to the HTTP server • Examples: breaking monolithic applications into micro-services
- 32. Use cases -Notifications • Change data capture • Listeners are frequently different tenants • Quotas needs to ensure producer is not affected 32 Event Pulsar topic Component 1 Component 2 Component 3 Listeners
- 33. Use cases -Notifications • Replicating DB transactions across regions • Notification to multiple interested tenants (example: new user account, …) 33
- 34. Use cases -Feedback system • Coordinate a large number of machines • Propagate state 34 External inputs Pulsar topic 1 Serving system Serving system Serving system Pulsar topic 2 Controller Updates Feedback
- 35.
- 36.
- 37. Pulsar — Conclusion •A fast durable, distributed pub/sub messaging • Flexible Traditional Messaging: Queuing and Pub/Sub • Focus on message dispatch and consumption • Remove data as soon as possible if they are not needed • It is backed by a scalable log store • durable message store, zero data loss • allow rewinding / reprocessing messages for backfill, bootstrap systems and stream computing • It carries all the fantastic features from the log store. 37