Download as PDF, PPTX
![GO CLIENT
• go get -u github.com/apache/pulsar/pulsar-client-go/pulsar
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: "pulsar://localhost:6650"
})
producer, err := client.CreateProducer(pulsar.ProducerOptions{
Topic: "my-topic",
})
for i := 0; i < 10; i++ {
err := producer.Send(context.Background(), pulsar.ProducerMessage{
Payload: []byte(fmt.Sprintf("hello-%d", i)),
})
}
• Based on C++ client library — Pure Go client is being worked on
20](https://image.slidesharecdn.com/pulsar-flexiblepub-subforinternetscale-181002061232/85/Pulsar-flexible-pub-sub-for-internet-scale-20-320.jpg)
Uploaded byMatteo Merli
Pulsar - flexible pub-sub for internet scale
Apache Pulsar is a flexible, multi-tenant pub-sub messaging system designed for large-scale internet applications, offering features like high throughput, low latency, and geo-replication. It evolved from initial development at Yahoo in 2012 to becoming an Apache project, addressing scalability and operational challenges faced by existing systems. The architecture includes separate layers for brokers and storage, and it supports multiple client libraries and functions for processing messages.
More Related Content
Similar to Pulsar - flexible pub-sub for internet scale
Pulsar - flexible pub-sub for internet scale
- 1. APACHE PULSAR Flexible Pub-Subsystem for Internet scale http://pulsar.apache.org Content on this presentation is licensed under a Creative Commons Attribution 4.0 International license
- 2. Pulsar graduates asTLPproject today!
- 3. WHO AM I? •Matteo Merli • Apache Pulsar PMC Chair • Member of Apache BookKeeper PMC • Co-Founder of Streamlio • Worked on Pulsar since its beginning atYahoo
- 4. WHAT IS APACHEPULSAR? “Pub-Sub messaging backed by durable log storage”
- 5. WHAT IS APACHEPULSAR? 5 Multi-tenancy A single cluster can support many tenants and use cases Ordering Guaranteed ordering Durability Data replicated and synced to disk Delivery Guarantees At least once, at most once and effectively once Highly scalable Can support millions of topics Unified messaging model Support both Topic & Queue semantic in a single model Geo-replication Out of box support for geographically distributed applications High throughput Can reach 1.8 M messages/s in a single partition Low Latency Low publish latency of 5ms at 99pct
- 6. WHY BUILD ANEW SYSTEM? • No existing solution to satisfy requirements • Multi tenant — 1M topics — Low latency — Durability — Geo replication • Other systems don’t scale well with many topics: • Storage model based on individual directory per topic partition • Durability kills the performance • Ability to manage large backlogs — Read old data without impacting writers • Many other choking points: getting stats, access to metadata, flow-control • Operations are not very convenient — replacing servers, expanding clusters, etc… 6
- 7. STATE OFTHE PROJECT •Project started atYahoo around 2012 and went through various iterations • Open-Sourced in September 2016 • Entered Apache Incubator in June 2017 • Graduated asTLP on September 2018 • 2249 Commits — 22Yahoo releases — 9 Apache releases • 59 Contributors 7
- 8. ARCHITECTURALVIEW Separate layers between brokersbookies • Broker and bookies can be added independently • Traffic can be shifted very quickly across brokers • New bookies will ramp up on traffic quickly
- 9. APACHE BOOKKEEPER Replicated logstorage • Low-latency durable writes • Simple repeatable read consistency • Highly available • Store many logs per node • I/O Isolation
- 10. SEGMENT CENTRIC STORAGE • In additionto partitioning, messages are stored in segments (based on time and size) • Segments are independent from each others and spread across all storage nodes
- 11.
- 12. DATA PATH 1 —Publisher sends message to broker
- 13. DATA PATH 2 —Broker writes in parallel to N replicas
- 14. DATA PATH 3 —Wait for a quorum of acks from bookies
- 15. DATA PATH 4 —Send ack to producer — Dispatch to consumer
- 16. BOOKKEEPER INTERNAL Storage optimizedfor sequential & immutable data • IO isolation between write and read operations • Slow consumers won’t impact latency • Very effective IO patterns: • Journal — append only and no reads • Storage device — bulk write and sequential reads • Number of files is independent from number of topics
- 17.
- 18. PULSAR CLIENT LIBRARY •Java — C++ — Python — Go — WebSocket APIs • Partitioned topics • Apache Kafka compatibility wrapper API • Transparent batching and compression • TLS encryption and authentication • End-to-end encryption 18
- 19. PYTHON CLIENT import pulsar client= pulsar.Client('pulsar://localhost:6650') producer = client.create_producer('my-topic') for i in range(10): producer.send(('Hello-%d' % i).encode('utf-8')) client.close() 19 • pip install pulsar-client
- 20. GO CLIENT • goget -u github.com/apache/pulsar/pulsar-client-go/pulsar client, err := pulsar.NewClient(pulsar.ClientOptions{ URL: "pulsar://localhost:6650" }) producer, err := client.CreateProducer(pulsar.ProducerOptions{ Topic: "my-topic", }) for i := 0; i < 10; i++ { err := producer.Send(context.Background(), pulsar.ProducerMessage{ Payload: []byte(fmt.Sprintf("hello-%d", i)), }) } • Based on C++ client library — Pure Go client is being worked on 20
- 21. MULTI-TENANCY • Authentication /Authorization / Namespaces / Admin APIs • I/O Isolations between writes and reads • Provided by BookKeeper - Ensure readers draining backlog won’t affect publishers • Soft isolation • Storage quotas — flow-control — back-pressure — rate limiting • Hardware isolation • Constrain some tenants on a subset of brokers or bookies 21
- 22. GEO REPLICATION Topic (T1)Topic (T1) Topic (T1) Subscription (S1) Subscription (S1) Producer (P1) Consumer (C1) Producer (P3) Producer (P2) Consumer (C2) Data Center A Data Center B Data Center C • Scalable asynchronous replication • Integrated in the broker message flow • Simple configuration to add/remove regions
- 23. SCHEMA REGISTRY • Storeinformation on the data structure — Stored in BookKeeper • Enforce data types on topic • Allow for compatible schema evolutions 23
- 24. TYPE-SAFE CLIENT API Producer<MyClass>producer = client .newProducer(Schema.JSON(MyClass.class)) .topic("my-topic") .create(); producer.send(new MyClass(1, 2)); 24 Consumer<MyClass> consumer = client .newConsumer(Schema.JSON(MyClass.class)) .topic("my-topic") .subscriptionName("my-subscription") .subscribe(); Message<MyClass> msg = consumer.receive(); • Integrated schema in API • End-to-end type safety — Enforced in Pulsar broker
- 25. PULSAR FUNCTIONS Managed lightweightcompute framework
- 26. PULSAR FUNCTIONS /1 • Simple compute against a consumed message • Managed or manual deployment • A function gets messages from 1 or more topics • An instance of the function is invoked to process the event • The output of the function is published on 1 or more topics 26
- 27. PULSAR FUNCTIONS /2 • Super simple to use — no SDK • Python example: def process(input): return input + '!' • Supports Java & Python — Go will come next 27
- 28. PULSAR FUNCTIONS /3 • Good use cases for functions: • ETL • Data enrichment • Data filtering • Routing 28
- 29. PULSAR FUNCTIONS /4 • Deployment modes: • Local run — Manually run a function, useful for dev mode • Managed — Worker service is running instances of functions 29
- 30. PULSAR IO Connector frameworkbased on Pulsar Functions
- 31. PULSAR IO • Source— Ingest data into a Pulsar topic • Sink — Reads data from topic and dump into external sink • Pulsar provides a set of built-in connectors • Users can submit customized connectors 31
- 32. TIERED STORAGE Unlimited topicstorage capacity Achieves the true “stream-storage”: keep the raw data forever in stream form
- 33. TIERED STORAGE • Leveragecloud storage services to offload cold data — Completely transparent to clients • Extremely cost effective — Backends (S3) (Coming GCS, HDFS) • Example: Retain all data for 1 month — Offload all messages older than 1 day to S3 33
- 34. PULSAR SQL • Comingvery soon in Pulsar 2.2 • Interactive SQL queries over data stored in Pulsar • Query old and real-time data 34
- 35. PULSAR SQL /2 • Based on Presto by Facebook — https://prestodb.io/ • Presto is a distributed query execution engine • Fetches the data from multiple sources (HDFS, S3, MySQL, …) • Full SQL compatibility 35
- 36. PULSAR SQL /3 • Pulsar connector for Presto • Read data directly from BookKeeper — bypass Pulsar Broker • Many-to-many data reads • Data is split even on a single partition — multiple workers can read data in parallel from single Pulsar partition • Time based indexing — Use “publishTime” in predicates to reduce data being read from disk 36
- 37.
- 38. BENCHMARK FRAMEWORK • Designedto measure performance of distributed messaging systems • Supports various “drivers” (Kafka, Pulsar, RocketMQ, RabbitMQ) • Automated deployment in EC2 • Configure workloads through aYAML file 38
- 39. DISTRIBUTED EXECUTION Coordinator willtake the workload definition and propagate to multiple workers — Collects and reports stats
- 40.
- 41. Latency at fixed throughput 50Kmsg/s 1Topic 1 Partition 1KB payload
- 42. Latency at fixed throughput — (includingKafka-sync) 50K msg/s 1Topic 1 Partition 1KB payload
- 43. Latency at fixed throughput — 99pct 50Kmsg/s 1Topic 1 Partition 1KB payload
- 44.