Downloaded 10 times
Uploaded byScyllaDB
Architecting a High-Performance (Open Source) Distributed Message Queuing System in C++
The document discusses BlazingMQ, a high-performance distributed message queuing system developed by Bloomberg, highlighting its features such as multi-producer/multi-consumer capability, high availability, and scalability. It has been successfully utilized in production for over eight years, processing billions of messages daily with low latency. Additionally, the system separates control and data planes for improved performance and is available as open source.
More Related Content
![[OSC2016] マイクロサービスを支える MQ を考える](https://cdn.slidesharecdn.com/ss_thumbnails/event-161105054759-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Architecting a High-Performance (Open Source) Distributed Message Queuing System in C++
Architecting a High-Performance (Open Source) Distributed Message Queuing System in C++
- 1. HOSTED BY Architecting aHigh-Performance Distributed Message Queuing System Vitaly Dzhitenov Senior Engineer at Bloomberg
- 2. Agenda ■ BlazingMQ overview ■Distributed architecture ■ Key performance-related ideas ■ Broker architecture ■ Fixing bottlenecks 2
- 3. What is BlazingMQ? ■Multi-producer, multi-consumer message queue ■ Physical decoupling, as well as temporal isolation, between the actors ■ Guaranteed acknowledgment ■ Message persistence and replication ■ High availability ■ Transport abstraction ■ Scalability (just add more workers / applications); high fan-out ratio (1:6,000+) 3
- 4. Distributed architecture 4 App App Proxy Node Node Node Node App App Proxy DataCenter 2 Data Center 1 Replica Leader Replica Primary
- 5. Queue trajectories 5 Primary Replica ReplicaReplica Proxy Proxy Proxy Proxy Proxy Consumer Consumer Consumer Consumer Consumer Consumer Consumer Proxy Producer Proxy Producer Producer Replica Replica PUTs PUSHes
- 6. BlazingMQ at Bloomberg ■Battle-tested in production for eight (8) years ■ 55,000+ queues ■ Processing billions messages and terabytes of data daily ■ Low Latency ● For 600,000 msg/sec to no persistence queue w/ fan-out ratio 5, the median is 1.7ms ● For 150,000 msg/sec over 10 persistent queues, the median is 1.4ms ● https://bloomberg.github.io/blazingmq/docs/performance/benchmarks/ 6
- 7. Performance ■ Actor threadmodel ■ Batching ■ Memory and Object pools, polymorphic allocators ■ No data copying 7
- 8. Actors ■ Client ● Reading/writingto client ● Statistics and validation ● Queue lookup ■ Queue ● Storage and replication ● Data routing ■ Cluster ● Reading/writing to cluster nodes ● Cluster health ● Primary node ● Queue lookup 8
- 9.
- 10. CLIENT CLUSTER DispatcherClient type: QUEUE EventDispatcher 10 ThreadPool Queue Processors: Monitored SingleConsumerQueues Clients EventPool
- 11. Batching ■ Batch buildersfor every data type ■ Flushing (to the network) on: ● Size limit, fixed or auto-tuning ● Dispatcher queue idleness ● Intelligent batching decisions: ■ Adjustable batch size ■ Interdependent flushing 11
- 12. Proxy Primary Replica Channel Advancedbatching 12 Cluster Client Queue network network Channel network network Client Client Cluster Queue network network PUT PUSH Replication PUSH PUSH PUT PUT Queue Cluster PUT PUT PUSH
- 13. Actor bottleneck 13 Cluster Client Queue ClientQueue Client Queue
- 14. The solution ■ SeparateControl and Data planes ■ Keep Cluster on the Control Plane and bypass it on the Data Plane ■ Queue takes over Context, Statistics, and Validation work on the Data Plane ■ Queue validates data using lockless synchronization with Cluster ● AtomicGate ■ Based on one atomic int ■ Multiple lockless, non-blocking AtomicGate::tryEnter ■ Single AtomicGate::open, AtomicGate::closeAndDrain 14
- 15. Published as OpenSource! ■ https://github.com/bloomberg/blazingmq ■ https://bloomberg.github.io/blazingmq ■ https://bloomberg.github.io/blazingmq/docs/performance/benchmarks/ 15
- 16.