Your SlideShare is downloading. ×
A Big, Fast and Persistent Queue
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

A Big, Fast and Persistent Queue

4,847
views

Published on

introdution to a light-weight, big, fast and persistent queue based on memory mapped file.

introdution to a light-weight, big, fast and persistent queue based on memory mapped file.

Published in: Technology

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,847
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
32
Comments
0
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. BigQueueA Big, Fast and PersistentQueue - by William http://bulldog2011.github.com/
  • 2. Feature Highlights Fast  enqueue and dequeue are close to O(1) direct memory access. Big  Only limited by available disk space. Persistent  Data is persisted on disk and is crash resistant. Reliable  OS will be responsible for message persistence even your process crashes. Realtime  Produced messages will be immediately visible to consumers Flexible Queue Semantics  Consume once queue, fanout queue, can even consume by index Memory-efficient  Automatic pagging & swapping algorithm, only most recently accessed data is kept in memory. Thread-safe  Multiple threads can concurrently enqueue and dequeue without data corruption. Simple & Light-weight  Current library jar is less than 40K.
  • 3. Performance Highlights In concurrent producing and consuming case, average throughput is around 166MBps. In sequential producing then consuming case, average throughput is around 333MBps.
  • 4. Design – Logical View Looks just like a big array or a circular array
  • 5. Design – Consume Once and FanoutSemantics Support
  • 6. Design – Physical View : Paged Index +Data File
  • 7. Design – Components View
  • 8. Design – Dynamic View :Memory Mapped Sliding Window
  • 9. Concurrency Produce(or append) is synchronized in the queue implementation Consume(or read) is already thread-safe
  • 10. Simple Interface Creation IBigQueue bigQueue = new BigQueueImpl("d:/bigqueue/tutorial", "demo"); Enqueue for(int i = 0; i < 10; i++) { String item = String.valueOf(i); bigQueue.enqueue(item.getBytes()); } Dequeue for(int i = 0; i < 5; i++) { String item = new String(bigQueue.dequeue()); } Peek byte[] data = bigQueue.peek();
  • 11. Fanout Queue Creation IFanOutQueue foQueue = new FanOutQueueImpl("d:/tutorial/fanout-queue", "demo"); Enqueue for(int i = 0; i < 10; i++) { String log = "log-" + i; foQueue.enqueue(log.getBytes()); } Fanout 1 Dequeue String fanoutId1 = "realtime"; while(!foQueue.isEmpty(fanoutId1)) { String item = new String(foQueue.dequeue(fanoutId1)); System.out.println(item); } Fanout 2 Dequeue String fanoutId2 = "offline"; while(!foQueue.isEmpty(fanoutId2)) { String item = new String(foQueue.dequeue(fanoutId2)); System.out.println(item); } Fanout 1 and Fanout 2 consuming are independent
  • 12. Fanout Queue Semantics
  • 13. Use Case 1 :Log Collecting & Consuming
  • 14. Use Case 2 :Big Data Sorting Queue Only Algorithm: 1. Put all data into a source queue 2. Build a queueOfSortedQueues by dividing and sorting the source queue 3. Merge sort the queueOfSortedQueues 4. The last one left in the queueOfSortedQueues is the final sorted queue
  • 15. Source, Samples, Docs and Tutorials https://github.com/bulldog2011/bigqueue
  • 16. Other Alternatives Apache ActiveMQ  http://activemq.apache.org RabbitMQ  http://www.rabbitmq.com/ ZeroMQ  http://www.zeromq.org Kestrel  https://github.com/robey/kestrel Apache Kafka  http://kafka.apache.org