Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
BigQueueA Big, Fast and PersistentQueue    - by William    http://bulldog2011.github.com/
Feature Highlights   Fast         enqueue and dequeue are close to O(1) direct memory access.   Big         Only limit...
Performance Highlights   In concurrent producing and consuming case,    average throughput is around 166MBps.   In seque...
Design – Logical View   Looks just like a big array or a circular array
Design – Consume Once and FanoutSemantics Support
Design – Physical View : Paged Index +Data File
Design – Components View
Design – Dynamic View :Memory Mapped Sliding Window
Concurrency   Produce(or append) is synchronized in the    queue implementation   Consume(or read) is already thread-safe
Simple Interface   Creation    IBigQueue bigQueue = new BigQueueImpl("d:/bigqueue/tutorial",    "demo");   Enqueue     f...
Fanout Queue   Creation    IFanOutQueue foQueue = new FanOutQueueImpl("d:/tutorial/fanout-queue", "demo");   Enqueue    ...
Fanout Queue Semantics
Use Case 1 :Log Collecting & Consuming
Use Case 2 :Big Data Sorting        Queue Only Algorithm:    1.     Put all data into a source queue    2.     Build a qu...
Source, Samples, Docs and Tutorials    https://github.com/bulldog2011/bigqueue
Other Alternatives   Apache ActiveMQ       http://activemq.apache.org   RabbitMQ       http://www.rabbitmq.com/   Zer...
Upcoming SlideShare
Loading in …5
×

A Big, Fast and Persistent Queue

8,554 views

Published on

introdution to a light-weight, big, fast and persistent queue based on memory mapped file.

Published in: Technology

A Big, Fast and Persistent Queue

  1. 1. BigQueueA Big, Fast and PersistentQueue - by William http://bulldog2011.github.com/
  2. 2. Feature Highlights Fast  enqueue and dequeue are close to O(1) direct memory access. Big  Only limited by available disk space. Persistent  Data is persisted on disk and is crash resistant. Reliable  OS will be responsible for message persistence even your process crashes. Realtime  Produced messages will be immediately visible to consumers Flexible Queue Semantics  Consume once queue, fanout queue, can even consume by index Memory-efficient  Automatic pagging & swapping algorithm, only most recently accessed data is kept in memory. Thread-safe  Multiple threads can concurrently enqueue and dequeue without data corruption. Simple & Light-weight  Current library jar is less than 40K.
  3. 3. Performance Highlights In concurrent producing and consuming case, average throughput is around 166MBps. In sequential producing then consuming case, average throughput is around 333MBps.
  4. 4. Design – Logical View Looks just like a big array or a circular array
  5. 5. Design – Consume Once and FanoutSemantics Support
  6. 6. Design – Physical View : Paged Index +Data File
  7. 7. Design – Components View
  8. 8. Design – Dynamic View :Memory Mapped Sliding Window
  9. 9. Concurrency Produce(or append) is synchronized in the queue implementation Consume(or read) is already thread-safe
  10. 10. Simple Interface Creation IBigQueue bigQueue = new BigQueueImpl("d:/bigqueue/tutorial", "demo"); Enqueue for(int i = 0; i < 10; i++) { String item = String.valueOf(i); bigQueue.enqueue(item.getBytes()); } Dequeue for(int i = 0; i < 5; i++) { String item = new String(bigQueue.dequeue()); } Peek byte[] data = bigQueue.peek();
  11. 11. Fanout Queue Creation IFanOutQueue foQueue = new FanOutQueueImpl("d:/tutorial/fanout-queue", "demo"); Enqueue for(int i = 0; i < 10; i++) { String log = "log-" + i; foQueue.enqueue(log.getBytes()); } Fanout 1 Dequeue String fanoutId1 = "realtime"; while(!foQueue.isEmpty(fanoutId1)) { String item = new String(foQueue.dequeue(fanoutId1)); System.out.println(item); } Fanout 2 Dequeue String fanoutId2 = "offline"; while(!foQueue.isEmpty(fanoutId2)) { String item = new String(foQueue.dequeue(fanoutId2)); System.out.println(item); } Fanout 1 and Fanout 2 consuming are independent
  12. 12. Fanout Queue Semantics
  13. 13. Use Case 1 :Log Collecting & Consuming
  14. 14. Use Case 2 :Big Data Sorting Queue Only Algorithm: 1. Put all data into a source queue 2. Build a queueOfSortedQueues by dividing and sorting the source queue 3. Merge sort the queueOfSortedQueues 4. The last one left in the queueOfSortedQueues is the final sorted queue
  15. 15. Source, Samples, Docs and Tutorials https://github.com/bulldog2011/bigqueue
  16. 16. Other Alternatives Apache ActiveMQ  http://activemq.apache.org RabbitMQ  http://www.rabbitmq.com/ ZeroMQ  http://www.zeromq.org Kestrel  https://github.com/robey/kestrel Apache Kafka  http://kafka.apache.org

×