Queue in the cloud with mongo db

517 views

Published on

Leveraging MongoDB as a queue infrastructure. This talk describes a low-overhead approach for building a queue as a service built on top of MongoDB, covering concerns of durability, long running processes and distributed processing.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
517
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Queue holds elements until they are required. Items in the queue are accessed from the head of the queue only – implied order.
  • Add capped collection
  • Queue in the cloud with mongo db

    1. 1. QUEUE IN THECLOUD WITHMONGODBMONGODB LA 2013NURI HALPERIN
    2. 2. QUEUE
    3. 3. USAGEOrdered executionBuffering consumer/producerWork distribution
    4. 4. GOALS OF PROJECTLeverage Mongo • Reduce ops overhead by reusing infrastructure • Map queue semantics to Mongo’s strengthsReliable • Durable - support long running process • Resilient to machine failure • Narrow down window of failure/ data loss.Centralized, distributed: • Multiple producers • Multiple consumers
    5. 5. ITERATION 0Capped collection – not the perfect choice • Tailing queue seems attractive, but… • Need external sync to avoid double-consume • Secondary indexes and updating are anti-patternRelaxing FIFO is OK • No guarantee that first-popped is first done • Multi-client is negated if they have to sync on execution order • Race condition for queue insertion has same effectConclusion: Project doesn’t use capped collection andrelaxes FIFO.
    6. 6. PARANOID BY DESIGN Network dies Process dies DB dies Machine dies Poison letter Dead letter
    7. 7. ITERATION 1db.q4foo.save({v:{f:1}})db.q4foo.findAndModify({query: {}, sort: {_id:1}, remove: true})Hot: quick and simpleNot: dead client, dead in transit, no trace
    8. 8. ARE WE THERE YET? Network dies Process dies DB dies Machine dies Poison letter Dead letter
    9. 9. QUEUE SEMANTICSLocal / Memory DistributedPush PutPop Get << visibility >><< exception >> Release << retry >> Delete << exception >>
    10. 10. ITERATION 2 db.q4foo.save({v:{f:1}, dq: null}) db.q4foo.findAndModify( { query: { dq: null}, sort: {_id:1}, update:{ $set: { dq: later(60)}}}) … If processing was success => delete.. Hot: If client dies, item remains in queue. Data not lost. Not: index on _id less useful in high volume.
    11. 11. ARE WE THERE YET? Network dies Process dies DB dies Machine dies Poison letter Dead letter
    12. 12. ITERATION 3db.q4foo.save({v:{f:1}, dq: null, pc: 0})db.q4foo.findAndModify({ query: { dq: null, pc:{$lt:3}}, sort: {_id:1}, update:{$set:{dq:later(60)},$inc:{pc:1}}}) // consumedb.q4foo.findAndModify({ query: {_id:"..."}, update:{$set:{dq: null}}}) // releaseHot: An item can be retried automatically (pc) after released. Exhausted item remains in queue.Not: Not strict FIFO.
    13. 13. ARE WE THERE? YES. Network dies Process dies DB dies Machine dies Poison letter Dead letter
    14. 14. ITERATION 4Ensure your queue writes use applicable durability • db.q4foo.save() + getLastError(…) • db.q4foo.findAndModify () + getLastError(…)Replica sets for durability only. No capacity or speed gain.
    15. 15. OTHER THOUGHTSCreate admin jobs to monitor queues: • Growth • Retries exhaustedConsider TTL risks (ex: client failure before calling Release())Consider idempotent operations when possibleDesign clients to back off pollingSeparate queue vs. extra “topic” fieldConsider dedicated DB for write-lock scopeCapped vs. regular collection – capped now can have _id, in-place update.
    16. 16. Q&A Thank you! Nuri Halperinnuri@plusnconsulting.com

    ×