Your SlideShare is downloading. ×
Queue in the cloud with mongo db
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Queue in the cloud with mongo db

292
views

Published on

Leveraging MongoDB as a queue infrastructure. This talk describes a low-overhead approach for building a queue as a service built on top of MongoDB, covering concerns of durability, long running …

Leveraging MongoDB as a queue infrastructure. This talk describes a low-overhead approach for building a queue as a service built on top of MongoDB, covering concerns of durability, long running processes and distributed processing.

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
292
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Queue holds elements until they are required. Items in the queue are accessed from the head of the queue only – implied order.
  • Add capped collection
  • Transcript

    • 1. QUEUE IN THECLOUD WITHMONGODBMONGODB LA 2013NURI HALPERIN
    • 2. QUEUE
    • 3. USAGEOrdered executionBuffering consumer/producerWork distribution
    • 4. GOALS OF PROJECTLeverage Mongo • Reduce ops overhead by reusing infrastructure • Map queue semantics to Mongo’s strengthsReliable • Durable - support long running process • Resilient to machine failure • Narrow down window of failure/ data loss.Centralized, distributed: • Multiple producers • Multiple consumers
    • 5. ITERATION 0Capped collection – not the perfect choice • Tailing queue seems attractive, but… • Need external sync to avoid double-consume • Secondary indexes and updating are anti-patternRelaxing FIFO is OK • No guarantee that first-popped is first done • Multi-client is negated if they have to sync on execution order • Race condition for queue insertion has same effectConclusion: Project doesn’t use capped collection andrelaxes FIFO.
    • 6. PARANOID BY DESIGN Network dies Process dies DB dies Machine dies Poison letter Dead letter
    • 7. ITERATION 1db.q4foo.save({v:{f:1}})db.q4foo.findAndModify({query: {}, sort: {_id:1}, remove: true})Hot: quick and simpleNot: dead client, dead in transit, no trace
    • 8. ARE WE THERE YET? Network dies Process dies DB dies Machine dies Poison letter Dead letter
    • 9. QUEUE SEMANTICSLocal / Memory DistributedPush PutPop Get << visibility >><< exception >> Release << retry >> Delete << exception >>
    • 10. ITERATION 2 db.q4foo.save({v:{f:1}, dq: null}) db.q4foo.findAndModify( { query: { dq: null}, sort: {_id:1}, update:{ $set: { dq: later(60)}}}) … If processing was success => delete.. Hot: If client dies, item remains in queue. Data not lost. Not: index on _id less useful in high volume.
    • 11. ARE WE THERE YET? Network dies Process dies DB dies Machine dies Poison letter Dead letter
    • 12. ITERATION 3db.q4foo.save({v:{f:1}, dq: null, pc: 0})db.q4foo.findAndModify({ query: { dq: null, pc:{$lt:3}}, sort: {_id:1}, update:{$set:{dq:later(60)},$inc:{pc:1}}}) // consumedb.q4foo.findAndModify({ query: {_id:"..."}, update:{$set:{dq: null}}}) // releaseHot: An item can be retried automatically (pc) after released. Exhausted item remains in queue.Not: Not strict FIFO.
    • 13. ARE WE THERE? YES. Network dies Process dies DB dies Machine dies Poison letter Dead letter
    • 14. ITERATION 4Ensure your queue writes use applicable durability • db.q4foo.save() + getLastError(…) • db.q4foo.findAndModify () + getLastError(…)Replica sets for durability only. No capacity or speed gain.
    • 15. OTHER THOUGHTSCreate admin jobs to monitor queues: • Growth • Retries exhaustedConsider TTL risks (ex: client failure before calling Release())Consider idempotent operations when possibleDesign clients to back off pollingSeparate queue vs. extra “topic” fieldConsider dedicated DB for write-lock scopeCapped vs. regular collection – capped now can have _id, in-place update.
    • 16. Q&A Thank you! Nuri Halperinnuri@plusnconsulting.com

    ×