Your SlideShare is downloading. ×
0
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Building a High-Performance Distributed Task Queue on MongoDB
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Building a High-Performance Distributed Task Queue on MongoDB

3,963

Published on

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,963
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
24
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Chapman: Building aDistributed Job Queuein MongoDBRick Copeland@rick446rick@synapp.io
  • 2. Getting to Know OneAnotherRick
  • 3. RoadmapDefine the problemSchema design & operationsTypes of tasksReducing Polling••••
  • 4. RequirementsDigital OceanRackspace USRackspace UKSMTPServerSMTPServerSMTPServerSMTPServerSMTPServerSMTPServerAppServer
  • 5. RequirementsGroupcheck_smtpAnalyzeResultsUpdateReportsPipeline
  • 6. RequirementsMongoDB(of course)
  • 7. Basic IdeasmsgChapman InsecuremsgmsgmsgmsgmsgTaskTaskWorkerProcess
  • 8. Job Queue Schema:Message{ _id: ObjectId(...),task_id: ObjectId(...),slot: run,s: {status: ready,ts: ISODateTime(...),q: chapman,pri: 10,w: ----------,},args: Binary(...),kwargs: Binary(...),send_args: Binary(...),send_kwargs: Binary(...)}Task method tobe runDestinationTaskSchedSynchroMessagearguments
  • 9. Job Queue Schema:TaskState{ _id: ObjectId(...),type: Group,parent_id: ObjectId(...),on_complete: ObjectId(...),mq: [ObjectId(...), ...],status: pending,options: {queue: chapman,priority: 10,immutable: false,ignore_result: true,}result: Binary(...),data: {...}}Python classregistered fortaskParent task(if any)Message to besent oncompletion
  • 10. Message State: NaiveApproachReserveMessageTry tolock taskUn-reserveMessageReserveMessageTry tolock taskUn-reserveMessageReserveMessageTry tolock taskUn-reserveMessageReserveMessageTry tolock taskUn-reserveMessageReserveMessageTry tolock taskUn-reserveMessageReserveMessageTry tolock taskUn-reserveMessageReserveMessageTry tolock taskUn-reserveMessageReserveMessageTry tolock taskUn-reserveMessageReserveMessageTry tolock taskUn-reserveMessageReserveMessageTry tolock taskUn-reserveMessageReserveMessageTry tolock taskUn-reserveMessageReserveMessageTry tolock taskUn-reserveMessageReserveMessageTry tolock taskUn-reserveMessage
  • 11. Message States:Reserve MessagefindAndModify (‘ready’)s.state => ‘q1’s.w => worker_id$push _id onto task’s mq fieldIf msg is first in mq, s.state =>‘busy’Start processing••••••readyq1busyq2nextpending
  • 12. Message States:Reserve MessagefindAndModify (‘next’)s.state => ‘busy’s.w => worker_idstart processing••••readyq1busyq2nextpending
  • 13. Message States:Retire MessagefindAndModify TaskState$pull message _id from ‘mq’findAndModify new first messagein ‘mq’ if its s.state is in [‘q1’, ‘q2’]s.state => ‘next’••••readyq1busyq2nextpending
  • 14. Task StatesStates mainlyadvisorysuccess,failuretransitions triggeron_completemessage‘chained’ is a tail-call optimization•••pendingactivechainedfailuresuccess
  • 15. Basic Tasks:FunctionTaskSimplest task: run a function tocompletion, set the result to the returnvalueIf a ‘Suspend’ exception is raised, movethe task to ‘chained’ statusOther exceptions set task to ‘failure’, save•••@task(ignore_result=True, priority=50)def track(event, user_id=None, **kwargs):log.info(track(%s, %s...), event, user_id)# ...
  • 16. Digression: TaskChainingTask state set to ‘chained’New “Chain” task is created that willCall the “next” taskWhen the “next” task completes, alsocomplete the “current” task••••@task(ignore_result=True,priority=50)def function_task(*args, **kwargs):# ...Chain.call(some_other_task)
  • 17. Composite Taskson_complete message for each subtaskwith slot=retire_subtask, specifying subtaskposition & the result of the subtaskDifferent composite tasks implement ‘run’and ‘retire_subtask’ differently••task_state.update({ _id: subtask_id },{ $set: {parent_id: composite_id,data.composite_position: position,options.ignore_result: false}})
  • 18. Composite Task:PipelineRunSend a ‘run’ message to the subtask withposition=0Retire_subtask(position, result)Send a ‘run’ message with the previousresult to the subtask with position =(position+1), OR retire the Pipeline if nomore tasks••••
  • 19. Composite Task:GroupRunSend a ‘run’ message to all subtasksRetire_subtask(position, result)Decrement the num_waiting counterIf num_waiting is 0, retire the groupCollect subtask results, complete••••••
  • 20. Reducing PollingReserving messages is expensiveUse Pub/Sub system insteadPublish to the channel whenever amessage is ready to be handledEach worker subscribes to the channel••••
  • 21. Pub/Sub for MongoDBCappedCollectionFixed sizeFast inserts“Tailable”cursors•••TailableCursor
  • 22. Getting a TailableCursordef get_cursor(collection, topic_re, await_data=True):options = { tailable: True }if await_data:options[await_data] = Truecur = collection.find({ k: topic_re },**options)cur = cur.hint([($natural, 1)]) # ensure we dont use any indexesreturn curimport re, timewhile True:cur = get_cursor(db.capped_collection,re.compile(^foo),await_data=True)for msg in cur:do_something(msg)time.sleep(0.1)Holds open cursor for awhileMake cursoDon’t use indexesStill some pollingproducer, so dotoo fast
  • 23. Building in retry...def get_cursor(collection, topic_re, last_id=-1, await_data=True):options = { tailable: True }spec = {id: { $gt: last_id }, # only new messagesk: topic_re }if await_data:options[await_data] = Truecur = collection.find(spec, **options)cur = cur.hint([($natural, 1)]) # ensure we dont use any indexesreturn curInteger au
  • 24. Building auto-incrementclass Sequence(object):...def next(self, sname, inc=1):doc = self._db[self._name].find_and_modify(query={_id: sname},update={$inc: { value: inc } },upsert=True,new=True)return doc[value]Atomically $inc thededicated document
  • 25. Ludicrous Speedfrom pymongo.cursor import _QUERY_OPTIONSdef get_cursor(collection, topic_re, last_id=-1, await_data=True):options = { tailable: True }spec = {ts: { $gt: last_id }, # only new messagesk: topic_re }if await_data:options[await_data] = Truecur = collection.find(spec, **options)cur = cur.hint([($natural, 1)]) # ensure we dont use any indexesif await:cur = cur.add_option(_QUERY_OPTIONS[oplog_replay])return curid ==> tsCo-opt theoplog_replaoption
  • 26. Performance
  • 27. Questions?Rick Copelandrick@synapp.io@rick446

×