This document discusses building a distributed job queue in MongoDB. It begins by outlining the requirements of processing tasks across multiple servers. It then describes the design of storing messages and tasks in MongoDB collections, including different message and task states. It also covers techniques like composite tasks, reducing polling through tailable cursors, and optimizing performance through retry logic and oplog replay. The overall aim is to build an asynchronous and scalable way to process jobs in a distributed manner using MongoDB.

1. Chapman: Building a
Distributed Job Queue
in MongoDB
Rick Copeland
@rick446
rick@synapp.io

2. Getting to Know One
Another
Rick

3. Roadmap
Define the problem
Schema design & operations
Types of tasks
Reducing Polling
•
•
•
•

4. Requirements
Digital Ocean
Rackspace US
Rackspace UK
SMTP
Server
SMTP
Server
SMTP
Server
SMTP
Server
SMTP
Server
SMTP
Server
App
Server

5. Requirements
Group
check_smtp
Analyze
Results
Update
Reports
Pipeline

6. Requirements
MongoDB
(of course)

7. Basic Ideas
msg
Chapman Insecure
msg
msg
msg
msg
msg
Task
Task
Worker
Process

8. Job Queue Schema:
Message
{ _id: ObjectId(...),
task_id: ObjectId(...),
slot: 'run',
s: {
status: 'ready',
ts: ISODateTime(...),
q: 'chapman',
pri: 10,
w: '----------',
},
args: Binary(...),
kwargs: Binary(...),
send_args: Binary(...),
send_kwargs: Binary(...)
}
Task method to
be run
Destination
Task
Sched
SynchroMessage
arguments

9. Job Queue Schema:
TaskState
{ _id: ObjectId(...),
type: 'Group',
parent_id: ObjectId(...),
on_complete: ObjectId(...),
mq: [ObjectId(...), ...],
status: 'pending',
options: {
queue: 'chapman',
priority: 10,
immutable: false,
ignore_result: true,
}
result: Binary(...),
data: {...}
}
Python class
registered for
task
Parent task
(if any)
Message to be
sent on
completion

10. Message State: Naive
Approach
Reserve
Message
Try to
lock task
Un-
reserve
Message
Reserve
Message
Try to
lock task
Un-
reserve
Message
Reserve
Message
Try to
lock task
Un-
reserve
Message
Reserve
Message
Try to
lock task
Un-
reserve
Message
Reserve
Message
Try to
lock task
Un-
reserve
Message
Reserve
Message
Try to
lock task
Un-
reserve
Message
Reserve
Message
Try to
lock task
Un-
reserve
Message
Reserve
Message
Try to
lock task
Un-
reserve
Message
Reserve
Message
Try to
lock task
Un-
reserve
Message
Reserve
Message
Try to
lock task
Un-
reserve
Message
Reserve
Message
Try to
lock task
Un-
reserve
Message
Reserve
Message
Try to
lock task
Un-
reserve
Message
Reserve
Message
Try to
lock task
Un-
reserve
Message

11. Message States:
Reserve Message
findAndModify (‘ready’)
s.state => ‘q1’
s.w => worker_id
$push _id onto task’s mq field
If msg is first in mq, s.state =>
‘busy’
Start processing
•
•
•
•
•
•
ready
q1
busy
q2
next
pending

12. Message States:
Reserve Message
findAndModify (‘next’)
s.state => ‘busy’
s.w => worker_id
start processing
•
•
•
•
ready
q1
busy
q2
next
pending

13. Message States:
Retire Message
findAndModify TaskState
$pull message _id from ‘mq’
findAndModify new first message
in ‘mq’ if its s.state is in [‘q1’, ‘q2’]
s.state => ‘next’
•
•
•
•
ready
q1
busy
q2
next
pending

14. Task States
States mainly
advisory
success,failure
transitions trigger
on_complete
message
‘chained’ is a tail-
call optimization
•
•
•
pending
active
chainedfailuresuccess

15. Basic Tasks:
FunctionTask
Simplest task: run a function to
completion, set the result to the return
value
If a ‘Suspend’ exception is raised, move
the task to ‘chained’ status
Other exceptions set task to ‘failure’, save
•
•
•
@task(ignore_result=True, priority=50)
def track(event, user_id=None, **kwargs):
log.info('track(%s, %s...)', event, user_id)
# ...

16. Digression: Task
Chaining
Task state set to ‘chained’
New “Chain” task is created that will
Call the “next” task
When the “next” task completes, also
complete the “current” task
•
•
•
•
@task(ignore_result=True,
priority=50)
def function_task(*args, **kwargs):
# ...
Chain.call(some_other_task)

17. Composite Tasks
on_complete message for each subtask
with slot=retire_subtask, specifying subtask
position & the result of the subtask
Different composite tasks implement ‘run’
and ‘retire_subtask’ differently
•
•
task_state.update(
{ '_id': subtask_id },
{ $set: {
'parent_id': composite_id,
'data.composite_position': position,
'options.ignore_result': false
}}
)

18. Composite Task:
Pipeline
Run
Send a ‘run’ message to the subtask with
position=0
Retire_subtask(position, result)
Send a ‘run’ message with the previous
result to the subtask with position =
(position+1), OR retire the Pipeline if no
more tasks
•
•
•
•

19. Composite Task:
Group
Run
Send a ‘run’ message to all subtasks
Retire_subtask(position, result)
Decrement the num_waiting counter
If num_waiting is 0, retire the group
Collect subtask results, complete
•
•
•
•
•
•

20. Reducing Polling
Reserving messages is expensive
Use Pub/Sub system instead
Publish to the channel whenever a
message is ready to be handled
Each worker subscribes to the channel
•
•
•
•

21. Pub/Sub for MongoDB
Capped
Collection
Fixed size
Fast inserts
“Tailable”
cursors
•
•
•
Tailable
Cursor

22. Getting a Tailable
Cursor
def get_cursor(collection, topic_re, await_data=True):
options = { 'tailable': True }
if await_data:
options['await_data'] = True
cur = collection.find(
{ 'k': topic_re },
**options)
cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes
return cur
import re, time
while True:
cur = get_cursor(
db.capped_collection,
re.compile('^foo'),
await_data=True)
for msg in cur:
do_something(msg)
time.sleep(0.1)
Holds open cursor for a
while
Make curso
Don’t use indexes
Still some polling
producer, so do
too fast

23. Building in retry...
def get_cursor(collection, topic_re, last_id=-1, await_data=True):
options = { 'tailable': True }
spec = {
'id': { '$gt': last_id }, # only new messages
'k': topic_re }
if await_data:
options['await_data'] = True
cur = collection.find(spec, **options)
cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes
return cur
Integer au

24. Building auto-increment
class Sequence(object):
...
def next(self, sname, inc=1):
doc = self._db[self._name].find_and_modify(
query={'_id': sname},
update={'$inc': { 'value': inc } },
upsert=True,
new=True)
return doc['value']
Atomically $inc the
dedicated document

25. Ludicrous Speed
from pymongo.cursor import _QUERY_OPTIONS
def get_cursor(collection, topic_re, last_id=-1, await_data=True):
options = { 'tailable': True }
spec = {
'ts': { '$gt': last_id }, # only new messages
'k': topic_re }
if await_data:
options['await_data'] = True
cur = collection.find(spec, **options)
cur = cur.hint([('$natural', 1)]) # ensure we don't use any indexes
if await:
cur = cur.add_option(_QUERY_OPTIONS['oplog_replay'])
return cur
id ==> ts
Co-opt the
oplog_repla
option

26. Performance

27. Questions?
Rick Copeland
rick@synapp.io
@rick446