The document discusses replacing RabbitMQ with MongoDB for queuing purposes. It outlines some of the key features of queuing like consumers, atomicity, speed and garbage collection and how MongoDB implements these features using findAndModify commands and daemon processes. It also covers scaling, fault tolerance using replica sets, and some potential problems with MongoDB like durability and global locking that may make RabbitMQ a better choice for queuing.

1. Replacing RabbitMQ with MongoDB

2. David Mytton
Woop Japan!

4. Queuing: Uses
www.flickr.com/photos/triplexpresso/496995086/

5. Queuing: Uses
• Background processing
www.flickr.com/photos/triplexpresso/496995086/

6. Queuing: Uses
• Background processing
• Sending notifications
www.flickr.com/photos/triplexpresso/496995086/

7. Queuing: Uses
• Background processing
• Sending notifications
• Event streaming
www.flickr.com/photos/triplexpresso/496995086/

8. Queuing: Features

9. Queuing: Features
• Consumers

10. Queuing: Features
• Consumers
• Atomic

11. Queuing: Features
• Consumers
• Atomic
• Speed

12. Queuing: Features
• Consumers
• Atomic
• Speed
• GC

13. Queuing: Features
•Consumers

14. Queuing: Features
•Consumers
MongoDB RabbitMQ
Mongo Wire
AMQP
Protocol

15. Queuing: Features
•Atomic
en.wikipedia.org/wiki/State_of_matter

16. Queuing: Features
•Atomic
MongoDB RabbitMQ
findAndModify consume/ack
en.wikipedia.org/wiki/State_of_matter

17. Queuing: Features
•Speed

18. Queuing: Features
•GC

19. Queuing: Features
•GC
MongoDB RabbitMQ
☹ consume/ack

20. It’s a little different.

21. Implementation
• Consumers

22. Implementation
• Consumers
db.runCommand(
{ findAndModify : <collection>, <options> } )

23. Implementation
• Consumers
db.runCommand(
{ findAndModify : <collection>, <options> } )
query: filter (WHERE)
{ query: { hats: 5 } }

24. Implementation
• Consumers
db.runCommand(
{ findAndModify : <collection>, <options> } )
sort: selects the first one on multi-match
{ sort: { added: -1 } }

25. Implementation
• Consumers
db.runCommand(
{ findAndModify : <collection>, <options> } )
update: modifier object
{ update: { $set: {inProg: true, start: new
Date()} } }

26. Implementation
• Consumers
db.runCommand(
{ findAndModify : <collection>, <options> } )
remove: true = deletes on return
new: true = returns modified object
fields: return specific fields
upsert: true = create object if !exists()

27. Implementation
• Consumers

28. Implementation
• Consumers
def run(self):
import time, sched
sc = sched.scheduler(time.time, time.sleep)
self.poll(sc)
sc.run()
def poll(self, sc):
doStuff()
sc.enter(10, 1, self.poll, (sc,))

29. Implementation
• Consumers

30. Implementation
• Consumers
from daemon import Daemon
class pantalaimon(Daemon):
def run(self): # Do stuff
pineMarten = pantalaimon('/path/to/pid.pid')
pineMarten.start()
https://github.com/boxedice/python-daemon

31. Implementation
• Consumers
from daemon import Daemon
class pantalaimon(Daemon):
def run(self):
import time, sched
sc = sched.scheduler(time.time, time.sleep)
self.poll(sc)
sc.run()
def poll(self, sc):
doStuff()
sc.enter(10, 1, self.poll, (sc,))
https://github.com/boxedice/python-daemon

32. Implementation
• GC

33. Implementation
• GC
now = datetime.datetime.now()
difference = datetime.timedelta(seconds=10)
timeout = now - difference
queue.find({'inProg' : True, 'start' :
{'$lte' : timeout} })

34. Fault tolerance
Replica sets

35. Fault tolerance
Replica sets
Basic automatic failover

36. Fault tolerance
Replica sets
Basic automatic failover
Writing to n slaves

37. Fault tolerance
Replica sets
Basic automatic failover
Writing to n slaves
Data centre awareness

38. Scaling
Capped Collections

39. Scaling
Capped Collections
Extremely fast

40. Scaling
Capped Collections
Extremely fast
Maintain insertion order (FIFO)

41. Scaling
Capped Collections
Extremely fast
Maintain insertion order (FIFO)
Don’t have to remove()

42. Scaling
Sharding

43. Scaling
Sharding
findAndModify() locks

44. Scaling
Sharding
findAndModify() locks
Scaling writes across machines

45. Scaling
Sharding

46. Scaling
Sharding
Not with capped collections

47. Problems / Mongo quirks

48. Problems / Mongo quirks
Meta

49. Problems / Mongo quirks
Meta
Durability

50. Problems / Mongo quirks
Meta
Durability
Global lock

51. Stick with RabbitMQ?

52. Stick with RabbitMQ?
QoS

53. Stick with RabbitMQ?
QoS
AMQP

54. Stick with RabbitMQ?
QoS
AMQP
Throttling

55. David Mytton
@davidmytton
david@boxedice.com
www.mongomonitor.com
Woop Japan!