Resque is a Redis-backed Ruby library for creating background jobs. Resque-mongo is an alternative implementation that uses MongoDB instead of Redis. It provides the same basic functionality as Resque but stores job queue data and metadata in MongoDB collections instead of Redis lists and hashes. Resque-mongo aims to be API compatible with Resque to simplify migration. It includes plugins for job grouping and scheduling that take advantage of MongoDB's features.

1. Mongo à la Resque
Nicolas Fouché
http:// .com
http://about.me/nfo

2. Say “rescue”
or “resquioux”
in French

3. Resque
One man:

4. Resque
One language:

5. Resque
One dependency:

6. Redis
Remote Dictionary Server
“memcached on steroids”
Initial release was March 2009

7. Redis
Redis is an open source, advanced key-
value store. It is often referred to as a data
structure server since keys can contain
strings, hashes, lists, sets and sorted sets.
http://redis.io

8. Redis
In RAM
Latency: 10ns VS 1ms =
1,000,000ms
≃ 5000 Gets/s
≃ 5000 Sets/s

9. Redis
Virtual memory
keys in memory
values as needed in memory
replication

10. Redis
well written ANSI C
and clients in all languages

11. Redis
keep in mind the complexity of each
operation

12. Resque
parent / child fork process

13. Resque
handles Unix signals
=> god / monit / bluepill friendly

14. Resque
Sinatra web UI, known as resque-web

15. Queues
Priority:
$ QUEUES=high,low rake resque:work

16. Queues
Consume all queues:
$ QUEUES=* rake resque:work

17. Enqueue jobs
Resque.enqueue(Archive, @repo.id, branch)

18. Workers
class Archive
@queue = :file_serve
def self.perform(repo_id, branch = nil)
repo = Repository.find(repo_id)
repo.create_archive(branch || "master")
end
end

19. Persistence
Jobs are stored as JSON
{
"class": "Archive",
"args": [ 44, "masterbrew" ]
}

20. Failure backends
redis (for resque-web)
hoptoad
and others (like exceptional)

21. Don’t like Ruby ?

22. Alternate implementations
https://github.com/defunkt/resque/wiki/alternate-implementations

23. resque-mongo

24. Translating queries
atomic operations
$set - set a particular value
$unset - set a particular value
$inc - increment a particular value by a certain amount
$push - append a value to an array
$pushAll - append several values to an array
$pull - remove a value(s) from an existing array
$pullAll - remove several value(s) from an existing array
$bit - bitwis

25. Translating queries
Push a job
redis.sadd(:queues, queue)
redis.rpush("queue:#{queue}", encode(item))
mongo << {:queue => queue, :item => item,
:date => Time.now}
mongo_queues.update(
{:queue => queue },
{'$inc' => {:count => 1}}
)

26. Translating queries
Pop a job
redis.lpop("queue:#{queue}")
doc = mongo.find_and_modify(
:query => {:queue => queue},
:sort => [[:date, 1]],
:remove => true)
mongo_queues.update(
{:queue => queue },
{'$inc' => {:count => -1}})

27. Translating queries
queue size
redis.llen("queue:#{queue}").to_i
mongo_queues.find_one(:queue => queue)['count']

28. Translating queries
Peek a job
list_range("queue:#{queue}", start, count)
start, count = [start, count].map { |n| Integer(n)
}
res = mongo.find(:queue => queue).sort([:date,
1]).skip(start).limit(count).to_a
res.collect! { |doc| doc['item'] }
count == 1 ? res.first : res

29. Translating queries
Remove queue
redis.srem(:queues, queue)
redis.del("queue:#{queue}")
mongo.remove({:queue => queue})
mongo_queues.remove({:queue => queue})

30. Mongo to the rescue ?
Sometimes you just know MongoDB
better than Redis
Most of query features of Redis can
be translated in Mongo queries
But for Redis addicts, MongoDB has
some secrets

31. Used to MongoDB ?

32. Used to MongoDB ?[lol@cats ~]# bin/mongo
MongoDB shell version: 1.6.4
connecting to: test
> show dbs
admin
local
monque
> use monque
switched to db monque
> db.getCollectionNames();
[
"delayed_queue",
"failures",
"job_groups",
"monque",
"queues",
"schedules",
"schedules_changed",
"stats",
"system.indexes",
"workers"
]
> db.workers.count();
40
> db.workers.findOne();
{
"_id" : ObjectId("4c863c6c89cb535954000001"),
"started" : "Tue Sep 07 2010 13:21:48 GMT+0000 (UTC)",
"worker" : "myserver:22868:facebook*"
}

33. Used to MongoDB ?
require 'rubygems'
require 'mongo'
include Mongo
db = Connection.new.db('sample-db')
coll = db.collection('test')
coll.remove
3.times do |i|
coll.insert({'a' => i+1})
end
puts "There are #{coll.count()} records. Here they are:"
coll.find().each { |doc| puts doc.inspect }

34. Really used to MongoDB ?

35. New features
Go take a look in this
JSON !

36. Search failed jobs

37. What’s happening right now ?
Use the mongo console !

38. Listen on queues by prefix
Queues prefix:
$ QUEUES=email*,analytics* rake
resque:work

39. Plugins
resque-mongo-groups
resque-mongo-
scheduler

40. resque-mongo-groups
by Catalin Bordianu, aka @omikronn

41. resque-mongo-groups
Resque.enqueue(
EatBreakfast,
:with => ['eggs', 'bacon'],
:group_id => 'meals:' + Time.now.utc.strftime('%Y%m%d')
)

42. resque-mongo-groups
class EatBreakfast
extend Resque::Plugins::Groups::TrackedJob
end

43. resque-mongo-groups
def self.perform
# ...
self.class.atomic_op_on_complete = {
'$inc' => { 'eggs' => 1, 'bacon' => 1 }
}
# ...
end

44. resque-mongo-groups
Resque.group_stats("meals:20110316")
{
'_id' => 'meals:20110316', 'total' => 4, 'delayed' => 2, 'completed' => 3,
'failed' => 1, 'exceptions' => ["#<TooMuchEggs: beware !>"]
'custom' => {:eggs => 12, :bacon => 1, :potatoes => 3}
}

45. resque-mongo-groups
gem "resque-mongo-groups"

46. resque-mongo-scheduler
Resque.enqueue_at(
1.hour.from_now,
Eat,
:what => 'leftovers'
)

47. resque-mongo-scheduler
breakfast:
cron: "0 8 * * *"
class: Eat
args: "bacon"
description: Breakfast

48. gem "resque-mongo-scheduler", :require => "resque-scheduler"
resque-mongo-scheduler

49. resque-mongo ⚡ resque
You cannot use resque and resque-mongo in
the same project

50. resque-mongo ♥ resque
migration to resque-mongo => free

51. resque-mongo ♥ resque
Still have to migrate from
to

52. resque-mongo ♥ resque
easy to merge the last
changes from resque
to resque-mongo
easy to port existing
plugins

53. Get the gem !
gem install nfo-resque-mongo
The maintainer still did not answer
to my ownership requests :(

54. and use it with Bundler
gem "nfo-resque-mongo", :require => "resque"

55. Get the source!
https://github.com/nfo/resque-
mongo

56. Help needed !
Add exclusive features to resque-web
Migrate existing plugins
Create new plugins
Promote the gem
Or simply use the gem, you ‘ll surely
send pull requests

57. That’s all folks
Credits for Flickr images: acezebragirl4j,
kiwitime, fulbert05, josh_exell, benheine,
tudacee