Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Mongo à la Resque
Nicolas Fouché
http:// .com
http://about.me/nfo
Say “rescue”
or “resquioux”
in French
Resque
One man:
Resque
One language:
Resque
One dependency:
Redis
Remote Dictionary Server
“memcached on steroids”
Initial release was March 2009
Redis
Redis is an open source, advanced key-
value store. It is often referred to as a data
structure server since keys ca...
Redis
In RAM
Latency: 10ns VS 1ms =
1,000,000ms
≃ 5000 Gets/s
≃ 5000 Sets/s
Redis
Virtual memory
keys in memory
values as needed in memory
replication
Redis
well written ANSI C
and clients in all languages
Redis
keep in mind the complexity of each
operation
Resque
parent / child fork process
Resque
handles Unix signals
=> god / monit / bluepill friendly
Resque
Sinatra web UI, known as resque-web
Queues
Priority:
$ QUEUES=high,low rake resque:work
Queues
Consume all queues:
$ QUEUES=* rake resque:work
Enqueue jobs
Resque.enqueue(Archive, @repo.id, branch)
Workers
class Archive

@queue = :file_serve



def self.perform(repo_id, branch = nil)

repo = Repository.find(repo_id)

r...
Persistence
Jobs are stored as JSON
{

"class": "Archive",

"args": [ 44, "masterbrew" ]

}
Failure backends
redis (for resque-web)
hoptoad
and others (like exceptional)
Don’t like Ruby ?
Alternate implementations
https://github.com/defunkt/resque/wiki/alternate-implementations
resque-mongo
Translating queries
atomic operations
$set - set a particular value
$unset - set a particular value
$inc - increment a par...
Translating queries
Push a job
redis.sadd(:queues, queue)
redis.rpush("queue:#{queue}", encode(item))
mongo << {:queue => ...
Translating queries
Pop a job
redis.lpop("queue:#{queue}")
doc = mongo.find_and_modify(
:query => {:queue => queue},
:sort...
Translating queries
queue size
redis.llen("queue:#{queue}").to_i
mongo_queues.find_one(:queue => queue)['count']
Translating queries
Peek a job
list_range("queue:#{queue}", start, count)
start, count = [start, count].map { |n| Integer(...
Translating queries
Remove queue
redis.srem(:queues, queue)

redis.del("queue:#{queue}")
mongo.remove({:queue => queue})

...
Mongo to the rescue ?
Sometimes you just know MongoDB
better than Redis
Most of query features of Redis can
be translated ...
Used to MongoDB ?
Used to MongoDB ?[lol@cats ~]# bin/mongo

MongoDB shell version: 1.6.4

connecting to: test

> show dbs

admin

local

mon...
Used to MongoDB ?
require 'rubygems'

require 'mongo'

include Mongo



db = Connection.new.db('sample-db')

coll = db.col...
Really used to MongoDB ?
New features
Go take a look in this
JSON !
Search failed jobs
What’s happening right now ?
Use the mongo console !
Listen on queues by prefix
Queues prefix:
$ QUEUES=email*,analytics* rake
resque:work
Plugins
resque-mongo-groups
resque-mongo-
scheduler
resque-mongo-groups
by Catalin Bordianu, aka @omikronn
resque-mongo-groups
Resque.enqueue(

EatBreakfast,

:with => ['eggs', 'bacon'],

:group_id => 'meals:' + Time.now.utc.strf...
resque-mongo-groups
class EatBreakfast

extend Resque::Plugins::Groups::TrackedJob

end

resque-mongo-groups
def self.perform

# ...

self.class.atomic_op_on_complete = {

'$inc' => { 'eggs' => 1, 'bacon' => 1 }...
resque-mongo-groups
Resque.group_stats("meals:20110316")

{

'_id' => 'meals:20110316', 'total' => 4, 'delayed' => 2, 'com...
resque-mongo-groups
gem "resque-mongo-groups"
resque-mongo-scheduler
Resque.enqueue_at(

1.hour.from_now,

Eat,

:what => 'leftovers'

)
resque-mongo-scheduler
breakfast:

cron: "0 8 * * *"

class: Eat

args: "bacon"

description: Breakfast
gem "resque-mongo-scheduler", :require => "resque-scheduler"
resque-mongo-scheduler
resque-mongo ⚡ resque
You cannot use resque and resque-mongo in
the same project
resque-mongo ♥ resque
migration to resque-mongo => free
resque-mongo ♥ resque
Still have to migrate from
to
resque-mongo ♥ resque
easy to merge the last
changes from resque
to resque-mongo
easy to port existing
plugins
Get the gem !
gem install nfo-resque-mongo
The maintainer still did not answer
to my ownership requests :(
and use it with Bundler
gem "nfo-resque-mongo", :require => "resque"
Get the source!
https://github.com/nfo/resque-
mongo
Help needed !
Add exclusive features to resque-web
Migrate existing plugins
Create new plugins
Promote the gem
Or simply u...
That’s all folks
Credits for Flickr images: acezebragirl4j,
kiwitime, fulbert05, josh_exell, benheine,
tudacee
Upcoming SlideShare
Loading in …5
×

Mongo à la Resque

352 views

Published on

Talk given at MongoFR 2011 in Paris, France.

http://lanyrd.com/2011/mongofr/speakers

Published in: Software
  • Be the first to comment

  • Be the first to like this

Mongo à la Resque

  1. 1. Mongo à la Resque Nicolas Fouché http:// .com http://about.me/nfo
  2. 2. Say “rescue” or “resquioux” in French
  3. 3. Resque One man:
  4. 4. Resque One language:
  5. 5. Resque One dependency:
  6. 6. Redis Remote Dictionary Server “memcached on steroids” Initial release was March 2009
  7. 7. Redis Redis is an open source, advanced key- value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. http://redis.io
  8. 8. Redis In RAM Latency: 10ns VS 1ms = 1,000,000ms ≃ 5000 Gets/s ≃ 5000 Sets/s
  9. 9. Redis Virtual memory keys in memory values as needed in memory replication
  10. 10. Redis well written ANSI C and clients in all languages
  11. 11. Redis keep in mind the complexity of each operation
  12. 12. Resque parent / child fork process
  13. 13. Resque handles Unix signals => god / monit / bluepill friendly
  14. 14. Resque Sinatra web UI, known as resque-web
  15. 15. Queues Priority: $ QUEUES=high,low rake resque:work
  16. 16. Queues Consume all queues: $ QUEUES=* rake resque:work
  17. 17. Enqueue jobs Resque.enqueue(Archive, @repo.id, branch)
  18. 18. Workers class Archive
 @queue = :file_serve
 
 def self.perform(repo_id, branch = nil)
 repo = Repository.find(repo_id)
 repo.create_archive(branch || "master")
 end
 end
  19. 19. Persistence Jobs are stored as JSON {
 "class": "Archive",
 "args": [ 44, "masterbrew" ]
 }
  20. 20. Failure backends redis (for resque-web) hoptoad and others (like exceptional)
  21. 21. Don’t like Ruby ?
  22. 22. Alternate implementations https://github.com/defunkt/resque/wiki/alternate-implementations
  23. 23. resque-mongo
  24. 24. Translating queries atomic operations $set - set a particular value $unset - set a particular value $inc - increment a particular value by a certain amount $push - append a value to an array $pushAll - append several values to an array $pull - remove a value(s) from an existing array $pullAll - remove several value(s) from an existing array $bit - bitwis
  25. 25. Translating queries Push a job redis.sadd(:queues, queue) redis.rpush("queue:#{queue}", encode(item)) mongo << {:queue => queue, :item => item, :date => Time.now} mongo_queues.update( {:queue => queue }, {'$inc' => {:count => 1}} )
  26. 26. Translating queries Pop a job redis.lpop("queue:#{queue}") doc = mongo.find_and_modify( :query => {:queue => queue}, :sort => [[:date, 1]], :remove => true)
 mongo_queues.update( {:queue => queue }, {'$inc' => {:count => -1}})
  27. 27. Translating queries queue size redis.llen("queue:#{queue}").to_i mongo_queues.find_one(:queue => queue)['count']
  28. 28. Translating queries Peek a job list_range("queue:#{queue}", start, count) start, count = [start, count].map { |n| Integer(n) }
 res = mongo.find(:queue => queue).sort([:date, 1]).skip(start).limit(count).to_a 
 res.collect! { |doc| doc['item'] }
 count == 1 ? res.first : res

  29. 29. Translating queries Remove queue redis.srem(:queues, queue)
 redis.del("queue:#{queue}") mongo.remove({:queue => queue})
 mongo_queues.remove({:queue => queue})

  30. 30. Mongo to the rescue ? Sometimes you just know MongoDB better than Redis Most of query features of Redis can be translated in Mongo queries But for Redis addicts, MongoDB has some secrets
  31. 31. Used to MongoDB ?
  32. 32. Used to MongoDB ?[lol@cats ~]# bin/mongo
 MongoDB shell version: 1.6.4
 connecting to: test
 > show dbs
 admin
 local
 monque
 > use monque
 switched to db monque
 > db.getCollectionNames();
 [
 "delayed_queue",
 "failures",
 "job_groups",
 "monque",
 "queues",
 "schedules",
 "schedules_changed",
 "stats",
 "system.indexes",
 "workers"
 ]
 > db.workers.count();
 40
 > db.workers.findOne();
 {
 "_id" : ObjectId("4c863c6c89cb535954000001"),
 "started" : "Tue Sep 07 2010 13:21:48 GMT+0000 (UTC)",
 "worker" : "myserver:22868:facebook*"
 }

  33. 33. Used to MongoDB ? require 'rubygems'
 require 'mongo'
 include Mongo
 
 db = Connection.new.db('sample-db')
 coll = db.collection('test')
 
 coll.remove
 3.times do |i|
 coll.insert({'a' => i+1})
 end
 puts "There are #{coll.count()} records. Here they are:"
 coll.find().each { |doc| puts doc.inspect }

  34. 34. Really used to MongoDB ?
  35. 35. New features Go take a look in this JSON !
  36. 36. Search failed jobs
  37. 37. What’s happening right now ? Use the mongo console !
  38. 38. Listen on queues by prefix Queues prefix: $ QUEUES=email*,analytics* rake resque:work
  39. 39. Plugins resque-mongo-groups resque-mongo- scheduler
  40. 40. resque-mongo-groups by Catalin Bordianu, aka @omikronn
  41. 41. resque-mongo-groups Resque.enqueue(
 EatBreakfast,
 :with => ['eggs', 'bacon'],
 :group_id => 'meals:' + Time.now.utc.strftime('%Y%m%d')
 )

  42. 42. resque-mongo-groups class EatBreakfast
 extend Resque::Plugins::Groups::TrackedJob
 end

  43. 43. resque-mongo-groups def self.perform
 # ...
 self.class.atomic_op_on_complete = {
 '$inc' => { 'eggs' => 1, 'bacon' => 1 }
 }
 # ...
 end
  44. 44. resque-mongo-groups Resque.group_stats("meals:20110316")
 {
 '_id' => 'meals:20110316', 'total' => 4, 'delayed' => 2, 'completed' => 3,
 'failed' => 1, 'exceptions' => ["#<TooMuchEggs: beware !>"]
 'custom' => {:eggs => 12, :bacon => 1, :potatoes => 3}
 }

  45. 45. resque-mongo-groups gem "resque-mongo-groups"
  46. 46. resque-mongo-scheduler Resque.enqueue_at(
 1.hour.from_now,
 Eat,
 :what => 'leftovers'
 )
  47. 47. resque-mongo-scheduler breakfast:
 cron: "0 8 * * *"
 class: Eat
 args: "bacon"
 description: Breakfast
  48. 48. gem "resque-mongo-scheduler", :require => "resque-scheduler" resque-mongo-scheduler
  49. 49. resque-mongo ⚡ resque You cannot use resque and resque-mongo in the same project
  50. 50. resque-mongo ♥ resque migration to resque-mongo => free
  51. 51. resque-mongo ♥ resque Still have to migrate from to
  52. 52. resque-mongo ♥ resque easy to merge the last changes from resque to resque-mongo easy to port existing plugins
  53. 53. Get the gem ! gem install nfo-resque-mongo The maintainer still did not answer to my ownership requests :(
  54. 54. and use it with Bundler gem "nfo-resque-mongo", :require => "resque"
  55. 55. Get the source! https://github.com/nfo/resque- mongo
  56. 56. Help needed ! Add exclusive features to resque-web Migrate existing plugins Create new plugins Promote the gem Or simply use the gem, you ‘ll surely send pull requests
  57. 57. That’s all folks Credits for Flickr images: acezebragirl4j, kiwitime, fulbert05, josh_exell, benheine, tudacee

×