Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mongo à la Resque

Talk given at MongoFR 2011 in Paris, France.

http://lanyrd.com/2011/mongofr/speakers

  • Be the first to comment

  • Be the first to like this

Mongo à la Resque

  1. 1. Mongo à la Resque Nicolas Fouché http:// .com http://about.me/nfo
  2. 2. Say “rescue” or “resquioux” in French
  3. 3. Resque One man:
  4. 4. Resque One language:
  5. 5. Resque One dependency:
  6. 6. Redis Remote Dictionary Server “memcached on steroids” Initial release was March 2009
  7. 7. Redis Redis is an open source, advanced key- value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. http://redis.io
  8. 8. Redis In RAM Latency: 10ns VS 1ms = 1,000,000ms ≃ 5000 Gets/s ≃ 5000 Sets/s
  9. 9. Redis Virtual memory keys in memory values as needed in memory replication
  10. 10. Redis well written ANSI C and clients in all languages
  11. 11. Redis keep in mind the complexity of each operation
  12. 12. Resque parent / child fork process
  13. 13. Resque handles Unix signals => god / monit / bluepill friendly
  14. 14. Resque Sinatra web UI, known as resque-web
  15. 15. Queues Priority: $ QUEUES=high,low rake resque:work
  16. 16. Queues Consume all queues: $ QUEUES=* rake resque:work
  17. 17. Enqueue jobs Resque.enqueue(Archive, @repo.id, branch)
  18. 18. Workers class Archive
 @queue = :file_serve
 
 def self.perform(repo_id, branch = nil)
 repo = Repository.find(repo_id)
 repo.create_archive(branch || "master")
 end
 end
  19. 19. Persistence Jobs are stored as JSON {
 "class": "Archive",
 "args": [ 44, "masterbrew" ]
 }
  20. 20. Failure backends redis (for resque-web) hoptoad and others (like exceptional)
  21. 21. Don’t like Ruby ?
  22. 22. Alternate implementations https://github.com/defunkt/resque/wiki/alternate-implementations
  23. 23. resque-mongo
  24. 24. Translating queries atomic operations $set - set a particular value $unset - set a particular value $inc - increment a particular value by a certain amount $push - append a value to an array $pushAll - append several values to an array $pull - remove a value(s) from an existing array $pullAll - remove several value(s) from an existing array $bit - bitwis
  25. 25. Translating queries Push a job redis.sadd(:queues, queue) redis.rpush("queue:#{queue}", encode(item)) mongo << {:queue => queue, :item => item, :date => Time.now} mongo_queues.update( {:queue => queue }, {'$inc' => {:count => 1}} )
  26. 26. Translating queries Pop a job redis.lpop("queue:#{queue}") doc = mongo.find_and_modify( :query => {:queue => queue}, :sort => [[:date, 1]], :remove => true)
 mongo_queues.update( {:queue => queue }, {'$inc' => {:count => -1}})
  27. 27. Translating queries queue size redis.llen("queue:#{queue}").to_i mongo_queues.find_one(:queue => queue)['count']
  28. 28. Translating queries Peek a job list_range("queue:#{queue}", start, count) start, count = [start, count].map { |n| Integer(n) }
 res = mongo.find(:queue => queue).sort([:date, 1]).skip(start).limit(count).to_a 
 res.collect! { |doc| doc['item'] }
 count == 1 ? res.first : res

  29. 29. Translating queries Remove queue redis.srem(:queues, queue)
 redis.del("queue:#{queue}") mongo.remove({:queue => queue})
 mongo_queues.remove({:queue => queue})

  30. 30. Mongo to the rescue ? Sometimes you just know MongoDB better than Redis Most of query features of Redis can be translated in Mongo queries But for Redis addicts, MongoDB has some secrets
  31. 31. Used to MongoDB ?
  32. 32. Used to MongoDB ?[lol@cats ~]# bin/mongo
 MongoDB shell version: 1.6.4
 connecting to: test
 > show dbs
 admin
 local
 monque
 > use monque
 switched to db monque
 > db.getCollectionNames();
 [
 "delayed_queue",
 "failures",
 "job_groups",
 "monque",
 "queues",
 "schedules",
 "schedules_changed",
 "stats",
 "system.indexes",
 "workers"
 ]
 > db.workers.count();
 40
 > db.workers.findOne();
 {
 "_id" : ObjectId("4c863c6c89cb535954000001"),
 "started" : "Tue Sep 07 2010 13:21:48 GMT+0000 (UTC)",
 "worker" : "myserver:22868:facebook*"
 }

  33. 33. Used to MongoDB ? require 'rubygems'
 require 'mongo'
 include Mongo
 
 db = Connection.new.db('sample-db')
 coll = db.collection('test')
 
 coll.remove
 3.times do |i|
 coll.insert({'a' => i+1})
 end
 puts "There are #{coll.count()} records. Here they are:"
 coll.find().each { |doc| puts doc.inspect }

  34. 34. Really used to MongoDB ?
  35. 35. New features Go take a look in this JSON !
  36. 36. Search failed jobs
  37. 37. What’s happening right now ? Use the mongo console !
  38. 38. Listen on queues by prefix Queues prefix: $ QUEUES=email*,analytics* rake resque:work
  39. 39. Plugins resque-mongo-groups resque-mongo- scheduler
  40. 40. resque-mongo-groups by Catalin Bordianu, aka @omikronn
  41. 41. resque-mongo-groups Resque.enqueue(
 EatBreakfast,
 :with => ['eggs', 'bacon'],
 :group_id => 'meals:' + Time.now.utc.strftime('%Y%m%d')
 )

  42. 42. resque-mongo-groups class EatBreakfast
 extend Resque::Plugins::Groups::TrackedJob
 end

  43. 43. resque-mongo-groups def self.perform
 # ...
 self.class.atomic_op_on_complete = {
 '$inc' => { 'eggs' => 1, 'bacon' => 1 }
 }
 # ...
 end
  44. 44. resque-mongo-groups Resque.group_stats("meals:20110316")
 {
 '_id' => 'meals:20110316', 'total' => 4, 'delayed' => 2, 'completed' => 3,
 'failed' => 1, 'exceptions' => ["#<TooMuchEggs: beware !>"]
 'custom' => {:eggs => 12, :bacon => 1, :potatoes => 3}
 }

  45. 45. resque-mongo-groups gem "resque-mongo-groups"
  46. 46. resque-mongo-scheduler Resque.enqueue_at(
 1.hour.from_now,
 Eat,
 :what => 'leftovers'
 )
  47. 47. resque-mongo-scheduler breakfast:
 cron: "0 8 * * *"
 class: Eat
 args: "bacon"
 description: Breakfast
  48. 48. gem "resque-mongo-scheduler", :require => "resque-scheduler" resque-mongo-scheduler
  49. 49. resque-mongo ⚡ resque You cannot use resque and resque-mongo in the same project
  50. 50. resque-mongo ♥ resque migration to resque-mongo => free
  51. 51. resque-mongo ♥ resque Still have to migrate from to
  52. 52. resque-mongo ♥ resque easy to merge the last changes from resque to resque-mongo easy to port existing plugins
  53. 53. Get the gem ! gem install nfo-resque-mongo The maintainer still did not answer to my ownership requests :(
  54. 54. and use it with Bundler gem "nfo-resque-mongo", :require => "resque"
  55. 55. Get the source! https://github.com/nfo/resque- mongo
  56. 56. Help needed ! Add exclusive features to resque-web Migrate existing plugins Create new plugins Promote the gem Or simply use the gem, you ‘ll surely send pull requests
  57. 57. That’s all folks Credits for Flickr images: acezebragirl4j, kiwitime, fulbert05, josh_exell, benheine, tudacee

×