ResqueBackground processingSystem administration on a multi-        instances setup      Experience feedback            Pl...
My project : bookandgolf.com
bookandgolf.comAt launch :  • more than 60 golfs to synchronise  • 4 different APIs  • More than 350 000 slots (starts) to...
nginx          resque-web:web             redis-server:db       worker 1                        Current       worker 2:app...
Resque job classclass MyKikoololSleepJob  @queue = :medium  def self.perform(sleep_duration)    sleep sleep_duration  ende...
Resque workersrake environment resque:work QUEUE=mediumrails@www:~$ ps aux | grep resquerails  8950  sh -c cd /home/rails/...
Resque workers Unix signals• QUIT - Wait for child to finish processing then exit• TERM / INT - Immediately kill child then...
Resque-web (Sinatra webapp)
God setup - my way!Versioned god config files by instance name, ie :•   god / app.god•   god / web.god•   god / db.god•   ...
God config# ResqueGod.watch do |w|  w.env = { RAILS_ROOT => rails_root,              RAILS_ENV  => rails_env }  w.name    ...
Workers graceful restartRake task to restart workers, the "graceful"waynamespace :resque do  task :restart_workers => :env...
Workers graceful restartnamespace :resque do  task :restart_workers, :roles => [:app, :db] do    rake "resque:restart_work...
Resque-web - Final setup• Launched externally with nginx reverse proxy (easy to add  http basic auth)• External Ruby confi...
Resque-web God monitoring%w{5678}.each do |port|  God.watch do |w|    w.env = { RAILS_ROOT => rails_root,              RAI...
ActiveRecord - timeout / stale :(Fast workaround...   ...   ActiveRecord::Base.connection.reconnect!   ...
Resque plugins• defunkt / resque-lockPrevent two workers from working on the sameJob class with same arguments
Resque plugins• jayniz / resque-lonerTIP :Do not forget to :  require "resque-loner"in your Resque-web config.rb file! or....
Other plugins...Look @ github.com/defunkt/resqueWiki!And remember that with a goodsystem architecture, Resque is just...
Background processing with Resque
Background processing with Resque
Background processing with Resque
Background processing with Resque
Upcoming SlideShare
Loading in …5
×

Background processing with Resque

14,846 views

Published on

System administration on a multi-instances architecture, experience feedback, plugins I like

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
14,846
On SlideShare
0
From Embeds
0
Number of Embeds
9,216
Actions
Shares
0
Downloads
32
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Background processing with Resque

  1. 1. ResqueBackground processingSystem administration on a multi- instances setup Experience feedback Plugins Nicolas Blanco Novapost
  2. 2. My project : bookandgolf.com
  3. 3. bookandgolf.comAt launch : • more than 60 golfs to synchronise • 4 different APIs • More than 350 000 slots (starts) to synchronize every 2 hoursAt first we tried to do the synchronization with cron tasks... but...
  4. 4. nginx resque-web:web redis-server:db worker 1 Current worker 2:app worker 3 architecture worker 4
  5. 5. Resque job classclass MyKikoololSleepJob  @queue = :medium  def self.perform(sleep_duration)    sleep sleep_duration  endendResque.enqueue(MyKikoololSleepJob, 10)
  6. 6. Resque workersrake environment resque:work QUEUE=mediumrails@www:~$ ps aux | grep resquerails  8950  sh -c cd /home/rails/www/socianalytics/current &&rake environment resque:work QUEUE=high,medium,low...rails  8951  resque-1.9.10: Forked 9105 at 1302095560               rails  9105  resque-1.9.10: Processing medium since 1302095560      
  7. 7. Resque workers Unix signals• QUIT - Wait for child to finish processing then exit• TERM / INT - Immediately kill child then exit• USR1 - Immediately kill child but dont exit• USR2 - Dont start to process any new jobs• CONT - Start to process new jobs again after a USR2
  8. 8. Resque-web (Sinatra webapp)
  9. 9. God setup - my way!Versioned god config files by instance name, ie :• god / app.god• god / web.god• god / db.god• ...God init.d file found on gist.github.com :)god.conf :GOD_CONFIG=/home/rails/www/bookgolf/current/god/db.godGOD_COMMAND="sudo -u rails /usr/local/bin/god"GOD_LOG=/home/rails/log/god.log
  10. 10. God config# ResqueGod.watch do |w|  w.env = { RAILS_ROOT => rails_root,              RAILS_ENV  => rails_env }  w.name          = "resque-worker"  w.interval      = 30.seconds  w.start         = "cd #{rails_root} && rakeenvironment resque:work QUEUE=high,medium,low"  w.start_grace   = 10.seconds    ...
  11. 11. Workers graceful restartRake task to restart workers, the "graceful"waynamespace :resque do  task :restart_workers => :environment do    pids = Array.new        Resque.workers.each do |worker|      pids << worker.to_s.split(/:/).second ifworker.to_s.include?(Settings.resque.localhost_name)    end        if pids.size > 0      system("kill -QUIT #{pids.join( )}")    end        system("rm /home/rails/.god/pids/resque-worker*.pid")  endend
  12. 12. Workers graceful restartnamespace :resque do  task :restart_workers, :roles => [:app, :db] do    rake "resque:restart_workers"  endend
  13. 13. Resque-web - Final setup• Launched externally with nginx reverse proxy (easy to add http basic auth)• External Ruby config file to change Redis setup, load plugins, etc.     resque-web config/resque-web.rb
  14. 14. Resque-web God monitoring%w{5678}.each do |port|  God.watch do |w|    w.env = { RAILS_ROOT => rails_root,              RAILS_ENV  => rails_env }    w.uid       = "rails"    w.name      = "resque-web"    w.interval  = 30.seconds    w.start     = "cd #{rails_root} && resque-web config/resque-web.rb"    w.start_grace   = 15.seconds    w.start_if do |start|      start.condition(:process_running) do |c|          c.interval = 5.seconds          c.running = false      end    end  endend
  15. 15. ActiveRecord - timeout / stale :(Fast workaround...   ...   ActiveRecord::Base.connection.reconnect!   ...
  16. 16. Resque plugins• defunkt / resque-lockPrevent two workers from working on the sameJob class with same arguments
  17. 17. Resque plugins• jayniz / resque-lonerTIP :Do not forget to :  require "resque-loner"in your Resque-web config.rb file! or... FAIL (cleared jobs wontbe relaunched again!)BAD :Need inheritance in job classes :(... But a fork exists thatdoesnt need it.
  18. 18. Other plugins...Look @ github.com/defunkt/resqueWiki!And remember that with a goodsystem architecture, Resque is just...

×