Background processing with Resque

ResqueBackground processingSystem administration on a multi- instances setup Experience feedback Plugins Nicolas Blanco Novapost

My project : bookandgolf.com

bookandgolf.comAt launch : • more than 60 golfs to synchronise • 4 different APIs • More than 350 000 slots (starts) to synchronize every 2 hoursAt first we tried to do the synchronization with cron tasks... but...

nginx resque-web:web redis-server:db worker 1 Current worker 2:app worker 3 architecture worker 4

Resque job classclass MyKikoololSleepJob @queue = :medium def self.perform(sleep_duration) sleep sleep_duration endendResque.enqueue(MyKikoololSleepJob, 10)

Resque workersrake environment resque:work QUEUE=mediumrails@www:~$ ps aux | grep resquerails 8950 sh -c cd /home/rails/www/socianalytics/current &&rake environment resque:work QUEUE=high,medium,low...rails 8951 resque-1.9.10: Forked 9105 at 1302095560 rails 9105 resque-1.9.10: Processing medium since 1302095560

Resque workers Unix signals• QUIT - Wait for child to ﬁnish processing then exit• TERM / INT - Immediately kill child then exit• USR1 - Immediately kill child but dont exit• USR2 - Dont start to process any new jobs• CONT - Start to process new jobs again after a USR2

Resque-web (Sinatra webapp)

God setup - my way!Versioned god config files by instance name, ie :• god / app.god• god / web.god• god / db.god• ...God init.d file found on gist.github.com :)god.conf :GOD_CONFIG=/home/rails/www/bookgolf/current/god/db.godGOD_COMMAND="sudo -u rails /usr/local/bin/god"GOD_LOG=/home/rails/log/god.log

God config# ResqueGod.watch do |w| w.env = { RAILS_ROOT => rails_root, RAILS_ENV => rails_env } w.name = "resque-worker" w.interval = 30.seconds w.start = "cd #{rails_root} && rakeenvironment resque:work QUEUE=high,medium,low" w.start_grace = 10.seconds ...

Workers graceful restartRake task to restart workers, the "graceful"waynamespace :resque do task :restart_workers => :environment do pids = Array.new Resque.workers.each do |worker| pids << worker.to_s.split(/:/).second ifworker.to_s.include?(Settings.resque.localhost_name) end if pids.size > 0 system("kill -QUIT #{pids.join( )}") end system("rm /home/rails/.god/pids/resque-worker*.pid") endend

Workers graceful restartnamespace :resque do task :restart_workers, :roles => [:app, :db] do rake "resque:restart_workers" endend

Resque-web - Final setup• Launched externally with nginx reverse proxy (easy to add http basic auth)• External Ruby config file to change Redis setup, load plugins, etc. resque-web conﬁg/resque-web.rb

Resque-web God monitoring%w{5678}.each do |port| God.watch do |w| w.env = { RAILS_ROOT => rails_root, RAILS_ENV => rails_env } w.uid = "rails" w.name = "resque-web" w.interval = 30.seconds w.start = "cd #{rails_root} && resque-web config/resque-web.rb" w.start_grace = 15.seconds w.start_if do |start| start.condition(:process_running) do |c| c.interval = 5.seconds c.running = false end end endend

ActiveRecord - timeout / stale :(Fast workaround... ... ActiveRecord::Base.connection.reconnect! ...

Resque plugins• defunkt / resque-lockPrevent two workers from working on the sameJob class with same arguments

Resque plugins• jayniz / resque-lonerTIP :Do not forget to : require "resque-loner"in your Resque-web config.rb file! or... FAIL (cleared jobs wontbe relaunched again!)BAD :Need inheritance in job classes :(... But a fork exists thatdoesnt need it.

Other plugins...Look @ github.com/defunkt/resqueWiki!And remember that with a goodsystem architecture, Resque is just...

http://www.nicolasblanco.fr	288
http://localhost:2999	136
http://localhost:3000	22
http://localhost:4567	10
http://nicolasblanco.fr	3
http://twitter.com	2
url_unknown	1

Background processing with Resque

by Nicolas Blanco on Apr 09, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

7 Embeds 462

Statistics

Background processing with Resque — Presentation Transcript