Resque is a background processing library for Ruby that allows processing of jobs asynchronously via a queue using Redis. It uses workers that listen for jobs on queues and process them, and includes plugins like resque-lock to prevent duplicate jobs and resque-loner to relaunch failed jobs. The document discusses setting up Resque with God for monitoring workers, handling signals, and restarting workers gracefully on the bookandgolf.com project which synchronizes golf course data from multiple APIs into Redis.

1. Resque
Background processing
System administration on a multi-
instances setup
Experience feedback
Plugins
Nicolas Blanco
Novapost

2. My project : bookandgolf.com

3. bookandgolf.com
At launch :
• more than 60 golfs to synchronise
• 4 different APIs
• More than 350 000 slots (starts) to synchronize every 2 hours
At first we tried to do the synchronization with cron tasks... but...

5. nginx resque-web
:web
redis-server
:db
worker 1
Current
worker 2
:app
worker 3
architecture
worker 4

6. Resque job class
class MyKikoololSleepJob
@queue = :medium
def self.perform(sleep_duration)
sleep sleep_duration
end
end
Resque.enqueue(MyKikoololSleepJob, 10)

7. Resque workers
rake environment resque:work QUEUE=medium
rails@www:~$ ps aux | grep resque
rails 8950 sh -c cd /home/rails/www/socianalytics/current &&
rake environment resque:work QUEUE=high,medium,low
...
rails 8951 resque-1.9.10: Forked 9105 at 1302095560
rails 9105 resque-1.9.10: Processing medium since 1302095560

8. Resque workers Unix signals
• QUIT - Wait for child to finish processing then exit
• TERM / INT - Immediately kill child then exit
• USR1 - Immediately kill child but don't exit
• USR2 - Don't start to process any new jobs
• CONT - Start to process new jobs again after a USR2

9. Resque-web (Sinatra webapp)

12. God setup - my way!
Versioned god config files by instance name, ie :
• god / app.god
• god / web.god
• god / db.god
• ...
God init.d file found on gist.github.com :)
god.conf :
GOD_CONFIG=/home/rails/www/bookgolf/current/god/db.god
GOD_COMMAND="sudo -u rails /usr/local/bin/god"
GOD_LOG=/home/rails/log/god.log

13. God config
# Resque
God.watch do |w|
w.env = { 'RAILS_ROOT' => rails_root,
'RAILS_ENV' => rails_env }
w.name = "resque-worker"
w.interval = 30.seconds
w.start = "cd #{rails_root} && rake
environment resque:work QUEUE=high,medium,low"
w.start_grace = 10.seconds
...

14. Workers graceful restart
Rake task to restart workers, the "graceful"
way
namespace :resque do
task :restart_workers => :environment do
pids = Array.new
Resque.workers.each do |worker|
pids << worker.to_s.split(/:/).second if
worker.to_s.include?(Settings.resque.localhost_name)
end
if pids.size > 0
system("kill -QUIT #{pids.join(' ')}")
end
system("rm /home/rails/.god/pids/resque-worker*.pid")
end
end

15. Workers graceful restart
namespace :resque do
task :restart_workers, :roles => [:app, :db] do
rake "resque:restart_workers"
end
end

16. Resque-web - Final setup
• Launched externally with nginx reverse proxy (easy to add
http basic auth)
• External Ruby config file to change Redis setup, load
plugins, etc.
resque-web config/resque-web.rb

17. Resque-web God monitoring
%w{5678}.each do |port|
God.watch do |w|
w.env = { 'RAILS_ROOT' => rails_root,
'RAILS_ENV' => rails_env }
w.uid = "rails"
w.name = "resque-web"
w.interval = 30.seconds
w.start = "cd #{rails_root} && resque-web config/resque-
web.rb"
w.start_grace = 15.seconds
w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = 5.seconds
c.running = false
end
end
end
end

18. ActiveRecord - timeout / stale :(
Fast workaround...
...
ActiveRecord::Base.connection.reconnect!
...

19. Resque plugins
• defunkt / resque-lock
Prevent two workers from working on the same
Job class with same arguments

20. Resque plugins
• jayniz / resque-loner
TIP :
Do not forget to :
require "resque-loner"
in your Resque-web config.rb file! or... FAIL (cleared jobs won't
be relaunched again!)
BAD :
Need inheritance in job classes :'(... But a fork exists that
doesn't need it.

21. Other plugins...
Look @ github.com/defunkt/resque
Wiki!
And remember that with a good
system architecture, Resque is just...