Background
Processing
Rob Mack
Austin on Rails - April 2009
Thursday, April 30, 2009

Why Background
Processing?
Thursday, April 30, 2009

Simple Background
Processing
Thursday, April 30, 2009

script/runner
From RAILS_ROOT:
• script/runner lib/do_work.rb
• script/runner ‘Worker.do_work’
Thursday, April 30, 2009

Rake
def call_rake(task,options={})
options[:rails_env] = RAILS_ENV
args = options.map{|n,v| \"#{n.to_s.upcase}='#{v}'\"}
system \"rake #{task} #{args.join(' ')} &\"
end
• Episode #127 Railscasts
Thursday, April 30, 2009

Daemons
http://daemons.rubyforge.org/
http://github.com/dougal/daemon_generator/
Thursday, April 30, 2009

Generate A Daemon
• script/generate daemon my_worker
create lib/daemons
create script/daemons
create lib/daemons/my_worker.rb
create lib/daemons/my_worker_ctl
create config/daemons.yml
Thursday, April 30, 2009

Do Some Work
$running = true
Signal.trap(\"TERM\") do
$running = false
end
while($running) do
# Do some work
sleep 10
end
lib/daemons/my_worker.rb
Thursday, April 30, 2009

Kick It Off
lib/daemons/my_worker_ctl start
lib/daemons/my_worker_ctl stop
Thursday, April 30, 2009

Spawn
http://github/tra/spawn
Thursday, April 30, 2009

Spawn - Do Some Work
def my_action
flash[:notice] = \"Processing ... \"
spawn do
# do some processing
end
end
Thursday, April 30, 2009

Simple Queues
Thursday, April 30, 2009

ar_mailer
setup:
gem install ar_mailer
ar_sendmail --create-migration
ar_sendmail --create-model
http://seattlerb.rubyforge.org/ar_mailer/
Thursday, April 30, 2009

ar_mailer Schema
create_table \"emails\" do |t|
t.string \"from\"
t.string \"to\"
t.integer \"last_send_attempt\", :default => 0
t.text \"mail\", :limit => 16777215
t.datetime \"created_on\"
end
Thursday, April 30, 2009

ar_mailer Setup
environment.rb
config.gem 'ar_mailer', :version => '1.3.1',
:lib => 'action_mailer/ar_mailer'
config.action_mailer.delivery_method = :activerecord
Mail Model:
class MyMailer < ActionMailer::ARMailer
Thursday, April 30, 2009

Mail Something
MyMailer.deliver_newsletter @recipients
Thursday, April 30, 2009

The Worker Process
ar_sendmail --daemonize --max-age 0
Thursday, April 30, 2009

BackgrounDRb
http://backgroundrb.rubyforge.org
http://github.com/gnufied/backgroundrb
Thursday, April 30, 2009

BackgrounDRb Setup
• sudo gem install chronic packet
• script/plugin install git://github.com/
gnufied/backgroundrb.git
• rake backgroundrb:setup
• rake db:migrate
• edit config/backgroundrb.yml
• script/generate worker my_worker
Thursday, April 30, 2009

BackgrounDRb Schema
create_table \"bdrb_job_queues\" do |t|
t.text \"args\"
t.string \"worker_name\"
t.string \"worker_method\"
t.string \"job_key\"
t.integer \"taken\"
t.integer \"finished\"
t.integer \"timeout\"
t.integer \"priority\"
t.datetime \"submitted_at\"
t.datetime \"started_at\"
t.datetime \"finished_at\"
t.datetime \"archived_at\"
t.string \"tag\"
t.string \"submitter_info\"
t.string \"runner_info\"
t.string \"worker_key\"
t.datetime \"scheduled_at\"
end
Thursday, April 30, 2009

Do Some Work
class MyWorker < BackgrounDRb::MetaWorker
set_worker_name :my_worker
def create(args = nil)
# initialization goes here
end
def do_work(user_id = nil)
# do some work
end
end
def my_action
flash[:notice] = \"Processing ...\"
MiddleMan.worker(:my_worker).async_do_work(:arg => @user.id)
end
Thursday, April 30, 2009

Persistent Work
def my_action
MiddleMan(:my_worker).enq_do_work(:job_key => \"unique_key\")
end
Thursday, April 30, 2009

Poll Status
class MyWorker < BackgrounDRb::MetaWorker
set_worker_name :my_worker
def create(args = nil)
# initialization goes here
end
def do_work(user_id = nil)
# do some work
register_status \"almost done\"
end
end
Thursday, April 30, 2009

Poll Status
def my_action
@job_key = MiddleMan.new_worker(
:worker => :my_worker,
:job_key => \"unique_key\"
)
MiddleMan.worker(:my_worker, @job_key).
do_work(@user.id)
end
def check_status
@status = MiddleMan.worker(:my_worker, @job_key).
ask_status
end
Thursday, April 30, 2009

The Worker Process
script/backgroundrb start
Thursday, April 30, 2009

BJ - Background Job
http://codeforpeople.rubyforge.org/svn/bj
Thursday, April 30, 2009

Bj - Setup
script/plugin install http://
codeforpeople.rubyforge.org/svn/rails/plugins/bj
script/bj setup
gem install bj
require ‘bj’
bj setup
Thursday, April 30, 2009

Bj Schema
create_table \"bj_job\", :primary_key => \"bj_job_id\" do |t|
t.text \"command\"
t.text \"state\"
t.integer \"priority\"
t.text \"tag\"
t.integer \"is_restartable\"
t.text \"submitter\"
t.text \"runner\"
t.integer \"pid\"
t.datetime \"submitted_at\"
t.datetime \"started_at\"
t.datetime \"finished_at\"
t.text \"env\"
t.text \"stdin\"
t.text \"stdout\"
t.text \"stderr\"
t.integer \"exit_status\"
end
Thursday, April 30, 2009

Bj Schema
create_table \"bj_job_archive\", :primary_key => \"bj_job_archive_id\" do |t|
t.text \"command\"
t.text \"state\"
t.integer \"priority\"
t.text \"tag\"
t.integer \"is_restartable\"
t.text \"submitter\"
t.text \"runner\"
t.integer \"pid\"
t.datetime \"submitted_at\"
t.datetime \"started_at\"
t.datetime \"finished_at\"
t.datetime \"archived_at\"
t.text \"env\"
t.text \"stdin\"
t.text \"stdout\"
t.text \"stderr\"
t.integer \"exit_status\"
end
Thursday, April 30, 2009

Do Some Work
def my_action
flash[:notice] = \"Processing ...\"
Bj.submit './script/runner /lib/do_some_work.rb'
end
def my_other_action
flash[:notice] = \"Processing ...\"
Bj.submit 'echo foobar', :tag => 'sample job'
end
Thursday, April 30, 2009

The Worker Process
bj run --forever --rails_env=development
Thursday, April 30, 2009

Delayed Job
http://github.com/tobi/delayed_job/
Thursday, April 30, 2009

Delayed Job Setup
• script/plugin install git://github.com/
tobi/delayed_job.git
• create a delayed jobs table
Thursday, April 30, 2009

Delayed Job Schema
create_table \"delayed_jobs\" do |t|
t.integer \"priority\", :default => 0
t.integer \"attempts\", :default => 0
t.text \"handler\"
t.string \"last_error\"
t.datetime \"run_at\"
t.datetime \"locked_at\"
t.datetime \"failed_at\"
t.string \"locked_by\"
t.datetime \"created_at\"
t.datetime \"updated_at\"
end
Thursday, April 30, 2009

Do Some Work
class MyWorker
attr_accessor :user_id
def initialize(user)
self.user_id = user_id
end
def perform
User.find user_id
# do some work
end
end
Thursday, April 30, 2009

Do Some Work
def my_action
flash[:notice] = \"Processing ...\"
Delayed::Job.enqueue MyWorker.new(@user.id)
end
def my_action
flash[:notice] = \"Processing ...\"
MyMailer.new(@user.id).send_later(:deliver_newsletter)
end
Thursday, April 30, 2009

The Worker Process
rake jobs:work
Thursday, April 30, 2009

Workling
http://github.com/purzelrakete/workling/
Thursday, April 30, 2009

Workling Install
script/plugin install git://github.com/purzelrakete/
workling.git
Thursday, April 30, 2009

Do Some Work
class MyWorker < Workling::Base
def do_something(options)
# do some work
end
end
def my_action
flash[:notice] = \"Processing ...\"
MyWorker.async_do_something :user_id => @user.id
end
Thursday, April 30, 2009

The Worker Process
with spawn:
script/workling_client start
with starling:
sudo starling -d
script/workling_client start
Thursday, April 30, 2009

Starling
http://github.com/starling/starling/
Thursday, April 30, 2009

Even More Options
• Background Fu - http://github.com/ncr/
background-fu
• Roofus Scheduler - http://github.com/
jmettraux/rufus-scheduler
• AP4R - http://rubyforge.org/projects/
ap4r/
Thursday, April 30, 2009

Advanced Queues
Thursday, April 30, 2009

Kestrel
http://github.com/robey/kestrel/
Thursday, April 30, 2009

Rabbit MQ
http://www.rabbitmq.com/
Thursday, April 30, 2009

Active MQ
http://activemq.apache.org/
Thursday, April 30, 2009

Amazon SQS
http://aws.amazon.com/sqs/
Thursday, April 30, 2009

Monitoring
• god - http://github.com/mojombo/
god/
• launchd - OS X
• monit - http://mmonit.com/monit/
Thursday, April 30, 2009

Persistant?
Yes
No
Scaling concerns or fear
I enjoy
of commitment?
managing a
process?
No Yes
No
Yes
Delayed Job
Background Job
Spawn Workling
Daemons
Are you the Kestrel
Starling Probably Not Yes
next Twitter? Rabbit MQ
Thursday, April 30, 2009

Thanks
rob@robmack.com
twitter.com/robmack
Thursday, April 30, 2009