Background Processing in Ruby on Rails

Background Processing Rob Mack Austin on Rails - April 2009 Thursday, April 30, 2009

Why Background Processing? Thursday, April 30, 2009

Simple Background Processing Thursday, April 30, 2009

script/runner From RAILS_ROOT: • script/runner lib/do_work.rb • script/runner ‘Worker.do_work’ Thursday, April 30, 2009

Rake def call_rake(task,options={}) options[:rails_env] = RAILS_ENV args = options.map{|n,v| quot;#{n.to_s.upcase}='#{v}'quot;} system quot;rake #{task} #{args.join(' ')} " end • Episode #127 Railscasts Thursday, April 30, 2009

Daemons http://daemons.rubyforge.org/ http://github.com/dougal/daemon_generator/ Thursday, April 30, 2009

Generate A Daemon • script/generate daemon my_worker create lib/daemons create script/daemons create lib/daemons/my_worker.rb create lib/daemons/my_worker_ctl create conﬁg/daemons.yml Thursday, April 30, 2009

Do Some Work $running = true Signal.trap(quot;TERMquot;) do $running = false end while($running) do # Do some work sleep 10 end lib/daemons/my_worker.rb Thursday, April 30, 2009

Kick It Off lib/daemons/my_worker_ctl start lib/daemons/my_worker_ctl stop Thursday, April 30, 2009

Spawn http://github/tra/spawn Thursday, April 30, 2009

Spawn - Do Some Work def my_action flash[:notice] = quot;Processing ... quot; spawn do # do some processing end end Thursday, April 30, 2009

Simple Queues Thursday, April 30, 2009

ar_mailer setup: gem install ar_mailer ar_sendmail --create-migration ar_sendmail --create-model http://seattlerb.rubyforge.org/ar_mailer/ Thursday, April 30, 2009

ar_mailer Schema create_table quot;emailsquot; do |t| t.string quot;fromquot; t.string quot;toquot; t.integer quot;last_send_attemptquot;, :default => 0 t.text quot;mailquot;, :limit => 16777215 t.datetime quot;created_onquot; end Thursday, April 30, 2009

ar_mailer Setup environment.rb config.gem 'ar_mailer', :version => '1.3.1', :lib => 'action_mailer/ar_mailer' config.action_mailer.delivery_method = :activerecord Mail Model: class MyMailer < ActionMailer::ARMailer Thursday, April 30, 2009

Mail Something MyMailer.deliver_newsletter @recipients Thursday, April 30, 2009

The Worker Process ar_sendmail --daemonize --max-age 0 Thursday, April 30, 2009

BackgrounDRb http://backgroundrb.rubyforge.org http://github.com/gnuﬁed/backgroundrb Thursday, April 30, 2009

BackgrounDRb Setup • sudo gem install chronic packet • script/plugin install git://github.com/ gnuﬁed/backgroundrb.git • rake backgroundrb:setup • rake db:migrate • edit conﬁg/backgroundrb.yml • script/generate worker my_worker Thursday, April 30, 2009

BackgrounDRb Schema create_table quot;bdrb_job_queuesquot; do |t| t.text quot;argsquot; t.string quot;worker_namequot; t.string quot;worker_methodquot; t.string quot;job_keyquot; t.integer quot;takenquot; t.integer quot;finishedquot; t.integer quot;timeoutquot; t.integer quot;priorityquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.datetime quot;archived_atquot; t.string quot;tagquot; t.string quot;submitter_infoquot; t.string quot;runner_infoquot; t.string quot;worker_keyquot; t.datetime quot;scheduled_atquot; end Thursday, April 30, 2009

Do Some Work class MyWorker < BackgrounDRb::MetaWorker set_worker_name :my_worker def create(args = nil) # initialization goes here end def do_work(user_id = nil) # do some work end end def my_action flash[:notice] = quot;Processing ...quot; MiddleMan.worker(:my_worker).async_do_work(:arg => @user.id) end Thursday, April 30, 2009

Persistent Work def my_action MiddleMan(:my_worker).enq_do_work(:job_key => quot;unique_keyquot;) end Thursday, April 30, 2009

Poll Status class MyWorker < BackgrounDRb::MetaWorker set_worker_name :my_worker def create(args = nil) # initialization goes here end def do_work(user_id = nil) # do some work register_status quot;almost donequot; end end Thursday, April 30, 2009

Poll Status def my_action @job_key = MiddleMan.new_worker( :worker => :my_worker, :job_key => quot;unique_keyquot; ) MiddleMan.worker(:my_worker, @job_key). do_work(@user.id) end def check_status @status = MiddleMan.worker(:my_worker, @job_key). ask_status end Thursday, April 30, 2009

The Worker Process script/backgroundrb start Thursday, April 30, 2009

BJ - Background Job http://codeforpeople.rubyforge.org/svn/bj Thursday, April 30, 2009

Bj - Setup script/plugin install http:// codeforpeople.rubyforge.org/svn/rails/plugins/bj script/bj setup gem install bj require ‘bj’ bj setup Thursday, April 30, 2009

Bj Schema create_table quot;bj_jobquot;, :primary_key => quot;bj_job_idquot; do |t| t.text quot;commandquot; t.text quot;statequot; t.integer quot;priorityquot; t.text quot;tagquot; t.integer quot;is_restartablequot; t.text quot;submitterquot; t.text quot;runnerquot; t.integer quot;pidquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.text quot;envquot; t.text quot;stdinquot; t.text quot;stdoutquot; t.text quot;stderrquot; t.integer quot;exit_statusquot; end Thursday, April 30, 2009

Bj Schema create_table quot;bj_job_archivequot;, :primary_key => quot;bj_job_archive_idquot; do |t| t.text quot;commandquot; t.text quot;statequot; t.integer quot;priorityquot; t.text quot;tagquot; t.integer quot;is_restartablequot; t.text quot;submitterquot; t.text quot;runnerquot; t.integer quot;pidquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.datetime quot;archived_atquot; t.text quot;envquot; t.text quot;stdinquot; t.text quot;stdoutquot; t.text quot;stderrquot; t.integer quot;exit_statusquot; end Thursday, April 30, 2009

Do Some Work def my_action flash[:notice] = quot;Processing ...quot; Bj.submit './script/runner /lib/do_some_work.rb' end def my_other_action flash[:notice] = quot;Processing ...quot; Bj.submit 'echo foobar', :tag => 'sample job' end Thursday, April 30, 2009

The Worker Process bj run --forever --rails_env=development Thursday, April 30, 2009

Delayed Job http://github.com/tobi/delayed_job/ Thursday, April 30, 2009

Delayed Job Setup • script/plugin install git://github.com/ tobi/delayed_job.git • create a delayed jobs table Thursday, April 30, 2009

Delayed Job Schema create_table quot;delayed_jobsquot; do |t| t.integer quot;priorityquot;, :default => 0 t.integer quot;attemptsquot;, :default => 0 t.text quot;handlerquot; t.string quot;last_errorquot; t.datetime quot;run_atquot; t.datetime quot;locked_atquot; t.datetime quot;failed_atquot; t.string quot;locked_byquot; t.datetime quot;created_atquot; t.datetime quot;updated_atquot; end Thursday, April 30, 2009

Do Some Work class MyWorker attr_accessor :user_id def initialize(user) self.user_id = user_id end def perform User.find user_id # do some work end end Thursday, April 30, 2009

Do Some Work def my_action flash[:notice] = quot;Processing ...quot; Delayed::Job.enqueue MyWorker.new(@user.id) end def my_action flash[:notice] = quot;Processing ...quot; MyMailer.new(@user.id).send_later(:deliver_newsletter) end Thursday, April 30, 2009

The Worker Process rake jobs:work Thursday, April 30, 2009

Workling http://github.com/purzelrakete/workling/ Thursday, April 30, 2009

Workling Install script/plugin install git://github.com/purzelrakete/ workling.git Thursday, April 30, 2009

Do Some Work class MyWorker < Workling::Base def do_something(options) # do some work end end def my_action flash[:notice] = quot;Processing ...quot; MyWorker.async_do_something :user_id => @user.id end Thursday, April 30, 2009

The Worker Process with spawn: script/workling_client start with starling: sudo starling -d script/workling_client start Thursday, April 30, 2009

Starling http://github.com/starling/starling/ Thursday, April 30, 2009

Even More Options • Background Fu - http://github.com/ncr/ background-fu • Roofus Scheduler - http://github.com/ jmettraux/rufus-scheduler • AP4R - http://rubyforge.org/projects/ ap4r/ Thursday, April 30, 2009

Advanced Queues Thursday, April 30, 2009

Kestrel http://github.com/robey/kestrel/ Thursday, April 30, 2009

Rabbit MQ http://www.rabbitmq.com/ Thursday, April 30, 2009

Active MQ http://activemq.apache.org/ Thursday, April 30, 2009

Amazon SQS http://aws.amazon.com/sqs/ Thursday, April 30, 2009

Monitoring • god - http://github.com/mojombo/ god/ • launchd - OS X • monit - http://mmonit.com/monit/ Thursday, April 30, 2009

Persistant? Yes No Scaling concerns or fear I enjoy of commitment? managing a process? No Yes No Yes Delayed Job Background Job Spawn Workling Daemons Are you the Kestrel Starling Probably Not Yes next Twitter? Rabbit MQ Thursday, April 30, 2009

Thanks rob@robmack.com twitter.com/robmack Thursday, April 30, 2009

http://blog.robmack.com	472
http://www.slideshare.net	80
http://209.85.229.132	1

Background Processing in Ruby on Rails

by robmack on Apr 30, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 553

Statistics

Background Processing in Ruby on Rails — Presentation Transcript