Background Processing in Ruby on Rails

9,055 views
8,963 views

Published on

Background processing is an important tool in web development. Some
things just can't optimized enough for the normal request/response cycle
of a web site and need to be run asynchronously. Ruby on Rails doesn't provide
you with any real out of the box solution. There are a lot of external
options available and this part of Rails is lacking any real convention
or standard.

In this presentation, we will talk about how to choose and implement a
solution that fits your needs. We will start with some basic options
using built in Rails tools and we will cover some of the more popular
solutions solutions out there such as BackgrounDRb, Background Job,
Delayed Job, Workling and more.

Rob Mack has been working with Rails professionally on and off since
2005. Rob currently works for VitalSource Technologies.

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
9,055
On SlideShare
0
From Embeds
0
Number of Embeds
1,388
Actions
Shares
0
Downloads
114
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Background Processing in Ruby on Rails

  1. 1. Background Processing Rob Mack Austin on Rails - April 2009 Thursday, April 30, 2009
  2. 2. Why Background Processing? Thursday, April 30, 2009
  3. 3. Simple Background Processing Thursday, April 30, 2009
  4. 4. script/runner From RAILS_ROOT: • script/runner lib/do_work.rb • script/runner ‘Worker.do_work’ Thursday, April 30, 2009
  5. 5. Rake def call_rake(task,options={}) options[:rails_env] = RAILS_ENV args = options.map{|n,v| quot;#{n.to_s.upcase}='#{v}'quot;} system quot;rake #{task} #{args.join(' ')} " end • Episode #127 Railscasts Thursday, April 30, 2009
  6. 6. Daemons http://daemons.rubyforge.org/ http://github.com/dougal/daemon_generator/ Thursday, April 30, 2009
  7. 7. Generate A Daemon • script/generate daemon my_worker create lib/daemons create script/daemons create lib/daemons/my_worker.rb create lib/daemons/my_worker_ctl create config/daemons.yml Thursday, April 30, 2009
  8. 8. Do Some Work $running = true Signal.trap(quot;TERMquot;) do $running = false end while($running) do # Do some work sleep 10 end lib/daemons/my_worker.rb Thursday, April 30, 2009
  9. 9. Kick It Off lib/daemons/my_worker_ctl start lib/daemons/my_worker_ctl stop Thursday, April 30, 2009
  10. 10. Spawn http://github/tra/spawn Thursday, April 30, 2009
  11. 11. Spawn - Do Some Work def my_action flash[:notice] = quot;Processing ... quot; spawn do # do some processing end end Thursday, April 30, 2009
  12. 12. Simple Queues Thursday, April 30, 2009
  13. 13. ar_mailer setup: gem install ar_mailer ar_sendmail --create-migration ar_sendmail --create-model http://seattlerb.rubyforge.org/ar_mailer/ Thursday, April 30, 2009
  14. 14. ar_mailer Schema create_table quot;emailsquot; do |t| t.string quot;fromquot; t.string quot;toquot; t.integer quot;last_send_attemptquot;, :default => 0 t.text quot;mailquot;, :limit => 16777215 t.datetime quot;created_onquot; end Thursday, April 30, 2009
  15. 15. ar_mailer Setup environment.rb config.gem 'ar_mailer', :version => '1.3.1', :lib => 'action_mailer/ar_mailer' config.action_mailer.delivery_method = :activerecord Mail Model: class MyMailer < ActionMailer::ARMailer Thursday, April 30, 2009
  16. 16. Mail Something MyMailer.deliver_newsletter @recipients Thursday, April 30, 2009
  17. 17. The Worker Process ar_sendmail --daemonize --max-age 0 Thursday, April 30, 2009
  18. 18. BackgrounDRb http://backgroundrb.rubyforge.org http://github.com/gnufied/backgroundrb Thursday, April 30, 2009
  19. 19. BackgrounDRb Setup • sudo gem install chronic packet • script/plugin install git://github.com/ gnufied/backgroundrb.git • rake backgroundrb:setup • rake db:migrate • edit config/backgroundrb.yml • script/generate worker my_worker Thursday, April 30, 2009
  20. 20. BackgrounDRb Schema create_table quot;bdrb_job_queuesquot; do |t| t.text quot;argsquot; t.string quot;worker_namequot; t.string quot;worker_methodquot; t.string quot;job_keyquot; t.integer quot;takenquot; t.integer quot;finishedquot; t.integer quot;timeoutquot; t.integer quot;priorityquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.datetime quot;archived_atquot; t.string quot;tagquot; t.string quot;submitter_infoquot; t.string quot;runner_infoquot; t.string quot;worker_keyquot; t.datetime quot;scheduled_atquot; end Thursday, April 30, 2009
  21. 21. Do Some Work class MyWorker < BackgrounDRb::MetaWorker set_worker_name :my_worker def create(args = nil) # initialization goes here end def do_work(user_id = nil) # do some work end end def my_action flash[:notice] = quot;Processing ...quot; MiddleMan.worker(:my_worker).async_do_work(:arg => @user.id) end Thursday, April 30, 2009
  22. 22. Persistent Work def my_action MiddleMan(:my_worker).enq_do_work(:job_key => quot;unique_keyquot;) end Thursday, April 30, 2009
  23. 23. Poll Status class MyWorker < BackgrounDRb::MetaWorker set_worker_name :my_worker def create(args = nil) # initialization goes here end def do_work(user_id = nil) # do some work register_status quot;almost donequot; end end Thursday, April 30, 2009
  24. 24. Poll Status def my_action @job_key = MiddleMan.new_worker( :worker => :my_worker, :job_key => quot;unique_keyquot; ) MiddleMan.worker(:my_worker, @job_key). do_work(@user.id) end def check_status @status = MiddleMan.worker(:my_worker, @job_key). ask_status end Thursday, April 30, 2009
  25. 25. The Worker Process script/backgroundrb start Thursday, April 30, 2009
  26. 26. BJ - Background Job http://codeforpeople.rubyforge.org/svn/bj Thursday, April 30, 2009
  27. 27. Bj - Setup script/plugin install http:// codeforpeople.rubyforge.org/svn/rails/plugins/bj script/bj setup gem install bj require ‘bj’ bj setup Thursday, April 30, 2009
  28. 28. Bj Schema create_table quot;bj_jobquot;, :primary_key => quot;bj_job_idquot; do |t| t.text quot;commandquot; t.text quot;statequot; t.integer quot;priorityquot; t.text quot;tagquot; t.integer quot;is_restartablequot; t.text quot;submitterquot; t.text quot;runnerquot; t.integer quot;pidquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.text quot;envquot; t.text quot;stdinquot; t.text quot;stdoutquot; t.text quot;stderrquot; t.integer quot;exit_statusquot; end Thursday, April 30, 2009
  29. 29. Bj Schema create_table quot;bj_job_archivequot;, :primary_key => quot;bj_job_archive_idquot; do |t| t.text quot;commandquot; t.text quot;statequot; t.integer quot;priorityquot; t.text quot;tagquot; t.integer quot;is_restartablequot; t.text quot;submitterquot; t.text quot;runnerquot; t.integer quot;pidquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.datetime quot;archived_atquot; t.text quot;envquot; t.text quot;stdinquot; t.text quot;stdoutquot; t.text quot;stderrquot; t.integer quot;exit_statusquot; end Thursday, April 30, 2009
  30. 30. Do Some Work def my_action flash[:notice] = quot;Processing ...quot; Bj.submit './script/runner /lib/do_some_work.rb' end def my_other_action flash[:notice] = quot;Processing ...quot; Bj.submit 'echo foobar', :tag => 'sample job' end Thursday, April 30, 2009
  31. 31. The Worker Process bj run --forever --rails_env=development Thursday, April 30, 2009
  32. 32. Delayed Job http://github.com/tobi/delayed_job/ Thursday, April 30, 2009
  33. 33. Delayed Job Setup • script/plugin install git://github.com/ tobi/delayed_job.git • create a delayed jobs table Thursday, April 30, 2009
  34. 34. Delayed Job Schema create_table quot;delayed_jobsquot; do |t| t.integer quot;priorityquot;, :default => 0 t.integer quot;attemptsquot;, :default => 0 t.text quot;handlerquot; t.string quot;last_errorquot; t.datetime quot;run_atquot; t.datetime quot;locked_atquot; t.datetime quot;failed_atquot; t.string quot;locked_byquot; t.datetime quot;created_atquot; t.datetime quot;updated_atquot; end Thursday, April 30, 2009
  35. 35. Do Some Work class MyWorker attr_accessor :user_id def initialize(user) self.user_id = user_id end def perform User.find user_id # do some work end end Thursday, April 30, 2009
  36. 36. Do Some Work def my_action flash[:notice] = quot;Processing ...quot; Delayed::Job.enqueue MyWorker.new(@user.id) end def my_action flash[:notice] = quot;Processing ...quot; MyMailer.new(@user.id).send_later(:deliver_newsletter) end Thursday, April 30, 2009
  37. 37. The Worker Process rake jobs:work Thursday, April 30, 2009
  38. 38. Workling http://github.com/purzelrakete/workling/ Thursday, April 30, 2009
  39. 39. Workling Install script/plugin install git://github.com/purzelrakete/ workling.git Thursday, April 30, 2009
  40. 40. Do Some Work class MyWorker < Workling::Base def do_something(options) # do some work end end def my_action flash[:notice] = quot;Processing ...quot; MyWorker.async_do_something :user_id => @user.id end Thursday, April 30, 2009
  41. 41. The Worker Process with spawn: script/workling_client start with starling: sudo starling -d script/workling_client start Thursday, April 30, 2009
  42. 42. Starling http://github.com/starling/starling/ Thursday, April 30, 2009
  43. 43. Even More Options • Background Fu - http://github.com/ncr/ background-fu • Roofus Scheduler - http://github.com/ jmettraux/rufus-scheduler • AP4R - http://rubyforge.org/projects/ ap4r/ Thursday, April 30, 2009
  44. 44. Advanced Queues Thursday, April 30, 2009
  45. 45. Kestrel http://github.com/robey/kestrel/ Thursday, April 30, 2009
  46. 46. Rabbit MQ http://www.rabbitmq.com/ Thursday, April 30, 2009
  47. 47. Active MQ http://activemq.apache.org/ Thursday, April 30, 2009
  48. 48. Amazon SQS http://aws.amazon.com/sqs/ Thursday, April 30, 2009
  49. 49. Monitoring • god - http://github.com/mojombo/ god/ • launchd - OS X • monit - http://mmonit.com/monit/ Thursday, April 30, 2009
  50. 50. Persistant? Yes No Scaling concerns or fear I enjoy of commitment? managing a process? No Yes No Yes Delayed Job Background Job Spawn Workling Daemons Are you the Kestrel Starling Probably Not Yes next Twitter? Rabbit MQ Thursday, April 30, 2009
  51. 51. Thanks rob@robmack.com twitter.com/robmack Thursday, April 30, 2009

×