Background
Processing
Kerry Buckley
IPRUG 3 November 2010
Why?
• Sending e-mail, SMS, tweets etc
• Resizing images
• Generating large PDFs
• Updating search indexes
• etc
Approaches
• Process/thread based
• Spawn
• Database-backed
• BackgroundDRb, BackgroundJob,
DelayedJob, JobFu, Background-...
Delayed::Job
Delayed::Job
• Extracted from Shopify
• Daemon runs with Rails environment
• Use collectiveidea’s fork
• 2.0 for Rails 2.x...
Setup
Gemfile:
gem "delayed_job",
:git =>
"git://github.com/collectiveidea/delayed_job.git"
Then run:
$ rails g delayed_job...
Delaying methods
Mailer.send_alerts(event)
Delaying methods
Mailer.delay.send_alerts(event)
Options
Mailer.delay(:priority => 3,
run_at => 1.hour.from_now).
send_alerts(event)
Running workers
Foreground
$ rake jobs:work
$ script/delayed_job run
Daemonised
$ script/delayed_job start
$ RAILS_ENV=pro...
Custom Jobs
class NewsletterJob < Struct.new(:text, :emails)
def perform
emails.each do |e|
NewsletterMailer.deliver_text_...
Hooks
def enqueue(job)
end
def before(job)
end
def after(job)
end
def success(job)
end
def error(job, exception)
end
def f...
Resque
Resque
• Created by GitHub
• Jobs persisted to Redis as JSON
• Multiple queues
• Parent/child forking architecture
• Web m...
Setup
Redis:
$ brew install redis
Gemfile:
gem "resque"
Rakefile:
require "resque/tasks"
task "resque:setup" => :environment
Delaying methods
class Archive
@queue = :file_serve
def self.perform(repo_id, branch = 'master')
repo = Repository.find(re...
Running queues
$ QUEUE=queue_name rake resque:work
$ QUEUES=file_serve,warm_cache rake resque:work
$ QUEUES=critical,high,...
Monitoring
$ resque-web
Which to use?
Delayed::Job Resque
How much of your site is background work?
Less than 50% More than 50%
Demo
Background processing
Upcoming SlideShare
Loading in...5
×

Background processing

2,537

Published on

Published in: Business, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,537
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
20
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Background processing

  1. 1. Background Processing Kerry Buckley IPRUG 3 November 2010
  2. 2. Why? • Sending e-mail, SMS, tweets etc • Resizing images • Generating large PDFs • Updating search indexes • etc
  3. 3. Approaches • Process/thread based • Spawn • Database-backed • BackgroundDRb, BackgroundJob, DelayedJob, JobFu, Background-Fu • Queue-backed • Sparrow, Starling, Kestrel, RabbitMQ, ActiveMQ, SQS, Beanstalkd, Resque
  4. 4. Delayed::Job
  5. 5. Delayed::Job • Extracted from Shopify • Daemon runs with Rails environment • Use collectiveidea’s fork • 2.0 for Rails 2.x • 2.1RC (from GitHub) for Rails 3 • ActiveRecord/MongoMapper/DataMapper
  6. 6. Setup Gemfile: gem "delayed_job", :git => "git://github.com/collectiveidea/delayed_job.git" Then run: $ rails g delayed_job $ rake db:migrate $ script/delayed_job start
  7. 7. Delaying methods Mailer.send_alerts(event)
  8. 8. Delaying methods Mailer.delay.send_alerts(event)
  9. 9. Options Mailer.delay(:priority => 3, run_at => 1.hour.from_now). send_alerts(event)
  10. 10. Running workers Foreground $ rake jobs:work $ script/delayed_job run Daemonised $ script/delayed_job start $ RAILS_ENV=production script/delayed_job -n 4 start
  11. 11. Custom Jobs class NewsletterJob < Struct.new(:text, :emails) def perform emails.each do |e| NewsletterMailer.deliver_text_to_email(text, e) end end end emails = Customers.all.collect(&:email) Delayed::Job.enqueue NewsletterJob.new(text, emails)
  12. 12. Hooks def enqueue(job) end def before(job) end def after(job) end def success(job) end def error(job, exception) end def failure end
  13. 13. Resque
  14. 14. Resque • Created by GitHub • Jobs persisted to Redis as JSON • Multiple queues • Parent/child forking architecture • Web monitoring interface
  15. 15. Setup Redis: $ brew install redis Gemfile: gem "resque" Rakefile: require "resque/tasks" task "resque:setup" => :environment
  16. 16. Delaying methods class Archive @queue = :file_serve def self.perform(repo_id, branch = 'master') repo = Repository.find(repo_id) repo.create_archive(branch) end end Resque.enqueue(Archive, repository.id, branch)
  17. 17. Running queues $ QUEUE=queue_name rake resque:work $ QUEUES=file_serve,warm_cache rake resque:work $ QUEUES=critical,high,low rake resque:work $ QUEUE=* rake resque:work $ COUNT=5 QUEUE=* rake resque:workers
  18. 18. Monitoring $ resque-web
  19. 19. Which to use? Delayed::Job Resque How much of your site is background work? Less than 50% More than 50%
  20. 20. Demo
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×