Background processing

2,925 views
2,748 views

Published on

Published in: Business, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,925
On SlideShare
0
From Embeds
0
Number of Embeds
30
Actions
Shares
0
Downloads
20
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Background processing

  1. 1. Background Processing Kerry Buckley IPRUG 3 November 2010
  2. 2. Why? • Sending e-mail, SMS, tweets etc • Resizing images • Generating large PDFs • Updating search indexes • etc
  3. 3. Approaches • Process/thread based • Spawn • Database-backed • BackgroundDRb, BackgroundJob, DelayedJob, JobFu, Background-Fu • Queue-backed • Sparrow, Starling, Kestrel, RabbitMQ, ActiveMQ, SQS, Beanstalkd, Resque
  4. 4. Delayed::Job
  5. 5. Delayed::Job • Extracted from Shopify • Daemon runs with Rails environment • Use collectiveidea’s fork • 2.0 for Rails 2.x • 2.1RC (from GitHub) for Rails 3 • ActiveRecord/MongoMapper/DataMapper
  6. 6. Setup Gemfile: gem "delayed_job", :git => "git://github.com/collectiveidea/delayed_job.git" Then run: $ rails g delayed_job $ rake db:migrate $ script/delayed_job start
  7. 7. Delaying methods Mailer.send_alerts(event)
  8. 8. Delaying methods Mailer.delay.send_alerts(event)
  9. 9. Options Mailer.delay(:priority => 3, run_at => 1.hour.from_now). send_alerts(event)
  10. 10. Running workers Foreground $ rake jobs:work $ script/delayed_job run Daemonised $ script/delayed_job start $ RAILS_ENV=production script/delayed_job -n 4 start
  11. 11. Custom Jobs class NewsletterJob < Struct.new(:text, :emails) def perform emails.each do |e| NewsletterMailer.deliver_text_to_email(text, e) end end end emails = Customers.all.collect(&:email) Delayed::Job.enqueue NewsletterJob.new(text, emails)
  12. 12. Hooks def enqueue(job) end def before(job) end def after(job) end def success(job) end def error(job, exception) end def failure end
  13. 13. Resque
  14. 14. Resque • Created by GitHub • Jobs persisted to Redis as JSON • Multiple queues • Parent/child forking architecture • Web monitoring interface
  15. 15. Setup Redis: $ brew install redis Gemfile: gem "resque" Rakefile: require "resque/tasks" task "resque:setup" => :environment
  16. 16. Delaying methods class Archive @queue = :file_serve def self.perform(repo_id, branch = 'master') repo = Repository.find(repo_id) repo.create_archive(branch) end end Resque.enqueue(Archive, repository.id, branch)
  17. 17. Running queues $ QUEUE=queue_name rake resque:work $ QUEUES=file_serve,warm_cache rake resque:work $ QUEUES=critical,high,low rake resque:work $ QUEUE=* rake resque:work $ COUNT=5 QUEUE=* rake resque:workers
  18. 18. Monitoring $ resque-web
  19. 19. Which to use? Delayed::Job Resque How much of your site is background work? Less than 50% More than 50%
  20. 20. Demo

×