• Like

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Background processing

  • 2,380 views
Uploaded on

 

More in: Business , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,380
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
17
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Background Processing Kerry Buckley IPRUG 3 November 2010
  • 2. Why? • Sending e-mail, SMS, tweets etc • Resizing images • Generating large PDFs • Updating search indexes • etc
  • 3. Approaches • Process/thread based • Spawn • Database-backed • BackgroundDRb, BackgroundJob, DelayedJob, JobFu, Background-Fu • Queue-backed • Sparrow, Starling, Kestrel, RabbitMQ, ActiveMQ, SQS, Beanstalkd, Resque
  • 4. Delayed::Job
  • 5. Delayed::Job • Extracted from Shopify • Daemon runs with Rails environment • Use collectiveidea’s fork • 2.0 for Rails 2.x • 2.1RC (from GitHub) for Rails 3 • ActiveRecord/MongoMapper/DataMapper
  • 6. Setup Gemfile: gem "delayed_job", :git => "git://github.com/collectiveidea/delayed_job.git" Then run: $ rails g delayed_job $ rake db:migrate $ script/delayed_job start
  • 7. Delaying methods Mailer.send_alerts(event)
  • 8. Delaying methods Mailer.delay.send_alerts(event)
  • 9. Options Mailer.delay(:priority => 3, run_at => 1.hour.from_now). send_alerts(event)
  • 10. Running workers Foreground $ rake jobs:work $ script/delayed_job run Daemonised $ script/delayed_job start $ RAILS_ENV=production script/delayed_job -n 4 start
  • 11. Custom Jobs class NewsletterJob < Struct.new(:text, :emails) def perform emails.each do |e| NewsletterMailer.deliver_text_to_email(text, e) end end end emails = Customers.all.collect(&:email) Delayed::Job.enqueue NewsletterJob.new(text, emails)
  • 12. Hooks def enqueue(job) end def before(job) end def after(job) end def success(job) end def error(job, exception) end def failure end
  • 13. Resque
  • 14. Resque • Created by GitHub • Jobs persisted to Redis as JSON • Multiple queues • Parent/child forking architecture • Web monitoring interface
  • 15. Setup Redis: $ brew install redis Gemfile: gem "resque" Rakefile: require "resque/tasks" task "resque:setup" => :environment
  • 16. Delaying methods class Archive @queue = :file_serve def self.perform(repo_id, branch = 'master') repo = Repository.find(repo_id) repo.create_archive(branch) end end Resque.enqueue(Archive, repository.id, branch)
  • 17. Running queues $ QUEUE=queue_name rake resque:work $ QUEUES=file_serve,warm_cache rake resque:work $ QUEUES=critical,high,low rake resque:work $ QUEUE=* rake resque:work $ COUNT=5 QUEUE=* rake resque:workers
  • 18. Monitoring $ resque-web
  • 19. Which to use? How much of your site is background work? Less than 50% More than 50% Delayed::Job Resque
  • 20. Demo