Asynchronous Processing with Ruby on Rails (RailsConf 2008)

Asynchronous Processing Jonathan Dahl (and RailSpikes, Slantwise, Zencoder, etc.)

I. What is it, and why should I care?

Wife: What are you talking about at RailsConf this year? Jon: Asynchronous Processing Wife: [changes subject]

important tool

(ahem...)

Related Concepts • Background Processing • Parallel Processing • Distributed Processing

has_attachment :storage => :s3

Image Upload ( 15 seconds) Send to S3 Browser Response

Image Upload Send to S3 ( 15 seconds) Browser Response

Image Upload Send to S3 Browser ( 3 seconds) Response

Image Upload Send to S3 Browser Response

has_attachment :storage => :file_system

Image Upload ( 15 seconds) Browser Response Send to S3

Image Upload Browser ( 3 seconds) Response Send to S3 ( who cares?)

has_attachment :storage => :s3, :thumbnails => { :thumb => '100x100!', :small => '240x180>', :medium => '500x500>' }

Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response

II. When do I need it?

Request • Method (GET, POST) • URI (host, port, path) • Parameters

Response • Status (200, 404, 500) • Metadata (content type, server info, etc.) • Body (xml, html, ﬁle)

Resources

Trigger

HTTP trigger - browser request

HTTP trigger - API request GET /photos/1.xml HTTP/1.1 Host: example.com:80

Human trigger - capistrano cap staging deploy

Human trigger - rake rake db:migrate

Human trigger - console $ script/console production Loading production environment (Rails 2.0.2) >> Photo.destroy_all

No trigger? - Send email in 2 hours - Sync data at 3:00am PST - Notify admin when disk is 90% full - Expire sessions that are inactive - Archive records that exceed quota

1. Time 2. Resources 3. Trigger

Concrete examples • Sending mail • Transcoding video/audio • Storing images on S3 • Receiving email • Synching with outside database • Complex computations

class Emailer < ActionMailer::ARMailer

Zencoder User Video Sharing Website Data Storage (Amazon S3) Zencoder Manager Worker Worker Worker Worker

class Photo < ActiveRecord::Base after_create :background_s3_upload def background_s3_upload Bj.submit quot;./script/runner ./jobs/send_to_s3.rb #{self.id}quot; end end

III. So how do you decide what to use?

be seamless

how reliable?

when should it run?

dependencies and system complexity

scaling and/or performance

IV. The simple solution: fork or thread

Parallel vs. Background

1. Stay within one request

2. thread.join

3. ActiveRecord ActiveRecord::Base.allow_concurrency = true

ﬁre and forget

Spawn spawn(:method => :fork) do # do something end

1. Time 2. Resources 3. Trigger

V. More robust solutions

Task Storage Task Trigger

Task Storage • task details (what happens?) • priority • when to run

Task Trigger • worker pulling jobs • time-based • execute immediately

Task Storage Task Trigger Database daemon Message Queue cron

create_table quot;jobsquot; do |t| t.text quot;commandquot; t.integer quot;priorityquot; t.integer quot;pidquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.text quot;resultquot; end

create_table quot;photosquot; do |t| t.string quot;filenamequot; t.datetime quot;created_atquot; t.datetime quot;processed_atquot; end

• Amazon SQS • Websphere MQ • Starling • JMS • beanstalkd

queue = SQS.get_queue(quot;task_listquot;)

put message queue.send_message quot;process:2872quot;

receive message message = queue.receive_message

Starling starling -h 192.168.1.1 -d

require 'memcache' starling = MemCache.new('192.168.1.1:22122') # Put messages onto a queue: starling.set('my_queue', 12345) # Get message from the queue: starling.get('my_queue')

storage choice? • queue: optimized for performance • database: you’ve already got one

daemon #!/usr/bin/env ruby class JobRequester < SimpleDaemon::Base def self.start loop { Job.process_next } end end JobRequester.daemonize

0 6 * * * script/runner jobs/send_emails.rb

cronedit require 'cronedit' CronEdit::Crontab.Add quot;send-emailsquot;, { :minute => 0, :hour => 6, :command => quot;script/runner jobs/send_emails.rbquot; } CronEdit::Crontab.Remove 'old-task'

trigger choice • process: always running • cron: as reliable as your operating system

BackgroundDRb class BillingWorker < BackgrounDRb::MetaWorker set_worker_name :billing_worker def create(args = nil) # this method is called when worker is loaded for the first time end def charge_customer(customer_id = nil) logger.info 'charging customer now' end end MiddleMan.worker(:billing_worker).charge_customer(current_customer.id)

:backgroundrb: :ip: 0.0.0.0 :development: :backgroundrb: :port: 11111 :log: foreground :production: :backgroundrb: :port: 22222 :lazy_load: true :debug_log: false ./script/backgroundrb start

AP4R def MyController def queue ap4r.async_to({:action => 'download'}, {:story => story.id, :url => params[:url]}) end def download # long-running task end end

Bj Acronym

create_table quot;bj_jobquot;, :primary_key => quot;bj t.text quot;commandquot; t.text quot;statequot; t.integer quot;priorityquot; t.text quot;tagquot; t.integer quot;is_restartablequot; t.text quot;submitterquot; t.text quot;runnerquot; t.integer quot;pidquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.text quot;envquot;

Bj.submit quot;./script/runner ./jobs/task.rbquot;

after_create :bj_send_to_s3 def bj_send_to_s3 Bj.submit quot;./script/runner ./jobs/send.rb #{id}quot; end

Workling # environment config Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new

# task class class ImageWorker < Workling::Base def send_to_s3(options = {}) # put file to S3 end end

# trigger asynchronous job ImageWorker.asynch_send_to_s3(:image_id => 2927)

script/workling_starling_client start

Pitfalls

race conditions

alive, but stalled

VI. some recommendations

general purpose Bj

distributed processing SQS (+ custom worker)

time-scheduled cron (+ rake or script)

speed + scalability Starling/Workling

Thanks! Jonathan Dahl Slides at RailSpikes http://railspikes.com

http://railspikes.com	3051
http://www.slideshare.net	58
http://www.railspikes.com	9
http://static.slideshare.net	3
http://mergejil.blogspot.com	2
http://203.208.37.132	1
http://74.125.77.132	1
file://	1
http://64.233.169.104	1
http://www.techgig.com	1

Asynchronous Processing with Ruby on Rails (RailsConf 2008)

by Jonathan Dahl on Jun 02, 2008

Accessibility

Categories

Tags

Upload Details

Usage Rights

10 Embeds 3,128

Statistics

Asynchronous Processing with Ruby on Rails (RailsConf 2008) — Presentation Transcript