Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Upcoming SlideShare
Loading in...5
×
 

Asynchronous Processing with Ruby on Rails (RailsConf 2008)

on

  • 27,400 views

RailsConf 2008 presentation: Asynchronous Processing with Ruby on Rails

RailsConf 2008 presentation: Asynchronous Processing with Ruby on Rails

Statistics

Views

Total Views
27,400
Views on SlideShare
23,164
Embed Views
4,236

Actions

Likes
33
Downloads
565
Comments
2

12 Embeds 4,236

http://railspikes.com 4157
http://www.slideshare.net 58
http://www.railspikes.com 9
http://static.slideshare.net 3
http://mergejil.blogspot.com 2
http://74.125.77.132 1
file:// 1
http://203.208.37.132 1
http://64.233.169.104 1
http://www.techgig.com 1
http://translate.googleusercontent.com 1
http://seekr-artemis.heroku.com 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Sorry,where download this presentation?
    Are you sure you want to
    Your message goes here
    Processing…
  • Need : 'Ruby on Rail Developer' at NJ USA,

    Position : Ruby on Rail Developer

    Location : NJ

    Duration : 6+ Months,



    Client is looking for “Ruby on Rail Developer” Experience with Ruby on Rail Developer, Heroku or Engine Factory, ect;

    Client is Looking more of an architect on such development methodology but he/she has to know how to code too.
    Any experience on Heroku or Engine Factory would be a plus.
    Kindly send your resume along with the following information……………………….Please,

    FULL NAME:

    Best contact number:

    Current Location:

    Visa Status (US Citizen/GC):

    Availability (specify the notice period, if so required):

    Expected Hourly / PERANNUM Rate (on W2 Please specify):

    Ready to relocate to NJ, if hired for this position (Yes/No):

    Ready to Face to Face Interview (Yes/No):

    Vijay Bhaskar



    Globalnest LLC

    281 state Route 79, Suite 208

    Morganville, NJ 07751



    Voice: (732)-333-4808

    Fax: (732) 333-5946

    Email: bhaskar@globalnest.com
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Asynchronous Processing with Ruby on Rails (RailsConf 2008) Asynchronous Processing with Ruby on Rails (RailsConf 2008) Presentation Transcript

  • Asynchronous Processing Jonathan Dahl (and RailSpikes, Slantwise, Zencoder, etc.)
  • I. What is it, and why should I care?
  • Wife: What are you talking about at RailsConf this year? Jon: Asynchronous Processing Wife: [changes subject]
  • important tool
  • (ahem...)
  • Related Concepts • Background Processing • Parallel Processing • Distributed Processing
  • has_attachment :storage => :s3
  • Image Upload ( 15 seconds) Send to S3 Browser Response
  • Image Upload Send to S3 ( 15 seconds) Browser Response
  • Image Upload Send to S3 Browser ( 3 seconds) Response
  • Image Upload Send to S3 Browser Response
  • has_attachment :storage => :file_system
  • Image Upload ( 15 seconds) Browser Response Send to S3
  • Image Upload Browser ( 3 seconds) Response Send to S3 ( who cares?)
  • has_attachment :storage => :s3, :thumbnails => { :thumb => '100x100!', :small => '240x180>', :medium => '500x500>' }
  • Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  • Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  • Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  • Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  • Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  • Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  • Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  • II. When do I need it?
  • Time
  • Request • Method (GET, POST) • URI (host, port, path) • Parameters
  • Response • Status (200, 404, 500) • Metadata (content type, server info, etc.) • Body (xml, html, file)
  • Resources
  • Trigger
  • HTTP trigger - browser request
  • HTTP trigger - API request GET /photos/1.xml HTTP/1.1 Host: example.com:80
  • Human trigger - capistrano cap staging deploy
  • Human trigger - rake rake db:migrate
  • Human trigger - console $ script/console production Loading production environment (Rails 2.0.2) >> Photo.destroy_all
  • No trigger? - Send email in 2 hours - Sync data at 3:00am PST - Notify admin when disk is 90% full - Expire sessions that are inactive - Archive records that exceed quota
  • 1. Time 2. Resources 3. Trigger
  • Concrete examples • Sending mail • Transcoding video/audio • Storing images on S3 • Receiving email • Synching with outside database • Complex computations
  • class Emailer < ActionMailer::ARMailer
  • Zencoder User Video Sharing Website Data Storage (Amazon S3) Zencoder Manager Worker Worker Worker Worker
  • class Photo < ActiveRecord::Base after_create :background_s3_upload def background_s3_upload Bj.submit quot;./script/runner ./jobs/send_to_s3.rb #{self.id}quot; end end
  • III. So how do you decide what to use?
  • be seamless
  • how reliable?
  • when should it run?
  • dependencies and system complexity
  • scaling and/or performance
  • IV. The simple solution: fork or thread
  • Parallel vs. Background
  • 1. Stay within one request
  • 2. thread.join
  • 3. ActiveRecord ActiveRecord::Base.allow_concurrency = true
  • fire and forget
  • Spawn spawn(:method => :fork) do # do something end
  • 1. Time 2. Resources 3. Trigger
  • V. More robust solutions
  • Task Storage Task Trigger
  • Task Storage • task details (what happens?) • priority • when to run
  • Task Trigger • worker pulling jobs • time-based • execute immediately
  • Task Storage Task Trigger Database daemon Message Queue cron
  • Task Storage Task Trigger Database daemon Message Queue cron
  • create_table quot;jobsquot; do |t| t.text quot;commandquot; t.integer quot;priorityquot; t.integer quot;pidquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.text quot;resultquot; end
  • create_table quot;photosquot; do |t| t.string quot;filenamequot; t.datetime quot;created_atquot; t.datetime quot;processed_atquot; end
  • create_table quot;photosquot; do |t| t.string quot;filenamequot; t.datetime quot;created_atquot; t.datetime quot;processed_atquot; end
  • Task Storage Task Trigger Database daemon Message Queue cron
  • Task Storage Task Trigger Database daemon Message Queue cron
  • • Amazon SQS • Websphere MQ • Starling • JMS • beanstalkd
  • queue = SQS.get_queue(quot;task_listquot;)
  • put message queue.send_message quot;process:2872quot;
  • receive message message = queue.receive_message
  • Starling starling -h 192.168.1.1 -d
  • require 'memcache' starling = MemCache.new('192.168.1.1:22122') # Put messages onto a queue: starling.set('my_queue', 12345) # Get message from the queue: starling.get('my_queue')
  • Task Storage Task Trigger Database daemon Message Queue cron
  • storage choice? • queue: optimized for performance • database: you’ve already got one
  • Task Storage Task Trigger Database daemon Message Queue cron
  • daemon #!/usr/bin/env ruby class JobRequester < SimpleDaemon::Base def self.start loop { Job.process_next } end end JobRequester.daemonize
  • Task Storage Task Trigger Database daemon Message Queue cron
  • 0 6 * * * script/runner jobs/send_emails.rb
  • cronedit require 'cronedit' CronEdit::Crontab.Add quot;send-emailsquot;, { :minute => 0, :hour => 6, :command => quot;script/runner jobs/send_emails.rbquot; } CronEdit::Crontab.Remove 'old-task'
  • trigger choice • process: always running • cron: as reliable as your operating system
  • BackgroundDRb class BillingWorker < BackgrounDRb::MetaWorker set_worker_name :billing_worker def create(args = nil) # this method is called when worker is loaded for the first time end def charge_customer(customer_id = nil) logger.info 'charging customer now' end end MiddleMan.worker(:billing_worker).charge_customer(current_customer.id)
  • :backgroundrb: :ip: 0.0.0.0 :development: :backgroundrb: :port: 11111 :log: foreground :production: :backgroundrb: :port: 22222 :lazy_load: true :debug_log: false ./script/backgroundrb start
  • AP4R def MyController def queue ap4r.async_to({:action => 'download'}, {:story => story.id, :url => params[:url]}) end def download # long-running task end end
  • Bj Acronym
  • create_table quot;bj_jobquot;, :primary_key => quot;bj t.text quot;commandquot; t.text quot;statequot; t.integer quot;priorityquot; t.text quot;tagquot; t.integer quot;is_restartablequot; t.text quot;submitterquot; t.text quot;runnerquot; t.integer quot;pidquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.text quot;envquot;
  • Bj.submit quot;./script/runner ./jobs/task.rbquot;
  • after_create :bj_send_to_s3 def bj_send_to_s3 Bj.submit quot;./script/runner ./jobs/send.rb #{id}quot; end
  • Workling # environment config Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
  • # task class class ImageWorker < Workling::Base def send_to_s3(options = {}) # put file to S3 end end
  • # trigger asynchronous job ImageWorker.asynch_send_to_s3(:image_id => 2927)
  • script/workling_starling_client start
  • Pitfalls
  • race conditions
  • alive, but stalled
  • VI. some recommendations
  • general purpose Bj
  • distributed processing SQS (+ custom worker)
  • time-scheduled cron (+ rake or script)
  • speed + scalability Starling/Workling
  • Thanks! Jonathan Dahl Slides at RailSpikes http://railspikes.com