Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Asynchronous Processing with Ruby on Rails (RailsConf 2008)

37,246 views

Published on

RailsConf 2008 presentation: Asynchronous Processing with Ruby on Rails

Published in: Technology, Art & Photos
  • Sorry,where download this presentation?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Need : 'Ruby on Rail Developer' at NJ USA,

    Position : Ruby on Rail Developer

    Location : NJ

    Duration : 6+ Months,



    Client is looking for “Ruby on Rail Developer” Experience with Ruby on Rail Developer, Heroku or Engine Factory, ect;

    Client is Looking more of an architect on such development methodology but he/she has to know how to code too.
    Any experience on Heroku or Engine Factory would be a plus.
    Kindly send your resume along with the following information……………………….Please,

    FULL NAME:

    Best contact number:

    Current Location:

    Visa Status (US Citizen/GC):

    Availability (specify the notice period, if so required):

    Expected Hourly / PERANNUM Rate (on W2 Please specify):

    Ready to relocate to NJ, if hired for this position (Yes/No):

    Ready to Face to Face Interview (Yes/No):

    Vijay Bhaskar



    Globalnest LLC

    281 state Route 79, Suite 208

    Morganville, NJ 07751



    Voice: (732)-333-4808

    Fax: (732) 333-5946

    Email: bhaskar@globalnest.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Asynchronous Processing with Ruby on Rails (RailsConf 2008)

  1. Asynchronous Processing Jonathan Dahl (and RailSpikes, Slantwise, Zencoder, etc.)
  2. I. What is it, and why should I care?
  3. Wife: What are you talking about at RailsConf this year? Jon: Asynchronous Processing Wife: [changes subject]
  4. important tool
  5. (ahem...)
  6. Related Concepts • Background Processing • Parallel Processing • Distributed Processing
  7. has_attachment :storage => :s3
  8. Image Upload ( 15 seconds) Send to S3 Browser Response
  9. Image Upload Send to S3 ( 15 seconds) Browser Response
  10. Image Upload Send to S3 Browser ( 3 seconds) Response
  11. Image Upload Send to S3 Browser Response
  12. has_attachment :storage => :file_system
  13. Image Upload ( 15 seconds) Browser Response Send to S3
  14. Image Upload Browser ( 3 seconds) Response Send to S3 ( who cares?)
  15. has_attachment :storage => :s3, :thumbnails => { :thumb => '100x100!', :small => '240x180>', :medium => '500x500>' }
  16. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  17. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  18. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  19. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  20. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  21. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  22. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  23. II. When do I need it?
  24. Time
  25. Request • Method (GET, POST) • URI (host, port, path) • Parameters
  26. Response • Status (200, 404, 500) • Metadata (content type, server info, etc.) • Body (xml, html, file)
  27. Resources
  28. Trigger
  29. HTTP trigger - browser request
  30. HTTP trigger - API request GET /photos/1.xml HTTP/1.1 Host: example.com:80
  31. Human trigger - capistrano cap staging deploy
  32. Human trigger - rake rake db:migrate
  33. Human trigger - console $ script/console production Loading production environment (Rails 2.0.2) >> Photo.destroy_all
  34. No trigger? - Send email in 2 hours - Sync data at 3:00am PST - Notify admin when disk is 90% full - Expire sessions that are inactive - Archive records that exceed quota
  35. 1. Time 2. Resources 3. Trigger
  36. Concrete examples • Sending mail • Transcoding video/audio • Storing images on S3 • Receiving email • Synching with outside database • Complex computations
  37. class Emailer < ActionMailer::ARMailer
  38. Zencoder User Video Sharing Website Data Storage (Amazon S3) Zencoder Manager Worker Worker Worker Worker
  39. class Photo < ActiveRecord::Base after_create :background_s3_upload def background_s3_upload Bj.submit quot;./script/runner ./jobs/send_to_s3.rb #{self.id}quot; end end
  40. III. So how do you decide what to use?
  41. be seamless
  42. how reliable?
  43. when should it run?
  44. dependencies and system complexity
  45. scaling and/or performance
  46. IV. The simple solution: fork or thread
  47. Parallel vs. Background
  48. 1. Stay within one request
  49. 2. thread.join
  50. 3. ActiveRecord ActiveRecord::Base.allow_concurrency = true
  51. fire and forget
  52. Spawn spawn(:method => :fork) do # do something end
  53. 1. Time 2. Resources 3. Trigger
  54. V. More robust solutions
  55. Task Storage Task Trigger
  56. Task Storage • task details (what happens?) • priority • when to run
  57. Task Trigger • worker pulling jobs • time-based • execute immediately
  58. Task Storage Task Trigger Database daemon Message Queue cron
  59. Task Storage Task Trigger Database daemon Message Queue cron
  60. create_table quot;jobsquot; do |t| t.text quot;commandquot; t.integer quot;priorityquot; t.integer quot;pidquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.text quot;resultquot; end
  61. create_table quot;photosquot; do |t| t.string quot;filenamequot; t.datetime quot;created_atquot; t.datetime quot;processed_atquot; end
  62. create_table quot;photosquot; do |t| t.string quot;filenamequot; t.datetime quot;created_atquot; t.datetime quot;processed_atquot; end
  63. Task Storage Task Trigger Database daemon Message Queue cron
  64. Task Storage Task Trigger Database daemon Message Queue cron
  65. • Amazon SQS • Websphere MQ • Starling • JMS • beanstalkd
  66. queue = SQS.get_queue(quot;task_listquot;)
  67. put message queue.send_message quot;process:2872quot;
  68. receive message message = queue.receive_message
  69. Starling starling -h 192.168.1.1 -d
  70. require 'memcache' starling = MemCache.new('192.168.1.1:22122') # Put messages onto a queue: starling.set('my_queue', 12345) # Get message from the queue: starling.get('my_queue')
  71. Task Storage Task Trigger Database daemon Message Queue cron
  72. storage choice? • queue: optimized for performance • database: you’ve already got one
  73. Task Storage Task Trigger Database daemon Message Queue cron
  74. daemon #!/usr/bin/env ruby class JobRequester < SimpleDaemon::Base def self.start loop { Job.process_next } end end JobRequester.daemonize
  75. Task Storage Task Trigger Database daemon Message Queue cron
  76. 0 6 * * * script/runner jobs/send_emails.rb
  77. cronedit require 'cronedit' CronEdit::Crontab.Add quot;send-emailsquot;, { :minute => 0, :hour => 6, :command => quot;script/runner jobs/send_emails.rbquot; } CronEdit::Crontab.Remove 'old-task'
  78. trigger choice • process: always running • cron: as reliable as your operating system
  79. BackgroundDRb class BillingWorker < BackgrounDRb::MetaWorker set_worker_name :billing_worker def create(args = nil) # this method is called when worker is loaded for the first time end def charge_customer(customer_id = nil) logger.info 'charging customer now' end end MiddleMan.worker(:billing_worker).charge_customer(current_customer.id)
  80. :backgroundrb: :ip: 0.0.0.0 :development: :backgroundrb: :port: 11111 :log: foreground :production: :backgroundrb: :port: 22222 :lazy_load: true :debug_log: false ./script/backgroundrb start
  81. AP4R def MyController def queue ap4r.async_to({:action => 'download'}, {:story => story.id, :url => params[:url]}) end def download # long-running task end end
  82. Bj Acronym
  83. create_table quot;bj_jobquot;, :primary_key => quot;bj t.text quot;commandquot; t.text quot;statequot; t.integer quot;priorityquot; t.text quot;tagquot; t.integer quot;is_restartablequot; t.text quot;submitterquot; t.text quot;runnerquot; t.integer quot;pidquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.text quot;envquot;
  84. Bj.submit quot;./script/runner ./jobs/task.rbquot;
  85. after_create :bj_send_to_s3 def bj_send_to_s3 Bj.submit quot;./script/runner ./jobs/send.rb #{id}quot; end
  86. Workling # environment config Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
  87. # task class class ImageWorker < Workling::Base def send_to_s3(options = {}) # put file to S3 end end
  88. # trigger asynchronous job ImageWorker.asynch_send_to_s3(:image_id => 2927)
  89. script/workling_starling_client start
  90. Pitfalls
  91. race conditions
  92. alive, but stalled
  93. VI. some recommendations
  94. general purpose Bj
  95. distributed processing SQS (+ custom worker)
  96. time-scheduled cron (+ rake or script)
  97. speed + scalability Starling/Workling
  98. Thanks! Jonathan Dahl Slides at RailSpikes http://railspikes.com

×