Asynchronous Processing
               Jonathan Dahl




              (and RailSpikes,
                    Slantwise,
   ...
I.
    What is it, and
why should I care?
Wife:   What are you talking about at
        RailsConf this year?

Jon:    Asynchronous Processing


Wife:   [changes sub...
important
     tool
(ahem...)
Related Concepts

• Background Processing
• Parallel Processing
• Distributed Processing
has_attachment :storage => :s3
Image Upload   (   15 seconds)



 Send to S3


  Browser
 Response
Image Upload



 Send to S3    (   15 seconds)


  Browser
 Response
Image Upload



 Send to S3


  Browser
               (   3 seconds)
 Response
Image Upload



 Send to S3


  Browser
 Response
has_attachment :storage => :file_system
Image Upload   (   15 seconds)


  Browser
 Response


 Send to S3
Image Upload


  Browser
               (   3 seconds)
 Response


 Send to S3    (   who cares?)
has_attachment :storage => :s3,
               :thumbnails => {
                 :thumb => '100x100!',
                 :s...
Image Upload

Generate 3 Thumbnails

Send Thumbnail A to S3

Send Thumbnail B to S3

Send Thumbnail C to S3

  Send Origin...
Image Upload

Generate 3 Thumbnails

Send Thumbnail A to S3

Send Thumbnail B to S3

Send Thumbnail C to S3

  Send Origin...
Image Upload

Generate 3 Thumbnails

Send Thumbnail A to S3

Send Thumbnail B to S3

Send Thumbnail C to S3

  Send Origin...
Image Upload

Generate 3 Thumbnails

Send Thumbnail A to S3

Send Thumbnail B to S3

Send Thumbnail C to S3

  Send Origin...
Image Upload

Generate 3 Thumbnails

Send Thumbnail A to S3

Send Thumbnail B to S3

Send Thumbnail C to S3

  Send Origin...
Image Upload

Generate 3 Thumbnails

Send Thumbnail A to S3

Send Thumbnail B to S3

Send Thumbnail C to S3

  Send Origin...
Image Upload

Generate 3 Thumbnails

Send Thumbnail A to S3

Send Thumbnail B to S3

Send Thumbnail C to S3

  Send Origin...
II.
When do I need it?
Time
Request

• Method (GET, POST)
• URI (host, port, path)
• Parameters
Response

• Status (200, 404, 500)
• Metadata (content type, server info,
  etc.)

• Body (xml, html, file)
Resources
Trigger
HTTP trigger - browser request
HTTP trigger - API request

      GET /photos/1.xml HTTP/1.1
      Host: example.com:80
Human trigger - capistrano

      cap staging deploy
Human trigger - rake

      rake db:migrate
Human trigger - console
     $ script/console production
     Loading production environment
     (Rails 2.0.2)
     >> Ph...
No trigger?



    - Send email in 2 hours
    - Sync data at 3:00am PST
    - Notify admin when disk is 90% full
    - Ex...
1. Time
2. Resources
3. Trigger
Concrete examples
• Sending mail
• Transcoding video/audio
• Storing images on S3
• Receiving email
• Synching with outsid...
class Emailer < ActionMailer::ARMailer
Zencoder
             User



         Video Sharing
            Website

                                  Data Storage
 ...
class Photo < ActiveRecord::Base
  after_create :background_s3_upload
  def background_s3_upload
    Bj.submit quot;./scri...
III.
     So how do you
decide what to use?
be seamless
how reliable?
when should it run?
dependencies and
system complexity
scaling
   and/or
performance
IV.
The simple solution:
      fork or thread
Parallel vs. Background
1. Stay within one
     request
2. thread.join
3. ActiveRecord
ActiveRecord::Base.allow_concurrency = true
fire and forget
Spawn


spawn(:method => :fork) do
  # do something
end
1. Time
2. Resources
  3. Trigger
V.
More robust
  solutions
Task Storage
Task Trigger
Task Storage

• task details (what happens?)
• priority
• when to run
Task Trigger

• worker pulling jobs
• time-based
• execute immediately
Task Storage   Task Trigger
   Database       daemon
Message Queue      cron
Task Storage   Task Trigger
   Database       daemon
Message Queue      cron
create_table   quot;jobsquot; do |t|
  t.text       quot;commandquot;
  t.integer    quot;priorityquot;
  t.integer    quo...
create_table   quot;photosquot; do |t|
  t.string     quot;filenamequot;
  t.datetime   quot;created_atquot;
  t.datetime ...
create_table   quot;photosquot; do |t|
  t.string     quot;filenamequot;
  t.datetime   quot;created_atquot;
  t.datetime ...
Task Storage   Task Trigger
   Database       daemon
Message Queue      cron
Task Storage   Task Trigger
   Database       daemon
Message Queue      cron
• Amazon SQS
• Websphere MQ
• Starling
• JMS
• beanstalkd
queue = SQS.get_queue(quot;task_listquot;)
put message


queue.send_message quot;process:2872quot;
receive message


message = queue.receive_message
Starling


starling -h 192.168.1.1 -d
require 'memcache'
starling = MemCache.new('192.168.1.1:22122')

# Put messages onto a queue:
starling.set('my_queue', 123...
Task Storage   Task Trigger
   Database       daemon
Message Queue      cron
storage choice?


• queue: optimized for performance
• database: you’ve already got one
Task Storage   Task Trigger
   Database       daemon
Message Queue      cron
daemon
#!/usr/bin/env ruby
class JobRequester < SimpleDaemon::Base
  def self.start
    loop { Job.process_next }
  end
en...
Task Storage   Task Trigger
   Database       daemon
Message Queue      cron
0 6 * * * script/runner jobs/send_emails.rb
cronedit
require 'cronedit'

CronEdit::Crontab.Add quot;send-emailsquot;, {
  :minute => 0,
  :hour => 6,
  :command => qu...
trigger choice


• process: always running
• cron: as reliable as your operating
  system
BackgroundDRb
class BillingWorker < BackgrounDRb::MetaWorker
  set_worker_name :billing_worker
  def create(args = nil)
  ...
:backgroundrb:
  :ip: 0.0.0.0

:development:
  :backgroundrb:
    :port: 11111
    :log: foreground

:production:
  :backg...
AP4R
def MyController
  def queue
    ap4r.async_to({:action => 'download'}, {:story =>
story.id, :url => params[:url]})
 ...
Bj


Acronym
create_table   quot;bj_jobquot;, :primary_key => quot;bj
  t.text       quot;commandquot;
  t.text       quot;statequot;
 ...
Bj.submit quot;./script/runner ./jobs/task.rbquot;
after_create :bj_send_to_s3

def bj_send_to_s3
  Bj.submit quot;./script/runner ./jobs/send.rb #{id}quot;
end
Workling

# environment config
Workling::Remote.dispatcher =
 Workling::Remote::Runners::StarlingRunner.new
# task class
class ImageWorker < Workling::Base
  def send_to_s3(options = {})
    # put file to S3
  end
end
# trigger asynchronous job
ImageWorker.asynch_send_to_s3(:image_id => 2927)
script/workling_starling_client start
Pitfalls
race conditions
alive, but stalled
VI.
         some
recommendations
general purpose


      Bj
distributed processing


         SQS
  (+ custom worker)
time-scheduled


      cron
(+ rake or script)
speed + scalability


Starling/Workling
Thanks!
 Jonathan Dahl



Slides at RailSpikes
http://railspikes.com
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Upcoming SlideShare
Loading in...5
×

Asynchronous Processing with Ruby on Rails (RailsConf 2008)

23,217

Published on

RailsConf 2008 presentation: Asynchronous Processing with Ruby on Rails

Published in: Technology, Art & Photos
2 Comments
33 Likes
Statistics
Notes
  • Sorry,where download this presentation?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Need : 'Ruby on Rail Developer' at NJ USA,

    Position : Ruby on Rail Developer

    Location : NJ

    Duration : 6+ Months,



    Client is looking for “Ruby on Rail Developer” Experience with Ruby on Rail Developer, Heroku or Engine Factory, ect;

    Client is Looking more of an architect on such development methodology but he/she has to know how to code too.
    Any experience on Heroku or Engine Factory would be a plus.
    Kindly send your resume along with the following information……………………….Please,

    FULL NAME:

    Best contact number:

    Current Location:

    Visa Status (US Citizen/GC):

    Availability (specify the notice period, if so required):

    Expected Hourly / PERANNUM Rate (on W2 Please specify):

    Ready to relocate to NJ, if hired for this position (Yes/No):

    Ready to Face to Face Interview (Yes/No):

    Vijay Bhaskar



    Globalnest LLC

    281 state Route 79, Suite 208

    Morganville, NJ 07751



    Voice: (732)-333-4808

    Fax: (732) 333-5946

    Email: bhaskar@globalnest.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
23,217
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
568
Comments
2
Likes
33
Embeds 0
No embeds

No notes for slide

Asynchronous Processing with Ruby on Rails (RailsConf 2008)

  1. 1. Asynchronous Processing Jonathan Dahl (and RailSpikes, Slantwise, Zencoder, etc.)
  2. 2. I. What is it, and why should I care?
  3. 3. Wife: What are you talking about at RailsConf this year? Jon: Asynchronous Processing Wife: [changes subject]
  4. 4. important tool
  5. 5. (ahem...)
  6. 6. Related Concepts • Background Processing • Parallel Processing • Distributed Processing
  7. 7. has_attachment :storage => :s3
  8. 8. Image Upload ( 15 seconds) Send to S3 Browser Response
  9. 9. Image Upload Send to S3 ( 15 seconds) Browser Response
  10. 10. Image Upload Send to S3 Browser ( 3 seconds) Response
  11. 11. Image Upload Send to S3 Browser Response
  12. 12. has_attachment :storage => :file_system
  13. 13. Image Upload ( 15 seconds) Browser Response Send to S3
  14. 14. Image Upload Browser ( 3 seconds) Response Send to S3 ( who cares?)
  15. 15. has_attachment :storage => :s3, :thumbnails => { :thumb => '100x100!', :small => '240x180>', :medium => '500x500>' }
  16. 16. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  17. 17. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  18. 18. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  19. 19. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  20. 20. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  21. 21. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  22. 22. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
  23. 23. II. When do I need it?
  24. 24. Time
  25. 25. Request • Method (GET, POST) • URI (host, port, path) • Parameters
  26. 26. Response • Status (200, 404, 500) • Metadata (content type, server info, etc.) • Body (xml, html, file)
  27. 27. Resources
  28. 28. Trigger
  29. 29. HTTP trigger - browser request
  30. 30. HTTP trigger - API request GET /photos/1.xml HTTP/1.1 Host: example.com:80
  31. 31. Human trigger - capistrano cap staging deploy
  32. 32. Human trigger - rake rake db:migrate
  33. 33. Human trigger - console $ script/console production Loading production environment (Rails 2.0.2) >> Photo.destroy_all
  34. 34. No trigger? - Send email in 2 hours - Sync data at 3:00am PST - Notify admin when disk is 90% full - Expire sessions that are inactive - Archive records that exceed quota
  35. 35. 1. Time 2. Resources 3. Trigger
  36. 36. Concrete examples • Sending mail • Transcoding video/audio • Storing images on S3 • Receiving email • Synching with outside database • Complex computations
  37. 37. class Emailer < ActionMailer::ARMailer
  38. 38. Zencoder User Video Sharing Website Data Storage (Amazon S3) Zencoder Manager Worker Worker Worker Worker
  39. 39. class Photo < ActiveRecord::Base after_create :background_s3_upload def background_s3_upload Bj.submit quot;./script/runner ./jobs/send_to_s3.rb #{self.id}quot; end end
  40. 40. III. So how do you decide what to use?
  41. 41. be seamless
  42. 42. how reliable?
  43. 43. when should it run?
  44. 44. dependencies and system complexity
  45. 45. scaling and/or performance
  46. 46. IV. The simple solution: fork or thread
  47. 47. Parallel vs. Background
  48. 48. 1. Stay within one request
  49. 49. 2. thread.join
  50. 50. 3. ActiveRecord ActiveRecord::Base.allow_concurrency = true
  51. 51. fire and forget
  52. 52. Spawn spawn(:method => :fork) do # do something end
  53. 53. 1. Time 2. Resources 3. Trigger
  54. 54. V. More robust solutions
  55. 55. Task Storage Task Trigger
  56. 56. Task Storage • task details (what happens?) • priority • when to run
  57. 57. Task Trigger • worker pulling jobs • time-based • execute immediately
  58. 58. Task Storage Task Trigger Database daemon Message Queue cron
  59. 59. Task Storage Task Trigger Database daemon Message Queue cron
  60. 60. create_table quot;jobsquot; do |t| t.text quot;commandquot; t.integer quot;priorityquot; t.integer quot;pidquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.text quot;resultquot; end
  61. 61. create_table quot;photosquot; do |t| t.string quot;filenamequot; t.datetime quot;created_atquot; t.datetime quot;processed_atquot; end
  62. 62. create_table quot;photosquot; do |t| t.string quot;filenamequot; t.datetime quot;created_atquot; t.datetime quot;processed_atquot; end
  63. 63. Task Storage Task Trigger Database daemon Message Queue cron
  64. 64. Task Storage Task Trigger Database daemon Message Queue cron
  65. 65. • Amazon SQS • Websphere MQ • Starling • JMS • beanstalkd
  66. 66. queue = SQS.get_queue(quot;task_listquot;)
  67. 67. put message queue.send_message quot;process:2872quot;
  68. 68. receive message message = queue.receive_message
  69. 69. Starling starling -h 192.168.1.1 -d
  70. 70. require 'memcache' starling = MemCache.new('192.168.1.1:22122') # Put messages onto a queue: starling.set('my_queue', 12345) # Get message from the queue: starling.get('my_queue')
  71. 71. Task Storage Task Trigger Database daemon Message Queue cron
  72. 72. storage choice? • queue: optimized for performance • database: you’ve already got one
  73. 73. Task Storage Task Trigger Database daemon Message Queue cron
  74. 74. daemon #!/usr/bin/env ruby class JobRequester < SimpleDaemon::Base def self.start loop { Job.process_next } end end JobRequester.daemonize
  75. 75. Task Storage Task Trigger Database daemon Message Queue cron
  76. 76. 0 6 * * * script/runner jobs/send_emails.rb
  77. 77. cronedit require 'cronedit' CronEdit::Crontab.Add quot;send-emailsquot;, { :minute => 0, :hour => 6, :command => quot;script/runner jobs/send_emails.rbquot; } CronEdit::Crontab.Remove 'old-task'
  78. 78. trigger choice • process: always running • cron: as reliable as your operating system
  79. 79. BackgroundDRb class BillingWorker < BackgrounDRb::MetaWorker set_worker_name :billing_worker def create(args = nil) # this method is called when worker is loaded for the first time end def charge_customer(customer_id = nil) logger.info 'charging customer now' end end MiddleMan.worker(:billing_worker).charge_customer(current_customer.id)
  80. 80. :backgroundrb: :ip: 0.0.0.0 :development: :backgroundrb: :port: 11111 :log: foreground :production: :backgroundrb: :port: 22222 :lazy_load: true :debug_log: false ./script/backgroundrb start
  81. 81. AP4R def MyController def queue ap4r.async_to({:action => 'download'}, {:story => story.id, :url => params[:url]}) end def download # long-running task end end
  82. 82. Bj Acronym
  83. 83. create_table quot;bj_jobquot;, :primary_key => quot;bj t.text quot;commandquot; t.text quot;statequot; t.integer quot;priorityquot; t.text quot;tagquot; t.integer quot;is_restartablequot; t.text quot;submitterquot; t.text quot;runnerquot; t.integer quot;pidquot; t.datetime quot;submitted_atquot; t.datetime quot;started_atquot; t.datetime quot;finished_atquot; t.text quot;envquot;
  84. 84. Bj.submit quot;./script/runner ./jobs/task.rbquot;
  85. 85. after_create :bj_send_to_s3 def bj_send_to_s3 Bj.submit quot;./script/runner ./jobs/send.rb #{id}quot; end
  86. 86. Workling # environment config Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
  87. 87. # task class class ImageWorker < Workling::Base def send_to_s3(options = {}) # put file to S3 end end
  88. 88. # trigger asynchronous job ImageWorker.asynch_send_to_s3(:image_id => 2927)
  89. 89. script/workling_starling_client start
  90. 90. Pitfalls
  91. 91. race conditions
  92. 92. alive, but stalled
  93. 93. VI. some recommendations
  94. 94. general purpose Bj
  95. 95. distributed processing SQS (+ custom worker)
  96. 96. time-scheduled cron (+ rake or script)
  97. 97. speed + scalability Starling/Workling
  98. 98. Thanks! Jonathan Dahl Slides at RailSpikes http://railspikes.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×