Asynchronous Processing with Ruby on Rails (RailsConf 2008)

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    21 Favorites

    Asynchronous Processing with Ruby on Rails (RailsConf 2008) - Presentation Transcript

    1. Asynchronous Processing Jonathan Dahl (and RailSpikes, Slantwise, Zencoder, etc.)
    2. I. What is it, and why should I care?
    3. Wife: What are you talking about at RailsConf this year? Jon: Asynchronous Processing Wife: [changes subject]
    4. important tool
    5. (ahem...)
    6. Related Concepts • Background Processing • Parallel Processing • Distributed Processing
    7. has_attachment :storage => :s3
    8. Image Upload ( 15 seconds) Send to S3 Browser Response
    9. Image Upload Send to S3 ( 15 seconds) Browser Response
    10. Image Upload Send to S3 Browser ( 3 seconds) Response
    11. Image Upload Send to S3 Browser Response
    12. has_attachment :storage => :file_system
    13. Image Upload ( 15 seconds) Browser Response Send to S3
    14. Image Upload Browser ( 3 seconds) Response Send to S3 ( who cares?)
    15. has_attachment :storage => :s3, :thumbnails => { :thumb => '100x100!', :small => '240x180>', :medium => '500x500>' }
    16. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
    17. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
    18. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
    19. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
    20. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
    21. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
    22. Image Upload Generate 3 Thumbnails Send Thumbnail A to S3 Send Thumbnail B to S3 Send Thumbnail C to S3 Send Original to S3 Browser Response
    23. II. When do I need it?
    24. Time
    25. Request • Method (GET, POST) • URI (host, port, path) • Parameters
    26. Response • Status (200, 404, 500) • Metadata (content type, server info, etc.) • Body (xml, html, file)
    27. Resources
    28. Trigger
    29. HTTP trigger - browser request
    30. HTTP trigger - API request GET /photos/1.xml HTTP/1.1 Host: example.com:80
    31. Human trigger - capistrano cap staging deploy
    32. Human trigger - rake rake db:migrate
    33. Human trigger - console $ script/console production Loading production environment (Rails 2.0.2) >> Photo.destroy_all
    34. No trigger? - Send email in 2 hours - Sync data at 3:00am PST - Notify admin when disk is 90% full - Expire sessions that are inactive - Archive records that exceed quota
    35. 1. Time 2. Resources 3. Trigger
    36. Concrete examples • Sending mail • Transcoding video/audio • Storing images on S3 • Receiving email • Synching with outside database • Complex computations
    37. class Emailer < ActionMailer::ARMailer
    38. Zencoder User Video Sharing Website Data Storage (Amazon S3) Zencoder Manager Worker Worker Worker Worker
    39. class Photo < ActiveRecord::Base after_create :background_s3_upload def background_s3_upload Bj.submit \"./script/runner ./jobs/send_to_s3.rb #{self.id}\" end end
    40. III. So how do you decide what to use?
    41. be seamless
    42. how reliable?
    43. when should it run?
    44. dependencies and system complexity
    45. scaling and/or performance
    46. IV. The simple solution: fork or thread
    47. Parallel vs. Background
    48. 1. Stay within one request
    49. 2. thread.join
    50. 3. ActiveRecord ActiveRecord::Base.allow_concurrency = true
    51. fire and forget
    52. Spawn spawn(:method => :fork) do # do something end
    53. 1. Time 2. Resources 3. Trigger
    54. V. More robust solutions
    55. Task Storage Task Trigger
    56. Task Storage • task details (what happens?) • priority • when to run
    57. Task Trigger • worker pulling jobs • time-based • execute immediately
    58. Task Storage Task Trigger Database daemon Message Queue cron
    59. Task Storage Task Trigger Database daemon Message Queue cron
    60. create_table \"jobs\" do |t| t.text \"command\" t.integer \"priority\" t.integer \"pid\" t.datetime \"submitted_at\" t.datetime \"started_at\" t.datetime \"finished_at\" t.text \"result\" end
    61. create_table \"photos\" do |t| t.string \"filename\" t.datetime \"created_at\" t.datetime \"processed_at\" end
    62. create_table \"photos\" do |t| t.string \"filename\" t.datetime \"created_at\" t.datetime \"processed_at\" end
    63. Task Storage Task Trigger Database daemon Message Queue cron
    64. Task Storage Task Trigger Database daemon Message Queue cron
    65. • Amazon SQS • Websphere MQ • Starling • JMS • beanstalkd
    66. queue = SQS.get_queue(\"task_list\")
    67. put message queue.send_message \"process:2872\"
    68. receive message message = queue.receive_message
    69. Starling starling -h 192.168.1.1 -d
    70. require 'memcache' starling = MemCache.new('192.168.1.1:22122') # Put messages onto a queue: starling.set('my_queue', 12345) # Get message from the queue: starling.get('my_queue')
    71. Task Storage Task Trigger Database daemon Message Queue cron
    72. storage choice? • queue: optimized for performance • database: you’ve already got one
    73. Task Storage Task Trigger Database daemon Message Queue cron
    74. daemon #!/usr/bin/env ruby class JobRequester < SimpleDaemon::Base def self.start loop { Job.process_next } end end JobRequester.daemonize
    75. Task Storage Task Trigger Database daemon Message Queue cron
    76. 0 6 * * * script/runner jobs/send_emails.rb
    77. cronedit require 'cronedit' CronEdit::Crontab.Add \"send-emails\", { :minute => 0, :hour => 6, :command => \"script/runner jobs/send_emails.rb\" } CronEdit::Crontab.Remove 'old-task'
    78. trigger choice • process: always running • cron: as reliable as your operating system
    79. BackgroundDRb class BillingWorker < BackgrounDRb::MetaWorker set_worker_name :billing_worker def create(args = nil) # this method is called when worker is loaded for the first time end def charge_customer(customer_id = nil) logger.info 'charging customer now' end end MiddleMan.worker(:billing_worker).charge_customer(current_customer.id)
    80. :backgroundrb: :ip: 0.0.0.0 :development: :backgroundrb: :port: 11111 :log: foreground :production: :backgroundrb: :port: 22222 :lazy_load: true :debug_log: false ./script/backgroundrb start
    81. AP4R def MyController def queue ap4r.async_to({:action => 'download'}, {:story => story.id, :url => params[:url]}) end def download # long-running task end end
    82. Bj Acronym
    83. create_table \"bj_job\", :primary_key => \"bj t.text \"command\" t.text \"state\" t.integer \"priority\" t.text \"tag\" t.integer \"is_restartable\" t.text \"submitter\" t.text \"runner\" t.integer \"pid\" t.datetime \"submitted_at\" t.datetime \"started_at\" t.datetime \"finished_at\" t.text \"env\"
    84. Bj.submit \"./script/runner ./jobs/task.rb\"
    85. after_create :bj_send_to_s3 def bj_send_to_s3 Bj.submit \"./script/runner ./jobs/send.rb #{id}\" end
    86. Workling # environment config Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
    87. # task class class ImageWorker < Workling::Base def send_to_s3(options = {}) # put file to S3 end end
    88. # trigger asynchronous job ImageWorker.asynch_send_to_s3(:image_id => 2927)
    89. script/workling_starling_client start
    90. Pitfalls
    91. race conditions
    92. alive, but stalled
    93. VI. some recommendations
    94. general purpose Bj
    95. distributed processing SQS (+ custom worker)
    96. time-scheduled cron (+ rake or script)
    97. speed + scalability Starling/Workling
    98. Thanks! Jonathan Dahl Slides at RailSpikes http://railspikes.com

    + jondahljondahl, 2 years ago

    custom

    11012 views, 21 favs, 8 embeds more stats

    RailsConf 2008 presentation: Asynchronous Processin more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 11012
      • 9741 on SlideShare
      • 1271 from embeds
    • Comments 0
    • Favorites 21
    • Downloads 412
    Most viewed embeds
    • 1253 views on http://railspikes.com
    • 9 views on http://www.railspikes.com
    • 3 views on http://static.slideshare.net
    • 2 views on http://mergejil.blogspot.com
    • 1 views on http://64.233.169.104

    more

    All embeds
    • 1253 views on http://railspikes.com
    • 9 views on http://www.railspikes.com
    • 3 views on http://static.slideshare.net
    • 2 views on http://mergejil.blogspot.com
    • 1 views on http://64.233.169.104
    • 1 views on file://
    • 1 views on http://74.125.77.132
    • 1 views on http://203.208.37.132

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories