Resque
High-performance asynchronous task queuing and
                  processing
Resque
High-performance asynchronous task queuing and
                  processing
Resque
High-performance asynchronous task queuing and
                  processing
Experts
Experts
‘Cause you might build the next GitHub
Three Things
1. Background
Why run background processes at all?
2. Delayed::Job
Because it’s the foundation for Resque
3. Resque
Process different.
1. Background

   Why?
1. Background

It’s all about the ms
1. Background

    It’s all about the ms
• Resize an image
1. Background

    It’s all about the ms
• Resize an image
• Send an e-mail
1. Background

    It’s all about the ms
• Resize an image
• Send an e-mail
• Communicate with a web service
1. Background

    It’s all about the ms
• Resize an image
• Send an e-mail
• Communicate with a web service
• Geocode a l...
1. Background

Do something later
    than now
1. Background
1. Background

 Webpages.
1. Background

       Webpages.
Rails serves webpages.
                     Memories. You're
                      talkin ...
1. Background
1. Background

1. Receive requests
1. Background

1. Receive requests
 2. Send responses
1. Background

Everything else can go
      elsewhere
1. Background

  Where?
1. Background

         Where?
• CRON
1. Background

               Where?
• CRON
• Beanstalkd
1. Background

               Where?
• CRON
• Beanstalkd
• Workling
1. Background

           Where?
• CRON
• Beanstalkd
• Workling
• BackgroundRB
1. Background

WTF my tasks r fail?!1
1. Background

WTF my tasks r fail?!1
• Silent failure
1. Background

WTF my tasks r fail?!1
• Silent failure
• No status checks
1. Background

WTF my tasks r fail?!1
• Silent failure
• No status checks
• Unstable processes
2. Delayed::Job

To the rescue!
2. Delayed::Job

To the rescue!
2. Delayed::Job

ActiveRecord::Base’d
2. Delayed::Job

 ActiveRecord::Base’d

• Persistence! At last!
2. Delayed::Job

 ActiveRecord::Base’d

• Persistence! At last!
• Ease of use! At last!
2. Delayed::Job

 How easy?
2. Delayed::Job

            Super easy.
# without delayed_job
Notifier.deliver_signup(@user)

# with delayed_job
Notifier...
2. Delayed::Job

Where do they go?
2. Delayed::Job
mysql> SELECT * FROM delayed_jobs;
*************************** 1. row ***************************
        ...
2. Delayed::Job

What do you do with
      them?
2. Delayed::Job

  What do you do with
        them?
$ script/delayed_job start
2. Delayed::Job

     Workers.
2. Delayed::Job

     Workers.
    • Live in their own thread
2. Delayed::Job

     Workers.
    • Live in their own thread
    • Query the database for jobs
2. Delayed::Job

     Workers.
    • Live in their own thread
    • Query the database for jobs
    • Perform the long-run...
The Old Way
The New Way


Delayed::Job
2. Delayed::Job


 Totally rad.
3. Resque

Isn’t Delayed::Job
      enough?
3. Resque

Isn’t Delayed::Job
      enough?
3. Resque


Apparently not.
3. Resque

   Resque is some
Enterprise Grade shit.
3. Resque


Runs on Redis
3. Resque


Runs on Redis
Tangent Alert: Redis!

Redis is a Key-Value
        Store
      Just like Memcached
Tangent Alert: Redis!

    Persistence
      Unlike Memcached
Tangent Alert: Redis!

               Scalability
           The Gem load balances for you


Resque.redis = DistRedis.new(...
Tangent Alert: Redis!

      Basically:
 Fast & Distributed
Tangent Alert: Redis!

Also, Rails-independent
No ActiveRecord/ActiveSupport means you can run
Resque (on Redis) in Sinatr...
3. Resque

Better Workers
3. Resque

      Better Workers
• Forked, sandboxed threads
3. Resque

      Better Workers
• Forked, sandboxed threads
• Failing tasks are tracked
3. Resque

      Better Workers
• Forked, sandboxed threads
• Failing tasks are tracked
• Kills timeouts
3. Resque

                 Queueing
          Separate queues for separate workers



fileserver$ QUEUE=zip_files rake re...
3. Resque

         No Marshaling
          Because it should be functional



Resque.enqueue(SignupNotifier, @user.id)
3. Resque

         No Marshaling
class SignupNotifier
  def self.perform(user_id)
    Notifier.deliver_signup(User.find(u...
3. Resque

                UI
That’s right. It has its own interface.
3. Resque

So... why?
3. Resque

           So... why?
• Many background tasks
3. Resque

            So... why?
• Many background tasks
• Distributed task servers
3. Resque

            So... why?
• Many background tasks
• Distributed task servers
• Stable workers
3. Resque

Trouble Points
3. Resque

      Trouble Points
• Complex and expensive
3. Resque

        Trouble Points
• Complex and expensive
• No scheduling or priority like in D::J
3. Resque

       Trouble Points
• Complex and expensive
• No scheduling or priority like in D::J
• Asynchronous Redis wri...
3. Resque

Live coding FTW!
    Here goes nothing...
Thnaks
Thanks for taking the time to listen to me blather
                 about Resque!
Me!
  Github:    http://github.com/flipsasser
Homepage:    http://x451.com
  Twitter:   @flipsasser
   E-mail:   flip@intride...
Bmore on Rails

  Meetups - 2nd Tuesday
  #OSHN - 4th Tuesday

http://www.meetup.com/bmore-on-rails/
Upcoming SlideShare
Loading in...5
×

Asynchronous Awesome

2,373

Published on

This is the presentation I gave to DCRUG on Resque. Read the presenter notes or it may not make a whole lot of sense.

Published in: Technology, Business
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,373
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
43
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide
  • Okay, my talk is called “...” which I think is pretty descriptive. Except that it doesn’t really demonstrate the pure awesome is that is Resque, so really, we’ll call it...
  • Yeah.
  • So, why should you care about what I’m going to say in the next 20 minutes? Because when you walk away from this room, you’ll all be experts at building a network of high-performance and fault-tolerant workers.
  • So we’re going to talk about three things today
  • First, we’re going to do a general background of background processing
  • Next, we’ll cover the basic concepts of Resque that are derived from Delayed::Job
  • Last, we’ll get into Resque, and how it’s different.
  • So, why run background tasks at all?
  • - Resize
    - Etc
    That’s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
  • - Resize
    - Etc
    That’s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
  • - Resize
    - Etc
    That’s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
  • - Resize
    - Etc
    That’s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
  • Let’s do something later than now. If my application doesn’t NEED to do this stuff right away, don’t keep the request tied up. After all, what IS a request for?
  • Webpages
    In case you’d forgotten
    - Rails serves webpages.
    It’s not built to process images or send out mass e-mails; it’s built to do two things:
  • Webpages
    In case you’d forgotten
    - Rails serves webpages.
    It’s not built to process images or send out mass e-mails; it’s built to do two things:
  • Webpages
    In case you’d forgotten
    - Rails serves webpages.
    It’s not built to process images or send out mass e-mails; it’s built to do two things:
  • Webpages
    In case you’d forgotten
    - Rails serves webpages.
    It’s not built to process images or send out mass e-mails; it’s built to do two things:
  • - Receive requests
    - Send responses
  • - Receive requests
    - Send responses
  • Everything else can go elsewhere
  • So... where should it go? There are a whole bunch of options.
    - CRON
    - Etc.
    But sometimes those setups have problems
  • So... where should it go? There are a whole bunch of options.
    - CRON
    - Etc.
    But sometimes those setups have problems
  • So... where should it go? There are a whole bunch of options.
    - CRON
    - Etc.
    But sometimes those setups have problems
  • So... where should it go? There are a whole bunch of options.
    - CRON
    - Etc.
    But sometimes those setups have problems
  • - They can fail without telling you why
    - You can’t get the status of one of a worker
    - Ruby processes can be unstable and crash-ey
    So maybe you need something a little bit better
  • - They can fail without telling you why
    - You can’t get the status of one of a worker
    - Ruby processes can be unstable and crash-ey
    So maybe you need something a little bit better
  • - They can fail without telling you why
    - You can’t get the status of one of a worker
    - Ruby processes can be unstable and crash-ey
    So maybe you need something a little bit better
  • Delayed::Job to the rescue!
  • A Delayed::Job is an ActiveRecord model, so it lives in the database.
    - You get persistence baked right in
    - It’s unbelievably easy to use

  • A Delayed::Job is an ActiveRecord model, so it lives in the database.
    - You get persistence baked right in
    - It’s unbelievably easy to use

  • How easy?
  • Super easy. I pulled this straight from the github page. This code shows how you would create a Delayed::Job
  • But where does that job go?
  • Here - your database. Notice that it simply marshals the object and stuffs in the database.
  • So what do you do with these jobs in the database?
    - Run the delayed_job script
    This will create...
  • Workers. They’re separate Ruby processes that
    - Live in their own thread
    - Find jobs in the database
    - Perform them separately from the request/response cycle
    They boot up your Rails environment to do so.
  • Workers. They’re separate Ruby processes that
    - Live in their own thread
    - Find jobs in the database
    - Perform them separately from the request/response cycle
    They boot up your Rails environment to do so.
  • Workers. They’re separate Ruby processes that
    - Live in their own thread
    - Find jobs in the database
    - Perform them separately from the request/response cycle
    They boot up your Rails environment to do so.
  • So before, you had requests coming from a browser into Rails, which dispatches to various services, gets results, and sends something back to the browser
  • Now you have Rails storing the jobs for later, and Delayed::Job comes along and does the processing Rails isn’t good at doing anyway.
  • So Delayed::Job is totally rad. It handles most of what you might want it to handle.
  • So when we start in with Resque, you might wonder, “Isn’t Delayed::Job enough?”
  • Apparently not. There are some situations where Delayed::Job isn’t sufficient.
  • Basically, Resque is enterprise grade shit.
  • So the first big difference between Resque and Delayed::Job is that Resque doesn’t rely on ActiveRecord. It runs on something called Redis, which probably deserves
    - its own tangent
  • Redis is an in-memory Key-Value Store similar to Memcached. But it offers
  • Redis is an in-memory Key-Value Store similar to Memcached. But it offers
  • Redis is an in-memory Key-Value Store similar to Memcached. But it offers
  • Redis is an in-memory Key-Value Store similar to Memcached. But it offers
  • Redis is an in-memory Key-Value Store similar to Memcached. But it offers
  • Redis is an in-memory Key-Value Store similar to Memcached. But it offers
  • Persistence! So you can reboot your server and Redis will reload your original queued jobs. It also is great at
  • Scalability! The Ruby gem will load-balance tasks across multiple Redis servers for you, meaning you spend less time configuring and more time queueing
  • So Redis makes Resque incredibly fast and easily distributed.
  • Also, using Redis means you don’t have to worry about ActiveRecord - so you can skip Rails entirely
  • The next big difference is Resque’s workers
    - When a worker finds a job to process, it creates a fork that it can monitor to process the job
    If a fork spirals out, it doesn’t take the worker with it
    - Workers are capable of moving failed jobs into the failed List and moving on with their lives
    - Workers are also capable of timing their own processes out and failing them
  • The next big difference is Resque’s workers
    - When a worker finds a job to process, it creates a fork that it can monitor to process the job
    If a fork spirals out, it doesn’t take the worker with it
    - Workers are capable of moving failed jobs into the failed List and moving on with their lives
    - Workers are also capable of timing their own processes out and failing them
  • The next big difference is Resque’s workers
    - When a worker finds a job to process, it creates a fork that it can monitor to process the job
    If a fork spirals out, it doesn’t take the worker with it
    - Workers are capable of moving failed jobs into the failed List and moving on with their lives
    - Workers are also capable of timing their own processes out and failing them
  • Resque also supports queues, so different types of tasks can be divided by disparate workers. A file server, for example, can have a worker that only zips files - while a web server can warm image caches.
  • Resque also skips the marshaling step, and instead opts for a class and a JSON-ifiable argument list. This will keep all instances current
  • The class’ “perform” method is called with the arguments you enqueued it with
  • Resque comes with its own interface for monitoring queues and tasks, and reviewing failed tasks.
  • So why use Resque? Use Resque if
    - You have many, many, many background tasks
    - You need to have different servers performing different types tasks
    - You need bullet-proof workers
  • So why use Resque? Use Resque if
    - You have many, many, many background tasks
    - You need to have different servers performing different types tasks
    - You need bullet-proof workers
  • So why use Resque? Use Resque if
    - You have many, many, many background tasks
    - You need to have different servers performing different types tasks
    - You need bullet-proof workers
  • Trouble
    - Complex / expensive
    - No scheduling or priority
    - Asynchronous Redis writes don’t guarantee persistence
  • Trouble
    - Complex / expensive
    - No scheduling or priority
    - Asynchronous Redis writes don’t guarantee persistence
  • Trouble
    - Complex / expensive
    - No scheduling or priority
    - Asynchronous Redis writes don’t guarantee persistence
  • Okay, if we have time I’ll show off Resque’s UI and do a quick comparison of it to D::J
  • Thanks for listening!
  • This is me - you can find me on Github or Twitter, and if you’re interested I’ll post these slides on Twitter.
  • Last but not least, I’m a visiting emissary from B’more on Rails, and I’d like to personally invite all of you to come up to our Meetups (on the 2nd tuesday of every month) and our Open Source Hack Nights (on the 4th Tuesday of every month).
  • Asynchronous Awesome

    1. 1. Resque High-performance asynchronous task queuing and processing
    2. 2. Resque High-performance asynchronous task queuing and processing
    3. 3. Resque High-performance asynchronous task queuing and processing
    4. 4. Experts
    5. 5. Experts ‘Cause you might build the next GitHub
    6. 6. Three Things
    7. 7. 1. Background Why run background processes at all?
    8. 8. 2. Delayed::Job Because it’s the foundation for Resque
    9. 9. 3. Resque Process different.
    10. 10. 1. Background Why?
    11. 11. 1. Background It’s all about the ms
    12. 12. 1. Background It’s all about the ms • Resize an image
    13. 13. 1. Background It’s all about the ms • Resize an image • Send an e-mail
    14. 14. 1. Background It’s all about the ms • Resize an image • Send an e-mail • Communicate with a web service
    15. 15. 1. Background It’s all about the ms • Resize an image • Send an e-mail • Communicate with a web service • Geocode a location
    16. 16. 1. Background Do something later than now
    17. 17. 1. Background
    18. 18. 1. Background Webpages.
    19. 19. 1. Background Webpages. Rails serves webpages. Memories. You're talkin about memories.
    20. 20. 1. Background
    21. 21. 1. Background 1. Receive requests
    22. 22. 1. Background 1. Receive requests 2. Send responses
    23. 23. 1. Background Everything else can go elsewhere
    24. 24. 1. Background Where?
    25. 25. 1. Background Where? • CRON
    26. 26. 1. Background Where? • CRON • Beanstalkd
    27. 27. 1. Background Where? • CRON • Beanstalkd • Workling
    28. 28. 1. Background Where? • CRON • Beanstalkd • Workling • BackgroundRB
    29. 29. 1. Background WTF my tasks r fail?!1
    30. 30. 1. Background WTF my tasks r fail?!1 • Silent failure
    31. 31. 1. Background WTF my tasks r fail?!1 • Silent failure • No status checks
    32. 32. 1. Background WTF my tasks r fail?!1 • Silent failure • No status checks • Unstable processes
    33. 33. 2. Delayed::Job To the rescue!
    34. 34. 2. Delayed::Job To the rescue!
    35. 35. 2. Delayed::Job ActiveRecord::Base’d
    36. 36. 2. Delayed::Job ActiveRecord::Base’d • Persistence! At last!
    37. 37. 2. Delayed::Job ActiveRecord::Base’d • Persistence! At last! • Ease of use! At last!
    38. 38. 2. Delayed::Job How easy?
    39. 39. 2. Delayed::Job Super easy. # without delayed_job Notifier.deliver_signup(@user) # with delayed_job Notifier.send_later :deliver_signup, @user
    40. 40. 2. Delayed::Job Where do they go?
    41. 41. 2. Delayed::Job mysql> SELECT * FROM delayed_jobs; *************************** 1. row *************************** id: 1 priority: 0 attempts: 0 handler: --- !ruby/object:WorkerBee <-- Notice the pretty Marshal dump n: 18 last_error: NULL run_at: 2010-03-10 22:44:30 locked_at: NULL failed_at: NULL locked_by: NULL created_at: 2010-03-10 22:44:30 updated_at: 2010-03-10 22:44:30 1 row in set (0.00 sec)
    42. 42. 2. Delayed::Job What do you do with them?
    43. 43. 2. Delayed::Job What do you do with them? $ script/delayed_job start
    44. 44. 2. Delayed::Job Workers.
    45. 45. 2. Delayed::Job Workers. • Live in their own thread
    46. 46. 2. Delayed::Job Workers. • Live in their own thread • Query the database for jobs
    47. 47. 2. Delayed::Job Workers. • Live in their own thread • Query the database for jobs • Perform the long-running task away from request/response
    48. 48. The Old Way
    49. 49. The New Way Delayed::Job
    50. 50. 2. Delayed::Job Totally rad.
    51. 51. 3. Resque Isn’t Delayed::Job enough?
    52. 52. 3. Resque Isn’t Delayed::Job enough?
    53. 53. 3. Resque Apparently not.
    54. 54. 3. Resque Resque is some Enterprise Grade shit.
    55. 55. 3. Resque Runs on Redis
    56. 56. 3. Resque Runs on Redis
    57. 57. Tangent Alert: Redis! Redis is a Key-Value Store Just like Memcached
    58. 58. Tangent Alert: Redis! Persistence Unlike Memcached
    59. 59. Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new( :hosts=> %w{192.168.1.22:6379 192.168.1.23:6379} )
    60. 60. Tangent Alert: Redis! Basically: Fast & Distributed
    61. 61. Tangent Alert: Redis! Also, Rails-independent No ActiveRecord/ActiveSupport means you can run Resque (on Redis) in Sinatra or any other Ruby app
    62. 62. 3. Resque Better Workers
    63. 63. 3. Resque Better Workers • Forked, sandboxed threads
    64. 64. 3. Resque Better Workers • Forked, sandboxed threads • Failing tasks are tracked
    65. 65. 3. Resque Better Workers • Forked, sandboxed threads • Failing tasks are tracked • Kills timeouts
    66. 66. 3. Resque Queueing Separate queues for separate workers fileserver$ QUEUE=zip_files rake resque:work webserver$ QUEUE=cache_images rake resque:work
    67. 67. 3. Resque No Marshaling Because it should be functional Resque.enqueue(SignupNotifier, @user.id)
    68. 68. 3. Resque No Marshaling class SignupNotifier def self.perform(user_id) Notifier.deliver_signup(User.find(user_id)) end end
    69. 69. 3. Resque UI That’s right. It has its own interface.
    70. 70. 3. Resque So... why?
    71. 71. 3. Resque So... why? • Many background tasks
    72. 72. 3. Resque So... why? • Many background tasks • Distributed task servers
    73. 73. 3. Resque So... why? • Many background tasks • Distributed task servers • Stable workers
    74. 74. 3. Resque Trouble Points
    75. 75. 3. Resque Trouble Points • Complex and expensive
    76. 76. 3. Resque Trouble Points • Complex and expensive • No scheduling or priority like in D::J
    77. 77. 3. Resque Trouble Points • Complex and expensive • No scheduling or priority like in D::J • Asynchronous Redis writes don’t guarantee persistence
    78. 78. 3. Resque Live coding FTW! Here goes nothing...
    79. 79. Thnaks Thanks for taking the time to listen to me blather about Resque!
    80. 80. Me! Github: http://github.com/flipsasser Homepage: http://x451.com Twitter: @flipsasser E-mail: flip@intridea.com
    81. 81. Bmore on Rails Meetups - 2nd Tuesday #OSHN - 4th Tuesday http://www.meetup.com/bmore-on-rails/
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×