Download as KEY, PPTX
Uploaded byFlip Sasser
Asynchronous Awesome
The document discusses Resque, a high-performance task queuing system that uses Redis, highlighting its advantages over Delayed::Job, such as better worker management and task tracking. Key topics include the necessity of background processes, the ease of use provided by Delayed::Job for processing tasks away from web requests, and the scalability and speed benefits of using Resque. Additionally, it addresses the challenges associated with both systems, including complexities and lack of scheduling features in Resque.
More Related Content
![[BDD 2025 - Mobile Development] Mobile Engineer and Software Engineer: Are we...](https://cdn.slidesharecdn.com/ss_thumbnails/md-mobileengineerandsoftwareengineerarewestillrelevantsidiqpermana-251127010650-55224ef1-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Mobile Development] Exploring Apple’s On-Device FoundationModels](https://cdn.slidesharecdn.com/ss_thumbnails/md-exploringappleson-devicefoundationmodels-251124030840-d690542c-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] The Modern Stack: Building Web & AI Appli...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-themodernstackbuildingwebaiapplicationswithserverless-251124030844-388cf04f-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Artificial Intelligence] Building AI Systems That Users (and Comp...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-buildingaisystemsthatusersandcompanieslove-251124030845-038f7732-thumbnail.jpg?width=640&height=640&fit=bounds)
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
Asynchronous Awesome
- 1. Resque High-performance asynchronous taskqueuing and processing
- 2. Resque High-performance asynchronous taskqueuing and processing
- 3. Resque High-performance asynchronous taskqueuing and processing
- 4.
- 5. Experts ‘Cause you mightbuild the next GitHub
- 6.
- 7. 1. Background Why runbackground processes at all?
- 8. 2. Delayed::Job Because it’sthe foundation for Resque
- 9.
- 10. 1. Background Why?
- 11. 1. Background It’s allabout the ms
- 12. 1. Background It’s all about the ms • Resize an image
- 13. 1. Background It’s all about the ms • Resize an image • Send an e-mail
- 14. 1. Background It’s all about the ms • Resize an image • Send an e-mail • Communicate with a web service
- 15. 1. Background It’s all about the ms • Resize an image • Send an e-mail • Communicate with a web service • Geocode a location
- 16. 1. Background Do somethinglater than now
- 17.
- 18.
- 19. 1. Background Webpages. Rails serves webpages. Memories. You're talkin about memories.
- 20.
- 21. 1. Background 1. Receiverequests
- 22. 1. Background 1. Receiverequests 2. Send responses
- 23. 1. Background Everything elsecan go elsewhere
- 24. 1. Background Where?
- 25. 1. Background Where? • CRON
- 26. 1. Background Where? • CRON • Beanstalkd
- 27. 1. Background Where? • CRON • Beanstalkd • Workling
- 28. 1. Background Where? • CRON • Beanstalkd • Workling • BackgroundRB
- 29. 1. Background WTF mytasks r fail?!1
- 30. 1. Background WTF mytasks r fail?!1 • Silent failure
- 31. 1. Background WTF mytasks r fail?!1 • Silent failure • No status checks
- 32. 1. Background WTF mytasks r fail?!1 • Silent failure • No status checks • Unstable processes
- 33. 2. Delayed::Job To therescue!
- 34. 2. Delayed::Job To therescue!
- 35.
- 36. 2. Delayed::Job ActiveRecord::Base’d •Persistence! At last!
- 37. 2. Delayed::Job ActiveRecord::Base’d •Persistence! At last! • Ease of use! At last!
- 38. 2. Delayed::Job Howeasy?
- 39. 2. Delayed::Job Super easy. # without delayed_job Notifier.deliver_signup(@user) # with delayed_job Notifier.send_later :deliver_signup, @user
- 40. 2. Delayed::Job Where dothey go?
- 41. 2. Delayed::Job mysql> SELECT* FROM delayed_jobs; *************************** 1. row *************************** id: 1 priority: 0 attempts: 0 handler: --- !ruby/object:WorkerBee <-- Notice the pretty Marshal dump n: 18 last_error: NULL run_at: 2010-03-10 22:44:30 locked_at: NULL failed_at: NULL locked_by: NULL created_at: 2010-03-10 22:44:30 updated_at: 2010-03-10 22:44:30 1 row in set (0.00 sec)
- 42. 2. Delayed::Job What doyou do with them?
- 43. 2. Delayed::Job What do you do with them? $ script/delayed_job start
- 44. 2. Delayed::Job Workers.
- 45. 2. Delayed::Job Workers. • Live in their own thread
- 46. 2. Delayed::Job Workers. • Live in their own thread • Query the database for jobs
- 47. 2. Delayed::Job Workers. • Live in their own thread • Query the database for jobs • Perform the long-running task away from request/response
- 48.
- 49.
- 50.
- 51. 3. Resque Isn’t Delayed::Job enough?
- 52. 3. Resque Isn’t Delayed::Job enough?
- 53.
- 54. 3. Resque Resque is some Enterprise Grade shit.
- 55. 3. Resque Runs onRedis
- 56. 3. Resque Runs onRedis
- 57. Tangent Alert: Redis! Redisis a Key-Value Store Just like Memcached
- 58. Tangent Alert: Redis! Persistence Unlike Memcached
- 59. Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new( :hosts=> %w{192.168.1.22:6379 192.168.1.23:6379} )
- 60. Tangent Alert: Redis! Basically: Fast & Distributed
- 61. Tangent Alert: Redis! Also,Rails-independent No ActiveRecord/ActiveSupport means you can run Resque (on Redis) in Sinatra or any other Ruby app
- 62.
- 63. 3. Resque Better Workers • Forked, sandboxed threads
- 64. 3. Resque Better Workers • Forked, sandboxed threads • Failing tasks are tracked
- 65. 3. Resque Better Workers • Forked, sandboxed threads • Failing tasks are tracked • Kills timeouts
- 66. 3. Resque Queueing Separate queues for separate workers fileserver$ QUEUE=zip_files rake resque:work webserver$ QUEUE=cache_images rake resque:work
- 67. 3. Resque No Marshaling Because it should be functional Resque.enqueue(SignupNotifier, @user.id)
- 68. 3. Resque No Marshaling class SignupNotifier def self.perform(user_id) Notifier.deliver_signup(User.find(user_id)) end end
- 69. 3. Resque UI That’s right. It has its own interface.
- 70.
- 71. 3. Resque So... why? • Many background tasks
- 72. 3. Resque So... why? • Many background tasks • Distributed task servers
- 73. 3. Resque So... why? • Many background tasks • Distributed task servers • Stable workers
- 74.
- 75. 3. Resque Trouble Points • Complex and expensive
- 76. 3. Resque Trouble Points • Complex and expensive • No scheduling or priority like in D::J
- 77. 3. Resque Trouble Points • Complex and expensive • No scheduling or priority like in D::J • Asynchronous Redis writes don’t guarantee persistence
- 78. 3. Resque Live codingFTW! Here goes nothing...
- 79. Thnaks Thanks for takingthe time to listen to me blather about Resque!
- 80. Me! Github: http://github.com/flipsasser Homepage: http://x451.com Twitter: @flipsasser E-mail: flip@intridea.com
- 81. Bmore on Rails Meetups - 2nd Tuesday #OSHN - 4th Tuesday http://www.meetup.com/bmore-on-rails/
Editor's Notes
- #2 Okay, my talk is called &#x201C;...&#x201D; which I think is pretty descriptive. Except that it doesn&#x2019;t really demonstrate the pure awesome is that is Resque, so really, we&#x2019;ll call it...
- #3 Yeah.
- #4 So, why should you care about what I&#x2019;m going to say in the next 20 minutes? Because when you walk away from this room, you&#x2019;ll all be experts at building a network of high-performance and fault-tolerant workers.
- #5 So we&#x2019;re going to talk about three things today
- #6 First, we&#x2019;re going to do a general background of background processing
- #7 Next, we&#x2019;ll cover the basic concepts of Resque that are derived from Delayed::Job
- #8 Last, we&#x2019;ll get into Resque, and how it&#x2019;s different.
- #9 So, why run background tasks at all?
- #10 - Resize - Etc That&#x2019;s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
- #11 - Resize - Etc That&#x2019;s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
- #12 - Resize - Etc That&#x2019;s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
- #13 - Resize - Etc That&#x2019;s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
- #14 Let&#x2019;s do something later than now. If my application doesn&#x2019;t NEED to do this stuff right away, don&#x2019;t keep the request tied up. After all, what IS a request for?
- #15 Webpages In case you&#x2019;d forgotten - Rails serves webpages. It&#x2019;s not built to process images or send out mass e-mails; it&#x2019;s built to do two things:
- #16 Webpages In case you&#x2019;d forgotten - Rails serves webpages. It&#x2019;s not built to process images or send out mass e-mails; it&#x2019;s built to do two things:
- #17 Webpages In case you&#x2019;d forgotten - Rails serves webpages. It&#x2019;s not built to process images or send out mass e-mails; it&#x2019;s built to do two things:
- #18 Webpages In case you&#x2019;d forgotten - Rails serves webpages. It&#x2019;s not built to process images or send out mass e-mails; it&#x2019;s built to do two things:
- #19 - Receive requests - Send responses
- #20 - Receive requests - Send responses
- #21 Everything else can go elsewhere
- #22 So... where should it go? There are a whole bunch of options. - CRON - Etc. But sometimes those setups have problems
- #23 So... where should it go? There are a whole bunch of options. - CRON - Etc. But sometimes those setups have problems
- #24 So... where should it go? There are a whole bunch of options. - CRON - Etc. But sometimes those setups have problems
- #25 So... where should it go? There are a whole bunch of options. - CRON - Etc. But sometimes those setups have problems
- #26 - They can fail without telling you why - You can&#x2019;t get the status of one of a worker - Ruby processes can be unstable and crash-ey So maybe you need something a little bit better
- #27 - They can fail without telling you why - You can&#x2019;t get the status of one of a worker - Ruby processes can be unstable and crash-ey So maybe you need something a little bit better
- #28 - They can fail without telling you why - You can&#x2019;t get the status of one of a worker - Ruby processes can be unstable and crash-ey So maybe you need something a little bit better
- #29 Delayed::Job to the rescue!
- #30 A Delayed::Job is an ActiveRecord model, so it lives in the database. - You get persistence baked right in - It&#x2019;s unbelievably easy to use
- #31 A Delayed::Job is an ActiveRecord model, so it lives in the database. - You get persistence baked right in - It&#x2019;s unbelievably easy to use
- #32 How easy?
- #33 Super easy. I pulled this straight from the github page. This code shows how you would create a Delayed::Job
- #34 But where does that job go?
- #35 Here - your database. Notice that it simply marshals the object and stuffs in the database.
- #36 So what do you do with these jobs in the database? - Run the delayed_job script This will create...
- #37 Workers. They&#x2019;re separate Ruby processes that - Live in their own thread - Find jobs in the database - Perform them separately from the request/response cycle They boot up your Rails environment to do so.
- #38 Workers. They&#x2019;re separate Ruby processes that - Live in their own thread - Find jobs in the database - Perform them separately from the request/response cycle They boot up your Rails environment to do so.
- #39 Workers. They&#x2019;re separate Ruby processes that - Live in their own thread - Find jobs in the database - Perform them separately from the request/response cycle They boot up your Rails environment to do so.
- #40 So before, you had requests coming from a browser into Rails, which dispatches to various services, gets results, and sends something back to the browser
- #41 Now you have Rails storing the jobs for later, and Delayed::Job comes along and does the processing Rails isn&#x2019;t good at doing anyway.
- #42 So Delayed::Job is totally rad. It handles most of what you might want it to handle.
- #43 So when we start in with Resque, you might wonder, &#x201C;Isn&#x2019;t Delayed::Job enough?&#x201D;
- #44 Apparently not. There are some situations where Delayed::Job isn&#x2019;t sufficient.
- #45 Basically, Resque is enterprise grade shit.
- #46 So the first big difference between Resque and Delayed::Job is that Resque doesn&#x2019;t rely on ActiveRecord. It runs on something called Redis, which probably deserves - its own tangent
- #47 Redis is an in-memory Key-Value Store similar to Memcached. But it offers
- #48 Redis is an in-memory Key-Value Store similar to Memcached. But it offers
- #49 Redis is an in-memory Key-Value Store similar to Memcached. But it offers
- #50 Redis is an in-memory Key-Value Store similar to Memcached. But it offers
- #51 Redis is an in-memory Key-Value Store similar to Memcached. But it offers
- #52 Redis is an in-memory Key-Value Store similar to Memcached. But it offers
- #53 Persistence! So you can reboot your server and Redis will reload your original queued jobs. It also is great at
- #54 Scalability! The Ruby gem will load-balance tasks across multiple Redis servers for you, meaning you spend less time configuring and more time queueing
- #55 So Redis makes Resque incredibly fast and easily distributed.
- #56 Also, using Redis means you don&#x2019;t have to worry about ActiveRecord - so you can skip Rails entirely
- #57 The next big difference is Resque&#x2019;s workers - When a worker finds a job to process, it creates a fork that it can monitor to process the job If a fork spirals out, it doesn&#x2019;t take the worker with it - Workers are capable of moving failed jobs into the failed List and moving on with their lives - Workers are also capable of timing their own processes out and failing them
- #58 The next big difference is Resque&#x2019;s workers - When a worker finds a job to process, it creates a fork that it can monitor to process the job If a fork spirals out, it doesn&#x2019;t take the worker with it - Workers are capable of moving failed jobs into the failed List and moving on with their lives - Workers are also capable of timing their own processes out and failing them
- #59 The next big difference is Resque&#x2019;s workers - When a worker finds a job to process, it creates a fork that it can monitor to process the job If a fork spirals out, it doesn&#x2019;t take the worker with it - Workers are capable of moving failed jobs into the failed List and moving on with their lives - Workers are also capable of timing their own processes out and failing them
- #60 Resque also supports queues, so different types of tasks can be divided by disparate workers. A file server, for example, can have a worker that only zips files - while a web server can warm image caches.
- #61 Resque also skips the marshaling step, and instead opts for a class and a JSON-ifiable argument list. This will keep all instances current
- #62 The class&#x2019; &#x201C;perform&#x201D; method is called with the arguments you enqueued it with
- #63 Resque comes with its own interface for monitoring queues and tasks, and reviewing failed tasks.
- #64 So why use Resque? Use Resque if - You have many, many, many background tasks - You need to have different servers performing different types tasks - You need bullet-proof workers
- #65 So why use Resque? Use Resque if - You have many, many, many background tasks - You need to have different servers performing different types tasks - You need bullet-proof workers
- #66 So why use Resque? Use Resque if - You have many, many, many background tasks - You need to have different servers performing different types tasks - You need bullet-proof workers
- #67 Trouble - Complex / expensive - No scheduling or priority - Asynchronous Redis writes don&#x2019;t guarantee persistence
- #68 Trouble - Complex / expensive - No scheduling or priority - Asynchronous Redis writes don&#x2019;t guarantee persistence
- #69 Trouble - Complex / expensive - No scheduling or priority - Asynchronous Redis writes don&#x2019;t guarantee persistence
- #70 Okay, if we have time I&#x2019;ll show off Resque&#x2019;s UI and do a quick comparison of it to D::J
- #71 Thanks for listening!
- #72 This is me - you can find me on Github or Twitter, and if you&#x2019;re interested I&#x2019;ll post these slides on Twitter.
- #73 Last but not least, I&#x2019;m a visiting emissary from B&#x2019;more on Rails, and I&#x2019;d like to personally invite all of you to come up to our Meetups (on the 2nd tuesday of every month) and our Open Source Hack Nights (on the 4th Tuesday of every month).