Asynchronous Awesome
•Download as KEY, PDF•
7 likes•1,156 views
This is the presentation I gave to DCRUG on Resque. Read the presenter notes or it may not make a whole lot of sense.
More Related Content
What's hot
What's hot (11)
Viewers also liked
Similar to Asynchronous Awesome
Similar to Asynchronous Awesome (20)
Recently uploaded
Recently uploaded (20)
Asynchronous Awesome
- 1. Resque High-performance asynchronous task queuing and processing
- 2. Resque High-performance asynchronous task queuing and processing
- 3. Resque High-performance asynchronous task queuing and processing
- 4. Experts
- 5. Experts ‘Cause you might build the next GitHub
- 6. Three Things
- 7. 1. Background Why run background processes at all?
- 8. 2. Delayed::Job Because it’s the foundation for Resque
- 10. 1. Background Why?
- 11. 1. Background It’s all about the ms
- 12. 1. Background It’s all about the ms • Resize an image
- 13. 1. Background It’s all about the ms • Resize an image • Send an e-mail
- 14. 1. Background It’s all about the ms • Resize an image • Send an e-mail • Communicate with a web service
- 15. 1. Background It’s all about the ms • Resize an image • Send an e-mail • Communicate with a web service • Geocode a location
- 16. 1. Background Do something later than now
- 17. 1. Background
- 19. 1. Background Webpages. Rails serves webpages. Memories. You're talkin about memories.
- 20. 1. Background
- 21. 1. Background 1. Receive requests
- 22. 1. Background 1. Receive requests 2. Send responses
- 23. 1. Background Everything else can go elsewhere
- 24. 1. Background Where?
- 25. 1. Background Where? • CRON
- 26. 1. Background Where? • CRON • Beanstalkd
- 27. 1. Background Where? • CRON • Beanstalkd • Workling
- 28. 1. Background Where? • CRON • Beanstalkd • Workling • BackgroundRB
- 29. 1. Background WTF my tasks r fail?!1
- 30. 1. Background WTF my tasks r fail?!1 • Silent failure
- 31. 1. Background WTF my tasks r fail?!1 • Silent failure • No status checks
- 32. 1. Background WTF my tasks r fail?!1 • Silent failure • No status checks • Unstable processes
- 33. 2. Delayed::Job To the rescue!
- 34. 2. Delayed::Job To the rescue!
- 36. 2. Delayed::Job ActiveRecord::Base’d • Persistence! At last!
- 37. 2. Delayed::Job ActiveRecord::Base’d • Persistence! At last! • Ease of use! At last!
- 38. 2. Delayed::Job How easy?
- 39. 2. Delayed::Job Super easy. # without delayed_job Notifier.deliver_signup(@user) # with delayed_job Notifier.send_later :deliver_signup, @user
- 40. 2. Delayed::Job Where do they go?
- 41. 2. Delayed::Job mysql> SELECT * FROM delayed_jobs; *************************** 1. row *************************** id: 1 priority: 0 attempts: 0 handler: --- !ruby/object:WorkerBee <-- Notice the pretty Marshal dump n: 18 last_error: NULL run_at: 2010-03-10 22:44:30 locked_at: NULL failed_at: NULL locked_by: NULL created_at: 2010-03-10 22:44:30 updated_at: 2010-03-10 22:44:30 1 row in set (0.00 sec)
- 42. 2. Delayed::Job What do you do with them?
- 43. 2. Delayed::Job What do you do with them? $ script/delayed_job start
- 44. 2. Delayed::Job Workers.
- 45. 2. Delayed::Job Workers. • Live in their own thread
- 46. 2. Delayed::Job Workers. • Live in their own thread • Query the database for jobs
- 47. 2. Delayed::Job Workers. • Live in their own thread • Query the database for jobs • Perform the long-running task away from request/response
- 48. The Old Way
- 50. 2. Delayed::Job Totally rad.
- 51. 3. Resque Isn’t Delayed::Job enough?
- 52. 3. Resque Isn’t Delayed::Job enough?
- 54. 3. Resque Resque is some Enterprise Grade shit.
- 55. 3. Resque Runs on Redis
- 56. 3. Resque Runs on Redis
- 57. Tangent Alert: Redis! Redis is a Key-Value Store Just like Memcached
- 58. Tangent Alert: Redis! Persistence Unlike Memcached
- 59. Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new( :hosts=> %w{192.168.1.22:6379 192.168.1.23:6379} )
- 60. Tangent Alert: Redis! Basically: Fast & Distributed
- 61. Tangent Alert: Redis! Also, Rails-independent No ActiveRecord/ActiveSupport means you can run Resque (on Redis) in Sinatra or any other Ruby app
- 63. 3. Resque Better Workers • Forked, sandboxed threads
- 64. 3. Resque Better Workers • Forked, sandboxed threads • Failing tasks are tracked
- 65. 3. Resque Better Workers • Forked, sandboxed threads • Failing tasks are tracked • Kills timeouts
- 66. 3. Resque Queueing Separate queues for separate workers fileserver$ QUEUE=zip_files rake resque:work webserver$ QUEUE=cache_images rake resque:work
- 67. 3. Resque No Marshaling Because it should be functional Resque.enqueue(SignupNotifier, @user.id)
- 68. 3. Resque No Marshaling class SignupNotifier def self.perform(user_id) Notifier.deliver_signup(User.find(user_id)) end end
- 69. 3. Resque UI That’s right. It has its own interface.
- 71. 3. Resque So... why? • Many background tasks
- 72. 3. Resque So... why? • Many background tasks • Distributed task servers
- 73. 3. Resque So... why? • Many background tasks • Distributed task servers • Stable workers
- 75. 3. Resque Trouble Points • Complex and expensive
- 76. 3. Resque Trouble Points • Complex and expensive • No scheduling or priority like in D::J
- 77. 3. Resque Trouble Points • Complex and expensive • No scheduling or priority like in D::J • Asynchronous Redis writes don’t guarantee persistence
- 78. 3. Resque Live coding FTW! Here goes nothing...
- 79. Thnaks Thanks for taking the time to listen to me blather about Resque!
- 80. Me! Github: http://github.com/flipsasser Homepage: http://x451.com Twitter: @flipsasser E-mail: flip@intridea.com
- 81. Bmore on Rails Meetups - 2nd Tuesday #OSHN - 4th Tuesday http://www.meetup.com/bmore-on-rails/
Editor's Notes
- Okay, my talk is called &#x201C;...&#x201D; which I think is pretty descriptive. Except that it doesn&#x2019;t really demonstrate the pure awesome is that is Resque, so really, we&#x2019;ll call it...
- Yeah.
- So, why should you care about what I&#x2019;m going to say in the next 20 minutes? Because when you walk away from this room, you&#x2019;ll all be experts at building a network of high-performance and fault-tolerant workers.
- So we&#x2019;re going to talk about three things today
- First, we&#x2019;re going to do a general background of background processing
- Next, we&#x2019;ll cover the basic concepts of Resque that are derived from Delayed::Job
- Last, we&#x2019;ll get into Resque, and how it&#x2019;s different.
- So, why run background tasks at all?
- - Resize - Etc That&#x2019;s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
- - Resize - Etc That&#x2019;s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
- - Resize - Etc That&#x2019;s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
- - Resize - Etc That&#x2019;s all really cool stuff, but it all takes time - sometimes a few seconds, sometimes up to 30 seconds. So, someone, somewhere, had an idea:
- Let&#x2019;s do something later than now. If my application doesn&#x2019;t NEED to do this stuff right away, don&#x2019;t keep the request tied up. After all, what IS a request for?
- Webpages In case you&#x2019;d forgotten - Rails serves webpages. It&#x2019;s not built to process images or send out mass e-mails; it&#x2019;s built to do two things:
- Webpages In case you&#x2019;d forgotten - Rails serves webpages. It&#x2019;s not built to process images or send out mass e-mails; it&#x2019;s built to do two things:
- Webpages In case you&#x2019;d forgotten - Rails serves webpages. It&#x2019;s not built to process images or send out mass e-mails; it&#x2019;s built to do two things:
- Webpages In case you&#x2019;d forgotten - Rails serves webpages. It&#x2019;s not built to process images or send out mass e-mails; it&#x2019;s built to do two things:
- - Receive requests - Send responses
- - Receive requests - Send responses
- Everything else can go elsewhere
- So... where should it go? There are a whole bunch of options. - CRON - Etc. But sometimes those setups have problems
- So... where should it go? There are a whole bunch of options. - CRON - Etc. But sometimes those setups have problems
- So... where should it go? There are a whole bunch of options. - CRON - Etc. But sometimes those setups have problems
- So... where should it go? There are a whole bunch of options. - CRON - Etc. But sometimes those setups have problems
- - They can fail without telling you why - You can&#x2019;t get the status of one of a worker - Ruby processes can be unstable and crash-ey So maybe you need something a little bit better
- - They can fail without telling you why - You can&#x2019;t get the status of one of a worker - Ruby processes can be unstable and crash-ey So maybe you need something a little bit better
- - They can fail without telling you why - You can&#x2019;t get the status of one of a worker - Ruby processes can be unstable and crash-ey So maybe you need something a little bit better
- Delayed::Job to the rescue!
- A Delayed::Job is an ActiveRecord model, so it lives in the database. - You get persistence baked right in - It&#x2019;s unbelievably easy to use
- A Delayed::Job is an ActiveRecord model, so it lives in the database. - You get persistence baked right in - It&#x2019;s unbelievably easy to use
- How easy?
- Super easy. I pulled this straight from the github page. This code shows how you would create a Delayed::Job
- But where does that job go?
- Here - your database. Notice that it simply marshals the object and stuffs in the database.
- So what do you do with these jobs in the database? - Run the delayed_job script This will create...
- Workers. They&#x2019;re separate Ruby processes that - Live in their own thread - Find jobs in the database - Perform them separately from the request/response cycle They boot up your Rails environment to do so.
- Workers. They&#x2019;re separate Ruby processes that - Live in their own thread - Find jobs in the database - Perform them separately from the request/response cycle They boot up your Rails environment to do so.
- Workers. They&#x2019;re separate Ruby processes that - Live in their own thread - Find jobs in the database - Perform them separately from the request/response cycle They boot up your Rails environment to do so.
- So before, you had requests coming from a browser into Rails, which dispatches to various services, gets results, and sends something back to the browser
- Now you have Rails storing the jobs for later, and Delayed::Job comes along and does the processing Rails isn&#x2019;t good at doing anyway.
- So Delayed::Job is totally rad. It handles most of what you might want it to handle.
- So when we start in with Resque, you might wonder, &#x201C;Isn&#x2019;t Delayed::Job enough?&#x201D;
- Apparently not. There are some situations where Delayed::Job isn&#x2019;t sufficient.
- Basically, Resque is enterprise grade shit.
- So the first big difference between Resque and Delayed::Job is that Resque doesn&#x2019;t rely on ActiveRecord. It runs on something called Redis, which probably deserves - its own tangent
- Redis is an in-memory Key-Value Store similar to Memcached. But it offers
- Redis is an in-memory Key-Value Store similar to Memcached. But it offers
- Redis is an in-memory Key-Value Store similar to Memcached. But it offers
- Redis is an in-memory Key-Value Store similar to Memcached. But it offers
- Redis is an in-memory Key-Value Store similar to Memcached. But it offers
- Redis is an in-memory Key-Value Store similar to Memcached. But it offers
- Persistence! So you can reboot your server and Redis will reload your original queued jobs. It also is great at
- Scalability! The Ruby gem will load-balance tasks across multiple Redis servers for you, meaning you spend less time configuring and more time queueing
- So Redis makes Resque incredibly fast and easily distributed.
- Also, using Redis means you don&#x2019;t have to worry about ActiveRecord - so you can skip Rails entirely
- The next big difference is Resque&#x2019;s workers - When a worker finds a job to process, it creates a fork that it can monitor to process the job If a fork spirals out, it doesn&#x2019;t take the worker with it - Workers are capable of moving failed jobs into the failed List and moving on with their lives - Workers are also capable of timing their own processes out and failing them
- The next big difference is Resque&#x2019;s workers - When a worker finds a job to process, it creates a fork that it can monitor to process the job If a fork spirals out, it doesn&#x2019;t take the worker with it - Workers are capable of moving failed jobs into the failed List and moving on with their lives - Workers are also capable of timing their own processes out and failing them
- The next big difference is Resque&#x2019;s workers - When a worker finds a job to process, it creates a fork that it can monitor to process the job If a fork spirals out, it doesn&#x2019;t take the worker with it - Workers are capable of moving failed jobs into the failed List and moving on with their lives - Workers are also capable of timing their own processes out and failing them
- Resque also supports queues, so different types of tasks can be divided by disparate workers. A file server, for example, can have a worker that only zips files - while a web server can warm image caches.
- Resque also skips the marshaling step, and instead opts for a class and a JSON-ifiable argument list. This will keep all instances current
- The class&#x2019; &#x201C;perform&#x201D; method is called with the arguments you enqueued it with
- Resque comes with its own interface for monitoring queues and tasks, and reviewing failed tasks.
- So why use Resque? Use Resque if - You have many, many, many background tasks - You need to have different servers performing different types tasks - You need bullet-proof workers
- So why use Resque? Use Resque if - You have many, many, many background tasks - You need to have different servers performing different types tasks - You need bullet-proof workers
- So why use Resque? Use Resque if - You have many, many, many background tasks - You need to have different servers performing different types tasks - You need bullet-proof workers
- Trouble - Complex / expensive - No scheduling or priority - Asynchronous Redis writes don&#x2019;t guarantee persistence
- Trouble - Complex / expensive - No scheduling or priority - Asynchronous Redis writes don&#x2019;t guarantee persistence
- Trouble - Complex / expensive - No scheduling or priority - Asynchronous Redis writes don&#x2019;t guarantee persistence
- Okay, if we have time I&#x2019;ll show off Resque&#x2019;s UI and do a quick comparison of it to D::J
- Thanks for listening!
- This is me - you can find me on Github or Twitter, and if you&#x2019;re interested I&#x2019;ll post these slides on Twitter.
- Last but not least, I&#x2019;m a visiting emissary from B&#x2019;more on Rails, and I&#x2019;d like to personally invite all of you to come up to our Meetups (on the 2nd tuesday of every month) and our Open Source Hack Nights (on the 4th Tuesday of every month).