Message Queues in Ruby - An Overview

  • 22,616 views
Uploaded on

First given at RubyConf India 2010

First given at RubyConf India 2010

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • The new kid on the block that these slides don’t mention: http://www.simpleworker.com. Job queueing, worker processing and scheduling as a service which makes it easier to use than all of the things mentioned because not setup, no maintenance, etc.
    Are you sure you want to
    Your message goes here
  • queue
    Are you sure you want to
    Your message goes here
  • Wow! Very well explained slide.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
22,616
On Slideshare
0
From Embeds
0
Number of Embeds
5

Actions

Shares
Downloads
286
Comments
3
Likes
41

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide












































  • by Blaine Cook
  • used it on early version of Presently, works just fine, very stable


  • based on memcache-client, which will do server-based hashing
  • can submit/get any kind of string. will have








  • by Blaine Cook
  • used it on early version of Presently, works just fine, very stable












































































  • want to get rid of these


Transcript

  • 1. Message Queueing in Ruby An Overview Pradeep Elankumaran @pradeep24 pradeep@intridea.com
  • 2. About Us ❖ Intridea is a high-end web consulting shop ❖ Based out of Washington DC, but dispersed all over the United States ❖ Our most popular product is Presently, an enterprise microblogging platform ❖ Grub.it is our latest product - mobile dish reviewing ❖ Various iPhone & Android apps (Car Finder, Grub.it) ❖ Specialize in web & mobile development using Ruby, Rails, Java, Erlang and beyond
  • 3. Some background...
  • 4. A request is supposed to return a response really fast.
  • 5. However, there are certain actions that may take longer than a few milliseconds
  • 6. A Few Examples ❖ Delivering e-mail ❖ Image processing (resizing, cropping etc.) ❖ Sending out notifications ❖ Geocoding a location ❖ Talking to other web services
  • 7. These actions, if not extracted out to run asynchronously, will slow down your app
  • 8. Bad user experience
  • 9. What is a Message Queue?
  • 10. A server that you submit asynchronous ‘jobs’ to Things to be done later
  • 11. Interfaces between your Rails app and the ‘workers’ that perform each ‘job’
  • 12. ‘Workers’ pull jobs from each queue and process them ASAP
  • 13. Queues need to be FAST & RELIABLE
  • 14. Queue 1: Delayed Job github.com/tobi/delayed_job
  • 15. Extracted out from Shopify.com
  • 16. Delayed Job is backed by your database.
  • 17. Serializes Ruby Job classes (YAML) into a ‘jobs’ table
  • 18. Easy-to-use syntax
  • 19. rake jobs:run
  • 20. Polls the database to get new jobs :(
  • 21. A Few Notes about DJ ❖ used by Github exhaustively till they moved to Resque ❖ will get you 80% of the way there, has some nice advanced features (priority, retries etc.) ❖ should be careful not to overload your jobs table ❖ backed up jobs can lead to query slowdown on your site ❖ suggested that you monitor and profile workers before you launch on production
  • 22. Queue 2: Starling github.com/starling/starling
  • 23. Extracted out (infamously) from early versions of Twitter
  • 24. Gets a bad rep for systems with high load, but pretty great for smaller apps
  • 25. Speaks Memcached Protocol
  • 26. Distributed from the client-side
  • 27. Easy-to-use Syntax
  • 28. Will have to write your own workers :(
  • 29. Will have to poll the queue to get new jobs :(
  • 30. A Few Notes about Starling ❖ persists to RAILS_ROOT/tmp/starling ❖ great for smaller apps, very lightweight ❖ again, will get you 80% of the way there ❖ writing robust, reliable workers is left up to you ❖ polling makes it slow compared to other queues
  • 31. Queue 3: ActiveMQ activemq.apache.org
  • 32. Mature Apache project.
  • 33. Been around for a while, used by many, many websites & companies
  • 34. Written in Java - however, can connect w/ Ruby using the STOMP protocol
  • 35. Scripts provided to stop/start queue server daemon bin/activemq stop|start|restart
  • 36. Use ActiveMessaging for easy integration or the STOMP gem for manual integration
  • 37. ActiveMessaging plugin ❖ developed by ThoughtWorks ❖ provides a mini-framework to publish and subscribe to various ActiveMQ queues and topics ❖ mature, well-tested
  • 38. Publish from any Ruby object
  • 39. Define your own Processors script/poller run
  • 40. Comes with a basic monitoring interface
  • 41. Evented!
  • 42. A Few Notes about ActiveMQ ❖ enterprise-grade ❖ can be clustered (edit XML files) ❖ using ActiveMessaging, no need to write your own workers ❖ STOMP gem is great if you do want to write your own workers
  • 43. Queue 4: RabbitMQ rabbitmq.com
  • 44. AMQP Advanced Message Queuing Protocol
  • 45. AMQP can do lots of stuff... ❖ Point-to-point communication ❖ One-to-many broadcasting (including multicast) ❖ Transactional publication and acknowledgement ❖ High-speed transient message flows ❖ Reliable persistent message delivery ❖ File streaming
  • 46. Fast, Reliable Erlang Queue
  • 47. Careful with the libraries.... ❖ @tmm1’s amqp gem - runs off EventMachine, doesn’t play well with Passenger (asynchronous) ❖ warren - the best of the bunch for Rails (synchronous) ❖ bunny - lightweight (synchronous)
  • 48. Evented!
  • 49. A Few Notes about RabbitMQ ❖ enterprise-grade ❖ can be clustered using Erlang’s built-in awesome clustering ❖ will have to write your own workers unless you use Workling ❖ very, very fast
  • 50. Queue 5: Resque github.com/defunkt/resque
  • 51. Resque sits on top of Redis
  • 52. Redis superfast data structures store like memcached, but smarter in-memory for the most part persisted to disk asynchronously sets, lists & corresponding operations
  • 53. created by @defunkt of github.com
  • 54. persists jobs to Redis as JSON objects
  • 55. Easy to define new Jobs any Ruby class that responds to ‘perform’ method
  • 56. Queuing jobs QUEUE=image_queue rake resque:work QUEUE=* rake resque:work
  • 57. Also comes with a Sinatra app that lets you monitor your queues
  • 58. A Few Notes about Resque ❖ has a dependency on Redis ❖ don’t have to write your own workers ❖ very nice worker fork() ❖ not evented :(
  • 59. Pick the right queue for the job
  • 60. Do your own load-tests on your own stack
  • 61. The Care & Feeding of Worker Daemons
  • 62. More often than not, the queues are not the problem.
  • 63. Writing workers in Ruby is easy...
  • 64. ... but writing Ruby workers that are long-running and reliable is NOT easy unless you’re using EventMachine or really know what you’re doing
  • 65. Why so hard to write Ruby workers? ❖ You will wake up to workers with memory leaks ❖ Workers sometimes lock up ❖ Loading the whole Rails stack for each worker ❖ In general, a pain to manage these worker daemons
  • 66. Use EventMachine Reactor Pattern Define callbacks on events pretty solid & fast
  • 67. fork( ) spin up the worker in the new process close when done can kill bothersome workers easier debugging
  • 68. Workling plugin github.com/purzelrakete/workling
  • 69. Questions? @pradeep24 pradeep@intridea.com
  • 70. Something new...
  • 71. Llama ❖ A port of Apache Camel to Ruby ❖ based off EventMachine, so it runs ROCK SOLID ❖ early alpha at this point, looking for helpers ❖ github.com/skyfallsin/llama
  • 72. Llama Code Example
  • 73. github.com/skyfallsin/llama