Message Queues in Ruby - An Overview


Published on

First given at RubyConf India 2010

Published in: Technology
  • The new kid on the block that these slides don’t mention: Job queueing, worker processing and scheduling as a service which makes it easier to use than all of the things mentioned because not setup, no maintenance, etc.
    Are you sure you want to  Yes  No
    Your message goes here
  • queue
    Are you sure you want to  Yes  No
    Your message goes here
  • Wow! Very well explained slide.
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

  • by Blaine Cook
  • used it on early version of Presently, works just fine, very stable

  • based on memcache-client, which will do server-based hashing
  • can submit/get any kind of string. will have

  • by Blaine Cook
  • used it on early version of Presently, works just fine, very stable

  • want to get rid of these

  • Message Queues in Ruby - An Overview

    1. 1. Message Queueing in Ruby An Overview Pradeep Elankumaran @pradeep24
    2. 2. About Us ❖ Intridea is a high-end web consulting shop ❖ Based out of Washington DC, but dispersed all over the United States ❖ Our most popular product is Presently, an enterprise microblogging platform ❖ is our latest product - mobile dish reviewing ❖ Various iPhone & Android apps (Car Finder, ❖ Specialize in web & mobile development using Ruby, Rails, Java, Erlang and beyond
    3. 3. Some background...
    4. 4. A request is supposed to return a response really fast.
    5. 5. However, there are certain actions that may take longer than a few milliseconds
    6. 6. A Few Examples ❖ Delivering e-mail ❖ Image processing (resizing, cropping etc.) ❖ Sending out notifications ❖ Geocoding a location ❖ Talking to other web services
    7. 7. These actions, if not extracted out to run asynchronously, will slow down your app
    8. 8. Bad user experience
    9. 9. What is a Message Queue?
    10. 10. A server that you submit asynchronous ‘jobs’ to Things to be done later
    11. 11. Interfaces between your Rails app and the ‘workers’ that perform each ‘job’
    12. 12. ‘Workers’ pull jobs from each queue and process them ASAP
    13. 13. Queues need to be FAST & RELIABLE
    14. 14. Queue 1: Delayed Job
    15. 15. Extracted out from
    16. 16. Delayed Job is backed by your database.
    17. 17. Serializes Ruby Job classes (YAML) into a ‘jobs’ table
    18. 18. Easy-to-use syntax
    19. 19. rake jobs:run
    20. 20. Polls the database to get new jobs :(
    21. 21. A Few Notes about DJ ❖ used by Github exhaustively till they moved to Resque ❖ will get you 80% of the way there, has some nice advanced features (priority, retries etc.) ❖ should be careful not to overload your jobs table ❖ backed up jobs can lead to query slowdown on your site ❖ suggested that you monitor and profile workers before you launch on production
    22. 22. Queue 2: Starling
    23. 23. Extracted out (infamously) from early versions of Twitter
    24. 24. Gets a bad rep for systems with high load, but pretty great for smaller apps
    25. 25. Speaks Memcached Protocol
    26. 26. Distributed from the client-side
    27. 27. Easy-to-use Syntax
    28. 28. Will have to write your own workers :(
    29. 29. Will have to poll the queue to get new jobs :(
    30. 30. A Few Notes about Starling ❖ persists to RAILS_ROOT/tmp/starling ❖ great for smaller apps, very lightweight ❖ again, will get you 80% of the way there ❖ writing robust, reliable workers is left up to you ❖ polling makes it slow compared to other queues
    31. 31. Queue 3: ActiveMQ
    32. 32. Mature Apache project.
    33. 33. Been around for a while, used by many, many websites & companies
    34. 34. Written in Java - however, can connect w/ Ruby using the STOMP protocol
    35. 35. Scripts provided to stop/start queue server daemon bin/activemq stop|start|restart
    36. 36. Use ActiveMessaging for easy integration or the STOMP gem for manual integration
    37. 37. ActiveMessaging plugin ❖ developed by ThoughtWorks ❖ provides a mini-framework to publish and subscribe to various ActiveMQ queues and topics ❖ mature, well-tested
    38. 38. Publish from any Ruby object
    39. 39. Define your own Processors script/poller run
    40. 40. Comes with a basic monitoring interface
    41. 41. Evented!
    42. 42. A Few Notes about ActiveMQ ❖ enterprise-grade ❖ can be clustered (edit XML files) ❖ using ActiveMessaging, no need to write your own workers ❖ STOMP gem is great if you do want to write your own workers
    43. 43. Queue 4: RabbitMQ
    44. 44. AMQP Advanced Message Queuing Protocol
    45. 45. AMQP can do lots of stuff... ❖ Point-to-point communication ❖ One-to-many broadcasting (including multicast) ❖ Transactional publication and acknowledgement ❖ High-speed transient message flows ❖ Reliable persistent message delivery ❖ File streaming
    46. 46. Fast, Reliable Erlang Queue
    47. 47. Careful with the libraries.... ❖ @tmm1’s amqp gem - runs off EventMachine, doesn’t play well with Passenger (asynchronous) ❖ warren - the best of the bunch for Rails (synchronous) ❖ bunny - lightweight (synchronous)
    48. 48. Evented!
    49. 49. A Few Notes about RabbitMQ ❖ enterprise-grade ❖ can be clustered using Erlang’s built-in awesome clustering ❖ will have to write your own workers unless you use Workling ❖ very, very fast
    50. 50. Queue 5: Resque
    51. 51. Resque sits on top of Redis
    52. 52. Redis superfast data structures store like memcached, but smarter in-memory for the most part persisted to disk asynchronously sets, lists & corresponding operations
    53. 53. created by @defunkt of
    54. 54. persists jobs to Redis as JSON objects
    55. 55. Easy to define new Jobs any Ruby class that responds to ‘perform’ method
    56. 56. Queuing jobs QUEUE=image_queue rake resque:work QUEUE=* rake resque:work
    57. 57. Also comes with a Sinatra app that lets you monitor your queues
    58. 58. A Few Notes about Resque ❖ has a dependency on Redis ❖ don’t have to write your own workers ❖ very nice worker fork() ❖ not evented :(
    59. 59. Pick the right queue for the job
    60. 60. Do your own load-tests on your own stack
    61. 61. The Care & Feeding of Worker Daemons
    62. 62. More often than not, the queues are not the problem.
    63. 63. Writing workers in Ruby is easy...
    64. 64. ... but writing Ruby workers that are long-running and reliable is NOT easy unless you’re using EventMachine or really know what you’re doing
    65. 65. Why so hard to write Ruby workers? ❖ You will wake up to workers with memory leaks ❖ Workers sometimes lock up ❖ Loading the whole Rails stack for each worker ❖ In general, a pain to manage these worker daemons
    66. 66. Use EventMachine Reactor Pattern Define callbacks on events pretty solid & fast
    67. 67. fork( ) spin up the worker in the new process close when done can kill bothersome workers easier debugging
    68. 68. Workling plugin
    69. 69. Questions? @pradeep24
    70. 70. Something new...
    71. 71. Llama ❖ A port of Apache Camel to Ruby ❖ based off EventMachine, so it runs ROCK SOLID ❖ early alpha at this point, looking for helpers ❖
    72. 72. Llama Code Example
    73. 73.
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.