Message Queueing in Ruby
          An Overview
       Pradeep Elankumaran
            @pradeep24
       pradeep@intridea.c...
About Us
❖   Intridea is a high-end web consulting shop

❖   Based out of Washington DC, but dispersed all over the United...
Some background...
A request is supposed to
return a response really fast.
However, there are certain
actions that may take longer
  than a few milliseconds
A Few Examples
❖   Delivering e-mail

❖   Image processing (resizing, cropping etc.)

❖   Sending out notifications

❖   Ge...
These actions, if not
  extracted out to run
asynchronously, will slow
     down your app
Bad user experience
What is a
Message Queue?
A server that you submit
 asynchronous ‘jobs’ to
        Things to be done later
Interfaces between your Rails
  app and the ‘workers’ that
      perform each ‘job’
‘Workers’ pull jobs from each
  queue and process them
           ASAP
Queues need to be
FAST & RELIABLE
Queue 1: Delayed Job
     github.com/tobi/delayed_job
Extracted out from
   Shopify.com
Delayed Job is backed by
     your database.
Serializes Ruby Job classes
  (YAML) into a ‘jobs’ table
Easy-to-use syntax
rake jobs:run
Polls the database
to get new jobs :(
A Few Notes about DJ
❖   used by Github exhaustively till they moved to Resque

❖   will get you 80% of the way there, has...
Queue 2: Starling
    github.com/starling/starling
Extracted out (infamously)
  from early versions of
         Twitter
Gets a bad rep for systems
with high load, but pretty
  great for smaller apps
Speaks Memcached Protocol
Distributed from the
     client-side
Easy-to-use Syntax
Will have to write your own
         workers :(
Will have to poll the queue
     to get new jobs :(
A Few Notes about Starling

❖   persists to RAILS_ROOT/tmp/starling

❖   great for smaller apps, very lightweight

❖   aga...
Queue 3: ActiveMQ
     activemq.apache.org
Mature Apache project.
Been around for a while, used
 by many, many websites &
        companies
Written in Java - however,
can connect w/ Ruby using
  the STOMP protocol
Scripts provided to stop/start
    queue server daemon
        bin/activemq stop|start|restart
Use ActiveMessaging for easy
 integration or the STOMP
 gem for manual integration
ActiveMessaging plugin


❖   developed by ThoughtWorks

❖   provides a mini-framework to publish and subscribe to
    vari...
Publish from any Ruby object
Define your own Processors




      script/poller run
Comes with a basic
monitoring interface
Evented!
A Few Notes about ActiveMQ

❖   enterprise-grade

❖   can be clustered (edit XML files)

❖   using ActiveMessaging, no need...
Queue 4: RabbitMQ
      rabbitmq.com
AMQP
Advanced Message Queuing Protocol
AMQP can do lots of stuff...
❖   Point-to-point communication

❖   One-to-many broadcasting (including multicast)

❖   Tran...
Fast, Reliable Erlang Queue
Careful with the libraries....


❖   @tmm1’s amqp gem - runs off EventMachine, doesn’t play
    well with Passenger (asynch...
Evented!
A Few Notes about
                RabbitMQ

❖   enterprise-grade

❖   can be clustered using Erlang’s built-in awesome clu...
Queue 5: Resque
   github.com/defunkt/resque
Resque sits on top of Redis
Redis
    superfast data structures store
     like memcached, but smarter
     in-memory for the most part
   persisted t...
created by @defunkt
     of github.com
persists jobs to Redis as
     JSON objects
Easy to define new Jobs

any Ruby class that responds
    to ‘perform’ method
Queuing jobs




QUEUE=image_queue rake resque:work
QUEUE=* rake resque:work
Also comes with a Sinatra
app that lets you monitor
       your queues
A Few Notes about Resque

❖   has a dependency on Redis

❖   don’t have to write your own workers

    ❖   very nice worke...
Pick the right queue for
        the job
Do your own load-tests
 on your own stack
The Care & Feeding of
  Worker Daemons
More often than not, the
queues are not the problem.
Writing workers
in Ruby is easy...
... but writing Ruby workers
  that are long-running and
     reliable is NOT easy
   unless you’re using EventMachine or
...
Why so hard to write Ruby workers?


❖   You will wake up to workers with memory leaks

❖   Workers sometimes lock up

❖  ...
Use EventMachine
     Reactor Pattern
 Define callbacks on events
    pretty solid & fast
fork( )
spin up the worker in the new process
           close when done
     can kill bothersome workers
           easie...
Workling plugin
  github.com/purzelrakete/workling
Questions?
    @pradeep24
pradeep@intridea.com
Something new...
Llama

❖   A port of Apache Camel to Ruby

❖   based off EventMachine, so it runs ROCK SOLID

❖   early alpha at this point...
Llama Code Example
github.com/skyfallsin/llama
Upcoming SlideShare
Loading in …5
×

Message Queues in Ruby - An Overview

25,378 views
24,937 views

Published on

First given at RubyConf India 2010

Published in: Technology
3 Comments
45 Likes
Statistics
Notes
  • The new kid on the block that these slides don’t mention: http://www.simpleworker.com. Job queueing, worker processing and scheduling as a service which makes it easier to use than all of the things mentioned because not setup, no maintenance, etc.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • queue
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Wow! Very well explained slide.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
25,378
On SlideShare
0
From Embeds
0
Number of Embeds
2,530
Actions
Shares
0
Downloads
299
Comments
3
Likes
45
Embeds 0
No embeds

No notes for slide












































  • by Blaine Cook
  • used it on early version of Presently, works just fine, very stable


  • based on memcache-client, which will do server-based hashing
  • can submit/get any kind of string. will have








  • by Blaine Cook
  • used it on early version of Presently, works just fine, very stable












































































  • want to get rid of these


  • Message Queues in Ruby - An Overview

    1. 1. Message Queueing in Ruby An Overview Pradeep Elankumaran @pradeep24 pradeep@intridea.com
    2. 2. About Us ❖ Intridea is a high-end web consulting shop ❖ Based out of Washington DC, but dispersed all over the United States ❖ Our most popular product is Presently, an enterprise microblogging platform ❖ Grub.it is our latest product - mobile dish reviewing ❖ Various iPhone & Android apps (Car Finder, Grub.it) ❖ Specialize in web & mobile development using Ruby, Rails, Java, Erlang and beyond
    3. 3. Some background...
    4. 4. A request is supposed to return a response really fast.
    5. 5. However, there are certain actions that may take longer than a few milliseconds
    6. 6. A Few Examples ❖ Delivering e-mail ❖ Image processing (resizing, cropping etc.) ❖ Sending out notifications ❖ Geocoding a location ❖ Talking to other web services
    7. 7. These actions, if not extracted out to run asynchronously, will slow down your app
    8. 8. Bad user experience
    9. 9. What is a Message Queue?
    10. 10. A server that you submit asynchronous ‘jobs’ to Things to be done later
    11. 11. Interfaces between your Rails app and the ‘workers’ that perform each ‘job’
    12. 12. ‘Workers’ pull jobs from each queue and process them ASAP
    13. 13. Queues need to be FAST & RELIABLE
    14. 14. Queue 1: Delayed Job github.com/tobi/delayed_job
    15. 15. Extracted out from Shopify.com
    16. 16. Delayed Job is backed by your database.
    17. 17. Serializes Ruby Job classes (YAML) into a ‘jobs’ table
    18. 18. Easy-to-use syntax
    19. 19. rake jobs:run
    20. 20. Polls the database to get new jobs :(
    21. 21. A Few Notes about DJ ❖ used by Github exhaustively till they moved to Resque ❖ will get you 80% of the way there, has some nice advanced features (priority, retries etc.) ❖ should be careful not to overload your jobs table ❖ backed up jobs can lead to query slowdown on your site ❖ suggested that you monitor and profile workers before you launch on production
    22. 22. Queue 2: Starling github.com/starling/starling
    23. 23. Extracted out (infamously) from early versions of Twitter
    24. 24. Gets a bad rep for systems with high load, but pretty great for smaller apps
    25. 25. Speaks Memcached Protocol
    26. 26. Distributed from the client-side
    27. 27. Easy-to-use Syntax
    28. 28. Will have to write your own workers :(
    29. 29. Will have to poll the queue to get new jobs :(
    30. 30. A Few Notes about Starling ❖ persists to RAILS_ROOT/tmp/starling ❖ great for smaller apps, very lightweight ❖ again, will get you 80% of the way there ❖ writing robust, reliable workers is left up to you ❖ polling makes it slow compared to other queues
    31. 31. Queue 3: ActiveMQ activemq.apache.org
    32. 32. Mature Apache project.
    33. 33. Been around for a while, used by many, many websites & companies
    34. 34. Written in Java - however, can connect w/ Ruby using the STOMP protocol
    35. 35. Scripts provided to stop/start queue server daemon bin/activemq stop|start|restart
    36. 36. Use ActiveMessaging for easy integration or the STOMP gem for manual integration
    37. 37. ActiveMessaging plugin ❖ developed by ThoughtWorks ❖ provides a mini-framework to publish and subscribe to various ActiveMQ queues and topics ❖ mature, well-tested
    38. 38. Publish from any Ruby object
    39. 39. Define your own Processors script/poller run
    40. 40. Comes with a basic monitoring interface
    41. 41. Evented!
    42. 42. A Few Notes about ActiveMQ ❖ enterprise-grade ❖ can be clustered (edit XML files) ❖ using ActiveMessaging, no need to write your own workers ❖ STOMP gem is great if you do want to write your own workers
    43. 43. Queue 4: RabbitMQ rabbitmq.com
    44. 44. AMQP Advanced Message Queuing Protocol
    45. 45. AMQP can do lots of stuff... ❖ Point-to-point communication ❖ One-to-many broadcasting (including multicast) ❖ Transactional publication and acknowledgement ❖ High-speed transient message flows ❖ Reliable persistent message delivery ❖ File streaming
    46. 46. Fast, Reliable Erlang Queue
    47. 47. Careful with the libraries.... ❖ @tmm1’s amqp gem - runs off EventMachine, doesn’t play well with Passenger (asynchronous) ❖ warren - the best of the bunch for Rails (synchronous) ❖ bunny - lightweight (synchronous)
    48. 48. Evented!
    49. 49. A Few Notes about RabbitMQ ❖ enterprise-grade ❖ can be clustered using Erlang’s built-in awesome clustering ❖ will have to write your own workers unless you use Workling ❖ very, very fast
    50. 50. Queue 5: Resque github.com/defunkt/resque
    51. 51. Resque sits on top of Redis
    52. 52. Redis superfast data structures store like memcached, but smarter in-memory for the most part persisted to disk asynchronously sets, lists & corresponding operations
    53. 53. created by @defunkt of github.com
    54. 54. persists jobs to Redis as JSON objects
    55. 55. Easy to define new Jobs any Ruby class that responds to ‘perform’ method
    56. 56. Queuing jobs QUEUE=image_queue rake resque:work QUEUE=* rake resque:work
    57. 57. Also comes with a Sinatra app that lets you monitor your queues
    58. 58. A Few Notes about Resque ❖ has a dependency on Redis ❖ don’t have to write your own workers ❖ very nice worker fork() ❖ not evented :(
    59. 59. Pick the right queue for the job
    60. 60. Do your own load-tests on your own stack
    61. 61. The Care & Feeding of Worker Daemons
    62. 62. More often than not, the queues are not the problem.
    63. 63. Writing workers in Ruby is easy...
    64. 64. ... but writing Ruby workers that are long-running and reliable is NOT easy unless you’re using EventMachine or really know what you’re doing
    65. 65. Why so hard to write Ruby workers? ❖ You will wake up to workers with memory leaks ❖ Workers sometimes lock up ❖ Loading the whole Rails stack for each worker ❖ In general, a pain to manage these worker daemons
    66. 66. Use EventMachine Reactor Pattern Define callbacks on events pretty solid & fast
    67. 67. fork( ) spin up the worker in the new process close when done can kill bothersome workers easier debugging
    68. 68. Workling plugin github.com/purzelrakete/workling
    69. 69. Questions? @pradeep24 pradeep@intridea.com
    70. 70. Something new...
    71. 71. Llama ❖ A port of Apache Camel to Ruby ❖ based off EventMachine, so it runs ROCK SOLID ❖ early alpha at this point, looking for helpers ❖ github.com/skyfallsin/llama
    72. 72. Llama Code Example
    73. 73. github.com/skyfallsin/llama

    ×