Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Message Queues in Ruby - An Overview

26,596 views

Published on

First given at RubyConf India 2010

Published in: Technology
  • DOWNLOAD THI5 BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • The new kid on the block that these slides don’t mention: http://www.simpleworker.com. Job queueing, worker processing and scheduling as a service which makes it easier to use than all of the things mentioned because not setup, no maintenance, etc.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Message Queues in Ruby - An Overview

  1. 1. Message Queueing in Ruby An Overview Pradeep Elankumaran @pradeep24 pradeep@intridea.com
  2. 2. About Us ❖ Intridea is a high-end web consulting shop ❖ Based out of Washington DC, but dispersed all over the United States ❖ Our most popular product is Presently, an enterprise microblogging platform ❖ Grub.it is our latest product - mobile dish reviewing ❖ Various iPhone & Android apps (Car Finder, Grub.it) ❖ Specialize in web & mobile development using Ruby, Rails, Java, Erlang and beyond
  3. 3. Some background...
  4. 4. A request is supposed to return a response really fast.
  5. 5. However, there are certain actions that may take longer than a few milliseconds
  6. 6. A Few Examples ❖ Delivering e-mail ❖ Image processing (resizing, cropping etc.) ❖ Sending out notifications ❖ Geocoding a location ❖ Talking to other web services
  7. 7. These actions, if not extracted out to run asynchronously, will slow down your app
  8. 8. Bad user experience
  9. 9. What is a Message Queue?
  10. 10. A server that you submit asynchronous ‘jobs’ to Things to be done later
  11. 11. Interfaces between your Rails app and the ‘workers’ that perform each ‘job’
  12. 12. ‘Workers’ pull jobs from each queue and process them ASAP
  13. 13. Queues need to be FAST & RELIABLE
  14. 14. Queue 1: Delayed Job github.com/tobi/delayed_job
  15. 15. Extracted out from Shopify.com
  16. 16. Delayed Job is backed by your database.
  17. 17. Serializes Ruby Job classes (YAML) into a ‘jobs’ table
  18. 18. Easy-to-use syntax
  19. 19. rake jobs:run
  20. 20. Polls the database to get new jobs :(
  21. 21. A Few Notes about DJ ❖ used by Github exhaustively till they moved to Resque ❖ will get you 80% of the way there, has some nice advanced features (priority, retries etc.) ❖ should be careful not to overload your jobs table ❖ backed up jobs can lead to query slowdown on your site ❖ suggested that you monitor and profile workers before you launch on production
  22. 22. Queue 2: Starling github.com/starling/starling
  23. 23. Extracted out (infamously) from early versions of Twitter
  24. 24. Gets a bad rep for systems with high load, but pretty great for smaller apps
  25. 25. Speaks Memcached Protocol
  26. 26. Distributed from the client-side
  27. 27. Easy-to-use Syntax
  28. 28. Will have to write your own workers :(
  29. 29. Will have to poll the queue to get new jobs :(
  30. 30. A Few Notes about Starling ❖ persists to RAILS_ROOT/tmp/starling ❖ great for smaller apps, very lightweight ❖ again, will get you 80% of the way there ❖ writing robust, reliable workers is left up to you ❖ polling makes it slow compared to other queues
  31. 31. Queue 3: ActiveMQ activemq.apache.org
  32. 32. Mature Apache project.
  33. 33. Been around for a while, used by many, many websites & companies
  34. 34. Written in Java - however, can connect w/ Ruby using the STOMP protocol
  35. 35. Scripts provided to stop/start queue server daemon bin/activemq stop|start|restart
  36. 36. Use ActiveMessaging for easy integration or the STOMP gem for manual integration
  37. 37. ActiveMessaging plugin ❖ developed by ThoughtWorks ❖ provides a mini-framework to publish and subscribe to various ActiveMQ queues and topics ❖ mature, well-tested
  38. 38. Publish from any Ruby object
  39. 39. Define your own Processors script/poller run
  40. 40. Comes with a basic monitoring interface
  41. 41. Evented!
  42. 42. A Few Notes about ActiveMQ ❖ enterprise-grade ❖ can be clustered (edit XML files) ❖ using ActiveMessaging, no need to write your own workers ❖ STOMP gem is great if you do want to write your own workers
  43. 43. Queue 4: RabbitMQ rabbitmq.com
  44. 44. AMQP Advanced Message Queuing Protocol
  45. 45. AMQP can do lots of stuff... ❖ Point-to-point communication ❖ One-to-many broadcasting (including multicast) ❖ Transactional publication and acknowledgement ❖ High-speed transient message flows ❖ Reliable persistent message delivery ❖ File streaming
  46. 46. Fast, Reliable Erlang Queue
  47. 47. Careful with the libraries.... ❖ @tmm1’s amqp gem - runs off EventMachine, doesn’t play well with Passenger (asynchronous) ❖ warren - the best of the bunch for Rails (synchronous) ❖ bunny - lightweight (synchronous)
  48. 48. Evented!
  49. 49. A Few Notes about RabbitMQ ❖ enterprise-grade ❖ can be clustered using Erlang’s built-in awesome clustering ❖ will have to write your own workers unless you use Workling ❖ very, very fast
  50. 50. Queue 5: Resque github.com/defunkt/resque
  51. 51. Resque sits on top of Redis
  52. 52. Redis superfast data structures store like memcached, but smarter in-memory for the most part persisted to disk asynchronously sets, lists & corresponding operations
  53. 53. created by @defunkt of github.com
  54. 54. persists jobs to Redis as JSON objects
  55. 55. Easy to define new Jobs any Ruby class that responds to ‘perform’ method
  56. 56. Queuing jobs QUEUE=image_queue rake resque:work QUEUE=* rake resque:work
  57. 57. Also comes with a Sinatra app that lets you monitor your queues
  58. 58. A Few Notes about Resque ❖ has a dependency on Redis ❖ don’t have to write your own workers ❖ very nice worker fork() ❖ not evented :(
  59. 59. Pick the right queue for the job
  60. 60. Do your own load-tests on your own stack
  61. 61. The Care & Feeding of Worker Daemons
  62. 62. More often than not, the queues are not the problem.
  63. 63. Writing workers in Ruby is easy...
  64. 64. ... but writing Ruby workers that are long-running and reliable is NOT easy unless you’re using EventMachine or really know what you’re doing
  65. 65. Why so hard to write Ruby workers? ❖ You will wake up to workers with memory leaks ❖ Workers sometimes lock up ❖ Loading the whole Rails stack for each worker ❖ In general, a pain to manage these worker daemons
  66. 66. Use EventMachine Reactor Pattern Define callbacks on events pretty solid & fast
  67. 67. fork( ) spin up the worker in the new process close when done can kill bothersome workers easier debugging
  68. 68. Workling plugin github.com/purzelrakete/workling
  69. 69. Questions? @pradeep24 pradeep@intridea.com
  70. 70. Something new...
  71. 71. Llama ❖ A port of Apache Camel to Ruby ❖ based off EventMachine, so it runs ROCK SOLID ❖ early alpha at this point, looking for helpers ❖ github.com/skyfallsin/llama
  72. 72. Llama Code Example
  73. 73. github.com/skyfallsin/llama

×