Message Queues in Ruby - An Overview
Upcoming SlideShare
Loading in...5
×
 

Message Queues in Ruby - An Overview

on

  • 25,479 views

First given at RubyConf India 2010

First given at RubyConf India 2010

Statistics

Views

Total Views
25,479
Views on SlideShare
23,006
Embed Views
2,473

Actions

Likes
40
Downloads
284
Comments
3

14 Embeds 2,473

http://abrdev.com 2313
http://www.slideshare.net 97
http://xlii.ghost.io 25
http://www.linkedin.com 9
url_unknown 8
https://apps.fujisan.co.jp 4
http://a0.twimg.com 4
http://paper.li 3
http://twitter.com 2
http://bundlr.com 2
http://us-w1.rockmelt.com 2
http://www.lmodules.com 2
http://hghltd.yandex.net 1
https://www.linkedin.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • The new kid on the block that these slides don’t mention: http://www.simpleworker.com. Job queueing, worker processing and scheduling as a service which makes it easier to use than all of the things mentioned because not setup, no maintenance, etc.
    Are you sure you want to
    Your message goes here
    Processing…
  • queue
    Are you sure you want to
    Your message goes here
    Processing…
  • Wow! Very well explained slide.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • by Blaine Cook <br />
  • used it on early version of Presently, works just fine, very stable <br />
  • <br /> <br />
  • based on memcache-client, which will do server-based hashing <br />
  • can submit/get any kind of string. will have <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • by Blaine Cook <br />
  • used it on early version of Presently, works just fine, very stable <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • <br /> <br />
  • want to get rid of these <br />
  • <br /> <br />

Message Queues in Ruby - An Overview Message Queues in Ruby - An Overview Presentation Transcript

  • Message Queueing in Ruby An Overview Pradeep Elankumaran @pradeep24 pradeep@intridea.com
  • About Us ❖ Intridea is a high-end web consulting shop ❖ Based out of Washington DC, but dispersed all over the United States ❖ Our most popular product is Presently, an enterprise microblogging platform ❖ Grub.it is our latest product - mobile dish reviewing ❖ Various iPhone & Android apps (Car Finder, Grub.it) ❖ Specialize in web & mobile development using Ruby, Rails, Java, Erlang and beyond
  • Some background...
  • A request is supposed to return a response really fast.
  • However, there are certain actions that may take longer than a few milliseconds
  • A Few Examples ❖ Delivering e-mail ❖ Image processing (resizing, cropping etc.) ❖ Sending out notifications ❖ Geocoding a location ❖ Talking to other web services
  • These actions, if not extracted out to run asynchronously, will slow down your app
  • Bad user experience
  • What is a Message Queue?
  • A server that you submit asynchronous ‘jobs’ to Things to be done later
  • Interfaces between your Rails app and the ‘workers’ that perform each ‘job’
  • ‘Workers’ pull jobs from each queue and process them ASAP
  • Queues need to be FAST & RELIABLE
  • Queue 1: Delayed Job github.com/tobi/delayed_job
  • Extracted out from Shopify.com
  • Delayed Job is backed by your database.
  • Serializes Ruby Job classes (YAML) into a ‘jobs’ table
  • Easy-to-use syntax
  • rake jobs:run
  • Polls the database to get new jobs :(
  • A Few Notes about DJ ❖ used by Github exhaustively till they moved to Resque ❖ will get you 80% of the way there, has some nice advanced features (priority, retries etc.) ❖ should be careful not to overload your jobs table ❖ backed up jobs can lead to query slowdown on your site ❖ suggested that you monitor and profile workers before you launch on production
  • Queue 2: Starling github.com/starling/starling
  • Extracted out (infamously) from early versions of Twitter
  • Gets a bad rep for systems with high load, but pretty great for smaller apps
  • Speaks Memcached Protocol
  • Distributed from the client-side
  • Easy-to-use Syntax
  • Will have to write your own workers :(
  • Will have to poll the queue to get new jobs :(
  • A Few Notes about Starling ❖ persists to RAILS_ROOT/tmp/starling ❖ great for smaller apps, very lightweight ❖ again, will get you 80% of the way there ❖ writing robust, reliable workers is left up to you ❖ polling makes it slow compared to other queues
  • Queue 3: ActiveMQ activemq.apache.org
  • Mature Apache project.
  • Been around for a while, used by many, many websites & companies
  • Written in Java - however, can connect w/ Ruby using the STOMP protocol
  • Scripts provided to stop/start queue server daemon bin/activemq stop|start|restart
  • Use ActiveMessaging for easy integration or the STOMP gem for manual integration
  • ActiveMessaging plugin ❖ developed by ThoughtWorks ❖ provides a mini-framework to publish and subscribe to various ActiveMQ queues and topics ❖ mature, well-tested
  • Publish from any Ruby object
  • Define your own Processors script/poller run
  • Comes with a basic monitoring interface
  • Evented!
  • A Few Notes about ActiveMQ ❖ enterprise-grade ❖ can be clustered (edit XML files) ❖ using ActiveMessaging, no need to write your own workers ❖ STOMP gem is great if you do want to write your own workers
  • Queue 4: RabbitMQ rabbitmq.com
  • AMQP Advanced Message Queuing Protocol
  • AMQP can do lots of stuff... ❖ Point-to-point communication ❖ One-to-many broadcasting (including multicast) ❖ Transactional publication and acknowledgement ❖ High-speed transient message flows ❖ Reliable persistent message delivery ❖ File streaming
  • Fast, Reliable Erlang Queue
  • Careful with the libraries.... ❖ @tmm1’s amqp gem - runs off EventMachine, doesn’t play well with Passenger (asynchronous) ❖ warren - the best of the bunch for Rails (synchronous) ❖ bunny - lightweight (synchronous)
  • Evented!
  • A Few Notes about RabbitMQ ❖ enterprise-grade ❖ can be clustered using Erlang’s built-in awesome clustering ❖ will have to write your own workers unless you use Workling ❖ very, very fast
  • Queue 5: Resque github.com/defunkt/resque
  • Resque sits on top of Redis
  • Redis superfast data structures store like memcached, but smarter in-memory for the most part persisted to disk asynchronously sets, lists & corresponding operations
  • created by @defunkt of github.com
  • persists jobs to Redis as JSON objects
  • Easy to define new Jobs any Ruby class that responds to ‘perform’ method
  • Queuing jobs QUEUE=image_queue rake resque:work QUEUE=* rake resque:work
  • Also comes with a Sinatra app that lets you monitor your queues
  • A Few Notes about Resque ❖ has a dependency on Redis ❖ don’t have to write your own workers ❖ very nice worker fork() ❖ not evented :(
  • Pick the right queue for the job
  • Do your own load-tests on your own stack
  • The Care & Feeding of Worker Daemons
  • More often than not, the queues are not the problem.
  • Writing workers in Ruby is easy...
  • ... but writing Ruby workers that are long-running and reliable is NOT easy unless you’re using EventMachine or really know what you’re doing
  • Why so hard to write Ruby workers? ❖ You will wake up to workers with memory leaks ❖ Workers sometimes lock up ❖ Loading the whole Rails stack for each worker ❖ In general, a pain to manage these worker daemons
  • Use EventMachine Reactor Pattern Define callbacks on events pretty solid & fast
  • fork( ) spin up the worker in the new process close when done can kill bothersome workers easier debugging
  • Workling plugin github.com/purzelrakete/workling
  • Questions? @pradeep24 pradeep@intridea.com
  • Something new...
  • Llama ❖ A port of Apache Camel to Ruby ❖ based off EventMachine, so it runs ROCK SOLID ❖ early alpha at this point, looking for helpers ❖ github.com/skyfallsin/llama
  • Llama Code Example
  • github.com/skyfallsin/llama