Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Art Of Message Queues


Published on

Published in: Technology

Art Of Message Queues

  1. 1. Mike Willbanks Sr. Software Engineer at CaringBridge Blog: Twitter : mwillbanks IRC : lubs on freenode Talk: The Art of Message Queues 2010
  2. 2. “ Message queues and mailboxes are software-engineering components used for interprocess communication, or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content.”
  3. 3. “ Messaging describes the sending and receiving of data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.”
  4. 4. A Basic Message Queue
  5. 5. What are Message Queues <ul><li>An application framework for sending and receiving messages. </li><ul><li>Message Oriented Middleware aka Enterprise Service Bus </li></ul><li>A way to communicate between applications / systems.
  6. 6. A way to decouple components.
  7. 7. A way to offload work. </li></ul>
  8. 8. Why are Message Queues Useful? <ul><li>Asynchronous Processing
  9. 9. Communication between Applications / Systems
  10. 10. Chat
  11. 11. Image Resizing
  12. 12. Video Processing
  13. 13. Sending out Emails
  14. 14. Auto-Scaling Virtual Instances
  15. 15. Log Analysis </li></ul>
  16. 16. You Might Use a Message Queue If... <ul><li>The request can be fulfilled in the background.
  17. 17. You need to communicate between multiple applications / systems.
  18. 18. You have a limited number of processing slots.
  19. 19. You need to support legacy applications or use multiple programming languages. </li></ul>
  20. 20. Message Queue Servers
  21. 21. Picking a Message Queue Server <ul><li>Durability (Memory, Disk, DBMS)
  22. 22. Scalability (Horizontal / Vertical)
  23. 23. Synchronous / Asynchronous
  24. 24. Batching Policies
  25. 25. Delivery Policies
  26. 26. Purging Policies
  27. 27. Routing Policies
  28. 28. Security Policies
  29. 29. Standards </li></ul>
  30. 30. Standard Protocols <ul><li>AMQP – Advanced Message Queue Protocol
  31. 31. STOMP – Streaming Text Oriented Messaging Protocol
  32. 32. XMPP - Extensible Messaging and Presence Protocol </li></ul>
  33. 33. What I'll Be Using <ul><li>RabbitMQ – AMQP based
  34. 34. ActiveMQ – Stomp Transport
  35. 35. Beanstalkd – Non-standard based </li></ul>
  36. 36. Installing Servers <ul><li>RabbitMQ – AMQP </li><ul><li>Download Server:
  37. 37. Experimental Client: php-rabbit </li><ul><li>Requires rabbitmq-c and rabbitmq-codegen
  38. 38. </li></ul></ul><li>ActiveMQ </li><ul><li>Download binary:
  39. 39. Client: pecl install stomp </li></ul><li>Beanstalkd </li><ul><li>Download from:
  40. 40. Client from: </li></ul></ul>
  41. 41. Some Code
  42. 42. A Simple Publish with AMQP
  43. 43. A Simple Receive with AMQP
  44. 44. A Simple Publish with Stomp
  45. 45. A Simple Receive with Stomp
  46. 46. A Simple Publish with Beanstalk
  47. 47. A Simple Receive with Beanstalk
  48. 48. How to Use Your Queues
  49. 49. Message Formatting <ul><li>Anything you want it to be! </li><ul><li>XML
  50. 50. JSON
  51. 51. … </li></ul><li>Remember the purpose is to decouple and application agnostic. </li></ul>
  52. 52. Workers / Consumers to 1 Task <ul><li>Workers / Consumers should be dumb, they should not need to do more than a single task. </li><ul><li>If they do, write another message to the queue for a separate worker to do the work. </li><ul><li>Exception: Legacy Code </li></ul></ul></ul>
  53. 53. A Simple Architecture Queue Receive Client / Publisher Message Queue Server Worker / Consumer
  54. 54. A Complex Architecture Client Message Queue Server Worker Worker Worker Worker Worker Message Queue Server Message Queue Server Client Client
  55. 55. Things to Keep in Mind <ul><li>Consumers and Servers should be monitored </li><ul><li>Supervisord ( is excellent at this.
  56. 56. Memory Usage of Consumers (in the event of leaks, kill and let supervisord re-create it) </li></ul><li>Persistence of Data </li><ul><li>Many have multiple backends or built-in backends. </li></ul><li>High Availability </li><ul><li>Not all clients have built in support for multiple servers, you may need to abstract it. </li></ul><li>Message queues do not handle scheduled processing (aka wait 4 days to process the message). </li></ul>
  57. 57. Use Cases
  58. 58. Email Notifications <ul><li>A user takes an action on a page that needs to notify all subscribing members.
  59. 59. Send a message to the queue regarding the action, then queue emails to send in another queue. </li><ul><li>Why 2 queues </li><ul><li>Allows us to handle getting everyone that needs the notification.
  60. 60. Then allows us to focus on sending email through a single queue with multiple workers. </li></ul></ul></ul>
  61. 61. Auto-Scaling <ul><li>A scheduled task polls the servers looking for potential load issues. If load exceeds 80% or 80% of web server processes are taken send a message to the queue to boot up a new instance.
  62. 62. The worker or consumer then utilizes a service call or logic to boot up a new instance. </li></ul>
  63. 63. Video / Photos <ul><li>Since videos and photos can take up a lot of memory and cpu time when resizing, you may not want to process them all at once.
  64. 64. When a new video or photo comes in, send a message to the queue to do the work. Keep only enough workers / consumers available to not strain your machine.
  65. 65. Remember to monitor how many jobs are getting backed up :) </li></ul>
  66. 66. Mike Willbanks Blog: Twitter : mwillbanks IRC : lubs on freenode Talk: Questions?