Mike Willbanks Sr. Software Engineer at CaringBridge Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://joind.in/1375 The Art of Message Queues ConFoo.ca 2010
“ Message queues and mailboxes are software-engineering components used for interprocess communication, or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content.” http://en.wikipedia.org/wiki/Message_queue
“ Messaging describes the sending and receiving of data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.” http://www.rabbitmq.com/faq.html#what-is-messaging
What are Message Queues <ul><li>An application framework for sending and receiving messages. </li><ul><li>Message Oriented Middleware aka Enterprise Service Bus </li></ul><li>A way to communicate between applications / systems.
… </li></ul><li>Remember the purpose is to decouple and application agnostic. </li></ul>
Workers / Consumers to 1 Task <ul><li>Workers / Consumers should be dumb, they should not need to do more than a single task. </li><ul><li>If they do, write another message to the queue for a separate worker to do the work. </li><ul><li>Exception: Legacy Code </li></ul></ul></ul>
A Simple Architecture Queue Receive Client / Publisher Message Queue Server Worker / Consumer
A Complex Architecture Client Message Queue Server Worker Worker Worker Worker Worker Message Queue Server Message Queue Server Client Client
Things to Keep in Mind <ul><li>Consumers and Servers should be monitored </li><ul><li>Supervisord (http://supervisord.org/) is excellent at this.
Memory Usage of Consumers (in the event of leaks, kill and let supervisord re-create it) </li></ul><li>Persistence of Data </li><ul><li>Many have multiple backends or built-in backends. </li></ul><li>High Availability </li><ul><li>Not all clients have built in support for multiple servers, you may need to abstract it. </li></ul><li>Message queues do not handle scheduled processing (aka wait 4 days to process the message). </li></ul>
Email Notifications <ul><li>A user takes an action on a page that needs to notify all subscribing members.
Send a message to the queue regarding the action, then queue emails to send in another queue. </li><ul><li>Why 2 queues </li><ul><li>Allows us to handle getting everyone that needs the notification.
Then allows us to focus on sending email through a single queue with multiple workers. </li></ul></ul></ul>
Auto-Scaling <ul><li>A scheduled task polls the servers looking for potential load issues. If load exceeds 80% or 80% of web server processes are taken send a message to the queue to boot up a new instance.
The worker or consumer then utilizes a service call or logic to boot up a new instance. </li></ul>
Video / Photos <ul><li>Since videos and photos can take up a lot of memory and cpu time when resizing, you may not want to process them all at once.
When a new video or photo comes in, send a message to the queue to do the work. Keep only enough workers / consumers available to not strain your machine.
Remember to monitor how many jobs are getting backed up :) </li></ul>
Mike Willbanks Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://joind.in/1375 Questions?