1. Mike Willbanks Sr. Software Engineer at CaringBridge Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://joind.in/1375 The Art of Message Queues ConFoo.ca 2010
2. “ Message queues and mailboxes are software-engineering components used for interprocess communication, or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content.” http://en.wikipedia.org/wiki/Message_queue
3. “ Messaging describes the sending and receiving of data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.” http://www.rabbitmq.com/faq.html#what-is-messaging
4. A Basic Message Queue
5. What are Message Queues <ul><li>An application framework for sending and receiving messages. </li><ul><li>Message Oriented Middleware aka Enterprise Service Bus </li></ul><li>A way to communicate between applications / systems.
6. A way to decouple components.
7. A way to offload work. </li></ul>
8. Why are Message Queues Useful? <ul><li>Asynchronous Processing
9. Communication between Applications / Systems
11. Image Resizing
12. Video Processing
13. Sending out Emails
14. Auto-Scaling Virtual Instances
15. Log Analysis </li></ul>
16. You Might Use a Message Queue If... <ul><li>The request can be fulfilled in the background.
17. You need to communicate between multiple applications / systems.
18. You have a limited number of processing slots.
19. You need to support legacy applications or use multiple programming languages. </li></ul>
20. Message Queue Servers
21. Picking a Message Queue Server <ul><li>Durability (Memory, Disk, DBMS)
49. Message Formatting <ul><li>Anything you want it to be! </li><ul><li>XML
51. … </li></ul><li>Remember the purpose is to decouple and application agnostic. </li></ul>
52. Workers / Consumers to 1 Task <ul><li>Workers / Consumers should be dumb, they should not need to do more than a single task. </li><ul><li>If they do, write another message to the queue for a separate worker to do the work. </li><ul><li>Exception: Legacy Code </li></ul></ul></ul>
53. A Simple Architecture Queue Receive Client / Publisher Message Queue Server Worker / Consumer
54. A Complex Architecture Client Message Queue Server Worker Worker Worker Worker Worker Message Queue Server Message Queue Server Client Client
55. Things to Keep in Mind <ul><li>Consumers and Servers should be monitored </li><ul><li>Supervisord (http://supervisord.org/) is excellent at this.
56. Memory Usage of Consumers (in the event of leaks, kill and let supervisord re-create it) </li></ul><li>Persistence of Data </li><ul><li>Many have multiple backends or built-in backends. </li></ul><li>High Availability </li><ul><li>Not all clients have built in support for multiple servers, you may need to abstract it. </li></ul><li>Message queues do not handle scheduled processing (aka wait 4 days to process the message). </li></ul>
57. Use Cases
58. Email Notifications <ul><li>A user takes an action on a page that needs to notify all subscribing members.
59. Send a message to the queue regarding the action, then queue emails to send in another queue. </li><ul><li>Why 2 queues </li><ul><li>Allows us to handle getting everyone that needs the notification.
60. Then allows us to focus on sending email through a single queue with multiple workers. </li></ul></ul></ul>
61. Auto-Scaling <ul><li>A scheduled task polls the servers looking for potential load issues. If load exceeds 80% or 80% of web server processes are taken send a message to the queue to boot up a new instance.
62. The worker or consumer then utilizes a service call or logic to boot up a new instance. </li></ul>
63. Video / Photos <ul><li>Since videos and photos can take up a lot of memory and cpu time when resizing, you may not want to process them all at once.
64. When a new video or photo comes in, send a message to the queue to do the work. Keep only enough workers / consumers available to not strain your machine.
65. Remember to monitor how many jobs are getting backed up :) </li></ul>
66. Mike Willbanks Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://joind.in/1375 Questions?