Your SlideShare is downloading. ×
0
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Art Of Message Queues
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Art Of Message Queues

8,785

Published on

Published in: Technology
0 Comments
18 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
8,785
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
199
Comments
0
Likes
18
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Mike Willbanks Sr. Software Engineer at CaringBridge Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://joind.in/1375 The Art of Message Queues ConFoo.ca 2010
  • 2. “ Message queues and mailboxes are software-engineering components used for interprocess communication, or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content.” http://en.wikipedia.org/wiki/Message_queue
  • 3. “ Messaging describes the sending and receiving of data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.” http://www.rabbitmq.com/faq.html#what-is-messaging
  • 4. A Basic Message Queue
  • 5. What are Message Queues <ul><li>An application framework for sending and receiving messages. </li><ul><li>Message Oriented Middleware aka Enterprise Service Bus </li></ul><li>A way to communicate between applications / systems.
  • 6. A way to decouple components.
  • 7. A way to offload work. </li></ul>
  • 8. Why are Message Queues Useful? <ul><li>Asynchronous Processing
  • 9. Communication between Applications / Systems
  • 10. Chat
  • 11. Image Resizing
  • 12. Video Processing
  • 13. Sending out Emails
  • 14. Auto-Scaling Virtual Instances
  • 15. Log Analysis </li></ul>
  • 16. You Might Use a Message Queue If... <ul><li>The request can be fulfilled in the background.
  • 17. You need to communicate between multiple applications / systems.
  • 18. You have a limited number of processing slots.
  • 19. You need to support legacy applications or use multiple programming languages. </li></ul>
  • 20. Message Queue Servers
  • 21. Picking a Message Queue Server <ul><li>Durability (Memory, Disk, DBMS)
  • 22. Scalability (Horizontal / Vertical)
  • 23. Synchronous / Asynchronous
  • 24. Batching Policies
  • 25. Delivery Policies
  • 26. Purging Policies
  • 27. Routing Policies
  • 28. Security Policies
  • 29. Standards </li></ul>http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
  • 30. Standard Protocols <ul><li>AMQP – Advanced Message Queue Protocol
  • 31. STOMP – Streaming Text Oriented Messaging Protocol
  • 32. XMPP - Extensible Messaging and Presence Protocol </li></ul>
  • 33. What I'll Be Using <ul><li>RabbitMQ – AMQP based
  • 34. ActiveMQ – Stomp Transport
  • 35. Beanstalkd – Non-standard based </li></ul>
  • 36. Installing Servers <ul><li>RabbitMQ – AMQP </li><ul><li>Download Server: http://www.rabbitmq.com/server.html
  • 37. Experimental Client: php-rabbit </li><ul><li>Requires rabbitmq-c and rabbitmq-codegen
  • 38. http://code.google.com/p/php-rabbit/ </li></ul></ul><li>ActiveMQ </li><ul><li>Download binary: http://activemq.apache.org/activemq-530-release.html
  • 39. Client: pecl install stomp </li></ul><li>Beanstalkd </li><ul><li>Download from: http://kr.github.com/beanstalkd/
  • 40. Client from: http://github.com/pda/pheanstalk/ </li></ul></ul>
  • 41. Some Code
  • 42. A Simple Publish with AMQP
  • 43. A Simple Receive with AMQP
  • 44. A Simple Publish with Stomp
  • 45. A Simple Receive with Stomp
  • 46. A Simple Publish with Beanstalk
  • 47. A Simple Receive with Beanstalk
  • 48. How to Use Your Queues
  • 49. Message Formatting <ul><li>Anything you want it to be! </li><ul><li>XML
  • 50. JSON
  • 51. … </li></ul><li>Remember the purpose is to decouple and application agnostic. </li></ul>
  • 52. Workers / Consumers to 1 Task <ul><li>Workers / Consumers should be dumb, they should not need to do more than a single task. </li><ul><li>If they do, write another message to the queue for a separate worker to do the work. </li><ul><li>Exception: Legacy Code </li></ul></ul></ul>
  • 53. A Simple Architecture Queue Receive Client / Publisher Message Queue Server Worker / Consumer
  • 54. A Complex Architecture Client Message Queue Server Worker Worker Worker Worker Worker Message Queue Server Message Queue Server Client Client
  • 55. Things to Keep in Mind <ul><li>Consumers and Servers should be monitored </li><ul><li>Supervisord (http://supervisord.org/) is excellent at this.
  • 56. Memory Usage of Consumers (in the event of leaks, kill and let supervisord re-create it) </li></ul><li>Persistence of Data </li><ul><li>Many have multiple backends or built-in backends. </li></ul><li>High Availability </li><ul><li>Not all clients have built in support for multiple servers, you may need to abstract it. </li></ul><li>Message queues do not handle scheduled processing (aka wait 4 days to process the message). </li></ul>
  • 57. Use Cases
  • 58. Email Notifications <ul><li>A user takes an action on a page that needs to notify all subscribing members.
  • 59. Send a message to the queue regarding the action, then queue emails to send in another queue. </li><ul><li>Why 2 queues </li><ul><li>Allows us to handle getting everyone that needs the notification.
  • 60. Then allows us to focus on sending email through a single queue with multiple workers. </li></ul></ul></ul>
  • 61. Auto-Scaling <ul><li>A scheduled task polls the servers looking for potential load issues. If load exceeds 80% or 80% of web server processes are taken send a message to the queue to boot up a new instance.
  • 62. The worker or consumer then utilizes a service call or logic to boot up a new instance. </li></ul>
  • 63. Video / Photos <ul><li>Since videos and photos can take up a lot of memory and cpu time when resizing, you may not want to process them all at once.
  • 64. When a new video or photo comes in, send a message to the queue to do the work. Keep only enough workers / consumers available to not strain your machine.
  • 65. Remember to monitor how many jobs are getting backed up :) </li></ul>
  • 66. Mike Willbanks Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://joind.in/1375 Questions?

×