Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Art of Message Queues


Published on

Goes through some simplistic items on what message queues are and how you can define them.

Published in: Technology

The Art of Message Queues

  1. 1. Mike Willbanks Blog: Twitter : mwillbanks IRC : lubs on freenode Talk: The Art of Message Queues MinneBar 2009
  2. 2. What are Message Queues <ul><li>The sending and/or receiving of data [messages] between systems.
  3. 3. An exchange of information between systems, programs and/or components.
  4. 4. The decoupling of application components.
  5. 5. A FIFO buffer
  6. 6. What is your definition? </li></ul>
  7. 7. Why are Message Queues Useful? <ul><li>Buzzwords </li><ul><li>Durability
  8. 8. Reliability
  9. 9. Scalability </li></ul><li>Offline Processing
  10. 10. Communication between Systems
  11. 11. Asynchronous Processing
  12. 12. Scheduling Operations
  13. 13. There are many more uses... </li></ul>
  14. 14. Usefulness By Example
  15. 15. Social Network Notification Emails <ul>Synopsis: When a user adds a photo an email is sent to each of their friends. Presently, this happens when the user adds the photo and complaints are taking place that it takes too long to add a photo or the system times out. Solution: When a photo is added add a message to the queue a worker retrieves the message to process sending out emails to the friends of the user. </ul>
  16. 16. Video Processing <ul>Synopsis: When a user uploads a video, this video needs to be converted and compressed by ffmpeg before adding it to the users area. Solution: Once the video has been uploaded a message is sent to the queue. A worker picks up the queue item when there is processing available, processes the video and marks it as active. </ul>
  17. 17. Asynchronous Processing <ul>Synopsis: Your E-commerce system needs to do several things: verify credit card details, hold an item, submit shipment details, etc. Your boss wants to do these operations at the same time. Solution: Send a message to the queue for each item do your additional processing and process the results as each item comes in. </ul>
  18. 18. Grid Computing <ul>Synopsis: You have a lot of hardware that during down periods you want to save on data center costs or many virtual machines. These are all running today and you want to add / remove them as load increases. Solution: Implement a scheduled task to poll the servers load, after a certain threshold is reached add a message to the queue. The worker could be created to add a new [virtual] machine to the stack. </ul>
  19. 19. Message Queue Servers / Services
  20. 20. Some of the Players <ul><li>Amazon SQS
  21. 21. Active MQ
  22. 22. Beanstalkd
  23. 23. Dropr
  24. 24. Gearman
  25. 25. JMS
  26. 26. MemcacheQ
  27. 27. Minimalist Queue Services
  28. 28. RabbitMQ
  29. 29. Spread </li></ul>
  30. 30. Message Formatting <ul><li>Anything you want it to be! </li><ul><li>Serialized Input
  31. 31. XML
  32. 32. JSON
  33. 33. … </li></ul><li>Although... for the sanity of others don't make a new language in the message :) </li></ul>
  34. 34. A Simple Architecture Queue Receive Client Message Queue Server Worker
  35. 35. A Complex Architecture Client Message Queue Server Worker Worker Worker Worker Worker Message Queue Server Message Queue Server Client Client
  36. 36. Demos
  37. 37. Mike Willbanks Blog: Twitter : mwillbanks IRC : lubs on freenode Talk: Questions?