Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Art of Message Queues - TEKX


Published on

Published in: Technology

The Art of Message Queues - TEKX

  1. 1. Mike Willbanks Sr. Software Engineer at CaringBridge Blog: Twitter : mwillbanks IRC : lubs on freenode Talk: The Art of Message Queues TEK-X
  2. 2. Who Am I? <ul><li>Sr. Sofware Engineer at CaringBridge
  3. 3. PHP monkey
  4. 4. MNPHP User Group
  5. 5. Interested in Scalability & Performance </li></ul>
  6. 6. “ Message queues and mailboxes are software-engineering components used for interprocess communication, or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content.”
  7. 7. “ Messaging describes the sending and receiving of data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.”
  8. 8. Messages are Everywhere
  9. 9. A Basic Message Queue
  10. 10. What are Message Queues <ul><li>A FIFO buffer
  11. 11. Asynchronous push / pull
  12. 12. An application framework for sending and receiving messages.
  13. 13. A way to communicate between applications / systems.
  14. 14. A way to decouple components.
  15. 15. A way to offload work. </li></ul>
  16. 16. Why are Message Queues Useful? <ul><li>Asynchronous Processing
  17. 17. Communication between Applications / Systems
  18. 18. Image Resizing
  19. 19. Video Processing
  20. 20. Sending out Emails
  21. 21. Auto-Scaling Virtual Instances
  22. 22. Log Analysis
  23. 23. The list goes on... </li></ul>
  24. 24. You Might Use a Message Queue If... <ul><li>The request can be fulfilled in the background.
  25. 25. You need to communicate between multiple applications / systems.
  26. 26. You have a limited number of processing slots.
  27. 27. You need to support legacy applications or use multiple programming languages. </li></ul>Good short presentation on why you might need it:
  28. 28. Going back to Unix <ul><li>“Write programs that work together.”
  29. 29. “Do it in the background.” </li></ul>
  30. 30. Why does that matter? <ul><li>Web systems need to be geared to run things asynchronously.
  31. 31. Distribution may also come into play. </li></ul>
  32. 32. Protocols
  33. 33. Standardized Protocols <ul><li>AMQP – Advanced Message Queue Protocol
  34. 34. STOMP – Streaming Text Oriented Messaging Protocol
  35. 35. XMPP - Extensible Messaging and Presence Protocol </li><ul><li>Related with patches, I will not be going into this. However, XMPP is the next topic in this room. </li></ul></ul>
  36. 36. AMQP
  37. 37. Overview <ul><li>AMQP Working Group (Community and Vendor)
  38. 38. Platform agnostic protocol.
  39. 39. Completely open, interoperable and broadly applicable.
  40. 40. Many severs available and many client libraries. </li></ul> For more information on the protocol.
  41. 41. How it works. <ul><li>AMQP utilizes exchanges, queues and bindings.
  42. 42. An exchange is routers with routing tables.
  43. 43. A queue is where the messages wait for a consumer.
  44. 44. A binding defines the routing rules. </li></ul>
  45. 45. In a nutshell....
  46. 46. Some of the Various Servers <ul><li>Apache Qpid -
  47. 47. OpenAMQ -
  48. 48. RabbitMQ -
  49. 49. ZeroMQ - </li></ul>
  50. 50. A PHP Client <ul><li>php-rabbit – </li><ul><li>Looks like it will become a PECL package at some point:
  51. 51. Installation can be a bit painful since the c client needs to be linked. </li></ul></ul>
  52. 52. Exchanges <ul><li>Fanout Exchange </li><ul><li>No routing keys involved. Any message that is sent to the exchange is sent to all queues bound to that exchange. </li></ul><li>Direct Exchange </li><ul><li>Routing keys involved. A queue binds to the exchange to request messages that match a routing key exactly. </li></ul><li>Topic Exchange </li><ul><li>Routing keys involved. A queue binds to the exchange to request messages that match a routing key pattern. </li></ul></ul>
  53. 53. Enough Text, lets see a picture
  54. 54. Your First Queue <ul><li>An exchange, queue and bindings must be defined first. Publishing can then commence after. </li><ul><li>Create the queue
  55. 55. Create the exchange
  56. 56. Bind to the queue. </li></ul></ul>
  57. 57. Do You Need Persistence? <ul><li>Just about all items have a level of persistence if you would like them to survive on a problem. This can be done by marking the exchanges, queues and messages to be DURABLE.
  58. 58. By default, nothing is made to be durable and will not survive in the event of the message server going down or a reboot or things of that nature. </li></ul>
  59. 59. A simple consumer:
  60. 60. A simple publisher:
  61. 61. Stomp
  62. 62. Overview <ul><li>STOMP = Streaming Text Oriented Messaging Protocol </li><ul><li>Otherwise known as TTMP. </li></ul><li>The protocol itself is extremely simplistic.
  63. 63. Several servers have plug-ins to handle the protocol. </li></ul>
  64. 64. How It Works <ul><li>Connect -> Send -> Disconnect
  65. 65. Connect -> Subscribe -> [Ack] -> Disconnect
  66. 66. When you send in a message, you tell it which queue to go to.
  67. 67. When you subscribe you request a queue. </li></ul>
  68. 68. Stomp Capable Servers <ul><li>ActiveMQ -
  69. 69. StompConnect -
  70. 70. StompServer -
  71. 71. Gozirra -
  72. 72. RabbitMQ - </li></ul>
  73. 73. Stomp PHP Clients <ul><li>PECL: Stomp </li><ul><li>pecl install stomp </li></ul><li>Zend_Queue_Stomp </li></ul>
  74. 74. A simple publisher
  75. 75. A simple consumer
  76. 76. Job Queues
  77. 77. Overview <ul><li>There are job servers available that are more specific to handling jobs.
  78. 78. These are basically message queues with specialized purposes.
  79. 79. Gearman –
  80. 80. Beanstalkd –
  81. 81. Zend Job Queue – Zend Platform & Zend Server </li></ul>
  82. 82. Gearman <ul><li>Gearman supports doing jobs both asynchronously as well as synchronously.
  83. 83. It can optionally have persistence and works on a job / worker model.
  84. 84. You can respond back to the application and check on the status of a job.
  85. 85. PECL has a great PHP extension.
  86. 86. Very cool stuff. </li></ul>
  87. 87. Gearman Client Example
  88. 88. Gearman Worker Example
  89. 89. Beanstalkd <ul><li>Beanstalkd has a single advantage over several other message queues, in the fact that it supports delays.
  90. 90. It has persistence and works on a job / worker model.
  91. 91. User land client available. </li><ul><li>Pheanstalk - </li></ul><li>Works great for scheduling jobs out (aka 4 days later). </li></ul>
  92. 92. Pheanstalk Client Example
  93. 93. Pheanstalk Worker Example
  94. 94. Picking a Message Queue
  95. 95. Various Message Queue Servers
  96. 96. Picking a Message Queue Server <ul><li>Durability (Memory, Disk, DBMS)
  97. 97. Scalability (Horizontal / Vertical)
  98. 98. Synchronous / Asynchronous
  99. 99. Batching Policies
  100. 100. Delivery Policies
  101. 101. Purging Policies
  102. 102. Routing Policies
  103. 103. Security Policies
  104. 104. Standards </li></ul>
  105. 105. How to Use Your Queues
  106. 106. Formatting your Messages <ul><li>Anything that you want. </li><ul><li>JSON
  107. 107. XML
  108. 108. … </li></ul><li>Remember the purpose is to decouple and make messages application agnostic. </li></ul>
  109. 109. Consumers to 1 Task <ul><li>Consumers should be dumb, they should not need to do more than a single task. </li><ul><li>If they do, write another message to a separate queue for a different consumer to utilize it. </li><ul><li>Exception: Legacy Code </li></ul></ul></ul>
  110. 110. A Simple Architecture Queue Receive Producer Message Queue Server Consumer
  111. 111. A Complex Architecture Producer Message Queue Server Consumer Consumer Consumer Consumer Consumer Message Queue Server Message Queue Server Producer Producer
  112. 112. Things to Keep in Mind <ul><li>Queues and Consumers should be monitored </li><ul><li>Supervisord ( can ensure there are always consumers.
  113. 113. Memory Usage of Consumers (in the event of leaks [memory_get_usage], kill and let supervisord re-create it) </li></ul><li>Persistence of Data </li><ul><li>Many have multiple backends or built-in backends but this does slow down messages. </li></ul><li>High Availability </li><ul><li>Not all clients have built in support for multiple servers, you may need to abstract it. </li></ul><li>Many message queues do not handle scheduled processing (aka wait 4 days to process the message). </li></ul>
  114. 114. Use Cases
  115. 115. Email Notifications <ul><li>A user takes an action on a page that needs to notify all subscribing members.
  116. 116. Send a message to the queue regarding the action, then queue emails to send in another queue. </li><ul><li>Why 2 queues </li><ul><li>Allows us to handle getting everyone that needs the notification.
  117. 117. Then allows us to focus on sending email through a single queue with multiple consumers.
  118. 118. “ Write programs to work together” </li></ul></ul></ul>
  119. 119. Auto-Scaling <ul><li>A scheduled task polls the servers looking for potential load issues. If load or web server processes are above a threshold send a message to the queue to have a consumer build out a new instance (assuming virtual machines or the cloud).
  120. 120. The consumer then utilizes a service call or whatever logic it needs to create the new instance.
  121. 121. Once the machine boots you could utilize another queue to send a message that states to put the server in the farm. </li></ul>
  122. 122. Video / Photos <ul><li>Since videos and photos can take up a lot of memory and cpu time when resizing, you may not want to process them all at once.
  123. 123. When a new video or photo comes in, send a message to the queue to do the work. Keep only enough workers / consumers available to not strain your machine.
  124. 124. Remember to monitor how many jobs are around. </li></ul>
  125. 125. Monitoring a Job <ul><li>Some jobs you need to know the status of, or have it notify that it has been completed for user feedback.
  126. 126. In the event that you are not using gearman, you can create another queue to hold status updates. One way of doing this is to record an id number and check for it's status by utilizing another queue or updating a record in a database to poll it.
  127. 127. There are numerous ways to handle this. </li></ul>
  128. 128. Apple APNs <ul><li>Apple's push notification service requires to run a daemon like process. To be connected to their server.
  129. 129. We could send the payload through the message queue to be handled by the worker that is consistently connected to the APNs server. </li></ul>
  130. 130. Mike Willbanks Blog: Twitter : mwillbanks IRC : lubs on freenode Talk: Thank You!