The document discusses message queues and their uses. Message queues allow for asynchronous communication between applications and components. They decouple systems, allow for background processing, and improve scalability. Common use cases for message queues include processing email notifications, auto-scaling cloud applications, handling image/video processing, and interacting with services like Apple Push Notifications.

1. Mike Willbanks Sr. Software Engineer at CaringBridge Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://joind.in/1586 The Art of Message Queues TEK-X

2. Who Am I? Sr. Sofware Engineer at CaringBridge

3. PHP monkey

4. MNPHP User Group

5. Interested in Scalability & Performance

6. “ Message queues and mailboxes are software-engineering components used for interprocess communication, or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content.” http://en.wikipedia.org/wiki/Message_queue

7. “ Messaging describes the sending and receiving of data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.” http://www.rabbitmq.com/faq.html#what-is-messaging

8. Messages are Everywhere

9. A Basic Message Queue

10. What are Message Queues A FIFO buffer

11. Asynchronous push / pull

12. An application framework for sending and receiving messages.

13. A way to communicate between applications / systems.

14. A way to decouple components.

15. A way to offload work.

16. Why are Message Queues Useful? Asynchronous Processing

17. Communication between Applications / Systems

18. Image Resizing

19. Video Processing

20. Sending out Emails

21. Auto-Scaling Virtual Instances

22. Log Analysis

23. The list goes on...

24. You Might Use a Message Queue If... The request can be fulfilled in the background.

25. You need to communicate between multiple applications / systems.

26. You have a limited number of processing slots.

27. You need to support legacy applications or use multiple programming languages. Good short presentation on why you might need it: blog.pasker.net/2008/06/16/you-might-need-messaging-if/

28. Going back to Unix “Write programs that work together.”

29. “Do it in the background.”

30. Why does that matter? Web systems need to be geared to run things asynchronously.

31. Distribution may also come into play.

32. Protocols

33. Standardized Protocols AMQP – Advanced Message Queue Protocol

34. STOMP – Streaming Text Oriented Messaging Protocol

35. XMPP - Extensible Messaging and Presence Protocol Related with patches, I will not be going into this. However, XMPP is the next topic in this room.

36. AMQP

37. Overview AMQP Working Group (Community and Vendor)

38. Platform agnostic protocol.

39. Completely open, interoperable and broadly applicable.

40. Many severs available and many client libraries. http://amqp.org For more information on the protocol.

41. How it works. AMQP utilizes exchanges, queues and bindings.

42. An exchange is routers with routing tables.

43. A queue is where the messages wait for a consumer.

44. A binding defines the routing rules.

45. In a nutshell....

46. Some of the Various Servers Apache Qpid - qpid.apache.org

47. OpenAMQ - openamq.org

48. RabbitMQ - rabbitmq.com

49. ZeroMQ - zeromq.org

50. A PHP Client php-rabbit – code.google.com/p/php-rabbit Looks like it will become a PECL package at some point: pecl.php.net/package/amqp.

51. Installation can be a bit painful since the c client needs to be linked.

52. Exchanges Fanout Exchange No routing keys involved. Any message that is sent to the exchange is sent to all queues bound to that exchange. Direct Exchange Routing keys involved. A queue binds to the exchange to request messages that match a routing key exactly. Topic Exchange Routing keys involved. A queue binds to the exchange to request messages that match a routing key pattern.

53. Enough Text, lets see a picture

54. Your First Queue An exchange, queue and bindings must be defined first. Publishing can then commence after. Create the queue

55. Create the exchange

56. Bind to the queue.

57. Do You Need Persistence? Just about all items have a level of persistence if you would like them to survive on a problem. This can be done by marking the exchanges, queues and messages to be DURABLE.

58. By default, nothing is made to be durable and will not survive in the event of the message server going down or a reboot or things of that nature.

59. A simple consumer:

60. A simple publisher:

61. Stomp

62. Overview STOMP = Streaming Text Oriented Messaging Protocol Otherwise known as TTMP. The protocol itself is extremely simplistic.

63. Several servers have plug-ins to handle the protocol.

64. How It Works Connect -> Send -> Disconnect

65. Connect -> Subscribe -> [Ack] -> Disconnect

66. When you send in a message, you tell it which queue to go to.

67. When you subscribe you request a queue.

68. Stomp Capable Servers ActiveMQ - activemq.apache.org

69. StompConnect - stomp.codehaus.org/StompConnect

70. StompServer - stompserver.rubyforge.org

71. Gozirra - germane-software.com/software/Java/Gozirra

72. RabbitMQ - rabbitmq.com

73. Stomp PHP Clients PECL: Stomp pecl install stomp Zend_Queue_Stomp

74. A simple publisher

75. A simple consumer

76. Job Queues

77. Overview There are job servers available that are more specific to handling jobs.

78. These are basically message queues with specialized purposes.

79. Gearman – gearman.org

80. Beanstalkd – kr.github.com/beanstalkd

81. Zend Job Queue – Zend Platform & Zend Server

82. Gearman Gearman supports doing jobs both asynchronously as well as synchronously.

83. It can optionally have persistence and works on a job / worker model.

84. You can respond back to the application and check on the status of a job.

85. PECL has a great PHP extension.

86. Very cool stuff.

87. Gearman Client Example

88. Gearman Worker Example

89. Beanstalkd Beanstalkd has a single advantage over several other message queues, in the fact that it supports delays.

90. It has persistence and works on a job / worker model.

91. User land client available. Pheanstalk - github.com/pda/pheanstalk Works great for scheduling jobs out (aka 4 days later).

92. Pheanstalk Client Example

93. Pheanstalk Worker Example

94. Picking a Message Queue

95. Various Message Queue Servers

96. Picking a Message Queue Server Durability (Memory, Disk, DBMS)

97. Scalability (Horizontal / Vertical)

98. Synchronous / Asynchronous

99. Batching Policies

100. Delivery Policies

101. Purging Policies

102. Routing Policies

103. Security Policies

104. Standards http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

105. How to Use Your Queues

106. Formatting your Messages Anything that you want. JSON

107. XML

108. … Remember the purpose is to decouple and make messages application agnostic.

109. Consumers to 1 Task Consumers should be dumb, they should not need to do more than a single task. If they do, write another message to a separate queue for a different consumer to utilize it. Exception: Legacy Code

110. A Simple Architecture Queue Receive Producer Message Queue Server Consumer

111. A Complex Architecture Producer Message Queue Server Consumer Consumer Consumer Consumer Consumer Message Queue Server Message Queue Server Producer Producer

112. Things to Keep in Mind Queues and Consumers should be monitored Supervisord (http://supervisord.org/) can ensure there are always consumers.

113. Memory Usage of Consumers (in the event of leaks [memory_get_usage], kill and let supervisord re-create it) Persistence of Data Many have multiple backends or built-in backends but this does slow down messages. High Availability Not all clients have built in support for multiple servers, you may need to abstract it. Many message queues do not handle scheduled processing (aka wait 4 days to process the message).

114. Use Cases

115. Email Notifications A user takes an action on a page that needs to notify all subscribing members.

116. Send a message to the queue regarding the action, then queue emails to send in another queue. Why 2 queues Allows us to handle getting everyone that needs the notification.

117. Then allows us to focus on sending email through a single queue with multiple consumers.

118. “ Write programs to work together”

119. Auto-Scaling A scheduled task polls the servers looking for potential load issues. If load or web server processes are above a threshold send a message to the queue to have a consumer build out a new instance (assuming virtual machines or the cloud).

120. The consumer then utilizes a service call or whatever logic it needs to create the new instance.

121. Once the machine boots you could utilize another queue to send a message that states to put the server in the farm.

122. Video / Photos Since videos and photos can take up a lot of memory and cpu time when resizing, you may not want to process them all at once.

123. When a new video or photo comes in, send a message to the queue to do the work. Keep only enough workers / consumers available to not strain your machine.

124. Remember to monitor how many jobs are around.

125. Monitoring a Job Some jobs you need to know the status of, or have it notify that it has been completed for user feedback.

126. In the event that you are not using gearman, you can create another queue to hold status updates. One way of doing this is to record an id number and check for it's status by utilizing another queue or updating a record in a database to poll it.

127. There are numerous ways to handle this.

128. Apple APNs Apple's push notification service requires to run a daemon like process. To be connected to their server.

129. We could send the payload through the message queue to be handled by the worker that is consistently connected to the APNs server.

130. Mike Willbanks Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://joind.in/1586 Thank You!