Interested in Scalability & Performance </li></ul>
“ Message queues and mailboxes are software-engineering components used for interprocess communication, or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content.” http://en.wikipedia.org/wiki/Message_queue
“ Messaging describes the sending and receiving of data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.” http://www.rabbitmq.com/faq.html#what-is-messaging
A PHP Client <ul><li>php-rabbit – code.google.com/p/php-rabbit </li><ul><li>Looks like it will become a PECL package at some point: pecl.php.net/package/amqp.
Installation can be a bit painful since the c client needs to be linked. </li></ul></ul>
Exchanges <ul><li>Fanout Exchange </li><ul><li>No routing keys involved. Any message that is sent to the exchange is sent to all queues bound to that exchange. </li></ul><li>Direct Exchange </li><ul><li>Routing keys involved. A queue binds to the exchange to request messages that match a routing key exactly. </li></ul><li>Topic Exchange </li><ul><li>Routing keys involved. A queue binds to the exchange to request messages that match a routing key pattern. </li></ul></ul>
Do You Need Persistence? <ul><li>Just about all items have a level of persistence if you would like them to survive on a problem. This can be done by marking the exchanges, queues and messages to be DURABLE.
By default, nothing is made to be durable and will not survive in the event of the message server going down or a reboot or things of that nature. </li></ul>
… </li></ul><li>Remember the purpose is to decouple and make messages application agnostic. </li></ul>
Consumers to 1 Task <ul><li>Consumers should be dumb, they should not need to do more than a single task. </li><ul><li>If they do, write another message to a separate queue for a different consumer to utilize it. </li><ul><li>Exception: Legacy Code </li></ul></ul></ul>
A Simple Architecture Queue Receive Producer Message Queue Server Consumer
A Complex Architecture Producer Message Queue Server Consumer Consumer Consumer Consumer Consumer Message Queue Server Message Queue Server Producer Producer
Things to Keep in Mind <ul><li>Queues and Consumers should be monitored </li><ul><li>Supervisord (http://supervisord.org/) can ensure there are always consumers.
Memory Usage of Consumers (in the event of leaks [memory_get_usage], kill and let supervisord re-create it) </li></ul><li>Persistence of Data </li><ul><li>Many have multiple backends or built-in backends but this does slow down messages. </li></ul><li>High Availability </li><ul><li>Not all clients have built in support for multiple servers, you may need to abstract it. </li></ul><li>Many message queues do not handle scheduled processing (aka wait 4 days to process the message). </li></ul>
Email Notifications <ul><li>A user takes an action on a page that needs to notify all subscribing members.
Send a message to the queue regarding the action, then queue emails to send in another queue. </li><ul><li>Why 2 queues </li><ul><li>Allows us to handle getting everyone that needs the notification.
Then allows us to focus on sending email through a single queue with multiple consumers.
“ Write programs to work together” </li></ul></ul></ul>
Auto-Scaling <ul><li>A scheduled task polls the servers looking for potential load issues. If load or web server processes are above a threshold send a message to the queue to have a consumer build out a new instance (assuming virtual machines or the cloud).
The consumer then utilizes a service call or whatever logic it needs to create the new instance.
Once the machine boots you could utilize another queue to send a message that states to put the server in the farm. </li></ul>
Video / Photos <ul><li>Since videos and photos can take up a lot of memory and cpu time when resizing, you may not want to process them all at once.
When a new video or photo comes in, send a message to the queue to do the work. Keep only enough workers / consumers available to not strain your machine.
Remember to monitor how many jobs are around. </li></ul>
Monitoring a Job <ul><li>Some jobs you need to know the status of, or have it notify that it has been completed for user feedback.
In the event that you are not using gearman, you can create another queue to hold status updates. One way of doing this is to record an id number and check for it's status by utilizing another queue or updating a record in a database to poll it.
There are numerous ways to handle this. </li></ul>
Apple APNs <ul><li>Apple's push notification service requires to run a daemon like process. To be connected to their server.
We could send the payload through the message queue to be handled by the worker that is consistently connected to the APNs server. </li></ul>
Mike Willbanks Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://joind.in/1586 Thank You!