The Art of Message Queues - TEKX

Mike Willbanks Sr. Software Engineer at CaringBridge Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://joind.in/1586 The Art of Message Queues TEK-X

Who Am I?
Sr. Sofware Engineer at CaringBridge
PHP monkey
MNPHP User Group
Interested in Scalability & Performance

“ Message queues and mailboxes are software-engineering components used for interprocess communication, or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content.” http://en.wikipedia.org/wiki/Message_queue

“ Messaging describes the sending and receiving of data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.” http://www.rabbitmq.com/faq.html#what-is-messaging

Messages are Everywhere

A Basic Message Queue

What are Message Queues
A FIFO buffer
Asynchronous push / pull
An application framework for sending and receiving messages.
A way to communicate between applications / systems.
A way to decouple components.
A way to offload work.

Why are Message Queues Useful?
Asynchronous Processing
Communication between Applications / Systems
Image Resizing
Video Processing
Sending out Emails
Auto-Scaling Virtual Instances
Log Analysis
The list goes on...

You Might Use a Message Queue If...
The request can be fulfilled in the background.
You need to communicate between multiple applications / systems.
You have a limited number of processing slots.
You need to support legacy applications or use multiple programming languages.
Good short presentation on why you might need it: blog.pasker.net/2008/06/16/you-might-need-messaging-if/

Going back to Unix
“Write programs that work together.”
“Do it in the background.”

Why does that matter?
Web systems need to be geared to run things asynchronously.
Distribution may also come into play.

Protocols

Standardized Protocols
AMQP – Advanced Message Queue Protocol
STOMP – Streaming Text Oriented Messaging Protocol
XMPP - Extensible Messaging and Presence Protocol
Related with patches, I will not be going into this. However, XMPP is the next topic in this room.

Overview
AMQP Working Group (Community and Vendor)
Platform agnostic protocol.
Completely open, interoperable and broadly applicable.
Many severs available and many client libraries.
http://amqp.org For more information on the protocol.

How it works.
AMQP utilizes exchanges, queues and bindings.
An exchange is routers with routing tables.
A queue is where the messages wait for a consumer.
A binding defines the routing rules.

In a nutshell....

Some of the Various Servers
Apache Qpid - qpid.apache.org
OpenAMQ - openamq.org
RabbitMQ - rabbitmq.com
ZeroMQ - zeromq.org

A PHP Client
php-rabbit – code.google.com/p/php-rabbit
Looks like it will become a PECL package at some point: pecl.php.net/package/amqp.
Installation can be a bit painful since the c client needs to be linked.

Exchanges
Fanout Exchange
No routing keys involved. Any message that is sent to the exchange is sent to all queues bound to that exchange.
Direct Exchange
Routing keys involved. A queue binds to the exchange to request messages that match a routing key exactly.
Topic Exchange
Routing keys involved. A queue binds to the exchange to request messages that match a routing key pattern.

Enough Text, lets see a picture

Your First Queue
An exchange, queue and bindings must be defined first. Publishing can then commence after.
Create the queue
Create the exchange
Bind to the queue.

Do You Need Persistence?
Just about all items have a level of persistence if you would like them to survive on a problem. This can be done by marking the exchanges, queues and messages to be DURABLE.
By default, nothing is made to be durable and will not survive in the event of the message server going down or a reboot or things of that nature.

A simple consumer:

A simple publisher:

Overview
STOMP = Streaming Text Oriented Messaging Protocol
Otherwise known as TTMP.
The protocol itself is extremely simplistic.
Several servers have plug-ins to handle the protocol.

How It Works
Connect -> Send -> Disconnect
Connect -> Subscribe -> [Ack] -> Disconnect
When you send in a message, you tell it which queue to go to.
When you subscribe you request a queue.

Stomp Capable Servers
ActiveMQ - activemq.apache.org
StompConnect - stomp.codehaus.org/StompConnect
StompServer - stompserver.rubyforge.org
Gozirra - germane-software.com/software/Java/Gozirra
RabbitMQ - rabbitmq.com

Stomp PHP Clients
PECL: Stomp
pecl install stomp
Zend_Queue_Stomp

A simple publisher

A simple consumer

Job Queues

Overview
There are job servers available that are more specific to handling jobs.
These are basically message queues with specialized purposes.
Gearman – gearman.org
Beanstalkd – kr.github.com/beanstalkd
Zend Job Queue – Zend Platform & Zend Server

Gearman
Gearman supports doing jobs both asynchronously as well as synchronously.
It can optionally have persistence and works on a job / worker model.
You can respond back to the application and check on the status of a job.
PECL has a great PHP extension.
Very cool stuff.

Gearman Client Example

Gearman Worker Example

Beanstalkd
Beanstalkd has a single advantage over several other message queues, in the fact that it supports delays.
It has persistence and works on a job / worker model.
User land client available.
Pheanstalk - github.com/pda/pheanstalk
Works great for scheduling jobs out (aka 4 days later).

Pheanstalk Client Example

Pheanstalk Worker Example

Picking a Message Queue

Various Message Queue Servers

Picking a Message Queue Server
Durability (Memory, Disk, DBMS)
Scalability (Horizontal / Vertical)
Synchronous / Asynchronous
Batching Policies
Delivery Policies
Purging Policies
Routing Policies
Security Policies
Standards
http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

How to Use Your Queues

Formatting your Messages
Anything that you want.
JSON
XML
…
Remember the purpose is to decouple and make messages application agnostic.

Consumers to 1 Task
Consumers should be dumb, they should not need to do more than a single task.
If they do, write another message to a separate queue for a different consumer to utilize it.
Exception: Legacy Code

A Simple Architecture Queue Receive Producer Message Queue Server Consumer

A Complex Architecture Producer Message Queue Server Consumer Consumer Consumer Consumer Consumer Message Queue Server Message Queue Server Producer Producer

Things to Keep in Mind
Queues and Consumers should be monitored
Supervisord (http://supervisord.org/) can ensure there are always consumers.
Memory Usage of Consumers (in the event of leaks [memory_get_usage], kill and let supervisord re-create it)
Persistence of Data
Many have multiple backends or built-in backends but this does slow down messages.
High Availability
Not all clients have built in support for multiple servers, you may need to abstract it.
Many message queues do not handle scheduled processing (aka wait 4 days to process the message).

Use Cases

Email Notifications
A user takes an action on a page that needs to notify all subscribing members.
Send a message to the queue regarding the action, then queue emails to send in another queue.
Why 2 queues
Allows us to handle getting everyone that needs the notification.
Then allows us to focus on sending email through a single queue with multiple consumers.
“ Write programs to work together”

Auto-Scaling
A scheduled task polls the servers looking for potential load issues. If load or web server processes are above a threshold send a message to the queue to have a consumer build out a new instance (assuming virtual machines or the cloud).
The consumer then utilizes a service call or whatever logic it needs to create the new instance.
Once the machine boots you could utilize another queue to send a message that states to put the server in the farm.

Video / Photos
Since videos and photos can take up a lot of memory and cpu time when resizing, you may not want to process them all at once.
When a new video or photo comes in, send a message to the queue to do the work. Keep only enough workers / consumers available to not strain your machine.
Remember to monitor how many jobs are around.

Monitoring a Job
Some jobs you need to know the status of, or have it notify that it has been completed for user feedback.
In the event that you are not using gearman, you can create another queue to hold status updates. One way of doing this is to record an id number and check for it's status by utilizing another queue or updating a record in a database to poll it.
There are numerous ways to handle this.

Apple APNs
Apple's push notification service requires to run a daemon like process. To be connected to their server.
We could send the payload through the message queue to be handled by the worker that is consistently connected to the APNs server.

Mike Willbanks Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://joind.in/1586 Thank You!

http://abrdev.com	4970
http://www.slideshare.net	32
url_unknown	24
http://translate.googleusercontent.com	3
http://webmail.k-p.com.ua	3
http://webcache.googleusercontent.com	2
http://127.0.0.1	2
http://webmail.blumenbecker.kiev.ua	2
http://207.46.192.232	1
http://abrdev.com HTTP	1

The Art of Message Queues - TEKX

by Mike Willbanks on May 20, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

10 Embeds 5,040

Statistics

The Art of Message Queues - TEKX — Presentation Transcript