The Art of Message Queues

Mike Willbanks Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://wiki.minnestar.org/MessageQueues The Art of Message Queues MinneBar 2009

What are Message Queues
The sending and/or receiving of data [messages] between systems.
An exchange of information between systems, programs and/or components.
The decoupling of application components.
A FIFO buffer
What is your definition?

Why are Message Queues Useful?
Buzzwords
Durability
Reliability
Scalability
Offline Processing
Communication between Systems
Asynchronous Processing
Scheduling Operations
There are many more uses...

Usefulness By Example

Social Network Notification Emails
Synopsis: When a user adds a photo an email is sent to each of their friends. Presently, this happens when the user adds the photo and complaints are taking place that it takes too long to add a photo or the system times out. Solution: When a photo is added add a message to the queue a worker retrieves the message to process sending out emails to the friends of the user.

Video Processing
Synopsis: When a user uploads a video, this video needs to be converted and compressed by ffmpeg before adding it to the users area. Solution: Once the video has been uploaded a message is sent to the queue. A worker picks up the queue item when there is processing available, processes the video and marks it as active.

Asynchronous Processing
Synopsis: Your E-commerce system needs to do several things: verify credit card details, hold an item, submit shipment details, etc. Your boss wants to do these operations at the same time. Solution: Send a message to the queue for each item do your additional processing and process the results as each item comes in.

Grid Computing
Synopsis: You have a lot of hardware that during down periods you want to save on data center costs or many virtual machines. These are all running today and you want to add / remove them as load increases. Solution: Implement a scheduled task to poll the servers load, after a certain threshold is reached add a message to the queue. The worker could be created to add a new [virtual] machine to the stack.

Message Queue Servers / Services

Some of the Players
Amazon SQS
Active MQ
Beanstalkd
Dropr
Gearman
JMS
MemcacheQ
Minimalist Queue Services
RabbitMQ
Spread

Message Formatting
Anything you want it to be!
Serialized Input
XML
JSON
…
Although... for the sanity of others don't make a new language in the message :)

A Simple Architecture Queue Receive Client Message Queue Server Worker

A Complex Architecture Client Message Queue Server Worker Worker Worker Worker Worker Message Queue Server Message Queue Server Client Client

Mike Willbanks Blog: http://blog.digitalstruct.com Twitter : mwillbanks IRC : lubs on freenode Talk: http://wiki.minnestar.org/MessageQueues Questions?

http://blog.digitalstruct.com	585
http://www.planet-php.net	150
http://www.slideshare.net	23
http://planet-php.org	16
http://www.planet-php.org	11
http://static.slideshare.net	7
http://www.politicalneedle.com	5
http://translate.googleusercontent.com	2
http://planet-php.net	1
http://xss.yandex.net	1
http://quiteso.nr7.net	1
http://phpeye.com	1
http://www.mefeedia.com	1

The Art of Message Queues

by Mike Willbanks on Nov 23, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

13 Embeds 804

Statistics

The Art of Message Queues — Presentation Transcript