Massaging the Pony: Message Queues and You

Massaging the Pony:Message Queues and You
Shawn Rider
PBS Education
DjangoCon 2010
http://www.flickr.com/photos/funtik/1175522045

What is a Message Queue?
A system for enabling asynchronous processing of discrete tasks.
Super awesome.
http://www.flickr.com/photos/gadl/89650415

What are they good for?
Alleviate server issues by offloading processing
http://www.flickr.com/photos/islandgyrl/3197469932

Make better user experiences
http://www.flickr.com/photos/bbaltimore/6779682

Increase the reliability of third party service integrations.

Background media processing
http://www.flickr.com/photos/factoryjoe/2882992091

Repeating Tasks can replace Cron jobs
Keep all your code in your project

Available MQ Solutions
There are many solutions out there. To name a few of them:
RabbitMQ
Amazon Simple Queue System
Apache MQ
Gearman
OpenAMQ
Peafowl
Q4M
Starling
Sun Java System Message Queue

Message Queue Protocols
AMQP: Advanced Message Queuing Protocol
JMS: Java Messaging Service
STOMP: Streaming Text Oriented Messaging Protocol

Criteria for Broker Selection
The following criteria were important to us in selecting a solution. You may have some different needs.
Handling exceptions and recovery
Low level of required maintenance
Ease of deployment
Persistent queuing
Community support
What language is it written in? How compatible is that with our current systems?
Cluster support

Failsafe MQ Brokers
Must be able to survive a server failure.
Upon restarting the MQ Broker, processing should resume from the last un-processed messages in queues

Queue Durability
The Messages Queue Broker must be able to reconstruct the queues when it is restarted.
http://www.flickr.com/photos/raul/846318014

Message Persistence
The Messages (tasks to be completed) must persist between shutdowns and restarts of MQ Server

Our Choice
Celery (a Django task management app)
Carrot (Django middleware)
PyAMQP (Python module)
RabbitMQ (MQ Broker software)

The Production System
Webapp
Rabbit
MQ
Celery
Worker
Server(s)
Database

MQ in Development Environs
It is important to be able to develop Message Queue tasks in Django without being concerned about running the broker process, queue setup, persistence etc.

MQ in Development Environs
CARROT_BACKEND = ‘qhettoq.taproot.Database’
Carrot can be set up to use different queuing backend for development purposes
GhettoQ provides database backend for carrot where a database serves as the queue storage
The broker is a simple Django application that monitors the queue in DB
Inefficient, therefore only suitable for development environments

Ease a future MQ implementation
Isolate functionality into reusable components.

Recycle Existing Code

Django, HTTP Requests & MQ
Most functionalities provided throughDjangoapplications are tightly coupled with HTTP requests e.g. form data processing
In order to asynchronously execute a task, Celery (any task management platform) must serialize & store parameters
HTTP requests are usually large and not easily serializable. e.g. WSGI requests

ThePickleableHTTPRequestObject

UsingPickleableHttpRequest
if request.GET.get(‘download’):
try:
result = EnrollmentCSVExportTask.delay(
PickleableHttpRequest(
request,
attributes=
[‘user’, ‘admin_current_organization’]
)
)

Making Message Queue Tasks
It is fast and easy to add a decorator to specific functions to create Message Queue tasks.
from celery.decoratorsimport task
@task
def add(x, y):
return x + y

Explicitly Inherit from Celery’s Task Object

Then What?
Some Message Queue tasks can complete with no notification required.
For other tasks:
Notify by email
Use an AJAX listener to notify users
Some other messaging

In Review
Choose your MQ solution wisely
Set up a robust system
Leverage your existing code
Replace trouble spots as they come up
Profit!

Message Queues Make Life Better

References and Links
Second Life’s Survey of Message Queues:http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
Ask SolemHoel’sGithub Repos: http://github.com/ask
Celery Project: http://celeryproject.org/
PickleableHttpRequest by NowellStrite (@nowells)
http://gist.github.com/571027
Special thanks to TarequeHossain (http://codexn.com)

Massaging the Pony: Message Queues and You

by Shawn Rider

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 23

Statistics