Slides from my presentation about Message Queues at Djangocon 2010. Covers selecting your solution, design patterns, etc.

1. Massaging the Pony:Message Queues and You Shawn Rider PBS Education DjangoCon 2010 http://www.flickr.com/photos/funtik/1175522045

2. What is a Message Queue? A system for enabling asynchronous processing of discrete tasks. Super awesome. http://www.flickr.com/photos/gadl/89650415

3. What are they good for? Alleviate server issues by offloading processing http://www.flickr.com/photos/islandgyrl/3197469932

4. What are they good for? Make better user experiences http://www.flickr.com/photos/bbaltimore/6779682

5. What are they good for? Increase the reliability of third party service integrations.

6. What are they good for? Background media processing http://www.flickr.com/photos/factoryjoe/2882992091

7. What are they good for? Repeating Tasks can replace Cron jobs Keep all your code in your project

8. Available MQ Solutions There are many solutions out there. To name a few of them: RabbitMQ Amazon Simple Queue System Apache MQ Gearman OpenAMQ Peafowl Q4M Starling Sun Java System Message Queue

9. Message Queue Protocols AMQP: Advanced Message Queuing Protocol JMS: Java Messaging Service STOMP: Streaming Text Oriented Messaging Protocol

10. Criteria for Broker Selection The following criteria were important to us in selecting a solution. You may have some different needs. Handling exceptions and recovery Low level of required maintenance Ease of deployment Persistent queuing Community support What language is it written in? How compatible is that with our current systems? Cluster support

11. Failsafe MQ Brokers Must be able to survive a server failure. Upon restarting the MQ Broker, processing should resume from the last un-processed messages in queues

12. Queue Durability The Messages Queue Broker must be able to reconstruct the queues when it is restarted. http://www.flickr.com/photos/raul/846318014

13. Message Persistence The Messages (tasks to be completed) must persist between shutdowns and restarts of MQ Server

14. Our Choice Celery (a Django task management app) Carrot (Django middleware) PyAMQP (Python module) RabbitMQ (MQ Broker software)

15. The Production System Webapp Rabbit MQ Celery Worker Server(s) Database

16. MQ in Development Environs It is important to be able to develop Message Queue tasks in Django without being concerned about running the broker process, queue setup, persistence etc.

17. MQ in Development Environs CARROT_BACKEND = ‘qhettoq.taproot.Database’ Carrot can be set up to use different queuing backend for development purposes GhettoQ provides database backend for carrot where a database serves as the queue storage The broker is a simple Django application that monitors the queue in DB Inefficient, therefore only suitable for development environments

18. Ease a future MQ implementation Isolate functionality into reusable components.

19. Recycle Existing Code

20. Django, HTTP Requests & MQ Most functionalities provided throughDjangoapplications are tightly coupled with HTTP requests e.g. form data processing In order to asynchronously execute a task, Celery (any task management platform) must serialize & store parameters HTTP requests are usually large and not easily serializable. e.g. WSGI requests

21. ThePickleableHTTPRequestObject

22. UsingPickleableHttpRequest if request.GET.get(‘download’): try: result = EnrollmentCSVExportTask.delay( PickleableHttpRequest( request, attributes=br /> [‘user’, ‘admin_current_organization’] ) )

23. Making Message Queue Tasks It is fast and easy to add a decorator to specific functions to create Message Queue tasks. from celery.decoratorsimport task @task def add(x, y): return x + y

24. Explicitly Inherit from Celery’s Task Object

25. Then What? Some Message Queue tasks can complete with no notification required. For other tasks: Notify by email Use an AJAX listener to notify users Some other messaging

26. In Review Choose your MQ solution wisely Set up a robust system Leverage your existing code Replace trouble spots as they come up Profit!

27. Message Queues Make Life Better

28. References and Links Second Life’s Survey of Message Queues:http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes Ask SolemHoel’sGithub Repos: http://github.com/ask Celery Project: http://celeryproject.org/ PickleableHttpRequest by NowellStrite (@nowells) http://gist.github.com/571027 Special thanks to TarequeHossain (http://codexn.com)