The document explains the concept of message queues as systems for asynchronous processing that offload tasks from servers, improve user experiences, and enhance the reliability of integrations. It discusses various solutions and protocols for implementing message queues, criteria for broker selection, and uses Celery with RabbitMQ for task management in Django applications. The document emphasizes the importance of developing message queue tasks in a controlled environment while maintaining robust functionality.

1. Massaging the Pony:Message Queues and YouShawn RiderPBS EducationDjangoCon 2010http://www.flickr.com/photos/funtik/1175522045

2. What is a Message Queue?A system for enabling asynchronous processing of discrete tasks.Super awesome.http://www.flickr.com/photos/gadl/89650415

3. What are they good for?Alleviate server issues by offloading processinghttp://www.flickr.com/photos/islandgyrl/3197469932

4. What are they good for?Make better user experienceshttp://www.flickr.com/photos/bbaltimore/6779682

5. What are they good for?Increase the reliability of third party service integrations.

6. What are they good for?Background media processinghttp://www.flickr.com/photos/factoryjoe/2882992091

7. What are they good for? Repeating Tasks can replace Cron jobs Keep all your code in your project

8. Available MQ SolutionsThere are many solutions out there. To name a few of them:RabbitMQAmazon Simple Queue SystemApache MQGearmanOpenAMQPeafowlQ4MStarlingSun Java System Message Queue

9. Message Queue ProtocolsAMQP: Advanced Message Queuing ProtocolJMS: Java Messaging ServiceSTOMP: Streaming Text Oriented Messaging Protocol

10. Criteria for Broker SelectionThe following criteria were important to us in selecting a solution. You may have some different needs.Handling exceptions and recoveryLow level of required maintenanceEase of deploymentPersistent queuingCommunity supportWhat language is it written in? How compatible is that with our current systems?Cluster support

11. Failsafe MQ BrokersMust be able to survive a server failure.Upon restarting the MQ Broker, processing should resume from the last un-processed messages in queues

12. Queue DurabilityThe Messages Queue Broker must be able to reconstruct the queues when it is restarted.http://www.flickr.com/photos/raul/846318014

13. Message PersistenceThe Messages (tasks to be completed) must persist between shutdowns and restarts of MQ Server

14. Our ChoiceCelery (a Django task management app)Carrot (Django middleware)PyAMQP (Python module)RabbitMQ (MQ Broker software)

15. The Production SystemWebappRabbitMQCeleryWorkerServer(s)Database

16. MQ in Development EnvironsIt is important to be able to develop Message Queue tasks in Django without being concerned about running the broker process, queue setup, persistence etc.

17. MQ in Development EnvironsCARROT_BACKEND = ‘qhettoq.taproot.Database’Carrot can be set up to use different queuing backend for development purposesGhettoQ provides database backend for carrot where a database serves as the queue storageThe broker is a simple Django application that monitors the queue in DBInefficient, therefore only suitable for development environments

18. Ease a future MQ implementationIsolate functionality into reusable components.

19. Recycle Existing Code

20. Django, HTTP Requests & MQMost functionalities provided throughDjangoapplications are tightly coupled with HTTP requests e.g. form data processingIn order to asynchronously execute a task, Celery (any task management platform) must serialize & store parametersHTTP requests are usually large and not easily serializable. e.g. WSGI requests

21. ThePickleableHTTPRequestObject

22. UsingPickleableHttpRequestif request.GET.get(‘download’):try: result = EnrollmentCSVExportTask.delay(PickleableHttpRequest( request, attributes=\ [‘user’, ‘admin_current_organization’] ) )

23. Making Message Queue TasksIt is fast and easy to add a decorator to specific functions to create Message Queue tasks.from celery.decoratorsimport task@taskdef add(x, y):return x + y

24. Explicitly Inherit from Celery’s Task Object

25. Then What?Some Message Queue tasks can complete with no notification required.For other tasks:Notify by emailUse an AJAX listener to notify usersSome other messaging

26. In ReviewChoose your MQ solution wiselySet up a robust systemLeverage your existing codeReplace trouble spots as they come upProfit!

27. Message Queues Make Life Better

28. References and LinksSecond Life’s Survey of Message Queues:http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_NotesAsk SolemHoel’sGithub Repos: http://github.com/askCelery Project: http://celeryproject.org/PickleableHttpRequest by NowellStrite (@nowells)http://gist.github.com/571027Special thanks to TarequeHossain (http://codexn.com)