Successfully reported this slideshow.
Massaging the Pony:Message Queues and You<br />Shawn Rider<br />PBS Education<br />DjangoCon 2010<br />http://www.flickr.c...
What is a Message Queue?<br />A system for enabling asynchronous processing of discrete tasks.<br />Super awesome.<br />ht...
What are they good for?<br />Alleviate server issues by offloading processing<br />http://www.flickr.com/photos/islandgyrl...
What are they good for?<br />Make better user experiences<br />http://www.flickr.com/photos/bbaltimore/6779682<br />
What are they good for?<br />Increase the reliability of third party service integrations.<br />
What are they good for?<br />Background media processing<br />http://www.flickr.com/photos/factoryjoe/2882992091<br />
What are they good for?<br /> Repeating Tasks can replace Cron jobs <br /> Keep all your code in your project<br />
Available MQ Solutions<br />There are many solutions out there. To name a few of them:<br />RabbitMQ<br />Amazon Simple Qu...
Message Queue Protocols<br />AMQP: Advanced Message Queuing Protocol<br />JMS: Java Messaging Service<br />STOMP: Streamin...
Criteria for Broker Selection<br />The following criteria were important to us in selecting a solution. You may have some ...
Failsafe MQ Brokers<br />Must be able to survive a server failure.<br />Upon restarting the MQ Broker, processing should r...
Queue Durability<br />The Messages Queue Broker must be able to reconstruct the queues when it is restarted.<br />http://w...
Message Persistence<br />The Messages (tasks to be completed) must persist between shutdowns and restarts of MQ Server<br />
Our Choice<br />Celery (a Django task management app)<br />Carrot (Django middleware)<br />PyAMQP (Python module)<br />Rab...
The Production System<br />Webapp<br />Rabbit<br />MQ<br />Celery<br />Worker<br />Server(s)<br />Database<br />
MQ in Development Environs<br />It is important to be able to develop Message Queue tasks in Django without being concerne...
MQ in Development Environs<br />CARROT_BACKEND = ‘qhettoq.taproot.Database’<br />Carrot can be set up to use different que...
Ease a future MQ implementation<br />Isolate functionality into reusable components.<br />
Recycle Existing Code<br />
Django, HTTP Requests & MQ<br />Most functionalities provided throughDjangoapplications are tightly coupled with HTTP requ...
ThePickleableHTTPRequestObject<br />
UsingPickleableHttpRequest<br />if request.GET.get(‘download’):<br />try:<br />        result = EnrollmentCSVExportTask.de...
Making Message Queue Tasks<br />It is fast and easy to add a decorator to specific functions to create Message Queue tasks...
Explicitly Inherit from Celery’s Task Object<br />
Then What?<br />Some Message Queue tasks can complete with no notification required.<br />For other tasks:<br />Notify by ...
In Review<br />Choose your MQ solution wisely<br />Set up a robust system<br />Leverage your existing code<br />Replace tr...
Message Queues Make Life Better<br />
References and Links<br />Second Life’s Survey of Message Queues:http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_...
Upcoming SlideShare
Loading in …5
×

Massaging the Pony: Message Queues and You

4,050 views

Published on

Slides from my presentation about Message Queues at Djangocon 2010. Covers selecting your solution, design patterns, etc.

Published in: Technology
  • Be the first to comment

Massaging the Pony: Message Queues and You

  1. 1. Massaging the Pony:Message Queues and You<br />Shawn Rider<br />PBS Education<br />DjangoCon 2010<br />http://www.flickr.com/photos/funtik/1175522045<br />
  2. 2. What is a Message Queue?<br />A system for enabling asynchronous processing of discrete tasks.<br />Super awesome.<br />http://www.flickr.com/photos/gadl/89650415<br />
  3. 3. What are they good for?<br />Alleviate server issues by offloading processing<br />http://www.flickr.com/photos/islandgyrl/3197469932<br />
  4. 4. What are they good for?<br />Make better user experiences<br />http://www.flickr.com/photos/bbaltimore/6779682<br />
  5. 5. What are they good for?<br />Increase the reliability of third party service integrations.<br />
  6. 6. What are they good for?<br />Background media processing<br />http://www.flickr.com/photos/factoryjoe/2882992091<br />
  7. 7. What are they good for?<br /> Repeating Tasks can replace Cron jobs <br /> Keep all your code in your project<br />
  8. 8. Available MQ Solutions<br />There are many solutions out there. To name a few of them:<br />RabbitMQ<br />Amazon Simple Queue System<br />Apache MQ<br />Gearman<br />OpenAMQ<br />Peafowl<br />Q4M<br />Starling<br />Sun Java System Message Queue<br />
  9. 9. Message Queue Protocols<br />AMQP: Advanced Message Queuing Protocol<br />JMS: Java Messaging Service<br />STOMP: Streaming Text Oriented Messaging Protocol<br />
  10. 10. Criteria for Broker Selection<br />The following criteria were important to us in selecting a solution. You may have some different needs.<br />Handling exceptions and recovery<br />Low level of required maintenance<br />Ease of deployment<br />Persistent queuing<br />Community support<br />What language is it written in? How compatible is that with our current systems?<br />Cluster support<br />
  11. 11. Failsafe MQ Brokers<br />Must be able to survive a server failure.<br />Upon restarting the MQ Broker, processing should resume from the last un-processed messages in queues<br />
  12. 12. Queue Durability<br />The Messages Queue Broker must be able to reconstruct the queues when it is restarted.<br />http://www.flickr.com/photos/raul/846318014<br />
  13. 13. Message Persistence<br />The Messages (tasks to be completed) must persist between shutdowns and restarts of MQ Server<br />
  14. 14. Our Choice<br />Celery (a Django task management app)<br />Carrot (Django middleware)<br />PyAMQP (Python module)<br />RabbitMQ (MQ Broker software)<br />
  15. 15. The Production System<br />Webapp<br />Rabbit<br />MQ<br />Celery<br />Worker<br />Server(s)<br />Database<br />
  16. 16. MQ in Development Environs<br />It is important to be able to develop Message Queue tasks in Django without being concerned about running the broker process, queue setup, persistence etc.<br />
  17. 17. MQ in Development Environs<br />CARROT_BACKEND = ‘qhettoq.taproot.Database’<br />Carrot can be set up to use different queuing backend for development purposes<br />GhettoQ provides database backend for carrot where a database serves as the queue storage<br />The broker is a simple Django application that monitors the queue in DB<br />Inefficient, therefore only suitable for development environments<br />
  18. 18. Ease a future MQ implementation<br />Isolate functionality into reusable components.<br />
  19. 19. Recycle Existing Code<br />
  20. 20. Django, HTTP Requests & MQ<br />Most functionalities provided throughDjangoapplications are tightly coupled with HTTP requests e.g. form data processing<br />In order to asynchronously execute a task, Celery (any task management platform) must serialize & store parameters<br />HTTP requests are usually large and not easily serializable. e.g. WSGI requests<br />
  21. 21. ThePickleableHTTPRequestObject<br />
  22. 22. UsingPickleableHttpRequest<br />if request.GET.get(‘download’):<br />try:<br /> result = EnrollmentCSVExportTask.delay(<br />PickleableHttpRequest(<br /> request, <br /> attributes=<br /> [‘user’, ‘admin_current_organization’]<br /> )<br /> )<br />
  23. 23. Making Message Queue Tasks<br />It is fast and easy to add a decorator to specific functions to create Message Queue tasks.<br />from celery.decoratorsimport task<br />@task<br />def add(x, y):<br />return x + y<br />
  24. 24. Explicitly Inherit from Celery’s Task Object<br />
  25. 25. Then What?<br />Some Message Queue tasks can complete with no notification required.<br />For other tasks:<br />Notify by email<br />Use an AJAX listener to notify users<br />Some other messaging<br />
  26. 26. In Review<br />Choose your MQ solution wisely<br />Set up a robust system<br />Leverage your existing code<br />Replace trouble spots as they come up<br />Profit!<br />
  27. 27. Message Queues Make Life Better<br />
  28. 28. References and Links<br />Second Life’s Survey of Message Queues:http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes<br />Ask SolemHoel’sGithub Repos: http://github.com/ask<br />Celery Project: http://celeryproject.org/<br />PickleableHttpRequest by NowellStrite (@nowells)<br />http://gist.github.com/571027<br />Special thanks to TarequeHossain (http://codexn.com)<br />

×