Your SlideShare is downloading. ×
0
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Massaging the Pony: Message Queues and You
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Massaging the Pony: Message Queues and You

3,523

Published on

Slides from my presentation about Message Queues at Djangocon 2010. Covers selecting your solution, design patterns, etc.

Slides from my presentation about Message Queues at Djangocon 2010. Covers selecting your solution, design patterns, etc.

Published in: Technology
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,523
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
101
Comments
0
Likes
7
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Example from TL about bulk refunds and paypal.
  • Cron jobs essentially pass control of your code to the operating system. Although the Cron job is a valuable tool, it’s not the only option.
  • Because a Message Queue Broker is a separate system that your project interfaces with, you could feasibly use many different apps created in many different languages.
  • Cluster support will be a priority for a lot of people, but it was not a priority for us. (Cluster support is the ability to have a cluster of MQ Brokers.)
  • To achieve this, we consider two concepts (on next slides).
  • This is what that looks like in production. Many of these components exist on dedicated servers and/or server clusters.
  • Some people who are more familiar with Carrot and Celery might wonder why not use the “always eager” setting, which processes MQ tasks in real time. However, in our experience that setting does not provide the effect we are after, and it is frustrating to get timeouts on long-running processes in development.
  • You want to replace trouble spots in your project as they come up. No need to rewrite your entire project right away.
  • The MockHTTP object always serializes GET and POST data. It then saves any additional HTTPRequest attributes specified when it is called.
  • Transcript

    • 1. Massaging the Pony:Message Queues and You
      Shawn Rider
      PBS Education
      DjangoCon 2010
      http://www.flickr.com/photos/funtik/1175522045
    • 2. What is a Message Queue?
      A system for enabling asynchronous processing of discrete tasks.
      Super awesome.
      http://www.flickr.com/photos/gadl/89650415
    • 3. What are they good for?
      Alleviate server issues by offloading processing
      http://www.flickr.com/photos/islandgyrl/3197469932
    • 4. What are they good for?
      Make better user experiences
      http://www.flickr.com/photos/bbaltimore/6779682
    • 5. What are they good for?
      Increase the reliability of third party service integrations.
    • 6. What are they good for?
      Background media processing
      http://www.flickr.com/photos/factoryjoe/2882992091
    • 7. What are they good for?
      Repeating Tasks can replace Cron jobs
      Keep all your code in your project
    • 8. Available MQ Solutions
      There are many solutions out there. To name a few of them:
      RabbitMQ
      Amazon Simple Queue System
      Apache MQ
      Gearman
      OpenAMQ
      Peafowl
      Q4M
      Starling
      Sun Java System Message Queue
    • 9. Message Queue Protocols
      AMQP: Advanced Message Queuing Protocol
      JMS: Java Messaging Service
      STOMP: Streaming Text Oriented Messaging Protocol
    • 10. Criteria for Broker Selection
      The following criteria were important to us in selecting a solution. You may have some different needs.
      Handling exceptions and recovery
      Low level of required maintenance
      Ease of deployment
      Persistent queuing
      Community support
      What language is it written in? How compatible is that with our current systems?
      Cluster support
    • 11. Failsafe MQ Brokers
      Must be able to survive a server failure.
      Upon restarting the MQ Broker, processing should resume from the last un-processed messages in queues
    • 12. Queue Durability
      The Messages Queue Broker must be able to reconstruct the queues when it is restarted.
      http://www.flickr.com/photos/raul/846318014
    • 13. Message Persistence
      The Messages (tasks to be completed) must persist between shutdowns and restarts of MQ Server
    • 14. Our Choice
      Celery (a Django task management app)
      Carrot (Django middleware)
      PyAMQP (Python module)
      RabbitMQ (MQ Broker software)
    • 15. The Production System
      Webapp
      Rabbit
      MQ
      Celery
      Worker
      Server(s)
      Database
    • 16. MQ in Development Environs
      It is important to be able to develop Message Queue tasks in Django without being concerned about running the broker process, queue setup, persistence etc.
    • 17. MQ in Development Environs
      CARROT_BACKEND = ‘qhettoq.taproot.Database’
      Carrot can be set up to use different queuing backend for development purposes
      GhettoQ provides database backend for carrot where a database serves as the queue storage
      The broker is a simple Django application that monitors the queue in DB
      Inefficient, therefore only suitable for development environments
    • 18. Ease a future MQ implementation
      Isolate functionality into reusable components.
    • 19. Recycle Existing Code
    • 20. Django, HTTP Requests & MQ
      Most functionalities provided throughDjangoapplications are tightly coupled with HTTP requests e.g. form data processing
      In order to asynchronously execute a task, Celery (any task management platform) must serialize & store parameters
      HTTP requests are usually large and not easily serializable. e.g. WSGI requests
    • 21. ThePickleableHTTPRequestObject
    • 22. UsingPickleableHttpRequest
      if request.GET.get(‘download’):
      try:
      result = EnrollmentCSVExportTask.delay(
      PickleableHttpRequest(
      request,
      attributes=
      [‘user’, ‘admin_current_organization’]
      )
      )
    • 23. Making Message Queue Tasks
      It is fast and easy to add a decorator to specific functions to create Message Queue tasks.
      from celery.decoratorsimport task
      @task
      def add(x, y):
      return x + y
    • 24. Explicitly Inherit from Celery’s Task Object
    • 25. Then What?
      Some Message Queue tasks can complete with no notification required.
      For other tasks:
      Notify by email
      Use an AJAX listener to notify users
      Some other messaging
    • 26. In Review
      Choose your MQ solution wisely
      Set up a robust system
      Leverage your existing code
      Replace trouble spots as they come up
      Profit!
    • 27. Message Queues Make Life Better
    • 28. References and Links
      Second Life’s Survey of Message Queues:http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
      Ask SolemHoel’sGithub Repos: http://github.com/ask
      Celery Project: http://celeryproject.org/
      PickleableHttpRequest by NowellStrite (@nowells)
      http://gist.github.com/571027
      Special thanks to TarequeHossain (http://codexn.com)

    ×