Massaging the Pony: Message Queues and You
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Massaging the Pony: Message Queues and You

  • 4,226 views
Uploaded on

Slides from my presentation about Message Queues at Djangocon 2010. Covers selecting your solution, design patterns, etc.

Slides from my presentation about Message Queues at Djangocon 2010. Covers selecting your solution, design patterns, etc.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
4,226
On Slideshare
4,197
From Embeds
29
Number of Embeds
4

Actions

Shares
Downloads
100
Comments
0
Likes
7

Embeds 29

http://lanyrd.com 26
https://lanyrd.com 1
http://www.linkedin.com 1
https://www.linkedin.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Example from TL about bulk refunds and paypal.
  • Cron jobs essentially pass control of your code to the operating system. Although the Cron job is a valuable tool, it’s not the only option.
  • Because a Message Queue Broker is a separate system that your project interfaces with, you could feasibly use many different apps created in many different languages.
  • Cluster support will be a priority for a lot of people, but it was not a priority for us. (Cluster support is the ability to have a cluster of MQ Brokers.)
  • To achieve this, we consider two concepts (on next slides).
  • This is what that looks like in production. Many of these components exist on dedicated servers and/or server clusters.
  • Some people who are more familiar with Carrot and Celery might wonder why not use the “always eager” setting, which processes MQ tasks in real time. However, in our experience that setting does not provide the effect we are after, and it is frustrating to get timeouts on long-running processes in development.
  • You want to replace trouble spots in your project as they come up. No need to rewrite your entire project right away.
  • The MockHTTP object always serializes GET and POST data. It then saves any additional HTTPRequest attributes specified when it is called.

Transcript

  • 1. Massaging the Pony:Message Queues and You
    Shawn Rider
    PBS Education
    DjangoCon 2010
    http://www.flickr.com/photos/funtik/1175522045
  • 2. What is a Message Queue?
    A system for enabling asynchronous processing of discrete tasks.
    Super awesome.
    http://www.flickr.com/photos/gadl/89650415
  • 3. What are they good for?
    Alleviate server issues by offloading processing
    http://www.flickr.com/photos/islandgyrl/3197469932
  • 4. What are they good for?
    Make better user experiences
    http://www.flickr.com/photos/bbaltimore/6779682
  • 5. What are they good for?
    Increase the reliability of third party service integrations.
  • 6. What are they good for?
    Background media processing
    http://www.flickr.com/photos/factoryjoe/2882992091
  • 7. What are they good for?
    Repeating Tasks can replace Cron jobs
    Keep all your code in your project
  • 8. Available MQ Solutions
    There are many solutions out there. To name a few of them:
    RabbitMQ
    Amazon Simple Queue System
    Apache MQ
    Gearman
    OpenAMQ
    Peafowl
    Q4M
    Starling
    Sun Java System Message Queue
  • 9. Message Queue Protocols
    AMQP: Advanced Message Queuing Protocol
    JMS: Java Messaging Service
    STOMP: Streaming Text Oriented Messaging Protocol
  • 10. Criteria for Broker Selection
    The following criteria were important to us in selecting a solution. You may have some different needs.
    Handling exceptions and recovery
    Low level of required maintenance
    Ease of deployment
    Persistent queuing
    Community support
    What language is it written in? How compatible is that with our current systems?
    Cluster support
  • 11. Failsafe MQ Brokers
    Must be able to survive a server failure.
    Upon restarting the MQ Broker, processing should resume from the last un-processed messages in queues
  • 12. Queue Durability
    The Messages Queue Broker must be able to reconstruct the queues when it is restarted.
    http://www.flickr.com/photos/raul/846318014
  • 13. Message Persistence
    The Messages (tasks to be completed) must persist between shutdowns and restarts of MQ Server
  • 14. Our Choice
    Celery (a Django task management app)
    Carrot (Django middleware)
    PyAMQP (Python module)
    RabbitMQ (MQ Broker software)
  • 15. The Production System
    Webapp
    Rabbit
    MQ
    Celery
    Worker
    Server(s)
    Database
  • 16. MQ in Development Environs
    It is important to be able to develop Message Queue tasks in Django without being concerned about running the broker process, queue setup, persistence etc.
  • 17. MQ in Development Environs
    CARROT_BACKEND = ‘qhettoq.taproot.Database’
    Carrot can be set up to use different queuing backend for development purposes
    GhettoQ provides database backend for carrot where a database serves as the queue storage
    The broker is a simple Django application that monitors the queue in DB
    Inefficient, therefore only suitable for development environments
  • 18. Ease a future MQ implementation
    Isolate functionality into reusable components.
  • 19. Recycle Existing Code
  • 20. Django, HTTP Requests & MQ
    Most functionalities provided throughDjangoapplications are tightly coupled with HTTP requests e.g. form data processing
    In order to asynchronously execute a task, Celery (any task management platform) must serialize & store parameters
    HTTP requests are usually large and not easily serializable. e.g. WSGI requests
  • 21. ThePickleableHTTPRequestObject
  • 22. UsingPickleableHttpRequest
    if request.GET.get(‘download’):
    try:
    result = EnrollmentCSVExportTask.delay(
    PickleableHttpRequest(
    request,
    attributes=
    [‘user’, ‘admin_current_organization’]
    )
    )
  • 23. Making Message Queue Tasks
    It is fast and easy to add a decorator to specific functions to create Message Queue tasks.
    from celery.decoratorsimport task
    @task
    def add(x, y):
    return x + y
  • 24. Explicitly Inherit from Celery’s Task Object
  • 25. Then What?
    Some Message Queue tasks can complete with no notification required.
    For other tasks:
    Notify by email
    Use an AJAX listener to notify users
    Some other messaging
  • 26. In Review
    Choose your MQ solution wisely
    Set up a robust system
    Leverage your existing code
    Replace trouble spots as they come up
    Profit!
  • 27. Message Queues Make Life Better
  • 28. References and Links
    Second Life’s Survey of Message Queues:http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
    Ask SolemHoel’sGithub Repos: http://github.com/ask
    Celery Project: http://celeryproject.org/
    PickleableHttpRequest by NowellStrite (@nowells)
    http://gist.github.com/571027
    Special thanks to TarequeHossain (http://codexn.com)