• Like
Queue Everything and Please Everyone
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Queue Everything and Please Everyone


My presentation at PyCon India 2012 on Queues in Web Applications.

My presentation at PyCon India 2012 on Queues in Web Applications.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Queue Everything &Please Everyone *** Vaidik Kapoor
  • 2. Who am I?➔ Undergraduate student.➔ Involved with: ✔ Mozilla as a Contributor and Rep ✔ Drupal as a Contributor➔ Twitter: @vaidikkapoor➔ Github: vaidikkp➔ Web: vaidikkapoor.info
  • 3. Web Applications
  • 4. Responsiveness
  • 5. Request Response Client Task 1 Task 2 Task 3 Task NWhy do everything at once?
  • 6. Break the Request-Response Cycle Request Response Client Task 1 Task N Task 2 Task 3
  • 7. Meet Queues
  • 8. What can Queues help with?➔ Background Processing ✔ Data Processing ✔ Media Processing ✔ Updating Caches➔ Anything that you want to offload off your server.➔ Improve the overall User Experience
  • 9. Redis +HotQueue / PyRes
  • 10. Redis + HotQueuePRODUCER:from hotqueue import HotQueuequeue = HotQueue("myqueue", host="localhost", port=6379, db=0)queue.put(message)CONSUMER / WORKER:from hotqueue import HotQueuequeue = HotQueue("myqueue", host="localhost", port=6379, db=0)while True: message = queue.get() # do something awesome
  • 11. Redis + PyRes➔ Python clone of Githubs Resque➔ Offers a lot more features than HotQueue ✔ Job failure handling ✔ Queue status information➔ Comes with Monitoring System written in Ruby
  • 12. Redis + PyResPRODUCER:from pyres import ResQr = Resq()class WebPage: queue = “screenies” @staticmethod def perform(urll): # save the screenshotr.enqueue(WebPage, http://python.org)CONSUMER:./pyres_worker screenies
  • 13. Message QueueA system for enabling asynchronous processing of discrete tasks.
  • 14. What can MQs help with?➔ Everything that generic Queues can help with.➔ Increase Reliability➔ Cron Jobs (Celery)
  • 15. Available MQ Solutions➔ RabbitMQ➔ Amazon Simple Queue➔ Apache MQ➔ Gearman➔ Starling➔ OpenAMQ➔ Sun Java Message Queue System
  • 16. Message Queue Protocols➔ AMQP: Advanced Message Queue Protocol➔ JMS: Java Messaging Service➔ STOMP: Streaming Text Oriented Messaging Protocol
  • 17. Criteria for Broker Selection➔ Difficulty in Recovery➔ Relatively low level of required maintenance➔ Ease of Deployment➔ Durability➔ Persistence➔ Community Support➔ Cluster Support➔ What language is it written in?
  • 18. RabbitMQ➔ A Message Broker➔ Implements AMQP
  • 19. Pika & Other AMQP Libraries
  • 20. Celery➔ An amazing Task Manager➔ Batteries Included➔ Uses RabbitMQ as broker (or almost anything)➔ Libraries for most of the common web frameworks like Django, Flask, Pyramid➔ Supports multiprocessing.➔ Distribute tasks easily.➔ Makes life easy
  • 21. Celery Examplefrom celery.task import task@task():def take_screenshot(url): # magical code to take screenshot comes here
  • 22. How to put it all together?
  • 23. The General Setup TaskWeb App Broker Manager WorkerDatabase Server (s)
  • 24. Then what?
  • 25. Notify Your Users
  • 26. Djangos Messaging System
  • 27. Flasks Flashes
  • 28. Poll using AJAX
  • 29. WebSockets
  • 30. Email
  • 31. Things to Remember
  • 32. IsolateIsolate code, make reusable components.
  • 33. RecycleRemove Request Dependency
  • 34. Unit Testing?
  • 35. Review➔ Choose a system according to needs➔ Build a robust system➔ Integrate seamlessly with your UI➔ Isolate & Recycle: make it a habit➔ Improve the overall UX!➔ Please everyone, even yourself!
  • 36. References➔ PyRes - http://pyres.readthedocs.org/➔ RabbitMQ - http://www.rabbitmq.com/➔ Celery - http://celeryproject.org/➔ Amazing Article - http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
  • 37. Thank You!