Queue Everything and Please Everyone
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


Queue Everything and Please Everyone



My presentation at PyCon India 2012 on Queues in Web Applications.

My presentation at PyCon India 2012 on Queues in Web Applications.



Total Views
Views on SlideShare
Embed Views



2 Embeds 19

http://www.linkedin.com 12
https://www.linkedin.com 7



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Queue Everything and Please Everyone Presentation Transcript

  • 1. Queue Everything &Please Everyone *** Vaidik Kapoor
  • 2. Who am I?➔ Undergraduate student.➔ Involved with: ✔ Mozilla as a Contributor and Rep ✔ Drupal as a Contributor➔ Twitter: @vaidikkapoor➔ Github: vaidikkp➔ Web: vaidikkapoor.info
  • 3. Web Applications
  • 4. Responsiveness
  • 5. Request Response Client Task 1 Task 2 Task 3 Task NWhy do everything at once?
  • 6. Break the Request-Response Cycle Request Response Client Task 1 Task N Task 2 Task 3
  • 7. Meet Queues
  • 8. What can Queues help with?➔ Background Processing ✔ Data Processing ✔ Media Processing ✔ Updating Caches➔ Anything that you want to offload off your server.➔ Improve the overall User Experience
  • 9. Redis +HotQueue / PyRes
  • 10. Redis + HotQueuePRODUCER:from hotqueue import HotQueuequeue = HotQueue("myqueue", host="localhost", port=6379, db=0)queue.put(message)CONSUMER / WORKER:from hotqueue import HotQueuequeue = HotQueue("myqueue", host="localhost", port=6379, db=0)while True: message = queue.get() # do something awesome
  • 11. Redis + PyRes➔ Python clone of Githubs Resque➔ Offers a lot more features than HotQueue ✔ Job failure handling ✔ Queue status information➔ Comes with Monitoring System written in Ruby
  • 12. Redis + PyResPRODUCER:from pyres import ResQr = Resq()class WebPage: queue = “screenies” @staticmethod def perform(urll): # save the screenshotr.enqueue(WebPage, http://python.org)CONSUMER:./pyres_worker screenies
  • 13. Message QueueA system for enabling asynchronous processing of discrete tasks.
  • 14. What can MQs help with?➔ Everything that generic Queues can help with.➔ Increase Reliability➔ Cron Jobs (Celery)
  • 15. Available MQ Solutions➔ RabbitMQ➔ Amazon Simple Queue➔ Apache MQ➔ Gearman➔ Starling➔ OpenAMQ➔ Sun Java Message Queue System
  • 16. Message Queue Protocols➔ AMQP: Advanced Message Queue Protocol➔ JMS: Java Messaging Service➔ STOMP: Streaming Text Oriented Messaging Protocol
  • 17. Criteria for Broker Selection➔ Difficulty in Recovery➔ Relatively low level of required maintenance➔ Ease of Deployment➔ Durability➔ Persistence➔ Community Support➔ Cluster Support➔ What language is it written in?
  • 18. RabbitMQ➔ A Message Broker➔ Implements AMQP
  • 19. Pika & Other AMQP Libraries
  • 20. Celery➔ An amazing Task Manager➔ Batteries Included➔ Uses RabbitMQ as broker (or almost anything)➔ Libraries for most of the common web frameworks like Django, Flask, Pyramid➔ Supports multiprocessing.➔ Distribute tasks easily.➔ Makes life easy
  • 21. Celery Examplefrom celery.task import task@task():def take_screenshot(url): # magical code to take screenshot comes here
  • 22. How to put it all together?
  • 23. The General Setup TaskWeb App Broker Manager WorkerDatabase Server (s)
  • 24. Then what?
  • 25. Notify Your Users
  • 26. Djangos Messaging System
  • 27. Flasks Flashes
  • 28. Poll using AJAX
  • 29. WebSockets
  • 30. Email
  • 31. Things to Remember
  • 32. IsolateIsolate code, make reusable components.
  • 33. RecycleRemove Request Dependency
  • 34. Unit Testing?
  • 35. Review➔ Choose a system according to needs➔ Build a robust system➔ Integrate seamlessly with your UI➔ Isolate & Recycle: make it a habit➔ Improve the overall UX!➔ Please everyone, even yourself!
  • 36. References➔ PyRes - http://pyres.readthedocs.org/➔ RabbitMQ - http://www.rabbitmq.com/➔ Celery - http://celeryproject.org/➔ Amazing Article - http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
  • 37. Thank You!