Queue Everything       &Please Everyone       ***   Vaidik Kapoor
Who am I?➔   Undergraduate student.➔   Involved with:    ✔   Mozilla as a Contributor and Rep    ✔   Drupal as a Contribut...
Web Applications
Responsiveness
Request                      Response                      Client    Task 1   Task 2   Task 3   Task NWhy do everything at...
Break the Request-Response Cycle       Request                   Response                      Client           Task 1    ...
Meet Queues
What can Queues help with?➔   Background Processing    ✔   Data Processing    ✔   Media Processing    ✔   Updating Caches➔...
Redis       +HotQueue / PyRes
Redis + HotQueuePRODUCER:from hotqueue import HotQueuequeue = HotQueue("myqueue", host="localhost", port=6379, db=0)queue....
Redis + PyRes➔   Python clone of Githubs Resque➔   Offers a lot more features than HotQueue    ✔   Job failure handling   ...
Redis + PyResPRODUCER:from pyres import ResQr = Resq()class WebPage:     queue = “screenies”    @staticmethod    def perfo...
Message QueueA system for enabling asynchronous processing               of discrete tasks.
What can MQs help with?➔   Everything that generic Queues can help with.➔   Increase Reliability➔   Cron Jobs (Celery)
Available MQ Solutions➔   RabbitMQ➔   Amazon Simple Queue➔   Apache MQ➔   Gearman➔   Starling➔   OpenAMQ➔   Sun Java Messa...
Message Queue Protocols➔   AMQP: Advanced Message Queue Protocol➔   JMS: Java Messaging Service➔   STOMP: Streaming Text O...
Criteria for Broker Selection➔   Difficulty in Recovery➔   Relatively low level of required maintenance➔   Ease of Deploym...
RabbitMQ➔   A Message Broker➔   Implements AMQP
Pika & Other AMQP Libraries
Celery➔   An amazing Task Manager➔   Batteries Included➔   Uses RabbitMQ as broker (or almost anything)➔   Libraries for m...
Celery Examplefrom celery.task import task@task():def take_screenshot(url):  # magical code to take screenshot comes here
How to put it all together?
The General Setup                   TaskWeb App                        Broker                  Manager                  Wo...
Then what?
Notify Your Users
Djangos Messaging System
Flasks Flashes
Poll using AJAX
WebSockets
Email
Things to Remember
IsolateIsolate code, make reusable components.
RecycleRemove Request Dependency
Unit Testing?
Review➔   Choose a system according to needs➔   Build a robust system➔   Integrate seamlessly with your UI➔   Isolate & Re...
References➔   PyRes - http://pyres.readthedocs.org/➔   RabbitMQ - http://www.rabbitmq.com/➔   Celery - http://celeryprojec...
Thank You!
Upcoming SlideShare
Loading in...5
×

Queue Everything and Please Everyone

1,710

Published on

My presentation at PyCon India 2012 on Queues in Web Applications.

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,710
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
11
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "Queue Everything and Please Everyone"

  1. 1. Queue Everything &Please Everyone *** Vaidik Kapoor
  2. 2. Who am I?➔ Undergraduate student.➔ Involved with: ✔ Mozilla as a Contributor and Rep ✔ Drupal as a Contributor➔ Twitter: @vaidikkapoor➔ Github: vaidikkp➔ Web: vaidikkapoor.info
  3. 3. Web Applications
  4. 4. Responsiveness
  5. 5. Request Response Client Task 1 Task 2 Task 3 Task NWhy do everything at once?
  6. 6. Break the Request-Response Cycle Request Response Client Task 1 Task N Task 2 Task 3
  7. 7. Meet Queues
  8. 8. What can Queues help with?➔ Background Processing ✔ Data Processing ✔ Media Processing ✔ Updating Caches➔ Anything that you want to offload off your server.➔ Improve the overall User Experience
  9. 9. Redis +HotQueue / PyRes
  10. 10. Redis + HotQueuePRODUCER:from hotqueue import HotQueuequeue = HotQueue("myqueue", host="localhost", port=6379, db=0)queue.put(message)CONSUMER / WORKER:from hotqueue import HotQueuequeue = HotQueue("myqueue", host="localhost", port=6379, db=0)while True: message = queue.get() # do something awesome
  11. 11. Redis + PyRes➔ Python clone of Githubs Resque➔ Offers a lot more features than HotQueue ✔ Job failure handling ✔ Queue status information➔ Comes with Monitoring System written in Ruby
  12. 12. Redis + PyResPRODUCER:from pyres import ResQr = Resq()class WebPage: queue = “screenies” @staticmethod def perform(urll): # save the screenshotr.enqueue(WebPage, http://python.org)CONSUMER:./pyres_worker screenies
  13. 13. Message QueueA system for enabling asynchronous processing of discrete tasks.
  14. 14. What can MQs help with?➔ Everything that generic Queues can help with.➔ Increase Reliability➔ Cron Jobs (Celery)
  15. 15. Available MQ Solutions➔ RabbitMQ➔ Amazon Simple Queue➔ Apache MQ➔ Gearman➔ Starling➔ OpenAMQ➔ Sun Java Message Queue System
  16. 16. Message Queue Protocols➔ AMQP: Advanced Message Queue Protocol➔ JMS: Java Messaging Service➔ STOMP: Streaming Text Oriented Messaging Protocol
  17. 17. Criteria for Broker Selection➔ Difficulty in Recovery➔ Relatively low level of required maintenance➔ Ease of Deployment➔ Durability➔ Persistence➔ Community Support➔ Cluster Support➔ What language is it written in?
  18. 18. RabbitMQ➔ A Message Broker➔ Implements AMQP
  19. 19. Pika & Other AMQP Libraries
  20. 20. Celery➔ An amazing Task Manager➔ Batteries Included➔ Uses RabbitMQ as broker (or almost anything)➔ Libraries for most of the common web frameworks like Django, Flask, Pyramid➔ Supports multiprocessing.➔ Distribute tasks easily.➔ Makes life easy
  21. 21. Celery Examplefrom celery.task import task@task():def take_screenshot(url): # magical code to take screenshot comes here
  22. 22. How to put it all together?
  23. 23. The General Setup TaskWeb App Broker Manager WorkerDatabase Server (s)
  24. 24. Then what?
  25. 25. Notify Your Users
  26. 26. Djangos Messaging System
  27. 27. Flasks Flashes
  28. 28. Poll using AJAX
  29. 29. WebSockets
  30. 30. Email
  31. 31. Things to Remember
  32. 32. IsolateIsolate code, make reusable components.
  33. 33. RecycleRemove Request Dependency
  34. 34. Unit Testing?
  35. 35. Review➔ Choose a system according to needs➔ Build a robust system➔ Integrate seamlessly with your UI➔ Isolate & Recycle: make it a habit➔ Improve the overall UX!➔ Please everyone, even yourself!
  36. 36. References➔ PyRes - http://pyres.readthedocs.org/➔ RabbitMQ - http://www.rabbitmq.com/➔ Celery - http://celeryproject.org/➔ Amazing Article - http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
  37. 37. Thank You!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×