Queue Everything and Please Everyone

  • 1,553 views
Uploaded on

My presentation at PyCon India 2012 on Queues in Web Applications.

My presentation at PyCon India 2012 on Queues in Web Applications.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,553
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
7
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Queue Everything &Please Everyone *** Vaidik Kapoor
  • 2. Who am I?➔ Undergraduate student.➔ Involved with: ✔ Mozilla as a Contributor and Rep ✔ Drupal as a Contributor➔ Twitter: @vaidikkapoor➔ Github: vaidikkp➔ Web: vaidikkapoor.info
  • 3. Web Applications
  • 4. Responsiveness
  • 5. Request Response Client Task 1 Task 2 Task 3 Task NWhy do everything at once?
  • 6. Break the Request-Response Cycle Request Response Client Task 1 Task N Task 2 Task 3
  • 7. Meet Queues
  • 8. What can Queues help with?➔ Background Processing ✔ Data Processing ✔ Media Processing ✔ Updating Caches➔ Anything that you want to offload off your server.➔ Improve the overall User Experience
  • 9. Redis +HotQueue / PyRes
  • 10. Redis + HotQueuePRODUCER:from hotqueue import HotQueuequeue = HotQueue("myqueue", host="localhost", port=6379, db=0)queue.put(message)CONSUMER / WORKER:from hotqueue import HotQueuequeue = HotQueue("myqueue", host="localhost", port=6379, db=0)while True: message = queue.get() # do something awesome
  • 11. Redis + PyRes➔ Python clone of Githubs Resque➔ Offers a lot more features than HotQueue ✔ Job failure handling ✔ Queue status information➔ Comes with Monitoring System written in Ruby
  • 12. Redis + PyResPRODUCER:from pyres import ResQr = Resq()class WebPage: queue = “screenies” @staticmethod def perform(urll): # save the screenshotr.enqueue(WebPage, http://python.org)CONSUMER:./pyres_worker screenies
  • 13. Message QueueA system for enabling asynchronous processing of discrete tasks.
  • 14. What can MQs help with?➔ Everything that generic Queues can help with.➔ Increase Reliability➔ Cron Jobs (Celery)
  • 15. Available MQ Solutions➔ RabbitMQ➔ Amazon Simple Queue➔ Apache MQ➔ Gearman➔ Starling➔ OpenAMQ➔ Sun Java Message Queue System
  • 16. Message Queue Protocols➔ AMQP: Advanced Message Queue Protocol➔ JMS: Java Messaging Service➔ STOMP: Streaming Text Oriented Messaging Protocol
  • 17. Criteria for Broker Selection➔ Difficulty in Recovery➔ Relatively low level of required maintenance➔ Ease of Deployment➔ Durability➔ Persistence➔ Community Support➔ Cluster Support➔ What language is it written in?
  • 18. RabbitMQ➔ A Message Broker➔ Implements AMQP
  • 19. Pika & Other AMQP Libraries
  • 20. Celery➔ An amazing Task Manager➔ Batteries Included➔ Uses RabbitMQ as broker (or almost anything)➔ Libraries for most of the common web frameworks like Django, Flask, Pyramid➔ Supports multiprocessing.➔ Distribute tasks easily.➔ Makes life easy
  • 21. Celery Examplefrom celery.task import task@task():def take_screenshot(url): # magical code to take screenshot comes here
  • 22. How to put it all together?
  • 23. The General Setup TaskWeb App Broker Manager WorkerDatabase Server (s)
  • 24. Then what?
  • 25. Notify Your Users
  • 26. Djangos Messaging System
  • 27. Flasks Flashes
  • 28. Poll using AJAX
  • 29. WebSockets
  • 30. Email
  • 31. Things to Remember
  • 32. IsolateIsolate code, make reusable components.
  • 33. RecycleRemove Request Dependency
  • 34. Unit Testing?
  • 35. Review➔ Choose a system according to needs➔ Build a robust system➔ Integrate seamlessly with your UI➔ Isolate & Recycle: make it a habit➔ Improve the overall UX!➔ Please everyone, even yourself!
  • 36. References➔ PyRes - http://pyres.readthedocs.org/➔ RabbitMQ - http://www.rabbitmq.com/➔ Celery - http://celeryproject.org/➔ Amazing Article - http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
  • 37. Thank You!