Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Asynchronous job queues with python-rq


Published on

A short presentation of Redis Queue ( and integrating it with Python/Django.

Watch the presentation here:

Published in: Technology
  • Be the first to comment

Asynchronous job queues with python-rq

  1. 1. AsynchronousJobQueueswith PythonRQ AshishAcharya
  2. 2. Outline ● Asynchronous job queuing ● Redis ● Redis Queue (RQ) ● django-rq
  3. 3. AboutMe ● Computer Engineering graduate from Pulchowk Campus ● Runs a Python shop called Thoplo ● Does a lot of Python/Django ● Uses far too many exec statements ● ● Not a designer
  4. 4. WhyAsynchronousJobQueues? ● Some problems take too long to solve within a single process ● Web applications need to provide an immediate response ● Sending emails, processing images, making API calls, etc. can take time ● So background processes and job queues
  5. 5. ToolsforparallelisminPython ● Celery ● IPython.parallel ● Parallel Python ● pymq ● snakeMQ ● Redis Queue
  6. 6. ChallengeswithCelery ● Lots of one-off configuration ● Difficult and time-consuming to set up ● Difficult to get right - lots of moving parts that can fail ● Time consuming to monitor and maintain ● Needs right kind of storage mechanism with the right kind of message queues
  7. 7. MainProblemwithCelery It’s OVERKILL for MOST web projects!
  8. 8. IntroducingRedisQueue ● RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. ● Has a low barrier to entry. ● Has been created as a lightweight alternative to existing queueing frameworks.
  9. 9. WhyRedisQueue? ● Can get it up and running in an afternoon ● No configuration file editing -- or very little ● Don’t have to configure a message broker -- just uses Redis
  10. 10. AboutRedis A server that lets you associate a data structure with a key ● Written in C ● Bindings in most popular languages ● Easy to access from Python ● Used extensively for caching in web applications ● I know almost nothing about it.
  11. 11. AboutRedis import redis connection = redis.Redis(‘localhost’) connection.set(‘key’, ‘value’) print(connection.get(‘key’))
  12. 12. doesRedisQueueuseRedis?Duh! A Python module for treating Redis like a queue ● Pure Python ● Provides a clean, well thought out API for humans
  13. 13. ASimpleExample from redis import Redis from rq import Queue import time q = Queue(connection=Redis()) job = q.enqueue(‘operator.add’, 2, 3) time.sleep(1) print(job.result)
  14. 14. What’sSoCoolAboutRedisQueue?
  15. 15. ...This Monitoring on a web interface: rq-dashboard
  16. 16. ...AlsoThis:django-rq ● Django integration with RQ ● Define queues in ● Add some urls ● Done! You can start queuing jobs immediately.
  17. 17. Django-rq ● Enqueue jobs like so: import django_rq django_rq.enqueue(func, foo, bar=baz) ● Has a @job decorator like Celery’s @task ● Has management commands for rqworker and rqscheduler python rqworker high default low python rqscheduler
  18. 18. Alsohasacooldashboard!
  19. 19. ● Procrastinate digitally...schedule a job for later. ● Send spam daily...and other periodic tasks ...AndThis:rq-scheduler
  20. 20. ...andfinally If you’re already using Redis for caching, you can use the same connection information and reap the benefits of optimization. “Two birds, one Redis.” -- A.A
  21. 21. Twoscoopsapproved
  22. 22. Demo
  23. 23. Tips Break jobs down into atomic packages that don’t do anything else. At a point when this doesn’t work for you anymore, it’s definitely time to upgrade to a more powerful, more complex tool -- Celery. (Or just hire someone to do it for you!)
  24. 24. Questions?