Django Celery

19,240 views

Published on

Quick overview of Celery for Django

Published in: Technology
0 Comments
17 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
19,240
On SlideShare
0
From Embeds
0
Number of Embeds
7,563
Actions
Shares
0
Downloads
257
Comments
0
Likes
17
Embeds 0
No embeds

No notes for slide

Django Celery

  1. 1. celery distributed task @matclayton
  2. 2. warning
  3. 3. background
  4. 4. what is celery?
  5. 5. what is celery? "celery is an open source asynchronous task queue/ job queue based on distributed message I. It is focused
  6. 6. what is celery? "celery is an open source asynchronous task queue/ job queue based on distributed message passing. It is but what does that
  7. 7. you can run • distributed • concurrently • in the background
  8. 8. use cases • external api calls (twitter) • long running tasks (transcoding) • concurrent execution (batch image resize) • load balancing (across servers)
  9. 9. sql django celery ORM carro AMQP SQL or STOMP
  10. 10. result • database • AMQP • cache • tokyo tyrant • redis • mongodb
  11. 11. components 1. views.py / management command 2. broker – RabbitMQ 3. workers
  12. 12. workflow http://robertpogorzelski.com/blog/2009/09/10/rabbitmq-celery-
  13. 13. rabbits and warrens
  14. 14. setting up the $ sudo apt-get install rabbitmq- server $ sudo pip install celery $ rabbitmqctl add_user myuser mypassword $ rabbitmqctl add_vhost myvhost $ rabbitmqctl set_permissions -p
  15. 15. setup INSTALLED_APPS += ("djcelery", ) settings.p BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = “myuser" BROKER_PASSWORD = “mypassword" BROKER_VHOST = “myvhost" CELERY_QUEUES = {     "regular_tasks": {         "binding_key": "task.#",     },     "twitter_tasks": {         "binding_key": "twitter.#",     },     "feed_tasks": {         "binding_key": "feed.#",     }, } $ python manage.py celeryd -B
  16. 16. hello from celery.decorators import task tasks.p @task def add(x, y): return x + y >>> result = add.delay(4, 4) >>> result.wait() # wait 8
  17. 17. post to from celery.task import Task tasks.p class UpdateStatus(Task):     name = "twitter.updatestatus"     routing_key = 'twitter.updatestatus'     ignore_result = True              def run(self, tweet, **kwargs):         post_to_twitter(tweet)          from twitter.tasks import UpdateStatus UpdateStatus.delay(tweet=‘hello world’) views.p
  18. 18. retry / rate from celery.task import Task tasks.p class UpdateStatus(Task):     name = "twitter.updatestatus"     routing_key = 'twitter.updatestatus'     ignore_result = True     default_retry_delay = 5 * 60     max_retries = 12 # 1 hour retry rate_limit = ‘10/s’          def run(self, tweet, **kwargs):         try: post_to_twitter(tweet)         except Exception, exc:             # If twitter crashes retry             self.retry([tweet,], kwargs, exc=exc) from twitter.tasks import UpdateStatus views.p UpdateStatus.delay(tweet=‘hello world’)
  19. 19. podcast from celery.task import PeriodicTask tasks.p class FeedImportPeriodicTask(PeriodicTask):     run_every = timedelta(hours=1)     routing_key = 'feed.periodic_import'     def run(self, **kwargs):         logger = self.get_logger(**kwargs)         logger.info("Running Periodic Feed Import task!")         update_podcasts(silent=False)
  20. 20. class FeedImporter(Task):     name = "feed.import"     routing_key = 'feed.import' tasks.p     ignore_result = True     default_retry_delay = 5 * 60 # retry in 5 minutes     max_retries = 72 # 6 Hours to cover major outages     def run(self, podcast_id, **kwargs):         try:             logger = self.get_logger(**kwargs)             # The cache key consists of the task name and the MD5 digest of the feed id.             lock_id = "%s-lock-%s" % (self.name, podcast_id)             is_locked = lambda: str(cache.get(lock_id)) == "true"             acquire_lock = lambda: cache.set(lock_id, "true", 300)             # memcache delete is very slow, so we'd rather set a false value             # with a very low expiry time.             release_lock = lambda: cache.set(lock_id, "nil", 1)                  logger.debug("Trying to import feed: %s" % podcast_id)             if is_locked():                 logger.debug("Feed %s is already being imported by another worker" % podcast_id)                 return             acquire_lock()             try:                 import_feed(logger, podcast_id)             finally:                 release_lock()         except Exception, exc:             logger.error(exc)
  21. 21. typical • running out of disk space == rabbitmq fail • queue priorities, difficult • non-pickle-able errors • crashing consumers
  22. 22. other cool • tasksets / callbacks • remote control tasks • abortable tasks • eta – run tasks at a set time • HttpDispatchTask • expiring tasks • celerymon • celeryev • ajax views
  23. 23. finding • http://github.com/ask/celery • http://github.com/ask/django- celery • irc.freenode.net #celery (asksol owner, always helpful and about)
  24. 24. @matclayton mat@mixcloud.com

×