Your SlideShare is downloading. ×
Django Celery
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Django Celery

16,505
views

Published on

Quick overview of Celery for Django

Quick overview of Celery for Django

Published in: Technology

0 Comments
15 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
16,505
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
229
Comments
0
Likes
15
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. celery distributed task @matclayton
  • 2. warning
  • 3. background
  • 4. what is celery?
  • 5. what is celery? "celery is an open source asynchronous task queue/ job queue based on distributed message I. It is focused
  • 6. what is celery? "celery is an open source asynchronous task queue/ job queue based on distributed message passing. It is but what does that
  • 7. you can run • distributed • concurrently • in the background
  • 8. use cases • external api calls (twitter) • long running tasks (transcoding) • concurrent execution (batch image resize) • load balancing (across servers)
  • 9. sql django celery ORM carro AMQP SQL or STOMP
  • 10. result • database • AMQP • cache • tokyo tyrant • redis • mongodb
  • 11. components 1. views.py / management command 2. broker – RabbitMQ 3. workers
  • 12. workflow http://robertpogorzelski.com/blog/2009/09/10/rabbitmq-celery-
  • 13. rabbits and warrens
  • 14. setting up the $ sudo apt-get install rabbitmq- server $ sudo pip install celery $ rabbitmqctl add_user myuser mypassword $ rabbitmqctl add_vhost myvhost $ rabbitmqctl set_permissions -p
  • 15. setup INSTALLED_APPS += ("djcelery", ) settings.p BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = “myuser" BROKER_PASSWORD = “mypassword" BROKER_VHOST = “myvhost" CELERY_QUEUES = {     "regular_tasks": {         "binding_key": "task.#",     },     "twitter_tasks": {         "binding_key": "twitter.#",     },     "feed_tasks": {         "binding_key": "feed.#",     }, } $ python manage.py celeryd -B
  • 16. hello from celery.decorators import task tasks.p @task def add(x, y): return x + y >>> result = add.delay(4, 4) >>> result.wait() # wait 8
  • 17. post to from celery.task import Task tasks.p class UpdateStatus(Task):     name = "twitter.updatestatus"     routing_key = 'twitter.updatestatus'     ignore_result = True              def run(self, tweet, **kwargs):         post_to_twitter(tweet)          from twitter.tasks import UpdateStatus UpdateStatus.delay(tweet=‘hello world’) views.p
  • 18. retry / rate from celery.task import Task tasks.p class UpdateStatus(Task):     name = "twitter.updatestatus"     routing_key = 'twitter.updatestatus'     ignore_result = True     default_retry_delay = 5 * 60     max_retries = 12 # 1 hour retry rate_limit = ‘10/s’          def run(self, tweet, **kwargs):         try: post_to_twitter(tweet)         except Exception, exc:             # If twitter crashes retry             self.retry([tweet,], kwargs, exc=exc) from twitter.tasks import UpdateStatus views.p UpdateStatus.delay(tweet=‘hello world’)
  • 19. podcast from celery.task import PeriodicTask tasks.p class FeedImportPeriodicTask(PeriodicTask):     run_every = timedelta(hours=1)     routing_key = 'feed.periodic_import'     def run(self, **kwargs):         logger = self.get_logger(**kwargs)         logger.info("Running Periodic Feed Import task!")         update_podcasts(silent=False)
  • 20. class FeedImporter(Task):     name = "feed.import"     routing_key = 'feed.import' tasks.p     ignore_result = True     default_retry_delay = 5 * 60 # retry in 5 minutes     max_retries = 72 # 6 Hours to cover major outages     def run(self, podcast_id, **kwargs):         try:             logger = self.get_logger(**kwargs)             # The cache key consists of the task name and the MD5 digest of the feed id.             lock_id = "%s-lock-%s" % (self.name, podcast_id)             is_locked = lambda: str(cache.get(lock_id)) == "true"             acquire_lock = lambda: cache.set(lock_id, "true", 300)             # memcache delete is very slow, so we'd rather set a false value             # with a very low expiry time.             release_lock = lambda: cache.set(lock_id, "nil", 1)                  logger.debug("Trying to import feed: %s" % podcast_id)             if is_locked():                 logger.debug("Feed %s is already being imported by another worker" % podcast_id)                 return             acquire_lock()             try:                 import_feed(logger, podcast_id)             finally:                 release_lock()         except Exception, exc:             logger.error(exc)
  • 21. typical • running out of disk space == rabbitmq fail • queue priorities, difficult • non-pickle-able errors • crashing consumers
  • 22. other cool • tasksets / callbacks • remote control tasks • abortable tasks • eta – run tasks at a set time • HttpDispatchTask • expiring tasks • celerymon • celeryev • ajax views
  • 23. finding • http://github.com/ask/celery • http://github.com/ask/django- celery • irc.freenode.net #celery (asksol owner, always helpful and about)
  • 24. @matclayton mat@mixcloud.com