• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Django Celery
 

Django Celery

on

  • 16,191 views

Quick overview of Celery for Django

Quick overview of Celery for Django

Statistics

Views

Total Views
16,191
Views on SlideShare
10,132
Embed Views
6,059

Actions

Likes
13
Downloads
199
Comments
0

18 Embeds 6,059

http://www.python.rk.edu.pl 2017
http://ask.github.com 1475
http://docs.celeryproject.org 1252
http://celery.readthedocs.org 925
http://packages.python.org 88
http://celeryproject.org 69
http://localhost 62
http://readthedocs.org 49
http://docs.celeryq.org 42
http://localhost:8080 23
http://python.rk.edu.pl 16
https://celery.readthedocs.org 13
http://celeryq.org 10
http://celery.github.com 8
http://python2web.com 6
http://www.linkedin.com 2
http://webcache.googleusercontent.com 1
http://python.mc69.org 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Django Celery Django Celery Presentation Transcript

    • celery distributed task @matclayton
    • warning
    • background
    • what is celery?
    • what is celery? "celery is an open source asynchronous task queue/ job queue based on distributed message I. It is focused
    • what is celery? "celery is an open source asynchronous task queue/ job queue based on distributed message passing. It is but what does that
    • you can run • distributed • concurrently • in the background
    • use cases • external api calls (twitter) • long running tasks (transcoding) • concurrent execution (batch image resize) • load balancing (across servers)
    • sql django celery ORM carro AMQP SQL or STOMP
    • result • database • AMQP • cache • tokyo tyrant • redis • mongodb
    • components 1. views.py / management command 2. broker – RabbitMQ 3. workers
    • workflow http://robertpogorzelski.com/blog/2009/09/10/rabbitmq-celery-
    • rabbits and warrens
    • setting up the $ sudo apt-get install rabbitmq- server $ sudo pip install celery $ rabbitmqctl add_user myuser mypassword $ rabbitmqctl add_vhost myvhost $ rabbitmqctl set_permissions -p
    • setup INSTALLED_APPS += ("djcelery", ) settings.p BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = “myuser" BROKER_PASSWORD = “mypassword" BROKER_VHOST = “myvhost" CELERY_QUEUES = {     "regular_tasks": {         "binding_key": "task.#",     },     "twitter_tasks": {         "binding_key": "twitter.#",     },     "feed_tasks": {         "binding_key": "feed.#",     }, } $ python manage.py celeryd -B
    • hello from celery.decorators import task tasks.p @task def add(x, y): return x + y >>> result = add.delay(4, 4) >>> result.wait() # wait 8
    • post to from celery.task import Task tasks.p class UpdateStatus(Task):     name = "twitter.updatestatus"     routing_key = 'twitter.updatestatus'     ignore_result = True              def run(self, tweet, **kwargs):         post_to_twitter(tweet)          from twitter.tasks import UpdateStatus UpdateStatus.delay(tweet=‘hello world’) views.p
    • retry / rate from celery.task import Task tasks.p class UpdateStatus(Task):     name = "twitter.updatestatus"     routing_key = 'twitter.updatestatus'     ignore_result = True     default_retry_delay = 5 * 60     max_retries = 12 # 1 hour retry rate_limit = ‘10/s’          def run(self, tweet, **kwargs):         try: post_to_twitter(tweet)         except Exception, exc:             # If twitter crashes retry             self.retry([tweet,], kwargs, exc=exc) from twitter.tasks import UpdateStatus views.p UpdateStatus.delay(tweet=‘hello world’)
    • podcast from celery.task import PeriodicTask tasks.p class FeedImportPeriodicTask(PeriodicTask):     run_every = timedelta(hours=1)     routing_key = 'feed.periodic_import'     def run(self, **kwargs):         logger = self.get_logger(**kwargs)         logger.info("Running Periodic Feed Import task!")         update_podcasts(silent=False)
    • class FeedImporter(Task):     name = "feed.import"     routing_key = 'feed.import' tasks.p     ignore_result = True     default_retry_delay = 5 * 60 # retry in 5 minutes     max_retries = 72 # 6 Hours to cover major outages     def run(self, podcast_id, **kwargs):         try:             logger = self.get_logger(**kwargs)             # The cache key consists of the task name and the MD5 digest of the feed id.             lock_id = "%s-lock-%s" % (self.name, podcast_id)             is_locked = lambda: str(cache.get(lock_id)) == "true"             acquire_lock = lambda: cache.set(lock_id, "true", 300)             # memcache delete is very slow, so we'd rather set a false value             # with a very low expiry time.             release_lock = lambda: cache.set(lock_id, "nil", 1)                  logger.debug("Trying to import feed: %s" % podcast_id)             if is_locked():                 logger.debug("Feed %s is already being imported by another worker" % podcast_id)                 return             acquire_lock()             try:                 import_feed(logger, podcast_id)             finally:                 release_lock()         except Exception, exc:             logger.error(exc)
    • typical • running out of disk space == rabbitmq fail • queue priorities, difficult • non-pickle-able errors • crashing consumers
    • other cool • tasksets / callbacks • remote control tasks • abortable tasks • eta – run tasks at a set time • HttpDispatchTask • expiring tasks • celerymon • celeryev • ajax views
    • finding • http://github.com/ask/celery • http://github.com/ask/django- celery • irc.freenode.net #celery (asksol owner, always helpful and about)
    • @matclayton mat@mixcloud.com