Django Celery
Upcoming SlideShare
Loading in...5
×
 

Django Celery

on

  • 16,507 views

Quick overview of Celery for Django

Quick overview of Celery for Django

Statistics

Views

Total Views
16,507
Slideshare-icon Views on SlideShare
10,404
Embed Views
6,103

Actions

Likes
13
Downloads
201
Comments
0

19 Embeds 6,103

http://www.python.rk.edu.pl 2054
http://ask.github.com 1475
http://docs.celeryproject.org 1257
http://celery.readthedocs.org 925
http://packages.python.org 88
http://celeryproject.org 69
http://localhost 62
http://readthedocs.org 49
http://docs.celeryq.org 42
http://localhost:8080 23
http://python.rk.edu.pl 16
https://celery.readthedocs.org 13
http://celeryq.org 10
http://celery.github.com 8
http://python2web.com 6
http://www.linkedin.com 2
https://www.linkedin.com 2
http://webcache.googleusercontent.com 1
http://python.mc69.org 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Django Celery Django Celery Presentation Transcript

    • celery distributed task @matclayton
    • warning
    • background
    • what is celery?
    • what is celery? "celery is an open source asynchronous task queue/ job queue based on distributed message I. It is focused
    • what is celery? "celery is an open source asynchronous task queue/ job queue based on distributed message passing. It is but what does that
    • you can run • distributed • concurrently • in the background
    • use cases • external api calls (twitter) • long running tasks (transcoding) • concurrent execution (batch image resize) • load balancing (across servers)
    • sql django celery ORM carro AMQP SQL or STOMP
    • result • database • AMQP • cache • tokyo tyrant • redis • mongodb
    • components 1. views.py / management command 2. broker – RabbitMQ 3. workers
    • workflow http://robertpogorzelski.com/blog/2009/09/10/rabbitmq-celery-
    • rabbits and warrens
    • setting up the $ sudo apt-get install rabbitmq- server $ sudo pip install celery $ rabbitmqctl add_user myuser mypassword $ rabbitmqctl add_vhost myvhost $ rabbitmqctl set_permissions -p
    • setup INSTALLED_APPS += ("djcelery", ) settings.p BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = “myuser" BROKER_PASSWORD = “mypassword" BROKER_VHOST = “myvhost" CELERY_QUEUES = {     "regular_tasks": {         "binding_key": "task.#",     },     "twitter_tasks": {         "binding_key": "twitter.#",     },     "feed_tasks": {         "binding_key": "feed.#",     }, } $ python manage.py celeryd -B
    • hello from celery.decorators import task tasks.p @task def add(x, y): return x + y >>> result = add.delay(4, 4) >>> result.wait() # wait 8
    • post to from celery.task import Task tasks.p class UpdateStatus(Task):     name = "twitter.updatestatus"     routing_key = 'twitter.updatestatus'     ignore_result = True              def run(self, tweet, **kwargs):         post_to_twitter(tweet)          from twitter.tasks import UpdateStatus UpdateStatus.delay(tweet=‘hello world’) views.p
    • retry / rate from celery.task import Task tasks.p class UpdateStatus(Task):     name = "twitter.updatestatus"     routing_key = 'twitter.updatestatus'     ignore_result = True     default_retry_delay = 5 * 60     max_retries = 12 # 1 hour retry rate_limit = ‘10/s’          def run(self, tweet, **kwargs):         try: post_to_twitter(tweet)         except Exception, exc:             # If twitter crashes retry             self.retry([tweet,], kwargs, exc=exc) from twitter.tasks import UpdateStatus views.p UpdateStatus.delay(tweet=‘hello world’)
    • podcast from celery.task import PeriodicTask tasks.p class FeedImportPeriodicTask(PeriodicTask):     run_every = timedelta(hours=1)     routing_key = 'feed.periodic_import'     def run(self, **kwargs):         logger = self.get_logger(**kwargs)         logger.info("Running Periodic Feed Import task!")         update_podcasts(silent=False)
    • class FeedImporter(Task):     name = "feed.import"     routing_key = 'feed.import' tasks.p     ignore_result = True     default_retry_delay = 5 * 60 # retry in 5 minutes     max_retries = 72 # 6 Hours to cover major outages     def run(self, podcast_id, **kwargs):         try:             logger = self.get_logger(**kwargs)             # The cache key consists of the task name and the MD5 digest of the feed id.             lock_id = "%s-lock-%s" % (self.name, podcast_id)             is_locked = lambda: str(cache.get(lock_id)) == "true"             acquire_lock = lambda: cache.set(lock_id, "true", 300)             # memcache delete is very slow, so we'd rather set a false value             # with a very low expiry time.             release_lock = lambda: cache.set(lock_id, "nil", 1)                  logger.debug("Trying to import feed: %s" % podcast_id)             if is_locked():                 logger.debug("Feed %s is already being imported by another worker" % podcast_id)                 return             acquire_lock()             try:                 import_feed(logger, podcast_id)             finally:                 release_lock()         except Exception, exc:             logger.error(exc)
    • typical • running out of disk space == rabbitmq fail • queue priorities, difficult • non-pickle-able errors • crashing consumers
    • other cool • tasksets / callbacks • remote control tasks • abortable tasks • eta – run tasks at a set time • HttpDispatchTask • expiring tasks • celerymon • celeryev • ajax views
    • finding • http://github.com/ask/celery • http://github.com/ask/django- celery • irc.freenode.net #celery (asksol owner, always helpful and about)
    • @matclayton mat@mixcloud.com