Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Asynchronous Task Queues with Celery


Published on

Asynchronous Task Queues with Celery

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Asynchronous Task Queues with Celery

  1. 1. Kishor Kumar Mahato Python Lead at Eagle Vision IT Python User Group Nepal Meetup #7 2nd August 2015 Celery Asynchronous Task Queues
  2. 2.  Distributed Asynchronous Task Queue For Python  It is a task queue which focus on real-time processing  It also supports task scheduling
  3. 3. • Anything that needs to run asynchronously, eg. outside of the request-response cycle. • Background computation of ‘expensive queries’ • Interactions with external API’s (eg. Twitter) • Periodic tasks (instead of cron & scripts) • Long-running actions with results displayed via AJAX. Potential Uses
  4. 4. Work Flow
  5. 5. Celery Workers  Run at least 1 celery worker server ( Redis / RabbitMQ)  Can be on different machines  Celery guarantees that tasks are only executed once
  6. 6. Task from celery.task import Task from celery.registry import tasks class FetchUserInfoTask(Task): def run(self, screen_name, **kwargs): logger = self.get_logger(**kwargs) try: user = logger.debug("Successfully fetched {0}".format(screen_name)) except TwitterError: logger.error("Unable to fetch {0}: {1}".format( screen_name, TwitterError)) raise return user tasks.register(FetchUserInfoTask)
  7. 7. >>> from myapp.tasks import FetchUserInfoTask >>> result = FetchUserInfoTask.delay(“kishor_14”) Run It!
  8. 8. Periodic Tasks from celery.task import PeriodicTask from celery.registry import tasks from datetime import timedelta class FetchMentionsTask(Task): run_every = timedelta(seconds=60) def run(self, **kwargs): logger = self.get_logger(**kwargs) mentions = twitter.statuses.mentions() for m in mentions: ProcessMentionTask.delay(m) return len(mentions) tasks.register(FetchMentionsTask)
  9. 9. Periodic Tasks from datetime import timedelta from celery.schedules import crontab CELERYBEAT_SCHEDULE = { 'order-reminder-every-1-hours': { 'task’: 'application.views.cron_job_for_application_processing', 'schedule': crontab(minute=0, hour='*/1') # 'schedule': timedelta(seconds=30) }, } CELERY_TIMEZONE = 'UTC'
  10. 10. Support  
  11. 11. Thank You! Kishor Kumar Mahato @kishor_14 fb:kishrorkumar