SlideShare a Scribd company logo
Celery
A Distributed Task Queue

                        Idan Gazit
 PyWeb-IL 8 / 29th September 2009
What is Celery?
Celery is a...
  Distributed
 Asynchronous
  Task Queue
  For Django
Celery is a...
  Distributed
 Asynchronous
  Task Queue
  For Django
Celery is a...
  Distributed
 Asynchronous
  Task Queue
  For Django
Celery is a...
  Distributed
 Asynchronous
  Task Queue
  For Django
Celery is a...
  Distributed
 Asynchronous
  Task Queue
              sin
  For Django 0.8 ce
What can I use it for?




                         http://www.flickr.com/photos/jabzg/2145312172/
Potential Uses
» Anything that needs to run
  asynchronously, e.g. outside of the
  request-response cycle.
» Background computation of ‘expensive
  queries’ (ex. denormalized counts)
» Interactions with external API’s
  (ex. Twitter)
» Periodic tasks (instead of cron & scripts)
» Long-running actions with results
  displayed via AJAX.
How does it work?




    http://www.flickr.com/photos/tomypelluz/14638999/
Celery Architecture
          AMQP      celery   task result
user
          broker   workers      store
Celery Architecture

user



       submit:
       tasks
       task sets
       periodic tasks
       retryable tasks
Celery Architecture
        AMQP          celery
        broker       workers




broker pushes
tasks to worker(s)
Celery Architecture
                     celery
                    workers




workers execute
tasks in parallel
(multiprocessing)
Celery Architecture
                             celery   task result
                            workers      store



task result (tombstone)
is written to task store:
‣RDBMS
‣memcached
‣Tokyo Tyrant
‣MongoDB
‣AMQP (new in 0.8)
Celery Architecture
                            task result
user
                               store
         read task result
Celery Architecture

Celery    uses...

Carrot    to talk to...

AMQP
Broker
Celery Architecture

Celery    pip install celery



Carrot    (dependency of celery)


AMQP
Broker
Celery Architecture

 Celery    pip install celery



 Carrot    (dependency of celery)



RabbitMQ   http://www.rabbitmq.com
AMQP is... Complex.
AMQP is Complex

» VHost        » Routing Keys

» Exchanges    » Bindings

 » Direct      » Queues

 » Fanout        » Durable

 » Topic         » Temporary

                 » Auto-Delete
bit.ly/amqp_intro
I Can Haz Celery?
Adding Celery

1. get & install an AMQP broker
  (pay attention to vhosts, permissions)

2. add Celery to INSTALLED_APPS

3. add a few settings:
  AMQP_SERVER = "localhost"
  AMQP_PORT = 5672
  AMQP_USER = "myuser"
  AMQP_PASSWORD = "mypassword"
  AMQP_VHOST = "myvhost"


4. manage.py syncdb
Celery Workers


» Run at least 1 celery worker server

» manage.py celeryd
  (--detatch for production)

» Can be on different machines

» Celery guarantees that tasks are only
  executed once
Tasks
Tasks


» Define tasks in your app

» app_name/tasks.py

» register & autodiscovery
  (like admin.py)
Task

from celery.task import Task
from celery.registry import tasks

class FetchUserInfoTask(Task):
    def run(self, screen_name, **kwargs):
        logger = self.get_logger(**kwargs)
        try:
             user = twitter.users.show(id=screen_name)
             logger.debug("Successfully fetched {0}".format(screen_name))
        except TwitterError:
             logger.error("Unable to fetch {0}: {1}".format(
                 screen_name, TwitterError))
            raise

        return user

tasks.register(FetchUserInfoTask)
Run It!




>>> from myapp.tasks import FetchUserInfoTask
>>> result = FetchUserInfoTask.delay('idangazit')
Task Result
» result.ready()
  true if task has finished

» result.result
  the return value of the task or exception
  instance if the task failed

» result.get()
  blocks until the task is complete then
  returns result or exception

» result.successful()
  returns True/False of task success
Why even check results?
Chained Tasks
from celery.task import Task
from celery.registry import tasks

class FetchUserInfoTask(Task):
    def run(self, screen_name, **kwargs):
        logger = self.get_logger(**kwargs)
        try:
             user = twitter.users.show(id=screen_name)
             logger.debug("Successfully fetched {0}".format(screen_name))
        except TwitterError:
             logger.error("Unable to fetch {0}: {1}".format(
                 screen_name, TwitterError))
            raise
        else:
             ProcessUserTask.delay(user)

        return user

tasks.register(FetchUserInfoTask)
Task Retries
Task Retries
from celery.task import Task
from celery.registry import tasks

class FetchUserInfoTask(Task):
    default_retry_delay = 5 * 60 # retry in 5 minutes
    max_retries = 5

   def run(self, screen_name, **kwargs):
       logger = self.get_logger(**kwargs)
       try:
            user = twitter.users.show(id=screen_name)
            logger.debug("Successfully fetched {0}".format(screen_name))
       except TwitterError, exc:
           self.retry(args=[screen_name,], kwargs=**kwargs, exc)
       else:
            ProcessUserTask.delay(user)

       return user

tasks.register(FetchUserInfoTask)
Periodic Tasks
Periodic Tasks

from celery.task import PeriodicTask
from celery.registry import tasks
from datetime import timedelta

class FetchMentionsTask(Task):
    run_every = timedelta(seconds=60)

   def run(self, **kwargs):
       logger = self.get_logger(**kwargs)
       mentions = twitter.statuses.mentions()
       for m in mentions:
           ProcessMentionTask.delay(m)

       return len(mentions)

tasks.register(FetchMentionsTask)
Task Sets
Task Sets


>>> from myapp.tasks import FetchUserInfoTask
>>> from celery.task import TaskSet
>>> ts = TaskSet(FetchUserInfoTask, args=(
            ['ahikman'], {},
            ['idangazit'], {},
            ['daonb'], {},
            ['dibau_naum_h'], {}))
>>> ts_result = ts.run()
>>> list_of_return_values = ts.join()
MOAR SELRY!
Celery.Views
Celery.Views

» Celery ships with some django views for
  launching /getting the status of tasks.
» JSON views perfect for use in your AJAX
  (err, AJAJ) calls.
» celery.views.apply(request, task_name, *args)

» celery.views.is_task_done(request, task_id)

» celery.views.task_status(request, task_id)
Routable Tasks
Routable Tasks

» "I want tasks of type X to only execute on
  this specific server"
» Some extra settings in settings.py:
  CELERY_AMQP_EXCHANGE = "tasks"
  CELERY_AMQP_PUBLISHER_ROUTING_KEY = "task.regular"
  CELERY_AMQP_EXCHANGE_TYPE = "topic"
  CELERY_AMQP_CONSUMER_QUEUE = "foo_tasks"
  CELERY_AMQP_CONSUMER_ROUTING_KEY = "foo.#"


» set the task's routing key:
  class MyRoutableTask(Task):
      routing_key = 'foo.bars'
like django, it's just python.
Support
             #celery on freenode
http://groups.google.com/group/celery-users/

    AskSol (the author) is friendly & helpful
Fin.
   @idangazit
idan@pixane.com

More Related Content

What's hot

Celery
CeleryCelery
Celery
Fatih Erikli
 
Scaling up task processing with Celery
Scaling up task processing with CeleryScaling up task processing with Celery
Scaling up task processing with Celery
Nicolas Grasset
 
詳説WebAssembly
詳説WebAssembly詳説WebAssembly
詳説WebAssembly
祐司 伊藤
 
php-src の歩き方
php-src の歩き方php-src の歩き方
php-src の歩き方
do_aki
 
Selenium WebDriver avec Java
Selenium WebDriver avec Java Selenium WebDriver avec Java
Selenium WebDriver avec Java
Ahmed HARRAK
 
Django Celery
Django Celery Django Celery
Django Celery
Mat Clayton
 
Java Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame GraphsJava Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame Graphs
Brendan Gregg
 
GraphQL IN Golang
GraphQL IN GolangGraphQL IN Golang
GraphQL IN Golang
Bo-Yi Wu
 
Celery with python
Celery with pythonCelery with python
Celery with python
Alexandre González Rodríguez
 
Ansible
AnsibleAnsible
Ansible
Vishal Yadav
 
Nuxt.js - Introduction
Nuxt.js - IntroductionNuxt.js - Introduction
Nuxt.js - Introduction
Sébastien Chopin
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
Brendan Gregg
 
Ansible - Hands on Training
Ansible - Hands on TrainingAnsible - Hands on Training
Ansible - Hands on Training
Mehmet Ali Aydın
 
InjectionIII의 Hot Reload를 이용하여 앱 개발을 좀 더 편하게 하기.pdf
InjectionIII의 Hot Reload를 이용하여 앱 개발을 좀 더 편하게 하기.pdfInjectionIII의 Hot Reload를 이용하여 앱 개발을 좀 더 편하게 하기.pdf
InjectionIII의 Hot Reload를 이용하여 앱 개발을 좀 더 편하게 하기.pdf
정민 안
 
Quick tour of PHP from inside
Quick tour of PHP from insideQuick tour of PHP from inside
Quick tour of PHP from inside
julien pauli
 
PHP と SAPI と ZendEngine3 と
PHP と SAPI と ZendEngine3 とPHP と SAPI と ZendEngine3 と
PHP と SAPI と ZendEngine3 と
do_aki
 
젠킨스 설치 및 설정
젠킨스 설치 및 설정젠킨스 설치 및 설정
젠킨스 설치 및 설정
중선 곽
 
ansible why ?
ansible why ?ansible why ?
ansible why ?
Yashar Esmaildokht
 
[231]운영체제 수준에서의 데이터베이스 성능 분석과 최적화
[231]운영체제 수준에서의 데이터베이스 성능 분석과 최적화[231]운영체제 수준에서의 데이터베이스 성능 분석과 최적화
[231]운영체제 수준에서의 데이터베이스 성능 분석과 최적화
NAVER D2
 
Why Task Queues - ComoRichWeb
Why Task Queues - ComoRichWebWhy Task Queues - ComoRichWeb
Why Task Queues - ComoRichWeb
Bryan Helmig
 

What's hot (20)

Celery
CeleryCelery
Celery
 
Scaling up task processing with Celery
Scaling up task processing with CeleryScaling up task processing with Celery
Scaling up task processing with Celery
 
詳説WebAssembly
詳説WebAssembly詳説WebAssembly
詳説WebAssembly
 
php-src の歩き方
php-src の歩き方php-src の歩き方
php-src の歩き方
 
Selenium WebDriver avec Java
Selenium WebDriver avec Java Selenium WebDriver avec Java
Selenium WebDriver avec Java
 
Django Celery
Django Celery Django Celery
Django Celery
 
Java Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame GraphsJava Performance Analysis on Linux with Flame Graphs
Java Performance Analysis on Linux with Flame Graphs
 
GraphQL IN Golang
GraphQL IN GolangGraphQL IN Golang
GraphQL IN Golang
 
Celery with python
Celery with pythonCelery with python
Celery with python
 
Ansible
AnsibleAnsible
Ansible
 
Nuxt.js - Introduction
Nuxt.js - IntroductionNuxt.js - Introduction
Nuxt.js - Introduction
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
 
Ansible - Hands on Training
Ansible - Hands on TrainingAnsible - Hands on Training
Ansible - Hands on Training
 
InjectionIII의 Hot Reload를 이용하여 앱 개발을 좀 더 편하게 하기.pdf
InjectionIII의 Hot Reload를 이용하여 앱 개발을 좀 더 편하게 하기.pdfInjectionIII의 Hot Reload를 이용하여 앱 개발을 좀 더 편하게 하기.pdf
InjectionIII의 Hot Reload를 이용하여 앱 개발을 좀 더 편하게 하기.pdf
 
Quick tour of PHP from inside
Quick tour of PHP from insideQuick tour of PHP from inside
Quick tour of PHP from inside
 
PHP と SAPI と ZendEngine3 と
PHP と SAPI と ZendEngine3 とPHP と SAPI と ZendEngine3 と
PHP と SAPI と ZendEngine3 と
 
젠킨스 설치 및 설정
젠킨스 설치 및 설정젠킨스 설치 및 설정
젠킨스 설치 및 설정
 
ansible why ?
ansible why ?ansible why ?
ansible why ?
 
[231]운영체제 수준에서의 데이터베이스 성능 분석과 최적화
[231]운영체제 수준에서의 데이터베이스 성능 분석과 최적화[231]운영체제 수준에서의 데이터베이스 성능 분석과 최적화
[231]운영체제 수준에서의 데이터베이스 성능 분석과 최적화
 
Why Task Queues - ComoRichWeb
Why Task Queues - ComoRichWebWhy Task Queues - ComoRichWeb
Why Task Queues - ComoRichWeb
 

Similar to An Introduction to Celery

Threads, Queues, and More: Async Programming in iOS
Threads, Queues, and More: Async Programming in iOSThreads, Queues, and More: Async Programming in iOS
Threads, Queues, and More: Async Programming in iOS
TechWell
 
[NDC 2019] Functions 2.0: Enterprise-Grade Serverless
[NDC 2019] Functions 2.0: Enterprise-Grade Serverless[NDC 2019] Functions 2.0: Enterprise-Grade Serverless
[NDC 2019] Functions 2.0: Enterprise-Grade Serverless
KatyShimizu
 
[NDC 2019] Enterprise-Grade Serverless
[NDC 2019] Enterprise-Grade Serverless[NDC 2019] Enterprise-Grade Serverless
[NDC 2019] Enterprise-Grade Serverless
KatyShimizu
 
Celery
CeleryCelery
Asynchronous Task Queues with Celery
Asynchronous Task Queues with CeleryAsynchronous Task Queues with Celery
Asynchronous Task Queues with Celery
Kishor Kumar
 
How and why i roll my own node.js framework
How and why i roll my own node.js frameworkHow and why i roll my own node.js framework
How and why i roll my own node.js framework
Ben Lin
 
Google App Engine Developer - Day3
Google App Engine Developer - Day3Google App Engine Developer - Day3
Google App Engine Developer - Day3
Simon Su
 
Fun Teaching MongoDB New Tricks
Fun Teaching MongoDB New TricksFun Teaching MongoDB New Tricks
Fun Teaching MongoDB New Tricks
MongoDB
 
Deixa para depois, Procrastinando com Celery em Python
Deixa para depois, Procrastinando com Celery em PythonDeixa para depois, Procrastinando com Celery em Python
Deixa para depois, Procrastinando com Celery em Python
Adriano Petrich
 
using Mithril.js + postgREST to build and consume API's
using Mithril.js + postgREST to build and consume API'susing Mithril.js + postgREST to build and consume API's
using Mithril.js + postgREST to build and consume API's
Antônio Roberto Silva
 
JavaCro'14 - Unit testing in AngularJS – Slaven Tomac
JavaCro'14 - Unit testing in AngularJS – Slaven TomacJavaCro'14 - Unit testing in AngularJS – Slaven Tomac
JavaCro'14 - Unit testing in AngularJS – Slaven Tomac
HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
 
JavaScript Growing Up
JavaScript Growing UpJavaScript Growing Up
JavaScript Growing Up
David Padbury
 
Slaven tomac unit testing in angular js
Slaven tomac   unit testing in angular jsSlaven tomac   unit testing in angular js
Slaven tomac unit testing in angular js
Slaven Tomac
 
GPerf Using Jesque
GPerf Using JesqueGPerf Using Jesque
GPerf Using Jesque
ctoestreich
 
apidays LIVE Australia 2020 - Building distributed systems on the shoulders o...
apidays LIVE Australia 2020 - Building distributed systems on the shoulders o...apidays LIVE Australia 2020 - Building distributed systems on the shoulders o...
apidays LIVE Australia 2020 - Building distributed systems on the shoulders o...
apidays
 
Build Widgets
Build WidgetsBuild Widgets
Build Widgets
scottw
 
Re-Design with Elixir/OTP
Re-Design with Elixir/OTPRe-Design with Elixir/OTP
Re-Design with Elixir/OTP
Mustafa TURAN
 
Forge - DevCon 2016: Building a Drone Imagery Service
Forge - DevCon 2016: Building a Drone Imagery ServiceForge - DevCon 2016: Building a Drone Imagery Service
Forge - DevCon 2016: Building a Drone Imagery Service
Autodesk
 
Javascript first-class citizenery
Javascript first-class citizeneryJavascript first-class citizenery
Javascript first-class citizenery
toddbr
 
Intoduction to Play Framework
Intoduction to Play FrameworkIntoduction to Play Framework
Intoduction to Play Framework
Knoldus Inc.
 

Similar to An Introduction to Celery (20)

Threads, Queues, and More: Async Programming in iOS
Threads, Queues, and More: Async Programming in iOSThreads, Queues, and More: Async Programming in iOS
Threads, Queues, and More: Async Programming in iOS
 
[NDC 2019] Functions 2.0: Enterprise-Grade Serverless
[NDC 2019] Functions 2.0: Enterprise-Grade Serverless[NDC 2019] Functions 2.0: Enterprise-Grade Serverless
[NDC 2019] Functions 2.0: Enterprise-Grade Serverless
 
[NDC 2019] Enterprise-Grade Serverless
[NDC 2019] Enterprise-Grade Serverless[NDC 2019] Enterprise-Grade Serverless
[NDC 2019] Enterprise-Grade Serverless
 
Celery
CeleryCelery
Celery
 
Asynchronous Task Queues with Celery
Asynchronous Task Queues with CeleryAsynchronous Task Queues with Celery
Asynchronous Task Queues with Celery
 
How and why i roll my own node.js framework
How and why i roll my own node.js frameworkHow and why i roll my own node.js framework
How and why i roll my own node.js framework
 
Google App Engine Developer - Day3
Google App Engine Developer - Day3Google App Engine Developer - Day3
Google App Engine Developer - Day3
 
Fun Teaching MongoDB New Tricks
Fun Teaching MongoDB New TricksFun Teaching MongoDB New Tricks
Fun Teaching MongoDB New Tricks
 
Deixa para depois, Procrastinando com Celery em Python
Deixa para depois, Procrastinando com Celery em PythonDeixa para depois, Procrastinando com Celery em Python
Deixa para depois, Procrastinando com Celery em Python
 
using Mithril.js + postgREST to build and consume API's
using Mithril.js + postgREST to build and consume API'susing Mithril.js + postgREST to build and consume API's
using Mithril.js + postgREST to build and consume API's
 
JavaCro'14 - Unit testing in AngularJS – Slaven Tomac
JavaCro'14 - Unit testing in AngularJS – Slaven TomacJavaCro'14 - Unit testing in AngularJS – Slaven Tomac
JavaCro'14 - Unit testing in AngularJS – Slaven Tomac
 
JavaScript Growing Up
JavaScript Growing UpJavaScript Growing Up
JavaScript Growing Up
 
Slaven tomac unit testing in angular js
Slaven tomac   unit testing in angular jsSlaven tomac   unit testing in angular js
Slaven tomac unit testing in angular js
 
GPerf Using Jesque
GPerf Using JesqueGPerf Using Jesque
GPerf Using Jesque
 
apidays LIVE Australia 2020 - Building distributed systems on the shoulders o...
apidays LIVE Australia 2020 - Building distributed systems on the shoulders o...apidays LIVE Australia 2020 - Building distributed systems on the shoulders o...
apidays LIVE Australia 2020 - Building distributed systems on the shoulders o...
 
Build Widgets
Build WidgetsBuild Widgets
Build Widgets
 
Re-Design with Elixir/OTP
Re-Design with Elixir/OTPRe-Design with Elixir/OTP
Re-Design with Elixir/OTP
 
Forge - DevCon 2016: Building a Drone Imagery Service
Forge - DevCon 2016: Building a Drone Imagery ServiceForge - DevCon 2016: Building a Drone Imagery Service
Forge - DevCon 2016: Building a Drone Imagery Service
 
Javascript first-class citizenery
Javascript first-class citizeneryJavascript first-class citizenery
Javascript first-class citizenery
 
Intoduction to Play Framework
Intoduction to Play FrameworkIntoduction to Play Framework
Intoduction to Play Framework
 

More from Idan Gazit

Datadesignmeaning
DatadesignmeaningDatadesignmeaning
Datadesignmeaning
Idan Gazit
 
Designers Make It Go to Eleven!
Designers Make It Go to Eleven!Designers Make It Go to Eleven!
Designers Make It Go to Eleven!
Idan Gazit
 
Web typography
Web typographyWeb typography
Web typography
Idan Gazit
 
CSS Extenders
CSS ExtendersCSS Extenders
CSS Extenders
Idan Gazit
 
CSS for Designers
CSS for DesignersCSS for Designers
CSS for Designers
Idan Gazit
 
CSS for Designers
CSS for DesignersCSS for Designers
CSS for Designers
Idan Gazit
 
CSS: selectors and the box model
CSS: selectors and the box modelCSS: selectors and the box model
CSS: selectors and the box model
Idan Gazit
 
CSS: selectors and the box model
CSS: selectors and the box modelCSS: selectors and the box model
CSS: selectors and the box model
Idan Gazit
 
Why Django
Why DjangoWhy Django
Why Django
Idan Gazit
 
Repeatable Deployments and Installations
Repeatable Deployments and InstallationsRepeatable Deployments and Installations
Repeatable Deployments and Installations
Idan Gazit
 
Django 1.1 Tour
Django 1.1 TourDjango 1.1 Tour
Django 1.1 Tour
Idan Gazit
 

More from Idan Gazit (11)

Datadesignmeaning
DatadesignmeaningDatadesignmeaning
Datadesignmeaning
 
Designers Make It Go to Eleven!
Designers Make It Go to Eleven!Designers Make It Go to Eleven!
Designers Make It Go to Eleven!
 
Web typography
Web typographyWeb typography
Web typography
 
CSS Extenders
CSS ExtendersCSS Extenders
CSS Extenders
 
CSS for Designers
CSS for DesignersCSS for Designers
CSS for Designers
 
CSS for Designers
CSS for DesignersCSS for Designers
CSS for Designers
 
CSS: selectors and the box model
CSS: selectors and the box modelCSS: selectors and the box model
CSS: selectors and the box model
 
CSS: selectors and the box model
CSS: selectors and the box modelCSS: selectors and the box model
CSS: selectors and the box model
 
Why Django
Why DjangoWhy Django
Why Django
 
Repeatable Deployments and Installations
Repeatable Deployments and InstallationsRepeatable Deployments and Installations
Repeatable Deployments and Installations
 
Django 1.1 Tour
Django 1.1 TourDjango 1.1 Tour
Django 1.1 Tour
 

Recently uploaded

Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Precisely
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
Shinana2
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Alex Pruden
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Postman
 
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process MiningTrusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
LucaBarbaro3
 
Azure API Management to expose backend services securely
Azure API Management to expose backend services securelyAzure API Management to expose backend services securely
Azure API Management to expose backend services securely
Dinusha Kumarasiri
 
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
alexjohnson7307
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying AheadDigital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Wask
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 

Recently uploaded (20)

Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
 
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process MiningTrusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
 
Azure API Management to expose backend services securely
Azure API Management to expose backend services securelyAzure API Management to expose backend services securely
Azure API Management to expose backend services securely
 
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying AheadDigital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 

An Introduction to Celery

  • 1. Celery A Distributed Task Queue Idan Gazit PyWeb-IL 8 / 29th September 2009
  • 3. Celery is a... Distributed Asynchronous Task Queue For Django
  • 4. Celery is a... Distributed Asynchronous Task Queue For Django
  • 5. Celery is a... Distributed Asynchronous Task Queue For Django
  • 6. Celery is a... Distributed Asynchronous Task Queue For Django
  • 7. Celery is a... Distributed Asynchronous Task Queue sin For Django 0.8 ce
  • 8. What can I use it for? http://www.flickr.com/photos/jabzg/2145312172/
  • 9. Potential Uses » Anything that needs to run asynchronously, e.g. outside of the request-response cycle. » Background computation of ‘expensive queries’ (ex. denormalized counts) » Interactions with external API’s (ex. Twitter) » Periodic tasks (instead of cron & scripts) » Long-running actions with results displayed via AJAX.
  • 10. How does it work? http://www.flickr.com/photos/tomypelluz/14638999/
  • 11. Celery Architecture AMQP celery task result user broker workers store
  • 12. Celery Architecture user submit: tasks task sets periodic tasks retryable tasks
  • 13. Celery Architecture AMQP celery broker workers broker pushes tasks to worker(s)
  • 14. Celery Architecture celery workers workers execute tasks in parallel (multiprocessing)
  • 15. Celery Architecture celery task result workers store task result (tombstone) is written to task store: ‣RDBMS ‣memcached ‣Tokyo Tyrant ‣MongoDB ‣AMQP (new in 0.8)
  • 16. Celery Architecture task result user store read task result
  • 17. Celery Architecture Celery uses... Carrot to talk to... AMQP Broker
  • 18. Celery Architecture Celery pip install celery Carrot (dependency of celery) AMQP Broker
  • 19. Celery Architecture Celery pip install celery Carrot (dependency of celery) RabbitMQ http://www.rabbitmq.com
  • 21. AMQP is Complex » VHost » Routing Keys » Exchanges » Bindings » Direct » Queues » Fanout » Durable » Topic » Temporary » Auto-Delete
  • 23. I Can Haz Celery?
  • 24. Adding Celery 1. get & install an AMQP broker (pay attention to vhosts, permissions) 2. add Celery to INSTALLED_APPS 3. add a few settings: AMQP_SERVER = "localhost" AMQP_PORT = 5672 AMQP_USER = "myuser" AMQP_PASSWORD = "mypassword" AMQP_VHOST = "myvhost" 4. manage.py syncdb
  • 25. Celery Workers » Run at least 1 celery worker server » manage.py celeryd (--detatch for production) » Can be on different machines » Celery guarantees that tasks are only executed once
  • 26. Tasks
  • 27. Tasks » Define tasks in your app » app_name/tasks.py » register & autodiscovery (like admin.py)
  • 28. Task from celery.task import Task from celery.registry import tasks class FetchUserInfoTask(Task): def run(self, screen_name, **kwargs): logger = self.get_logger(**kwargs) try: user = twitter.users.show(id=screen_name) logger.debug("Successfully fetched {0}".format(screen_name)) except TwitterError: logger.error("Unable to fetch {0}: {1}".format( screen_name, TwitterError)) raise return user tasks.register(FetchUserInfoTask)
  • 29. Run It! >>> from myapp.tasks import FetchUserInfoTask >>> result = FetchUserInfoTask.delay('idangazit')
  • 30. Task Result » result.ready() true if task has finished » result.result the return value of the task or exception instance if the task failed » result.get() blocks until the task is complete then returns result or exception » result.successful() returns True/False of task success
  • 31. Why even check results?
  • 32. Chained Tasks from celery.task import Task from celery.registry import tasks class FetchUserInfoTask(Task): def run(self, screen_name, **kwargs): logger = self.get_logger(**kwargs) try: user = twitter.users.show(id=screen_name) logger.debug("Successfully fetched {0}".format(screen_name)) except TwitterError: logger.error("Unable to fetch {0}: {1}".format( screen_name, TwitterError)) raise else: ProcessUserTask.delay(user) return user tasks.register(FetchUserInfoTask)
  • 34. Task Retries from celery.task import Task from celery.registry import tasks class FetchUserInfoTask(Task): default_retry_delay = 5 * 60 # retry in 5 minutes max_retries = 5 def run(self, screen_name, **kwargs): logger = self.get_logger(**kwargs) try: user = twitter.users.show(id=screen_name) logger.debug("Successfully fetched {0}".format(screen_name)) except TwitterError, exc: self.retry(args=[screen_name,], kwargs=**kwargs, exc) else: ProcessUserTask.delay(user) return user tasks.register(FetchUserInfoTask)
  • 36. Periodic Tasks from celery.task import PeriodicTask from celery.registry import tasks from datetime import timedelta class FetchMentionsTask(Task): run_every = timedelta(seconds=60) def run(self, **kwargs): logger = self.get_logger(**kwargs) mentions = twitter.statuses.mentions() for m in mentions: ProcessMentionTask.delay(m) return len(mentions) tasks.register(FetchMentionsTask)
  • 38. Task Sets >>> from myapp.tasks import FetchUserInfoTask >>> from celery.task import TaskSet >>> ts = TaskSet(FetchUserInfoTask, args=( ['ahikman'], {}, ['idangazit'], {}, ['daonb'], {}, ['dibau_naum_h'], {})) >>> ts_result = ts.run() >>> list_of_return_values = ts.join()
  • 41. Celery.Views » Celery ships with some django views for launching /getting the status of tasks. » JSON views perfect for use in your AJAX (err, AJAJ) calls. » celery.views.apply(request, task_name, *args) » celery.views.is_task_done(request, task_id) » celery.views.task_status(request, task_id)
  • 43. Routable Tasks » "I want tasks of type X to only execute on this specific server" » Some extra settings in settings.py: CELERY_AMQP_EXCHANGE = "tasks" CELERY_AMQP_PUBLISHER_ROUTING_KEY = "task.regular" CELERY_AMQP_EXCHANGE_TYPE = "topic" CELERY_AMQP_CONSUMER_QUEUE = "foo_tasks" CELERY_AMQP_CONSUMER_ROUTING_KEY = "foo.#" » set the task's routing key: class MyRoutableTask(Task): routing_key = 'foo.bars'
  • 44. like django, it's just python.
  • 45. Support #celery on freenode http://groups.google.com/group/celery-users/ AskSol (the author) is friendly & helpful
  • 46. Fin. @idangazit idan@pixane.com