SlideShare a Scribd company logo
1 of 25
celery
distributed task




                   @matclayton
warning
background
what is celery?
what is celery?



  "celery is an open source asynchronous task queue/
job
  queue based on distributed message I. It is focused
what is celery?



  "celery is an open source asynchronous task queue/
job
  queue based on distributed message passing. It is




but what does that
you can run


• distributed
• concurrently
• in the background
use cases


  • external api calls (twitter)
  • long running tasks (transcoding)
  • concurrent execution (batch image
    resize)
  • load balancing (across servers)
sql

      django   celery

      ORM      carro

               AMQP
      SQL        or
               STOMP
result


•   database
•   AMQP
•   cache
•   tokyo tyrant
•   redis
•   mongodb
components


1. views.py / management
   command
2. broker – RabbitMQ
3. workers
workflow




http://robertpogorzelski.com/blog/2009/09/10/rabbitmq-celery-
rabbits and
warrens
setting up the

 $ sudo apt-get install rabbitmq-
  server
 $ sudo pip install celery


$ rabbitmqctl add_user myuser
 mypassword
$ rabbitmqctl add_vhost myvhost
$ rabbitmqctl set_permissions -p
setup
        INSTALLED_APPS += ("djcelery", )      settings.p
        BROKER_HOST = "localhost"
        BROKER_PORT = 5672
        BROKER_USER = “myuser"
        BROKER_PASSWORD = “mypassword"
        BROKER_VHOST = “myvhost"

        CELERY_QUEUES = {
            "regular_tasks": {
                "binding_key": "task.#",
            },
            "twitter_tasks": {
                "binding_key": "twitter.#",
            },
            "feed_tasks": {
                "binding_key": "feed.#",
            },
        }


        $ python manage.py celeryd -B
hello

    from celery.decorators import task
                                         tasks.p
    @task
    def add(x, y):
      return x + y




    >>> result = add.delay(4, 4)
    >>> result.wait() # wait
    8
post to

from celery.task import Task
                                             tasks.p
class UpdateStatus(Task):
    name = "twitter.updatestatus"
    routing_key = 'twitter.updatestatus'
    ignore_result = True
        
    def run(self, tweet, **kwargs):
        post_to_twitter(tweet)
        

from twitter.tasks import UpdateStatus
UpdateStatus.delay(tweet=‘hello world’)
                                             views.p
retry / rate
from celery.task import Task
                                                    tasks.p
class UpdateStatus(Task):
    name = "twitter.updatestatus"
    routing_key = 'twitter.updatestatus'
    ignore_result = True
    default_retry_delay = 5 * 60
    max_retries = 12 # 1 hour retry
     rate_limit = ‘10/s’
    
    def run(self, tweet, **kwargs):
        try:
            post_to_twitter(tweet)
        except Exception, exc:
            # If twitter crashes retry
            self.retry([tweet,], kwargs, exc=exc)

from twitter.tasks import UpdateStatus              views.p
UpdateStatus.delay(tweet=‘hello world’)
podcast


from celery.task import PeriodicTask
                                                 tasks.p
class FeedImportPeriodicTask(PeriodicTask):
    run_every = timedelta(hours=1)
    routing_key = 'feed.periodic_import'

    def run(self, **kwargs):
        logger = self.get_logger(**kwargs)
        logger.info("Running Periodic Feed Import task!")
        update_podcasts(silent=False)
class FeedImporter(Task):
    name = "feed.import"
    routing_key = 'feed.import'
                                                                            tasks.p
    ignore_result = True
    default_retry_delay = 5 * 60 # retry in 5 minutes
    max_retries = 72 # 6 Hours to cover major outages

    def run(self, podcast_id, **kwargs):
        try:
            logger = self.get_logger(**kwargs)
            # The cache key consists of the task name and the MD5 digest of the feed id.
            lock_id = "%s-lock-%s" % (self.name, podcast_id)
            is_locked = lambda: str(cache.get(lock_id)) == "true"
            acquire_lock = lambda: cache.set(lock_id, "true", 300)
            # memcache delete is very slow, so we'd rather set a false value
            # with a very low expiry time.
            release_lock = lambda: cache.set(lock_id, "nil", 1)
    
            logger.debug("Trying to import feed: %s" % podcast_id)
            if is_locked():
                logger.debug("Feed %s is already being imported by another worker" % podcast_id)
                return

            acquire_lock()
            try:
                import_feed(logger, podcast_id)
            finally:
                release_lock()
        except Exception, exc:
            logger.error(exc)
typical




• running out of disk space ==
  rabbitmq fail
• queue priorities, difficult
• non-pickle-able errors
• crashing consumers
other cool


   •   tasksets / callbacks
   •   remote control tasks
   •   abortable tasks
   •   eta – run tasks at a set time
   •   HttpDispatchTask
   •   expiring tasks
   •   celerymon
   •   celeryev
   •   ajax views
finding




• http://github.com/ask/celery
• http://github.com/ask/django-
  celery
• irc.freenode.net #celery (asksol
  owner, always helpful and about)
@matclayton
mat@mixcloud.com

More Related Content

What's hot

Introduction to Python Celery
Introduction to Python CeleryIntroduction to Python Celery
Introduction to Python CeleryMahendra M
 
Hello Armeria, Bye Spring
Hello Armeria, Bye SpringHello Armeria, Bye Spring
Hello Armeria, Bye SpringGihwan Kim
 
Node.js and Selenium Webdriver, a journey from the Java side
Node.js and Selenium Webdriver, a journey from the Java sideNode.js and Selenium Webdriver, a journey from the Java side
Node.js and Selenium Webdriver, a journey from the Java sideMek Srunyu Stittri
 
Celery - A Distributed Task Queue
Celery - A Distributed Task QueueCelery - A Distributed Task Queue
Celery - A Distributed Task QueueDuy Do
 
Workshop spring session 2 - La persistance au sein des applications Java
Workshop spring   session 2 - La persistance au sein des applications JavaWorkshop spring   session 2 - La persistance au sein des applications Java
Workshop spring session 2 - La persistance au sein des applications JavaAntoine Rey
 
CQRS and what it means for your architecture
CQRS and what it means for your architectureCQRS and what it means for your architecture
CQRS and what it means for your architectureRichard Banks
 
A New Era of SSRF - Exploiting URL Parser in Trending Programming Languages! ...
A New Era of SSRF - Exploiting URL Parser in Trending Programming Languages! ...A New Era of SSRF - Exploiting URL Parser in Trending Programming Languages! ...
A New Era of SSRF - Exploiting URL Parser in Trending Programming Languages! ...CODE BLUE
 
Functional Programming in JavaScript by Luis Atencio
Functional Programming in JavaScript by Luis AtencioFunctional Programming in JavaScript by Luis Atencio
Functional Programming in JavaScript by Luis AtencioLuis Atencio
 
Kubernetes DNS Horror Stories
Kubernetes DNS Horror StoriesKubernetes DNS Horror Stories
Kubernetes DNS Horror StoriesLaurent Bernaille
 
Why Task Queues - ComoRichWeb
Why Task Queues - ComoRichWebWhy Task Queues - ComoRichWeb
Why Task Queues - ComoRichWebBryan Helmig
 
우아한 모노리스
우아한 모노리스우아한 모노리스
우아한 모노리스Arawn Park
 
Java 8-streams-collectors-patterns
Java 8-streams-collectors-patternsJava 8-streams-collectors-patterns
Java 8-streams-collectors-patternsJosé Paumard
 
You code sucks, let's fix it
You code sucks, let's fix itYou code sucks, let's fix it
You code sucks, let's fix itRafael Dohms
 
Life in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoLife in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoTareque Hossain
 

What's hot (20)

Introduction to Python Celery
Introduction to Python CeleryIntroduction to Python Celery
Introduction to Python Celery
 
Flower and celery
Flower and celeryFlower and celery
Flower and celery
 
Hello Armeria, Bye Spring
Hello Armeria, Bye SpringHello Armeria, Bye Spring
Hello Armeria, Bye Spring
 
Node.js and Selenium Webdriver, a journey from the Java side
Node.js and Selenium Webdriver, a journey from the Java sideNode.js and Selenium Webdriver, a journey from the Java side
Node.js and Selenium Webdriver, a journey from the Java side
 
Celery - A Distributed Task Queue
Celery - A Distributed Task QueueCelery - A Distributed Task Queue
Celery - A Distributed Task Queue
 
PHP MVC
PHP MVCPHP MVC
PHP MVC
 
Workshop spring session 2 - La persistance au sein des applications Java
Workshop spring   session 2 - La persistance au sein des applications JavaWorkshop spring   session 2 - La persistance au sein des applications Java
Workshop spring session 2 - La persistance au sein des applications Java
 
CQRS and what it means for your architecture
CQRS and what it means for your architectureCQRS and what it means for your architecture
CQRS and what it means for your architecture
 
Java 8 Workshop
Java 8 WorkshopJava 8 Workshop
Java 8 Workshop
 
OpenID Connect 4 SSI
OpenID Connect 4 SSIOpenID Connect 4 SSI
OpenID Connect 4 SSI
 
Introduction to Celery
Introduction to CeleryIntroduction to Celery
Introduction to Celery
 
A New Era of SSRF - Exploiting URL Parser in Trending Programming Languages! ...
A New Era of SSRF - Exploiting URL Parser in Trending Programming Languages! ...A New Era of SSRF - Exploiting URL Parser in Trending Programming Languages! ...
A New Era of SSRF - Exploiting URL Parser in Trending Programming Languages! ...
 
Functional Programming in JavaScript by Luis Atencio
Functional Programming in JavaScript by Luis AtencioFunctional Programming in JavaScript by Luis Atencio
Functional Programming in JavaScript by Luis Atencio
 
Kubernetes DNS Horror Stories
Kubernetes DNS Horror StoriesKubernetes DNS Horror Stories
Kubernetes DNS Horror Stories
 
Why Task Queues - ComoRichWeb
Why Task Queues - ComoRichWebWhy Task Queues - ComoRichWeb
Why Task Queues - ComoRichWeb
 
우아한 모노리스
우아한 모노리스우아한 모노리스
우아한 모노리스
 
Java 8-streams-collectors-patterns
Java 8-streams-collectors-patternsJava 8-streams-collectors-patterns
Java 8-streams-collectors-patterns
 
You code sucks, let's fix it
You code sucks, let's fix itYou code sucks, let's fix it
You code sucks, let's fix it
 
API
APIAPI
API
 
Life in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoLife in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with django
 

Viewers also liked

Understanding Non Blocking I/O with Python
Understanding Non Blocking I/O with PythonUnderstanding Non Blocking I/O with Python
Understanding Non Blocking I/O with PythonVaidik Kapoor
 
Building Distributed System with Celery on Docker Swarm
Building Distributed System with Celery on Docker SwarmBuilding Distributed System with Celery on Docker Swarm
Building Distributed System with Celery on Docker SwarmWei Lin
 
Celery in the Django
Celery in the DjangoCelery in the Django
Celery in the DjangoWalter Liu
 
Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...
Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...
Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...Jimmy DeadcOde
 
Resftul API Web Development with Django Rest Framework & Celery
Resftul API Web Development with Django Rest Framework & CeleryResftul API Web Development with Django Rest Framework & Celery
Resftul API Web Development with Django Rest Framework & CeleryRidwan Fadjar
 
Distributed Task Processing with Celery - PyZH
Distributed Task Processing with Celery - PyZHDistributed Task Processing with Celery - PyZH
Distributed Task Processing with Celery - PyZHCesar Cardenas Desales
 
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016Wei Lin
 
An Introduction to Celery
An Introduction to CeleryAn Introduction to Celery
An Introduction to CeleryIdan Gazit
 
Queue Everything and Please Everyone
Queue Everything and Please EveryoneQueue Everything and Please Everyone
Queue Everything and Please EveryoneVaidik Kapoor
 

Viewers also liked (10)

Understanding Non Blocking I/O with Python
Understanding Non Blocking I/O with PythonUnderstanding Non Blocking I/O with Python
Understanding Non Blocking I/O with Python
 
Building Distributed System with Celery on Docker Swarm
Building Distributed System with Celery on Docker SwarmBuilding Distributed System with Celery on Docker Swarm
Building Distributed System with Celery on Docker Swarm
 
Celery in the Django
Celery in the DjangoCelery in the Django
Celery in the Django
 
Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...
Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...
Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitM...
 
Celery by dummy
Celery by dummyCelery by dummy
Celery by dummy
 
Resftul API Web Development with Django Rest Framework & Celery
Resftul API Web Development with Django Rest Framework & CeleryResftul API Web Development with Django Rest Framework & Celery
Resftul API Web Development with Django Rest Framework & Celery
 
Distributed Task Processing with Celery - PyZH
Distributed Task Processing with Celery - PyZHDistributed Task Processing with Celery - PyZH
Distributed Task Processing with Celery - PyZH
 
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
 
An Introduction to Celery
An Introduction to CeleryAn Introduction to Celery
An Introduction to Celery
 
Queue Everything and Please Everyone
Queue Everything and Please EveryoneQueue Everything and Please Everyone
Queue Everything and Please Everyone
 

Similar to Django Celery

Testing My Patience
Testing My PatienceTesting My Patience
Testing My PatienceAdam Lowry
 
Background Jobs with Resque
Background Jobs with ResqueBackground Jobs with Resque
Background Jobs with Resquehomanj
 
Debugging: Rules And Tools - PHPTek 11 Version
Debugging: Rules And Tools - PHPTek 11 VersionDebugging: Rules And Tools - PHPTek 11 Version
Debugging: Rules And Tools - PHPTek 11 VersionIan Barber
 
Debugging: Rules & Tools
Debugging: Rules & ToolsDebugging: Rules & Tools
Debugging: Rules & ToolsIan Barber
 
Threads, Queues, and More: Async Programming in iOS
Threads, Queues, and More: Async Programming in iOSThreads, Queues, and More: Async Programming in iOS
Threads, Queues, and More: Async Programming in iOSTechWell
 
Google App Engine Developer - Day3
Google App Engine Developer - Day3Google App Engine Developer - Day3
Google App Engine Developer - Day3Simon Su
 
Introduction to Protractor
Introduction to ProtractorIntroduction to Protractor
Introduction to ProtractorJie-Wei Wu
 
Akka Cluster in Java - JCConf 2015
Akka Cluster in Java - JCConf 2015Akka Cluster in Java - JCConf 2015
Akka Cluster in Java - JCConf 2015Jiayun Zhou
 
Ten useful JavaScript tips & best practices
Ten useful JavaScript tips & best practicesTen useful JavaScript tips & best practices
Ten useful JavaScript tips & best practicesAnkit Rastogi
 
Automation with Ansible and Containers
Automation with Ansible and ContainersAutomation with Ansible and Containers
Automation with Ansible and ContainersRodolfo Carvalho
 
DjangoCon US 2011 - Monkeying around at New Relic
DjangoCon US 2011 - Monkeying around at New RelicDjangoCon US 2011 - Monkeying around at New Relic
DjangoCon US 2011 - Monkeying around at New RelicGraham Dumpleton
 
Djangocon11: Monkeying around at New Relic
Djangocon11: Monkeying around at New RelicDjangocon11: Monkeying around at New Relic
Djangocon11: Monkeying around at New RelicNew Relic
 
Тестирование и Django
Тестирование и DjangoТестирование и Django
Тестирование и DjangoMoscowDjango
 
Deixa para depois, Procrastinando com Celery em Python
Deixa para depois, Procrastinando com Celery em PythonDeixa para depois, Procrastinando com Celery em Python
Deixa para depois, Procrastinando com Celery em PythonAdriano Petrich
 
Python magicmethods
Python magicmethodsPython magicmethods
Python magicmethodsdreampuf
 

Similar to Django Celery (20)

Testing My Patience
Testing My PatienceTesting My Patience
Testing My Patience
 
Celery
CeleryCelery
Celery
 
Background Jobs with Resque
Background Jobs with ResqueBackground Jobs with Resque
Background Jobs with Resque
 
Debugging: Rules And Tools - PHPTek 11 Version
Debugging: Rules And Tools - PHPTek 11 VersionDebugging: Rules And Tools - PHPTek 11 Version
Debugging: Rules And Tools - PHPTek 11 Version
 
Debugging: Rules & Tools
Debugging: Rules & ToolsDebugging: Rules & Tools
Debugging: Rules & Tools
 
Threads, Queues, and More: Async Programming in iOS
Threads, Queues, and More: Async Programming in iOSThreads, Queues, and More: Async Programming in iOS
Threads, Queues, and More: Async Programming in iOS
 
How to fake_properly
How to fake_properlyHow to fake_properly
How to fake_properly
 
Google App Engine Developer - Day3
Google App Engine Developer - Day3Google App Engine Developer - Day3
Google App Engine Developer - Day3
 
Introduction to Protractor
Introduction to ProtractorIntroduction to Protractor
Introduction to Protractor
 
Akka Cluster in Java - JCConf 2015
Akka Cluster in Java - JCConf 2015Akka Cluster in Java - JCConf 2015
Akka Cluster in Java - JCConf 2015
 
Curator intro
Curator introCurator intro
Curator intro
 
Ten useful JavaScript tips & best practices
Ten useful JavaScript tips & best practicesTen useful JavaScript tips & best practices
Ten useful JavaScript tips & best practices
 
Automation with Ansible and Containers
Automation with Ansible and ContainersAutomation with Ansible and Containers
Automation with Ansible and Containers
 
DjangoCon US 2011 - Monkeying around at New Relic
DjangoCon US 2011 - Monkeying around at New RelicDjangoCon US 2011 - Monkeying around at New Relic
DjangoCon US 2011 - Monkeying around at New Relic
 
Djangocon11: Monkeying around at New Relic
Djangocon11: Monkeying around at New RelicDjangocon11: Monkeying around at New Relic
Djangocon11: Monkeying around at New Relic
 
Тестирование и Django
Тестирование и DjangoТестирование и Django
Тестирование и Django
 
Deixa para depois, Procrastinando com Celery em Python
Deixa para depois, Procrastinando com Celery em PythonDeixa para depois, Procrastinando com Celery em Python
Deixa para depois, Procrastinando com Celery em Python
 
Queue your work
Queue your workQueue your work
Queue your work
 
DevOps with Fabric
DevOps with FabricDevOps with Fabric
DevOps with Fabric
 
Python magicmethods
Python magicmethodsPython magicmethods
Python magicmethods
 

Recently uploaded

Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...FIDO Alliance
 
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...WSO2
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuidePixlogix Infotech
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxFIDO Alliance
 
Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Zilliz
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)Samir Dash
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Paige Cruz
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Victor Rentea
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMKumar Satyam
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsLeah Henrickson
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityWSO2
 
Navigating Identity and Access Management in the Modern Enterprise
Navigating Identity and Access Management in the Modern EnterpriseNavigating Identity and Access Management in the Modern Enterprise
Navigating Identity and Access Management in the Modern EnterpriseWSO2
 
Quantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingQuantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingWSO2
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityVictorSzoltysek
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewDianaGray10
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Victor Rentea
 
AI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAnitaRaj43
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxFIDO Alliance
 

Recently uploaded (20)

Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
Navigating Identity and Access Management in the Modern Enterprise
Navigating Identity and Access Management in the Modern EnterpriseNavigating Identity and Access Management in the Modern Enterprise
Navigating Identity and Access Management in the Modern Enterprise
 
Quantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingQuantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation Computing
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps Productivity
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering Developers
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
AI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by Anitaraj
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 

Django Celery

  • 5. what is celery? "celery is an open source asynchronous task queue/ job queue based on distributed message I. It is focused
  • 6. what is celery? "celery is an open source asynchronous task queue/ job queue based on distributed message passing. It is but what does that
  • 7. you can run • distributed • concurrently • in the background
  • 8. use cases • external api calls (twitter) • long running tasks (transcoding) • concurrent execution (batch image resize) • load balancing (across servers)
  • 9. sql django celery ORM carro AMQP SQL or STOMP
  • 10. result • database • AMQP • cache • tokyo tyrant • redis • mongodb
  • 11. components 1. views.py / management command 2. broker – RabbitMQ 3. workers
  • 14.
  • 15. setting up the $ sudo apt-get install rabbitmq- server $ sudo pip install celery $ rabbitmqctl add_user myuser mypassword $ rabbitmqctl add_vhost myvhost $ rabbitmqctl set_permissions -p
  • 16. setup INSTALLED_APPS += ("djcelery", ) settings.p BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = “myuser" BROKER_PASSWORD = “mypassword" BROKER_VHOST = “myvhost" CELERY_QUEUES = {     "regular_tasks": {         "binding_key": "task.#",     },     "twitter_tasks": {         "binding_key": "twitter.#",     },     "feed_tasks": {         "binding_key": "feed.#",     }, } $ python manage.py celeryd -B
  • 17. hello from celery.decorators import task tasks.p @task def add(x, y): return x + y >>> result = add.delay(4, 4) >>> result.wait() # wait 8
  • 18. post to from celery.task import Task tasks.p class UpdateStatus(Task):     name = "twitter.updatestatus"     routing_key = 'twitter.updatestatus'     ignore_result = True              def run(self, tweet, **kwargs):         post_to_twitter(tweet)          from twitter.tasks import UpdateStatus UpdateStatus.delay(tweet=‘hello world’) views.p
  • 19. retry / rate from celery.task import Task tasks.p class UpdateStatus(Task):     name = "twitter.updatestatus"     routing_key = 'twitter.updatestatus'     ignore_result = True     default_retry_delay = 5 * 60     max_retries = 12 # 1 hour retry rate_limit = ‘10/s’          def run(self, tweet, **kwargs):         try: post_to_twitter(tweet)         except Exception, exc:             # If twitter crashes retry             self.retry([tweet,], kwargs, exc=exc) from twitter.tasks import UpdateStatus views.p UpdateStatus.delay(tweet=‘hello world’)
  • 20. podcast from celery.task import PeriodicTask tasks.p class FeedImportPeriodicTask(PeriodicTask):     run_every = timedelta(hours=1)     routing_key = 'feed.periodic_import'     def run(self, **kwargs):         logger = self.get_logger(**kwargs)         logger.info("Running Periodic Feed Import task!")         update_podcasts(silent=False)
  • 21. class FeedImporter(Task):     name = "feed.import"     routing_key = 'feed.import' tasks.p     ignore_result = True     default_retry_delay = 5 * 60 # retry in 5 minutes     max_retries = 72 # 6 Hours to cover major outages     def run(self, podcast_id, **kwargs):         try:             logger = self.get_logger(**kwargs)             # The cache key consists of the task name and the MD5 digest of the feed id.             lock_id = "%s-lock-%s" % (self.name, podcast_id)             is_locked = lambda: str(cache.get(lock_id)) == "true"             acquire_lock = lambda: cache.set(lock_id, "true", 300)             # memcache delete is very slow, so we'd rather set a false value             # with a very low expiry time.             release_lock = lambda: cache.set(lock_id, "nil", 1)                  logger.debug("Trying to import feed: %s" % podcast_id)             if is_locked():                 logger.debug("Feed %s is already being imported by another worker" % podcast_id)                 return             acquire_lock()             try:                 import_feed(logger, podcast_id)             finally:                 release_lock()         except Exception, exc:             logger.error(exc)
  • 22. typical • running out of disk space == rabbitmq fail • queue priorities, difficult • non-pickle-able errors • crashing consumers
  • 23. other cool • tasksets / callbacks • remote control tasks • abortable tasks • eta – run tasks at a set time • HttpDispatchTask • expiring tasks • celerymon • celeryev • ajax views
  • 24. finding • http://github.com/ask/celery • http://github.com/ask/django- celery • irc.freenode.net #celery (asksol owner, always helpful and about)