SlideShare a Scribd company logo
Advanced Task Management in Celery


           Mahendra M
           @mahendra
    https://github.com/mahendra
@mahendra
●   Python developer for 6 years
●   FOSS enthusiast/volunteer for 14 years
    ●   Bangalore LUG and Infosys LUG
    ●   FOSS.in and LinuxBangalore/200x
●   Celery user for 3 years
●   Contributions
    ●   patches, testing new releases
    ●   Zookeeper msg transport for kombu
    ●   Kafka support (in-progress)
Quick Intro to Celery
●   Asynchronous task/job queue
●   Uses distributed message passing
●   Tasks are run asynchronously on worker nodes
●   Results are passed back to the caller (if any)
Overview

                    Worker 1



                    Worker 2
Sender    Msg Q
                        .
                        .
                        .

                    Worker N
Sample Code
from celery.task import task


@task
def add(x, y):
   return x + y


result = add.delay(5,6)
result.get()
Uses of Celery
●   Asynchronous task processing
●   Handling long running / heavy jobs
    ●   Image resizing, video transcode, PDF generation
●   Offloading heavy web backend operations
●   Scheduling tasks to be run at a particular time
    ●   Cron for python
Advanced Uses
●   Task Routing
●   Task retries, timeout and revoking
●   Task Canvas – combining tasks
    ●   Task co-ordination
    ●   Dependencies
    ●   Task trees or graphs
    ●   Batch tasks
    ●   Progress monitoring
●   Tricks
    ●   DB conflict management
Sending tasks to a particular worker

                                  Worker 1
                                 (Windows)

                       windows
                                  Worker 2
                       windows   (Windows)
     Sender    Msg Q
                                     .
                        linux
                                     .
                                     .
                                 Worker N
                                  (Linux)
Routing tasks – Use cases
●   Priority execution
●   Based on hardware capabilities
    ●   Special cards available for video capture
    ●   Making use of GPUs (CUDA)
●   Based on OS (for eg. Playready encryption)
●   Based on location
    ●   Moving compute closer to data (Hadoop-ish)
    ●   Sending tasks to different data centers
●   Sequencing operations (CouchDB conflicts)
Sample Code
from celery.task import task


@task(queue = 'windows')
def drm_encrypt(audio_file, key_phrase):
   ...


r = drm_encrypt.apply_async( args = [afile, key],
                               queue = 'windows' )


#Start celery worker with queues options
$ celery worker -Q windows
Retrying tasks
@task( default_retry_delay = 60,
      max_retries = 3 )
def drm_encrypt(audio_file, key_phrase):
   try:
       playready.encrypt(...)
   except Exception, exc:
       raise drm_encrypt.retry(exc=exc, countdown=5)
Retrying tasks
●   You can specify the number of times a task can
    be retried.
●   The cases for retrying a task must be handled
    within code. Celery will not do it automatically
●   The tasks should be designed to be idempotent
Handling worker failures
@task( acks_late = True )
def drm_encrypt(audio_file, key_phrase):
     try:
          playready.encrypt(...)
     except Exception, exc:
          raise drm_encrypt.retry(exc=exc, countdown=5)



●   This is used where the task must be resend in case of
    worker or node failure
●   The ack message to the message queue is sent after the
    task finishes executing
Worker processes

                                 Worker 1
                                (Windows)

                      windows
                                 Worker 2
                      windows   (Windows)
Sender        Msg Q
                                    .
                       linux
                                    .
                                    .
                                Worker N
                                 (Linux)
                                    Process 1
                                    Process 2

                                    Process N
Worker processes

                                 Worker 1
                                (Windows)

                      windows
                                 Worker 2
                      windows   (Windows)
Sender        Msg Q
                                    .
                       linux
                                    .
                                    .
                                Worker N
                                 (Linux)
                                    Process 1
                                    Process 2

                                    Process N
Worker process
●   In every worker node, celery starts a pool of
    worker processes
●   The number is determined by the concurrency
    setting (or autodetected – for full CPU usage)
●   Each processes can be configured to restart
    after running x number of tasks
    ●   Disabled by default
●   Alternately eventlet can be used instead of
    processes (discuss later)
Revoking tasks
celery.control.revoke( task_id,
                        terminate = False,
                        signal = 'SIGKILL' )
●
    revoke() works by sending a broadcast
    message to all workers
●   If a task has not yet run, workers will keep this
    task_id in memory and ensure that it does not
    run
●   If a task is running, revoke() will not work
    unless terminate = True
Task expiration
task.apply_async( expires = x )
        x can be
        * in seconds
        * a specific datetime()


●   Global time limits can be configured in settings
    ●   Soft time limit – the task receives an exception
        which can be used to cleanup
    ●   Hard time limit – the worker running the task is
        killed and is replaced with another one.
Handling soft time limit
@task()
def drm_encrypt(audio_file, key_phrase):
   Try:
          setup_tmp_files()
           SoftTimeLimitExceeded:

          playready.encrypt(...)
   except SoftTimeLimitExceeded:
          cleanup_tmp_files()
   except Exception, exc:
          raise drm_encrypt.retry(exc=exc, countdown=5)
Task Canvas
●   Chains – Linking one task to another
●   Groups – Execute several tasks in parallel
●   Chord – execute a task after a set of tasks has
    finished
●   Map and starmap – Similar to map() function
●   Chunks – divide an iterable of work into chunks
●   Chunks + Chord/chain can be used for map-
    reduce
                Best shown in a demo
Task trees

[ task 1 ] --- spawns --- [ task 2 ] ---- spawns -->   [ task 2_1 ]
                  |                                    [ task 2_3 ]
                  |
                  +------ [ task 3 ] ---- spawns -->   [ task 3_1 ]
                  |                                    [ task 3_2 ]
                  |
                  +------ [ task 4 ] ---- links ---> [ task 5 ]
                                                         |(spawns)
                                                         |
                                                         |
                          [ task 8 ] <--- links <--- [ task 6 ]
                                                         |(spawns)
                                                     [ task 7 ]
Task Trees
●   Home grown solution (our current approach)
    ●   Use db models and keep track of trees
●   Better approach
    ●   Use celery-tasktree
    ●   http://pypi.python.org/pypi/celery-tasktree
Celery Batches
●   Collect jobs and execute it in a batch.
●   Can be used for stats collection
●   Batch execution is done once
    ●   a configured timeout is reached OR
    ●   a configured number of tasks have been received
●   Useful for reducing n/w and db loads
Celery Batches
from celery.contrib.batches import Batches
@task( base=Batches, flush_every=50, flush_interval=10 )
def collect_stats( requests ):
   items = {}
   for request in requests:
       item_id = request.kwargs['item_id']
       items[ item_id ] = get_obj( item_id )
       items[ item_id ].count += 1
   # Sync to db


collect_stats.delay( item_id = 45 )
collect_stats.delay( item_id = 57 )
Celery monitoring
●   Celery Flower
    https://github.com/mher/flower
●   Django admin monitor
●   Celery jobstatic
    http://pypi.python.org/pypi/jobtastic
Celery deployment
●   Cyme – celery instance manager
    https://github.com/celery/cyme
●   Celery autoscaling
●   Use celery eventlet where required

More Related Content

What's hot

Introduction to Celery
Introduction to CeleryIntroduction to Celery
Introduction to Celery
Chathuranga Bandara
 
Data processing with celery and rabbit mq
Data processing with celery and rabbit mqData processing with celery and rabbit mq
Data processing with celery and rabbit mq
Jeff Peck
 
Building Distributed System with Celery on Docker Swarm
Building Distributed System with Celery on Docker SwarmBuilding Distributed System with Celery on Docker Swarm
Building Distributed System with Celery on Docker Swarm
Wei Lin
 
Flask Introduction - Python Meetup
Flask Introduction - Python MeetupFlask Introduction - Python Meetup
Flask Introduction - Python Meetup
Areski Belaid
 
Europython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & CeleryEuropython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & CeleryMauro Rocco
 
2017 Pycon KR - Django/AWS 를 이용한 쇼핑몰 서비스 구축
2017 Pycon KR - Django/AWS 를 이용한 쇼핑몰 서비스 구축2017 Pycon KR - Django/AWS 를 이용한 쇼핑몰 서비스 구축
2017 Pycon KR - Django/AWS 를 이용한 쇼핑몰 서비스 구축
Youngil Cho
 
Celery - A Distributed Task Queue
Celery - A Distributed Task QueueCelery - A Distributed Task Queue
Celery - A Distributed Task Queue
Duy Do
 
Airflow를 이용한 데이터 Workflow 관리
Airflow를 이용한  데이터 Workflow 관리Airflow를 이용한  데이터 Workflow 관리
Airflow를 이용한 데이터 Workflow 관리
YoungHeon (Roy) Kim
 
Rust system programming language
Rust system programming languageRust system programming language
Rust system programming language
robin_sy
 
Drools
DroolsDrools
Drools
John Paulett
 
Formation jpa-hibernate-spring-data
Formation jpa-hibernate-spring-dataFormation jpa-hibernate-spring-data
Formation jpa-hibernate-spring-data
Lhouceine OUHAMZA
 
Spring boot
Spring bootSpring boot
Spring boot
sdeeg
 
Why Task Queues - ComoRichWeb
Why Task Queues - ComoRichWebWhy Task Queues - ComoRichWeb
Why Task Queues - ComoRichWeb
Bryan Helmig
 
The Rust Programming Language
The Rust Programming LanguageThe Rust Programming Language
The Rust Programming Language
Mario Alexandro Santini
 
(참고) Elk stack 설치 및 kafka
(참고) Elk stack 설치 및 kafka(참고) Elk stack 설치 및 kafka
(참고) Elk stack 설치 및 kafka
NoahKIM36
 
Deep dive into Coroutines on JVM @ KotlinConf 2017
Deep dive into Coroutines on JVM @ KotlinConf 2017Deep dive into Coroutines on JVM @ KotlinConf 2017
Deep dive into Coroutines on JVM @ KotlinConf 2017
Roman Elizarov
 
Database connectivity in python
Database connectivity in pythonDatabase connectivity in python
Database connectivity in python
baabtra.com - No. 1 supplier of quality freshers
 
Java 8, Streams & Collectors, patterns, performances and parallelization
Java 8, Streams & Collectors, patterns, performances and parallelizationJava 8, Streams & Collectors, patterns, performances and parallelization
Java 8, Streams & Collectors, patterns, performances and parallelization
José Paumard
 
Asynchronous JS in Odoo
Asynchronous JS in OdooAsynchronous JS in Odoo
Asynchronous JS in Odoo
Odoo
 
Asynchronous JavaScript Programming with Callbacks & Promises
Asynchronous JavaScript Programming with Callbacks & PromisesAsynchronous JavaScript Programming with Callbacks & Promises
Asynchronous JavaScript Programming with Callbacks & Promises
Hùng Nguyễn Huy
 

What's hot (20)

Introduction to Celery
Introduction to CeleryIntroduction to Celery
Introduction to Celery
 
Data processing with celery and rabbit mq
Data processing with celery and rabbit mqData processing with celery and rabbit mq
Data processing with celery and rabbit mq
 
Building Distributed System with Celery on Docker Swarm
Building Distributed System with Celery on Docker SwarmBuilding Distributed System with Celery on Docker Swarm
Building Distributed System with Celery on Docker Swarm
 
Flask Introduction - Python Meetup
Flask Introduction - Python MeetupFlask Introduction - Python Meetup
Flask Introduction - Python Meetup
 
Europython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & CeleryEuropython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & Celery
 
2017 Pycon KR - Django/AWS 를 이용한 쇼핑몰 서비스 구축
2017 Pycon KR - Django/AWS 를 이용한 쇼핑몰 서비스 구축2017 Pycon KR - Django/AWS 를 이용한 쇼핑몰 서비스 구축
2017 Pycon KR - Django/AWS 를 이용한 쇼핑몰 서비스 구축
 
Celery - A Distributed Task Queue
Celery - A Distributed Task QueueCelery - A Distributed Task Queue
Celery - A Distributed Task Queue
 
Airflow를 이용한 데이터 Workflow 관리
Airflow를 이용한  데이터 Workflow 관리Airflow를 이용한  데이터 Workflow 관리
Airflow를 이용한 데이터 Workflow 관리
 
Rust system programming language
Rust system programming languageRust system programming language
Rust system programming language
 
Drools
DroolsDrools
Drools
 
Formation jpa-hibernate-spring-data
Formation jpa-hibernate-spring-dataFormation jpa-hibernate-spring-data
Formation jpa-hibernate-spring-data
 
Spring boot
Spring bootSpring boot
Spring boot
 
Why Task Queues - ComoRichWeb
Why Task Queues - ComoRichWebWhy Task Queues - ComoRichWeb
Why Task Queues - ComoRichWeb
 
The Rust Programming Language
The Rust Programming LanguageThe Rust Programming Language
The Rust Programming Language
 
(참고) Elk stack 설치 및 kafka
(참고) Elk stack 설치 및 kafka(참고) Elk stack 설치 및 kafka
(참고) Elk stack 설치 및 kafka
 
Deep dive into Coroutines on JVM @ KotlinConf 2017
Deep dive into Coroutines on JVM @ KotlinConf 2017Deep dive into Coroutines on JVM @ KotlinConf 2017
Deep dive into Coroutines on JVM @ KotlinConf 2017
 
Database connectivity in python
Database connectivity in pythonDatabase connectivity in python
Database connectivity in python
 
Java 8, Streams & Collectors, patterns, performances and parallelization
Java 8, Streams & Collectors, patterns, performances and parallelizationJava 8, Streams & Collectors, patterns, performances and parallelization
Java 8, Streams & Collectors, patterns, performances and parallelization
 
Asynchronous JS in Odoo
Asynchronous JS in OdooAsynchronous JS in Odoo
Asynchronous JS in Odoo
 
Asynchronous JavaScript Programming with Callbacks & Promises
Asynchronous JavaScript Programming with Callbacks & PromisesAsynchronous JavaScript Programming with Callbacks & Promises
Asynchronous JavaScript Programming with Callbacks & Promises
 

Similar to Advanced task management with Celery

SWT Tech Sharing: Node.js + Redis
SWT Tech Sharing: Node.js + RedisSWT Tech Sharing: Node.js + Redis
SWT Tech Sharing: Node.js + Redis
Infinity Levels Studio
 
Internals
InternalsInternals
Internals
Sandeep Purohit
 
Apache Spark Internals
Apache Spark InternalsApache Spark Internals
Apache Spark Internals
Knoldus Inc.
 
Celery
CeleryCelery
Concurrency in Swift
Concurrency in SwiftConcurrency in Swift
Concurrency in Swift
Seven Peaks Speaks
 
Trevor McDonald - Nagios XI Under The Hood
Trevor McDonald  - Nagios XI Under The HoodTrevor McDonald  - Nagios XI Under The Hood
Trevor McDonald - Nagios XI Under The Hood
Nagios
 
Dragoncraft Architectural Overview
Dragoncraft Architectural OverviewDragoncraft Architectural Overview
Dragoncraft Architectural Overview
jessesanford
 
Robust C++ Task Systems Through Compile-time Checks
Robust C++ Task Systems Through Compile-time ChecksRobust C++ Task Systems Through Compile-time Checks
Robust C++ Task Systems Through Compile-time Checks
Stoyan Nikolov
 
Celery introduction
Celery introductionCelery introduction
Celery introduction
Ionel Mărieș Cristian
 
Batch Processing with Amazon EC2 Container Service
Batch Processing with Amazon EC2 Container ServiceBatch Processing with Amazon EC2 Container Service
Batch Processing with Amazon EC2 Container Service
Amazon Web Services
 
Async and parallel patterns and application design - TechDays2013 NL
Async and parallel patterns and application design - TechDays2013 NLAsync and parallel patterns and application design - TechDays2013 NL
Async and parallel patterns and application design - TechDays2013 NL
Arie Leeuwesteijn
 
Troubleshooting MySQL from a MySQL Developer Perspective
Troubleshooting MySQL from a MySQL Developer PerspectiveTroubleshooting MySQL from a MySQL Developer Perspective
Troubleshooting MySQL from a MySQL Developer Perspective
Marcelo Altmann
 
NovaProva, a new generation unit test framework for C programs
NovaProva, a new generation unit test framework for C programsNovaProva, a new generation unit test framework for C programs
NovaProva, a new generation unit test framework for C programs
Greg Banks
 
Scaling Django with gevent
Scaling Django with geventScaling Django with gevent
Scaling Django with geventMahendra M
 
Kernel Recipes 2018 - KernelShark 1.0; What's new and what's coming - Steven ...
Kernel Recipes 2018 - KernelShark 1.0; What's new and what's coming - Steven ...Kernel Recipes 2018 - KernelShark 1.0; What's new and what's coming - Steven ...
Kernel Recipes 2018 - KernelShark 1.0; What's new and what's coming - Steven ...
Anne Nicolas
 
Programming with Threads in Java
Programming with Threads in JavaProgramming with Threads in Java
Programming with Threads in Javakoji lin
 
Introduction to map reduce
Introduction to map reduceIntroduction to map reduce
Introduction to map reduce
M Baddar
 
Yevhen Tatarynov "From POC to High-Performance .NET applications"
Yevhen Tatarynov "From POC to High-Performance .NET applications"Yevhen Tatarynov "From POC to High-Performance .NET applications"
Yevhen Tatarynov "From POC to High-Performance .NET applications"
LogeekNightUkraine
 
Numpy Meetup 07/02/2013
Numpy Meetup 07/02/2013Numpy Meetup 07/02/2013
Numpy Meetup 07/02/2013Francesco
 

Similar to Advanced task management with Celery (20)

SWT Tech Sharing: Node.js + Redis
SWT Tech Sharing: Node.js + RedisSWT Tech Sharing: Node.js + Redis
SWT Tech Sharing: Node.js + Redis
 
Internals
InternalsInternals
Internals
 
Apache Spark Internals
Apache Spark InternalsApache Spark Internals
Apache Spark Internals
 
internals
internalsinternals
internals
 
Celery
CeleryCelery
Celery
 
Concurrency in Swift
Concurrency in SwiftConcurrency in Swift
Concurrency in Swift
 
Trevor McDonald - Nagios XI Under The Hood
Trevor McDonald  - Nagios XI Under The HoodTrevor McDonald  - Nagios XI Under The Hood
Trevor McDonald - Nagios XI Under The Hood
 
Dragoncraft Architectural Overview
Dragoncraft Architectural OverviewDragoncraft Architectural Overview
Dragoncraft Architectural Overview
 
Robust C++ Task Systems Through Compile-time Checks
Robust C++ Task Systems Through Compile-time ChecksRobust C++ Task Systems Through Compile-time Checks
Robust C++ Task Systems Through Compile-time Checks
 
Celery introduction
Celery introductionCelery introduction
Celery introduction
 
Batch Processing with Amazon EC2 Container Service
Batch Processing with Amazon EC2 Container ServiceBatch Processing with Amazon EC2 Container Service
Batch Processing with Amazon EC2 Container Service
 
Async and parallel patterns and application design - TechDays2013 NL
Async and parallel patterns and application design - TechDays2013 NLAsync and parallel patterns and application design - TechDays2013 NL
Async and parallel patterns and application design - TechDays2013 NL
 
Troubleshooting MySQL from a MySQL Developer Perspective
Troubleshooting MySQL from a MySQL Developer PerspectiveTroubleshooting MySQL from a MySQL Developer Perspective
Troubleshooting MySQL from a MySQL Developer Perspective
 
NovaProva, a new generation unit test framework for C programs
NovaProva, a new generation unit test framework for C programsNovaProva, a new generation unit test framework for C programs
NovaProva, a new generation unit test framework for C programs
 
Scaling Django with gevent
Scaling Django with geventScaling Django with gevent
Scaling Django with gevent
 
Kernel Recipes 2018 - KernelShark 1.0; What's new and what's coming - Steven ...
Kernel Recipes 2018 - KernelShark 1.0; What's new and what's coming - Steven ...Kernel Recipes 2018 - KernelShark 1.0; What's new and what's coming - Steven ...
Kernel Recipes 2018 - KernelShark 1.0; What's new and what's coming - Steven ...
 
Programming with Threads in Java
Programming with Threads in JavaProgramming with Threads in Java
Programming with Threads in Java
 
Introduction to map reduce
Introduction to map reduceIntroduction to map reduce
Introduction to map reduce
 
Yevhen Tatarynov "From POC to High-Performance .NET applications"
Yevhen Tatarynov "From POC to High-Performance .NET applications"Yevhen Tatarynov "From POC to High-Performance .NET applications"
Yevhen Tatarynov "From POC to High-Performance .NET applications"
 
Numpy Meetup 07/02/2013
Numpy Meetup 07/02/2013Numpy Meetup 07/02/2013
Numpy Meetup 07/02/2013
 

Recently uploaded

Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
Sri Ambati
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
OnBoard
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
Abida Shariff
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
DianaGray10
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
Ralf Eggert
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
Prayukth K V
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
Cheryl Hung
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Inflectra
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
BookNet Canada
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
Laura Byrne
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
Thijs Feryn
 

Recently uploaded (20)

Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)PHP Frameworks: I want to break free (IPC Berlin 2024)
PHP Frameworks: I want to break free (IPC Berlin 2024)
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
 
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and SalesThe Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
 

Advanced task management with Celery

  • 1. Advanced Task Management in Celery Mahendra M @mahendra https://github.com/mahendra
  • 2. @mahendra ● Python developer for 6 years ● FOSS enthusiast/volunteer for 14 years ● Bangalore LUG and Infosys LUG ● FOSS.in and LinuxBangalore/200x ● Celery user for 3 years ● Contributions ● patches, testing new releases ● Zookeeper msg transport for kombu ● Kafka support (in-progress)
  • 3. Quick Intro to Celery ● Asynchronous task/job queue ● Uses distributed message passing ● Tasks are run asynchronously on worker nodes ● Results are passed back to the caller (if any)
  • 4. Overview Worker 1 Worker 2 Sender Msg Q . . . Worker N
  • 5. Sample Code from celery.task import task @task def add(x, y): return x + y result = add.delay(5,6) result.get()
  • 6. Uses of Celery ● Asynchronous task processing ● Handling long running / heavy jobs ● Image resizing, video transcode, PDF generation ● Offloading heavy web backend operations ● Scheduling tasks to be run at a particular time ● Cron for python
  • 7. Advanced Uses ● Task Routing ● Task retries, timeout and revoking ● Task Canvas – combining tasks ● Task co-ordination ● Dependencies ● Task trees or graphs ● Batch tasks ● Progress monitoring ● Tricks ● DB conflict management
  • 8. Sending tasks to a particular worker Worker 1 (Windows) windows Worker 2 windows (Windows) Sender Msg Q . linux . . Worker N (Linux)
  • 9. Routing tasks – Use cases ● Priority execution ● Based on hardware capabilities ● Special cards available for video capture ● Making use of GPUs (CUDA) ● Based on OS (for eg. Playready encryption) ● Based on location ● Moving compute closer to data (Hadoop-ish) ● Sending tasks to different data centers ● Sequencing operations (CouchDB conflicts)
  • 10. Sample Code from celery.task import task @task(queue = 'windows') def drm_encrypt(audio_file, key_phrase): ... r = drm_encrypt.apply_async( args = [afile, key], queue = 'windows' ) #Start celery worker with queues options $ celery worker -Q windows
  • 11. Retrying tasks @task( default_retry_delay = 60, max_retries = 3 ) def drm_encrypt(audio_file, key_phrase): try: playready.encrypt(...) except Exception, exc: raise drm_encrypt.retry(exc=exc, countdown=5)
  • 12. Retrying tasks ● You can specify the number of times a task can be retried. ● The cases for retrying a task must be handled within code. Celery will not do it automatically ● The tasks should be designed to be idempotent
  • 13. Handling worker failures @task( acks_late = True ) def drm_encrypt(audio_file, key_phrase): try: playready.encrypt(...) except Exception, exc: raise drm_encrypt.retry(exc=exc, countdown=5) ● This is used where the task must be resend in case of worker or node failure ● The ack message to the message queue is sent after the task finishes executing
  • 14. Worker processes Worker 1 (Windows) windows Worker 2 windows (Windows) Sender Msg Q . linux . . Worker N (Linux) Process 1 Process 2 Process N
  • 15. Worker processes Worker 1 (Windows) windows Worker 2 windows (Windows) Sender Msg Q . linux . . Worker N (Linux) Process 1 Process 2 Process N
  • 16. Worker process ● In every worker node, celery starts a pool of worker processes ● The number is determined by the concurrency setting (or autodetected – for full CPU usage) ● Each processes can be configured to restart after running x number of tasks ● Disabled by default ● Alternately eventlet can be used instead of processes (discuss later)
  • 17. Revoking tasks celery.control.revoke( task_id, terminate = False, signal = 'SIGKILL' ) ● revoke() works by sending a broadcast message to all workers ● If a task has not yet run, workers will keep this task_id in memory and ensure that it does not run ● If a task is running, revoke() will not work unless terminate = True
  • 18. Task expiration task.apply_async( expires = x ) x can be * in seconds * a specific datetime() ● Global time limits can be configured in settings ● Soft time limit – the task receives an exception which can be used to cleanup ● Hard time limit – the worker running the task is killed and is replaced with another one.
  • 19. Handling soft time limit @task() def drm_encrypt(audio_file, key_phrase): Try: setup_tmp_files() SoftTimeLimitExceeded: playready.encrypt(...) except SoftTimeLimitExceeded: cleanup_tmp_files() except Exception, exc: raise drm_encrypt.retry(exc=exc, countdown=5)
  • 20. Task Canvas ● Chains – Linking one task to another ● Groups – Execute several tasks in parallel ● Chord – execute a task after a set of tasks has finished ● Map and starmap – Similar to map() function ● Chunks – divide an iterable of work into chunks ● Chunks + Chord/chain can be used for map- reduce Best shown in a demo
  • 21. Task trees [ task 1 ] --- spawns --- [ task 2 ] ---- spawns --> [ task 2_1 ] | [ task 2_3 ] | +------ [ task 3 ] ---- spawns --> [ task 3_1 ] | [ task 3_2 ] | +------ [ task 4 ] ---- links ---> [ task 5 ] |(spawns) | | [ task 8 ] <--- links <--- [ task 6 ] |(spawns) [ task 7 ]
  • 22. Task Trees ● Home grown solution (our current approach) ● Use db models and keep track of trees ● Better approach ● Use celery-tasktree ● http://pypi.python.org/pypi/celery-tasktree
  • 23. Celery Batches ● Collect jobs and execute it in a batch. ● Can be used for stats collection ● Batch execution is done once ● a configured timeout is reached OR ● a configured number of tasks have been received ● Useful for reducing n/w and db loads
  • 24. Celery Batches from celery.contrib.batches import Batches @task( base=Batches, flush_every=50, flush_interval=10 ) def collect_stats( requests ): items = {} for request in requests: item_id = request.kwargs['item_id'] items[ item_id ] = get_obj( item_id ) items[ item_id ].count += 1 # Sync to db collect_stats.delay( item_id = 45 ) collect_stats.delay( item_id = 57 )
  • 25. Celery monitoring ● Celery Flower https://github.com/mher/flower ● Django admin monitor ● Celery jobstatic http://pypi.python.org/pypi/jobtastic
  • 26. Celery deployment ● Cyme – celery instance manager https://github.com/celery/cyme ● Celery autoscaling ● Use celery eventlet where required