Celery is a distributed task queue system that allows putting tasks in a task queue to be processed asynchronously by worker processes or threads. It supports common features like events, workflows, rate limits, retries, and routing. Celery can use many different brokers like RabbitMQ, Redis, MongoDB, and SQL databases to store tasks and results. It is extensible and works well for asynchronous tasks like emails, image processing, data imports, and API calls.

1. Celery
Distributed task queue system

2. What is celery
● it’s a distributed task queue system
○ abstractions to put tasks in the task queue
○ abstractions to manage task results
○ services to run tasks from the task queue
○ monitoring tools
● can use almost any database or message
queue
● works best with a task queue but can use
other non-traditional transports like a sql db
or redis

3. Possible uses
● asynchronous processing
○ emails
○ image processing
○ transcoding
○ batch import / processing
○ that slow api you need to use
● scrapping
● processing large amount of data

4. What is celery
task queue
(“broker”)
result
backend
worker 1
worker 2
worker 1
worker 2
machine 1
machine 2
webserver
resize
picture

5. Usage

6. Usage
……

7. Configuration

8. Notable features
● events
● workflow
● rate limits
● time limits
● retry
● acks (reliable execution)
● routing
● task schedulers (like cron)
● various worker pools: process (fork),
threaded, eventlet, gevent
● extensible

9. Brokers & backends overview
● rabbitmq
● redis
● mongodb
● sql (sqlalchemy or django)
● beanstalk

10. The canvas
Workflow primitives:
● group
○ apply tasks in parallel
● chain
○ apply tasks in sequence
● chord
○ apply task after group or task completes
● map, starmap, chunks
○ abstractions over group

11. Rate limits
● can use “/h”, “/m”

12. Revokes & time limits
●
●

13. Similar projects & alternatives
● Kuyruk
● huey
● pyres
● rq