This document provides an overview of Celery, an asynchronous task queue/job queue based on distributed message passing. It discusses offline execution vs online, installing and configuring Celery, creating simple tasks, integrating with Django, deployment considerations, and how Celery is used at Aviso. The agenda includes an introduction to Celery, installing it, creating simple tasks, using Celery Canvas to divide work, and deployment best practices.

1. VEBSnet2
Celery Workshop
A Gentle Introduction
Eswar Vandanapu
Chief Technology Architect

2. VEBSnet2
Objective
At the end of the session you can expect to
understand modes of off-line execution, what is
celery and how to use it with Django, and
considerations around deployment.
This is HANDS ON session. We will spend time
on getting it working for everybody in
attendance.

3. VEBSnet2
Agenda
• Types of offline Execution
• Why Celery
• Installing Celery
• A simple celery task
• Integrating with Django Framework
• Celery Canvas
• Deployment Considerations
• How is Celery used in Aviso

4. VEBSnet2
Types of Execution
• Online
• Offline
– Daily/Monthly Processing
– Event Driven
– Batch Processing

5. VEBSnet2
Why Offline?
• Resource Constraints
• Long Running Tasks
• Bracing for Impact
• Serializing Access
• Windows of Opportunity
• Reaction to events

6. VEBSnet2
Installing Celery
• Requirements
– Python
– Message Broker
• Using MongoDB is simpler in windows
Create a Virtual Environment and
$ pip install celery
Run ‘celery’ to validate

7. VEBSnet2
First Steps
http://celery.readthedocs.org/en/latest/getting-
started/first-steps-with-celery.html

8. VEBSnet2
Inspect and Control
Add a sleep in your celery task for a minute of so
so that we can use the celery tools:
From Command Line
• celery inspect
• celery control
From Python Shell

9. VEBSnet2
Canvas – A way to divide the work
• A Naïve method is to create a master task,
that uses .get() on the results to orchestrate.
• It is better to leave the control with celery
though
• What is partial? Let’s try functools.partial
http://celery.readthedocs.org/en/latest/usergui
de/canvas.html

10. VEBSnet2
Canvas Primitives
map, startmap and chunks or basically for convenience.

11. VEBSnet2
Deployment
• Make it into a service
– In Unix we use Upstart
– Create a separate virtual environment and run as
non-sudoer account.
• Fail-over of message store is important
• Design the tasks to be reentrant
– Easier said than done 

12. VEBSnet2
At AVISO
• In use from last 3 years
• Custom web interface allows anyone with
permissions to launch worker pools for their jobs
• Hosted RabbitMQ for backend to reduce the
headaches of message queue clustering, security
etc.
• All logs harvested using Loggly
• Flower to monitor didn’t work at scale, but it is
good for debugging locally.
• Custom UI to track the tasks by customer,
submitter, pool used etc and to kill if necessary