Celery is a task queue that allows for asynchronous and distributed processing of long-running tasks. It uses a broker to add tasks to a queue where workers pick them up to process reliably and with retries. Origami Logic uses Celery to fetch marketing metrics from social networks daily in a periodic, reliable, distributed, and scalable way. A simple example shows adding two numbers asynchronously by defining a task, running a Celery worker, and triggering the task.
2. Who am I
● Os Shiowattana (@djshiow)
● Software engineer for ~10 years
● Currently at Origami Logic
○ We’re looking for great developers ;-)
● Python experience: 3 months
○ Teaching is a good way to learn
3. Outline
● What is Celery?
● What is it for?
● Simple Example
http://www.myhdiet.com/sharons-raw-celery-soup/
4. What is celery?
● Task queue
○ Distributed
○ Asynchronous
● Create tasks and
add it to the queue
● Workers pick them
up & work on them
5. How does it work?
Broker
Worker
Worker
Worker
….
Producer
Result
Store
6. What is it for?
● Long running asynchronous task
● Execute a task “reliably”
○ Asynchronous + Retries
○ e.g. Interactions with external APIs
● Scheduling Periodic tasks
7. How does Origami Logic use it?
● Fetch marketing metrics from all your social
network daily
○ Periodic
○ Reliable (with retries) / Traceable
○ Distributed (multiple machines) & Scalable
○ Rate limits