Celery is a task queue that allows for asynchronous and distributed processing of long-running tasks. It uses a broker to add tasks to a queue where workers pick them up to process reliably and with retries. Origami Logic uses Celery to fetch marketing metrics from social networks daily in a periodic, reliable, distributed, and scalable way. A simple example shows adding two numbers asynchronously by defining a task, running a Celery worker, and triggering the task.

1. By
Os Shiowattana (@djshiow)

2. Who am I
● Os Shiowattana (@djshiow)
● Software engineer for ~10 years
● Currently at Origami Logic
○ We’re looking for great developers ;-)
● Python experience: 3 months
○ Teaching is a good way to learn

3. Outline
● What is Celery?
● What is it for?
● Simple Example
http://www.myhdiet.com/sharons-raw-celery-soup/

4. What is celery?
● Task queue
○ Distributed
○ Asynchronous
● Create tasks and
add it to the queue
● Workers pick them
up & work on them

5. How does it work?
Broker
Worker
Worker
Worker
….
Producer
Result
Store

6. What is it for?
● Long running asynchronous task
● Execute a task “reliably”
○ Asynchronous + Retries
○ e.g. Interactions with external APIs
● Scheduling Periodic tasks

7. How does Origami Logic use it?
● Fetch marketing metrics from all your social
network daily
○ Periodic
○ Reliable (with retries) / Traceable
○ Distributed (multiple machines) & Scalable
○ Rate limits

8. Instagram
● Populate follower feed
>7 M followers
Source: http://blogs.vmware.com/vfabric/2013/04/how-instagram-feeds-work-celery-and-rabbitmq.html

9. Simple Example
● Adding two numbers asynchronously !!
● Setup:
○ Install RabbitMQ
○ Install Celery
From: http://celery.readthedocs.org/en/latest/getting-started/first-steps-with-celery.html

10. First, define Celery app & tasks
from celery import Celery
#setup app
app = Celery('tasks', backend='amqp' , broker='amqp://guest@localhost//')
#Define task
@app.task
def add(x, y):
return x + y
Then, run celery worker
celery -A tasks worker --loglevel=info
Decorator to create a task class

11. Lastly, trigger the task
from tasks import add
#run the add task asynchronously 3 seconds later
#passing (4,4) as param
add.apply_async((4, 4) , countdown=3)
[2015-03-15 14:34:10,397: INFO/MainProcess]Received task: tasks.add[b68b0109-7e99-473b-8991-1da8adaa50e8]
eta:[2015-03-15 14:34:13.385448+00:00]
[2015-03-15 14:34:14,836: INFO/MainProcess] Tasktasks.add[b68b0109-7e99-473b-8991-1da8adaa50e8] succeeded in
0.0209855769999s: 8

12. Reference
● http://www.slideshare.net/idangazit/an-introduction-to-celery
● http://blogs.vmware.com/vfabric/2013/04/how-instagram-feeds-work-
celery-and-rabbitmq.html
● http://celery.readthedocs.org/en/latest/