This document discusses using Celery, an asynchronous task queue, to build a distributed workflow for baking pies. It describes Celery's architecture and components like brokers, workers, tasks, and queues. It provides examples of defining tasks, building workflows with primitives like groups and chords, and routing tasks to different queues. The document also covers options for asynchronous and synchronous task execution, periodic tasks, concurrency models, and Celery signals.
7. Celery: Distributed Task Queue
Celery is an asynchronous task queue/job queue based on distributed message
passing. It is focused on real-time operation, but supports scheduling as well.
The execution units, called tasks, are executed concurrently on a single or more
worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute
asynchronously (in the background) or synchronously (wait until ready).
Celery is used in production systems to process millions of tasks a day.
8.
9. Ask Solem Hoel
https://github.com/ask
Principal Software Engineer at Robinhood
San Francisco, CA
Staff Engineer, RabbitMQ
VMware
май 2011 – май 2013 (2 года 1 месяц)
Open source and consulting work on Celery, RabbitMQ,
Celery Дата начала: май 2009
10. Celery
Kombu (Messaging library for Python) 41991 (22541) lines
Billiard (Python multiprocessing fork with improvements and bugfixes) 19191(13115)
Vine (promise, async, future) 2921
Celery 104296 (37495)
Total 168399(76072)
19. Tasks
from celery.app.task import Task
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
class GetIngredient(Task):
name = 'get_ingredients'
def run(self, ingredient_name):
logger.info('Getting {}'.format(ingredient_name))
l # код получения ингредиента
return ingredient
20. Workflow
from celery.app.task import Task
from celery import canvas, signature
class OrderPie(Task):
name = 'order_pie'
def run(self):
get_meat = signature('get_ingredients', args=('meat',))
...
dough_chord = canvas.chord([get_eggs, get_milk, get_flour], create_dough)
components_group = canvas.group(dough_chord, get_meat)
return (components_group | seal_pie | bake_pie).delay()
21. Client side
>>> from tasks.order_pie import OrderPie
>>> r = OrderPie().delay()
>>> r.id
'58c5bb47-8fb3-4d4a-b2f5-8899520b5179'
>>> r.ready()
True
>>> r.get()
[['091f9ae7-561a-4781-a4d9-47bbcb121360', [['0865f66b-b89d-4ff3-a272-1f6b01d0a11f', None],
None]], None]
>>> r.backend.get_task_meta(r.get()[0][0])
{'task_id': '091f9ae7-561a-4781-a4d9-47bbcb121360', 'children': [], 'traceback': None, 'result':
'baked pie with meat', 'status': 'SUCCESS'}
25. Gevent
gevent is a coroutine -based Python networking library that uses greenlet to
provide a high-level synchronous API on top of the libev event loop
gevent.monkey – Make the standard library cooperative
def _patch_gevent():
from gevent import monkey, signal as gsignal, version_info
monkey.patch_all()
26. Polling
def run(self, *ingredients):
logger.info('Ingredients: {}'.format(ingredients))
id = create_dough(ingredients) # Отправка запроса на приготовление теста
while True:
time.sleep(polling_timeout)
if ready(id): # Проверяем готово ли тесто
dough = get_dough(id) # Если готово, то забираем
return dough
27. Calling Tasks
apply
Execute this task locally, by blocking until the task returns.
apply_async
Apply tasks asynchronously by sending a message.
delay
Shortcut to send a task message, but does not support execution options.
retry
Retry the task.
28. Options
Linking (callbacks/errbacks)
link
link_error
ETA and countdown
countdown
eta
Expiration
expires
Retry
retry=True,
retry_policy={
'max_retries': 3,
'interval_start': 0,
'interval_step': 0.2,
'interval_max': 0.2,
})
Serializers
Compression
Routing options
29. Callbacks
options = {
'link': app.signature('seal_pie'),
'link_error': app.signature('order_error')
}
for sub_task in options.values():
sub_task.set(
**app.amqp.router.route(sub_task.options, sub_task.task, sub_task.args,
sub_task.kwargs)
)
