Manage operations with queues in Drupal
•
1 like•736 views
Session from Drupal Camp Lviv 2015.
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Viewers also liked (20)
Similar to Manage operations with queues in Drupal
Similar to Manage operations with queues in Drupal (20)
More from Yuriy Gerasimov
More from Yuriy Gerasimov (16)
Recently uploaded
Recently uploaded (20)
Manage operations with queues in Drupal
- 1. Delayed operations with queues Yuriy Gerasimov Frédéric G. Marand
- 2. Yuriy Gerasimov ygerasimov • FFW • Drupal architect & developer • Contrib 7 modules: services, draggableviews • Founder at Backtrac.io
- 3. Frédéric G. Marand fgm • OSInet: performance/architecture consulting • for internal teams at larger accounts • Core contributor 4.7 to 8.0.x, MongoDB + XMLRPC maintainer + others • Already 4 D8 customer projects before 8.0.0 • Customer D8 in production since 07/2015 • Frequently adds queueing to larger Drupal projects
- 4. Why use queues ? To have websites which are : • Faster for visitors • Snappier for editors • More scaleable To process time-consuming jobs : • Video encoding • High-resolution gallery uploads and processing
- 5. Concrete use cases • Prepare content for non-Drupal front-ends • Anticipate content generation • Deferred submits, e.g. comments handling • Slow operations: node saves, previews, image processing • External data sources: pull, push • Multi-step operations: batch
- 11. Image processing
- 12. Job servers • How to get results • Rerun failed jobs • Separate queue for failed jobs • Monitoring queues, workers • Supervisor
- 14. Queue API methods: Queue QueueInterface • Q::createItem(mixed $data: void • Q::claimItem($lease_time = 3600: mixed $item FALSE | stdClass + [item_id => int, data => mixed, created => timestamp] $lease_time → Assumptions for runner, currently not used • Q::deleteItem($item): void -> work done • Q::releaseItem($item): bool • Q::numberOfItems(): int → best guess, unreliable • Q::createQueue() / Q::deleteQueue() ReliableQueueInterface: ordering, single execution
- 15. Queue Runners Core / Contrib • Core Cron / Elysia Cron / Queue_Runner • Drush: queue-list / queue-run Similar limitations: • Default on in D6 / D7, default off in D8 • Single threaded, single process across queues Custom runners • Provided by queue modules or per-project one-offs • Preemption, parallel execution...
- 16. Queue API limitations • Exception handling • Priority management • Tagging • Delay, burying ... • No monitoring, peek(), deduplication
- 17. Questions ?