Drupal 7 Queues

Drupal Queues New to Drupal 7 First in first out data structure Used internally by Drupal Uses OO principles Fully customisable

Why Use Queues? Batch processing of large amounts of data/items Delay processing of complex calculations so system load is more stable Sequential processing of items Preventing API service black listing

Drupal 7 Implementations Batch API Cron Aggregator Module Update Module

Drupal Queues Found in the file: /modules/system/system.queue.inc Classes are used to wrap queue functionality Reliable vs Non-reliable

Reliable Generally kept in database table Every item will be executed at least once Items exist over several requests If the request fails then queue remains intact

Non-reliable Generally kept in memory only All items might exist in a single request No guarantee that all queue items will be executed No guarantee that all items will be executed in order If request fails then queue might be lost

Drupal Queue Classes

DrupalQueue Single static method get() DrupalQueue::get('my_queue'); Returns a queue object of a given name and a given type (reliable or non-reliable) Force reliable queue by passing TRUE as second parameter Default returned class is SystemQueue Essentially a queue object factory

SystemQueue Example of a reliable queue class Implements DrupalReliableQueueInterface Uses the database table queue to store and retrieve the queue Items are 'leased' to ensure no two processes get the same queue item Default class for new queues

queue Table Created at Drupal install

MemoryQueue Example of an non-reliable queue class All queue items are stored in memory $queue parameter in class contains the queue Also implements item leasing

System Variables Three system variables are used by DrupalQueue to decide what sort of object to return 'queue_class_' . $name Default is NULL queue_default_class Default is SystemQueue queue_default_reliable_class Default is SystemQueue

Using Drupal Queues

Create A Queue $queue = DrupalQueue::get('my_queue', TRUE); $item = array( 'dataitem1' => 'something', 'int' => 123 ); $queue->createItem($item); echo $queue->numberOfItems(); // 1

Create A MemoryQueue variable_set('queue_default_class', 'MemoryQueue'); $queue = DrupalQueue::get('my_queue'); $item = array( 'dataitem1' => 'something', 'int' => 123 ); $queue->createItem($item); echo $queue->numberOfItems(); // 1

Get Item From Queue $queue = DrupalQueue::get('my_queue'); $got_item = $queue->claimItem(); echo $got_item->data['dataitem1'];

Retrieved Item Structure stdClass Object ( [data] => Array ( [dataitem1] => something [qwe] => 123 ) [item_id] => 89 )

Change Lease Length Can pass the lease time (in seconds) to claimItem() $got_item = $queue->claimItem( 100 );

Release Or Delete releaseItem() resets the lease time $queue->releaseItem($got_item); deleteItem() removes item from queue $queue->deleteItem($got_item);

Customizing Create custom queue class Create variable called “queue_class_” . $name Value is the class name of your queue class Call DrupalQueue::get() with the variable $name as a string

Customizing Reliable class MUST implement DrupalReliableQueueInterface Can also extend SystemQueue All Queue classes should at least implement DrupalQueueInterface

Using Custom Classes variable_set('queue_class_mycustom', 'MyCustomQueueClass'); $queue = DrupalQueue::get('mycustom');

Stack Class Last in first out implementation class Stack extends SystemQueue { public function claimItem($lease_time = 30) { while (TRUE) { $item = db_query_range('SELECT data, item_id FROM {queue} q WHERE expire = 0 AND name = :name ORDER BY created DESC ', 0, 1, array(':name' => $this->name))->fetchObject(); // ...

EventQueue Class Required as part of the NWDUG website Any items added would not be available for 45 minutes to give a period of grace after creating an event node class EventQueue extends SystemQueue { public function claimItem($lease_time = 30) { while (TRUE) { $item = db_query_range('SELECT data, item_id FROM {queue} q WHERE expire = 0 AND name = :name AND created >= UNIX_TIMESTAMP(DATE_ADD(NOW(), INTERVAL 2700 SECOND)) ORDER BY created ASC', 0, 1, array(':name' => $this->name))->fetchObject();

WatchdogSystemQueue Extends SystemQueue and creates a log every time anything is done Maintains SystemQueue functionality

WatchdogSystemQueue class WatchdogSystemQueue extends SystemQueue { public function __construct($name) { watchdog('queue', '%name queue created', array('%name' => $name)); parent::__construct($name); } public function createItem($data) { watchdog('queue', 'Item created : %data', array('%data' => print_r($data, TRUE))); parent::createItem($data); } public function claimItem($lease_time = 3600) { $return_value = parent::claimItem($lease_time); watchdog('queue', 'Item claimed %item (lease time = %lease)', array( '%item' => print_r($return_value, TRUE), '%lease' => $lease_time)); return $return_value; } //....

RandomMemoryQueue Extends the MemoryQueue class Items are added in order but are retrieved in random order

RandomMemoryQueue class RandomMemoryQueue extends MemoryQueue { public function claimItem($lease_time = 30) { $available_items = array(); // Extract the remining available items foreach ($this->queue as $key => $item) { if ($item->expire == 0) { $available_items[] = $item; } } // Randomly select one (if available) if (count($available_items) > 0) { $queue_length = count($this->queue); $rand_item = rand(0, $queue_length - 1); $item = $available_items[$rand_item]; $item->expire = time() + $lease_time; return $item; } return FALSE; } }

Creating And Destroying DrupalQueueInterface has two methods available for creating and destroying the queue createQueue() Should be called within an install hook deleteQueue() Should be called within an uninstall hook

Tips Unless you really need to rewrite the entire class it is best to extend SystemQueue or MemoryQueue For open source projects try to keep the same retrieved item structure as the system queues No checks for unique items in default system queues

Resources Drupal Queues API http://api.drupal.org/api/drupal/modules--system--system.queue.inc/group/queue/7 http://bit.ly/kuphnc Queue UI Module http://drupal.org/project/queue_ui Source code is well documented /modules/system/system.queue.inc Full write up of this talk on #! code http://www.hashbangcode.com/

Questions? $queue = DrupalQueue::get('questions'); $question = $queue->claimItem(); echo $question->data['question'];

Blog http:///www.norton42.org.uk/ Twitter @philipnorton42 #! code http://www.hashbangcode.com/ #! code on Twitter @hashbangcode Philip Norton

Drupal 7 Queues

by Philip Norton on May 23, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 7

Statistics

Drupal 7 Queues — Presentation Transcript

http://paper.li	6
http://twitter.com	1