SlideShare a Scribd company logo
Drupal 7 Queues
Drupal Queues New to Drupal 7 First in first out data structure Used internally by Drupal Uses OO principles Fully customisable
Why Use Queues? Batch processing of large amounts of data/items Delay processing of complex calculations so system load is more stable Sequential processing of items Preventing API service black listing
Drupal 7 Implementations Batch API Cron Aggregator Module Update Module
Drupal Queues Found in the file: /modules/system/system.queue.inc Classes are used to wrap queue functionality Reliable vs Non-reliable
Reliable Generally kept in database table Every item will be executed at least once Items exist over several requests If the request fails then queue remains intact
Non-reliable Generally kept in memory only All items might exist in a single request No guarantee that all queue items will be executed No guarantee that all items will be executed in order If request fails then queue might be lost
Drupal Queue Classes
DrupalQueue Single static method get() DrupalQueue::get('my_queue'); Returns a queue object of a given name and a given type (reliable or non-reliable) Force reliable queue by passing TRUE as second parameter Default returned class is SystemQueue  Essentially a queue object factory
SystemQueue Example of a reliable queue class Implements  DrupalReliableQueueInterface Uses the database table  queue  to store and retrieve the queue Items are 'leased' to ensure no two processes get the same queue item Default class for new queues
queue Table Created at Drupal install
MemoryQueue Example of an non-reliable queue class All queue items are stored in memory $queue  parameter in class contains the queue Also implements item leasing
System Variables Three system variables are used by DrupalQueue to decide what sort of object to return 'queue_class_' . $name Default is NULL queue_default_class Default is SystemQueue queue_default_reliable_class Default is SystemQueue
Using Drupal Queues
Create A Queue $queue = DrupalQueue::get('my_queue', TRUE); $item = array( 'dataitem1' => 'something', 'int' => 123 ); $queue->createItem($item); echo $queue->numberOfItems(); // 1
Create A MemoryQueue variable_set('queue_default_class', 'MemoryQueue'); $queue = DrupalQueue::get('my_queue'); $item = array( 'dataitem1' => 'something', 'int' => 123 ); $queue->createItem($item); echo $queue->numberOfItems(); // 1
Get Item From Queue $queue = DrupalQueue::get('my_queue'); $got_item = $queue->claimItem(); echo $got_item->data['dataitem1'];
Retrieved Item Structure stdClass Object ( [data] => Array ( [dataitem1] => something [qwe] => 123 ) [item_id] => 89 )
Change Lease Length Can pass the lease time (in seconds) to claimItem() $got_item = $queue->claimItem( 100 );
Release Or Delete releaseItem() resets the lease time $queue->releaseItem($got_item); deleteItem() removes item from queue $queue->deleteItem($got_item);
Customizing Create custom queue class Create variable called “queue_class_” . $name Value is the class name of your queue class Call  DrupalQueue::get()  with the variable $name as a string
Customizing Reliable class MUST implement  DrupalReliableQueueInterface Can also extend SystemQueue All Queue classes should at least implement  DrupalQueueInterface
Using Custom Classes variable_set('queue_class_mycustom', 'MyCustomQueueClass'); $queue = DrupalQueue::get('mycustom');
Stack Class Last in first out implementation class Stack extends SystemQueue { public function claimItem($lease_time = 30) { while (TRUE) { $item = db_query_range('SELECT data, item_id FROM {queue} q WHERE  expire = 0 AND name = :name ORDER BY created  DESC ', 0, 1, array(':name' => $this->name))->fetchObject(); // ...
EventQueue Class Required as part of the NWDUG website Any items added would not be available for 45 minutes to give a period of grace after creating an event node class EventQueue extends SystemQueue { public function claimItem($lease_time = 30) { while (TRUE) { $item = db_query_range('SELECT data, item_id FROM {queue} q WHERE expire = 0 AND name = :name  AND created >= UNIX_TIMESTAMP(DATE_ADD(NOW(), INTERVAL 2700 SECOND))  ORDER BY created ASC', 0, 1, array(':name' => $this->name))->fetchObject();
WatchdogSystemQueue Extends SystemQueue and creates a log every time anything is done Maintains SystemQueue functionality
WatchdogSystemQueue class WatchdogSystemQueue extends SystemQueue { public function __construct($name) { watchdog('queue', '%name queue created',  array('%name' => $name)); parent::__construct($name); } public function createItem($data) { watchdog('queue', 'Item created : %data',  array('%data' => print_r($data, TRUE))); parent::createItem($data); } public function claimItem($lease_time = 3600) { $return_value = parent::claimItem($lease_time); watchdog('queue', 'Item claimed %item (lease time = %lease)',  array( '%item' => print_r($return_value, TRUE), '%lease' => $lease_time));  return $return_value; } //....
RandomMemoryQueue Extends the MemoryQueue class Items are added in order but are retrieved in random order
RandomMemoryQueue class RandomMemoryQueue extends MemoryQueue { public function claimItem($lease_time = 30) {  $available_items = array(); // Extract the remining available items foreach ($this->queue as $key => $item) { if ($item->expire == 0) { $available_items[] = $item; } } // Randomly select one (if available) if (count($available_items) > 0) { $queue_length = count($this->queue);  $rand_item = rand(0, $queue_length - 1); $item = $available_items[$rand_item]; $item->expire = time() + $lease_time; return $item; } return FALSE; } }
Creating And Destroying DrupalQueueInterface has two methods available for creating and destroying the queue createQueue() Should be called within an install hook deleteQueue() Should be called within an uninstall hook
Tips Unless you really need to rewrite the entire class it is best to extend SystemQueue or MemoryQueue For open source projects try to keep the same retrieved item structure as the system queues No checks for unique items in default system queues
Resources Drupal Queues API http://api.drupal.org/api/drupal/modules--system--system.queue.inc/group/queue/7 http://bit.ly/kuphnc Queue UI Module http://drupal.org/project/queue_ui Source code is well documented /modules/system/system.queue.inc Full write up of this talk on #! code http://www.hashbangcode.com/
Questions? $queue = DrupalQueue::get('questions'); $question = $queue->claimItem(); echo $question->data['question'];
Blog http:///www.norton42.org.uk/ Twitter @philipnorton42 #! code http://www.hashbangcode.com/ #! code on Twitter @hashbangcode Philip Norton

More Related Content

Similar to Drupal 7 Queues

Nickolay Shmalenuk.Render api eng.DrupalCamp Kyiv 2011
Nickolay Shmalenuk.Render api eng.DrupalCamp Kyiv 2011Nickolay Shmalenuk.Render api eng.DrupalCamp Kyiv 2011
Nickolay Shmalenuk.Render api eng.DrupalCamp Kyiv 2011
camp_drupal_ua
 
Lazy evaluation drupal camp moscow 2014
Lazy evaluation drupal camp moscow 2014Lazy evaluation drupal camp moscow 2014
Lazy evaluation drupal camp moscow 2014
Evgeny Nikitin
 
MUC - Moodle Universal Cache
MUC - Moodle Universal CacheMUC - Moodle Universal Cache
MUC - Moodle Universal Cache
Tim Hunt
 
Power shell examples_v4
Power shell examples_v4Power shell examples_v4
Power shell examples_v4
JoeDinaso
 
Intro to The PHP SPL
Intro to The PHP SPLIntro to The PHP SPL
Intro to The PHP SPL
Chris Tankersley
 
Introduction to cron queue
Introduction to cron queueIntroduction to cron queue
Introduction to cron queue
ADCI Solutions
 
ZendCon2010 The Doctrine Project
ZendCon2010 The Doctrine ProjectZendCon2010 The Doctrine Project
ZendCon2010 The Doctrine Project
Jonathan Wage
 
Advanced symfony Techniques
Advanced symfony TechniquesAdvanced symfony Techniques
Advanced symfony Techniques
Kris Wallsmith
 
The state of hooking into Drupal - DrupalCon Dublin
The state of hooking into Drupal - DrupalCon DublinThe state of hooking into Drupal - DrupalCon Dublin
The state of hooking into Drupal - DrupalCon Dublin
Nida Ismail Shah
 
DRUPAL 8 STORAGES OVERVIEW
DRUPAL 8 STORAGES OVERVIEWDRUPAL 8 STORAGES OVERVIEW
DRUPAL 8 STORAGES OVERVIEW
DrupalCamp Kyiv
 
Introducing CakeEntity
Introducing CakeEntityIntroducing CakeEntity
Introducing CakeEntity
Basuke Suzuki
 
OOP in PHP.pptx
OOP in PHP.pptxOOP in PHP.pptx
OOP in PHP.pptx
switipatel4
 
Advanced PHPUnit Testing
Advanced PHPUnit TestingAdvanced PHPUnit Testing
Advanced PHPUnit Testing
Mike Lively
 
JavaScript in Drupal 7: What developers need to know
JavaScript in Drupal 7: What developers need to knowJavaScript in Drupal 7: What developers need to know
JavaScript in Drupal 7: What developers need to know
Work at Play
 
JavaScript in Drupal 7: What developers need to know
JavaScript in Drupal 7: What developers need to knowJavaScript in Drupal 7: What developers need to know
JavaScript in Drupal 7: What developers need to know
katbailey
 
Unittests für Dummies
Unittests für DummiesUnittests für Dummies
Unittests für Dummies
Lars Jankowfsky
 
Advanced moduledevelopment d6_slideshare
Advanced moduledevelopment d6_slideshareAdvanced moduledevelopment d6_slideshare
Advanced moduledevelopment d6_slideshare
Opevel
 
Easy rest service using PHP reflection api
Easy rest service using PHP reflection apiEasy rest service using PHP reflection api
Easy rest service using PHP reflection api
Matthieu Aubry
 
WordPress for developers - phpday 2011
WordPress for developers -  phpday 2011WordPress for developers -  phpday 2011
WordPress for developers - phpday 2011
Maurizio Pelizzone
 
Php on the desktop and php gtk2
Php on the desktop and php gtk2Php on the desktop and php gtk2
Php on the desktop and php gtk2
Elizabeth Smith
 

Similar to Drupal 7 Queues (20)

Nickolay Shmalenuk.Render api eng.DrupalCamp Kyiv 2011
Nickolay Shmalenuk.Render api eng.DrupalCamp Kyiv 2011Nickolay Shmalenuk.Render api eng.DrupalCamp Kyiv 2011
Nickolay Shmalenuk.Render api eng.DrupalCamp Kyiv 2011
 
Lazy evaluation drupal camp moscow 2014
Lazy evaluation drupal camp moscow 2014Lazy evaluation drupal camp moscow 2014
Lazy evaluation drupal camp moscow 2014
 
MUC - Moodle Universal Cache
MUC - Moodle Universal CacheMUC - Moodle Universal Cache
MUC - Moodle Universal Cache
 
Power shell examples_v4
Power shell examples_v4Power shell examples_v4
Power shell examples_v4
 
Intro to The PHP SPL
Intro to The PHP SPLIntro to The PHP SPL
Intro to The PHP SPL
 
Introduction to cron queue
Introduction to cron queueIntroduction to cron queue
Introduction to cron queue
 
ZendCon2010 The Doctrine Project
ZendCon2010 The Doctrine ProjectZendCon2010 The Doctrine Project
ZendCon2010 The Doctrine Project
 
Advanced symfony Techniques
Advanced symfony TechniquesAdvanced symfony Techniques
Advanced symfony Techniques
 
The state of hooking into Drupal - DrupalCon Dublin
The state of hooking into Drupal - DrupalCon DublinThe state of hooking into Drupal - DrupalCon Dublin
The state of hooking into Drupal - DrupalCon Dublin
 
DRUPAL 8 STORAGES OVERVIEW
DRUPAL 8 STORAGES OVERVIEWDRUPAL 8 STORAGES OVERVIEW
DRUPAL 8 STORAGES OVERVIEW
 
Introducing CakeEntity
Introducing CakeEntityIntroducing CakeEntity
Introducing CakeEntity
 
OOP in PHP.pptx
OOP in PHP.pptxOOP in PHP.pptx
OOP in PHP.pptx
 
Advanced PHPUnit Testing
Advanced PHPUnit TestingAdvanced PHPUnit Testing
Advanced PHPUnit Testing
 
JavaScript in Drupal 7: What developers need to know
JavaScript in Drupal 7: What developers need to knowJavaScript in Drupal 7: What developers need to know
JavaScript in Drupal 7: What developers need to know
 
JavaScript in Drupal 7: What developers need to know
JavaScript in Drupal 7: What developers need to knowJavaScript in Drupal 7: What developers need to know
JavaScript in Drupal 7: What developers need to know
 
Unittests für Dummies
Unittests für DummiesUnittests für Dummies
Unittests für Dummies
 
Advanced moduledevelopment d6_slideshare
Advanced moduledevelopment d6_slideshareAdvanced moduledevelopment d6_slideshare
Advanced moduledevelopment d6_slideshare
 
Easy rest service using PHP reflection api
Easy rest service using PHP reflection apiEasy rest service using PHP reflection api
Easy rest service using PHP reflection api
 
WordPress for developers - phpday 2011
WordPress for developers -  phpday 2011WordPress for developers -  phpday 2011
WordPress for developers - phpday 2011
 
Php on the desktop and php gtk2
Php on the desktop and php gtk2Php on the desktop and php gtk2
Php on the desktop and php gtk2
 

More from Philip Norton

ReactPHP
ReactPHPReactPHP
ReactPHP
Philip Norton
 
Getting Into Drupal 8 Configuration
Getting Into Drupal 8 ConfigurationGetting Into Drupal 8 Configuration
Getting Into Drupal 8 Configuration
Philip Norton
 
Drupal 8 Services And Dependency Injection
Drupal 8 Services And Dependency InjectionDrupal 8 Services And Dependency Injection
Drupal 8 Services And Dependency Injection
Philip Norton
 
Drupal 8 Services
Drupal 8 ServicesDrupal 8 Services
Drupal 8 Services
Philip Norton
 
Webform and Drupal 8
Webform and Drupal 8Webform and Drupal 8
Webform and Drupal 8
Philip Norton
 
Acquia Drupal Certification
Acquia Drupal CertificationAcquia Drupal Certification
Acquia Drupal Certification
Philip Norton
 
Becoming A Drupal Master Builder
Becoming A Drupal Master BuilderBecoming A Drupal Master Builder
Becoming A Drupal Master Builder
Philip Norton
 
Drupal Performance : DrupalCamp North
Drupal Performance : DrupalCamp NorthDrupal Performance : DrupalCamp North
Drupal Performance : DrupalCamp North
Philip Norton
 
Drupal 8 Configuration Management
Drupal 8 Configuration ManagementDrupal 8 Configuration Management
Drupal 8 Configuration Management
Philip Norton
 
Getting Started With Jenkins And Drupal
Getting Started With Jenkins And DrupalGetting Started With Jenkins And Drupal
Getting Started With Jenkins And Drupal
Philip Norton
 
Drupal theming
Drupal themingDrupal theming
Drupal theming
Philip Norton
 
Drush
DrushDrush
Making The Drupal Pill Easier To Swallow
Making The Drupal Pill Easier To SwallowMaking The Drupal Pill Easier To Swallow
Making The Drupal Pill Easier To Swallow
Philip Norton
 

More from Philip Norton (13)

ReactPHP
ReactPHPReactPHP
ReactPHP
 
Getting Into Drupal 8 Configuration
Getting Into Drupal 8 ConfigurationGetting Into Drupal 8 Configuration
Getting Into Drupal 8 Configuration
 
Drupal 8 Services And Dependency Injection
Drupal 8 Services And Dependency InjectionDrupal 8 Services And Dependency Injection
Drupal 8 Services And Dependency Injection
 
Drupal 8 Services
Drupal 8 ServicesDrupal 8 Services
Drupal 8 Services
 
Webform and Drupal 8
Webform and Drupal 8Webform and Drupal 8
Webform and Drupal 8
 
Acquia Drupal Certification
Acquia Drupal CertificationAcquia Drupal Certification
Acquia Drupal Certification
 
Becoming A Drupal Master Builder
Becoming A Drupal Master BuilderBecoming A Drupal Master Builder
Becoming A Drupal Master Builder
 
Drupal Performance : DrupalCamp North
Drupal Performance : DrupalCamp NorthDrupal Performance : DrupalCamp North
Drupal Performance : DrupalCamp North
 
Drupal 8 Configuration Management
Drupal 8 Configuration ManagementDrupal 8 Configuration Management
Drupal 8 Configuration Management
 
Getting Started With Jenkins And Drupal
Getting Started With Jenkins And DrupalGetting Started With Jenkins And Drupal
Getting Started With Jenkins And Drupal
 
Drupal theming
Drupal themingDrupal theming
Drupal theming
 
Drush
DrushDrush
Drush
 
Making The Drupal Pill Easier To Swallow
Making The Drupal Pill Easier To SwallowMaking The Drupal Pill Easier To Swallow
Making The Drupal Pill Easier To Swallow
 

Recently uploaded

Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Ana-Maria Mihalceanu
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
Claudio Di Ciccio
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Rohit Gautam
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Edge AI and Vision Alliance
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Adtran
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
SOFTTECHHUB
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
James Anderson
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
Removing Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software FuzzingRemoving Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software Fuzzing
Aftab Hussain
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 

Recently uploaded (20)

Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
Removing Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software FuzzingRemoving Uninteresting Bytes in Software Fuzzing
Removing Uninteresting Bytes in Software Fuzzing
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 

Drupal 7 Queues

  • 2. Drupal Queues New to Drupal 7 First in first out data structure Used internally by Drupal Uses OO principles Fully customisable
  • 3. Why Use Queues? Batch processing of large amounts of data/items Delay processing of complex calculations so system load is more stable Sequential processing of items Preventing API service black listing
  • 4. Drupal 7 Implementations Batch API Cron Aggregator Module Update Module
  • 5. Drupal Queues Found in the file: /modules/system/system.queue.inc Classes are used to wrap queue functionality Reliable vs Non-reliable
  • 6. Reliable Generally kept in database table Every item will be executed at least once Items exist over several requests If the request fails then queue remains intact
  • 7. Non-reliable Generally kept in memory only All items might exist in a single request No guarantee that all queue items will be executed No guarantee that all items will be executed in order If request fails then queue might be lost
  • 9. DrupalQueue Single static method get() DrupalQueue::get('my_queue'); Returns a queue object of a given name and a given type (reliable or non-reliable) Force reliable queue by passing TRUE as second parameter Default returned class is SystemQueue Essentially a queue object factory
  • 10. SystemQueue Example of a reliable queue class Implements DrupalReliableQueueInterface Uses the database table queue to store and retrieve the queue Items are 'leased' to ensure no two processes get the same queue item Default class for new queues
  • 11. queue Table Created at Drupal install
  • 12. MemoryQueue Example of an non-reliable queue class All queue items are stored in memory $queue parameter in class contains the queue Also implements item leasing
  • 13. System Variables Three system variables are used by DrupalQueue to decide what sort of object to return 'queue_class_' . $name Default is NULL queue_default_class Default is SystemQueue queue_default_reliable_class Default is SystemQueue
  • 15. Create A Queue $queue = DrupalQueue::get('my_queue', TRUE); $item = array( 'dataitem1' => 'something', 'int' => 123 ); $queue->createItem($item); echo $queue->numberOfItems(); // 1
  • 16. Create A MemoryQueue variable_set('queue_default_class', 'MemoryQueue'); $queue = DrupalQueue::get('my_queue'); $item = array( 'dataitem1' => 'something', 'int' => 123 ); $queue->createItem($item); echo $queue->numberOfItems(); // 1
  • 17. Get Item From Queue $queue = DrupalQueue::get('my_queue'); $got_item = $queue->claimItem(); echo $got_item->data['dataitem1'];
  • 18. Retrieved Item Structure stdClass Object ( [data] => Array ( [dataitem1] => something [qwe] => 123 ) [item_id] => 89 )
  • 19. Change Lease Length Can pass the lease time (in seconds) to claimItem() $got_item = $queue->claimItem( 100 );
  • 20. Release Or Delete releaseItem() resets the lease time $queue->releaseItem($got_item); deleteItem() removes item from queue $queue->deleteItem($got_item);
  • 21. Customizing Create custom queue class Create variable called “queue_class_” . $name Value is the class name of your queue class Call DrupalQueue::get() with the variable $name as a string
  • 22. Customizing Reliable class MUST implement DrupalReliableQueueInterface Can also extend SystemQueue All Queue classes should at least implement DrupalQueueInterface
  • 23. Using Custom Classes variable_set('queue_class_mycustom', 'MyCustomQueueClass'); $queue = DrupalQueue::get('mycustom');
  • 24. Stack Class Last in first out implementation class Stack extends SystemQueue { public function claimItem($lease_time = 30) { while (TRUE) { $item = db_query_range('SELECT data, item_id FROM {queue} q WHERE expire = 0 AND name = :name ORDER BY created DESC ', 0, 1, array(':name' => $this->name))->fetchObject(); // ...
  • 25. EventQueue Class Required as part of the NWDUG website Any items added would not be available for 45 minutes to give a period of grace after creating an event node class EventQueue extends SystemQueue { public function claimItem($lease_time = 30) { while (TRUE) { $item = db_query_range('SELECT data, item_id FROM {queue} q WHERE expire = 0 AND name = :name AND created >= UNIX_TIMESTAMP(DATE_ADD(NOW(), INTERVAL 2700 SECOND)) ORDER BY created ASC', 0, 1, array(':name' => $this->name))->fetchObject();
  • 26. WatchdogSystemQueue Extends SystemQueue and creates a log every time anything is done Maintains SystemQueue functionality
  • 27. WatchdogSystemQueue class WatchdogSystemQueue extends SystemQueue { public function __construct($name) { watchdog('queue', '%name queue created', array('%name' => $name)); parent::__construct($name); } public function createItem($data) { watchdog('queue', 'Item created : %data', array('%data' => print_r($data, TRUE))); parent::createItem($data); } public function claimItem($lease_time = 3600) { $return_value = parent::claimItem($lease_time); watchdog('queue', 'Item claimed %item (lease time = %lease)', array( '%item' => print_r($return_value, TRUE), '%lease' => $lease_time)); return $return_value; } //....
  • 28. RandomMemoryQueue Extends the MemoryQueue class Items are added in order but are retrieved in random order
  • 29. RandomMemoryQueue class RandomMemoryQueue extends MemoryQueue { public function claimItem($lease_time = 30) { $available_items = array(); // Extract the remining available items foreach ($this->queue as $key => $item) { if ($item->expire == 0) { $available_items[] = $item; } } // Randomly select one (if available) if (count($available_items) > 0) { $queue_length = count($this->queue); $rand_item = rand(0, $queue_length - 1); $item = $available_items[$rand_item]; $item->expire = time() + $lease_time; return $item; } return FALSE; } }
  • 30. Creating And Destroying DrupalQueueInterface has two methods available for creating and destroying the queue createQueue() Should be called within an install hook deleteQueue() Should be called within an uninstall hook
  • 31. Tips Unless you really need to rewrite the entire class it is best to extend SystemQueue or MemoryQueue For open source projects try to keep the same retrieved item structure as the system queues No checks for unique items in default system queues
  • 32. Resources Drupal Queues API http://api.drupal.org/api/drupal/modules--system--system.queue.inc/group/queue/7 http://bit.ly/kuphnc Queue UI Module http://drupal.org/project/queue_ui Source code is well documented /modules/system/system.queue.inc Full write up of this talk on #! code http://www.hashbangcode.com/
  • 33. Questions? $queue = DrupalQueue::get('questions'); $question = $queue->claimItem(); echo $question->data['question'];
  • 34. Blog http:///www.norton42.org.uk/ Twitter @philipnorton42 #! code http://www.hashbangcode.com/ #! code on Twitter @hashbangcode Philip Norton