Your SlideShare is downloading. ×
0
PLEASE HOLD: YOUR                     CALL IS IN A QUEUE                     An introduction to queueing in Drupal        ...
QUEUING IN 30 MINUTES...    •I     hate waiting in line...    • Drupal         Queue API    • Beyond         Core    • Que...
I HATE WAITING IN LINE...Sunday, 3 March 13
Sunday, 3 March 13
Sunday, 3 March 13
Sunday, 3 March 13
Sunday, 3 March 13
QUEUES WILL FORM    WHEN PROCESSES WITH VARIABILITY      ARE LOADED TO HIGH LEVELS             OF UTILISATION       WITHOU...
SOMETHING WILL QUEUE                SOMEWHERESunday, 3 March 13
SOMETHING WILL QUEUE                SOMEWHERE                     Whether you like it or notSunday, 3 March 13
DO YOU WANT TO BE IN                          CONTROL?Sunday, 3 March 13
<silly_question>                     DO YOU WANT TO BE IN                          CONTROL?                           </si...
DRUPAL TO THE              RESCUE!Sunday, 3 March 13
CORE QUEUE API    • Two            kinds of queue: reliable and non-reliable    • Provides           interface to common q...
WHERE CAN I FIND QUEUE                   API?    • Drupal         7:         • ./modules/system/system.queue.inc    • Drup...
QUEUE API METHODS    • createQueue()           • numberOfItems()    • createItem()            • deleteItem()    • claimIte...
Code time.Sunday, 3 March 13
/**                * Create a queue.                */              function queues_create_queue() {                 $queu...
/**               * Add an item to a queue.               */              function queues_create_queueitem() {            ...
/**               * Process a queue item               */              function queues_process_queueitem() {              ...
/**                * Implements hook_cron_queue_info().                */              function queues_cron_queue_info() {...
/**                * Implements hook_queue_info() from queue_ui module.                */              function queues_que...
That’s it!Sunday, 3 March 13
BEYOND CORE    •   Drupal Queue (backport to D6) - http://drupal.org/project/drupal_queue    •   Queue UI - http://drupal....
CUSTOM BACKENDS    • Default        queue class:         • $conf[queue_default_class]   = SystemQueue;         • $conf[‘qu...
CUSTOM BACKENDS    • Default        queue class:         • $conf[queue_default_class]   = SystemQueue;         • $conf[‘qu...
QUEUES AND DRUSH    • drush          queue-list    • drush          queue-run [queue name]    • ...use         with cautio...
WHEN TO QUEUE    • Remote           service call triggered by user interface    • Complex           business rules around ...
SOME EXAMPLES    • Publishing         user information to a remote CRM    • Backend          order fulfilment    • Email   ...
GOTCHASSunday, 3 March 13
MULTIPLE CONSUMERSSunday, 3 March 13
MULTIPLE CONSUMERS    • Identify          what it is you’re processing    • Use            Lock API to secure a lock on it...
DEBUGGING YOUR QUEUESunday, 3 March 13
DEBUGGING YOUR QUEUE    • watchdog()          is your friend    • Log            EVERYTHING    • Abstract          it so y...
THINGS CAN GO WRONG...Sunday, 3 March 13
THINGS CAN GO WRONG...    • Monitor          your queue depths with Munin or Nagios    • Alert          if they get too de...
THINGS BREAK.                      BUT WHEN THEY DO, KNOW THAT                     EVERY ITEM IN YOUR QUEUE               ...
IMPLEMENTING QUEUEING IN YOUR APPLICATION               SMOOTHS THE PEAKS,Sunday, 3 March 13
IMPLEMENTING QUEUEING IN YOUR APPLICATION                SMOOTHS THE PEAKS,           MAKE THROUGHPUT PREDICTABLE,Sunday, ...
IMPLEMENTING QUEUEING IN YOUR APPLICATION                SMOOTHS THE PEAKS,           MAKE THROUGHPUT PREDICTABLE,        ...
IMPLEMENTING QUEUEING IN YOUR APPLICATION                SMOOTHS THE PEAKS,           MAKE THROUGHPUT PREDICTABLE,        ...
IN OTHER WORDS...Sunday, 3 March 13
Sunday, 3 March 13
QUESTIONS?   Tweet: @tsphethean  Drupal.org: tsphethean     IRC: tsphetheanSunday, 3 March 13
IMAGE ATTRIBUTIONS    •   Image of supermarket queue - http://petoneponderings.blogspot.co.uk/2012_12_01_archive.html    •...
Upcoming SlideShare
Loading in...5
×

Please hold: your call is in a queue. An introduction to queuing in Drupal.

2,056

Published on

Please hold: your call is in a queue. An introduction to queuing in Drupal.

My talk from Drupalcamp London 1st-3rd March 2013. A version of this talk with minor modifications was also given at the Drupal Learning April 2013 meetup in London (http://www.meetup.com/Learning-Drupal-Meetup/events/111107222/).

It's not just the Brits that love queues, Drupal loves them too.

Maybe your website is making requests to another internal or third party system. Maybe a third party is sending messages to you. Either way, as you begin to grow your services are increasingly likely to run into problems completing these requests in a predictable amount of time and with guaranteed success.

Queueing is one solution to this, balancing load, ensuring jobs are processed successfully - but the uses don't end there.

This session will provide an overview of queueing patterns, discuss some scenarios where you may want to implement queues, and look at how Drupal implements queueing - as well as some of the gotchas in implementing queue processing at scale.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,056
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
9
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Please hold: your call is in a queue. An introduction to queuing in Drupal."

  1. 1. PLEASE HOLD: YOUR CALL IS IN A QUEUE An introduction to queueing in Drupal by Tom PhetheanSunday, 3 March 13
  2. 2. QUEUING IN 30 MINUTES... •I hate waiting in line... • Drupal Queue API • Beyond Core • Queue jumpers and other gotchas • QuestionsSunday, 3 March 13
  3. 3. I HATE WAITING IN LINE...Sunday, 3 March 13
  4. 4. Sunday, 3 March 13
  5. 5. Sunday, 3 March 13
  6. 6. Sunday, 3 March 13
  7. 7. Sunday, 3 March 13
  8. 8. QUEUES WILL FORM WHEN PROCESSES WITH VARIABILITY ARE LOADED TO HIGH LEVELS OF UTILISATION WITHOUT CONSTRAINTSunday, 3 March 13
  9. 9. SOMETHING WILL QUEUE SOMEWHERESunday, 3 March 13
  10. 10. SOMETHING WILL QUEUE SOMEWHERE Whether you like it or notSunday, 3 March 13
  11. 11. DO YOU WANT TO BE IN CONTROL?Sunday, 3 March 13
  12. 12. <silly_question> DO YOU WANT TO BE IN CONTROL? </silly_question>Sunday, 3 March 13
  13. 13. DRUPAL TO THE RESCUE!Sunday, 3 March 13
  14. 14. CORE QUEUE API • Two kinds of queue: reliable and non-reliable • Provides interface to common queue functions • MemoryQueue implements QueueInterface • SystemQueue implements ReliableQueueInterface • In Drupal 7 (and 8), Batch API extends SystemQueueSunday, 3 March 13
  15. 15. WHERE CAN I FIND QUEUE API? • Drupal 7: • ./modules/system/system.queue.inc • Drupal 8: • ./core/lib/Drupal/Core/QueueSunday, 3 March 13
  16. 16. QUEUE API METHODS • createQueue() • numberOfItems() • createItem() • deleteItem() • claimItem() • deleteQueue() • releaseItem()Sunday, 3 March 13
  17. 17. Code time.Sunday, 3 March 13
  18. 18. /** * Create a queue. */ function queues_create_queue() { $queue = DrupalQueue::get(my_queue); $queue->createQueue(); }Sunday, 3 March 13
  19. 19. /** * Add an item to a queue. */ function queues_create_queueitem() { $queue = DrupalQueue::get(my_queue); $queue_data = array ( stuff => test, ); return $queue->createItem($queue_data); } /** * Check number of items on a queue. */ function queues_count_queueitems() { $queue = DrupalQueue::get(my_queue); return $queue->numberOfItems(); }Sunday, 3 March 13
  20. 20. /** * Process a queue item */ function queues_process_queueitem() { $queue = DrupalQueue::get(my_queue); $item = $queue->claimItem(30); if ($item && $item->data[stuff] == test) { // Success - we dont need this item anymore. $queue->deleteItem($item); } else { // Something unexpected happened, we still need this item. $queue->releaseItem($item); } }Sunday, 3 March 13
  21. 21. /** * Implements hook_cron_queue_info(). */ function queues_cron_queue_info() { $queues[my_queue] = array( worker callback => queues_process_queueitem, time => 60, // Time is NOT the lease item. // Time is the amount of time cron will spend processing this queue. ); return $queues; }Sunday, 3 March 13
  22. 22. /** * Implements hook_queue_info() from queue_ui module. */ function queues_queue_info() { return array( my_queue => array( title => t(My test queue), batch => array( operations => array(array(queues_batch_process, array())), finished => queues_batch_finished, title => t(Processing my test queue), ), cron => array( callback =>queues_process_queueitem, ), ), ); }Sunday, 3 March 13
  23. 23. That’s it!Sunday, 3 March 13
  24. 24. BEYOND CORE • Drupal Queue (backport to D6) - http://drupal.org/project/drupal_queue • Queue UI - http://drupal.org/project/queue_ui • Examples - http://drupal.org/project/examples • Queue backends: • Beanstalkd - http://drupal.org/project/beanstalkd • STOMP - http://drupal.org/project/stomp • Redis - http://drupal.org/project/redis_queueSunday, 3 March 13
  25. 25. CUSTOM BACKENDS • Default queue class: • $conf[queue_default_class] = SystemQueue; • $conf[‘queue_default_reliable_class’] = ‘SystemQueue’; • Override queue classes: • $conf[queue_class_{queue name}] = StompQueue;Sunday, 3 March 13
  26. 26. CUSTOM BACKENDS • Default queue class: • $conf[queue_default_class] = SystemQueue; • $conf[‘queue_default_reliable_class’] = ‘SystemQueue’; • Override queue classes: • $conf[queue_class_{queue name}] = RedisQueue;Sunday, 3 March 13
  27. 27. QUEUES AND DRUSH • drush queue-list • drush queue-run [queue name] • ...use with caution: queue-run will always delete your queue itemSunday, 3 March 13
  28. 28. WHEN TO QUEUE • Remote service call triggered by user interface • Complex business rules around data processing • Need guaranteed successSunday, 3 March 13
  29. 29. SOME EXAMPLES • Publishing user information to a remote CRM • Backend order fulfilment • Email sending (see notifications module)Sunday, 3 March 13
  30. 30. GOTCHASSunday, 3 March 13
  31. 31. MULTIPLE CONSUMERSSunday, 3 March 13
  32. 32. MULTIPLE CONSUMERS • Identify what it is you’re processing • Use Lock API to secure a lock on it: • lock_acquire(‘my_id’) • lock_release(‘my_id’) • Check the “thing” is in a state you expect it to be • Release or Delete from queue as appropriateSunday, 3 March 13
  33. 33. DEBUGGING YOUR QUEUESunday, 3 March 13
  34. 34. DEBUGGING YOUR QUEUE • watchdog() is your friend • Log EVERYTHING • Abstract it so you can toggle verbose logging • Use Graylog, LogStash etc to save your database from death by logsSunday, 3 March 13
  35. 35. THINGS CAN GO WRONG...Sunday, 3 March 13
  36. 36. THINGS CAN GO WRONG... • Monitor your queue depths with Munin or Nagios • Alert if they get too deep • Know what the alert means (and listen to it)!Sunday, 3 March 13
  37. 37. THINGS BREAK. BUT WHEN THEY DO, KNOW THAT EVERY ITEM IN YOUR QUEUE IS A REQUEST WHICH WOULD OTHERWISE HAVE BEEN LOST.Sunday, 3 March 13
  38. 38. IMPLEMENTING QUEUEING IN YOUR APPLICATION SMOOTHS THE PEAKS,Sunday, 3 March 13
  39. 39. IMPLEMENTING QUEUEING IN YOUR APPLICATION SMOOTHS THE PEAKS, MAKE THROUGHPUT PREDICTABLE,Sunday, 3 March 13
  40. 40. IMPLEMENTING QUEUEING IN YOUR APPLICATION SMOOTHS THE PEAKS, MAKE THROUGHPUT PREDICTABLE, REMOVES PROCESSING TIME FROM THE USER EXPERIENCESunday, 3 March 13
  41. 41. IMPLEMENTING QUEUEING IN YOUR APPLICATION SMOOTHS THE PEAKS, MAKE THROUGHPUT PREDICTABLE, REMOVES PROCESSING TIME FROM THE USER EXPERIENCE AND ENSURES THAT PROBLEMS ARE HANDLED GRACEFULLYSunday, 3 March 13
  42. 42. IN OTHER WORDS...Sunday, 3 March 13
  43. 43. Sunday, 3 March 13
  44. 44. QUESTIONS? Tweet: @tsphethean Drupal.org: tsphethean IRC: tsphetheanSunday, 3 March 13
  45. 45. IMAGE ATTRIBUTIONS • Image of supermarket queue - http://petoneponderings.blogspot.co.uk/2012_12_01_archive.html • Post Office queue - http://www.flickr.com/photos/welshkaren/5367517662/ • Apple Store queueing http://www.telegraph.co.uk/technology/apple/7854982/Apple-iPhone-4-more- than-a-quarter-of-users-think-video-calling-is-a-gimmick.html • Supermarket crush - http://www.dailymail.co.uk/news/article-2078597/Boxing-Day-sales-Record- numbers-shops-open-80-push-grab-shoppers.html • Drupal Superhero - http://www.origineight.net/blog-post/web-sites-you-manage • Lots of Shoppers - http://www.dailymail.co.uk/news/article-2253523/United-Nations-Bargain-Hunters- Britons-the-queue-China-Middle-East-lead-rush-designer-brands.html • Confused baby - http://thementalpausechronicles.blogspot.co.uk/2008/08/head-scratching.html • Mark Sonnabaum - http://www.whatwouldmarksonnabaumdo.comSunday, 3 March 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×