Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Message Queues and Drupal

Presented at Oklahoma Drupal Users Group on November 21, 2013.

  • Be the first to comment

Message Queues and Drupal

  1. 1. Message Queues and Drupal
  2. 2. About Me ● ● ● ● Name: Brian Altenhofel IRC: VeggieMeat Twitter: @BrianAltenhofel Drupal: #550616 What I Do: Commerce, Third-party APIs, Migrations, Devops, anything with data... Favorite Tools: Vim, Scotch, Padron Serie 1926 No. 1 Maduro
  3. 3. What are message queues?
  4. 4. Consumer Producer Producer Queue Consumer Consumer Producer What are message queues? Publisher Publisher Publisher Subscriber Queue Subscriber Subscriber
  5. 5. You already use queues.
  6. 6. In Drupal...
  7. 7. Queues are pluggable Backends available on Drupal.org: AWS SQS – http://drupal.org/project/aws_sqs Beanstalkd – http://drupal.org/project/beanstalkd Redis – http://drupal.org/project/redis STOMP – http://drupal.org/project/stomp (And.... still in sandbox but ready to go...) Openstack Marconi http://drupal.org/sandbox/BrianAltenhofel/2107851
  8. 8. Using a Different Queue As easy as: ● Global: $conf['queue_default_class'] = 'MarconiQueue'; ● Specific queue $conf['queue_class_{queue_name}'] = 'MarconiQueue';
  9. 9. Queues and Drush ● drush queue-list ● drush queue-run [queue_name]
  10. 10. Code.
  11. 11. Code. function my_module_create_queue() {   $queue = DrupalQueue::get('my_queue');   $queue­>createQueue(); }
  12. 12. Code. function my_module_create_queue_item() {   $queue = DrupalQueue::get('my_queue');   $queue_data = array(     'my_key' => 'my_value',   );   return $queue­>createItem($queue_data); }
  13. 13. Code. function my_module_count_queue_items() {   $queue = DrupalQueue::get('my_queue');      return $queue­>numberOfItems(); }
  14. 14. Code. function my_module_process_queue_item() {   $queue = DrupalQueue::get('my_queue');   $item = $queue­>claimItem(30);   if ($item && $item['my_key'] == 'my_value') {     $queue­>deleteItem($item);   }   else {     $queue­>releaseItem($item);   } }
  15. 15. Code. function my_module_cron_queue_info() {   $queues['my_queue'] = array(     'worker callback' =>                                 'my_module_process_queue_item',     'time' => 60,     // Time for cron to spend on this queue, not     // the lease time.   );   return $queues; }
  16. 16. It's really that simple.
  17. 17. Debugging ● watchdog() is a Drupaler's best friend ● Log everything (for both your sanity and insanity) ● (probably want to ship these logs if you're processing a lot of items) (if you want to get started on that): http://www.youtube.com/watch?v=p0Av29yaBEI
  18. 18. Monitoring ● Monitor queue depths with your favorite tool (Nagios, Zabbix, Zenoss, etc.) ● In a pinch, you can get an idea with $queue->numberOfItems(). ● Take action if queues get too deep
  19. 19. Any questions? @BrianAltenhofel IRC: VeggieMeat (MANY channels) brian.altenhofel@vmdoh.com

×