Message Queues and Drupal
About Me
●
●
●
●

Name: Brian Altenhofel
IRC: VeggieMeat
Twitter: @BrianAltenhofel
Drupal: #550616

What I Do: Commerce, T...
What are message queues?
Consumer

Producer
Producer

Queue

Consumer
Consumer

Producer

What are message queues?
Publisher
Publisher
Publisher

S...
You already use queues.
In Drupal...
Queues are pluggable
Backends available on Drupal.org:
AWS SQS – http://drupal.org/project/aws_sqs
Beanstalkd – http://dru...
Using a Different Queue
As easy as:

●

Global:
$conf['queue_default_class'] = 'MarconiQueue';

●

Specific queue
$conf['q...
Queues and Drush
●

drush queue-list

●

drush queue-run [queue_name]
Code.
Code.
function my_module_create_queue() {
  $queue = DrupalQueue::get('my_queue');
  $queue­>createQueue();
}
Code.
function my_module_create_queue_item() {
  $queue = DrupalQueue::get('my_queue');
  $queue_data = array(
    'my_key...
Code.
function my_module_count_queue_items() {
  $queue = DrupalQueue::get('my_queue');
  
  return $queue­>numberOfItems(...
Code.
function my_module_process_queue_item() {
  $queue = DrupalQueue::get('my_queue');
  $item = $queue­>claimItem(30);
...
Code.
function my_module_cron_queue_info() {
  $queues['my_queue'] = array(
    'worker callback' =>                      ...
It's really that simple.
Debugging
●

watchdog() is a Drupaler's best friend

●

Log everything (for both your sanity and insanity)

●

(probably w...
Monitoring
●

Monitor queue depths with your favorite tool (Nagios, Zabbix,
Zenoss, etc.)

●

In a pinch, you can get an i...
Any questions?
@BrianAltenhofel
IRC: VeggieMeat (MANY channels)
brian.altenhofel@vmdoh.com
Message Queues and Drupal
Upcoming SlideShare
Loading in …5
×

Message Queues and Drupal

2,171 views

Published on

Presented at Oklahoma Drupal Users Group on November 21, 2013.

Published in: Technology
  • Be the first to comment

Message Queues and Drupal

  1. 1. Message Queues and Drupal
  2. 2. About Me ● ● ● ● Name: Brian Altenhofel IRC: VeggieMeat Twitter: @BrianAltenhofel Drupal: #550616 What I Do: Commerce, Third-party APIs, Migrations, Devops, anything with data... Favorite Tools: Vim, Scotch, Padron Serie 1926 No. 1 Maduro
  3. 3. What are message queues?
  4. 4. Consumer Producer Producer Queue Consumer Consumer Producer What are message queues? Publisher Publisher Publisher Subscriber Queue Subscriber Subscriber
  5. 5. You already use queues.
  6. 6. In Drupal...
  7. 7. Queues are pluggable Backends available on Drupal.org: AWS SQS – http://drupal.org/project/aws_sqs Beanstalkd – http://drupal.org/project/beanstalkd Redis – http://drupal.org/project/redis STOMP – http://drupal.org/project/stomp (And.... still in sandbox but ready to go...) Openstack Marconi http://drupal.org/sandbox/BrianAltenhofel/2107851
  8. 8. Using a Different Queue As easy as: ● Global: $conf['queue_default_class'] = 'MarconiQueue'; ● Specific queue $conf['queue_class_{queue_name}'] = 'MarconiQueue';
  9. 9. Queues and Drush ● drush queue-list ● drush queue-run [queue_name]
  10. 10. Code.
  11. 11. Code. function my_module_create_queue() {   $queue = DrupalQueue::get('my_queue');   $queue­>createQueue(); }
  12. 12. Code. function my_module_create_queue_item() {   $queue = DrupalQueue::get('my_queue');   $queue_data = array(     'my_key' => 'my_value',   );   return $queue­>createItem($queue_data); }
  13. 13. Code. function my_module_count_queue_items() {   $queue = DrupalQueue::get('my_queue');      return $queue­>numberOfItems(); }
  14. 14. Code. function my_module_process_queue_item() {   $queue = DrupalQueue::get('my_queue');   $item = $queue­>claimItem(30);   if ($item && $item['my_key'] == 'my_value') {     $queue­>deleteItem($item);   }   else {     $queue­>releaseItem($item);   } }
  15. 15. Code. function my_module_cron_queue_info() {   $queues['my_queue'] = array(     'worker callback' =>                                 'my_module_process_queue_item',     'time' => 60,     // Time for cron to spend on this queue, not     // the lease time.   );   return $queues; }
  16. 16. It's really that simple.
  17. 17. Debugging ● watchdog() is a Drupaler's best friend ● Log everything (for both your sanity and insanity) ● (probably want to ship these logs if you're processing a lot of items) (if you want to get started on that): http://www.youtube.com/watch?v=p0Av29yaBEI
  18. 18. Monitoring ● Monitor queue depths with your favorite tool (Nagios, Zabbix, Zenoss, etc.) ● In a pinch, you can get an idea with $queue->numberOfItems(). ● Take action if queues get too deep
  19. 19. Any questions? @BrianAltenhofel IRC: VeggieMeat (MANY channels) brian.altenhofel@vmdoh.com

×