• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Message Queues and Drupal

Message Queues and Drupal



Presented at Oklahoma Drupal Users Group on November 21, 2013.

Presented at Oklahoma Drupal Users Group on November 21, 2013.



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as OpenOffice

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Commonly used message queues include Redis – RabbitMQ – IronMQ (service) – Amazon SQS/SNS <br />
  • - a datastore for messages to be consumed – like a mailbox <br /> - used for parallel processing, ensuring that a task is performed without blocking the user experience <br /> - scalability – typically think vertical (server size) or horizontal (more servers) – queues allow you to scale time and smooth spikes... great for rate-limited APIs <br /> - decoupling <br />
  • Many configurations possible... two most common <br /> Producer/Consumer – producers create messages, consumers claim them when they begin working on them, release them on failure, delete on completion – i.e. Parallel processing <br /> Publisher/Subscriber – publishers create messages, subscribers consume them as long as they are in the queue, message eventually expires and disappears from queue – i.e. Data feeds <br />
  • - uploading videos that need to be encoded <br /> - GitHub uses queues extensively... most of the work their service does is in the background <br /> - Drupal. Several modules implement queues (i.e. Notifications, Batch API) – every time you enable a module, it gets added to a queue to check for updates <br />
  • Queue-list will list all available queues to run <br /> Queue-run will run them (but will always delete items) – advantage over cron is that cron processes can only be run 1 at a time – queue-run can be invoked as much as you want <br />
  • Note: this is only necessary if Drupal is also acting as the consumer/subscriber. This is not necessary is Drupal is just producing or publishing messages to a queue to be consumed by something else. <br />
  • Actions: Send alert with PagerDuty... spin up new workers... <br />

Message Queues and Drupal Message Queues and Drupal Presentation Transcript

  • Message Queues and Drupal
  • About Me ● ● ● ● Name: Brian Altenhofel IRC: VeggieMeat Twitter: @BrianAltenhofel Drupal: #550616 What I Do: Commerce, Third-party APIs, Migrations, Devops, anything with data... Favorite Tools: Vim, Scotch, Padron Serie 1926 No. 1 Maduro
  • What are message queues?
  • Consumer Producer Producer Queue Consumer Consumer Producer What are message queues? Publisher Publisher Publisher Subscriber Queue Subscriber Subscriber
  • You already use queues.
  • In Drupal...
  • Queues are pluggable Backends available on Drupal.org: AWS SQS – http://drupal.org/project/aws_sqs Beanstalkd – http://drupal.org/project/beanstalkd Redis – http://drupal.org/project/redis STOMP – http://drupal.org/project/stomp (And.... still in sandbox but ready to go...) Openstack Marconi http://drupal.org/sandbox/BrianAltenhofel/2107851
  • Using a Different Queue As easy as: ● Global: $conf['queue_default_class'] = 'MarconiQueue'; ● Specific queue $conf['queue_class_{queue_name}'] = 'MarconiQueue';
  • Queues and Drush ● drush queue-list ● drush queue-run [queue_name]
  • Code.
  • Code. function my_module_create_queue() {   $queue = DrupalQueue::get('my_queue');   $queue­>createQueue(); }
  • Code. function my_module_create_queue_item() {   $queue = DrupalQueue::get('my_queue');   $queue_data = array(     'my_key' => 'my_value',   );   return $queue­>createItem($queue_data); }
  • Code. function my_module_count_queue_items() {   $queue = DrupalQueue::get('my_queue');      return $queue­>numberOfItems(); }
  • Code. function my_module_process_queue_item() {   $queue = DrupalQueue::get('my_queue');   $item = $queue­>claimItem(30);   if ($item && $item['my_key'] == 'my_value') {     $queue­>deleteItem($item);   }   else {     $queue­>releaseItem($item);   } }
  • Code. function my_module_cron_queue_info() {   $queues['my_queue'] = array(     'worker callback' =>                                 'my_module_process_queue_item',     'time' => 60,     // Time for cron to spend on this queue, not     // the lease time.   );   return $queues; }
  • It's really that simple.
  • Debugging ● watchdog() is a Drupaler's best friend ● Log everything (for both your sanity and insanity) ● (probably want to ship these logs if you're processing a lot of items) (if you want to get started on that): http://www.youtube.com/watch?v=p0Av29yaBEI
  • Monitoring ● Monitor queue depths with your favorite tool (Nagios, Zabbix, Zenoss, etc.) ● In a pinch, you can get an idea with $queue->numberOfItems(). ● Take action if queues get too deep
  • Any questions? @BrianAltenhofel IRC: VeggieMeat (MANY channels) brian.altenhofel@vmdoh.com