Your SlideShare is downloading. ×
Message Queues and Drupal
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Message Queues and Drupal


Published on

Presented at Oklahoma Drupal Users Group on November 21, 2013.

Presented at Oklahoma Drupal Users Group on November 21, 2013.

Published in: Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • Commonly used message queues include Redis – RabbitMQ – IronMQ (service) – Amazon SQS/SNS
  • - a datastore for messages to be consumed – like a mailbox
    - used for parallel processing, ensuring that a task is performed without blocking the user experience
    - scalability – typically think vertical (server size) or horizontal (more servers) – queues allow you to scale time and smooth spikes... great for rate-limited APIs
    - decoupling
  • Many configurations possible... two most common
    Producer/Consumer – producers create messages, consumers claim them when they begin working on them, release them on failure, delete on completion – i.e. Parallel processing
    Publisher/Subscriber – publishers create messages, subscribers consume them as long as they are in the queue, message eventually expires and disappears from queue – i.e. Data feeds
  • - uploading videos that need to be encoded
    - GitHub uses queues extensively... most of the work their service does is in the background
    - Drupal. Several modules implement queues (i.e. Notifications, Batch API) – every time you enable a module, it gets added to a queue to check for updates
  • Queue-list will list all available queues to run
    Queue-run will run them (but will always delete items) – advantage over cron is that cron processes can only be run 1 at a time – queue-run can be invoked as much as you want
  • Note: this is only necessary if Drupal is also acting as the consumer/subscriber. This is not necessary is Drupal is just producing or publishing messages to a queue to be consumed by something else.
  • Actions: Send alert with PagerDuty... spin up new workers...
  • Transcript

    • 1. Message Queues and Drupal
    • 2. About Me ● ● ● ● Name: Brian Altenhofel IRC: VeggieMeat Twitter: @BrianAltenhofel Drupal: #550616 What I Do: Commerce, Third-party APIs, Migrations, Devops, anything with data... Favorite Tools: Vim, Scotch, Padron Serie 1926 No. 1 Maduro
    • 3. What are message queues?
    • 4. Consumer Producer Producer Queue Consumer Consumer Producer What are message queues? Publisher Publisher Publisher Subscriber Queue Subscriber Subscriber
    • 5. You already use queues.
    • 6. In Drupal...
    • 7. Queues are pluggable Backends available on AWS SQS – Beanstalkd – Redis – STOMP – (And.... still in sandbox but ready to go...) Openstack Marconi
    • 8. Using a Different Queue As easy as: ● Global: $conf['queue_default_class'] = 'MarconiQueue'; ● Specific queue $conf['queue_class_{queue_name}'] = 'MarconiQueue';
    • 9. Queues and Drush ● drush queue-list ● drush queue-run [queue_name]
    • 10. Code.
    • 11. Code. function my_module_create_queue() {   $queue = DrupalQueue::get('my_queue');   $queue­>createQueue(); }
    • 12. Code. function my_module_create_queue_item() {   $queue = DrupalQueue::get('my_queue');   $queue_data = array(     'my_key' => 'my_value',   );   return $queue­>createItem($queue_data); }
    • 13. Code. function my_module_count_queue_items() {   $queue = DrupalQueue::get('my_queue');      return $queue­>numberOfItems(); }
    • 14. Code. function my_module_process_queue_item() {   $queue = DrupalQueue::get('my_queue');   $item = $queue­>claimItem(30);   if ($item && $item['my_key'] == 'my_value') {     $queue­>deleteItem($item);   }   else {     $queue­>releaseItem($item);   } }
    • 15. Code. function my_module_cron_queue_info() {   $queues['my_queue'] = array(     'worker callback' =>                                 'my_module_process_queue_item',     'time' => 60,     // Time for cron to spend on this queue, not     // the lease time.   );   return $queues; }
    • 16. It's really that simple.
    • 17. Debugging ● watchdog() is a Drupaler's best friend ● Log everything (for both your sanity and insanity) ● (probably want to ship these logs if you're processing a lot of items) (if you want to get started on that):
    • 18. Monitoring ● Monitor queue depths with your favorite tool (Nagios, Zabbix, Zenoss, etc.) ● In a pinch, you can get an idea with $queue->numberOfItems(). ● Take action if queues get too deep
    • 19. Any questions? @BrianAltenhofel IRC: VeggieMeat (MANY channels)