Joomla Day Florida 2017 - Background processing queues with Beanstalkd

2. DON’T MAKE ME
WAIT
Let’s get queued up
Chad Windnagle - @drmmr763

3. INTRODUCTION
• Recovering Joomla Addict (Clean
2yrs)
• Joomla GSoC Admin (2012-2015)
• Joomla Resource Team Editor
• Joomla!Day, JWC, & JAB Speaker
• JWC 2014 Keynote Speaker
Chad Windnagle - @drmmr763

4. JOOMLA WAS MY
PLAYGROUND

5. Chad Windnagle - @drmmr763
JOOMLA INSPIRED
ME

6. AUDIENCE
INTRODUCTION
Chad Windnagle - @drmmr763

7. WHAT ARE QUEUES
ANYWAY?

8. Chad Windnagle - @drmmr763
SYNCHRONOUS FLOW
User Sends
Request
Server Sends
Response
Response
Rendered

9. Chad Windnagle - @drmmr763
A-SYNCHRONOUS
FLOW
User Sends
Request
Server Sends
Response
Response
Rendered
Background Task
Initialized
Background Task
Executed
Notify User
Task Complete

10. Chad Windnagle - @drmmr763
WHAT SHOULD WE
QUEUE?•Notifications (Email /
SMS)
•Statistical Calculations
•Data Matching
Algorithms
•Regularly Scheduled
Tasks
•Image / Video
Processing
•Data Indexing
•Traditional Page
Requests
•User File Uploads
•Login / Logout
Functions
•File Downloads
•Add to Cart Functions
•REST API Endpoints
Use Queues: Don’t Use Queues:

11. COMPONENTS OF A
QUEUE SYSTEM
Chad Windnagle - @drmmr763

12. THE MESSAGE
A message is string-based content which informs
the application which task to perform, and
encapsulates any data needed to perform that task.
Chad Windnagle - @drmmr763

13. {
"messageType" : "sendEmail",
"email" : {
"to" : "speakers@jdayflorida.com",
"from" : "chad@chadwindnagle.com",
"subject": "Hello speakers!",
"body" : "Can't wait to see you all."
}
}
MESSAGE
EXAMPLE
Chad Windnagle - @drmmr763

14. THE QUEUE /
TUBE
The message queue, or tube, contains a collection
of messages that need to be processed by the
queue system.
Chad Windnagle - @drmmr763
Newest Oldest
Typical Processing Order

15. SERVICE
WORKER
The service worker is code that is responsible for
executing the tasks on the queue.
Chad Windnagle - @drmmr763
Newest Oldest
Typical Processing Order

16. EXECUTION
CYCLE
Chad Windnagle - @drmmr763
User Requests
Queueable Task
Message with Data
Generated
Message Added
To Queue
Server Boots
Service Worker
Service Worker
Performs Task
Message Removed
From Queue

17. QUEUE SYSTEM
VENDORS

18. VARIOUS
VENDORS
Beanstalkd
Laravel
Redis
Gearman
RabbitMQ
AmazonSQS
IronMQ
Symfony
Chad Windnagle - @drmmr763
Beanstalkd

19. BEANSTALKD
BENEFITS
Chad Windnagle - @drmmr763
RAM Based Data Storage - High Performance!
HTTP Protocol - Allows Multi-Server Set Up
PHP Composer Library Available - Fast
Development
Community Resources (Blogs, Documentation,
Etc.)

20. Chad Windnagle - @drmmr763
STACK OVERVIEW
PHP Application (Joomla!)
PHP Library (Pheanstalk)
Beanstalkd Service
Queue Jobs
Queue / Execute Job

21. Chad Windnagle - @drmmr763
apt-get install beanstalkd
INSTALLING
BEANSTALKD

22. Chad Windnagle - @drmmr763
php composer.phar require
pda/pheanstalk
INSTALL PHEANSTALK

23. RUNNING
BEANSTALKD
Chad Windnagle - @drmmr763
./beanstalkd -l 127.0.0.1 -p 11300
Options:
-l Specify the address to listen on
-p specify the port to listen on

24. CURRENT STATUS?
Chad Windnagle - @drmmr763
Beanstalkd is running:
- It can add jobs to a queue
- Returns job details to service worker
- Jobs can be removed from the queue

25. ADD MESSAGE TO
QUEUE
Chad Windnagle - @drmmr763
<?php
use PheanstalkPheanstalk;
// connect to Beanstalkd Service
$pheanstalk = new Pheanstalk(‘127.0.0.1');
// add the job to the queue
$pheanstalk->put(‘a special message’);

26. GET MESSAGE FROM
QUEUE
Chad Windnagle - @drmmr763
<?php
use PheanstalkPheanstalk;
// connect to Beanstalkd Service
$pheanstalk = new Pheanstalk(‘127.0.0.1');
// get next available job from queue
$job = $pheanstalk
->watch(‘default’) // specific tube
->reserve(); // job in progress

27. EXECUTE A TASK
Chad Windnagle - @drmmr763
<?php
// get the message from the queued job
$jobData = $job->getData();
// prints “a special message”
print $jobData;
// remove job from queue
$job->delete();

28. MORE USEFUL
EXAMPLE
Chad Windnagle - @drmmr763
$message = [
‘messageType => ‘sendEmail’,
‘email’ => [
‘to’ => ‘speakers@jdayflorida.com’,
‘from’ => ‘chad@chadwindnagle.com’,
‘subject => ‘Hello speakers!’,
‘body’ => ‘Please come to my session!’
]
];

29. MORE USEFUL
EXAMPLE
Chad Windnagle - @drmmr763
<?php
// add the job to the queue
$pheanstalk->put(json_encode($message));
Application code would add tasks to the queue

30. MORE USEFUL
EXAMPLE
Chad Windnagle - @drmmr763
// get the message from the queued job
$jobData = json_decode($job->getData());
$messageType = $jobData[‘messageType’];
if ($messageType == ‘sendEmail’ {
// “to”, “from”, “subject”, & “body”
$this->sendEmail($jobData[‘email’]);
}
Service worker executes based on message type

31. TRIGGERING SERVICE
WORKER
Chad Windnagle - @drmmr763
php ./service-worker.php

32. Chad Windnagle - @drmmr763
github.com/drmmr763/jdayflorida
PHP Project
Composer Based
Uses Pheanstalk
Uses Beanstalkd
Command Line Utilities
PHP Class Message
Types
EXAMPLE GITHUB CODE

33. ANY QUESTIONS?
Chad Windnagle - @drmmr763

34. ICON CREDITS
Kirby Wu
Schmidt
Sergey
IconSpher
e
Carl
Holdnerss
ProSymbol
s
Above icons licensed via Creative Commons, sourced from TheNounProject.com
Gregor
Cresnar
Olivia
Stoian
Rudolf
Horaczek
ProSymbol
s
icon 54
ChangHoo
n
Baek
Daniel
Baker

35. Chad Windnagle - @drmmr763
THANK YOU!Code:
github.com/drmmr763/jdayflorida
Follow Me: @drmmr763
Slides available on slideshare