A document about queues discusses what queues are, why they are used, common use cases, implementation patterns, protocols, considerations when implementing queues, and how to handle issues that may arise. Queues act as buffers that allow different applications or systems to communicate asynchronously by passing messages. They help decouple components, distribute load, and improve reliability and user experience. Common examples of messages that may be queued include emails, images, videos, and IoT data.

1. DOYOU QUEUE?
ZendCon 2015

2. I AM MIKE WILLBANKS
• Father, Husband, Developer
• VP of Development at
Packet Power
• Twitter: @mwillbanks

3. Task
Producer ConsumerMessages
Messages
Messages
WHAT IS A QUEUE?
• Pub/Sub
• FIFO buffer
• Push / Pull
• A way to communicate
between applications / systems.
• A way to decouple
components.
• A way to offload work.

4. WHY QUEUE?

5. THE LONG RUNNING CASE
Request Long-process
Send ResponseResponse

6. THE INTEROP CASE
Request Conditioning
Service ResponseResponse
Web Service Call
Send Response

7. NOTYOUR DATABASE
Request Process
Response Send Response
DB

8. SO WHY QUEUE
• User experience
• System Security
• Load Distribution
• System Reliability

9. IN PRACTICE
You’ve seen this before…

13. WHATYOU MIGHT QUEUE

14. COMMUNICATIONS
• Emails
• SMS
• Push Notifications

15. IMAGES
• Conversions
• Resize
• Thumbnail
• Watermark

16. VIDEO
• Conversion
• Resampling
• Audio overlay

17. IOT
• Receive messages from
devices and process
responses

18. PATTERNS

19. POINTTO POINT
Point to Point
Channel
Receiver
Sender

20. PUBLISH / SUBSCRIBE
Publiser Subscribe Channel
Subscriber
Subscriber
Subscriber

21. MESSAGE BUS
Application
ApplicationApplication
Message Bus

22. PIPELINE
Sender
Receiver
Point to Point
Channel
Receiver
Point to Point
Channel

23. INVALID MESSAGE
Channel
Receiver
Sender
X
Invalid Message
Channel

24. PROTOCOLS
Or implementations for that matter.

25. AMQP
• AMQP Working Group
(Community andVendor)
• Platform agnostic protocol.
• Completely open,
interoperable and broadly
applicable.
• Many severs available and
many client libraries.

27. STOMP
• Simple protocol
• Behaviors follow very
simple commands.
• Most message queues can
communicate over STOMP.

28. Connect Send Disconnect
/queue/
msg
P
H
P
S
T
O
M
P
S
E
R
V
E
R
Connect Subscribe Disconnect
/queue/
msg
Read
Ack

29. SQS
• Fairly simple protocol
• Supports delays, timers, and
multiple policies.

31. SPECIAL
PURPOSE
Many queue implementations
exist that don’t necessarily sit
under standards…

32. XMPP
• Best for real-time data.
• Leveraging pub/sub can
turn it into more of a
generic message system.
• Multiple libraries available.

33. SOCKET.IO
• New comer
• Real-time bidirectional
event-based communication
• Largely leverages pub/sub

34. ZEROMQ
• The ultimate in message
queue flexibility.
• Socket library that acts as a
concurrency framework.

35. GEARMAN
• Application framework for
farming out work.
• Job sever for asynchronous
or synchronous messages.

36. BEANSTALKD
• Asynchronous Job Queue
• Supports delays
• Many PHP clients exist

37. CONSIDERATIONS
How do we evaluate our options…

38. PULLVS. PUSH
• Always PULL, whenever
possible.
• Push eliminates several
benefits.

39. DURABILITY
• Memory residence
• Persistence
• Restart survival

40. SECURITY
• Authentication
• Queue permissions /
restrictions

41. DELIVERY
• Is the delivery guaranteed?
• If a message cannot be
delivered how it it handled?

42. ROUTING
• Multiple routing scenarios
• Fanout
• Direct
• Topic
• Broadcast

43. BATCHING
• Do it later but in bulk
(credit card processing)
• Can be done via scheduling
(cron)

44. RECEIPT
• Do you get an
acknowledgement of
receipt?

45. IMPLEMENTING QUEUES

46. STARTING POINTS

47. PUSHING MESSAGES
<?php
class UserService {
public function save($user) {
$this->db->save($user);
$stomp = new Stomp('tcp://localhost:61613');
$stomp->send('/queue/email', [
'to' => $user->getEmail(),
'subject' => 'Welcome',
'message' => 'Welcome',
'headers' => [],
]);
}
}

48. HANDLING MESSAGES
<?php
$stomp = new Stomp('tcp://localhost:61613');
$stomp->subscribe('/queue/email');
while (true) {
if (!$stomp->hasFrame()) {
sleep(2);
continue ;
}
$stomp->readFrame();
$email = json_decode($frame->body);
mail($email->to, $email->subject, $email->message, $email->headers);
}

49. MESSAGES

50. MESSAGE CONSIDERATIONS
• Message Format
• Message Contents

51. SERIALIZE
O:7:"Message":1:{s:7:"content";a:1:{s:3:"foo";a:1:{s:3:"bar";a:1:{i:0;s:
3:"baz";}}}}

52. WORKERS

53. WORKER CONSIDERATIONS
• Should do ONE thing and
ONE thing well.
• Should attempt to be as
quick as possible in handling
that type.
• Should be able to be scaled
horizontally.

54. HANDLING WORKERS
• Prevent Memory Leaks
• memory_get_usage
• Handle Signals!
• pcntl_signal

55. ABSTRACTIONS
We want to make this easy…

56. <?php
interface QueueInterface {
public function __construct(Stomp $stomp, $queue);
public function dispatch();
public function publish(array $message);
public function work(StompFrame $message);
}

57. <?php
class AbstractQueue implements QueueInterface {
protected $stomp;
protected $queue;
protected $signal;
public function __construct(Stomp $stomp, $queue) {
$this->stomp = $stomp;
$this->queue = $queue;
}
protected function prepare() {
if (php_sapi_name() != 'cli') {
throw new RuntimeException('You cannot dispatch outside of the CLI');
}
if (function_exists('pcntl_signal')) {
pcntl_signal(SIGTERM, array($this, 'signal'));
pcntl_signal(SIGINT, array($this, 'signal'));
pcntl_signal(SIGHUP, array($this, 'signal'));
}
}
protected function signal($signal) {
$this->signal = $signal;
}

58. public function dispatch() {
$this->prepare();
while (true) {
if ($this->signal) {
break ;
}
if (!$this->stomp->hasFrame()) {
$this->wait();
continue ;
}
$frame = $this->stomp->readFrame();
if ($this->validate($frame)) {
$this->work($frame);
}
$this->stomp->ack($frame);
}
}
protected function wait() {
sleep(1);
}
protected function validate(StompFrame $message) {
return false;
}
public function publish(array $message) {
return $this->stomp->send($this->queue, json_encode($message));
}

59. <?php
class EmailQueue extends AbstractQueue {
public function validate(StompFrame $message) {
if (!array_key_exists('to', $message)) {
return false;
}
return true;
}
public function work(StompFrame $message) {
$mail = json_decode($message);
mail($mail->to, $mail->subject, $mail->message);
}
}

60. BOOTSTRAPPING
Leverage your existing infrastructure as much as possible!

61. <?php
declare(ticks=1);
include 'vendor/autoload.php';
$app = ZendMvcApplication::init(include 'config/application.config.php');
$sm = $app->getServiceManager();
if (!isset($argv[1])) {
fprintf(STDERR, "Syntax: worker <name>nn");
exit(1);
}
$name = $argv[1];
try {
echo "Starting worker: " . $name . ' as ' . get_current_user() . PHP_EOL;
$consumer = $sm->get($name);
$consumer->dispatch();
} catch (Exception $e) {
fprintf(STDERR, "%sn", $msg);
exit(1);
}
$consumer = null;
echo 'Shutdown ' . $name . ' worker gracefully.' . PHP_EOL;
exit(0);

62. EVENTS

63. SERVICESTRIGGER EVENTS
<?php
use ZendEventManagerEventManagerAwareTrait;
class UserService
{
use EventManagerAwareTrait;
public function save($user) {
$this->db->save($user);
$this->getEventManager()->trigger('save', null, ['user' => $user]);
}
}

64. ATTACH EVENTS
<?php
use ZendServiceManagerServiceManager;
$sm = new ServiceManager();
$service = $sm->get('UserService');
$queue = $sm->get('EmailQueue');
$service->getEventManager()->attach('save', function($e) use ($queue) {
$params = $e->getParams();
$queue->publish([
'to' => $params['user']['email'],
'subject' => 'Welcome',
'message' => 'Welcome',
'headers' => [],
]);
});

65. HANDLING
PROGRESS
• Keep track by using a
generic handler
• Or, keep track via your
database.

66. TOOLING

67. SUPERVISOR
• Daemon that runs on the
server.
• Monitors programs and
keeps them running in case
of failure.
• Handles logging.

68. PAINLESS INSTALLATION
sudo easy_install supervisor
sudo echo_supervisord_conf > /etc/supervisord.conf
sudo service supervisor start

69. EXAMPLE PROGRAM
CONFIGURATION
[program:emailworker]
command=/usr/bin/php /var/www/worker "MyProjectQueueEmail"
process_name=%(program_name)s_%(process_num)d
numprocs=2
numprocs_start=2
user=www-data
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=10 ; number of secs prog must stay running (def. 10)
startretries=5 ; max # of serial start failures (default 3)
log_stdout=true ; if true, log program stdout (default true)
log_stderr=true ; if true, log program stderr (def false)
redirect_stderr=true ; if true, redirect stderr to stdout
stdout_logfile=/var/www/logs/worker-panoramaqueuekrpano.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=15

70. SUPERVISORD MULTI SERVER MONITORING
TOOL
https://github.com/mlazarov/supervisord-monitor

71. WHEN BADTHINGS HAPPEN

72. QUEUE BACKUP
• Most of all issues with
queues are that a queue has
backed up.

73. WORKER EXCEPTIONS
• Broken code causes
hardships, but recoverable!

74. • https://pixabay.com/en/autobahn-accident-germany-car-road-837643/
• https://pixabay.com/en/traffic-rent-a-car-traffic-jam-637118/
• https://pixabay.com/en/airplanes-line-runway-military-713662/
• https://pixabay.com/en/leo-animal-savannah-lioness-safari-350690/
• https://pixabay.com/en/user-top-view-office-keyboard-154199/
• https://pixabay.com/en/mechanics-engine-springs-mechanic-424130/
• https://pixabay.com/en/laughter-fun-happiness-boy-child-449781/
• https://pixabay.com/en/umbrellas-red-blue-patterns-205386/
• https://pixabay.com/en/spot-runs-start-la-stadion-862274/
• https://pixabay.com/en/artistic-the-art-of-abstraction-948588/
• https://pixabay.com/en/boots-work-boots-shoes-647035/
• https://pixabay.com/en/meerkat-watch-guard-cute-676944/
• https://pixabay.com/en/broken-window-hole-glass-damage-960188/
• https://pixabay.com/en/police-security-safety-protection-869216/
• https://pixabay.com/en/parcel-package-packaging-box-575623/
• https://pixabay.com/en/directory-signposts-trail-direction-494457/
• https://pixabay.com/en/cookies-chocolate-chip-food-dessert-28423/
• https://pixabay.com/en/phone-communication-call-select-735060/
• https://pixabay.com/en/receipt-note-paper-bill-document-575750/
• https://pixabay.com/en/tools-construct-craft-repair-864983/
• https://pixabay.com/en/moore-oklahoma-tornado-disaster-112781/
• https://pixabay.com/en/network-iot-internet-of-things-782707/
• https://pixabay.com/en/mobile-phone-smartphone-app-426559/
• https://pixabay.com/en/mr-site-build-crane-baukran-462074/
• https://pixabay.com/en/film-projector-movie-projector-738806/
• https://pixabay.com/en/calves-legs-human-standing-on-540519/
• https://pixabay.com/en/notebook-pages-opened-paper-note-820078/
• https://pixabay.com/en/letters-penpal-cards-leave-stack-566810/
• https://pixabay.com/en/spray-household-surface-shine-315164/
• https://pixabay.com/en/industry-crafts-gears-mechanical-94448/
• https://pixabay.com/en/hornet-wasp-insect-sting-macro-11514/
• https://pixabay.com/en/honey-bees-bees-hive-bee-hive-401238/
• https://pixabay.com/en/temple-china-door-handle-840526/
• https://pixabay.com/en/no-button-push-sign-icon-symbol-685042/
Image Credits
THANKYOU!
http://joind.in/talk/view/15538