The document discusses using queue systems to execute tasks asynchronously in the background to improve application performance and scalability. It provides an overview of different types of queue systems including dedicated job queues like Gearman and Beanstalkd, message queues like RabbitMQ, and software-as-a-service queues like Amazon SQS. It also discusses using databases like Redis as queues. The document then dives deeper into examples of using Gearman and Beanstalkd in PHP applications and compares their performance. It also discusses using queue abstraction layers and best practices for queueing jobs.

1. Queue your work
Jurian Sluiman - uncon session PHPBenelux 2014

2. About me
•
Jurian Sluiman
•
Founder @ soflomo (Delft, The Netherlands)
•
Web development (eCommerce, health care)
•
Zend Framework 2 contributor
•
Blogger: http://juriansluiman.nl
•
Twitter: @juriansluiman

3. Queue systems
•
Execute tasks in background
•
Return your response fast!
request
Server
response

4. Queue systems
•
Execute tasks in background
•
Return your response fast!
request
response
Server
(Producer)
Queue
•
Producer: piece of code pushing new jobs
•
Worker:
piece of code consuming jobs
Worker

5. Queue systems
request
response
Worker
Server
(Producer)
Queue
request
response
Server
(Producer)
Worker
request
response
Worker
Server
(Producer)
Queue
Worker

6. So why?
Advantages
Disadvantages
•
Asynchronous
•
Asynchronous
•
Resilience
•
Complexity (is it?)
•
Scalable
•
Atomic

7. Types
1.Dedicated job queues
•
Gearman, beanstalkd, Celery
2.Message queues
•
RabbitMQ, ZeroMQ, ActiveMQ
3.SaaS queues
•
Amazon SQS, IronMQ
4.Databases
•
Redis, (My)SQL
A list of most of the queues: http://queues.io

8. Job queues
1.Gearman
•
Widely known
•
PECL extension available (http://php.net/gearman)
2.Beanstalkd
•
Small footprint
•
Very easy setup

9. Gearman
1. Run task in background
// producer
$client = new GearmanClient();
$client->addServer();
// 127.0.0.1:4730
$client->doBackground('email', $workload);
// worker
$worker = new GearmanWorker();
$worker->addServer();
$worker->addFunction('email', function($job) {
$params = $job->getWorkLoad();
mail($params['to'], $params['subject'], $params['msg']);
};
while($worker->work());

10. Gearman
2. Normal tasks
$client = new GearmanClient();
$client->addServer();
$result = $client->doNormal('generate-key');
// use $result

11. Gearman
2. Normal tasks
$client = new GearmanClient();
$client->addServer();
$result = $client->doNormal('generate-key');
// use $result
3. Multiple servers
$servers = array('192.168.1.200', '192.168.1.201');
shuffle($servers);
$client = new GearmanClient();
$client->addServers(implode(',', $servers));

12. Gearman
Advantages
Disadvantages
•
Multi-tenant
•
Requires compilation!
•
Job status
•
Pre-defined functions
•
Priorities
•
Opt-in persistency
•
Persistent

13. Beanstalkd
1. Run task in background
// producer
$client = new Pheanstalk_Pheanstalk('127.0.0.1');
$client->put($data);
// worker
$worker = new Pheanstalk_Pheanstalk('127.0.0.1');
while($job = $worker->reserve()) {
$data
= $job->getData();
$function = $data['function'];
$args
= $data['args'];
call_user_func_array($function, $args);
}

14. Beanstalkd
2. Priority & delayed task
$client = new Pheanstalk_Pheanstalk('127.0.0.1');
$prio
= Pheanstalk_PheanstalkInterface::DEFAULT_PRIORITY;
$delay = 5 * 60;
$client->put($data, $prio, $delay);

15. Beanstalkd
2. Priority & delayed task
$client = new Pheanstalk_Pheanstalk('127.0.0.1');
$prio
= Pheanstalk_PheanstalkInterface::DEFAULT_PRIORITY;
$delay = 5 * 60;
$client->put($data, $prio, $delay);
3. Using tubes
// producer
$client = new Pheanstalk_Pheanstalk('127.0.0.1');
$client->putInTube('image', $data);
// worker
$client = new Pheanstalk_Pheanstalk('127.0.0.1');
$job = $client->reserveFromTube('image');

16. Beanstalkd
Advantages
Disadvantages
•
Fast!
•
Single-tenant
•
Priorities & delays
•
Fairly unknown
•
Job life cycle
•
Time-to-run
•
Persistent
•
Flexible payload

17. How fast?
Gearman
•
•
•
1mln jobs
Push:~205 seconds
~4900 ops/sec
Pull: ~ 180 seconds
~ 5500 ops /sec
Beanstalkd
•
•
•
1mln jobs
Push:~120 seconds
~ 8300 ops/sec
Pull: ~ 150 seconds
~ 6600 ops/sec
Tested on a 2.5GHz VPS, 1 core, 1GB RAM – http://gist.github.com/juriansluiman/8593421

18. Message queues
1.RabbitMQ
•
AMQP implementation (Message-Oriented-Middleware)
•
High availability, multi-tenancy, persistency
2.ØMQ “ZeroMQ”
•
Higher throughput than TCP
•
Flexible socket library, create your own patterns

19. ØMQ
Example
// producer
$context = new ZMQContext();
$producer = new ZMQSocket($context, ZMQ::SOCKET_REQ);
$producer->bind(“tcp://localhost:5555”);
$producer->send($payload);
// worker
$context = new ZMQContext();
$worker = new ZMQSocket($context, ZMQ::SOCKET_REP);
$worker->connect(“tcp://*:5555”);
while(true) {
$payload = $worker->recv();
}

20. Message queues
Advantages
Disadvantages
•
Extremely flexible
•
Extremely flexible
•
AMQP
•
DIY
•
Message broker
•
No “best way”
•
Extreme & advanced
•
Routing, pub/sub,
ventilators, channels

21. Software-as-a-Service
1.Amazon Simple Queue Service (SQS)
•
Useful within EC2 instances
•
http://aws.amazon.com/sqs
2.IronMQ
•
They say they're better than SQS
•
http://iron.io/mq

22. Software-as-a-Service
Advantages
Disadvantages
•
No maintenance
•
HTTP latency
•
Easy to set-up
•
Distributed (SQS)
•
Easy to scale
•
Cost

23. Databases
1.Redis
•
Extremely light-weight key/value store
•
BRPOPLPUSH
•
Watch back Ross Tuck @ PHPBenelux 2013
2.(My)SQL
•
Only if you have to

24. Databases
Advantages
Disadvantages
•
Well known set-up
•
Not meant for jobs
•
OK for shared hosting
•
DIY

25. Queue abstraction layers
1.SlmQueue (April 2012)
2.php-queue (September 2012)
3.BBQ (June 2013)
4.Laravel Queue component

26. SlmQueue
•
Supports Beanstalkd, SQS and Doctrine
•
Redis and Gearman support coming
•
Integrated with ZF2, but not required
•
Packagist: slm/queue + slm/queue-beanstalkd
•
GitHub: http://github.com/juriansluiman/SlmQueue

27. SlmQueue
SlmQueueJobJobInterface
interface JobInterface
{
public function execute();
}

28. SlmQueue
SlmQueueJobJobInterface
interface JobInterface
{
public function execute();
}
SlmQueueQueueQueueInterface
interface QueueInterface
{
public function push(JobInterface $job, $options);
public function pop();
}

29. SlmQueue
SlmQueueWorkerWorkerInterface
interface WorkerInterface
{
public function processQueue($name, $options);
public function processJob(JobInterface $job);
}

30. Workers
Run via ZF2 app
php public/index.php queue beanstalkd default
Configuration
•
Time-out for blocking calls
•
Maximum number of cycles
•
Maximum memory usage
•
Signal handlers for SIGTERM and SIGINT

31. Dependency injection
MyModuleJobEmail
class Email extends AbstractJob
{
protected $transport;
public function __construct(Transport $transport)
{
$this->transport = $transport;
}
public function execute()
{
$payload = $this->getContent();
$message = new Message;
$message->setTo($payload['to']);
$message->setMessage($payload['message']);
}
}
$this->transport->send($message);

32. Dependency injection
MyModuleFactoryEmailJobFactory
use MyModuleJobEmail as EmailJob;
class EmailJobFactory implements FactoryInterface
{
public function createService(ServiceLocator $sl)
{
$transport = $sl->get('MyEmailTransport');
return new EmailJob($transport);
}
}
module.config.php
'slm_queue' => [
'job_manager' => [
'MyEmailJob' => 'MyModuleFactoryEmailJobFactory'
]
]

33. Dependency injection
Example: ZF2 Controller
public function fooAction()
{
$payload = array(
'to'
=> 'jurian@juriansluiman.nl',
'message' => 'Hi there!',
);
$this->queue('default')
->push('MyEmailJob', $payload);
}

34. Queue aware jobs
MyModuleJobFoo
class Foo extends AbstractJob implements QueueAwareInterface
{
use QueueAwareTrait;
public function execute()
{
// work here
}
}
$job = new BarJob();
$this->getQueue()->push($job);

35. Pro-tips
1.Start experimenting now!
2.Atomic jobs
3.Log everything
4.Use a control system like supervisord

36. Questions?
Jurian Sluiman - uncon session PHPBenelux 2014

37. Jobs in services
MyModuleServiceFoo
class Foo
{
protected $queue;
public function __construct(QueueInterface $queue)
{
$this->queue = $queue;
}
public function doSomething()
{
// work here
$job = new BarJob;
$this->queue->push($job);
}
}

38. Lazy services
class Buzzer
{
public function __construct()
{
sleep(5);
}
}
public function buzz()
{
// do something
}
Lazy services with a Proxy pattern by Marco Pivetta (Ocramius)

39. Lazy services
class BuzzerProxy extends Buzzer
{
private $sl;
private $instance;
public function __construct(ServiceLocator $sl)
{
$this->sl = $sl;
}
private function initialize()
{
$this->initialized = true;
$this->original = $this->sl->get('Buzzer');
}
}
public function buzz()
{
if (!$this->initialized) { $this->initialize(); }
return $this->instance->buzz();
}