Queue your work
Upcoming SlideShare
Loading in...5
×
 

Queue your work

on

  • 1,350 views

Uncon talk from PHPBenelux 2014.

Uncon talk from PHPBenelux 2014.

Statistics

Views

Total Views
1,350
Views on SlideShare
1,334
Embed Views
16

Actions

Likes
6
Downloads
9
Comments
0

3 Embeds 16

http://librosweb.es 9
https://twitter.com 6
http://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Queue your work Queue your work Presentation Transcript

  • Queue your work Jurian Sluiman - uncon session PHPBenelux 2014
  • About me • Jurian Sluiman • Founder @ soflomo (Delft, The Netherlands) • Web development (eCommerce, health care) • Zend Framework 2 contributor • Blogger: http://juriansluiman.nl • Twitter: @juriansluiman
  • Queue systems • Execute tasks in background • Return your response fast! request Server response
  • Queue systems • Execute tasks in background • Return your response fast! request response Server (Producer) Queue • Producer: piece of code pushing new jobs • Worker: piece of code consuming jobs Worker
  • Queue systems request response Worker Server (Producer) Queue request response Server (Producer) Worker request response Worker Server (Producer) Queue Worker
  • So why? Advantages Disadvantages • Asynchronous • Asynchronous • Resilience • Complexity (is it?) • Scalable • Atomic
  • Types 1.Dedicated job queues • Gearman, beanstalkd, Celery 2.Message queues • RabbitMQ, ZeroMQ, ActiveMQ 3.SaaS queues • Amazon SQS, IronMQ 4.Databases • Redis, (My)SQL A list of most of the queues: http://queues.io
  • Job queues 1.Gearman • Widely known • PECL extension available (http://php.net/gearman) 2.Beanstalkd • Small footprint • Very easy setup
  • Gearman 1. Run task in background // producer $client = new GearmanClient(); $client->addServer(); // 127.0.0.1:4730 $client->doBackground('email', $workload); // worker $worker = new GearmanWorker(); $worker->addServer(); $worker->addFunction('email', function($job) { $params = $job->getWorkLoad(); mail($params['to'], $params['subject'], $params['msg']); }; while($worker->work());
  • Gearman 2. Normal tasks $client = new GearmanClient(); $client->addServer(); $result = $client->doNormal('generate-key'); // use $result
  • Gearman 2. Normal tasks $client = new GearmanClient(); $client->addServer(); $result = $client->doNormal('generate-key'); // use $result 3. Multiple servers $servers = array('192.168.1.200', '192.168.1.201'); shuffle($servers); $client = new GearmanClient(); $client->addServers(implode(',', $servers));
  • Gearman Advantages Disadvantages • Multi-tenant • Requires compilation! • Job status • Pre-defined functions • Priorities • Opt-in persistency • Persistent
  • Beanstalkd 1. Run task in background // producer $client = new Pheanstalk_Pheanstalk('127.0.0.1'); $client->put($data); // worker $worker = new Pheanstalk_Pheanstalk('127.0.0.1'); while($job = $worker->reserve()) { $data = $job->getData(); $function = $data['function']; $args = $data['args']; call_user_func_array($function, $args); }
  • Beanstalkd 2. Priority & delayed task $client = new Pheanstalk_Pheanstalk('127.0.0.1'); $prio = Pheanstalk_PheanstalkInterface::DEFAULT_PRIORITY; $delay = 5 * 60; $client->put($data, $prio, $delay);
  • Beanstalkd 2. Priority & delayed task $client = new Pheanstalk_Pheanstalk('127.0.0.1'); $prio = Pheanstalk_PheanstalkInterface::DEFAULT_PRIORITY; $delay = 5 * 60; $client->put($data, $prio, $delay); 3. Using tubes // producer $client = new Pheanstalk_Pheanstalk('127.0.0.1'); $client->putInTube('image', $data); // worker $client = new Pheanstalk_Pheanstalk('127.0.0.1'); $job = $client->reserveFromTube('image');
  • Beanstalkd Advantages Disadvantages • Fast! • Single-tenant • Priorities & delays • Fairly unknown • Job life cycle • Time-to-run • Persistent • Flexible payload
  • How fast? Gearman • • • 1mln jobs Push:~205 seconds ~4900 ops/sec Pull: ~ 180 seconds ~ 5500 ops /sec Beanstalkd • • • 1mln jobs Push:~120 seconds ~ 8300 ops/sec Pull: ~ 150 seconds ~ 6600 ops/sec Tested on a 2.5GHz VPS, 1 core, 1GB RAM – http://gist.github.com/juriansluiman/8593421
  • Message queues 1.RabbitMQ • AMQP implementation (Message-Oriented-Middleware) • High availability, multi-tenancy, persistency 2.ØMQ “ZeroMQ” • Higher throughput than TCP • Flexible socket library, create your own patterns
  • ØMQ Example // producer $context = new ZMQContext(); $producer = new ZMQSocket($context, ZMQ::SOCKET_REQ); $producer->bind(“tcp://localhost:5555”); $producer->send($payload); // worker $context = new ZMQContext(); $worker = new ZMQSocket($context, ZMQ::SOCKET_REP); $worker->connect(“tcp://*:5555”); while(true) { $payload = $worker->recv(); }
  • Message queues Advantages Disadvantages • Extremely flexible • Extremely flexible • AMQP • DIY • Message broker • No “best way” • Extreme & advanced • Routing, pub/sub, ventilators, channels
  • Software-as-a-Service 1.Amazon Simple Queue Service (SQS) • Useful within EC2 instances • http://aws.amazon.com/sqs 2.IronMQ • They say they're better than SQS • http://iron.io/mq
  • Software-as-a-Service Advantages Disadvantages • No maintenance • HTTP latency • Easy to set-up • Distributed (SQS) • Easy to scale • Cost
  • Databases 1.Redis • Extremely light-weight key/value store • BRPOPLPUSH • Watch back Ross Tuck @ PHPBenelux 2013 2.(My)SQL • Only if you have to
  • Databases Advantages Disadvantages • Well known set-up • Not meant for jobs • OK for shared hosting • DIY
  • Queue abstraction layers 1.SlmQueue (April 2012) 2.php-queue (September 2012) 3.BBQ (June 2013) 4.Laravel Queue component
  • SlmQueue • Supports Beanstalkd, SQS and Doctrine • Redis and Gearman support coming • Integrated with ZF2, but not required • Packagist: slm/queue + slm/queue-beanstalkd • GitHub: http://github.com/juriansluiman/SlmQueue
  • SlmQueue SlmQueueJobJobInterface interface JobInterface { public function execute(); }
  • SlmQueue SlmQueueJobJobInterface interface JobInterface { public function execute(); } SlmQueueQueueQueueInterface interface QueueInterface { public function push(JobInterface $job, $options); public function pop(); }
  • SlmQueue SlmQueueWorkerWorkerInterface interface WorkerInterface { public function processQueue($name, $options); public function processJob(JobInterface $job); }
  • Workers Run via ZF2 app php public/index.php queue beanstalkd default Configuration • Time-out for blocking calls • Maximum number of cycles • Maximum memory usage • Signal handlers for SIGTERM and SIGINT
  • Dependency injection MyModuleJobEmail class Email extends AbstractJob { protected $transport; public function __construct(Transport $transport) { $this->transport = $transport; } public function execute() { $payload = $this->getContent(); $message = new Message; $message->setTo($payload['to']); $message->setMessage($payload['message']); } } $this->transport->send($message);
  • Dependency injection MyModuleFactoryEmailJobFactory use MyModuleJobEmail as EmailJob; class EmailJobFactory implements FactoryInterface { public function createService(ServiceLocator $sl) { $transport = $sl->get('MyEmailTransport'); return new EmailJob($transport); } } module.config.php 'slm_queue' => [ 'job_manager' => [ 'MyEmailJob' => 'MyModuleFactoryEmailJobFactory' ] ]
  • Dependency injection Example: ZF2 Controller public function fooAction() { $payload = array( 'to' => 'jurian@juriansluiman.nl', 'message' => 'Hi there!', ); $this->queue('default') ->push('MyEmailJob', $payload); }
  • Queue aware jobs MyModuleJobFoo class Foo extends AbstractJob implements QueueAwareInterface { use QueueAwareTrait; public function execute() { // work here } } $job = new BarJob(); $this->getQueue()->push($job);
  • Pro-tips 1.Start experimenting now! 2.Atomic jobs 3.Log everything 4.Use a control system like supervisord
  • Questions? Jurian Sluiman - uncon session PHPBenelux 2014
  • Jobs in services MyModuleServiceFoo class Foo { protected $queue; public function __construct(QueueInterface $queue) { $this->queue = $queue; } public function doSomething() { // work here $job = new BarJob; $this->queue->push($job); } }
  • Lazy services class Buzzer { public function __construct() { sleep(5); } } public function buzz() { // do something } Lazy services with a Proxy pattern by Marco Pivetta (Ocramius)
  • Lazy services class BuzzerProxy extends Buzzer { private $sl; private $instance; public function __construct(ServiceLocator $sl) { $this->sl = $sl; } private function initialize() { $this->initialized = true; $this->original = $this->sl->get('Buzzer'); } } public function buzz() { if (!$this->initialized) { $this->initialize(); } return $this->instance->buzz(); }