Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Queue your work

5,305 views

Published on

Uncon talk from PHPBenelux 2014.

Published in: Technology

Queue your work

  1. 1. Queue your work Jurian Sluiman - uncon session PHPBenelux 2014
  2. 2. About me • Jurian Sluiman • Founder @ soflomo (Delft, The Netherlands) • Web development (eCommerce, health care) • Zend Framework 2 contributor • Blogger: http://juriansluiman.nl • Twitter: @juriansluiman
  3. 3. Queue systems • Execute tasks in background • Return your response fast! request Server response
  4. 4. Queue systems • Execute tasks in background • Return your response fast! request response Server (Producer) Queue • Producer: piece of code pushing new jobs • Worker: piece of code consuming jobs Worker
  5. 5. Queue systems request response Worker Server (Producer) Queue request response Server (Producer) Worker request response Worker Server (Producer) Queue Worker
  6. 6. So why? Advantages Disadvantages • Asynchronous • Asynchronous • Resilience • Complexity (is it?) • Scalable • Atomic
  7. 7. Types 1.Dedicated job queues • Gearman, beanstalkd, Celery 2.Message queues • RabbitMQ, ZeroMQ, ActiveMQ 3.SaaS queues • Amazon SQS, IronMQ 4.Databases • Redis, (My)SQL A list of most of the queues: http://queues.io
  8. 8. Job queues 1.Gearman • Widely known • PECL extension available (http://php.net/gearman) 2.Beanstalkd • Small footprint • Very easy setup
  9. 9. Gearman 1. Run task in background // producer $client = new GearmanClient(); $client->addServer(); // 127.0.0.1:4730 $client->doBackground('email', $workload); // worker $worker = new GearmanWorker(); $worker->addServer(); $worker->addFunction('email', function($job) { $params = $job->getWorkLoad(); mail($params['to'], $params['subject'], $params['msg']); }; while($worker->work());
  10. 10. Gearman 2. Normal tasks $client = new GearmanClient(); $client->addServer(); $result = $client->doNormal('generate-key'); // use $result
  11. 11. Gearman 2. Normal tasks $client = new GearmanClient(); $client->addServer(); $result = $client->doNormal('generate-key'); // use $result 3. Multiple servers $servers = array('192.168.1.200', '192.168.1.201'); shuffle($servers); $client = new GearmanClient(); $client->addServers(implode(',', $servers));
  12. 12. Gearman Advantages Disadvantages • Multi-tenant • Requires compilation! • Job status • Pre-defined functions • Priorities • Opt-in persistency • Persistent
  13. 13. Beanstalkd 1. Run task in background // producer $client = new Pheanstalk_Pheanstalk('127.0.0.1'); $client->put($data); // worker $worker = new Pheanstalk_Pheanstalk('127.0.0.1'); while($job = $worker->reserve()) { $data = $job->getData(); $function = $data['function']; $args = $data['args']; call_user_func_array($function, $args); }
  14. 14. Beanstalkd 2. Priority & delayed task $client = new Pheanstalk_Pheanstalk('127.0.0.1'); $prio = Pheanstalk_PheanstalkInterface::DEFAULT_PRIORITY; $delay = 5 * 60; $client->put($data, $prio, $delay);
  15. 15. Beanstalkd 2. Priority & delayed task $client = new Pheanstalk_Pheanstalk('127.0.0.1'); $prio = Pheanstalk_PheanstalkInterface::DEFAULT_PRIORITY; $delay = 5 * 60; $client->put($data, $prio, $delay); 3. Using tubes // producer $client = new Pheanstalk_Pheanstalk('127.0.0.1'); $client->putInTube('image', $data); // worker $client = new Pheanstalk_Pheanstalk('127.0.0.1'); $job = $client->reserveFromTube('image');
  16. 16. Beanstalkd Advantages Disadvantages • Fast! • Single-tenant • Priorities & delays • Fairly unknown • Job life cycle • Time-to-run • Persistent • Flexible payload
  17. 17. How fast? Gearman • • • 1mln jobs Push:~205 seconds ~4900 ops/sec Pull: ~ 180 seconds ~ 5500 ops /sec Beanstalkd • • • 1mln jobs Push:~120 seconds ~ 8300 ops/sec Pull: ~ 150 seconds ~ 6600 ops/sec Tested on a 2.5GHz VPS, 1 core, 1GB RAM – http://gist.github.com/juriansluiman/8593421
  18. 18. Message queues 1.RabbitMQ • AMQP implementation (Message-Oriented-Middleware) • High availability, multi-tenancy, persistency 2.ØMQ “ZeroMQ” • Higher throughput than TCP • Flexible socket library, create your own patterns
  19. 19. ØMQ Example // producer $context = new ZMQContext(); $producer = new ZMQSocket($context, ZMQ::SOCKET_REQ); $producer->bind(“tcp://localhost:5555”); $producer->send($payload); // worker $context = new ZMQContext(); $worker = new ZMQSocket($context, ZMQ::SOCKET_REP); $worker->connect(“tcp://*:5555”); while(true) { $payload = $worker->recv(); }
  20. 20. Message queues Advantages Disadvantages • Extremely flexible • Extremely flexible • AMQP • DIY • Message broker • No “best way” • Extreme & advanced • Routing, pub/sub, ventilators, channels
  21. 21. Software-as-a-Service 1.Amazon Simple Queue Service (SQS) • Useful within EC2 instances • http://aws.amazon.com/sqs 2.IronMQ • They say they're better than SQS • http://iron.io/mq
  22. 22. Software-as-a-Service Advantages Disadvantages • No maintenance • HTTP latency • Easy to set-up • Distributed (SQS) • Easy to scale • Cost
  23. 23. Databases 1.Redis • Extremely light-weight key/value store • BRPOPLPUSH • Watch back Ross Tuck @ PHPBenelux 2013 2.(My)SQL • Only if you have to
  24. 24. Databases Advantages Disadvantages • Well known set-up • Not meant for jobs • OK for shared hosting • DIY
  25. 25. Queue abstraction layers 1.SlmQueue (April 2012) 2.php-queue (September 2012) 3.BBQ (June 2013) 4.Laravel Queue component
  26. 26. SlmQueue • Supports Beanstalkd, SQS and Doctrine • Redis and Gearman support coming • Integrated with ZF2, but not required • Packagist: slm/queue + slm/queue-beanstalkd • GitHub: http://github.com/juriansluiman/SlmQueue
  27. 27. SlmQueue SlmQueueJobJobInterface interface JobInterface { public function execute(); }
  28. 28. SlmQueue SlmQueueJobJobInterface interface JobInterface { public function execute(); } SlmQueueQueueQueueInterface interface QueueInterface { public function push(JobInterface $job, $options); public function pop(); }
  29. 29. SlmQueue SlmQueueWorkerWorkerInterface interface WorkerInterface { public function processQueue($name, $options); public function processJob(JobInterface $job); }
  30. 30. Workers Run via ZF2 app php public/index.php queue beanstalkd default Configuration • Time-out for blocking calls • Maximum number of cycles • Maximum memory usage • Signal handlers for SIGTERM and SIGINT
  31. 31. Dependency injection MyModuleJobEmail class Email extends AbstractJob { protected $transport; public function __construct(Transport $transport) { $this->transport = $transport; } public function execute() { $payload = $this->getContent(); $message = new Message; $message->setTo($payload['to']); $message->setMessage($payload['message']); } } $this->transport->send($message);
  32. 32. Dependency injection MyModuleFactoryEmailJobFactory use MyModuleJobEmail as EmailJob; class EmailJobFactory implements FactoryInterface { public function createService(ServiceLocator $sl) { $transport = $sl->get('MyEmailTransport'); return new EmailJob($transport); } } module.config.php 'slm_queue' => [ 'job_manager' => [ 'MyEmailJob' => 'MyModuleFactoryEmailJobFactory' ] ]
  33. 33. Dependency injection Example: ZF2 Controller public function fooAction() { $payload = array( 'to' => 'jurian@juriansluiman.nl', 'message' => 'Hi there!', ); $this->queue('default') ->push('MyEmailJob', $payload); }
  34. 34. Queue aware jobs MyModuleJobFoo class Foo extends AbstractJob implements QueueAwareInterface { use QueueAwareTrait; public function execute() { // work here } } $job = new BarJob(); $this->getQueue()->push($job);
  35. 35. Pro-tips 1.Start experimenting now! 2.Atomic jobs 3.Log everything 4.Use a control system like supervisord
  36. 36. Questions? Jurian Sluiman - uncon session PHPBenelux 2014
  37. 37. Jobs in services MyModuleServiceFoo class Foo { protected $queue; public function __construct(QueueInterface $queue) { $this->queue = $queue; } public function doSomething() { // work here $job = new BarJob; $this->queue->push($job); } }
  38. 38. Lazy services class Buzzer { public function __construct() { sleep(5); } } public function buzz() { // do something } Lazy services with a Proxy pattern by Marco Pivetta (Ocramius)
  39. 39. Lazy services class BuzzerProxy extends Buzzer { private $sl; private $instance; public function __construct(ServiceLocator $sl) { $this->sl = $sl; } private function initialize() { $this->initialized = true; $this->original = $this->sl->get('Buzzer'); } } public function buzz() { if (!$this->initialized) { $this->initialize(); } return $this->instance->buzz(); }

×