Gearman jobqueue

1,776 views

Published on

0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,776
On SlideShare
0
From Embeds
0
Number of Embeds
49
Actions
Shares
0
Downloads
0
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Gearman jobqueue

  1. 1. Gearman Job Queue.Раскрываем недокументированные возможности
  2. 2. Синхронное выполнение задач Client Client Client Client Client Client Client Task
  3. 3. Без очередей — хаос!
  4. 4. Очереди к ограниченным ресурсам необходимы! ● Пред-расчет данных ● Операции импорта/экспорта данных ● Обработка логов системы ● Маркетинговый анализ базы данных ● Почтовые рассылки ● Map/Reduce ● и многое другое...
  5. 5. Выполнение задачи в асинхронном режиме Client Job Server Worker add job job handle execute success
  6. 6. Client Client Client Client Client Client Client Task managerWorker Worker Worker Worker Worker
  7. 7. About project Gearman Gearman = Manager 2005: originally Perl implementation by Danga Interactive 2008: rewrite in C by Brian Aker Used in: Digg (45+ servers, 400K jobs/day) Yahoo (60+ servers, 6M jobs/day) Xing.com Grooveshark ...
  8. 8. Преимущества и недостатки● Высокая скорость ● Мониторинг, администрирование● Масштабируемость ● Слабая документация● Балансировка нагрузки ● Работа с памятью● Параллельное и асинхронное ● Отличия API в различных версиях выполнение задач● Защита от потери очереди● Multi-language API
  9. 9. Gearman server monitoring Administrative Protocol > telnet localhost 4730 > Trying ::1... > Connected to localhost. > Escape character is ^]. > wokers FD IP-ADDRESS CLIENT-ID : FUNCTION > status FUNCTION TOTAL RUNNING AVAILABLE_WORKERS GearUp: Gearman based monitoring service https://launchpad.net/gearup Mod Gearman http://labs.consol.de/nagios/mod-gearman/
  10. 10. Client Client Client Client Client Client Client API Client API Gearman Gearman Worker API Worker APIWorker Worker Worker Worker Worker
  11. 11. client.php <?php # create our object $client= new GearmanClient(); # add the default server $client->addServer(); $workload = array( email => nobody@example.com , subject => the subject , message => the message ); # send email $client->doBackground( "send_mail" , json_encode($workload)); if ($client->returnCode() != GEARMAN_SUCCESS) { echo "bad return coden" ; exit; } ?>
  12. 12. worker.php <?php # create the worker $worker = new GearmanWorker(); # add the default job server (localhost) $worker->addServer(); # add the reverse function $worker->addFunction( "send_mail" , "sendMail" ); # start to worker listening for job submissions while ($worker->work()); function sendMail($job) { $arg = json_decode($job->workload(), true); mail($arg[ email], $arg[ subject ], $arg[ message ]); echo "email has been sentn" ; } ?>
  13. 13. Client Gearman Worker Task register tasks work() doBackground() run task execute success
  14. 14. worker.php <?php #... function sendMail($job) { $arg = json_decode($job->workload(), true); # connect to remote service $mail = new Mail($arg[email], $arg[subject], $arg[message]); # send email if (!$mail->send()){ # it needs to be solved } else { echo "email has been sentn"; } } ?>
  15. 15. Client Gearman Worker Task doBackground() run task execute failure ???
  16. 16. worker.php <?php #... function sendMail($job) { #... # send email if (!$mail->send()){ sleep(20); $client= new GearmanClient(); $client->addServer(); $client->doBackground("send_mail", $job->workload()); echo "error has happened n"; } else { echo "email has been sentn"; } } ?>
  17. 17. Client Gearman Worker Task doBackground() run task execute sleep(20) doBackground() failure run task execute sleep(20) doBackground() failure run task execute success
  18. 18. worker.php <?php #... function sendMail($job) { #... # send email if (!$mail->send()){ setCron("send_mail", $job->workload(), 20); echo "error has happened n"; } else { echo "email has been sentn"; } } ?>
  19. 19. Client Gearman Worker Task Cron doBackground() run task execute failure do postponed delay doBackground() run task execute do postponed failure delay doBackground() run task execute success do postponed
  20. 20. Solution Credit:— In ver. 0.20 added when_to_run tasks functionality— But public API doesnt support that functionality— Neither does PECL Gearman module— Documentation also is silent about it John Ewart Software Engineer
  21. 21. worker.php <?php #... function sendMail($job) { #... # send email if (!$mail->send()){ $client= new GearmanClient(); $client->addServer(); $time = time() + 20; $client->doSuspended("send_mail", $job->workload(), NULL, $time); echo "error has happened n" ; } else { echo "email has been sentn" ; } } ?>
  22. 22. Suspended task execution Client Gearman Worker Task doBackground() run task execute doSuspended() failure delay run task execute doSuspended() failure delay run task execute success
  23. 23. client.php <?php $client= new GearmanClient(); $client->addServer(); $workload = array(value => 1, limit_attempts => 3); $client->doBackground( "remote_procedure_call", json_encode($workload)); if ($client->returnCode() != GEARMAN_SUCCESS) { echo "bad return coden" ; exit; } ?>
  24. 24. worker.php <?php #... $worker->addFunction( remote_procedure_call , rpcJobHandler ); while ($worker->work()); function rpcJobHandler($job) { $args = json_decode($job->workload(), true); try { // Execute task remoteProcedureCall($args[ value]); } catch (Exception $e) { echo "Failure, reason: " . $e->getMessage() . "n"; } } function remoteProcedureCall(&$value) { // Accept value only less than 3 if ($value < 3) { $value++; throw new Exception ("Service is unavailable" ); } } ?>
  25. 25. worker.php — break of task loop<?php #... function rpcJobHandler($job) { $args = json_decode($job->workload(), true); try { remoteProcedureCall($args[ value]); } catch (Exception $e) { if (!isset($args[attempt ])) $args[ attempt ] = 0; if (empty($args[limit_attempts ]) or $args[attempt ] <= $args[ limit_attempts ]) { $client = new GearmanClient(); $client->addServer(); if (method_exists($client, doSuspended )) { $time = getExecutionTime($args[attempt ]); $args[ attempt ]++; $client->doSuspended(run, json_encode($args), null, $time); } } else { echo "Failure, reason: " . $e->getMessage() . "n"; } } }?>
  26. 26. Geometric progression <?php #... function getExecutionTime($attempt) { return time() + pow(2, (int)$attempt) * 10; } ?>attempt seconds delta time0 10 10s1 20 20s2 40 40s3 80 1m 20s4 160 2m 40s5 320 5m 20s6 640 10m 40s7 1280 21m 20s8 2560 42m 40s9 5120 1h 25m 20s10 10240 2h 50m 40s
  27. 27. Q &A

×