CakeFest 2012 – Manchester, UK                     Job Queues              Letting your servers breathe                   ...
So you build a signupclass UsersController extends AppController {    public function add() {        if ($this->request->i...
PLEASE STOP!BAD IDEA
Bad ideas●   Email sending●   Notifications●   Messaging    –   To multiple users●   Imports    –   Oh yes, imports●   Cou...
The #1 Rule of Fight ClubYOU DO NOT MAKE THE USER WAIT
The solution
Show me how, commarade                             Could someone please                                send this email?   ...
Ingredients●   Channel for dispatching tasks●   Job queue●   Workers that wait for jobs●   Beer●   CakeResque: github://ka...
A Harry Potter momentGEARMANMANAGER
Break it down●   Client API to dispatch tasks●   Job server (Gearmand)●   Worker API to, well, work on jobs●   Multi-langu...
Super scalable            Gearmand #1                   Gearmand #2Worker #1     Worker #2   Worker #2   Worker #2   Worke...
Client & Worker$client = new GearmanClient();$client->addServer(127.0.0.1);$result = $client->doBackground(hello, Mariano)...
Getting jiggy with it●   Start $ gearmand --verbose DEBUG●   $ telnet 4730 and type in STATUS    –   Gives you registered ...
Show me the money
Get the money back$client = new GearmanClient();$client->addServer(127.0.0.1);$result = $client->doNormal(hello, Mariano);...
Show me the money
I got 99 problems
<3 supervisor[supervisord]logfile=/var/log/supervisor/supervisord.loglogfile_maxbytes = 50MBpidfile=/var/run/supervisord.p...
<3 supervisor$ sudo supervisord -c supervisor.conf
Job statuses         [file1.jpg,             [file1.jpg,         file2.jpg,              file2.jpg,         file3.jpg]    ...
Job statuses$worker = new GearmanWorker();$worker->addServer(127.0.0.1);$worker->addFunction(process, function($job) {    ...
Job statuses$client = new GearmanClient();$client->addServer(127.0.0.1);$handle = $client->doBackground(process, json_enco...
Parallel tasks●   They are not as unusual as you think●   Resource usage optimization●   Divide and conquer <BuzzWord>Map ...
Parallel tasks$client->setStatusCallback(function($task) {    echo "PROCESSED {$task->taskNumerator()} of {$task->taskDeno...
Parallel tasks       worker-1.log                 worker-2.logProcessing a1.jpg... DONE   Processing file1.jpg... DONEProc...
What about CakePHP?
Bake it!class JobsController extends AppController {   public function trigger() {       $client = new GearmanClient();   ...
Bake it!class WorkerShell extends AppShell {   public function start() {       $this->out(Connecting to Gearman... , 0, Sh...
Bake it!public function _run(GearmanJob $job) {    $params = json_decode($job->workload(), true);    $this->out(Got new jo...
Lessons learned●   Processes stay alive, which mean resource    timeouts    –   Check your DB for reconnection!●   Workers...
BONUS TRACK
I can haz non-blocking!                                             (Sort of)$worker->addOptions(GEARMAN_WORKER_NON_BLOCKI...
CakeFest 2012 – Manchester, UK                      Questions?                          Beers?                            ...
Upcoming SlideShare
Loading in...5
×

Job queues with Gearman

9,790

Published on

Published in: Technology, Business
1 Comment
12 Likes
Statistics
Notes
  • Hola Mariano, disculpa la molestia, estoy intentando obtener el result de una background task, y no hay caso... Sabes si esto es posible y como se tendria que hacer?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
9,790
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
61
Comments
1
Likes
12
Embeds 0
No embeds

No notes for slide
  • * PRO TIP: If you are doing signup forms, integrate with facebook and linkedin. Both provide emails that you know are already validated
  • * Job queues are not only about performance. They are also about stopping BAD IDEAS * For example: signup form, on $_POST, sends email. Good idea or bad idea? * There are several things that can go wrong: timeouts, errors while connecting, errors while rendering email, errors in email component (I KNOW ;)
  • * Performance is not just about caching, its about the right architecture * Surprisingly, right architectures are not that hard to build
  • * Communism is a good thing on the Internet * Outsource anything you can to Workers (ehem ehem Workana)
  • * I think of controllers as sexy secretaries. * When looking for pictures of “Hot secretaries”, Justin Bieber showed up. Can somebody please explain that?
  • * In essence you could do this with RabbitMQ, ActiveMQ, Redis pub/sub, heck even MySQL ** Yes, MySQL (robot plugin) * CakeResque: CakePHP 2.0+, uses Redis
  • * Remember when Harry was told TOM MARVOLO RIDDLE == LORD VOLDEMORT?
  • * Multi-language: PHP, Perl, Java, Ruby, Python * Multi-platform: Linux, OSX, Windows (cygwin) * Synchronous execution: execute, and get the result, waiting for it * Job status callbacks: register callbacks for when the task is completed, or when the job sends a status update (using the worker API)
  • * Why not multiple workers on same machine on multi-core CPUs? Supervisord
  • * addServer(): you can attach your worker to multiple servers * PRIORITIES: doLowBackground(), doBackground(), doHighBackground()
  • * PRIORITIES: doLow(), doNormal(), doHigh()
  • * What happens if your worker fatals? You got it, the whole worker goes bananas. * We need to ensure there&apos;s always a worker active
  • * You send status through sendStatus() * You can also send data through sendData()
  • * NOT UNUSUAL: if they can be done independently, they can be done in parallel. EG: send email / pre-create profile page / update stat counters * Resource usage optimization: some “expensive” machines for heavy tasks, others for easier tasks. But we still need to know when all the tasks are DONE * Map &amp; Reduce: Mapper on one Gearman server, reducer on the other addTask: has priorities (addTask, addTaskLow, addTaskHigh)
  • * worker3.php: we use the same exact worker from before, no need to change it for switching to a parallel work mode
  • * Workers KILLED: monit that issues a ping dummy task for each worker, synchronously * PROCESS switches: do an usleep() between loop changes. It won&apos;t kill you.
  • * wait(): wait for activity from server * PHP BUG: #60764
  • Job queues with Gearman

    1. 1. CakeFest 2012 – Manchester, UK Job Queues Letting your servers breathe @mgiglesias
    2. 2. So you build a signupclass UsersController extends AppController { public function add() { if ($this->request->is(post)) { $this->User->save($this->request->data); $email = new CakeEmail(); $email->to($this->request->data[User][email]) ->subject("Validate em emails") ->template(activate) ->viewVars(array(id => $this->User->id)) ->send(); $this->Session->setFlash("Giddy up!"); return $this->redirect(array(action =>index)); } else { $this->Session->setFlash("Fix em typos"); } }}
    3. 3. PLEASE STOP!BAD IDEA
    4. 4. Bad ideas● Email sending● Notifications● Messaging – To multiple users● Imports – Oh yes, imports● Counter caches
    5. 5. The #1 Rule of Fight ClubYOU DO NOT MAKE THE USER WAIT
    6. 6. The solution
    7. 7. Show me how, commarade Could someone please send this email? Controller{ { to: happy@unicorn.com, to: angry@dwarf.com, subject: FYI, subject: Yo, dawg, body: Unicorns sux body: Whatup} }
    8. 8. Ingredients● Channel for dispatching tasks● Job queue● Workers that wait for jobs● Beer● CakeResque: github://kamisama/Cake-Resque
    9. 9. A Harry Potter momentGEARMANMANAGER
    10. 10. Break it down● Client API to dispatch tasks● Job server (Gearmand)● Worker API to, well, work on jobs● Multi-language● Multi-platform● Synchronous execution● Job status callbacks
    11. 11. Super scalable Gearmand #1 Gearmand #2Worker #1 Worker #2 Worker #2 Worker #2 Worker #2
    12. 12. Client & Worker$client = new GearmanClient();$client->addServer(127.0.0.1);$result = $client->doBackground(hello, Mariano);var_dump($result);$worker = new GearmanWorker();$worker->addServer(127.0.0.1);$worker->addFunction(hello, function($job) { $who = $job->workload(); sleep(1); echo "Hello {$who}!" . "n"; sleep(1); return true;});while($worker->work()) { usleep(50000);}
    13. 13. Getting jiggy with it● Start $ gearmand --verbose DEBUG● $ telnet 4730 and type in STATUS – Gives you registered functions, number of workers and items in queue
    14. 14. Show me the money
    15. 15. Get the money back$client = new GearmanClient();$client->addServer(127.0.0.1);$result = $client->doNormal(hello, Mariano);var_dump($result);$worker = new GearmanWorker();$worker->addServer(127.0.0.1);$worker->addFunction(hello, function($job) { $who = $job->workload(); return "Hello {$who}!";});while($worker->work()) { usleep(50000);}
    16. 16. Show me the money
    17. 17. I got 99 problems
    18. 18. <3 supervisor[supervisord]logfile=/var/log/supervisor/supervisord.loglogfile_maxbytes = 50MBpidfile=/var/run/supervisord.pid[inet_http_server]port = 127.0.0.1:9001[supervisorctl]serverurl = unix:///var/run/supervisor.sockprompt = supervisor[program:worker]command=/usr/bin/php -f /home/mariano/worker.phpprocess_name=%(program_name)s #%(process_num)snumprocs=4autostart=trueautorestart=truestdout_logfile=/var/log/supervisor/worker-%(process_num)s.logstderr_logfile=/var/log/supervisor/worker-%(process_num)s-error.log
    19. 19. <3 supervisor$ sudo supervisord -c supervisor.conf
    20. 20. Job statuses [file1.jpg, [file1.jpg, file2.jpg, file2.jpg, file3.jpg] file3.jpg] (1, 3) (1, 3) WorkerClient (2, 3) (2, 3) Gearman (3, 3) (3, 3)
    21. 21. Job statuses$worker = new GearmanWorker();$worker->addServer(127.0.0.1);$worker->addFunction(process, function($job) { $files = json_decode($job->workload(), true); $i = 0; $count = count($files); foreach($files as $file) { echo "Processing {$file}... "; $job->sendStatus(++$i, $count); echo "DONEn"; sleep(1); }});while($worker->work()) { usleep(50000);}
    22. 22. Job statuses$client = new GearmanClient();$client->addServer(127.0.0.1);$handle = $client->doBackground(process, json_encode(array( file1.jpg, file2.jpg, file3.jpg)));$done = 0;do { list($queued, $running, $processed, $total) = $client->jobStatus($handle); if (!$queued) { break; } if ($processed != $done) { $done = $processed; echo "PROCESSED {$done} of {$total}n"; } usleep(50000);} while(true);echo DONE;
    23. 23. Parallel tasks● They are not as unusual as you think● Resource usage optimization● Divide and conquer <BuzzWord>Map & Reduce</BuzzWord>● Can even prioritize the tasks● Its parallel, so its faster :)
    24. 24. Parallel tasks$client->setStatusCallback(function($task) { echo "PROCESSED {$task->taskNumerator()} of {$task->taskDenominator()}n";});$client->setCompleteCallback(function($task) { echo "COMPLETED: {$task->unique()}: {$task->data()}n";});$client->addTask(process, json_encode(array( file1.jpg, file2.jpg, file3.jpg)));$client->addTask(process, json_encode(array( a1.jpg, b.jpg)));$client->runTasks();
    25. 25. Parallel tasks worker-1.log worker-2.logProcessing a1.jpg... DONE Processing file1.jpg... DONEProcessing b.jpg... DONE Processing file2.jpg... DONE Processing file3.jpg... DONE
    26. 26. What about CakePHP?
    27. 27. Bake it!class JobsController extends AppController { public function trigger() { $client = new GearmanClient(); $client->addServer(127.0.0.1); $client->doNormal(run, json_encode(array( action => get_called_class() . ::hello, params => array(Mariano) ))); echo DONE; exit; } public static function hello($who) { echo "ttHello {$who}!n"; }}
    28. 28. Bake it!class WorkerShell extends AppShell { public function start() { $this->out(Connecting to Gearman... , 0, Shell::VERBOSE); $worker = new GearmanWorker(); $worker->addServer(127.0.0.1); $worker->addFunction(run, array($this, _run)); $this->out(DONE, 1, Shell::VERBOSE); $this->out(Waiting for jobs, 1, Shell::VERBOSE); while($worker->work()) { usleep(50000); } }}
    29. 29. Bake it!public function _run(GearmanJob $job) { $params = json_decode($job->workload(), true); $this->out(Got new job: . $job->workload(), 1, Shell::VERBOSE); if (empty($params) || !isset($params[action])) { throw new InvalidArgumentException(Invalid job request); } else if ( preg_match(/^(.+?Controller)::/, $params[action], $match) && !class_exists($match[1]) ) { App::uses($match[1], Controller); } $result = call_user_func_array($params[action], $params[params]); $this->out(Job done, 1, Shell::VERBOSE); return $result;}
    30. 30. Lessons learned● Processes stay alive, which mean resource timeouts – Check your DB for reconnection!● Workers can be killed, easily● Be mindful of your memory usage● Account for process switches
    31. 31. BONUS TRACK
    32. 32. I can haz non-blocking! (Sort of)$worker->addOptions(GEARMAN_WORKER_NON_BLOCKING);while ($worker->work() || ( $worker->returnCode() == GEARMAN_IO_WAIT || $worker->returnCode() == GEARMAN_NO_JOBS)) { if ($worker->returnCode() == GEARMAN_SUCCESS) { continue; } if (!$worker->wait()) { if ($worker->returnCode() == GEARMAN_NO_ACTIVE_FDS) { echo "Got disconnected, so waiting forserver...n"; sleep(5); continue; } break; }}
    33. 33. CakeFest 2012 – Manchester, UK Questions? Beers? @mgiglesias
    1. Gostou de algum slide específico?

      Recortar slides é uma maneira fácil de colecionar informações para acessar mais tarde.

    ×