High gear PHP with Gearman

High gear PHP with Gearman Felix De Vliegher PHPBenelux meeting - 13/10/2009 zondag 11 oktober 2009

Contents • What is Gearman? • Distributed work • Once upon a time • Advantages • Gearman 101 • PHP interface • Examples! • Application areas • Other bits 2 zondag 11 oktober 2009

What is Gearman? And why should I care about it? 3 zondag 11 oktober 2009

4 zondag 11 oktober 2009

Quote “ The way I like to think of Gearman is as a massively distributed, massively fault tolerant fork mechanism. - Joe Stump, Digg ” 5 zondag 11 oktober 2009

A grain of truth The name is an anagram for “Manager,” since it dispatches jobs to be done, but does not do anything useful itself. - From Gearman website 6 zondag 11 oktober 2009

What is Gearman? It’s an Application framework to distribute work 7 zondag 11 oktober 2009

Some keywords • Open source • Multi-language • Parallel • Asynchronous • Multi threaded -> blazingly fast • Persistent queues • Decentralized • Fault tolerant • tcp 4730 • multiple client api’s (c, php, perl, drizzle, ...) 8 zondag 11 oktober 2009

Distributed processing Or: how to make others do all the work 9 zondag 11 oktober 2009

“Distributed computing” are “autonomous computational entries” (computers or nodes) which are communicating by message passing 10 zondag 11 oktober 2009

Distributed processing • Actual work is distributed • Handled by a number of “computational entities” • Some sort of RPC • Distributed parallel processing • Shared nothing • Decentralized 11 zondag 11 oktober 2009

Distributed processing • Actual work is distributed • Handled by a number of “computational entities” • Some sort of RPC • Distributed parallel processing • Shared nothing • Decentralized Gearman! 11 zondag 11 oktober 2009

Once upon a time Gearman history 12 zondag 11 oktober 2009

Once upon a time • 2005: http://brad.livejournal.com/2106943.html • Originally a Perl implementation • Created by Danga Interactive • LiveJournal • SixApart • Guys behind Memcache and MogileFS 13 zondag 11 oktober 2009

Once upon a time • 2008: Rewrite in C by Brian Aker • PHP Extension by James Luedke • Gearman powers some of the largest sites around: • Digg: 45+ servers, 400K jobs/day • Yahoo: 60+ servers, 6M jobs/day • Xing.com • Maybe Netlog? ;-) 14 zondag 11 oktober 2009

Advantages 15 zondag 11 oktober 2009

Advantages • Speed up work • Load balance work • Parallel and asynchronous work • Scales well • Architecture-based workload distributing • Call functionality in other programming languages • Doesn’t block your apache processes! 16 zondag 11 oktober 2009

Gearman 101 Terminology and architecture 17 zondag 11 oktober 2009

Terminology 18 zondag 11 oktober 2009

Terminology Create jobs to be run and send Client them to a job server Register with a job server and Worker grab jobs to run Coordinates assignment from Job Server clients to workers, handles restarts 18 zondag 11 oktober 2009

Tasks vs Jobs •A job is always a task •A task is not always a job • Single interface: jobs • Concurrent interface: tasks • Clients deal with tasks • Workers deal with jobs 19 zondag 11 oktober 2009

Architecture Client application Gearman Client API Gearman Job server (gearmand) Gearman Worker API Gearman Worker API Gearman Worker API Worker application Worker application Worker application 20 zondag 11 oktober 2009

Installing Gearman • Job Server: gearmand • Get it from https://launchpad.net/gearmand/ • extract, configure, make, make install: dev:/usr/local/src# wget http://launchpad.net/gearmand/trunk/0.10/+download/ gearmand-0.10.tar.gz dev:/usr/local/src# tar -xzvf gearmand-0.10.tar.gz dev:/usr/local/src# cd gearmand-0.10/ dev:/usr/local/src/gearmand-0.10# ./configure --prefix=/usr/local/ dev:/usr/local/src/gearmand-0.10# make dev:/usr/local/src/gearmand-0.10# make install dev:/usr/local/src/gearmand-0.10# /usr/local/sbin/gearmand --help 21 zondag 11 oktober 2009

Gearmand usage /usr/local/sbin/gearmand -d -u <user> -L 127.0.0.1 -p 7003 -d Start as daemon in background -u <user> Run as the specified user -L <host> Only listen on the specified host or IP -p <port> Listen on the specified port 22 zondag 11 oktober 2009

Commandline gearman • Client mode • ls | gearman -f function • gearman -f function < file • gearman -f function “foo data” • Worker mode • gearman -w -f function -- wc -l • gearman -w -f function ./script.sh • Example: dev:~/gearman# gearman -w -f foo -- grep GearmanClient & dev:~/gearman# cat client.php | gearman -p 7003 -f foo $client= new GearmanClient(); 23 zondag 11 oktober 2009

PHP Interface 24 zondag 11 oktober 2009

PHP: Worker API Possible to add multiple servers: 25 zondag 11 oktober 2009

PHP: Worker API Registering functions and any valid callback: Pass application data to the functions: # php -q client.php Sending job Count: 1: HELLO! Count: 2: WORLD! 26 zondag 11 oktober 2009

PHP: Worker API Put it all together: 27 zondag 11 oktober 2009

PHP: Job API Status notifications: GearmanJob::sendStatus(int $numerator, int $denominator) GearmanJob::sendWarning(string $warning) GearmanJob::sendComplete(string $result) GearmanJob::sendFail(void) GearmanJob::sendException(string $exception) 28 zondag 11 oktober 2009

PHP: Callbacks Provide functionality on different moments in the process: GearmanClient::setDataCallback GearmanClient::setCompleteCallback GearmanClient::setCreatedCallback GearmanClient::setExceptionCallback GearmanClient::setFailCallback GearmanClient::setStatusCallback GearmanClient::setWorkloadCallback 29 zondag 11 oktober 2009

PHP: Job API Client receiving the status notifications: dev:~/gearman# php -q client.php Running: true, numerator: 0, denomintor: 2 Running: true, numerator: 1, denomintor: 2 Running: false, numerator: 0, denomintor: 0 30 zondag 11 oktober 2009

PHP: Client API Setting up the client: 31 zondag 11 oktober 2009

PHP: Client API Job priorities and synchronous vs asynchronous: $status returns an array containing status information for the job corresponding to the supplied job handle. The first array element is a boolean indicating whether the job is even known, the second is a boolean indicating whether the job is still running, and the third and fourth elements correspond to the numerator and denominator of the fractional completion percentage, respectively. 32 zondag 11 oktober 2009

Tasks vs Jobs 33 zondag 11 oktober 2009

Application areas 34 zondag 11 oktober 2009

Application areas • Map/Reduce • Log analysis and aggregation • Image resizing • Asynchronous queues • URL processing • Cache warm-up • Database jobs • Cloud services and applications 35 zondag 11 oktober 2009

Map / Reduce • Client sends request to intermediate job server • Map / reduce worker splits up job (MAP) • Each part is processed by workers • Response is sent back to map / reduce worker • Results are collected and aggregated (REDUCE) • Client receives one reduced response 36 zondag 11 oktober 2009

Map / Reduce Client Gearman Job server Map/Reduce Worker Client Client Client Gearman Job server Worker Worker Worker Worker Worker 37 zondag 11 oktober 2009

Image resizing: worker 38 zondag 11 oktober 2009

Image resizing: Client 39 zondag 11 oktober 2009

Image resizing: output $ php -q resize_worker.php & $ php -q resize_client.php image.jpg H:dev:39 - creating: small_image.jpg x:200 y: Status: 1/1 $ ls *.jpg image.jpg small_image.jpg 40 zondag 11 oktober 2009

Other bits 41 zondag 11 oktober 2009

Other bits GearUp: Gearman based monitoring service • No code yet, but looks promising • http://launchpad.net/gearup Apache logging through Gearman: 42 zondag 11 oktober 2009

Other bits Gearman and MySQL with Drizzle • http://krow.livejournal.com/628025.html • todo 43 zondag 11 oktober 2009

Links & sources Credits: - Red gears: http://lineymachine.googlepages.com/ - Distributed computing: http://www.theleadblog.com/2009/06/ lead-distribution-creating-right-sales.html - Old gears: http://decorate.pebblez.com Gearman online: - http://www.danga.com/gearman/ - http://gearman.org/ - http://pecl.php.net/package/gearman/ 44 zondag 11 oktober 2009

Questions ? 45 zondag 11 oktober 2009

Thank you! Contact details: Email: felix@phpbenelux.eu Twitter: felixdv zondag 11 oktober 2009

http://www.slideshare.net	49
http://wiki.simplexi.com	33
http://203.231.63.114	3
http://blog.w3classroom.com	2
https://team.theflowr.com	1
http://wiki.simplexi.com:8080	1
http://a0.twimg.com	1

High gear PHP with Gearman

by Felix De Vliegher on Oct 25, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

7 Embeds 90

Statistics

High gear PHP with Gearman — Presentation Transcript