Search

High gear PHP with Gearman

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    Notes on slide 1

    2 Favorites

    High gear PHP with Gearman - Presentation Transcript

    1. High gear PHP with Gearman Felix De Vliegher PHPBenelux meeting - 13/10/2009 zondag 11 oktober 2009
    2. Contents • What is Gearman? • Distributed work • Once upon a time • Advantages • Gearman 101 • PHP interface • Examples! • Application areas • Other bits 2 zondag 11 oktober 2009
    3. What is Gearman? And why should I care about it? 3 zondag 11 oktober 2009
    4. 4 zondag 11 oktober 2009
    5. Quote “ The way I like to think of Gearman is as a massively distributed, massively fault tolerant fork mechanism. - Joe Stump, Digg ” 5 zondag 11 oktober 2009
    6. A grain of truth The name is an anagram for “Manager,” since it dispatches jobs to be done, but does not do anything useful itself. - From Gearman website 6 zondag 11 oktober 2009
    7. What is Gearman? It’s an Application framework to distribute work 7 zondag 11 oktober 2009
    8. Some keywords • Open source • Multi-language • Parallel • Asynchronous • Multi threaded -> blazingly fast • Persistent queues • Decentralized • Fault tolerant • tcp 4730 • multiple client api’s (c, php, perl, drizzle, ...) 8 zondag 11 oktober 2009
    9. Distributed processing Or: how to make others do all the work 9 zondag 11 oktober 2009
    10. “Distributed computing” are “autonomous computational entries” (computers or nodes) which are communicating by message passing 10 zondag 11 oktober 2009
    11. Distributed processing • Actual work is distributed • Handled by a number of “computational entities” • Some sort of RPC • Distributed parallel processing • Shared nothing • Decentralized 11 zondag 11 oktober 2009
    12. Distributed processing • Actual work is distributed • Handled by a number of “computational entities” • Some sort of RPC • Distributed parallel processing • Shared nothing • Decentralized Gearman! 11 zondag 11 oktober 2009
    13. Once upon a time Gearman history 12 zondag 11 oktober 2009
    14. Once upon a time • 2005: http://brad.livejournal.com/2106943.html • Originally a Perl implementation • Created by Danga Interactive • LiveJournal • SixApart • Guys behind Memcache and MogileFS 13 zondag 11 oktober 2009
    15. Once upon a time • 2008: Rewrite in C by Brian Aker • PHP Extension by James Luedke • Gearman powers some of the largest sites around: • Digg: 45+ servers, 400K jobs/day • Yahoo: 60+ servers, 6M jobs/day • Xing.com • Maybe Netlog? ;-) 14 zondag 11 oktober 2009
    16. Advantages 15 zondag 11 oktober 2009
    17. Advantages • Speed up work • Load balance work • Parallel and asynchronous work • Scales well • Architecture-based workload distributing • Call functionality in other programming languages • Doesn’t block your apache processes! 16 zondag 11 oktober 2009
    18. Gearman 101 Terminology and architecture 17 zondag 11 oktober 2009
    19. Terminology 18 zondag 11 oktober 2009
    20. Terminology Create jobs to be run and send Client them to a job server Register with a job server and Worker grab jobs to run Coordinates assignment from Job Server clients to workers, handles restarts 18 zondag 11 oktober 2009
    21. Terminology Create jobs to be run and send Client them to a job server Register with a job server and Worker grab jobs to run Coordinates assignment from Job Server clients to workers, handles restarts 18 zondag 11 oktober 2009
    22. Terminology Create jobs to be run and send Client them to a job server Register with a job server and Worker grab jobs to run Coordinates assignment from Job Server clients to workers, handles restarts 18 zondag 11 oktober 2009
    23. Tasks vs Jobs •A job is always a task •A task is not always a job • Single interface: jobs • Concurrent interface: tasks • Clients deal with tasks • Workers deal with jobs 19 zondag 11 oktober 2009
    24. Architecture Client application Gearman Client API Gearman Job server (gearmand) Gearman Worker API Gearman Worker API Gearman Worker API Worker application Worker application Worker application 20 zondag 11 oktober 2009
    25. Installing Gearman • Job Server: gearmand • Get it from https://launchpad.net/gearmand/ • extract, configure, make, make install: dev:/usr/local/src# wget http://launchpad.net/gearmand/trunk/0.10/+download/ gearmand-0.10.tar.gz dev:/usr/local/src# tar -xzvf gearmand-0.10.tar.gz dev:/usr/local/src# cd gearmand-0.10/ dev:/usr/local/src/gearmand-0.10# ./configure --prefix=/usr/local/ dev:/usr/local/src/gearmand-0.10# make dev:/usr/local/src/gearmand-0.10# make install dev:/usr/local/src/gearmand-0.10# /usr/local/sbin/gearmand --help 21 zondag 11 oktober 2009
    26. Gearmand usage /usr/local/sbin/gearmand -d -u <user> -L 127.0.0.1 -p 7003 -d Start as daemon in background -u <user> Run as the specified user -L <host> Only listen on the specified host or IP -p <port> Listen on the specified port 22 zondag 11 oktober 2009
    27. Commandline gearman • Client mode • ls | gearman -f function • gearman -f function < file • gearman -f function “foo data” • Worker mode • gearman -w -f function -- wc -l • gearman -w -f function ./script.sh • Example: dev:~/gearman# gearman -w -f foo -- grep GearmanClient & dev:~/gearman# cat client.php | gearman -p 7003 -f foo $client= new GearmanClient(); 23 zondag 11 oktober 2009
    28. PHP Interface 24 zondag 11 oktober 2009
    29. PHP: Worker API Possible to add multiple servers: 25 zondag 11 oktober 2009
    30. PHP: Worker API Registering functions and any valid callback: Pass application data to the functions: # php -q client.php Sending job Count: 1: HELLO! Count: 2: WORLD! 26 zondag 11 oktober 2009
    31. PHP: Worker API Put it all together: 27 zondag 11 oktober 2009
    32. PHP: Job API Status notifications: GearmanJob::sendStatus(int $numerator, int $denominator) GearmanJob::sendWarning(string $warning) GearmanJob::sendComplete(string $result) GearmanJob::sendFail(void) GearmanJob::sendException(string $exception) 28 zondag 11 oktober 2009
    33. PHP: Callbacks Provide functionality on different moments in the process: GearmanClient::setDataCallback GearmanClient::setCompleteCallback GearmanClient::setCreatedCallback GearmanClient::setExceptionCallback GearmanClient::setFailCallback GearmanClient::setStatusCallback GearmanClient::setWorkloadCallback 29 zondag 11 oktober 2009
    34. PHP: Job API Client receiving the status notifications: dev:~/gearman# php -q client.php Running: true, numerator: 0, denomintor: 2 Running: true, numerator: 1, denomintor: 2 Running: false, numerator: 0, denomintor: 0 30 zondag 11 oktober 2009
    35. PHP: Client API Setting up the client: 31 zondag 11 oktober 2009
    36. PHP: Client API Job priorities and synchronous vs asynchronous: $status returns an array containing status information for the job corresponding to the supplied job handle. The first array element is a boolean indicating whether the job is even known, the second is a boolean indicating whether the job is still running, and the third and fourth elements correspond to the numerator and denominator of the fractional completion percentage, respectively. 32 zondag 11 oktober 2009
    37. Tasks vs Jobs 33 zondag 11 oktober 2009
    38. Application areas 34 zondag 11 oktober 2009
    39. Application areas • Map/Reduce • Log analysis and aggregation • Image resizing • Asynchronous queues • URL processing • Cache warm-up • Database jobs • Cloud services and applications 35 zondag 11 oktober 2009
    40. Map / Reduce • Client sends request to intermediate job server • Map / reduce worker splits up job (MAP) • Each part is processed by workers • Response is sent back to map / reduce worker • Results are collected and aggregated (REDUCE) • Client receives one reduced response 36 zondag 11 oktober 2009
    41. Map / Reduce Client Gearman Job server Map/Reduce Worker Client Client Client Gearman Job server Worker Worker Worker Worker Worker 37 zondag 11 oktober 2009
    42. Image resizing: worker 38 zondag 11 oktober 2009
    43. Image resizing: Client 39 zondag 11 oktober 2009
    44. Image resizing: output $ php -q resize_worker.php & $ php -q resize_client.php image.jpg H:dev:39 - creating: small_image.jpg x:200 y: Status: 1/1 $ ls *.jpg image.jpg small_image.jpg 40 zondag 11 oktober 2009
    45. Other bits 41 zondag 11 oktober 2009
    46. Other bits GearUp: Gearman based monitoring service • No code yet, but looks promising • http://launchpad.net/gearup Apache logging through Gearman: 42 zondag 11 oktober 2009
    47. Other bits Gearman and MySQL with Drizzle • http://krow.livejournal.com/628025.html • todo 43 zondag 11 oktober 2009
    48. Links & sources Credits: - Red gears: http://lineymachine.googlepages.com/ - Distributed computing: http://www.theleadblog.com/2009/06/ lead-distribution-creating-right-sales.html - Old gears: http://decorate.pebblez.com Gearman online: - http://www.danga.com/gearman/ - http://gearman.org/ - http://pecl.php.net/package/gearman/ 44 zondag 11 oktober 2009
    49. Questions ? 45 zondag 11 oktober 2009
    50. Thank you! Contact details: Email: felix@phpbenelux.eu Twitter: felixdv zondag 11 oktober 2009

    + Felix De VliegherFelix De Vliegher, 1 month ago

    custom

    173 views, 2 favs, 0 embeds more stats

    Gearman provides a generic application framework to more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 173
      • 173 on SlideShare
      • 0 from embeds
    • Comments 0
    • Favorites 2
    • Downloads 0
    Most viewed embeds

    more

    All embeds

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories

    Tags

    -->
    Search
    RSS Feed
    What's new?

    © 2009 SlideShare Inc. All Rights Reserved