• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
High gear PHP with Gearman (Confoo 2010)
 

High gear PHP with Gearman (Confoo 2010)

on

  • 8,150 views

Gearman is an application framework for distributing work to other machines and processors which are better suited for the job. It can be used in a variety of applications, from high-availability web ...

Gearman is an application framework for distributing work to other machines and processors which are better suited for the job. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. In other words, it is the nervous system for how distributed processing communicates. With things like scalability and distributed computing becoming more and more important to today's web applications, Gearman and its PHP interface can prove quite useful to us in a variety of situations. In this talk, we'll first have a look at what distributed processing exactly means, and then looking at what Gearman actually is and does, and how it can power up your application using the Gearman PHP extension. By showing different examples and application area's, you'll get a good feeling of what Gearman is capable of and why it can be a valuable asset to your next PHP project.

Statistics

Views

Total Views
8,150
Views on SlideShare
7,916
Embed Views
234

Actions

Likes
21
Downloads
176
Comments
0

5 Embeds 234

http://george.mihailoff.com 116
http://w3devnote.wordpress.com 56
http://www.slideshare.net 55
http://www.thewebhatesme.com 5
http://www.slashdocs.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    High gear PHP with Gearman (Confoo 2010) High gear PHP with Gearman (Confoo 2010) Presentation Transcript

    • High gear PHP with Gearman Felix De Vliegher 11/03/2010 - Confoo 2010 donderdag 11 maart 2010
    • Whoami? • Consultant and software engineer at Ibuildings • Long time PHP developer • PHPBenelux • Interest in PHP QA, high performance & scalability 2 donderdag 11 maart 2010
    • Whoami? • Software engineer and consultant at Ibuildings • Long time PHP developer • PHPBenelux • Interest in PHP QA, high performance & scalability • Belgium (beer)! 3 donderdag 11 maart 2010
    • What is Gearman? And why should I care about it? 4 donderdag 11 maart 2010
    • GEARMAN! 5 donderdag 11 maart 2010
    • A grain of truth The name is an anagram for “Manager,” since it dispatches jobs to be done, but does not do anything useful itself. - From Gearman website 6 donderdag 11 maart 2010
    • What is Gearman? It’s an Application framework to distribute work 7 donderdag 11 maart 2010
    • Distribute work? • Actual work is farmed out • Handled by a number of nodes • RPC • Distributed parallel processing • Shared nothing 8 donderdag 11 maart 2010
    • Traditional architecture Application View Controller f Model 9 donderdag 11 maart 2010
    • New architecture Model Client application Gearman Client API Gearman Job server (gearmand) Gearman Worker API Gearman Worker API Gearman Worker API Worker application Worker application Worker application 10 donderdag 11 maart 2010
    • Terminology 11 donderdag 11 maart 2010
    • Terminology Create jobs to be run and send Client them to a job server Register with a job server and Worker grab jobs to run Coordinates assignment from Job Server clients to workers, handles restarts 11 donderdag 11 maart 2010
    • Terminology Create jobs to be run and send Client them to a job server Register with a job server and Worker grab jobs to run Coordinates assignment from Job Server clients to workers, handles restarts 11 donderdag 11 maart 2010
    • Terminology Create jobs to be run and send Client them to a job server Register with a job server and Worker grab jobs to run Coordinates assignment from Job Server clients to workers, handles restarts 11 donderdag 11 maart 2010
    • Application areas • Image resizing / generating • Log analysis and aggregation • Asynchronous queues • Map/Reduce • URL processing • Cache warm-up 12 donderdag 11 maart 2010
    • Multiple client & worker API’s 13 donderdag 11 maart 2010
    • Multiple client & worker API’s 13 donderdag 11 maart 2010
    • Advantages • Speed up work • Parallel and asynchronous work • Doesn’t block your apache processes • Scales well • Architecture-based workload distributing • Legacy code 14 donderdag 11 maart 2010
    • Once upon a time Gearman history 15 donderdag 11 maart 2010
    • Once upon a time • 2005: http://brad.livejournal.com/2106943.html • Originally a Perl implementation • Created by Danga Interactive • Guys behind Memcache and MogileFS 16 donderdag 11 maart 2010
    • Once upon a time • 2008: Rewrite in C by Brian Aker • PHP Extension by James Luedke • Gearman powers some of the largest sites around: • Digg: 45+ servers, 400K jobs/day • Yahoo: 60+ servers, 6M jobs/day • Xing.com 17 donderdag 11 maart 2010
    • Installing Gearman • Job Server: gearmand • Get it from https://launchpad.net/gearmand/ • extract, configure, make, make install: dev:/usr/local/src# wget http://launchpad.net/gearmand/trunk/0.10/+download/ gearmand-0.10.tar.gz dev:/usr/local/src# tar -xzvf gearmand-0.10.tar.gz dev:/usr/local/src# cd gearmand-0.10/ dev:/usr/local/src/gearmand-0.10# ./configure --prefix=/usr/local/ dev:/usr/local/src/gearmand-0.10# make dev:/usr/local/src/gearmand-0.10# make install dev:/usr/local/src/gearmand-0.10# /usr/local/sbin/gearmand --help 18 donderdag 11 maart 2010
    • Gearmand usage /usr/local/sbin/gearmand -d -u <user> -L 127.0.0.1 -p 7003 -d Start as daemon in background -u <user> Run as the specified user -L <host> Only listen on the specified host or IP -p <port> Listen on the specified port -t <threads> Number of threads to use -v(vv) Verbose (useful for debugging) 19 donderdag 11 maart 2010
    • Commandline gearman • Client mode • ls | gearman -f processFiles • gearman -f processFiles < file • gearman -f processFiles “foo data” • Worker mode • gearman -w -f myFunction -- wc -l • gearman -w -c 100 -f myFunction ./script.sh • Example: dev:~/gearman# gearman -w -f foo -- grep GearmanClient & dev:~/gearman# cat demo.php | gearman -f foo $client= new GearmanClient(); 20 donderdag 11 maart 2010
    • PHP Interface 21 donderdag 11 maart 2010
    • PHP: 2 options • Pecl extension: $ pecl install channel://pecl.php.net/gearman-0.6.0 $ php -i | grep "gearman support" gearman support => enabled • Net_Gearman PEAR Library: $ pear install Net_Gearman • Net_Gearman_Job • Net_Gearman_Worker • Net_Gearman_Task • Net_Gearman_Set • Net_Gearman_Client 22 donderdag 11 maart 2010
    • Simplest example Client: Worker: 23 donderdag 11 maart 2010
    • Client API Setting up the client: 24 donderdag 11 maart 2010
    • Client API Job priorities and synchronous vs asynchronous: 25 donderdag 11 maart 2010
    • GearmanClient::jobStatus() array(4) { [0]=> Job is known? bool(true) [1]=> Job still running? bool(true) [2]=> Numerator int(2) [3]=> Denominator int(5) } 26 donderdag 11 maart 2010
    • Notifying the client Client receiving the status notifications: dev:~/gearman# php -q client.php Running: true, numerator: 0, denomintor: 2 Running: true, numerator: 1, denomintor: 2 Running: false, numerator: 0, denomintor: 0 27 donderdag 11 maart 2010
    • Worker API Possible to add multiple servers: 28 donderdag 11 maart 2010
    • Worker API Registering functions using a callback: Pass application data to the functions: # php -q client.php Sending job Count: 1: HELLO! Count: 2: WORLD! 29 donderdag 11 maart 2010
    • Worker API Put it all together: 30 donderdag 11 maart 2010
    • Notifying the client Status notifications: GearmanJob::sendStatus(int $numerator, int $denominator) GearmanJob::sendWarning(string $warning) GearmanJob::sendComplete(string $result) GearmanJob::sendFail(void) GearmanJob::sendException(string $exception) 31 donderdag 11 maart 2010
    • Jobs vs Tasks 32 donderdag 11 maart 2010
    • Callbacks Provide feedback on different moments in the process: GearmanClient::setDataCallback GearmanClient::setCompleteCallback GearmanClient::setCreatedCallback GearmanClient::setExceptionCallback GearmanClient::setFailCallback GearmanClient::setStatusCallback GearmanClient::setWorkloadCallback 33 donderdag 11 maart 2010
    • What about persistence? By default, jobs are stored in memory (fast) gearmand --queue-type (-q) • libdrizzle • postgresql • libmemcached • libsqlite3 Only for background jobs 34 donderdag 11 maart 2010
    • Sqlite3 example gearmand --queue-type=libsqlite3 --libsqlite3-db=/tmp/jobs.sqlite --libsqlite3-table=gearman_jobs Table structure: sqlite> CREATE TABLE gearman_jobs ( ...> unique_key TEXT PRIMARY_KEY, ...> function_name TEXT, ...> priority INTEGER, ...> data BLOB ...> ); 35 donderdag 11 maart 2010
    • How to do storage? 36 donderdag 11 maart 2010
    • How to do storage? Distributed 36 donderdag 11 maart 2010
    • How to do storage? Distributed Need to share storage (most of the time) 36 donderdag 11 maart 2010
    • How to do storage? Distributed Need to share storage (most of the time) Some options: 36 donderdag 11 maart 2010
    • How to do storage? Distributed Need to share storage (most of the time) Some options: • NFS 36 donderdag 11 maart 2010
    • How to do storage? Distributed Need to share storage (most of the time) Some options: • NFS • MogileFS 36 donderdag 11 maart 2010
    • How to do storage? Distributed Need to share storage (most of the time) Some options: • NFS • MogileFS • DR:BD 36 donderdag 11 maart 2010
    • So, can I kick out my crons? 37 donderdag 11 maart 2010
    • So, can I kick out my crons? Not quite :) 37 donderdag 11 maart 2010
    • So, can I kick out my crons? Not quite :) Scheduled execution (cron) 37 donderdag 11 maart 2010
    • So, can I kick out my crons? Not quite :) Scheduled execution (cron) delayed execution (at) 37 donderdag 11 maart 2010
    • So, can I kick out my crons? Not quite :) Scheduled execution (cron) delayed execution (at) */15 * * * * /usr/local/bin/gearman -f cronTask < /tmp/input.txt 37 donderdag 11 maart 2010
    • So, can I kick out my crons? Not quite :) Scheduled execution (cron) delayed execution (at) */15 * * * * /usr/local/bin/gearman -f cronTask < /tmp/input.txt at functionality is considered for inclusion 37 donderdag 11 maart 2010
    • So, can I kick out my crons? Not quite :) Scheduled execution (cron) delayed execution (at) */15 * * * * /usr/local/bin/gearman -f cronTask < /tmp/input.txt at functionality is considered for inclusion See http://groups.google.com/group/gearman/browse_thread/thread/b9891649fb08d16b# 37 donderdag 11 maart 2010
    • HTTP protocol Start gearmand with -r http POST /reverse HTTP/1.1 Content-Length: 13 Hello confoo! HTTP/1.0 200 OK X-Gearman-Job-Handle: H:gman:4 Content-Length: 13 Server: Gearman/0.9 !oofnoc olleH Use X-Gearman-* headers to modify job 38 donderdag 11 maart 2010
    • Application areas 39 donderdag 11 maart 2010
    • Map / Reduce Client Gearman Job server Map/Reduce Worker Client Client Client Gearman Job server Worker Worker Worker Worker Worker 40 donderdag 11 maart 2010
    • Image resizing: Client 41 donderdag 11 maart 2010
    • Image resizing: worker 42 donderdag 11 maart 2010
    • Other bits 43 donderdag 11 maart 2010
    • Other bits GearUp: Monitoring service for gearman ‣ No code yet, but looks promising ‣ http://launchpad.net/gearup 44 donderdag 11 maart 2010
    • Other bits GearUp: Monitoring service for gearman ‣ No code yet, but looks promising ‣ http://launchpad.net/gearup 44 donderdag 11 maart 2010
    • Other bits GearUp: Monitoring service for gearman ‣ No code yet, but looks promising ‣ http://launchpad.net/gearup Apache logging through Gearman: 44 donderdag 11 maart 2010
    • Other bits GearUp: Monitoring service for gearman ‣ No code yet, but looks promising ‣ http://launchpad.net/gearup Apache logging through Gearman: 44 donderdag 11 maart 2010
    • Other bits Supervisord: can manage workers [program:gearman-foobar-worker] command=/usr/local/bin/php -q /home/foo/workers/foobar.php process_name=%(program_name)s_%(process_num)02d autostart=true autorestart=true numprocs=100 redirect_stderr=true stdout_logfile=/var/log/gearman-foobar-worker.log stdout_logfile_maxbytes=5MB stdout_logfile_backups=10 Combine with get_memory_usage() to restart workers http://supervisord.org/ 45 donderdag 11 maart 2010
    • Alternatives Most similar: Beanstalkd • PHP Client: pheanstalk • Web interface in Django • http://kr.github.com/beanstalkd/ Zend Server job queue • Has job scheduling and dependencies built in • Not free • http://www.zend.com/en/products/server/zend-server-job- queue 46 donderdag 11 maart 2010
    • Questions ? http://joind.in/1336 47 donderdag 11 maart 2010
    • Links & sources Credits: - Gear man: http://agearman.com/ - Distributed computing: http://www.theleadblog.com/2009/06/ lead-distribution-creating-right-sales.html - Old gears: http://decorate.pebblez.com - Army of elephpants: http://www.flickr.com/people/dragonbe/ Gearman online: - http://www.danga.com/gearman/ - http://gearman.org/ - http://pecl.php.net/package/gearman - IRC: #gearman on irc.freenode.net - ML: http://groups.google.com/group/gearman 48 donderdag 11 maart 2010
    • Thank you! Contact details: Felix De Vliegher <felix@ibuildings.nl> Twitter: felixdv donderdag 11 maart 2010