High gear PHP with Gearman




                                   Felix De Vliegher
                          11/03/2010 -...
Whoami?

                 • Consultant and software engineer at Ibuildings

                 • Long time PHP developer

  ...
Whoami?

                 • Software engineer and consultant at Ibuildings

                 • Long time PHP developer

  ...
What is Gearman?
                          And why should I care about it?




                                           ...
GEARMAN!




                                     5
donderdag 11 maart 2010
A grain of truth




                          The name is an anagram for
                          “Manager,” since it di...
What is Gearman?


               It’s an

               Application framework
               to

               distribu...
Distribute work?



          •    Actual work is farmed out
          •    Handled by a number of nodes
          •    RP...
Traditional architecture

                          Application

                                          View


        ...
New architecture
         Model                            Client application
                                         Gea...
Terminology




                          11
donderdag 11 maart 2010
Terminology



                                   Create jobs to be run and send
                          Client
        ...
Terminology



                                   Create jobs to be run and send
                          Client
        ...
Terminology



                                   Create jobs to be run and send
                          Client
        ...
Application areas

          • Image           resizing / generating

          • Log           analysis and aggregation

...
Multiple client & worker API’s




                                           13
donderdag 11 maart 2010
Multiple client & worker API’s




                                           13
donderdag 11 maart 2010
Advantages

          • Speed up work

          • Parallel and asynchronous work

          • Doesn’t block your apache p...
Once upon a time
                          Gearman history




                                             15
donderdag 1...
Once upon a time




          • 2005:         http://brad.livejournal.com/2106943.html
          • Originally       a Per...
Once upon a time




          • 2008:          Rewrite in C by Brian Aker
          • PHP           Extension by James Lu...
Installing Gearman

          • Job           Server: gearmand


          • Get           it from https://launchpad.net/g...
Gearmand usage

          /usr/local/sbin/gearmand -d -u <user> -L 127.0.0.1 -p 7003



                           -d     ...
Commandline gearman

          •    Client mode
               •    ls | gearman -f processFiles
               •    gearm...
PHP Interface




                                          21
donderdag 11 maart 2010
PHP: 2 options

           •    Pecl extension:

          $ pecl install channel://pecl.php.net/gearman-0.6.0

          ...
Simplest example

                             Client:




                             Worker:




                      ...
Client API

          Setting up the client:




                                   24
donderdag 11 maart 2010
Client API

          Job priorities and synchronous vs asynchronous:




                                                ...
GearmanClient::jobStatus()



                array(4) {
                  [0]=>        Job is known?
                  bo...
Notifying the client

          Client receiving the status notifications:




                  dev:~/gearman# php -q cli...
Worker API

          Possible to add multiple servers:




                                              28
donderdag 11 ...
Worker API

          Registering functions using a callback:




          Pass application data to the functions:




  ...
Worker API
          Put it all together:




                                 30
donderdag 11 maart 2010
Notifying the client
          Status notifications:
          GearmanJob::sendStatus(int $numerator, int $denominator)
  ...
Jobs vs Tasks




                          32
donderdag 11 maart 2010
Callbacks

          Provide feedback on different moments in the process:

           GearmanClient::setDataCallback

   ...
What about persistence?

               By default, jobs are stored in memory (fast)


               gearmand --queue-typ...
Sqlite3 example

               gearmand
                 --queue-type=libsqlite3
                 --libsqlite3-db=/tmp/jo...
How to do storage?




                               36
donderdag 11 maart 2010
How to do storage?

               Distributed




                               36
donderdag 11 maart 2010
How to do storage?

               Distributed

               Need to share storage (most of the time)




              ...
How to do storage?

               Distributed

               Need to share storage (most of the time)

               So...
How to do storage?

               Distributed

               Need to share storage (most of the time)

               So...
How to do storage?

               Distributed

               Need to share storage (most of the time)

               So...
How to do storage?

               Distributed

               Need to share storage (most of the time)

               So...
So, can I kick out my crons?




                                         37
donderdag 11 maart 2010
So, can I kick out my crons?

               Not quite :)




                                         37
donderdag 11 maa...
So, can I kick out my crons?

               Not quite :)
               Scheduled execution (cron)




                  ...
So, can I kick out my crons?

               Not quite :)
               Scheduled execution (cron)
               delayed...
So, can I kick out my crons?

               Not quite :)
               Scheduled execution (cron)
               delayed...
So, can I kick out my crons?

               Not quite :)
               Scheduled execution (cron)
               delayed...
So, can I kick out my crons?

               Not quite :)
               Scheduled execution (cron)
               delayed...
HTTP protocol

               Start gearmand with -r http

                           POST /reverse HTTP/1.1
             ...
Application areas




                                              39
donderdag 11 maart 2010
Map / Reduce

                                                   Client


                                             Gea...
Image resizing: Client




                                   41
donderdag 11 maart 2010
Image resizing: worker




                                   42
donderdag 11 maart 2010
Other bits




                                       43
donderdag 11 maart 2010
Other bits

               GearUp: Monitoring service for gearman
                    ‣ No code yet, but looks promising
 ...
Other bits

               GearUp: Monitoring service for gearman
                    ‣ No code yet, but looks promising
 ...
Other bits

               GearUp: Monitoring service for gearman
                    ‣ No code yet, but looks promising
 ...
Other bits

               GearUp: Monitoring service for gearman
                    ‣ No code yet, but looks promising
 ...
Other bits

               Supervisord: can manage workers


             [program:gearman-foobar-worker]
             com...
Alternatives

               Most similar: Beanstalkd
               • PHP Client: pheanstalk
               • Web interfa...
Questions ?




                          http://joind.in/1336


                                                 47
donde...
Links & sources
                          Credits:
                          - Gear man: http://agearman.com/
            ...
Thank you!
                                                 Contact details:

                          Felix De Vliegher ...
Upcoming SlideShare
Loading in...5
×

High gear PHP with Gearman (Confoo 2010)

7,009

Published on

Gearman is an application framework for distributing work to other machines and processors which are better suited for the job. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. In other words, it is the nervous system for how distributed processing communicates. With things like scalability and distributed computing becoming more and more important to today's web applications, Gearman and its PHP interface can prove quite useful to us in a variety of situations. In this talk, we'll first have a look at what distributed processing exactly means, and then looking at what Gearman actually is and does, and how it can power up your application using the Gearman PHP extension. By showing different examples and application area's, you'll get a good feeling of what Gearman is capable of and why it can be a valuable asset to your next PHP project.

Published in: Technology, Business
0 Comments
21 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,009
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
183
Comments
0
Likes
21
Embeds 0
No embeds

No notes for slide

High gear PHP with Gearman (Confoo 2010)

  1. 1. High gear PHP with Gearman Felix De Vliegher 11/03/2010 - Confoo 2010 donderdag 11 maart 2010
  2. 2. Whoami? • Consultant and software engineer at Ibuildings • Long time PHP developer • PHPBenelux • Interest in PHP QA, high performance & scalability 2 donderdag 11 maart 2010
  3. 3. Whoami? • Software engineer and consultant at Ibuildings • Long time PHP developer • PHPBenelux • Interest in PHP QA, high performance & scalability • Belgium (beer)! 3 donderdag 11 maart 2010
  4. 4. What is Gearman? And why should I care about it? 4 donderdag 11 maart 2010
  5. 5. GEARMAN! 5 donderdag 11 maart 2010
  6. 6. A grain of truth The name is an anagram for “Manager,” since it dispatches jobs to be done, but does not do anything useful itself. - From Gearman website 6 donderdag 11 maart 2010
  7. 7. What is Gearman? It’s an Application framework to distribute work 7 donderdag 11 maart 2010
  8. 8. Distribute work? • Actual work is farmed out • Handled by a number of nodes • RPC • Distributed parallel processing • Shared nothing 8 donderdag 11 maart 2010
  9. 9. Traditional architecture Application View Controller f Model 9 donderdag 11 maart 2010
  10. 10. New architecture Model Client application Gearman Client API Gearman Job server (gearmand) Gearman Worker API Gearman Worker API Gearman Worker API Worker application Worker application Worker application 10 donderdag 11 maart 2010
  11. 11. Terminology 11 donderdag 11 maart 2010
  12. 12. Terminology Create jobs to be run and send Client them to a job server Register with a job server and Worker grab jobs to run Coordinates assignment from Job Server clients to workers, handles restarts 11 donderdag 11 maart 2010
  13. 13. Terminology Create jobs to be run and send Client them to a job server Register with a job server and Worker grab jobs to run Coordinates assignment from Job Server clients to workers, handles restarts 11 donderdag 11 maart 2010
  14. 14. Terminology Create jobs to be run and send Client them to a job server Register with a job server and Worker grab jobs to run Coordinates assignment from Job Server clients to workers, handles restarts 11 donderdag 11 maart 2010
  15. 15. Application areas • Image resizing / generating • Log analysis and aggregation • Asynchronous queues • Map/Reduce • URL processing • Cache warm-up 12 donderdag 11 maart 2010
  16. 16. Multiple client & worker API’s 13 donderdag 11 maart 2010
  17. 17. Multiple client & worker API’s 13 donderdag 11 maart 2010
  18. 18. Advantages • Speed up work • Parallel and asynchronous work • Doesn’t block your apache processes • Scales well • Architecture-based workload distributing • Legacy code 14 donderdag 11 maart 2010
  19. 19. Once upon a time Gearman history 15 donderdag 11 maart 2010
  20. 20. Once upon a time • 2005: http://brad.livejournal.com/2106943.html • Originally a Perl implementation • Created by Danga Interactive • Guys behind Memcache and MogileFS 16 donderdag 11 maart 2010
  21. 21. Once upon a time • 2008: Rewrite in C by Brian Aker • PHP Extension by James Luedke • Gearman powers some of the largest sites around: • Digg: 45+ servers, 400K jobs/day • Yahoo: 60+ servers, 6M jobs/day • Xing.com 17 donderdag 11 maart 2010
  22. 22. Installing Gearman • Job Server: gearmand • Get it from https://launchpad.net/gearmand/ • extract, configure, make, make install: dev:/usr/local/src# wget http://launchpad.net/gearmand/trunk/0.10/+download/ gearmand-0.10.tar.gz dev:/usr/local/src# tar -xzvf gearmand-0.10.tar.gz dev:/usr/local/src# cd gearmand-0.10/ dev:/usr/local/src/gearmand-0.10# ./configure --prefix=/usr/local/ dev:/usr/local/src/gearmand-0.10# make dev:/usr/local/src/gearmand-0.10# make install dev:/usr/local/src/gearmand-0.10# /usr/local/sbin/gearmand --help 18 donderdag 11 maart 2010
  23. 23. Gearmand usage /usr/local/sbin/gearmand -d -u <user> -L 127.0.0.1 -p 7003 -d Start as daemon in background -u <user> Run as the specified user -L <host> Only listen on the specified host or IP -p <port> Listen on the specified port -t <threads> Number of threads to use -v(vv) Verbose (useful for debugging) 19 donderdag 11 maart 2010
  24. 24. Commandline gearman • Client mode • ls | gearman -f processFiles • gearman -f processFiles < file • gearman -f processFiles “foo data” • Worker mode • gearman -w -f myFunction -- wc -l • gearman -w -c 100 -f myFunction ./script.sh • Example: dev:~/gearman# gearman -w -f foo -- grep GearmanClient & dev:~/gearman# cat demo.php | gearman -f foo $client= new GearmanClient(); 20 donderdag 11 maart 2010
  25. 25. PHP Interface 21 donderdag 11 maart 2010
  26. 26. PHP: 2 options • Pecl extension: $ pecl install channel://pecl.php.net/gearman-0.6.0 $ php -i | grep "gearman support" gearman support => enabled • Net_Gearman PEAR Library: $ pear install Net_Gearman • Net_Gearman_Job • Net_Gearman_Worker • Net_Gearman_Task • Net_Gearman_Set • Net_Gearman_Client 22 donderdag 11 maart 2010
  27. 27. Simplest example Client: Worker: 23 donderdag 11 maart 2010
  28. 28. Client API Setting up the client: 24 donderdag 11 maart 2010
  29. 29. Client API Job priorities and synchronous vs asynchronous: 25 donderdag 11 maart 2010
  30. 30. GearmanClient::jobStatus() array(4) { [0]=> Job is known? bool(true) [1]=> Job still running? bool(true) [2]=> Numerator int(2) [3]=> Denominator int(5) } 26 donderdag 11 maart 2010
  31. 31. Notifying the client Client receiving the status notifications: dev:~/gearman# php -q client.php Running: true, numerator: 0, denomintor: 2 Running: true, numerator: 1, denomintor: 2 Running: false, numerator: 0, denomintor: 0 27 donderdag 11 maart 2010
  32. 32. Worker API Possible to add multiple servers: 28 donderdag 11 maart 2010
  33. 33. Worker API Registering functions using a callback: Pass application data to the functions: # php -q client.php Sending job Count: 1: HELLO! Count: 2: WORLD! 29 donderdag 11 maart 2010
  34. 34. Worker API Put it all together: 30 donderdag 11 maart 2010
  35. 35. Notifying the client Status notifications: GearmanJob::sendStatus(int $numerator, int $denominator) GearmanJob::sendWarning(string $warning) GearmanJob::sendComplete(string $result) GearmanJob::sendFail(void) GearmanJob::sendException(string $exception) 31 donderdag 11 maart 2010
  36. 36. Jobs vs Tasks 32 donderdag 11 maart 2010
  37. 37. Callbacks Provide feedback on different moments in the process: GearmanClient::setDataCallback GearmanClient::setCompleteCallback GearmanClient::setCreatedCallback GearmanClient::setExceptionCallback GearmanClient::setFailCallback GearmanClient::setStatusCallback GearmanClient::setWorkloadCallback 33 donderdag 11 maart 2010
  38. 38. What about persistence? By default, jobs are stored in memory (fast) gearmand --queue-type (-q) • libdrizzle • postgresql • libmemcached • libsqlite3 Only for background jobs 34 donderdag 11 maart 2010
  39. 39. Sqlite3 example gearmand --queue-type=libsqlite3 --libsqlite3-db=/tmp/jobs.sqlite --libsqlite3-table=gearman_jobs Table structure: sqlite> CREATE TABLE gearman_jobs ( ...> unique_key TEXT PRIMARY_KEY, ...> function_name TEXT, ...> priority INTEGER, ...> data BLOB ...> ); 35 donderdag 11 maart 2010
  40. 40. How to do storage? 36 donderdag 11 maart 2010
  41. 41. How to do storage? Distributed 36 donderdag 11 maart 2010
  42. 42. How to do storage? Distributed Need to share storage (most of the time) 36 donderdag 11 maart 2010
  43. 43. How to do storage? Distributed Need to share storage (most of the time) Some options: 36 donderdag 11 maart 2010
  44. 44. How to do storage? Distributed Need to share storage (most of the time) Some options: • NFS 36 donderdag 11 maart 2010
  45. 45. How to do storage? Distributed Need to share storage (most of the time) Some options: • NFS • MogileFS 36 donderdag 11 maart 2010
  46. 46. How to do storage? Distributed Need to share storage (most of the time) Some options: • NFS • MogileFS • DR:BD 36 donderdag 11 maart 2010
  47. 47. So, can I kick out my crons? 37 donderdag 11 maart 2010
  48. 48. So, can I kick out my crons? Not quite :) 37 donderdag 11 maart 2010
  49. 49. So, can I kick out my crons? Not quite :) Scheduled execution (cron) 37 donderdag 11 maart 2010
  50. 50. So, can I kick out my crons? Not quite :) Scheduled execution (cron) delayed execution (at) 37 donderdag 11 maart 2010
  51. 51. So, can I kick out my crons? Not quite :) Scheduled execution (cron) delayed execution (at) */15 * * * * /usr/local/bin/gearman -f cronTask < /tmp/input.txt 37 donderdag 11 maart 2010
  52. 52. So, can I kick out my crons? Not quite :) Scheduled execution (cron) delayed execution (at) */15 * * * * /usr/local/bin/gearman -f cronTask < /tmp/input.txt at functionality is considered for inclusion 37 donderdag 11 maart 2010
  53. 53. So, can I kick out my crons? Not quite :) Scheduled execution (cron) delayed execution (at) */15 * * * * /usr/local/bin/gearman -f cronTask < /tmp/input.txt at functionality is considered for inclusion See http://groups.google.com/group/gearman/browse_thread/thread/b9891649fb08d16b# 37 donderdag 11 maart 2010
  54. 54. HTTP protocol Start gearmand with -r http POST /reverse HTTP/1.1 Content-Length: 13 Hello confoo! HTTP/1.0 200 OK X-Gearman-Job-Handle: H:gman:4 Content-Length: 13 Server: Gearman/0.9 !oofnoc olleH Use X-Gearman-* headers to modify job 38 donderdag 11 maart 2010
  55. 55. Application areas 39 donderdag 11 maart 2010
  56. 56. Map / Reduce Client Gearman Job server Map/Reduce Worker Client Client Client Gearman Job server Worker Worker Worker Worker Worker 40 donderdag 11 maart 2010
  57. 57. Image resizing: Client 41 donderdag 11 maart 2010
  58. 58. Image resizing: worker 42 donderdag 11 maart 2010
  59. 59. Other bits 43 donderdag 11 maart 2010
  60. 60. Other bits GearUp: Monitoring service for gearman ‣ No code yet, but looks promising ‣ http://launchpad.net/gearup 44 donderdag 11 maart 2010
  61. 61. Other bits GearUp: Monitoring service for gearman ‣ No code yet, but looks promising ‣ http://launchpad.net/gearup 44 donderdag 11 maart 2010
  62. 62. Other bits GearUp: Monitoring service for gearman ‣ No code yet, but looks promising ‣ http://launchpad.net/gearup Apache logging through Gearman: 44 donderdag 11 maart 2010
  63. 63. Other bits GearUp: Monitoring service for gearman ‣ No code yet, but looks promising ‣ http://launchpad.net/gearup Apache logging through Gearman: 44 donderdag 11 maart 2010
  64. 64. Other bits Supervisord: can manage workers [program:gearman-foobar-worker] command=/usr/local/bin/php -q /home/foo/workers/foobar.php process_name=%(program_name)s_%(process_num)02d autostart=true autorestart=true numprocs=100 redirect_stderr=true stdout_logfile=/var/log/gearman-foobar-worker.log stdout_logfile_maxbytes=5MB stdout_logfile_backups=10 Combine with get_memory_usage() to restart workers http://supervisord.org/ 45 donderdag 11 maart 2010
  65. 65. Alternatives Most similar: Beanstalkd • PHP Client: pheanstalk • Web interface in Django • http://kr.github.com/beanstalkd/ Zend Server job queue • Has job scheduling and dependencies built in • Not free • http://www.zend.com/en/products/server/zend-server-job- queue 46 donderdag 11 maart 2010
  66. 66. Questions ? http://joind.in/1336 47 donderdag 11 maart 2010
  67. 67. Links & sources Credits: - Gear man: http://agearman.com/ - Distributed computing: http://www.theleadblog.com/2009/06/ lead-distribution-creating-right-sales.html - Old gears: http://decorate.pebblez.com - Army of elephpants: http://www.flickr.com/people/dragonbe/ Gearman online: - http://www.danga.com/gearman/ - http://gearman.org/ - http://pecl.php.net/package/gearman - IRC: #gearman on irc.freenode.net - ML: http://groups.google.com/group/gearman 48 donderdag 11 maart 2010
  68. 68. Thank you! Contact details: Felix De Vliegher <felix@ibuildings.nl> Twitter: felixdv donderdag 11 maart 2010
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×