r man
          Gea
Sunday, 25 October 2009
Gearman: a technology
                     for distributed
                       computing
                              ...
Mainframe

                                                             terminal
                          Mainframe

    ...
Mini computers
               Mainframe                      Mini           terminal


                                   ...
Networks
                   server                                   personal
                                            ...
Web applications
              web server                                browser




                                     ...
service
                          Cloud applications
        provider service
                 provider
              serv...
Some actors

                • memcached
                • gearman

                          Used in production by Facebo...
GEARMAN


Sunday, 25 October 2009
!= G ER M A N


Sunday, 25 October 2009
G E A R M A N?


Sunday, 25 October 2009
MANAGER

Sunday, 25 October 2009
job
                                   task
                                                   request




            ser...
Distributed




Sunday, 25 October 2009
Multiple operating systems




Sunday, 25 October 2009
multiple languages




Sunday, 25 October 2009
freedom of
                             choice



Sunday, 25 October 2009
redundancy



Sunday, 25 October 2009
USING GEARMAN
                    • Server: gearmand
                    • Client libraries:
                     • C/C++
...
Simple usage
                            GEARMAN


                    • Command line client and worker



Sunday, 25 Octo...
starting the server
  /usr/local/sbin/gearmand -d

  # started as daemon.
  # No feedback given on the command line




Su...
starting the server (2)
  /usr/local/sbin/gearmand -v -v
   INFO Starting up
   INFO Listening on :::4730 (5)
   INFO List...
starting the worker
  gearman -w -h hostname -p 4730 
    -f conta wc

  #      -w               =   act as worker
  #    ...
what the server says
  /usr/local/sbin/gearmand -v -v
   INFO Starting up
   INFO Listening on :::4730 (5)
   INFO Listeni...
starting the client
  gearman -h hostname -p 4730 
    -f conta < ~/.bashrc
        57     135    2149   # <- output
     ...
what the server says
  /usr/local/sbin/gearmand -v -v
   INFO Starting up
   INFO Listening on :::4730 (5)
   INFO Listeni...
What happened
            1             server start


                          listen to port 4730




Sunday, 25 Octobe...
What happened
            2              worker starts


                          registers function 'conta' to server


...
What happened
            3             client starts


                          requires function 'conta' from server

 ...
What happened
            4             server sends client request to worker


                              passes all i...
What happened
            5             worker receives request and data


                                  processes inp...
What happened
            6             worker returns processed data


                             server passes it to c...
What happened
            7             client receives processed data


                               client displays re...
A simple Perl worker



Sunday, 25 October 2009
A simple perl worker

                    1. add server
                    2. add function
                    3. loop
  ...
simple worker (1)
  use           strict;
  use           warnings;
  use           Gearman::XS qw(:constants);
  use     ...
simple worker (2)
  my $options = '';

  $ret = $worker->add_function(
    "reverse",   # public function name
    0,     ...
simple worker (3)
  while (1) {
    my $ret = $worker->work();
    if ($ret != GEARMAN_SUCCESS) {
      printf(STDERR "%sn...
simple worker (4)
  sub myreverse {
    my ($job) = @_;

         my $workload = $job->workload();
         my $result   =...
A simple Perl client



Sunday, 25 October 2009
A simple perl client


                    • add server
                    • run a task


Sunday, 25 October 2009
simple client (1)
  use           strict;
  use           warnings;
  use           Gearman::XS qw(:constants);
  use     ...
simple client (2)
  my $input = shift || 'teststring';

  my ($return, $result) =
    $client->do("reverse", $input);
  if...
A sample run



Sunday, 25 October 2009
host 1
  perl worker.pl




Sunday, 25 October 2009
host 2
  perl client.pl
  Result=gnirtstset




Sunday, 25 October 2009
host 1
  perl worker.pl
  Job=H:gmac3.local:4 F_Name=reverse
  Workload=teststring Result=gnirtstset




Sunday, 25 Octobe...
more client functions

                    • do_background
                    • add_task
                    • run_tasks
...
Some advanced usage

                    • DBIx::SQLCrosstab
                    • Data cubes
                    • Perl o...
Sunday, 25 October 2009
Image processing

                    • CPU intensive
                    • Large storage needed
                    • App...
Sunday, 25 October 2009
Sunday, 25 October 2009
Remote sandboxes
               Using GEARMAN
                     client server architecture
                     N serve...
host1                            worker1                    host2
                                                   worke...
host1                                           worker1                 host2
                                            ...
Remote sandbox
                             worker

                   •add server
                   •add function
      ...
remote sandbox worker (2)
  $ret = $worker->add_function(
    "make_sandbox",
    0,
    &make_sandbox,
    '');
  if ($re...
remote sandbox worker (4)
  sub make_sandbox {
      my ($job, $options) = @_;

                my $command     = $job->wo...
Client to install
                          Remote sandboxes

                   •add server
                   •run remot...
sample call to remote worker
  my $ret =
    qx(echo "$MASTER_SANDBOX" |
    gearman -h $master_ip -f make_sandbox ) ;

  ...
More on similar matters

     • FOSS.MY, Kuala Lumpur, Malaysia, 25-Oct-2009
     • CodeBits, Lisbon, Portugal, 3-4-5-Dec-...
THANKS
                                                                              Let's talk!




This work is licensed...
Upcoming SlideShare
Loading in...5
×

Gearman For Beginners

6,570

Published on

With Gearman, a client/server infrastructure for generic tasks, you can work on distributed servers easily, with little worry about the details.
No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more.

Published in: Technology

Gearman For Beginners

  1. 1. r man Gea Sunday, 25 October 2009
  2. 2. Gearman: a technology for distributed computing Giuseppe Maxia MySQL Community Team Lead Sun Microsystems Sunday, 25 October 2009
  3. 3. Mainframe terminal Mainframe terminal operating system client terminal hardware application terminal 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  4. 4. Mini computers Mainframe Mini terminal terminal operating system Mini client hardware terminal application terminal 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  5. 5. Networks server personal computer personal operating operating computer system system personal client hardware hardware computer hardware personal application 0 computer 100 USER FRIENDLINESS Sunday, 25 October 2009
  6. 6. Web applications web server browser INTERNET browser operating operating operating system system systemoperating system client browser hardware hardware hardware application browser 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  7. 7. service Cloud applications provider service provider service provider service browser provider web INTERNET server browser operating operating operating system system system client browser hardware hardware hardware application applicationapplication browser 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  8. 8. Some actors • memcached • gearman Used in production by Facebook and Digg Sunday, 25 October 2009
  9. 9. GEARMAN Sunday, 25 October 2009
  10. 10. != G ER M A N Sunday, 25 October 2009
  11. 11. G E A R M A N? Sunday, 25 October 2009
  12. 12. MANAGER Sunday, 25 October 2009
  13. 13. job task request server worker client http://gearman.org Sunday, 25 October 2009
  14. 14. Distributed Sunday, 25 October 2009
  15. 15. Multiple operating systems Sunday, 25 October 2009
  16. 16. multiple languages Sunday, 25 October 2009
  17. 17. freedom of choice Sunday, 25 October 2009
  18. 18. redundancy Sunday, 25 October 2009
  19. 19. USING GEARMAN • Server: gearmand • Client libraries: • C/C++ • Java • Perl • PHP • Python Sunday, 25 October 2009
  20. 20. Simple usage GEARMAN • Command line client and worker Sunday, 25 October 2009
  21. 21. starting the server /usr/local/sbin/gearmand -d # started as daemon. # No feedback given on the command line Sunday, 25 October 2009
  22. 22. starting the server (2) /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) # started as normal application # verbose output requested Sunday, 25 October 2009
  23. 23. starting the worker gearman -w -h hostname -p 4730 -f conta wc # -w = act as worker # -f = function # conta = function name # wc = command to execute when function # 'conta' is called Sunday, 25 October 2009
  24. 24. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected Sunday, 25 October 2009
  25. 25. starting the client gearman -h hostname -p 4730 -f conta < ~/.bashrc 57 135 2149 # <- output # from worker # -f = function # conta = function name # < ~/.bashrc = input data Sunday, 25 October 2009
  26. 26. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected … INFO Accepted connection from 127.0.0.1:5181 INFO [ 0] 127.0.0.1:5181 Connected INFO [ 0] 127.0.0.1:5181 Disconnected Sunday, 25 October 2009
  27. 27. What happened 1 server start listen to port 4730 Sunday, 25 October 2009
  28. 28. What happened 2 worker starts registers function 'conta' to server Sunday, 25 October 2009
  29. 29. What happened 3 client starts requires function 'conta' from server provides input data Sunday, 25 October 2009
  30. 30. What happened 4 server sends client request to worker passes all input data to worker Sunday, 25 October 2009
  31. 31. What happened 5 worker receives request and data processes input Sunday, 25 October 2009
  32. 32. What happened 6 worker returns processed data server passes it to client Sunday, 25 October 2009
  33. 33. What happened 7 client receives processed data client displays result Sunday, 25 October 2009
  34. 34. A simple Perl worker Sunday, 25 October 2009
  35. 35. A simple perl worker 1. add server 2. add function 3. loop 4. function definition Sunday, 25 October 2009
  36. 36. simple worker (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Worker; my $host = '127.0.0.1'; my $port = 4730; my $worker = new Gearman::XS::Worker; my $ret = $worker->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); exit(1); } Sunday, 25 October 2009
  37. 37. simple worker (2) my $options = ''; $ret = $worker->add_function( "reverse", # public function name 0, # timeout &myreverse, # reference to function $options); # function arguments if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Sunday, 25 October 2009
  38. 38. simple worker (3) while (1) { my $ret = $worker->work(); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } } Sunday, 25 October 2009
  39. 39. simple worker (4) sub myreverse { my ($job) = @_; my $workload = $job->workload(); my $result = reverse($workload); printf( "Job=%s F_Name=%s Workload=%s Result=%sn", $job->handle(), $job->function_name(), $job->workload(), $result); return $result; } Sunday, 25 October 2009
  40. 40. A simple Perl client Sunday, 25 October 2009
  41. 41. A simple perl client • add server • run a task Sunday, 25 October 2009
  42. 42. simple client (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Client; my $client = new Gearman::XS::Client; my $host = '127.0.0.1'; my $port = 4730; my $ret = $client->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $client->error()); exit(1); } Sunday, 25 October 2009
  43. 43. simple client (2) my $input = shift || 'teststring'; my ($return, $result) = $client->do("reverse", $input); if ($return == GEARMAN_SUCCESS) { printf("Result=%sn", $result); } Sunday, 25 October 2009
  44. 44. A sample run Sunday, 25 October 2009
  45. 45. host 1 perl worker.pl Sunday, 25 October 2009
  46. 46. host 2 perl client.pl Result=gnirtstset Sunday, 25 October 2009
  47. 47. host 1 perl worker.pl Job=H:gmac3.local:4 F_Name=reverse Workload=teststring Result=gnirtstset Sunday, 25 October 2009
  48. 48. more client functions • do_background • add_task • run_tasks Sunday, 25 October 2009
  49. 49. Some advanced usage • DBIx::SQLCrosstab • Data cubes • Perl only Sunday, 25 October 2009
  50. 50. Sunday, 25 October 2009
  51. 51. Image processing • CPU intensive • Large storage needed • Application is OS specific Sunday, 25 October 2009
  52. 52. Sunday, 25 October 2009
  53. 53. Sunday, 25 October 2009
  54. 54. Remote sandboxes Using GEARMAN client server architecture N servers N workers N clients Supports multiple languages Sunday, 25 October 2009
  55. 55. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman server func3 worker4 func1 func3 Gearman server client worker5 func1 host3 func3 host4 Sunday, 25 October 2009
  56. 56. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman Gearman Master server server Slave1 func3 worker4 func1 Gearman func3 server client worker5 func1 host3 Slave2 func3 host4 Sunday, 25 October 2009
  57. 57. Remote sandbox worker •add server •add function •loop •function definition Sunday, 25 October 2009
  58. 58. remote sandbox worker (2) $ret = $worker->add_function( "make_sandbox", 0, &make_sandbox, ''); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Sunday, 25 October 2009
  59. 59. remote sandbox worker (4) sub make_sandbox { my ($job, $options) = @_; my $command = $job->workload(); chomp $command; print STDERR "<$command>n"; my $result = qx(make_sandbox $command) ; if ($?) { print $result, "n"; return "Error ($!)n"; } else { return "okn" } } Sunday, 25 October 2009
  60. 60. Client to install Remote sandboxes •add server •run remote installation on master •run remote installation on each slave •run configuration queries on slaves Sunday, 25 October 2009
  61. 61. sample call to remote worker my $ret = qx(echo "$MASTER_SANDBOX" | gearman -h $master_ip -f make_sandbox ) ; # using the gearman command line utility Sunday, 25 October 2009
  62. 62. More on similar matters • FOSS.MY, Kuala Lumpur, Malaysia, 25-Oct-2009 • CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009 http://datacharmer.blogspot.com http://gearman.org Sunday, 25 October 2009
  63. 63. THANKS Let's talk! This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http:// creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Sunday, 25 October 2009
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×