Gearman For Beginners

7,687 views
7,394 views

Published on

With Gearman, a client/server infrastructure for generic tasks, you can work on distributed servers easily, with little worry about the details.
No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more.

Published in: Technology

Gearman For Beginners

  1. 1. r man Gea Sunday, 25 October 2009
  2. 2. Gearman: a technology for distributed computing Giuseppe Maxia MySQL Community Team Lead Sun Microsystems Sunday, 25 October 2009
  3. 3. Mainframe terminal Mainframe terminal operating system client terminal hardware application terminal 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  4. 4. Mini computers Mainframe Mini terminal terminal operating system Mini client hardware terminal application terminal 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  5. 5. Networks server personal computer personal operating operating computer system system personal client hardware hardware computer hardware personal application 0 computer 100 USER FRIENDLINESS Sunday, 25 October 2009
  6. 6. Web applications web server browser INTERNET browser operating operating operating system system systemoperating system client browser hardware hardware hardware application browser 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  7. 7. service Cloud applications provider service provider service provider service browser provider web INTERNET server browser operating operating operating system system system client browser hardware hardware hardware application applicationapplication browser 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  8. 8. Some actors • memcached • gearman Used in production by Facebook and Digg Sunday, 25 October 2009
  9. 9. GEARMAN Sunday, 25 October 2009
  10. 10. != G ER M A N Sunday, 25 October 2009
  11. 11. G E A R M A N? Sunday, 25 October 2009
  12. 12. MANAGER Sunday, 25 October 2009
  13. 13. job task request server worker client http://gearman.org Sunday, 25 October 2009
  14. 14. Distributed Sunday, 25 October 2009
  15. 15. Multiple operating systems Sunday, 25 October 2009
  16. 16. multiple languages Sunday, 25 October 2009
  17. 17. freedom of choice Sunday, 25 October 2009
  18. 18. redundancy Sunday, 25 October 2009
  19. 19. USING GEARMAN • Server: gearmand • Client libraries: • C/C++ • Java • Perl • PHP • Python Sunday, 25 October 2009
  20. 20. Simple usage GEARMAN • Command line client and worker Sunday, 25 October 2009
  21. 21. starting the server /usr/local/sbin/gearmand -d # started as daemon. # No feedback given on the command line Sunday, 25 October 2009
  22. 22. starting the server (2) /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) # started as normal application # verbose output requested Sunday, 25 October 2009
  23. 23. starting the worker gearman -w -h hostname -p 4730 -f conta wc # -w = act as worker # -f = function # conta = function name # wc = command to execute when function # 'conta' is called Sunday, 25 October 2009
  24. 24. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected Sunday, 25 October 2009
  25. 25. starting the client gearman -h hostname -p 4730 -f conta < ~/.bashrc 57 135 2149 # <- output # from worker # -f = function # conta = function name # < ~/.bashrc = input data Sunday, 25 October 2009
  26. 26. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected … INFO Accepted connection from 127.0.0.1:5181 INFO [ 0] 127.0.0.1:5181 Connected INFO [ 0] 127.0.0.1:5181 Disconnected Sunday, 25 October 2009
  27. 27. What happened 1 server start listen to port 4730 Sunday, 25 October 2009
  28. 28. What happened 2 worker starts registers function 'conta' to server Sunday, 25 October 2009
  29. 29. What happened 3 client starts requires function 'conta' from server provides input data Sunday, 25 October 2009
  30. 30. What happened 4 server sends client request to worker passes all input data to worker Sunday, 25 October 2009
  31. 31. What happened 5 worker receives request and data processes input Sunday, 25 October 2009
  32. 32. What happened 6 worker returns processed data server passes it to client Sunday, 25 October 2009
  33. 33. What happened 7 client receives processed data client displays result Sunday, 25 October 2009
  34. 34. A simple Perl worker Sunday, 25 October 2009
  35. 35. A simple perl worker 1. add server 2. add function 3. loop 4. function definition Sunday, 25 October 2009
  36. 36. simple worker (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Worker; my $host = '127.0.0.1'; my $port = 4730; my $worker = new Gearman::XS::Worker; my $ret = $worker->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); exit(1); } Sunday, 25 October 2009
  37. 37. simple worker (2) my $options = ''; $ret = $worker->add_function( "reverse", # public function name 0, # timeout &myreverse, # reference to function $options); # function arguments if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Sunday, 25 October 2009
  38. 38. simple worker (3) while (1) { my $ret = $worker->work(); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } } Sunday, 25 October 2009
  39. 39. simple worker (4) sub myreverse { my ($job) = @_; my $workload = $job->workload(); my $result = reverse($workload); printf( "Job=%s F_Name=%s Workload=%s Result=%sn", $job->handle(), $job->function_name(), $job->workload(), $result); return $result; } Sunday, 25 October 2009
  40. 40. A simple Perl client Sunday, 25 October 2009
  41. 41. A simple perl client • add server • run a task Sunday, 25 October 2009
  42. 42. simple client (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Client; my $client = new Gearman::XS::Client; my $host = '127.0.0.1'; my $port = 4730; my $ret = $client->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $client->error()); exit(1); } Sunday, 25 October 2009
  43. 43. simple client (2) my $input = shift || 'teststring'; my ($return, $result) = $client->do("reverse", $input); if ($return == GEARMAN_SUCCESS) { printf("Result=%sn", $result); } Sunday, 25 October 2009
  44. 44. A sample run Sunday, 25 October 2009
  45. 45. host 1 perl worker.pl Sunday, 25 October 2009
  46. 46. host 2 perl client.pl Result=gnirtstset Sunday, 25 October 2009
  47. 47. host 1 perl worker.pl Job=H:gmac3.local:4 F_Name=reverse Workload=teststring Result=gnirtstset Sunday, 25 October 2009
  48. 48. more client functions • do_background • add_task • run_tasks Sunday, 25 October 2009
  49. 49. Some advanced usage • DBIx::SQLCrosstab • Data cubes • Perl only Sunday, 25 October 2009
  50. 50. Sunday, 25 October 2009
  51. 51. Image processing • CPU intensive • Large storage needed • Application is OS specific Sunday, 25 October 2009
  52. 52. Sunday, 25 October 2009
  53. 53. Sunday, 25 October 2009
  54. 54. Remote sandboxes Using GEARMAN client server architecture N servers N workers N clients Supports multiple languages Sunday, 25 October 2009
  55. 55. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman server func3 worker4 func1 func3 Gearman server client worker5 func1 host3 func3 host4 Sunday, 25 October 2009
  56. 56. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman Gearman Master server server Slave1 func3 worker4 func1 Gearman func3 server client worker5 func1 host3 Slave2 func3 host4 Sunday, 25 October 2009
  57. 57. Remote sandbox worker •add server •add function •loop •function definition Sunday, 25 October 2009
  58. 58. remote sandbox worker (2) $ret = $worker->add_function( "make_sandbox", 0, &make_sandbox, ''); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Sunday, 25 October 2009
  59. 59. remote sandbox worker (4) sub make_sandbox { my ($job, $options) = @_; my $command = $job->workload(); chomp $command; print STDERR "<$command>n"; my $result = qx(make_sandbox $command) ; if ($?) { print $result, "n"; return "Error ($!)n"; } else { return "okn" } } Sunday, 25 October 2009
  60. 60. Client to install Remote sandboxes •add server •run remote installation on master •run remote installation on each slave •run configuration queries on slaves Sunday, 25 October 2009
  61. 61. sample call to remote worker my $ret = qx(echo "$MASTER_SANDBOX" | gearman -h $master_ip -f make_sandbox ) ; # using the gearman command line utility Sunday, 25 October 2009
  62. 62. More on similar matters • FOSS.MY, Kuala Lumpur, Malaysia, 25-Oct-2009 • CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009 http://datacharmer.blogspot.com http://gearman.org Sunday, 25 October 2009
  63. 63. THANKS Let's talk! This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http:// creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Sunday, 25 October 2009

×