Gearman For Beginners
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Gearman For Beginners

on

  • 7,780 views

With Gearman, a client/server infrastructure for generic tasks, you can work on distributed servers easily, with little worry about the details....

With Gearman, a client/server infrastructure for generic tasks, you can work on distributed servers easily, with little worry about the details.
No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more.

Statistics

Views

Total Views
7,780
Views on SlideShare
7,479
Embed Views
301

Actions

Likes
6
Downloads
274
Comments
0

4 Embeds 301

http://www.scoop.it 140
http://www.symfonylab.com 133
http://www.slideshare.net 27
http://localhost 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Gearman For Beginners Presentation Transcript

  • 1. r man Gea Sunday, 25 October 2009
  • 2. Gearman: a technology for distributed computing Giuseppe Maxia MySQL Community Team Lead Sun Microsystems Sunday, 25 October 2009
  • 3. Mainframe terminal Mainframe terminal operating system client terminal hardware application terminal 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  • 4. Mini computers Mainframe Mini terminal terminal operating system Mini client hardware terminal application terminal 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  • 5. Networks server personal computer personal operating operating computer system system personal client hardware hardware computer hardware personal application 0 computer 100 USER FRIENDLINESS Sunday, 25 October 2009
  • 6. Web applications web server browser INTERNET browser operating operating operating system system systemoperating system client browser hardware hardware hardware application browser 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  • 7. service Cloud applications provider service provider service provider service browser provider web INTERNET server browser operating operating operating system system system client browser hardware hardware hardware application applicationapplication browser 0 100 USER FRIENDLINESS Sunday, 25 October 2009
  • 8. Some actors • memcached • gearman Used in production by Facebook and Digg Sunday, 25 October 2009
  • 9. GEARMAN Sunday, 25 October 2009
  • 10. != G ER M A N Sunday, 25 October 2009
  • 11. G E A R M A N? Sunday, 25 October 2009
  • 12. MANAGER Sunday, 25 October 2009
  • 13. job task request server worker client http://gearman.org Sunday, 25 October 2009
  • 14. Distributed Sunday, 25 October 2009
  • 15. Multiple operating systems Sunday, 25 October 2009
  • 16. multiple languages Sunday, 25 October 2009
  • 17. freedom of choice Sunday, 25 October 2009
  • 18. redundancy Sunday, 25 October 2009
  • 19. USING GEARMAN • Server: gearmand • Client libraries: • C/C++ • Java • Perl • PHP • Python Sunday, 25 October 2009
  • 20. Simple usage GEARMAN • Command line client and worker Sunday, 25 October 2009
  • 21. starting the server /usr/local/sbin/gearmand -d # started as daemon. # No feedback given on the command line Sunday, 25 October 2009
  • 22. starting the server (2) /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) # started as normal application # verbose output requested Sunday, 25 October 2009
  • 23. starting the worker gearman -w -h hostname -p 4730 -f conta wc # -w = act as worker # -f = function # conta = function name # wc = command to execute when function # 'conta' is called Sunday, 25 October 2009
  • 24. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected Sunday, 25 October 2009
  • 25. starting the client gearman -h hostname -p 4730 -f conta < ~/.bashrc 57 135 2149 # <- output # from worker # -f = function # conta = function name # < ~/.bashrc = input data Sunday, 25 October 2009
  • 26. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected … INFO Accepted connection from 127.0.0.1:5181 INFO [ 0] 127.0.0.1:5181 Connected INFO [ 0] 127.0.0.1:5181 Disconnected Sunday, 25 October 2009
  • 27. What happened 1 server start listen to port 4730 Sunday, 25 October 2009
  • 28. What happened 2 worker starts registers function 'conta' to server Sunday, 25 October 2009
  • 29. What happened 3 client starts requires function 'conta' from server provides input data Sunday, 25 October 2009
  • 30. What happened 4 server sends client request to worker passes all input data to worker Sunday, 25 October 2009
  • 31. What happened 5 worker receives request and data processes input Sunday, 25 October 2009
  • 32. What happened 6 worker returns processed data server passes it to client Sunday, 25 October 2009
  • 33. What happened 7 client receives processed data client displays result Sunday, 25 October 2009
  • 34. A simple Perl worker Sunday, 25 October 2009
  • 35. A simple perl worker 1. add server 2. add function 3. loop 4. function definition Sunday, 25 October 2009
  • 36. simple worker (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Worker; my $host = '127.0.0.1'; my $port = 4730; my $worker = new Gearman::XS::Worker; my $ret = $worker->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); exit(1); } Sunday, 25 October 2009
  • 37. simple worker (2) my $options = ''; $ret = $worker->add_function( "reverse", # public function name 0, # timeout &myreverse, # reference to function $options); # function arguments if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Sunday, 25 October 2009
  • 38. simple worker (3) while (1) { my $ret = $worker->work(); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } } Sunday, 25 October 2009
  • 39. simple worker (4) sub myreverse { my ($job) = @_; my $workload = $job->workload(); my $result = reverse($workload); printf( "Job=%s F_Name=%s Workload=%s Result=%sn", $job->handle(), $job->function_name(), $job->workload(), $result); return $result; } Sunday, 25 October 2009
  • 40. A simple Perl client Sunday, 25 October 2009
  • 41. A simple perl client • add server • run a task Sunday, 25 October 2009
  • 42. simple client (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Client; my $client = new Gearman::XS::Client; my $host = '127.0.0.1'; my $port = 4730; my $ret = $client->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $client->error()); exit(1); } Sunday, 25 October 2009
  • 43. simple client (2) my $input = shift || 'teststring'; my ($return, $result) = $client->do("reverse", $input); if ($return == GEARMAN_SUCCESS) { printf("Result=%sn", $result); } Sunday, 25 October 2009
  • 44. A sample run Sunday, 25 October 2009
  • 45. host 1 perl worker.pl Sunday, 25 October 2009
  • 46. host 2 perl client.pl Result=gnirtstset Sunday, 25 October 2009
  • 47. host 1 perl worker.pl Job=H:gmac3.local:4 F_Name=reverse Workload=teststring Result=gnirtstset Sunday, 25 October 2009
  • 48. more client functions • do_background • add_task • run_tasks Sunday, 25 October 2009
  • 49. Some advanced usage • DBIx::SQLCrosstab • Data cubes • Perl only Sunday, 25 October 2009
  • 50. Sunday, 25 October 2009
  • 51. Image processing • CPU intensive • Large storage needed • Application is OS specific Sunday, 25 October 2009
  • 52. Sunday, 25 October 2009
  • 53. Sunday, 25 October 2009
  • 54. Remote sandboxes Using GEARMAN client server architecture N servers N workers N clients Supports multiple languages Sunday, 25 October 2009
  • 55. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman server func3 worker4 func1 func3 Gearman server client worker5 func1 host3 func3 host4 Sunday, 25 October 2009
  • 56. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman Gearman Master server server Slave1 func3 worker4 func1 Gearman func3 server client worker5 func1 host3 Slave2 func3 host4 Sunday, 25 October 2009
  • 57. Remote sandbox worker •add server •add function •loop •function definition Sunday, 25 October 2009
  • 58. remote sandbox worker (2) $ret = $worker->add_function( "make_sandbox", 0, &make_sandbox, ''); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Sunday, 25 October 2009
  • 59. remote sandbox worker (4) sub make_sandbox { my ($job, $options) = @_; my $command = $job->workload(); chomp $command; print STDERR "<$command>n"; my $result = qx(make_sandbox $command) ; if ($?) { print $result, "n"; return "Error ($!)n"; } else { return "okn" } } Sunday, 25 October 2009
  • 60. Client to install Remote sandboxes •add server •run remote installation on master •run remote installation on each slave •run configuration queries on slaves Sunday, 25 October 2009
  • 61. sample call to remote worker my $ret = qx(echo "$MASTER_SANDBOX" | gearman -h $master_ip -f make_sandbox ) ; # using the gearman command line utility Sunday, 25 October 2009
  • 62. More on similar matters • FOSS.MY, Kuala Lumpur, Malaysia, 25-Oct-2009 • CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009 http://datacharmer.blogspot.com http://gearman.org Sunday, 25 October 2009
  • 63. THANKS Let's talk! This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http:// creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Sunday, 25 October 2009