Search

Gearman For Beginners

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    Notes on slide 1

    1 Favorite

    Gearman For Beginners - Presentation Transcript

    1. r man Gea Sunday, 25 October 2009
    2. Gearman: a technology for distributed computing Giuseppe Maxia MySQL Community Team Lead Sun Microsystems Sunday, 25 October 2009
    3. Mainframe terminal Mainframe terminal operating system client terminal hardware application terminal 0 100 USER FRIENDLINESS Sunday, 25 October 2009
    4. Mini computers Mainframe Mini terminal terminal operating system Mini client hardware terminal application terminal 0 100 USER FRIENDLINESS Sunday, 25 October 2009
    5. Networks server personal computer personal operating operating computer system system personal client hardware hardware computer hardware personal application 0 computer 100 USER FRIENDLINESS Sunday, 25 October 2009
    6. Web applications web server browser INTERNET browser operating operating operating system system systemoperating system client browser hardware hardware hardware application browser 0 100 USER FRIENDLINESS Sunday, 25 October 2009
    7. service Cloud applications provider service provider service provider service browser provider web INTERNET server browser operating operating operating system system system client browser hardware hardware hardware application applicationapplication browser 0 100 USER FRIENDLINESS Sunday, 25 October 2009
    8. Some actors • memcached • gearman Used in production by Facebook and Digg Sunday, 25 October 2009
    9. GEARMAN Sunday, 25 October 2009
    10. != G ER M A N Sunday, 25 October 2009
    11. G E A R M A N? Sunday, 25 October 2009
    12. MANAGER Sunday, 25 October 2009
    13. job task request server worker client http://gearman.org Sunday, 25 October 2009
    14. Distributed Sunday, 25 October 2009
    15. Multiple operating systems Sunday, 25 October 2009
    16. multiple languages Sunday, 25 October 2009
    17. freedom of choice Sunday, 25 October 2009
    18. redundancy Sunday, 25 October 2009
    19. USING GEARMAN • Server: gearmand • Client libraries: • C/C++ • Java • Perl • PHP • Python Sunday, 25 October 2009
    20. Simple usage GEARMAN • Command line client and worker Sunday, 25 October 2009
    21. starting the server /usr/local/sbin/gearmand -d # started as daemon. # No feedback given on the command line Sunday, 25 October 2009
    22. starting the server (2) /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) # started as normal application # verbose output requested Sunday, 25 October 2009
    23. starting the worker gearman -w -h hostname -p 4730 -f conta wc # -w = act as worker # -f = function # conta = function name # wc = command to execute when function # 'conta' is called Sunday, 25 October 2009
    24. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected Sunday, 25 October 2009
    25. starting the client gearman -h hostname -p 4730 -f conta < ~/.bashrc 57 135 2149 # <- output # from worker # -f = function # conta = function name # < ~/.bashrc = input data Sunday, 25 October 2009
    26. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected … INFO Accepted connection from 127.0.0.1:5181 INFO [ 0] 127.0.0.1:5181 Connected INFO [ 0] 127.0.0.1:5181 Disconnected Sunday, 25 October 2009
    27. What happened 1 server start listen to port 4730 Sunday, 25 October 2009
    28. What happened 2 worker starts registers function 'conta' to server Sunday, 25 October 2009
    29. What happened 3 client starts requires function 'conta' from server provides input data Sunday, 25 October 2009
    30. What happened 4 server sends client request to worker passes all input data to worker Sunday, 25 October 2009
    31. What happened 5 worker receives request and data processes input Sunday, 25 October 2009
    32. What happened 6 worker returns processed data server passes it to client Sunday, 25 October 2009
    33. What happened 7 client receives processed data client displays result Sunday, 25 October 2009
    34. A simple Perl worker Sunday, 25 October 2009
    35. A simple perl worker 1. add server 2. add function 3. loop 4. function definition Sunday, 25 October 2009
    36. simple worker (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Worker; my $host = '127.0.0.1'; my $port = 4730; my $worker = new Gearman::XS::Worker; my $ret = $worker->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); exit(1); } Sunday, 25 October 2009
    37. simple worker (2) my $options = ''; $ret = $worker->add_function( "reverse", # public function name 0, # timeout &myreverse, # reference to function $options); # function arguments if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Sunday, 25 October 2009
    38. simple worker (3) while (1) { my $ret = $worker->work(); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } } Sunday, 25 October 2009
    39. simple worker (4) sub myreverse { my ($job) = @_; my $workload = $job->workload(); my $result = reverse($workload); printf( "Job=%s F_Name=%s Workload=%s Result=%sn", $job->handle(), $job->function_name(), $job->workload(), $result); return $result; } Sunday, 25 October 2009
    40. A simple Perl client Sunday, 25 October 2009
    41. A simple perl client • add server • run a task Sunday, 25 October 2009
    42. simple client (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Client; my $client = new Gearman::XS::Client; my $host = '127.0.0.1'; my $port = 4730; my $ret = $client->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $client->error()); exit(1); } Sunday, 25 October 2009
    43. simple client (2) my $input = shift || 'teststring'; my ($return, $result) = $client->do("reverse", $input); if ($return == GEARMAN_SUCCESS) { printf("Result=%sn", $result); } Sunday, 25 October 2009
    44. A sample run Sunday, 25 October 2009
    45. host 1 perl worker.pl Sunday, 25 October 2009
    46. host 2 perl client.pl Result=gnirtstset Sunday, 25 October 2009
    47. host 1 perl worker.pl Job=H:gmac3.local:4 F_Name=reverse Workload=teststring Result=gnirtstset Sunday, 25 October 2009
    48. more client functions • do_background • add_task • run_tasks Sunday, 25 October 2009
    49. Some advanced usage • DBIx::SQLCrosstab • Data cubes • Perl only Sunday, 25 October 2009
    50. Sunday, 25 October 2009
    51. Image processing • CPU intensive • Large storage needed • Application is OS specific Sunday, 25 October 2009
    52. Sunday, 25 October 2009
    53. Sunday, 25 October 2009
    54. Remote sandboxes Using GEARMAN client server architecture N servers N workers N clients Supports multiple languages Sunday, 25 October 2009
    55. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman server func3 worker4 func1 func3 Gearman server client worker5 func1 host3 func3 host4 Sunday, 25 October 2009
    56. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman Gearman Master server server Slave1 func3 worker4 func1 Gearman func3 server client worker5 func1 host3 Slave2 func3 host4 Sunday, 25 October 2009
    57. Remote sandbox worker •add server •add function •loop •function definition Sunday, 25 October 2009
    58. remote sandbox worker (2) $ret = $worker->add_function( "make_sandbox", 0, &make_sandbox, ''); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Sunday, 25 October 2009
    59. remote sandbox worker (4) sub make_sandbox { my ($job, $options) = @_; my $command = $job->workload(); chomp $command; print STDERR "<$command>n"; my $result = qx(make_sandbox $command) ; if ($?) { print $result, "n"; return "Error ($!)n"; } else { return "okn" } } Sunday, 25 October 2009
    60. Client to install Remote sandboxes •add server •run remote installation on master •run remote installation on each slave •run configuration queries on slaves Sunday, 25 October 2009
    61. sample call to remote worker my $ret = qx(echo "$MASTER_SANDBOX" | gearman -h $master_ip -f make_sandbox ) ; # using the gearman command line utility Sunday, 25 October 2009
    62. More on similar matters • FOSS.MY, Kuala Lumpur, Malaysia, 25-Oct-2009 • CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009 http://datacharmer.blogspot.com http://gearman.org Sunday, 25 October 2009
    63. THANKS Let's talk! This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http:// creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Sunday, 25 October 2009

    + Giuseppe MaxiaGiuseppe Maxia, 3 weeks ago

    custom

    244 views, 1 favs, 0 embeds more stats

    With Gearman, a client/server infrastructure for ge more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 244
      • 244 on SlideShare
      • 0 from embeds
    • Comments 0
    • Favorites 1
    • Downloads 8
    Most viewed embeds

    more

    All embeds

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories

    Tags

    -->
    Search
    RSS Feed
    What's new?

    © 2009 SlideShare Inc. All Rights Reserved