Search

Gearman for MySQL

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    Notes on slide 1

    2 Favorites

    Gearman for MySQL - Presentation Transcript

    1. Gearman for MySQL Giuseppe Maxia MySQL Community Team Lead Sun Microsystems This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Thursday, 12 November 2009 1
    2. Mainframe terminal Mainframe terminal operating system client terminal hardware application terminal 0 100 USER FRIENDLINESS Thursday, 12 November 2009 2
    3. Mini computers Mainframe Mini terminal terminal operating system Mini client hardware terminal application terminal 0 100 USER FRIENDLINESS Thursday, 12 November 2009 3
    4. Networks server personal computer personal operating operating computer system system personal client hardware hardware computer hardware personal application 0 computer 100 USER FRIENDLINESS Thursday, 12 November 2009 4
    5. Web applications web server browser INTERNET browser operating operating operating system system systemoperating system client browser hardware hardware hardware application browser 0 100 USER FRIENDLINESS Thursday, 12 November 2009 5
    6. service Cloud applications provider service provider service provider service browser provider web INTERNET server browser operating operating operating system system system client browser hardware hardware hardware application applicationapplication browser 0 100 USER FRIENDLINESS Thursday, 12 November 2009 6
    7. Some actors • memcached • gearman Used in production by Facebook and Digg Thursday, 12 November 2009 7
    8. GEARMAN Thursday, 12 November 2009 8
    9. != G ER M A N Thursday, 12 November 2009 9
    10. G E A R M A N? Thursday, 12 November 2009 10
    11. MANAGER Thursday, 12 November 2009 11
    12. job task request server worker client http://gearman.org Thursday, 12 November 2009 12
    13. Gearman: a technology for distributed computing Thursday, 12 November 2009 13
    14. Distributed Thursday, 12 November 2009 14
    15. Multiple operating systems Thursday, 12 November 2009 15
    16. multiple languages Thursday, 12 November 2009 16
    17. freedom of choice Thursday, 12 November 2009 17
    18. redundancy Thursday, 12 November 2009 18
    19. USING GEARMAN • Server: gearmand • Client libraries: • C/C++ • Java • Perl • PHP • Python Thursday, 12 November 2009 19
    20. Simple usage GEARMAN • Command line client and worker Thursday, 12 November 2009 20
    21. starting the server /usr/local/sbin/gearmand -d # started as daemon. # No feedback given on the command line Thursday, 12 November 2009 21
    22. starting the server (2) /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) # started as normal application # verbose output requested Thursday, 12 November 2009 22
    23. starting the worker gearman -w -h hostname -p 4730 -f conta wc # -w = act as worker # -f = function # conta = function name # wc = command to execute when function # 'conta' is called Thursday, 12 November 2009 23
    24. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected Thursday, 12 November 2009 24
    25. starting the client gearman -h hostname -p 4730 -f conta < ~/.bashrc 57 135 2149 # <- output # from worker # -f = function # conta = function name # < ~/.bashrc = input data Thursday, 12 November 2009 25
    26. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected … INFO Accepted connection from 127.0.0.1:5181 INFO [ 0] 127.0.0.1:5181 Connected INFO [ 0] 127.0.0.1:5181 Disconnected Thursday, 12 November 2009 26
    27. What happened 1 server start listen to port 4730 Thursday, 12 November 2009 27
    28. What happened 2 worker starts registers function 'conta' to server Thursday, 12 November 2009 28
    29. What happened 3 client starts requires function 'conta' from server provides input data Thursday, 12 November 2009 29
    30. What happened 4 server sends client request to worker passes all input data to worker Thursday, 12 November 2009 30
    31. What happened 5 worker receives request and data processes input Thursday, 12 November 2009 31
    32. What happened 6 worker returns processed data server passes it to client Thursday, 12 November 2009 32
    33. What happened 7 client receives processed data client displays result Thursday, 12 November 2009 33
    34. A simple Perl worker Thursday, 12 November 2009 34
    35. A simple perl worker 1. add server 2. add function 3. loop 4. function definition Thursday, 12 November 2009 35
    36. simple worker (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Worker; my $host = '127.0.0.1'; my $port = 4730; my $worker = new Gearman::XS::Worker; my $ret = $worker->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); exit(1); } Thursday, 12 November 2009 36
    37. simple worker (2) my $options = ''; $ret = $worker->add_function( "reverse", # public function name 0, # timeout &myreverse, # reference to function $options); # function arguments if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Thursday, 12 November 2009 37
    38. simple worker (3) while (1) { my $ret = $worker->work(); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } } Thursday, 12 November 2009 38
    39. simple worker (4) sub myreverse { my ($job) = @_; my $workload = $job->workload(); my $result = reverse($workload); printf( "Job=%s F_Name=%s Workload=%s Result=%sn", $job->handle(), $job->function_name(), $job->workload(), $result); return $result; } Thursday, 12 November 2009 39
    40. A simple Perl client Thursday, 12 November 2009 40
    41. A simple perl client • add server • run a task Thursday, 12 November 2009 41
    42. simple client (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Client; my $client = new Gearman::XS::Client; my $host = '127.0.0.1'; my $port = 4730; my $ret = $client->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $client->error()); exit(1); } Thursday, 12 November 2009 42
    43. simple client (2) my $input = shift || 'teststring'; my ($return, $result) = $client->do("reverse", $input); if ($return == GEARMAN_SUCCESS) { printf("Result=%sn", $result); } Thursday, 12 November 2009 43
    44. A sample run Thursday, 12 November 2009 44
    45. host 1 perl worker.pl Thursday, 12 November 2009 45
    46. host 2 perl client.pl Result=gnirtstset Thursday, 12 November 2009 46
    47. host 1 perl worker.pl Job=H:gmac3.local:4 F_Name=reverse Workload=teststring Result=gnirtstset Thursday, 12 November 2009 47
    48. more client functions • do_background • add_task • run_tasks Thursday, 12 November 2009 48
    49. Some advanced usage • DBIx::SQLCrosstab • Data cubes • Perl only Thursday, 12 November 2009 49
    50. Thursday, 12 November 2009 50
    51. Image processing • CPU intensive • Large storage needed • Application is OS specific Thursday, 12 November 2009 51
    52. Thursday, 12 November 2009 52
    53. Remote sandboxes Using GEARMAN client server architecture N servers N workers N clients Supports multiple languages Thursday, 12 November 2009 53
    54. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman server func3 worker4 func1 func3 Gearman server client worker5 func1 host3 func3 host4 Thursday, 12 November 2009 54
    55. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman Gearman Master server server Slave1 func3 worker4 func1 Gearman func3 server client worker5 func1 host3 Slave2 func3 host4 Thursday, 12 November 2009 55
    56. Remote sandbox worker •add server •add function •loop •function definition Thursday, 12 November 2009 56
    57. remote sandbox worker (2) $ret = $worker->add_function( "make_sandbox", 0, &make_sandbox, ''); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Thursday, 12 November 2009 57
    58. remote sandbox worker (4) sub make_sandbox { my ($job, $options) = @_; my $command = $job->workload(); chomp $command; print STDERR "<$command>n"; my $result = qx(make_sandbox $command) ; if ($?) { print $result, "n"; return "Error ($!)n"; } else { return "okn" } } Thursday, 12 November 2009 58
    59. Client to install Remote sandboxes •add server •run remote installation on master •run remote installation on each slave •run configuration queries on slaves Thursday, 12 November 2009 59
    60. sample call to remote worker my $ret = qx(echo "$MASTER_SANDBOX" | gearman -h $master_ip -f make_sandbox ) ; # using the gearman command line utility Thursday, 12 November 2009 60
    61. Gearman for the MySQL server Thursday, 12 November 2009 61
    62. Gearman UDF Thursday, 12 November 2009 62
    63. More power to MySQL • Perl/PHP/Python functions • Shell access (you can send and receive email!) • filesystem access • advanced monitoring through Gearman features Thursday, 12 November 2009 63
    64. WARNING! You can easily shoot yourself in the foot Thursday, 12 November 2009 64
    65. Using MySQL UDF •Install Gearman •Get MySQL server source (needed to compile the UDF) •Get the UDF source •https://launchpad.net/gearman-mysql-udf •Follow the instructions (good luck) :-) Thursday, 12 November 2009 65
    66. Installation (1) export DEPS_LIBS="-L/usr/local/lib -lgearman" export DEPS_CFLAGS="-I/usr/local/include/ libgearman-server/ -I/usr/local/include/libgearman" ./configure --with-mysql=$MYSQL_BIN/bin/mysql_config --libdir=$MYSQL_BIN/lib/mysql/plugin/ make && make install Thursday, 12 November 2009 66
    67. Installation (2) mysql> CREATE FUNCTION gman_do RETURNS STRING SONAME "libgearman_mysql_udf.so"; mysql> CREATE FUNCTION gman_servers_set RETURNS STRING SONAME "libgearman_mysql_udf.so"; # there are more functions, but these two # are the MUST-have ones Thursday, 12 November 2009 67
    68. Installation (3) # register at least one server mysql> select gman_servers_set ('192.168.1.45:4730'); # optionally, a server can be associated # with a function mysql> select gman_servers_set ('192.168.1.45:4730', 'shell'); Thursday, 12 November 2009 68
    69. Create a worker (1) my @functions = ( ['reverse', &myreverse], ['count', &mycount], ['shell', &myshell], ['eval', &myeval], ['store', &mystore], ); # see the full code here: #http://forge.mysql.com/tools/tool.php?id=235 Thursday, 12 November 2009 69
    70. Create a worker (2) for my $func (@functions) { $ret = $worker->add_function( $func->[0], 0, $func->[1], $options); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "error with function %s - %sn", $func->[0], $worker->error()); } } Thursday, 12 November 2009 70
    71. Create a worker (3) sub myshell { my $job = shift; my $workload = $job->workload(); my $result = qx($workload); return $result; } # WARNING! # You can shoot yourself in the foot! Thursday, 12 November 2009 71
    72. Create a worker (4) sub myeval { my $job = shift; my $workload = $job->workload(); my $result = eval $workload; return $result; } # WARNING! # You can machine gun yourself # in the foot! Thursday, 12 November 2009 72
    73. Create a worker (4) sub myeval { my $job = shift; my $workload = $job->workload(); my $result = eval $workload; return $result; } # WARNING! # You can machine gun yourself # in the foot! Thursday, 12 November 2009 72
    74. Using the UDF mysql> select gman_do('reverse','abcd') as test; +------+ | test | +------+ | dcba | +------+ Thursday, 12 November 2009 73
    75. Using the UDF mysql> SELECT gman_do('shell', concat(' ls -lh ', (select variable_value from information_schema.global_variables where variable_name = "datadir" )))G total 40976 -rw-rw---- 1 gmax staff 5.0M Nov 11 13:34 ib_logfile0 -rw-rw---- 1 gmax staff 5.0M Nov 11 13:34 ib_logfile1 -rw-rw---- 1 gmax staff 10M Nov 11 13:34 ibdata1 -rw-rw---- 1 gmax staff 1.2K Nov 11 13:34 msandbox.err drwx------ 2 gmax staff 2.4K Nov 11 13:34 mysql -rw-rw---- 1 gmax staff 6B Nov 11 13:34 mysql_sandbox5140.pid drwx------ 2 gmax staff 68B Nov 11 13:34 test Thursday, 12 November 2009 74
    76. Using the UDF mysql> select gman_do('eval','2 * 3') ; +-------------------------+ | gman_do('eval','2 * 3') | +-------------------------+ | 6 | +-------------------------+ Thursday, 12 November 2009 75
    77. Using the UDF mysql> select gman_do('eval', concat('$_="',host,'";tr/a-z/b-za/; $_')) as test from mysql.user; +-------------+ | test | +-------------+ | % | | mpdbmiptu | +-------------+ Thursday, 12 November 2009 76
    78. Replication scenario Master slave slave Thursday, 12 November 2009 77
    79. Replication scenario slaves status Master slave 1 slave 2 slave slave Thursday, 12 November 2009 78
    80. More on similar matters • CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009 • Linux.Conf.Au, Wellington, NZ, 18-19-Jan-2010 http://datacharmer.blogspot.com http://gearman.org Thursday, 12 November 2009 79
    81. THANKS Let's talk! This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http:// creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Thursday, 12 November 2009 80
    SlideShare Zeitgeist 2009

    + Giuseppe MaxiaGiuseppe Maxia Nominate

    custom

    792 views, 2 favs, 8 embeds more stats

    Gearman is a client/server infrastructure for gener more

    More info about this document

    CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

    Go to text version

    • Total Views 792
      • 709 on SlideShare
      • 83 from embeds
    • Comments 0
    • Favorites 2
    • Downloads 45
    Most viewed embeds
    • 73 views on http://www.ningoo.net
    • 3 views on http://www.zhuaxia.com
    • 2 views on http://xianguo.com
    • 1 views on http://reader.youdao.com
    • 1 views on http://ningoo.net

    more

    All embeds
    • 73 views on http://www.ningoo.net
    • 3 views on http://www.zhuaxia.com
    • 2 views on http://xianguo.com
    • 1 views on http://reader.youdao.com
    • 1 views on http://ningoo.net
    • 1 views on http://203.208.37.132
    • 1 views on http://cache.baidu.com
    • 1 views on http://blog.zdnet.com.cn

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories

    Tags

    -->
    Search
    RSS Feed
    What's new?

    © 2009 SlideShare Inc. All Rights Reserved