Gearman for MySQL
Upcoming SlideShare
Loading in...5
×
 

Gearman for MySQL

on

  • 12,888 views

Gearman is a client/server infrastructure for generic tasks, usable on distributed servers, with little worry about the details. No matter what language you speak, Gearman can meet your needs in C, ...

Gearman is a client/server infrastructure for generic tasks, usable on distributed servers, with little worry about the details. No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more. Gearman can also work in conjunction with MySQL, either using UDFs, or simply through its basic architecture.
This talk will show examples of how to use Gearman for remote installation and how to call functions written in Perl from any other language or from inside MySQL server, with no knowledge of Perl at all.

Statistics

Views

Total Views
12,888
Views on SlideShare
10,653
Embed Views
2,235

Actions

Likes
16
Downloads
330
Comments
0

28 Embeds 2,235

http://www.ningoo.net 1480
http://blogread.cn 190
http://soft.zdnet.com.cn 179
http://www.symfonylab.com 129
http://blog.bypat.com 103
http://wiki.simplexi.com 43
http://www.slideshare.net 28
http://codebits.eu 15
http://nwlinux.com 12
http://cache.baidu.com 10
https://codebits.eu 7
http://xianguo.com 4
http://www.builder.com.cn 4
http://www.aicuxiao.com 4
http://translate.googleusercontent.com 3
http://203.231.63.114 3
http://blog.zdnet.com.cn 3
http://cache.baiducontent.com 3
http://www.zhuaxia.com 3
http://reader.youdao.com 2
http://webcache.googleusercontent.com 2
http://www.blogread.cn 2
http://203.208.39.132 1
http://wiki.simplexi.com:8080 1
http://203.208.37.132 1
http://ningoo.net 1
http://buckycomputing.net 1
http://rbucky.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Gearman for MySQL Gearman for MySQL Presentation Transcript

    • Gearman for MySQL Giuseppe Maxia MySQL Community Team Lead Sun Microsystems This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Thursday, 12 November 2009 1
    • Mainframe terminal Mainframe terminal operating system client terminal hardware application terminal 0 100 USER FRIENDLINESS Thursday, 12 November 2009 2
    • Mini computers Mainframe Mini terminal terminal operating system Mini client hardware terminal application terminal 0 100 USER FRIENDLINESS Thursday, 12 November 2009 3
    • Networks server personal computer personal operating operating computer system system personal client hardware hardware computer hardware personal application 0 computer 100 USER FRIENDLINESS Thursday, 12 November 2009 4
    • Web applications web server browser INTERNET browser operating operating operating system system systemoperating system client browser hardware hardware hardware application browser 0 100 USER FRIENDLINESS Thursday, 12 November 2009 5
    • service Cloud applications provider service provider service provider service browser provider web INTERNET server browser operating operating operating system system system client browser hardware hardware hardware application applicationapplication browser 0 100 USER FRIENDLINESS Thursday, 12 November 2009 6
    • Some actors • memcached • gearman Used in production by Facebook and Digg Thursday, 12 November 2009 7
    • GEARMAN Thursday, 12 November 2009 8
    • != G ER M A N Thursday, 12 November 2009 9
    • G E A R M A N? Thursday, 12 November 2009 10
    • MANAGER Thursday, 12 November 2009 11
    • job task request server worker client http://gearman.org Thursday, 12 November 2009 12
    • Gearman: a technology for distributed computing Thursday, 12 November 2009 13
    • Distributed Thursday, 12 November 2009 14
    • Multiple operating systems Thursday, 12 November 2009 15
    • multiple languages Thursday, 12 November 2009 16
    • freedom of choice Thursday, 12 November 2009 17
    • redundancy Thursday, 12 November 2009 18
    • USING GEARMAN • Server: gearmand • Client libraries: • C/C++ • Java • Perl • PHP • Python Thursday, 12 November 2009 19
    • Simple usage GEARMAN • Command line client and worker Thursday, 12 November 2009 20
    • starting the server /usr/local/sbin/gearmand -d # started as daemon. # No feedback given on the command line Thursday, 12 November 2009 21
    • starting the server (2) /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) # started as normal application # verbose output requested Thursday, 12 November 2009 22
    • starting the worker gearman -w -h hostname -p 4730 -f conta wc # -w = act as worker # -f = function # conta = function name # wc = command to execute when function # 'conta' is called Thursday, 12 November 2009 23
    • what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected Thursday, 12 November 2009 24
    • starting the client gearman -h hostname -p 4730 -f conta < ~/.bashrc 57 135 2149 # <- output # from worker # -f = function # conta = function name # < ~/.bashrc = input data Thursday, 12 November 2009 25
    • what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected … INFO Accepted connection from 127.0.0.1:5181 INFO [ 0] 127.0.0.1:5181 Connected INFO [ 0] 127.0.0.1:5181 Disconnected Thursday, 12 November 2009 26
    • What happened 1 server start listen to port 4730 Thursday, 12 November 2009 27
    • What happened 2 worker starts registers function 'conta' to server Thursday, 12 November 2009 28
    • What happened 3 client starts requires function 'conta' from server provides input data Thursday, 12 November 2009 29
    • What happened 4 server sends client request to worker passes all input data to worker Thursday, 12 November 2009 30
    • What happened 5 worker receives request and data processes input Thursday, 12 November 2009 31
    • What happened 6 worker returns processed data server passes it to client Thursday, 12 November 2009 32
    • What happened 7 client receives processed data client displays result Thursday, 12 November 2009 33
    • A simple Perl worker Thursday, 12 November 2009 34
    • A simple perl worker 1. add server 2. add function 3. loop 4. function definition Thursday, 12 November 2009 35
    • simple worker (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Worker; my $host = '127.0.0.1'; my $port = 4730; my $worker = new Gearman::XS::Worker; my $ret = $worker->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); exit(1); } Thursday, 12 November 2009 36
    • simple worker (2) my $options = ''; $ret = $worker->add_function( "reverse", # public function name 0, # timeout &myreverse, # reference to function $options); # function arguments if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Thursday, 12 November 2009 37
    • simple worker (3) while (1) { my $ret = $worker->work(); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } } Thursday, 12 November 2009 38
    • simple worker (4) sub myreverse { my ($job) = @_; my $workload = $job->workload(); my $result = reverse($workload); printf( "Job=%s F_Name=%s Workload=%s Result=%sn", $job->handle(), $job->function_name(), $job->workload(), $result); return $result; } Thursday, 12 November 2009 39
    • A simple Perl client Thursday, 12 November 2009 40
    • A simple perl client • add server • run a task Thursday, 12 November 2009 41
    • simple client (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Client; my $client = new Gearman::XS::Client; my $host = '127.0.0.1'; my $port = 4730; my $ret = $client->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $client->error()); exit(1); } Thursday, 12 November 2009 42
    • simple client (2) my $input = shift || 'teststring'; my ($return, $result) = $client->do("reverse", $input); if ($return == GEARMAN_SUCCESS) { printf("Result=%sn", $result); } Thursday, 12 November 2009 43
    • A sample run Thursday, 12 November 2009 44
    • host 1 perl worker.pl Thursday, 12 November 2009 45
    • host 2 perl client.pl Result=gnirtstset Thursday, 12 November 2009 46
    • host 1 perl worker.pl Job=H:gmac3.local:4 F_Name=reverse Workload=teststring Result=gnirtstset Thursday, 12 November 2009 47
    • more client functions • do_background • add_task • run_tasks Thursday, 12 November 2009 48
    • Some advanced usage • DBIx::SQLCrosstab • Data cubes • Perl only Thursday, 12 November 2009 49
    • Thursday, 12 November 2009 50
    • Image processing • CPU intensive • Large storage needed • Application is OS specific Thursday, 12 November 2009 51
    • Thursday, 12 November 2009 52
    • Remote sandboxes Using GEARMAN client server architecture N servers N workers N clients Supports multiple languages Thursday, 12 November 2009 53
    • host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman server func3 worker4 func1 func3 Gearman server client worker5 func1 host3 func3 host4 Thursday, 12 November 2009 54
    • host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman Gearman Master server server Slave1 func3 worker4 func1 Gearman func3 server client worker5 func1 host3 Slave2 func3 host4 Thursday, 12 November 2009 55
    • Remote sandbox worker •add server •add function •loop •function definition Thursday, 12 November 2009 56
    • remote sandbox worker (2) $ret = $worker->add_function( "make_sandbox", 0, &make_sandbox, ''); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Thursday, 12 November 2009 57
    • remote sandbox worker (4) sub make_sandbox { my ($job, $options) = @_; my $command = $job->workload(); chomp $command; print STDERR "<$command>n"; my $result = qx(make_sandbox $command) ; if ($?) { print $result, "n"; return "Error ($!)n"; } else { return "okn" } } Thursday, 12 November 2009 58
    • Client to install Remote sandboxes •add server •run remote installation on master •run remote installation on each slave •run configuration queries on slaves Thursday, 12 November 2009 59
    • sample call to remote worker my $ret = qx(echo "$MASTER_SANDBOX" | gearman -h $master_ip -f make_sandbox ) ; # using the gearman command line utility Thursday, 12 November 2009 60
    • Gearman for the MySQL server Thursday, 12 November 2009 61
    • Gearman UDF Thursday, 12 November 2009 62
    • More power to MySQL • Perl/PHP/Python functions • Shell access (you can send and receive email!) • filesystem access • advanced monitoring through Gearman features Thursday, 12 November 2009 63
    • WARNING! You can easily shoot yourself in the foot Thursday, 12 November 2009 64
    • Using MySQL UDF •Install Gearman •Get MySQL server source (needed to compile the UDF) •Get the UDF source •https://launchpad.net/gearman-mysql-udf •Follow the instructions (good luck) :-) Thursday, 12 November 2009 65
    • Installation (1) export DEPS_LIBS="-L/usr/local/lib -lgearman" export DEPS_CFLAGS="-I/usr/local/include/ libgearman-server/ -I/usr/local/include/libgearman" ./configure --with-mysql=$MYSQL_BIN/bin/mysql_config --libdir=$MYSQL_BIN/lib/mysql/plugin/ make && make install Thursday, 12 November 2009 66
    • Installation (2) mysql> CREATE FUNCTION gman_do RETURNS STRING SONAME "libgearman_mysql_udf.so"; mysql> CREATE FUNCTION gman_servers_set RETURNS STRING SONAME "libgearman_mysql_udf.so"; # there are more functions, but these two # are the MUST-have ones Thursday, 12 November 2009 67
    • Installation (3) # register at least one server mysql> select gman_servers_set ('192.168.1.45:4730'); # optionally, a server can be associated # with a function mysql> select gman_servers_set ('192.168.1.45:4730', 'shell'); Thursday, 12 November 2009 68
    • Create a worker (1) my @functions = ( ['reverse', &myreverse], ['count', &mycount], ['shell', &myshell], ['eval', &myeval], ['store', &mystore], ); # see the full code here: #http://forge.mysql.com/tools/tool.php?id=235 Thursday, 12 November 2009 69
    • Create a worker (2) for my $func (@functions) { $ret = $worker->add_function( $func->[0], 0, $func->[1], $options); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "error with function %s - %sn", $func->[0], $worker->error()); } } Thursday, 12 November 2009 70
    • Create a worker (3) sub myshell { my $job = shift; my $workload = $job->workload(); my $result = qx($workload); return $result; } # WARNING! # You can shoot yourself in the foot! Thursday, 12 November 2009 71
    • Create a worker (4) sub myeval { my $job = shift; my $workload = $job->workload(); my $result = eval $workload; return $result; } # WARNING! # You can machine gun yourself # in the foot! Thursday, 12 November 2009 72
    • Create a worker (4) sub myeval { my $job = shift; my $workload = $job->workload(); my $result = eval $workload; return $result; } # WARNING! # You can machine gun yourself # in the foot! Thursday, 12 November 2009 72
    • Using the UDF mysql> select gman_do('reverse','abcd') as test; +------+ | test | +------+ | dcba | +------+ Thursday, 12 November 2009 73
    • Using the UDF mysql> SELECT gman_do('shell', concat(' ls -lh ', (select variable_value from information_schema.global_variables where variable_name = "datadir" )))G total 40976 -rw-rw---- 1 gmax staff 5.0M Nov 11 13:34 ib_logfile0 -rw-rw---- 1 gmax staff 5.0M Nov 11 13:34 ib_logfile1 -rw-rw---- 1 gmax staff 10M Nov 11 13:34 ibdata1 -rw-rw---- 1 gmax staff 1.2K Nov 11 13:34 msandbox.err drwx------ 2 gmax staff 2.4K Nov 11 13:34 mysql -rw-rw---- 1 gmax staff 6B Nov 11 13:34 mysql_sandbox5140.pid drwx------ 2 gmax staff 68B Nov 11 13:34 test Thursday, 12 November 2009 74
    • Using the UDF mysql> select gman_do('eval','2 * 3') ; +-------------------------+ | gman_do('eval','2 * 3') | +-------------------------+ | 6 | +-------------------------+ Thursday, 12 November 2009 75
    • Using the UDF mysql> select gman_do('eval', concat('$_="',host,'";tr/a-z/b-za/; $_')) as test from mysql.user; +-------------+ | test | +-------------+ | % | | mpdbmiptu | +-------------+ Thursday, 12 November 2009 76
    • Replication scenario Master slave slave Thursday, 12 November 2009 77
    • Replication scenario slaves status Master slave 1 slave 2 slave slave Thursday, 12 November 2009 78
    • More on similar matters • CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009 • Linux.Conf.Au, Wellington, NZ, 18-19-Jan-2010 http://datacharmer.blogspot.com http://gearman.org Thursday, 12 November 2009 79
    • THANKS Let's talk! This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http:// creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Thursday, 12 November 2009 80