Gearman for MySQL

Gearman for MySQL Giuseppe Maxia MySQL Community Team Lead Sun Microsystems This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Thursday, 12 November 2009 1

Mainframe terminal Mainframe terminal operating system client terminal hardware application terminal 0 100 USER FRIENDLINESS Thursday, 12 November 2009 2

Mini computers Mainframe Mini terminal terminal operating system Mini client hardware terminal application terminal 0 100 USER FRIENDLINESS Thursday, 12 November 2009 3

Networks server personal computer personal operating operating computer system system personal client hardware hardware computer hardware personal application 0 computer 100 USER FRIENDLINESS Thursday, 12 November 2009 4

Web applications web server browser INTERNET browser operating operating operating system system systemoperating system client browser hardware hardware hardware application browser 0 100 USER FRIENDLINESS Thursday, 12 November 2009 5

service Cloud applications provider service provider service provider service browser provider web INTERNET server browser operating operating operating system system system client browser hardware hardware hardware application applicationapplication browser 0 100 USER FRIENDLINESS Thursday, 12 November 2009 6

Some actors • memcached • gearman Used in production by Facebook and Digg Thursday, 12 November 2009 7

GEARMAN Thursday, 12 November 2009 8

!= G ER M A N Thursday, 12 November 2009 9

G E A R M A N? Thursday, 12 November 2009 10

MANAGER Thursday, 12 November 2009 11

job task request server worker client http://gearman.org Thursday, 12 November 2009 12

Gearman: a technology for distributed computing Thursday, 12 November 2009 13

Distributed Thursday, 12 November 2009 14

Multiple operating systems Thursday, 12 November 2009 15

multiple languages Thursday, 12 November 2009 16

freedom of choice Thursday, 12 November 2009 17

redundancy Thursday, 12 November 2009 18

USING GEARMAN • Server: gearmand • Client libraries: • C/C++ • Java • Perl • PHP • Python Thursday, 12 November 2009 19

Simple usage GEARMAN • Command line client and worker Thursday, 12 November 2009 20

starting the server /usr/local/sbin/gearmand -d # started as daemon. # No feedback given on the command line Thursday, 12 November 2009 21

starting the server (2) /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) # started as normal application # verbose output requested Thursday, 12 November 2009 22

starting the worker gearman -w -h hostname -p 4730 -f conta wc # -w = act as worker # -f = function # conta = function name # wc = command to execute when function # 'conta' is called Thursday, 12 November 2009 23

what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected Thursday, 12 November 2009 24

starting the client gearman -h hostname -p 4730 -f conta < ~/.bashrc 57 135 2149 # <- output # from worker # -f = function # conta = function name # < ~/.bashrc = input data Thursday, 12 November 2009 25

what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected … INFO Accepted connection from 127.0.0.1:5181 INFO [ 0] 127.0.0.1:5181 Connected INFO [ 0] 127.0.0.1:5181 Disconnected Thursday, 12 November 2009 26

What happened 1 server start listen to port 4730 Thursday, 12 November 2009 27

What happened 2 worker starts registers function 'conta' to server Thursday, 12 November 2009 28

What happened 3 client starts requires function 'conta' from server provides input data Thursday, 12 November 2009 29

What happened 4 server sends client request to worker passes all input data to worker Thursday, 12 November 2009 30

What happened 5 worker receives request and data processes input Thursday, 12 November 2009 31

What happened 6 worker returns processed data server passes it to client Thursday, 12 November 2009 32

What happened 7 client receives processed data client displays result Thursday, 12 November 2009 33

A simple Perl worker Thursday, 12 November 2009 34

A simple perl worker 1. add server 2. add function 3. loop 4. function deﬁnition Thursday, 12 November 2009 35

simple worker (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Worker; my $host = '127.0.0.1'; my $port = 4730; my $worker = new Gearman::XS::Worker; my $ret = $worker->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); exit(1); } Thursday, 12 November 2009 36

simple worker (2) my $options = ''; $ret = $worker->add_function( "reverse", # public function name 0, # timeout &myreverse, # reference to function $options); # function arguments if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Thursday, 12 November 2009 37

simple worker (3) while (1) { my $ret = $worker->work(); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } } Thursday, 12 November 2009 38

simple worker (4) sub myreverse { my ($job) = @_; my $workload = $job->workload(); my $result = reverse($workload); printf( "Job=%s F_Name=%s Workload=%s Result=%sn", $job->handle(), $job->function_name(), $job->workload(), $result); return $result; } Thursday, 12 November 2009 39

A simple Perl client Thursday, 12 November 2009 40

A simple perl client • add server • run a task Thursday, 12 November 2009 41

simple client (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Client; my $client = new Gearman::XS::Client; my $host = '127.0.0.1'; my $port = 4730; my $ret = $client->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $client->error()); exit(1); } Thursday, 12 November 2009 42

simple client (2) my $input = shift || 'teststring'; my ($return, $result) = $client->do("reverse", $input); if ($return == GEARMAN_SUCCESS) { printf("Result=%sn", $result); } Thursday, 12 November 2009 43

A sample run Thursday, 12 November 2009 44

host 1 perl worker.pl Thursday, 12 November 2009 45

host 2 perl client.pl Result=gnirtstset Thursday, 12 November 2009 46

host 1 perl worker.pl Job=H:gmac3.local:4 F_Name=reverse Workload=teststring Result=gnirtstset Thursday, 12 November 2009 47

more client functions • do_background • add_task • run_tasks Thursday, 12 November 2009 48

Some advanced usage • DBIx::SQLCrosstab • Data cubes • Perl only Thursday, 12 November 2009 49

Thursday, 12 November 2009 50

Image processing • CPU intensive • Large storage needed • Application is OS speciﬁc Thursday, 12 November 2009 51

Thursday, 12 November 2009 52

Remote sandboxes Using GEARMAN client server architecture N servers N workers N clients Supports multiple languages Thursday, 12 November 2009 53

host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman server func3 worker4 func1 func3 Gearman server client worker5 func1 host3 func3 host4 Thursday, 12 November 2009 54

host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman Gearman Master server server Slave1 func3 worker4 func1 Gearman func3 server client worker5 func1 host3 Slave2 func3 host4 Thursday, 12 November 2009 55

Remote sandbox worker •add server •add function •loop •function definition Thursday, 12 November 2009 56

remote sandbox worker (2) $ret = $worker->add_function( "make_sandbox", 0, &make_sandbox, ''); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Thursday, 12 November 2009 57

remote sandbox worker (4) sub make_sandbox { my ($job, $options) = @_; my $command = $job->workload(); chomp $command; print STDERR "<$command>n"; my $result = qx(make_sandbox $command) ; if ($?) { print $result, "n"; return "Error ($!)n"; } else { return "okn" } } Thursday, 12 November 2009 58

Client to install Remote sandboxes •add server •run remote installation on master •run remote installation on each slave •run configuration queries on slaves Thursday, 12 November 2009 59

sample call to remote worker my $ret = qx(echo "$MASTER_SANDBOX" | gearman -h $master_ip -f make_sandbox ) ; # using the gearman command line utility Thursday, 12 November 2009 60

Gearman for the MySQL server Thursday, 12 November 2009 61

Gearman UDF Thursday, 12 November 2009 62

More power to MySQL • Perl/PHP/Python functions • Shell access (you can send and receive email!) • ﬁlesystem access • advanced monitoring through Gearman features Thursday, 12 November 2009 63

WARNING! You can easily shoot yourself in the foot Thursday, 12 November 2009 64

Using MySQL UDF •Install Gearman •Get MySQL server source (needed to compile the UDF) •Get the UDF source •https://launchpad.net/gearman-mysql-udf •Follow the instructions (good luck) :-) Thursday, 12 November 2009 65

Installation (1) export DEPS_LIBS="-L/usr/local/lib -lgearman" export DEPS_CFLAGS="-I/usr/local/include/ libgearman-server/ -I/usr/local/include/libgearman" ./configure --with-mysql=$MYSQL_BIN/bin/mysql_config --libdir=$MYSQL_BIN/lib/mysql/plugin/ make && make install Thursday, 12 November 2009 66

Installation (2) mysql> CREATE FUNCTION gman_do RETURNS STRING SONAME "libgearman_mysql_udf.so"; mysql> CREATE FUNCTION gman_servers_set RETURNS STRING SONAME "libgearman_mysql_udf.so"; # there are more functions, but these two # are the MUST-have ones Thursday, 12 November 2009 67

Installation (3) # register at least one server mysql> select gman_servers_set ('192.168.1.45:4730'); # optionally, a server can be associated # with a function mysql> select gman_servers_set ('192.168.1.45:4730', 'shell'); Thursday, 12 November 2009 68

Create a worker (1) my @functions = ( ['reverse', &myreverse], ['count', &mycount], ['shell', &myshell], ['eval', &myeval], ['store', &mystore], ); # see the full code here: #http://forge.mysql.com/tools/tool.php?id=235 Thursday, 12 November 2009 69

Create a worker (2) for my $func (@functions) { $ret = $worker->add_function( $func->[0], 0, $func->[1], $options); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "error with function %s - %sn", $func->[0], $worker->error()); } } Thursday, 12 November 2009 70

Create a worker (3) sub myshell { my $job = shift; my $workload = $job->workload(); my $result = qx($workload); return $result; } # WARNING! # You can shoot yourself in the foot! Thursday, 12 November 2009 71

Create a worker (4) sub myeval { my $job = shift; my $workload = $job->workload(); my $result = eval $workload; return $result; } # WARNING! # You can machine gun yourself # in the foot! Thursday, 12 November 2009 72

Using the UDF mysql> select gman_do('reverse','abcd') as test; +------+ | test | +------+ | dcba | +------+ Thursday, 12 November 2009 73

Using the UDF mysql> SELECT gman_do('shell', concat(' ls -lh ', (select variable_value from information_schema.global_variables where variable_name = "datadir" )))G total 40976 -rw-rw---- 1 gmax staff 5.0M Nov 11 13:34 ib_logfile0 -rw-rw---- 1 gmax staff 5.0M Nov 11 13:34 ib_logfile1 -rw-rw---- 1 gmax staff 10M Nov 11 13:34 ibdata1 -rw-rw---- 1 gmax staff 1.2K Nov 11 13:34 msandbox.err drwx------ 2 gmax staff 2.4K Nov 11 13:34 mysql -rw-rw---- 1 gmax staff 6B Nov 11 13:34 mysql_sandbox5140.pid drwx------ 2 gmax staff 68B Nov 11 13:34 test Thursday, 12 November 2009 74

Using the UDF mysql> select gman_do('eval','2 * 3') ; +-------------------------+ | gman_do('eval','2 * 3') | +-------------------------+ | 6 | +-------------------------+ Thursday, 12 November 2009 75

Replication scenario Master slave slave Thursday, 12 November 2009 77

Replication scenario slaves status Master slave 1 slave 2 slave slave Thursday, 12 November 2009 78

More on similar matters • CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009 • Linux.Conf.Au, Wellington, NZ, 18-19-Jan-2010 http://datacharmer.blogspot.com http://gearman.org Thursday, 12 November 2009 79

THANKS Let's talk! This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http:// creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Thursday, 12 November 2009 80

http://www.ningoo.net	1319
http://blogread.cn	173
http://soft.zdnet.com.cn	113
http://www.symfonylab.com	108
http://blog.bypat.com	103
http://wiki.simplexi.com	43
http://www.slideshare.net	28
http://codebits.eu	15
http://nwlinux.com	11
http://cache.baidu.com	10
https://codebits.eu	7
http://www.aicuxiao.com	4
http://www.builder.com.cn	4
http://xianguo.com	4
http://blog.zdnet.com.cn	3
http://translate.googleusercontent.com	3
http://203.231.63.114	3
http://www.zhuaxia.com	3
http://webcache.googleusercontent.com	2
http://www.blogread.cn	2
http://reader.youdao.com	2
http://ningoo.net	1
http://203.208.37.132	1
http://203.208.39.132	1
http://buckycomputing.net	1
http://rbucky.com	1
http://wiki.simplexi.com:8080	1
http://cache.baiducontent.com	1

Gearman for MySQL

by Giuseppe Maxia on Nov 12, 2009

Accessibility

Categories

Upload Details

Usage Rights

28 Embeds 1,967

Statistics

Gearman for MySQL Presentation Transcript