Your SlideShare is downloading. ×
0
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Gearman for MySQL
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Gearman for MySQL

10,707

Published on

Gearman is a client/server infrastructure for generic tasks, usable on distributed servers, with little worry about the details. No matter what language you speak, Gearman can meet your needs in C, …

Gearman is a client/server infrastructure for generic tasks, usable on distributed servers, with little worry about the details. No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more. Gearman can also work in conjunction with MySQL, either using UDFs, or simply through its basic architecture.
This talk will show examples of how to use Gearman for remote installation and how to call functions written in Perl from any other language or from inside MySQL server, with no knowledge of Perl at all.

Published in: Technology
0 Comments
17 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,707
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
343
Comments
0
Likes
17
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  1. Gearman for MySQL Giuseppe Maxia MySQL Community Team Lead Sun Microsystems This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Thursday, 12 November 2009 1
  2. Mainframe terminal Mainframe terminal operating system client terminal hardware application terminal 0 100 USER FRIENDLINESS Thursday, 12 November 2009 2
  3. Mini computers Mainframe Mini terminal terminal operating system Mini client hardware terminal application terminal 0 100 USER FRIENDLINESS Thursday, 12 November 2009 3
  4. Networks server personal computer personal operating operating computer system system personal client hardware hardware computer hardware personal application 0 computer 100 USER FRIENDLINESS Thursday, 12 November 2009 4
  5. Web applications web server browser INTERNET browser operating operating operating system system systemoperating system client browser hardware hardware hardware application browser 0 100 USER FRIENDLINESS Thursday, 12 November 2009 5
  6. service Cloud applications provider service provider service provider service browser provider web INTERNET server browser operating operating operating system system system client browser hardware hardware hardware application applicationapplication browser 0 100 USER FRIENDLINESS Thursday, 12 November 2009 6
  7. Some actors • memcached • gearman Used in production by Facebook and Digg Thursday, 12 November 2009 7
  8. GEARMAN Thursday, 12 November 2009 8
  9. != G ER M A N Thursday, 12 November 2009 9
  10. G E A R M A N? Thursday, 12 November 2009 10
  11. MANAGER Thursday, 12 November 2009 11
  12. job task request server worker client http://gearman.org Thursday, 12 November 2009 12
  13. Gearman: a technology for distributed computing Thursday, 12 November 2009 13
  14. Distributed Thursday, 12 November 2009 14
  15. Multiple operating systems Thursday, 12 November 2009 15
  16. multiple languages Thursday, 12 November 2009 16
  17. freedom of choice Thursday, 12 November 2009 17
  18. redundancy Thursday, 12 November 2009 18
  19. USING GEARMAN • Server: gearmand • Client libraries: • C/C++ • Java • Perl • PHP • Python Thursday, 12 November 2009 19
  20. Simple usage GEARMAN • Command line client and worker Thursday, 12 November 2009 20
  21. starting the server /usr/local/sbin/gearmand -d # started as daemon. # No feedback given on the command line Thursday, 12 November 2009 21
  22. starting the server (2) /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) # started as normal application # verbose output requested Thursday, 12 November 2009 22
  23. starting the worker gearman -w -h hostname -p 4730 -f conta wc # -w = act as worker # -f = function # conta = function name # wc = command to execute when function # 'conta' is called Thursday, 12 November 2009 23
  24. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected Thursday, 12 November 2009 24
  25. starting the client gearman -h hostname -p 4730 -f conta < ~/.bashrc 57 135 2149 # <- output # from worker # -f = function # conta = function name # < ~/.bashrc = input data Thursday, 12 November 2009 25
  26. what the server says /usr/local/sbin/gearmand -v -v INFO Starting up INFO Listening on :::4730 (5) INFO Listening on 0.0.0.0:4730 (6) … INFO Accepted connection from 127.0.0.1:4994 INFO [ 0] 127.0.0.1:4994 Connected … INFO Accepted connection from 127.0.0.1:5181 INFO [ 0] 127.0.0.1:5181 Connected INFO [ 0] 127.0.0.1:5181 Disconnected Thursday, 12 November 2009 26
  27. What happened 1 server start listen to port 4730 Thursday, 12 November 2009 27
  28. What happened 2 worker starts registers function 'conta' to server Thursday, 12 November 2009 28
  29. What happened 3 client starts requires function 'conta' from server provides input data Thursday, 12 November 2009 29
  30. What happened 4 server sends client request to worker passes all input data to worker Thursday, 12 November 2009 30
  31. What happened 5 worker receives request and data processes input Thursday, 12 November 2009 31
  32. What happened 6 worker returns processed data server passes it to client Thursday, 12 November 2009 32
  33. What happened 7 client receives processed data client displays result Thursday, 12 November 2009 33
  34. A simple Perl worker Thursday, 12 November 2009 34
  35. A simple perl worker 1. add server 2. add function 3. loop 4. function definition Thursday, 12 November 2009 35
  36. simple worker (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Worker; my $host = '127.0.0.1'; my $port = 4730; my $worker = new Gearman::XS::Worker; my $ret = $worker->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); exit(1); } Thursday, 12 November 2009 36
  37. simple worker (2) my $options = ''; $ret = $worker->add_function( "reverse", # public function name 0, # timeout &myreverse, # reference to function $options); # function arguments if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Thursday, 12 November 2009 37
  38. simple worker (3) while (1) { my $ret = $worker->work(); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } } Thursday, 12 November 2009 38
  39. simple worker (4) sub myreverse { my ($job) = @_; my $workload = $job->workload(); my $result = reverse($workload); printf( "Job=%s F_Name=%s Workload=%s Result=%sn", $job->handle(), $job->function_name(), $job->workload(), $result); return $result; } Thursday, 12 November 2009 39
  40. A simple Perl client Thursday, 12 November 2009 40
  41. A simple perl client • add server • run a task Thursday, 12 November 2009 41
  42. simple client (1) use strict; use warnings; use Gearman::XS qw(:constants); use Gearman::XS::Client; my $client = new Gearman::XS::Client; my $host = '127.0.0.1'; my $port = 4730; my $ret = $client->add_server($host, $port); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $client->error()); exit(1); } Thursday, 12 November 2009 42
  43. simple client (2) my $input = shift || 'teststring'; my ($return, $result) = $client->do("reverse", $input); if ($return == GEARMAN_SUCCESS) { printf("Result=%sn", $result); } Thursday, 12 November 2009 43
  44. A sample run Thursday, 12 November 2009 44
  45. host 1 perl worker.pl Thursday, 12 November 2009 45
  46. host 2 perl client.pl Result=gnirtstset Thursday, 12 November 2009 46
  47. host 1 perl worker.pl Job=H:gmac3.local:4 F_Name=reverse Workload=teststring Result=gnirtstset Thursday, 12 November 2009 47
  48. more client functions • do_background • add_task • run_tasks Thursday, 12 November 2009 48
  49. Some advanced usage • DBIx::SQLCrosstab • Data cubes • Perl only Thursday, 12 November 2009 49
  50. Thursday, 12 November 2009 50
  51. Image processing • CPU intensive • Large storage needed • Application is OS specific Thursday, 12 November 2009 51
  52. Thursday, 12 November 2009 52
  53. Remote sandboxes Using GEARMAN client server architecture N servers N workers N clients Supports multiple languages Thursday, 12 November 2009 53
  54. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman server func3 worker4 func1 func3 Gearman server client worker5 func1 host3 func3 host4 Thursday, 12 November 2009 54
  55. host1 worker1 host2 worker2 func1 worker0 func1 func2 func3 func1 func3 worker3 func1 Gearman Gearman Master server server Slave1 func3 worker4 func1 Gearman func3 server client worker5 func1 host3 Slave2 func3 host4 Thursday, 12 November 2009 55
  56. Remote sandbox worker •add server •add function •loop •function definition Thursday, 12 November 2009 56
  57. remote sandbox worker (2) $ret = $worker->add_function( "make_sandbox", 0, &make_sandbox, ''); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } Thursday, 12 November 2009 57
  58. remote sandbox worker (4) sub make_sandbox { my ($job, $options) = @_; my $command = $job->workload(); chomp $command; print STDERR "<$command>n"; my $result = qx(make_sandbox $command) ; if ($?) { print $result, "n"; return "Error ($!)n"; } else { return "okn" } } Thursday, 12 November 2009 58
  59. Client to install Remote sandboxes •add server •run remote installation on master •run remote installation on each slave •run configuration queries on slaves Thursday, 12 November 2009 59
  60. sample call to remote worker my $ret = qx(echo "$MASTER_SANDBOX" | gearman -h $master_ip -f make_sandbox ) ; # using the gearman command line utility Thursday, 12 November 2009 60
  61. Gearman for the MySQL server Thursday, 12 November 2009 61
  62. Gearman UDF Thursday, 12 November 2009 62
  63. More power to MySQL • Perl/PHP/Python functions • Shell access (you can send and receive email!) • filesystem access • advanced monitoring through Gearman features Thursday, 12 November 2009 63
  64. WARNING! You can easily shoot yourself in the foot Thursday, 12 November 2009 64
  65. Using MySQL UDF •Install Gearman •Get MySQL server source (needed to compile the UDF) •Get the UDF source •https://launchpad.net/gearman-mysql-udf •Follow the instructions (good luck) :-) Thursday, 12 November 2009 65
  66. Installation (1) export DEPS_LIBS="-L/usr/local/lib -lgearman" export DEPS_CFLAGS="-I/usr/local/include/ libgearman-server/ -I/usr/local/include/libgearman" ./configure --with-mysql=$MYSQL_BIN/bin/mysql_config --libdir=$MYSQL_BIN/lib/mysql/plugin/ make && make install Thursday, 12 November 2009 66
  67. Installation (2) mysql> CREATE FUNCTION gman_do RETURNS STRING SONAME "libgearman_mysql_udf.so"; mysql> CREATE FUNCTION gman_servers_set RETURNS STRING SONAME "libgearman_mysql_udf.so"; # there are more functions, but these two # are the MUST-have ones Thursday, 12 November 2009 67
  68. Installation (3) # register at least one server mysql> select gman_servers_set ('192.168.1.45:4730'); # optionally, a server can be associated # with a function mysql> select gman_servers_set ('192.168.1.45:4730', 'shell'); Thursday, 12 November 2009 68
  69. Create a worker (1) my @functions = ( ['reverse', &myreverse], ['count', &mycount], ['shell', &myshell], ['eval', &myeval], ['store', &mystore], ); # see the full code here: #http://forge.mysql.com/tools/tool.php?id=235 Thursday, 12 November 2009 69
  70. Create a worker (2) for my $func (@functions) { $ret = $worker->add_function( $func->[0], 0, $func->[1], $options); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "error with function %s - %sn", $func->[0], $worker->error()); } } Thursday, 12 November 2009 70
  71. Create a worker (3) sub myshell { my $job = shift; my $workload = $job->workload(); my $result = qx($workload); return $result; } # WARNING! # You can shoot yourself in the foot! Thursday, 12 November 2009 71
  72. Create a worker (4) sub myeval { my $job = shift; my $workload = $job->workload(); my $result = eval $workload; return $result; } # WARNING! # You can machine gun yourself # in the foot! Thursday, 12 November 2009 72
  73. Create a worker (4) sub myeval { my $job = shift; my $workload = $job->workload(); my $result = eval $workload; return $result; } # WARNING! # You can machine gun yourself # in the foot! Thursday, 12 November 2009 72
  74. Using the UDF mysql> select gman_do('reverse','abcd') as test; +------+ | test | +------+ | dcba | +------+ Thursday, 12 November 2009 73
  75. Using the UDF mysql> SELECT gman_do('shell', concat(' ls -lh ', (select variable_value from information_schema.global_variables where variable_name = "datadir" )))G total 40976 -rw-rw---- 1 gmax staff 5.0M Nov 11 13:34 ib_logfile0 -rw-rw---- 1 gmax staff 5.0M Nov 11 13:34 ib_logfile1 -rw-rw---- 1 gmax staff 10M Nov 11 13:34 ibdata1 -rw-rw---- 1 gmax staff 1.2K Nov 11 13:34 msandbox.err drwx------ 2 gmax staff 2.4K Nov 11 13:34 mysql -rw-rw---- 1 gmax staff 6B Nov 11 13:34 mysql_sandbox5140.pid drwx------ 2 gmax staff 68B Nov 11 13:34 test Thursday, 12 November 2009 74
  76. Using the UDF mysql> select gman_do('eval','2 * 3') ; +-------------------------+ | gman_do('eval','2 * 3') | +-------------------------+ | 6 | +-------------------------+ Thursday, 12 November 2009 75
  77. Using the UDF mysql> select gman_do('eval', concat('$_="',host,'";tr/a-z/b-za/; $_')) as test from mysql.user; +-------------+ | test | +-------------+ | % | | mpdbmiptu | +-------------+ Thursday, 12 November 2009 76
  78. Replication scenario Master slave slave Thursday, 12 November 2009 77
  79. Replication scenario slaves status Master slave 1 slave 2 slave slave Thursday, 12 November 2009 78
  80. More on similar matters • CodeBits, Lisbon, Portugal, 3-4-5-Dec-2009 • Linux.Conf.Au, Wellington, NZ, 18-19-Jan-2010 http://datacharmer.blogspot.com http://gearman.org Thursday, 12 November 2009 79
  81. THANKS Let's talk! This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. To view a copy of this license, visit http:// creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Thursday, 12 November 2009 80

×