Sharding for the masses
                   Introducing the Spider
                 storage engine, and more



           ...
What is sharding?



                       "shard" is a piece of broken ceramic or glass

                       "Shardin...
WHY SHARDING?




Thursday, 15 October 2009
WHY SHARDING?
                       SCALING

                       SCALING

                        SCALING
            ...
Scaling: the problem


                       You start with one server

                            Too much data

      ...
Scaling: the solution


                       The MySQL way

                       Also known as the Yahoo and Google wa...
Replication: how it works
                                            Master
                            WRITE

          ...
Replication: how it scales
                                                   Master
                            WRITE

  ...
Replication: how it chokes
                                                    Master
                                    ...
Homemade sharding
       client
                                           Master1
                                       ...
How homemade sharding
                        breaks
       client
                                          Master1
     ...
How homemade sharding
                        breaks
       client
                                          Master1
     ...
How homemade sharding
                        breaks
       client



        WRITE
        READ
                         ...
How homemade sharding
                       breaks (2)
       client
                                          Master1
  ...
How homemade sharding
                       breaks (2)
       client
                                                Mast...
The quest for magic
                                 sharding




Thursday, 15 October 2009
The quest for magic
                                 sharding

                       MySQL Proxy

                       ...
The quest for magic
                                 sharding

                       MySQL Proxy

                       ...
Horizontal
                  partitioning
Thursday, 15 October 2009
Enter Spider




Thursday, 15 October 2009
Introducing Spider
                       A MySQL storage engine

                       Developed by Kentoku Shiba

     ...
Note about partitions
                       A feature introduced in MySQL 5.1

                       Horizontal partitio...
host2   MySQL server
                                             without
                                             SPI...
host2   MySQL server
                                                         without
                                    ...
select * from
                              employees where
                            date = '1998-01-01'   host3   MySQ...
host2   MySQL server
                                                            without
                               se...
host2   MySQL server
                                             without
                                             SPI...
select * from        host2   MySQL server
                                                    without
                empl...
host2   MySQL server
                                             without
                                             SPI...
host8              MySQL server with SPIDER   host2   MySQL server
                                                       ...
data               data   data   data   data




Thursday, 15 October 2009
data               data   data   data   data




Thursday, 15 October 2009
data               data   data   data   data




Thursday, 15 October 2009
data               data   data   data   data




Thursday, 15 October 2009
data               data   data   data   data




Thursday, 15 October 2009
data               data   data     data   data




                                   client

Thursday, 15 October 2009
data               data   data        data   data




                                   client
                          ...
data           data   data   data   data
                HA          HA     HA     HA     HA

             data           ...
INSTALLATION




Thursday, 15 October 2009
INSTALLATION (1)

                       Get the source code for MySQL 5.1.39

                            http://dev.mysq...
INSTALLATION (2)
                       Unpack MySQL source code

                       Unpack Spider source code and doc...
/    l
            .'     :                     __.....__..._ ____
           / /                    _.-"              "-....
MASTER

                                                     remote
                  SELECT * FROM sometable            s...
MASTER

                                                     remote
                  SELECT * FROM sometable            s...
INSTALLATION (3)
                       Move the spider directory into MySQL source
                       code




      ...
INSTALLATION (4)
                       Apply the Spider patches to MySQL code




       cd mysql-5.1.39
       patch -p2...
INSTALLATION (5)
                       Compile MySQL code (see the docs for details)
       autoconf
       automake
    ...
INSTALLATION (6)
                       create a binary tarball




       ./scripts/make_binary_distribution




Thursday...
INSTALLATION (7)
                       Install manually in your main server

                       OR

                 ...
SETUP




Thursday, 15 October 2009
SETUP (1)
              Get the SQL from the docs

              or get it from my site

                      http://data...
SETUP (2)
                       Check the engines


       ./use
       select engine,support,transactions,xa
           ...
USING SPIDER
                             (Simple case)



Thursday, 15 October 2009
Preparing remote servers
                             (1)
              Create three servers using MySQL Sandbox




     ...
Preparing remote servers
                             (2)
              Check the port numbers


       ~/sandboxes/spider...
Preparing remote servers
                             (3)
              create table definition (tablea.sql)

       drop ...
Preparing remote servers
                             (4)
              create table in remote servers

       cd $HOME/sa...
setting the main server (1)
              create table definition (tablea_main.sql)

       drop schema if exists   myspid...
setting the main server (2)
              create table definition (tablea_main.sql) (continues)

Connection ' table "tbl_a...
setting the main server (3)
              create table

./use < tablea_main.sql




Thursday, 15 October 2009
(Finally) using it (1)
              in the main server

./use myspider
insert into tbl_b values (500,1), 
(1500,2), (5000...
(Finally) using it (2)
              in the main server

  select * from tbl_b;
+-------+-------+
| col_a | col_b |
+-----...
WHERE IS THE DATA?


                       500 < 1000 = host 1

                       1500 < 2000 = host2

             ...
Looking for the data
              in the "remote" servers
 $HOME/sandboxes/spider_dir/use_all 
  "select * from myspider....
Using Spider
                            (more complex case)



Thursday, 15 October 2009
Setting more remote
                                 servers (1)
              in the main server
./use myspider
drop tabl...
Setting more remote
                                 servers (2)
              create 20 remote servers
make_multiple_sand...
Setting more remote
                                 servers (3)
              create tables for the employees database
cd...
the test employees database
          http://launchpad.net/test-db




Thursday, 15 October 2009
Setting the main server (1)
              create tables for the employees database
cd $HOME/sandboxes/spider_main
wget htt...
checking the remote
                                       servers
              see how many rows have you got after load...
Running remote commands
                   with Spider



Thursday, 15 October 2009
data                     data          data   data     data



              UDF
                 CREATE TABLE
           ...
data                     data          data   data     data



              UDF
               INSERT INTO db.t1
        ...
data                    data         data          data     data




                            CREATE TEMPORARY TABLE
  ...
data               data     data      data     data



              UDF
               SELECT * FROM db.t1
              ...
creating a remote table

           select spider_direct_sql(
           "create table myspider.t1 (id int)",
           "...
inserting records into
                                remote table
           select spider_direct_sql(
           "inser...
getting remote records (1)

      create temporary table remote6001 (i int);

      select spider_direct_sql(
      "selec...
getting remote records (2)

      select * from remote6001;
      +------+
      | i    |
      +------+
      | 1000 |
  ...
LIVE DEMO
 Where everything can go wrong!


Thursday, 15 October 2009
Vertical
                  partitioning
Thursday, 15 October 2009
The vertical partition engine


                       Same author of the Spider engine

                       Open sourc...
Original table

                                          employees

                       id       name    salary dept  ...
Split tables
                             empl2                               empl1
             id             name      ...
Split tables
                             empl2                               empl1
             id             name      ...
Vertical partitioning
                                 installation
                       Similar to Spider:

           ...
vertical partitioning syntax
CREATE TABLE empl1 (
   id int not null,
   name varchar(50),
   salary decimal(10,3),
   pri...
vertical partitioning syntax
CREATE TABLE employees (
  id int not null,
  name varchar(50),
  salary decimal(10,3),
  dep...
inserting into the vertical
                        partitioning engine
insert into employees values (1, 'Joe',
1300,1,'jo...
retrieving data from the
                vertical partitioning engine
select * from employees;
+----+------+----------+---...
READING MORE
                       my blog (search for Spider)

                            http://datacharmer.blogspot.c...
THANKS




                             Q&A
Thursday, 15 October 2009
Sharding for the masses
Sharding for the masses
Sharding for the masses
Sharding for the masses
Upcoming SlideShare
Loading in...5
×

Sharding for the masses

6,298

Published on

An introduction to Spider, a MySQL storage engine for transparent sharding, and the vertical partitioning engine

Published in: Technology, Business
2 Comments
19 Likes
Statistics
Notes
No Downloads
Views
Total Views
6,298
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
414
Comments
2
Likes
19
Embeds 0
No embeds

No notes for slide

Sharding for the masses

  1. 1. Sharding for the masses Introducing the Spider storage engine, and more Giuseppe Maxia MySQL Community Team Lead datacharmer@sun.com Thursday, 15 October 2009
  2. 2. What is sharding? "shard" is a piece of broken ceramic or glass "Sharding" means breaking a database to pieces Thursday, 15 October 2009
  3. 3. WHY SHARDING? Thursday, 15 October 2009
  4. 4. WHY SHARDING? SCALING SCALING SCALING SCALING SCALING Thursday, 15 October 2009
  5. 5. Scaling: the problem You start with one server Too much data Too much traffic Now what? Thursday, 15 October 2009
  6. 6. Scaling: the solution The MySQL way Also known as the Yahoo and Google way Replication Thursday, 15 October 2009
  7. 7. Replication: how it works Master WRITE client slave slave READ load balancer slave Thursday, 15 October 2009
  8. 8. Replication: how it scales Master WRITE client slave slave client slave slave slave slave slave slave client client client client READ load balancer slave slave slave slave Thursday, 15 October 2009
  9. 9. Replication: how it chokes Master > data WRITE client slave slave client slave slave slave slave slave slave client client client client READ load balancer slave slave slave slave Thursday, 15 October 2009
  10. 10. Homemade sharding client Master1 rule1 WRITE READ Master2 rule2 application logic MasterN MasterN MasterN MasterN ruleN ruleN ruleN ruleN Thursday, 15 October 2009
  11. 11. How homemade sharding breaks client Master1 rule1 WRITE READ Master2 rule2 application logic MasterN MasterN MasterN MasterN ruleN ruleN ruleN ruleN Thursday, 15 October 2009
  12. 12. How homemade sharding breaks client Master1 rule1 WRITE READ Master2 rule2 application logic MasterN MasterN MasterN MasterN ruleN ruleN ruleN ruleN Thursday, 15 October 2009
  13. 13. How homemade sharding breaks client WRITE READ application logic Master1 Master2 MasterN MasterN rule1 MasterN rule2 MasterN ruleN ruleN ruleN ruleN Thursday, 15 October 2009
  14. 14. How homemade sharding breaks (2) client Master1 rule1 rule1 rule1 rule1 rule1 WRITE READ Master2 rule2 rule2 application rule2 rule2 rule2 logic MasterN MasterN MasterN MasterN ruleN ruleN ruleN ruleN ruleNruleN ruleN ruleN Thursday, 15 October 2009
  15. 15. How homemade sharding breaks (2) client Master1 rule1 rule1 rule1 rule1 rule1 WRITE READ Master2 rule2 rule2 rule2 rule2 rule2 application logic MasterN MasterN MasterN MasterN ruleN ruleN ruleN ruleN ruleNruleN ruleN ruleN Thursday, 15 October 2009
  16. 16. The quest for magic sharding Thursday, 15 October 2009
  17. 17. The quest for magic sharding MySQL Proxy HSCALE SpockProxy Thursday, 15 October 2009
  18. 18. The quest for magic sharding MySQL Proxy HSCALE SpockProxy DON'T SCALE (SPoF) Thursday, 15 October 2009
  19. 19. Horizontal partitioning Thursday, 15 October 2009
  20. 20. Enter Spider Thursday, 15 October 2009
  21. 21. Introducing Spider A MySQL storage engine Developed by Kentoku Shiba Built on top of the partitions engine Associates a partition with a remote server Transparent to user Easy to expand Independent from application Thursday, 15 October 2009
  22. 22. Note about partitions A feature introduced in MySQL 5.1 Horizontal partitioning Transparent to users Increases insertion and selection performance presentations http://tinyurl.com/mysql-partition-tut http://tinyurl.com/mysql-partition-perf Thursday, 15 October 2009
  23. 23. host2 MySQL server without SPIDER Spider conceptual model host3 MySQL server without SPIDER table employees host4 MySQL server partition 1 year 1997 host2 without SPIDER partition 2 year 1998 host3 partition 3 year 1999 host4 host5 MySQL server without partition 4 year 2000 host5 SPIDER partition 5 year 2001 host6 partition 6 year 2002 host7 host6 MySQL server without SPIDER host1 MySQL server with SPIDER host7 MySQL server without SPIDER Thursday, 15 October 2009
  24. 24. host2 MySQL server without SPIDER Spider conceptual model host3 MySQL server without SPIDER table employees host4 MySQL server partition 1 year 1997 host2 without SPIDER partition 2 year 1998 host3 partition 3 year 1999 host4 host5 MySQL server without partition 4 year 2000 host5 SPIDER partition 5 year 2001 host6 partition 6 year 2002 host7 host6 MySQL server without SPIDER host1 MySQL server with SPIDER host7 MySQL server no without SPIDER data here Thursday, 15 October 2009
  25. 25. select * from employees where date = '1998-01-01' host3 MySQL server without SPIDER table employees host4 MySQL server partition 1 year 1997 host2 without SPIDER partition 2 year 1998 host3 partition 3 year 1999 host4 host5 MySQL server without partition 4 year 2000 host5 SPIDER partition 5 year 2001 host6 partition 6 year 2002 host7 host6 MySQL server without SPIDER host1 MySQL server with SPIDER host7 MySQL server without SPIDER Thursday, 15 October 2009
  26. 26. host2 MySQL server without select * from SPIDER employees where date = '1998-01-01' host3 MySQL server without SPIDER table employees host4 MySQL server partition 1 year 1997 host2 without SPIDER partition 2 year 1998 host3 partition 3 year 1999 host4 host5 MySQL server without partition 4 year 2000 host5 SPIDER partition 5 year 2001 host6 partition 6 year 2002 host7 host6 MySQL server without SPIDER host1 MySQL server with SPIDER host7 MySQL server without SPIDER Thursday, 15 October 2009
  27. 27. host2 MySQL server without SPIDER host3 MySQL server without SPIDER table employees host4 MySQL server partition 1 year 1997 host2 without SPIDER partition 2 year 1998 host3 partition 3 year 1999 host4 host5 MySQL server without partition 4 year 2000 host5 SPIDER partition 5 year 2001 host6 partition 6 year 2002 host7 host6 MySQL server without SPIDER host1 MySQL server with SPIDER host7 MySQL server without SPIDER Thursday, 15 October 2009
  28. 28. select * from host2 MySQL server without employees where date = SPIDER '1998-01-01' limit 0, 9223372036854775807 host3 MySQL server without SPIDER table employees host4 MySQL server partition 1 year 1997 host2 without SPIDER partition 2 year 1998 host3 partition 3 year 1999 host4 host5 MySQL server without partition 4 year 2000 host5 SPIDER partition 5 year 2001 host6 partition 6 year 2002 host7 host6 MySQL server without SPIDER host1 MySQL server with SPIDER host7 MySQL server without SPIDER Thursday, 15 October 2009
  29. 29. host2 MySQL server without SPIDER host3 MySQL server without SPIDER table employees host4 MySQL server partition 1 year 1997 host2 without SPIDER partition 2 year 1998 host3 partition 3 year 1999 host4 host5 MySQL server without partition 4 year 2000 host5 SPIDER partition 5 year 2001 host6 partition 6 year 2002 host7 host6 MySQL server without SPIDER host1 MySQL server with SPIDER host7 MySQL server without SPIDER Thursday, 15 October 2009
  30. 30. host8 MySQL server with SPIDER host2 MySQL server without SPIDER table employees partition 1 year 1997 host2 host3 MySQL server partition 2 year 1998 host3 without SPIDER partition 3 year 1999 host4 partition 4 year 2000 host5 host4 MySQL server partition 5 year 2001 host6 without SPIDER partition 6 year 2002 host7 host5 MySQL server host1 MySQL server with SPIDER without SPIDER table employees host6 MySQL server partition 1 year 1997 host2 without SPIDER partition 2 year 1998 host3 partition 3 year 1999 host4 host7 MySQL server partition 4 year 2000 host5 without SPIDER partition 5 year 2001 host6 partition 6 year 2002 host7 Thursday, 15 October 2009
  31. 31. data data data data data Thursday, 15 October 2009
  32. 32. data data data data data Thursday, 15 October 2009
  33. 33. data data data data data Thursday, 15 October 2009
  34. 34. data data data data data Thursday, 15 October 2009
  35. 35. data data data data data Thursday, 15 October 2009
  36. 36. data data data data data client Thursday, 15 October 2009
  37. 37. data data data data data client client client client Thursday, 15 October 2009
  38. 38. data data data data data HA HA HA HA HA data data data data data Thursday, 15 October 2009
  39. 39. INSTALLATION Thursday, 15 October 2009
  40. 40. INSTALLATION (1) Get the source code for MySQL 5.1.39 http://dev.mysql.com/downloads Get the source code for Spider 2.5 http://launchpad.net/spiderformysql Get the patch for condition pushdown https://launchpad.net/ partitionconditionpushdownformysql Thursday, 15 October 2009
  41. 41. INSTALLATION (2) Unpack MySQL source code Unpack Spider source code and docs Unpack the condition pushdown patch mkdir spider cd spider tar -xzf mysql-5.1.39.tar.gz tar -xzf spider-src-1.0-for-5.1.39.tgz tar -xzf spider-doc-1.0-for-5.1.39.tgz tar -xzf partition_cond_push-0.1- for-5.1.36.tgz Thursday, 15 October 2009
  42. 42. / l .' : __.....__..._ ____ / / _.-" "-. "" "-. (`-: .---: .--.' _....J. "-. """y ,.' __..--"" `+""--. `. : .'/ .-"""-. _. `. "-. `._.._ ; _.'.' .-j `. "-. "-._`. : / .-" : `-. `- "-. ; /.' ; :; ." `, :_:/ :: ;: ( / .-" .') ; STOP! ;-" ; "-. / ; .^. .' .' / .-" / .- : `. '. : .- / __.-j.'.' .-" /.---' / / `,. .' "":' /-" .' __.' : : ,"" ; .' .' .-"" _J ; ; `. /.' _/ .-" / "-: /"--.b-..-' .' ; / / ""-..' .--'.-'/ , : :`. : / : `-i" ,',_: _ : '._ :__; .'.-"; ; ; j `.l "-._ `" :_/ :_/ `.; "-._ What the hell is a :_"-._ "-. `. l "-. ) `. ""^--""^-. : "; : `._ "condition ; / `._ ""---. / / _ `.--.__.' : : / ; :". ; ; : : ; `. `. / ; : ; : `. `. / /: ; : ; "-' pushdown"? :_.' ; ; ; : / / :_l `-' Thursday, 15 October 2009
  43. 43. MASTER remote SELECT * FROM sometable server WHERE col1 = 2 management SELECT col1,col2,col3 FROM sometable LIMIT 0,9999999 Without condition REMOTE SERVER pushdown Thursday, 15 October 2009
  44. 44. MASTER remote SELECT * FROM sometable server WHERE col1 = 2 management SELECT col1,col2,col3 FROM sometable WHERE col1 = 2 With LIMIT 0,9999999 condition pushdown REMOTE SERVER Thursday, 15 October 2009
  45. 45. INSTALLATION (3) Move the spider directory into MySQL source code mv spider mysql-5.1.39/storage Thursday, 15 October 2009
  46. 46. INSTALLATION (4) Apply the Spider patches to MySQL code cd mysql-5.1.39 patch -p2 < ../mysql-5.1.39.spider.diff patch -p2 < ../mysql-5.1.36.partition_cond_push.diff Thursday, 15 October 2009
  47. 47. INSTALLATION (5) Compile MySQL code (see the docs for details) autoconf automake ./configure --enable-thread-safe-client --enable-local-infile --with-pic --with-fast-mutexes --with-client-ldflags=-static --with-mysqld-ldflags=-static --with-zlib-dir=bundled --with-big-tables --with-ssl --with-readline --with-embedded-server --with-partition --with-innodb --without-ndbcluster --without-archive-storage-engine --without-blackhole-storage-engine --with-csv-storage-engine --without-example-storage-engine --without-federated-storage-engine --with-extra-charsets=complex && make Thursday, 15 October 2009
  48. 48. INSTALLATION (6) create a binary tarball ./scripts/make_binary_distribution Thursday, 15 October 2009
  49. 49. INSTALLATION (7) Install manually in your main server OR use MySQL Sandbox make_sandbox $PWD/mysql-5.1.39-osx10.5-i386.tar.gz --sandbox_directory=spider_main Thursday, 15 October 2009
  50. 50. SETUP Thursday, 15 October 2009
  51. 51. SETUP (1) Get the SQL from the docs or get it from my site http://datacharmer.org/downloads/spider_setup.sql Run it cd $HOME/sandboxes/spider_main wget http://datacharmer.org/downloads/ spider_setup.sql ./use < spider_setup.sql Thursday, 15 October 2009
  52. 52. SETUP (2) Check the engines ./use select engine,support,transactions,xa -> from information_schema.engines; +------------+---------+--------------+------+ | engine | support | transactions | xa | +------------+---------+--------------+------+ | SPIDER | YES | YES | YES | | MRG_MYISAM | YES | NO | NO | | CSV | YES | NO | NO | | MyISAM | DEFAULT | NO | NO | | InnoDB | YES | YES | YES | | MEMORY | YES | NO | NO | +------------+---------+--------------+------+ Thursday, 15 October 2009
  53. 53. USING SPIDER (Simple case) Thursday, 15 October 2009
  54. 54. Preparing remote servers (1) Create three servers using MySQL Sandbox make_multiple_sandbox --group_directory=spider_dir --sandbox_base_port=6000 --check_base_port 5.1.39 Thursday, 15 October 2009
  55. 55. Preparing remote servers (2) Check the port numbers ~/sandboxes/spider_dir/use_all "show variables like 'port'" # server: 1: Variable_name Value port 6001 # server: 2: Variable_name Value port 6002 # server: 3: Variable_name Value port 6003 Thursday, 15 October 2009
  56. 56. Preparing remote servers (3) create table definition (tablea.sql) drop schema if exists myspider; create schema myspider; use myspider; Create table tbl_a( col_a int, col_b int, primary key(col_a) ); Thursday, 15 October 2009
  57. 57. Preparing remote servers (4) create table in remote servers cd $HOME/sandboxes/spider_dir ./use_all "source tablea.sql" Thursday, 15 October 2009
  58. 58. setting the main server (1) create table definition (tablea_main.sql) drop schema if exists myspider; create schema myspider; use myspider; Create table tbl_b( col_a int, col_b int, primary key(col_a) ) engine = Spider -- continues ... Thursday, 15 October 2009
  59. 59. setting the main server (2) create table definition (tablea_main.sql) (continues) Connection ' table "tbl_a", user "msandbox", password "msandbox" ' partition by range( col_a ) ( partition pt1 values less than (1000) comment 'host "127.0.0.1", port "6001"', partition pt2 values less than (2000) comment 'host "127.0.0.1", port "6002"', partition pt3 values less than (MAXVALUE) comment 'host "127.0.0.1", port "6003"' ); Thursday, 15 October 2009
  60. 60. setting the main server (3) create table ./use < tablea_main.sql Thursday, 15 October 2009
  61. 61. (Finally) using it (1) in the main server ./use myspider insert into tbl_b values (500,1), (1500,2), (5000,3); Query OK, 3 rows affected (0.01 sec) Records: 3 Duplicates: 0 Warnings: 0 Thursday, 15 October 2009
  62. 62. (Finally) using it (2) in the main server select * from tbl_b; +-------+-------+ | col_a | col_b | +-------+-------+ | 500 | 1 | | 1500 | 2 | | 5000 | 3 | +-------+-------+ 3 rows in set (0.01 sec) Thursday, 15 October 2009
  63. 63. WHERE IS THE DATA? 500 < 1000 = host 1 1500 < 2000 = host2 5000 < MAXVALUE = host3 Thursday, 15 October 2009
  64. 64. Looking for the data in the "remote" servers $HOME/sandboxes/spider_dir/use_all "select * from myspider.tbl_a" # server: 1: col_a col_b 500 1 # server: 2: col_a col_b 1500 2 # server: 3: col_a col_b 5000 3 Thursday, 15 October 2009
  65. 65. Using Spider (more complex case) Thursday, 15 October 2009
  66. 66. Setting more remote servers (1) in the main server ./use myspider drop table tbl_b; Thursday, 15 October 2009
  67. 67. Setting more remote servers (2) create 20 remote servers make_multiple_sandbox --how_many_nodes=20 --group_directory=spider_dir --sandbox_base_port=6000 5.1.39 Thursday, 15 October 2009
  68. 68. Setting more remote servers (3) create tables for the employees database cd $HOME/sandboxes/spider_dir wget http://datacharmer.org/downloads/ spider_remote_employees.sql ./use_all "source spider_remote_employees.sql" # see also http://launchpad.net/test-db Thursday, 15 October 2009
  69. 69. the test employees database http://launchpad.net/test-db Thursday, 15 October 2009
  70. 70. Setting the main server (1) create tables for the employees database cd $HOME/sandboxes/spider_main wget http://datacharmer.org/downloads/ spider_main_employees.sql ./use < spider_main_employees.sql # see also http://launchpad.net/test-db Thursday, 15 October 2009
  71. 71. checking the remote servers see how many rows have you got after loading cd $HOME/sandboxes/spider_dir ./use_all "select count(*) from employees.salaries" # server: 1: count(*) 0 # server: 2: count(*) 18293 # server: 3: count(*) 37957 # server: 4: count(*) 57440 ... Thursday, 15 October 2009
  72. 72. Running remote commands with Spider Thursday, 15 October 2009
  73. 73. data data data data data UDF CREATE TABLE db.t1(i int) "" host "127.0.0.1", port "6001", user "msandbox" 1 = success 0 = failure Thursday, 15 October 2009
  74. 74. data data data data data UDF INSERT INTO db.t1 VALUES (1000), (2000), (3000) "" host "127.0.0.1", port "6001", user "msandbox" 1 = success 0 = failure Thursday, 15 October 2009
  75. 75. data data data data data CREATE TEMPORARY TABLE my_temp_table (i int) my_temp_table Thursday, 15 October 2009
  76. 76. data data data data data UDF SELECT * FROM db.t1 "my_temp_table" host "127.0.0.1", port "6001", user "msandbox" my_temp_table 1000 2000 3000 Thursday, 15 October 2009
  77. 77. creating a remote table select spider_direct_sql( "create table myspider.t1 (id int)", "", "port '6001', host '127.0.0.1', user 'msandbox', password 'msandbox'"); # result: 1 Thursday, 15 October 2009
  78. 78. inserting records into remote table select spider_direct_sql( "insert into myspider.t1 values (1000), (2000),(3000)", "", "port '6001', host '127.0.0.1', user 'msandbox', password 'msandbox'"); # result 1 Thursday, 15 October 2009
  79. 79. getting remote records (1) create temporary table remote6001 (i int); select spider_direct_sql( "select * from myspider.t1", "remote6001", "port '6001', host '127.0.0.1', user 'msandbox', password 'msandbox'"); # result: 1 Thursday, 15 October 2009
  80. 80. getting remote records (2) select * from remote6001; +------+ | i | +------+ | 1000 | | 2000 | | 3000 | +------+ Thursday, 15 October 2009
  81. 81. LIVE DEMO Where everything can go wrong! Thursday, 15 October 2009
  82. 82. Vertical partitioning Thursday, 15 October 2009
  83. 83. The vertical partition engine Same author of the Spider engine Open source https://launchpad.net/vpformysql Simple concept Thursday, 15 October 2009
  84. 84. Original table employees id name salary dept email 1 Joe 1300 1 joe@company.com 2 Rick 1250 4 rick@other.com 3 Fred 1600 11 fred@some.org Thursday, 15 October 2009
  85. 85. Split tables empl2 empl1 id name salary id dept email 1 Joe 1300 1 1 joe@company.com 2 Rick 1250 2 4 rick@other.com 3 Fred 1600 3 11 fred@some.org employees id name salary dept email 1 Joe 1300 1 joe@company.com 2 Rick 1250 4 rick@other.com 3 Fred 1600 11 fred@some.org Thursday, 15 October 2009
  86. 86. Split tables empl2 empl1 id name salary id dept email 1 Joe 1300 1 1 joe@company.com 2 Rick 1250 2 4 rick@other.com 3 Fred 1600 3 11 fred@some.org employees id name salary dept email 1 Joe 1300 1 joe@company.com no data 2 Rick 1250 4 rick@other.com here 3 Fred 1600 11 fred@some.org Thursday, 15 October 2009
  87. 87. Vertical partitioning installation Similar to Spider: download the MySQL source code download the engine source from https://launchpad.net/vpformysql copy source under $basedir/storage apply patch compile Thursday, 15 October 2009
  88. 88. vertical partitioning syntax CREATE TABLE empl1 ( id int not null, name varchar(50), salary decimal(10,3), primary key (id) ); CREATE TABLE empl2 ( id int not null, dept int, email varchar(100), primary key (id) ); Thursday, 15 October 2009
  89. 89. vertical partitioning syntax CREATE TABLE employees ( id int not null, name varchar(50), salary decimal(10,3), dept int, email varchar(100), primary key (id) ) engine=VP COMMENT='table_name_list "empl1 empl2"'; Thursday, 15 October 2009
  90. 90. inserting into the vertical partitioning engine insert into employees values (1, 'Joe', 1300,1,'joe@company.com'); Query OK, 1 row affected (0.00 sec) insert into employees values (2, 'Rick', 1250,4,'rick@other.com'); Query OK, 1 row affected (0.00 sec) Thursday, 15 October 2009
  91. 91. retrieving data from the vertical partitioning engine select * from employees; +----+------+----------+------+-----------------+ | id | name | salary | dept | email | +----+------+----------+------+-----------------+ | 1 | Joe | 1300.000 | 1 | joe@company.com | | 2 | Rick | 1250.000 | 4 | rick@other.com | +----+------+----------+------+-----------------+ The data is actually in empl1 and empl2 (check the data directory file sizes to make sure) Thursday, 15 October 2009
  92. 92. READING MORE my blog (search for Spider) http://datacharmer.blogspot.com home of the engines http://launchpad.net/spiderformysql http://launchpad.net/vpformysql Look for these slides: http://slideshare.net/datacharmer Thursday, 15 October 2009
  93. 93. THANKS Q&A Thursday, 15 October 2009
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×