Boost performance
with
MySQL 5.1 partitions

March 19, 2009
Giuseppe Maxia
MySQL Community Team Lead
Sun Microsystems
Who's this guy?




about me
                  2
Giuseppe Maxia
    a.k.a. The Data Charmer
•
    MySQL Community Team Lead
•
    Long time hacking with MySQL features
•
 ...
MySQL 5.1 GA




               4
MySQL 5.1 GA




               5
MySQL 5.1 GA




               5
Defining the problem




   YOUR
   NEEDS               6
The problem(s)
    Too much data
•
    Not enough RAM
•
    Historical data
•
    Growing data
•
    Rotating data
•




 ...
Too much data




         data




                8
Not enough RAM



                       RAM


                       INDEXES
             INDEXES
      data




        ...
Not enough RAM




                       INDEXES
                       RAM
             INDEXES
      data




         ...
MySQL 5.1 partitions




     WHAT
                       11
What is it?
• Logical splitting of tables
• Transparent to user




                                12
Logical splitting
• No need to create separate tables
• No need to move chunks of data across files




                  ...
MySQL 5.1 partitions




          SQL
 TRANSPARENT
                       14
COMPARED TO MERGE TABLES
                     separate tables
                 •
   MERGE TABLE
                     risk ...
COMPARED TO MERGE TABLES
                        One table
                    •
PARTITIONED TABLE
                       ...
Wait a minute ...
• WHAT THE HELL DOES quot;LOGICAL SPLITquot;
  REALLY MEANS?
• LET ME EXPLAIN ...




                  ...
Physical partitioning (1)
• Take a map




                            18
Physical partitioning (2)
• cut it into pieces




                            19
Physical partitioning (3)
• What you have, is several different pieces




                                               ...
Physical partitioning (4)
• If you want the map back, you need some
  application (adhesive tape) and you may get it
  wro...
Logical partitioning (1)
• Take a map




                           22
Logical partitioning (2)
• fold it to show the piece you need




                                       23
Logical partitioning (3)
• what you have is still a map, even if you see only
  one part.




                            ...
Logical partitioning (4)
• if you unfold the map, you still have the whole thing




                                     ...
What partitions can do
    logical split
•
    but you can also split the data physically
•
    granular partition (subpar...
MySQL 5.1 partitions




        WHY
                       27
4 main reasons to use partitions
    To make single inserts and selects faster
•
    To make range selects faster
•
    To...
MySQL 5.1 partitions




       HOW
                       29
HOW TO MAKE PARTITIONS




                         30
HOW TO MAKE PARTITIONS

•RTFM ...



                         30
HOW TO MAKE PARTITIONS

•RTFM ...
• No, seriously, the manual has everything




                                         ...
HOW TO MAKE PARTITIONS

•RTFM ...
• No, seriously, the manual has everything
• But if you absolutely insist ...




      ...
HOW TO MAKE PARTITIONS
CREATE TABLE t1 (
   id int
) ENGINE=InnoDB
   # or MyISAM, ARCHIVE
PARTITION BY RANGE (id)
(
  PAR...
HOW TO MAKE PARTITIONS
CREATE TABLE t1 (
   id int
) ENGINE=InnoDB
PARTITION BY LIST (id)
(
  PARTITION P1 VALUES IN (1,2,...
HOW TO MAKE PARTITIONS
CREATE TABLE t1 (
  id int not null primary key
) ENGINE=InnoDB
PARTITION BY HASH (id)
PARTITIONS 1...
HOW TO MAKE PARTITIONS
CREATE TABLE t1 (
  id int not null primary key
) ENGINE=InnoDB
PARTITION BY KEY ()
PARTITIONS 10;
...
Limitations
• Can partition only by INTEGER columns
• OR you can partition by an expression, which must
  return an intege...
PARTITIONING BY DATE
CREATE TABLE t1 (
  d date
) ENGINE=InnoDB
PARTITION BY RANGE (year(d))
(
  PARTITION P1 VALUES
     ...
PARTITIONING BY DATE
CREATE TABLE t1 (
  d date
) ENGINE=InnoDB
PARTITION BY RANGE (to_days(d))
(
  PARTITION P1 VALUES
  ...
MySQL 5.1 partitions




     WHEN
                       38
When to use partitions?
• if you have large tables
• if you know that you will always query for the
  partitioning column
...
MySQL 5.1 partitions




   HANDS
    ON                 40
components used for testing




                              41
components used for testing
• MySQL Sandbox
 > created MySQL instances in seconds
 > http://launchpad.net/mysql-sandbox


...
components used for testing
• MySQL Sandbox
  > created MySQL instances in seconds
  > http://launchpad.net/mysql-sandbox
...
components used for testing
• MySQL Sandbox
  > created MySQL instances in seconds
  > http://launchpad.net/mysql-sandbox
...
employees test database




                          42
How many partitions
from information_schema.partitions
+-------+-----------------+----------+
| pname | expr            | ...
How many records
select count(*) from salaries;
 +----------+
 | count(*) |
 +----------+
 | 2844047 |
 +----------+
1 row...
How many records in 1998
not partitioned
select count(*) from salaries
where from_date between
'1998-01-01' and '1998-12-3...
How many records in 1998
PARTITIONED
select count(*) from salaries
where from_date between
'1998-01-01' and '1998-12-31';
...
How many records in 1999
not partitioned
select count(*) from salaries
where from_date between
'1999-01-01' and '1999-12-3...
How many records in 1999
PARTITIONED
select count(*) from salaries
where from_date between
'1999-01-01' and '1999-12-31';
...
Deleting 1998 records
not partitioned
delete from salaries where
from_date between '1998-01-01'
and '1998-12-31';
Query OK...
Deleting 1998 records
partitioned
alter table salaries drop
partition p15;
Query OK, 0 rows affected (1.35
sec)




      ...
MySQL 5.1 partitions




 LEVERAGING
 REPLICATION
                       51
Replication schemes
                     INNODB                   INNODB
                     NOT                      NOT...
Replication schemes
                              concurrent insert
                INNODB
                PARTITIONED
 MA...
MySQL 5.1 partitions




PITFALLS
                       54
Expressions
• Partition by function
• Search by column




                          55
WRONG
PARTITION BY RANGE(year(from_date))

select count(*) from salaries where
year(from_date) = 1998;
+----------+
| coun...
RIGHT
PARTITION BY RANGE(year(from_date))

select count(*) from salaries where
from_date between '1998-01-01' and
'1998-12...
EXPLAIN
explain partitions select count(*)
from salaries where year(from_date)
= 1998G
***** 1. row ****
            id: 1...
EXPLAIN
explain partitions select count(*)
from salaries where from_date
between '1998-01-01' and
'1998-12-31'G
***** 1. r...
MySQL 5.1 partitions




   BENCHMARKS

                       60
Partitions benchmarks (laptop)


 engine                                storage
                                        (M...
Benchmarking results (laptop)
   engine                query year   query year
                         2000         2002
...
Partitioning storage (huge server)


 engine                              storage
                                       (...
Benchmarking results (huge server)
engine                         6 month range
InnoDB                              4 min ...
MySQL 5.1 partitions




   TOOLS
                       65
The partition helper
• The INFORMATION_SCHEMA.PARTITIONS table
• The partition helper
  > http://datacharmer.blogspot.com/...
MySQL 5.1 partitions




        TIPS
                       67
TIPS
• For large table ( indexes > available RAM)
  > DO NOT USE INDEXES
  > PARTITIONS ARE MORE EFFICIENT




           ...
TIPS
• Before adopting partitions in production, benchmark!
• you can create partitions of different sizes
  > in historic...
TIPS
• For large historical tables, consider using ARCHIVE
  + partitions
  > You see benefits for very large amounts of d...
Thank you!



Giuseppe Maxia
http://datacharmer@blogspot.com
datacharmer@sun.com
MySQL 5.1 partitions




        BONUS
        SLIDES
                       72
MySQL 5.1 partitions




ANNOYANCES
                       73
Annoyances with partitions
• CREATE TABLE STATEMENT hard to read




                                        74
Annoyances with partitions
• CREATE TABLE STATEMENT hard to read




                                        74
Annoyances with partitions
• hard to read (FIXED!! in 5.1.31)




                                     75
Annoyances with partitions
• hard to read (FIXED!! in 5.1.31)




                                     75
Annoyances with partitions
• CREATE TABLE only keeps the result of the
  expression for each partition.
• (you can use the...
Annoyances with partitions
CREATE TABLE t1 (
  d date
) ENGINE=InnoDB
PARTITION BY RANGE (to_days(d))
(
  PARTITION P1 VAL...
Annoyances with partitions
CREATE TABLE `t1` (
  `d` date DEFAULT NULL
) ENGINE=InnoDB DEFAULT
CHARSET=latin1
/*!50100 PAR...
Fixing annoyances with partitions
select partition_name part,
partition_expression expr,
partition_description val
from in...
fixing annoyances with partitions
select partition_name part ,
partition_expression expr,
from_days(partition_description)...
Upcoming SlideShare
Loading in...5
×

Boost Performance With My S Q L 51 Partitions

4,149

Published on

Published in: Technology

Boost Performance With My S Q L 51 Partitions

  1. 1. Boost performance with MySQL 5.1 partitions March 19, 2009 Giuseppe Maxia MySQL Community Team Lead Sun Microsystems
  2. 2. Who's this guy? about me 2
  3. 3. Giuseppe Maxia a.k.a. The Data Charmer • MySQL Community Team Lead • Long time hacking with MySQL features • Formerly, database consultant, designer, coder. • A passion for QA • An even greater passion for open source • ... and community • Passionate blogger • http://datacharmer.blogspot.com • 3
  4. 4. MySQL 5.1 GA 4
  5. 5. MySQL 5.1 GA 5
  6. 6. MySQL 5.1 GA 5
  7. 7. Defining the problem YOUR NEEDS 6
  8. 8. The problem(s) Too much data • Not enough RAM • Historical data • Growing data • Rotating data • 7
  9. 9. Too much data data 8
  10. 10. Not enough RAM RAM INDEXES INDEXES data 9
  11. 11. Not enough RAM INDEXES RAM INDEXES data 10
  12. 12. MySQL 5.1 partitions WHAT 11
  13. 13. What is it? • Logical splitting of tables • Transparent to user 12
  14. 14. Logical splitting • No need to create separate tables • No need to move chunks of data across files 13
  15. 15. MySQL 5.1 partitions SQL TRANSPARENT 14
  16. 16. COMPARED TO MERGE TABLES separate tables • MERGE TABLE risk of duplicates • insert in each table • no constraints • 15
  17. 17. COMPARED TO MERGE TABLES One table • PARTITIONED TABLE No risk of duplicates • insert in one table • constraints enforced • 16
  18. 18. Wait a minute ... • WHAT THE HELL DOES quot;LOGICAL SPLITquot; REALLY MEANS? • LET ME EXPLAIN ... 17
  19. 19. Physical partitioning (1) • Take a map 18
  20. 20. Physical partitioning (2) • cut it into pieces 19
  21. 21. Physical partitioning (3) • What you have, is several different pieces 20
  22. 22. Physical partitioning (4) • If you want the map back, you need some application (adhesive tape) and you may get it wrong 21
  23. 23. Logical partitioning (1) • Take a map 22
  24. 24. Logical partitioning (2) • fold it to show the piece you need 23
  25. 25. Logical partitioning (3) • what you have is still a map, even if you see only one part. 24
  26. 26. Logical partitioning (4) • if you unfold the map, you still have the whole thing 25
  27. 27. What partitions can do logical split • but you can also split the data physically • granular partition (subpartitioning) • different methods (range, list, hash, key) • 26
  28. 28. MySQL 5.1 partitions WHY 27
  29. 29. 4 main reasons to use partitions To make single inserts and selects faster • To make range selects faster • To help split the data across different paths • To store historical data efficiently • If you need to delete large chunks of data • INSTANTLY 28
  30. 30. MySQL 5.1 partitions HOW 29
  31. 31. HOW TO MAKE PARTITIONS 30
  32. 32. HOW TO MAKE PARTITIONS •RTFM ... 30
  33. 33. HOW TO MAKE PARTITIONS •RTFM ... • No, seriously, the manual has everything 30
  34. 34. HOW TO MAKE PARTITIONS •RTFM ... • No, seriously, the manual has everything • But if you absolutely insist ... 30
  35. 35. HOW TO MAKE PARTITIONS CREATE TABLE t1 ( id int ) ENGINE=InnoDB # or MyISAM, ARCHIVE PARTITION BY RANGE (id) ( PARTITION P1 VALUES LESS THAN (10), PARTITION P2 VALUES LESS THAN (20) ) 31
  36. 36. HOW TO MAKE PARTITIONS CREATE TABLE t1 ( id int ) ENGINE=InnoDB PARTITION BY LIST (id) ( PARTITION P1 VALUES IN (1,2,4), PARTITION P2 VALUES IN (3,5,9) ) 32
  37. 37. HOW TO MAKE PARTITIONS CREATE TABLE t1 ( id int not null primary key ) ENGINE=InnoDB PARTITION BY HASH (id) PARTITIONS 10; 33
  38. 38. HOW TO MAKE PARTITIONS CREATE TABLE t1 ( id int not null primary key ) ENGINE=InnoDB PARTITION BY KEY () PARTITIONS 10; 34
  39. 39. Limitations • Can partition only by INTEGER columns • OR you can partition by an expression, which must return an integer • Maximum 1024 partitions • If you have a Unique Key or PK, the partition column must be part of that key • No Foreign Key support • No Fulltext or GIS support 35
  40. 40. PARTITIONING BY DATE CREATE TABLE t1 ( d date ) ENGINE=InnoDB PARTITION BY RANGE (year(d)) ( PARTITION P1 VALUES LESS THAN (1999), PARTITION P2 VALUES LESS THAN (2000) 36 )
  41. 41. PARTITIONING BY DATE CREATE TABLE t1 ( d date ) ENGINE=InnoDB PARTITION BY RANGE (to_days(d)) ( PARTITION P1 VALUES LESS THAN (to_days('1999-01-01')), PARTITION P2 VALUES LESS THAN (to_days('2000-01-01')) 37 )
  42. 42. MySQL 5.1 partitions WHEN 38
  43. 43. When to use partitions? • if you have large tables • if you know that you will always query for the partitioning column • if you have historical tables that you want to purge quickly • if your indexes are larger than the available RAM 39
  44. 44. MySQL 5.1 partitions HANDS ON 40
  45. 45. components used for testing 41
  46. 46. components used for testing • MySQL Sandbox > created MySQL instances in seconds > http://launchpad.net/mysql-sandbox 41
  47. 47. components used for testing • MySQL Sandbox > created MySQL instances in seconds > http://launchpad.net/mysql-sandbox • MySQL Employees Test Database > has ~ 4 mil records in 6 tables > http://launchpad.net/test-db 41
  48. 48. components used for testing • MySQL Sandbox > created MySQL instances in seconds > http://launchpad.net/mysql-sandbox • MySQL Employees Test Database > has ~ 4 mil records in 6 tables > http://launchpad.net/test-db • Command line ability > fingers > ingenuity 41
  49. 49. employees test database 42
  50. 50. How many partitions from information_schema.partitions +-------+-----------------+----------+ | pname | expr | descr | +-------+-----------------+----------+ | p01 | year(from_date) | 1985 | | p02 | year(from_date) | 1986 | ... | p13 | year(from_date) | 1997 | | p14 | year(from_date) | 1998 | | p15 | year(from_date) | 1999 | | p16 | year(from_date) | 2000 | ... | p19 | year(from_date) | MAXVALUE | +-------+-----------------+----------+ 43
  51. 51. How many records select count(*) from salaries; +----------+ | count(*) | +----------+ | 2844047 | +----------+ 1 row in set (0.01 sec) 44
  52. 52. How many records in 1998 not partitioned select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'; +----------+ | count(*) | +----------+ | 247489 | +----------+ 1 row in set (1.52 sec) # NOT PARTITIONED 45
  53. 53. How many records in 1998 PARTITIONED select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'; +----------+ | count(*) | +----------+ | 247489 | +----------+ 1 row in set (0.41 sec) # partition p15 46
  54. 54. How many records in 1999 not partitioned select count(*) from salaries where from_date between '1999-01-01' and '1999-12-31'; +----------+ | count(*) | +----------+ | 260957 | +----------+ 1 row in set (1.54 sec) # NOT PARTITIONED 47
  55. 55. How many records in 1999 PARTITIONED select count(*) from salaries where from_date between '1999-01-01' and '1999-12-31'; +----------+ | count(*) | +----------+ | 260957 | +----------+ 1 row in set (0.17 sec) # partition p16 48
  56. 56. Deleting 1998 records not partitioned delete from salaries where from_date between '1998-01-01' and '1998-12-31'; Query OK, 247489 rows affected (19.13 sec) # NOT PARTITIONED 49
  57. 57. Deleting 1998 records partitioned alter table salaries drop partition p15; Query OK, 0 rows affected (1.35 sec) 50
  58. 58. MySQL 5.1 partitions LEVERAGING REPLICATION 51
  59. 59. Replication schemes INNODB INNODB NOT NOT MASTER PARTITIONED PARTITIONED SLAVE concurrent insert concurrent read INNODB MyISAM PARTITIONED PARTITIONED BY RANGE BY RANGE SLAVE SLAVE 52 concurrent batch processing large batch processing
  60. 60. Replication schemes concurrent insert INNODB PARTITIONED MASTER INNODB NON PARTITIONED MyISAM PARTITIONED SLAVE SLAVE 53 batch processing concurrent reads
  61. 61. MySQL 5.1 partitions PITFALLS 54
  62. 62. Expressions • Partition by function • Search by column 55
  63. 63. WRONG PARTITION BY RANGE(year(from_date)) select count(*) from salaries where year(from_date) = 1998; +----------+ | count(*) | +----------+ | 247489 | +----------+ 1 row in set (2.25 sec) 56
  64. 64. RIGHT PARTITION BY RANGE(year(from_date)) select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'; +----------+ | count(*) | +----------+ | 247489 | +----------+ 1 row in set (0.46 sec) 57
  65. 65. EXPLAIN explain partitions select count(*) from salaries where year(from_date) = 1998G ***** 1. row **** id: 1 select_type: SIMPLE table: salaries partitions: p01,p02,p03,p04,p05,p06,p07,p08,p09 ,p10,p11,p12,p13,p14,p15,p16,p17,p1 8,p19 type: index 58 ...
  66. 66. EXPLAIN explain partitions select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'G ***** 1. row **** id: 1 select_type: SIMPLE table: salaries partitions: p14,p15 ... 59
  67. 67. MySQL 5.1 partitions BENCHMARKS 60
  68. 68. Partitions benchmarks (laptop) engine storage (MB) innodb 221 myisam 181 archive 74 innodb partitioned (whole) 289 innodb partitioned (file per table) 676 myisam partitioned 182 archive partitioned 72
  69. 69. Benchmarking results (laptop) engine query year query year 2000 2002 InnoDB 1.25 1.25 MyISAM 1.72 1.73 Archive 2.47 2.45 InnoDB partitioned 0.24 0.10 whole InnoDB Partitioned 0.45 0.10 (file per table) MyISAM partitioned 0.18 0.12 Archive partitioned 0.22 0.12
  70. 70. Partitioning storage (huge server) engine storage (GB) innodb (with PK) 330 myisam (with PK) 141 archive 13 innodb partitioned (no PK) 237 myisam partitioned (no PK) 107 archive partitioned 13
  71. 71. Benchmarking results (huge server) engine 6 month range InnoDB 4 min 30s MyISAM 25.03s Archive 22 min 25s InnoDB partitioned by month 13.19 MyISAM partitioned by year 6.31 MyISAM partitioned by month 4.45 Archive partitioned by year 16.67 Archive partitioned by month 8.97
  72. 72. MySQL 5.1 partitions TOOLS 65
  73. 73. The partition helper • The INFORMATION_SCHEMA.PARTITIONS table • The partition helper > http://datacharmer.blogspot.com/2008/12/partition- helper-improving-usability.html > A Perl script that creates partitioning statements • ... anything you are creating right now :) 66
  74. 74. MySQL 5.1 partitions TIPS 67
  75. 75. TIPS • For large table ( indexes > available RAM) > DO NOT USE INDEXES > PARTITIONS ARE MORE EFFICIENT 68
  76. 76. TIPS • Before adopting partitions in production, benchmark! • you can create partitions of different sizes > in historical tables, you can partition – current data by day – recent data by month – distant past data by year > ALL IN THE SAME TABLE! • If you want to enforce a constraint on a integer column, you may set a partition by list, with only the values you want to admit. 69
  77. 77. TIPS • For large historical tables, consider using ARCHIVE + partitions > You see benefits for very large amounts of data ( > 500 MB) > Very effective when you run statistics > Almost the same performance of MyISAM > but 1/10 of storage 70
  78. 78. Thank you! Giuseppe Maxia http://datacharmer@blogspot.com datacharmer@sun.com
  79. 79. MySQL 5.1 partitions BONUS SLIDES 72
  80. 80. MySQL 5.1 partitions ANNOYANCES 73
  81. 81. Annoyances with partitions • CREATE TABLE STATEMENT hard to read 74
  82. 82. Annoyances with partitions • CREATE TABLE STATEMENT hard to read 74
  83. 83. Annoyances with partitions • hard to read (FIXED!! in 5.1.31) 75
  84. 84. Annoyances with partitions • hard to read (FIXED!! in 5.1.31) 75
  85. 85. Annoyances with partitions • CREATE TABLE only keeps the result of the expression for each partition. • (you can use the information_schema to ease the pain in this case) 76
  86. 86. Annoyances with partitions CREATE TABLE t1 ( d date ) ENGINE=InnoDB PARTITION BY RANGE (to_days(d)) ( PARTITION P1 VALUES LESS THAN (to_days('1999-01-01')), PARTITION P2 VALUES LESS THAN (to_days('2000-01-01')) 77 )
  87. 87. Annoyances with partitions CREATE TABLE `t1` ( `d` date DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1 /*!50100 PARTITION BY RANGE (to_days(d)) (PARTITION P1 VALUES LESS THAN (730120) ENGINE = InnoDB, PARTITION P2 VALUES LESS THAN (730485) ENGINE = InnoDB) */ 78
  88. 88. Fixing annoyances with partitions select partition_name part, partition_expression expr, partition_description val from information_schema.partitions where table_name='t1'; +------+------------+--------+ | part | expr | val | +------+------------+--------+ | P1 | to_days(d) | 730120 | | P2 | to_days(d) | 730485 | 79 +------+------------+--------+
  89. 89. fixing annoyances with partitions select partition_name part , partition_expression expr, from_days(partition_description) val from information_schema.partitions where table_name='t1'; +------+------------+------------+ | part | expr | val | +------+------------+------------+ | P1 | to_days(d) | 1999-01-01 | | P2 | to_days(d) | 2000-01-01 | +------+------------+------------+ 80
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×