Partitions Performance with MySQL 5.1 and 5.5

25,540
-1

Published on

learn the basics of MySQL 5.1 partitions and the MySQL 5.5. enhancements

2 Comments
39 Likes
Statistics
Notes
No Downloads
Views
Total Views
25,540
On Slideshare
0
From Embeds
0
Number of Embeds
31
Actions
Shares
0
Downloads
697
Comments
2
Likes
39
Embeds 0
No embeds

No notes for slide

Partitions Performance with MySQL 5.1 and 5.5

  1. 1. ith ew nc .5 ma d 5 for an st per 5.1 B oo SQL ing M y rtit ion pa Giuseppe Maxia MySQL Community Team Lead Sun Microsystems Wednesday, 13 January 2010
  2. 2. Who's this guy? about me 2 Wednesday, 13 January 2010
  3. 3. Giuseppe Maxia • a.k.a. The Data Charmer • MySQL Community Team Lead • Long time hacking with MySQL features • Formerly, database consultant, designer, coder. • A passion for QA • An even greater passion for open source • ... and community • Passionate blogger • http://datacharmer.blogspot.com 3 Wednesday, 13 January 2010
  4. 4. MySQL 5.1 GA 4 Wednesday, 13 January 2010
  5. 5. MySQL 5.1 GA 5 Wednesday, 13 January 2010
  6. 6. Defining the problem YOUR NEEDS 6 Wednesday, 13 January 2010
  7. 7. The problem(s) • Too much data • Not enough RAM • Historical data • Growing data • Rotating data 7 Wednesday, 13 January 2010
  8. 8. Too much data data 8 Wednesday, 13 January 2010
  9. 9. Not enough RAM RAM INDEXES data INDEXES 9 Wednesday, 13 January 2010
  10. 10. Not enough RAM INDEXES RAM data INDEXES 10 Wednesday, 13 January 2010
  11. 11. MySQL 5.1 partitions WHAT 11 Wednesday, 13 January 2010
  12. 12. What is it? • Logical splitting of tables • Transparent to user 12 Wednesday, 13 January 2010
  13. 13. Logical splitting • No need to create separate tables • No need to move chunks of data across files 13 Wednesday, 13 January 2010
  14. 14. MySQL 5.1 partitions TRANSPARENT SQL 14 Wednesday, 13 January 2010
  15. 15. COMPARED TO MERGE TABLES MERGE TABLE • separate tables • risk of duplicates • insert in each table • no constraints 15 Wednesday, 13 January 2010
  16. 16. COMPARED TO MERGE TABLES PARTITIONED TABLE • One table • No risk of duplicates • insert in one table • constraints enforced 16 Wednesday, 13 January 2010
  17. 17. Wait a minute ... • WHAT THE HELL DOES "LOGICAL SPLIT" REALLY MEANS? • LET ME EXPLAIN ... 17 Wednesday, 13 January 2010
  18. 18. Physical partitioning (1) • Take a map 18 Wednesday, 13 January 2010
  19. 19. Physical partitioning (2) • cut it into pieces 19 Wednesday, 13 January 2010
  20. 20. Physical partitioning (3) • What you have, is several different pieces 20 Wednesday, 13 January 2010
  21. 21. Physical partitioning (4) • If you want the map back, you need some application (adhesive tape) and you may get it wrong 21 Wednesday, 13 January 2010
  22. 22. Logical partitioning (1) • Take a map 22 Wednesday, 13 January 2010
  23. 23. Logical partitioning (2) • fold it to show the piece you need 23 Wednesday, 13 January 2010
  24. 24. Logical partitioning (3) • what you have is still a map, even if you see only one part. 24 Wednesday, 13 January 2010
  25. 25. Logical partitioning (4) • if you unfold the map, you still have the whole thing 25 Wednesday, 13 January 2010
  26. 26. What partitions can do • logical split • but you can also split the data physically • granular partition (subpartitioning) • different methods (range, list, hash, key) 26 Wednesday, 13 January 2010
  27. 27. Partition pruning 1a - unpartitioned table - SINGLE RECORD select * from table_name where colx = 120 Wednesday, 13 January 2010
  28. 28. Partition pruning 1b - unpartitioned table - SINGLE RECORD select * from table_name where colx = 350 Wednesday, 13 January 2010
  29. 29. Partition pruning 1c - unpartitioned table - RANGE select * from table_name where colx between 120 and 230 Wednesday, 13 January 2010
  30. 30. Partition pruning 2a - table partitioned by colx - SINGLE REC 1-99 100-199 select * from 200-299 table_name where colx = 300-399 120 400-499 500-599 Wednesday, 13 January 2010
  31. 31. Partition pruning 2b - table partitioned by colx - SINGLE REC 1-99 100-199 select * from 200-299 table_name where colx = 300-399 350 400-499 500-599 Wednesday, 13 January 2010
  32. 32. Partition pruning 2c - table partitioned by colx - RANGE 1-99 select * 100-199 from table_name 200-299 where colx 300-399 between 120 and 230 400-499 500-599 Wednesday, 13 January 2010
  33. 33. MySQL 5.1 partitions WHY 33 Wednesday, 13 January 2010
  34. 34. 4 main reasons to use partitions • To make single inserts and selects faster • To make range selects faster • To help split the data across different paths • To store historical data efficiently • If you need to delete large chunks of data INSTANTLY 34 Wednesday, 13 January 2010
  35. 35. MySQL 5.1 partitions HOW 35 Wednesday, 13 January 2010
  36. 36. HOW TO MAKE PARTITIONS •RTFM ... • No, seriously, the manual has everything • But if you absolutely insist ... 36 Wednesday, 13 January 2010
  37. 37. HOW TO MAKE PARTITIONS CREATE TABLE t1 ( id int ) ENGINE=InnoDB # or MyISAM, ARCHIVE PARTITION BY RANGE (id) ( PARTITION P1 VALUES LESS THAN (10), PARTITION P2 VALUES LESS THAN (20) ) 37 Wednesday, 13 January 2010
  38. 38. HOW TO MAKE PARTITIONS CREATE TABLE t1 ( id int ) ENGINE=InnoDB PARTITION BY LIST (id) ( PARTITION P1 VALUES IN (1,2,4), PARTITION P2 VALUES IN (3,5,9) ) 38 Wednesday, 13 January 2010
  39. 39. HOW TO MAKE PARTITIONS CREATE TABLE t1 ( id int not null primary key ) ENGINE=InnoDB PARTITION BY HASH (id) PARTITIONS 10; 39 Wednesday, 13 January 2010
  40. 40. HOW TO MAKE PARTITIONS CREATE TABLE t1 ( id int not null primary key ) ENGINE=InnoDB PARTITION BY KEY () PARTITIONS 10; 40 Wednesday, 13 January 2010
  41. 41. Limitations • Can partition only by INTEGER columns • OR you can partition by an expression, which must return an integer • Maximum 1024 partitions • If you have a Unique Key or PK, the partition column must be part of that key • No Foreign Key support • No Fulltext or GIS support 41 Wednesday, 13 January 2010
  42. 42. PARTITIONING BY DATE CREATE TABLE t1 ( d date ) ENGINE=InnoDB PARTITION BY RANGE (year(d)) ( PARTITION P1 VALUES LESS THAN (1999), PARTITION P2 VALUES LESS THAN (2000) 42 ) Wednesday, 13 January 2010
  43. 43. PARTITIONING BY DATE CREATE TABLE t1 ( d date ) ENGINE=InnoDB PARTITION BY RANGE (to_days(d)) ( PARTITION P1 VALUES LESS THAN (to_days('1999-01-01')), PARTITION P2 VALUES LESS THAN (to_days('2000-01-01')) 43 ) Wednesday, 13 January 2010
  44. 44. MySQL 5.1 partitions WHEN 44 Wednesday, 13 January 2010
  45. 45. When to use partitions? • if you have large tables • if you know that you will always query for the partitioning column • if you have historical tables that you want to purge quickly • if your indexes are larger than the available RAM 45 Wednesday, 13 January 2010
  46. 46. MySQL 5.1 partitions HANDS ON 46 Wednesday, 13 January 2010
  47. 47. components used for testing • MySQL Sandbox > created MySQL instances in seconds > http://launchpad.net/mysql-sandbox • MySQL Employees Test Database > has ~ 4 mil records in 6 tables > http://launchpad.net/test-db • Command line ability > fingers > ingenuity 47 Wednesday, 13 January 2010
  48. 48. employees test database 48 Wednesday, 13 January 2010
  49. 49. How many partitions from information_schema.partitions +-------+-----------------+----------+ | pname | expr | descr | +-------+-----------------+----------+ | p01 | year(from_date) | 1985 | | p02 | year(from_date) | 1986 | ... | p13 | year(from_date) | 1997 | | p14 | year(from_date) | 1998 | | p15 | year(from_date) | 1999 | | p16 | year(from_date) | 2000 | ... | p19 | year(from_date) | MAXVALUE | +-------+-----------------+----------+ 49 Wednesday, 13 January 2010
  50. 50. How many records select count(*) from salaries; +----------+ | count(*) | +----------+ | 2844047 | +----------+ 1 row in set (0.01 sec) 50 Wednesday, 13 January 2010
  51. 51. How many records in 1998 not partitioned select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'; +----------+ | count(*) | +----------+ | 247489 | +----------+ 1 row in set (1.52 sec) # NOT PARTITIONED 51 Wednesday, 13 January 2010
  52. 52. How many records in 1998 PARTITIONED select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'; +----------+ | count(*) | +----------+ | 247489 | +----------+ 1 row in set (0.41 sec) # partition p15 52 Wednesday, 13 January 2010
  53. 53. How many records in 1999 not partitioned select count(*) from salaries where from_date between '1999-01-01' and '1999-12-31'; +----------+ | count(*) | +----------+ | 260957 | +----------+ 1 row in set (1.54 sec) # NOT PARTITIONED 53 Wednesday, 13 January 2010
  54. 54. How many records in 1999 PARTITIONED select count(*) from salaries where from_date between '1999-01-01' and '1999-12-31'; +----------+ | count(*) | +----------+ | 260957 | +----------+ 1 row in set (0.17 sec) # partition p16 54 Wednesday, 13 January 2010
  55. 55. Deleting 1998 records not partitioned delete from salaries where from_date between '1998-01-01' and '1998-12-31'; Query OK, 247489 rows affected (19.13 sec) # NOT PARTITIONED 55 Wednesday, 13 January 2010
  56. 56. Deleting 1998 records partitioned alter table salaries drop partition p15; Query OK, 0 rows affected (1.35 sec) 56 Wednesday, 13 January 2010
  57. 57. MySQL 5.1 partitions LEVERAGING REPLICATION 57 Wednesday, 13 January 2010
  58. 58. Replication schemes INNODB INNODB NOT NOT MASTER PARTITIONED PARTITIONED concurrent insert SLAVE concurrent read INNODB MyISAM PARTITIONED PARTITIONED BY RANGE BY RANGE SLAVE SLAVE 58 concurrent batch processing large batch processing Wednesday, 13 January 2010
  59. 59. Replication schemes concurrent insert INNODB PARTITIONED MASTER INNODB NON MyISAM PARTITIONED SLAVE PARTITIONED SLAVE 59 batch processing concurrent reads Wednesday, 13 January 2010
  60. 60. MySQL 5.1 partitions PITFALLS 60 Wednesday, 13 January 2010
  61. 61. Expressions • Partition by function • Search by column 61 Wednesday, 13 January 2010
  62. 62. WRONG PARTITION BY RANGE(year(from_date)) select count(*) from salaries where year(from_date) = 1998; +----------+ | count(*) | +----------+ | 247489 | +----------+ 1 row in set (2.25 sec) 62 Wednesday, 13 January 2010
  63. 63. RIGHT PARTITION BY RANGE(year(from_date)) select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'; +----------+ | count(*) | +----------+ | 247489 | +----------+ 1 row in set (0.46 sec) 63 Wednesday, 13 January 2010
  64. 64. EXPLAIN explain partitions select count(*) from salaries where year(from_date) = 1998G ***** 1. row **** id: 1 select_type: SIMPLE table: salaries partitions: p01,p02,p03,p04,p05,p06,p07,p08,p09 ,p10,p11,p12,p13,p14,p15,p16,p17,p1 8,p19 type: index 64 ... Wednesday, 13 January 2010
  65. 65. EXPLAIN explain partitions select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'G ***** 1. row **** id: 1 select_type: SIMPLE table: salaries partitions: p14,p15 ... 65 Wednesday, 13 January 2010
  66. 66. MySQL 5.1 partitions BENCHMARKS 66 Wednesday, 13 January 2010
  67. 67. Partitions benchmarks (laptop) engine storage (MB) innodb 221 myisam 181 archive 74 innodb partitioned (whole) 289 innodb partitioned (file per table) 676 myisam partitioned 182 archive partitioned 72 Wednesday, 13 January 2010
  68. 68. Benchmarking results (laptop) engine query year query year 2000 2002 InnoDB 1.25 1.25 MyISAM 1.72 1.73 Archive 2.47 2.45 InnoDB partitioned 0.24 0.10 whole InnoDB Partitioned 0.45 0.10 (file per table) MyISAM partitioned 0.18 0.12 Archive partitioned 0.22 0.12 Wednesday, 13 January 2010
  69. 69. Partitioning storage (huge server) engine storage (GB) innodb (with PK) 330 myisam (with PK) 141 archive 13 innodb partitioned (no PK) 237 myisam partitioned (no PK) 107 archive partitioned 13 Wednesday, 13 January 2010
  70. 70. Benchmarking results (huge server) engine 6 month range InnoDB 4 min 30s MyISAM 25.03s Archive 22 min 25s InnoDB partitioned by month 13.19 MyISAM partitioned by year 6.31 MyISAM partitioned by month 4.45 Archive partitioned by year 16.67 Archive partitioned by month 8.97 Wednesday, 13 January 2010
  71. 71. MySQL 5.1 partitions TOOLS 71 Wednesday, 13 January 2010
  72. 72. The partition helper • The INFORMATION_SCHEMA.PARTITIONS table • The partition helper > http://datacharmer.blogspot.com/2008/12/partition- helper-improving-usability.html > A Perl script that creates partitioning statements • ... anything you are creating right now :) 72 Wednesday, 13 January 2010
  73. 73. MySQL 5.1 partitions TIPS 73 Wednesday, 13 January 2010
  74. 74. TIPS • For large table ( indexes > available RAM) > DO NOT USE INDEXES > PARTITIONS ARE MORE EFFICIENT 74 Wednesday, 13 January 2010
  75. 75. TIPS • Before adopting partitions in production, benchmark! • you can create partitions of different sizes > in historical tables, you can partition – current data by day – recent data by month – distant past data by year > ALL IN THE SAME TABLE! • If you want to enforce a constraint on a integer column, you may set a partition by list, with only the values you want to admit. 75 Wednesday, 13 January 2010
  76. 76. TIPS • For large historical tables, consider using ARCHIVE + partitions > You see benefits for very large amounts of data ( > 500 MB) > Very effective when you run statistics > Almost the same performance of MyISAM > but 1/10 of storage 76 Wednesday, 13 January 2010
  77. 77. MySQL 5.5 partitions NEW FEATURES 77 Wednesday, 13 January 2010
  78. 78. MySQL 5.5 enhancements • PARTITION BY RANGE COLUMNS • PARTITION BY LIST COLUMNS • TO_SECONDS 78 Wednesday, 13 January 2010
  79. 79. R E MySQL 5.5 enhancements CREATE TABLE t ( FO dt date B E 5.1 ) PARTITION BY RANGE (TO_DAYS(dt)) ( PARTITION p01 VALUES LESS THAN (TO_DAYS('2007-01-01')), PARTITION p02 VALUES LESS THAN (TO_DAYS('2008-01-01')), PARTITION p03 VALUES LESS THAN (TO_DAYS('2009-01-01')), PARTITION p04 VALUES LESS THAN (MAXVALUE)); 79 Wednesday, 13 January 2010
  80. 80. R E MySQL 5.5 enhancements FO SHOW CREATE TABLE t G B E 5.1 Table: t Create Table: CREATE TABLE `t` ( `dt` date DEFAULT NULL ) ENGINE=MyISAM DEFAULT CHARSET=latin1 /*!50100 PARTITION BY RANGE (TO_DAYS (dt)) (PARTITION p01 VALUES LESS THAN (733042) ENGINE = MyISAM, […] 80 Wednesday, 13 January 2010
  81. 81. MySQL 5.5 enhancements E R CREATE TABLE t ( FT dt date A 5.5 ) PARTITION BY RANGE COLUMNS (dt) ( PARTITION p01 VALUES LESS THAN ('2007-01-01'), PARTITION p02 VALUES LESS THAN ('2008-01-01'), PARTITION p03 VALUES LESS THAN ('2009-01-01'), PARTITION p04 VALUES LESS THAN (MAXVALUE)); 81 Wednesday, 13 January 2010
  82. 82. MySQL 5.5 enhancements E R FT SHOW CREATE TABLE t Table: t A 5.5 Create Table: CREATE TABLE `t` ( `dt` date DEFAULT NULL ) ENGINE=MyISAM DEFAULT CHARSET=latin1 /*!50500 PARTITION BY RANGE COLUMNS(dt) (PARTITION p01 VALUES LESS THAN ('2007-01-01') ENGINE = MyISAM, […] 82 Wednesday, 13 January 2010
  83. 83. MySQL 5.5 - Multiple columns CREATE TABLE t ( a int, b int )PARTITION BY RANGE COLUMNS (a,b) ( PARTITION p01 VALUES LESS THAN (10,1), PARTITION p02 VALUES LESS THAN (10,10), PARTITION p03 VALUES LESS THAN (10,20), PARTITION p04 VALUES LESS THAN (MAXVALUE, MAXVALUE)); 83 Wednesday, 13 January 2010
  84. 84. partition definition by range (a,b) records partition LESS THAN a b p01 10 10 1 10 p02 10 20 10 9 p03 10 30 10 10 p04 10 MAXVALUE 10 11 p05 MAXVALUE MAXVALUE Wednesday, 13 January 2010
  85. 85. partition definition by range (a,b) records partition LESS THAN a b p01 10 10 1 10 p02 10 20 10 9 p03 10 30 10 10 p04 10 MAXVALUE 10 11 p05 MAXVALUE MAXVALUE (1,10) < (10,10) ? (a < 10) OR ((a = 10) AND (b < 10)) (1 < 10) OR TRUE ((1 = 10) AND (10 < 10)) Wednesday, 13 January 2010
  86. 86. partition definition by range (a,b) records partition LESS THAN a b p01 10 10 1 10 p02 10 20 10 9 p03 10 30 10 10 p04 10 MAXVALUE 10 11 p05 MAXVALUE MAXVALUE (10,9) < (10,10) ? (a < 10) OR ((a = 10) AND (b < 10)) (10 < 10) OR TRUE ((10 = 10) AND (9 < 10)) Wednesday, 13 January 2010
  87. 87. partition definition by range (a,b) records partition LESS THAN a b p01 10 10 1 10 p02 10 20 10 9 p03 10 30 10 10 p04 10 MAXVALUE 10 11 p05 MAXVALUE MAXVALUE (10,10) < (10,10) ? (a < 10) OR ((a = 10) AND (b < 10)) (10 < 10) OR FALSE ((10 = 10) AND (10 < 10)) Wednesday, 13 January 2010
  88. 88. partition definition by range (a,b) records partition LESS THAN a b p01 10 10 1 10 p02 10 20 10 9 p03 10 30 10 10 p04 10 MAXVALUE 10 11 p05 MAXVALUE MAXVALUE (10,10) < (10,20) ? (a < 10) OR ((a = 10) AND (b < 20)) (10 < 10) OR TRUE ((10 = 10) AND (10 < 20)) Wednesday, 13 January 2010
  89. 89. CREATE TABLE employees ( emp_no int(11) NOT NULL, birth_date date NOT NULL, first_name varchar(14) NOT NULL, last_name varchar(16) NOT NULL, gender char(1) DEFAULT NULL, hire_date date NOT NULL ) ENGINE=MyISAM PARTITION BY RANGE COLUMNS(gender,hire_date) (PARTITION p01 VALUES LESS THAN ('F','1990-01-01'), PARTITION p02 VALUES LESS THAN ('F','2000-01-01'), PARTITION p03 VALUES LESS THAN ('F',MAXVALUE), PARTITION p04 VALUES LESS THAN ('M','1990-01-01'), PARTITION p05 VALUES LESS THAN ('M','2000-01-01'), PARTITION p06 VALUES LESS THAN ('M',MAXVALUE), PARTITION p07 VALUES LESS THAN (MAXVALUE,MAXVALUE) 89 Wednesday, 13 January 2010
  90. 90. MySQL 5.5 enhancements • TRUNCATE PARTITION • TO_SECONDS() 90 Wednesday, 13 January 2010
  91. 91. More new features COMING SOON 91 Wednesday, 13 January 2010
  92. 92. PLANNED FEATURES - DISCUSS • ALTER TABLE t1 EXCHANGE PARTITION p WITH TABLE t2 • SELECT * FROM T (PARTITION p) WHERE ... 92 Wednesday, 13 January 2010
  93. 93. Read more • articles http://dev.mysql.com • official docs http://dev.mysql.com/doc • my blog http://datacharmer.blogspot.com 93 Wednesday, 13 January 2010
  94. 94. Thank you! Giuseppe Maxia http://datacharmer.blogspot.com datacharmer@sun.com Wednesday, 13 January 2010
  95. 95. MySQL 5.1 partitions BONUS SLIDES 95 Wednesday, 13 January 2010
  96. 96. MySQL 5.1 partitions ANNOYANCES 96 Wednesday, 13 January 2010
  97. 97. Annoyances with partitions • CREATE TABLE STATEMENT hard to read 97 Wednesday, 13 January 2010
  98. 98. Annoyances with partitions • hard to read (FIXED!! in 5.1.31) 98 Wednesday, 13 January 2010
  99. 99. Annoyances with partitions • CREATE TABLE only keeps the result of the expression for each partition. • (you can use the information_schema to ease the pain in this case) 99 Wednesday, 13 January 2010
  100. 100. Annoyances with partitions CREATE TABLE t1 ( d date ) ENGINE=InnoDB PARTITION BY RANGE (to_days(d)) ( PARTITION P1 VALUES LESS THAN (to_days('1999-01-01')), PARTITION P2 VALUES LESS THAN (to_days('2000-01-01')) 100 ) Wednesday, 13 January 2010
  101. 101. Annoyances with partitions CREATE TABLE `t1` ( `d` date DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1 /*!50100 PARTITION BY RANGE (to_days (d)) (PARTITION P1 VALUES LESS THAN (730120) ENGINE = InnoDB, PARTITION P2 VALUES LESS THAN (730485) ENGINE = InnoDB) */ 101 Wednesday, 13 January 2010
  102. 102. Fixing annoyances with partitions select partition_name part, partition_expression expr, partition_description val from information_schema.partitions where table_name='t1'; +------+------------+--------+ | part | expr | val | +------+------------+--------+ | P1 | to_days(d) | 730120 | | P2 | to_days(d) | 730485 | 102 +------+------------+--------+ Wednesday, 13 January 2010
  103. 103. fixing annoyances with partitions select partition_name part , partition_expression expr, from_days (partition_description) val from information_schema.partitions where table_name='t1'; +------+------------+------------+ | part | expr | val | +------+------------+------------+ | P1 | to_days(d) | 1999-01-01 | | P2 | to_days(d) | 2000-01-01 | +------+------------+------------+ 103 Wednesday, 13 January 2010
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×