0
Partitioning in MySQL 5.1
                                      and onwards

                          Dr. Mikael Ronström...
Why Partitioning?


                   • For Large Tables:
                          •   ”Divide and Conquer”
            ...
Partitioned Tables

                     • Partitioned Tables are standard MySQL tables
                          – Can be...
Partition Management

                          •   Fast Deletion of entire parts of a table
                          •  ...
MySQL Partitioning Types

      • Range Partitioning

      • List Partitioning

      • Hash Partitioning

      • Compos...
Storage Engines
    • Partition applies to all storage engines
           –   MyISAM
           –   InnoDB
           –   ...
Key Partitioning


                   • Definition
                          – Same as Hash partitioning except that MySQL...
LINEAR HASH/KEY partitioning

           • The normal HASH/KEY uses a modulo function to
             spread records => Ev...
Partition Function
            • PARTITION BY RANGE(f(f1,f2,f3))
            • Partition function must always deliver inte...
Partition Options

                     MAX_ROWS
                     MIN_ROWS
                     NODEGROUP (only MySQL ...
INFORMATION SCHEMA for
                                Partitioning
      • Provides information of:
             – Partit...
EXPLAIN for queries using
                             Partitioned Tables

   • Explains which partitions will be actually...
Partition Management

   •    ALTER TABLE t1 DROP PARTITION p0;
   •    ALTER TABLE t1 ADD PARTITION (PARTITION p1);
   • ...
Drop RANGE/LIST Partition


ALTER TABLE t1 DROP PARTITION 2002




            2002              2003     2004            ...
Add RANGE/LIST Partition

          ALTER TABLE t1 ADD PARTITION (PARTITION 2006 VALUES IN (2006));




           2003   ...
ADD HASH/KEY partition(s)
                          ALTER TABLE t1 ADD PARTITION (PARTITION p4);

                        ...
ADD LINEAR HASH/KEY
                                         partition(s)
                          ALTER TABLE t1 ADD PAR...
COALESCE HASH/KEY partition(s)
                          ALTER TABLE t1 COALESCE PARTITION 1;




                        ...
COALESCE LINEAR HASH/KEY
                                 partition(s)

                           ALTER TABLE t1 COALESCE...
Using REORGANIZE partition to
                                 SPLIT a partition
                ALTER TABLE t1 REORGANIZE...
Using REORGANIZE partition to
                                 MERGE partitions
                ALTER TABLE t1 REORGANIZE ...
Using REORGANIZE partition to
                                 balance partitions
           ALTER TABLE t1 REORGANIZE PAR...
Using REORGANIZE partition to move
                   partitions to new disk device
              ALTER TABLE t1 REORGANIZ...
Using REBUILD partition to recreate
                             partition
                          ALTER TABLE t1 REBUIL...
Partition Pruning

   • Only scan needed partitions
         – Range optimisations for single-field function, YEAR(date) a...
Partition Pruning
  SELECT * FROM Cars                             Where PRICE > 9000


      8000-TSU564-1988    7000-SAG...
Partition Pruning
 Select *FROM Cars Where PRICE >9000
 and COLOR= Red
      8000-TSU564-1988    7000-SAG293-2004       12...
Dynamic Partitioning Pruning
   SELECT * FROM t1, t2 WHERE t2.a = t1.a;

   If t1 is inner loop it is possible to select o...
Partitioning and NULL values

   • NULL values are allowed in partitioning fields
   • If a partitioning function is evalu...
Partititioning Implementation

                                        TABLE t1 (Abstract table,
                         ...
index_read Algorithm
                                             Handler output

                                        ...
Insert in Partitioning Table



      YELLOW                 GREEN                                BLUE
                   ...
Updating Partition



      Yellow              Blue          Red                    Green




  Delete                   ...
Partitioning for MySQL Cluster
   • Default for MySQL Cluster:
         – All tables in MySQL Cluster are partitioned
    ...
Cluster: Partitioning Controls
                              Physical Distribution
          •    Accessing relevant parti...
Cluster: Partition by Key
        CREATE TABLE service(user int unsigned,
                                                ...
Cluster: Default Tables

   • ENGINE=NDB is translated to PARTITION BY KEY()
     ENGINE=NDB
         – PARTITION BY KEY()...
Backup and Restore for partitioned
                      tables in MySQL Cluster

   SCENARIO DESCRIPTION:
   • Backup tak...
Nodegroup maps for ndb_restore

   • Using ndb_restore it is possible to specify a mapping
     between node groups as spe...
Future Roadmap
     • Global Indexes
           – Unique indexes fully supported
          Foreign Key support
     •
    ...
Partitioning Limitations (5.1)

    • Partitioning Function must return integer result
    • All partitions must use the s...
More information on Partitioning

                          • Documentation
                            – http://dev.mysql...
Thank You!

                          Senior Software Architect
                            Dr. Mikael Ronström
          ...
Upcoming SlideShare
Loading in...5
×

Partitioning 20061205

3,553

Published on

partitioning in mysql

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,553
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
167
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Partitioning 20061205"

  1. 1. Partitioning in MySQL 5.1 and onwards Dr. Mikael Ronström, Senior Software Architect MySQL AB mikael@mysql.com Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 1
  2. 2. Why Partitioning? • For Large Tables: • ”Divide and Conquer” • Easier maintenance • Performance improvement for queries • Control data placement on disk devices • Control data placement on Cluster nodes • Preparatory step for Parallel Queries Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 2
  3. 3. Partitioned Tables • Partitioned Tables are standard MySQL tables – Can be part of all constructs where tables are used – Complex select queries, Stored Procedures, Triggers, Views • Easy to ALTER TABLE into partitioned table – Syntax from CREATE TABLE can also be used in ALTER TABLE • Easy to drop partitioning on a table – ALTER TABLE t1 REMOVE PARTITIONING; Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 3
  4. 4. Partition Management • Fast Deletion of entire parts of a table • Fast reorganisation of one part of a table • Fast reorganisation of partition boundaries • Fast adding of new partitions • Fast split/merge of partitions Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 4
  5. 5. MySQL Partitioning Types • Range Partitioning • List Partitioning • Hash Partitioning • Composite Partitioning Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 5
  6. 6. Storage Engines • Partition applies to all storage engines – MyISAM – InnoDB – Archive – NDB Cluster – Falcon • Exceptions – Merge • Don’t mix storage engines in one table for now (5.1 limitation) – Example: – Archive Engine for really old data (> 10 years) – MyISAM for medium old data (> 1 year) – InnoDB for current data Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 6
  7. 7. Key Partitioning • Definition – Same as Hash partitioning except that MySQL decides the hash function using the given fields • Benefits – Very good hash function – Tight integration with MySQL Cluster partitioning • Drawbacks – Same as for Hash partitioning Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 7
  8. 8. LINEAR HASH/KEY partitioning • The normal HASH/KEY uses a modulo function to spread records => Even distribution of data among partitions • Also leads to full rebuild of table for ADD/COALESCE Partition • LINEAR HASH/KEY partitions use an algorithm based on linear hashing which means that some partitions will have twice as much data as others if the number of partitions is not on the form 2 ^ n • LINEAR HASH/KEY only requires rebuild of a number of the partitions, thus faster partition management at the cost of a slight imbalance Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 8
  9. 9. Partition Function • PARTITION BY RANGE(f(f1,f2,f3)) • Partition function must always deliver integer result (5.1 limitation) • Partition function can contain any number of fields, but must contain at least one field • If primary key is defined, no fields outside of primary key is allowed in partition function • Partition function can contain a large variety of character functions, date functions and mathematical functions • If unique key is defined on a table, no fields outside of unique key is allowed in partition function (this limitation does not exist for tables in MySQL Cluster) Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 9
  10. 10. Partition Options MAX_ROWS MIN_ROWS NODEGROUP (only MySQL Cluster) DATA DIRECTORY (only MyISAM) INDEX DIRECTORY (only MyISAM) COMMENT Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 10
  11. 11. INFORMATION SCHEMA for Partitioning • Provides information of: – Partition names – Where partitions are stored – Information about partition types – Information about partition functions – Number of rows per partition – Average row length in partitions – Other attributes of partitions (timestamps, …) • Support SHOW CREATE TABLE • Support SHOW TABLE STATUS – Will be displayed as Create option PARTITIONED Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 11
  12. 12. EXPLAIN for queries using Partitioned Tables • Explains which partitions will be actually scanned in a query – Gives you everything that EXPLAIN provides – Plus a list of partition names used in the query • Use EXPLAIN PARTITIONS to understand: – How partition pruning affects the query – Which indexes are used • might be affected by partitioning – If there is a better partitioning strategy Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 12
  13. 13. Partition Management • ALTER TABLE t1 DROP PARTITION p0; • ALTER TABLE t1 ADD PARTITION (PARTITION p1); • ALTER TABLE t1 REORGANIZE PARTITION ….; • ALTER TABLE t1 COALESCE PARTITION 1; • ALTER TABLE t1 REBUILD PARTITION p0; • ALTER TABLE t1 OPTIMIZE PARTITION p0; • ALTER TABLE t1 CHECK PARTITION p0; • ALTER TABLE t1 ANALYZE PARTITION p0; • ALTER TABLE t1 REPAIR PARTITION p0; • ALTER TABLE t1 TRUNCATE PARTITION p0; Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 13
  14. 14. Drop RANGE/LIST Partition ALTER TABLE t1 DROP PARTITION 2002 2002 2003 2004 2005 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 14
  15. 15. Add RANGE/LIST Partition ALTER TABLE t1 ADD PARTITION (PARTITION 2006 VALUES IN (2006)); 2003 2004 2005 2006 2006 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 15
  16. 16. ADD HASH/KEY partition(s) ALTER TABLE t1 ADD PARTITION (PARTITION p4); Old Partitions p1 p2 p3 p1 p2 p3 p4 New Partitions Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 16
  17. 17. ADD LINEAR HASH/KEY partition(s) ALTER TABLE t1 ADD PARTITION (PARTITION p4); Old Partitions p1 p2 p3 p2 p4 New Partitions Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 17
  18. 18. COALESCE HASH/KEY partition(s) ALTER TABLE t1 COALESCE PARTITION 1; Old Partitions p1 p2 p3 p4 p1 p2 p3 New Partitions Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 18
  19. 19. COALESCE LINEAR HASH/KEY partition(s) ALTER TABLE t1 COALESCE PARTITION 1; Old Partitions p1 p3 p2 p4 New Partitions p2 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 19
  20. 20. Using REORGANIZE partition to SPLIT a partition ALTER TABLE t1 REORGANIZE PARTITION 2006_2007 INTO (PARTITION 2006 VALUES LESS THAN (2007), PARTITION 2007 VALUES LESS THAN (2008)); Before_2002 2002_2003 2004_2005 2006_2007 2007 2006 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 20
  21. 21. Using REORGANIZE partition to MERGE partitions ALTER TABLE t1 REORGANIZE PARTITION 2006,2007 INTO (PARTITION 2006_2007 VALUES LESS THAN (2008)); Before_2004 2004_2005 2006 2007 2006_2007 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 21
  22. 22. Using REORGANIZE partition to balance partitions ALTER TABLE t1 REORGANIZE PARTITION 2006_Q1,2006_Q2 INTO (PARTITION 2006_M1_4 VALUES LESS THAN (DATE(’2006-05-01’), PARTITION 2006_M_5_8 VALUES LESS THAN (DATE(’2006-09-01’)); Before_2004 2005 2006_Q1 2006-Q2 2006_M_5_8 2006_M_1_4 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 22
  23. 23. Using REORGANIZE partition to move partitions to new disk device ALTER TABLE t1 REORGANIZE PARTITION 2006 INTO (PARTITION 2006 DATA DIRECTORY ’/home/user/2006/data_file’ INDEX DIRECTORY ’/home/user/2006/index_file’ VALUES LESS THAN (2007)); Before_2004 2005 2006 2007 2006 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 23
  24. 24. Using REBUILD partition to recreate partition ALTER TABLE t1 REBUILD PARTITION 2006; Before_2004 2005 2006 2007 2006 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 24
  25. 25. Partition Pruning • Only scan needed partitions – Range optimisations for single-field function, YEAR(date) and DATE_TO_DAYS(date) functions (RANGE partitioning) – Range optimisations for all LIST/HASH/KEY partitions • Best use case: – Full table scans on non-indexed fields • Example: – CREATE TABLE t1 (a int, index(a)) PARTITION BY HASH (a) partitions 4; – CREATE TABLE t2 (a int, index(a)); Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 25
  26. 26. Partition Pruning SELECT * FROM Cars Where PRICE > 9000 8000-TSU564-1988 7000-SAG293-2004 12000-FBI007-2004 1000-YUK333-1981 11000-YRG213-2005 21100-GRO293-1956 34000-SIE568-2004 13000-KAR365-2001 8500-KHO297-2004 Results Price >9000 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 26
  27. 27. Partition Pruning Select *FROM Cars Where PRICE >9000 and COLOR= Red 8000-TSU564-1988 7000-SAG293-2004 12000-FBI007-2004 1000-YUK333-1981 11000-YRG213-2005 21100-GRO293-1956 34000-SIE568-2004 13000-KAR365-2001 8500-KHO297-2004 Results Price >9000 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 27
  28. 28. Dynamic Partitioning Pruning SELECT * FROM t1, t2 WHERE t2.a = t1.a; If t1 is inner loop it is possible to select only one partition in each of its scan (one scan per record in outer table t2). If t1 is outer loop it has to scan all partitions. Explanation: This works since there is an index on ‘a’ that contains all partitioning fields and this is bound for each scan in inner loop Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 28
  29. 29. Partitioning and NULL values • NULL values are allowed in partitioning fields • If a partitioning function is evaluated where one field is NULL the result of the partitioning will be the smallest integer • This also holds for VALUES IN (NULL) in List partitioned tables • VALUES LESS THAN (NULL) is not allowed • VALUES LESS THAN MAXVALUE will include all integers upto the biggest integer Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 29
  30. 30. Partititioning Implementation TABLE t1 (Abstract table, frm-file for t1 exists, Implemented by Partition handler) Partition p1 Handler Table (no frm for partitions) MyISAM/ InnoDB/ Federated/ ….. Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 30
  31. 31. index_read Algorithm Handler output Merge Sort Part Sorted Sorted Sorted Sorted Output stream Output stream Output stream Output stream From Partition 1 From Partition 2 From Partition 3 From Partition 4 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 31
  32. 32. Insert in Partitioning Table YELLOW GREEN BLUE RED Insert GOX 123 $ 3000 2001 YELLOW Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 32
  33. 33. Updating Partition Yellow Blue Red Green Delete Insert GOX 123 $ 3000 2001 Yellow GOX 123 $ 3000 2001 Green Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 33
  34. 34. Partitioning for MySQL Cluster • Default for MySQL Cluster: – All tables in MySQL Cluster are partitioned – Default for an ENGINE=NDB is PARTITION BY KEY() • User Defined Partitioning for MySQL Cluster – Supports same partitioning types as rest of MySQL (Beta using –new) – PARTITION BY KEY is fully integrated with NDB kernel – Partitions defined in MySQL mapped to partitions in NDB storage engine, thus no extra overhead for many partitions – Partitioning makes manual placement of partitions possible • NOTE: – No support for DROP PARTITION – Full table copy is always employed for Partition Management • Locks entire table (=> not on-line) • More memory resources used • More processing resources Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 34
  35. 35. Cluster: Partitioning Controls Physical Distribution • Accessing relevant partitions only (partition pruning) for a query optimizes communication costs • A table no longer has to be distributed on all nodes • Altering the table partitioning can change the physical distribution – Altering table partitioning is currently done as a copying ALTER TABLE • Node groups can be populated on-line – ALTER TABLE account ADD PARTITION (PARTITION P2 NODEGROUP 2) – With future support for adding nodes (node groups) on- line, tables can be re-partitioned to populate the added nodes Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 35
  36. 36. Cluster: Partition by Key CREATE TABLE service(user int unsigned, service varchar, parameter int) PRIMARY KEY (user, service) PARTITION BY KEY (user) ENGINE=NDBCLUSTER; • PARTITION BY KEY enables us to place all user records from all tables in the same node group – Better locality of access for transactions towards a particular user • Default number of partitions in MySQL Cluster == Number of nodes in cluster Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 36
  37. 37. Cluster: Default Tables • ENGINE=NDB is translated to PARTITION BY KEY() ENGINE=NDB – PARTITION BY KEY() means by fields in primary key • if no primary key exists it means by Hidden Key – a unique identifier generated by MySQL for tables without primary keys in MySQL Cluster – PARTITION BY KEY() can be used also for non-clustered tables • Only partition by primary key (no hidden key stuff) • Default number of partitions for non-clustered partitioned tables is 1 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 37
  38. 38. Backup and Restore for partitioned tables in MySQL Cluster SCENARIO DESCRIPTION: • Backup taken in cluster with 4 node groups • Restore performed in cluster with 2 node groups • Assume table manually partitioned to be in node group 0 and 3 • What can be done for the partition in node group 3? • For mysqldump and Cluster Replication the table must be created by the user before applying the user records/binlog Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 38
  39. 39. Nodegroup maps for ndb_restore • Using ndb_restore it is possible to specify a mapping between node groups as specified in backup and the node groups set up by the restore program. • Use parameter --ndb-nodegroup-map ’(3,1)(2,0)’ • It needs a list of mappings from old nodegroup to new nodegroup • An old nodegroup map can be mapped to more than one nodegroup, in that case mappings are done on a round robin basis for those partitions • Tables that haven’t explicitly set a nodegroup is not affected by this mapping Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 39
  40. 40. Future Roadmap • Global Indexes – Unique indexes fully supported Foreign Key support • Import/Export Partitions to/from other tables • Even more on-line changes • Optimised locking/opening of tables • – Better performance • Mix storage engines in one table – Example: – Archive Engine for really old data (> 10 years) – MyISAM for medium old data (> 1 year) – InnoDB for current data • Enable queries on ”broken” tables – Query even when a partition is not available Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 40
  41. 41. Partitioning Limitations (5.1) • Partitioning Function must return integer result • All partitions must use the same Storage Engine (with same options) • Use of a partitioned table => open all partitions ⇒Increases response time for partitioned tables with many partitions (execution overhead very little) • Foreign Keys support is not available for partitioned tables (on the roadmap for future release) Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 41
  42. 42. More information on Partitioning • Documentation – http://dev.mysql.com/doc/refman/5.1/en/partitioni ng.html • Blog – http://mikaelronstrom.blogspot.com • Forums – http://forums.mysql.com/list.php?106 Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 42
  43. 43. Thank You! Senior Software Architect Dr. Mikael Ronström MySQL AB mikael@mysql.com Copyright 2005 MySQL AB The World’s Most Popular Open Source Database 43
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×