0
Covering Indexes:
Orders-of-Magnitude Improvements

         Bradley C. Kuszmaul
           Chief Architect




  Percona ...
A Performance Example
A fact table drawn from iiBench, except no indexes.
Like TPCH, the int values are essentially random...
The Query
A simple query:
mysql> select prod_id from facts
              where cust_id = 50000;
 prod id
     525
     654...
An Index Doubles Speed
mysql> alter table add index facts
             cust_idx(cust_id);
mysql> select prod_id from facts...
Covering Index
A covering index is an index in which all the
neccessary columns are part of the key.
mysql> alter table ad...
Covering Index Is 1300x Faster
mysql> alter table add index facts
             cust_prod_idx (cust_id, prod_id);
mysql> se...
Outline
For a database that doesn’t fit in main memory, how
can we predict performance?

   • The Disk Access Model (DAM).
...
The Disk-Access Model
                                        Disk

                                                      ...
Analysis for No Index
                                      xid cust id prod id
                                        1 ...
Analysis for Index
    cust idx                                   facts
    cust id                    xid                ...
Analysis With Covering Index
    cust prod idx                                     facts
    cust id prod id              ...
Which Indexes?
Problem: MySQL only allows 16 columns in an
index (32 in TokuDB).
Covering indexes speed up queries.
But wh...
TPC-H Q17
We’ve been trying to figure out how to make
TPC-H-like queries run faster, so we picked Q17,
which is one of the ...
Indexes Are Expensive (or Are They?)
The downside of maintaining indexes is that
insertions are more expensive.
Fractal Tr...
Other Materialization Ideas
Better tools are needed to help maintain other
interesting materializations for MySQL. For
exa...
iiBench on RAID10 and SSD
                     35000


                     30000


                     25000
    Inserti...
Upcoming SlideShare
Loading in...5
×

Covering Indexes Ordersof Magnitude Improvements

2,026

Published on

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,026
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
29
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Covering Indexes Ordersof Magnitude Improvements"

  1. 1. Covering Indexes: Orders-of-Magnitude Improvements Bradley C. Kuszmaul Chief Architect Percona Performance Conference 2009
  2. 2. A Performance Example A fact table drawn from iiBench, except no indexes. Like TPCH, the int values are essentially random. Create Table: CREATE TABLE ‘facts‘ ( ‘xid‘ int(11) NOT NULL AUTO_INCREMENT, ‘dateandtime‘ datetime DEFAULT NULL, ‘cashregisterid‘ int(11) NOT NULL, ‘cust_id‘ int(11) NOT NULL, ‘prod_id‘ int(11) NOT NULL, ‘price‘ float NOT NULL, PRIMARY KEY (‘xid‘), ) ENGINE=TOKUDB Populated with 1 billion rows. Bradley C. Kuszmaul Covering Indexes 2
  3. 3. The Query A simple query: mysql> select prod_id from facts where cust_id = 50000; prod id 525 654 704 . . 984 276 576 10014 rows in set (11 min 26.26 sec) Implemented via table scan. 14.6 rows/s. Bradley C. Kuszmaul Covering Indexes 3
  4. 4. An Index Doubles Speed mysql> alter table add index facts cust_idx(cust_id); mysql> select prod_id from facts where cust_id = 50000; prod id 525 654 704 . . 984 276 576 10014 rows in set (5 min 48.94 sec) Looks at 0.001% of the data and get 2x speedup. 29 rows/s. Bradley C. Kuszmaul Covering Indexes 4
  5. 5. Covering Index A covering index is an index in which all the neccessary columns are part of the key. mysql> alter table add index facts cust_prod_idx (cust_id, prod_id); Even if we have a key that happens to be unique, we can throw in some more keys (which doesn’t change the index order) to make it a covering index. Bradley C. Kuszmaul Covering Indexes 5
  6. 6. Covering Index Is 1300x Faster mysql> alter table add index facts cust_prod_idx (cust_id, prod_id); mysql> select prod_id from facts where cust_id = 50000; prod id 0 0 . . 999 999 10014 rows in set (0.26 sec) Note different row order. Bradley C. Kuszmaul Covering Indexes 6
  7. 7. Outline For a database that doesn’t fit in main memory, how can we predict performance? • The Disk Access Model (DAM). • Analysis using the DAM. • Which indexes should we maintain? • Another brief example: TPC-H. • Does SSD help? In this talk, I’ll describe a theoretical model for predicting performance, and show how to use it. Bradley C. Kuszmaul Covering Indexes 7
  8. 8. The Disk-Access Model Disk A theoretical model Main memory for understanding performance of data Processor structures on disk. B Memory is organized in blocks B . . . of size B. B Blocks are transferred between memory and disk. Count only the number of block transfers. Model can predict the performance of a query plan. Bradley C. Kuszmaul Covering Indexes 8
  9. 9. Analysis for No Index xid cust id prod id 1 42 501 Block 0 (size B) . . 6044 50000 525 . . Block 1 20480 50000 654 . . 109 rows Block 109/B − 2 44921 50000 704 . . 999703368 50000 984 . Block 109/B − 1 . 999850921 50000 276 . . Block 109/B 999923451 50000 576 . . 9 Table scan requires O(10 /B) transfers. Bradley C. Kuszmaul Covering Indexes 9
  10. 10. Analysis for Index cust idx facts cust id xid xid cust id prod id 42 1 1 42 501 . . . . 50000 6044 6044 50000 525 . 50000 20480 . 50000 44921 20480 50000 654 . . . . 50000 999703368 44921 50000 704 . 50000 999850921 . 50000 999923451 999703368 50000 984 . . . . 999850921 50000 276 . Fetch only 10014 rows, . 999923451 50000 576 but mostly in different . . blocks, so O(10014) memory transfers. Bradley C. Kuszmaul Covering Indexes 10
  11. 11. Analysis With Covering Index cust prod idx facts cust id prod id xid xid cust id prod id 42 501 1 1 42 501 . 50000 276 999850921 . 50000 525 6044 6044 50000 525 . 50000 576 999923451 . 50000 654 20480 20480 50000 654 . 50000 704 44921 . 50000 984 999703368 44921 50000 704 . . Answer directly out of 999703368 50000 984 . . the index: O(10014/B) 999850921 50000 276 transfers. . . (The prod ids are sorted 999923451 50000 576 for each customer.) . . Bradley C. Kuszmaul Covering Indexes 11
  12. 12. Which Indexes? Problem: MySQL only allows 16 columns in an index (32 in TokuDB). Covering indexes speed up queries. But which columns should we throw in? Solution: We defined clustering indexes. A clustering index is an index which includes all the columns in the index. mysql> alter table facts add clustering index cust cluster idx (customerid); Materializes a table, sorted in a different order, clustered on the index. A clustering index is a covering index for all queries. Bradley C. Kuszmaul Covering Indexes 12
  13. 13. TPC-H Q17 We’ve been trying to figure out how to make TPC-H-like queries run faster, so we picked Q17, which is one of the slowest in MySQL. Results: With a clustering index on (L_PARTNUM, L_QTY): Scale Standard Clustering SF10 (10GB) > 3600s 101s (> 36x speedup) SF100 (100GB) 680533s 773s (770x speedup) I’ll write more about this in blog tokuview.com. Bradley C. Kuszmaul Covering Indexes 13
  14. 14. Indexes Are Expensive (or Are They?) The downside of maintaining indexes is that insertions are more expensive. Fractal Tree indexes speed up insertions by orders of magnitude, however. 9 For B = 4096 rows and N = 10 rows, the number of memory transfers per operation is B-Trees Fractal Trees log N Point Query O ≥ 1 O(logB N) ≥ 1 log B Range Query O(S/B) O(S/B) log N log N Insertion O ≥1O = 0.007 log B B TokuDB, the Tokutek storage engine, implements Fractal Tree indexes. Bradley C. Kuszmaul Covering Indexes 14
  15. 15. Other Materialization Ideas Better tools are needed to help maintain other interesting materializations for MySQL. For example: • Denormalization: prejoining some columns. • Multidimensional indexing: often where clauses look like range queries on multiple columns. 38<=a and a<=42 and 90<=b and b<=99 Partions can provide a painful substitute for multidimensional indexing. Fractal Tree indexes can help solve these problems. Bradley C. Kuszmaul Covering Indexes 15
  16. 16. iiBench on RAID10 and SSD 35000 30000 25000 Insertion Rate TokuDB 20000 FusionIO X25E RAID10 15000 10000 InnoDB 5000 FusionIO X25-E RAID10 0 0 5e+07 1e+08 1.5e+08 Cummulative Insertions Percona measured TokuDB and InnoDB on iiBench on RAID 10 disks, Intel X25-E 32GB SSD, and FusionIO 160GB SSD. These SSDs provide surprisingly little performance. Bradley C. Kuszmaul Covering Indexes 16
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×