SlideShare a Scribd company logo
1 of 72
Download to read offline
How to Design
                             Indexes, Really
                            Bill Karwin, Percona Inc.




Tuesday, December 4, 2012
It’s About Performance
       • What’s the most frequent recommendation in
                database performance audits?
       • What’s the easiest way to speed up SQL queries,
          without schema changes or code changes?
       • What’s a commonly neglected part of database
          development and maintenance?



                              indexes.
                                                www.percona.com
Tuesday, December 4, 2012
Indexing Mistakes


                                           www.percona.com
Tuesday, December 4, 2012
Index Shotgun


                “Indexes improve performance,
                    so I index every column.”




                                            www.percona.com
Tuesday, December 4, 2012
Index Shotgun
       • Many such indexes are never used.
       • Indexes occupy space on disk in memory buffers.
       • Each index needs to be modified during INSERT,
                UPDATE, DELETE.
       • Query optimizer considers indexes during every
                query, more indexes makes this work harder.




                                                   www.percona.com
Tuesday, December 4, 2012
Index Aversion


                       “Indexes have overhead,
                     so I never create any indexes.”




                                              www.percona.com
Tuesday, December 4, 2012
Index Aversion
       • The right indexes are crucial to speed up queries.
       • Most workloads are read-heavy, so the net benefit
           of indexes outweighs the overhead of updating.
       • Indexes are compact, so they are a more efficient
           use of buffer memory.




                                                www.percona.com
Tuesday, December 4, 2012
SQL Aversion


                            “I use a NoSQL database
                              because it just works.” *



                      (* after I create the right indexes.)

                                                     www.percona.com
Tuesday, December 4, 2012
SQL Aversion
       • Many non-relational databases require you to
                define indexes explicitly too.
       • The ANSI/ISO SQL standard doesn’t specify
          anything about indexes.
       • So if you want to use NoSQL, just use indexes! :-)




                                                 www.percona.com
Tuesday, December 4, 2012
Naive Questions


                             “Here’s my schema...

                            what indexes do I need?”



                                Copyright 2012 Gene Ontology Consortium http://www.geneontology.org/
                                                                                                       www.percona.com
Tuesday, December 4, 2012
Naive Questions
       • Index choice depends on...
              • What tables and columns you need to query
              • What JOINs you need to perform
              • What GROUP BY’s and ORDER BY’s you need
       • Index design is not implicit from table design.




                                                    www.percona.com
Tuesday, December 4, 2012
Lesson #1


              relational
                                         index
               schema
              design is
              based on
                              but       design is
                                        based on
                                        queries.
                data.
                                            www.percona.com
Tuesday, December 4, 2012
Where Are the Queries?

                              Slow query log
                             General query log
                             Application logging
                                  Tcpdump
                                 Binary log
                                Process list
                                                www.percona.com
Tuesday, December 4, 2012
Collecting Log of Queries
       • Enable slow-query log:
              mysql> SET GLOBAL slow_query_log = ON;
       • Include all queries, not just those that are slow:
              mysql> SET GLOBAL long_query_time = 0;
       • Percona Server can log more information:
              mysql> SET GLOBAL log_slow_verbosity =
               'full';




                                                  www.percona.com
Tuesday, December 4, 2012
Where to Start?



                                pt-query-digest



                    http://www.percona.com/doc/percona-toolkit/pt-query-digest.html

                                                                      www.percona.com
Tuesday, December 4, 2012
pt-query-digest
       • Analyzes your query log.
       • Reports the queries accounting for greatest load.
              • Slow queries.
              • Quick queries that run frequently.
       • Outputs a ranked list of top query patterns.
              $ pt-query-digest 
               /var/lib/mysql/mysql-slow.log 
               > ~/pqd.txt



                                                     www.percona.com
Tuesday, December 4, 2012
Ranked Queries
   # Profile
   # Rank Query ID             Response time     Calls   R/Call    Apdx V/M     Item
   # ==== ==================   ===============   =====   =======   ==== =====   =======
   #    1 0xA8D2BBDE7EBE7822   4932.2992 28.8%      78   63.2346   0.00 5.22    SELECT person_info
   #    2 0xFE25DAF5DBB71F49   4205.2160 24.6%     130   32.3478   0.00 3.47    SELECT title
   #    3 0x70DAC639802CA233   1299.6269 7.6%       14   92.8305   0.00 0.17    SELECT cast_info
   #    4 0xE336B880F4FEC4B8   1184.5101 6.9%      294    4.0289   0.36 2.29    SELECT cast_info
   #    5 0x60550B93960F1837    905.1648 5.3%       60   15.0861   0.05 1.33    SELECT name
   #    6 0xF46D5C09B4E0CA2F    777.2446 4.5%    16340    0.0476   1.00 0.17    SELECT char_name
   #    7 0x09FCFFF0E5BC929F    747.4346 4.4%      130    5.7495   0.53 7.69    SELECT name
   #    8 0x9433950BE12B9470    744.1755 4.4%    14368    0.0518   1.00 0.18    SELECT name
   #    9 0x4DC0E044996DA715    448.5637 2.6%      130    3.4505   0.65 8.31    SELECT title
   #   10 0x09FB72D72ED18E93    361.1904 2.1%       78    4.6306   0.28 1.89    SELECT cast_info title




                                                                                 www.percona.com
Tuesday, December 4, 2012
Slow Queries
   # Profile
   # Rank Query ID              Response time     Calls   R/Call    Apdx V/M     Item
   # ==== ==================    ===============   =====   =======   ==== =====   =======
   #    1 0xA8D2BBDE7EBE7822    4932.2992 28.8%      78   63.2346   0.00 5.22    SELECT person_info
   #    2 0xFE25DAF5DBB71F49    4205.2160 24.6%     130   32.3478   0.00 3.47    SELECT title
   #    3 0x70DAC639802CA233    1299.6269 7.6%       14   92.8305   0.00 0.17    SELECT cast_info
   #    4 0xE336B880F4FEC4B8    1184.5101 6.9%      294    4.0289   0.36 2.29    SELECT cast_info
   #    5 0x60550B93960F1837     905.1648 5.3%       60   15.0861   0.05 1.33    SELECT name
   #    6 0xF46D5C09B4E0CA2F     777.2446 4.5%    16340    0.0476   1.00 0.17    SELECT char_name
   #    7 0x09FCFFF0E5BC929F     747.4346 4.4%      130    5.7495   0.53 7.69    SELECT name
   #    8 0x9433950BE12B9470     744.1755 4.4%    14368    0.0518   1.00 0.18    SELECT name
   #    9 0x4DC0E044996DA715     448.5637 2.6%      130    3.4505   0.65 8.31    SELECT title
   #   10 0x09FB72D72ED18E93     361.1904 2.1%       78    4.6306   0.28 1.89    SELECT cast_info title



                       this query executed a few times,
                         but each takes over 1 minute


                                                                                  www.percona.com
Tuesday, December 4, 2012
Quick Queries
   # Profile
   # Rank Query ID             Response time     Calls   R/Call    Apdx V/M     Item
   # ==== ==================   ===============   =====   =======   ==== =====   =======
   #    1 0xA8D2BBDE7EBE7822   4932.2992 28.8%      78   63.2346   0.00 5.22    SELECT person_info
   #    2 0xFE25DAF5DBB71F49   4205.2160 24.6%     130   32.3478   0.00 3.47    SELECT title
   #    3 0x70DAC639802CA233   1299.6269 7.6%       14   92.8305   0.00 0.17    SELECT cast_info
   #    4 0xE336B880F4FEC4B8   1184.5101 6.9%      294    4.0289   0.36 2.29    SELECT cast_info
   #    5 0x60550B93960F1837    905.1648 5.3%       60   15.0861   0.05 1.33    SELECT name
   #    6 0xF46D5C09B4E0CA2F    777.2446 4.5%    16340    0.0476   1.00 0.17    SELECT char_name
   #    7 0x09FCFFF0E5BC929F    747.4346 4.4%      130    5.7495   0.53 7.69    SELECT name
   #    8 0x9433950BE12B9470    744.1755 4.4%    14368    0.0518   1.00 0.18    SELECT name
   #    9 0x4DC0E044996DA715    448.5637 2.6%      130    3.4505   0.65 8.31    SELECT title
   #   10 0x09FB72D72ED18E93    361.1904 2.1%       78    4.6306   0.28 1.89    SELECT cast_info title



                                             another query executed
                                          frequently, each call is quick

                                                                                 www.percona.com
Tuesday, December 4, 2012
Understanding
                            Indexes by Analogy



                                            www.percona.com
Tuesday, December 4, 2012
This is a Telephone Book

       • A telephone book is like an index sorted
          by last name first, first name last.
       • Also includes the person’s phone number.




                                  Thomas Siencki
                                  http://en.wikipedia.org/wiki/File:Telefonbog_ubt-1.JPG

                                                                                           www.percona.com
Tuesday, December 4, 2012
Implementation
       • In MySQL syntax, this would look like the
                following:
              CREATE INDEX phone_idx ON TelephoneBook
               (last_name, first_name, phone_number);




                                                www.percona.com
Tuesday, December 4, 2012
Simple Searches
       • Since the index is pre-sorted, it benefits queries
                for last name:
              ! SELECT * FROM TelephoneBook
                WHERE last_name = 'Smith';
       • By benefit, we mean that it helps to narrow down
                the search quickly, because the entries are
                already in sorted order.




                                                     www.percona.com
Tuesday, December 4, 2012
Compound Searches
       • The sort order benefits us further if we search on
                last name and first name together:
              ! SELECT * FROM TelephoneBook
                WHERE last_name = 'Smith'
                AND first_name = 'John';




                                                     www.percona.com
Tuesday, December 4, 2012
When the Index Fails
       • But if we search only on first_name, the sort order
                doesn’t help:
              ! SELECT * FROM TelephoneBook
                WHERE first_name = 'John';
       • The ‘John’ entries occur throughout the book, in
                an unpredictable distribution. Anyone’s first
                name can be ‘John’. We have to search every
                page of the book.




                                                    www.percona.com
Tuesday, December 4, 2012
Order Matters
       • From this, we conclude that the order we define
                the columns in an index matters.
       • An index with the columns declared in the
                opposite order would benefit a search for first
                name alone, but then it wouldn’t help a search
                for last name alone.
              CREATE INDEX phone_idx2 ON TelephoneBook
               (first_name, last_name, phone_number);




                                                      www.percona.com
Tuesday, December 4, 2012
Both Indexes Can Be Helpful
       • If both criteria use equality comparisons, then the
                order of columns shouldn’t matter.
       • Either index would benefit a query that searches
                for a specific value in both columns.
              ! SELECT * FROM TelephoneBook
                WHERE last_name = 'Smith'
                AND first_name = 'John';




                                                        www.percona.com
Tuesday, December 4, 2012
Range Comparisons
       • Searching for multiple names is another case.
       • An index still helps us to narrow the search even if
                we search for all names that match a pattern:
              ! SELECT * FROM TelephoneBook
                WHERE last_name LIKE 'S%';
       • The book groups all the names starting with ‘S’
                together, so the index can help us to avoid
                reading the whole book.



                                                      www.percona.com
Tuesday, December 4, 2012
Compound Range Comparisons
       • If we search by a range of last names, but a
                specific first name, then the first names are
                again distributed in an unpredictable way among
                the matching last names:
              ! SELECT * FROM TelephoneBook
                WHERE last_name LIKE 'S%'
                AND first_name = 'John';
       • The ‘John’ entries are not grouped together
                among all ‘S’ names, so we have to search
                manually every ‘S’ name to find the ones we
                want.

                                                    www.percona.com
Tuesday, December 4, 2012
Order Matters
       • From this we see that once we do a range
                comparison on a column in an index, any criteria
                we have for subsequent columns cannot benefit
                from the index.
              CREATE INDEX phone_idx ON TelephoneBook
               (last_name, first_name, phone_number);


   this column in the index         but then subsequent columns
   helps the range search               do not help searching



                                                     www.percona.com
Tuesday, December 4, 2012
Sorting by Index
       • Since the telephone book is pre-sorted, we can
                rely on this if we need to read through the
                entries in that order.
              ! SELECT * FROM TelephoneBook
                WHERE last_name = 'Smith'
                ORDER BY first_name;
       • There’s no extra work to do here; we can simply
                read the entries in the order they are naturally
                written in the book.
              • So the ORDER BY is a no-op.

                                                       www.percona.com
Tuesday, December 4, 2012
When the Index Can’t Help Sorting
       • Sorting by any other column is another story.
              ! SELECT * FROM TelephoneBook
                WHERE last_name = 'Smith'
                ORDER BY phone_number;
       • The book is already ordered by last name and
                then by first name, but not numerically by phone
                number.
       • To get the result sorted in the manner above, we
                could copy the ‘Smith’ entries onto sticky notes,
                then re-order the sticky notes by phone number
                manually.
                                                      www.percona.com
Tuesday, December 4, 2012
Order Matters
       • From this, we conclude that an index helps if we
                sort by the column immediately following the
                columns in the search criteria.
       • But if we need to sort by another column later in
                the index, or by a column that isn’t in the index
                at all, we have to do it the hard way.




                                                       www.percona.com
Tuesday, December 4, 2012
Index-Only Searches
       • Once we find the matching entries, we may need
                other fields of information:
              ! SELECT phone_number
                FROM TelephoneBook
                WHERE last_name = 'Smith'
                AND first_name = 'John';
       • Because the phone number is included in the
                index entry, we get it for free. The entry
                includes all three fields, even if our query didn’t
                search for the phone number explicitly.


                                                        www.percona.com
Tuesday, December 4, 2012
When the Index Can’t Help
       • If we need other information that isn’t included in
                the index, we may need to do extra work.
              ! SELECT business_hours
                FROM TelephoneBook
                WHERE last_name = 'Smith Plumbing';
       • In this case, we’d have to use the phone number
                of the business, and call them to ask their hours.
              • That’s the extra work.



                                                      www.percona.com
Tuesday, December 4, 2012
Columns Matter
       • From this, we conclude that putting columns in an
                index can be very important for certain queries,
                even if that query doesn’t use the columns for
                searching or sorting.
       • This is called a covering index.




                                                      www.percona.com
Tuesday, December 4, 2012
Lesson #2


                             rate your
                               indexes
                              using the
                            star system.
                                           www.percona.com
Tuesday, December 4, 2012
The Star System
       • Rate an index with respect to a
                given query by three “stars”
                which describe effectiveness.
       • This technique is covered in
                Relational Database Index
                Design and the Optimizers by
                Tapio Lahdenmäki and
                Michael Leach.
              • http://www.wiley.com/WileyCDA/WileyTitle/
                       productCd-0471719994.html




                                                            www.percona.com
Tuesday, December 4, 2012
The Star System
       ★ First star:
                    Rows referenced by your query are grouped
                    together in the index.
       ★ Second star:
          Rows referenced by your query are ordered in
          the index the way you want them.
       ★ Third star:
          The index contains all columns referenced by
          your query (covering index).


                                                     www.percona.com
Tuesday, December 4, 2012
Look at this Terrible Query
           mysql> EXPLAIN SELECT person_id,
             person_role_id FROM cast_info
             WHERE movie_id = 91280 AND role_id = 1
             ORDER BY nr_order ASCG

                      id:      1         scans whole table
             select_type:      SIMPLE
                   table:      cast_info    uses no index
                    type:      ALL
           possible_keys:      NULL
                                              scans a lot of rows
                     key:      NULL
                 key_len:      NULL            sorts the hard way
                     ref:      NULL
                    rows:      24008907
                   Extra:      Using where; Using filesort
                                                      www.percona.com
Tuesday, December 4, 2012
★ First Star
       ★ Pick all columns from equality predicates.
                    Define the index with these columns first.
              ! SELECT person_id, person_role_id
                FROM cast_info
                WHERE movie_id = 91280 AND role_id = 1
                ORDER BY nr_order ASC;

                                          equality predicates

              ALTER TABLE cast_info ADD INDEX
               (movie_id, role_id);

                                                         www.percona.com
Tuesday, December 4, 2012
★ Second Star
       ★ Add the column(s) in the GROUP BY or ORDER
                    BY clause, if the query has one.
              ! SELECT person_id, person_role_id
                FROM cast_info
                WHERE movie_id = 91280 AND role_id = 1
                ORDER BY nr_order ASC;

                                                 sorting column

              ALTER TABLE cast_info ADD INDEX
               (movie_id, role_id, nr_order);

                                                         www.percona.com
Tuesday, December 4, 2012
★ Third Star
       ★ Add any remaining columns referenced in the
                    SELECT list.
              ! SELECT person_id, person_role_id
                FROM cast_info
                WHERE movie_id = 91280 AND role_id = 1
                ORDER BY nr_order ASC;

                                   select-list columns

              ALTER TABLE cast_info ADD INDEX
               (movie_id, role_id, nr_order,
               person_id, person_role_id);
                                                         www.percona.com
Tuesday, December 4, 2012
Explain That
              mysql> EXPLAIN SELECT person_id,
               person_role_id FROM cast_info
               WHERE movie_id = 91280 AND role_id = 1
               ORDER BY nr_order ASCG
                                           ★1: effective index
                         id:   1
                select_type:   SIMPLE
                      table:   cast_info                ★2: no filesort
                       type:   ref
              possible_keys:   movie_id
                        key:   movie_id    ★3: covering index
                    key_len:   8
                        ref:   const,const
                       rows:   57
                      Extra:   Using where; Using index
                                                        www.percona.com
Tuesday, December 4, 2012
Complications
       • Can’t achieve First-Star indexes when:
              • WHERE clause has more complex expressions
                       including OR predicates (disjunction).
              ! SELECT * FROM TelephoneBook
                WHERE last_name = 'Smith'
                OR first_name = 'John'


                                    disjunction



                                                                www.percona.com
Tuesday, December 4, 2012
Complications
       • Can’t achieve Second-Star indexes when:
              • WHERE clause includes range predicates
                       (>, <, !=, NOT IN, BETWEEN, LIKE, etc.).
              • Query includes both GROUP BY and ORDER BY
                 referencing different columns.
              • Query must sort by multiple columns over different
                 tables (can’t make a single index span tables).
              • Query includes ORDER BY in a different order than
                       the order in which rows are accessed.



                                                               www.percona.com
Tuesday, December 4, 2012
Complications
       • Can’t achieve Third-Star indexes when:
              • Query requires more columns than the maximum of
                 16 columns per index.
              • Query requires BLOB, TEXT, or VARCHAR columns
                 longer than the maximum of 1000 bytes per index
                 (utf8 counts three bytes per character).




                                                      www.percona.com
Tuesday, December 4, 2012
Example Query




                                            www.percona.com
Tuesday, December 4, 2012
Example Query
       mysql> EXPLAIN SELECT STRAIGHT_JOIN n.*
         FROM char_name AS c
         INNER JOIN cast_info AS i
         ! ON c.id=i.person_role_id
         INNER JOIN name AS n
         ! ON i.person_id=n.id
         WHERE c.name = 'James Bond'
         ORDER BY n.nameG


                     char_name       cast_info     name




                                                 www.percona.com
Tuesday, December 4, 2012
Not Optimized
           ***************************   1. row ***************************
                      id: 1
             select_type: SIMPLE
                   table: c
                    type: ALL
           possible_keys: PRIMARY                        2.4 million × 21.7 million
                     key: NULL
                 key_len: NULL                           = 52 trillion comparisons
                     ref: NULL
                    rows: 2402968
                   Extra: Using where;   Using temporary; Using filesort
           ***************************   2. row ***************************
                      id: 1
             select_type: SIMPLE
                   table: i
                    type: index
           possible_keys: person_id
                     key: person_id
                 key_len: 9
                     ref: NULL
                    rows: 21676661
                   Extra: Using where;   Using index; Using join buffer
           ***************************   3. row ***************************
                      id: 1
             select_type: SIMPLE
                   table: n
                   type: eq_ref
                                                                              www.percona.com
Tuesday, December 4, 2012
Add Indexes
       ★ First-star index:
              mysql> ALTER TABLE char_name
               ADD INDEX n (name(20));
       ★ Third-star index:
              mysql> ALTER TABLE cast_info
               ADD INDEX pr_p (person_role_id, person_id);



                     char_name      cast_info     name




                                                www.percona.com
Tuesday, December 4, 2012
Optimized
           ***************************   1. row ***************************
                      id: 1
             select_type: SIMPLE
                   table: c
                    type: ref
           possible_keys: PRIMARY,n                       improved by a factor
                     key: n
                 key_len: 62                                 of 480 million!
                     ref: const
                    rows: 1
                   Extra: Using where;   Using temporary; Using filesort
           ***************************   2. row ***************************
                      id: 1
             select_type: SIMPLE
                   table: i
                    type: ref
           possible_keys: pr_p
                     key: pr_p
                 key_len: 5
                     ref: imdb.c.id
                    rows: 108383
                   Extra: Using where;   Using index
           ***************************   3. row ***************************
                      id: 1
             select_type: SIMPLE
                   table: n
                    type: eq_ref
                                                                              www.percona.com
Tuesday, December 4, 2012
Why Not Second Star?
       • Not possible to create a second-star index here.
       • The order of the join column isn’t necessarily the
                same as the order of the sort column:
       +---------+--------------------------+
       | id      | name                     |
       +---------+--------------------------+
       | 127817 | Connery, Sean             |
       | 201693 | Lazenby, George           |
       | 213877 | Moore, Roger              |
       | 228614 | Dalton, Timothy           |
       | 581060 | Brosnan, Pierce           |
       | 707448 | Craig, Daniel             |
       +---------+--------------------------+
                                                        www.percona.com
Tuesday, December 4, 2012
Lesson #3


                              tidy up
                             indexes
                            you don’t
                               need.
                                         www.percona.com
Tuesday, December 4, 2012
Indexes You Don’t Need?


                create                         drop
               indexes                       indexes
                 that
              help your
                                    and     that don’t
                                            help your
               queries                       queries
                                                www.percona.com
Tuesday, December 4, 2012
Redundant Indexes



                       pt-duplicate-key-checker



                      http://www.percona.com/doc/percona-toolkit/pt-duplicate-key-checker.html

                                                                               www.percona.com
Tuesday, December 4, 2012
Redundant Indexes
       mysql> create table foo (a int, b int,
         key (a), key (b), key (a,b));
       $ pt-duplicate-key-checker h=localhost
       # #############################################################
       # test.foo
       # #############################################################

       # a is a left-prefix of a_2
       # Key definitions:
       #   KEY `a` (`a`),
       #   KEY `a_2` (`a`,`b`)
       # Column types:
       #! `a` int(11) default null
       #! `b` int(11) default null
       # To remove this duplicate index, execute:
       ALTER TABLE `test`.`foo` DROP INDEX `a`;

                                                         www.percona.com
Tuesday, December 4, 2012
Unused Indexes



                                     pt-index-usage



                            http://www.percona.com/doc/percona-toolkit/pt-index-usage.html

                                                                                www.percona.com
Tuesday, December 4, 2012
Unused Indexes
       $ sudo pt-index-usage /var/lib/mysql/mysql-slow.log
       /var/lib/mysql/mysql-slow.log: 12% 01:07 remain
       /var/lib/mysql/mysql-slow.log: 25% 00:57 remain
       /var/lib/mysql/mysql-slow.log: 37% 00:49 remain
       /var/lib/mysql/mysql-slow.log: 49% 00:41 remain
       /var/lib/mysql/mysql-slow.log: 61% 00:31 remain
       /var/lib/mysql/mysql-slow.log: 73% 00:21 remain
       /var/lib/mysql/mysql-slow.log: 84% 00:12 remain
       /var/lib/mysql/mysql-slow.log: 97% 00:02 remain

       . . .
       ALTER TABLE `imdb`.`cast_info` DROP KEY `movie_id`;
         -- type:non-unique
       . . .



                                                  www.percona.com
Tuesday, December 4, 2012
Index Usage Statistics



                                Index Statistics
                              in Percona Server


                    http://www.percona.com/doc/percona-server/5.5/diagnostics/user_stats.html

                                                                              www.percona.com
Tuesday, December 4, 2012
Index Usage Statistics
       • INFORMATION_SCHEMA.INDEX_STATISTICS
       • Enable with:
              mysql> SET GLOBAL userstat = ON;
       • Statistics are cleared when you restart mysqld.
       • Negligible performance impact:
              • http://www.mysqlperformanceblog.com/2012/06/02/how-expensive-is-
                       user_statistics/




                                                                        www.percona.com
Tuesday, December 4, 2012
Index Usage Statistics
       • Find indexes that were never used:
       mysql> SELECT CONCAT(
         !    'ALTER TABLE `', s.table_schema, '`.`', s.table_name,
         !    ' DROP KEY `', s.index_name, '`;') AS ddl
         FROM INFORMATION_SCHEMA.STATISTICS AS s
         LEFT OUTER JOIN INFORMATION_SCHEMA.INDEX_STATISTICS AS i
         !    USING (table_schema, table_name, index_name)
         WHERE s.table_schema = 'imdb' AND i.rows_read IS NULL;
       +---------------------------------------------------------+
       | ddl                                                     |
       +---------------------------------------------------------+
       | ALTER TABLE `imdb`.`cast_info DROP KEY `movie_id`;      |
       | ALTER TABLE `imdb`.`cast_info DROP KEY `person_id`;     |
       | ALTER TABLE `imdb`.`comp_cast_type DROP KEY `kind`;     |
       | ALTER TABLE `imdb`.`company_type DROP KEY `kind`;       |
       . . .


                                                         www.percona.com
Tuesday, December 4, 2012
Lesson #4




                            make it a
                             habit.

                                        www.percona.com
Tuesday, December 4, 2012
Review Queries Regularly
       • Repeat the query analysis periodically:
              • As new application code introduces new queries.
              • As the volume of data grows, making trivial queries
                 more expensive.
              • As site traffic changes, making some queries more
                 frequent.




                                                        www.percona.com
Tuesday, December 4, 2012
Review Queries Regularly
       • Track query types you have reviewed before:
              • pt-query-digest --review
                       Saves query types to a table, so you can add notes
                       to them, mark them as reviewed, omit them from
                       future reports.
              • pt-query-digest --review-history
                       Saves query performance statistics to a table, so
                       you can analyze trends in the data over time.




                                                              www.percona.com
Tuesday, December 4, 2012
Review Indexes Regularly
       • Run pt-duplicate-key-checker periodically:
              • After schema changes.
              • You can run this on a development or test instance.




                                                        www.percona.com
Tuesday, December 4, 2012
Review Indexes Regularly
       • Run pt-index-usage or review
                INDEX_STATISTICS periodically:
              • On a regular schedule (e.g. weekly).
              • Especially after application code changes.




                                                        www.percona.com
Tuesday, December 4, 2012
Lessons Redux


                            identify   rate your
                            queries.    indexes.


                                       make it a
                            tidy up.
                                        habit.
                                               www.percona.com
Tuesday, December 4, 2012
Percona Training for MySQL




                               Senior Industry Experts
                            In-Person and Online Classes
                               Custom Onsite Training
                                http://percona.com/training


                                                              www.percona.com
Tuesday, December 4, 2012
Visit Our Sponsors

       diamond


       platinum


       exhibitors



       and the rest!


                                                 www.percona.com
Tuesday, December 4, 2012
SQL Antipatterns




                            http://www.pragprog.com/titles/bksqla/
                                                            www.percona.com
Tuesday, December 4, 2012
License and Copyright
                                Copyright 2012 Bill Karwin
                               www.slideshare.net/billkarwin

                              Released under a Creative Commons 3.0 License:
                               http://creativecommons.org/licenses/by-nc-nd/3.0/
                                 You are free to share - to copy, distribute and
                                transmit this work, under the following conditions:




                    Attribution.                  Noncommercial.           No Derivative Works.
               You must attribute this work   You may not use this work   You may not alter, transform,
                     to Bill Karwin.           for commercial purposes.     or build upon this work.


                                                                                    www.percona.com
Tuesday, December 4, 2012

More Related Content

What's hot

Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to RedisDvir Volk
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDBMongoDB
 
C* Summit 2013: The World's Next Top Data Model by Patrick McFadin
C* Summit 2013: The World's Next Top Data Model by Patrick McFadinC* Summit 2013: The World's Next Top Data Model by Patrick McFadin
C* Summit 2013: The World's Next Top Data Model by Patrick McFadinDataStax Academy
 
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQLTop 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQLJim Mlodgenski
 
MySQL Index Cookbook
MySQL Index CookbookMySQL Index Cookbook
MySQL Index CookbookMYXPLAIN
 
MongoDB Fundamentals
MongoDB FundamentalsMongoDB Fundamentals
MongoDB FundamentalsMongoDB
 
MySQL Performance Tuning: Top 10 Tips
MySQL Performance Tuning: Top 10 TipsMySQL Performance Tuning: Top 10 Tips
MySQL Performance Tuning: Top 10 TipsOSSCube
 
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Jaime Crespo
 
Data Quality With or Without Apache Spark and Its Ecosystem
Data Quality With or Without Apache Spark and Its EcosystemData Quality With or Without Apache Spark and Its Ecosystem
Data Quality With or Without Apache Spark and Its EcosystemDatabricks
 
Database Anti Patterns
Database Anti PatternsDatabase Anti Patterns
Database Anti PatternsRobert Treat
 
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...Databricks
 
Understanding SQL Trace, TKPROF and Execution Plan for beginners
Understanding SQL Trace, TKPROF and Execution Plan for beginnersUnderstanding SQL Trace, TKPROF and Execution Plan for beginners
Understanding SQL Trace, TKPROF and Execution Plan for beginnersCarlos Sierra
 
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...Aaron Shilo
 
Introduction to Apache ZooKeeper
Introduction to Apache ZooKeeperIntroduction to Apache ZooKeeper
Introduction to Apache ZooKeeperSaurav Haloi
 
Indexing with MongoDB
Indexing with MongoDBIndexing with MongoDB
Indexing with MongoDBMongoDB
 
MySQL: Indexing for Better Performance
MySQL: Indexing for Better PerformanceMySQL: Indexing for Better Performance
MySQL: Indexing for Better Performancejkeriaki
 
MongoDB presentation
MongoDB presentationMongoDB presentation
MongoDB presentationHyphen Call
 

What's hot (20)

Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDB
 
C* Summit 2013: The World's Next Top Data Model by Patrick McFadin
C* Summit 2013: The World's Next Top Data Model by Patrick McFadinC* Summit 2013: The World's Next Top Data Model by Patrick McFadin
C* Summit 2013: The World's Next Top Data Model by Patrick McFadin
 
MongoDB
MongoDBMongoDB
MongoDB
 
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQLTop 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
 
MySQL Index Cookbook
MySQL Index CookbookMySQL Index Cookbook
MySQL Index Cookbook
 
MongoDB Fundamentals
MongoDB FundamentalsMongoDB Fundamentals
MongoDB Fundamentals
 
MySQL Performance Tuning: Top 10 Tips
MySQL Performance Tuning: Top 10 TipsMySQL Performance Tuning: Top 10 Tips
MySQL Performance Tuning: Top 10 Tips
 
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
 
Data Quality With or Without Apache Spark and Its Ecosystem
Data Quality With or Without Apache Spark and Its EcosystemData Quality With or Without Apache Spark and Its Ecosystem
Data Quality With or Without Apache Spark and Its Ecosystem
 
Sql query patterns, optimized
Sql query patterns, optimizedSql query patterns, optimized
Sql query patterns, optimized
 
Database Anti Patterns
Database Anti PatternsDatabase Anti Patterns
Database Anti Patterns
 
Survey of Percona Toolkit
Survey of Percona ToolkitSurvey of Percona Toolkit
Survey of Percona Toolkit
 
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
 
Understanding SQL Trace, TKPROF and Execution Plan for beginners
Understanding SQL Trace, TKPROF and Execution Plan for beginnersUnderstanding SQL Trace, TKPROF and Execution Plan for beginners
Understanding SQL Trace, TKPROF and Execution Plan for beginners
 
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
 
Introduction to Apache ZooKeeper
Introduction to Apache ZooKeeperIntroduction to Apache ZooKeeper
Introduction to Apache ZooKeeper
 
Indexing with MongoDB
Indexing with MongoDBIndexing with MongoDB
Indexing with MongoDB
 
MySQL: Indexing for Better Performance
MySQL: Indexing for Better PerformanceMySQL: Indexing for Better Performance
MySQL: Indexing for Better Performance
 
MongoDB presentation
MongoDB presentationMongoDB presentation
MongoDB presentation
 

Similar to How to Design Indexes, Really

Tools and Techniques for Index Design
Tools and Techniques for Index DesignTools and Techniques for Index Design
Tools and Techniques for Index DesignMYXPLAIN
 
Scaling MySQL Strategies for Developers
Scaling MySQL Strategies for DevelopersScaling MySQL Strategies for Developers
Scaling MySQL Strategies for DevelopersJonathan Levin
 
Database Optimization (MySQL)
Database Optimization (MySQL)Database Optimization (MySQL)
Database Optimization (MySQL)Oleksii Prohonnyi
 
Profiling and Tuning a Web Application - The Dirty Details
Profiling and Tuning a Web Application - The Dirty DetailsProfiling and Tuning a Web Application - The Dirty Details
Profiling and Tuning a Web Application - The Dirty DetailsAchievers Tech
 
Self Service Analytics enabled by Data Virtualization from Denodo
Self Service Analytics enabled by Data Virtualization from DenodoSelf Service Analytics enabled by Data Virtualization from Denodo
Self Service Analytics enabled by Data Virtualization from DenodoDenodo
 
SQL Server Worst Practices - EN
SQL Server Worst Practices - ENSQL Server Worst Practices - EN
SQL Server Worst Practices - ENGianluca Sartori
 
Better front-end development in Atlassian plugins
Better front-end development in Atlassian pluginsBetter front-end development in Atlassian plugins
Better front-end development in Atlassian pluginsAtlassian
 
Lucene BootCamp
Lucene BootCampLucene BootCamp
Lucene BootCampGokulD
 
[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails App
[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails App[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails App
[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails AppSrijan Technologies
 
Unit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentUnit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentDiego Delon
 
Unit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentUnit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentDiego Delon
 
Unit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentUnit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentDiego Delon
 
Unit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentUnit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentDiego Delon
 
Unit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentUnit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentDiego Delon
 
Unit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentUnit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentDiego Delon
 
Who wants to be a DBA? Roles and Responsibilities
Who wants to be a DBA? Roles and ResponsibilitiesWho wants to be a DBA? Roles and Responsibilities
Who wants to be a DBA? Roles and ResponsibilitiesKevin Kline
 
Index the obvious and not so obvious
Index the obvious and not so obviousIndex the obvious and not so obvious
Index the obvious and not so obviousHarry Zheng
 
Web Page Test - Beyond the Basics
Web Page Test - Beyond the BasicsWeb Page Test - Beyond the Basics
Web Page Test - Beyond the BasicsAndy Davies
 

Similar to How to Design Indexes, Really (20)

Tools and Techniques for Index Design
Tools and Techniques for Index DesignTools and Techniques for Index Design
Tools and Techniques for Index Design
 
Understanding indices
Understanding indicesUnderstanding indices
Understanding indices
 
Scaling MySQL Strategies for Developers
Scaling MySQL Strategies for DevelopersScaling MySQL Strategies for Developers
Scaling MySQL Strategies for Developers
 
Database story by DevOps
Database story by DevOpsDatabase story by DevOps
Database story by DevOps
 
Database Optimization (MySQL)
Database Optimization (MySQL)Database Optimization (MySQL)
Database Optimization (MySQL)
 
Profiling and Tuning a Web Application - The Dirty Details
Profiling and Tuning a Web Application - The Dirty DetailsProfiling and Tuning a Web Application - The Dirty Details
Profiling and Tuning a Web Application - The Dirty Details
 
Self Service Analytics enabled by Data Virtualization from Denodo
Self Service Analytics enabled by Data Virtualization from DenodoSelf Service Analytics enabled by Data Virtualization from Denodo
Self Service Analytics enabled by Data Virtualization from Denodo
 
SQL Server Worst Practices - EN
SQL Server Worst Practices - ENSQL Server Worst Practices - EN
SQL Server Worst Practices - EN
 
Better front-end development in Atlassian plugins
Better front-end development in Atlassian pluginsBetter front-end development in Atlassian plugins
Better front-end development in Atlassian plugins
 
Lucene BootCamp
Lucene BootCampLucene BootCamp
Lucene BootCamp
 
[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails App
[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails App[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails App
[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails App
 
Unit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentUnit Test for ZF SlideShare Component
Unit Test for ZF SlideShare Component
 
Unit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentUnit Test for ZF SlideShare Component
Unit Test for ZF SlideShare Component
 
Unit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentUnit Test for ZF SlideShare Component
Unit Test for ZF SlideShare Component
 
Unit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentUnit Test for ZF SlideShare Component
Unit Test for ZF SlideShare Component
 
Unit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentUnit Test for ZF SlideShare Component
Unit Test for ZF SlideShare Component
 
Unit Test for ZF SlideShare Component
Unit Test for ZF SlideShare ComponentUnit Test for ZF SlideShare Component
Unit Test for ZF SlideShare Component
 
Who wants to be a DBA? Roles and Responsibilities
Who wants to be a DBA? Roles and ResponsibilitiesWho wants to be a DBA? Roles and Responsibilities
Who wants to be a DBA? Roles and Responsibilities
 
Index the obvious and not so obvious
Index the obvious and not so obviousIndex the obvious and not so obvious
Index the obvious and not so obvious
 
Web Page Test - Beyond the Basics
Web Page Test - Beyond the BasicsWeb Page Test - Beyond the Basics
Web Page Test - Beyond the Basics
 

More from Karwin Software Solutions LLC (14)

Recursive Query Throwdown
Recursive Query ThrowdownRecursive Query Throwdown
Recursive Query Throwdown
 
InnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick FiguresInnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick Figures
 
SQL Outer Joins for Fun and Profit
SQL Outer Joins for Fun and ProfitSQL Outer Joins for Fun and Profit
SQL Outer Joins for Fun and Profit
 
Extensible Data Modeling
Extensible Data ModelingExtensible Data Modeling
Extensible Data Modeling
 
Schemadoc
SchemadocSchemadoc
Schemadoc
 
Percona toolkit
Percona toolkitPercona toolkit
Percona toolkit
 
MySQL 5.5 Guide to InnoDB Status
MySQL 5.5 Guide to InnoDB StatusMySQL 5.5 Guide to InnoDB Status
MySQL 5.5 Guide to InnoDB Status
 
Requirements the Last Bottleneck
Requirements the Last BottleneckRequirements the Last Bottleneck
Requirements the Last Bottleneck
 
Mentor Your Indexes
Mentor Your IndexesMentor Your Indexes
Mentor Your Indexes
 
Models for hierarchical data
Models for hierarchical dataModels for hierarchical data
Models for hierarchical data
 
Sql Injection Myths and Fallacies
Sql Injection Myths and FallaciesSql Injection Myths and Fallacies
Sql Injection Myths and Fallacies
 
Full Text Search In PostgreSQL
Full Text Search In PostgreSQLFull Text Search In PostgreSQL
Full Text Search In PostgreSQL
 
Practical Object Oriented Models In Sql
Practical Object Oriented Models In SqlPractical Object Oriented Models In Sql
Practical Object Oriented Models In Sql
 
Sql Antipatterns Strike Back
Sql Antipatterns Strike BackSql Antipatterns Strike Back
Sql Antipatterns Strike Back
 

Recently uploaded

A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
Visualising and forecasting stocks using Dash
Visualising and forecasting stocks using DashVisualising and forecasting stocks using Dash
Visualising and forecasting stocks using Dashnarutouzumaki53779
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesExploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesSanjay Willie
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo Day
H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo DayH2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo Day
H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo DaySri Ambati
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 

Recently uploaded (20)

A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
Visualising and forecasting stocks using Dash
Visualising and forecasting stocks using DashVisualising and forecasting stocks using Dash
Visualising and forecasting stocks using Dash
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesExploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo Day
H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo DayH2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo Day
H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo Day
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 

How to Design Indexes, Really

  • 1. How to Design Indexes, Really Bill Karwin, Percona Inc. Tuesday, December 4, 2012
  • 2. It’s About Performance • What’s the most frequent recommendation in database performance audits? • What’s the easiest way to speed up SQL queries, without schema changes or code changes? • What’s a commonly neglected part of database development and maintenance? indexes. www.percona.com Tuesday, December 4, 2012
  • 3. Indexing Mistakes www.percona.com Tuesday, December 4, 2012
  • 4. Index Shotgun “Indexes improve performance, so I index every column.” www.percona.com Tuesday, December 4, 2012
  • 5. Index Shotgun • Many such indexes are never used. • Indexes occupy space on disk in memory buffers. • Each index needs to be modified during INSERT, UPDATE, DELETE. • Query optimizer considers indexes during every query, more indexes makes this work harder. www.percona.com Tuesday, December 4, 2012
  • 6. Index Aversion “Indexes have overhead, so I never create any indexes.” www.percona.com Tuesday, December 4, 2012
  • 7. Index Aversion • The right indexes are crucial to speed up queries. • Most workloads are read-heavy, so the net benefit of indexes outweighs the overhead of updating. • Indexes are compact, so they are a more efficient use of buffer memory. www.percona.com Tuesday, December 4, 2012
  • 8. SQL Aversion “I use a NoSQL database because it just works.” * (* after I create the right indexes.) www.percona.com Tuesday, December 4, 2012
  • 9. SQL Aversion • Many non-relational databases require you to define indexes explicitly too. • The ANSI/ISO SQL standard doesn’t specify anything about indexes. • So if you want to use NoSQL, just use indexes! :-) www.percona.com Tuesday, December 4, 2012
  • 10. Naive Questions “Here’s my schema... what indexes do I need?” Copyright 2012 Gene Ontology Consortium http://www.geneontology.org/ www.percona.com Tuesday, December 4, 2012
  • 11. Naive Questions • Index choice depends on... • What tables and columns you need to query • What JOINs you need to perform • What GROUP BY’s and ORDER BY’s you need • Index design is not implicit from table design. www.percona.com Tuesday, December 4, 2012
  • 12. Lesson #1 relational index schema design is based on but design is based on queries. data. www.percona.com Tuesday, December 4, 2012
  • 13. Where Are the Queries? Slow query log General query log Application logging Tcpdump Binary log Process list www.percona.com Tuesday, December 4, 2012
  • 14. Collecting Log of Queries • Enable slow-query log: mysql> SET GLOBAL slow_query_log = ON; • Include all queries, not just those that are slow: mysql> SET GLOBAL long_query_time = 0; • Percona Server can log more information: mysql> SET GLOBAL log_slow_verbosity = 'full'; www.percona.com Tuesday, December 4, 2012
  • 15. Where to Start? pt-query-digest http://www.percona.com/doc/percona-toolkit/pt-query-digest.html www.percona.com Tuesday, December 4, 2012
  • 16. pt-query-digest • Analyzes your query log. • Reports the queries accounting for greatest load. • Slow queries. • Quick queries that run frequently. • Outputs a ranked list of top query patterns. $ pt-query-digest /var/lib/mysql/mysql-slow.log > ~/pqd.txt www.percona.com Tuesday, December 4, 2012
  • 17. Ranked Queries # Profile # Rank Query ID Response time Calls R/Call Apdx V/M Item # ==== ================== =============== ===== ======= ==== ===== ======= # 1 0xA8D2BBDE7EBE7822 4932.2992 28.8% 78 63.2346 0.00 5.22 SELECT person_info # 2 0xFE25DAF5DBB71F49 4205.2160 24.6% 130 32.3478 0.00 3.47 SELECT title # 3 0x70DAC639802CA233 1299.6269 7.6% 14 92.8305 0.00 0.17 SELECT cast_info # 4 0xE336B880F4FEC4B8 1184.5101 6.9% 294 4.0289 0.36 2.29 SELECT cast_info # 5 0x60550B93960F1837 905.1648 5.3% 60 15.0861 0.05 1.33 SELECT name # 6 0xF46D5C09B4E0CA2F 777.2446 4.5% 16340 0.0476 1.00 0.17 SELECT char_name # 7 0x09FCFFF0E5BC929F 747.4346 4.4% 130 5.7495 0.53 7.69 SELECT name # 8 0x9433950BE12B9470 744.1755 4.4% 14368 0.0518 1.00 0.18 SELECT name # 9 0x4DC0E044996DA715 448.5637 2.6% 130 3.4505 0.65 8.31 SELECT title # 10 0x09FB72D72ED18E93 361.1904 2.1% 78 4.6306 0.28 1.89 SELECT cast_info title www.percona.com Tuesday, December 4, 2012
  • 18. Slow Queries # Profile # Rank Query ID Response time Calls R/Call Apdx V/M Item # ==== ================== =============== ===== ======= ==== ===== ======= # 1 0xA8D2BBDE7EBE7822 4932.2992 28.8% 78 63.2346 0.00 5.22 SELECT person_info # 2 0xFE25DAF5DBB71F49 4205.2160 24.6% 130 32.3478 0.00 3.47 SELECT title # 3 0x70DAC639802CA233 1299.6269 7.6% 14 92.8305 0.00 0.17 SELECT cast_info # 4 0xE336B880F4FEC4B8 1184.5101 6.9% 294 4.0289 0.36 2.29 SELECT cast_info # 5 0x60550B93960F1837 905.1648 5.3% 60 15.0861 0.05 1.33 SELECT name # 6 0xF46D5C09B4E0CA2F 777.2446 4.5% 16340 0.0476 1.00 0.17 SELECT char_name # 7 0x09FCFFF0E5BC929F 747.4346 4.4% 130 5.7495 0.53 7.69 SELECT name # 8 0x9433950BE12B9470 744.1755 4.4% 14368 0.0518 1.00 0.18 SELECT name # 9 0x4DC0E044996DA715 448.5637 2.6% 130 3.4505 0.65 8.31 SELECT title # 10 0x09FB72D72ED18E93 361.1904 2.1% 78 4.6306 0.28 1.89 SELECT cast_info title this query executed a few times, but each takes over 1 minute www.percona.com Tuesday, December 4, 2012
  • 19. Quick Queries # Profile # Rank Query ID Response time Calls R/Call Apdx V/M Item # ==== ================== =============== ===== ======= ==== ===== ======= # 1 0xA8D2BBDE7EBE7822 4932.2992 28.8% 78 63.2346 0.00 5.22 SELECT person_info # 2 0xFE25DAF5DBB71F49 4205.2160 24.6% 130 32.3478 0.00 3.47 SELECT title # 3 0x70DAC639802CA233 1299.6269 7.6% 14 92.8305 0.00 0.17 SELECT cast_info # 4 0xE336B880F4FEC4B8 1184.5101 6.9% 294 4.0289 0.36 2.29 SELECT cast_info # 5 0x60550B93960F1837 905.1648 5.3% 60 15.0861 0.05 1.33 SELECT name # 6 0xF46D5C09B4E0CA2F 777.2446 4.5% 16340 0.0476 1.00 0.17 SELECT char_name # 7 0x09FCFFF0E5BC929F 747.4346 4.4% 130 5.7495 0.53 7.69 SELECT name # 8 0x9433950BE12B9470 744.1755 4.4% 14368 0.0518 1.00 0.18 SELECT name # 9 0x4DC0E044996DA715 448.5637 2.6% 130 3.4505 0.65 8.31 SELECT title # 10 0x09FB72D72ED18E93 361.1904 2.1% 78 4.6306 0.28 1.89 SELECT cast_info title another query executed frequently, each call is quick www.percona.com Tuesday, December 4, 2012
  • 20. Understanding Indexes by Analogy www.percona.com Tuesday, December 4, 2012
  • 21. This is a Telephone Book • A telephone book is like an index sorted by last name first, first name last. • Also includes the person’s phone number. Thomas Siencki http://en.wikipedia.org/wiki/File:Telefonbog_ubt-1.JPG www.percona.com Tuesday, December 4, 2012
  • 22. Implementation • In MySQL syntax, this would look like the following: CREATE INDEX phone_idx ON TelephoneBook (last_name, first_name, phone_number); www.percona.com Tuesday, December 4, 2012
  • 23. Simple Searches • Since the index is pre-sorted, it benefits queries for last name: ! SELECT * FROM TelephoneBook WHERE last_name = 'Smith'; • By benefit, we mean that it helps to narrow down the search quickly, because the entries are already in sorted order. www.percona.com Tuesday, December 4, 2012
  • 24. Compound Searches • The sort order benefits us further if we search on last name and first name together: ! SELECT * FROM TelephoneBook WHERE last_name = 'Smith' AND first_name = 'John'; www.percona.com Tuesday, December 4, 2012
  • 25. When the Index Fails • But if we search only on first_name, the sort order doesn’t help: ! SELECT * FROM TelephoneBook WHERE first_name = 'John'; • The ‘John’ entries occur throughout the book, in an unpredictable distribution. Anyone’s first name can be ‘John’. We have to search every page of the book. www.percona.com Tuesday, December 4, 2012
  • 26. Order Matters • From this, we conclude that the order we define the columns in an index matters. • An index with the columns declared in the opposite order would benefit a search for first name alone, but then it wouldn’t help a search for last name alone. CREATE INDEX phone_idx2 ON TelephoneBook (first_name, last_name, phone_number); www.percona.com Tuesday, December 4, 2012
  • 27. Both Indexes Can Be Helpful • If both criteria use equality comparisons, then the order of columns shouldn’t matter. • Either index would benefit a query that searches for a specific value in both columns. ! SELECT * FROM TelephoneBook WHERE last_name = 'Smith' AND first_name = 'John'; www.percona.com Tuesday, December 4, 2012
  • 28. Range Comparisons • Searching for multiple names is another case. • An index still helps us to narrow the search even if we search for all names that match a pattern: ! SELECT * FROM TelephoneBook WHERE last_name LIKE 'S%'; • The book groups all the names starting with ‘S’ together, so the index can help us to avoid reading the whole book. www.percona.com Tuesday, December 4, 2012
  • 29. Compound Range Comparisons • If we search by a range of last names, but a specific first name, then the first names are again distributed in an unpredictable way among the matching last names: ! SELECT * FROM TelephoneBook WHERE last_name LIKE 'S%' AND first_name = 'John'; • The ‘John’ entries are not grouped together among all ‘S’ names, so we have to search manually every ‘S’ name to find the ones we want. www.percona.com Tuesday, December 4, 2012
  • 30. Order Matters • From this we see that once we do a range comparison on a column in an index, any criteria we have for subsequent columns cannot benefit from the index. CREATE INDEX phone_idx ON TelephoneBook (last_name, first_name, phone_number); this column in the index but then subsequent columns helps the range search do not help searching www.percona.com Tuesday, December 4, 2012
  • 31. Sorting by Index • Since the telephone book is pre-sorted, we can rely on this if we need to read through the entries in that order. ! SELECT * FROM TelephoneBook WHERE last_name = 'Smith' ORDER BY first_name; • There’s no extra work to do here; we can simply read the entries in the order they are naturally written in the book. • So the ORDER BY is a no-op. www.percona.com Tuesday, December 4, 2012
  • 32. When the Index Can’t Help Sorting • Sorting by any other column is another story. ! SELECT * FROM TelephoneBook WHERE last_name = 'Smith' ORDER BY phone_number; • The book is already ordered by last name and then by first name, but not numerically by phone number. • To get the result sorted in the manner above, we could copy the ‘Smith’ entries onto sticky notes, then re-order the sticky notes by phone number manually. www.percona.com Tuesday, December 4, 2012
  • 33. Order Matters • From this, we conclude that an index helps if we sort by the column immediately following the columns in the search criteria. • But if we need to sort by another column later in the index, or by a column that isn’t in the index at all, we have to do it the hard way. www.percona.com Tuesday, December 4, 2012
  • 34. Index-Only Searches • Once we find the matching entries, we may need other fields of information: ! SELECT phone_number FROM TelephoneBook WHERE last_name = 'Smith' AND first_name = 'John'; • Because the phone number is included in the index entry, we get it for free. The entry includes all three fields, even if our query didn’t search for the phone number explicitly. www.percona.com Tuesday, December 4, 2012
  • 35. When the Index Can’t Help • If we need other information that isn’t included in the index, we may need to do extra work. ! SELECT business_hours FROM TelephoneBook WHERE last_name = 'Smith Plumbing'; • In this case, we’d have to use the phone number of the business, and call them to ask their hours. • That’s the extra work. www.percona.com Tuesday, December 4, 2012
  • 36. Columns Matter • From this, we conclude that putting columns in an index can be very important for certain queries, even if that query doesn’t use the columns for searching or sorting. • This is called a covering index. www.percona.com Tuesday, December 4, 2012
  • 37. Lesson #2 rate your indexes using the star system. www.percona.com Tuesday, December 4, 2012
  • 38. The Star System • Rate an index with respect to a given query by three “stars” which describe effectiveness. • This technique is covered in Relational Database Index Design and the Optimizers by Tapio Lahdenmäki and Michael Leach. • http://www.wiley.com/WileyCDA/WileyTitle/ productCd-0471719994.html www.percona.com Tuesday, December 4, 2012
  • 39. The Star System ★ First star: Rows referenced by your query are grouped together in the index. ★ Second star: Rows referenced by your query are ordered in the index the way you want them. ★ Third star: The index contains all columns referenced by your query (covering index). www.percona.com Tuesday, December 4, 2012
  • 40. Look at this Terrible Query mysql> EXPLAIN SELECT person_id, person_role_id FROM cast_info WHERE movie_id = 91280 AND role_id = 1 ORDER BY nr_order ASCG id: 1 scans whole table select_type: SIMPLE table: cast_info uses no index type: ALL possible_keys: NULL scans a lot of rows key: NULL key_len: NULL sorts the hard way ref: NULL rows: 24008907 Extra: Using where; Using filesort www.percona.com Tuesday, December 4, 2012
  • 41. ★ First Star ★ Pick all columns from equality predicates. Define the index with these columns first. ! SELECT person_id, person_role_id FROM cast_info WHERE movie_id = 91280 AND role_id = 1 ORDER BY nr_order ASC; equality predicates ALTER TABLE cast_info ADD INDEX (movie_id, role_id); www.percona.com Tuesday, December 4, 2012
  • 42. ★ Second Star ★ Add the column(s) in the GROUP BY or ORDER BY clause, if the query has one. ! SELECT person_id, person_role_id FROM cast_info WHERE movie_id = 91280 AND role_id = 1 ORDER BY nr_order ASC; sorting column ALTER TABLE cast_info ADD INDEX (movie_id, role_id, nr_order); www.percona.com Tuesday, December 4, 2012
  • 43. ★ Third Star ★ Add any remaining columns referenced in the SELECT list. ! SELECT person_id, person_role_id FROM cast_info WHERE movie_id = 91280 AND role_id = 1 ORDER BY nr_order ASC; select-list columns ALTER TABLE cast_info ADD INDEX (movie_id, role_id, nr_order, person_id, person_role_id); www.percona.com Tuesday, December 4, 2012
  • 44. Explain That mysql> EXPLAIN SELECT person_id, person_role_id FROM cast_info WHERE movie_id = 91280 AND role_id = 1 ORDER BY nr_order ASCG ★1: effective index id: 1 select_type: SIMPLE table: cast_info ★2: no filesort type: ref possible_keys: movie_id key: movie_id ★3: covering index key_len: 8 ref: const,const rows: 57 Extra: Using where; Using index www.percona.com Tuesday, December 4, 2012
  • 45. Complications • Can’t achieve First-Star indexes when: • WHERE clause has more complex expressions including OR predicates (disjunction). ! SELECT * FROM TelephoneBook WHERE last_name = 'Smith' OR first_name = 'John' disjunction www.percona.com Tuesday, December 4, 2012
  • 46. Complications • Can’t achieve Second-Star indexes when: • WHERE clause includes range predicates (>, <, !=, NOT IN, BETWEEN, LIKE, etc.). • Query includes both GROUP BY and ORDER BY referencing different columns. • Query must sort by multiple columns over different tables (can’t make a single index span tables). • Query includes ORDER BY in a different order than the order in which rows are accessed. www.percona.com Tuesday, December 4, 2012
  • 47. Complications • Can’t achieve Third-Star indexes when: • Query requires more columns than the maximum of 16 columns per index. • Query requires BLOB, TEXT, or VARCHAR columns longer than the maximum of 1000 bytes per index (utf8 counts three bytes per character). www.percona.com Tuesday, December 4, 2012
  • 48. Example Query www.percona.com Tuesday, December 4, 2012
  • 49. Example Query mysql> EXPLAIN SELECT STRAIGHT_JOIN n.* FROM char_name AS c INNER JOIN cast_info AS i ! ON c.id=i.person_role_id INNER JOIN name AS n ! ON i.person_id=n.id WHERE c.name = 'James Bond' ORDER BY n.nameG char_name cast_info name www.percona.com Tuesday, December 4, 2012
  • 50. Not Optimized *************************** 1. row *************************** id: 1 select_type: SIMPLE table: c type: ALL possible_keys: PRIMARY 2.4 million × 21.7 million key: NULL key_len: NULL = 52 trillion comparisons ref: NULL rows: 2402968 Extra: Using where; Using temporary; Using filesort *************************** 2. row *************************** id: 1 select_type: SIMPLE table: i type: index possible_keys: person_id key: person_id key_len: 9 ref: NULL rows: 21676661 Extra: Using where; Using index; Using join buffer *************************** 3. row *************************** id: 1 select_type: SIMPLE table: n type: eq_ref www.percona.com Tuesday, December 4, 2012
  • 51. Add Indexes ★ First-star index: mysql> ALTER TABLE char_name ADD INDEX n (name(20)); ★ Third-star index: mysql> ALTER TABLE cast_info ADD INDEX pr_p (person_role_id, person_id); char_name cast_info name www.percona.com Tuesday, December 4, 2012
  • 52. Optimized *************************** 1. row *************************** id: 1 select_type: SIMPLE table: c type: ref possible_keys: PRIMARY,n improved by a factor key: n key_len: 62 of 480 million! ref: const rows: 1 Extra: Using where; Using temporary; Using filesort *************************** 2. row *************************** id: 1 select_type: SIMPLE table: i type: ref possible_keys: pr_p key: pr_p key_len: 5 ref: imdb.c.id rows: 108383 Extra: Using where; Using index *************************** 3. row *************************** id: 1 select_type: SIMPLE table: n type: eq_ref www.percona.com Tuesday, December 4, 2012
  • 53. Why Not Second Star? • Not possible to create a second-star index here. • The order of the join column isn’t necessarily the same as the order of the sort column: +---------+--------------------------+ | id | name | +---------+--------------------------+ | 127817 | Connery, Sean | | 201693 | Lazenby, George | | 213877 | Moore, Roger | | 228614 | Dalton, Timothy | | 581060 | Brosnan, Pierce | | 707448 | Craig, Daniel | +---------+--------------------------+ www.percona.com Tuesday, December 4, 2012
  • 54. Lesson #3 tidy up indexes you don’t need. www.percona.com Tuesday, December 4, 2012
  • 55. Indexes You Don’t Need? create drop indexes indexes that help your and that don’t help your queries queries www.percona.com Tuesday, December 4, 2012
  • 56. Redundant Indexes pt-duplicate-key-checker http://www.percona.com/doc/percona-toolkit/pt-duplicate-key-checker.html www.percona.com Tuesday, December 4, 2012
  • 57. Redundant Indexes mysql> create table foo (a int, b int, key (a), key (b), key (a,b)); $ pt-duplicate-key-checker h=localhost # ############################################################# # test.foo # ############################################################# # a is a left-prefix of a_2 # Key definitions: # KEY `a` (`a`), # KEY `a_2` (`a`,`b`) # Column types: #! `a` int(11) default null #! `b` int(11) default null # To remove this duplicate index, execute: ALTER TABLE `test`.`foo` DROP INDEX `a`; www.percona.com Tuesday, December 4, 2012
  • 58. Unused Indexes pt-index-usage http://www.percona.com/doc/percona-toolkit/pt-index-usage.html www.percona.com Tuesday, December 4, 2012
  • 59. Unused Indexes $ sudo pt-index-usage /var/lib/mysql/mysql-slow.log /var/lib/mysql/mysql-slow.log: 12% 01:07 remain /var/lib/mysql/mysql-slow.log: 25% 00:57 remain /var/lib/mysql/mysql-slow.log: 37% 00:49 remain /var/lib/mysql/mysql-slow.log: 49% 00:41 remain /var/lib/mysql/mysql-slow.log: 61% 00:31 remain /var/lib/mysql/mysql-slow.log: 73% 00:21 remain /var/lib/mysql/mysql-slow.log: 84% 00:12 remain /var/lib/mysql/mysql-slow.log: 97% 00:02 remain . . . ALTER TABLE `imdb`.`cast_info` DROP KEY `movie_id`; -- type:non-unique . . . www.percona.com Tuesday, December 4, 2012
  • 60. Index Usage Statistics Index Statistics in Percona Server http://www.percona.com/doc/percona-server/5.5/diagnostics/user_stats.html www.percona.com Tuesday, December 4, 2012
  • 61. Index Usage Statistics • INFORMATION_SCHEMA.INDEX_STATISTICS • Enable with: mysql> SET GLOBAL userstat = ON; • Statistics are cleared when you restart mysqld. • Negligible performance impact: • http://www.mysqlperformanceblog.com/2012/06/02/how-expensive-is- user_statistics/ www.percona.com Tuesday, December 4, 2012
  • 62. Index Usage Statistics • Find indexes that were never used: mysql> SELECT CONCAT( ! 'ALTER TABLE `', s.table_schema, '`.`', s.table_name, ! ' DROP KEY `', s.index_name, '`;') AS ddl FROM INFORMATION_SCHEMA.STATISTICS AS s LEFT OUTER JOIN INFORMATION_SCHEMA.INDEX_STATISTICS AS i ! USING (table_schema, table_name, index_name) WHERE s.table_schema = 'imdb' AND i.rows_read IS NULL; +---------------------------------------------------------+ | ddl | +---------------------------------------------------------+ | ALTER TABLE `imdb`.`cast_info DROP KEY `movie_id`; | | ALTER TABLE `imdb`.`cast_info DROP KEY `person_id`; | | ALTER TABLE `imdb`.`comp_cast_type DROP KEY `kind`; | | ALTER TABLE `imdb`.`company_type DROP KEY `kind`; | . . . www.percona.com Tuesday, December 4, 2012
  • 63. Lesson #4 make it a habit. www.percona.com Tuesday, December 4, 2012
  • 64. Review Queries Regularly • Repeat the query analysis periodically: • As new application code introduces new queries. • As the volume of data grows, making trivial queries more expensive. • As site traffic changes, making some queries more frequent. www.percona.com Tuesday, December 4, 2012
  • 65. Review Queries Regularly • Track query types you have reviewed before: • pt-query-digest --review Saves query types to a table, so you can add notes to them, mark them as reviewed, omit them from future reports. • pt-query-digest --review-history Saves query performance statistics to a table, so you can analyze trends in the data over time. www.percona.com Tuesday, December 4, 2012
  • 66. Review Indexes Regularly • Run pt-duplicate-key-checker periodically: • After schema changes. • You can run this on a development or test instance. www.percona.com Tuesday, December 4, 2012
  • 67. Review Indexes Regularly • Run pt-index-usage or review INDEX_STATISTICS periodically: • On a regular schedule (e.g. weekly). • Especially after application code changes. www.percona.com Tuesday, December 4, 2012
  • 68. Lessons Redux identify rate your queries. indexes. make it a tidy up. habit. www.percona.com Tuesday, December 4, 2012
  • 69. Percona Training for MySQL Senior Industry Experts In-Person and Online Classes Custom Onsite Training http://percona.com/training www.percona.com Tuesday, December 4, 2012
  • 70. Visit Our Sponsors diamond platinum exhibitors and the rest! www.percona.com Tuesday, December 4, 2012
  • 71. SQL Antipatterns http://www.pragprog.com/titles/bksqla/ www.percona.com Tuesday, December 4, 2012
  • 72. License and Copyright Copyright 2012 Bill Karwin www.slideshare.net/billkarwin Released under a Creative Commons 3.0 License: http://creativecommons.org/licenses/by-nc-nd/3.0/ You are free to share - to copy, distribute and transmit this work, under the following conditions: Attribution. Noncommercial. No Derivative Works. You must attribute this work You may not use this work You may not alter, transform, to Bill Karwin. for commercial purposes. or build upon this work. www.percona.com Tuesday, December 4, 2012