15 protips for mysql users pfz

15 Pro-tips for MySQL Users PFCongress - 17 september 2011 Utrecht - Netherlands

Joshua Thijssen Senior Software Engineer @ Enrise Development in PHP, Python, Perl, C, Java.... Email: [email_address] Twitter: @jaytaph Blogs: http://www.adayinthelifeof.nl http://www.enrise.com/blog

15 MySQL Pro-tips
No “theoretical tips”, all taken from the field.
Starting simple - ending “complex”
What are we going to discuss?
QUESTIONS? RAISE YOUR HAND OR YELL LOUD

0) Use the correct MySQL version
Tip 0
MySQL 5.0 != 5.5

RHEL 5-7: 5.0.77
RHEL 6-1: 5.1.52
Debian (lenny) 5.0.51a
Debian (squeeze): 5.1.49
Debian (sid): 5.1.58
Tip 0: Use the correct MySQL version http://distrowatch.com/table.php?distribution=redhat http://distrowatch.com/table.php?distribution=debian

1) Know how to use explain (and profiler)
Tip 1
EXPLAIN IS YOUR BESTEST FRIEND

I will not show you how to use EXPLAIN.
Use EXPLAIN and EXPLAIN EXTENDED/ SHOW WARNINGS;
Tip 1: Know your EXPLAIN (1)

Tip 1: Know your EXPLAIN (2) mysql> desc varchartest;+----------+--------------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+----------+--------------+------+-----+---------+-------+| id | int(11) | NO | PRI | 0 | || name | varchar(255) | NO | MUL | NULL | || utf8name | varchar(255) | NO | MUL | NULL | |+----------+--------------+------+-----+---------+-------+3 rows in set (0.01 sec)

Tip 1: Know your EXPLAIN (4) mysql> SHOW WARNINGSg+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| Level | Code | Message |+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| Note | 1003 | select `pfz`.`varchartest`.ìd` AS ìd`,`pfz`.`varchartest`.`name` AS `name`,`pfz`.`varchartest`.ùtf8name` AS ùtf8name` from `pfz`.`varchartest` where (`pfz`.`varchartest`.`name` like 'joshua') |+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Tip 1: Know your PROFILER (1) mysql> SET profiling=1;mysql> SELECT * FROM table; mysql> SHOW PROFILE CPU, BLOCK IO; +--------------------------------+----------+----------+------------+--------------+---------------+| Status | Duration | CPU_user | CPU_system | Block_ops_in | Block_ops_out |+--------------------------------+----------+----------+------------+--------------+---------------+| starting | 0.000930 | 0.000000 | 0.000000 | 0 | 8 || checking query cache for query | 0.000547 | 0.000000 | 0.000000 | 0 | 0 || checking permissions | 0.000045 | 0.000000 | 0.000000 | 0 | 0 || Opening tables | 0.000142 | 0.000000 | 0.000000 | 0 | 0 || System lock | 0.000030 | 0.000000 | 0.000000 | 0 | 0 || Table lock | 0.000045 | 0.000000 | 0.000000 | 0 | 0 || init | 0.000207 | 0.000000 | 0.000000 | 0 | 0 || optimizing | 0.000007 | 0.000000 | 0.000000 | 0 | 0 || statistics | 0.000068 | 0.000000 | 0.000000 | 0 | 0 || preparing | 0.001393 | 0.004000 | 0.000000 | 0 | 0 || Creating tmp table | 0.001658 | 0.000000 | 0.000000 | 0 | 8 || executing | 0.000005 | 0.000000 | 0.000000 | 0 | 0 || Copying to tmp table | 0.000834 | 0.000000 | 0.004001 | 0 | 8 || Sorting result | 0.000179 | 0.000000 | 0.000000 | 0 | 0 || Sending data | 0.000089 | 0.000000 | 0.000000 | 0 | 0 || end | 0.000022 | 0.000000 | 0.000000 | 0 | 0 || removing tmp table | 0.000089 | 0.000000 | 0.000000 | 0 | 0 || end | 0.000005 | 0.000000 | 0.000000 | 0 | 0 || query end | 0.000004 | 0.000000 | 0.000000 | 0 | 0 || freeing items | 0.000844 | 0.000000 | 0.000000 | 0 | 0 || logging slow query | 0.000023 | 0.000000 | 0.000000 | 0 | 0 || logging slow query | 0.000060 | 0.000000 | 0.000000 | 0 | 8 || cleaning up | 0.000007 | 0.000000 | 0.000000 | 0 | 0 |+--------------------------------+----------+----------+------------+--------------+---------------+

2) Know the most basic my.cnf settings
Tip 2
THERE ARE ONLY A FEW “BASIC” ONES.

Tip 2: My.cnf settings (1) Know the most important ones: key_buffer_size, innodb_buffer_pool_size, sort_buffer_size, max_connections

Some settings work on global level, some per connection!
Know some quirks: (max_heap_table_size vs tmp_table_size, binlog-do-db, replicate-ignore-db etc)
Tip 2: My.cnf settings (2)

http://www.omh.cc/mycnf/
http://rackerhacker.com/mysqltuner/
http://www.day32.com/MySQL/
phpmyadmin
Tip 2: My.cnf settings (3)

3) Backup on table level
Tip 3
RESTORING JUST ONE TABLE CAN BE PAINFUL OTHERWISE

mysqldump can dump per database OR by table.
Simple scripts to scan/dump tables.
Easy restore for single table (or part of table)
Tip 3: Backup on table level (1)
COULD YOU RESTORE TABLE x? YES! YES I CAN!

4) Don’t use “SELECT *” when you only need one or two fields.
Tip 4
DON’T ASK WHAT YOU DON’T NEED

Much more data to be read from disk
Much more data will be send over, thus slower (blobs/texts)
Cannot use covering indices
Tip 4: Select * (1)
DON’T ASK WHAT YOU DON’T NEED

Tip 4: Select * (2) mysql> SHOW FULL COLUMNS FROM `covering`;+--------------+------------------+------------------+------+-----+---------+----------------+ | Field | Type | Collation | Null | Key | Default | Extra | +--------------+------------------+------------------+------+-----+---------+----------------+| id | int(10) unsigned | NULL | NO | PRI | NULL | auto_increment || email | varchar(255) | ascii_general_ci | NO | MUL | NULL | | | want_mailing | tinyint(1) | NULL | NO | MUL | NULL | | | extra_info | varchar(255) | ascii_general_ci | NO | MUL | NULL | |+--------------+------------------+------------------+------+-----+---------+----------------+3 rows in set (0.00 sec)mysql> SHOW INDEXES FROM `covering`;+----------+------------+------------+--------------+--------------+-----------+-------------+| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | +----------+------------+------------+--------------+--------------+-----------+-------------+| covering | 0 | PRIMARY | 1 | id | A | 3 || covering | 1 | idx_email | 1 | email | A | 3 || covering | 1 | idx_email | 2 | want_mailing | A | 3 || covering | 1 | idx_email2 | 1 | want_mailing | A | 1 || covering | 1 | idx_email2 | 2 | email | A | 3 |+----------+------------+------------+--------------+--------------+-----------+-------------+5 rows in set (0.01 sec)

5) Use triggers and stored procedures
Tip 5
ENFORCE CONSISTENCY

6 triggers per table (insert, update, delete, before and after the mutation)
3rd party tools (phpmyadmin etc) can also use the database without loosing data consistency.
Watch out with (phpmyadmin) table dumps!
Tip 5: Triggers and stored procedures (1)
ENFORCE CONSISTENCY

6) Don’t use FULLTEXT searches
Tip 6
THERE ARE MUCH BETTER SOLUTIONS

They only work for MyISAM tables.
Not compatible with other DB’s.
Slow (especially compared to Solr, Sphinx).
No extra features (faceted search, spell checking etc).
Tip 6: Don’t use FULLTEXT search (1)

7) Some wildcard searches (%item%) are bad for performance
Tip 7
IT LOOKS LIKE YOU NEED MORE ADVANCED SEARCH?

MySQL cannot use indexing!
Revert your data: search for ‘moc.esirne@%’ instead of ‘%@enrise.com’.
Use a better solution (solr, sphinx). You probably want it (no, really)
Tip 7: Wildcard searches (1)

8) Shard your volatile and non-volatile data.
Tip 8
MAKE CACHING AND LOCKING HAPPY AGAIN

Tip 8: Sharding (1) mysql> SHOW FULL COLUMNS FROM `pages`;+------------+------------------+-------------------+------+-----+-------------------+-------+| Field | Type | Collation | Null | Key | Default | Extra |+------------+------------------+-------------------+------+-----+-------------------+-------+| page_id | int(10) unsigned | NULL | NO | PRI | NULL | || created_dt | timestamp | NULL | NO | | CURRENT_TIMESTAMP | || creator_id | int(11) | NULL | NO | | NULL | || title | varchar(100) | latin1_swedish_ci | NO | | NULL | || contents | text | latin1_swedish_ci | NO | | NULL | || hit_count | int(11) | NULL | NO | | 0 | |+------------+------------------+-------------------+------+-----+-------------------+-------+6 rows in set (0.01 sec)

Remember: an update on a table will invalidate ALL caches referring to that table.
UPDATE pages SET hit_count = hit_count + 1;
Thus: page table will NEVER be cached.
Tip 8: Sharding (2)

Define hot data (volatile, changes often) and cold data (static, changes never or infrequently)
move to different tables
UPDATE page_stats SET hit_count = hit_count + 1;
Query cache is happy again
Tip 8: Sharding (3)

MySQL 5.1 and higher has got partitioning (but you should shard anyway).
Tip 8: Sharding (4)

9) Don’t use a large primary key for InnoDB tables.
Tip 9
PK’S ARE ON EVERY INDEX

InnoDB adds the primary key to EACH index.
No primary key given? It uses an internal 6(!)-byte key.
Tip 9: Large primary keys (1)

10) Don’t “SELECT COUNT(*) FROM TABLE” on InnoDB.
Tip 10
COUNT(*) => MYISAM = FAST
COUNT(*) => INNODB = SLOW

InnoDB implements MVCC (multi-version concurrency control).
COUNT(*) must be counted and is not fetched from metadata.
Tip 10: SELECT COUNT(*) (1)

What do you want to COUNT(*)?
Just for displaying purposes (there are X amount of pages): do you need the EXACT amount? (guess &| cache)
Tip 10: SELECT COUNT(*) (2)

11) Don’t rely on the VARCHAR()
Tip 11
IT ISN’T THAT VARIABLE AS YOU MIGHT THINK

Tip 11: VARCHAR() (1)
IT DOESN’T SOLVE YOUR PROBLEMS

DON’T WORRY ABOUT THE UTF-8, I’LL BASH THAT LATER
Tip 11: VARCHAR() (3) mysql> SHOW FULL COLUMNS FROM `varchartest`;+----------+--------------+-------------------+------+-----+---------+-------+| Field | Type | Collation | Null | Key | Default | Extra |+----------+--------------+-------------------+------+-----+---------+-------+| id | int(11) | NULL | NO | PRI | 0 | || name | varchar(255) | latin1_swedish_ci | NO | MUL | NULL | || utf8name | varchar(255) | utf8_general_ci | NO | MUL | NULL | |+----------+--------------+-------------------+------+-----+---------+-------+3 rows in set (0.01 sec)mysql> SHOW INDEXES FROM `varchartest`;+-------------+------------+--------------+--------------+-------------+-----------+-------------+| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality |+-------------+------------+--------------+--------------+-------------+-----------+-------------+| varchartest | 0 | PRIMARY | 1 | id | A | 3 || varchartest | 1 | idx_name | 1 | name | A | 3 || varchartest | 1 | idx_utf8name | 1 | utf8name | A | 3 |+-------------+------------+--------------+--------------+-------------+-----------+-------------+3 rows in set (0.00 sec)

12) UTF-8 is not the enemy, but it certainly isn’t your friend.
Tip 12
DON’T EXCHANGE ONE PROBLEM FOR ANOTHER

Dr Jeckyl and Mr Hyde
Solves all your multi-language problems!
But gives back performance issues.
Tip 12: UTF-8 (1)

Tip 12: UTF-8 (1) mysql> SHOW FULL COLUMNS FROM `varchartest`;+----------+--------------+-------------------+------+-----+---------+-------+| Field | Type | Collation | Null | Key | Default | Extra |+----------+--------------+-------------------+------+-----+---------+-------+| id | int(11) | NULL | NO | PRI | 0 | || name | varchar(255) | latin1_swedish_ci | NO | MUL | NULL | || utf8name | varchar(255) | utf8_general_ci | NO | MUL | NULL | |+----------+--------------+-------------------+------+-----+---------+-------+3 rows in set (0.01 sec)mysql> SHOW INDEXES FROM `varchartest`;+-------------+------------+--------------+--------------+-------------+-----------+-------------+| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality |+-------------+------------+--------------+--------------+-------------+-----------+-------------+| varchartest | 0 | PRIMARY | 1 | id | A | 3 || varchartest | 1 | idx_name | 1 | name | A | 3 || varchartest | 1 | idx_utf8name | 1 | utf8name | A | 3 |+-------------+------------+--------------+--------------+-------------+-----------+-------------+3 rows in set (0.00 sec)

ALL temporary buffers are allocated for worst-case scenario’s.
This means a varchar(255) in UTF-8 uses 255*3 + 2 = 767 bytes PER row, even if you have only 1 single char inside.
Tip 12: UTF-8 (1)

13) Know your cardinality & selectivity
Tip 13

Cardinality: the number of unique entries inside the index.
Selectivity: percentage of unique entries.
S(I) = cardinality / count * 100%
Tip 13: Cardinality & Selectivity (1)

with 10 records: 5/10 * 100% = 50%
with 1000 records: 75/1000 * 100 = 7.5%
with 10.000 records: 200/10000 * 100% = 2%
Tip 13: Cardinality & Selectivity (3) country_id (max +-200, but effectively +- 50, maybe less)

A selectivity < 30% ? Full table scan!
ANALYZE TABLE frequently.

Adding records changes your cardinality and thus selectivity.
Develop against a “real” dataset.

14) Non-deterministic functions do not go well with query caching
Tip 14
NOW(), RAND(), UUID(), CONNECTION_ID() ETC..

Tip 14: Query caching (1) mysql> show status like '%qcache%';+-------------------------+----------+| Variable_name | Value |+-------------------------+----------+| Qcache_free_blocks | 1 || Qcache_free_memory | 16768400 || Qcache_hits | 3860 || Qcache_inserts | 975 || Qcache_lowmem_prunes | 0 || Qcache_not_cached | 486 || Qcache_queries_in_cache | 0 || Qcache_total_blocks | 1 |+-------------------------+----------+8 rows in set (0.00 sec) mysql> select * from varchartest;+----+------+----------+| id | name | utf8name |+----+------+----------+| 1 | j | joshua || 2 | j | jeroen || 3 | d | david |+----+------+----------+3 rows in set (0.00 sec)mysql> show status like '%qcache%';+-------------------------+----------+| Variable_name | Value |+-------------------------+----------+| Qcache_free_blocks | 1 || Qcache_free_memory | 16766864 || Qcache_hits | 3860 || Qcache_inserts | 976 || Qcache_lowmem_prunes | 0 || Qcache_not_cached | 486 || Qcache_queries_in_cache | 1 || Qcache_total_blocks | 4 |+-------------------------+----------+8 rows in set (0.00 sec)

Tip 14: Query caching (2) mysql> select * from varchartest;+----+------+----------+| id | name | utf8name |+----+------+----------+| 1 | j | joshua || 2 | j | jeroen || 3 | d | david |+----+------+----------+3 rows in set (0.00 sec)mysql> show status like '%qcache%';+-------------------------+----------+| Variable_name | Value |+-------------------------+----------+| Qcache_free_blocks | 1 || Qcache_free_memory | 16766864 || Qcache_hits | 3861 || Qcache_inserts | 976 || Qcache_lowmem_prunes | 0 || Qcache_not_cached | 486 || Qcache_queries_in_cache | 1 || Qcache_total_blocks | 4 |+-------------------------+----------+8 rows in set (0.00 sec) mysql> select * from varchartest;+----+------+----------+| id | name | utf8name |+----+------+----------+| 1 | j | joshua || 2 | j | jeroen || 3 | d | david |+----+------+----------+3 rows in set (0.00 sec)mysql> show status like '%qcache%';+-------------------------+----------+| Variable_name | Value |+-------------------------+----------+| Qcache_free_blocks | 1 || Qcache_free_memory | 16766864 || Qcache_hits | 3860 || Qcache_inserts | 976 || Qcache_lowmem_prunes | 0 || Qcache_not_cached | 486 || Qcache_queries_in_cache | 1 || Qcache_total_blocks | 4 |+-------------------------+----------+8 rows in set (0.00 sec)

Tip 14: Query caching (3) mysql> select * from varchartest;+----+------+----------+| id | name | utf8name |+----+------+----------+| 1 | j | joshua || 2 | j | jeroen || 3 | d | david |+----+------+----------+3 rows in set (0.00 sec)mysql> show status like '%qcache%';+-------------------------+----------+| Variable_name | Value |+-------------------------+----------+| Qcache_free_blocks | 1 || Qcache_free_memory | 16766864 || Qcache_hits | 3861 || Qcache_inserts | 976 || Qcache_lowmem_prunes | 0 || Qcache_not_cached | 486 || Qcache_queries_in_cache | 1 || Qcache_total_blocks | 4 |+-------------------------+----------+8 rows in set (0.00 sec) mysql> select * from varchartest ORDER BY RAND();+----+------+----------+| id | name | utf8name |+----+------+----------+| 2 | j | jeroen || 1 | j | joshua || 3 | d | david |+----+------+----------+3 rows in set (0.05 sec)mysql> show status like '%qcache%';+-------------------------+----------+| Variable_name | Value |+-------------------------+----------+| Qcache_free_blocks | 1 || Qcache_free_memory | 16766864 || Qcache_hits | 3861 || Qcache_inserts | 976 || Qcache_lowmem_prunes | 0 || Qcache_not_cached | 487 || Qcache_queries_in_cache | 1 || Qcache_total_blocks | 4 |+-------------------------+----------+8 rows in set (0.00 sec)

SELECT * FROM table WHERE YEAR(created_dt) < YEAR(NOW()); vs SELECT * FROM table WHERE YEAR(created_dt) < ‘2010’; Tip 14: Query caching (4)

15) Certify yourself as a DBA and/or DBE.
Tip 15
AND GET SOME NICE TITLES WHILE YOU’RE AT IT...

Oracle Certified MySQL Associate
Oracle Certified Professional MySQL 5.0 Developer
Oracle Certified Professional MySQL 5.0 Database Administrator
Oracle Certified Expert, MySQL 5.1 Cluster Database Administrator
“ Old”, but still, get them all!
Tip 15: Certify yourself (1)
THEY ARE NOT VERY EASY EXAMS, BUT WELL WORTH IT

Let’s summarize
Know how to use explain.
Know the most basic my.cnf settings.
Backup on table level.
Don’t use “SELECT *” when you only need 1 or 2 fields.
Use triggers and stored procedures.
Don’t use FULLTEXT searches.
Wildcard searches (%item%) are bad for performance.
Shard your volatile and non-volatile data.
Don’t use a large Primary Key for InnoDB tables.Don’t “Select COUNT(*)” on InnoDB.
Don’t rely on the VARCHAR().
UTF-8 is not the enemy, but it certainly isn’t your friend.Know your cardinality & selectivity.Non-deterministic functions do not go well with query caching. Certify yourself as a DBA and/or DBE.

Any questions?
QUESTIONS?
http://farm1.static.flickr.com/73/163450213_18478d3aa6_d.jpg

Daycamp for developers http://www.enrise.com http://www.daycamp4developers.com /

THANK YOU FOR YOUR ATTENTION
Please rate my talk: http://joind.in/3662

15 protips for mysql users pfz

by Joshua Thijssen on Sep 17, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

15 protips for mysql users pfz — Presentation Transcript