• Like
  • Save
Quick Wins with Third Party Patches
Upcoming SlideShare
Loading in...5
×
 

Quick Wins with Third Party Patches

on

  • 2,746 views

An overview of some of the really easy to use enhancements that aren't in a standard MySQL 5.1 release.

An overview of some of the really easy to use enhancements that aren't in a standard MySQL 5.1 release.

Statistics

Views

Total Views
2,746
Views on SlideShare
2,743
Embed Views
3

Actions

Likes
1
Downloads
37
Comments
0

2 Embeds 3

http://www.slideshare.net 2
http://www.slashdocs.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Quick Wins with Third Party Patches Quick Wins with Third Party Patches Presentation Transcript

    • Quick Wins with Third Party Patches Morgan Tocker, firstname at percona dot com Consultant, Percona Inc.
    • Introduction • There are a lot of great enhancements available (by default) in a MySQL release. – Many of these are stable. – Many of these are disabled by default.
    • Introduction (cont.) • For some of the enhancements, the change is just that a constant becomes a configuration item. – Not all of these are documented well :( – When Facebook writes a patch, they care about their needs, not Average Joe’s Wordpress install.
    • Who’s writing the patches • Various individuals and companies. • They are (mostly) incremental improvements, and addressing their business needs. • We’re not really talking about a major fork, it’s more a delta.
    • How can you get the patches? • Biased answer: – Percona releases binaries for a limited number of platforms. see http://www.percona.com/docs/wiki/ release:start • Other answers: – OurDelta. See http://ourdelta.org/ – MariaDB. See http://askmonty.org/wiki/index.php/ MariaDB:Download – Compile your own.
    • On to the main event... • The aim of this talk is to show you how to retrofit some quick wins to your application performance. • For most of these wins, some involvement is required...
    • InnoDB (High Level) Log Files SELECT * FROM City WHERE CountryCode=ʼAUSʼ Tablespace Buffer Pool 7
    • InnoDB (High Level) Log Files SELECT * FROM City WHERE CountryCode=ʼAUSʼ Tablespace Buffer Pool 7
    • InnoDB (High Level) Log Files SELECT * FROM City WHERE CountryCode=ʼAUSʼ Tablespace Buffer Pool 7
    • InnoDB (High Level) Log Files SELECT * FROM City WHERE CountryCode=ʼAUSʼ Tablespace Buffer Pool 7
    • InnoDB (High Level) Log Files SELECT * FROM City WHERE CountryCode=ʼAUSʼ Tablespace Buffer Pool 7
    • InnoDB (High Level) Log Files SELECT * FROM City WHERE CountryCode=ʼAUSʼ Tablespace Buffer Pool 7
    • InnoDB (cont.) Log Files UPDATE City SET name = 'Morgansville' WHERE name = 'Brisbane' AND CountryCode='AUS' Tablespace Buffer Pool 8
    • InnoDB (cont.) Log Files UPDATE City SET name = 'Morgansville' WHERE name = 'Brisbane' AND CountryCode='AUS' Tablespace Buffer Pool 8
    • InnoDB (cont.) Log Files UPDATE City SET name = 'Morgansville' WHERE name = 'Brisbane' AND CountryCode='AUS' Tablespace Buffer Pool 8
    • InnoDB (cont.) Log Files UPDATE City SET name = 'Morgansville' WHERE name = 'Brisbane' AND CountryCode='AUS' Tablespace Buffer Pool 8
    • InnoDB (cont.) 01010 Log Files UPDATE City SET name = 'Morgansville' WHERE name = 'Brisbane' AND CountryCode='AUS' Tablespace Buffer Pool 8
    • InnoDB (cont.) 01010 Log Files UPDATE City SET name = 'Morgansville' WHERE name = 'Brisbane' AND CountryCode='AUS' Tablespace Buffer Pool 8
    • InnoDB (cont.) 01010 Log Files UPDATE City SET name = 'Morgansville' WHERE name = 'Brisbane' AND CountryCode='AUS' Tablespace Buffer Pool 8
    • InnoDB (cont.) 01010 Log Files UPDATE City SET name = 'Morgansville' WHERE name = 'Brisbane' AND CountryCode='AUS' Tablespace Buffer Pool 8
    • #1 - New InnoDB Configuration • innodb_io_capacity - Ability to set the number of IOPS that your hardware is capable of. Used by InnoDB for choosing how much work to do in background operations (old behaviour: fixed at 100 IOPS). • innodb_write_io_threads / innodb_read_io_threads - Ability to specify how many concurrent threads to use in background operations. For many powerful IO devices, you need more than 1 thread to extract all capacity. Patch authored by Google. Also available in InnoDB Plugin.
    • New InnoDB Config (cont.) • The background algorithms are never perfect: Source: http://www.mysqlperformanceblog.com/2008/11/13/adaptive-checkpointing/
    • New InnoDB Config (cont.) • innodb_adaptive_flushing / innodb_adaptive_checkpoint - Change InnoDB to get more aggressive at background work. Get more aggressive at flushing when reaching the end of a log file to prevent DoS.
    • Example Configuration: • Server with ~32G memory. Hardware RAID 10 (x8 15K RPM disks): – [mysqld] innodb_flush_method=O_DIRECT innodb_buffer_pool_size=26G innodb_io_capacity=600 innodb_write_io_threads=4 innodb_read_io_threads=8 innodb_adaptive_checkpoint=1 • Your mileage may vary!
    • #2 - Fast Crash Recovery ★ Crash recovery in InnoDB can be very slow. From MySQL BUG #29847: [28 Oct 2008 21:40] James Day One reported effect of this performance limitation is that a system with 24GB buffer pool size could only recover 10% after 2 hour. With a 4G buffer pool and innodb_flush_method=O_DIRECT removed the system recovered completely in 30 minutes. Partial workarounds. 1. During recovery, temporarily reduce innodb_buffer_pool_size to force InnoDB to flush pages from the flush list. A value of 4G is likely to be reasonable. 2. During recovery, temporarily remove O_DIRECT so that the operating system can cache changes during recovery. 13
    • Fast Recovery (cont.) ★ Is Single Threaded. ★ Depends on the size of --innodb_log_file_size ✦ It’s not linear - 256M can be a lot worse than 128M, and most values > 512M mean “take a vacation”. ★ Depends on the size of the updates in the log file, and how far behind dirty page flushing was at the time of crash. ✦ Seen in SHOW INNODB STATUS under “LOG”: Log sequence number 84 3000620880 Log flushed up to 84 3000611265 Last checkpoint at 84 2939889199 14
    • Fast Recovery (cont.) ★ In our own tests, was x10 improvement. ★ Not enabled by default - can be turned on with: ★ --innodb_fast_recovery=1 15
    • #3 - Slow Query Filtering • Let me tell a story first: – mysql> select * from employees WHERE birth_date BETWEEN '1960-01-01' AND '1960-03-31' ORDER by RAND(); 117138 rows in set (0.63 sec) – mysql> select * from employees WHERE birth_date BETWEEN '1960-01-01' AND '1960-03-31'; 117138 rows in set (0.25 sec)
    • First One: • mysql> EXPLAIN select * from employees WHERE birth_date BETWEEN '1960-01-01' AND '1960-03-31' ORDER by RAND()G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: employees type: range possible_keys: birth_date key: birth_date key_len: 3 ref: NULL rows: 11554 Extra: Using where; Using temporary; Using filesort 1 row in set (0.00 sec)
    • Second One • mysql> EXPLAIN select * from employees WHERE birth_date BETWEEN '1960-01-01' AND '1960-03-31'G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: employees type: range possible_keys: birth_date key: birth_date key_len: 3 ref: NULL rows: 11554 Extra: Using where 1 row in set (0.00 sec)
    • ... what do we see? • So the query that doesn’t have to sort records is slightly faster. No surprises here. • What about the difference in scalability between these two queries. – Astute audience members will realize this is not the same question at all, i.e. scalability != performance.
    • The empirical test • #!/bin/sh ITERATIONS=10 CONCURRENCY="1,2,4,8,16,32,64,128" mkdir -p results ./mysqlslap -q "select * from employees WHERE birth_date BETWEEN '1960-01-01' AND '1960-03-31' ORDER by RAND()" -- create-schema=employees -i $ITERATIONS -c $CONCURRENCY > results/ORDER_BY_rand.txt ./mysqlslap -q "select * from employees WHERE birth_date BETWEEN '1960-01-01' AND '1960-03-31'" --create- schema=employees -i $ITERATIONS -c $CONCURRENCY > results/ ORDER_BY_null.txt
    • Results 65.00 60.05 48.75 32.50 26.96 16.25 18.56 14.39 9.81 5.16 0 1.22 2.40 0.89 4.88 1.76 3.48 4.55 0.63 0.25 0.47 1 2 4 8 16 32 64 128 RAND() No Rand
    • Why is this? • The query that requires the sort hits a different bottleneck, the sort. – Sorts in memory cause a lot of CPU pressure. • Might not scale. – Temporary tables or sorts on disk cause IO pressure. • Certainly won’t scale.
    • Why is this (cont.)? • Sometimes it’s just as important to monitor “what queries take a long time” as “what may not work with concurrency”. • Examples of things that might be expensive and reduce concurrency are.....
    • Expensive Things... qc_miss The query was not found in the query cache. full_scan The query performed a full table scan. full_join The query performed a full join (a join without indexes). tmp_table The query created an implicit internal temporary table. tmp_table_on_disk The query's temporary table was stored on disk. filesort The query used a filesort. filesort_on_disk The filesort was performed on disk.
    • Introducing Log slow filter... • [mysqld] log_slow_filter=tmp_table_on_disk,filesort_on_disk qc_miss The query was not found in the query cache. full_scan The query performed a full table scan. full_join The query performed a full join (a join without indexes). tmp_table The query created an implicit internal temporary table. tmp_table_on_disk The query's temporary table was stored on disk. filesort The query used a filesort. filesort_on_disk The filesort was performed on disk.
    • Extra Feature #1 • log_slow_verbosity - You can also *get* more information written to the log: microtime Log queries with microsecond precision (mandatory). query_plan Log information about the query's execution plan (optional). innodb Log InnoDB statistics (optional). # User@Host: mailboxer[mailboxer] @ [192.168.10.165] # Thread_id: 11167745 Schema: board # QC_Hit: No Full_scan: No Full_join: No Tmp_table: Yes Disk_tmp_table: No # Filesort: Yes Disk_filesort: No Merge_passes: 0 # Query_time: 0.000659 Lock_time: 0.000070 Rows_sent: 0 Rows_examined: 30 Rows_affected: 0 Rows_read: 30 # InnoDB_IO_r_ops: 1 InnoDB_IO_r_bytes: 16384 InnoDB_IO_r_wait: 0.028487 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 5 select count(distinct author_id) from art87.article87 force index (forum_id) where forum_id = 240215 and thread_id = '710575'
    • Extra Feature #2 • It also allows you to set the long_query_time to microseconds in MySQL 5.0: – long_query_time = 100000 • If we think that it takes an average of up to 7* queries to generate a page, 1 second is too long. – MySQL finally fixed this in MySQL 5.1 * Source: Brian Aker somewhere.
    • More information • This patch was originally authored by Percona • More Information: http://www.percona.com/docs/wiki/patches:microslow_innodb
    • #4 - Index Statistics • What are the possible indexes? • mysql> EXPLAIN SELECT Name FROM Country WHERE continent = 'Asia' AND population > 5000000 ORDER BY NameG *************************** 1. row *************************** id: 1 select_type: SIMPLE table: Country type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 239 Extra: Using where; Using filesort 1 row in set (0.00 sec)
    • Answers • The following indexes work (to varying degrees): – (continent) – (continent, population) – (continent, population, name).
    • Answers (cont.) • The following answers are incorrect: – name. Using an index on name would avoid the sort, but it would actually be quicker to table scan. – population. Using an index on population is not very helpful, since it doesn’t filter enough rows (most countries have populations > 5M).
    • How would you tell... • .. if you added the wrong index? You know, given that indexes will hurt things like write performance.
    • The simple answer... • You can’t! – There is no way in MySQL to find “dead indexes”. • Other than drop all indexes and start adding again ;) – Did I mention index selection changes over time based on data distribution? • .. and it’s really annoying.
    • Introducing Index Statistics • SELECT DISTINCT s.TABLE_SCHEMA, s.TABLE_NAME, s.INDEX_NAME FROM information_schema.statistics `s` LEFT JOIN information_schema.index_statistics IS ON (s.TABLE_SCHEMA = IS.TABLE_SCHEMA AND s.TABLE_NAME=IS.TABLE_NAME AND s.INDEX_NAME=IS.INDEX_NAME) WHERE IS.TABLE_SCHEMA IS NULL; +--------------+---------------------------+-----------------+ | TABLE_SCHEMA | TABLE_NAME                | INDEX_NAME      | +--------------+---------------------------+-----------------+ | art100       | article100                | ext_key         | | art100       | article100                | site_id         | | art100       | article100                | hash            | | art100       | article100                | forum_id_2      | | art100       | article100                | published       | | art100       | article100                | inserted        | | art100       | article100                | site_id_2       | | art100       | author100                 | PRIMARY         | | art100       | author100                 | site_id         | ... +--------------+---------------------------+-----------------+ 1150 rows IN SET (1 min 44.23 sec) Source: http://www.mysqlperformanceblog.com/2008/09/12/unused-indexes-by- single-query/
    • Index statistics is... • A very simple statistics tool that increments counters as index rows are read. – Allows you to figure out which indexes should be dropped or tweaked. – Very little overhead.
    • More information • Actually comes as part of a patch for INDEX_STATISTICS, USER_STATISTICS, TABLE_STATISTICS. • Patch originally authored by Google. • More Information: http://www.percona.com/docs/wiki/ patches:userstatv2
    • #5 - Global long query time. • The rules for the slow query time are actually defined from when the connection starts. • This ones a small one, but a life saver for applications that use connection pooling ;) • See: http://www.percona.com/docs/wiki/ patches:use_global_long_query_time
    • #6 - Buffer Pool Dump/Restore • Exports the contents of the buffer pool (space_id, page_no) to a file called ‘ib_lru_dump’: – SELECT * FROM information_schema.XTRADB_ADMIN_COMMAND /*! XTRA_LRU_DUMP*/; • Restored with: – SELECT * FROM information_schema.XTRADB_ADMIN_COMMAND /*! XTRA_LRU_RESTORE*/;
    • Buffer Pool Dump/Restore (cont.) • [Tip] Can be changed to sequential IO reload with a small python script (lrusort.py). • For more information, see: – http://www.mysqlperformanceblog.com/2010/01/20/ xtradb-feature-save-restore-buffer-pool/
    • The End. • Many of the examples I use show up in some form on our blog: http://www.mysqlperformanceblog.com/ • Questions?