Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Reducing Risk When Upgrading MySQL

1,127 views

Published on

Upgrading MySQL databases do not come without risk. There is no guarantee that no problems will happen if you move to a new major MySQL version.

Should we just upgrade and rollback immediately if problems occur? But what if these problems only happen a few days after migrating to this new version?

You might have a database environment that is risk-adverse, where you really have to be sure that this new MySQL version will handle the workload properly.

Examples:

- Both MySQL 5.6 and 5.7 have a lot of changes in the MySQL Optimizer. It is expected that this improves performance of my queries, but is it really the case? What if there is a performance regression? How will this affect my database performance?
- Also, there are a lot of incompatible changes which are documented in the release notes, how do I know if I'm affected by this in my workload? It's a lot to read..
- Can I go immediately from MySQL 5.5 to 5.7 and skip MySQL 5.6 even though the MySQL documentation states that this is not supported?
- Many companies have staging environments, but is there a QA team and do they really test all functionality, under a similar workload?

This presentation will show you a process, using open source tools, of these types of migrations with a focus on assessing risk and fixing any problems you might run into prior to the migration.

This process can then be used for various changes:

- MySQL upgrades for major version upgrades
- Switching storage engines
- Changing hardware architecture

Additionally, we will describe ways to do the actual migration and rollback with the least amount of downtime.

Published in: Internet
  • Be the first to comment

Reducing Risk When Upgrading MySQL

  1. 1. Reducing Risk When Upgrading Your MySQL Environment   Kenny Gryp MySQL Practice Manager
  2. 2. My Experience as MySQL Consultant On Upgrading MySQL it's quite complex... Kenny Gryp MySQL Practice Manager
  3. 3. Table of Contents The O cial Documentation Make Your Own Documentation Potential Risks Establish Upgrade Method For A Single Server Rollback Scenario Testing Test Writes Test Individual Reads Workload Testing Establish (& Test) Migration Process Migration In Production (Rollback) Post-Migration Assessment 3 / 77
  4. 4. The O cial Documentation 4 / 77
  5. 5. Oracle's Recommended Process Backup your data Read all release notes and assess https://dev.mysql.com/doc/relnotes/mysql/5.7/en/ Read Changes Affecting Upgrades to MySQL 5.7 https://dev.mysql.com/doc/refman/5.7/en/upgrading-from- previous-series.html 5 / 77
  6. 6. Oracle's Recommended Process Upgrade Slaves First 6 / 77
  7. 7. Oracle's Recommended Process Upgrade Slaves First In-Place Upgrade: Clean shutdown (innodb_fast_shutdown=0) Run mysql_upgrade Logical Upgrade: mysqldump data Import data again Run mysql_upgrade to x mysql schema http://dev.mysql.com/doc/refman/5.7/en/upgrading.html 7 / 77
  8. 8. Oracle's Recommended Process (cont.) A Lot of Risk: No guarantee queries will execute the same No guarantee queries will be same speed or faster No guarantee all your queries will still work (new default stricter sql_mode) There is no o cial support to upgrade from <5.6 to 5.7 but we might actually be able to do that 8 / 77
  9. 9. do-it-yourself Documenting The Process 9 / 77
  10. 10. Documenting The Process PEBKAC: Human errors happen and create issues import data using wrong character set setting up replica using wrong binlog le/pos ... Document every step, we need to repeat it multiple times 10 / 77
  11. 11. making you afraid to upgrade by describing Potential Risks 11 / 77
  12. 12. Optimizer Changes Example: index_merge_intersection Often seen during migrations to MySQL 5.6 Affects environments with sub-optimal indexing Queries with c1='a' AND c2='b' when composite index (c1,c2) is missing Is often slower when selectivity with 1 of the 2 columns is bad (and it happens frequently) Result: a lot of queries were slower in new environment Need SELECT performance tests between versions https://www.percona.com/blog/2012/12/14/the-optimization-that-often-isnt-index-merge-intersection/ 12 / 77
  13. 13. New Defaults In MySQL 5.7 The new defaults in MySQL 5.7 make a lot of sense: More use of available features and performance enhancements out of the box More strictness with data/query validation New Reserved words Applications might not be ready for it. Drupal 7 - https://www.drupal.org/node/2545480 They will/might break the application more easily: sql_mode=ONLY_FULL_GROUP_BY, STRICT_TRANS_TABLES, NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER, NO_ENGINE_SUBSTITUTION innodb_strict_mode=1 Needs SELECT & DML query validity tests between versions 13 / 77
  14. 14. Other Changes in MySQL 5.7 Passwords that use the older pre-4.1 password hashing format is removed. 14 / 77
  15. 15. MySQL 5.0.37 +-------+ | 0 | +-------+ MySQL 5.0.45 +-------+ | 1 | +-------+ Minor Versions Also At Risk CREATE TABLE date (d DATE); INSERT INTO date VALUES ('2017-04-19'); SELECT COUNT(*) FROM date WHERE d < NOW()-INTERVAL 1 DAY; Seen with DELETE FROM date WHERE d < NOW()- INTERVAL 1 DAY in binlog_format=STATEMENT environments. Needs SELECT & DML query result tests between versions 15 / 77
  16. 16. Workload SYNC_BINLOG=1 in MySQL 5.7 Can impact certain environments, might not be noticed when looking at a single query InnoDB LRU Flushing changes require tuning for heavy workloads in 5.6 (innodb_lru_scan_depth) When switching to MySQL 8.0 with the new data dictionary ... Need to do Workload Testing between versions http://mysqlentomologist.blogspot.com/2015/10/fun-with-bugs-38-regression-bugs-in.html http://lefred.be/content/sync_binlog-1-in-5-7/ 16 / 77
  17. 17. How Do We Reduce All This Risk? 17 / 77
  18. 18. Testing! 18 / 77
  19. 19. establish Upgrade Method For A Single Server 19 / 77
  20. 20. Upgrade Method For A Single Server Follow MySQL documentation: http://dev.mysql.com/doc/refman/5.7/en/upgrading.html Ensure to document every command Restore from backup Or take a replica you can miss 20 / 77
  21. 21. Upgrade Method For A Single Server 21 / 77
  22. 22. Upgrade Method For A Single Server 22 / 77
  23. 23. Replication Consistency Testing Writes 23 / 77
  24. 24. Writes - Replication Consistency pt-table-checksum: validate consistency in a replication topology Identify problems caused by PEBKAC Ensure events replicate properly (binlog_format=STATEMENT) Upgrade a replica or add a replica which is using the modi ed version. Do it on production, will have no result in test/staging https://www.percona.com/doc/percona-toolkit/3.0/pt-table-checksum.html 24 / 77
  25. 25. Writes - Replicate Test Server 25 / 77
  26. 26. often left behind is Rollback Scenario Testing 26 / 77
  27. 27. Rollback Scenario Testing Possibility to fall back in case something went wrong during migration Can be done using replication, but has to be tested! 27 / 77
  28. 28. Writes - Rollback Testing 28 / 77
  29. 29. Rollback Scenario Testing You might need to change some settings to your new my.cnf to be able to support replicating back. Example: binlog_checksum = NONE binlog_row_image = FULL binlog_rows_query_log_events = OFF log_bin_use_v1_row_events = 1 gtid_mode = OFF log_slave_updates=1 skip-slave-start 29 / 77
  30. 30. Writes - Checksums - GTID 30 / 77
  31. 31. Writes - Checksums - Non-GTID 31 / 77
  32. 32. Writes - Checkums - ROW 32 / 77
  33. 33. Writes - Checkums - ROW 33 / 77
  34. 34. Where To Run pt-table-checksum? GTID: pt-table-checksum can only be run on Master (Errant Transactions) Or scratch the pt-table-checksum host after tests non-GTID: pt-table-checksum can be run on intermediate master binlog_format=ROW: only 1 tier below can be checksummed run on every tier that has a replica (for rollback) pt-table-checksum can bring prod overhead when run on active master Let replication run for a while before checksumming 34 / 77
  35. 35. pt-table-checksum results On every replica (including rollback): SELECT db, tbl, SUM(this_cnt) AS total_rows, COUNT(*) AS chunks FROM percona.checksum WHERE (master_cnt <> this_cnt OR master_crc <> this_crc OR ISNULL(master_crc) <> ISNULL(this_crc)) GROUP BY db, tbl; +----+-----------------+------------+--------+ | db | tbl | total_rows | chunks | +----+-----------------+------------+--------+ | db | telephone_debit | 44342 | 1 | | db | orderline | 21451 | 3 | | db | orders | 25125215 | 12 | +----+-----------------+------------+--------+ 35 / 77
  36. 36. pt-table-checksum - Analysis Troubleshooting starts now... What went wrong? 36 / 77
  37. 37. pt-table-checksum - Analysis Which chunks failed? db: db tbl: telephone_debit chunk: 100 chunk_time: 0.4956125 chunk_index: PRIMARY lower_boundary: 5014733 upper_boundary: 5059074 this_crc: 7fd37eb9 this_cnt: 44342 master_crc: b7babd94 master_cnt: 44342 ts: 2013-02-05 01:59:48 37 / 77
  38. 38. pt-table-checksum - Analysis Which chunks failed? db: db tbl: telephone_debit chunk: 100 chunk_time: 0.4956125 chunk_index: PRIMARY lower_boundary: 5014733 upper_boundary: 5059074 this_crc: 7fd37eb9 this_cnt: 44342 master_crc: b7babd94 master_cnt: 44342 ts: 2013-02-05 01:59:48 38 / 77
  39. 39. pt-table-checksum - Analysis SELECT * INTO outfile '/tmp/telephone_debit_mysql56' FROM db.telephone_debit WHERE id BETWEEN 5014733 AND 5059074; SELECT * INTO outfile '/tmp/telephone_debit_mysql57' FROM db.telephone_debit WHERE id BETWEEN 5014733 AND 5059074; # diff -u /tmp/telephone_debit_mysql5{6,7} 39 / 77
  40. 40. pt-table-checksum - Analysis SELECT * INTO outfile '/tmp/telephone_debit_mysql56' FROM db.telephone_debit WHERE id BETWEEN 5014733 AND 5059074; SELECT * INTO outfile '/tmp/telephone_debit_mysql57' FROM db.telephone_debit WHERE id BETWEEN 5014733 AND 5059074; # diff -u /tmp/telephone_debit_mysql5{6,7} Use twindb_table_compare! https://github.com/twindb/twindb_table_compare 40 / 77
  41. 41. pt-table-checksum - Analysis Wrong upgrade method backups wrong replication le/pos ... binlog_format=STATEMENT using (UUID()...) Common Seen Issues replicating older versions: Floating point differences: Storing currencies in a DOUBLE Temporal data types Invalid dates converted to zero dates Trailing spaces in CHAR elds 41 / 77
  42. 42. Testing Writes Consistency Checks Process: Checksum Check for differences On new environment On rollback environment For each inconsistency Analyze diff Find root cause Fix problem Document problem & solution Repeat checksum again 42 / 77
  43. 43. Testing Individual Reads 43 / 77
  44. 44. Testing Reads - Collect Queries 44 / 77
  45. 45. Testing Reads - Collect Queries Collection Techniques: Slow Query Log long_query_time=0 Careful when ~+10000 QPS Percona Server: log_slow_rate_limit tcpdump 'packets lost' in libpcap Application/Load Balancer queries Ensure: Get the full workload (long enough) Get data from Master & Replicas Collect batchjob queries running at night https://www.percona.com/doc/percona-server/5.7/diagnostics/slow_extended.html 45 / 77
  46. 46. Testing Reads - Setup 2 Environments 46 / 77
  47. 47. Testing Reads - Setup 2 Environments Need 2 Test Servers: Reuse servers from checksum + rollback Ensure they have the same data (break replication at same time) Same HW speci cations Similar Con gurations on buffer pool, flatc... Fast enough to more or less resemble production Optionally can be done using 1 machine (pt-upgrade --save-results) 47 / 77
  48. 48. Testing Reads - pt-upgrade 48 / 77
  49. 49. Testing Reads - pt-upgrade pt-upgrade: runs one query at a time on both test environments compares differences: warnings/errors resultset (even different order) query response time Run pt-upgrade on third host with similar network latency Run twice to warm up buffer pool rst (need to be equal) Can also compare writes for execution time & warnings Filter slowlog initially to limit similar queries pt-query-digest --no-report --output slowlog --samples 20 https://www.percona.com/doc/percona-toolkit/3.0/pt-upgrade.html 49 / 77
  50. 50. Testing Reads - pt-upgrade Reporting class because there are 1000 row diffs. Total queries 10 Unique queries 10 Discarded queries 0 select ... from ... ## ## Row diffs: 10 ## -- 1. @ row 2 < 13178,"dim0",37,2,21,,,0,0,0,1,NULL,NULL > 13178,"dimø",37,2,21,,,0,0,0,1,NULL,NULL ... 50 / 77
  51. 51. Testing Reads - pt-upgrade Reporting class because it has diffs, but hasn't been reported yet. SELECT * FROM `database`.table WHERE treeid = '' AND productid='0' ## Warning diffs: 2 Code: 1366 Level: Warning Message: Incorrect integer value: '' for column 'treeid' at row 1 vs. No warning 1366 51 / 77
  52. 52. Testing Reads - pt-upgrade SELECT * FROM `database`.client_orders WHERE client=? AND blacklist=? LIMIT ? ## Query time diffs: 1 -- 1. 0.000513 vs. 0.036395 seconds (70.9x increase) SELECT * FROM `database`.client_orders WHERE client=57450 AND blacklist=1 LIMIT 1 52 / 77
  53. 53. Testing Reads Process Collect queries Run pt-upgrade (twice) For each entry in report Figure out why it is reported Deploy x in Prod Application Make schema changes Document analysis Run pt-upgrade again 53 / 77
  54. 54. one of the most challenging is Testing Workload 54 / 77
  55. 55. Workload Testing - Percona Playback 55 / 77
  56. 56. Workload Testing - Query Playback Uses slowlog to replay queries Needs long_query_time=0 - challenging on busy servers Enough data during peak workload Tries to execute workload as realistically as possible same connections, same transactions, same delays between queries Run against both environments, compare speed Think about preloading buffer on both the same way Active development by Marius Wachtler (ex)-DropBox! Thank you! (uno cal product of Percona, no support) 56 / 77
  57. 57. Workload Testing - ProxySQL Mirroring 57 / 77
  58. 58. Workload Testing - ProxySQL Mirroring Mirror queries from Load Balancer to test environment Good Blogpost: https://www.pythian.com/blog/using-proxysql-validate-mysql- updates/ 58 / 77
  59. 59. establish (& test) Migration Process 59 / 77
  60. 60. Migration Process Create Migration Plan Different for every environment/application Upgrade a replica rst for a couple of days/weeks? How to switch masters? How is failover being handled nowadays? MHA, Orchestrator, Manual, GTID/msyqlrpladmin...? Test in staging! 60 / 77
  61. 61. the actual Migration In Production 61 / 77
  62. 62. Migration - Create Slave Environments 62 / 77
  63. 63. Migration - Redirect Read Tra c 63 / 77
  64. 64. Migration - Application Switchover - 1 64 / 77
  65. 65. Migration - Application Switchover - 2 65 / 77
  66. 66. you (think you) will never need to do a Rollback 66 / 77
  67. 67. Rollback 67 / 77
  68. 68. Rollback What went wrong? I did not follow the full process! (or I forgot to document it) Do consistency checks again! 68 / 77
  69. 69. after all that testing, it's ok to spend time doing Post-Migration Assessment 69 / 77
  70. 70. Post-Migration Check trending for different behavior more cpu load? more disk IO? higher amount of innodb_rows_* and handler_* threads_running stability? do some query optimization If all looks good, scratch the 5.6 rollback & make it 5.7 Remove the rollback speci c con guration options 70 / 77
  71. 71. Post Migration Cleanup 71 / 77
  72. 72. small recap Summary 72 / 77
  73. 73. Multi-Use (Minor MySQL version upgrades) Major MySQL version upgrades Switching Hardware from Intel -> AMD archicture Using a new kernel/libc/memory allocator Switching storage engines MariaDB/Percona Server/MySQL ... 73 / 77
  74. 74. Do I really have to go through this? Many success stories: Have done several MySQL upgrades from 4.1 -> 5.5 without intermediate slaves Upgraded environments with major schema changes in the mix (mssql-style environments using stored procedures only) Found numerous application bugs using this process Optimized many customers schemas/queries in the meantime As long as you follow this process completely, the risk of running into problems is quite small. 74 / 77
  75. 75. Do I really have to go through this? It Depends: Your business might be risk-averse: every change has to be thoroughly tested Other companies just upgrade a replica in production and see how it goes My suggestion to do this at least for: Major MySQL version upgrades Switching storage engines 75 / 77
  76. 76. Summary Test Step Skip? Document Upgrade Single Server Really? Why? Rollback Scenarios Not Recommended Consistency Checks Required, No Debate! Read Tests Strongly Suggested Workload Tests Possible (Early Adopter Alert) Migration Tests Not Recommended To Skip 76 / 77
  77. 77. Reducing Risk When Upgrading Your MySQL Environment Q&A! Kenny Gryp MySQL Practice Manager

×