Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MySQL Performance Tuning: The Perfect Scalability (OOW2019)

760 views

Published on

“When data grows, MySQL multiplies.” When data grows, a standard configuration does not fit anymore. This hands-on lab walks through the ways MySQL has to adapt to an ever-increasing amount of data in context of tuning, under the heading “multiplication”:
—Scaling up through partitioning and how partitioning can help manage/archive data
—How to relocate binlog, undo, redo, tablespaces, and more on different mount points to take advantage of multiple storages
—All improvements to parallel slave replication
—A quick OS approach, to verify swapping and affinity tuning take the most out of the machine.

Published in: Data & Analytics
  • Be the first to comment

MySQL Performance Tuning: The Perfect Scalability (OOW2019)

  1. 1. Copyright © 2019 Oracle and/or its affiliates. MySQL Performance Tuning: The Perfect Scalability (HOL3025) MySQL Principal Technical Support Engineer Oracle MySQL Support September 17, 2019 Mirko Ortensi
  2. 2. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.
  3. 3. Our Goal When dataset and concurrency increase, horizontal and vertical scaling can address service growth. But in order to achieve perfect scalability, I/O, CPU and Network bound workloads can meet additional improvements at no expense to get the most out of MySQL Server. Copyright © 2019 Oracle and/or its affiliates.
  4. 4. Why This Matters Scaling up and out a database implies more performing storage, more processing capacity, additional memory and higher bandwidth. Keeping adjustements to infrastructure as little as possible helps keeping operational expenditure limited. Copyright © 2019 Oracle and/or its affiliates.
  5. 5. “Query is slow Copyright © 2019 Oracle and/or its affiliates. Anonymous
  6. 6. Copyright © 2019 Oracle and/or its affiliates. Mirko Ortensi MySQL Principal Technical Support Engineer
  7. 7. Agenda • Horizontal vs Vertical scaling • I/O Bound Workload • CPU Bound Workload • Network Bound Workload • OS-wise • Conclusions Copyright © 2019 Oracle and/or its affiliates.
  8. 8. Data and Concurrency grow Service is growing: more data and more active connections will grow the database «old» Copyright © 2019 Oracle and/or its affiliates. THREAD THREAD THREAD THREAD CLIENT CLIENT CLIENT CLIENT THREAD THREAD
  9. 9. I/O, CPU and NETWORK bound More locks + scans Buffer pool should be increased More connections threads I/O throughput Copyright © 2019 Oracle and/or its affiliates.
  10. 10. Agenda • Horizontal vs Vertical scaling • I/O Bound Workload • CPU Bound Workload • Network Bound Workload • OS-wise • Conclusions Copyright © 2019 Oracle and/or its affiliates.
  11. 11. Copyright © 2019 Oracle and/or its affiliates. Scaling out READS With additional slaves MySQL Router WRITE READ REPL
  12. 12. Copyright © 2019 Oracle and/or its affiliates. Scaling out READS With additional slaves • Create slave server (clone(), MEB, mysqldump) • Distribute read and write operations accordingly • Slaves can be delayed
  13. 13. Copyright © 2019 Oracle and/or its affiliates. Scaling out READS With InnoDB Cluster MySQL Router WRITE READ REPL
  14. 14. Copyright © 2019 Oracle and/or its affiliates. Scaling out READS With InnoDB Cluster • Highly available with automatic failover • Integrated solution to scale and balance reads • Up to 9 replicas • Writes do not scale (single or multi master) • Read your own writes
  15. 15. Copyright © 2019 Oracle and/or its affiliates. Scaling out WRITES Sharding • MySQL Server does not support sharding • Only auto-sharding solution is NDB Cluster • Custom / third party ID SERVICE DATA 1773467253 chat xxx 6257346892 location xxx 1773467253 chat xxx 7234782739 photos xxx 8437829249 location xxx
  16. 16. Copyright © 2019 Oracle and/or its affiliates. Scaling Up? • More sockets / cores • Faster disks / SSD • More memory • More bandwidth
  17. 17. MySQL Multiplies! Take advantage of: • Multiple storages • Thread pooling • Parallel slave And tune as usual: • Configuration • Queries Copyright © 2019 Oracle and/or its affiliates. CLIENT CLIENT CLIENT CLIENT THREAD THREAD THREAD CLIENT CLIENT WORKER WORKER MASTER
  18. 18. 1 2 3 Copyright © 2019 Oracle and/or its affiliates. If Data and Concurrency grows, "MySQL multiplies" IO bound • Single storage • Temporary tables • Set to prevent data loss • Insufficient Buffer pool • Write workload CPU bound • Active connections • Redundant indexes • sort/ joins • Scans • Background tasks Network bound • Uncompressed • Huge recordsets
  19. 19. • Linux (native/VM) • MySQL 8 • sysstat • world.sql https://bit.ly/2kO70nY HOL3025 Copyright © 2019 Oracle and/or its affiliates.
  20. 20. Agenda • Horizontal vs Vertical scaling • I/O Bound Workload • CPU Bound Workload • Network Bound Workload • OS-wise • Conclusions Copyright © 2019 Oracle and/or its affiliates.
  21. 21. Copyright © 2019 Oracle and/or its affiliates. I/O Bound Monitoring • iostat • SHOW ENGINE INNODB STATUS • Performance schema (MEM) • SYS schema select * from sys.io_by_thread_by_latency; select * from sys.io_global_by_file_by_bytes; select * from sys.io_global_by_file_by_latency; select * from sys.waits_global_by_latency; select * from sys.statements_with_temp_tables;
  22. 22. Reconfigure Move ib_logfile files Permissions START MySQL Copyright © 2019 Oracle and/or its affiliates. I/O Bound REDO Log Stop the instance, edit Custom innodb_log_group_ho me_dir and save changes. From data directory to the new directory, The files are usually ib_logfile0 and ib_logfile1, but they could be more, depending on the value of innodb_log_files_in_g roup Make sure that the MySQL server has the proper access rights to read/create files in the log directory. It will read the configuration file with the new location.
  23. 23. On storage devices with cache, data loss is possible if data files and redo log files reside on different storage devices, and a crash occurs before data file writes are flushed from the device cache. If you use or intend to use different storage devices for redo logs and data files, use O_DIRECT instead. I/O Bound REDO Log Copyright © 2019 Oracle and/or its affiliates.
  24. 24. I/O Bound Binary Logs Copyright © 2019 Oracle and/or its affiliates. Custom log-bin 1. Stop: systemctl stop mysqld 2. Create: /opt/mysql/binlog 3. Edit my.cnf: log_bin=/opt/mysql/binlog/binlog 4. Move binlog.* to /opt/mysql/binlog 5. Edit: binlog.index to adopt absolute paths (e.g. /opt/mysql/binlog/binlog.000011) 6. Start: systemctl start mysqld
  25. 25. I/O Bound Binary Logs Copyright © 2019 Oracle and/or its affiliates. TEST 1. FLUSH BINARY LOGS; 2. SELECT @@log_bin_basename; 3. RESET MASTER;
  26. 26. I/O Bound UNDO Logs Copyright © 2019 Oracle and/or its affiliates. Custom innodb_undo_directory 1. Stop: systemctl stop mysqld 2. Create: /opt/mysql/undo 3. Relocate undo_00* 4. Start: systemctl start mysqld 5. Check: 1. select @@innodb_undo_directory
  27. 27. I/O Bound EXTERNAL tablespace Copyright © 2019 Oracle and/or its affiliates. • Creating a file-per-table tablespace outside of the data directory. • The DATA DIRECTORY clause cannot be used with ALTER TABLE to change the location later. • CREATE TABLE..DATA DIRECTORY TEST mysql> USE test; Database changed mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY) DATA DIRECTORY = '/opt/mysql/tablespaces';
  28. 28. I/O Bound GENERAL tablespace Copyright © 2019 Oracle and/or its affiliates. • Acts as a container of tablespaces, same as system tablespace • Can be located outside of the data directory • From file-per-table or system to general tablespace (ALTER) • From system or general (shared) to file-per-table (ALTER) • Directory must be known: set innodb_directories (read-only) TEST CREATE TABLESPACE `ts1` ADD DATAFILE '/opt/mysql/tablespaces/ts1.ibd' Engine=InnoDB; CREATE TABLE t1 (c1 INT PRIMARY KEY) TABLESPACE ts1;
  29. 29. I/O Bound Partitioning Copyright © 2019 Oracle and/or its affiliates. • Horizontal partitioning—that is, different rows of a table may be assigned to different physical partitions. • Partitioning makes it possible to store more data in one table than can be held on a single disk or file system partition. TEST CREATE TABLE th (id INT, name VARCHAR(30), adate DATE) PARTITION BY LIST(YEAR(adate)) ( PARTITION p1999 VALUES IN (1995, 1999, 2003) DATA DIRECTORY = '/opt/mysql/tablespaces/', PARTITION p2000 VALUES IN (1996, 2000, 2004) DATA DIRECTORY = '/opt/mysql/tablespaces/' );
  30. 30. I/O Bound Session and Global Temporary Tablespace Copyright © 2019 Oracle and/or its affiliates. • If no value is specified for innodb_temp_data_file_path, the default behavior is to create an auto-extending data file named ibtmp1 in the innodb_data_home_dir (system tablespace location) • Not a dynamic setting • At database start a pool of session 10 temporary tables is created • At database start, ibtmp1 is created to • Temp tables rollback segments in ibtmp1 • Example: --innodb-temp-data-file- path=../../../tmp/ibtmp1:12M:autoextend
  31. 31. I/O Bound Session and Global Temporary Tablespace Copyright © 2019 Oracle and/or its affiliates. TEST • mysql> select @@innodb_temp_data_file_path; • mysql> SELECT * FROM INFORMATION_SCHEMA.FILES WHERE TABLESPACE_NAME='innodb_temporary'G • shell> ls –l <location> • Stop Server and list directory again
  32. 32. I/O Bound Logging General log general_log_file Slow Query Log slow_query_log_file Audit Log audit_log_file Copyright © 2019 Oracle and/or its affiliates.
  33. 33. I/O Bound Flush method Copyright © 2019 Oracle and/or its affiliates. Data can travel through different layer before reaching the durable storage.
  34. 34. I/O Bound Flush method Copyright © 2019 Oracle and/or its affiliates. Flush method: innodb_flush_method (not dynamic) • fsync is default: fsync is used for both data and log file • O_DIRECT can help to avoid double buffering between the InnoDB buffer pool and the operating system file system cache. But is buffer pool is not enough, buffer cache won’t come to help. fsync is used for both data and log file • O_DIRECT_NO_FSYNC same as before, but skips fsyncs • O_DSYNC: InnoDB uses O_SYNC to open and flush the log files, and fsync() to flush the data files, only. Based on O_SYNC: nfsync(), write() calls go to buffer cache, blocks until writes are into the physical storage.
  35. 35. I/O Bound Flush method monitoring Copyright © 2019 Oracle and/or its affiliates. TEST 1 1. sync; echo 1 > /proc/sys/vm/drop_caches 2. vmstat 1 30 3. mysqlslap -uroot -h127.0.0.1 -uroot -p --auto-generate-sql -- concurrency=20 --iterations=100 —number-of-queries=10 TEST 2 At different innodb_flush_method: • mysql> show status like "Innodb_data_fsyncs"; • mysql> show status like "Innodb_data_pending_fsyncs"; • bash> free -m
  36. 36. I/O Bound Miscellaneous Copyright © 2019 Oracle and/or its affiliates. An all-comprehensive discussion in our official online documentation. • Disable logging of compressed pages • Act on I/O capacity for background tasks: innodb_io_capacity and innodb_flush_sync • Threshold on write buffer before fsync(): innodb_fsync_threshold • Data size • ...
  37. 37. Agenda • Horizontal vs Vertical scaling • I/O Bound Workload • CPU Bound Workload • Network Bound Workload • OS-wise • Conclusions Copyright © 2019 Oracle and/or its affiliates.
  38. 38. Copyright © 2019 Oracle and/or its affiliates. CPU Bound Monitoring • vmstat / mpstat • Status SHOW STATUS LIKE "Handler%"; SHOW STATUS LIKE "select_full_join"; • sys schema select * from sys.schema_redundant_indexes; select * from sys.schema_unused_indexes; select * from sys.statements_with_sorting; CALL sys.diagnostics(60, 60, 'current');
  39. 39. Copyright © 2019 Oracle and/or its affiliates. CPU Bound Thread Pool High CPU may also be a good sign, no long IO wait or heavy locking. But can also be for context switches by CPU scheduler. 1. Too many thread stacks make CPU caches almost useless in highly parallel execution workloads 2. With too many threads executing in parallel, context switching overhead is high Thread pool promotes thread stack reuse The thread pool controls when transactions start to ensure that not too many execute in parallel.
  40. 40. Copyright © 2019 Oracle and/or its affiliates. CPU Bound Query Tuning TEST 1. Import world.sql 2. Monitor: mpstat 1 30 3. mysqlslap -uroot -p --no-drop --create-schema=world -- query="select * from world.city where Name='Kabul';" -- concurrency=10 --iterations=1000 4. EXPLAIN SELECT * FROM world.city WHERE Name='Kabul'; 5. SELECT * FROM sys.schema_tables_with_full_table_scans; 6. CREATE INDEX city_idx ON world.city(Name); 7. Repeat the test
  41. 41. Copyright © 2019 Oracle and/or its affiliates. CPU Bound Replication • In 5.6 we introduced parallel slave, that could run load for different databases in parallel. With operations split on several DBs, this could be a simple option. • In 5.7 we introduced LOGICAL_CLOCK algorithm, that allows running many transactions in parallel if they were run in parallel on master. • In 8.0 version we have introduced write sets, that allows to run some statements in parallel, even if they were executed by the same thread on master.
  42. 42. Copyright © 2019 Oracle and/or its affiliates. CPU Bound Miscellaneous • Tablespace compression • Protocol compression • Thread cache • SSL • Background Checkpoint Purge Automatic Statistics Calculation
  43. 43. Agenda • Horizontal vs Vertical scaling • I/O Bound Workload • CPU Bound Workload • Network Bound Workload • OS-wise • Conclusions Copyright © 2019 Oracle and/or its affiliates.
  44. 44. Copyright © 2019 Oracle and/or its affiliates. NETWORK Bound Compression --compress: Compress all information sent between the client and the server if possible TEST • mysql –uroot –compress –p • mysqlslap -uroot -h127.0.0.1 -uroot -p --auto-generate-sql -- concurrency=20 --iterations=100 —number-of-queries=10 –compress • sar -n DEV 3 30 • mpstat 1 30 Look at averages
  45. 45. Agenda • Horizontal vs Vertical scaling • I/O Bound Workload • CPU Bound Workload • Network Bound Workload • OS-wise • Conclusions Copyright © 2019 Oracle and/or its affiliates.
  46. 46. Copyright © 2019 Oracle and/or its affiliates. OS-wise SWAP, SWAPPINESS and AFFINITY SWAP & SWAPPINESS cat /proc/sys/vm/swappiness AFFINITY At the linux OS level, you can limit the number of processors that the kernel will utilize for a user space process using numactl, or more commonly taskset.
  47. 47. Copyright © 2019 Oracle and/or its affiliates. OS-wise SWAP, SWAPPINESS and AFFINITY TEST • Identify systemd configuration file: /etc/systemd/system/multi-user.target.wants/mysqld.service • Modify: [Service] CPUAffinity=1 • bash> systemctl daemon-reload
  48. 48. CPU monitor Press f Last CPU used Esc OS-wise AFFINITY top -H -p <pid> To get the list of options Scroll down and select "P = Last Used Cpu (SMP)", press space bar Column on the right side will show what CPU has just been used Copyright © 2019 Oracle and/or its affiliates.
  49. 49. Agenda • Horizontal vs Vertical scaling • I/O Bound Workload • CPU Bound Workload • Network Bound Workload • OS-wise • Conclusions Copyright © 2019 Oracle and/or its affiliates.
  50. 50. Conclusion Setup the most reliable production-like traffic generator Test and monitor at different concurrencies Experiment! Copyright © 2019 Oracle and/or its affiliates.
  51. 51. Thank You Copyright © 2019 Oracle and/or its affiliates. MySQL Principal Technical Support Engineer Oracle MySQL Support September 17, 2019 Mirko Ortensi
  52. 52. Session Survey Help us make the content even better. Please complete the session survey in the Mobile App. Copyright © 2019 Oracle and/or its affiliates.
  53. 53. What’s Ahead Tuesday 5pm-5.45pm Thursday Copyright © 2019 Oracle and/or its affiliates. NDB, Simply the World’s Highest- Throughput Transactional Datastore [DEV2554] Moscone South - Room 205 10am-10:45am What’s New in MySQL Optimizer and Executor? [DEV2077] Moscone South - Room 205
  54. 54. The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.

×