Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MySQL Performance Tuning 101

151 views

Published on

Here's a brief hands-on to get started with MySQL Server performance tuning. I'll show basic options to get started and have the right basics settings. Hope you'll enjoy!

Published in: Engineering
  • Be the first to comment

MySQL Performance Tuning 101

  1. 1. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Copyright © 2017, Oracle and/or its affiliates. All rights reserved. MySQL Performance Tuning 101 Hands-on-Lab Mirko Ortensi Senior Support Engineer MySQL Support @ Oracle
  2. 2. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Understand what are the most important configuration parameters to get started with MySQL 2 Find out what to look at when performance is not the expected Better hardware does not always mean scalability. Find out what are the typical bottlenecks Objectives
  3. 3. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | “The beginning is the most important part of the work” – Plato 3
  4. 4. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Hands-On Lab Agenda Introduction Connections Threads model REDO log InnoDB buffer pool Execution plan 1 2 3 4 5 4 6
  5. 5. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Introduction Get ready for speed in 5 steps 5
  6. 6. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Introduction • shell$ yum install sysstat • shell$ wget https://tinyurl.com/employeesdb Setup
  7. 7. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Introduction TEST DATABASE The Employees sample database provides a combination of a large base of data (approximately 160MB) spread over six separate tables and consisting of 4 million records in total. BENCHMARKING strategies: single operation vs workload mysqlslap is a diagnostic program designed to emulate client load for a MySQL server and to report the timing of each stage. It works as if multiple clients are accessing the server. Test Database and Benchmarking InnoDB storage engine will be used for the following exercises
  8. 8. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Connections The more, the merrier 8
  9. 9. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Connections THREAD THREAD THREAD THREAD CLIENT CLIENT CLIENT CLIENT THREAD THREAD • Each client connection is associated to a thread • Thread handles authentication and the request • Manager thread handles creation of the thread
  10. 10. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Connections • max_connections – The maximum permitted number of simultaneous client connections. By default, this is 151. • Max_used_connections – The maximum number of connections that have been in use simultaneously since the server started. • Connection_errors_max_connections – The number of connections refused because the server max_connections limit was reached. Basic parameters and configuration
  11. 11. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Connections mysql> select @@max_connections; +-------------------+ | @@max_connections | +-------------------+ | 151 | +-------------------+ 1 row in set (0.00 sec) mysql> show global status like 'Max_used_connections'; +----------------------+-------+ | Variable_name | Value | +----------------------+-------+ | Max_used_connections | 1 | +----------------------+-------+ 1 row in set (0.01 sec) Exercise 1 – READ CONFIGURATION
  12. 12. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Connections [hol@localhost ~]$ mysqlslap --auto-generate-sql --concurrency=100 --iterations=10 Benchmark Average number of seconds to run all queries: 0.758 seconds Minimum number of seconds to run all queries: 0.620 seconds Maximum number of seconds to run all queries: 0.967 seconds Number of clients running queries: 100 Average number of queries per client: 0 mysql> show global status like 'Max_used_connections'; +----------------------+-------+ | Variable_name | Value | +----------------------+-------+ | Max_used_connections | 101 | +----------------------+-------+ 1 row in set (0.01 sec) Exercise 1 – ADD LOAD, 100 CONNECTIONS
  13. 13. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Connections [hol@localhost ~]$ mysqlslap --auto-generate-sql --concurrency=170 --iterations=10 mysqlslap: Error when connecting to server: 1040 Too many connections mysqlslap: Error when connecting to server: 1040 Too many connections mysqlslap: Error when connecting to server: 1040 Too many connections mysqlslap: Error when connecting to server: 1040 Too many connections mysqlslap: Error when connecting to server: 1040 Too many connections mysqlslap: Error when connecting to server: 1040 Too many connections mysqlslap: Error when connecting to server: 1040 Too many connections mysqlslap: Error when connecting to server: 1040 Too many connections mysqlslap: Error when connecting to server: 1040 Too many connections Exercise 1 – MORE LOAD, 170 CONNECTIONS
  14. 14. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Connections mysql> show global status like 'Connection_errors_max_connections'; +-----------------------------------+-------+ | Variable_name | Value | +-----------------------------------+-------+ | Connection_errors_max_connections | 213 | +-----------------------------------+-------+ 1 row in set (0.00 sec) mysql> SET GLOBAL max_connections=200; Query OK, 0 rows affected (0.00 sec) Exercise 1 – ALLOW MORE CONNECTIONS
  15. 15. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Connections [hol@localhost ~]$ mysqlslap --auto-generate-sql --concurrency=170 --iterations=10 Benchmark Average number of seconds to run all queries: 1.563 seconds Minimum number of seconds to run all queries: 1.367 seconds Maximum number of seconds to run all queries: 1.800 seconds Number of clients running queries: 170 Average number of queries per client: 0 Exercise 1 – RETEST
  16. 16. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Connections • Connection pooling is a technique of creating and managing a pool of connections that are ready for use by any thread that needs them • Connection pooling can greatly increase the performance of your application, while reducing overall resource usage – Connector/J – Connector/Python – Connector/Net – Connector/ODBC Connection pooling
  17. 17. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Threads model Reuse for good 17
  18. 18. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Threads model 18 THREAD THREAD THREAD THREAD CLIENT CLIENT CLIENT CLIENT THREAD THREAD • Manager threads create a new thread when necessary but try to avoid doing so by consulting the thread cache • When a connection ends, its thread is returned to the thread cache if the cache is not full.
  19. 19. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Threads model mysql> select @@thread_cache_size; +---------------------+ | @@thread_cache_size | +---------------------+ | 9 | +---------------------+ 1 row in set (0.00 sec) mysql> show global status like 'Threads_%'; +-------------------+-------+ | Variable_name | Value | +-------------------+-------+ | Threads_cached | 8 | | Threads_connected | 1 | | Threads_created | 2816 | | Threads_running | 1 | +-------------------+-------+ 4 rows in set (0.00 sec) Exercise 2 – UNDERSTAND THE ROLE OF THREAD CACHE
  20. 20. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Threads model [hol@localhost ~]$ mysqlslap --auto-generate-sql --concurrency=100 --iterations=10 Benchmark Average number of seconds to run all queries: 0.682 seconds Minimum number of seconds to run all queries: 0.517 seconds Maximum number of seconds to run all queries: 0.838 seconds Number of clients running queries: 100 Average number of queries per client: 0 mysql> show global status like 'Threads_created'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | Threads_created | 3727 | +-----------------+-------+ 1 row in set (0.00 sec) Exercise 2 – CHECK THREADS CREATION
  21. 21. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Threads model mysql> SET GLOBAL thread_cache_size=100; Query OK, 0 rows affected (0.00 sec) [hol@localhost ~]$ mysqlslap --auto-generate-sql --concurrency=100 --iterations=10 Benchmark Average number of seconds to run all queries: 0.732 seconds Minimum number of seconds to run all queries: 0.655 seconds Maximum number of seconds to run all queries: 0.818 seconds Number of clients running queries: 100 Average number of queries per client: 0 mysql> show global status like 'Threads_created'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | Threads_created | 3819 | +-----------------+-------+ 1 row in set (0.00 sec) Exercise 2 – INCREASE THREAD CACHE
  22. 22. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Threads model • The default thread-handling model in MySQL Server executes statements using one thread per client connection • As more clients connect to the server and execute statements, overall performance degrades • The thread pool addresses several problems of the one thread per connection model MySQL Enterprise Thread pool
  23. 23. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | REDO log Sustainable IO throughput 23
  24. 24. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | REDO log • The redo log is a disk-based data structure used during crash recovery • Redo log encodes requests to change InnoDB table data • Redo log is flushed before a transaction is committed • MySQL writes to the redo log files in a circular fashion 24 What is it? CLIENT BUFFER POOL REDO LOG CLIENT
  25. 25. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | REDO log • innodb_log_buffer_size – The size in bytes of the buffer that InnoDB uses to write to the log files on disk • innodb_log_file_size – The size in bytes of each log file in a log group • innodb_log_files_in_group – The number of log files in the log group • innodb_flush_log_at_trx_commit – ACID compliance for commit operations vs higher performance Configuration
  26. 26. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | REDO log • shell$ mysqlslap --auto-generate-sql -- concurrency=100 --iterations=100 • shell$ iostat -xdh /dev/sda 2 – Execute on other console session as mysqlslap is running Exercise 3 – SHOW REDO LOGGING IO IMPACT %util Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). w/s The number of write requests that were issued to the device per second.
  27. 27. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | REDO log Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 263.93 0.00 742.08 0.0k 3.9M 10.79 0.35 0.48 0.00 0.48 0.47 34.97 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 251.87 0.00 708.56 0.0k 3.7M 10.72 0.34 0.49 0.00 0.49 0.48 34.01 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 246.52 0.00 693.05 0.0k 3.6M 10.76 0.34 0.49 0.00 0.49 0.48 33.58 Exercise 3 – IOSTAT OUTPUT
  28. 28. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | REDO log • mysql> set global innodb_flush_log_at_trx_commit=0; • shell$ mysqlslap --auto-generate-sql -- concurrency=100 --iterations=100 • shell$ iostat -xdh /dev/sda 2 – Execute on other console session as mysqlslap is running Exercise 3 – REDUCE IO PRESSURE With innodb_flush_log_at_trx_commit = 0 you can lose up to a second of transactions with any mysqld process crash. Use with test instances, restores, replication slaves...
  29. 29. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | REDO log Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 135.45 0.00 124.87 0.0k 1.9M 31.49 0.08 0.61 0.00 0.61 0.61 7.57 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 129.95 0.00 121.93 0.0k 1.9M 31.79 0.09 0.74 0.00 0.74 0.72 8.77 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 102.12 0.00 97.88 0.0k 1.5M 31.70 0.06 0.63 0.00 0.63 0.61 5.98 Exercise 3 – AFTER innodb_flush_log_at_trx_commit
  30. 30. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | REDO log • mysql> show global status like 'Innodb_os_log_written'; • SYS schema: the host_summary_by_file_io_type view Exercise 3 – MEASURE LATENCY mysql> select * from sys.host_summary_by_file_io_type; +------------+--------------------------------------+-------+---------------+-------------+ | host | event_name | total | total_latency | max_latency | +------------+--------------------------------------+-------+---------------+-------------+ | background | wait/io/file/innodb/innodb_data_file | 1607 | 332.96 ms | 39.41 ms | | background | wait/io/file/innodb/innodb_log_file | 171 | 237.56 ms | 17.46 ms | | background | wait/io/file/sql/FRM | 2135 | 1.98 ms | 67.37 us | | background | wait/io/file/sql/casetest | 15 | 926.63 us | 833.10 us | | background | wait/io/file/sql/file_parser | 204 | 296.84 us | 3.58 us | | background | wait/io/file/myisam/kfile | 33 | 51.20 us | 5.19 us | | background | wait/io/file/sql/pid | 3 | 39.02 us | 27.48 us |
  31. 31. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | InnoDB Buffer Pool In memory it’s faster 31
  32. 32. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | InnoDB Buffer Pool 32 What is it? • InnoDB maintains a storage area called the buffer pool for caching data and indexes in memory • Used to keep frequently accessed data in memory CLIENT BUFFER POOL REDO LOG CLIENT
  33. 33. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | InnoDB Buffer Pool mysql> show global status like 'innodb_buffer_pool_pages_total'; mysql> show global status like 'innodb_buffer_pool_pages_free'; mysql> show global status like 'Innodb_buffer_pool_reads'; mysql> show global status like 'Innodb_buffer_pool_read_requests'; mysql> select * from information_schema.INNODB_BUFFER_POOL_STATSG; mysql> select * from sys.schema_table_statistics_with_bufferG; How to Monitor the Buffer Pool
  34. 34. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | InnoDB Buffer Pool mysqlslap --no-drop --create-schema=employees --query="select * from employees.employees;" --concurrency=1 --iterations=100 Benchmark Average number of seconds to run all queries: 0.321 seconds Minimum number of seconds to run all queries: 0.265 seconds Maximum number of seconds to run all queries: 0.591 seconds Number of clients running queries: 1 Average number of queries per client: 1 Exercise 4 – STRESS TEST
  35. 35. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | InnoDB Buffer Pool mysql> select * from information_schema.INNODB_BUFFER_POOL_STATSG; *************************** 1. row *************************** POOL_ID: 0 POOL_SIZE: 8191 FREE_BUFFERS: 5469 DATABASE_PAGES: 2720 […] PAGES_WRITTEN_RATE: 0 NUMBER_PAGES_GET: 23128215 HIT_RATE: 1000 YOUNG_MAKE_PER_THOUSAND_GETS: 0 NOT_YOUNG_MAKE_PER_THOUSAND_GETS: 0 NUMBER_PAGES_READ_AHEAD: 757 NUMBER_READ_AHEAD_EVICTED: 0 READ_AHEAD_RATE: 0 READ_AHEAD_EVICTED_RATE: 0 LRU_IO_TOTAL: 0 LRU_IO_CURRENT: 0 UNCOMPRESS_TOTAL: 0 UNCOMPRESS_CURRENT: 0 1 row in set (0.00 sec) Exercise 4 – CHECK BUFFER POOL HIT RATE
  36. 36. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | InnoDB Buffer Pool EDIT /ETC/MY.CNF AND RESTART THE SERVER 1. innodb_buffer_pool_chunk_size=1048576 2. innodb_buffer_pool_size=5242880 3. shell$ sudo service mysld restart Run mysqlslap and monitor the stats of InnoDB Buffer Pool on a a different console • shell$ mysqlslap --no-drop --create-schema=employees - -query="select * from employees.employees;" -- concurrency=1 --iterations=100 Exercise 4 – SHRINK BUFFER POOL
  37. 37. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | InnoDB Buffer Pool mysql> select * from information_schema.INNODB_BUFFER_POOL_STATSG; *************************** 1. row *************************** POOL_ID: 0 POOL_SIZE: 320 FREE_BUFFERS: 0 DATABASE_PAGES: 320 OLD_DATABASE_PAGES: 0 [...] PAGES_CREATE_RATE: 0 PAGES_WRITTEN_RATE: 0 NUMBER_PAGES_GET: 774836 HIT_RATE: 975 YOUNG_MAKE_PER_THOUSAND_GETS: 0 NOT_YOUNG_MAKE_PER_THOUSAND_GETS: 0 NUMBER_PAGES_READ_AHEAD: 19360 NUMBER_READ_AHEAD_EVICTED: 475 READ_AHEAD_RATE: 2639.34016495876 READ_AHEAD_EVICTED_RATE: 65.983504123969 LRU_IO_TOTAL: 1659 LRU_IO_CURRENT: 193 UNCOMPRESS_TOTAL: 0 UNCOMPRESS_CURRENT: 0 1 row in set (0.00 sec) Exercise 4 – CHECK AGAIN BUFFER POOL
  38. 38. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Execution plan Don’t waste your time 38
  39. 39. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Execution plan • Different aspects to consider to speed up a query – Concurrency, caching, locking, IO…and execution plan • Tools to investigate a slow query – SHOW ENGINE INNODB STATUS – SHOW PROCESSLIST – EXPLAIN – SLOW QUERY LOG – SYS SCHEMA 39
  40. 40. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Execution plan mysql> explain select * from employees.employees where adddate(hire_date,INTERVAL 18 YEAR)>=NOW(); +----+-------------+-----------+------------+------+---------------+------+---------+------+--------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-----------+------------+------+---------------+------+---------+------+--------+----------+-------------+ | 1 | SIMPLE | employees | NULL | ALL | NULL | NULL | NULL | NULL | 291892 | 100.00 | Using where | +----+-------------+-----------+------------+------+---------------+------+---------+------+--------+----------+-------------+ 1 row in set, 1 warning (0.00 sec) mysql> create index hire_date_idx on employees(hire_date); Query OK, 0 rows affected (0.99 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> explain select * from employees.employees where adddate(hire_date,INTERVAL 18 YEAR)>=NOW(); +----+-------------+-----------+------------+------+---------------+------+---------+------+--------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-----------+------------+------+---------------+------+---------+------+--------+----------+-------------+ | 1 | SIMPLE | employees | NULL | ALL | NULL | NULL | NULL | NULL | 291892 | 100.00 | Using where | +----+-------------+-----------+------------+------+---------------+------+---------+------+--------+----------+-------------+ 1 row in set, 1 warning (0.00 sec) EXERCISE 5 – HAVE A BAD EXECUTION PLAN?
  41. 41. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Execution plan mysql> ALTER TABLE employees ADD COLUMN past_date date GENERATED ALWAYS AS (adddate(hire_date,INTERVAL 18 YEAR)) VIRTUAL; Query OK, 0 rows affected (0.03 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> CREATE INDEX past_date_idx on employees(past_date); Query OK, 0 rows affected (1.01 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> explain select * from employees.employees where adddate(hire_date,INTERVAL 18 YEAR)>=NOW(); +----+-------------+-----------+------------+-------+---------------+---------------+---------+------+------+----------+------------------------ + | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+-----------+------------+-------+---------------+---------------+---------+------+------+----------+------------------------ + | 1 | SIMPLE | employees | NULL | range | past_date_idx | past_date_idx | 4 | NULL | 111 | 100.00 | Using where; Using MRR | +----+-------------+-----------+------------+-------+---------------+---------------+---------+------+------+----------+------------------------ + 1 row in set, 1 warning (0.00 sec) EXERCISE 5 –FUNCTION GENERATED INDEX
  42. 42. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Wrapping up 1. Use connection pool 2. Tune thread cache / use Thread Pooling plugin (Enterprise version) 3. Configure REDO logging 4. Configure buffer pool to maximize memory usage 5. Analyze execution plan for queries
  43. 43. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Thank You • MySQL Documentation dev.mysql.com • Support Blog dev.mysql.com/support/blogs 43 Q&A

×