SlideShare a Scribd company logo
MYSQL PERFORMANCE TUNING
I. MySQL Configuration (incl. MySQL 5.7)
Aurimas Mikalauskas
@ Speedemy
AURIMAS MIKALAUSKAS
• MySQL hacker since 1999 (yep, v3.23)
• In love with performance & scalability
since 2004 Danga Interactive talk on
LiveJournal’s Backend at LISA04
• Percona Senior Performance
Consultant & Architect 2006-2015
• Performance Engineer at
EstanteVirtual.com.br
• Independent data performance
consultant, instructor at Speedemy.com
ON THE AGENDA FOR TODAY
How to Choose The Right MySQL Distribution
Essentials of MySQL Configuration Tuning
17 Key MySQL Settings for Best Performance
DISTROS
MySQL? Percona Server?
MariaDB? WebScaleSQL?
COMMUNITY MYSQL
• Community MySQL 5.1
• innodb plugin not enabled by default
• very poor InnoDB scalability (by default)
• Community MySQL 5.5
• still missing a number of performance improvements
• stability issues with high amounts of writes
• Community MySQL 5.6
• not many reasons to use alternative distro
• still not perfectly stable with high end hardware in a write-
intensive environment
COMMUNITY MYSQL 5.7
• Current GA release
• Rock solid
• New features:
• multi-source replication
• JSON support
• proper multi-threaded replication
• online buffer pool resize
• spatial data types for InnoDB
• sys schema
• etc. (see full list)
PERCONA SERVER
• Launched July 2008
• as Percona patches for MySQL
• in reality earlier than that
• Always up to date with upstream
• i.e. Percona Server 5.1.73 = MySQL 5.1.73 + Percona code
• also always backwards compatible
• meaning no problem to keep switching between Percona Server and
MySQL. No upgrade/downgrade scripts need to be used.
• some exceptions to backwards compatibility in the past
• none of these options were enabled by default
• very clear when it’s the case
• Makes it easy to try it out and switch back at any point
PERCONA SERVER VERSION
• Using MySQL 5.1?
• definitely switch to Percona Server 5.1
• Using MySQL 5.5? Switch to Percona Server 5.5 if…
• adaptive hash index is a bottle-neck
• you want faster checksums (hardware accelerated)
• you have write-intensive workload
• you need PAM authentication, audit log, thread pool
• Using MySQL 5.6? Switch to Percona Server 5.6 if…
• you have very write-intensive workload and high end hardware
• you need some of the features mentioned with 5.5
• MySQL 5.7? Use MySQL, Percona Server 5.7 coming soon.
PERCONA SERVER KEYSTONE FEATURE: EXTENDED SLOW QUERY LOG
# Time: 150811 12:15:06.959000
# User@Host: root[root] @ localhost [localhost] Id: 62591948
# Schema: Core_DB Last_errno: 0 Killed: 0
# Query_time: 2.411376 Lock_time: 0.002444 Rows_sent: 0 Rows_examined: 10811959 Rows_affected: 0
# Bytes_sent: 0 Tmp_tables: 1 Tmp_disk_tables: 0 Tmp_table_sizes: 0
# QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: No
# Filesort: No Filesort_on_disk: No Merge_passes: 0
# InnoDB_trx_id: 20A40488F
# InnoDB_IO_r_ops: 37 InnoDB_IO_r_bytes: 606208 InnoDB_IO_r_wait: 0.000225
# InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
# InnoDB_pages_distinct: 1747
SELECT ...
# Time: 150811 12:15:06
# User@Host: root[root] @ localhost [localhost] Id: 62591948
# Schema: Core_DB Last_errno: 1051 Killed: 0
# Query_time: 2.411376 Lock_time: 0.002444 Rows_sent: 0 Rows_examined: 10811959
SELECT ...
-VS-
microsecond TS
EXTENDED SLOW QUERY LOG (2)
# Time: 150811 12:15:06.959000
# User@Host: root[root] @ localhost [localhost] Id: 62591948
# Schema: Core_DB Last_errno: 0 Killed: 0
# Query_time: 2.411376 Lock_time: 0.002444 Rows_sent: 0 Rows_examined: 10811959 Rows_affected: 0
# Bytes_sent: 0 Tmp_tables: 1 Tmp_disk_tables: 0 Tmp_table_sizes: 0
# QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: No
# Filesort: No Filesort_on_disk: No Merge_passes: 0
# InnoDB_trx_id: 20A40488F
# InnoDB_IO_r_ops: 37 InnoDB_IO_r_bytes: 606208 InnoDB_IO_r_wait: 0.000225
# InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
# InnoDB_pages_distinct: 1747
SELECT ...
# Time: 150811 12:15:06
# User@Host: root[root] @ localhost [localhost] Id: 62591948
# Schema: Core_DB Last_errno: 1051 Killed: 0
# Query_time: 2.411376 Lock_time: 0.002444 Rows_sent: 0 Rows_examined: 10811959
SELECT ...
-VS-
query plan
EXTENDED SLOW QUERY LOG (3)
# Time: 150811 12:15:06.959000
# User@Host: root[root] @ localhost [localhost] Id: 62591948
# Schema: Core_DB Last_errno: 0 Killed: 0
# Query_time: 2.411376 Lock_time: 0.002444 Rows_sent: 0 Rows_examined: 10811959 Rows_affected: 0
# Bytes_sent: 0 Tmp_tables: 1 Tmp_disk_tables: 0 Tmp_table_sizes: 0
# QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: No
# Filesort: No Filesort_on_disk: No Merge_passes: 0
# InnoDB_trx_id: 20A40488F
# InnoDB_IO_r_ops: 37 InnoDB_IO_r_bytes: 606208 InnoDB_IO_r_wait: 0.000225
# InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
# InnoDB_pages_distinct: 1747
SELECT ...
# Time: 150811 12:15:06
# User@Host: root[root] @ localhost [localhost] Id: 62591948
# Schema: Core_DB Last_errno: 1051 Killed: 0
# Query_time: 2.411376 Lock_time: 0.002444 Rows_sent: 0 Rows_examined: 10811959
SELECT ...
-VS-
extra innodb info
MARIADB
• Made by Monty Widenius
• creator of MySQL and MyISAM
• Advertised as the good MySQL
• as opposed to bad MySQL that’s being “killed” by Evil Corp Oracle
• MariaDB 5.5 = MySQL 5.5 + XtraDB + MariaDB 5.3:
• query optimizer improvements
• multi-master replication
• group commit fix
• MariaDB 10
• real fork of MySQL 5.6
• may become “backwards” incompatible
MARIADB 10 FEATURES
• Parallel replication
• Multi-source replication
• Cassandra, Spider, TokuDB storage engines
• and few others.
SWITCH TO MARIADB?
• Think twice before switching
• No great advantages over MySQL 5.7 or Percona Server
• And yet a number of performance improvements from 5.7 will
be missed in MariaDB 10
• Some features may be appealing, if that’s your case - go for it
• Backwards incompatibility is a bit worrying
• full mysqldump->import should still be possible
WEBSCALESQL
15
• Collaboration by a few heavy-duty MySQL users:
• Alibaba, Facebook, Google, Linkedin & Twitter
• MySQL 5.6 at its core
• good stuff being back-ported from MySQL 5.7
• Has very special mission
• is not meant to be general purpose MySQL Server
• addresses needs of running MySQL at SCALE
FEW WEBSCALESQL FEATURES
16
• Ability to specify millisecond timeouts
• Super read-only mode
• Ability to disable deadlock detection
• Prefix index query optimization
• Performance Schema not compiled in
• InnoDB flushing performance fixes
read this article by Laurynas @ Percona to learn more.
SWITCH TO WEBSCALESQL?
17
• Think thrice
• Only switch if you REALLY understand what it is
• Great performance features that don’t hurt functionality will be
ported to other variants (some already are)
THE ESSENTIALS
Of MySQL Configuration Tuning
MYSQL DEFAULTS ARE POOR
• Okay for development, not for busy production db
• Good news:
• MySQL 5.6 defaults are better than versions < 5.6
• MySQL 5.7 defaults are better than 5.6
• MySQL is super easy to configure
MYSQL CONFIGURATION FILE
• One configuration file (unless it contains include directives)
• Location:
• /etc/my.cnf - most systems
• /etc/mysql/my.cnf - Debian style
• On Windows - best use data directory (create the file)
COMMON MISTAKES
• Using Trial and Error approach
• change something, see if it feels better
• don’t do it
• Asking Google for performance advice
• answers often lack context
• use case could be benchmarks - often non-production suitable
• settings are hardware (or else) dependant
• Obsessing about fine-tuning the my.cnf
• 10-15 variables is often all you need to change
• fine-tuning won’t give you significant wins
COMMON MISTAKES (2)
• Changing many things at once
• makes it very hard to figure out what caused which effect
• instead, change one thing, then measure impact
• Not keeping my.cnf in sync with the changes you make
• changing settings online is convenient, but…
• don’t forget to update my.cnf
• Redundant entries in my.cnf
• MySQL won’t mind them
• Last value will be used
• “-” and “_” can be used interchangeably
COMMON MISTAKES (3)
• Multiplying buffer sizes
• don’t do it
• some buffers are local, some server-wide
• few variables need to be increased after hardware upgrades
• Using the wrong my.cnf section
• [mysql], [client], [mysqld_safe] — all are incorrect choices
• [mysqld] — put ALL of the server configuration here
CHANGING CONFIGURATION ONLINE
• Many things can be changed online (dynamically)
• Even innodb_buffer_pool_size starting with MySQL 5.7
• Example of changing innodb_thread_concurrency online:
mysql> show global variables like 'innodb%';
+-------------------------------------------+------------------------+
| Variable_name | Value |
+-------------------------------------------+------------------------+
...
| innodb_thread_concurrency | 0 |
...
+-------------------------------------------+------------------------+
140 rows in set (0.02 sec)
mysql> select @@global.innodb_thread_concurrency;
+------------------------------------+
| @@global.innodb_thread_concurrency |
+------------------------------------+
| 8 |
+------------------------------------+
1 row in set (0.00 sec)
mysql> set global innodb_thread_concurrency = 8;
Query OK, 0 rows affected (0.01 sec)
GLOBAL -VS- LOCAL SCOPE
• In many cases, you only want to change local session buffers
and leave global configuration as is
• (sort|join|read|read_rnd)_buffer_size are all good examples
• So:
• For a query that needs to sort a lot of data, before you run it:
set sort_buffer_size = 64 * 1024 * 1024;
• Some queries would find index merge intersect optimization
harmful, so for such queries you can just:
set optimizer_switch = 'index_merge_intersection=off';
17
Key MySQL Settings
READY TO USE MY.CNF WITH ALL SETTINGS
I have prepared a my.cnf with short handy descriptions near each
variable and appropriate links to learn more for your
convenience. Download it here:
http://www.speedemy.com/17
1. DEFAULT_STORAGE_ENGINE – CHOOSE THE RIGHT ENGINE FIRST
• Already on InnoDB? Skip to next variable
• Otherwise, bear with me.
STORAGE ENGINE WHAT?
• MySQL uses pluggable storage engines since the beginning
• Different storage engines store data in different ways
• some may even store it in a remote server (e.g. Federated)
• others may only pretend they are storing data (e.g. Blackhole)
• Two most commonly used ones:
• MyISAM
• InnoDB
• Few other famous SEs:
• Blackhole
• Archive
• CSV
• TokuDB
WHAT’S WRONG WITH MYISAM
• Non transactional:
• no Atomicity
• no Consistency
• no Isolation
• no Durability
• Table level locks only
• Not scalable when it comes to write intensive workload
• with an exception of INSERT-only workload
MEET INNODB
• Shipped with MySQL since 2001
• Fully ACID transactional storage engine, designed to handle
highly concurrent workload and scale.
• Road to multi-core multi-disk systems was bumpy
• handles high concurrency pretty well since MySQL 5.0.30
• is hard to beat in many ways nowadays
• Default MySQL storage engine since version 5.5.5
HERE’S STORAGE ENGINES YOU ARE USING
• Run this on your server:
mysql> SELECT engine,
count(*) as TABLES,
concat(round(sum(table_rows)/1000000,2),'M') rows,
concat(round(sum(data_length)/(1024*1024*1024),2),'G') DATA,
concat(round(sum(index_length)/(1024*1024*1024),2),'G') idx,
concat(round(sum(data_length+index_length)/(1024*1024*1024),2),'G') total_size,
round(sum(index_length)/sum(data_length),2) idxfrac
FROM information_schema.TABLES
WHERE table_schema not in ('mysql', 'performance_schema', 'information_schema')
GROUP BY engine
ORDER BY sum(data_length+index_length) DESC LIMIT 10;
+--------+--------+---------+--------+--------+------------+---------+
| engine | TABLES | rows | DATA | idx | total_size | idxfrac |
+--------+--------+---------+--------+--------+------------+---------+
| InnoDB | 181 | 457.58M | 92.34G | 54.58G | 146.92G | 0.59 |
| MyISAM | 13 | 22.91M | 7.85G | 2.12G | 9.97G | 0.27 |
+--------+--------+---------+--------+--------+------------+---------+
2 rows in set (0.22 sec)
* no need to rewrite this, it will be included in the ebook you will get at speedemy.com/17
HERE’S MYISAM TABLES ON YOUR SERVER
• Run this:
mysql> SELECT
concat(table_schema, '.', table_name) tbl,
engine,
concat(round(table_rows/1000000,2),'M') rows,
concat(round(data_length/(1024*1024*1024),2),'G') DATA,
concat(round(index_length/(1024*1024*1024),2),'G') idx,
concat(round((data_length+index_length)/(1024*1024*1024),2),'G') total_size,
round(index_length/data_length,2) idxfrac
FROM information_schema.TABLES
WHERE table_schema not in ('mysql', 'performance_schema', 'information_schema')
AND engine = 'MyISAM'
ORDER BY data_length+index_length DESC;
CONVERTING TO INNODB
• It’s not enough to change default-storage-engine
• Tables need to be converted. One. By. One.
• Simply converting largest tables is not enough:
• if at least one table in a join is MyISAM, the entire query is
using table level locks. So having even a small MyISAM table in
a large join can be very bad for concurrency
• Convert only when the server is configured for InnoDB
properly
CONVERTING EACH TABLE TO INNODB
• This will give you a list of commands to run:
SET @DB_NAME = 'your_database';
SELECT CONCAT('ALTER TABLE `', TABLE_NAME, '` ENGINE=InnoDB;') AS sql_statements
FROM INFORMATION_SCHEMA.TABLES AS tb
WHERE TABLE_SCHEMA = @DB_NAME
AND ENGINE = 'MyISAM'
AND TABLE_TYPE = 'BASE TABLE'
ORDER BY TABLE_NAME DESC;
• To use InnoDB even when MyISAM is specified as a storage
engine, in Percona Server you can set this in my.cnf:
enforce_storage_engine = InnoDB
2. INNODB_BUFFER_POOL_SIZE – GET THE BEST OUT OF YOUR MEMORY
• Most important variable for InnoDB
• InnoDB Buffer Pool is:
• a cache for read data (these are stored in pages of 16kb)
• also cache for indexes, modified (dirty) data
• and place for many internal InnoDB structures
QUITE A TYPICAL BUFFER POOL DISTRIBUTION
INNODB_BUFFER_POOL_SIZE = ?
• On a dedicated server, set to 80% of total memory
• leaves room for other internal structures outside the buffer pool
• query execution (session buffers)
• OS cache (binary logs, relay logs, innodb transaction log, etc.)
• OS memory structures
• even on servers with 256-512GB of RAM
• Avoid swapping at all costs!
• swapping is not the same as reading data from disk
• it’s much much worse
INNODB_BUFFER_POOL_SIZE = ?
• On a shared server:
• check total size of db, maybe it all fits in, say, 50% of RAM?
• don’t strive to make it perfect
• again, avoid swapping at all costs (use vmstat 1 to check)
• Number of random reads from disk into BP can be checked
by:
$ mysqladmin ext -ri1 | grep Innodb_buffer_pool_reads
| Innodb_buffer_pool_reads | 1832098003 |
| Innodb_buffer_pool_reads | 595 |
| Innodb_buffer_pool_reads | 915 |
| Innodb_buffer_pool_reads | 734 |
| Innodb_buffer_pool_reads | 622 |
| Innodb_buffer_pool_reads | 710 |
| Innodb_buffer_pool_reads | 664 |
...
CHANGING INNODB_BUFFER_POOL_SIZE
• On MySQL 5.7, can be done online:
mysql> set global innodb_buffer_pool_size = size_in_bytes;
• Don’t forget to update my.cnf after this
• MySQL log will go like:
[Note] InnoDB: Resizing buffer pool from 134217728 to 21474836480. (unit=134217728)
[Note] InnoDB: disabled adaptive hash index.
[Note] InnoDB: buffer pool 0 : 159 chunks (1302369 blocks) were added.
[Note] InnoDB: buffer pool 0 : hash tables were resized.
[Note] InnoDB: Resized hash tables at lock_sys, adaptive hash index, dictionary.
[Note] InnoDB: Completed to resize buffer pool from 134217728 to 21474836480.
[Note] InnoDB: Re-enabled adaptive hash index.
• On any previous version of MySQL:
• set an appropriate innodb_buffer_pool_size in my.cnf
• restart MySQL server
3. INNODB_LOG_FILE_SIZE – ROOM FOR MYSQL’S REDO LOG
• Size for redo (a.k.a. transaction) logs
• 5MB up until MySQL 5.6.8 — way too small for just about
any workload
• Current default of 48MB still too small for most
• What’s innodb log file?
• next slide, please
WHAT IS INNODB REDO LOG?
• Transactional storage engines (including InnoDB) need both
undo and redo functions, similar to image editor
• well not exactly; let me explain
• Undo happens when rollback occurs
• Implicitly or explicitly
• Redo happens after a database crash
• Also when preparing a backup and other similar cases
UNDO
• When record is changed (but before COMMIT)
• changes are not written to data files right away
• first they are written to transaction log
• and data is modified in memory
• original unmodified copy is moved to rollback segment
• if Rollback occurs:
• Undo needs to happen, to restore data to previous state
• InnoDB removes the copy from the rollback segment (and
copies it back if need be), removes the dirty page
• and marks in transaction log that change was rolled back
• That’s Undo.
REDO
• After COMMIT, changes are ready to be written to data files
• but they aren’t actually written (yet). This would be inefficient.
• Instead, changes are written to Redo log (and modified pages
are only stored in memory)
• If MySQL crashes (before changes are written to data files):
• Redo needs to happen
• else, these changes would be lost forever
• After restart, InnoDB finds the last checkpoint position
• and re-applies the same changes it had in memory before restart
• It is a gross oversimplification, but yeah, that’s Redo.
REDO LOG SIZE
• Size matters:
• Small log files make writes slower and crash recovery faster
• Large log files make writes faster and crash recovery slower
• Log file is like a buffer, so:
• small log files = small buffer, so flushing needs to happen often
• big log files = big buffer, hence flushing is more streamlined
• But:
• with large log files, in case of crash, more work needs to be done
to restore the database to the consistent state (i.e. the Redo
operation)
SIZING REDO LOG
• Rule of Thumb:
• Check that total size of your Redo logs fits in 1-2h worth of
writes during your busy period
mysql> pager grep seq
mysql> show engine innodb statusG select sleep(60); show engine innodb statusG
Log sequence number 1777308180429
...
Log sequence number 1777354541591
mysql> nopager
mysql> select (1777354541591-1777308180429)*60/1024/1024;
+--------------------------------------------+
| (1777354541591-1777308180429)*60/1024/1024 |
+--------------------------------------------+
| 2652.80696869 |
+--------------------------------------------+
1 row in set (0.00 sec)
• Based on this 60s sample, InnoDB could be writing around
2.6GB per hour, so innodb_log_file_size=2560M sounds like a
good start to get 5GB worth of redo logs total
CHANGING THE REDO LOG SIZE
• MySQL ≥ 5.6 - change my.cnf and restart the server.
• MySQL < 5.6, my.cnf change is not enough:
1. change innodb_log_file_size in my.cnf
2. stop MySQL server
3. ensure MySQL had a clean shutdown (mysql log is your
friend)
4. remove old log files, usually by running the following
command: rm -f /var/lib/mysql/ib_logfile*
5. start MySQL server – it should take a bit longer to start
because it is going to be creating new transaction log files
• MySQL < 5.6.2 only supports 4GB total!
4. INNODB_FLUSH_LOG_AT_TRX_COMMIT – DURABLE OR NOT? THAT IS THE QUESTION!
• innodb_flush_log_at_trx_commit=1 by default:
• FLUSH and SYNC after EVERY transaction commit - full durability
• INSERT/UPDATE/DELETE is a transaction if autocommit=1 (default
behaviour)
• SYNC is often expensive - it’s synchronisation to disks
• Exception is if you have non-volatile cache
• e.g. battery-backup unit (BBU) protected write-back cache
• or Super-capacitor with a flash memory chip
• Alternative values for innodb_flush_log_at_trx_commit:
• 0 means FLUSH the buffers to OS, but DO NOT SYNC (no actual IO
is performed on commit)
• 2 means DON’T FLUSH and DON’T SYNC (again no actual IO is
performed on commit)
INNODB_FLUSH_LOG_AT_TRX_COMMIT = ?
• When 0 or 2 is chosen, SYNC is performed once per second:
• Means you may lose up to 1s worth of committed data
• Once per second is not guaranteed*
• innodb_flush_log_at_trx_commit=1 is full durability
• Required for bank transactions and similar financial operations
• Many websites use 0 or 2 instead
• After all, MyISAM would lose up to 30s worth of data in case of crash and
it’s been default for many many years
• So then, 0 or 2?
• Small difference, because neither causes SYNC to disk
• 0 is good in that no data is lost if MySQL crashes (but the
machine stays ON)
* I’v seen SYNC delays due to mutex contention issues
5. SYNC_BINLOG – THAT’S FOR DURABLE BINLOG
• sync_binlog=1 makes binary logs durable
• Therefore:
• if you have no slaves & no backups - use sync_binlog=0 (default)
• if you do have replication and/or backups, but you don’t mind
losing few events (on slave or otherwise) in case of server power
loss in order to gain better performance, sync_binlog=0
• if consistency is really important and you also use
innodb_flush_log_at_trx_commit=1 anyway, do use sync_binlog=1
and make sure you run MySQL 5.6+, Percona Server 5.6+ or
MariaDB 5.5+ as these versions have a binlog group commit fix
(meaning that sync calls are grouped together)
6. INNODB_FLUSH_METHOD – AVOID DOUBLE BUFFERING
• Set innodb_flush_method=O_DIRECT
• Only supported in Linux
• Overcomes OS cache for reads and for writes
• If not used, double buffering occurs
• Bad because memory is wasted on storing the same data in
memory twice (InnoDB buffer pool and OS cache)
• Very few exceptions to this
• If you’re not sure, you’re not an exception.
7. INNODB_BUFFER_POOL_INSTANCES – REDUCE MUTEX CONTENTION
• Introduced in MySQL 5.5
• Reduces global buffer pool mutex contention
• splits buffer pool into multiple buffer pools
• On MySQL 5.5 (including variants) be more conservative:
• innodb_buffer_pool_instances=4 should be good enough
• On MySQL 5.6+ (including variants):
• innodb_buffer_pool_instances=8 or even 16 is a safe bet
8. INNODB_THREAD_CONCURRENCY – CONTROL YOUR THREADS
• innodb_thread_concurrency=0 is default and often used in
benchmarks, but
• with high workloads, setting a limit may work much better
• How does it work? I wish I could show you, but let me at
least describe it.
• next slide, please
INNODB CONCURRENCY MECHANISM
• innodb_thread_concurrency controls how many threads can be
executing in parallel
• if 0, all requests will be served immediately
• that’s fine if you have 32 CPU cores and 4 requests
• not so if you have 32 CPU intensive requests and 4 CPU cores
• can become a mess when all are executing at the same time and new
requests keep coming in
• number > 0 caps the number of threads that are executing at
the same time
• But it’s not a simple FIFO queue
• Each request is given a certain number of tickets
• 500 by default on MySQL 5.5 and earlier
• 5000 by default on MySQL 5.6 and newer
INNODB CONCURRENCY MECHANISM (2)
• A thread waits in the queue for a slot to become available
• once it starts executing, it starts using the tickets
• one ticket is used for every row read, insert, update, etc.
• when all tickets are used, a thread is sent to the back of the
queue and the cycle repeats until operation is complete
• The advantage is that long running queries don’t prevent
quick queries from ever getting into the queue, even if that
does prolong their execution.
• And of course that there’s no fighting for resources
INNODB CONCURRENCY CONTROLS
• innodb_concurrency_tickets determines the number of tickets
given
• increase for long queries to run longer before letting others in
• innodb_thread_sleep_delay sets amount of sleep time before
joining the innodb queue (in microseconds)
• innodb_thread_concurrency sets how many slots are available for
execution
• use if you have spikes or hardware is often saturated
• set to 8 and go up until you see good hardware utilization
• it is a dynamic variable:
• set global innodb_thread_concurrency=8;
MONITORING THE QUEUE
• You can see the number of requests queued up and executing
in the “show engine innodb statusG” output.
• Look for something like:
22 queries inside InnoDB, 104 queries in queue
9. SKIP_NAME_RESOLVE – DO SKIP THAT REVERSE IP LOOKUP
• Add skip_name_resolve to avoid DNS resolution on connect
• No impact when all is working fine
• When DNS server fails, takes a long time to figure things out
• slow connections due to DNS failure don’t help to solve this
faster
• Exception: local hosts file based names
HOW ABOUT A DEEP BREATH
and a quick stretch ?
BETTER?
OK, LET’S CONTINUE
10. INNODB_IO_CAPACITY(_MAX)? – CAP INNODB IO USAGE
• innodb_io_capacity controls how many write IO requests per
second (IOPS) will MySQL issue when flushing the dirty data
• innodb_io_capacity_max controls how many write IOPS will
MySQL issue flushing the dirty data when it’s under stress
• IO activity related to background writes only
• Under stress means MySQL is behind with flushing activity and
needs to shift gears or things may go bad
• Set innodb_io_capacity to 50-75% of write capacity
• and innodb_io_capacity_max to 100% or close
• write capacity = number of random write iops server can handle
(more on it here)
11. INNODB_STATS_ON_METADATA – TURN THEM OFF!
• OFF by default on MySQL 5.6 and 5.7
• Safe to switch OFF on MySQL 5.5 and 5.1 too, so in my.cnf:
• innodb_stats_on_metadata = 0
• can be changed online too
• Makes “show table status” and some queries against
INFORMATION_SCHEMA faster.
• InnoDB stats are still going to be updated, don’t worry!
12. INNODB_BUFFER_POOL_(DUMP_AT_SHUTDOWN|LOAD_AT_STARTUP)
• Makes warm-up much faster
• by loading contents of buffer pool on server startup
• works even with SSDs
• Only page reference numbers are stored in a file
• Asynchronous activity, no direct performance impact
• innodb_buffer_pool_dump_pct in MySQL 5.7 to control how
much of the buffer pool to dump (25 by default, I recommend
75-100)
• Supported in MySQL since 5.6, in Percona Server - since 5.1
13. INNODB_ADAPTIVE_HASH_INDEX_PARTS – SPLIT THE AHI MUTEX
• Adaptive Hash Index is ON by default
• Dynamic Hash index maintained by InnoDB to improve
certain query patterns. Usually very helpful
• except when requests for mutex start backfiring
• Starting with MySQL 5.7 mutex can be split (and is by default
split into 8 partitions), i.e.
• innodb_adaptive_hash_index_parts=8
• Using MySQL 5.6 and earlier?
• Switch to an appropriate Percona Server or MariaDB version and
use innodb_adaptive_hash_index_partitions
14. QUERY_CACHE_TYPE – ON? OFF? ON DEMAND?
• Before it becomes a bottle-neck, query cache is great with:
• small databases with few updates
• low concurrency workload
• read-only databases
• Even if it’s helpful, don’t exceed query_cache_size=256M
• wait time on invalidation increases significantly
• innocent queries get blocked
• With high concurrency workload, often a bottle-neck
• Domas Mituzas suggests using this query cache tuner (next
slide, please)
QUERY CACHE TUNING
• More seriously though, mutex is still locked even if
query_cache_size=0
• Use the following configuration:
• query_cache_size=0
• query_cache_type=OFF
• Requires a MySQL restart to disable mutex
• Works with MySQL 5.5 or newer
• If using MySQL 5.1, switch to Percona Server 5.1 to get the
same effect
15. INNODB_CHECKSUM_ALGORITHM – THE SECRET HARDWARE ACCELERATION TRICK
• Old checksum algorithm is expensive (CPU overhead)
• crc32 can use native CPU instructions. That’s a YUUGE win.
• Use innodb_checksum_algorithm=crc32 with MySQL 5.6
• used by default in MySQL 5.7
• Safe to change, no need to reload data
• Can be changed online in fact
16. TABLE_OPEN_CACHE_INSTANCES – IT’S THERE FOR A REASON
• Introduced in MySQL 5.6.6 to split open cache instances
• Avoid server lock-up when opening many tables or when
opening a table is slow
• Starting with MySQL 5.7.8, table_open_cache_instances=16
• Set this manually in MySQL 5.6
17. INNODB_(READ|WRITE)_IO_THREADS – LAST AND, YES, LEAST
• MySQL supports Asynchronous IO on Linux since MySQL
5.5, so this is not as important as it may seem.
• These threads are used for background activities only
• Set it to match number of bearing read/write disks
• e.g. innodb_read_io_threads=8 and innodb_write_io_threads=4 on
RAID10 with 8 disks.
• on SSDs, set at 32/16 (or similar) respectively
NOT THAT I DIDN’T MENTION IT, BUT
I have prepared a my.cnf with short handy descriptions near each
variable and appropriate links to learn more for your
convenience. Download it here:
http://www.speedemy.com/17
FINAL THOUGHTS
• Configuration file is only part of the deal
• More often than not, real the devil is in queries
• To learn about query optimization, register for a free webinar
on query optimization:
http://www.speedemy.com/webinars
END
Speedemy
Speedemy_com
Speedemy.com
P.S. I have prepared a video where I have discussed MySQL
configuration in great detail. It also contains an additional section
about reading MySQL Status Counters, in case you’re interested.
Register here to get the free video once post production is finished.

More Related Content

What's hot

MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
Jean-François Gagné
 
Troubleshooting PostgreSQL Streaming Replication
Troubleshooting PostgreSQL Streaming ReplicationTroubleshooting PostgreSQL Streaming Replication
Troubleshooting PostgreSQL Streaming Replication
Alexey Lesovsky
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
oysteing
 
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
Jesmar Cannao'
 
Solving PostgreSQL wicked problems
Solving PostgreSQL wicked problemsSolving PostgreSQL wicked problems
Solving PostgreSQL wicked problems
Alexander Korotkov
 
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQLTop 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
Jim Mlodgenski
 
The Full MySQL and MariaDB Parallel Replication Tutorial
The Full MySQL and MariaDB Parallel Replication TutorialThe Full MySQL and MariaDB Parallel Replication Tutorial
The Full MySQL and MariaDB Parallel Replication Tutorial
Jean-François Gagné
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
Altinity Ltd
 
How to set up orchestrator to manage thousands of MySQL servers
How to set up orchestrator to manage thousands of MySQL serversHow to set up orchestrator to manage thousands of MySQL servers
How to set up orchestrator to manage thousands of MySQL servers
Simon J Mudd
 
MySQL Performance Schema in 20 Minutes
 MySQL Performance Schema in 20 Minutes MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 Minutes
Sveta Smirnova
 
MAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19cMAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19c
Markus Michalewicz
 
What is new in PostgreSQL 14?
What is new in PostgreSQL 14?What is new in PostgreSQL 14?
What is new in PostgreSQL 14?
Mydbops
 
How to upgrade like a boss to MySQL 8.0 - PLE19
How to upgrade like a boss to MySQL 8.0 -  PLE19How to upgrade like a boss to MySQL 8.0 -  PLE19
How to upgrade like a boss to MySQL 8.0 - PLE19
Alkin Tezuysal
 
Sharding Methods for MongoDB
Sharding Methods for MongoDBSharding Methods for MongoDB
Sharding Methods for MongoDB
MongoDB
 
Percona Live 2022 - MySQL Architectures
Percona Live 2022 - MySQL ArchitecturesPercona Live 2022 - MySQL Architectures
Percona Live 2022 - MySQL Architectures
Frederic Descamps
 
MariaDB 마이그레이션 - 네오클로바
MariaDB 마이그레이션 - 네오클로바MariaDB 마이그레이션 - 네오클로바
MariaDB 마이그레이션 - 네오클로바
NeoClova
 
Linux tuning to improve PostgreSQL performance
Linux tuning to improve PostgreSQL performanceLinux tuning to improve PostgreSQL performance
Linux tuning to improve PostgreSQL performance
PostgreSQL-Consulting
 
OLTP+OLAP=HTAP
 OLTP+OLAP=HTAP OLTP+OLAP=HTAP
OLTP+OLAP=HTAP
EDB
 
Presto: SQL-on-anything
Presto: SQL-on-anythingPresto: SQL-on-anything
Presto: SQL-on-anything
DataWorks Summit
 
MySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptxMySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptx
NeoClova
 

What's hot (20)

MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
 
Troubleshooting PostgreSQL Streaming Replication
Troubleshooting PostgreSQL Streaming ReplicationTroubleshooting PostgreSQL Streaming Replication
Troubleshooting PostgreSQL Streaming Replication
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
 
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
 
Solving PostgreSQL wicked problems
Solving PostgreSQL wicked problemsSolving PostgreSQL wicked problems
Solving PostgreSQL wicked problems
 
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQLTop 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
 
The Full MySQL and MariaDB Parallel Replication Tutorial
The Full MySQL and MariaDB Parallel Replication TutorialThe Full MySQL and MariaDB Parallel Replication Tutorial
The Full MySQL and MariaDB Parallel Replication Tutorial
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
 
How to set up orchestrator to manage thousands of MySQL servers
How to set up orchestrator to manage thousands of MySQL serversHow to set up orchestrator to manage thousands of MySQL servers
How to set up orchestrator to manage thousands of MySQL servers
 
MySQL Performance Schema in 20 Minutes
 MySQL Performance Schema in 20 Minutes MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 Minutes
 
MAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19cMAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19c
 
What is new in PostgreSQL 14?
What is new in PostgreSQL 14?What is new in PostgreSQL 14?
What is new in PostgreSQL 14?
 
How to upgrade like a boss to MySQL 8.0 - PLE19
How to upgrade like a boss to MySQL 8.0 -  PLE19How to upgrade like a boss to MySQL 8.0 -  PLE19
How to upgrade like a boss to MySQL 8.0 - PLE19
 
Sharding Methods for MongoDB
Sharding Methods for MongoDBSharding Methods for MongoDB
Sharding Methods for MongoDB
 
Percona Live 2022 - MySQL Architectures
Percona Live 2022 - MySQL ArchitecturesPercona Live 2022 - MySQL Architectures
Percona Live 2022 - MySQL Architectures
 
MariaDB 마이그레이션 - 네오클로바
MariaDB 마이그레이션 - 네오클로바MariaDB 마이그레이션 - 네오클로바
MariaDB 마이그레이션 - 네오클로바
 
Linux tuning to improve PostgreSQL performance
Linux tuning to improve PostgreSQL performanceLinux tuning to improve PostgreSQL performance
Linux tuning to improve PostgreSQL performance
 
OLTP+OLAP=HTAP
 OLTP+OLAP=HTAP OLTP+OLAP=HTAP
OLTP+OLAP=HTAP
 
Presto: SQL-on-anything
Presto: SQL-on-anythingPresto: SQL-on-anything
Presto: SQL-on-anything
 
MySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptxMySQL8.0_performance_schema.pptx
MySQL8.0_performance_schema.pptx
 

Similar to MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)

12 Things about Oracle WebLogic Server 12c
12 Things	 about Oracle WebLogic Server 12c12 Things	 about Oracle WebLogic Server 12c
12 Things about Oracle WebLogic Server 12c
Guatemala User Group
 
My sql 56_roadmap_april2012
My sql 56_roadmap_april2012My sql 56_roadmap_april2012
My sql 56_roadmap_april2012
sqlhjalp
 
Configuring Sage 500 for Performance
Configuring Sage 500 for PerformanceConfiguring Sage 500 for Performance
Configuring Sage 500 for Performance
RKLeSolutions
 
MySQL Performance Tuning at COSCUP 2014
MySQL Performance Tuning at COSCUP 2014MySQL Performance Tuning at COSCUP 2014
MySQL Performance Tuning at COSCUP 2014
Ryusuke Kajiyama
 
MySQL Strategy&Roadmap
MySQL Strategy&RoadmapMySQL Strategy&Roadmap
MySQL Strategy&Roadmap
slidethanks
 
My sqlstrategyroadmap
My sqlstrategyroadmapMy sqlstrategyroadmap
My sqlstrategyroadmap
slidethanks
 
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
Insight Technology, Inc.
 
MySQL 5.7 what's new
MySQL 5.7 what's newMySQL 5.7 what's new
MySQL 5.7 what's new
Ricky Setyawan
 
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Jaime Crespo
 
Percona Server for MySQL 8.0 @ Percona Live 2019
Percona Server for MySQL 8.0 @ Percona Live 2019Percona Server for MySQL 8.0 @ Percona Live 2019
Percona Server for MySQL 8.0 @ Percona Live 2019
Laurynas Biveinis
 
MySQL 5.7: What's New, Nov. 2015
MySQL 5.7: What's New, Nov. 2015MySQL 5.7: What's New, Nov. 2015
MySQL 5.7: What's New, Nov. 2015
Mario Beck
 
Disaster Recovery Site Implementation with MySQL
Disaster Recovery Site Implementation with MySQLDisaster Recovery Site Implementation with MySQL
Disaster Recovery Site Implementation with MySQL
Syed Jahanzaib Bin Hassan - JBH Syed
 
How to upgrade like a boss to my sql 8.0?
How to upgrade like a boss to my sql 8.0?How to upgrade like a boss to my sql 8.0?
How to upgrade like a boss to my sql 8.0?
Alkin Tezuysal
 
MySQL 5.6 - Operations and Diagnostics Improvements
MySQL 5.6 - Operations and Diagnostics ImprovementsMySQL 5.6 - Operations and Diagnostics Improvements
MySQL 5.6 - Operations and Diagnostics Improvements
Morgan Tocker
 
Pre and post tips to installing sql server correctly
Pre and post tips to installing sql server correctlyPre and post tips to installing sql server correctly
Pre and post tips to installing sql server correctly
Antonios Chatzipavlis
 
Running MySQL on Linux
Running MySQL on LinuxRunning MySQL on Linux
Running MySQL on Linux
Great Wide Open
 
Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016
Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016
Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016
Dylan Butler
 
2010 12 mysql_clusteroverview
2010 12 mysql_clusteroverview2010 12 mysql_clusteroverview
2010 12 mysql_clusteroverview
Dimas Prasetyo
 
My sql 56_roadmap_april2012_zht2
My sql 56_roadmap_april2012_zht2My sql 56_roadmap_april2012_zht2
My sql 56_roadmap_april2012_zht2
Ivan Tu
 
MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012
Colin Charles
 

Similar to MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7) (20)

12 Things about Oracle WebLogic Server 12c
12 Things	 about Oracle WebLogic Server 12c12 Things	 about Oracle WebLogic Server 12c
12 Things about Oracle WebLogic Server 12c
 
My sql 56_roadmap_april2012
My sql 56_roadmap_april2012My sql 56_roadmap_april2012
My sql 56_roadmap_april2012
 
Configuring Sage 500 for Performance
Configuring Sage 500 for PerformanceConfiguring Sage 500 for Performance
Configuring Sage 500 for Performance
 
MySQL Performance Tuning at COSCUP 2014
MySQL Performance Tuning at COSCUP 2014MySQL Performance Tuning at COSCUP 2014
MySQL Performance Tuning at COSCUP 2014
 
MySQL Strategy&Roadmap
MySQL Strategy&RoadmapMySQL Strategy&Roadmap
MySQL Strategy&Roadmap
 
My sqlstrategyroadmap
My sqlstrategyroadmapMy sqlstrategyroadmap
My sqlstrategyroadmap
 
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
 
MySQL 5.7 what's new
MySQL 5.7 what's newMySQL 5.7 what's new
MySQL 5.7 what's new
 
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
 
Percona Server for MySQL 8.0 @ Percona Live 2019
Percona Server for MySQL 8.0 @ Percona Live 2019Percona Server for MySQL 8.0 @ Percona Live 2019
Percona Server for MySQL 8.0 @ Percona Live 2019
 
MySQL 5.7: What's New, Nov. 2015
MySQL 5.7: What's New, Nov. 2015MySQL 5.7: What's New, Nov. 2015
MySQL 5.7: What's New, Nov. 2015
 
Disaster Recovery Site Implementation with MySQL
Disaster Recovery Site Implementation with MySQLDisaster Recovery Site Implementation with MySQL
Disaster Recovery Site Implementation with MySQL
 
How to upgrade like a boss to my sql 8.0?
How to upgrade like a boss to my sql 8.0?How to upgrade like a boss to my sql 8.0?
How to upgrade like a boss to my sql 8.0?
 
MySQL 5.6 - Operations and Diagnostics Improvements
MySQL 5.6 - Operations and Diagnostics ImprovementsMySQL 5.6 - Operations and Diagnostics Improvements
MySQL 5.6 - Operations and Diagnostics Improvements
 
Pre and post tips to installing sql server correctly
Pre and post tips to installing sql server correctlyPre and post tips to installing sql server correctly
Pre and post tips to installing sql server correctly
 
Running MySQL on Linux
Running MySQL on LinuxRunning MySQL on Linux
Running MySQL on Linux
 
Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016
Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016
Transitioning From SQL Server to MySQL - Presentation from Percona Live 2016
 
2010 12 mysql_clusteroverview
2010 12 mysql_clusteroverview2010 12 mysql_clusteroverview
2010 12 mysql_clusteroverview
 
My sql 56_roadmap_april2012_zht2
My sql 56_roadmap_april2012_zht2My sql 56_roadmap_april2012_zht2
My sql 56_roadmap_april2012_zht2
 
MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012
 

Recently uploaded

20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
Pixlogix Infotech
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Neo4j
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Zilliz
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
ControlCase
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
名前 です男
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Neo4j
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
SOFTTECHHUB
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Edge AI and Vision Alliance
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
DianaGray10
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 

Recently uploaded (20)

20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 

MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)

  • 1. MYSQL PERFORMANCE TUNING I. MySQL Configuration (incl. MySQL 5.7) Aurimas Mikalauskas @ Speedemy
  • 2. AURIMAS MIKALAUSKAS • MySQL hacker since 1999 (yep, v3.23) • In love with performance & scalability since 2004 Danga Interactive talk on LiveJournal’s Backend at LISA04 • Percona Senior Performance Consultant & Architect 2006-2015 • Performance Engineer at EstanteVirtual.com.br • Independent data performance consultant, instructor at Speedemy.com
  • 3. ON THE AGENDA FOR TODAY How to Choose The Right MySQL Distribution Essentials of MySQL Configuration Tuning 17 Key MySQL Settings for Best Performance
  • 5. COMMUNITY MYSQL • Community MySQL 5.1 • innodb plugin not enabled by default • very poor InnoDB scalability (by default) • Community MySQL 5.5 • still missing a number of performance improvements • stability issues with high amounts of writes • Community MySQL 5.6 • not many reasons to use alternative distro • still not perfectly stable with high end hardware in a write- intensive environment
  • 6. COMMUNITY MYSQL 5.7 • Current GA release • Rock solid • New features: • multi-source replication • JSON support • proper multi-threaded replication • online buffer pool resize • spatial data types for InnoDB • sys schema • etc. (see full list)
  • 7. PERCONA SERVER • Launched July 2008 • as Percona patches for MySQL • in reality earlier than that • Always up to date with upstream • i.e. Percona Server 5.1.73 = MySQL 5.1.73 + Percona code • also always backwards compatible • meaning no problem to keep switching between Percona Server and MySQL. No upgrade/downgrade scripts need to be used. • some exceptions to backwards compatibility in the past • none of these options were enabled by default • very clear when it’s the case • Makes it easy to try it out and switch back at any point
  • 8. PERCONA SERVER VERSION • Using MySQL 5.1? • definitely switch to Percona Server 5.1 • Using MySQL 5.5? Switch to Percona Server 5.5 if… • adaptive hash index is a bottle-neck • you want faster checksums (hardware accelerated) • you have write-intensive workload • you need PAM authentication, audit log, thread pool • Using MySQL 5.6? Switch to Percona Server 5.6 if… • you have very write-intensive workload and high end hardware • you need some of the features mentioned with 5.5 • MySQL 5.7? Use MySQL, Percona Server 5.7 coming soon.
  • 9. PERCONA SERVER KEYSTONE FEATURE: EXTENDED SLOW QUERY LOG # Time: 150811 12:15:06.959000 # User@Host: root[root] @ localhost [localhost] Id: 62591948 # Schema: Core_DB Last_errno: 0 Killed: 0 # Query_time: 2.411376 Lock_time: 0.002444 Rows_sent: 0 Rows_examined: 10811959 Rows_affected: 0 # Bytes_sent: 0 Tmp_tables: 1 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: No # Filesort: No Filesort_on_disk: No Merge_passes: 0 # InnoDB_trx_id: 20A40488F # InnoDB_IO_r_ops: 37 InnoDB_IO_r_bytes: 606208 InnoDB_IO_r_wait: 0.000225 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 1747 SELECT ... # Time: 150811 12:15:06 # User@Host: root[root] @ localhost [localhost] Id: 62591948 # Schema: Core_DB Last_errno: 1051 Killed: 0 # Query_time: 2.411376 Lock_time: 0.002444 Rows_sent: 0 Rows_examined: 10811959 SELECT ... -VS- microsecond TS
  • 10. EXTENDED SLOW QUERY LOG (2) # Time: 150811 12:15:06.959000 # User@Host: root[root] @ localhost [localhost] Id: 62591948 # Schema: Core_DB Last_errno: 0 Killed: 0 # Query_time: 2.411376 Lock_time: 0.002444 Rows_sent: 0 Rows_examined: 10811959 Rows_affected: 0 # Bytes_sent: 0 Tmp_tables: 1 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: No # Filesort: No Filesort_on_disk: No Merge_passes: 0 # InnoDB_trx_id: 20A40488F # InnoDB_IO_r_ops: 37 InnoDB_IO_r_bytes: 606208 InnoDB_IO_r_wait: 0.000225 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 1747 SELECT ... # Time: 150811 12:15:06 # User@Host: root[root] @ localhost [localhost] Id: 62591948 # Schema: Core_DB Last_errno: 1051 Killed: 0 # Query_time: 2.411376 Lock_time: 0.002444 Rows_sent: 0 Rows_examined: 10811959 SELECT ... -VS- query plan
  • 11. EXTENDED SLOW QUERY LOG (3) # Time: 150811 12:15:06.959000 # User@Host: root[root] @ localhost [localhost] Id: 62591948 # Schema: Core_DB Last_errno: 0 Killed: 0 # Query_time: 2.411376 Lock_time: 0.002444 Rows_sent: 0 Rows_examined: 10811959 Rows_affected: 0 # Bytes_sent: 0 Tmp_tables: 1 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: No # Filesort: No Filesort_on_disk: No Merge_passes: 0 # InnoDB_trx_id: 20A40488F # InnoDB_IO_r_ops: 37 InnoDB_IO_r_bytes: 606208 InnoDB_IO_r_wait: 0.000225 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 1747 SELECT ... # Time: 150811 12:15:06 # User@Host: root[root] @ localhost [localhost] Id: 62591948 # Schema: Core_DB Last_errno: 1051 Killed: 0 # Query_time: 2.411376 Lock_time: 0.002444 Rows_sent: 0 Rows_examined: 10811959 SELECT ... -VS- extra innodb info
  • 12. MARIADB • Made by Monty Widenius • creator of MySQL and MyISAM • Advertised as the good MySQL • as opposed to bad MySQL that’s being “killed” by Evil Corp Oracle • MariaDB 5.5 = MySQL 5.5 + XtraDB + MariaDB 5.3: • query optimizer improvements • multi-master replication • group commit fix • MariaDB 10 • real fork of MySQL 5.6 • may become “backwards” incompatible
  • 13. MARIADB 10 FEATURES • Parallel replication • Multi-source replication • Cassandra, Spider, TokuDB storage engines • and few others.
  • 14. SWITCH TO MARIADB? • Think twice before switching • No great advantages over MySQL 5.7 or Percona Server • And yet a number of performance improvements from 5.7 will be missed in MariaDB 10 • Some features may be appealing, if that’s your case - go for it • Backwards incompatibility is a bit worrying • full mysqldump->import should still be possible
  • 15. WEBSCALESQL 15 • Collaboration by a few heavy-duty MySQL users: • Alibaba, Facebook, Google, Linkedin & Twitter • MySQL 5.6 at its core • good stuff being back-ported from MySQL 5.7 • Has very special mission • is not meant to be general purpose MySQL Server • addresses needs of running MySQL at SCALE
  • 16. FEW WEBSCALESQL FEATURES 16 • Ability to specify millisecond timeouts • Super read-only mode • Ability to disable deadlock detection • Prefix index query optimization • Performance Schema not compiled in • InnoDB flushing performance fixes read this article by Laurynas @ Percona to learn more.
  • 17. SWITCH TO WEBSCALESQL? 17 • Think thrice • Only switch if you REALLY understand what it is • Great performance features that don’t hurt functionality will be ported to other variants (some already are)
  • 18. THE ESSENTIALS Of MySQL Configuration Tuning
  • 19. MYSQL DEFAULTS ARE POOR • Okay for development, not for busy production db • Good news: • MySQL 5.6 defaults are better than versions < 5.6 • MySQL 5.7 defaults are better than 5.6 • MySQL is super easy to configure
  • 20. MYSQL CONFIGURATION FILE • One configuration file (unless it contains include directives) • Location: • /etc/my.cnf - most systems • /etc/mysql/my.cnf - Debian style • On Windows - best use data directory (create the file)
  • 21. COMMON MISTAKES • Using Trial and Error approach • change something, see if it feels better • don’t do it • Asking Google for performance advice • answers often lack context • use case could be benchmarks - often non-production suitable • settings are hardware (or else) dependant • Obsessing about fine-tuning the my.cnf • 10-15 variables is often all you need to change • fine-tuning won’t give you significant wins
  • 22. COMMON MISTAKES (2) • Changing many things at once • makes it very hard to figure out what caused which effect • instead, change one thing, then measure impact • Not keeping my.cnf in sync with the changes you make • changing settings online is convenient, but… • don’t forget to update my.cnf • Redundant entries in my.cnf • MySQL won’t mind them • Last value will be used • “-” and “_” can be used interchangeably
  • 23. COMMON MISTAKES (3) • Multiplying buffer sizes • don’t do it • some buffers are local, some server-wide • few variables need to be increased after hardware upgrades • Using the wrong my.cnf section • [mysql], [client], [mysqld_safe] — all are incorrect choices • [mysqld] — put ALL of the server configuration here
  • 24. CHANGING CONFIGURATION ONLINE • Many things can be changed online (dynamically) • Even innodb_buffer_pool_size starting with MySQL 5.7 • Example of changing innodb_thread_concurrency online: mysql> show global variables like 'innodb%'; +-------------------------------------------+------------------------+ | Variable_name | Value | +-------------------------------------------+------------------------+ ... | innodb_thread_concurrency | 0 | ... +-------------------------------------------+------------------------+ 140 rows in set (0.02 sec) mysql> select @@global.innodb_thread_concurrency; +------------------------------------+ | @@global.innodb_thread_concurrency | +------------------------------------+ | 8 | +------------------------------------+ 1 row in set (0.00 sec) mysql> set global innodb_thread_concurrency = 8; Query OK, 0 rows affected (0.01 sec)
  • 25. GLOBAL -VS- LOCAL SCOPE • In many cases, you only want to change local session buffers and leave global configuration as is • (sort|join|read|read_rnd)_buffer_size are all good examples • So: • For a query that needs to sort a lot of data, before you run it: set sort_buffer_size = 64 * 1024 * 1024; • Some queries would find index merge intersect optimization harmful, so for such queries you can just: set optimizer_switch = 'index_merge_intersection=off';
  • 27. READY TO USE MY.CNF WITH ALL SETTINGS I have prepared a my.cnf with short handy descriptions near each variable and appropriate links to learn more for your convenience. Download it here: http://www.speedemy.com/17
  • 28. 1. DEFAULT_STORAGE_ENGINE – CHOOSE THE RIGHT ENGINE FIRST • Already on InnoDB? Skip to next variable • Otherwise, bear with me.
  • 29. STORAGE ENGINE WHAT? • MySQL uses pluggable storage engines since the beginning • Different storage engines store data in different ways • some may even store it in a remote server (e.g. Federated) • others may only pretend they are storing data (e.g. Blackhole) • Two most commonly used ones: • MyISAM • InnoDB • Few other famous SEs: • Blackhole • Archive • CSV • TokuDB
  • 30. WHAT’S WRONG WITH MYISAM • Non transactional: • no Atomicity • no Consistency • no Isolation • no Durability • Table level locks only • Not scalable when it comes to write intensive workload • with an exception of INSERT-only workload
  • 31. MEET INNODB • Shipped with MySQL since 2001 • Fully ACID transactional storage engine, designed to handle highly concurrent workload and scale. • Road to multi-core multi-disk systems was bumpy • handles high concurrency pretty well since MySQL 5.0.30 • is hard to beat in many ways nowadays • Default MySQL storage engine since version 5.5.5
  • 32. HERE’S STORAGE ENGINES YOU ARE USING • Run this on your server: mysql> SELECT engine, count(*) as TABLES, concat(round(sum(table_rows)/1000000,2),'M') rows, concat(round(sum(data_length)/(1024*1024*1024),2),'G') DATA, concat(round(sum(index_length)/(1024*1024*1024),2),'G') idx, concat(round(sum(data_length+index_length)/(1024*1024*1024),2),'G') total_size, round(sum(index_length)/sum(data_length),2) idxfrac FROM information_schema.TABLES WHERE table_schema not in ('mysql', 'performance_schema', 'information_schema') GROUP BY engine ORDER BY sum(data_length+index_length) DESC LIMIT 10; +--------+--------+---------+--------+--------+------------+---------+ | engine | TABLES | rows | DATA | idx | total_size | idxfrac | +--------+--------+---------+--------+--------+------------+---------+ | InnoDB | 181 | 457.58M | 92.34G | 54.58G | 146.92G | 0.59 | | MyISAM | 13 | 22.91M | 7.85G | 2.12G | 9.97G | 0.27 | +--------+--------+---------+--------+--------+------------+---------+ 2 rows in set (0.22 sec) * no need to rewrite this, it will be included in the ebook you will get at speedemy.com/17
  • 33. HERE’S MYISAM TABLES ON YOUR SERVER • Run this: mysql> SELECT concat(table_schema, '.', table_name) tbl, engine, concat(round(table_rows/1000000,2),'M') rows, concat(round(data_length/(1024*1024*1024),2),'G') DATA, concat(round(index_length/(1024*1024*1024),2),'G') idx, concat(round((data_length+index_length)/(1024*1024*1024),2),'G') total_size, round(index_length/data_length,2) idxfrac FROM information_schema.TABLES WHERE table_schema not in ('mysql', 'performance_schema', 'information_schema') AND engine = 'MyISAM' ORDER BY data_length+index_length DESC;
  • 34. CONVERTING TO INNODB • It’s not enough to change default-storage-engine • Tables need to be converted. One. By. One. • Simply converting largest tables is not enough: • if at least one table in a join is MyISAM, the entire query is using table level locks. So having even a small MyISAM table in a large join can be very bad for concurrency • Convert only when the server is configured for InnoDB properly
  • 35. CONVERTING EACH TABLE TO INNODB • This will give you a list of commands to run: SET @DB_NAME = 'your_database'; SELECT CONCAT('ALTER TABLE `', TABLE_NAME, '` ENGINE=InnoDB;') AS sql_statements FROM INFORMATION_SCHEMA.TABLES AS tb WHERE TABLE_SCHEMA = @DB_NAME AND ENGINE = 'MyISAM' AND TABLE_TYPE = 'BASE TABLE' ORDER BY TABLE_NAME DESC; • To use InnoDB even when MyISAM is specified as a storage engine, in Percona Server you can set this in my.cnf: enforce_storage_engine = InnoDB
  • 36. 2. INNODB_BUFFER_POOL_SIZE – GET THE BEST OUT OF YOUR MEMORY • Most important variable for InnoDB • InnoDB Buffer Pool is: • a cache for read data (these are stored in pages of 16kb) • also cache for indexes, modified (dirty) data • and place for many internal InnoDB structures
  • 37. QUITE A TYPICAL BUFFER POOL DISTRIBUTION
  • 38. INNODB_BUFFER_POOL_SIZE = ? • On a dedicated server, set to 80% of total memory • leaves room for other internal structures outside the buffer pool • query execution (session buffers) • OS cache (binary logs, relay logs, innodb transaction log, etc.) • OS memory structures • even on servers with 256-512GB of RAM • Avoid swapping at all costs! • swapping is not the same as reading data from disk • it’s much much worse
  • 39. INNODB_BUFFER_POOL_SIZE = ? • On a shared server: • check total size of db, maybe it all fits in, say, 50% of RAM? • don’t strive to make it perfect • again, avoid swapping at all costs (use vmstat 1 to check) • Number of random reads from disk into BP can be checked by: $ mysqladmin ext -ri1 | grep Innodb_buffer_pool_reads | Innodb_buffer_pool_reads | 1832098003 | | Innodb_buffer_pool_reads | 595 | | Innodb_buffer_pool_reads | 915 | | Innodb_buffer_pool_reads | 734 | | Innodb_buffer_pool_reads | 622 | | Innodb_buffer_pool_reads | 710 | | Innodb_buffer_pool_reads | 664 | ...
  • 40. CHANGING INNODB_BUFFER_POOL_SIZE • On MySQL 5.7, can be done online: mysql> set global innodb_buffer_pool_size = size_in_bytes; • Don’t forget to update my.cnf after this • MySQL log will go like: [Note] InnoDB: Resizing buffer pool from 134217728 to 21474836480. (unit=134217728) [Note] InnoDB: disabled adaptive hash index. [Note] InnoDB: buffer pool 0 : 159 chunks (1302369 blocks) were added. [Note] InnoDB: buffer pool 0 : hash tables were resized. [Note] InnoDB: Resized hash tables at lock_sys, adaptive hash index, dictionary. [Note] InnoDB: Completed to resize buffer pool from 134217728 to 21474836480. [Note] InnoDB: Re-enabled adaptive hash index. • On any previous version of MySQL: • set an appropriate innodb_buffer_pool_size in my.cnf • restart MySQL server
  • 41. 3. INNODB_LOG_FILE_SIZE – ROOM FOR MYSQL’S REDO LOG • Size for redo (a.k.a. transaction) logs • 5MB up until MySQL 5.6.8 — way too small for just about any workload • Current default of 48MB still too small for most • What’s innodb log file? • next slide, please
  • 42. WHAT IS INNODB REDO LOG? • Transactional storage engines (including InnoDB) need both undo and redo functions, similar to image editor • well not exactly; let me explain • Undo happens when rollback occurs • Implicitly or explicitly • Redo happens after a database crash • Also when preparing a backup and other similar cases
  • 43. UNDO • When record is changed (but before COMMIT) • changes are not written to data files right away • first they are written to transaction log • and data is modified in memory • original unmodified copy is moved to rollback segment • if Rollback occurs: • Undo needs to happen, to restore data to previous state • InnoDB removes the copy from the rollback segment (and copies it back if need be), removes the dirty page • and marks in transaction log that change was rolled back • That’s Undo.
  • 44. REDO • After COMMIT, changes are ready to be written to data files • but they aren’t actually written (yet). This would be inefficient. • Instead, changes are written to Redo log (and modified pages are only stored in memory) • If MySQL crashes (before changes are written to data files): • Redo needs to happen • else, these changes would be lost forever • After restart, InnoDB finds the last checkpoint position • and re-applies the same changes it had in memory before restart • It is a gross oversimplification, but yeah, that’s Redo.
  • 45. REDO LOG SIZE • Size matters: • Small log files make writes slower and crash recovery faster • Large log files make writes faster and crash recovery slower • Log file is like a buffer, so: • small log files = small buffer, so flushing needs to happen often • big log files = big buffer, hence flushing is more streamlined • But: • with large log files, in case of crash, more work needs to be done to restore the database to the consistent state (i.e. the Redo operation)
  • 46. SIZING REDO LOG • Rule of Thumb: • Check that total size of your Redo logs fits in 1-2h worth of writes during your busy period mysql> pager grep seq mysql> show engine innodb statusG select sleep(60); show engine innodb statusG Log sequence number 1777308180429 ... Log sequence number 1777354541591 mysql> nopager mysql> select (1777354541591-1777308180429)*60/1024/1024; +--------------------------------------------+ | (1777354541591-1777308180429)*60/1024/1024 | +--------------------------------------------+ | 2652.80696869 | +--------------------------------------------+ 1 row in set (0.00 sec) • Based on this 60s sample, InnoDB could be writing around 2.6GB per hour, so innodb_log_file_size=2560M sounds like a good start to get 5GB worth of redo logs total
  • 47. CHANGING THE REDO LOG SIZE • MySQL ≥ 5.6 - change my.cnf and restart the server. • MySQL < 5.6, my.cnf change is not enough: 1. change innodb_log_file_size in my.cnf 2. stop MySQL server 3. ensure MySQL had a clean shutdown (mysql log is your friend) 4. remove old log files, usually by running the following command: rm -f /var/lib/mysql/ib_logfile* 5. start MySQL server – it should take a bit longer to start because it is going to be creating new transaction log files • MySQL < 5.6.2 only supports 4GB total!
  • 48. 4. INNODB_FLUSH_LOG_AT_TRX_COMMIT – DURABLE OR NOT? THAT IS THE QUESTION! • innodb_flush_log_at_trx_commit=1 by default: • FLUSH and SYNC after EVERY transaction commit - full durability • INSERT/UPDATE/DELETE is a transaction if autocommit=1 (default behaviour) • SYNC is often expensive - it’s synchronisation to disks • Exception is if you have non-volatile cache • e.g. battery-backup unit (BBU) protected write-back cache • or Super-capacitor with a flash memory chip • Alternative values for innodb_flush_log_at_trx_commit: • 0 means FLUSH the buffers to OS, but DO NOT SYNC (no actual IO is performed on commit) • 2 means DON’T FLUSH and DON’T SYNC (again no actual IO is performed on commit)
  • 49. INNODB_FLUSH_LOG_AT_TRX_COMMIT = ? • When 0 or 2 is chosen, SYNC is performed once per second: • Means you may lose up to 1s worth of committed data • Once per second is not guaranteed* • innodb_flush_log_at_trx_commit=1 is full durability • Required for bank transactions and similar financial operations • Many websites use 0 or 2 instead • After all, MyISAM would lose up to 30s worth of data in case of crash and it’s been default for many many years • So then, 0 or 2? • Small difference, because neither causes SYNC to disk • 0 is good in that no data is lost if MySQL crashes (but the machine stays ON) * I’v seen SYNC delays due to mutex contention issues
  • 50. 5. SYNC_BINLOG – THAT’S FOR DURABLE BINLOG • sync_binlog=1 makes binary logs durable • Therefore: • if you have no slaves & no backups - use sync_binlog=0 (default) • if you do have replication and/or backups, but you don’t mind losing few events (on slave or otherwise) in case of server power loss in order to gain better performance, sync_binlog=0 • if consistency is really important and you also use innodb_flush_log_at_trx_commit=1 anyway, do use sync_binlog=1 and make sure you run MySQL 5.6+, Percona Server 5.6+ or MariaDB 5.5+ as these versions have a binlog group commit fix (meaning that sync calls are grouped together)
  • 51. 6. INNODB_FLUSH_METHOD – AVOID DOUBLE BUFFERING • Set innodb_flush_method=O_DIRECT • Only supported in Linux • Overcomes OS cache for reads and for writes • If not used, double buffering occurs • Bad because memory is wasted on storing the same data in memory twice (InnoDB buffer pool and OS cache) • Very few exceptions to this • If you’re not sure, you’re not an exception.
  • 52. 7. INNODB_BUFFER_POOL_INSTANCES – REDUCE MUTEX CONTENTION • Introduced in MySQL 5.5 • Reduces global buffer pool mutex contention • splits buffer pool into multiple buffer pools • On MySQL 5.5 (including variants) be more conservative: • innodb_buffer_pool_instances=4 should be good enough • On MySQL 5.6+ (including variants): • innodb_buffer_pool_instances=8 or even 16 is a safe bet
  • 53. 8. INNODB_THREAD_CONCURRENCY – CONTROL YOUR THREADS • innodb_thread_concurrency=0 is default and often used in benchmarks, but • with high workloads, setting a limit may work much better • How does it work? I wish I could show you, but let me at least describe it. • next slide, please
  • 54. INNODB CONCURRENCY MECHANISM • innodb_thread_concurrency controls how many threads can be executing in parallel • if 0, all requests will be served immediately • that’s fine if you have 32 CPU cores and 4 requests • not so if you have 32 CPU intensive requests and 4 CPU cores • can become a mess when all are executing at the same time and new requests keep coming in • number > 0 caps the number of threads that are executing at the same time • But it’s not a simple FIFO queue • Each request is given a certain number of tickets • 500 by default on MySQL 5.5 and earlier • 5000 by default on MySQL 5.6 and newer
  • 55. INNODB CONCURRENCY MECHANISM (2) • A thread waits in the queue for a slot to become available • once it starts executing, it starts using the tickets • one ticket is used for every row read, insert, update, etc. • when all tickets are used, a thread is sent to the back of the queue and the cycle repeats until operation is complete • The advantage is that long running queries don’t prevent quick queries from ever getting into the queue, even if that does prolong their execution. • And of course that there’s no fighting for resources
  • 56. INNODB CONCURRENCY CONTROLS • innodb_concurrency_tickets determines the number of tickets given • increase for long queries to run longer before letting others in • innodb_thread_sleep_delay sets amount of sleep time before joining the innodb queue (in microseconds) • innodb_thread_concurrency sets how many slots are available for execution • use if you have spikes or hardware is often saturated • set to 8 and go up until you see good hardware utilization • it is a dynamic variable: • set global innodb_thread_concurrency=8;
  • 57. MONITORING THE QUEUE • You can see the number of requests queued up and executing in the “show engine innodb statusG” output. • Look for something like: 22 queries inside InnoDB, 104 queries in queue
  • 58. 9. SKIP_NAME_RESOLVE – DO SKIP THAT REVERSE IP LOOKUP • Add skip_name_resolve to avoid DNS resolution on connect • No impact when all is working fine • When DNS server fails, takes a long time to figure things out • slow connections due to DNS failure don’t help to solve this faster • Exception: local hosts file based names
  • 59. HOW ABOUT A DEEP BREATH and a quick stretch ?
  • 61. 10. INNODB_IO_CAPACITY(_MAX)? – CAP INNODB IO USAGE • innodb_io_capacity controls how many write IO requests per second (IOPS) will MySQL issue when flushing the dirty data • innodb_io_capacity_max controls how many write IOPS will MySQL issue flushing the dirty data when it’s under stress • IO activity related to background writes only • Under stress means MySQL is behind with flushing activity and needs to shift gears or things may go bad • Set innodb_io_capacity to 50-75% of write capacity • and innodb_io_capacity_max to 100% or close • write capacity = number of random write iops server can handle (more on it here)
  • 62. 11. INNODB_STATS_ON_METADATA – TURN THEM OFF! • OFF by default on MySQL 5.6 and 5.7 • Safe to switch OFF on MySQL 5.5 and 5.1 too, so in my.cnf: • innodb_stats_on_metadata = 0 • can be changed online too • Makes “show table status” and some queries against INFORMATION_SCHEMA faster. • InnoDB stats are still going to be updated, don’t worry!
  • 63. 12. INNODB_BUFFER_POOL_(DUMP_AT_SHUTDOWN|LOAD_AT_STARTUP) • Makes warm-up much faster • by loading contents of buffer pool on server startup • works even with SSDs • Only page reference numbers are stored in a file • Asynchronous activity, no direct performance impact • innodb_buffer_pool_dump_pct in MySQL 5.7 to control how much of the buffer pool to dump (25 by default, I recommend 75-100) • Supported in MySQL since 5.6, in Percona Server - since 5.1
  • 64. 13. INNODB_ADAPTIVE_HASH_INDEX_PARTS – SPLIT THE AHI MUTEX • Adaptive Hash Index is ON by default • Dynamic Hash index maintained by InnoDB to improve certain query patterns. Usually very helpful • except when requests for mutex start backfiring • Starting with MySQL 5.7 mutex can be split (and is by default split into 8 partitions), i.e. • innodb_adaptive_hash_index_parts=8 • Using MySQL 5.6 and earlier? • Switch to an appropriate Percona Server or MariaDB version and use innodb_adaptive_hash_index_partitions
  • 65. 14. QUERY_CACHE_TYPE – ON? OFF? ON DEMAND? • Before it becomes a bottle-neck, query cache is great with: • small databases with few updates • low concurrency workload • read-only databases • Even if it’s helpful, don’t exceed query_cache_size=256M • wait time on invalidation increases significantly • innocent queries get blocked • With high concurrency workload, often a bottle-neck • Domas Mituzas suggests using this query cache tuner (next slide, please)
  • 66.
  • 67. QUERY CACHE TUNING • More seriously though, mutex is still locked even if query_cache_size=0 • Use the following configuration: • query_cache_size=0 • query_cache_type=OFF • Requires a MySQL restart to disable mutex • Works with MySQL 5.5 or newer • If using MySQL 5.1, switch to Percona Server 5.1 to get the same effect
  • 68. 15. INNODB_CHECKSUM_ALGORITHM – THE SECRET HARDWARE ACCELERATION TRICK • Old checksum algorithm is expensive (CPU overhead) • crc32 can use native CPU instructions. That’s a YUUGE win. • Use innodb_checksum_algorithm=crc32 with MySQL 5.6 • used by default in MySQL 5.7 • Safe to change, no need to reload data • Can be changed online in fact
  • 69. 16. TABLE_OPEN_CACHE_INSTANCES – IT’S THERE FOR A REASON • Introduced in MySQL 5.6.6 to split open cache instances • Avoid server lock-up when opening many tables or when opening a table is slow • Starting with MySQL 5.7.8, table_open_cache_instances=16 • Set this manually in MySQL 5.6
  • 70. 17. INNODB_(READ|WRITE)_IO_THREADS – LAST AND, YES, LEAST • MySQL supports Asynchronous IO on Linux since MySQL 5.5, so this is not as important as it may seem. • These threads are used for background activities only • Set it to match number of bearing read/write disks • e.g. innodb_read_io_threads=8 and innodb_write_io_threads=4 on RAID10 with 8 disks. • on SSDs, set at 32/16 (or similar) respectively
  • 71. NOT THAT I DIDN’T MENTION IT, BUT I have prepared a my.cnf with short handy descriptions near each variable and appropriate links to learn more for your convenience. Download it here: http://www.speedemy.com/17
  • 72. FINAL THOUGHTS • Configuration file is only part of the deal • More often than not, real the devil is in queries • To learn about query optimization, register for a free webinar on query optimization: http://www.speedemy.com/webinars
  • 74. P.S. I have prepared a video where I have discussed MySQL configuration in great detail. It also contains an additional section about reading MySQL Status Counters, in case you’re interested. Register here to get the free video once post production is finished.