MariaDB Performance Tuning Crash Course

krzysztof@severalnines.com
Copyright 2018 Severalnines AB
Presenter
Krzysztof Książek, Senior Support Engineer @Severalnines
MariaDB Performance Tuning
7th November 2018

•Tuning process - how to make sure you make correct changes
•Configuration tuning for MariaDB
•InnoDB internals and contentions
Agenda

Tuning Process

•Never-ending story which starts once you install MariaDB on the host
•You have to tune for a specific workload
•Workload may change in time
More data can make it I/O-bound
Different query mix may increase CPU load and put stress on different parts of the
InnoDB
•Keep in mind that configuration tuning is not likely to give you a huge increase in
performance (except if the server is really badly configured)
•Make sure you understand why a configuration change ended up with a given result
Tuning process

Tuning process

•You need a deterministic, test environment to make sure you can measure the impact of the
changes
•Environment should mirror production as close as possible, to make it more relevant
•Changes should be introduced one at a time to ensure you understand the impact of each of
them
•Benchmark the system using queries as close to production as possible
•Restore it to the original state for another round of tweaking
•Rinse and repeat until you are happy with results
Tuning process

•Grab a backup of your production systems
•Restore it on a host, restart MariaDB or reboot the host itself to clear caches
•Capture real-world queries using slow log or tcpdump
•Do a baseline run, replay queries using Percona Playback or pt-upgrade
•Restore backup again, restart MariaDB or reboot the host itself to clear caches
•Make _one_ change in my.cnf or OS settings
•Replay queries using Percona Playback or pt-upgrade
•Measure the difference, repeat the process by restoring the backup if you want to make one
more change
Tuning process

Tuning MariaDB configuration

Disable Query Cache
Default 10.3 settings
Avg 17987.648 QPS
Default 10.3 settings
Query Cache disabled
Avg 29299.84 QPS
•Query cache - optimize it away by disabling
Use external caching layer (ProxySQL, Redis, Memcached)
All tests were done on r5d.4xlarge

•InnoDB buffer pool - used to cache data
and store dirty pages
•More is better but you need to leave some
memory for other buffers
Per join buffers
Per session buffers
Temporary tables
•You may have heard about 80% rule
It’s more like 90% for large (i.e. 128GB)
hosts
Tuning MariaDB configuration - memory
•Make sure you err on the side of ‘too small’
•Unless you run recent MariaDB (10.2 and
up) where you can resize InnoDB buffer
pool dynamically, without restart
•For fairly loaded (~20-30 running threads)
host with 128GB of memory it should be ok
to leave ~15GB of memory free
•All depends on the workload so your
mileage may vary

•Per-session buffers in InnoDB:
sort_buffer_size, read_buffer_size, read_rnd_buffer_size
•Per-join buffer: join_buffer_size
•By default - small values
•More _not_ always better
At 256KB the way how memory allocates change
smaller chunks use malloc() which is faster than mmap()
•Make sure to benchmark your system after any change to those settings
Tuning MariaDB configuration - memory

Buffers
innodb_buffer_pool_size = 100G
innodb_buffer_pool_instances = 16
innodb_log_file_size = 4G
Avg 169621.54 QPS

Buffers
join_buffer_size = 128M
read_buffer_size = 128M
read_rnd_buffer_size = 128M
Avg 169018.30 QPS

Buffers
Avg 169340.18 QPS

Buffers - sysbench-tpcc
Avg 66324.898 QPS
Avg 64663.332 QPS
Avg 60105.784 QPS

•innodb_flush_log_at_trx_commit - governs the durability in InnoDB
1 - full ACID compliance
2 - you may lose up to 1s of transactions when hardware crashes
0 - you may lose up to 1s of transactions when MariaDB crashes
•Significant change in the I/O performance - less flushes means less I/O and less overhead
•Pick whatever you like and whatever you need
Slaves may not require full durability if you have many of them
Galera Cluster nodes may also not require full durability
Tuning MariaDB configuration - I/O performance

•innodb_io_capacity, innodb_io_capacity_max and innodb_lru_scan_depth - define number
of disk operations InnoDB can execute
•Set it too low and you may not fully utilize your hardware
•More not always better - aggressive flushing is not always the best option
Redo logs are there for a reason - to minimize number of writes to tablespaces
•innodb_flush_method:
O_DIRECT for BBU-backed hardware
O_DSYNC may work better with SAN
Benchmark your setup before you go live

I/O settings
innodb_buffer_pool_instances=4
innodb_log_file_size = 128M
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT
Avg 35504.52 QPS

I/O settings
Avg 60321.126 QPS

innodb_flush_method = O_DSYNC
Avg 59452.728 QPS

I/O settings
innodb_flush_method = fsync (default)
Avg 54398.378 QPS

I/O settings
innodb_io_capacity_max = 16000
Avg 42521.23 QPS

•InnoDB Redo Logs are used to store write transactions and they are written sequentially
•MariaDB must not run out of space in them
•Larger logs help with better write merging
•Larger logs help with more stable flushing
•Larger logs may seriously impact recovery time in case of a crash
•The rule of thumb is to make them large enough to store at least 1h of writes

•max_connections - keep it large enough to handle incoming connections
•If you need to handle thousands of connections, check the connection pooling options or a
proxy, ideally with connection multiplexing (ProxySQL)
•log_bin - you want to have binlogs enabled
Consider sync_binlog=1 Less performance, more durability
•skip_name_resolve - just to make sure your database won’t suffer when DNS will not be
reachable
Tuning MariaDB configuration

InnoDB Internals

InnoDB Internals
MariaDB [(none)]> select * from performance_schema.events_waits_summary_global_by_event_name WHERE
EVENT_NAME like '%mutex%' and count_star > 0 ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;
+-------------------------------------------+------------+----------------+----------------+----------------+----------------+
| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | MIN_TIMER_WAIT | AVG_TIMER_WAIT | MAX_TIMER_WAIT |
+-------------------------------------------+------------+----------------+----------------+----------------+----------------+
| wait/synch/mutex/sql/THD::LOCK_thd_data | 2508482 | 699876746415 | 17325 | 278740 | 129770289110 |
| wait/synch/mutex/innodb/buf_pool_mutex | 877021 | 357991729095 | 17325 | 408100 | 21343581105 |
| wait/synch/mutex/sql/THD::LOCK_thd_kill | 585288 | 255054289875 | 17325 | 435435 | 62933799005 |
| wait/synch/mutex/sql/LOCK_table_cache | 1170607 | 209109547570 | 17325 | 178255 | 24837530795 |
| wait/synch/mutex/innodb/fil_system_mutex | 625176 | 128177721980 | 17325 | 204820 | 24410468390 |
| wait/synch/mutex/innodb/srv_sys_mutex | 3095 | 32831559145 | 18480 | 10607905 | 31612420455 |
| wait/synch/mutex/innodb/dict_sys_mutex | 405 | 30690657305 | 19635 | 75779165 | 5380793495 |
| wait/synch/mutex/mysys/BITMAP::mutex | 83582 | 19281719380 | 21175 | 230615 | 6739930890 |
| wait/synch/mutex/innodb/srv_threads_mutex | 36193 | 17454819690 | 18480 | 482020 | 9705892735 |
| wait/synch/mutex/innodb/log_sys_mutex | 155877 | 13870954790 | 16170 | 88935 | 4442297475 |
+-------------------------------------------+------------+----------------+----------------+----------------+----------------+
10 rows in set (0.004 sec)
performance_schema=ON
performance-schema-instrument='%=ON'

InnoDB Internals
root@vagrant:~# for mutex in $(mysql -e "SHOW ENGINE INNODB MUTEXG" | grep Name
| cut -d : -f 3,4 | sort | uniq) ; do cnt=$(mysql -e "SHOW ENGINE INNODB MUTEX;" | grep
${mutex} | cut -d = -f 2 | cut -d ' ' -f 1 | paste -sd+ | bc) ; echo "${mutex}: ${cnt}" ; done;
btr0sea.cc:243:66226
buf0buf.cc:1638:67368
dict0dict.cc:2461:1259
fil0fil.cc:1475:1751
hash0hash.cc:189:4284
ibuf0ibuf.cc:568:10
log0log.cc:644:154
trx0purge.cc:178:1

•Once you get the output, you can consult source code for your given MariaDB version
Find the lock, understand the context it is located
Decide if there is an option to improve
•In our case, btr0sea.cc:243 points towards the adaptive hash index. Maybe some tuning will
reduce the locking?
InnoDB Internals

•innodb_buffer_pool_instances, table_open_cache_instances
•metadata_locks_hash_instances, innodb_adaptive_hash_index_partitions
•Those options can help you to reduce contention on some of those structures
•Increase number of buffer pools or adaptive hash index partitions if you notice a congestion
on them
•Or, preemptively, if you have to handle highly concurrent traffic
•Don’t use buffer pool instances smaller than 1GB (use 2GB+, too many small instances can
slow down the system)
InnoDB Internals

Summary

•Make sure you approach the tuning with a correct process
It requires patience
You should understand the results before making another change
•Think before you act
What workload I have?
What is the bottleneck that I’m facing?
•Proper trending system is a great help
•Don’t forget about other areas to improve
SQL, index hints, optimizer switches
Summary

Thank You!
Contact: krzysztof@severalnines.com
Q&A

MariaDB Performance Tuning Crash Course

More Related Content

What's hot

Similar to MariaDB Performance Tuning Crash Course

More from Severalnines

Recently uploaded

MariaDB Performance Tuning Crash Course