Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning

2. Copyright 2018 Severalnines AB I'm JJ from the Severalnines Team and I'm your host for today's webinar! Feel free to ask any questions in the Questions section of this application or via the Chat box. You can also contact me directly via the chat box or via email: jj@severalnines.com during or after the webinar. Your host & some logistics

4. Copyright 2017 Severalnines AB Automation & Management Deployment # Deploy a Cluster in Minutes # On-Prem or Cloud (AWS/Azure/Google)  Monitoring # Systems View with 1 sec Resolution # DB / OS stats & Performance Advisors # Configurable Dashboards # Query Analyzer # Real-time / historical Management # Backup Management # Upgrades & Patching # Security & Compliance # Operational Reports # Automatic Recovery & Repair # Performance Management # Automatic Performance Advisors

8. Copyright 2018 Severalnines AB •What to tune and why? •Tuning process •Operating system tuning Memory I/O performance Agenda •MySQL configuration tuning Memory I/O performance •Useful tools •Do’s and do not’s of MySQL tuning •What’s new in MySQL 8.0?

9. Copyright 2018 Severalnines AB •Operating system Swapping I/O scheduler CPU scheduler TCP/IP stack Configuration tuning - what to tune and why? MySQL configuration •Per-session buffers •InnoDB Buffer Pool •InnoDB I/O settings

11. Copyright 2018 Severalnines AB •Never-ending story which starts once you install MySQL on the host •You have to tune for a specific workload •Workload may change in time More data can make it I/O-bound Different query mix may increase CPU load and put stress on different parts of the InnoDB •Keep in mind that configuration tuning is not likely to give you a huge increase in performance (except if the server is really badly configured) Tuning process

13. Copyright 2018 Severalnines AB •You need a deterministic, test environment to make sure you can measure the impact of the changes •Environment should mirror production as close as possible, to make it more relevant •Changes should be introduced one at a time to ensure you understand the impact of each of them •Benchmark the system using queries as close to production as possible •Restore it to the original state for another round of tweaking •Rinse and repeat until you are happy with results Tuning process

14. Copyright 2018 Severalnines AB •Grab a backup of your production systems •Restore it on a host, restart MySQL or reboot the host itself to clear caches •Capture real-world queries using slow log or tcpdump •Do a baseline run, replay queries using Percona Playback or pt-upgrade •Restore backup again, restart MySQL or reboot the host itself to clear caches •Make _one_ change in my.cnf or OS settings •Replay queries using Percona Playback or pt-upgrade •Measure the difference, repeat the process by restoring the backup if you want to make one more change Tuning process

16. Copyright 2018 Severalnines AB •Minimize the flushing operations in OS to ensure no stalls are happening when dirty data is flushed to disk vm.dirty_ratio - a hard limit of the memory that can be used to cache dirty disk pages vm.dirty_background_ratio - percentage of system memory that can be used to cache modified (“dirty”) pages before the background flush process kicks in •We want them to be set to low numbers - 5 - 10% to trigger flushing often and make it quick •Default settings may result in a situation where 40% of your memory has to be flushed to disk at once For large instances (128GB) it can be even >50GB of data to flush Tuning the operating system for MySQL - memory

17. Copyright 2018 Severalnines AB •Swapping - a process of moving data from memory to disk •MySQL doesn’t like active swapping - we want data in memory, not on disk but we even more don’t want to be killed when system runs out of memory •vm.swappiness = 1 - starting from kernel 3.5 this is the setting you want to use - allow some swapping but only when there’s no other option •You can also deprioritize MySQL when it comes to OOM Killer: echo -1000 > /proc/[pid]/oom_score_adj should help in environments shared with other processes Tuning the operating system for MySQL - memory

18. Copyright 2018 Severalnines AB •When running MySQL on bare iron, you need to make sure you have NUMA interleave policy set •Starting from 5.6.27 a variable is available: innodb_numa_interleave •Before that you could run MySQL as: numactl --interleave=all $command •Memory allocation library may impact performance of MySQL Many choices: glibc malloc, tcmalloc and jemalloc Use LD_PRELOAD or malloc-lib variable in [mysqld_safe] section of my.cnf Tuning the operating system for MySQL - memory

19. Copyright 2018 Severalnines AB •Increase buffers: net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 # Interface buffering net.core.netdev_max_backlog = 2500 Tuning the operating system for MySQL - network •Increase connection backlog: net.ipv4.tcp_max_syn_backlog = 4096 net.core.somaxconn = 1024 •Reduce timeouts: net.ipv4.tcp_syn_retries = 0 net.ipv4.tcp_synack_retries = 0 net.ipv4.tcp_keepalive_time = 30 net.ipv4.tcp_keepalive_intvl = 1 net.ipv4.tcp_keepalive_probes = 2

20. Copyright 2018 Severalnines AB •Disk schedulers may impact performance of I/O-bound workloads •CFQ is tuned for desktop workloads •Noop or deadline are usually better for MySQL Benchmark to see which one of them will work better for you •Filesystem - most often you’ll use EXT4 or XFS Performance depends on workload, hardware and kernel version Benchmark to see which one will work better in your environment Tuning the operating system for MySQL - I/O

22. Copyright 2018 Severalnines AB •InnoDB buffer pool - used to cache data and store dirty pages •More is better but you need to leave some memory for other buffers Per join buffers Per session buffers Temporary tables •You may have heard about 80% rule It’s more like 90% for large (i.e. 128GB) hosts Tuning MySQL configuration - memory •Make sure you err on the side of ‘too small’ •Unless you run MySQL 5.7 or 8.0 where you can resize InnoDB buffer pool dynamically, without restart •For fairly loaded (~20-30 running threads) host with 128GB of memory it should be ok to leave ~15GB of memory free •All depends on the workload so your mileage may vary

23. Copyright 2018 Severalnines AB •Per-session buffers in InnoDB: sort_buffer_size, read_buffer_size, read_rnd_buffer_size •Per-join buffer: join_buffer_size •By default - small values •More _not_ always better At 256KB the way how memory allocates change smaller chunks use malloc() which is faster than mmap() •Make sure to benchmark your system after any change to those settings Tuning MySQL configuration - memory

24. Copyright 2018 Severalnines AB •innodb_flush_log_at_trx_commit - governs the durability in InnoDB 1 - full ACID compliance 2 - you may lose up to 1s of transactions when hardware crashes 0 - you may lose up to 1s of transactions when MySQL crashes •Significant change in the I/O performance - less flushes means less I/O and less overhead •Pick whatever you like and whatever you need Slaves may not require full durability if you have many of them Galera Cluster nodes may also not require full durability Tuning MySQL configuration - I/O performance

25. Copyright 2018 Severalnines AB •innodb_io_capacity, innodb_io_capacity_max and innodb_lru_scan_depth - define number of disk operations InnoDB can execute •Set it too low and you may not fully utilize your hardware •More not always better - aggressive flushing is not always the best option Redo logs are there for a reason - to minimize number of writes to tablespaces •innodb_flush_method: O_DIRECT for BBU-backed hardware O_DSYNC may work better with SAN Benchmark your setup before you go live Tuning MySQL configuration - I/O performance

26. Copyright 2018 Severalnines AB •InnoDB Redo Logs are used to store write transactions and they are written sequentially •MySQL must not run out of space in them •Larger logs help with better write merging •Larger logs help with more stable flushing •Larger logs may seriously impact recovery time in case of a crash •The rule of thumb is to make them large enough to store at least 1h of writes Tuning MySQL configuration - I/O performance

28. Copyright 2018 Severalnines AB •innodb_buffer_pool_instances, table_open_cache_instances •metadata_locks_hash_instances, innodb_adaptive_hash_index_partitions •Those options can help you to reduce contention on some of those structures •Increase number of buffer pools or adaptive hash index partitions if you notice a congestion on them •Or, preemptively, if you have to handle highly concurrent traffic •Don’t use buffer pool instances smaller than 1GB (use 2GB+, more instances can slow down the system) Tuning MySQL configuration - contentions

29. Copyright 2018 Severalnines AB •max_connections - keep it large enough to handle incoming connections •If you need to handle thousands of connections, check the connection pooling options or a proxy, ideally with connection multiplexing (ProxySQL) •log_bin - you want to have binlogs enabled Consider sync_binlog=0 for MySQL 5.7 - it used to be a default on MySQL 5.6, it has been changed in 5.7.7. Better performance, less durability •skip_name_resolve - just to make sure your database won’t suffer when DNS will not be reachable •Query cache - optimize it away by disabling Use external caching layer (ProxySQL, Redis, Memcached) Tuning MySQL configuration

31. Copyright 2018 Severalnines AB •Percona developed two great tools, part of the Percona Toolkit suite, which helps you to collect useful data about MySQL and underlying operating system •pt-summary collects data about hardware and OS settings CPU, memory, disk, load on the system •pt-mysql-summary connects to MySQL instance and gives you data about workload and configuration Type of queries that are running, handlers, configuration snapshot of current traffic and many others Tools useful in tuning the configuration

36. Copyright 2018 Severalnines AB •Tuning MySQL is not a simple task - it’s very time-consuming to go through the process of change - test - change - test… •People like to simplify things and look for shortcuts •Auto-tuning tools have been created mysqltuner.pl •Configurators and wizards have been created (i.e. by Percona) •Monitoring and management tools offer advisors (i.e. ClusterControl) What to avoid when tuning MySQL?

37. Copyright 2018 Severalnines AB •Some of those tools base on counters from MySQL internal status and this is tricky The fact you have joins performed without indices doesn’t justify increasing join_buffer_size •Sometimes making a good change is a bad decision If you run MySQL without binlog enabled, without double_buffer or without XA enabled, you may see a suggestion to enable those features Binlog is great for point-in-time recovery double_buffer and XA helps keep your data safe Enabling them will add serious I/O load on the system - it may not handle it What to avoid when tuning MySQL?

38. Copyright 2018 Severalnines AB •Make sure you follow the tuning process as we described - one change at a time •If you use external tools that provide suggestions, take them with a grain of salt How competent is the vendor in databases? Even if the vendor has the competence - do I understand what this change is and how it may impact my system? •If you setup a new host, you may use external tools more freely This would be initial tuning, not changing the way how application already works •Make sure you use common sense and each of your changes has a reason What to keep in mind when tuning MySQL?

40. Copyright 2018 Severalnines AB •Undo tablespaces are now configurable at runtime •innodb_dedicated_server=1, autotunes: innodb_buffer_pool_size innodb_log_file_size innodb_flush_method •Configuration persistence with SET PERSIST What’s new in MySQL 8.0? •innodb_log_buffer_size is now dynamic •Spin delay is now configurable for threads waiting for redo log flushing innodb_log_wait_for_flush_spin_hwm innodb_log_spin_cpu_abs_lwm innodb_log_spin_cpu_pct_hwm

41. Copyright 2018 Severalnines AB •Blogs in “Become a MySQL DBA” series: http://www.severalnines.com/blog/become-mysql-dba-blog-series-query-tuning-process http://www.severalnines.com/blog/become-mysql-dba-blog-series-configuration-tuning-performance •How to Benchmark Performance of MySQL & MariaDB using SysBench https://severalnines.com/blog/how-benchmark-performance-mysql-mariadb-using-sysbench •How to Design Highly Available Open Source Database Environments https://severalnines.com/resources/whitepapers/how-design-highly-available-open-source-database- environments •MySQL Replication Blueprint http://severalnines.com/blog/new-whitepaper-mysql-replication-blueprint • Contact: jj@severalnines.com Thank you!

Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning

You are reading a preview.

Check these out next

Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning

More Related Content

Slideshows for you (20)

Similar to Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning (20)

More from Severalnines (15)

Recently uploaded (20)