Linux and Mysql TuningGetting the most from your hardware
The Ubiquitous “About Me”Ran datacenters for: National Geographic ExpertCity (now Citrix Online) Fastclick (now Valueclick) University of California (now broke);Now CEO/founder of LogicMonitor.comProbably just like you. (Not a kernel committer,or poring over MySQL source code).
Your Mileage May VaryNothing I am going to say will have any impactin your environment.Unless it does.So...test, measure, rinse, repeat.
Why?Limits are always one of CPU, RAM, Disk,Network.But with better <CPU|RAM|Disk|network> thelimit will move to next objectDefault configurations may be for:Your server:
IO Scheduler TuningKernel decides how to order disk operations, inorder to combine IO ops into one op, and alsominimize disk seeks, or not.Defaults work. No benefit in other policies, onlyharm. Dont bother testing.But, its trivial to test, so dont take my word forit. With battery backed RAID cache, othershave reported good results with noop.Kernel, hardware dependent. Test!echo noop > /sys/block/sda/queue/scheduler
VM on ESX with no-cache RAID1Switching from CFQ to NoopAmount of non-merged writes overwhelmeddisk. S S w w i i t t c c h h t t o o N N o o O O P P
VM on ESX no-cache RAID1Switch from anticipatory to CFQ at 13:10CFQ has slightly more reads merged, slightlybetter performance time. S w i t c h t o C F Q
Lots of IO where ordering doesnt matter4000 disk operations to SSDSwitching from CFQ to NoOP – slightly worseIO completion time. S S w w i it t c c h h t o t N o o O N P o O P
But it saves CPU, right?Not significantly, if at all. P o l i c y s w i t c h e d
Innodb Thread ConcurrencyIf database is slow, but CPU not busy...thismay be issue.Defaults range from 4 (in some distros) toinfinite, depending on version. 8 seems mostcommon default in real world.Default probably too low for “real” DB server.
Test! Benchmark first, then real life Use a threaded benchmark. sysbench --test=oltp --mysql-table- engine=innodb --mysql- socket=/var/lib/mysql/mysql.sock --max- time=300 --db-driver=mysql --num-threads=10 --max-requests=300000 run Use same benchmark, vary server config.
Results will vary with # usersHow many user threads to test? Use real data.
My results, not yoursThis is true on some hardware, with someversion of MySQL, with 10 client threads.Your mileage WILL vary. Test.Higher settings not always better.
More InnoDB threads – more efficient use of CPU 16 Innodb ∞ Innodb Threads Threads 8 Innodb threads 4 Innodb Threads
SwappingUse of swap not bad – swapping is bad.OK (but not ideal):Performance Killer:
Easy things firstecho 0 > /proc/sys/vm/swappinessecho “vm.swappiness = 0” >> /etc/sysctl.conf;sysctl -pThis may be enough. If so, KISS.
NUMA factorsSystem tries to allocate memory local to theCPU a process runs on.In dual processor system, if one process triesto grab > 50% of systems memory, it will useall free memory local to one processor.other process on that CPU requesting memorymay cause swap activity to get local memory –even though there is free non-local memory.http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/
Numa, III couldnt reproduce: Run mysql with affinity to a physical CPU, with buffer pool that consumes all local memory. Start Java process with affinity to same CPU. Mem was allocated from other core. But.. no swap used.But just in case...easy to address:numactl --interleave all mysqldIf you are you are allocating more than 50%memory to innodb (or any other singleprocess), you are guaranteed non-localmemory access. So may as well bedeterministic, and possibly benefit from thisswap-fix.
HugePagesCurrently huge pages are not swapped (maychange)Will have some (probably very minor)performance benefit (big in some cases)But.. more complex (and thus risky.)Need to enable in kernel, configure kernel,ulimits, security groups, configure mysql. Seehttp://dev.mysql.com/doc/refman/5.0/en/large-page-support.html
HugePagesSet nr_hugepages too low – Mysql wont use,and you just wasted memory. Ditto too high.Pins Innodb buffer pool cache – but willincrease pressure on other memory if swappingis triggeredWill reduce TLB cache lookups/misses. Doesthat matter? Test!
SummaryMy app/kernel/hardware/DB version is notyours. YMMV.Test IO scheduler, as its so easy.Test InnoDB thread concurrency.Address swapping in the simplest way you can.Trend everything. All the time. Compare pre-and post releases.