Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Database Hardware Benchmarking

6,120 views

Published on

Greg Smith

Cover how to use simple low-level tools such as memtest86, dd, bonnie++, and sysbench to benchmark the hardware of a server intended for database use. A heavy dose of vendor management suggestions will be included as well, for the inevitable time when your shiny new server fails to deliver the performance it should.

  • Be the first to comment

Database Hardware Benchmarking

  1. 1. Database Hardware Benchmarking Greg Smith 2ndQuadrant US 03/25/2009 Greg Smith Database Hardware Benchmarking
  2. 2. About this presentation The master source for these slides is http://www.2ndquadrant.com/talks Slides are released under the Creative Commons Attribution 3.0 United States License http://creativecommons.org/licenses/by/3.0/us Greg Smith Database Hardware Benchmarking
  3. 3. Why should you always benchmark your hardware? Many useful tests will only run when the server isn’t being used yet Software stacks are complicated Spending money on upgrades only helps if you upgrade the right thing usefully Vendors lie Greg Smith Database Hardware Benchmarking
  4. 4. Systematic Benchmarking Memory CPU Disk Database server Application Greg Smith Database Hardware Benchmarking
  5. 5. Memory Tests DOS: memtest86+ (also on many Linux CDs, like the Ubuntu installer) Windows: SiSoftware Sandra UNIX: sysbench, stream Greg Smith Database Hardware Benchmarking
  6. 6. sysbench memory write and CPU tests sysbench --test=memory --memory-oper=write --memory-block-size=64MB --memory-total-size=1024MB run sysbench --test=cpu run Greg Smith Database Hardware Benchmarking
  7. 7. stream memory read test wget http://www.cs.virginia.edu/stream/FTP/Code/stream.c gcc -O stream.c -o stream ./stream Greg Smith Database Hardware Benchmarking
  8. 8. Memory test comparisons Processor Frequency RAM Speed memtest86+ sysbench stream T7200 2.0GHz DDR2/667 2563 1360 2818 Q6600 2.4GHz DDR2/800 3352 2009 4386 T9400 2.53GHz DDR3/1066 3743 2226 ? Xeon X5450 3.00GHz DDR2/1333 3575 2487 ? Xeon E5520 2.27GHz DDR3/1333 9252 4230 ? Xeon X5550 2.67GHz DDR3/1333 10885 5493 ? i7 860 2.8GHz DDR3/1600 13716 5559 10033 Greg Smith Database Hardware Benchmarking
  9. 9. Advanced Clustering STREAM Benchmarking http://www.advancedclustering.com/company-blog/ Greg Smith Database Hardware Benchmarking
  10. 10. Oracle Calling Center OLTP Benchmark http://it.anandtech.com/IT/showdoc.aspx?i=3769&p=4 Greg Smith Database Hardware Benchmarking
  11. 11. Sources for slow memory results Single channel RAM/slot mistakes Incorrect SPD/timing/voltage Bad RAM/CPU multiplier combination Poor quality RAM Greg Smith Database Hardware Benchmarking
  12. 12. PostgreSQL and the CPU PostgreSQL uses only a single CPU per query Queries executing against cached data will bottleneck on CPU COPY is CPU intensive Greg Smith Database Hardware Benchmarking
  13. 13. CPU Tests Windows: SiSoftware Sandra UNIX: sysbench CPU test Custom test with timing and generate series pgbench select-only on small database Vary pgbench client count to test single or multiple CPUs Greg Smith Database Hardware Benchmarking
  14. 14. sysbench CPU comparisons Processor Frequency sysbench CPU seconds Intel Q6600 2.40GHz 19.6 Intel T7200 2.0GHz 18.9 Intel T9400 2.53GHz 12.0 Intel Xeon X5450 3.00GHz 8.4 Intel Xeon E5520 2.27GHz 11.7 Intel Xeon X5550 2.67GHz 8.7 Intel i7 860 2.8GHz 9.0 Greg Smith Database Hardware Benchmarking
  15. 15. Sources for slow CPU results Slow memory Power management throttling Linux: /proc/cpuinfo shows 1000MHz suggests you need to adjust the CPUFreq Governor to “performance” Greg Smith Database Hardware Benchmarking
  16. 16. Disk Tests Sequential write: INSERT, COPY FROM (when not CPU limited) Sequential read: SELECT * FROM and similar table sequential scans Seeks: SELECT using index, UPDATE Commit fsync rate: INSERT, UPDATE Greg Smith Database Hardware Benchmarking
  17. 17. dd test Compute 2X the size of your RAM in 8KB blocks blocks = 250,000 * gigabytes of RAM time sh -c "dd if=/dev/zero of=bigfile bs=8k count=X && sync" time dd if=bigfile of=/dev/null bs=8k Watch vmstat and/or iostat during disk tests vmstat’s bi and bo will match current read/write rate Note the CPU percentage required to reach the peak rate Greg Smith Database Hardware Benchmarking
  18. 18. bonnie++ ./bonnie++ bon csv2html Ignore the per-character and create results, look at the block output/input ones Random Seeks: The test runs SeekProcCount processes (default 3) in parallel, doing a total of 8000 random seek reads to locations in the file. In 10% of cases, the block read is changed and written back. Greg Smith Database Hardware Benchmarking
  19. 19. bonnie++ ZCAV ./zcav -f/dev/sda > t500 Must get a recent version of bonnie++ for ZCAV to scale properly for TB drives (1.03e works) ZCAV on experimental branch (1.96) gave useless results for me Download somewhat broken gnuplot script sample and typical results from: http://www.coker.com.au/bonnie++/zcav/results.html Greg Smith Database Hardware Benchmarking
  20. 20. Improved bonnie++ ZCAV gnuplot script unset autoscale x set autoscale xmax unset autoscale y set autoscale ymax set xlabel "Position GB" set ylabel "MB/s" set key right bottom set terminal png set output "zcav.png" plot "raid0" title "7200RPM RAID 0 3 Spindles", "single" title "7200RPM Single Drive" Greg Smith Database Hardware Benchmarking
  21. 21. bonnie++ ZCAV: Laptop 7200RPM Disk Greg Smith Database Hardware Benchmarking
  22. 22. Scaling of 3-Disk RAID0 with 7200RPM SATA Disks Greg Smith Database Hardware Benchmarking
  23. 23. 3-Disk Short-Stroked RAID0, larger 7200RPM SATA Greg Smith Database Hardware Benchmarking
  24. 24. Read seeks/second - sysbench THREADS=1 GB=10 MODE=rndrd OPTIONS="--test=fileio --num-threads=$THREADS --file-block-size=8K --file-test-mode=$MODE --file-num=$GB --file-total-size=${GB}G --file-fsync-freq=0 --file-fsync-end=no" sysbench prepare $OPTIONS sysbench run --max-time=60 $OPTIONS sysbench cleanup $OPTIONS Greg Smith Database Hardware Benchmarking
  25. 25. Sample sysbench random read results Read 78.125Mb Written 0b Total transferred 78.125Mb (1.0059Mb/sec) 128.75 Requests/sec executed That’s 128.75 seeks/second over 10GB, resulting in a net throughput of 128.75 * 8KB/s = 1.01MB/s Consider both the size of the disk used and the number of clients doing seeks Greg Smith Database Hardware Benchmarking
  26. 26. More customizable seek tests bonnie++ experimental (currently at 1.96) iozone fio Windows: HD Tune does everything but commit rate Greg Smith Database Hardware Benchmarking
  27. 27. Sources for slow disk results Poor mapping to underlying hardware Buggy driver Insufficient bandwidth to storage Bottlenecking at CPU/memory limits Bad performing filesystem or filesystem misaligned with stripe sizes Writes faster than reads? Probably low read-ahead settings somewhere. Vibration: don’t shout at your JBODs! They don’t like it! http://it.toolbox.com/blogs/database-soup/the-problem-with-iscsi-30602 http://blog.endpoint.com/2008/09/filesystem-io-what-we-presented.html http://www.youtube.com/watch?v=tDacjrSCeq4 Greg Smith Database Hardware Benchmarking
  28. 28. fsync tests sysbench --test=fileio --file-fsync-freq=1 --file-num=1 --file-total-size=16384 --file-test-mode=rndwr run | grep "Requests/sec" pgbench insert-only test PostgreSQL contrib/test fsync might work, but isn’t really reliable Greg Smith Database Hardware Benchmarking
  29. 29. Sample laptop disk specification ST9320423AS Momentus 7200.4 320GB 7200 RPM 16MB Cache Average seek: 11ms read/13ms write Average rotational latency: 4.17ms Greg Smith Database Hardware Benchmarking
  30. 30. Computed parameters Rotational latency = 1 / RPM / 60 / 2 IOPS=1/(latency+seek) IOPS = 1/(((1/(RPM/60))/2) + S) IOPS = 1/(4.17ms + 11ms) = 65.9 IOPS Greg Smith Database Hardware Benchmarking
  31. 31. IOPS Calculators and Info http://www.wmarow.com/strcalc/ http://www.dbasupport.com/oracle/ora10g/disk_IO_02.shtml http://storageadvisors.adaptec.com/2007/03/20/sata-iops-measurement/ Greg Smith Database Hardware Benchmarking
  32. 32. Sample disk results Disk Seq Seq bonnie++ Read-only Commit Drive Count Read Write seeks seeks Rate Model 1 71 58 232 @ 4GB 194 @ 4GB 105/s 7200.4 1 59 54 177 @ 16GB 56 @ 100GB 10212/s WD160 3 125 119 371 @ 16GB 60 @ 100GB 10855/s RAID0 Commit rate for 7200.4 laptop drive is 1048/s with unsafe volatile write cache Non-laptop drives include a 256MB battery-backed write cache, Linux SW RAID Greg Smith Database Hardware Benchmarking
  33. 33. Custom PostgreSQL tests Quick CPU test (about 1140ms on my laptop): timing select sum(generate series) from generate series(1,1000000); Quick insert/plan test: timing CREATE TABLE test (id INTEGER PRIMARY KEY); INSERT INTO test VALUES (generate series(1,100000)); EXPLAIN ANALYZE SELECT COUNT(*) FROM test; Greg Smith Database Hardware Benchmarking
  34. 34. Benchmarking functions http://justatheory.com/computers/databases/postgresql/benchmarking_upc_validation.html http://justatheory.com/computers/databases/postgresql/benchmarking_functions.html pg stat user functions handles this specific job in 8.4 The general technique is applicable for all sorts of custom benchmarks Greg Smith Database Hardware Benchmarking
  35. 35. pgbench select only test set naccounts 100000 * :scale setrandom aid 1 :naccounts SELECT abalance FROM accounts WHERE aid = :aid; Greg Smith Database Hardware Benchmarking
  36. 36. pgbench custom test: insert only, to measure commit rate set nbranches :scale set ntellers 10 * :scale set naccounts 100000 * :scale setrandom aid 1 :naccounts setrandom aid 1 :naccounts setrandom bid 1 :nbranches setrandom tid 1 :ntellers setrandom delta -5000 5000 BEGIN INSERT INTO history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT TIMESTAMP); END; Greg Smith Database Hardware Benchmarking
  37. 37. pgbench standard test BEGIN; UPDATE accounts SET abalance = abalance + :delta WHERE aid = :aid; SELECT abalance FROM accounts WHERE aid = :aid; UPDATE tellers SET tbalance = tbalance + :delta WHERE tid = :tid; UPDATE branches SET bbalance = bbalance + :delta WHERE bid = :bid; INSERT INTO history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT TIMESTAMP); END; Greg Smith Database Hardware Benchmarking
  38. 38. pgbench-tools Available from git.postgresql.org and github Set of shell scripts to automate running many pgbench tests Results are saved into a database for analysis Saves TPS, latency, background writer statistics Most of the cool features require PostgreSQL 8.3 Inspired by the dbt2 benchmark framework Greg Smith Database Hardware Benchmarking
  39. 39. Server configuration for pgbench results Quad-Core Intel Q6600 8GB DDR2-800 RAM Areca ARC-1210 SATA II PCI-e x8 RAID controller, 256MB write cache DB: 3x640GB Western Digital SATA disks, short-stroked, Linux software RAID-0 WAL: 160GB Western Digital SATA disk CentOS 5.4, Linux Kernel 2.6.18-164.11.1.el5xen x86 64 OS on separate disk Untuned ext3 filesystems Greg Smith Database Hardware Benchmarking
  40. 40. PostgreSQL Configuration PostgreSQL 9.0-alpha4 (with multi-threaded pgbench patch) shared buffers = 2GB checkpoint segments = 32 checkpoint completion target = 0.9 wal buffers = 16MB max connections = 300 Greg Smith Database Hardware Benchmarking
  41. 41. pgbench read-only size scaling Greg Smith Database Hardware Benchmarking
  42. 42. pgbench read-only client scaling Greg Smith Database Hardware Benchmarking
  43. 43. pgbench TCP-B-like size scaling Greg Smith Database Hardware Benchmarking
  44. 44. pgbench TPC-B-like client scaling Greg Smith Database Hardware Benchmarking
  45. 45. What should you do? Trust no one Don’t start on application benchmarks until you’ve proven basic performance Don’t spend too long on basic performance if you can switch to application benchmarks Vendors alternate among lying, misunderstanding what you want, and trying to make you feel dumb Use simple, standard tools whenever possible to minimize vendor disputes Be prepared to translate to your vendor’s language and subvert their agenda Never spend real money on hardware unless you can return it if it sucks Greg Smith Database Hardware Benchmarking
  46. 46. Credits Special thanks to: Mark Wong and the other contributors to dbt-2 Truviso for providing some of the test hardware included here Greg Smith Database Hardware Benchmarking
  47. 47. For more information... Some slides are taken from the upcoming book: “High Performance PostgreSQL” by Greg Smith Planned for release this summer by Packt Publishing Performance tuning of PostgreSQL 8.1 through 9.0, from hardware to scaling via replication Greg Smith Database Hardware Benchmarking
  48. 48. Questions? Remember that your coffee break awaits you Greg Smith Database Hardware Benchmarking

×