Tuning Storage
Subsystem for
Databases
Angelo Rajadurai
Agenda

Performance issues in Storage
Hybrid Storage (Disks, SSDs, Memory)
ZFS - Not Just Another File System
Tuning for d...
Why?
• Some very practical advice based on
  > recent test results
     > Improved pgbench results from 70 tps for pure di...
Storage Performance

                                                                       Cache

                       ...
Latency Comparison
Bridging the DRAM to HDD Gap




     1S

  100mS

    10m
      S
    1mS

   100uS

    10uS         ...
Storage Technology
                Price, Performance & Capacity


                Capacity     Latency             Cost/I...
Incorporating Flash
     Storage Hierarchy
Hybrid Storage
                 Flash as Cache


                   Application


         DRAM
       Level 1 Cache




 ...
ZFS - Last Word in Filesystem
     Pooled Storage Vs Traditional Volumes
Data Management Unit
  Smarts Built Right Into the Filesystem
Administering ZFS in two slides
                            As easy as pie


• zpool commands
 > create a single disk pool...
Administering ZFS in two slides
                            As easy as pie


• zfs commands
 > create a filesystem:
      ...
ZFS and Hybrid Storage
                          As easy as pie


• Read side
  > Add ssd as a read side cache
  > # zpool...
ZFS Performance Features
• Copy-on-write
  > Turns Random writes to Sequential writes
• Dynamic Striping across all device...
Databases
                      Not Just Another Application
• Most Databases do their own buffering
  > Filesystem cachin...
Tuning ZFS for Databases
                        Tuning is Evil - Long live Tuning


• In general tuning is evil. Let ZFS ...
ZFS tuning for MySQL
• Many tuning depends on storage engine
• For Innodb
  > Prefer to cache in Innodb rather than ARC
  ...
ZFS tuning for MySQL

• More tuning for Innodb
  > Some device vendors flush cache even
    when not needed. (eg. battery ...
ZFS tuning for PostgreSQL

• Postgres tuning hints
  > Set recordsize to 8k
     zfs set recordsize=8k poolname/database
 ...
ZFS tuning for Oracle

• Oracle tuning hints
  > Set recordsize to match db_block_size (default 8k)
     zfs set recordsiz...
Benchmark results

• Hardware
  > Sun x4150 2 x Quad core 2.3 GHz Xeon
     12 GB ram
      3 x 10000 rpm drives
      3 x...
Benchmark results

• pgbench & Postgres
  > command line: pgbench -c 10 -s 10 -t 10000 pgbench
                Description...
Benchmark results

• sysbench & mysql 5.4
  > read/write test: sysbench --max-time=300 --max-requests=0 --test=oltp --
   ...
Benchmark results

• sysbench & mysql 5.4
  > read test: sysbench --max-time=300 --max-requests=0 --test=oltp --oltp-
    ...
!"#$!%&'()*$'&(+,(-$.//012.$


Sun Unified Storage
                        *#%'345*6*5$!%(#+(5&#*
         :$;<$=*='&-$>$?...
Getting these systems at a discount
Sun Startup Essentials
                         •   Exclusive program for startups
   ...
Resources
• ZFS info: http://www.opensolaris.org/os/community/zfs/
• ZFS Best Practices Guide:
  http://www.solarisinterna...
Tuning Storage
Subsystem for
Databases
Angelo Rajadurai
angelo@sun.com
http://blogs.sun.com/angelo
twitter: rajadurai
Upcoming SlideShare
Loading in …5
×

SSD based storage tuning for databases

6,199 views
6,093 views

Published on

Databases are a key part of any application. The storage subsystem contributes most to performance of the database. In recent days, new storage technologies like Solid State Storage (SSD) and high performance drives are becoming cheaper and more accessible, but it takes a lot of planning to use these technologies in a cost effective way for best price-performance.

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,199
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
134
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

SSD based storage tuning for databases

  1. 1. Tuning Storage Subsystem for Databases Angelo Rajadurai
  2. 2. Agenda Performance issues in Storage Hybrid Storage (Disks, SSDs, Memory) ZFS - Not Just Another File System Tuning for databases (General principles) Tuning for MySQL Tuning for PostgreSQL Tuning for Oracle
  3. 3. Why? • Some very practical advice based on > recent test results > Improved pgbench results from 70 tps for pure disk to 5003 tps with SSD and tuning > Improved sysbench results from 425 tps to 1811 tps with SSD and tuning for read/write. > Improved sysbench results from 786 tps to 3085 tps with SSD and tuning for read. > collection of tuning knowledge from Sun performance engineers and the community • Some very good resources at the end of the talk for further study
  4. 4. Storage Performance Cache Memory 0,0 00 X t ial 10 en fer e dif nc Disk r ma High Cache r fo Performance Disks Pe Large Capacity Disks
  5. 5. Latency Comparison Bridging the DRAM to HDD Gap 1S 100mS 10m S 1mS 100uS 10uS TAPE 1u HDD S 100nS FLASH/ SSD 10nS DRAM 1n S CPU
  6. 6. Storage Technology Price, Performance & Capacity Capacity Latency Cost/IOPS Cost/GB Technologies IOPs (GB) (microS) ($) ($) Cloud Storage Unlimited 60,000 20 17c/GB 0.15/month Capacity HDDs 2,500 12,000 250 1.67 0.15 Performance HDDs 300 7,000 500 1.52 1.30 SSDs (write) 64 300 5000 0.20 13 SSDs (read only) 64 45 30,000 0.03 13 DRAM 8 0.005 500,000 0.001 52
  7. 7. Incorporating Flash Storage Hierarchy
  8. 8. Hybrid Storage Flash as Cache Application DRAM Level 1 Cache Write Read Flash Write side Log Flash Level 2 Cache Disk Primary Storage
  9. 9. ZFS - Last Word in Filesystem Pooled Storage Vs Traditional Volumes
  10. 10. Data Management Unit Smarts Built Right Into the Filesystem
  11. 11. Administering ZFS in two slides As easy as pie • zpool commands > create a single disk pool: # zpool create newpool diskname > create a pool with a mirror # zpool create newpool mirror disk1name disk2name > Add device to a pool: # zpool add poolname diskname > Replace a bad disk # zpool replace poolname baddiskname newdiskname > History of commands on the pool: # zpool history poolname > How is my pool performing: # zpool iostat poolname No format command, No fdisk partitions, No volumes
  12. 12. Administering ZFS in two slides As easy as pie • zfs commands > create a filesystem: # zfs create poolname/fs-name > set filesystem property: # zfs set quota=size poolname/fs-name # zfs set compression=on poolname/fs-name # zfs set nfsshare=on poolname/fs-name # zfs set recordsize=16k poolname/fs-name > get filesystem property: # zfs get compressratio poolname/fs-name # zfs get all poolname/fs-name > snapshot the filesystem: # zfs snapshot poolname/fs-name@snapshotname No newfs, No mkfs, No /etc/vfstab, No fsck
  13. 13. ZFS and Hybrid Storage As easy as pie • Read side > Add ssd as a read side cache > # zpool add poolname cache ssd-device • Write side > Add SSD as a ZFS Intent Log device > # zpool add poolname log ssd-device
  14. 14. ZFS Performance Features • Copy-on-write > Turns Random writes to Sequential writes • Dynamic Striping across all devices > Maximize throughput • Multiple Block Sizes > Automatically chosen to match workload • IO Pipelining > Priority/Deadline scheduling, sorting, aggregation • Intelligent prefetch • Compression - Improves performance & Capacity • Can safely use write cache on disks
  15. 15. Databases Not Just Another Application • Most Databases do their own buffering > Filesystem caching can get in the way > “double buffer” problem • Most Databases do “prefetch” > Filesystems prefetch can cause extra IO > “directio” gets filesystem out of the way • Have their own “log” mechanism. > Interesting interaction with a transaction based filesystem • Multiple blocks sizes > Database & Transaction log, block sizes are normally different
  16. 16. Tuning ZFS for Databases Tuning is Evil - Long live Tuning • In general tuning is evil. Let ZFS do it for you. • A few fine tuning tips for databases > Get to the latest update of OS > Set the recordsize to match database block size > Separate Transaction logs and data onto separate zpools > [Note: This will be addressed with the ZIL bypass property fix] > Reduce the impact of double buffering by changing the caching method to “metadata only” > Use separate ZIL (ZFS Intent Log) preferably SSD > Use SSD as secondary cache - L2ARC (Level 2. Adaptive Replacement Cache)
  17. 17. ZFS tuning for MySQL • Many tuning depends on storage engine • For Innodb > Prefer to cache in Innodb rather than ARC zfs set primarycache=metadata poolname/database > Set recordsize to 16k for data and 128k for log zfs set recordsize=16k poolname/database (Note: do this before you load any data) > Turn off prefetch set zfs:zfs_prefetch_disable = 1 (in /etc/system) (File level prefetch not triggered if you change record size to 16k) > Use raid0 or mirror over raidz raidz is no suitable for random IO > Add SSDs for either read side or write side based on workload zpool add datapool cache ssd-disk zpool create logpool ssd-disk3 In my.cnf set innodb_data_home_dir & innodb_log_group_home_dir
  18. 18. ZFS tuning for MySQL • More tuning for Innodb > Some device vendors flush cache even when not needed. (eg. battery backed cache) set zfs:zfs_nocacheflush = 1 > Turn on compression zfs set compression=on poolname/database ZFS does not turn on compression if less than 12.5% saving. IO reduction may offset the extra cpu cost > Disable double writes innodb_doublewrite=0 (in my.cnf) ZFS does not allow any partial writes so no need to guard against it.
  19. 19. ZFS tuning for PostgreSQL • Postgres tuning hints > Set recordsize to 8k zfs set recordsize=8k poolname/database > Turn down ARC cache. set zfs:zfs_arc_max in /etc/system > Add SSDs for either read side or write side based on workload zpool add poolname cache ssd-name zpool add poolname log ssd-name > Use separate pool for log (preferably one with SSD) & data initdb -X log_directory_name create tablespace datatbs location 'database_directory_name' create database mydb with  tablespace datatbs > Don’t forget to basic Postgres tuning on Solaris - (huge gains) Set shared_buffers, temp_buffers, work_mem, maintenance_work_mem, wal_sync_method, synchronous_commits etc see: http://blogs.sun.com/jkshah/entry/best_practices_with_postgresql_8
  20. 20. ZFS tuning for Oracle • Oracle tuning hints > Set recordsize to match db_block_size (default 8k) zfs set recordsize=8k poolname/database > Use separate pool for Oracle logs make sure record size of the log filesystem is left to the 128k default > Add SSDs for either read side or write side based on workload zpool add poolname cache ssd-name zpool add poolname log ssd-name
  21. 21. Benchmark results • Hardware > Sun x4150 2 x Quad core 2.3 GHz Xeon 12 GB ram 3 x 10000 rpm drives 3 x 32 GB SSDs • Software > OpenSolaris 2009.06 > Postgres 8.3.7 > MySQL 5.4 beta
  22. 22. Benchmark results • pgbench & Postgres > command line: pgbench -c 10 -s 10 -t 10000 pgbench Description TPS Single disk ZFS 72 tps 2 Raid 0 disk + SSD as level 2 cache 241 tps Above + general postgres optimization 2026 tps + all the data on SSD 2603 tps + data on hdd & log on SSD 4372 tps + primarycache=metadata 5003 tps
  23. 23. Benchmark results • sysbench & mysql 5.4 > read/write test: sysbench --max-time=300 --max-requests=0 --test=oltp -- oltp-dist-type=special --oltp-table-size=10000000 --num-threads=20 run Description TPS Single disk ZFS 425 tps raid0 ZFS 670 tps + SSD cache 788 tps + Separate intent log 1352 tps + With optimization 1809 tps
  24. 24. Benchmark results • sysbench & mysql 5.4 > read test: sysbench --max-time=300 --max-requests=0 --test=oltp --oltp- dist-type=special --oltp-table-size=10000000 --num-threads=20 --oltp-read- only=on run Description TPS Single disk ZFS 786 tps 2 disk raid0 ZFS 1501 tps + SSD cache 1981 tps + Separate intent log on SSD 2567 tps + optimization 3065 tps
  25. 25. !"#$!%&'()*$'&(+,(-$.//012.$ Sun Unified Storage *#%'345*6*5$!%(#+(5&#* :$;<$=*='&-$>$?@A9BB;<$!6!$1)CDC 78)74+*#!8%3$!9(5(:5* E($+'$F@;<$=*='&-G$142A?H<$!6H6$1)CDC E($+'$IA?:$;< write-optimized SSDs *#%'345*6*5$95"!%*'49(-(:5* E($+'$F@;<$=*='&-G$ZFA?H<$!6H6$1)CDC E($+'$FA?BB;<$&*,1.?FA?:;<$J&)+*$!!5C 6K+)L*M6K+)L*$/2"C+*&)#N$0'&$O.6 !9(5(:5*$95"!%*'49(-(:5* !%(#+('+$@*(%"'*!$>(55$,&+*5!? 622$5,+,$%&'+'K'2C$,#1$5,+,$!*&L)K*C$V#K2"1*1 E($+'$?I:;<$=*='&-G$I::A?H<$!6H6$1)CDC S6#,2-+)KCG$!#,(CR'+CG$7*(2)K,+)'#G$/'=(&*CC)'#G$4U!G$/VU!G$)!/!VWT E($+'$FA?BB;<$&*,1.?FA?:;<$J&)+*$!!5C <")2+M)#$@A?$;P$Q+R*&#*+$('&+C 6K+)L*M6K+)L*$/2"C+*&)#N$0'&$O.6 7*='+*$/'#C'2*$S!*&),2$'&$Q+R*&#*+T X)NR+CM'"+$Y,#,N*=*#+ &-%8&#(5$#*%;&'<8#)$=$9&##*9%868%3$>(55$,&+*5!? IA?B$;P$Q+R*&#*+$S'(+)K,2T @A?$;P$Q+R*&#*+$SK'((*&T$>$IA?;P$Q+R*&#*+$S'(+)K,2T U/$'&$!/!V$O<6$0'&$+,(*$P,KD"( !"#$ %&'(&)*+,&-./'#0)1*#+),23$ 456$ 7*8")&*1 9
  26. 26. Getting these systems at a discount Sun Startup Essentials • Exclusive program for startups • Eligibility <6 yrs. Old, <150 sun.com/startup employees • Co-marketing opportunities • Funding assistance • Deeply discounted storage and servers certified for Linux, Windows, and Solaris • Hosting starting at $40 • Open source software, and discounted MySQL • Free email based tech support • Free and discounted training on Sun technologies • Member-only webinars
  27. 27. Resources • ZFS info: http://www.opensolaris.org/os/community/zfs/ • ZFS Best Practices Guide: http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide • ZFS Evil Tuning Guide: http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide • Blogs of note: > All things performance tuning: http://blogs.sun.com/realneel http://blogs.sun.com/roch > Postgres tuning - Jignesh’s Blog http://blogs.sun.com/jkshah > Angelo’s blog http://blogs.sun.com/angelo
  28. 28. Tuning Storage Subsystem for Databases Angelo Rajadurai angelo@sun.com http://blogs.sun.com/angelo twitter: rajadurai

×