Optimizing Oracle databases with SSD

Optimizing Oracle database with SSD Guy Harrison, Dell Software Group

Optimizing Oracle databasewith SSDGuy HarrisonExecutive Director, R&DInformation management group

Introductions www.guyharrison.net guy_harrison@dell.com http://twitter.com/guyharrison3 Software Group

Dell, Quest and Toad4 Software Group

Quest Software is now part of Dell5 Software Group

Agenda• Brief History of Magnetic Disk• Solid State Disk (SSD) technologies• SSD internals• Oracle DB flash cache architecture• Performance comparisons• Recommendations and Suggestions6 Software Group

A brief history of disk7 Software Group

5MB HDD circa 19568 Software Group

28MB HDD - 1961• 1800 RPM• 100,000 times smaller than a cheap 3 TB drive• BUT spinning on 10 times slower than that drive9 Software Group

The more that things change....10 Software Group

Moore’s law• Transistor density doubles every 18 months• Exponential growth is observed in most electronic components: –CPU clock speeds –RAM –Hard Disk Drive storage density• But not in mechanical components –Service time (Seek latency) – limited by actuator arm speed and disk circumference –Throughput (rotational latency) – limited by speed of rotation, circumference and data density11 Software Group

Disk trends 2001-2009 2,000 1,500 1,000%age change 500 260 1,635 1,013 0 -630 -390 -500 -1,000 IO Rate Disk Capacity IO/Capacity CPU IO/CPU 12 Software Group

Solid State Disk to the rescue?13 Software Group

Seek timesSSD DDR-RAM 15 SSD PCI flash 25SSD SATA Flash 80 Magnetic Disk 4,000 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 Seek time (us) 14 Software Group

Power consumptionStart up 20 0.15 Seek 10 Flash SSD SATA HDD 0.08 Idle 8 0.01 0.1 1 10 Watts (logarithmic scale) 15 Software Group

Economics of SSD $/GB 0.00 10.00 20.00 30.00 40.00 50.00 60.00 0.06 FusionIO PCI SLC SSD 53.44 0.06FusionIO PCI MLC Duo SSD 24.92 0.05 Intel SLC SATA SSD 21.88 $/IOP 0.05 $/GB Intel MLC SATA SSD 6.88 Seagate SAS HDD 1.00 1.53 Seagate SATA HDD 0.09 2.38 0.00 0.50 1.00 1.50 2.00 2.50 $/IOPS 16 Software Group

Tiered storage management Main Memory DDR SSD Flash SSD $/GB $/IOP Fast Disk (SAS, RAID 0+1) Slow Disk (SATA, RAID 5) Tape, Flat Files, Hadoop17 Software Group

Storage Tiering Storage Tiering For Dummies,® Oracle Special Edition, Wiley 201118 Software Group

SSD technology and internals19 Software Group

Flavours of Flash SSD• DDR RAM Drive• SATA flash drive• PCI flash drive• SSD storage Server20 Software Group

Flash SSD is the most cost- effective SSD technology21 Software Group

PCI SSD vs SATA SSD • PCI vs SATA – SATA was designed for traditional disk drives with high latencies – PCI is designed for high speed devices – PCI SSD has latency ~ 1/3rd of SATA22 Software Group

Dell Express flash• PCI flash performance can normally only be achieved by attaching a PCI card directly to the server motherboard• Dell express flash exposes the interfaces the PCI bus to front loading drive slots allowing hot swap and install of PCI flash23 Software Group

Flash SSD Technology Storage Hierarchy: • Cell: One (SLC) or Two (MLC) bits • Page: Typically 4K • Block: Typically 128-512K Writes: • Read and first write require single page IO • Overwriting a page requires an erase & overwrite of the block Write endurance: • 100,000 erase cycles for SLC before failure • 5,000 – 10,000 erase cycles for MLC25 Software Group

Flash SSD performanceUpdate (256K block erase) 2000First insert (4k page write) 250 Read (4k page seek) 25 0 200 400 600 800 1000 1200 1400 1600 1800 2000 Microseconds26 Software Group

Flash Disk write degradation Empty Partially Full • All Blocks empty: Write time=250 us • 25% part full: – Write time= ( ¾ * 250 us + 1/4 * 2000 us) = 687 us • 75% part full – Write time = ( ¼ * 250 us + ¾ * 2000 us ) = 1562 us27 Software Group

Data Insert Free Block Pool Insert SSD Controller Used Block Pool Empty Data Page Valid Data Page InValid Data Page28 Software Group

Data Update Free Block Pool Update SSD Controller Used Block Pool Empty Data Page Valid Data Page Invalid Data Page29 Software Group

Garbage Collection Free Block Pool SSD Controller Used Block Pool Empty Data Page Valid Data Page Invalid Data Page30 Software Group

31 Software Group

11g DB flash Cache32 Software Group

Oracle DB flash cache• Introduced in 11gR2 for OEL and Solaris only• Secondary cache maintained by the DBWR, but only when idle cycles permit• Architecture is tolerant of poor flash write performance33 Software Group

Buffer cache and Free buffer waits Buffer Read from buffer cache Free cache Oracle process Buffer Write to buffer cache Waits Free buffer waits often occur DBWR when reads are much faster than writes.... Read from disk Database files Write dirty blocks to disk34 Software Group

Flash Cache Buffer Read from buffer cache cache Oracle process Write to buffer cache Read from flash cache Flash Cache DBWR Write clean blocks (time permitting) DB Flash cache architecture is designed to accelerate buffered reads Read from disk Write dirty blocks to disk Database files35 Software Group

Configuration • Create filesystem from flash device • Set DB_FLASH_CACHE_FILE and DB_FLASH_CACHE_SIZE. • Consider Filesystemio_options=setall36 Software Group

Flash KEEP pool• You can prioritise blocks for important objects using the FLASH_CACHE clause:37 Software Group

Oracle Db flash cache statistics http://guyharrison.squarespace.com/storage/flash_insert_stats.sql38 Software Group

Flash Cache Efficiency http://guyharrison.squarespace.com/storage/flash_time_savings.sql39 Software Group

Flash cache Contents http://guyharrison.squarespace.com/storage/flashContents.sql40 Software Group

Performance tests41 Software Group

Test systems • Low end system: – Dell Optiplex dual-core 4GB RAM – 2xSeagate 7500RPM Baracuda SATA HDD – Intel X-25E SLC SATA SSD • Higher end system: – Dell R510 2xquad core, 32 GB RAM – 4x300GB 15K RPM,6Gbps Dell SAS HDD – 1xFusionIO ioDrive SLC PCI SSD42 Software Group

Performance: indexed reads(X-25)Flash tablespace 48.17 Flash cache 143.27 CPU db file IO flash cache IO Other No Flash 529.7 0 100 200 300 400 500 600 Elapsed (s) 43 Software Group

Performance: Read/Write (X-25)Flash tablespace 200 CPU Flash Cache 1,693 db file IO write complete free buffer flash cache IO Other 3,289 No Flash 0 500 1000 1500 2000 2500 3000 3500 Elapsed time (s) 44 Software Group

Random reads – FusionIO Table on SSD 121SAS disk, flash cache 583 CPU Other DB File IO Flash cache IO SAS disk, no flash 2,211 cache 0 500 1000 1500 2000 2500 Elapsed time (s) 45 Software Group

Updates – Fusion IO Table on SSD 529 DB CPU SAS disk, flash cache 1,934 db file IO log file IO flash cache free buffer waits OtherSAS disk, no flash cache 6,219 0 1000 2000 3000 4000 5000 6000 7000 Elapsed Time (s) 46 Software Group

Full table scan – FusionIO Table on SSD 72 SAS disk, flash cache 398 CPU Other DB File IO Flash Cache IOSAS disk, no flash cache 418 0 50 100 150 200 250 300 350 400 450 Elasped time (s) 47 Software Group

Sorting – what we expect Multi-pass Disk SortTime Single Pass Disk Sort Memory Sort PGA Memory available (MB) Table/Index IO CPU Time Temp Segment IO 48 Software Group

Disk Sorts – temp tablespace SSD vs HDD 4000 3500 Multi-pass 3000 Disk Sort 2500 Elapsed time (s) 2000 1500 1000 Single Pass Disk Sort 500 0300 250 200 150 100 50 0 Sort Area Size 49 Software Group SAS based TTS SSD based TTS

Redo performance – Fusion IOFlash based redo log 291.93 CPU Log IO SAS based redo log 292.39 0 50 100 150 200 250 300 350 50 Software Group Elapsed time (s)

Concurrent redo workload (x10)Flash based redo log 1,637 331 1,681 CPU Other Log File IO SAS based redo log 1,605 397 1,944 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 51 Software Group Elapsed time (s) 51

Redo logs - redo size• Marcelle Kratochvil has reported significant improvements for SSD redo when applying LOB updates• Performance for SSD writing small OLTP style transactions may differ significantly from large LOB updates: – Small transactions will hit the same block repeatedly, resulting in block erase overheads for most writes. – When the redo size exceeds the SSD page size then this overhead is avoided and redo performance on SSD may exceed HDD52 Software Group

Buffer Cache bottlenecks• Flash cache architecture avoids ‘free buffer waits’ due to flash IO, but write complete waits can still occur on hot blocks.• Free buffer waits are still likely against the database files, due to high physical read rates created by the flash cache 53 Software Group

Write degradation• In theory, high sustained write IO can lead to SSD degradation when GC fails to cope with the block erase/update cycle• In practice, this is rarely noticeable from Oracle: –Oracle write IO is largely asynchronous (DBWR) –Almost all write activity has at least an equal amount of read activity – Garbage collection and wear levelling algorithms are sophisticated in decent SSD drives54 Software Group

55 Software Group

56 Software Group

Flash caching technologies Dell FluidCache, FusionIO DirectCache, etc. File System/ Raw File System/ Raw Devices/ ASM Devices/ ASM•TempTablespace• Hot Segments Regular Block Device Caching Block Device Read-• Hot Partitions intensive, po• DB Flash tentiallyCache Device Driver FluidCache Driver massive tablespaces(limited to thesize of the SSD) LUN 57 Software Group 57

Fusion IO direct cache – Table scansdirect cache on 2nd scan 36 direct cache on 1st scan 147 CPU IO No cache 2nd scan 147 Other No cache 1st scan 147 0 20 40 60 80 100 120 140 160 Elapsed time (s) 58 Software Group

Exadata59 Software Group 59

Exadata v2 flash storage• 4x96GB PCI Flash drives on each storage server (4x increase in X3)• Flash can be configured as: – Exadata Smart Flash Cache (ESFC) – Solid State Disk available to ASM disk groups• ESFC is not the same as the DB flash cache: – Maintained by cellsrv, not DBWR – DOES support full table scans – DOES NOT support smart scans – Unless CELL_FLASH_CACHE= KEEP, – Statistics accessed via the cellcli program• Considerations for cache vs. SSD are similar61 Software Group

Exadata: Flash grid disk vs ESFCSSD disks (no flash cache) 119SAS disk with flash cache 429 IO CPU SAS disk no flash cache 1,240 0 200 400 600 800 1000 1200 1400 Seconds62 100M row table, 200,000 random PK lookups, 1M possible keys Software Group

Exadata: SSD disk for Redo SSD redo 1 concurrent 1,007 SAS redo 1 concurrent 236 DB CPU Log IOSSD redo 10 concurrent (avg) Other 6,885SAS redo 10 concurrent (avg) 352 0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 Elapsed time (s) 63 Software Group

Exadata: the effect of redo block size SSD disk blocksize 4k 309.68SSD disk blocksize 512 1,009.67 CPU Redo IO Time SAS disk blocksize 4K 243.79SAS disk blocksize 512 236.05 0.00 200.00 400.00 600.00 800.00 1,000.00 1,200.00 Elasped time (s) CellCLI: Release 11.2.2.3.264 Software Group

Exadata Smart FlashLog65 Software Group

Smart Flash Log• Designed to reduce “outlier” redo log sync waits DB Node• Redo is written simultaneously to disk and flash• First write to complete wins CellSrv• Introduced in Exadata storage software 11.2.2.4 Disk Flash 66 Software Group

All Redo log writes (16M log writes) Flash Min Median Mean 99% Max Log ON 1.0 650 723 1,656 75,740 OFF 1.0 627 878 4,662 291,80067 Software Group

Redo log outliersWAIT #47124064145648: nam=log file sync ela= 710 buffer#=129938 sync scn=1266588258 p3=0 obj#=-1tim=1347583167579790WAIT #47124064145648: nam=log file sync ela= 733 buffer#=130039 sync scn=1266588297 p3=0 obj#=-1tim=1347583167580808WAIT #47124064145648: nam=log file sync ela= 621 buffer#=130124 sync scn=1266588332 p3=0 obj#=-1tim=1347583167581695WAIT #47124064145648: nam=log file sync ela= 507 buffer#=130231 sync scn=1266588371 p3=0 obj#=-1tim=1347583167582486WAIT #47124064145648: nam=log file sync ela= 683 buffer#=101549 sync scn=1266588404 p3=0 obj#=-1tim=1347583167583398WAIT #47124064145648: nam=log file sync ela= 2084 buffer#=130410 sync scn=1266588442 p3=0 obj#=-1tim=1347583167585748WAIT #47124064145648: nam=log file sync ela= 798 buffer#=130535 sync scn=1266588488 p3=0 obj#=-1tim=1347583167586864WAIT #47124064145648: nam=log file sync ela= 1043 buffer#=101808 sync scn=1266588527 p3=0 obj#=-1tim=1347583167588250WAIT #47124064145648: nam=log file sync ela= 2394 buffer#=130714 sync scn=1266588560 p3=0 obj#=-1tim=1347583167590888WAIT #47124064145648: nam=log file sync ela= 932 buffer#=101989 sync scn=1266588598 p3=0 obj#=-1tim=1347583167592057WAIT #47124064145648: nam=log file sync ela= 291780 buffer#=102074 sync scn=1266588637 p3=0 obj#=-1tim=1347583167884090WAIT #47124064145648: nam=log file sync ela= 671 buffer#=102196 sync scn=1266588697 p3=0 obj#=-1tim=1347583167885294WAIT #47124064145648: nam=log file sync ela= 957 buffer#=102294 sync scn=1266588730 p3=0 obj#=-1tim=1347583167886575WAIT #47124064145648: nam=log file sync ela= 852 buffer#=120 sync scn=1266588778 p3=0 obj#=-1tim=1347583167887763WAIT #47124064145648: nam=log file sync ela= 639 buffer#=214 sync scn=1266588826 p3=0 obj#=-1tim=1347583167888778WAIT #47124064145648: nam=log file sync ela= 699 buffer#=300 sync scn=1266588853 p3=0 obj#=-1tim=1347583167889767WAIT #47124064145648: nam=log file sync ela= 819 buffer#=102647 sync scn=1266588886 p3=0 obj#=-1tim=1347583167890829 68 Software Group

Top 10,000 waits69 Software Group

Exadata 12c (X3) Smart Flash Cache Write-back• Database writes go to flash cache – LRU aging to mag disk – Reads serviced by flash prior to age out – Similar restrictions to flach cache (smart scans, etc)• Performance claims: – On X3 system, 1M random single block write IOPS – On X3 system, 1.5M random single block read IOPS – 500K write IOPS claimed for Exadata V2 – Note: random IO writes are less problematic for flash than sequential writes.70 Software Group

Summary71 Software Group

Recommendations• Don’t wait for SSD to become as cheap as HDD – Magnetic HDD will always be cheaper per GB, SSD cheaper per IO• Consider a mixed or tiered storage strategy – Using DB flash cache, selective SSD tablespaces or partitions – Use SSD where your IO bottleneck is greatest and SSD advantage is significant• DB flash cache offers an easy way to leverage SSD for OLTP workloads, but has few advantages for OLAP or Data Warehouse72 Software Group

How to use SSD• Database flash cache – If your bottleneck is single block (indexed reads) and you are on OEL or Solaris 11GR2• Flash tablespace – Optimize read/writes against “hot” segments or partitions• Flash temp tablespace – If multi-pass disk sorts or hash joins are your bottleneck• Device cache (Dell FluidCache, FusionIO direct cache) – If you want to optimize both scans and index reads OR you are not on OEL/Solaris 11GR273 Software Group

Shameless Plugs74 Software Group

75 Software Group

76 Software Group

77 Software Group

References• Latest version of this presentation: http://www.slideshare.net/gharriso/ssd-and-the-db-flash-cache• Quest whitepaper: • http://www.quest.com/documents/landing.aspx?id=15423• Guy’s SSD guide • http://guyharrison.squarespace.com/ssdguide/ • Kai Yu’s whitepaper on optimizing Oracle OLTP with Flash: • http://i.dell.com/sites/doccontent/shared-content/data- sheets/en/Documents/PCIe_SSD_for_oracle_database_performance .pdf78 Software Group

THANK YOUGuy_harrison@dell.com@guyharrisonguyharrison.net

THANK YOUGuy_harrison@dell.com @guyharrison guyharrison.net

http://kojishinkubo.blogspot.com	53
http://www.insight-tec.com	46
http://kojishinkubo.blogspot.jp	39
http://202.58.19.123	2
https://twitter.com	2
http://webcache.googleusercontent.com	1

Optimizing Oracle databases with SSD

by Guy Harrison

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 143

Statistics

Optimizing Oracle databases with SSD Presentation Transcript