Demystifying Storage

Yottabytes and Beyond Demystifying Storage and Building large Storage Networks Part I by Bhavin Turakhia, CEO, Directi bhavin.t@directi.com

Why is storage important?
Web 2.0 applications are an extension of your Desktop
SaaS is here and growing
Broadband is a reality
Storage costs are dropping
Everyone expects near-unlimited storage online – Youtube, Flickr, Facebook et al are storing your life online*
(.. And yea … lets not forget your personal bit-torrent collection)
* it would take 1400 TB to store your entire life in video. 5700 TB if you want to know what was happening around you. Another 73 TB for the audio files of everything you heard (MP3 quality). That’s about 6000 TB for a copy of your life

Agenda
Hard disks
SATA, SAS, FC, Solidstate
RAID
DAS
SAN

“ Large scale storage requires careful planning”

Choosing your Hard Disk
(SATA, FC, SAS, SCSI, Solidstate)

Introduction to Hard Drives
Basic physical storage unit (aka Physical block device)
Variables to consider when selecting a drive
Type (SAS, SATA, FC)
RPM
Capacity
MTBF (Mean Time between Failures)
Life Expectancy

Hard Disk types SATA (Serial ATA) SAS (Serial Attached SCSI) FC (Fibre Channel) Typical Use
low-cost, high-volume, low-speed, large-storage environments
CDP / Backups
Replacement for SCSI
High performance transaction oriented applications with high IOPs requirement
High performance transaction oriented applications with high IOPs requirement
Performance
Average
Typically 7200 RPM
Good (Similar to FC)
10k / 15k RPM
Good (Similar to SAS)
10k / 15k RPM
Hard drive capacities Typically - 250 GB, 500 GB, 750 GB, 1TB Typically – 73 GB, 146 GB, 300 GB, 400 GB Typically – 73 GB, 146 GB, 300 GB, 400 GB

Hard Disk types SATA (Serial ATA) SAS (Serial Attached SCSI) FC (Fibre Channel) Price per Gig (based on max drive capacity retail web price) $ 0.33 $2 $3 Misc -
Backward compatible with SATA
Allows mixing SATA drives on same backplane
-

Hard Disk Conclusions
For high IOPs, database applications, low-storage requirements – you have a choice between FC and SAS
SAS currently seems like the better option
Future SAS standards promise to be faster than FC (though it is likely they may remain neck to neck)
For high-storage requirements (video server, file servers, photo storage, archivals, mail servers, backup servers) SATA is the way to go
One may combine SAS and SATA to reduce average cost and achieve your goals – especially since the backplanes are cross-compatible
Readup the spec sheet of the hard drives you plan on using for determining specifics

Solid State Drives
Uses solid state memory to store persistent data
Eliminates mechanical parts
Useful for creating efficient in-between caches or storing small to mid-sized high performance databases

Solid State Drives
References
Intro - http://en.wikipedia.org/wiki/Solid_state_disk
RAM vs Flash based - http://www.storagesearch.com/ssd-ram-v-flash.html
SSD based SAN!!!  - http://www.superssd.com/
Advantages Disadvantages
Faster startup – no spinning
Significantly faster on Random IO (From 250x to 1000x+)
Extremely low latency (25x to 200x better)
No noise
Lower power consumption
Lesser heat production
Significantly more expensive ($10-30/GB for Flash based, $100-200/GB for DDR RAM based)
Slightly slower on large sequential reads
Slower random write speeds incase of Flash based storage

RAID Primer
(0, 1, 2, 3, 4, 5, 6, TP, 0+1, 10, 50, 60)

Introduction to RAID
allows multiple disks to appear as a single contiguous physical block device
provides redundancy / high availability
A raid group appears as a single physical block device
HD1 HD2 HD1 HD2 RAID

Comparison of Single RAID Levels RAID 0 RAID 1 RAID 5 RAID 6 Diagram Description Striping Mirroring Striping with Parity Striping with Dual Parity Minimum Disks 2 2 3 4 Maximum Disks Controller Dependant 2 Controller Dependant Controller Dependant Array Capacity No. of Drives x Drive Capacity Drive Capacity (No. of Drives - 1) x Drive Capacity (No. of Drives - 2) x Drive Capacity

Comparison of Single RAID Levels RAID 0 RAID 1 RAID 5 RAID 6 Storage Efficiency 100% 50% (Num of drives – 1) / Num of drives (Num of drives – 2) / Num of drives Fault Tolerance None 1 Drive failure 1 Drive failure 2 Drive failures High Availability None Good Good Very Good Degradation during rebuild NA
Slight degradation
Rebuilds very fast
High degradation
Slow Rebuild
(due to write penalty of parity)
Very High degradation
Very Slow Rebuild
(due to write penalty of dual parity)

Comparison of Single RAID Levels RAID 0 RAID 1 RAID 5 RAID 6 Random Read Performance Very Good Good Very Good Very Good Random Write Performance Very Good Good (slightly worse than single drive) Fair (Parity overhead) Poor (Dual Parity Overhead) Sequential Read Performance Very Good Fair Good Good Sequential Write Performance Very Good Good Fair Fair Cost Lowest High Moderate Moderate+

Comparison of Single RAID Levels RAID 0 RAID 1 RAID 5 RAID 6 Use Case
Non critical data
High speed requirements
Data backed up elsewhere
Typically used as RAID 10 in OLTP / OLAP applications
Non-write intensive OLTP applications / file servers etc Non-write intensive OLTP applications / file servers etc Misc - - Parity can considerably slow down system Not supported on all RAID cards

Understanding the Parity Penalty
RAID 5 and RAID 6 store parity information against data for rebuild
Single Parity can be calculated using a simple XOR
eg– “abcdefghijkl” on a 4 disk RAID 5 array
If Disk 2 fails then the data “B” can be recalculated as (01000001 XOR 01000011 XOR 01000000) => 01000010 => B
+12124286429 Disk 1 Disk 2 Disk 3 Disk 4 A (01000001) B (01000010) C (01000011) {P – 01000000} Parity {P} D E F G Parity {P} H I J K Parity {P} L

Steps to change “B” to “X” on Disk 2
Read A, C and {P}
Recalculate {P} as ‘A’ XOR ‘X’ XOR ‘C’
Write ‘X’ and {P}
A single update required 3 reads and 2 writes
Random writes in RAID 5 and RAID 6 are very very expensive
Disk 1 Disk 2 Disk 3 Disk 4 A (01000001) B->X (01000010) -> (01011000) C (01000011) {P – 01000000}

Rebuilding in RAID 5 and RAID 6 is expensive
The cost increases with increase in number of disks
As if this isnt enough there is an additional penalty
All the writes after the computation (ie parity and the changed block) must be simultaneous (involving a two-phase commit operation)
The impact can be marginally reduced through write-back caching

Comparison of Nested RAID Levels RAID 10 RAID 50 Diagram Description Mirroring then Striping Striping with Parity then Striping without parity Minimum Disks Even number > 4 > 6 Maximum Disks Controller Dependant Controller Dependant Array Capacity (Size of Drive) * (Number of Drives ) / 2 (Size of Drive) * (No. of Drives In Each RAID 5 Set - 1) * (No of RAID 5 Sets)

Comparison of Nested RAID Levels RAID 10 RAID 50 Storage Efficiency 50% ((No. of Drives In Each RAID 5 Set - 1) / No. of Drives In Each RAID 5 Set) Fault Tolerance Multiple drive failure as long as 2 drives from same RAID 1 set do not fail Multiple drive failure as long as 2 drives from same RAID 5 set do not fail High Availability Excellent Excellent Degradation during rebuild Minor
Moderate degradation
Slow Rebuild
(due to write penalty of parity)

Comparison of Nested RAID Levels RAID 10 RAID 50 Read Performance Very Good Very Good Write Performance Very Good Good Use Case OLTP / OLAP applications Medium-write intensive OLTP / OLAP applications

Nested RAID Misc Notes
RAID 10 is faster and better than RAID 0+1 for the same cost
RAID 60 is similar to RAID 50 except that the striped sets with parity contain dual parity
Ideally RAID 10 and RAID 50 will be the only nested RAID levels you will use

RAID Considerations
Select your Stripe Size by empirical testing
smaller stripe size increases transfer performance, decreases positioning performance, and vice versa
ideal stripe sizes depend on your application, typical data read in a read, sequential vs random reads etc
Try and select hard drives from separate production batches
Maintain sufficient Spares in a large array (typically 1 per 10-15 disks is sufficient)
Use Global spares across RAID groups if your controller supports it

RAID Considerations
Use hardware RAID unless performance is not a consideration
Especially nested RAID levels or parity based RAID – consume more CPU cycles and increase rebuild time if implemented in software
General rule about Controller Cache – the higher the better
Ensure the controller has battery backup to retain its cache in case of power failure
For internal RAID Controller cards use faster PCI buses (PCI-x)

The Fun starts – Lets build our storage system

Passive Disk Enclosure based Direct Attached Storage (PDE based DAS)

Passive Disk Enclosure based DAS
DAS – Direct Attached storage
RAID controller inside host machine
External chasis is simply a JBOD (Just a Bunch Of Disks)
(or what I’d like to call Passive Disk Enclosure or PDE)
PDE enables stringing larger number of drives together as compared to internal RAID array
Eg Dell Powervault MD1000

Passive Disk Enclosure can consist of SAS, SATA or FC drives
Passive Disk Enclosure to RAID Controller connectivity can be SAS, FC, SCSI (possibly different from the backplane)
Multiple PDEs can be daisy chained if they support it
RAID card is a single point of failure
Only one host machine supported
Array of disks can be divided into multiple RAID groups

Array of disks can be divided into multiple heterogeneous RAID groups
Size and type of a RAID group depends on RAID card
PDE may have multiple paths to system with possibility of multiplexing for increased speed
Global spares can be defined on the RAID card
Maximum storage size = maximum number of PDEs that can be daisy chained x size of drives

Performance Considerations
Drives
RAID configuration
PDE Interconnect
PDE to RAID Card connect
RAID card config (cache etc)
PCI bus

Active Disk Enclosure based Direct Attached Storage (ADE based DAS)

Active Disk Enclosure based DAS
ADE Difference -> RAID Card is not in the host machine but in the enclosure
Host machine has a SAS/FC Host Bus Adaptor (HBA) depending on ADE to Host connectivity support
Some ADEs may support multiple connection protocols
ADE may support SAS/FC/SATA drives
ADE can support daisy-chaining PDEs
Eg of ADE – Dell MD 3000, Infortrend eonstor devices, Nexsan Satabeast and Sataboy etc

Active Disk Enclosure based DAS
ADE may support dual RAID Controllers
RAID Controllers can be used as Active-Active (incase of multiple RAID Groups) – otherwise as Active Passive
RAID Controller to HBA connectivity can be multiplexed - if supported - for higher throughput
ADEs are wrongly but commonly referred as SAN (SAN device would still be alright)

Partitioning and Mounting

Logical Volumes
A RAID Group is a physical unit of storage
At the Operating System a Logical Group can be created out of multiple RAID Groups
Each Logical Group can be further divided into Logical Volumes
Each Logical Volume represents a mountable block device
In Linux this is done using LVM
In LVM Logical Volumes are resizable

SAN (Storage Area Network)

SAN
Multiple host machines connected to an ADE through a SAN switch
SAN refers to the interconnect + Switch + ADE + PDE
Switch and HBA can be SAS / FC depending on interconnect type supported by ADE
ADE would support creation of Volumes
These can be mounted onto Client and further subdivided

SAN
Care must be taken to mount each Logical Volume onto a single client (unless you are running a Clustered File System)
This can be achieved by host masking supported by ADE and/or the Switch
Without careful host masking and mounting data corruption can take place

SAN
Complex SAN configs include multiple hosts and multiple ADEs connected to active-active switches with multiplexed connections
Client hosts can be of heterogeneous operating systems
(Funnily ADE to PDE paths sometimes are not be multiplexed)

SAN
While this looks complex – just think of it as removing hard disks from the machine and hosting them outside in separate enclosures
Each machine mounts an independent partition from the SAN

SAN
Performance Considerations
All variables we covered before
Switch config
Ensure that switch / HBA / interconnect does not become the bottleneck and full hdd throughput can be utilized

Throughput Calculations
Hard disk performance – Type, RPM etc
Data distribution and Type of Data access
RAID performance, number of drives, RAID type
RAID card performance – cache, active-active config etc
ADE to switch connection speed
Switch to HBA connection speed
HBA to PCI bus speed

That’s all Folks
“ Lets go build out our Yottabyte arrays and fill ‘em up”
[Considerably exaggerated hyperbole given that the combined space of all computers in the world today (2007) doesn’t add up to 1 Yottabyte (2 ^ 80 bytes). Infact the entire worlds storage is projected to hit 988 exabytes (2 ^ 60) by 2010] [6 th Sep 2007 - http://www.networkworld.com/newsletters/stor/2007/0903stor2.html – Nanotech breakthrough could put entire YouTube contents on an iPod-size device]

Part II sneak preview
Complex SAN configurations
iSCSI
NAS
Clustered Storage
GFS
Backups
Storage Monitoring
Storage Benchmarking
Some Commercial storage vendors

Shameless HR Propaganda Slide
Directi builds cool Web products
Deployed on distributed architecture
Using terrabytes of storage
Used by millions of users
Generating billions of pageviews and transactions
Spanning every possible software engineering technology
For more info visit http://careers.directi.com Blog: http://bhavin.directi.com Mail: [email_address]

http://bhavin.directi.com	156
http://www.kvaes.be	74
http://www.slideshare.net	9
http://192.168.10.100	1
http://www.lmodules.com	1

Demystifying Storage

by bhavintu79 on Oct 26, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 241

Statistics

Demystifying Storage — Presentation Transcript