A Technical Overview of Enterprise Level Storage Systems Amnon Naamad Open Lecture at EMC St. Petersburg Development Centre May, 15, 2008

Enterprise Level Storage Systems Some History Storage Systems’ Topologies and operations What is performance? Challenges: What is expected of Enterprise Level Storage Innovation today and in the future

Some History

Storage Systems’ Topologies and operations

What is performance?

Challenges: What is expected of Enterprise Level Storage

Innovation today and in the future

History

Storage Technology Has Gone Through Dramatic Changes Since 1990

IBM 3390 (1990) Total Capacity 22.7 GB Price $275,000 Price per GB $10,000 Protection None Max Transfer 24 MB / sec Max IO/s 360

What contributed to the big improvements RAID Redundant Array of Inexpensive Disks Improvements in technology Competition Huge investments in Engineering Market demand for storage grows 40-60% per year External Storage Market size is about $40B / Year

RAID

Redundant Array of Inexpensive Disks

Improvements in technology

Competition

Huge investments in Engineering

Market demand for storage grows 40-60% per year

External Storage Market size is about $40B / Year

RAID 5 3:1 Layout Detail Data Parity 25, 26, 27, 28 1 - 12 13, 14, 15, 16 37, 38, 39, 40 Disk 1 29, 30, 31, 32 1, 2, 3, 4 13 - 24 41, 42, 43, 44 25 - 36 5, 6, 7, 8 17, 18, 19, 20 45, 46, 47, 48 33, 34, 35, 36 9, 10, 11, 12 21, 22, 23, 24 37 - 48 Disk 2 Disk 3 Disk 4

Topologies and Operations

Storage System Architecture Front-End Director Host Channel Front-End Director Front-End Director Front-End Director Cache Back-End Director Back-End Director Back-End Director Back-End Director

Role of Various Components Front-End Directors Interpret the host commands Data transfer between Hosts and cache Cache Keeps user data Keeps meta data Back-End Director Data transfer between cache and disks Directors divided into several mostly independent CPU units (slices)

Front-End Directors

Interpret the host commands

Data transfer between Hosts and cache

Cache

Keeps user data

Keeps meta data

Back-End Director

Data transfer between cache and disks

Directors divided into several mostly independent CPU units (slices)

EMC Symmetrix DMX-3000

DMX-4

Data Organization The data in the system is organized and accessed by Logical volumes In the past, disks were also called volume Number of logical volumes can be several thousands Two types of data organization - Mainframe - Open Systems Enterprise storage supports both Mainframe and Open Systems volumes

The data in the system is organized and accessed by Logical volumes

In the past, disks were also called volume

Number of logical volumes can be several thousands

Two types of data organization

- Mainframe

- Open Systems

Enterprise storage supports both Mainframe and Open Systems volumes

Mainframe Still Reflects the structure of the 3390 The 3390 had Each disk (volume) had 3 GB of data Track contained 56 KB of data 8 plates. Each plate had data on both sides Cylinder is a collection of 15 tracks having the exact same radius Today’s disks have a totally different physical structure (next slide), but Logical volume size is 3 GB / 9 GB / 27 GB Track remains 56 KB, 15 tracks per cylinder Accessing data is by: Volume #, Cylinder #, track #, record # Structures of records may vary. The exact format is part of the data. The Mainframe data organization is also called CKD: Count Key Data

The 3390 had

Each disk (volume) had 3 GB of data

Track contained 56 KB of data

8 plates. Each plate had data on both sides

Cylinder is a collection of 15 tracks having the exact same radius

Today’s disks have a totally different physical structure (next slide), but

Logical volume size is 3 GB / 9 GB / 27 GB

Track remains 56 KB, 15 tracks per cylinder

Accessing data is by: Volume #, Cylinder #, track #, record #

Structures of records may vary. The exact format is part of the data.

The Mainframe data organization is also called CKD: Count Key Data

Open Systems Data Organization The basic unit of data is a block of 512 bytes Addressing is by: Volume #, Block #, length There is no notion of Cylinder or track In Symmetrix data is organized in “logical tracks” Logical Volume size not restricted Each vendor has some limit

The basic unit of data is a block of 512 bytes

Addressing is by: Volume #, Block #, length

There is no notion of Cylinder or track

In Symmetrix data is organized in “logical tracks”

Logical Volume size not restricted

Each vendor has some limit

Cache Management Cache contains: Read Data (LRU) Write Pending data Perma Cache data Meta Data User data in Cache is divided into Slots Mainframe slot size is 56 KB In Open Systems, the slot size varies DMX1,2: 32 KB DMX3,4: 64KB The size of the slot is the same as the size of “logical track” At any given moment, a slot holds data from one track – not necessarily all the data in the track Main Decisions you need to make Which data to bring into cache Which data to discard when we need a new slot

Cache contains:

Read Data (LRU)

Write Pending data

Perma Cache data

Meta Data

User data in Cache is divided into Slots

Mainframe slot size is 56 KB

In Open Systems, the slot size varies

DMX1,2: 32 KB

DMX3,4: 64KB

The size of the slot is the same as the size of “logical track”

At any given moment, a slot holds data from one track – not necessarily all the data in the track

Main Decisions you need to make

Which data to bring into cache

Which data to discard when we need a new slot

The Various I/O Commands From the server’s perspective there are: Reads and Writes From the storage system’s perspective there are: Read Hit, Read Miss, Sequential Read Sequential Write, Random Write Fast Write, Delayed Fast Write

From the server’s perspective there are:

Reads and Writes

From the storage system’s perspective there are:

Read Hit, Read Miss, Sequential Read

Sequential Write, Random Write

Fast Write, Delayed Fast Write

Read Hit Front-End Director Cache Host Channel Front-End Director Back-End Director Front-End Director Front-End Director Back-End Director Back-End Director Back-End Director Cache Hit

Staging Front-End Director Cache Host Channel Front-End Director Back-End Director Front-End Director Front-End Director Back-End Director Back-End Director Back-End Director Load Balance Disks Read Miss

Prefetch Front-End Director Cache Host Channel Front-End Director Back-End Director Front-End Director Front-End Director Back-End Director Back-End Director Back-End Director Load Balance Disks Sequential Read / Prefetch

De-staging (later) Front-End Director Cache Host Channel Front-End Director Back-End Director Front-End Director Front-End Director Back-End Director Back-End Director Back-End Director Fast Write to Cache Fast Write is Done in Two Stages

De-staging (later)

Fast Write to Cache

Write Pending Limits “ The mall parking lot can accommodate 1 million cars…. 5 thousand at a time” De-staging Front-End Director Cache Host Channel Front-End Director Back-End Director Front-End Director Front-End Director Back-End Director Back-End Director Back-End Director

“ The mall parking lot can accommodate 1 million cars…. 5 thousand at a time”

What is Performance?

Some Performance Envelop Numbers (DMX1, 2003) Read Hit 12,000 per FE CPU ~370,000 per system Read Miss Up to 350 per disk 3,000 per BE CPU Max 90,000 per system Bandwidth: Read Hit: 3 GB/sec Sequential Read: 2 GB/sec Random Write: 25% - 50% of Read, depends on protection scheme

Read Hit

12,000 per FE CPU

~370,000 per system

Read Miss

Up to 350 per disk

3,000 per BE CPU

Max 90,000 per system

Bandwidth:

Read Hit: 3 GB/sec

Sequential Read: 2 GB/sec

Random Write:

25% - 50% of Read, depends on protection scheme

Performance is Multidimensional Response times charts Maximum IO/s Bandwidth Mix of I/O Types Degraded Modes Internal Applications and operations, time and impact # of I/Os with response times above X ms Phase Transition Cache Management SymmOptimizer Adaptive Algorithms Differentiated Performance (QoS) Performance planning, monitoring and troubleshooting tools

Response times charts

Maximum IO/s

Bandwidth

Mix of I/O Types

Degraded Modes

Internal Applications and operations, time and impact

# of I/Os with response times above X ms

Phase Transition

Cache Management

SymmOptimizer

Adaptive Algorithms

Differentiated Performance (QoS)

Performance planning, monitoring and troubleshooting tools

Response Time Charts

DMX Prefetch Algorithm is Adaptive XXX Approach When two consecutive I/Os are detected, 1 MB is prefetched DMX Approach Each LUN has its own sequential patterns Collect statistics that will help to determine quickly when a sequence is being read Prefetch more as the sequence gets longer Get rid of sequential data in cache quickly Sequential Length Frequency

XXX Approach

When two consecutive I/Os are detected, 1 MB is prefetched

DMX Approach

Each LUN has its own sequential patterns

Collect statistics that will help to determine quickly when a sequence is being read

Prefetch more as the sequence gets longer

Get rid of sequential data in cache quickly

Prefetch – XXX Long and Short Sequences Response Time in ms I/Os per Second

Prefetch – DMX Long and Short Sequences Response Time in ms I/Os per Second

DMX Delayed Fast Write

In Heavy Write Scenarios XXX Arrays STALL! Once the write pending limit is reached, the array stops accepting writes from the host for 40 seconds. 32 Gbytes of cache X a write pending limit of 70% is 22.4 Gbytes. Divided by 2 is approximately 11+ Gbytes of dual write usable cache. A cache slot size of 64Kbytes yields approximately 180,000 slots. The NSC consumes the write cache in 2.5 seconds at 68,000 IOs per second.

32 Gbytes of cache X a write pending limit of 70% is 22.4 Gbytes.

Divided by 2 is approximately 11+ Gbytes of dual write usable cache.

A cache slot size of 64Kbytes yields approximately 180,000 slots.

The NSC consumes the write cache in 2.5 seconds at 68,000 IOs per second.

What is Expected of Enterprise Storage Systems

Enterprise Storage - Challenges Remote protection and replication Local replication Protection against any single HW failure Minimal impact on performance Predictable Performance 24 x 7 x 365 No down time - ever Non disruptive maintenance and upgrades Self Healing Data unavailability is very expensive ($1M / Hour) Inventory is very expensive; you need a lot to verify good quality Need to be backwards compatible to retain existing customers

Remote protection and replication

Local replication

Protection against any single HW failure

Minimal impact on performance

Predictable Performance

24 x 7 x 365

No down time - ever

Non disruptive maintenance and upgrades

Self Healing

Data unavailability is very expensive ($1M / Hour)

Inventory is very expensive; you need a lot to verify good quality

Need to be backwards compatible to retain existing customers

No single point of failure Dual Memory Dual Initiator RAID Bug-Free Software The software is written in such a way that it can recover from HW problems More than 80% of the software deals with problems and possible problems in the hardware, to support non disruptive maintenance, and to guarantee backwards compatibility

Dual Memory

Dual Initiator

RAID

Bug-Free Software

The software is written in such a way that it can recover from HW problems

More than 80% of the software deals with problems and possible problems in the hardware, to support non disruptive maintenance, and to guarantee backwards compatibility

Innovation Today and in the Future

Customers Want Efficiency and Effectiveness Buy as little as possible Always available with no down time Secure and protect Simplify Power Performance New Features to enable new services (SRDF/A)

Buy as little as possible

Always available with no down time

Secure and protect

Simplify

Power

Performance

New Features to enable new services (SRDF/A)

Many Innovations in the Past Year (3/2007 - 3/2008) (Partial List) Security RAID 6 Dynamic Cache Partitioning Priority QoS Symmetrix Management Console (SMC) Flash Drives Virtual Provisioning Mainframe Enhancements Catching up with IBM Remote Replication Improvements

Security

RAID 6

Dynamic Cache Partitioning

Priority QoS

Symmetrix Management Console (SMC)

Flash Drives

Virtual Provisioning

Mainframe Enhancements

Catching up with IBM

Remote Replication Improvements

Dynamic Cache Partitioning The effect of Cache Partition Shared OLTP and Backup-to-Disk Workload

Symmetrix Priority Controls

Symmetrix Priority Controls

RAID 6 Provides Excellent Protection – With a Performance Penalty In RAID 1 and RAID 5 dual drive failures may result in data loss RAID 6 provides protection against 2 drive failures RAID 6 reduces the probability of data loss (because of disk failure) by at least 3 orders of magnitude over RAID 1 or RAID 5 Impact on write operations

In RAID 1 and RAID 5 dual drive failures may result in data loss

RAID 6 provides protection against 2 drive failures

RAID 6 reduces the probability of data loss (because of disk failure) by at least 3 orders of magnitude over RAID 1 or RAID 5

Impact on write operations

Technologies We are Investigating (partial list) Flash Technology Power savings De-Dupe, Compression Application Awareness Enhancements to remote replication Goal driven Storage / performance management

Flash Technology

Power savings

De-Dupe, Compression

Application Awareness

Enhancements to remote replication

Goal driven Storage / performance management

Cache Simulation helps to optimize algorithms Optimal cache slot size (change to 64 k) Effect of cache size on read hit In general Per customer’s request Evaluation of TBC Algorithm and parameters Destage: Optimal write delay time, Value of destaging in time order / location order Is ARC better than TBC? More ….

Optimal cache slot size (change to 64 k)

Effect of cache size on read hit

In general

Per customer’s request

Evaluation of TBC Algorithm and parameters

Destage:

Optimal write delay time,

Value of destaging in time order / location order

Is ARC better than TBC?

More ….