Masters stretched svc-cluster-2012-04-13 v2

ATS Masters - Storage
© 2012 IBM Corporation
Stretched SVC Cluster

Agenda
 A little history
 Options and issues
 Requirements and restrictions

© 2012 IBM Corporation3
Terminology
 SVC Split I/O Group = SVC Stretched Cluster = SVC Split Cluster
– Two independent SVC nodes in two independent sites + one independent site for Quorum
– Acts just like a single I/O Group with distributed high availability
 Distributed I/O groups – NOT a HA Configuration and not recommended, if one site failed:
– Manual volume move required
– Some data still in cache of offline I/O Group
I/O Group 1 I/O Group 1
Site 1 Site 2
I/O Group 1
I/O Group 2
I/O Group 1 I/O Group 2
Site 1 Site 2
Site 2
Site 1
 Storwize V7000 Split I/O Group not an option:
– Single enclosure includes both nodes
– Physical distribution across two sites not possible
Site 1 Site 2

Early Days - separate racks in machine room
Servers
SVC + UPS SVC + UPS
Fabric
A
Fabric
B
Protected from physical problems
Plumbing leak, fire
“A chance to tackle the guy with a
chainsaw”
Where’s the quorum disk?
Lots of cross-cabling

Fabric presence on both sides of machine room
Servers
SVC + UPS SVC + UPS
Fabric
A
Fabric
B
Cross cabling only needed
for fabrics and SVC nodes
Fabric
B
Fabric
A
Requirement for zero-hop between nodes in an I/O Group
Needed to “zone away” ISL connections within an I/O Group
Each fabric has two sets of zones for the two sets of ports.
Quorum concern remains

Server cluster also stretched across machine room
Server Cluster
SVC + UPS SVC + UPS
Fabric
A
Fabric
B
Fabric
B
Fabric
A
Server Cluster
Can do cluster failover
Where’s the storage?

SVC V4.3 Vdisk (Volume) Mirroring
Server Cluster
SVC + UPS SVC + UPS
Fabric
A
Fabric
B
Fabric
B
Fabric
A
Server Cluster
SVC Volume has a copy
on either side of machine room
Can do cluster failover and
Mirroring allows single Volume
to be visible on both sides

SVC V5.1 – put LW SFPs in nodes for 10km distance
Server Cluster
SVC + UPS SVC + UPS
Fabric
A
Fabric
B
LW SFPs w/SM fibre allows up to 10km
Fabric
B
Fabric
A
Server Cluster
SVC Volume has a copy
at each site
Where’s the quorum?

SVC V5.1 – stretched cluster with 3rd
site -1
Server Cluster
SVC + UPS SVC + UPS
Fabric
A
Fabric
B
Fabric
B
Fabric
A
Server Cluster
Active / passive storage devices
(like DS3/4/5K):
Each quorum disk storage
controller must be connected to
both sites
Ok, but?

site -2
Server Cluster
SVC + UPS SVC + UPS
Fabric
A
Fabric
B
Fabric
B
Fabric
A
Server Cluster
(like DS3/4/5K):
controller must be
connected to both sites

site -3
Server Cluster
SVC + UPS SVC + UPS
Fabric
A
Fabric
B
Fabric
B
Fabric
A
Server Cluster
(like DS3/4/5K):
controller must be
LOTS OF CROSS CABLING!

SVC V6.3-option 1: Same as V5 but farther using DWDM
Server Cluster
SVC + UPS SVC + UPS
Fabric
A
Fabric
B
DWDM allows up to 40km
Speed drops after 10km
Fabric
B
Fabric
A
Server Cluster
(like DS3/4/5K):
controller must be

SVC V6.3-option 1 (cont)
Server Cluster
SVC + UPS
Server Cluster
SVC + UPS
Fabric
A
Fabric
A
Fabric
B
Fabric
B
User chooses number of ISLs on SAN
Still no hops between nodes in an I/O Group
These connections can be on DWDM too.
Two ports per SVC node attached to local switchess
Two ports per SVC node attached to remote switches via DWDM
Hosts and storage attached to local switches, need enough ISLs
3rd
site quorum (not shown) attached to both fabrics
Active or
passive
DWDM over
shared single
mode fibre(s)
0-10 KM Fibre Channel distance
supported up to 8Gbps
11-20KM Fibre Channel distance
21-40KM Fibre Channel distance
User chooses number of ISLs on SAN
Still no hops between nodes in an I/O Group
These connections can be on DWDM too.

SVC V6.3 option 2: Dedicated ISLs for nodes (can use DWDM)
Server Cluster
SVC + UPS
Server Cluster
SVC + UPS
Private
Fabric
C
Private
Fabric
C
Private
Fabric
D
Private
Fabric
D
Public
Fabric
A
Public
Fabric
B
Public
Fabric
B
Public
Fabric
A
At least 1 ISL
Trunk if more than 1
At least 1 ISL
Trunk if more than 1
User chooses number of ISLs on public SAN
Only half of all SVC ports used for host I/O
Two ports per SVC node attached to public fabrics
Two ports per SVC node attached to dedicated fabrics
Hosts and storage attached to public fabrics
3rd
site quorum (not shown) attached to public fabrics
Only half of all SVC ports used for host I/O
Distance now up to 300km
Apps may require less

Configuration Using Brocade Virtual Fabrics
Server Cluster
SVC + UPS
Server Cluster
SVC + UPS
Public
Fabric
A
Private
Fabric
A
Public
Fabric
B
Private
Fabric
B
Private
Fabric
A
Public
Fabric
A
Private
Fabric
B
Public
Fabric
B
Physical switches are partitioned into
two logical switches, two virtual fabrics
Note ISLs/Trunks for private fabrics are dedicated
rather than being shared to guarantee dedicated
bandwidth is available for node to node traffic
Note ISLs/Trunks for private
SANs are dedicated rather than
being shared to guarantee
dedicated bandwidth available
for node to node traffic

Configuration Using CISCO VSANs
Server Cluster
SVC + UPS
Server Cluster
SVC + UPS
Public
VSAN
A
Private
VSAN
A
Public
VSAN
B
Private
VSAN
B
Private
VSAN
A
Public
VSAN
A
Private
VSAN
B
Public
VSAN
B
Note ISLs/Trunks for private
VSANs are dedicated rather
than being shared to guarantee
dedicated bandwidth available
for node to node traffic
Switches/fabrics are partitioned using VSANs
1 ISL per I/O group
Configured as trunk
1 ISL per I/O group
Configured as trunk
Two ports per SVC node attached to public VSANs
Two ports per SVC node attached to private VSANs
Hosts and storage attached to public VSANs
3rd
site quorum (not shown) attached to public VSANs

Split I/O Group – Distance
 The new Split I/O Group configurations will support
distances of up to 300km (same recommendation as for
Metro Mirror)
 However for the typical deployment of a split I/O group
only 1/2 or 1/3rd
of this distance is recommended because
there will be 2 or 3 times as much latency depending on
what distance extension technology is used
 The following charts explain why

Metro/Global Mirror
o Technically SVC supports distances up to 8000km

SVC will tolerate a round trip delay of up to 80ms between
nodes
o The same code is used for all inter-node communication
• Global Mirror, Metro Mirror, Cache Mirroring, Clustering
• SVCs proprietary SCSI protocol only has 1 round trip
o In practice Applications are not designed to support a Write
I/O latency of 80ms

Hence Metro Mirror is deployed for shorter distances (up to
300km) and Global Mirror is used for longer distances

M/etro Mirror: Application Latency = 1 long distance round trip
IBM Presentation Template Full Version
Data center 1 Data center 2
4) Metro Mirror Data transfer to remote site
5) Acknowledgment
Steps 1–6 affect application latency
Steps 7–10 should not affect the application
Server Cluster 1
1) Write request from host
2) Xfer ready to host
3) Data transfer from host
6) Write completed to host
7a) Write request from SVC
8a) Xfer ready to SVC
9a) Data transfer from SVC
10a) Write completed to SVC
SVC Cluster 1
Server Cluster 2
7b) Write request from SVC
8b) Xfer ready to SVC
9b) Data transfer from SVC
10b) Write completed to SVC
SVC Cluster 2
1 round trip

Split I/O Group – Preferred Node local, Write uses 1 round trip
Server Cluster 1
Server Cluster 2
4) Cache Mirror Data transfer to remote site
5) Acknowledgment
SVC Split I/O Group
1 round trip
2 round trips – but SVC
write cache hides this
latency from the host
Node 1 Node 2

Split I/O Group – Preferred node remote, Write = 3 round trips
Server Cluster 1
Server Cluster 2
5) Acknowledgment
SVC Split I/O Group
2 round trips
1 round trip
2 round trips – but SVC
write cache hides this
latency from the host
Node 1 Node 2

Help with some round trips

Some switches and distance extenders use extra buffers and
proprietary protocols to eliminate one of the round trips worth
of latency for SCSI Write commands

These devices are already supported for use with SVC

No benefit or impact inter-node communication

Does benefit Host to remote SVC I/Os

Does benefit SVC to remote Storage Controller I/Os

Split I/O Group – Preferred Node Remote with help, 2 round trips
Steps 1 to 12 affect application latency
Steps 13 to 22 should not affect the application
Server Cluster 1
Server Cluster 2
4) Write+ data transfer to remote site
9) Acknowledgment
SVC Split I/O Group
11) Write completion to remote site
21) Write completion to remote site
16) Write+ data transfer to remote site
Distance Extenders
5) Write request to SVC
6) Xfer ready from SVC
7) Data transfer to SVC
10) Write completed from SVC
13) Write request from SVC
14) Xfer ready to SVC
15) Data transfer from SVC
22) Write completed to SVC
17) Write request to storage
18) Xfer ready from storage
19) Data transfer to storage
20) Write completed from
storage
1 round trip
1 round trip
1 round trip – hidden
from the host
Node 1 Node 2

Long Distance Impact
 Additional latency because of long distance
 Light speed in glass: ~ 200.000 km/sec
– 1 km distance = 2 km round trip
 Additional round trip time because of distance:
– 1 km = 0.01 ms
– 10 km = 0.10 ms
– 25 km = 0.25 ms
– 100 km = 1.00 ms
– 300 km = 3.00 ms
 SCSI protocol:
– Read: 1 I/O operation = 0.01 ms / km
• Initiator requests data and target provides data
– Write: 2 I/O operations = 0.02 ms / km
• Initiator announces amount of data, target acknowledges
• Initiator send data, target acknowledge
– SVC’s proprietary SCSI protocol for node-to-node traffic has only 1 round trip
 Fibre channel frame:
– User data per FC frame (Fibre channel payload): up to 2048 bytes = 2KB
• Also for very small user data (< 2KB) a complete frame is required
• Large user data is split across multiple frames

SVC Split I/O Group – Quorum Disk
 SVC creates three Quorum disk candidates on the first three
managed MDisks
 One Quorum disk is active
 SVC 5.1 and later:
– SVC is able to handle the Quorum disk management in a very flexible
way, but in a Split I/O Group configuration a well defined setup is
required.
– -> Disable the dynamic quorum feature using the “override” flag for V6.2
and later
• svctask chquorum -MDisk <mdisk_id or name> -override yes
• This flag is currently not configurable in the GUI
 “Split Brain” situation:
– SVC uses the quorum disk to decide which SVC node(s) should survive
 No access to the active Quorum disk:
– In a standard situation (no split brain): SVC will select one of the other
Quorum candidates as active Quorum

 Quorum disk requirements: Only certain storage supported
– Must be placed in a third, independent site
– Storage box must be fibre channel connected
– ISLs with one hop to Quorum storage system are supported
 Supported infrastructure:
– WDM equipment similar to Metro Mirror
– Link requirement similar to Metro Mirror
• Max round trip delay time is 80 ms, 40 ms each direction
– FCIP to Quorum disk can be used with the following requirements:
• Max round trip delay time is 80 ms, 40 ms each direction
• If fabrics are not merged, routers required
• Independent long distance equipment from each site to Site 3 is required
 iSCSI storage not supported
 Requirement for active / passive storage devices (like DS3/4/5K):
– Each quorum disk storage controller must be connected to both sites
26
SVC Split I/O Group – Quorum Disk

3rd
-site quorum supports Extended Quorum

Split I/O Group without ISLs between SVC
nodes
Minimum
distance
Maximum
distance
Maximum
Link
Speed
>= 0 km = 10 km 8 Gbps
> 10 km = 20 km 4 Gbps
> 20km = 40km 2 Gbps
 SVC 6.3:
– Similar to the support statement in SVC 6.2
– Additional: support for active WDM devices
– Quorum disk requirement similar to Remote Copy
(MM/GM) requirments:
• Max. 80 ms Round Trip delay time, 40 ms each
direction
• FCIP connectivity supported for quorum disk
• No support for iSCSI storage system
Split I/O Group without ISLs between SVC nodes (Classic Split I/O Group)
 SVC 6.2 and earlier:
– Two ports on each SVC node needed to be connected to the “remote” switch
– No ISLs between SVC nodes
– Third site required for Quorum disk
– ISLs with max. 1 hop can be used for storage traffic and Quorum disk attachment
 SVC 6.2 (late) update:
– Distance extension to max. 40 km with passive WDM devices
• Up to 20km at 4Gb/s or up to 40km at 2Gb/s.
• LongWave SFPs for long distances required
• LongWave SFPs must be supported from the switch and WDM vendor

S w it c h 1
S w it c h 2
S w it c h 3
S w it c h 4
A c t iv e
Q u o r u m
S V C n o d e 1 S V C n o d e 2
S e r v e r 1 S e r v e r 2
S t o r a g e S t o r a g e
S i t e 1 S i t e 2
S i t e 3
29
nodes

nodes
S w it c h 1
S w it c h 2
S w it c h 3
S w it c h 4
S w it c h 5 S w it c h 6
I S L ( S e r v e r )
S t o r a g e 3 S t o r a g e 2
S i t e 1 S i t e 2
S i t e 3
A c t . Q u o r u m

nodes
S w it c h 1
S w it c h 2
S w it c h 3
S w it c h 4
S w it c h 5 S w it c h 6
S t o r a g e 3 S t o r a g e 2
S i t e 1 S i t e 2
S i t e 3
D S 4 7 0 0
A c t . Q u o r u m
C t l . A C t l . B

SAN and Buffer-to-Buffer Credits
 Buffer-to-Buffer (B2B) credits
– Are used as a flow control method by Fibre Channel technology and
represent the number of frames a port can store
• Provides best performance
 Light must cover the distance 2 times
– Submit data from Node 1 to Node 2
– Submit acknowledge from Node 2 back to Node 1
 B2B Calculation depends on link speed and distance
– Number of multiple frames in flight increase equivalent to the link speed

Split I/O Group without ISLs: Long distance
configuration
 SVC Buffer to Buffer credits
– 2145–CF8 / CG8 have 41 B2B credits
•Enough for 10km at 8Gb/sec with 2 KB payload
– All earlier models:
•Use 1/2/4Gb/sec fibre channel adapters
•Have 8 B2B credits which is enough for 4km at 4Gb/sec
 Recommendation 1:
– Use CF8 / CG8 nodes for more than 4km distance for best performance
 Recommendation 2:
– SAN switches do not auto-negotiate B2B credits and 8 B2B credits is the default setting so change
the B2B credits in the switch to 41 as well
L in k s p e e d F C fr a m e
le n g th
R e q u ir e d B 2 B
c r e d its fo r 1 0 k m
d is ta n c e
M a x d is ta n c e w ith
8 B 2 B c r e d its
1 G b /s e c 1 k m 5 1 6 k m
2 G b /s e c 0 .5 k m 1 0 8 k m
4 G b /s e c 0 .2 5 k m 2 0 4 k m
8 G b /s e c 0 .1 2 5 k m 4 0 2 k m
L in k s p e e d F C fr a m e
le n g th
R e q u ir e d B 2 B
c r e d its fo r 1 0 k m
d is ta n c e
M a x d is ta n c e w ith
8 B 2 B c r e d its
1 G b /s e c 1 k m 5 1 6 k m
2 G b /s e c 0 .5 k m 1 0 8 k m
4 G b /s e c 0 .2 5 k m 2 0 4 k m
8 G b /s e c 0 .1 2 5 k m 4 0 2 k m

Split I/O Group with ISLs between SVC nodes
P r iv . S A N 1
P u b l. S A N 2
P r iv . S A N 1
P u b l. S A N 2
S w it c h S w it c h
S V C - 0 1 S V C - 0 2
I S L
I S L
I S L
I S L
S e r v e r 1
S e r v e r 2
S e r v e r 3
S e r v e r 4
P u b l. S A N 1 P u b l. S A N 1
I S L
P r iv . S A N 2 P r iv . S A N 2
I S L
S it e 1 S it e 2
S it e 3
A c t . Q u o r u m
C t l . A C t l . B
Q u o r u m c a n d i d a t e
S t o r a g e
Q u o r u m c a n d i d a t e
S t o r a g e
W D M W D M
W D M W D M

Long distance with ISLs between SVC nodes
 Some switches and distance extenders use extra buffers and proprietary
protocols to eliminate one of the round trips worth of latency for SCSI Write
commands
– These devices are already supported for use with SVC
– No benefit or impact inter-node communication
– Does benefit Host to remote SVC I/Os
– Does benefit SVC to remote Storage Controller I/Os
 Consequences:
– Metro Mirror is deployed for shorter distances (up to 300km)
– Global Mirror is used for longer distances
– Split I/O Group supported distance will depend on application latency restrictions
• 100km for live data mobility (150km with distance extenders)
• 300km for fail-over / recovery scenarios
• SVC supports up to 80ms latency, far greater than most application workloads would tolerate

Split I/O Group Configuration: Examples
P r iv .S A N 1
P u b l.S A N 2
P r iv. S A N 1
P u b l.S A N 2
S w itc h S w itc h
S V C - 0 1 S V C - 0 2
IS L
IS L
IS L
IS L
S e r v e r 1
S e r v e r 2
S e r v e r 3
S e r v e r 4
P u b l. S A N 1 P u b l.S A N 1
IS L
P r iv.S A N 2 P r iv.S A N 2
IS L
S it e 1 S it e 2
S it e 3
A c t . Q u o r u m
C t l. A C t l. B
Q u o r u m c a n d id a t e
S t o r a g e
Q u o r u m c a n d id a t e
S to r a g e
W D M W D M
W D M W D M
Example 3)
Configuration without live data mobility :
 VMware ESX with SRM, AIX HACMP, or MS Cluster
 Distance between sites: 180km
-> Only SVC 6.3 Split I/O Group with ISLs is supported or
-> Metro Mirror configuration
 Because of long distances: only in active / passive configuration
Example 1)
Configuration with live data mobility:
 VMware ESX with VMotion or AIX with live partition mobility
-> SVC 6.3: Configuration with or without ISLs are supported
-> SVC 6.2: Only configuration without ISLs is supported
Example 2)
Configuration with live data mobility :
 VMware ESX with VMotion or AIX with live partition mobility
-> Only SVC 6.3 Split I/O Group with ISLs is supported.

Split I/O Group - Disaster Recovery
 Split I/O groups provide distributed HA functionality
 Usage of Metro Mirror / Global Mirror is recommended for disaster
protection
 Both major Split I/O Group sites must be connected to the MM / GM
infrastructure
 Without ISLs between SVC nodes:
– All SVC ports can be used for MM / GM connectivity
 With ISLs between SVC nodes:
– Only MM / GM connectivity to the public SAN network is supported
– Only 2 FC ports per SVC node will be available for MM or GM and will also be
used for host to SVC and SVC to disk system I/O
• Thus limited capability currently
• Congestion on GM ports would affect host I/O, but not node-to-node (heartbeats,
etc.)
• Might need more than one cluster to handle all traffic
– More expensive, more ports and paths to deal with

Summary
 SVC Split I/O Group:
– Is a very powerful solution for automatic and fast handling of storage failures
– Transparent for servers
– Perfect fit in a vitualized environment (like VMware VMotion, AIX Live Partition
Mobility)
– Transparent for all OS based clusters
– Distances up to 300 km (SVC 6.3) are supported
 Two possible scenarios:
– Without ISLs between SVC nodes (classic SVC Split I/O Group)
• Up to 40 km distance with support for active (SVC 6.3) and passive (SVC 6.2) WDM
– With ISLs between SVC nodes:
• Up to 100 km distance for live data mobility (150 km with distance extenders)
• Up to 300 km for fail-over / recovery scenarios
 Long distance performance impact can be optimized by:
– Load distribution across both sites
– Appropriate SAN Buffer to Buffer credits

Masters stretched svc-cluster-2012-04-13 v2

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Masters stretched svc-cluster-2012-04-13 v2

Similar to Masters stretched svc-cluster-2012-04-13 v2 (20)

More from solarisyougood

More from solarisyougood (20)

Recently uploaded

Recently uploaded (20)

Masters stretched svc-cluster-2012-04-13 v2