FC/FCoE - Topologies, Protocols, and Limitations ( EMC World 2012 )

FC/FCoE
Topologies, Protocols
and Limitations

Erik Smith
Consulting Technologist – Connectrix BU

© Copyright 2012 EMC Corporation. All rights reserved. 1

Abstract
 An in-depth discussion of FC and FCoE
protocols focusing on
– topologies currently supported
– topologies under development
– known issues
 Review of current EMC SAN best practices
and reasons behind them


Goals
 Describe technical benefits and limitations of
both FC and FCoE
 Describe currently supported FC and FCoE
topologies and EMC-recommended best
practices
 Discuss known limitations with FC and FCoE


Agenda
 FC or Ethernet
 FC vs. FCoE vs. iSCSI
 Supported Topologies
 Best Practices
 Futures


FC or Ethernet

Today


FC or Ethernet
32G ??
Fibre Channel
16G

Today


FC or Ethernet
32G ??
Fibre Channel
16G

Today
??
10G 40/100G Ethernet


FC or Ethernet
32G ??
Fibre Channel
16G

Today
??
10G 40/100G Ethernet

 If you’re asking yourself this question, you’re
not alone


FC or Ethernet
 You don’t really need to decide right now
Physical
Fibre Channel
Today Time Line
Physical
Ethernet


FC or Ethernet
Physical
Fibre Channel
Today Time Line
Physical
Ethernet

Host Network Storage


FC or Ethernet
Physical
Fibre Channel
Today
Physical
Ethernet



FC or Ethernet
Physical
Fibre Channel
Today
Physical
Ethernet

FC FC FC



FC or Ethernet
Physical
Fibre Channel
Today
Physical
Ethernet

FCoE FC


FC or Ethernet
Physical
Fibre Channel
Today
Physical
Ethernet



FC or Ethernet Summary
 Ethernet wins eventually
– There’s still plenty of time to decide what this
means to you
 Migrating to Ethernet does not equal rip and
replace all FC
– Evolutionary versus Revolutionary


Agenda
 FC or Ethernet
– FC/FCoE vs. iSCSI
– FC vs. FCoE
 Best Practices
 Futures


FC/FCoE vs. iSCSI
 It’s more about what’s right for your
environment and less about which protocol
is better


FC/FCoE vs. iSCSI
Latency (usec)

Store and Forward

Cut through

0 5 10 15 20 25


FC/FCoE vs. iSCSI
Latency (usec)

SAS

FC

SSD
>=10x
Store and Forward

Cut through

0 1000 2000 3000 4000 5000 6000 7000


FC/FCoE vs. iSCSI
Latency (usec)

SAS

FC

SSD
>=325x
Store and Forward

Cut through

0 1000 2000 3000 4000 5000 6000 7000

 With block I/O, uncongested network latency
is practically a rounding error


FC/FCoE vs. iSCSI
Host 1

A Fabric (FC or DCB)
B

Host 2

A Storage 1A
B

Host 3
Storage 2A
A
B

Host 4
Storage 1B
A
B

Host 5 Storage 2B

A
B

Host 6

A
B


FC/FCoE vs. iSCSI
Host 1

B

Host 2

A Storage 1A
B
Zone 1

Host 3
Storage 2A
A Zone 2
B

Host 4
Storage 1B
A
B

Host 5 Storage 2B

A

Zones are created by grouping the WWPNs of the
B

Host 6 host interface and storage interface into a “zone”.
A
B
The set of zones created are put into a “zone set”
and activated on the fabric.


FC/FCoE vs. iSCSI
Host 1

B

Host 2

A Storage 1A
B
Zone 1
Zone 3
Host 3
Storage 2A
Zone 5
A Zone 2
B
Zone 4

Zone 7 6
Zone
Host 4
Storage 1B
Zone 9
A
B Zone 11
Zone 8

Zone 10
Host 5 Storage 2B

A
Zone 12
B

Host 6

A
The number of zones in the fabric should always be
B greater than, or equal to, the number of initiators in
the fabric.


FC/FCoE vs. iSCSI
Host 1

A LAN
B

Host 2

A Storage 1A
B

Host 3
Storage 2A
A
B

Host 4
Storage 1B
A
B

Host 5 Storage 2B

A
B

Host 6
With iSCSI, each host needs to be individually and
manually pointed at a storage port by specifying
A
B

either an IP Address, IQN, or both.


FC/FCoE vs. iSCSI
FC vs. FCoE vs. iSCSI provisioning steps…


FC/FCoE vs. iSCSI
...on …on ...on
Total
host network storage


FC/FCoE vs. iSCSI
...on …on ...on
OS Total

Windows

Linux

VMware


FC/FCoE vs. iSCSI
...on …on ...on
OS Protocol Total
FC
Windows FCoE
iSCSI
FC
Linux FCoE
iSCSI
FC
VMware FCoE
iSCSI


FC/FCoE vs. iSCSI
...on …on ...on
OS Protocol Total
FC 7
Windows FCoE 7
iSCSI 7
FC 7
Linux FCoE 7
iSCSI 7
FC 7
VMware FCoE 7
iSCSI 7


FC/FCoE vs. iSCSI
...on …on ...on
OS Protocol Total
FC 2 7
Windows FCoE 7
iSCSI 7
FC 1 7
Linux FCoE 7
iSCSI 7
FC 1 7
VMware FCoE 7
iSCSI 7


FC/FCoE vs. iSCSI
...on …on ...on
OS Protocol Total
FC 2 5 7
Windows FCoE 7
iSCSI 7
FC 1 5 7
Linux FCoE 7
iSCSI 7
FC 1 5 7
VMware FCoE 7
iSCSI 7


FC/FCoE vs. iSCSI
...on …on ...on
OS Protocol Total
FC 2 5 7 14
Windows FCoE 7
iSCSI 7
FC 1 5 7 13
Linux FCoE 7
iSCSI 7
FC 1 5 7 13
VMware FCoE 7
iSCSI 7


FC/FCoE vs. iSCSI
...on …on ...on
OS Protocol Total
FC 2 5 7 14
Windows FCoE 7
iSCSI 19 7
FC 1 5 7 13
Linux FCoE 7
iSCSI 14 7
FC 1 5 7 13
VMware FCoE 7
iSCSI 23 7


FC/FCoE vs. iSCSI
...on …on ...on
OS Protocol Total
FC 2 5 7 14
Windows FCoE 7
iSCSI 19 7 7
FC 1 5 7 13
Linux FCoE 7
iSCSI 14 7 7
FC 1 5 7 13
VMware FCoE 7
iSCSI 23 7 7


FC/FCoE vs. iSCSI
...on …on ...on
OS Protocol Total
FC 2 5 7 14
Windows FCoE 7
iSCSI 19 7 7 33
FC 1 5 7 13
Linux FCoE 7
iSCSI 14 7 7 28
FC 1 5 7 13
VMware FCoE 7
iSCSI 23 7 7 37


FC/FCoE vs. iSCSI
...on …on ...on
OS Protocol Total
FC 2 5 7 14
Windows FCoE 2 7
iSCSI 19 7 7 33
FC 1 5 7 13
Linux FCoE 1 7
iSCSI 14 7 7 28
FC 1 5 7 13
VMware FCoE 1 7
iSCSI 23 7 7 37


FC/FCoE vs. iSCSI
...on …on ...on
OS Protocol Total
FC 2 5 7 14
Windows FCoE 2 37 7
iSCSI 19 7 7 33
FC 1 5 7 13
Linux FCoE 1 37 7
iSCSI 14 7 7 28
FC 1 5 7 13
VMware FCoE 1 37 7
iSCSI 23 7 7 37


FC/FCoE vs. iSCSI
...on …on ...on
OS Protocol Total
FC 2 5 7 14
Windows FCoE 2 37 7 46
iSCSI 19 7 7 33
FC 1 5 7 13
Linux FCoE 1 37 7 45
iSCSI 14 7 7 28
FC 1 5 7 13
VMware FCoE 1 37 7 45
iSCSI 23 7 7 37


FC/FCoE vs. iSCSI
 Management during runtime
– FC/FCoE perform some amount of self
documentation due to FC Login and Name
Server registrations


FC/FCoE vs. iSCSI Summary
 It’s more about what’s right for your
environment and less about which protocol
is better
– Network/Network stack latency are not
currently the best place to focus on to reduce
response time
– EMC believes
▪ Network-centric is better suited for larger
environments
▪ End-node-centric is better suited for smaller
environments


FC vs. FCoE
 Essentially the same in terms of
– Network-centric
– Similar management tools
– Same multipathing software (for iSCSI as well)
– Similar basic port types
▪ N_Ports / F_Ports vs. VN_Ports and VF_Ports
▪ E_Ports vs. VE_Ports
– Same scalability limits
▪ Number of domains
▪ Number of N_Ports / VN_Ports
▪ Number of hops


FC vs. FCoE
 Completely different transports
– Physical FC versus Physical Ethernet
– Point-to-point links cannot be assumed with FCoE
– FCoE uses PFC instead of BB_Credit
▪ Distance implications


FC vs. FCoE
– Point-to-point links cannot be assumed


FC vs. FCoE

Must support FIP Snooping


FC vs. FCoE


FC vs. FCoE
VLAN 1

VLAN 2


FC vs. FCoE
– Point to point links cannot be assumed

DO NOT DO THIS! Only one Fabric per VLAN


FC vs. FCoE
– FCoE uses PFC instead of BB_Credit
▪ Distance implications


FC vs. FCoE

+16

+16
15
14

0
Waiting for
R_RDY


FC vs. FCoE


FC vs. FCoE

Buffer reaches
High water mark


FC vs. FCoE

Effective
bandwidth
10G

Distance
1k 5k Max Max+x

FC vs. FCoE Summary
 FCoE is FC
– Management tools
– Basic concepts
– Multipathing
– Scalability limits
 Transport use has implications
– Just because you can, doesn’t mean you should
▪ One fabric per VLAN
▪ Pay attention to topologies that are being created
– Do not use FCoE for distances that exceed the
maximum supported by the hardware vendor


Agenda
 FC or Ethernet
– General Guidelines
– Vendor-Specific
 Best Practices
 Futures


Supported Topologies
General Guidelines

 FC
– Maximum 5 hops
– Maximum 55 domains
– Maximum 6000 N_Ports
– Maximum 10,000 N_Ports per L3 SAN
– Avoid heterogeneous FC-SW interop (please)
▪ Use NPIV if possible


General Guidelines

 FCoE
– All FC guidelines apply (including multi-hop)
– Heterogeneous FC-SW interop is not supported
– When possible, connect directly to an FCF
– When not possible, use a FIP Snooping Bridge


Vendor-Specific

 There are at least 100 possible topologies
– Refer to either of the EMC FCoE TechBooks


Vendor-Specific – Cisco – End Device Connectivity

EMC

N7k MDS

N5k
FC
N2k
FCoE
Host



EMC

N7k MDS

N5k
FC
N2k
FCoE
Host


Vendor-Specific – Cisco – Network Connectivity
FC
FCoE

N7k MDS

N5k FC
Fabric
N2k


FC
FCoE

N7k MDS

N5k FC
Fabric
N2k


Vendor-Specific – Cisco – FIP Snooping Bridge Support

FCoE

N7k MDS

N5k

FSB


Cisco Summary

 Over 100 supported Cisco topologies are
described in the FCoE TechBook
 Extensive support for multi-hop FCoE
 Support for FIP Snooping Bridges
 Connectivity to existing FC SAN does not
require the use of an FC router


Vendor-Specific – Brocade – End Device Connectivity

EMC

DCX
VDX
8000
FC

FCoE
Host



EMC

DCX
VDX
8000
FC

FCoE
Host


Vendor-Specific – Brocade – Network Connectivity
FC
FCoE
EEx
DCX
8000
FC
Fabric
VCS EEx
VDX IR

VDX


FC
FCoE
EEx

8000 Ex DCX
FC
Fabric
VCS EEx
VDX IR
Ex
VDX


Vendor-Specific – Brocade – FIP Snooping Bridge

FCoE

Brocade

FSB


Brocade Summary

 A number of supported Brocade topologies
are described in the FCoE TechBook
 Support for multi-hop FCoE in VCS only
 No support for FIP Snooping Bridges
 Connectivity to existing FC SAN requires the
use of an FC router


Agenda
 FC or Ethernet
 Best Practices
 Futures


Best Practices
 Maximum hops
 Maximum N_Ports
 Single Initiator Zoning
 Monitor for bit errors


Best Practice – Max Hops
 Best practice
– Do not exceed 5 hops
 Reason
– Concern about data corruption
– Concern about fabric segmentation in certain
circumstances



Example: 4 - Layer 2 Ethernet Hops

4 Ethernet switches



Example: 3 FC Hops

4 FC switches


 Switches contain queues
– Used for buffering
– These queues have a “hold time” value
associated with them

 Queue hold time
– Length of time a frame is held before it’s discarded
– Typically between 500ms and 2 seconds


 R_A_TOV
– Maximum life span of a frame x 2 (FC-FS)
– Typically 10 seconds

R_A_TOV >= (max hops * Hold time) * 2


Ethernet (Row) FC SAN

Sw mod

Host

Blade Server

Rack Rack



ToR

1

Sw mod

Host

Blade Server

Rack Rack



ToR 2
EoR
1

Sw mod

Host

Blade Server

Rack Rack



ToR 2 3 SAN
EoR
Core
1

Sw mod

Host

Blade Server

Rack Rack



ToR 2 3 SAN
EoR
Core
1
4
Sw mod
SAN
Host Core

Blade Server

Rack Rack



ToR 2 3 SAN
EoR
Core
1
4
Sw mod
SAN
Host Core

5
Blade Server
Storage

Rack Rack


Best Practice – Max Hops Summary
 Do not exceed 5 hops
– If you must, ensure that the configuration will
never create a condition where frames older than
½ of R_A_TOV will exist
▪ IOW, you need to know the hold time for every element
▪ Consider error conditions and failures

 Links between any network elements that
contain buffers must be counted as a FC Hop
– This includes FSBs and NPIV Gateways


Best Practice – Max N_Ports
 Best practice
– The number of N_Ports should not exceed 6000
 Reason
– Originally
▪ Testing revealed fabric segmentation due to HLO
timeout between switches with around 1000 N_Ports


Best Practice – Max N_Ports Summary
 Do not exceed the maximum number of
N_Ports supported
– Results will be unpredictable
– In extreme cases, fabric segmentation can result


Best Practice – Single Initiator Zoning
 Best practice
– Each zone should only contain one initiator and
the targets it will access
 Reason
– Originally
▪ Concern about initiators logging in to other initiators
– Today
▪ Limits the number of name server queries that are sent
to the switch
▪ Related to Max N_Ports
▪ Recommended still due to its use in test configurations


Host 1

B

Host 2

A Storage 1A
B

Host 3
Storage 2A
A
B

Host 4
Storage 1B
A
B

Host 5 Storage 2B

A
B

Host 6

A
B


Host 1

B

Host 2

A Storage 1A
B

Host 3
Storage 2A
A
B

Host 4
Storage 1B
A
B

Host 5 Storage 2B

A
B

Host 6
Without zoning, the response would include information about
15 other N_Ports.
A
B


Host 1

B

Host 2

A Storage 1A
B

Host 3
Storage 2A
A
B

Host 4
Storage 1B
A
B

Host 5 Storage 2B

A
B

Host 6

A
For each N_Port returned in the NS Response, host will query
B the Name Server for additional information. If a host only
needs to access 1 Target this means 14 extra NS queries per
N_Port.


 Initiator login with Single Initiator Zoning


 Initiator login without Zoning


Host 1

B

Host 2

A Storage 1A
B

Host 3
Storage 2A
A
B

Host 4
Storage 1B
A
B

Host 5 Storage 2B

A
B

Host 6

A
Host will attempt PLOGI/PRLI with all N_Ports and perform
B report LUNs with all Targets.


Single Initiator Zoning Summary
 Use single Initiator / single target Zoning if
at all possible
 If not possible or practical
– e.g., some VMware and RecoverPoint
environments
– Learn to watch for the signs
▪ Randomly missing N_Port logins during power on or
after link events
▪ Elongated boot times
– Target Driven Zoning will help when available!


Monitor for Bit Errors
 Best practice
– Monitor for bit errors
 Reason
– VERY bad for performance
– FC – Buffer loss
– FCoE – Bit errors can cause unintentional data
distribution
▪ An I/O convergence issue not an FCoE one


Agenda
 FC or Ethernet
 Best Practices
 Futures
– TDZ
– Network Virtualization


Futures — Target Driven Zoning (TDZ)
• TDZ is an EMC initiative aimed at
automating the provisioning of networked
storage
• TDZ allows for the SAN to automatically
configure zoning based on information
provided to it by a storage port
• TDZ makes use of Peer Zoning
– Approved in FC-GS-6 and FC-SW-6


Futures – Network Virtualization
• Network Virtualization
– Generic term being used to describe “Overlay
Networks”
– Encapsulations being discussed to support this
functionality are
• NVGRE
• VXLAN
• STT
– Could be very disruptive to today’s SANs


Futures
• Join us for more information!
• Birds-of-a-Feather: Storage Networking for
the Future
– Time: Tuesday 1:30p
– Room: Marcello 4401 A


In Summary
 Physical FC will be around for a while
 Migrating to Ethernet
– Can be done at any time
– Does not require rip and replace
 Follow Best practices
– Unless you have a compelling reason not to
– Understand the risks
 The EMC FCoE Tech books
– contain hundreds of supported topologies; and
– detailed installation instructions


Additional information
 Please check out my blog
www.brasstacksblog.typepad.com


Additional information
 Or follow me on twitter!
@ErikSmith7


Q&A


Provide Feedback & Win!

 125 attendees will receive
$100 iTunes gift cards. To
enter the raffle, simply
complete:
– 5 sessions surveys
– The conference survey
 Download the EMC World
Conference App to learn
more: emcworld.com/app


FC/FCoE - Topologies, Protocols, and Limitations ( EMC World 2012 )

FC/FCoE - Topologies, Protocols, and Limitations ( EMC World 2012 )

More Related Content

What's hot

Similar to FC/FCoE - Topologies, Protocols, and Limitations ( EMC World 2012 )

More from EMC

Recently uploaded

FC/FCoE - Topologies, Protocols, and Limitations ( EMC World 2012 )