FCoE - Topologies, Protocol, and Limitations

FCoE ─ Topologies,
Protocols, and
Limitations
Erik Smith

© Copyright 2011 EMC Corporation. All rights reserved.

Objectives
•  Understand topologies that are currently supported
and what is currently being developed
•  Understand the essential components of FCoE
including DCBX, PFC, FIP, and DCB
•  Understand the known limitations when using FCoE

© Copyright 2011 EMC Corporation. All rights reserved. 2

EMC Product Roadmap & Futures
Disclaimer
•  EMC makes no representation and undertakes no
obligations with regard to product planning
information, anticipated product characteristics,
performance specifications, or anticipated release
dates (collectively, “Roadmap Information”).

•  Roadmap Information is provided by EMC as an
accommodation to the recipient solely for
purposes of discussion and without intending to be
bound thereby.


Agenda
•  Topology overview
–  Fibre Channel (FC)
–  Fibre Channel over Ethernet (FCoE) overview
•  Enabling technologies
–  DCBX
–  PFC
–  ETS
–  FCoE Initialization Protocol (FIP)
•  ENode-to-ENode / Direct connect configuration
•  Zero-hop
•  Multi-hop
•  Real world use cases


Topology Overview – Fibre Channel
SAN A

Switch 1A Switch 2A
Storage
Host
HBA A A

HBA B SAN B
B

Switch 1B Switch 2B


SAN A

N_Port N_Port
Switch 1A Switch 2A
Storage
Host
HBA A A
N_Port

HBA B SAN B
B

Switch 1B Switch 2B
N_Port


F_Port F_Port
SAN A

N_Port N_Port
Switch 1A Switch 2A
Storage
Host
HBA A A
N_Port

HBA B SAN B
B

Switch 1B Switch 2B
N_Port

F_Port F_Port


E_Port E_Port

F_Port F_Port
SAN A

N_Port N_Port
Switch 1A Switch 2A
Storage
Host
HBA A A
N_Port

HBA B SAN B
B

Switch 1B Switch 2B
N_Port

F_Port F_Port
E_Port E_Port


Topology Overview – FCoE
VE_Port VE_Port

VF_Port VF_Port
SAN A

VN_Port VN_Port
FCF 1A FCF 2A
Storage
Host
CNA A A
VN_Port

CNA B SAN B
B

FCF 1B FCF 2B
VN_Port

VF_Port VF_Port
VE_Port VE_Port


VE_Port VE_Port

VF_Port VF_Port
SAN A

VN_Port VN_Port
FCF 1A FCF 2A
Storage
Host
CNA A A
VN_Port

CNA B SAN B
B

NIC 1
FCF 1B FCF 2B
NIC 2
VN_Port
NIC n

VF_Port VF_Port
VE_Port VE_Port


VE_Port VE_Port

VF_Port VF_Port
SAN A

VN_Port VN_Port
FCF 1A FCF 2A
Storage
Host
CNA A A
VN_Port

CNA B SAN B
B

NIC 1
FCF 1B FCF 2B
NIC 2
VN_Port
NIC n

VF_Port VF_Port
VE_Port VE_Port

LAN


VE_Port VE_Port

VF_Port VF_Port
SAN A

VN_Port VN_Port
FCF 1A FCF 2A
Storage
Host
CNA A A
VN_Port

CNA B SAN B
B

NIC 1
FCF 1B FCF 2B
NIC 2
VN_Port
NIC n

VF_Port VF_Port
VE_Port VE_Port

VLAN 1
VLAN 2
VLAN n

LAN


VE_Port VE_Port

VF_Port VF_Port
SAN A

VN_Port Ethernet Ethernet VN_Port
Switch / Switch /
FCF 1A FCF 2A Storage
Host
NIC / A
CNA A
SAN B
B

NIC / Ethernet Ethernet
CNA B Switch / Switch /
FCF 1B FCF 2B VN_Port

VN_Port VF_Port VF_Port
VE_Port VE_Port

VLAN 1
VLAN 2
VLAN n

LAN


vPC Domain

SAN A
vPC
Ethernet Ethernet
Switch / Switch /
FCF 1A FCF 2A Storage
Host
NIC / A
CNA A
SAN B
B

NIC / Ethernet Ethernet
CNA B Switch / Switch /
FCF 1B FCF 2B

VLAN 1
VLAN 2
VLAN n

LAN


Agenda
–  DCBX
–  PFC
–  ETS
•  Zero-hop
•  Multi-hop


DCBX ─ Data Center Bridging Capability
eXchange Protocol
Lossless
ENode Ethernet
switch

LINK UP
Each DCBX DCBX DCBX
frame contains DCBX DCBX
priority map
TLVs for both
FIP and FCoE FIP VLAN D
is covery

•  An extension of the Link Layer Discovery Protocol (LLDP)
•  Allows for the exchange of priority map values for both FCoE and the
FCoE Initialization Protocol (FIP)
•  Enables lossless behavior


PFC ─ Priority Flow Control
Transmit Queues Receive Buffers
Ethernet Link

Eight
Virtual
Lanes

Ÿ  Necessary since FC requires a lossless environment to operate properly
Ÿ  Without PFC, normal periodic SAN congestion will cause frames to drop and the
entire exchange will need to be retransmitted
–  Can take up to 60 seconds, depending on which frame in the exchange is lost


ETS – Enhanced Transmission Selection
Desired Traffic ETS configuration 10GbE Link Actual Throughput

iSCSI 3 Gbps 3 Gbps 3 Gbps 2 Gb 3 Gbps 3 Gbps
2 Gbps

Priority (1)
3 Gbps
3 Gbps 3 Gbps
FCoE 3 Gbps 3 Gbps 3 Gbps 3 Gb
Priority (3) 3 Gbps 5 Gbps
4 Gbps

6 Gbps
5 Gb
4 Gbps
LAN 3 Gbps 9 Gbps 10 Gbps 10 Gbps

Priority (5)
t1 t2 t3 t1 t2 t3

Ÿ  ETS information is exchanged in DCBX
Ÿ  Ensures storage traffic has a guaranteed minimum amount of bandwidth
Ÿ  Utilized by all FCoE devices that are currently supported by EMC


FIP ─ FCoE Initialization Protocol
FIP bridges the gap between expectations of the FC protocol & the reality
of an FCoE Toplogy
•  ENode can discover who to log in with
•  LKA (Link Keep Alive) and FIP CVL (Clear Virtual Links) allows for logout from
the fabric should the logical link be lost
•  Implicit security (man-in-the-middle is difficult)
–  Provided that FIP snooping and Dynamic ACLs are implemented

For an in-depth description of the FIP protocol, visit
www.brasstacksblog.typepad.com


Agenda
–  DCBX
–  PFC
–  ETS
•  Zero-hop
•  Multi-hop


Enode-to-ENode – Direct Connect
Important information
CNA CNA CNA CNA
Ÿ  Not supported today
Server Storage
Ÿ  FC-BB-5 requires an FCF
to be present
Ÿ  FC-BB-6 removes this
requirement and allows
for the PT2PT / VN2VN
protocol to be used
Si

CNA CNA CNA CNA CNA CNA CNA CNA

Server Server Storage Storage


Agenda
–  DCBX
–  PFC
–  ETS
•  Zero-hop
•  Multi-hop


Zero-hop Topology ─ Available Today
Features
Ÿ  Benefits
–  SAN A / SAN B
–  Easier to manage SAN
–  Congestion is less likely (optional)
Ÿ  Physical topologies supported
–  Top of Rack
–  End of Row / Middle of Row
–  Out of Row FC FCFs
Ÿ  Products available
Si Si
–  Brocade
§  8000
§  DCX
Com
FCoE
i
§  VCS/VDX g so n CNA CNA
on
–  Cisco
§  7000
§  5000 Server
Storage
§  5000 / 2000
EMC makes no representation and undertakes no obligations with regard to product planning information, anticipated product
characteristics, performance specifications, or anticipated release dates (collectively, “Roadmap Information”). Roadmap Information
is provided by EMC as an accommodation to the recipient solely for purposes of discussion and without intending to be bound
thereby.

Zero-hop ─ Top of Rack (ToR)

Server 1

Server 2

Server n

RACK



FCF 1A
FCF 1B

Server 1

Server 2

Server n

RACK



FCF 1A
FCF 1B

Server 1

Server 2

Server n

RACK

Ÿ  Physical connectivity options
–  Twinax, Optical fiber
Ÿ  Scale
–  Up to 96 ports (native FCoE)



FCF 1A FCF 2A FCF nA
FCF 1B FCF 2B FCF nB

Server 1 Server 1 Server 1


Server n Server n Server n

RACK 1 RACK 2 RACK n

Ÿ  Scale




Ethernet
Server 2 Server 2 Server 2 switch A

Ethernet
switch B


RACK 1 RACK 2 RACK n End of Row

Ÿ  Scale




Ethernet
Storage
Ethernet
switch B


RACK 1 RACK 2 RACK n Storage End of Row

Ÿ  Scale




Ethernet
Storage
Ethernet
switch B



Ÿ  Scale




SAN A
Ethernet
Storage
Ethernet
switch B
SAN B



Ÿ  Scale
–  Up to 50 Racks (per SAN)




SAN A
Ethernet
Storage
Ethernet
switch B
SAN B



Ÿ  Physical connectivity options Ÿ  Storage connectivity
–  Twinax, Optical fiber –  ToR
Ÿ  Scale




SAN A
Ethernet
Storage
Ethernet
switch B
SAN B



–  Twinax, Optical fiber –  ToR, OoR
Ÿ  Scale


Zero-hop
End of Row (EoR) / Middle of Row (MoR)

SAN A
FCF A
Storage

FCF B

SAN B



Ÿ  Scale Ÿ  Storage connectivity
–  Up to 512 ports (native FCoE) –  EoR, OoR
–  Up to 512 servers
–  Twinax (<=10m), Optical fiber


Zero-hop ─ Out of Row (OoR)

SAN A
FCF A
Storage

FCF B
SAN B


RACK 1 RACK 2 RACK n Storage Out of
Row Row
Ÿ  Scale Ÿ  Storage connectivity
–  Up to 512 ports (native FCoE) –  EoR, OoR
–  Up to 512 servers
Ÿ  Cost comparison to FC
Ÿ  Physical connectivity options –  X % +- (Elias)
–  Twinax (<=10m), Optical fiber


Zero-hop ─ Out of Row (OoR)
Server 1 Server 1 Server 1 SAN A SAN B



Row 1 Existing
Data Center
patch panel



Row 2

FCF A

Storage
FCF B



Row n


Agenda
–  DCBX
–  PFC
–  ETS
•  Zero-hop
•  Multi-hop


Multi-hop Topology ─ Available Today
Features
Ÿ  Benefits SAN
(optional)
–  Increased scalability

–  Top of Rack
FC
–  End of Row / Middle of Row Si Si

–  Out of Row

Ÿ  Products available FCFs
–  Brocade
§  DCX (FC ISLs only) Si Si

§  8000 (FC ISLs only
§  VDX (coming soon) FCoE
–  Cisco CNA CNA
§  MDS
§  7000 Server Storage
§  5000
EMC makes no representation and undertakes no obligations with regard to product planning information, anticipated product
characteristics, performance specifications, or anticipated release dates (collectively, “Roadmap Information”). Roadmap Information
is provided by EMC as an accommodation to the recipient solely for purposes of discussion and without intending to be bound
thereby.

Features
(optional)

–  Top of Rack
FC

–  Out of Row
VE_Ports
Ÿ  Products available FCFs
–  Brocade

§  MDS
§  5000


Features
(optional)

–  Top of Rack
FC

–  Out of Row
FCFs ISLs
Ÿ  Products available
–  Brocade

§  MDS
§  5000


Multi-hop (Network Admin POV)
L2/L3 L2/L3 Distribution Layer
Ethernet Ethernet (EoR/OoR)
Switch Switch


Switch Switch

L2/L3 L2/L3
Access Layer
Ethernet Ethernet (ToR/EoR/OoR)
Switch Switch


Switch Switch

L2/L3 L2/L3
Access Layer
Ethernet Ethernet (ToR/EoR/OoR)
Switch Switch

FCoE
Servers Storage
Row 1


Switch Switch

L2/L3 L2/L3 L2/L3 L2/L3 8
x
N7K-‐F132XP-‐
L2/L3 L2/L3 L2/L3 L2/L3
Access Layer
Ethernet Ethernet Ethernet Ethernet
15
Ethernet
2
x
N7K-‐SUP1 Ethernet Ethernet Ethernet (ToR/EoR/OoR)
Switch Switch Switch Switch Switch Switch Switch Switch

FCoE FCoE FCoE FCoE
Servers Storage Servers Servers Storage Servers Storage
Storage
Row 1 Row 2 Row 3 Row n


Switch Switch

L2/L3 8
x
N7K-‐F132XP-‐
L2/L3 L2/L3 L2/L3
15
Ethernet Ethernet
2
x
N7K-‐SUP1 Ethernet Ethernet
Switch Switch Switch Switch
Access Layer
(ToR/EoR/OoR)

L2/L3 L2/L3 L2/L3 L2/L3

FCoE FCoE FCoE FCoE
Storage


Switch Switch

L2/L3 8
x
N7K-‐F132XP-‐
L2/L3 L2/L3 L2/L3
15
Ethernet Ethernet
2
x
N7K-‐SUP1 Ethernet Ethernet
Access Layer
(ToR/EoR/OoR)

L2/L3 L2/L3 L2/L3 L2/L3

FCoE FCoE FCoE FCoE
Storage


Multi-hop (Storage Admin POV)
Switch Switch

FCF FCF
FCF FCF FCF
Access Layer
(ToR/EoR/OoR)
SAN A SAN B
FCF FCF FCF FCF

FCoE FCoE FCoE FCoE
Storage


Agenda
–  DCBX
–  PFC
–  ETS
•  Zero-hop
•  Multi-hop


Case Study Overview
•  Each of the topologies will
–  Assume a server to storage ratio of 10 server ports to
one storage port
–  Provide the number of server ports available
–  Provide the oversubscription ratio
–  Use Cisco products


Case Study 1
MDS Reference Architecture

Server 1

Server 2

Server 16

RACK 1


Case Study 1




RACK 1 RACK 2 RACK 21

21 Racks
336 Servers


Case Study 1

MDS
Server 2 Server 2 Server 2 9513 A

MDS
9513 B


Each MDS:
21 Racks (7) 48 Port line cards = 336 ports
336 Servers - Servers
- 4:1 oversubscription
(4) 24 port line cards = 96 ports
- Storage
- 1:1 with proper layout


Case Study 1
Optical – MMF
672 OM3 (30m)

MDS
Server 2 Server 2 Server 2 336 9513 A

MDS
336 9513 B


Each MDS:
21 Racks (7) 48 Port line cards = 336 ports
- 4:1 oversubscription
- Storage


Case Study 1
Optical – MMF
672 OM3 (30m)
66 FC storage
ports

MDS
Server 2 Server 2 Server 2 336 9513 A 33

MDS
336 9513 B 33


RACK 1 RACK 2 RACK 21 EMC Storage
21 Racks
Each MDS: Arrays
(7) 48 Port line cards = 336 ports
Server to Storage port ratio = 10:1 - 4:1 oversubscription
- Storage


Case Study 2
Nexus 7000

Optical – MMF
768 OM3 (30m)

Server 1 Server 1 Server 1 Nexus
Server 2 Server 2 Server 2 384 7018 39
A
Nexus
384 7018 39
B


Each Nexus 7018:
24 Racks (14) 32 Port 10G “F1” = 448 ports
384 Servers - 32:23 oversubscription
Server to Storage port ratio = 10:1 (4) spare slots
(25) spare ports for VE_Ports


Case Study 3
Nexus 5596 / 2232

2232 1A 2232 1B

Server 1 Server 1

Server 2 Server 2

Server 16 Server 16

RACK 1A RACK 1B


Case Study 3
Nexus 5596 / 2232

2232 1A 2232 1B

FCoE Twinax

Server 1 Server 1

Server 2 Server 2

Server 16 Server 16

RACK 1A RACK 1B


Case Study 3
Nexus 5596 / 2232

2232 1A 2232 1B 2232 15A 2232 15B

FCoE Twinax

Server 1 Server 1 Server 1 Server 1


30 Racks
Server 16 Server 16 Server 16 Server 16 480 Servers

RACK 1A RACK 1B RACK 15A RACK 15B


Case Study 3
Nexus 5596 / 2232

Nexus 5596 1A
Nexus 5596 1B

8 8
Nexus 5596 2A Nexus 5596 2B

2232 1A 2232 1B 2232 15A 2232 15B

FCoE Twinax



30 Racks
Server 16 Server 16 Server 16 Server 16 480 Servers



Case Study 3
Nexus 5596 / 2232

8 Nexus 5596 1A
Nexus 5596 1B 8
8 8
Nexus 5596 2A 8 8 Nexus 5596 2B

Optical – FET - MMF
240 OM3 (30m)

2232 1A 2232 1B 2232 15A 2232 15B

FCoE Twinax



30 Racks
Server 16 Server 16 Server 16 Server 16 480 Servers – 4:1
oversubscribed


Case Study 3
Nexus 5596 / 2232
96 FCoE storage ports

8 24 24
Nexus 5596 1A
Nexus 5596 1B 8
8 24 24 8
Nexus 5596 2A 8 8 Nexus 5596 2B

240 OM3 (30m)

2232 1A 2232 1B 2232 15A 2232 15B

FCoE Twinax

FCoE Optical

30 Racks
oversubscribed
RACK 1A RACK 1B RACK 15A RACK 15B Server to Storage
port ratio = 10:1


Case Study 3
Nexus 5596 / 2232

8 24 24
Nexus 5596 1A
Nexus 5596 1B 8
4 4
LAN 8 24 24 8 LAN
4 4
Nexus 5596 2A 8 8 Nexus 5596 2B

240 OM3 (30m)

2232 1A 2232 1B 2232 15A 2232 15B

FCoE Twinax

FCoE Optical
10GbE Optical

30 Racks
oversubscribed
port ratio = 10:1


Case study summary
•  It is possible to create a fully converged FCoE
based solution for about half of what it costs to
deploy a similar topology based on MDS and 6509 .
–  Considering port count only.
•  When you evaluate each topology on a per
connected server port basis in terms of dollars per
MB/s and Watts per MB/s, a fully converged
topology costs less and saves power.
•  Check this out for yourselves using the Cisco “End-
to-End Unified Fabric TCO Calculator”
–  www.cisco.com/go/fcoe


Available Now!
EMC Fibre Channel over Ethernet (FCoE)
TechBook
Authored by Erik Smith, Mark Lippitt, Erik Paine,
Mark Anthony De Castro, and Shreedhan Nikam Upd
ated
!
Available for purchase at EMC World

Various new E-Lab TechBooks,
created from the former
EMC Networking Topology Guide,
are available at
elabNavigator.EMC.com
Topology Resource Center Tab


Want More?
Visit my blog!
www.brasstacksblog.typepad.com
Contact me via email!
erik.smith@emc.com

Drop by the EMC Select booth!


Summary and Questions
•  Discussed topologies currently supported and what
is being developed
•  Explained essential components of FCoE, including
–  DCBX
–  PFC
–  FIP
–  DCB Clouds
•  Highlighted known limitations when using FCoE
•  Explained three real world use cases


Related Sessions at EMC World
Session Title
Type
Lecture Converged Data Center: FCoE, iSCSI, and the Future of Storage
Networking
Birds-of-a-
The Future of Storage Networking
Feather
Solutions
EMC Select
Pavilion


Related Technical Documentation
TechBooks
Fibre Channel over Ethernet (FCoE) - Data Center Bridging (DCB) Concepts and
Protocols

Fibre Channel SAN Topologies

Networked Storage Concepts and Protocols


THANK YOU


Backup Slides


FIP ─ FCoE Initialization Protocol (cont.)

FCF
DCB Cloud Priority = 1

CNA: Lossless
Universal-MAC Fabric
Ethernet
ENode-MAC FCF-MAC WWNN = FABRIC-WWNN
VN_Port-MAC switch

FCF
Priority =
128

FIP allows an ENode to
Ÿ  Perform VLAN and FCF discovery
Ÿ  Ensure that Layer 2 network is capable of supporting mini-jumbo frames
Ÿ  Perform fabric login
Ÿ  Use LKA (Link Keep Alive) to maintain the virtual link with the FCF & vice versa


FIP ─ VLAN Request
FIP VLAN Request:
DA = ALL-FCF-MACs
SA = ENode MAC
802.1Q Tag = (Untagged)
MAC Address descriptor = ENode-MAC
FCF
DCB Cloud e st
qu Priority = 1
Re

CNA: Request Lossless
Ethernet
VN_Port-MAC switch

Re FCF
qu
e st Priority =
128

Ÿ  Multicast
Ÿ  Allows a CNA to discover which VLANs FCoE services are being provided
Ÿ  All FIP requests and responses use a pre-defined set of TLV (Type, Length,
Value) data structures


FIP ─ VLAN Notification
FIP VLAN Notification:
DA = ENode MAC
SA = FCF-MAC
802.1Q Tag = (untagged)
MAC Address descriptor = FCF-MAC
FCoE VID = 100
n FCF
DCB Cloud a tio
fic Priority = 1
N oti

Notification
CNA: Lossless
Ethernet
VN_Port-MAC switch
Notification

No FCF
tific
ati Priority =
on
128
FIP VLAN Notification:
DA = ENode MAC
SA = FCF-MAC
802.1Q Tag = VLAN 1
Ÿ  Unicast FCoE VID = 100

Ÿ  Both FCFs respond
Ÿ  Note the 802.1Q tag and the FCoE VID TLV


FIP ─ Solicitation
Solicitation:
DA = ALL-FCF-MACs
SA = ENode MAC
802.1Q Tag= VLAN 100
MAC Address descriptor = FIP-MAC
Name Identifier = WWNN
Max FCoE size = 2240
FCF
DCB Cloud ion
itat Priority = 1
S olic

CNA: Solicitation Lossless
Ethernet
VN_Port-MAC switch

So FCF
l i ci
tat Priority =
i on
128

Ÿ  Multicast
Ÿ  Allows the CNA to discover which FCFs are available for login
Ÿ  Note the 802.1Q Tag and the Max FCoE size field


FIP ─ Advertisement
Advertisement:
Ÿ  Unicast DA = FIP-MAC
SA = FCF-MAC
Ÿ  Note the Priority, Name ID, and Max FCoE size Priority = 1
Name Identifier = SWITCH-WWNN
Ÿ  Max FCoE size is a field padded to the proper size Max FCoE size = 2158

Ÿ  Dynamic ACL updated
t FCF
en
–  FIP snooping DCB Cloud i se
m
Priority = 1
v ert
Ad
Advertisement
CNA: Lossless
Ethernet
VN_Port-MAC switch
Advertisement
Ad
ve r FCF
Internal FCF list: tise
me Priority =
nt
Entry 1: 128
Priority = 1
DA = FCF-MAC
Max FCoE size verified = 1 Advertisement:
DA = FIP-MAC
Entry 2: SA = FCF-MAC
Priority = 128 802.1Q Tag= VLAN 100
Name Identifier = SWITCH-WWNN Priority = 128
DA = FCF-MAC MAC Address descriptor = FCF-MAC
Max FCoE size verified = 1 Name Identifier = SWITCH-WWNN
Max FCoE size = 2158


FIP ─ FLOGI
FIP FLOGI:
DA = FCF-MAC (priority 1)
SA = ENode MAC
FIP FLOGI descriptor = FLOGI frame
FCF
GI
FLO Priority = 1

CNA: FLOGI Lossless
Ethernet
VN_Port-MAC switch

FCF
Priority =
128
Internal FCF list:

Entry 1:
Priority = 1
DA = FCF-MAC
Max FCoE size verified = 1

Entry 2:
Priority = 128 Ÿ  Unicast
DA = FCF-MAC
Max FCoE size verified = 1 Ÿ  Since both are connected to the same Fabric (as determined
by the Name ID field), the FCF with the lower priority is sent
the FLOGI


FIP ─ FLOGI Accept
FIP FLOGI ACC:
DA = ENode MAC
SA = FCF-MAC (priority 1)
FIP FLOGI descriptor = FLOGI frame
MAC Descriptor = VN_Port MAC
CC FCF
DCB Cloud A
GI Priority = 1
FLO

FLOGI ACC
CNA: Lossless
Ethernet
VN_Port-MAC switch

FCF
Priority =
128
Ÿ  Unicast
Ÿ  Dynamic ACLs updated
Ÿ  CNA will use the Fabric Provided MAC Address (FPMA) and FCoE Ethertype
frames to log in with the Name Server and perform discovery


FCoE - Topologies, Protocol, and Limitations

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (17)

More from EMC

More from EMC (20)

Recently uploaded

Recently uploaded (20)

FCoE - Topologies, Protocol, and Limitations