• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
FCoE ─Topologies, Protocols, and Limitations ( 2010 EMC World )
 

FCoE ─Topologies, Protocols, and Limitations ( 2010 EMC World )

on

  • 644 views

FCoE ─Topologies, Protocols, and Limitations

FCoE ─Topologies, Protocols, and Limitations

Statistics

Views

Total Views
644
Views on SlideShare
644
Embed Views
0

Actions

Likes
0
Downloads
39
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    FCoE ─Topologies, Protocols, and Limitations ( 2010 EMC World ) FCoE ─Topologies, Protocols, and Limitations ( 2010 EMC World ) Presentation Transcript

    • FCoE ─ Topologies, Protocols, and Limitations Erik Smith© Copyright 2010 EMC Corporation. All rights reserved.
    • Roadmap Information Disclaimer EMC makes no representation and undertakes no obligations with regard to product planning information, anticipated product characteristics, performance specifications, or anticipated release dates (collectively, “Roadmap Information”). Roadmap Information is provided by EMC as an accommodation to the recipient solely for purposes of discussion and without intending to be bound thereby. Roadmap information is EMC Restricted Confidential and is provided under the terms, conditions and restrictions defined in the EMC Non- Disclosure Agreement in place with your organization.© Copyright 2009 EMC Corporation. All rights reserved. 2
    • Objectives Understand topologies that are currently supported and what is currently being developed Understand the essential components of FCoE including DCBX, PFC, FIP, and DCB (CEE) Clouds Understand the known limitations when using FCoE© Copyright 2009 EMC Corporation. All rights reserved. 3
    • Agenda Fibre Channel over Ethernet (FCoE) overview – I/O consolidation – FCoE basics FCoE ecosystem components – Encapsulation – Data Center Bridging Capability eXchange Protocol (DCBX) – FCoE Initialization Protocol (FIP) – Priority Flow Control (PFC) – Enhanced Transmission Selection (ETS) – TRILL (TRansparent Interconnection of Lots of Links) FCoE Topology timeline – Top of Rack (ToR) – End of Row (EoR) When and where FCoE makes sense© Copyright 2009 EMC Corporation. All rights reserved. 4
    • FCoE ─ An I/O Consolidation Technology Without FCoE With FCoE ESX Server ESX Server VMnics VMHBAs VMnics VMHBAs Hypervisor Hypervisor Physical Physical Physical NICs HBAs CNAs Lossless FCoE FCoE Ethernet Switch Switch LAN SAN A SAN B LAN SAN A SAN B Storage Storage© Copyright 2009 EMC Corporation. All rights reserved. 5
    • FCoE ─ An I/O Consolidation Technology Without FCoE With FCoE ESX Server ESX Server VMnics VMHBAs VMnics VMHBAs Hypervisor Hypervisor Physical Physical Physical NICs HBAs CNAs Lossless FCoE FCoE Ethernet Switch Switch LAN SAN A SAN B LAN SAN A SAN B Storage Storage© Copyright 2009 EMC Corporation. All rights reserved. 6
    • Topology Overview ─ Three Layer Hierarchical Model ─ Core MAN routed links Core Layer Si Si Layer 3 = Layer 3 function = Layer 2 function© Copyright 2009 EMC Corporation. All rights reserved. 7
    • Topology Overview ─ Three Layer Hierarchical Model ─ Distribution MAN routed links Core Layer Si Si Layer 3 routed links Distribution Layer Layer 3 Si Si = Layer 3 function = Layer 2 function© Copyright 2009 EMC Corporation. All rights reserved. 8
    • Topology Overview ─ Three Layer Hierarchical Model ─ Access MAN routed links Core Layer Si Si Layer 3 routed links Distribution Layer Layer 3 Si Si Layer 2/3 Access Layer Si No ST Si Si ST Si NIC NIC Si Si Server / TCP Storage NIC NIC = Layer 3 function Server / = Layer 2 function TCP Storage© Copyright 2009 EMC Corporation. All rights reserved. 9
    • Topology Overview ─ Adding FCoE MAN routed links Core Layer Si Si Layer 3 routed links Distribution Layer Layer 3 Si Si Layer 2/3 Fabric A Access Layer Fabric B No ST ST NIC NIC FIP FIP Server / TCP Storage NIC NIC = FCF Server / TCP StorageFIP = FIP Snooping Bridge © Copyright 2009 EMC Corporation. All rights reserved. 10
    • Topology Overview ─ Adding FCoE (cont.) MAN routed links Core Layer Si Si Layer 3 routed links Distribution Layer Layer 3 Si Si Layer 2/3 Fabric A Access Layer Fabric B No ST ST NIC NIC FIP FIP Server / TCP Storage NIC NIC = FCF Server /FIP = FIP Snooping Bridge TCP Storage © Copyright 2009 EMC Corporation. All rights reserved. 11
    • Topology Overview ─ Adding FCoE (cont.) MAN routed links Core Layer Si Si Layer 3 routed links Distribution Layer Layer 3 Si Si Layer 2/3 Fabric A Access Layer Fabric B No ST ST NIC NIC FIP FIP Server / TCP Storage NIC NIC = FCF Server /FIP = FIP Snooping Bridge TCP Storage © Copyright 2009 EMC Corporation. All rights reserved. 12
    • Topology Overview ─ Adding FCoE (cont.) MAN routed links Core Layer Si Si Layer 3 routed links Distribution Layer Layer 3 Si Si Layer 2/3 Fabric A Access Layer Fabric B No ST ST CNA CNA FIP FIP Server / TCP Storage CNA CNA = FCF Server /FIP = FIP Snooping Bridge TCP Storage © Copyright 2009 EMC Corporation. All rights reserved. 13
    • Topology Overview ─ Adding FCoE (cont.) MAN routed links Core Layer Si Si Layer 3 routed links Distribution Layer Layer 3 Si Si Layer 2/3 Fabric A Access Layer Fabric B No ST ST or TRILL FCoE CNA CNA FIP FIP Server / TCP Storage FCoE CNA CNA = FCF Server /FIP = FIP Snooping Bridge TCP Storage © Copyright 2009 EMC Corporation. All rights reserved. 14
    • Topology Overview ─ Crossing FCoE Domains Using iSCSI MAN routed links Core Layer Si Si Layer 3 routed links Distribution Layer Layer 3 Si Si Layer 2/3 Fabric A iSCSI Access Layer Fabric B No ST ST or TRILL FCoE CNA CNA FIP FIP Server FCoE CNA CNA = FCFFIP = FIP Snooping Bridge Storage © Copyright 2009 EMC Corporation. All rights reserved. 15
    • Topology Overview ─ Crossing FCoE Domains Using Fibre Channel Fabric A Fabric B Fibre Channel No ST ST or TRILL FCoE CNA CNA FIP FIP Server FCoE CNA CNA Storage = FCFFIP = FIP Snooping Bridge © Copyright 2009 EMC Corporation. All rights reserved. 16
    • Terminology  CNA – Converged Network Adapter Fabric A Fabric B No ST ST or TRILL FCoE CNA CNA FIP FIP Server FCoE = FCF CNA CNAFIP = FIP Snooping Bridge Storage © Copyright 2009 EMC Corporation. All rights reserved. 17
    • Terminology  CNA – Converged Network Adapter  FIP snooping bridge Fabric A Fabric B No ST ST or TRILL FCoE CNA CNA FIP FIP Server FCoE = FCF CNA CNAFIP = FIP Snooping Bridge Storage © Copyright 2009 EMC Corporation. All rights reserved. 18
    • Terminology  CNA – Converged Network Adapter  FIP snooping bridge Fabric A Fabric B  FCF – Fibre Channel Forwarder No ST ST or TRILL FCoE CNA CNA FIP FIP Server FCoE = FCF CNA CNAFIP = FIP Snooping Bridge Storage © Copyright 2009 EMC Corporation. All rights reserved. 19
    • Fibre Channel vs. FCoE Fibre Channel FC – Switched Fabric© Copyright 2009 EMC Corporation. All rights reserved. 20
    • Fibre Channel vs. FCoE Fibre Channel FC – Switched Fabric FC – Arbitrated Loop© Copyright 2009 EMC Corporation. All rights reserved. 21
    • Fibre Channel vs. FCoE Fibre Channel FC – Switched Fabric FC – Arbitrated Loop FC – Point-to-Point© Copyright 2009 EMC Corporation. All rights reserved. 22
    • Fibre Channel vs. FCoE Fibre Channel FCoE FC – Switched Fabric DCB (CEE) Cloud FC – Arbitrated Loop FC – Point-to-Point DCB (CEE) Cloud – Zero hop – Multi hop© Copyright 2009 EMC Corporation. All rights reserved. 23
    • Fibre Channel vs. FCoE Fibre Channel FCoE FC – Switched Fabric DCB (CEE) Cloud FC – Arbitrated Loop VN2VN (proposed) FC – Point-to-Point Point-to-Point (proposed) DCB (CEE) Cloud – Zero hop – Multi hop VN2VN and Point-to-Point – FC-BB-6 (Next generation FCoE ~2011) – Will probably use the same discovery protocol – If Point-to-Point fails, VN2VN takes over© Copyright 2009 EMC Corporation. All rights reserved. 24
    • Agenda Fibre Channel over Ethernet (FCoE) overview – I/O consolidation – FCoE basics FCoE ecosystem components – Encapsulation – Data Center Bridging Capability eXchange Protocol (DCBX) – FCoE Initialization Protocol (FIP) – Priority Flow Control (PFC) – Enhanced Transmission Selection (ETS) – TRILL (TRansparent Interconnection of Lots of Links) FCoE Topology timeline – Top of Rack (ToR) – End of Row (EoR) When and where FCoE makes sense© Copyright 2009 EMC Corporation. All rights reserved. 25
    • Encapsulation Normal Ethernet frame; Ethertype = FCoE Same as physical FC frame Control information: version, ordered sets (SOF, EOF) Maximum size is 2240 bytes – Mini-jumbo frame support is required – Not an issue for 10 GbE FCoE aware switches E-Lab tested© Copyright 2009 EMC Corporation. All rights reserved. 26
    • Encapsulation (cont.) Reserved fields ensure minimum Ethernet frame length of 64 bytes 802.1Q Tag carries COS information (used for PFC) FC frame contains a CRC – CRC is stomped if FCS is bad© Copyright 2009 EMC Corporation. All rights reserved. 27
    • DCBX ─ Data Center Bridging Capability eXchange Protocol Lossless ENode Ethernet switch LINK UP Each DCBX DCBX DCBX frame contains DCBX DCBX priority map TLVs for both FIP and FCoE FIP VLAN D is covery An extension of the Link Layer Discovery Protocol (LLDP) Allows for the exchange of priority map values for both FCoE and the FCoE Initialization Protocol (FIP) Enables lossless behavior FCoE virtual links should not be instantiated until DCBX has been performed© Copyright 2009 EMC Corporation. All rights reserved. 28
    • FIP ─ FCoE Initialization Protocol FC Protocol Relies on the presence of a physical link between two devices End devices implicitly know who to log in with One-to-one relationship between a N_Port and a the Fabric being logged into – No ambiguity Implicit logout from the fabric should physical link be lost Implicit security – Man-in-the-middle is very difficult© Copyright 2009 EMC Corporation. All rights reserved. 29
    • FIP ─ FCoE Initialization Protocol FC Protocol FCoE Topology Relies on the presence of a physical Provides some challenges: link between two devices  End devices do not know the MAC End devices implicitly know who to Address of the FCF that will allow log in with login, or even if one exists One-to-one relationship between a  No one-to-one relationship N_Port and a the Fabric being logged – Could be a good thing into  Not necessarily a single physical link – No ambiguity – No implicit logout Implicit logout from the fabric should physical link be lost  No implicit security – Man-in-the-middle is theoretically Implicit security possible – Man-in-the-middle is very difficult© Copyright 2009 EMC Corporation. All rights reserved. 30
    • FIP ─ FCoE Initialization Protocol (cont.)FIP bridges the gap between expectations of FC & the reality of FCoE ENode can discover who to log in with One-to-many relationship is built in LKA (Link Keep Alive) and FIP CVL (Clear Virtual Links) allows for logout from the fabric should the logical link be lost Implicit security (man-in-the-middle is difficult) – Provided that FIP snooping and Dynamic ACLs are implementedBe aware… An FCoE environment using FIP is much more complex than a native FC environment – Opportunity for an interoperability issue is slightly increased New users to FCoE need to understand the differences between FC and FCoE FIP and be prepared to deal with configuration and compatibility issues –E-Lab has published a configuration guide, including a troubleshooting section. Tests and configurations that are known to be compatible are also published.© Copyright 2009 EMC Corporation. All rights reserved. 31
    • FIP ─ FCoE Initialization Protocol (cont.) FCF DCB Cloud Priority = 1 CNA: Lossless Universal-MAC Fabric Ethernet ENode-MAC FCF-MAC WWNN = FABRIC-WWNN VN_Port-MAC switch FCF Priority = 128FIP allows an ENode to Perform VLAN and FCF discovery Ensure that Layer 2 network is capable of supporting mini-jumbo frames Perform fabric login Use LKA (Link Keep Alive) to maintain the virtual link with the FCF & vice versa© Copyright 2009 EMC Corporation. All rights reserved. 32
    • FIP ─ VLAN Request FIP VLAN Request: DA = ALL-FCF-MACs SA = ENode MAC 802.1Q Tag = (Untagged) MAC Address descriptor = ENode-MAC t FCF DCB Cloud s q ue Priority = 1 Re CNA: Request Lossless Universal-MAC Fabric Ethernet ENode-MAC FCF-MAC WWNN = FABRIC-WWNN VN_Port-MAC switch Re FCF qu e st Priority = 128 Multicast Allows a CNA to discover which VLANs FCoE services are being provided All FIP requests and responses use a pre-defined set of TLV (Type, Length, Value) data structures© Copyright 2009 EMC Corporation. All rights reserved. 33
    • FIP ─ VLAN Notification FIP VLAN Notification: DA = ENode MAC SA = FCF-MAC 802.1Q Tag = (untagged) MAC Address descriptor = FCF-MAC FCoE VID = 100 n FCF tio DCB Cloud ca tifi Priority = 1 No Notification CNA: Lossless Universal-MAC Fabric Ethernet ENode-MAC FCF-MAC WWNN = FABRIC-WWNN VN_Port-MAC switch Notification No FCF tific ati Priority = on 128 FIP VLAN Notification: DA = ENode MAC SA = FCF-MAC 802.1Q Tag = VLAN 1 MAC Address descriptor = FCF-MAC Unicast FCoE VID = 100 Both FCFs respond Note the 802.1Q tag and the FCoE VID TLV© Copyright 2009 EMC Corporation. All rights reserved. 34
    • FIP ─ Solicitation Solicitation: DA = ALL-FCF-MACs SA = ENode MAC 802.1Q Tag= VLAN 100 MAC Address descriptor = FIP-MAC Name Identifier = WWNN Max FCoE size = 2240 FCF DCB Cloud ion lic itat Priority = 1 So CNA: Solicitation Lossless Universal-MAC Fabric Ethernet ENode-MAC FCF-MAC WWNN = FABRIC-WWNN VN_Port-MAC switch So FCF l i ci tat Priority = i on 128 Multicast Allows the CNA to discover which FCFs are available for login Note the 802.1Q Tag and the Max FCoE size field© Copyright 2009 EMC Corporation. All rights reserved. 35
    • FIP ─ Advertisement Unicast Advertisement: DA = FIP-MAC Note the Priority, Name ID, and Max FCoE size SA = FCF-MAC 802.1Q Tag= VLAN 100 Priority = 1 Max FCoE size is a field padded to the proper size MAC Address descriptor = FCF-MAC Name Identifier = SWITCH-WWNN Max FCoE size = 2158 Dynamic ACL updated – FIP snooping nt FCF DCB Cloud me i se Priority = 1 v ert Ad Advertisement CNA: Lossless Universal-MAC Fabric Ethernet ENode-MAC FCF-MAC WWNN = FABRIC-WWNN VN_Port-MAC switch Advertisement Ad ve r FCF Internal FCF list: tise me Priority = nt Entry 1: 128 Priority = 1 Name Identifier = SWITCH-WWNN DA = FCF-MAC Max FCoE size verified = 1 Advertisement: DA = FIP-MAC Entry 2: SA = FCF-MAC Priority = 128 802.1Q Tag= VLAN 100 Name Identifier = SWITCH-WWNN Priority = 128 DA = FCF-MAC MAC Address descriptor = FCF-MAC Max FCoE size verified = 1 Name Identifier = SWITCH-WWNN Max FCoE size = 2158© Copyright 2009 EMC Corporation. All rights reserved. 36
    • FIP ─ FLOGI FIP FLOGI: DA = FCF-MAC (priority 1) SA = ENode MAC 802.1Q Tag= VLAN 100 FIP FLOGI descriptor = FLOGI frame FCF GI FLO Priority = 1 CNA: FLOGI Lossless Universal-MAC Fabric Ethernet ENode-MAC FCF-MAC WWNN = FABRIC-WWNN VN_Port-MAC switch FCF Priority = 128Internal FCF list:Entry 1:Priority = 1Name Identifier = SWITCH-WWNNDA = FCF-MAC  UnicastMax FCoE size verified = 1Entry 2:  Since both are connected to the same Fabric (as determinedPriority = 128Name Identifier = SWITCH-WWNN by the Name ID field), the FCF with the lower priority is sentDA = FCF-MACMax FCoE size verified = 1 the FLOGI© Copyright 2009 EMC Corporation. All rights reserved. 37
    • FIP ─ FLOGI Accept FIP FLOGI ACC: DA = ENode MAC SA = FCF-MAC (priority 1) 802.1Q Tag= VLAN 100 FIP FLOGI descriptor = FLOGI frame MAC Descriptor = VN_Port MAC CC FCF DCB Cloud A GI Priority = 1 FLO FLOGI ACC CNA: Lossless Universal-MAC Fabric Ethernet ENode-MAC FCF-MAC WWNN = FABRIC-WWNN VN_Port-MAC switch FCF Priority = 128 Unicast Dynamic ACLs updated CNA will use the Fabric Provided MAC Address (FPMA) and FCoE Ethertype frames to log in with the Name Server and perform discovery© Copyright 2009 EMC Corporation. All rights reserved. 38
    • PFC ─ Priority Flow Control Transmit Queues Receive Buffers Ethernet Link Eight Virtual Lanes Necessary since FC requires a lossless environment to operate properly Without PFC, normal periodic SAN congestion will cause frames to drop and the entire exchange will need to be retransmitted – Can take up to 60 seconds, depending on which frame in the exchange is lost© Copyright 2009 EMC Corporation. All rights reserved. 39
    • PFC ─ Priority Flow Control Buffer Full High Lossless ENode Ethernet Frame Frame Frame switch Frame Low Frame Buffer Frame Frame Frame Buffer Empty© Copyright 2009 EMC Corporation. All rights reserved. 40
    • PFC ─ Priority Flow Control (cont.) Pause class 3 for x quanta (512 bit times) Buffer Full Max multimode distances supported with Gen 2 CNAs High Frame Pause Lossless Frame ENode Ethernet Frame Frame Frame switch Frame Low Frame Buffer Frame Frame Frame Buffer Empty© Copyright 2009 EMC Corporation. All rights reserved. 41
    • PFC ─ Priority Flow Control (cont.) Pause class 3 for zero quanta Unpause Buffer Full High Pause Lossless ENode Ethernet Frame Frame Frame switch Low Frame Buffer Frame Frame Frame Buffer Empty© Copyright 2009 EMC Corporation. All rights reserved. 42
    • ETS – Enhanced Transmission Selection Desired Traffic ETS configuration 10GbE Link Actual Throughput 2 GbpsiSCSI 3 Gbps 3 Gbps 3 Gbps 2 Gb 3 Gbps 3 GbpsPriority (1) 3 Gbps 3 Gbps 3 GbpsFCoE 3 Gbps 3 Gbps 3 Gbps 3 GbPriority (3) 3 Gbps 5 Gbps 4 Gbps 6 Gbps 5 Gb 4 Gbps 9 Gbps 10 Gbps 10 Gbps LAN 3 GbpsPriority (5) t1 t2 t3 t1 t2 t3  ETS information is exchanged in DCBX  Ensures storage traffic has a guaranteed minimum amount of bandwidth  Utilized by all FCoE devices that are currently supported by EMC © Copyright 2009 EMC Corporation. All rights reserved. 43
    • TRILL – Transparent Interconnection of Lots of LinksTo avoid forwarding loops, the Spanning Tree Protocol (STP) Elects a Root switch Disables redundant paths in order to create a tree topology Creates a few problems, such as – Inefficient use of bandwidth Root Si Si X X X X X Si Si Si Si Si© Copyright 2009 EMC Corporation. All rights reserved. 44
    • TRILL – Transparent Interconnection of Lots of Links Provides a solution by allowing redundant links to remain active Key concepts – RBridges (RB)  Backward compatible, terminate STP – Designated RBridge (DRB) – IS-IS routing – Self-configuring – TRILL header and encapsulation DRB RB – ESADI  End Station Address Distribution Information Si Si  Think RSCN, but better! Si Si Si Si Si© Copyright 2009 EMC Corporation. All rights reserved. RB RB RB RB RB 45
    • Agenda Fibre Channel over Ethernet (FCoE) overview – I/O consolidation – FCoE basics FCoE ecosystem components – Encapsulation – Data Center Bridging Capability eXchange Protocol (DCBX) – FCoE Initialization Protocol (FIP) – Priority Flow Control (PFC) – Enhanced Transmission Selection (ETS) – TRILL (TRansparent Interconnection of Lots of Links) FCoE Topology timeline – Top of Rack (ToR) – End of Row (EoR) When and where FCoE makes sense© Copyright 2009 EMC Corporation. All rights reserved. 46
    • FCoE Switch Topology Timeline Legend: Available today Coming soon 1350+ Requires FC-BB-6 FCoE 450physical ports 150 Cisco Nexus Brocade 50x0 8000 50 0 8 9 9 9 0 0 0 0 + -0 -0 -0 -0 -1 -1 -1 -1 11 4 2 3 4 1 2 3 4 – Q Q Q Q Q Q Q Q 1 Q EMC GA date© Copyright 2009 EMC Corporation. All rights reserved. 47
    • FCoE Switch Topology Timeline Legend: Available today Coming soon 1350+ Requires FC-BB-6 450 Support for zero FCoE hop DCB (CEE)physical topology ports 150 Cisco Nexus Brocade 50x0 8000 50 0 9 9 9 0 08 0 0 0 + -0 -0 -0 -1 -1 -1 -1 11 – 2 3 4 1 2 3 4 – 4 Q Q Q Q Q Q Q Q 1 Q EMC GA date© Copyright 2009 EMC Corporation. All rights reserved. 48
    • Zero Hop DCB (CEE) ─ Available Today FCoE CNA CNA Server / Storage Benefits Be aware  CNA is directly connected to a  Native FCoE to Native FCoE switch that contains an FCF without the use of the fabric is not – Prevents man-in-the-middle attacks by supported pinning the VN_Port-MAC to the Ethernet interface where FLOGI was  Scalability via FC only performed  Two physical topologies supported – Top of Rack – End of Row© Copyright 2009 EMC Corporation. All rights reserved. 49
    • Four Interface Types FC – NPV / AG and FC-SW – Direct attach storage Ethernet uplink FCoE interface – Brocade – Emulex – QLogic – Cisco – Open FCoE (under development) 5000 - 8000 Ethernet end device interface© Copyright 2009 EMC Corporation. All rights reserved. 50
    • Structured Cabling ─ Based on TIA-942 Carrier equipment Carriers Comm. Room Carriers and demarcation Data Center© Copyright 2009 EMC Corporation. All rights reserved. 51
    • Structured Cabling ─ Based on TIA-942 (cont.) Carrier equipment Carriers Comm. Room Carriers and demarcation Backbone Core: cabling Main Distribution L3 LAN and SAN Area Data Switches Center© Copyright 2009 EMC Corporation. All rights reserved. 52
    • Structured Cabling ─ Based on TIA-942 (cont.) Carrier equipment Carriers Comm. Room Carriers and demarcation Backbone Core: cabling Main Distribution L3 LAN and SAN Area Data Switches Center Distribution: Horizontal Horizontal HorizontalRow Distribution Area Distribution Area Distribution Area L2/L3 LAN and SAN Switches © Copyright 2009 EMC Corporation. All rights reserved. 53
    • Structured Cabling ─ Based on TIA-942 (cont.) Carrier equipment Carriers Comm. Room Carriers and demarcation Backbone Core: cabling Main Distribution L3 LAN and SAN Area Data Switches Center Distribution: Horizontal Horizontal HorizontalRow Distribution Area Distribution Area Distribution Area L2/L3 LAN and SAN Switches Horiz. cabling Equipment Equipment EquipmentRack Distribution Area Distribution Area Distribution Area Access: Servers / L2 LAN Equipment Equipment EquipmentRack Distribution Area Distribution Area Distribution Area© Copyright 2009 EMC Corporation. All rights reserved. 54
    • Zero Hop DCB Cloud ─ Top of Rack (TOR) EDA-1 EDA-2 EDA-3 EDA-4 EDA-5 EDA-6 EDA-7 EDA-8 HDA© Copyright 2009 EMC Corporation. All rights reserved. 55
    • Zero Hop DCB Cloud ─ Top of Rack (cont.) Patch to MDA Ethernet Switch EDA-1 EDA-2 EDA-3 EDA-4 EDA-5 EDA-6 EDA-7 EDA-8 HDA© Copyright 2009 EMC Corporation. All rights reserved. 56
    • Zero Hop DCB Cloud ─ Top of Rack (cont.) Patch to MDA Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Ethernet Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Switch Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 EDA-1 EDA-2 EDA-3 EDA-4 EDA-5 EDA-6 EDA-7 EDA-8 HDA© Copyright 2009 EMC Corporation. All rights reserved. 57
    • Zero Hop DCB Cloud ─ Top of Rack (cont.) FCF 1A FCF 2A FCF 3A FCF 4A FCF 5A FCF 6A FCF 7A FCF 8A Patch to FCF 1B FCF 2B FCF 3B FCF 4B FCF 5B FCF 6B FCF 7B FCF 8B MDA Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Ethernet Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Switch Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 EDA-1 EDA-2 EDA-3 EDA-4 EDA-5 EDA-6 EDA-7 EDA-8 HDA© Copyright 2009 EMC Corporation. All rights reserved. 58
    • Zero Hop DCB Cloud ─ Top of Rack (cont.) FCF 1A FCF 2A FCF 3A FCF 4A FCF 5A FCF 6A FCF 7A FCF 8A Patch to FCF 1B FCF 2B FCF 3B FCF 4B FCF 5B FCF 6B FCF 7B FCF 8B MDA Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Ethernet Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Switch Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 EDA-1 EDA-2 EDA-3 EDA-4 EDA-5 EDA-6 EDA-7 EDA-8 HDA© Copyright 2009 EMC Corporation. All rights reserved. 59
    • Zero Hop DCB Cloud ─ Top of Rack (cont.) FC FC FCF 1A FCF 2A FCF 3A FCF 4A FCF 5A FCF 6A FCF 7A FCF 8A Patch to FCF 1B FCF 2B FCF 3B FCF 4B FCF 5B FCF 6B FCF 7B FCF 8B MDA Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Ethernet Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Switch Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 EDA-1 EDA-2 EDA-3 EDA-4 EDA-5 EDA-6 EDA-7 EDA-8 HDA© Copyright 2009 EMC Corporation. All rights reserved. 60
    • Zero Hop DCB Cloud ─ Top of Rack (cont.) 10GbE 10GbE FC FC FCF 1A FCF 2A FCF 3A FCF 4A FCF 5A FCF 6A FCF 7A FCF 8A Patch to FCF 1B FCF 2B FCF 3B FCF 4B FCF 5B FCF 6B FCF 7B FCF 8B MDA Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Ethernet Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Switch Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 EDA-1 EDA-2 EDA-3 EDA-4 EDA-5 EDA-6 EDA-7 EDA-8 HDA© Copyright 2009 EMC Corporation. All rights reserved. 61
    • Zero Hop DCB Cloud ─ End of Row (EOR) 10GbE - FCoE 10GbE - FCoE Patch to MDA Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 FCF A Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 FCF B Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 EDA-1 EDA-2 EDA-3 EDA-4 EDA-5 EDA-6 EDA-7 EDA-8 HDA© Copyright 2009 EMC Corporation. All rights reserved. 62
    • FCoE Switch Topology Timeline Legend: Available today Coming soon 1350+ Requires FC-BB-6 Cisco Nexus 4000 / FCoE 450 50x0physical ports 150 Cisco Nexus Brocade 50x0 8000 50 0 8 9 9 9 0 0 0 0 + -0 -0 -0 -0 -1 -1 -1 -1 11 4 2 3 4 1 2 3 4 – Q Q Q Q Q Q Q Q 1 Q EMC GA date© Copyright 2009 EMC Corporation. All rights reserved. 63
    • FCoE Switch Topology Timeline Legend: Available today Support for multi Coming soon hop DCB (CEE) 1350+ topology Requires FC-BB-6 Cisco Nexus 4000 / FCoE 450 50x0physical ports 150 Cisco Nexus Brocade 50x0 8000 50 0 8 9 9 9 0 0 0 0 + -0 -0 -0 -0 -1 -1 -1 -1 11 4 2 3 4 1 2 3 4 – Q Q Q Q Q Q Q Q 1 Q EMC GA date© Copyright 2009 EMC Corporation. All rights reserved. 64
    • Multi-hop DCB CloudFCFs ST or TRILL FIP Snooping FIP FIP bridges FCoE CNA CNA Server / Storage FIP Snooping bridges are FCoE-aware – FCoE stack – Dynamic ACLs (Access Control Lists)© Copyright 2009 EMC Corporation. All rights reserved. 65
    • Multi-hop DCB Cloud ─ Nexus 4001i Example LAN SAN Nexus- Nexus- 50x0-1 50x0-2 CEE Nexus Nexus 4001i 4001i Cloud Server Server IBM Blade Server 11 12 Chassis© Copyright 2009 EMC Corporation. All rights reserved. 66
    • FCoE Switch Topology Timeline Legend: Available today Coming soon 1350+ Cisco Nexus Requires FC-BB-6 Cisco 50x0 / 2232 Nexus 4000 / FCoE 450 50x0physical ports 150 Cisco Nexus Brocade 50x0 8000 50 0 8 9 9 9 0 0 0 0 + -0 -0 -0 -0 -1 -1 -1 -1 11 4 2 3 4 1 2 3 4 – Q Q Q Q Q Q Q Q 1 Q EMC GA date© Copyright 2009 EMC Corporation. All rights reserved. 67
    • Cisco Nexus 5000 / 2232 5000 – Two in the core (redundancy) – VPCs supported 2232 – 10G FEX (Fabric EXtender) – 32 10 G downlinks 5000 5000 F F – 8 uplinks – No local switching Si Si Scale – At 4:1 oversubscription – 5000s (up to 384 ports) Si Si Si Si 2232 2232 2232 2232© Copyright 2009 EMC Corporation. All rights reserved. 68
    • Cisco Nexus 5000 / 2232 – Physical Layout 2232 1A 2232 2A 2232 3A 2232 4A 2232 5A 2232 6A 2232 7A 2232 8A 5000 2232 1B 2232 2B 2232 3B 2232 4B 2232 5B 2232 6B 2232 7B 2232 8B 5000 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-1 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-2 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-3 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Server-4 Patch to MDA Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-5 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-6 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-7 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-8 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 Server-9 EDA-1 EDA-2 EDA-3 EDA-4 EDA-5 EDA-6 EDA-7 EDA-8 HDA© Copyright 2009 EMC Corporation. All rights reserved. 69
    • FCoE Switch Topology Timeline Legend: Available today Coming soon 1350+ Cisco Nexus Requires FC-BB-6 Cisco 50x0 / 2232 Nexus 4000 / FCoE 450 50x0 Brocadephysical DCX blade ports 150 Cisco Nexus Brocade 50x0 8000 50 0 8 9 9 9 0 0 0 0 + -0 -0 -0 -0 -1 -1 -1 -1 11 4 2 3 4 1 2 3 4 – Q Q Q Q Q Q Q Q 1 Q EMC GA date© Copyright 2009 EMC Corporation. All rights reserved. 70
    • Brocade DCX Blade (1) DCX – (4) FCoE-10-24 blades per chassis DCX-A DCX-C – Chassis connected by standard ISLs Scale F F F F F F F F – At ~4:1 oversubscription on the C o C o C o C o F C F C C C C C o o o o chassis E E E E E E E E – 384 minium (to start) – Same as FC (very soon) F F F F F F F F C C C C F F C C C C o o o o C C o o o o E E E E E E E E DCX-B DCX-D© Copyright 2009 EMC Corporation. All rights reserved. 71
    • Brocade DCX Blade (2) DCX – (4) FCoE-10-24 blades per chassis – Chassis connected by VE_Ports over DCX-A DCX-C FCoE Scale F C F C F C F C F C F C F C F C o o o o – At ~4:1 oversubscription on the E E E E o E o E o E o E chassis – Same as FC (planned) – 384 minimum F F F F F F F F C C C C C C C C o o o o o o o o E E E E E E E E DCX-B DCX-D© Copyright 2009 EMC Corporation. All rights reserved. 72
    • FCF LocationFCF in the core or on the edge? DCX-A DCX-C Cisco – Moving into the core – FCoE is just another protocol F F F F F F F F – Advantage: No FC-SW interop on the edge C o C o C o C o C C C C o o o o E E E E Brocade E E E E – Keeping it on the edge – Native FCoE scale via VE_Ports – Advantage: Less concern about security risks 50x0 50x0 F F Si Si F F F F F F F F C C C C C C C C o o o o o o o o E E E E E E E E Si Si Si Si DCX-B DCX-D 2232 2232 2232 2232© Copyright 2009 EMC Corporation. All rights reserved. 73
    • FCoE Switch Topology Timeline Legend: Available today Coming soon 1350+ Cisco Nexus Requires FC-BB-6 Cisco 50x0 / 2232 Nexus 4000 / FCoE 450 50x0 Brocadephysical DCX blade ports 150 Cisco Native FCoE Nexus Brocade Targets 50x0 8000 50 10G DCB LOM 0 8 9 9 9 0 0 0 0 + -0 -0 -0 -0 -1 -1 -1 -1 11 4 2 3 4 1 2 3 4 – Q Q Q Q Q Q Q Q 1 Q EMC GA date© Copyright 2009 EMC Corporation. All rights reserved. 74
    • Native Targets and 10 G DCB LOM Native Targets – Symmetrix and CLARiiON – UNH-IOL FCoE Plugfest 10 G DCB LOM – 10 G – DCB – Data Center Bridging – LOM – LAN on Motherboard – Windows and Open FCoE© Copyright 2009 EMC Corporation. All rights reserved. 75
    • FCoE Switch Topology Timeline Legend: Multi-hop Available today DCB cloud Coming soon 1350+ Cisco Nexus Requires FC-BB-6 Cisco 50x0 / 2232 Nexus 4000 / FCoE 450 50x0 Brocadephysical DCX blade ports 150 Cisco Native FCoE Nexus Brocade Targets 50x0 8000 VN2VN 50 10G DCB LOM Point to Point 0 8 9 9 9 0 0 0 0 + -0 -0 -0 -0 -1 -1 -1 -1 11 4 2 3 4 1 2 3 4 – Q Q Q Q Q Q Q Q 1 Q EMC GA date© Copyright 2009 EMC Corporation. All rights reserved. 76
    • Agenda Fibre Channel over Ethernet (FCoE) overview – I/O consolidation – FCoE basics FCoE ecosystem components – Encapsulation – Data Center Bridging Capability eXchange Protocol (DCBX) – FCoE Initialization Protocol (FIP) – Priority Flow Control (PFC) – Enhanced Transmission Selection (ETS) – TRILL (TRansparent Interconnection of Lots of Links) FCoE Topology timeline – Top of Rack (ToR) – End of Row (EoR) When and where FCoE makes sense© Copyright 2009 EMC Corporation. All rights reserved. 77
    • When and where FCoE makes sense You are using VMware or Hyper-V You are building a new datacenter and want to future proof your investment You are building out a new server farm and want to reduce large quantities of FC and 1GbE cabling and connectivity You anticipate 10GbE will be ubiquitous in your data center and are ready to take big steps toward that goal. You have applications that require block I/O and an environment that requires centralized control You already have an investment in Fibre Channel and would like to seamlessly migrate both data and skill sets You want to save on CAPEX and OPEX now…© Copyright 2009 EMC Corporation. All rights reserved. 78
    • Summary and Questions Discussed topologies currently supported and what is currently being developed Explained the essential components of FCoE – Including DCBX, PFC, FIP, and DCB Clouds Highlighted known limitations when using FCoE Described when and where it’s best to use FCoE© Copyright 2009 EMC Corporation. All rights reserved. 79
    • Related Sessions at EMC WorldOther E-Lab presentations: Managing Dynamic Storage Devices in a Linux Environment Vblock Use Case: VMware View 4 Characterization of Microsoft Hyper-V in a Symmetrix Environment© Copyright 2009 EMC Corporation. All rights reserved. 80
    • Related Technical Documentation EMC Networked Storage Topology Guide EMC Fibre Channel over Ethernet (FCoE) TechBook Authored by Erik Smith,Mark Lippitt, Erik Paine, and Mark Anthony De Castro Available for purchase at EMC World Documents available at Powerlink.EMC.com elabNavigator.EMC.com© Copyright 2009 EMC Corporation. All rights reserved. 81