Fibre Channel over Ethernet (FCoE)

5,990 views

Published on

1 Comment
5 Likes
Statistics
Notes
No Downloads
Views
Total views
5,990
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
649
Comments
1
Likes
5
Embeds 0
No embeds

No notes for slide

Fibre Channel over Ethernet (FCoE)

  1. 1. Fibre Channel over Ethernet (FCoE) John L Hufferd, Brocade
  2. 2. SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced without modification The SNIA must be acknowledged as source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee. Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 2
  3. 3. Abstract A new concept is currently moving through the Fibre Channel (T11) standards committee called Fire Channel over Ethernet (FCoE). The FCoE standard will specify the encapsulation of Fibre Channel frames into Ethernet Frames and the amalgamation of these technologies into a network fabric that can support Fibre Channel protocols and other protocols such as TCP/IP, UDP/IP etc. The tutorial will show the Fundamentals of the FCoE concept and describe how it might be exploited in a Data Center environment. Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 3
  4. 4. Agenda Introduction Goals & Requirements Consolidation Architecture Discovery & Link Instantiation Topologies Scenarios Summary Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 4
  5. 5. Introduction This presentation provides an overview of Fibre Channel over Ethernet (FCoE) One should think about FCoE as placing the FC protocol on a new physical link Ethernet links instead of physical FC links But it is still Fibre Channel The protocol is being defined in the INCITS Fibre Channel (T11) technical committee Many details of the protocol still need to be defined but the significant major issues have been resolved Target T11 Standards completion is the 2H08 Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 5
  6. 6. The Origins of Data Center Ethernet 2008-2009 CEE (Converged Enhanced Ethernet) 10Gbps Increasing Performance Integration of Carrier-grade features 1Gbps Incremental Protocol Enhancements 100Mbps Logical Partitioning 10Mbps Evolution from shared media to dedicated media Introduction of Ethernet 1973 Increasing Scalability, Feature, Function The technology has evolved continuously, showing a great ability to adapt to new technologies and increasing business requirements Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 6
  7. 7. Goals/Requirements Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 7
  8. 8. FCoE Goals/Requirements (part 1) FCoE Fabrics must be built with FCoE Switches Switches with Ethernet ports that provide FCoE capabilities and services – Called, in the standard, an FC Forwarder FCF Switches that include the functions of traditional FC switches Standard Ethernet switches may also exist in the fabric but switches with FCoE capabilities are required FCoE fabrics must operate seamlessly with real FC Fabrics FC services must operate identically on FCoE fabrics and Fibre Channel fabrics FCoE must support all Fibre Channel advanced features (e.g. virtual fabrics, IFR, security, etc.) transparently FCoE is NOT a replacement for FCIP or iFCP FCIP & iFCP use TCP/IP FCIP/iFCP is for inter-switch links beyond the Data Center Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 8
  9. 9. Integrated Ethernet & FCoE switch (FCF) with FC connections Some implementations may combine the features and capabilities of an Ethernet Switch with the features and capabilities of a FC switch • Support Ethernet and IP standards for switching, pathing and routing • Support FC standards for switching, pathing and routing CEE Ethernet Ports • Support current and enhanced Ethernet Standards (with IP & FCoE VF_Port & FC FC FC VE_Port capabilities) • Adapt between FCoE and FC An FCoE Port (N_Port, F_Port or E_Port) • Has the same function as in FC • But is layered on top of Ethernet • Called VN_Port, VF_Port and VE_Port Because many logical (virtual) Ports can share one physical port Note: FCF Pathing and Forwarding utilizes the FSPF (Fabric Shortest Path First) protocol Non-FCoE Ethernet traffic is relayed using conventional 802.1 defined mechanisms such as STP (Spanning Tree Protocol) and MSTP (Multiple STP) See additional FSPF and STP explanation in Appendix Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 9
  10. 10. Connections to a Combo FCoE Switch • Fibre Channel is carried over lossless Ethernet as a L3 protocol SCSI SCSI Applications iSCSI Fibre Channel UDP TCP IP FCoE Lossless Ethernet MAC (CEE) IP address 123.45.67.89 (FCoE VN_Port) Ethernet port with IP & FCoE VF_Port capabilities Combo Lossless Ethernet (CEE) Switch with FCoE Switch (FCF) capabilities Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 10
  11. 11. FCoE Goals/Requirements (part 2) FCoE requires specific Ethernet extensions to be implemented Lossless switches and fabrics (e.g., supporting IEEE 802.3 PAUSE) configurations are required Jumboframe support is required (not a standard, but widely available) Deployments of FCoE should utilize the advances in Ethernet currently being discussed in IEEE 802.1, specifically: Priority-based Flow Control (PFC) Enhanced Transmission Selection (ETS) DCB Capability eXchange Protocol (DCB CXP) These 802.1 advances are important for Consolidated Flows (Messaging, Clustering and Storage) This set of functions has been called Data Center Ethernet, or CEE – Converged Enhanced Ethernet (intended for a Data Center Environment) FCoE should require no changes to FC software Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 11
  12. 12. Consolidation Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 12
  13. 13. High End 10GE Server & NIC/HBA Consolidation OS1 Web Server OS2 App Server OS3 DB Server With CEE Hyper Visor (vmWare, Xen, etc) OS2 FC OS2 OS3 App Server HBA App Server DB Server OS1 OS1 OS3 Web Server Web Server DB Server Messaging Ethernet FC SAN •NIC E-HBA E-HBA •TCP acceleration. TCP/IP (CEE) E-HBA (CEE) •MPI, RDMA (CEE) MPI RDMA over Ethernet •FCoE Today IB/Ethernet Cluster Data Center CEE Network Data Center CEE Network Dramatic reduction in adapter, switch ports and cabling 4-6 cables to 2 cables per server Seamless connection to the installed based of existing SANs and LANs Requires high performance lightweight frame mappers vs. heavy weight gateways FCoE has no need to terminate and re-initiate a SCSI connection (e.g. iSCSI to FC) Effective sharing of high bandwidth links Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 13
  14. 14. Architecture Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 14
  15. 15. FC Encapsulation Into Network Packets (2 FCoE Related Packet types) FC-4 FC-4 FC Levels FC-3 FC-3 (Unchanged) FC-2 FC-2 FCoE Mapping FC-1 MAC IEEE 802.3 FC-0 PHY Layers Fibre Channel over Ethernet (FCoE) Packets Ethernet FCoE FC Header SCSI Commands/Data FCS Header Header FC Imbedded Frames: Same as in Physical FC Ethertype Protocol control information: Version, SOF, EOF, etc. “FCoE” Frame Check Sequence Ethernet Header provides things needed for the (CRC) physical network, including “Ethertype” FCoE Initialization Protocol (FIP) Packets Ethernet FIP Header Descriptors FCS Header Discovery and Login/Logout Parameters Ethertype Protocol control information: Version, Op-codes, etc. “FIP” Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 15
  16. 16. FIP Protocol and FCoE Protocol Discovery Phase FCFs Discovery each other and form a Fabric ENodes and FCFs Discover Potential VN_Port VF_Port pairing Capabilities of Potential pairing Login Phase FCoE ENodes chose among discovered FCFs’ Ports Initialization Creates association between ENode Ports and FCF Ports Protocol (FIP) VN_Port VF_Port Logical FC Link Two allowed alternatives for the ENode MAC Addressing Fabric Provided MAC Addresses (FPMA) Server Provided MAC Addresses (SPMA) Chosen by FCF (FPMA & SPMA Described latter) Uses: FLOGI, FLOGI ACC, LOGO, … Data Transfer Phase FCoE Protocol PLOGI/PRLI All other FC protocol frames (ELS, FC4 ULPs. etc.) Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 16
  17. 17. FIP Operation Format Word 31-24 23-16 15-8 7-0 0 Destination MAC Address (6 Bytes) 1 Optional IEEE 802.1q 2 Source MAC Address (6 Bytes) 4 Byte Tag goes here 3 ET=FIP (16 bits) Ver (4b) Reserved (12 bits) 4 FIP Operation Code Reserved FIP Operation Code FIP SubCode See Appendix for Descriptor list items Reserved List Length Descriptor FIP subcode Flags Descriptor List Length 5 FP SP S F Flags F S S F 6 P P Descriptor list varies Descriptor List In size … n PAD to minimum length or mini-Jumbo length Ethernet frame size n+1 Ethernet FCS Is 64Bytes to 2220Bytes Capability Bits (SPMA or FPMA) Solicited bit FCF bit Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 17
  18. 18. FC’s Encapsulation in Ethernet (FCoE) Word 31-24 23-16 15-8 7-0 0 Destination MAC Address (6 Bytes) 1 Optional IEEE 802.1q 2 Source MAC Address (6 Bytes) 4 Byte Tag goes here 3 ET=FCoE (16 bits) Ver (4b) Reserved (12 bits) 4 Reserved 5 Reserved 6 Reserved SOF (8 bits) 7 Encapsulated FC Frame This field varies … FC Frame = Minimum 28 Bytes (7 Words) In size Maximum 2180 Bytes (545 Words) n (including FC-CRC) n+1 EOF (8 bits) Reserved Ethernet frame n+2 Ethernet FCS size Is 64Bytes to 2220Bytes Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 18
  19. 19. FCF Model Link End Point (LEP) Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 19
  20. 20. ENode (HBA) Model Link End Point (LEP) Each ENode (HBA) may have multiple Physical Ethernet Ports Each Physical Port may have multiple Logical VN_Ports Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 20
  21. 21. Model of the ENODE with Multiple Logical FC interfaces In this Multiple FC NPIV model this instances on a single FC-3 /FC-4s is where logical FC Host interface FC-3 /FC-4s FC-2 functions • For each logical N_Port live (VN_Port) there is one FLOGI In this model this and perhaps 100’s of FDISC VN_Port FC ... VN_Port FC is where the Entity Entity Encapsulation /De-Encapsulation • Each VN_Port is seen by FCoE_LEP FCoE FCoE_LEP FCoE functions live Entity the Host as a separate Entity FCoE (logical) FC connection Controller MAC Address of FCoE_LEP (VN_Port) May or may not be • The number of (logical) FC Lossless Ethernet MAC Ethernet_Port the same as the FCoE controller connections is MAC Address of “Burnt-in implementation dependent MAC • Only one MAC Address is required for the FCoE Controller and the VN_Ports on a single physical MAC (aka Server Provided MAC Address – SPMA) • FCF may chose to specify new MAC addresses for each VN_Port (aka Fabric Provided MAC Address – FPMA) Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 21
  22. 22. Multiple Logical FC connections via a single Ethernet MAC Examples of single MACs with connections to two different FCFs Switch The Logical FC Link is defined by a MAC Address pair • A VN_Port MAC Address • A VF_Port MAC Address For a logical FC link the FCoE Frames are always sent to and received from a specific FCF’s MAC Address • Therefore, pathing to and from the FC driver is always defined by the MAC Address of the partner FCF’s VF_Port Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 22
  23. 23. Functions of an FCoE Initiator ASIC A B C Host PCIe interface • Has a Normal NIC interface FCoE (A) to the Host ASIC • Has one or more Normal FC interfaces (B,C) to the function Host • FCoE functions not seen by FCoE NIC Function FC the Host Function • FCoE functions perform the Encapsulation and De- Lossless encapsulation Ethernet MAC • The FCoE function External Instantiates a Logical FC Port N_Port, called a VN_Port Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 23
  24. 24. HBA with Multiple Logical FC Interfaces • The FCoE controller will perform the FIP functions and will instantiate new VN_Ports as FCoE Link End Point (LEP) • With the same MAC address as the FCoE Controller (SPMA) • Or with a new MAC address specified by the FCF (FPMA) A B C FCoE Chip MAC Address of “Burnt-in • MAY have one “burnt-in MAC MAC address for both IP FCoE_LEP VN_Port and FCoE/FIP packets FC-3 /FC-4s Entity Entity FCoE FC Or NIC Controller FCoE FCoE Function • MAY have different “Burnt- Etype= in” MACs for IP and FC-3 /FC-4s FCoE_LEP VN_Port FCoE or FCoE/FIP packets FIP? • Used to separate HW Entity Entity FCoE based FCoE from other FC Lossless Ethernet MAC Ethernet_Port Ethernet Traffic • Most NICs come with MAC several “Burnt-in” MAC Address of “Burnt-in MAC Addresses Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 24
  25. 25. FIP (FCoE Initiation Protocol) Discovery and Link Instantiation Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 25
  26. 26. ENode to FCF Logical Links An ENode must Locate FCFs with a Discovery protocol and then establish a Logical FC Link with an FCF (VN_Port VF_Port) before an Normal FC frame flow VF_Port VN_Port FCoE_LEPs FCoE_LEP FCF-MAC(A) MAC(H1) FCF A VF_Port H1 FCoE_LEPs FC fabric VN_Port FCF-MAC(B) FCoE_LEP Lossless VF_Port Note: The FCFs must VN_Port Ethernet FCoE_LEPs discover each other and MAC(H2) Switch create a Fabric before the FCoE_LEP FCF B ENode Discover FCFs VF_Port H2 FCoE_LEPs Classical Ethernet VN_Port FCoE_LEP VF_Port, LAN VF_Port_Name VN_Ports, FCF-MAC(x): A MAC address of a Lossless Ethernet port of FCF x VN_Port_Names MAC(y): A MAC address of a Lossless Ethernet port of ENode y Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 26
  27. 27. Initial Login Flow ladder (2 Phases) FCoE Initialization End-Node FCF Protocol Discovery Multicas to “ALL t FCFs” Discovery Unicast FCF MAC Address et.al. in Jumbo Frame Phase FLOGI FLOGI ACC with the FCF’s Login Phase chosen VN_Port MAC address as a descriptor value FC Command (Using the FCF Normal FC selected MAC Address as the SA) Processing FC Command responses (using the command’s SA as a DA) See Appendix for more details in Discovery Consideration and Actions FCOE Protocol Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 27
  28. 28. Subsequent Login Flow ladder (1 Phase) FCoE Initialization End-Node FCF Protocol FLOGI FLOGI ACC with the FCF’s Login Phase chosen VN_Port MAC address as a descriptor value FC Command (Using the FCF selected MAC Normal FC Address as the SA) Processing FC Command responses (using the command’s SA as a DA) See Appendix for more details in Discovery Consideration and Actions Fibre Channel over Ethernet (FCoE) FCOE Protocol 28 © 2008 Storage Networking Industry Association. All Rights Reserved.
  29. 29. Topologies Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 29
  30. 30. FCoE Switch & FC Fabric Note: FCoE servers and storage will probably use an FCoE HBA (or chip set) FCoE Switch FC FC An FCoE Switch may connect to a normal FC switch Via the FC E-Port Ethernet FC Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 30
  31. 31. Multiple FCoE Switch (FCF) Topologies FCoE Switch Lossless FC FC Ethernet FC Lossless Ethernet FC Fabric Fabric Lossless Ethernet switches configured into a Lossless Ethernet A Lossless Ethernet Fabric can be made up of Combo FCoE Switches Fabric can Front the FCoE Switch Lossless Ethernet 3 FCoE FCoE Switch Switch FCoE Switch Lossless FC FCoE FCoE Ethernet FC 2 Switch Switch Lossless FCoE Ethernet Lossless Ethernet FC 1 Switch FC Lossless Ethernet 4 FCoE Switches deployed at the edges of the Lossless Ethernet Fabric A VE_Port in an FCF may connect to other VE_Port in another FCF FCoE Switches connected via VE_Ports and Lossless Ethernet And an FCF FC E_Port may connect to an FC switch E_Port Fibre Channel over Ethernet (FCoE) Ethernet © 2008 Storage Networking Industry Association. All Rights Reserved. FC 31
  32. 32. Status of Current Data Center Networking Data Center Outfacing (IP) Network Server & Storage Network Management Group Management Group There can be 3 different System Area Network (SyAN) networks •Clustering Fabric System Area Network (SyAN) •InfiniBand •Myrinet Used for Clustering/Low Latency •Ethernet •CTC Storage area Network (SAN) Used to Access to Storage Remote Offices LAN/WAN External Networks Used for General messaging Used for Client-Server Messaging LAN/WAN Used for NAS •Messaging Data Center •NAS Often divided into at least 3 management domains Storage Area Network - Fibre Channel Data Center Server (clustering) Network Data Center Storage Network Outfacing (IP) Network LAN/WANs Note: with multiple Data Centers there may also be interconnects with DWDM, FCIP/iFCP, etc. – Messaging – NAS Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 32
  33. 33. Current Fabrics Remote Clustering Network Offices iSCSI Storage Mainframe Focus: Low Ficon Latency & High Bandwidth FC Outfacing FC FC IP Network FC Network (LAN/WAN) •Messaging •NAS Ficon Local & Remote Business Campus Ficon Storage Controller FC Link Focus: Protection, Clustering Network Ficon Bandwidth/Congestion Link File Storage Management Arrays (NAS) Ethernet Link Data Center Server & Storage Network Management Group IP Network Management Group There is a FC Storage Network, a Clustering Network & an “Outfacing” Network Note: with multiple Data Centers there may also be interconnects with DWDM, FCIP/iFCP, etc. Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 33
  34. 34. View of the Future Data Center Networks Data Center Server & Outfacing (IP) Network Storage Network Management Group Management Group The Server/Storage Networks will become a Consolidated Fabric Remote Managed by the Data Center Offices System and Storage Management Group – Includes Storage and Storage & System clustering provisioning Other IP Nets Area Network •Logically Single Fabric (LAN/WAN) – SAN and SyAN managed •Messaging (with FCoE) •NAS as a single fabric Focus: High Bandwidth and Low Latency The Outfacing (IP) Management Group remains the same Note: with multiple Data Centers there may also Focus: Protection, Bandwidth be interconnects with DWDM, FCIP/iFCP, etc. and Congestion Management Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 34
  35. 35. Phase in: FC & High Performance Lossless Ethernet Unify into a Data Center Fabric Mainframe Remote Offices Ficon iSCSI Storage Ethernet SW FC DataCenter Outfacing FC & Ethernet Fabric IP Network FC SW Blades (LAN/WAN) Including iSCSI •Messaging Gateways •NAS FCoE Ficon FC Ethernet Business SW Campus with iSCSI Ficon Storage connections File Storage Controller Arrays (NAS) Note: with multiple Data Centers there may also be interconnects with DWDM, FCIP/iFCP, etc. • FCoE permits intermixing of multiple Connection types/protocols FC Link • Clustering messaging, General Messaging, and Storage Ficon Link • The DataCenter Fabric will “Trunk” to the “Outfacing” Network (including iSCSI sys) Ethernet Link • Some Customers may want keep a mixed environment on-going Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 35
  36. 36. Scenarios Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 36
  37. 37. Scenario 1: FCoE & IP Flows Classical Internet Ethernet Lossless Ethernet Lossless FCoE Ethernet Switches FC FC FCoE Switch FCoE Flows IP Flows Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 37
  38. 38. Scenario 2: FCoE Right & Wrong Classical Internet Ethernet Lossless Ethernet Lossless FCoE Ethernet Switches FC FC FCoE Switch FCoE Flows Invalid FCoE Flows Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 38
  39. 39. Summary Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 39
  40. 40. Summary FCoE is a simple, efficient mechanism for encapsulating Fibre Channel in Ethernet frames FCoE is being standardized in INCITS Fibre Channel (T11) technical committee Target completion is 2H08 Maximum benefit of Fibre Channel is achieved: Evolutionary model of FC Switches and FC SANs Emphasis placed on capitalizing on the benefits of Converged Enhanced Ethernet (CEE) Being discussed in the IEEE 802.1 standards working group Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 40
  41. 41. Thank You!
  42. 42. Q&A / Feedback Please send any questions or comments on this presentation to SNIA: tracknetworking@snia.org Many thanks to the following individuals for their contributions to this tutorial. SNIA Education Committee Claudio DeSanti Howard Goldstein Walter Dey Robert Snively Suresh Vobbilisetty Silvano Gai Joe Pelissier John Hufferd For additional information refer to http://www.t11.org/fcoe Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 42
  43. 43. Appendix Additional Requirements FCoE Relation to ISO Layers Flows Additional Topologies FSPF and STP FIP Considerations and Actions FIP Descriptors Pause vs. BB_Credit Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 43
  44. 44. FCoE Goals/Requirements (part 3) FCoE must be a direct mapping of Fibre Channel over an Ethernet network FCoE must be layered on top of Ethernet FSPF used to route FCoE packets Ethernet Spanning Tree (STP), MSTP, etc, is at a layer below FCoE to allow an evolutionary approach towards consolidation of fabrics The Fibre Channel N_Port, F_Port and E-Port constructs must be retained With FCoE, ports may be connected with Logical Ethernet Links – May pass through Ethernet switches – Identified by pairs of end point MAC addresses Physical Ethernet Links can replace physical FC Links Physical Ethernet Links can carry all Ethernet traffic, including FCoE, but combined traffic needs the CEE capabilities Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 44
  45. 45. FCoE Goals/Requirements (part 4) “Combo FCoE Switches” may be built that support normal Ethernet traffic, FCoE traffic , & FC traffic The FCoE solutions should appear as a Fibre Channel to a Fibre Channel experienced customer FCoE should keep the Fibre Channel operations independent from Ethernet forwarding Keeps management /Troubleshooting simple Common physical structures, different logical structures Based on Ethertype (Ethertype = FCoE) Storage Management should be unchanged Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 45
  46. 46. FCoE Relation to ISO Layers Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 46
  47. 47. Flows Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 47
  48. 48. Logical Fabric Topology An FCoE Switch receives FCoE frames addressed to its FC-MAC address and forwards them based on the D_ID of the encapsulated FC frame Logical Lossless Transaction Path FCoE Ethernet Ethernet Switch FC A H2 #2 FCoE Switch #1 C Lossless #3 FC FC Lossless Ethernet #4 B Ethernet FCoE Switch H4 Lossless S2 Ethernet H1 H3 S1 An FCoE Switch rewrites the SA and DA of an FCoE frame Path #1 Path #2 Path #3 Path #4 Ethernet Destination FCoE-A MAC FCoE-B MAC FCoE-C MAC FCoE-S1 MAC & Source FCoE-H2 MAC FCoE-A MAC FCoE-B MAC FCoE-C MAC Encapsulated FC Frame D_ID FC_ID for S1 FC_ID for S1 FC_ID for S1 FC_ID for S1 S_ID FC_ID for H2 FC_ID for H2 FC_ID for H2 FC_ID for H2 Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 48
  49. 49. Single Ethernet Fabric with FCoE Switches H3 Ethernet H2 Ethernet Switch Switch FCoE #1 Switch FC FC Ethernet Switch C FCoE A Switch #2 #3 S2 Ethernet Ethernet Switch Lossless Ethernet Switch H1 S1 Path #1 Path #2 Path #3 Ethernet Destination FCoE-A MAC FCoE-C MAC FCoE-S1 MAC & Source FCoE-H2 MAC FCoE-A MAC FCoE-C MAC Encapsulated FC Frame D_ID FC_ID for S1 FC_ID for S1 FC_ID for S1 Logical Transaction Path S_ID FC_ID for H2 FC_ID for H2 FC_ID for H2 Ethernet FC Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 49
  50. 50. FC Host to FCoE Storage H3 S2 H5 Ethernet #1 H2 Ethernet #3 Switch FCoE Switch Switch FC Ethernet C FC FCoE Switch A Switch #2 Ethernet Ethernet S3 Switch Lossless Ethernet Switch H1 S1 Path #3 Path #2 Ethernet Destination FCoE-S2 MAC FCoE-A MAC & Source FCoE-A MAC FCoE-C MAC Path #1 Encapsulated FC Frame D_ID FC_ID for S2 FC_ID for S2 FC_ID for S2 Logical Transaction Path S_ID FC_ID for H5 FC_ID for H5 FC_ID for H5 Ethernet FC Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 50
  51. 51. Additional Topologies Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 51
  52. 52. Additional Topologies (1) Lossless Ethernet FCoE FCoE Switch Switch (E) Switch (A) FC FC FCoE Lossless FCoE Switch (B) Ethernet Switch (F) Switch Example of Topologies with Rack Mount servers Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 52
  53. 53. Additional Topologies (2) Lossless Ethernet FCoE FCoE Switch Switch (A) Switch (E) FC FC Lossless FCoE FCoE Ethernet Switch (B) Switch (F) Switch Example of Topologies with Blade Servers Equivalent to Blade servers with N_Port_ID Virtualization (NPIV) Support Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 53
  54. 54. FSPF and STP Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 54
  55. 55. FSPF & STP Concepts with FCoE (basic) Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 55
  56. 56. FSPF & STP Concepts with FCoE (interconnected) Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 56
  57. 57. Equivalent FC topology Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 57
  58. 58. FIP Considerations and Actions Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 58
  59. 59. FCoE Discovery Considerations The Discovery phase of the FCoE Initialization Protocol (FIP) uses two types of messages, Solicitations and Advertisements The FIP Discovery Phase helps define the FCF Ports that are available for the Link instantiation Phase The ENodes discover the FCF ports that can become VF_Ports and FCFs discover other FCF ports that can become VE_Ports ENodes Solicit (via Multicast) Advertisements from FCFs while specifying their capabilities In response FCF’s Advertise their availability and capabilities back to the ENodes FCF’s Multicast their existent to other FCF The FIP Discovery phase exchanges solicitation and/or Advertisements between (HBA and/or FCF) “FCoE Controllers” Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 59
  60. 60. FCF Discovery Actions FCF supporting VE_Ports: Discovers other VE_Port capable FCF-MACs, connected to the same Lossless Ethernet segment, by: Transmitting a multicast Solicitation to ‘All-FCFs’ (with the FCF bit set to one) Receives back Jumbo Unicast Advertisements from VE capable MACs To verify the support of Ethernet Jumbo frames in the path In response to receipt of a Multicast to “All-FCFs” from another FCF Has its own VE capable MACs sends their own Jumbo Unicast Advertisements Instantiates VE_Port to VE_Port connections and Exchanges FC ELP (Extended Link Protocol) and Fabric configuration (using Ethertype=FCoE) with the other FCFs VE_Port capable MACs Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 60
  61. 61. ENode Actions When an ENode becomes operational: The ENode discovers the VF_Port capable FCF-MACs connected to the same Lossless Ethernet segment Transmits a multicast Solicitation to ‘All-FCFs’ (with the FCF bit set to zero) Receives Jumbo unicast Advertisements from select compatible VF_Port capable FCF-MACs Stores the discovered FCF-MACs in an FCF list When an ENode receives an Advertisement that a new FCF is available, it may send a unicast Solicitation to it and receive a Jumbo unicast Advertisement in reply To verify the support of Ethernet Jumbo frames in the path May then perform FLOGIs (with Ethertype-FIP) to a vendor specific subset of the FCF-MACs in the FCF list Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 61
  62. 62. FIP Descriptors Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 62
  63. 63. FIP Descriptors (1) Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 63
  64. 64. FIP Descriptors (2) Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 64
  65. 65. Pause vs. BB_Credit Both mechanisms are used to avoid dropping frames With different trade-offs The Pause mechanism requires at least the (2 x RTT x bandwidth) product on a link as buffer space But allows Buffer handling in an arbitrary way Well suited for networks with limited (bandwidth x delay) product (e.g. within the data center) The Pause frame is handled by the MAC layer Similar to the R_RDY handling by the FC-1 level The BB_Credit mechanism prevents loosing frames over any link But links go under-utilized if link credits (& buffers) are < that needed for (RTT x BW) Requires buffer handling in maximum frame size units Fibre Channel over Ethernet (FCoE) © 2008 Storage Networking Industry Association. All Rights Reserved. 65

×