Converged Networks: FCoE, iSCSI and the Future of Storage Networking


Published on

Presentation from EMC World 2010 (Boston)

Published in: Technology
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Converged Networks: FCoE, iSCSI and the Future of Storage Networking

  1. 1. Converged Data Center: FCoE, iSCSI, and the Future of Storage Networking Stuart Miniman, Technologist, Office of the CTO EMC Corporation
  2. 2. Agenda <ul><li>The Journey to Convergence </li></ul><ul><li>Protocols & Standards Update </li></ul><ul><li>Solution Evolution </li></ul><ul><li>Conclusion and Summary </li></ul>
  3. 3. Rack Server Environment Today <ul><li>Servers connect to LAN, NAS and iSCSI SAN with NICs </li></ul><ul><li>Servers connect to FC SAN with HBAs </li></ul><ul><li>Many environments today are still 1 Gigabit Ethernet </li></ul><ul><li>Multiple server adapters, multiple cables, power and cooling costs </li></ul><ul><ul><li>Storage is a separate network (including iSCSI) </li></ul></ul>Rack-mounted servers Ethernet LAN 1 Gigabit Ethernet 1 Gigabit Ethernet NICs Today < 30% of servers in the data center are SAN attached to storage Ethernet Fibre Channel Storage Fibre Channel SAN Fibre Channel HBAs 1 Gigabit Ethernet iSCSI SAN
  4. 4. <ul><li>Transport storage (SCSI) over standard Ethernet </li></ul><ul><li>Reliability through TCP </li></ul><ul><li>SCSI has limited distance, iSCSI provides even more flexibility than FC due to IP routing </li></ul><ul><li>1Gb iSCSI has good performance </li></ul><ul><li>iSCSI has thrived, especially where the server, storage and network administrators are the same person </li></ul>The iSCSI Story Link IPsec IP TCP iSCSI SCSI Initiator IP Network
  5. 5. Why a New Option for FC Customers? <ul><li>FC has a large and well managed install base </li></ul><ul><ul><li>Want a solution that is attractive for customers with FC expertise / investment </li></ul></ul><ul><ul><li>Previous convergence options did not allow for incremental adoption </li></ul></ul><ul><li>Requirement for a Data Center solution that can provide I/O consolidation </li></ul><ul><li>Leveraging Ethernet infrastructure and skill set has always been attractive </li></ul>FCoE allows an Ethernet-based SAN to be introduced into the FC-based Data Center without breaking existing administrative tools and workflows
  6. 6. Non-Ethernet Convergence Options <ul><li>InfiniBand </li></ul><ul><ul><li>Used broadly for High Performance Computing (HPC) environments </li></ul></ul><ul><ul><li>Low cost and ultra-low latency geared for server to server cluster </li></ul></ul><ul><ul><li>Separate use from general network (Ethernet) or storage (FC or Ethernet) </li></ul></ul>Extend server’s PCI bus to appliance Appliance connects to existing infrastructure (FC, Ethernet) <ul><li>PCIe </li></ul><ul><ul><li>Extension of the server bus to an I/O aggregation box </li></ul></ul><ul><ul><ul><li>Single network from server -> top of rack, similar to early FCoE deployments </li></ul></ul></ul><ul><ul><li>Not a standard (parallel effort to MR-IOV standard), small players </li></ul></ul><ul><ul><li>Still using Ethernet and FC network and storage from the aggregation box </li></ul></ul>
  7. 7. 10Gb Ethernet allows for Converged Data Center <ul><li>Maturation of 10 Gigabit Ethernet </li></ul><ul><ul><li>10 Gigabit Ethernet allows replacement of n x 1Gb with a much smaller number (start with 2) of 10Gb Adapters </li></ul></ul><ul><ul><li>Single network allows for easier mobility for virtualization/cloud deployments </li></ul></ul><ul><li>10 Gigabit Ethernet simplifies server, network and storage infrastructure </li></ul><ul><ul><li>Reduces the number of cables and server adapters </li></ul></ul><ul><ul><li>Lowers capital expenditures and administrative costs </li></ul></ul><ul><ul><li>Reduces server power and cooling costs </li></ul></ul><ul><ul><li>Blade servers and server virtualization drive consolidated bandwidth </li></ul></ul>10 Gigabit Ethernet is the answer! iSCSI and FCoE both leverage this inflection point Single Wire for Network and Storage 10 GbE LAN SAN
  8. 8. FCoE Extends FC on a Single Network FC network FC storage Ethernet Network FCoE Switch Lossless Ethernet Links 2 options SAN sees host as FC Network Driver FC Driver Converged Network Adapter Server sees storage traffic as FC Ethernet FC FCoE SW Stack Standard 10G NIC
  9. 9. Time To Widespread Adoption 1990 2000 2010 1980 Defined 73 Standard 83 Widespread 93 Defined 85 Standard 94 Widespread 03 iSCSI Defined 00 02 Widespread 08 Standard Standard 10 Gigabit Ethernet 02 09 Widespread 07 09 ?? Defined Standard Widespread
  10. 10. Future <ul><li>40 & 100 Gb Ethernet (IEEE) standards will be completed in June 2010 </li></ul><ul><li>16Gb FC (T11) standard is targeted for completion at the end of 2010 </li></ul><ul><li>Server adoption of FC ~ 3+ years, of Ethernet ~ 5+ years </li></ul><ul><ul><li>Backbone typically faster adoption </li></ul></ul>16 Gb FC 40/100 Gb Ethernet 32 Gb FC
  11. 11. Ethernet Cabling Different short-distance option (QSFP) Low power 5-10m distance (Rack solution) N/A Twinax Copper / SFP+DA (direct attach) Expect shift to optical w/ 40/100Gb OM3 100m OM4 125m Most backbone deployments are optical OM2 82m OM3 300m Rare < 1% Ethernet, but Standard for FC OM2 ( orange ) OM3 ( aqua ) OM4* ( aqua ) Optical (multimode) / LC Not supported in initial standard Some products on market, but not for FCoE yet Cat6 55m; Cat 6a 100m > 99% of existing cabling (lots of Cat 5e) Cat6 or Cat6a Copper (10GBase-T) / RJ-45 40/100Gb 10Gb 1Gb Cable Type / Connector
  12. 12. Agenda <ul><li>The Journey to Convergence </li></ul><ul><li>Protocols & Standards Update </li></ul><ul><li>Solution Evolution </li></ul><ul><li>Conclusion and Summary </li></ul>
  13. 13. Standards for Next Generation Data Center <ul><li>Fibre Channel over Ethernet (FCoE) protocol </li></ul><ul><ul><li>Developed by International Committee for Information Technology Standards (INCITS) T11 Fibre Channel Interfaces Technical Committee </li></ul></ul><ul><ul><li>Fibre Channel over Ethernet allows native Fibre Channel to travel unaltered over Ethernet </li></ul></ul><ul><ul><li>FC-BB-5 standard ratified in June 2009 </li></ul></ul><ul><ul><li>FC-BB-6 in process to expand solution </li></ul></ul><ul><li>Converged Enhanced Ethernet (CEE) </li></ul><ul><ul><li>Developed by IEEE Data Center Bridging (DCB) Task Group </li></ul></ul><ul><ul><li>DCB/CEE creates an Ethernet environment that drops frames as rarely as Fibre Channel </li></ul></ul><ul><ul><li>Technology commonly referred to as Lossless Ethernet </li></ul></ul><ul><ul><li>IEEE standards targeting ratification in mid 2010 </li></ul></ul><ul><ul><li>Requirement for FCoE; Enhancement for iSCSI </li></ul></ul>Two parallel industry standards seek to drive I/O consolidation in large data centers over time: <ul><li>Companies working on the standard committees </li></ul><ul><ul><li>Key participants: Brocade, Cisco, EMC, Emulex, HP, IBM, Intel, QLogic, Oracle(Sun), others </li></ul></ul>
  14. 14. <ul><li>iSCSI is SCSI functionality transported using TCP/IP for delivery and routing in a standard Ethernet/IP environment </li></ul>iSCSI and FCoE Framing <ul><li>FCoE is FC frames encapsulated in Layer 2 Ethernet frames designed to utilize a Lossless Ethernet environment </li></ul><ul><ul><li>Large maximum size of FC requires Ethernet Jumbo Frames </li></ul></ul><ul><ul><li>No TCP, so Lossless environment required </li></ul></ul><ul><ul><li>No IP routing </li></ul></ul>FCoE Frame iSCSI Frame Data FC Frame CRC Ethernet Header Ethernet Header FCoE Header FC Header FC Payload CRC EOF FCS IP TCP iSCSI
  15. 15. FCoE Frame Formats <ul><li>Ethernet frames give a 1:1 encapsulation of FC frames </li></ul><ul><ul><li>No segmenting FC frames across multiple Ethernet frames </li></ul></ul><ul><ul><li>FCoE flow control is Ethernet based </li></ul></ul><ul><ul><ul><li>BB Credit/R_RDY replaced by Pause/PFC mechanism </li></ul></ul></ul><ul><li>FC frames are large, require Jumbo frames </li></ul><ul><ul><li>Max FC payload size is 2180 bytes </li></ul></ul><ul><ul><li>Max FCoE frame size is 2240 bytes </li></ul></ul><ul><li>FCoE Initialization Protocol (FIP) used for discovery and login </li></ul>Destination MAC Address Source MAC Address IEEE 802.1Q Tag ET = FCoE Ver Reserved Reserved Reserved SOF Encapsulated FC Frame (Including FC-CRC) EOF Reserved FCS Reserved FCoE Frame Format Bit 0 Bit 31
  16. 16. Storage Drivers and Server Virtualization NIC NIC vNIC vNIC vSCSI vSCSI *iSCSI initiator can also be in the VM virtual switch Hypervisor driver FC HBA FC HBA LAN traffic CNA CNA FCoE follows FC path Hypervisor iSCSI traffic
  17. 17. Storage Drivers and Server Virtualization NIC NIC vNIC vNIC vSCSI vSCSI FCoE software in the guest would send traffic through the vSwitch to the vNIC FC HBA FC HBA Hypervisor SW FCoE SW FCoE Hypervisor driver No FCoE access here currently virtual switch vSwitches from ESX (including Cisco 1000v option) and Hyper-V are not Lossless
  18. 18. FC-BB-6 <ul><li>Not required for multi-hop FCoE or other current deployments </li></ul><ul><li>Currently in the “herding cats” phase of defining goals </li></ul><ul><li>Likely to support point-to-point configuration which allows 2 FCoE devices to communicate without going through an FCF (or switch) </li></ul>For more, see Erik Smith of EMC E-Lab’s presentation FCoE - Topologies, Protocol, and Limitations Tues 8am and Thurs 8:30am
  19. 19. Lossless Ethernet <ul><li>IEEE 802.1 Data Center Bridging ( DCB ) is the standards task group </li></ul><ul><li>Converged Enhanced Ethernet ( CEE ) is an industry consensus term </li></ul><ul><li>Link level enhancements (Priority Flow Control, Enhanced Transmission Selection, Data Center Bridging Exchange Protocol) are shipping in products today </li></ul><ul><ul><li>Standards expected to be ratified in June ‘10 </li></ul></ul><ul><li>The “CEE cloud” or DCB-enabled LAN is only for the portion of your network that requires lossless functionality </li></ul><ul><ul><li>Currently limited to multimode (300m) distances per link (no singlemode) </li></ul></ul><ul><ul><li>Limit the environment to the Data Center; Layer 2 (no routing) </li></ul></ul>Enhanced Ethernet provides the Lossless Infrastructure which enables FCoE
  20. 20. PAUSE and Priority Flow Control <ul><li>PAUSE transforms Ethernet into a lossless fabric </li></ul><ul><li>Classical 802.3x PAUSE is rarely implemented since it stops all traffic </li></ul><ul><li>A new PAUSE known as Priority Flow Control (PFC) function that can halt traffic according to priority tag while allowing traffic at other priority levels to continue </li></ul><ul><ul><li>Creates lossless virtual lanes </li></ul></ul><ul><ul><li>PFC will be limited to Data Center </li></ul></ul><ul><li>Per priority link level flow control </li></ul><ul><ul><li>Only affect traffic that needs it </li></ul></ul><ul><ul><li>Ability to enable it per priority </li></ul></ul><ul><ul><li>Not simply 8 x 802.3x PAUSE </li></ul></ul>Switch A Switch B
  21. 21. Enhanced Transmission Selection and Data Center Bridging Exchange Protocol (DCBX) <ul><li>Enhanced Transmission Selection (ETS) provides a common management framework for bandwidth management </li></ul><ul><li>Allows configuration of HPC & storage traffic to have appropriately higher priority </li></ul><ul><li>When a given load in a class does not fully utilize its allocated bandwidth, ETS allows other traffic classes to use the available bandwidth </li></ul><ul><li>Maintain low latency treatment of certain traffic classes </li></ul><ul><li>Data Center Bridging Exchange Protocol (DCBX) is responsible for configuration of link parameters for DCB functions </li></ul><ul><li>Determines which devices support Enhanced Ethernet functions </li></ul>Offered Traffic t1 t2 t3 10 GE Link Realized Traffic Utilization 3G/s HPC Traffic 3G/s 2G/s 3G/s Storage Traffic 3G/s 3G/s LAN Traffic 4G/s 5 G/s 3G/s t1 t2 t3 3G/s 3G/s 3G/s 3G/s 3G/s 2G/s 3G/s 4G/s 6G/s
  22. 22. Beyond Link Level <ul><li>Congestion notification </li></ul><ul><ul><li>IEEE 802.1Qau ratified </li></ul></ul><ul><ul><ul><li>Allows a switch to notify attached ports to slow down transmission due to heavy traffic, in order to reduce the chances of packet drops or network deadlocks </li></ul></ul></ul><ul><ul><ul><li>Moves the management of congestion back to the edge, which helps alleviate network-wide bottlenecks </li></ul></ul></ul><ul><li>Layer 2 multipathing </li></ul><ul><ul><li>IETF TRILL - TR ansparent I nterconnection of L ots of L inks </li></ul></ul><ul><ul><ul><li>Used with the Spanning Tree Protocol to provide more efficient bridging and bandwidth aggregation </li></ul></ul></ul><ul><ul><ul><li>Focuses on a bridging capability that will increase bandwidth by allowing and aggregating multiple network paths </li></ul></ul></ul><ul><ul><ul><li>Standards are stable; products are coming soon </li></ul></ul></ul>Throttle Switch Transmit Queue Switch Receive Buffer Throttle
  23. 23. Agenda <ul><li>The Journey to Convergence </li></ul><ul><li>Protocols & Standards Update </li></ul><ul><li>Solution Evolution </li></ul><ul><li>Conclusion and Summary </li></ul>
  24. 24. iSCSI Deployment <ul><li>iSCSI was > 15% of revenue ($1.8B in ‘09) and > 20% capacity in SAN market in 2009 * </li></ul><ul><li>10 Gb iSCSI solutions are available </li></ul><ul><ul><li>Can work on both Traditional Ethernet (recover from dropped packets using TCP) or Lossless Ethernet (DCB) environment </li></ul></ul><ul><li>iSCSI natively routable (IP) </li></ul><ul><li>iSCSI solutions are much smaller scale than FC </li></ul><ul><ul><li>A single FC director is larger than most iSCSI environments </li></ul></ul>* According to IDC, 2009 Ethernet iSCSI SAN
  25. 25. FCoE Solutions in 2009 <ul><li>FCoE with direct attach of server to Converged Network Switch at top of rack or end of row </li></ul><ul><li>Tightly controlled solution </li></ul><ul><li>Server 10 GE adapters may be CNA or NIC </li></ul><ul><li>Storage is still a separate network </li></ul>Converged Network Switch FC Attach Rack Mounted Servers 10 GbE CNAs Ethernet LAN Storage Fibre Channel SAN Ethernet FC
  26. 26. Expansion of FCoE beyond a single switch <ul><li>First solutions are with an FCoE aware Ethernet switch (also known as FIP snooping ) </li></ul><ul><li>Enables rack bundle solutions </li></ul>Rack Area Network (RAN) allow for the rack as a unit of design and a unit of management FC switch FCoE switch FCoE aware Ethernet switch (embedded) CNA (embedded) FCoE enabled Blade Server storage
  27. 27. CX4-480 or Celerra NS-960 <ul><li>2 x Nexus 6140 </li></ul><ul><li>FCoE, Ethernet, FC ports </li></ul><ul><li>8 x UCS 5100 chassis </li></ul><ul><li>Includes CNA blade (2 x FCoE) </li></ul><ul><li>vSphere/ESX 4.0 </li></ul><ul><li>16:1 consolidation </li></ul><ul><li>1024 VMs </li></ul>Vblock1 example in 3 floor tiles <ul><li>Cisco UCS </li></ul><ul><ul><li>Optimized compute environment utilizing FCoE technology </li></ul></ul><ul><ul><li>Includes embedded FCoE adapter and ToR switch equivalent, plus the option for embedded FCoE switch </li></ul></ul><ul><li>Vblock </li></ul><ul><ul><li>Integrated Virtual Computing Environment combining best-in-class networking, compute, storage, security and management solutions </li></ul></ul><ul><ul><li>Cisco UCS + EMC storage + VMware vSphere/ESX </li></ul></ul>Cisco & VCE FCoE Rack Area Network (RAN)
  28. 28. IBM & HP FCoE offerings <ul><li>IBM BladeServers w/ embedded FCoE adapter for pass-thru switch module or an embedded FCoE switch module which attaches to ToR switch </li></ul>1x CNA Card 2-port 10Gb CNA 2 x 10Gb Switch 10-port 10Gb Switch or 14-port IBM 10Gb Pass-Thru TOR switch splits LAN & SAN traffic Blade Switch Top of Rack CNAs Converge traffic at the server End of Row Aggregation Core Converged edge will evolve from ToR/EoR to aggregation and core layers
  29. 29. Challenges for FCoE <ul><li>FCoE solution development will take time to expand and mature, just as with other technologies – customers looking to create topologies similar to existing FC configurations: </li></ul><ul><ul><li>Director support </li></ul></ul><ul><ul><li>Edge-core support (multi-hop) </li></ul></ul>Storage Network vs <ul><li>Organizational domain overlap </li></ul>
  30. 30. EMC and Ethernet <ul><li>Best Practices </li></ul><ul><ul><li>Google “ FCoE Tech Book ” (FCoE & Ethernet) </li></ul></ul><ul><li>Services </li></ul><ul><ul><li>Design, Implementation, Performance and Security offerings for networks </li></ul></ul><ul><li>Products </li></ul><ul><ul><li>Ethernet equipment for creating Converged Network Environments </li></ul></ul>
  31. 31. FCoE Timeline Supported 2010 FCoE top-of-rack switches 2nd Generation CNAs Windows, Linux, VMware Cisco UCS and Vblock IBM BladeCenter Native FCoE storage FCoE blades for FC and Ethernet directors UNIX support Open FCoE Expanded multi-hop solutions More embedded/server solutions 10Gb DCB LOMs 10G-BaseT w/ FCoE TRILL solutions Direct Connect (FC-BB-6) 40GbE/100GbE Future
  32. 32. Agenda <ul><li>The Journey to Convergence </li></ul><ul><li>Protocols & Standards Update </li></ul><ul><li>Solution Evolution </li></ul><ul><li>Conclusion and Summary </li></ul>
  33. 33. Next Generation Data Center <ul><li>10 Gigabit Ethernet </li></ul>Fibre Channel FCoE NAS iSCSI LAN <ul><li>EMC is working with the standards communities and partners to deliver the same reliability and robustness in the next generation virtual data center that we deliver today </li></ul>The Converged Data Center sets the operational and capital efficiency foundations for the virtual data center and private clouds Private Cloud Virtualized Data Center Cloud Computing virtualization common infrastructure common management
  34. 34. Summary <ul><li>A converged data center environment can be built using 10Gb Ethernet </li></ul><ul><li>Achieving a converged network requires consideration of technology, processes/best practices and organizational dynamics </li></ul><ul><li>10 Gigabit Ethernet solutions are maturing </li></ul><ul><ul><li>Active industry participation is creating standards that allow solutions that can integrate into existing data centers </li></ul></ul><ul><ul><li>Continued use of FC and adoption of FCoE can be flexible due to shared management </li></ul></ul><ul><ul><li>FCoE and iSCSI will follow the Ethernet roadmap to 40 and 100 Gigabit in the future </li></ul></ul>
  35. 35. Related References <ul><li>Full collection of FCoE references (blog, video, whitepaper, presentations and other EMC links) http:// </li></ul><ul><li>Industry site with consolidated information </li></ul><ul><li>T11 FCoE activity </li></ul><ul><li>IEEE 802.1 Data Center Bridging task group page </li></ul><ul><li>Other EMC Bloggers covering these technologies: </li></ul><ul><ul><li>Chad Sakac </li></ul></ul><ul><ul><li>Chuck Hollis </li></ul></ul><ul><ul><li>David Graham http:// / </li></ul></ul>