Converged Data Center: FCoE, iSCSI, and the Future of Storage Networking ( EMCWorld 2010 )

1,716 views

Published on

Converged Data Center: FCoE, iSCSI, and the Future of Storage Networking ( EMCWorld 2010 )

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,716
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
2
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Converged Data Center: FCoE, iSCSI, and the Future of Storage Networking ( EMCWorld 2010 )

  1. 1. Converged Data Center: FCoE, iSCSI, and the Future of Storage Networking Stuart Miniman, Technologist, Office of the CTO EMC Corporation© Copyright 2010 EMC Corporation. All rights reserved. 1
  2. 2. Roadmap Information Disclaimer EMC makes no representation and undertakes no obligations with regard to product planning information, anticipated product characteristics, performance specifications, or anticipated release dates (collectively, “Roadmap Information”). Roadmap Information is provided by EMC as an accommodation to the recipient solely for purposes of discussion and without intending to be bound thereby. Roadmap information is EMC Restricted Confidential and is provided under the terms, conditions and restrictions defined in the EMC Non- Disclosure Agreement in place with your organization.© Copyright 2010 EMC Corporation. All rights reserved. 2
  3. 3. Agenda  The Journey to Convergence  Protocols & Standards Update  Solution Evolution  Conclusion and Summary© Copyright 2010 EMC Corporation. All rights reserved. 3
  4. 4. Rack Server Environment Today Ethernet Fibre Channel iSCSI SAN  Servers connect to LAN, NAS and iSCSI SAN with NICs 1 Gigabit Ethernet  Servers connect to FC SAN with HBAs  Many environments today are still 1 Gigabit Ethernet 1 Gigabit Ethernet  Multiple server adapters, multiple 1 Gigabit Fibre Ethernet LAN cables, power and cooling costs Channel Ethernet HBAs – Storage is a separate network NICs (including iSCSI) Fibre Channel SAN Note: NAS will continue to be part of the solution. Everywhere that you Ethernet30% of servers in the Today < see data centeror 10Gb attached toin this are SAN Ethernet Storage presentation, NAS can be considered storage Rack-mounted servers part of the unified storage solution© Copyright 2010 EMC Corporation. All rights reserved. 4
  5. 5. The iSCSI Story Transport storage (SCSI) over standard Ethernet Initiator Reliability through TCP SCSI SCSI has limited distance, iSCSI provides even more flexibility than FC due to IP routing iSCSI 1Gb iSCSI has good performance TCP iSCSI has thrived, especially where the server, storage and network administrators are the same person IP IPsec Link IP Network© Copyright 2010 EMC Corporation. All rights reserved. 5
  6. 6. Why a New Option for FC Customers? FC has a large and well managed install base – Want a solution that is attractive for customers with FC expertise / investment – Previous convergence options did not allow for incremental adoption Requirement for a Data Center solution that can provide I/O consolidation Leveraging Ethernet infrastructure and skill set has always been attractive FCoE allows an Ethernet-based SAN to be introduced into the FC-based Data Center without breaking existing administrative tools and workflows© Copyright 2010 EMC Corporation. All rights reserved. 6
  7. 7. Non-Ethernet Convergence Options InfiniBand – Used broadly for High Performance Computing (HPC) environments – Low cost and ultra-low latency geared for server to server cluster – Separate use from general network (Ethernet) or storage (FC or Ethernet) Appliance connects to  PCIe existing infrastructure – Extension of the server bus to an I/O (FC, Ethernet) aggregation box  Single network from server -> top of rack, similar to early FCoE deployments Extend server‟s – Not a standard (parallel effort to MR- PCI bus to IOV standard), small players appliance – Still using Ethernet and FC network and storage from the aggregation box© Copyright 2010 EMC Corporation. All rights reserved. 7
  8. 8. 10Gb Ethernet allows for Converged Data Center Maturation of 10 Gigabit Ethernet – 10 Gigabit Ethernet allows replacement of n x 1Gb with a much smaller number (start with 2) of 10Gb Adapters – Single network allows for easier mobility for virtualization/cloud deployments Single Wire for SAN 10 GbE Network and Storage LAN 10 Gigabit Ethernet simplifies server, network and storage infrastructure – Reduces the number of cables and server adapters – Lowers capital expenditures and administrative costs – Reduces server power and cooling costs – Blade servers and server virtualization drive consolidated bandwidth 10 Gigabit Ethernet is the answer! iSCSI and FCoE both leverage this inflection point© Copyright 2010 EMC Corporation. All rights reserved. 8
  9. 9. FCoE Extends FC on a Single NetworkServer sees storage traffic as FC Network FC Ethernet Driver Driver Network FCoE SW Stack Standard Converged 10G NIC Network Adapter FC storage 2 Lossless Ethernet Linksoptions FC network FCoE Switch Ethernet FC © Copyright 2010 EMC Corporation. All rights reserved. SAN sees host as FC 9
  10. 10. Time To Widespread Adoption 1980 1990 2000 2010 10 Gigabit Ethernet 02 09 Ethernet Standard Widespread 73 83 93 iSCSIDefined Standard Widespread 00 02 08 Defined Widespread Standard Fibre Channel 85 94 03 FCoE Defined Standard Widespread 07 09 ?? Defined Standard Widespread© Copyright 2010 EMC Corporation. All rights reserved. 10
  11. 11. Future 40/100 Gb Ethernet 16 Gb 32 Gb FC FC 40 & 100 Gb Ethernet (IEEE) standards will be completed in June 2010 16Gb FC (T11) standard is targeted for completion at the end of 2010 Server adoption of FC ~ 3+ years, of Ethernet ~ 5+ years – Backbone typically faster adoption© Copyright 2010 EMC Corporation. All rights reserved. 11
  12. 12. Ethernet Cabling Type / Cable 1Gb 10Gb 40/100Gb Connector Copper Cat6 or > 99% of Some products Not (10GBase-T) / Cat6a existing on market, but supported RJ-45 cabling not for FCoE yet in initial (lots of standard Cat 5e) Cat6 55m; Cat 6a 100m Optical OM2 Rare Most backbone Expect shift (multimode) / (orange) deployments to optical w/ LC OM3 < 1% are optical 40/100Gb (aqua) Ethernet, but OM2 82m OM3 100m OM4* Standard (aqua) for FC OM3 300m OM4 125m Copper / Twinax N/A Low power Different SFP+DA short- (direct 5-10m distance distance attach) (Rack solution) option (QSFP)© Copyright 2010 EMC Corporation. All rights reserved. 12
  13. 13. Agenda  The Journey to Convergence  Protocols & Standards Update  Solution Evolution  Conclusion and Summary© Copyright 2010 EMC Corporation. All rights reserved. 13
  14. 14. Standards for Next Generation Data Center Two parallel industry standards seek to drive I/O consolidation in large data centers over time: Fibre Channel over Ethernet  Converged Enhanced Ethernet (FCoE) protocol (CEE) – Developed by International – Developed by IEEE Data Center Committee for Information Bridging (DCB) Task Group Technology Standards (INCITS) – DCB/CEE creates an Ethernet T11 Fibre Channel Interfaces environment that drops frames as Technical Committee rarely as Fibre Channel – Fibre Channel over Ethernet – Technology commonly referred to allows native Fibre Channel to as Lossless Ethernet travel unaltered over Ethernet – IEEE standards targeting – FC-BB-5 standard ratified in June ratification in mid 2010 2009 – Requirement for FCoE; – FC-BB-6 in process to expand Enhancement for iSCSI solution Companies working on the standard committeesKey participants: Brocade, Cisco, EMC, Emulex, HP, IBM, Intel, QLogic, Oracle(Sun), others© Copyright 2010 EMC Corporation. All rights reserved. 14
  15. 15. iSCSI and FCoE Framing iSCSI is SCSI functionality transported using TCP/IP for delivery and routing in a standard Ethernet/IP environment Ethernet TCP iSCSI Data CRCiSCSI Frame IP Header FCoE is FC frames encapsulated in Layer 2 Ethernet frames designed to utilize a Lossless Ethernet environment – Large maximum size of FC requires Ethernet Jumbo Frames – No TCP, so Lossless environment required – No IP routing Ethernet Header Header Header FCoE CRC EOF FCS FC FCoE Frame FC Payload© Copyright 2010 EMC Corporation. All rights reserved. FC Frame 15
  16. 16. FCoE Frame Formats FCoE Frame Format  Ethernet frames give a 1:1Bit 0 Bit 31 encapsulation of FC frames Destination MAC Address – No segmenting FC frames across Source MAC Address multiple Ethernet frames – FCoE flow control is Ethernet based IEEE 802.1Q Tag  BB Credit/R_RDY replaced by ET = FCoE Ver Reserved Pause/PFC mechanism Reserved Reserved  FC frames are large, require Jumbo Reserved SOF frames – Max FC payload size is 2180 bytes Encapsulated FC Frame (Including FC-CRC) – Max FCoE frame size is 2240 bytes EOF Reserved  FCoE Initialization Protocol (FIP) FCS used for discovery and login © Copyright 2010 EMC Corporation. All rights reserved. 16
  17. 17. Storage Drivers and Server Virtualization vNIC vSCSI vNIC vSCSI virtual switch Hypervisor Hypervisor driver NIC C FC NIC C FC N HBA N HBA A A LAN traffic LAN traffic iSCSI trafficFCoE follows FC iSCSI traffic FC traffic path *iSCSI initiator can also be in the VM© Copyright 2010 EMC Corporation. All rights reserved. 17
  18. 18. Storage Drivers and Server Virtualization SW SW FCoE FCoE vNIC vSCSI vNIC vSCSI vSwitches from ESX (including virtual switch Hypervisor HypervisorCisco 1000v option) driverand Hyper-V are not No FCoE access Lossless here currently NIC FC NIC FC HBA HBA FCoE software in the guest would send traffic through the vSwitch to the vNIC © Copyright 2010 EMC Corporation. All rights reserved. 18
  19. 19. FC-BB-6 Not required for multi-hop FCoE or other current deployments Currently in the “herding cats” phase of defining goals Likely to support point-to-point configuration which allows 2 FCoE devices to communicate without going through an FCF (or switch) For more, see Erik Smith of EMC E-Lab‟s presentation FCoE - Topologies, Protocol, and Limitations Tues 8am and Thurs 8:30am© Copyright 2010 EMC Corporation. All rights reserved. 19
  20. 20. Lossless Ethernet IEEE 802.1 Data Center Bridging (DCB) is the standards task group Converged Enhanced Ethernet (CEE) is an industry consensus term Link level enhancements (Priority Flow Control, Enhanced Transmission Selection, Data Center Bridging Exchange Protocol) are shipping in products today – Standards expected to be ratified in June „10 The “CEE cloud” or DCB-enabled LAN is only for the portion of your network that requires lossless functionality – Currently limited to multimode (300m) distances per link (no singlemode) – Limit the environment to the Data Center; Layer 2 (no routing) Enhanced Ethernet provides the Lossless Infrastructure which enables FCoE© Copyright 2010 EMC Corporation. All rights reserved. 20
  21. 21. PAUSE and Priority Flow Control PAUSE transforms Ethernet into a lossless fabric Classical 802.3x PAUSE is rarely implemented since it stops all traffic A new PAUSE known as Priority Flow Control (PFC) function that can halt traffic according to priority tag while allowing traffic at other priority levels to continue – Creates lossless virtual lanes – PFC will be limited to Data Center Per priority link level flow control – Only affect traffic that needs it – Ability to enable it per priority – Not simply 8 x 802.3x PAUSE Switch A Switch B© Copyright 2010 EMC Corporation. All rights reserved. 21
  22. 22. Enhanced Transmission Selection and Data Center Bridging Exchange Protocol (DCBX)Enhanced Transmission Selection (ETS) provides a commonmanagement framework for bandwidth management Allows configuration Offered Traffic 10 GE Link Realized Traffic Utilization of HPC & storage traffic to have 3G/s HPC Traffic 2G/s appropriately higher 3G/s 3G/s 2G/s 3G/s priority When a given load in 3G/s 3G/s a class does not fully Storage Traffic 3G/s 3G/s 3G/s 3G/s utilize its allocated bandwidth, ETS allows other traffic classes to use the available 3G/s 4G/s 6G/s 3G/s LAN Traffic 5G/s bandwidth 4G/s Maintain low latency treatment of certain t1 t2 t3 t1 t2 t3 traffic classesData Center Bridging Exchange Protocol (DCBX) is responsible for configuration of link parameters for DCB functions Determines which devices support Enhanced Ethernet functions © Copyright 2010 EMC Corporation. All rights reserved. 22
  23. 23. Beyond Link Level Switch  Congestion notification – IEEE 802.1Qau ratified Transmit Queue Switch  Allows a switch to notify attached ports to slow down transmission due to heavy traffic, in order to reduce the chances of packet drops or Receive Buffer network deadlocks  Moves the management of congestion back to the edge, which helps alleviate network-wide bottlenecks  Layer 2 multipathing – IETF TRILL - TRansparent Interconnection of Lots of Links  Used with the Spanning Tree Protocol to provide more efficient bridging and bandwidth aggregation  Focuses on a bridging capability that will increase bandwidth by allowing and aggregating multiple network paths  Standards are stable; products are coming soon© Copyright 2010 EMC Corporation. All rights reserved. 23
  24. 24. Agenda  The Journey to Convergence  Protocols & Standards Update  Solution Evolution  Conclusion and Summary© Copyright 2010 EMC Corporation. All rights reserved. 24
  25. 25. iSCSI Deployment  iSCSI was > 15% of revenue ($1.8B in „09) and > 20% capacity in SAN market in 2009 *  10 Gb iSCSI solutions are available – Can work on both Traditional Ethernet (recover from dropped packets using TCP) or Lossless Ethernet (DCB) environment  iSCSI natively routable (IP) Ethernet  iSCSI solutions are much smaller scale than iSCSI SAN FC – A single FC director is larger than most iSCSI environments * According to IDC, 2009© Copyright 2010 EMC Corporation. All rights reserved. 25
  26. 26. FCoE Solutions in 2009 FCoE with direct attach of server to Converged Network Switch at top of rack or end of row Tightly controlled solution Ethernet LAN Server 10 GE adapters may be CNA or NIC Storage is still a separate network Converged Network Switch FC Attach Fibre Channel SAN 10 GbE CNAs Storage Ethernet Rack Mounted FC Servers© Copyright 2010 EMC Corporation. All rights reserved. 26
  27. 27. Expansion of FCoE beyond a single switch First solutions are with an FCoE storage aware Ethernet switch (also known as FIP snooping) Enables rack bundle solutions FC switch FCoE aware CNA Ethernet switch FCoE switch (embedded) (embedded) FCoE enabled Blade Server Rack Area Network (RAN) allow for the rack as a unit of design and a unit of management© Copyright 2010 EMC Corporation. All rights reserved. 27
  28. 28. Cisco & VCE FCoE Rack Area Network (RAN) CX4-480 or 2 x Nexus 6140  Cisco UCS Celerra NS-960 •FCoE, Ethernet, FC ports – Optimized compute environment utilizing FCoE technology 8 x UCS 5100 – Includes embedded FCoE adapter and ToR switch chassis•Includes CNA blade (2 x FCoE) equivalent, plus the option for embedded FCoE switch  Vblock vSphere/ESX 4.0 – Integrated Virtual Computing •16:1 consolidation •1024 VMs Environment combining best-in-class networking, compute, storage, security and management solutions – Cisco UCS + EMC storage + Vblock1 example in VMware vSphere/ESX 3 floor tiles © Copyright 2010 EMC Corporation. All rights reserved. 28
  29. 29. IBM & HP FCoE offerings TOR switch splits LAN & SAN traffic Converged edge will evolve from ToR/EoR to aggregation and core Blade Switch layers 2 x 10Gb Switch Top of Rack 10-port 10Gb Switch or 14-port IBM 10Gb 1x CNA Card Core Pass-Thru Aggregation 2-port End of Row 10Gb CNA IBM BladeServers w/ embedded FCoE adapter for pass-thru switch module or an embedded CNAs Converge traffic at the server FCoE switch module which attachesEMC Corporation. All rights reserved. © Copyright 2010 to ToR switch 29
  30. 30. Challenges for FCoE FCoE solution development will take time to expand and mature, just as with other technologies – customers looking to create topologies similar to existing FC configurations: – Director support – Edge-core support (multi-hop) Organizational domain overlap Storage Network© Copyright 2010 EMC Corporation. All rights reserved. 30
  31. 31. EMC and Ethernet  Best Practices – Google “FCoE Tech Book” (FCoE & Ethernet)  Services – Design, Implementation, Performance and Security offerings for networks  Products – Ethernet equipment for creating Converged Network Environments© Copyright 2010 EMC Corporation. All rights reserved. 31
  32. 32. FCoE Timeline Supported 2010 Future FCoE top-of-rack Native FCoE storage 10Gb DCB LOMs switches FCoE blades for FC and Ethernet 10G-BaseT w/ FCoE 2nd Generation CNAs directors TRILL solutions Windows, Linux, UNIX support VMware Direct Connect (FC-BB-6) Open FCoE Cisco UCS and Vblock 40GbE/100GbE Expanded multi-hop solutions IBM BladeCenter More embedded/server solutions© Copyright 2010 EMC Corporation. All rights reserved. 32
  33. 33. Agenda  The Journey to Convergence  Protocols & Standards Update  Solution Evolution  Conclusion and Summary© Copyright 2010 EMC Corporation. All rights reserved. 33
  34. 34. Next Generation Data Center 10 Gigabit  EMC is working with the standards Ethernet Fibre communities and Channel partners to deliver the same reliability and robustness in the next generation common management virtual data center common infrastructure that we deliver today virtualization Private Cloud The Converged Data Center sets theoperational and capital efficiency foundations Cloudfor the virtual data center and private clouds Virtualized Data Center Computing© Copyright 2010 EMC Corporation. All rights reserved. 34
  35. 35. Summary A converged data center environment can be built using 10Gb Ethernet Achieving a converged network requires consideration of technology, processes/best practices and organizational dynamics 10 Gigabit Ethernet solutions are maturing – Active industry participation is creating standards that allow solutions that can integrate into existing data centers – Continued use of FC and adoption of FCoE can be flexible due to shared management – FCoE and iSCSI will follow the Ethernet roadmap to 40 and 100 Gigabit in the future© Copyright 2010 EMC Corporation. All rights reserved. 35
  36. 36. Related References Full collection of FCoE references (blog, video, whitepaper, presentations and other EMC links) http://blogstu.wordpress.com/tag/fcoe Industry site with consolidated information http://www.fcoe.com/ T11 FCoE activity http://www.t11.org/fcoe IEEE 802.1 Data Center Bridging task group page http://www.ieee802.org/1/pages/dcbridges.html Other EMC Bloggers covering these technologies: – Chad Sakac http://virtualgeek.typepad.com/ – Chuck Hollis http://chucksblog.typepad.com/ – David Graham http://flickerdown.com/© Copyright 2010 EMC Corporation. All rights reserved. 36

×