Converged Data Center: FCoE, iSCSI, and the Future of Storage Networking ( EMCWorld 2010 )

Converged Data Center:
FCoE, iSCSI, and the
Future of Storage Networking

Stuart Miniman, Technologist, Office of the CTO
EMC Corporation

© Copyright 2010 EMC Corporation. All rights reserved. 1

Roadmap Information Disclaimer

 EMC makes no representation and undertakes no obligations with
regard to product planning information, anticipated product
characteristics, performance specifications, or anticipated release dates
(collectively, “Roadmap Information”).

 Roadmap Information is provided by EMC as an accommodation to the
recipient solely for purposes of discussion and without intending to be
bound thereby.

 Roadmap information is EMC Restricted Confidential and is provided
under the terms, conditions and restrictions defined in the EMC Non-
Disclosure Agreement in place with your organization.


Agenda

 The Journey to Convergence
 Protocols & Standards Update
 Solution Evolution
 Conclusion and Summary


Rack Server Environment Today

Ethernet
Fibre Channel iSCSI SAN  Servers connect to LAN, NAS and
iSCSI SAN with NICs
1 Gigabit Ethernet  Servers connect to FC SAN with
HBAs
 Many environments today are still
1 Gigabit Ethernet 1 Gigabit Ethernet
 Multiple server adapters, multiple
1 Gigabit Fibre Ethernet LAN cables, power and cooling costs
Channel
Ethernet
HBAs – Storage is a separate network
NICs
(including iSCSI)
Fibre Channel SAN Note: NAS will continue to be part
of the solution. Everywhere that you
Ethernet30% of servers in the
Today <
see data centeror 10Gb attached toin this
are SAN
Ethernet
Storage presentation, NAS can be considered
storage
Rack-mounted
servers
part of the unified storage solution

The iSCSI Story

 Transport storage (SCSI) over standard Ethernet Initiator
 Reliability through TCP
SCSI
 SCSI has limited distance, iSCSI provides even more
flexibility than FC due to IP routing iSCSI
 1Gb iSCSI has good performance TCP
 iSCSI has thrived, especially where the server, storage
and network administrators are the same person IP

IPsec

Link
IP Network


Why a New Option for FC Customers?

 FC has a large and well managed install base
– Want a solution that is attractive for customers with FC expertise /
investment
– Previous convergence options did not allow for incremental adoption
 Requirement for a Data Center solution that can provide I/O
consolidation
 Leveraging Ethernet infrastructure and skill set has always been
attractive

FCoE allows an Ethernet-based SAN to be introduced
into the FC-based Data Center
without breaking existing administrative tools and workflows


Non-Ethernet Convergence Options

 InfiniBand
– Used broadly for High Performance
Computing (HPC) environments
– Low cost and ultra-low latency geared for
server to server cluster
– Separate use from general network
(Ethernet) or storage (FC or Ethernet)
Appliance connects to  PCIe
existing infrastructure – Extension of the server bus to an I/O
(FC, Ethernet) aggregation box
 Single network from server -> top of
rack, similar to early FCoE deployments
Extend server‟s
– Not a standard (parallel effort to MR-
PCI bus to
IOV standard), small players
appliance
– Still using Ethernet and FC network and
storage from the aggregation box


10Gb Ethernet allows for Converged Data Center

 Maturation of 10 Gigabit Ethernet
– 10 Gigabit Ethernet allows replacement of n x 1Gb with a much smaller
number (start with 2) of 10Gb Adapters
– Single network allows for easier mobility for virtualization/cloud deployments

Single Wire for SAN
10 GbE
Network and Storage
LAN

 10 Gigabit Ethernet simplifies server, network and storage infrastructure
– Reduces the number of cables and server adapters
– Lowers capital expenditures and administrative costs
– Reduces server power and cooling costs
– Blade servers and server virtualization drive consolidated bandwidth

10 Gigabit Ethernet is the answer!
iSCSI and FCoE both leverage this inflection point

FCoE Extends FC on a Single Network

Server sees storage traffic as FC

Network FC Ethernet
Driver Driver Network

FCoE SW Stack

Standard Converged
10G NIC Network Adapter FC storage

2 Lossless Ethernet Links
options
FC network

FCoE
Switch
Ethernet
FC
© Copyright 2010 EMC Corporation. All rights reserved.
SAN sees host as FC 9

Time To Widespread Adoption

1980 1990 2000 2010
10 Gigabit Ethernet
02 09
Ethernet Standard Widespread

73 83 93 iSCSI
Defined Standard Widespread 00 02 08
Defined Widespread
Standard

Fibre Channel
85 94 03 FCoE
Defined Standard Widespread
07 09 ??
Defined
Standard
Widespread


Future

40/100 Gb
Ethernet

16 Gb 32 Gb
FC FC

 40 & 100 Gb Ethernet (IEEE) standards will be completed in June 2010
 16Gb FC (T11) standard is targeted for completion at the end of 2010
 Server adoption of FC ~ 3+ years, of Ethernet ~ 5+ years
– Backbone typically faster adoption


Ethernet Cabling

Type / Cable 1Gb 10Gb 40/100Gb
Connector
Copper Cat6 or > 99% of Some products Not
(10GBase-T) / Cat6a existing on market, but supported
RJ-45 cabling not for FCoE yet in initial
(lots of standard
Cat 5e) Cat6 55m; Cat
6a 100m
Optical OM2 Rare Most backbone Expect shift
(multimode) / (orange) deployments to optical w/
LC OM3 < 1% are optical 40/100Gb
(aqua) Ethernet,
but OM2 82m OM3 100m
OM4* Standard
(aqua) for FC OM3 300m OM4 125m

Copper / Twinax N/A Low power Different
SFP+DA short-
(direct 5-10m distance distance
attach) (Rack solution) option
(QSFP)

Agenda



Standards for Next Generation Data Center

Two parallel industry standards seek to drive I/O consolidation
in large data centers over time:
 Fibre Channel over Ethernet  Converged Enhanced Ethernet
(FCoE) protocol (CEE)
– Developed by International – Developed by IEEE Data Center
Committee for Information Bridging (DCB) Task Group
Technology Standards (INCITS) – DCB/CEE creates an Ethernet
T11 Fibre Channel Interfaces environment that drops frames as
Technical Committee rarely as Fibre Channel
– Fibre Channel over Ethernet – Technology commonly referred to
allows native Fibre Channel to as Lossless Ethernet
travel unaltered over Ethernet
– IEEE standards targeting
– FC-BB-5 standard ratified in June ratification in mid 2010
2009
– Requirement for FCoE;
– FC-BB-6 in process to expand Enhancement for iSCSI
solution
Companies working on the standard committees
Key participants: Brocade, Cisco, EMC, Emulex, HP, IBM, Intel, QLogic, Oracle(Sun), others

iSCSI and FCoE Framing

 iSCSI is SCSI functionality transported using TCP/IP for delivery and
routing in a standard Ethernet/IP environment

Ethernet TCP iSCSI Data CRC
iSCSI Frame IP
Header

 FCoE is FC frames encapsulated in Layer 2 Ethernet frames designed to
utilize a Lossless Ethernet environment
– Large maximum size of FC requires Ethernet Jumbo Frames
– No TCP, so Lossless environment required
– No IP routing
Ethernet
Header

Header

Header
FCoE

CRC
EOF
FCS
FC

FCoE Frame FC Payload

© Copyright 2010 EMC Corporation. All rights reserved. FC Frame 15

FCoE Frame Formats

FCoE Frame Format
 Ethernet frames give a 1:1
Bit 0 Bit 31
encapsulation of FC frames
Destination MAC Address
– No segmenting FC frames across
Source MAC Address
multiple Ethernet frames
– FCoE flow control is Ethernet based
IEEE 802.1Q Tag
 BB Credit/R_RDY replaced by
ET = FCoE Ver Reserved
Pause/PFC mechanism
Reserved

Reserved  FC frames are large, require Jumbo
Reserved SOF frames
– Max FC payload size is 2180 bytes
Encapsulated FC Frame
(Including FC-CRC) – Max FCoE frame size is 2240 bytes
EOF Reserved  FCoE Initialization Protocol (FIP)
FCS used for discovery and login


Storage Drivers and Server Virtualization

vNIC vSCSI vNIC vSCSI

virtual switch Hypervisor Hypervisor
driver

NIC C FC NIC C FC
N HBA N HBA
A A
LAN traffic
LAN traffic iSCSI trafficFCoE follows FC
iSCSI traffic
FC traffic
path

*iSCSI initiator can also be in the VM

Storage Drivers and Server Virtualization

SW SW
FCoE FCoE
vNIC vSCSI vNIC vSCSI

vSwitches from
ESX (including virtual switch Hypervisor Hypervisor
Cisco 1000v option) driver
and Hyper-V are not No FCoE access
Lossless here currently
NIC FC NIC FC
HBA HBA

FCoE software in the guest would send traffic through the vSwitch to the vNIC


FC-BB-6

 Not required for multi-hop FCoE or other current deployments
 Currently in the “herding cats” phase of defining goals
 Likely to support point-to-point configuration which allows 2 FCoE
devices to communicate without going through an FCF (or switch)

For more, see Erik Smith of EMC E-Lab‟s presentation
FCoE - Topologies, Protocol, and Limitations
Tues 8am and Thurs 8:30am


Lossless Ethernet

 IEEE 802.1 Data Center Bridging (DCB) is the standards task group
 Converged Enhanced Ethernet (CEE) is an industry consensus term
 Link level enhancements (Priority Flow Control, Enhanced Transmission
Selection, Data Center Bridging Exchange Protocol) are shipping in
products today
– Standards expected to be ratified in June „10
 The “CEE cloud” or DCB-enabled LAN is only for the portion of your
network that requires lossless functionality
– Currently limited to multimode (300m) distances per link (no singlemode)
– Limit the environment to the Data Center; Layer 2 (no routing)

Enhanced Ethernet provides the Lossless Infrastructure
which enables FCoE

PAUSE and Priority Flow Control

 PAUSE transforms Ethernet into a lossless fabric
 Classical 802.3x PAUSE is rarely implemented since it stops all traffic
 A new PAUSE known as Priority Flow Control (PFC) function that can
halt traffic according to priority tag while allowing traffic at other priority
levels to continue
– Creates lossless virtual lanes
– PFC will be limited to Data Center
 Per priority link level flow control
– Only affect traffic that needs it
– Ability to enable it per priority
– Not simply 8 x 802.3x PAUSE

Switch A Switch B


Enhanced Transmission Selection and Data
Center Bridging Exchange Protocol (DCBX)

Enhanced Transmission Selection (ETS) provides a common
management framework for bandwidth management
 Allows configuration Offered Traffic 10 GE Link Realized Traffic Utilization
of HPC & storage
traffic to have
3G/s HPC Traffic 2G/s
appropriately higher 3G/s 3G/s 2G/s
3G/s
priority
 When a given load in 3G/s 3G/s
a class does not fully Storage Traffic
3G/s 3G/s 3G/s 3G/s
utilize its allocated
bandwidth, ETS allows
other traffic classes to
use the available 3G/s 4G/s 6G/s 3G/s LAN Traffic 5G/s
bandwidth 4G/s
 Maintain low latency
treatment of certain t1 t2 t3 t1 t2 t3
traffic classes
Data Center Bridging Exchange Protocol (DCBX) is responsible for
configuration of link parameters for DCB functions
 Determines which devices support Enhanced Ethernet functions

Beyond Link Level

Switch  Congestion notification
– IEEE 802.1Qau ratified
Transmit Queue
Switch
 Allows a switch to notify attached ports to slow
down transmission due to heavy traffic, in order
to reduce the chances of packet drops or
Receive Buffer network deadlocks
 Moves the management of congestion back to
the edge, which helps alleviate network-wide
bottlenecks
 Layer 2 multipathing
– IETF TRILL - TRansparent Interconnection of
Lots of Links
 Used with the Spanning Tree Protocol to
provide more efficient bridging and bandwidth
aggregation
 Focuses on a bridging capability that will
increase bandwidth by allowing and aggregating
multiple network paths
 Standards are stable; products are coming soon


Agenda



iSCSI Deployment

 iSCSI was > 15% of revenue ($1.8B in „09)
and > 20% capacity in SAN market in 2009 *
 10 Gb iSCSI solutions are available
– Can work on both Traditional Ethernet (recover
from dropped packets using TCP) or Lossless
Ethernet (DCB) environment

 iSCSI natively routable (IP)
Ethernet  iSCSI solutions are much smaller scale than
iSCSI SAN FC
– A single FC director is larger than most iSCSI
environments

* According to IDC, 2009

FCoE Solutions in 2009

 FCoE with direct attach of server to Converged Network
Switch at top of rack or end of row
 Tightly controlled solution Ethernet LAN

 Server 10 GE adapters may be CNA or NIC
 Storage is still a separate network

Converged Network
Switch
FC Attach Fibre Channel SAN

10 GbE CNAs
Storage

Ethernet Rack Mounted
FC Servers


Expansion of FCoE beyond a single switch

 First solutions are with an FCoE storage
aware Ethernet switch (also known
as FIP snooping)
 Enables rack bundle solutions
FC switch

FCoE aware
CNA Ethernet switch FCoE switch
(embedded) (embedded)

FCoE enabled Blade Server

Rack Area Network (RAN) allow for the rack as a
unit of design and a unit of management

Cisco & VCE FCoE Rack Area Network (RAN)

CX4-480 or 2 x Nexus 6140  Cisco UCS
Celerra NS-960 •FCoE, Ethernet, FC ports – Optimized compute
environment utilizing FCoE
technology
8 x UCS 5100 – Includes embedded FCoE
adapter and ToR switch
chassis
•Includes CNA blade (2 x FCoE) equivalent, plus the option
for embedded FCoE switch
 Vblock
vSphere/ESX 4.0 – Integrated Virtual Computing
•16:1 consolidation
•1024 VMs
Environment combining
best-in-class networking,
compute, storage, security
and management solutions
– Cisco UCS + EMC storage +
Vblock1 example in VMware vSphere/ESX
3 floor tiles


IBM & HP FCoE offerings

TOR switch splits LAN & SAN traffic Converged edge will evolve from
ToR/EoR to aggregation and core
Blade Switch
layers

2 x 10Gb Switch
Top of Rack

10-port 10Gb Switch
or 14-port IBM 10Gb
1x CNA Card Core
Pass-Thru
Aggregation

2-port End of Row
10Gb CNA

 IBM BladeServers w/ embedded
FCoE adapter for pass-thru
switch module or an embedded
CNAs Converge traffic at the server
FCoE switch module which
attachesEMC Corporation. All rights reserved.
© Copyright 2010 to ToR switch 29

Challenges for FCoE

 FCoE solution development will take time to expand and mature, just as
with other technologies – customers looking to create topologies similar
to existing FC configurations:
– Director support
– Edge-core support (multi-hop)
 Organizational domain overlap
Storage Network


EMC and Ethernet

 Best Practices
– Google “FCoE Tech Book” (FCoE & Ethernet)

 Services
– Design, Implementation, Performance and
Security offerings for networks

 Products
– Ethernet equipment for creating Converged
Network Environments


FCoE Timeline

Supported 2010 Future

FCoE top-of-rack Native FCoE storage 10Gb DCB LOMs
switches FCoE blades for FC and Ethernet 10G-BaseT w/ FCoE
2nd Generation CNAs directors TRILL solutions
Windows, Linux, UNIX support
VMware Direct Connect (FC-BB-6)
Open FCoE
Cisco UCS and Vblock 40GbE/100GbE
Expanded multi-hop solutions
IBM BladeCenter More embedded/server solutions


Agenda



Next Generation Data Center

10 Gigabit  EMC is working with
the standards
Ethernet Fibre communities and
Channel partners to deliver
the same reliability
and robustness in
the next generation
common management virtual data center
common infrastructure that we deliver today

virtualization Private
Cloud
The Converged Data Center sets the
operational and capital efficiency foundations
Cloud
for the virtual data center and private clouds Virtualized
Data Center Computing

Summary

 A converged data center environment can be built using 10Gb Ethernet
 Achieving a converged network requires consideration of technology,
processes/best practices and organizational dynamics
 10 Gigabit Ethernet solutions are maturing
– Active industry participation is creating standards that allow solutions that can
integrate into existing data centers
– Continued use of FC and adoption of FCoE can be flexible due to shared
management
– FCoE and iSCSI will follow the Ethernet roadmap to 40 and 100 Gigabit in the future


Related References

 Full collection of FCoE references (blog, video,
whitepaper, presentations and other EMC links)
http://blogstu.wordpress.com/tag/fcoe
 Industry site with consolidated information
http://www.fcoe.com/

 T11 FCoE activity http://www.t11.org/fcoe
 IEEE 802.1 Data Center Bridging task group page
http://www.ieee802.org/1/pages/dcbridges.html

 Other EMC Bloggers covering these technologies:
– Chad Sakac http://virtualgeek.typepad.com/
– Chuck Hollis http://chucksblog.typepad.com/
– David Graham http://flickerdown.com/


Converged Data Center: FCoE, iSCSI, and the Future of Storage Networking ( EMCWorld 2010 )

Converged Data Center: FCoE, iSCSI, and the Future of Storage Networking ( EMCWorld 2010 )

More Related Content

What's hot

Viewers also liked

Similar to Converged Data Center: FCoE, iSCSI, and the Future of Storage Networking ( EMCWorld 2010 )

More from EMC

Recently uploaded

Converged Data Center: FCoE, iSCSI, and the Future of Storage Networking ( EMCWorld 2010 )