Converged Data Center: FCoE, iSCSI and the Future of Storage Networking

Converged Data
Center:

FCoE, iSCSI and the
Future of Storage Networking
David L. Black, Distinguished Engineer
Office of the CTO

© Copyright 2011 EMC Corporation. All rights reserved. 1

Agenda
•  Network Convergence
•  Protocols & Standards
•  Solution Evolution
•  Conclusion and
Summary


10Gb Ethernet Converged Data Center
•  Maturation of 10 Gigabit Ethernet
–  Replace multiple 1Gb adapters with fewer 10Gb adapters (start with 2)
–  Single network simplifies mobility for virtualization/cloud deployments

Single Wire for Network SAN
10 GbE and Storage
LAN

•  10 Gigabit Ethernet simplifies infrastructure
–  Reduces number of cables and server adapters
–  Lowers capital expenditures and administrative costs
–  Reduces server power and cooling costs
–  Blade servers and server virtualization drive consolidated bandwidth

FCoE and iSCSI both leverage this inflection point


Conventional Rack Servers
Ethernet
Fibre Channel iSCSI SAN •  Servers connect to LAN, NAS and
iSCSI SAN with NICs
1 Gigabit Ethernet •  Servers connect to FC SAN with
HBAs
•  Many environments today are still
1 Gigabit Ethernet 1 Gigabit Ethernet
•  Multiple server adapters, multiple
cables, power and cooling costs
Fibre Ethernet LAN
1 Gigabit –  Storage is a separate network (including
Channel
Ethernet
HBAs iSCSI)
NICs

Note: NAS will continue to be part of the
Fibre Channel SAN solution. Everywhere that you see
Ethernet or 10Gb Ethernet in this
presentation, NAS can be considered
part of the unified storage solution
Storage
Rack-mounted
servers


FCoE: Why a New Option for FC
Customers?
•  FC: large and well managed installed base
–  Leverage FC expertise / investment
–  Other convergence options not incremental for existing FC

•  Data Center solution for I/O consolidation
•  Leverage Ethernet infrastructure and skill set

FCoE allows an Ethernet-based SAN to be introduced
into an FC-based Data Center
without breaking existing administrative tools and workflows


FCoE Extends FC on a Single Network
Server sees storage traffic as FC

Network FC Ethernet
Driver Driver Network

Converged FC storage
Network Adapter
SAN sees host as FC
Lossless Ethernet
FC network

Ethernet FCoE
FC Switch


Time to Widespread Adoption
1980 1990 2000 2010
10 Gigabit Ethernet
02 09
Ethernet Standard Widespread

73 83 93 iSCSI
Defined Standard Widespread 00 04 08
Defined Widespread
Standard

Fibre Channel
85 94 03 FCoE
Defined Standard Widespread
07 09 ??
Defined
Standard
Widespread


Further Developments: What’s Next?

40/100 Gb
Ethernet

16 32
GFC GFC

•  40 & 100 Gb Ethernet (IEEE) standards completed in June 2010
•  16GFC (T11) standard completed in January 2011, 32GFC is next (2012
target)
–  FC throughput doubles, encoding change optimizes analog bandwidth
•  8GFC: 800 MB/sec max, 8.5 Ghz
•  16GFC: 1600 MB/sec max, 14.025 Ghz


Agenda

•  Conclusion and Summary


iSCSI Introduction
•  Transport storage (SCSI) over standard Ethernet
–  Reliability through TCP

•  More flexible than FC due to IP routing
SCSI
•  Good performance
iSCSI
•  iSCSI has thrived
–  Especially where the server, storage and network TCP
administrators are the same person
IP

Link

IP Network


iSCSI Introduction (continued)
•  Standardized in 2004: IETF RFC 3720
–  Stable: No major changes since 2004
–  iSCSI Corrections and Clarifications: IETF RFC 5048 (2007)

•  iSCSI Session: One Initiator and one Target
–  Multiple TCP connections allowed in a session

•  Important iSCSI additions to SCSI
–  Immediate and unsolicited data to avoid round trip
–  Login phase for connection setup
–  Explicit logout for clean teardown


iSCSI Read Example
Initiator Target
SCSI Read
Data in PDU
Command

Target
Data in PDU

Receive Data in PDU
Data

Status
Command
Complete
Optimization: Good
status can be
included with last
“Data in” PDU


iSCSI Write Example
Initiator Target

Ready to
SCSI Write
Transmit
Command
(R2T)
Optimization:
Data out PDU Immediate or
Receive
Data unsolicited data
Data out PDU avoids a round trip

Data out PDU R2T

Data out PDU

Receive
Data

Command
Status
Complete


iSCSI and FCoE Framing
•  iSCSI is SCSI functionality transported using TCP/IP for delivery and
routing in a standard Ethernet/IP environment

Ethernet Data
iSCSI Frame IP TCP iSCSI CRC
Header

Ÿ  FCoE is FC frames encapsulated in Layer 2 Ethernet frames
–  No TCP, so Lossless Ethernet required
FC Frame
–  No IP routing Header

FCoE Frame FC Payload
Ethernet

Header
Header

CRC
FCoE

EOF
FC

FCS

FCoE Frame Format
Bit 0 Bit 31
Destination MAC Address

•  1:1 encapsulation of FC frames
Source MAC Address
–  No segmenting of FC frames across
IEEE 802.1Q Tag
multiple Ethernet frames
–  FCoE flow control is Ethernet based
ET = FCoE Ver Reserved

Reserved •  FCoE requires Mini Jumbo
Reserved frames
Reserved SOF –  Max FC payload size: 2180 bytes
–  Max FCoE frame size: 2240 bytes
Encapsulated FC Frame
(Including FC-CRC) •  FCoE Initialization Protocol (FIP)
–  Discovery: VLAN and FCoE
EOF Reserved
switches
FCS
–  FC login to discovered FCoE
switches


FCoE Initialization Protocol
Ethernet is more than a cable
•  Native Fiber Channel link: Fiber has exactly 2 endpoints (simple)
–  Discovery: Who is at the other end?
•  Response to FC login contains answer
–  Liveness: Is the other end still there?
•  Is the optical link lit and synchronized?

•  FCoE virtual link: Ethernet LAN or VLAN has more than 2 endpoints
–  Discovery: Choice of endpoints (FCoE switches)
•  Where should the FC login be sent?
–  Liveness: FCoE virtual link may span multiple Ethernet links
•  What if attached link is ok, but some other link is not?

•  FCoE configuration concern: Do mini jumbo frames (2.5k) work?
•  FIP: FCoE Initialization Protocol
–  Discover other endpoint, create and initialize virtual link with FCoE switch
–  Mini jumbo frame support: Large frame is part of discovery
–  Periodic LKA (Link Keep Alive) messages after initialization


FCoE Switch Discovery FCoE/FC
Step 1: FIP Solicitation Switches

Server
DCB FC SAN
Ethernet

Solicitation

Ÿ  Select FCoE VLAN first (pre-configured or use FIP)
Ÿ  Solicitation is multicast: Server can discover multiple switches
Ÿ  Solicitation identifies Server (FC WWN for FCoE CNA)
–  CNA = Converged Network Adapter (FCoE analog of HBA)
–  Switch may choose which servers to respond to (default: respond to all)


FCoE Switch Discovery FCoE/FC
Step 2: FIP Advertisement Switches

Server
DCB FC SAN
Ethernet
Advertisement
Priority =
1
Advertisement

Priority = 25
Ÿ  Advertisement identifies switch (FC WWN)
–  Multiple switches may respond, advertisement includes priority
–  Server chooses FCoE switch by priority (smallest number wins)
Ÿ  Advertisement padded to max FC frame size: Test mini jumbo frame support


FIP Switch Discovery FCoE/FC
Step 3: FIP-based FC Login Switches

Server
DCB FC SAN
Ethernet

FLOGI
Priority =
FLOGI 1
ACC

Priority = 25
Ÿ  FIP encapsulated FC Login
–  Server sends FC Fabric Login (FLOGI) to selected switch
–  Switch responds with FC FLOGI ACC (accept) that contains assigned FCID
Ÿ  Subsequent traffic: Standard FC frames (FCoE encapsulated)


FCoE and Ethernet Standards
Two complementary standards efforts
Fiber Channel over Ethernet Data Center Bridging Ethernet
(FCoE) (DCB)
–  Developed by International –  Developed by IEEE Data Center
Committee for Information Bridging (DCB) Task Group
Technology Standards (INCITS) –  DCB Ethernet drops frames as
T11 Fiber Channel Interfaces rarely as Fiber Channel
Technical Committee –  Technology commonly referred to as
–  Fiber Channel over Ethernet allows Lossless Ethernet
Fiber Channel traffic over Ethernet –  IEEE standards: final approval
–  FC-BB-5 standard ratified June March 2011
2009 –  DCB: Required for FCoE
–  FC-BB-6 in process to expand –  DCB: Enhancement for iSCSI
solution
Companies working on the standard committees
Key participants: Brocade, Cisco, EMC, Emulex, HP, IBM, Intel, QLogic, Oracle(Sun), others


FC-BB-6 – Major features
•  Direct connection of servers to storage
–  PT2PT [point to point]: Single cable
–  VN2VN [VN_Port to VN_Port]: Dedicated Ethernet LAN or VLAN

•  Better support for FC fabric scaling
–  Distribute logical FC fabric switch functionality
–  Enables every DCB Ethernet switch to participate in FCoE

For more, see Erik Smith’s (EMC E-Lab) presentation:
FCoE - Topologies, Protocol, and Limitations
Tues 5:00pm and Wed 4:15pm


Lossless Ethernet (DCB)
•  IEEE 802.1 Data Center Bridging (DCB)
•  Link enhancements: standardized, initial products available
1.  Enhanced Transmission Selection (ETS)
2.  Priority Flow Control (PFC)
3.  Data Center Bridging Exchange Protocol (DCBX)

•  DCB-enabled LAN: Network portion that must be lossless
–  Generally limited to data center distances per link
–  Can use long-distance optics, but uncommon in practice

Enhanced Ethernet provides the Lossless Infrastructure
that enables FCoE


Enhanced Transmission Selection
DCB part 1: IEEE 802.1Qaz
Management framework for link bandwidth

•  Priority configuration and bandwidth reservation
–  E.g., HPC & storage traffic: higher priority & reserved bandwidth

•  Bandwidth utilization Offered Traffic 10 GE Link Realized Traffic Utilization
–  Unused higher priority 3G/s HPC Traffic 2G/s
3G/s 3G/s 2G/s
bandwidth available 3G/s

to other traffic
3G/s Storage Traffic 3G/s
3G/s 3G/s 3G/s 3G/s
•  Low latency assured to
higher priority traffic
3G/s 4G/s 6G/s 3G/s LAN Traffic 5G/s
4G/s

t1 t2 t3 t1 t2 t3


PAUSE and Priority Flow Control
DCB part 2: IEEE 802.1Qbb & 802.3bd
•  Classic PAUSE can produce lossless Ethernet behavior
–  Classic 802.3x PAUSE stops all traffic: Rarely implemented
•  New PAUSE: Priority Flow Control (PFC)
–  Pause per priority level
–  No effect on traffic at other priority levels
–  Creates lossless virtual lanes
•  Per-priority link flow control
–  Enabled/disabled by priority
•  Only affect traffic that needs it
–  More than 8-way 802.3x PAUSE

Switch A Switch B


DCBX ─ Data Center Bridging Capability
eXchange
FCoE/FC
DCB part 3: IEEE 802.1Qaz (again)
Switches

Server
DCB FC SAN
Ethernet

DCB
X

•  Ethernet Link configuration (single link)
–  Extends Link Layer Discovery Protocol (LLDP)
•  Reliably enables lossless behavior (DCB)
–  e.g., exchange Ethernet priority values for FCoE and FIP
•  FCoE virtual links should not be instantiated without DCBX


Ethernet Spanning Trees and FCoE
•  Reminder: FCoE is Ethernet only, no IP routing
–  Ethernet (layer 2) is bridged, not routed
•  Spanning Tree Protocol (STP): Prevents (deadly) forwarding loops
–  Elects a Root Switch, disables redundant paths to create a tree
•  Causes problems in large Ethernet networks
–  No network multipathing
–  Inefficient link utilization
Root Switchè Si Si

Si Si Si Si Si


TRILL – Transparent Interconnection of
Lots of Links
•  Layer 2 routing among Ethernet switches
–  In contrast to IP routing at layer 3
–  IS-IS routing protocol for inter-switch Ethernet traffic
–  Blocks Spanning Tree Protocol

•  TRILL encapsulates Ethernet frames
–  Not used with end systems (NICs)
–  NICs use link teaming/aggregation
Si Si

All links active è

Si Si Si Si Si


Ethernet Cabling Choices
Type /
Connector Cable 1Gb 10Gb 40/100Gb
Copper Cat6 or Majority of Some products Not
(10GBase-T) / Cat6a existing on market, but supported
RJ-45 cabling not for FCoE
(e.g., Cat 5e) yet
Cat 6 55m
Cat 6a 100m

Optical OM2 Rare for Most Expect shift
(multimode) / (orange) Ethernet backbone to optical w/
LC OM3 deployments 40/100Gb
(aqua) Typical for are optical
FC OM3 100m
OM4 OM2 82m
(aqua) OM4 125m
OM3 300m

Copper / SFP Twinax N/A Low power Different
+DA (direct short-
attach) 5-10m distance
distance (Rack option
solution) (QSFP)


Virtual Machines and Storage Resources
Private Storage Resources
–  Accessed directly by VM
•  Device driver in VM’s OS
–  Managed as part of VM
•  Not visible to virtualization
Private management (e.g., vCenter)
–  If disk is local: No vMotion

Shared Storage Resources
–  Accessed by Hypervisor
•  Device driver in hypervisor
Shared –  Managed as part of
virtualization (e.g., vCenter)


Storage Drivers and Server Virtualization

vNIC vSCSI vNIC vSCSI

virtual switch Hypervisor Hypervisor
driver

NIC FC NIC FC
HBA HBA

LAN traffic
iSCSI traffic FC traffic
*iSCSI initiator can also be in the VM (Private
Storage)


Storage Drivers and Server Virtualization


virtual switch Hypervisor Hypervisor
driver

NIC C FC NIC C FC
N HBA N HBA
A A
LAN traffic iSCSI traffic FCoE follows FC
path
*iSCSI initiator can also be in the VM (Private
Storage)


Software FCoE and Server Virtualization

SW SW
FCoE FCoE

Virtual Switches in
ESX/ESXi
(including Cisco virtual switch Hypervisor Hypervisor
Nexus 1000v) and driver
Hyper-V are not
Lossless (not DCB)
Not a problem for
iSCSI, NFS or NIC FC NIC FC
HBA HBA
CIFS in a Virtual
Machine FCoE software in VMs would send traffic
through the virtual switch to the NICs


Storage Virtual Appliance (SVA):
Sharing Private Storage

Storage Virtual Appliance
–  Virtual Machine that provides
storage to hypervisors
–  Direct physical storage access
•  e.g., RDM (Raw Device
Mapping)
–  Exports storage to hypervisors
iSCSI •  ESX & ESXi: iSCSI (or NFS)
–  Avoid vMotion for SVAs

This SVA example: Simplified
–  Availability: Multiple SVAs
mirror or RAID across servers
–  Scale: SVAs provide shared
storage to vSphere server
cluster


Agenda

•  Conclusion and Summary


FCoE and iSCSI

FCoE iSCSI
Ethernet
No FC expertise needed
FC expertise / install base
Leverage
FC management Ethernet/IP expertise
Supports distance
Layer 2 Ethernet 10 Gigabit Ethernet (Layer 3 IP routing)
Use FCIP for distance Lossless Ethernet Strong virtualization affinity


iSCSI Deployment
•  10 Gb iSCSI solutions available
–  Traditional Ethernet
•  TCP recovers from dropped packets
–  Lossless Ethernet (DCB)

•  iSCSI: natively routable (IP)
–  Can use VLAN(s) to isolate traffic

Ethernet •  iSCSI solutions: smaller scale than
FC
iSCSI SAN
–  Single FC director : larger than most
iSCSI environments


FCoE Server Phase (1)
•  FCoE with Converged Network Switch
at top of rack or end of row
•  Tightly controlled solution
Ethernet LAN
•  Server 10 GE adapters may be CNA or
NIC
•  Storage still a separate network
Ethernet
FC
Converged Network
Switch

FC
Attach Fiber Channel SAN

1 Gb NICs FC
10 GbE CNAs HBAs
Storage
Rack
Mounted
Servers


FCoE Network Phase (2)
Ÿ  Converged Network Switches move out of
rack into unified network
Ÿ  Maintains existing LAN and SAN management Ethernet LAN

Overlapping admin domains may compel cultural adjustments

Ethernet Network
Ethernet (IP, FCoE) Converged Network
Switch
FC
Converged Network
Switch

FC
Attach Fiber Channel SAN

10 GbE CNAs
Storage
Rack
Mounted
Servers


FCoE Storage Phase (3)
•  Single Ethernet network for IP and storage traffic
•  End-to-End Ethernet with native FCoE
•  FC/FCoE configured and managed as an FC SAN Ethernet LAN
–  Leverage FC management skills and procedures

Converged Network
Ethernet Switch
FCoE
FC Storage

Fiber Channel
& FCoE FC & FCoE SAN
attach

10 GbE CNAs
Storage
Rack
Mounted
Servers


Convergence at 10 Gigabit Ethernet
•  Two paths to a Converged Network
–  iSCSI purely Ethernet
–  FCoE enables mix of FC and Ethernet (or all
Ethernet) Ethernet LAN
•  FC that you have today or buy tomorrow is compatible

•  Choose based on scalability,
management, Converged Network
and skill set
Ethernet Switch

FC iSCSI/FCoE
Storage

Fiber Channel
& FCoE
attach

10 GbE CNAs
FC & FCoE SAN

Rack
Mounted
Servers


EMC and Ethernet
•  Best Practices
–  Google “FCoE Tech Book”
(FCoE & Ethernet)

•  Services
–  Design, Implementation,
Performance and Security offerings
for networks

•  Products
–  Ethernet equipment for creating
Converged Network Environments


Agenda

•  Conclusion and
Summary


Summary
•  Converged data center environments can be built using
10Gb Ethernet
•  Achieving a converged network requires consideration of
technology, processes/best practices and organizational
dynamics
•  10 Gigabit Ethernet solutions are maturing
–  Active industry participation is creating standards that allow
solutions that can integrate into existing data centers
–  Continued use of FC and adoption of FCoE can be flexible due to
shared management
–  FCoE and iSCSI will follow Ethernet roadmap to 40 and 100
Gigabits/sec


Related Session and Resources
•  FCoE - Topologies, Protocol, and Limitations
–  Tuesday 5:00p & Wednesday 4:15p
•  Birds of a Feather: The Future of Storage Networking
–  Wednesday 8:30a
•  Cisco - Building Cloud-Ready Storage with Cisco and EMC
–  Tuesday 10:00a
•  FCoE in the EMC Topology Guide
–  http://elabnavigator.emc.com

•  EMC FCoE Videos: Search for “FCoE” on YouTube
•  EMC FCoE Introduction whitepaper
–  http://www.emc.com/collateral/hardware/white-papers/h5916-intro-to-fcoe-wp.pdf

•  FCoE Blog by Erik Smith (E-Lab)
–  http://www.brasstacksblog.typepad.com


Q&A


THANK YOU


Converged Data Center: FCoE, iSCSI and the Future of Storage Networking

More Related Content

What's hot

Viewers also liked

Similar to Converged Data Center: FCoE, iSCSI and the Future of Storage Networking

More from EMC

Recently uploaded

Converged Data Center: FCoE, iSCSI and the Future of Storage Networking