PLNOG 13: Artur Pająk: Storage w sieciach Ethernet, czyli coś o iSCSI I FCoE

Storage w sieciach Ethernet, czyli coś o iSCSI i FCoE
HUAWEI TECHNOLOGIES CO., LTD.
HUAWEI ENTERPRISE A BETTER WAY
www.huawei.com
2014/09/30
Artur Pająk
Senior IT Solution Architect

Background of Storage

History of Storage Technologies
3
Embedded Storage (ES)
Deployment Characteristics
Storage devices are
deployed in servers
Storage medium cannot be
shared, and scalability is low
Servers are directly
connected to a storage
array
Storage medium cannot be
shared, and scalability is
limited by access capabilities
of storage devices
Direct Storage (DS)
Servers are connected to
a storage array through
an IP network
Shared storage medium,
high scalability, and file
storage
Network Attached Storage (NAS)
Servers are connected to
a storage array through
an IP or FC network
Shared storage medium,
high scalability, and highly
efficient block storage
Storage Area Network (SAN)

E_PORT
FC SAN Networking
SAN Characteristics
 Dual planes ensure reliability: A SAN has two
independent planes (fabric networks). Each server and
storage device connects to the fabric networks through
two independent links. When any node or link fails, the
server software switches to traffic to the other link.
 Auto terminal registration: Switches provide services,
as well as registration and state maintenance functions
for server and storage devices.
4
FC switch
Server
F_PORT
Fabric B
N_PORT
 Network-side port on user device
 User-side port on FC switch
 Interconnection port
of FC switch
Fabric A
Storage
array
RSCN NS
FSPF Zone

WWN
address
FC Addressing
WorldWide Name (WWN)
 WWN: (similar to MAC address) is a globally unique
64-bit address defined by a device vendor to identify all
devices (FC NICs and switch ports) on an FC network
5
FCID Domain Area Port
Fiber Channel Identifier (FCID)
 FCID: Similar to an IP address. A 24-bit address used
for data forwarding on an FC network. Each FCID
identifies as a user node and is dynamically allocated
by an FC switch when a user registers with the FC
switch.
 FCID Fields: The FCID consists of domain, area, and
port fields. The domain field identifies an FC switch and
accommodates a maximum of 236 domain IDs (similar
to masks used on an IP network).
FCID: used for forwarding on a fabric network.

FC Terminal Registration
6
Storage array
FC switch
Fabric B
Server (ENode)
Fabric
Login
Port Login
FC data stream
Fabric Login (FLOGI)
 FLOGI: Used to request service from an FC switch and obtain
an FCID when the user connects to a fabric network.
Port Login (PLOGI)
 PLOGI: Used to send a session request to the destination
node. The user can access the destination node after a
session is set up.
FC communication
 After a server and a storage array set up a session through
PLOGI, they can exchange FC data. FC switches on the
fabric network search for transmission paths for FC data
based on FCIDs of the source and destination nodes.
Fabric A

FC Distributed Network Services (Fabric Services)
NS
 Name Service: A service that an FC switch provides to
enable users to query information such as FCID-WWN
mapping.
FSPF
 Fabric Shortest Path First: Similar to OSPF, this
protocol forwards FC data based on the domain ID in
the destination FCID.
RSCN
 Registered State Change Notification: When a
user’s state changes, the FC switch connected to the
user sends an RSCN message to notify the FC
network of the change.
7
Switch1
Fabric A Fabric B
Switch2
Fabric Login
NS:
FCID1<->WWN1
WWN1: 24.00.XX
WWN2: 36.00.XX
FSPF:
Domain in
D-FCID=switch 1
FC data
D_FCID：disk1
HBA
HBA

Storage and iSCSI

Why iSCSI?
FC SAN IP SAN
Converged with data
network
N Y
Network Interface Card FC HBA NIC Ethernet NIC
Flow control FC TCP/IP
Transmit efficiency high low
Transmit delay low high
Impact on CPU less more
Deploy and maintain
costs
high low
Commercial maturity mature mature
Comparison of FCSAN, and IPSAN

iSCSI Addressing
10
Name (three types)
 iQN: (similar to WWN address)
example
iqn.2006-
07.pl.iscg:storage.disks.rack5.shelf2.vdisk2
 EUI: (similar to WWN address)
exemple: eui.1234567890ABCDEF
 NAA: (similar to WWN address)
example naa.FEDCBA0987654321
iSNS
 iSNS: Similar to DNS Server.

iSCSI target and initiator
11
Target (two types)
 Hardware (hidden software): disk
arrays
 Software: example StarWind, MS
WSS
Initiator
Mostly software

iSCSI frame size
12
Jumbo Frame
Jumbo Frame: extended size of
frame abow 1500 bytes (max size.
9k)
Function Jumbo frames must be
set on each devices in path

iSCSI Security
13
CHAP
 CHAP: password on target and
initiator , default not used
IPSec
 IPSec: encryption transmition, default
not used.
Data Encryption
 Data encryption: data encryption
example: BitLocker.
Recomendation: use dedicated network.

Converged Network

Converged Data Center Network
15
Traditional DC Network Architecture
Front-end
Network
Server
Cluster
Back-end
Network
LAN
SAN
Converged DC Network Architecture
Converged
Network
Server Cluster
& Disk Array
Converged

IB
FC
GE
Network Convergence Trend
PCIe IPC/SAN/LAN
Future
Now
Past
Multiple networks
converge into one
Multiple network
adapters converge into
one, and multiple
network converge at
access layer
Multiple network adapters of
various types connect to
multiple physical networks
Server
IPC
SAN
LAN
PCIe
Three elements for
network convergence
IPC
IPC
Server
Converged
Server
IPC
SAN
LAN
PCIe CNA
CNA
 High bandwidth: 10GE/40GE/100GE
converged Ethernet network
 Low latency: CutThrough forwarding
 No packet loss: lossless congestion
control mechanism

FCoE Technology

Why FCoE?
FC SAN IP SAN
Converged with data
network
N Y
Network Interface
Card
FC HBA NIC Ethernet NIC
Flow control FC TCP/IP
Transmit efficiency high low
Transmit delay low high
Impact on CPU less more
Deploy and
maintain costs
high low
Commercial
maturity
mature mature
 FCoE inherits the advantages of FC SAN network, such as , high transmission efficiency, low propagation delay and packet loss, etc. ；
 FCoE use ethernet to bear the storage service, which reduce the cost of deployment and maintain;
 FCoE is compatible with FC SAN, which can take full use of existing storage network;
FCoE
Y
CNA NIC
DCB
high
low
less
medium
medium

10 GE and Lossless Enhanced Ethernet Technologies have been standardized, while
FCoE is still being optimized
Technology and Standard Evolution
19
Large Scale 10 GE Access
 10 GE CNA: Well-developed technology with many products
on the market
 40 GE/100 GE network products have emerged
Lossless Enhanced Ethernet
 DCB: A published standard that defines service based back-pressure
congestion control mechanisms based on FC BB-Credit
and EE-Credit
Fiber Channel over Ethernet
 FC-BB-5: A published FCoE standard that defines the
encapsulation format, signaling control, and forwarding model
of FC packets over Ethernet
 FC-BB-6: A draft standard that optimizes the local forwarding
mechanism and defines FDF local forwarding entities.

Type=FCoE_Type (8906h)
VLAN
Ethernet frame header: 12-byte MAC address + 4-byte VLAN tag
FCoE Type VER SOF FC Frame EOF
FCoE header: 16-byte (16-bit Ether-Type, 4-bit VER, 8-bit SOF)
EOF: 1 byte + 3-byte reserved field
FC SEQ_ID
FCoE Encapsulation
FCoE Packet Format
 DA: Destination MAC address, indicating the MAC address of
the next FCF hop in unicast forwarding, or reserved MAC
address in multicast forwarding.
 SA: Source MAC address, indicating the MAC address of
each FCF or terminal hop.
 VLAN: FIP-specified VLAN or data VLAN for FCoE packets.
 Type: Ethernet type. The value 8906h indicates FCoE packets.
 FCS: Ethernet frame check sequence.
20
 VER: Version number
 SOF: Start of frame
 EOF: End of frame
 D_ID: Destination FCID
 S_ID: Source FCID
 SEQ_ID: Sequence ID
Ethernet DA SA Ethernet Payload FCS
D_ID S_ID SCSI Command and data CRC
FC header: 24 bytes
FC payload: more than 2112 bytes

FCoE Networking and FCoE Switches
FCoE Networking
 Servers use Converged Network Adapters (CNAs) to connect
to FCoE switches.
 FCoE switches differentiate and distribute Ethernet and FC
service flows.
 The switches use the FIP protocol to complete FCoE
initialization and allocate FCIDs and MAC addresses to users.
FIP Protocol
 FCoE Initialization Protocol: Allocates MAC addresses and
VLAN IDs to online users.
Types of FCoE Switches
 Classification: FCoE switches are classified into FCF, NPV,
FDF, and FSB switches, based on the functions they provide.
FCFs are core components on an FCoE network and provide
key functions such as FIP and fabric services.
21
Ethernet
network
FC Fabric network
FCoE switch
Link type:
FCoE switch
Converged link (FCoE & Ethernet)
FC FCoE
Server
Ethernet
FC switch
CNA CNA

FCoE Switch: FSB (FIP Snooping Bridge)
22
FSB functions
 FSB: FIP Snooping Bridge
 Control plane: Does not participate in FIP protocol control,
but listens to FIP messages and controls link access based on
FIP messages to enhance security.
 Forwarding plane: Forwards packets based on MAC
addresses.
 Function: Provides 10 GE FCoE access and transparently
transmits FC data over Ethernet.
FSB advantages and disadvantages
 Advantages: Easy to deploy. Highly efficiency MAC-based
forwarding.
 Disadvantages: Cannot be deployed independently. An
FSB supports FCoE ports only and is not compatible with
FC networks. FC traffic must be transmitted by FCFs
(circuitous forwarding paths).

FCoE Switch: N_Port Virtualizer (NPV)
23
NPV functions
 NPV: N_Port Virtualizer
 Control plane: Participates in FIP protocol control. Functions
as a proxy agent for servers to send FIP requests.
 Forwarding plane: Forwards packets based on FCIDs.
 Function: Provides FCoE access and supports connection of
FC ports to traditional FC networks.
NPV advantages and disadvantages
 Advantage: Compatible with FC networks.
 Disadvantages: Cannot be deployed independently. FC
traffic must be transmitted by FCFs (circuitous forwarding
paths).

FCoE Switch: FC Forwarder (FCF)
24
FCF functions
 FCF: FC Forwarder.
 Control plane: Provides FIP registration service, and
provides fabric networks with the Naming Service (NS),
Registered State Change Notification (RSCN), and FC
Shortest Path First (FSPF) services.
 Forwarding plane: Forwards packets based on FCIDs and
supports local forwarding.
 Function: Provides FCoE initialization function (FIP) and
fabric service.
FCF advantages and disadvantages
 Advantage: Compatible with FC networks and can be
deployed independently.
 Disadvantage: Domain ID specification limits network
scale.

FCoE Switch: FCoE Data Forwarder (FDF)
25
FDF functions
 FDF: FCoE Data Forwarder.
 Control plane: Participates in FIP protocol control but does
not provide fabric service.
 Forwarding plane: Forwards packets based on FCIDs and
supports local forwarding.
 Function: Provides FCoE access and FC ports to connect to
traditional FC networks, and supports local forwarding.
FDF advantages and disadvantages
 Advantage: Compatible with traditional FC networks and
supports local forwarding to avoid circuitous forwarding paths.
 Disadvantage: Cannot be deployed independently and must
work with FCFs.
FDF is defined in FC-BB-6 but has not
been standardized

Changes brought by FCoE
Converged Network
 Server do not need both Ethernet NIC and FC HBA but one CNA converged card, which can reduce the CAPEX of NIC;
 Data network and Storage network begin to converge, which reduce the CAPEX of network;
 Take full use of existing FC SAN network, which protect the CAPEX of network;
 Switches support FSB/NPV function, use single or multiple hop mechanism to access storage, the scale of storage become larger;
 Switches support DCB function, which give a lossless Ethernet to bear storage service;
FCF FCF
FSB/NPV
FSB/NPV
FC SAN Storage
Data Network
Storage Network
26

FCoE Forwarding Path Optimization
27
FCF
FSBs and NPVs forward traffic through
circuitous path
 FC service flows are forwarded by FCFs or FC switches based
on FCIDs.
 Servers and storage devices connected to the same FSB or
NPV cannot communicate directly, and data exchanged
between them must be forwarded by the upstream FCF or FC
switch.
FDFs support local forwarding
 A fabric network can be expanded by adding FDFs, which do not
occupy domain IDs.
 FDFs support local forwarding. Data exchanged between servers and
storage devices under the same FDF is forwarded by the FDF, and
data exchanged between those under different FDFs are forwarded
by the upstream FCF.
FSB NPV
FDF
FDF
FCF

FCoE Switches Comparisons
28
Support
Traditional FC
Forwarding
Mechanism
Shortest Path
First
Distributed
Fabric service
Use Domain ID Scalability Complexity
FSB No MAC No No No Good Low
NPV Yes FCID No No No Good Medium
FCF Yes FCID Yes Yes Use
exclusively
Poor High
FDF Yes FCID Yes No Share Good Medium
FCF
FSB
FCF
NPV
FCF
FCF
FCF
FDF
FCF
FDF
?
FSB NPV
FCF + FSB networking
No forwarding path
optimization
FCF + NPV networking
No forwarding path
optimization
FCF networking
Optimized forwarding paths
FCF + FDF networking
Optimized forwarding paths

Enhanced Ethernet Technology

Ethernet Technology Improvements
31
Traditional Ethernet
 Congestion Control: Uses TCP
retransmission and sliding window
mechanisms to perform flow control
based on destinations.
 Pause Mechanism: Traditional
Ethernet switches send Pause frames
to request downstream devices to stop
sending frames.
Traditional FC
 Congestion Control: Uses buffer-to-buffer
credit (BB-Credit) and End-to-
End credit (EE_Credit) mechanisms.
(BB_Credit and EE_Credit are
similar to the token bucket
mechanism and perform flow control
based on bandwidth capabilities
between nodes.)
Enhanced Ethernet
 Congestion Control: Combines the
Ethernet Pause mechanism and FC
flow control to define the multi-level
congestion control mechanism based
on PFC, ETS, and the CN based end-to-
end congestion control mechanism.
 Storage services require lossless transmission.
 Traditional FC technology has Credit mechanism to ensure lossless transmission, but traditional Ethernet only
implements “best effort” transmission.
 DCB technology is introduced in Ethernet ensures FCoE (QoS).

E Ethernet service packet
FFFF F F
Priority Flow Control (PFC)
FC storage F service packet I IPC service packet
F
Ethernet 0/1
Buffering queue
Ethernet 0/2
Buffering queue
pause pause pause
Ethernet 0/1 Ethernet 0/2
Ethernet 0/1
Buffering queue
Ethernet 0/2
Buffering queue
Ethernet 0/1 Ethernet 0/2
Server FCOE switch FCOE switch Storage array
PFC service identification
 Different priorities (0 – 7) and queue thresholds are configured
for various services in network planning.
 Switches perform congestion control based on service priorities.
When a service traffic priority exceeds the threshold, excess
packets are dropped. Packets of other services are forwarded
normally.
Multi-level back pressure
 Ethernet Pause mechanism: When congestion occurs, switches send
back-pressure signals to downstream devices, requesting downstream
devices to stop sending service packets.

Enhanced Transmission Selection (ETS)
33
ETS service identification
 Different priorities, scheduling modes, and bandwidth thresholds
are configured for various services in network planning.
 Priority queuing is performed for latency sensitive IPC service,
and polling scheduling is performed for SAN and LAN services.
Cooperation with CN and PFC
 ETS uses PFC or CN back-pressure for services requiring low latency
and high reliability, such as IPC and SAN. When the rate of such a
service exceeds the bandwidth threshold, ETS sends a back-pressure
signal to the downstream node to reduce loads on the local device.
 For latency insensitive LAN service, retransmission mechanism of
TCP/IP is used, instead of the back pressure mechanism.
 The TCP/IP retransmission is used for latency-insensitive LAN services.

CN(Congestion Notification)
Congested Queue
Source node address
…
Ethernet 0/2 Ethernet 0/1 Ethernet 0/2
Page 34
Server A
FCOE switch FCOE switch Storage Array
Server B
Server C
Server A
Server C
CN implementation
 Implementation: An FCoE switch searches for the source node
that is causing the congestion and sends CN messages to that
node until the congestion is removed.
CN characteristics
 Finds the source node that is causing the congestion and prevents the
node from sending packets to the network.
FCoE switch forward the FCoE frame hop by hop, the source MAC address may be not the MAC address of initial sender, so in this case,
there is few difference between CN and PFC.

Huawei Converged Network Solution

Aggregation
Switch
CE12800
Illegal user
MAC spoofing
MAC:0066-6666-666
Access
Switch
CE6800
10 GE
Server
CNA CNA CNA CNA CNA CNA
Huawei FCoE Solution
37
Fabric A
Fabric B
40 GE trunk
FCF
FC
Storage
Ethernet
FSB
eNode
VN_PORT1：
MAC:0066-6666-666
Ethernet link FCoE link Ethernet & FCoE converged link
High reliability
 Double Fabric Planes: Each server connects to two
fabric planes through double CNAs to ensure reliability of
storage services.
 Multiple Ethernet links: Each fabric network has multiple
Ethernet links bundled into a trunk link to improve link
reliability.
No packet loss
 DCB: Deployed between servers, FSBs, and FCFs to
prevent packet loss for FC storage services on the
Ethernet network.
 10 GE access: Next generation data center 10 GE
switches provide sufficient bandwidth for FCoE access.
High security
 FSB: Listens to FIP messages to control user access.
Links are open only to authorized users.
FC SAN

Huawei Converged Network Evolution Roadmap
38
Servers
Core switch
CE12800
Access
Switch
CE6800
FC/FCoE switch(FCF)
FC
Storage
Ethernet port
FCoE/FC port
SAN
Access and storage network convergence
 TOR switches support FCoE (V1R1:FSB) / FC (V1R2: FCF/NPV) ports
and DCB.
 TOR switches distribute storage traffic to the SAN.
Core and storage network convergence
 TOR and core switches support FCoE/FC ports and DCB.
 Core switches distribute storage traffic to the SAN.

Huawei FCoE Configuration Example

Huawei FCoE Solution Topology Example
 Topology statement
― CE6850 connects to Server A and Server B
through port 10GE1/0/1 and 10GE1/0/2
respectively；
― CE6850 connects to FCF through port 10GE1/0/3
to access SAN storage;
― CE6850 connects to data network through port
40GE
 Port Statement
― VNP-port：the port of CE6850 connecting to FCF
― Enode-facing： the port of CE6850 connecting to
Server

Huawei FCoE Solution Example Configuration step
——FSB configuration
 Configure an FC instance: Configure an FC instance FSB on SwitchA and specify FCoE VLAN 2094 in this instance;
 Configure port roles: Configure 10GE1/0/3 on SwitchA to allow VLAN 2094. The configurations of 10GE1/0/1 and 10GE1/0/2
are similar to the configuration of 0GE1/0/3, and are not mentioned here. An interface is an ENode-facing port by default, so the
configurations of 10GE1/0/1 and 10GE1/0/2 are not mentioned.

——Priority configuration
 Configure priority mapping:
― create a DiffServ domain ds1and map 802.1p priorities of received server packets to CoS values.
― Bind the DiffServ domain to 10GE1/0/1, 10GE1/0/2, and 10GE1/0/3.

——DCBX and PFC Configuration
 Configure DCBX
 Configure PFC:
― Enable PFC for queues with priority 3. By default, PFC has been enabled for queues with priority 3.
― Enable PFC on an interface.

——ETS Configuration
 Configure ETS:
― Create an ETS profile.
― Map queue 3 to PG1, queue 7 to PG15, and other queues to PG0. Queue 3 maps PG1, and queues 6 and 7 map PG15 by
default, so you only need to add queue 6 to PG0.
― Configure flow control based on the priority group and set DRR weights of PG0 and PG1 to 60% and 40% respectively.
― Apply the ETS profile to 10GE1/0/1 and 10GE1/0/2.

——FCF Configuration
 Configure CEE profile, map profile, and so on.
 FCoE profile must be configured on the port connecting to CE6800
 FCoE profile must be configured on the port connecting to SAN storage.
 Zone must be created on FCF. Host and storage must be added in zone. The default rule of
zone must be set as “permit”
 LUN, host group, host must be created on SAN storage, FC initiator must be created and
indicated as WWN server.
 FC initiator must be added to host;

Are we ready for use FCoE????

Huawei Enterprise A Better Way

PLNOG 13: Artur Pająk: Storage w sieciach Ethernet, czyli coś o iSCSI I FCoE

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to PLNOG 13: Artur Pająk: Storage w sieciach Ethernet, czyli coś o iSCSI I FCoE

Similar to PLNOG 13: Artur Pająk: Storage w sieciach Ethernet, czyli coś o iSCSI I FCoE (20)

Recently uploaded

Recently uploaded (20)

PLNOG 13: Artur Pająk: Storage w sieciach Ethernet, czyli coś o iSCSI I FCoE