VMware EMC Service Talk

Copyright © 2013 EMC Corporation. All Rights Reserved.
VMware EMC Service
Talk
May 2014
Kevin Wang

Question 1: The detailed procedure for HBA
initiates login to SAN to complete the connection to
the SP (Answers cover both FC and FCoE)

This lesson covers the following topics:
• FC protocol make-up
• FC components
Lesson 1: Fibre Channel Protocol Overview
Fibre Channel and FCoE Technology 3

Fibre Channel Layered Architecture: FC-0
ULP
Upper Layer Protocol
SCSI-3, IP, ESCON/FIPS, etc.
FC-4
FCP (SCSI-3 Fibre Channel Link Encapsulation),
Single-Byte Command Code Sets (FICON), etc.
FC-3 Common Services (Not Used)
FC-2
Exchange and Sequence Management,
Frame Structure ,Flow Control
FC-1
Encode/Decode, Link Control Protocols,
Ordered Sets
FC-0
Physical Interface, Optical and Electrical Interfaces, Cables,
Connectors, etc.

Transceivers Cable Connectors Comments
SFP, SFP+ LC
• Most popular for Fibre Channel
today
• SFPs up to 8 Gbps
• SFP+ > 8Gbps
• versions for different distances
mSFP mLC
• mini-SFP and mini-LC
• Designed by Brocade for 64-port
blade
GBIC SC
• Technology found in older 1 Gbps
equipment
• Replaced by LC technology
Optical Transceivers and Cable Connectors

Fibre Channel Cables and Distances
Transceive
r Type
Speed
Multimode media maximum distance
62.5 um
(OM1)
50 um
(OM2)
50 um
(OM3)
50 um
(OM4)
SW
2 Gbps 150m 300m 500m 500m
4 Gbps 70m 150m 380m 400m
8 Gbps 21m 50m 150m 190m
10 Gbps 33m 82m 300m 550m
16 Gbps 15m 35m 100m 125m
Single-mode media maximum distance (9 um core optic fiber)
LWL
4 Gbps
10 km to 40 km depending on vendor
8 Gbps
10 Gbps
16 Gbps
Note: For greater distances, you may connect using DWDM / CWDM between switches .

Basic Port Types
Node Port: Transmits and receives Fibre Channel data frames in a
Switched Fabric environment (N_Port)
Fabric Port: Port on a switch that N_Port connects to. (F_Port)
Expansion Port: Port on a switch that has another switch plugged into
it (two switches that are merged) (E_Port)
Node Loop Port: Transmits and receives Fibre Channel frames in an
Arbitrated Loop (private) environment (NL_Port)
Fabric Loop Port: Allows a Fabric to communicate with an Arbitrated
Loop (public) (FL_Port)
Generic Port: Port on a switch that detects the attached port type and
auto-configures to match that port type (G_Port)

The Host Bus Adapter
Fabric
Target
Storage Array
Server
HBA
NIC
LAN
See EMC Support Matrix at https://elabnavigator.emc.com

ULP
FC-4
FC-3
FC-2
FC-1
Encode/Decode, Link Control Protocols,
Ordered Sets
FC-0
Data encoding/decoding
• Less than 10 Gbps (20% overhead for encoding)
 8-bits  10-bits
• 10 Gbps and 16 Gbps (3% overhead for encoding)
 64-bits  66-bits
• Special Characters
 Used with data characters to create ordered sets
 Ordered sets include: Idle, SOF, EOF, R_RDY

ULP
FC-4
FC-3
FC-2
Exchange and Sequence Management,
Frame Structure ,Flow Control
FC-1
FC-0
• Defines structure of Fibre Channel frame
 Frames, Sequences, Exchanges
• Flow control mechanisms
• Error handling and recovery
• Provides class of services
 Class 1, Class 2, Class 3, Class 4, Class 6, and Class F

FC-2
• Defines the structure of the Fibre channel frame
 How information to large to fit into a single frame is transferred by
using a sequence of frames
 How operations that consist of exchanges of sequences of frames
are managed
• Describes how the rate of frame transmission is controlled by the
flow control mechanisms
• Error handling and recovery
• Class of Service
 Class 1, Class 2, Class 3, Class 4, Class 6, and Class F

Exchange, Sequence, Frame
• Exchange is an operation manager
 SCSI task, an IP connection
 Has an Originator
 Has a Responder
 Has one or more Sequences
• Sequence is a ULP object to be transmitted in one direction that
may be one frame or longer
• Frame is the unit of information transfer
• Manages flow control (buffer-to-buffer and end-to-end)

Conversations
Session
Exchange Exchange Exchange
Seq Sequence Sequence SequenceSeqSeq
Frame #1 Frame #1
Frame #2
Frame #3
Frame #1
Frame #2
Frame #3
Frame #4
Frame #5
Frame #6
Frame #1
Frame #2
Frame #3
Frame #4
Frame #1
Frame #2
Frame #1

Fibre Channel Frame
Data Field
End-of-Frame
CRC
Start-of-Frame
Header
Optional
Header
Fill
Bytes
Payload
(information being transported)
Optional
Header
R_CTL
CS_CTL
TYPE
SEQ_ID DF_CTL SEQ_CNT
Frame Control (F_CNT)
Source Address (S_ID)
Destination Address (D_ID)
OX_ID RX_ID
Parameter Field (PARM)

Credit-Based Flow Control
Fibre Channel
Switch
Physical ISL
FC Frame R_RDY
Credit = 161516
Fibre Channel
Switch
Long Distance links may require additional BB_Credit

ULP
FC-4
FCP (SCSI-3 Fibre Channel Protocol) mapping,
Single-Byte Command Code Sets (used for FICON) mapping
FC-3 Common Services (Not Used)
FC-2
FC-1
FC-0
• Mapping Layer
 Maps Upper Level Protocol to lower levels
 SAN typically uses SCSI FCP mapping

FC-4 and Upper Level Protocol
Exchange, Sequence, and Frame
...Information Unit # 1 Information Unit # 3Information Unit # 2...FC-4
FC-2
Sequence # 1 Sequence # 3Sequence # 2... ...
Frame 1
...
Frame 1 Frame 2 Frame N Frame 1
Payload: SCSI CDB DATA Payload: Status
Exchange (Operation)
DATA DATA
SCSI Command SCSI ResponseSCSI DATA
ULP
SCSI-3:
One SCSI Read I/O Operation

EMC Fibre Channel Implementations
• Point-to-Point – Link between two
ports using Arbitrated Loop
• Arbitrated Loop (FC-AL) – Shared
interconnect between 2 to 126 nodes
• Switched Fabric (FC-SW) – Dynamic
connectivity support for more than 15
million nodes

Fibre Channel Fabric
Target
Storage Array
Fabric
Server
HBA
Server
HBA
ISL

Fibre Channel Address (FCID)
• Required to route frames from source to target
• 24-bits (three-bytes) - specified in Hex
 Assigned at Fabric Login (FLOGI)
• Divided into three parts:
Domain_ID Area_ID Port_ID
Domain (01-EF)
(1-239 in decimal)
00-FF 00-FF
Example: 6b 04 00

Common Fabric Services: Well Known Addresses
Service Name
Well Known
Address
Description
Login Service
(F_Port Server)
FFFFFE
All nodes register in login service when
performing FLOGI
Name Service FFFFFC
Stores information about nodes attached
to fabric during the Fabric Login process
Fabric Controller FFFFFD
State change notification to nodes
registered in the fabric and inter-switch
communications

World Wide Name
• A World Wide Name (WWN) is a 64-bit value used in Fibre
Channel networks to uniquely identify each element in the
network
• Usually assigned to a Host Bus Adapter (HBA), switch port, or
storage port by the vendor at the time of manufacture
• Often linked within a component, such as an HBA
Node WWN – A name for all ports within the single HBA
Port WWN – A name for each individual port

WWPN Examples
EMC VNX World Wide Port Name Example
5 0 0 6 0 1 6 0 4 7 2 0 5 5 9 0
EMC Company ID (24-bits) Port VNX Seed (from SPE/DPE midplane)
EMC VMAX World Wide Port Name Example
5 0 0 0 0 9 7 2 0 8 1 3 4 9 a d
EMC Company ID for VMAX (24-bits) VMAX Serial Number - Decodes to country
where manufactured/Model / etc.
(HK192601234)
CPU/Dir/P
(12g port B)
Emulex World Wide Port Name Example
1 0 0 0 0 0 0 0 c 9 2 0 d c 4 0
Reserved (12-bits) Company OUI (24-bits) Company Specific (24-bits)

FC Port Initialization Process - Step 11 - Link online
2 - Fabric Login - get FCID
3 - Port Login to fabric name server
4 - Registrations with name server
5 - Query Name server for FC-4 devices
6 - State Change Registration
Repeat for each Target:
 7 Port Login to Target port
 8 Process Login to Target port
 9 SCSI INQ
Fabric 1
5
1 2
3 4
HBA
WWN = 50:06:04:8a:d5:2e:76:c4
FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7
FCID = ??????
6

 7 - Port Login to Target port
 8 - Process Login to Target port
 9 - SCSI INQ
Fabric 1
5
1 2
3 4
HBA
FLOGI
ACC
Accept frame
includes FCID
for HBA 030500
FLOGI Frame
Destination Address =
FFFFFE
Source Address =
000000
WWN = 50:06:04:8a:d5:2e:76:c4
FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7
FCID = ??????
6

 9 - SCSI INQ
Fabric 1
WWN = 50:06:04:8a:d5:2e:76:c4
FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7
FCID = 030500
5
1 2
3 4
HBA
PLOGI
ACC
PLOGI Frame
FFFFFC
Source Address =
030500
6

 9 - SCSI INQ
Fabric 1
5
1 2
3 4
HBA
WWN = 50:06:04:8a:d5:2e:76:c4
FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7
FCID = 030500
6
Fibre Channel Services Frames
• Register Class of Service
• Register FC-4 Type
• Register Port type
• Register other (vendor dependent)
FC
Serv
ACC
FC
Serv
ACC

 9 - SCSI INQ
Fabric 1
5
1 2
3 4
HBA
WWN = 50:06:04:8a:d5:2e:76:c4
FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7
FCID = 030500
6
Fibre Channel Services Frames
• Query: Get PIDs
• Query: Get Port WWN for PID1
• Query: Get Port WWN for PIDn
FC
Serv
ACC
FC
Serv
ACC
Name Server Database:
• PID = 040600
• FC-4 = SCSI FCP
• WWPN = 50:06:04:8a:d5:2e:76:c4
• . . .
• PID = 030100 . . .
• PID = 040400 . . .

 9 - SCSI INQ
Fabric 1
5
1 2
3 4
HBA
WWN = 50:06:04:8a:d5:2e:76:c4
FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7
FCID = 030500
6
SCR
ACC
SCR Frame
FFFFFD
Source Address =
030500

 9 - SCSI INQ
Fabric 1
5
1 2
3 4
HBA
WWN = 50:06:04:8a:d5:2e:76:c4
FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7
FCID = 030500
6

 9 - SCSI INQ
Fabric 1
5
1 2
3 4
HBA
WWN = 50:06:04:8a:d5:2e:76:c4
FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7
FCID = 030500
6
SCSI
Process
SCSI
Process

Fibre Channel Login at a Glance
Process FLOGI PLOGI PRLI
Requirements - Link initialization
- Cable
- HBA and driver
- Switch port
- FLOGI
- Zoning
- Persistent binding
- Driver setting
- PLOGI
- Device masking (Target)
- Device mapping (Initiator)
- Driver setting (Initiator)
Information passed - WWN
- S_ID
- Protocol
- Class
- Zoning
- WWN
- S_ID
- ULP
- Class
- BB Credit
- LUN
Session members - N_Port to F_Port - N_Port to N_Port - ULP (SCSI-3 to SCSI-3)
Verification UNIX
WINDOWS
- Syslog
- Switch Utilities
- Syslog
- Driver Utilities
- Syslog
- Host based volume management
- Event Viewer
- Switch Viewer
- Driver utilities - Driver utilities
- Host based volume management
- Device manager
Fibre Channel and FCoE Technology - 33

Zone Set: Fabric 1
Zone A: Initiator A; Target w
Zone B: Initiator B; Target x
Zone C: Initiator C; Target y; Target z
. . .
Zoning Overview
Storage
(Targets)
Hosts
(Initiators)
I
T
Initiator (HBA) Target
Zone A
Fabric 1

Fabric 1
How to Define Zone Members
I
T
Target Port WWPN:
50:06:04:8a:d5:2e:76:c4
Zone A
Initiator Port WWPN:
10:00:00:00:c9:48:f1:c7
5 3
1 2
3 4
Zone A
10:00:00:00:c9:48:f1:c7;
50:06:04:8a:d5:2e:76:c4
Zone Members Defined by WWPN
Zone A
0305ef; (Domain_ID 03, port_05)
0403ef; (Domain_ID 04, port_03)
Zone Members Defined by Domain/Port

• Architecture of an Ethernet solution
Lesson 2: Ethernet Overview

Ethernet and the OSI Model
7 - Application
6 - Presentation
5 - Session
4 - Transport
3 - Network
2 - Data Link
1 - Physical
LLC - Logical Link Control
MAC - Media Access Control
Ethernet is defined at
Layer 2 of the OSI model

Ethernet PDU
Destination
MAC
6 bytes
Source
MAC
6 bytes
802.1Q
tag
(optional)
4 bytes
Type
2 bytes
Data Payload
variable # bytes
FCS
4 bytes
Ethernet_II Frame Format
Header

Ethernet MAC Address
I
G
G
L
Organizationally Unique
Identifier (OUI)
(Assigned by IEEE)
Vendor assigned
24 bits 24 bits
Examples:
Broadcast address: ff:ff:ff:ff:ff:ff
Physical address of port on PC: 00-26-5E-82-3E-6E
47 46 45 24 23 0

MAC Address Used to Direct Frame Through
Network
MAC: 00-26-5E-82-3E-6E MAC: 00-26-5E-77-2F-58
…2F:58 …3E:6E 08-00 Payload FCS
Ethernet Frame
This is the source address This is the destination address

Switching Frames Using MAC Addresses - Step 1
MAC Table
Port 1 -Empty
Port 2 - Empty
Port 3 - Empty
Port 4 - Empty
PC A PC C
PC B
PC D
Port 1
Port 2
Port 3
Port 4
Frame
Dest Src
PC A

MAC Table
Port 1 -PC A
Port 2 - Empty
Port 3 - Empty
Port 4 - Empty
PC C
PC B
PC D
Port 1
Port 2
Port 3
Port 4
No entry for PC
D, so flood the
frame out all
ports
PC A

MAC Table
Port 1 -PC A
Port 2 - Empty
Port 3 - Empty
Port 4 - Empty
PC A PC C
PC B
PC D
Port 1
Port 2
Port 3
Port 4
Frame
Dest Src
PC A PC D
PC D

The Problem of Loops and Broadcast Storms
Switch A
Switch C
Switch B
Broadcast

How STP Solves the Problem
• STP is a loop-prevention
protocol
• Uses spanning tree
algorithm to elect a
Root Bridge and block
certain ports.
Switch A
Switch C
Switch B
Block

How Does STP Protocol Elect a Root Bridge?
Switch A
Switch C
Switch B
Priority = 32768
MAC = 0000.1111.2222
Priority = 32768
MAC = 0000.2222.3333
Priority = 32768
MAC = 0000.2222.2222
• Switch with lowest
Bridge ID (BID) wins
election
• If all switches have
equal priority, then
switch with lowest MAC
address wins
Since all switches are set to
default priority, this switch
will be elected Root Bridge
because it has the lowest
MAC address
Bridge ID

Result of running STP Protocol
Switch A
Switch C
Switch B
Root
Bridge
Designated
Port (F)
Root Port (F)
Alternate
Port (D)
Nonroot
Bridge
Nonroot
Bridge
Root Port (F)
Designated
Port (F)
Designated
Port (F)
• Forwarding ports
 Root ports
 Designated ports
• Discarding port
 Alternate

Ethernet Hardware Overview
• Ethernet hardware supports the following speeds
 10 Mbps
 100 Mbps
 1 Gbps
 10 Gbps
 40 Gbps
 100 Gbps
Hardware Popular Implementations
NIC
(Network Interface
Controller)
• Stand-alone card
• Built into host motherboard
• Wireless
Cables
• Copper UTP (Unshielded Twisted Pair)
• Copper Twinax
• Optic
Connectors
• RJ45 (for copper UTP cables)
• SFP+ (for copper twinax cables)
• LC (for fiber optic cables)
Most common
Data Center Core (Backbone)

Network Interface Controller - NIC
• Provides access to the network
• Different implementations include:
 Stand-alone card (could have multiple ports)
 Built into motherboard (most common now)
 Wireless
• Ports can be “teamed” to act as a single logical interface for
failover
NIC LAN

Example Ethernet Cabling
UTP
(Unshielded Twisted
Pair)
Twinax with SFP+
Fiber Optic Cable
with LC Connector
Media Copper Copper Optic
IEEE Standard 1000BaseT (802.3ab)
10GSFP+CU (Direct
Attach)
1000BaseLX (802.3z)
Distance/speed 100 meters @1 Gbps 5 meters @ 10 Gbps 10 km @ 1 Gbps
Cat 5
RJ45
Twinax / SFP+ 9/125 SM Fiber
LC Connector

Issues With a Flat Switched Network
• Single broadcast
domain
• No Security between
users
Broadcast

VLAN
200
VLAN
200
VLAN
100
VLAN
100
VLAN Overview
• Each VLAN is a separate
broadcast domain
• Each switch port is assigned to a
single VLAN
• Devices in different VLANs must
use a router to communicate
• VLAN Trunks allow the data
from multiple VLANs to pass on
a single link between switches

How a VLAN Solves the Problems of the Flat
Network
• Smaller broadcast
domains
• Security between VLANs
VLAN
100
VLAN
200
VLAN
100
VLAN
200
VLAN
100
VLAN
200
Trunk
Broadcast Broadcast

How it Works: VLAN Tagging
• 802.1Q - IEEE Standard
 Each port on switch assigned to specific VLAN ID
 Unassigned ports automatically belong to Default VLAN (VLAN ID = 0001)
• When port receives frame:
 Compare VLAN ID to filter table.
 If valid, and MAC on another switch - forward to trunk port
 If destined to exit an access port on switch, remove VLAN ID
• Trunk carries both tagged and untagged traffic
Destination
MAC
6 bytes
Source
MAC
6 bytes
802.1Q
tag
(optional)
4 bytes
Type
2 bytes
Data Payload
variable # bytes
FCS
4 bytes
Ethernet_II Frame Format
VLAN Tag

• FCoE Protocol and FCoE Fabric design basics
Lesson 3: FCoE Protocol Overview

FCoE Protocol Stack
Volume managers, File systems, etc.
SCSI Command Descriptor Blocks, data, and
responses
FCoE frame encapsulation / de-capsulation,
FCoE entities
Frame switching, MAC address, transport
Connection to the Physical Layer
Fibre Channel Protocol frame construction
FC-4, FC-3, FC-2
Application
SCSI
FCP
FCoE
Enhanced
Ethernet
FCoE Protocol Stack

FCoE Encapsulation
SOF
EOF
Ethernet
Header
FCoE
Header
Encapsulated FC Frame
(including FC-CRC)
FCS
FC
Header
Payload
Fill
Dest.
MAC
Address
Source
MAC
Address
Version Reserved
VLAN
TAG
CRC
Ether
Type
Fibre Channel Frame
Ethernet Frame
FCoE PDU

FCoE Frames
• Frame Efficiency
 iSCSI, FCIP, iFCP rely on IP over Ethernet which result in additional
overhead in frames
 FCoE frames only use Ethernet protocol
 Efficiency can be calculated by the formula
• Jumbo Frames
 Ethernet frames with more than 1500 bytes of payload
 Considered unreliable on regular Ethernet
 CEE allows reliable use of Jumbo Frames
 FCoE must use jumbo frames (2240 bytes)
Number of Data Bytes
Total Bytes Sent

Converged Network Adapter (CNA) Overview
CNA
CNA = HBA + NIC + FCoE
Host

• CNA is both a
 Network Adapter
 Fibre Channel adapter
• Host sends I/O to appropriate
adapter:
 NIC – NFS, Web, Email, etc.
 HBA – SCSI block data
• Host receives data through
lossless Ethernet MAC
 Directs I/O to appropriate adapter
 Based on Ethernet “Type”
Converged Network Adapter Architecture
2 x 10 Gbps Ethernet Ports
Ethernet NIC
Host Networking
Drivers
FCoE Encapsulation
Existing FC
Host Drivers
CNA Adapter
Lossless Ethernet MAC
Is it FCoE?NO YES
HBA Architecture

Fibre Channel Forwarder – FCF
• FCF function provided by FCoE switch or FCoE blade
 Services FCoE Login requests
 Provides services associated with a Fibre Channel switch
Stand-alone
FCoE Switch
Fibre Channel Switch
with FCoE blade

• Lossless Ethernet:
 FCoE requires reliable frame delivery
 Can’t drop frames from buffer overflow
• Improves regular Ethernet:
 Eliminates Ethernet loss, therefore more
reliable
 Speed increased to 10 Gbps
• Additions to Ethernet:
 Priority-based Flow Control (PFC)
 Enhanced Transmission Selection (ETS)
 DCB Capability Exchange (DCBX)
 Notify neighbor node of DCB support
Data Center Bridging (DCB)

Enhanced Ethernet Features
Feature Enhancement
Priority Flow Control
IEEE 802.1Qbb – class of service flow control by
enabling PAUSE functionality on IEEE 802.1p
lanes
Data Center Bridging Exchange
Auto-negotiation of Enhanced Ethernet
capabilities DCBX (switch to NIC)
Bandwidth Management
IEEE 802.1Qaz Enhanced Transmission Selection –
manage bandwidth and assign priorities to
groups of IEEE 802.1p lanes based on class of
traffic.

Priority-based Flow Control
STOP
Transmit Queues Receive Buffers
0
1
2
3
4
5
6
7
Buffers are
almost full -
send “PAUSE”
Priority

Enhanced Ethernet - DCBX
• DCBX runs between the switch and
CNA
• DCBX negotiates capabilities
• Switch distributes configuration
to all attached adapters
• Devices need to discover the
capabilities of its peers.
• DCBCX utilizes the link-layer
discovery protocol and handles
local operational configuration
for each feature
Rules for Capabilities Result
Capability and configuration
match
Enabled
Adapter configured to accept
switch config.
Enabled
Adapter not configured to
accept switch config.
Disabled
Adapter does not
support/implement DCBX
Disabled

Enhanced Ethernet – Bandwidth Management
• Enables consistent
management of QoS
• Provides consistent scheduling
of different traffic types

Addressing
MAC = A
FCID = 050100
MAC = B
Domain ID = 05 Domain ID = 01
FCID = 011C00
FC SAN
Converged
Network
FC Frame
D_ID = 011C00
S_ID = 050100
Dest = MAC B
Source = MAC A
D_ID = 011C00
S_ID = 050100
Ethernet Frame

A Note About CNA MAC Addresses
FCoE
Controller
Lossless Ethernet MAC
FCoE_LEP
VN_Port
FC-3 / FC-4
Layers
Ethernet_Port
Ethernet NIC
Host Networking
Drivers
Existing FC
Host Drivers
Physical MAC
ENode MAC
FPMA MAC
CNA

How are MAC Addresses Assigned to the VN_Port?
• Server-provided MAC Address (SPMA):
 Uses single MAC Address for all FCoE traffic
 FCoE has MAC address independent of other protocols
 Support for different MAC addresses for each Virtual Port
 This method isn’t used
• Fabric-provided MAC Address (FPMA):
 Uses virtual port MAC addresses
 Each virtual port MAC address is supplied by the fabric
 Most common method
 Have the following format:

Server Provided MAC Addresses (SPMA)
• Adapter uses burned-in or configured MAC address
 Consistent with the Ethernet model
• FCF needs a table to map between MAC addresses and FC_IDs
FCoE
Node Port
MAC
00.05.69.00.00.03
MAC Address

Fabric Provided MAC Addresses (FPMA)
• MAC address assigned for each FC_ID:
 Consistent with the Fibre Channel model
• Uses OUIs with U/L = 1 FC-MAP (Local addressing)
• No table needed for encapsulation
FC-MAP
(0E-FC-00)
FCID
07.08.09
24 bits 24 bits
48 bit MAC Address
FC-MAP
(0E-FC-00)
FCID
07.08.09

FCoE Initialization Protocol (FIP) and FC Logins
CNA
CNA (ENode)
Lossless Ethernet
Switch (FIP Snooping
Bridge)
FCoE Switch
(FCF)
Storage
FCoE Switch
(FCF)
FC SAN
FC SAN
Storage
ENode (FCoE Controller)
Send FIP VLAN Request
Send FIP Solicitation
Send FIP FLOGI
FCF
Reply FIP VLAN Notification
Reply FIP Advertisement
Reply FIP FLOGI ACC
FIP Multicast

FIP, the FCoE Initialization Protocol – How does it
work?

FIP VLAN Request

FIP Discovery Solicitation

FIP Discovery Advertisement

FIP Virtual Link Instantiation Request (FLOGI)

FIP Virtual Link Instantiation Request (FLOGI ACC)

FIP Snooping Bridge (FSB)

Question 2: How SP coordinates the host to handle
hardware assist locks (Reservation, ATS)

Verify VAAI Support Details
• The compare write(F1) operation is exclusively used by VAAI
enabled VMware host

VAAI – Hardware Assisted Locking
• Scalable Lock Management – Atomic Test and Set (ATS)
• A number of VMFS operations cause the LUN to temporarily
become locked for exclusive write use by one of the ESXi nodes,
including:
– Moving a VM with vMotion
– Creating a new VM or deploying a VM from a template
– Powering a VM on or off
– Creating a template
– Creating or deleting a file, including snapshots
• VAAI feature, atomic_test_and_set (ATS) allows the ESXi host to
offload the management of the required locks to the storage and
avoids locking the entire VMFS file system

Atomic Test & Set
• Original file locking technique
1. Acquire SCSI reservation
2. Acquire file lock
3. Release SCSI reservation
4. Do work on VMFS file/metadata
5. Release file lock
• New file locking technique
1. Acquire ATS lock
2. Acquire file lock
3. Release ATS lock
4. Do work on VMFS file/metadata
5. Release file lock
Note - The main difference with using the ATS lock is that it
does not affect the other ESXi hosts sharing the datastore

VMFS Scalability with Atomic Test and
Set
Makes VMFS more scalable overall, by offloading block locking
mechanism
Using Atomic Test and Set (ATS) capability provides an
alternate option to use of SCSI reservations to protect the
VMFS metadata from being written to by two separate ESXi
hosts at one time.
Normal VMware
Locking (No ATS)
Enhanced VMware
Locking (With ATS)

How to resolve SCSI Reserves on
VMware ESXi servers
• Refer to the following VMware KB articles:
– SCSI Reservation Issue with Fibre Channel HBAs (4365932)
– Troubleshooting SCSI Reservation failures on Virtual Infrastructure 3.x and
vSphere 4.x (1005009)
– VMware KB Article 1002293
• Warning! DO NOT release the reservation with INLINES when
customer is using vSphere.
• It is also NOT recommended to try to release via Solutions
Enabler
• The CORRECT way to release a SCSI Reserve in ESXi is to ask the
customer to use OS tools as described in the VMware
documentation and KB articles that are listed.

Question 3: The terms or tricks to match WWPN
with SP port, which would help us to understand
mpath failover sequence

During this lesson the following topics are covered:
• PowerPath configuration and PowerPath’s function in the I/O
path
• How PowerPath supports EMC and non-EMC arrays
• Key functionality included in current releases of PowerPath
• How PowerPath is licensed
• Hardware, software, clusters, and storage arrays that PowerPath
supports
Lesson 1: PowerPath Architecture
87

EMC PowerPath - A Family of Products
• PowerPath traditionally viewed only as “Path Management Software”
 Performed Multipathing, load balancing, failover, etc.
• Functionality Expanded
 PowerPath Family now has 4 distinct products
 PowerPath can now be used for solutions previously handled by separate
software
 Common Interface allows for simplified management
• Optimized for heterogeneous host / storage environments
88
PowerPath Multipathing Path
Management /
OptimizationPowerPath/VE
PowerPath Migration
Enabler Data Protection
PowerPath Encryption

• PowerPath sits
between the
applications and the
HBA driver
• Applications direct
I/O to PowerPath
• PowerPath directs
I/O to an optimal
path based on
current workload
and path availability
PowerPath Configuration
89
OpenSystemsHost
Logical volume manager
File system
PowerPath Driver
Applications
Management
utilitiesDBMS
Storage Array
Host bus
adapter
Host bus
adapter
Host bus
adapter
Host bus
adapter
Storage Area Network

• PowerPath keeps a
table of paths
• Presents a single
device to the
application
• Combines
equivalent paths
into a single “Path
Set”
Volume Path Set
90
PowerPath
HBAHBA
SERVERSTORAGE
Front-end
Ports
Storage Area
Network
Volume Path Set
Application
HBAHBA
POWERPATH
“Power”
Device

PowerPath Array Support Concepts
91
SERVERSTORAGE
SERVERSTORAGE
Active-Active Arrays Active-Passive Arrays
Owned by SP-A
HBA HBA HBAHBA
SAN
SP-A
Active
Active
SP-B
Passive
Passive
PowerPath
0 1 0 1
Host Application(s)
Mapped to all
Front-end ports
Active
Active
Active
Active
SAN
Front-end Ports
HBA HBA HBAHBA
PowerPath
Host Application(s)

• Dead
 indicates the path is
not usable.
 PowerPath driver
will not direct user
I/O to this logical
I/O path.
• Alive
 indicates the path is
usable
 The PowerPath
driver can direct I/O
to this logical I/O
path.
Device (LUN) Path States
92
SERVERSTORAGE
Front-end
Ports
Storage Area
Network
Host Application(s)
HBA HBA HBAHBA Host Bus
Adapter
PowerPath
A
B

PowerPath Licensing
• PowerPath
 Unlocks the full load balancing and path failover capabilities of PowerPath and
supports all EMC and certain non-EMC arrays
 Configures optimized default load balancing policies for EMC VMAX, VNX and
supported third-party arrays
• PowerPath/VE
 Full feature set specifically for virtual server environments based on VMware
vSphere and Windows Hyper-V
• PowerPath SE
 Backend (between switch and array) failover only
 Single HBA supported only
 No load balancing available
 Path Failover across two backend paths
93
PowerPath SE

During this lesson the following topics are covered:
• Benefits of PowerPath/VE
• Features and functionality of PowerPath/VE
• PowerPath/VE configuration in an ESXi environment
Lesson 2: Multipathing in ESXi
94

What is PowerPath/VE?
• PowerPath advanced multipathing capabilities for virtual environments hosted on
VMware vSphere
• Features include:
 Dynamic load balancing
 Device prioritization
 Automated optimization of server, storage, and path utilization
 Dynamic path failover and path recovery
 Monitoring performance and path status, I/O statistics and reports
 Automatic path testing
 Automatic path restore
95
x86 Architecture
PowerPath

PowerPath/VE Key Benefits
• Automates the virtual server-storage connection:
 Assures tunable, predictable performance
 Optimizes server, storage, and data-path utilization
 Maximizes data availability
• Uses standard management features across physical and virtual
environments:
 PowerPath/VE experience similar to rest of environment (physical servers)
 Common user experience across all classes of supported platforms
• Saves Money:
 Simple to implement application service levels for performance and
availability
 Eliminates need to monitor and rebalance the dynamic environment
 Reduces number of multi-pathing software variants to manage
96

• PowerPath manages complexity:
 Constantly adjusts I/O path usage
and changes in I/O loads from VMs
 Simplifies provisioning by pooling
all connections
 Optimizes overall I/O performance
in VMware ESX environments
PowerPath/VE Load Balancing
97
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
STORAGE
SAN
PowerPath PowerPath PowerPath PowerPath

PowerPath/VE supports:
• Changes to Permanent
Device Loss (PDL) handling
• VMFS 5 Filesystem
vSphere 5.0 Kernel Support
98
EMC and Non-EMC Storage
Host
Bus
Adapte
r
Host
Bus
Adapte
r
Host
Bus
Adapte
r
Host
Bus
Adapte
r
vSphere Kernel
PowerPath/VE
Guest OS
Applications
DBMS Management
Utilities
File System
Logical Volume Manager
PowerPath/VE
runs in the kernel

PowerPath/VE – Two Major Components
99
PowerPath Remote
Management Server
ESX Hosts
 PowerPath remote tools
(rpowermt) installed
 Used to remotely manage
PowerPath/VE on any number of
ESX hosts
 PowerPath software is installed

PowerPath Remote Management Server
• Supported platforms:
 Can be a physical or virtual host
 Check ESM for version compatibility
• Software:
 rpowermt
 Sends PowerPath management commands to a remote ESX host
 Is the only way to manage PowerPath/VE (no local commands)
 VMware vSphere CLI
 Installs PowerPath remotely onto an ESX host
 License file
 Uses rpowermt to register license with an ESX host
• Can manage any number of ESX hosts.
100
PowerPath Remote
Management Server

PowerPath Remote Tools
• Are installed on the PowerPath Remote Management Server
• Provide remote versions of existing PowerPath commands:
 Example:
 Local command used by previous versions of PowerPath:
powermt display dev=all
 Remote command, supported by PowerPath/VE:
rpowermt host=<hostname> display dev=all
• Note: Remote Tools version and PowerPath version must be
compatible:
 Possible to update either Remote Tools or PowerPath, without updating the
other
101

Installation and Configuration Steps
1. Upload Files to ESXi Server
2. Install PowerPath software on the ESXi server
3. Install PowerPath Remote Tools on PowerPath Remote
Management Server (Optional)
4. Configure and register license file
102

Upload Files to ESXi Server
103
Create
New Folder
Upload File
or Folder

Install PowerPath on ESXi
104
# cd /vmfs/volumes/Storage1/PPVE/
# ls
PowerPath_VE_5_7_for_VMWARE_vSphere-Install_SW_Bundle.zip
# unzip PowerPath_VE_5_7_for_VMWARE_vSphere-Install_SW_Bundle.zip
. . .
# esxcli software vib install --depot=/vmfs/volumes/Storage1/PPVE/
PowerPath_VE_5_7_for_VMWARE_vSphere-Install_SW_Bundle/EMCPower.
VMWARE.5.7.b173.zip
Installation Result
Message: The update completed successfully, but the system needs
to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: EMC_bootbank_powerpath.cim.esx_5.7.0.00.00-b173,
EMC_bootbank_powerpath.lib.esx_5.7.0.00.00-b173,
EMC_bootbank_powerpath.plugin.esx_5.7.0.00.00-b173
VIBs Removed:
VIBs Skipped:

Reboot ESXi Server
105

Verify PowerPath /VE Installation
106
# cd /opt/emc/powerpath/bin
/opt/emc/powerpath/bin # ls
powermt
/opt/emc/powerpath/bin # ./powermt version
EMC powermt for PowerPath (c) client Version 5.7 (build 173)
EMC PowerPath (c) Version 5.7 (build 173)
Warning: License not installed.
/opt/emc/powerpath/bin #

THANK YOU

VMware EMC Service Talk

More Related Content

What's hot

Similar to VMware EMC Service Talk

More from Yunchao (Kevin) Wang

VMware EMC Service Talk