1.
using Cisco MDS 9396S
Designing Scalable SAN
Paresh Gupta, paregupt@cisco.com
Technical Marketing Engineer, Cisco
August 2015

2.
Cisco Multi-Protocol Product Portfolio:
SAN, LAN, and Compute
10+ Years of Proven NX-OS Operating System
Cisco Prime Data Center Network Manager (DCNM)
CONSISTENT AND SIMPLIFIED
Features, Management, and Programmability
Cisco UCS C-Series
Rack Servers
Cisco UCS B-Series
Blade Servers
Cisco UCS
Fabric
Interconnects
Cisco UCS
6248UP
Cisco UCS
6296UP
LAN/SAN
Cisco Nexus
9000
Cisco Nexus
7000
Cisco Nexus
5600
Cisco Nexus
5500
Cisco
Nexus 3000
Cisco
Nexus 2000
Cisco MDS
9700
48x16G
Line-Rate
FC
Cisco MDS
9250i
Cisco MDS
9148S
48x10G
Line-Rate
FCoE
SAN 16G COMPUTE
Cisco MDS
9396S
July
2015

3.
Methodology for a continuous life-cycle of a SAN
Prepare
Plan
Design
Implement
Operate
Optimize
Storage Area Network

4.
Propose an architecture to meet business
requirements and goals
Preparation phase Prepare
Top quality user experience and application performance
Maximize availability (Reduced downtime)
Lower TCO (Reduced CapEx & OpEx)
Expandable architecture to support future business growth
Define
Business
Goals
Create business plan with cost-benefit analysis

5.
Gather network requirements, existing gap
analysis and project plan
Planning phase
Human resource planning
Environmental planning (Power, Real State, etc)
Gap analysis, Risks and Constraints
Project management, timeline and mile stones
Create Project
Plan
Sourcing options (Vendor selection)
Plan

6.
MDS 9396S achieves your business goals
Prepare Plan

7.
• Best of both worlds: Performance of a Director in a fabric switch!
• 96 ports at line rate 16 Gbps Fibre Channel in compact 2 RU
• Non-blocking, non-oversubscribed architecture
• Consistent and Predictable performance across all ports
• No local switching dependency and restrictions
Top Quality User Experience & Application Performance
MDS 9700
48 port, 16Gbps
FC line card
MDS 9396S
delivered
by
High Performance
of
MDS 9396S

8.
• Two hot swappable fan trays
• Choice of reversible airflow
• Port side exhaust at FCS
• 2 hot swappable power
supplies
• 1+1 grid redundancy
• In Service Software Upgrade
• 16 member ports in single port-channel (Link Aggregation)
• High Availability is achieved by spreading members across different
ASIC, port-groups, etc
• Add or remove links as needed without traffic loss
• Dual Redundant Power Supply (for Grid Redundancy)
• Dual Redundant Fan Trays
Maximum availability
delivered
by
Robust Architecture
of
MDS 9396S

9.
• Base configuration: 48 ports or 96 ports
• 48 port base can be upgraded with 12 port on demand license
• Optional Licenses :
• Enterprise License for Extended B2B credits, QOS, IVR, VSAN
based access control
• DCNM License for Fabric Management
• DCNM + Enterprise license bundle
Expandable architecture: Support future business Growth
delivered
by
Flexible Licensing
48 Ports
12 Ports
Increment
12 Ports
Increment
12 Ports
Increment
12 Ports
Increment

10.
• Leadership across Collaboration, Security, High-End Routing,
Switching, Wireless, Cloud and Data Center Verticals, not just
point products
• Leadership across LAN, SAN & Compute in Data Center
• SAN: Industry partnership and Ecosystem
• SAN: Strong Investment Protection, backward compatibility
Solution provider, not just a point product vendor
delivered
by
Cisco’s ability to
develop and support
complete enterprise
IT infrastructure
Programmable: Integration with
Industry-Leading Cloud Platforms
EMC ViPR
Software Defined Storage
OpenStack
Microsoft System
Center VMM
IBM PowerVC
Automate Zoning for
faster application
deployments
User-Centric Models for Cloud-
Scale Workloads
Cisco MDS 9000 Family:
Extensive Industry Partnerships
Cisco
Channel Partners
MDS 9300/9200/9100 Only
Other Flash
Vendors
Storage Array interop only
The Power of Ecosystem
Vblock
FlexPod HDS
UCP
Select
VSPEX
Versa
Stack
UCS for Red
Hat
OpenStack
Smart
Stack
Pre-Packaged Converged
Infrastructures

11.
Tight integration between Cisco UCS and MDS 9396S
F-Port Channel
VSAN Trunking
F-Port Trunking for Multi-Tenancy
• Servers log in & participate in separate VSANs
• Any UCS blade can connect over any N-Port uplink across
which the VSAN is trunked
F-Port Channel for Resiliency
• Port Channel between UCS & MDS: No Port-Pinning
• No host relogin on individual link failure in Port Channel
• No Manual re-balancing when new link added or removed
Operational Simplicity: NX-OS, DCNM
• Common operating system for MDS & UCS Fabric
Interconnect (FI)
• Visibility of UCS FI and MDS in DCNM
• Cisco TAC as common support system
MDS 9396S
UCS FI
Cisco UCS and Cisco MDS 9000 Family: Better Together
White Paper
http://www.cisco.com/c/en/us/products/collateral/storage-networking/mds-9700-series-multilayer-directors/white-
paper-c11-734381.html

12.
Designing a network to meet Business and
technical Requirements
Design phase Design
High Availability
Scalability
Security
Manageability
Design the
network
Flexibility

13.
Designing small single-switch SAN using MDS 9396S
Consolidation of SAN Islands
E-mail
Database
BackupE-mail E-mail
Database
Start with 48 port base
license with single VSAN for
E-Mail Servers
Add 12 port license to
accommodate Database
servers
Add another 12 port license
to add Backup servers
MDS 9396S
Minimizes no. of SAN switches Centralized management
Blue VSAN
Red VSAN
Yellow VSAN
Reduced TCO with best user experience & application performance
Secure & Flexible
Design
48 – 96 Ports

14.
Designing small multi-switch SAN using MDS 9396S
Reduced TCO with support for business expansion
Secure & Flexible
Design
MDS 9396S MDS 9396S
• Deploy in N-Port Virtualization (NPV) or N-Port ID Virtualization (NPIV) mode to reduce
number of managed switches
• Maximize localization of servers and Storage on single switch: Reduced ISL traffic
BackupE-mail
MDS 9396S
MDS 9148S
E-mail Database
12 – 192 Ports

15.
Designing enterprise class SAN using MDS 9396S
Reduced OpEx & CapEx by reduction in no. of edge switches, power,
cooling &cabling requirements with readiness for future growth
Massive Scalable
Design
• MDS 9396S as Middle of Row (MoR) switch to consolidate server connections
• Even reduced number of managed devices with MDS 9396S as NPV & MDS 9700 as NPIV
• Quality of Service (QoS) for traffic prioritization
MDS 9700
MDS 9396S
MDS 9396S
MDS 9700
Edge-Core Architecture – Consolidation at Access Layer
1000s of Ports

16.
SAN Extension using MDS 9396S
Business Continuity and Disaster Recovery
Design to connect
distant SAN islands
• Up to 25 KM: Use of CWDM optics at 8G or ELW optics at 16G
• 512 KM @ 16G FC with DWDM infrastructure (Up to 4095 B2B credits per port with
enterprise license)
Primary
Site
MDS 9396S MDS 9396S
MDS 9250i MDS 9250i
Native Fibre Channel based
MAN / WAN
IP MAN / WAN for FCIP
Backup
Site
Disk replication, Tape backups
FC TrustSec
Encryption
IPSec
Encryption

17.
MDS 9396S achieves your business goals
Prepare Plan Design

18.
Quick provisioning of network as per
design
Implementation phase Implement
Connect the devices as per design
Configure the devices
Validate the configuration and connectivity
Training of support personals
Implement the
Design
Quick & automated implementation to minimize time to deploy

19.
• Intuitive approach to configure and Manage a Single Switch
• Point browser to the management IP address of the switch to access Device
Manager
Cisco Device Manager
Watch Full demo (and many other videos) at youtube.com/ciscomds9000

20.
• Unlike LAN switches, SAN switches require zoning for Servers and Storage
Array to communicate.
• Quick Config Wizard configures zoning within minutes
Quick Configuration Wizard
Expedite Return on Investment

21.
Quick Configuration Wizard

22.
As number of switches increase, Configuring Single
switch at a time is not a viable option

23.
Power on Auto Provisioning (POAP) on MDS 9396S
Rapid Error Free
Deployment
• Automation
• Consistency
• ‘Go-Live’ time in minutes
• No Human Errors
• Accuracy
• Reliability
MDS 9396S
…………………………………………………………………..
Provides:
•IP Address, Gateway
•Software Image
•Configuration File

24.
Maintain Network Health to support business
operations
Operating a SAN Operate
Provisioning of new Servers & Storage
Performance Monitoring
Health Monitoring
Incident Management
Day-to-Day
Operations
Fault Detection

25.
MDS 9396S achieves your business goals
Prepare Plan Design Operate

26.
• Options of zoning via Intuitive NX-OS CLI
• Zoning can be done easily via DCNM
• Zoning Automation available via Industry leading Platforms: Cisco UCS
Director, EMC ViPR, Microsoft System Center VMM, IBM Power PC,
OpenStack.
Simplified & Efficient Zoning operations using MDS 9396S

27.
Cisco Enhanced zoning on MDS 9396S
Secure and Resilient OperationsEnhanced Zoning
Serializes Configuration to Streamline Parallel Config Attempts from Multiple Users
HOST1
Users Acquire Fabric Lock
One at a Time to Make
Zone Config Changes
TARGET1
TARGET2
TARGET3
SWITCH1
SWITCH3
SWITCH2
SWITCH4
HOST2
HOST3
HOST4
TARGET4

28.
• 8 Initiators & 4 targets in single zone
create 132 entries (12 x (12-1)) within a
SAN switch
• Ideal number of entries = 32
• SAN admins pay the price on internal
inefficiency by creating 2-member zones.
Too much of overhead for day-to-day
operation!
• Solution: Cisco Smart Zoning
• Create single zone with all initiators and
targets. MDS 9396S is smart to create just
ideal number of entries (32)
Cisco Smart Zoning on MDS 9396S
Simpler, Efficient, IntuitiveSmart Zoning
I
T
I
I
I
I
I
I
I
T
T
T

29.
Visualizing the benefit of Smart Zoning on MDS 9396S
Smart Zoning
Requires only 1 zone with 12 hosts and 2 X-Brick
Zoneset name Smart-VDI vsan 200
zone name x-brick1 vsan 200
pwwn 15:10:10:10:10:10:10:01 initiator
pwwn 15:10:10:10:10:10:10:02 initiator
pwwn 15:10:10:10:10:10:10:03 initiator
pwwn 15:10:10:10:10:10:10:04 initiator
……..
pwwn 15:10:10:10:10:10:10:12 initiator
pwwn 25:20:20:20:20:20:20:01 target
pwwn 25:20:20:20:20:20:20:02 target
Traditional Zoning
(Requires 24 two-member zones)
Zoneset name VDI vsan 100
zone name host1-x-brick1a vsan 100
pwwn 10:10:10:10:10:10:10:01
pwwn 20:20:20:20:20:20:20:01
zone name host1-x-brick1b vsan 100
pwwn 10:10:10:10:10:10:10:01
pwwn 20:20:20:20:20:20:20:02
zone name host2-x-brick1a vsan 100
pwwn 10:10:10:10:10:10:10:02
pwwn 20:20:20:20:20:20:20:01
zone name host2-x-brick1b vsan 100
pwwn 10:10:10:10:10:10:10:02
pwwn 20:20:20:20:20:20:20:02
. . . . . . .
zone name host12-x-brick1a vsan 100
pwwn 10:10:10:10:10:10:10:12
pwwn 20:20:20:20:20:20:20:01
zone name host12-x-brick1b vsan 100
pwwn 10:10:10:10:10:10:10:12
pwwn 20:20:20:20:20:20:20:02
Adding another VDI cluster would require an
additional 72 zones to be configured per fabric.
Adding another VDI cluster would
require adding the new storage ports
and new hosts to the existing zone or
adding the 2 new storage pwwns to
the first zone and creating one
additional zone with all 4 storage ports
together with the new hosts.

30.
Resilient Operations on MDS 9396S Operate
• Faulty equipment, loose SFPs,
dirty or damaged cables, can
result in packets getting
corrupted
• CRC checking drops corrupted
frames from end devices or
internally corrupted frames
• FEC corrects frames corrupted
in-flight to preserve frames
Ingress
CRC
checking
Internal
CRC
checking
Forward
Error
Correction
Drop
frame
Drop
frame
CRC Detection and 16 Gbps FC Forward Error Correction (FEC)

31.
MDS 9396S for Self Healing SAN
Misbehaving device “slowly drains” frames destined to it. Unrelated flows and devices get affected
MDS auto-detects slow drain devices and takes recovery actions
Potential Reasons
for Slow-Drain Devices
• Speed Mismatch
• HBA issues
• Server Performance issues – application/OS
• Non graceful Virtual Machine exit
Affected
Host
Misbehaving
HBA
Detection
2.5 µs
Automatic Recovery
Immediate
Troubleshooting
DCNM Slow Drain
Diagnostics

32.
Detecting and Troubleshooting Slow Drain using DCNM
Automates
Troubleshooting
Reduces False
Positives
Shows Fluctuations
in Counters
Watch Full demo (and many other videos) at youtube.com/ciscomds9000

33.
Programmability using MDS 9396S
OpenStack Integration
Full benefits of OpenStack
along with seamless zoning
Embedded Event Manager (EEM)
NX-OS infrastructure to take
user defined events on actions
Native on-switch TCL Interpreter
Near-Infinite possibilities with
your own custom script
Scheduler
Capability to schedule any task on
the switch (like Linux cron)

34.
Programmability Use Case: Config Checkpoint
Config backup at remote
server after each change
Switchname_Timestamp.cfg
Functionality
Behind the
Scene
EEM catches
config change
event
Triggers TCL script on
switch bootflash
Script copies
running_config to
remote server

35.
Maximize the performance with proactive
management
Optimizing a SAN
Network Capacity Trending
Network Health Trending
Aim to improve uptime
Aim to reduce number of incidents
Day-to-Day
Operations
Take all the learnings to Prepare Phase
Optimize

36.
MDS 9396S Health Score on DCNM
HEALTH SCORE
TRENDING
Identify and Remediate
HEALTH SCORE
TRENDING
IF YOU CAN MEASURE IT, YOU CAN FIX IT
80% Health 60 % Health
Policy-Based Health Index
Quickly Determine Level of Risk
View Alerts Contributing to Low Score
Measure Success Overtime
Restore Health back Normal
MDS 9396S MDS 9396S

37.
MDS 9396S achieves your business goals
Prepare Plan Design OperateImplement Optimize

38.
• Current Infrastructure unable to support user growth and data explosion
• Planning for Access Network Consolidation to reduce CapEx
• Already have large UCS install base and growing
• Due to pressure on OpEx: Need for Programmability, Automation and Manageability
• Unhappy with long history of Fork Lift upgrades with every speed change.
Business Goals
• Best in class user experience: Support high demanding application (Today & Future)
• Investment Protection for 10+ years
• Minimize CapEx and OpEx
• Consolidated global IT strategy and architecture
Design under exploration: Large 16 Gbps FC customer
Prepare
Plan

39.
Design under exploration
MDS 9396S
MDS 9706
MDS 9710
. . . . . 80
• MDS 9706 as Storage Edge
• MDS 9710 as Core (Logically Server Edge)
• MDS 9396S as Top of Row (TOR) switch shared
among servers at multiple racks
• Single rack has 20 – 30 servers
• F-Port Channel between NPV MDS 9396S &
MDS 9710 NPIV for high availability
• Single NPIV MDS9710 will be attached to 80
NPV MDS 9396S in a single fabric
• POAP to provision hundreds of MDS 9396S
Design

40.
Customer meets Business Goals using Cisco MDS
MDS 9396S
MDS 9706
MDS 9710
. . . . . 80
Best in Class User
Experience
High performance,
non-oversubscribed,
no-local switching
Cisco MDS as choice for multiple
data centers across globe along with
Cisco Nexus and UCS
Consolidated global
IT strategy
Large CapEx saving by using MDS
9396S as Server Edge. OpEx saving
by POAP, DCNM
Minimize CapEx &
OpEx
MDS 9700 ready for 32 G FC & 40 G
FCoE today. ISLs can easily be
upgraded without forklift
Investment
Protection
By high performing, non-
oversubscribed, predictable,
consistent, no-local switching
architecture
Best in Class User
Experience

41.
• Existing Infrastructure unable to support user growth and data explosion
• Existing infrastructure has shown inconsistent and unpredictable behavior under load
• Existing infrastructure known to forward corrupt frames causing fabric wide disruption
Business Goals
• Increase efficiency of VDI application running at Point of Sale (POS) to reduce time for
each customer
• Stores located every few KMs. Maintain backup at central Data Center
• Reduced CapEx and OpEx
• Capacity for future expansion
Design under exploration: Flash Storage for VDI
Prepare
Plan

42.
Flash Storage for VDI using MDS 9396S
• MDS 9396S as SAN switch between VDI application
servers and Flash Storage
• Predictable & Consistent Performance provides
freedom to connect Storage & Servers to any port on
the switch without worrying local-switching complexity.
• Ingress CRC checking & dropping of corrupt frames
will limit failure domains just to the affected device
• Operational simplicity by Smart zoning
• End-to-End visibility (VM & LUN awareness) by
DCNM.
• MDS 9396S for long distance FC native synchronous
replication over dark fiber to central data center
Design
Access VSAN
Replication VSAN
Dark Fibre to central Data
Center – Continuous backup of
store operations
FC TrustSec Encryption

43.
MDS 9396S achieves Business Goals for Flash Storage
Best in Class User
Experience
High performance,
non-oversubscribed,
no-local switching
By unmatched licensing flexibility of MDS
9396S. Customer starts with 48-port base &
increments with 12-ports on-demand license
Capacity for
Future
Expansion
By consolidation of Access & Replication
SAN on same switch using Cisco VSANs.
OpEx saving by DCNM
Minimize
CapEx & OpEx
By synchronous replication between store &
Central Data Center. Long distance FC link
possible due to large number of B2B credits
on MDS 9396S
Backup at
Central Data
Center
By high performing, non-oversubscribed,
predictable, consistent, no-local switching
architecture of MDS 9396S. Cisco MDS +
UCS + EMC XtremeIO has been validated
for > 1M IOPS
Increased VDI
Application
Performance
Access VSAN
Replication VSAN
Dark Fibre to central Data
Center – Continuous backup of
store operations
FC TrustSec Encryption

44.
Summary

45.
Prepare
Plan
Design
Implement
Operate
Optimize
MDS 9396S
Coordinated Planning and Strategy
Assess
Readiness
Design the
Solution
Implement the Solution
Maintain
Network Health
Operational
Excellence

46.
2RU form factor
NPIV
No Oversubscription
POAP
Slow Drain Automatic Recovery
Robust Architecture
FEC
Industry wide EcoSystems
16 Gbps FC Line Rate
Affordable
Non-Disruptive Software Upgrade
Flexible Licensing
NPV
16 member PortChannel
Link Encryption
Predictable Performance
Redundant Fan Trays
Quick ConfigurationFEC Consistent Latency
Cisco MDS + UCS Synergies
Slow Drain detection at 2.5µs
MDS 9396SSmart Zoning
Deploy
today! 96 ports

47.
Reference

48.
B2B credit requirement increases with
Extending optical FC SAN : B2B credit requirement
Frame Size 1 Gbps 2 Gbps 4 Gbps 8 Gbps 10 Gbps 16 Gbps
512 Bytes 2 BB/km 4 BB/km 8 BB/km 16 BB/km 24 BB/km 32 BB/km
1024 Bytes 1 BB/km 2 BB/km 4 BB/km 8 BB/km 12 BB/km 16 BB/km
2112 Bytes 0.5 BB/km 1 BB/km 2 BB/km 4 BB/km 6 BB/km 8 BB/km
Distance Speed Frame size
B2Bcredit
requirement

49.
Understanding B2B credits on MDS 9396S
• All ports in a port group can have maximum of 500 B2B credits
• Enterprise license enables extended credits that means up to 4095 B2B per
port in a port group
• Best Practice: Avoid grouping all E ports in same port group/IOSlice
• Generic Formula: 0.5 B2B credits are required for every 1 KM distance at
1Gbps for standard FC frame (2112 bytes). Refer to previous slide
• CLI command: show port-resources module 1
Port Group
1
Port Group
2
Port Group
3
Port Group
4
Port Group
5
Port Group
6