Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SAN Extension Design and Solutions

1,550 views

Published on




SAN Extension Design and Solutions

Help your organization ensure data is backed up and available, both locally and remotely. Expert guidance on the pros and cons of various SAN design options, and which to choose
Learn from Cisco experts in this technical session as we cover the following:
•How to achieve the desired recovery time objective and recovery point objective for the business
•Pros and cons of various SAN extension solution designs, and which one to choose
•Cisco solutions and products for SAN extension using native Fibre Channel (FC), Fibre Channel over Ethernet (FCoE), and Fibre Channel over Internet Protocol (FCIP) protocols
•Best practices collected from a decade’s worth of experience with some of the largest deployments in the world
•Configuration guidelines/best practices to increase the return on your investment
•Guidelines for increasing performance and security while lowering solution costs

Published in: Technology

SAN Extension Design and Solutions

  1. 1. Design and Solutions SAN Extension Paresh Gupta, Technical Marketing Engineer, Cisco Mark Allen, Manager Technical Marketing, Cisco January 2017
  2. 2. Resilience Safety Restoration Business Continuity Industry wide Ecosystems Infrastructure Planning Competitive Edge Redundancy 16 member PortChannel Disaster Recovery Incident Management Cloud Storage Protection Customer SatisfactionStability Restoration Crisis Management Investment Protection Risk Why SAN Extension
  3. 3. Hope for the best, plan for the worst- Lee Child
  4. 4. Recovery Point and Recovery Time Objective Time Disaster Strikes Recovery Point Last Point Where Data in Usable State Recovery Time System Recovered and Operational How far back? How long to recover? Shorter RPO/RTO ̶ Higher $$$ ̶ Replication ̶ Hot standby systems Longer RPO/RTO ̶ Lower $$$ ̶ Tape backup/restore ̶ Cold standby systems
  5. 5. SAN Extension Minimize RPO Data should be in sync before & after disaster Minimize RTO Recovery should be quick Investment Protection Investment for more than a decade Choice of Protocol FC or FCoE or FCIP Acceptable Latency Latency within acceptable limit of Replication or Backup application Distance How far is the recovery site? $$ Security Link Encryption for security of data in motion Cost Design within the budget High Availability Increased availability Considerations for SAN Extension Design
  6. 6. SAN Extension Design Options
  7. 7. Cisco Multi-Protocol Product Portfolio 12+ Years of Proven NX-OS Operating System Cisco Prime Data Center Network Manager (DCNM) Cisco MDS 9700 48x16G Line-Rate FC LAN/SAN SAN COMPUTE Cisco UCS C-Series Rack Servers Cisco UCS B-Series Blade Servers Cisco UCS 6300 Series FI Cisco UCS 6200 UP Cisco Nexus 9000 Cisco Nexus 7000 Cisco Nexus 5600 Cisco Nexus 5500 Cisco Nexus 3000 Cisco Nexus 2000 24 x40G FCoE Cisco MDS 9250i Cisco MDS 9148S 48x10G Line-Rate FCoE Cisco MDS 9396S Nexus 5672UP-16G 16G FC: Nexus 2348UPQ 16G FC, 40G FCIP Consistent and Simplified Features, Management, and Programmability
  8. 8. Cisco MDS 9000 Switch Family 9RU MDS 9710 MDS 9706 4 module slots Up to 192 ports 14RU 26RU MDS 9718 8 module slots Up to 384 ports FCIP SAN Extension 16 module slots Up to 768 ports MDS 9148S MDS 9396S MDS 9250i 48 x 16G FC 48 x 10Gbps FCoE 24 x 40 Gbps FCoE 24 x 16G FC, 8 x 1/10 GE & 2 x 40 GE SAN Directors Director Modules Fabric Switches
  9. 9. Typical SAN Design • Dual fabric design : 2 Fibre Channel connections from Server to Storage • Multipath software provides high availability • Separate Access and Replication fabrics • Dual fabrics maintained over SAN extension Replication FabricReplication FabricAccess Fabric “B” Fabric “A” FabricDC Interconnect Network SiteA SiteB MDS MDS MDS MDS MDS MDS
  10. 10. Introducing Virtual SAN (VSAN) • Dual fabrics (E.g., yellow VSAN and red VSAN) over distance • Inter Switch Link (ISL) carry multiple VSANs (known as trunking) • Each VSAN maintains it’s own fabric services • FSPF: ‘Fabric Shortest Path first’ for route calculation • Name server, zoning database, etc. Replication VSANReplication VSANAccess VSAN “B” Fabric “A” FabricDC Interconnect Network SiteA SiteB MDS MDS MDS MDS VSANs – Increased redundancy, scalability and reduced cost
  11. 11. High Availability (HA) replication design • Client based protection by • PortChannel • Storage arrays • Rerouting by FSPF • Network based Protection by • Optical protection schemes Replication VSANReplication VSANAccess VSAN “B” Fabric “A” Fabric SiteA SiteB MDS MDS MDS MDS
  12. 12. Link HA via PortChannel • Multi-protocol support : FC, FCoE or FCIP • Up to 16 members in a port channel • Increased Resilience and availability • Single logical link • No FSPF re-calculations when members go down • Route member links over diverse geographic paths Replication VSANReplication VSANAccess VSAN “B” Fabric “A” Fabric SiteA SiteB MDS MDS MDS MDS
  13. 13. Extending optical FC SAN : B2B credit requirement Frame Size 1 Gbps 2 Gbps 4 Gbps 8 Gbps 10 Gbps 16 Gbps 512 Bytes 2 BB/km 4 BB/km 8 BB/km 16 BB/km 24 BB/km 32 BB/km 1024 Bytes 1 BB/km 2 BB/km 4 BB/km 8 BB/km 12 BB/km 16 BB/km 2112 Bytes 0.5 BB/km 1 BB/km 2 BB/km 4 BB/km 6 BB/km 8 BB/km B2B credit requirement increases with Distance Speed Frame size B2Bcredit requirement
  14. 14. SAN Extension over Dark Fiber Replication VSANReplication VSANAccess VSAN “B” Fabric “A” Fabric SiteA SiteB • 1/2/4/8/10/16 Gbps FC • Distance : limited due to optics and fiber cable • SW or LW (10 KM) or ER (40KM) optics. OM1, OM2, OM3 or OM4 cables • Client protection only : PortChannel / Storage arrays / Rerouting by FSPF • Loss of path reduces bandwidth of only one fabric by 50% • Cost: Low
  15. 15. SAN Extension over CWDM Network • Colored CWDM SFPs (8G FC) used in FC switches (no transponder required) • Distance : limited due to optics and fiber cable and dB loss in MUX (max 40 KM) • Client protection only : PortChannel / Storage arrays / Rerouting by FSPF • Loss of path reduces bandwidth of both “A” and “B” fabrics by 50% • No topology change (no FSPF recalculation) • Cost: Fair Replication VSANReplication VSANAccess VSAN “B” Fabric “A” Fabric SiteA SiteB MUX MUX MUX MUX MDS MDS MDS MDS
  16. 16. Dense Wavelength Division Multiplexing (DWDM) • Up to 32 channels per fiber • Longer Distance than CWDM : Use of Erbium-Doped Fiber Amplifiers (EDFA) • Multi Protocol Capability for data center to data center connectivity • 1, 2, 4, 8, 10 or 16 Gbps FC, FICON, GigE, 10GigE, ESCON, IBM GDPS • Client Protection : PortChannel / Storage arrays / Rerouting by FSPF • As well as Network Protection : Splitter / Line card Optical Splitter Protection Protected Lambda Optical Splitter Working Lambda MDS MDS Linecard or Y-Cable Protection Y-cable MDS MDS Single transponder required Dual transponders required, more expensive Protects against fiber breaks Protects against fiber breaks and Line card failure
  17. 17. SAN Extension over DWDM Network – Option 1 • Optical network sharing by both fabrics • Client protection via PortChannel – Recommended • Single fiber cut will not affect fabric • Loss of path reduces bandwidth of both “A” and “B” fabrics by 50% • Cost : High Replication VSANReplication VSANAccess VSAN “B” Fabric “A” Fabric SiteA SiteB DWDM Ring MDS MDS MDS MDS
  18. 18. SAN Extension over DWDM Network – Option 2 • Dedicated optical network per fabric • Client protection via PortChannel – Recommended • Single fiber cut will not affect fabric • Loss of path reduces bandwidth of only one fabric by 50% • Cost : High+ Replication VSANReplication VSANAccess VSAN “B” Fabric “A” Fabric SiteA SiteB DWDM Ring DWDM Ring MDS MDS MDS MDS
  19. 19. SAN Extension Technology Options Limited by Optics (Power Budget)Dark Fiber 1/2/4/8/10/16G FC, 10GE FCoE CWDM 1/2/4/8G FC, 10GE FCoE DWDM 1/2/4/8G FC, 10GE FCoE SONET/SDH 1/2/4G FC Data Center Campus Metro Regional National Increasing Distance Sync Sync Sync Limited by Optics (Power Budget) Limited by B2B_Credits Optical Async Global Sync Protection Client Network Cost Cost Cost Cost
  20. 20. Fibre Channel over Internet Protocol (FCIP) SiteB FC SANSiteA FC SAN IP Network FCIP Tunnel MDS MDS Single FSPF routing domain • IETF standard for Linking Fibre Channel SANs over IP (RFCs 3821 & 3643) • Point-to-point tunnel between FCIP link end-points
  21. 21. FCIP Frame Details • Segmentation and reassembly at default MTU of 1500 B (performance hit) • Recommendation: End to end IP MTU of 2300 bytes • All Cisco FCIP products support jumbo frames + + FCIP Header Ethernet Header IP Header TCP Header TCP Opts FC Frame Ethernet CRC32 14 20 20 12 28 4 94 EISL Hdr SOF 4 8 2172 VSAN Routing for TE port opt Hdr 0-16 RTTM is constantly measured for Round Trip Time 2270= 4
  22. 22. SAN Extension over FCIP • Client protection via PortChannel – Recommended • Portchannel individual FCIP links to separate Ethernet switches/routers • Each WAN link carries two FCIP tunnels • Global reach : Reliable delivery by TCP, No B2B credit requirement on FCIP link IP network Replication VSANReplication VSANAccess VSAN “B” Fabric “A” Fabric SiteA SiteB MDS MDS MDS MDS
  23. 23. SAN Extension Technology Options Limited by Optics (Power Budget)Dark Fiber 1/2/4/8/10/16G FC, 10GE FCoE CWDM 1/2/4/8G FC, 10GE FCoE DWDM 1/2/4/8G FC, 10GE FCoE SONET/SDH 1/2/4G FC Data Center Campus Metro Regional National Increasing Distance Sync Sync Sync Limited by Optics (Power Budget) Limited by B2B_Credits Optical Async Global Sync Protection Client Network Cost Cost Cost Cost 23 Async (WAN)MDS9000 FCIP GE, 10GE IP Sync (Metro Eth) Cost
  24. 24. Native FCoE SAN Extension FC DCB/FCoE Ethernet FCoE Replication VSANReplication VSANAccess VSAN “B” Fabric “A” Fabric SiteA SiteB Nexus Nexus Nexus • FCoE SAN Extension uses same design principles as FC or FCIP • Separate VLANs/VSANs for Host and replication traffic • Multiple geographical diverse paths • Client (PortChannel) or Network Protection (DWDM or SONET/SDH) • Distance depends on underlying media and Buffer (instead of B2B credits) • Typical FCoE SAN Extension rely on FC or FCIP to transport data over distance
  25. 25. Native FCoE SAN Extension FC DCB/FCoE Ethernet FCoE • FCoE Attached disk replication uses FC or FCIP transport network • SAN Extension design criteria based on FC or FCIP interconnect network Replication VSANReplication VSANAccess VSAN “B” Fabric “A” Fabric SiteA SiteB Nexus MDS 9250i MDS 9700 Nexus MDS 9250i MDS 9700
  26. 26. SAN Extension – Advanced Features
  27. 27. Link Layer Security IP NetworkDWDM Name: XYZ SSN: 1234567890 Amount: $123,456 Status: Gold @!$%!%!%!%%^& *&^%$#&%$#$%*!^ @*%$*^^^^%$@*) %#*@(*$%%%%#@ FC TrustSec IPSec Name: XYZ SSN: 1234567890 Amount: $123,456 Status: Gold Name: XYZ SSN: 1234567890 Amount: $123,456 Status: Gold @!$%!%!%!%%^& *&^%$#&%$#$%*!^ @*%$*^^^^%$@*) %#*@(*$%%%%#@ Primary DC Secondary DC Backup DC • Hardware supported, no additional latency • DH-CHAP used for peer authentication • Encryption: AES 128 bit key Fibre Channel TrustSec • Hardware support, no additional latency • Encryption: AES (128 or 256 bit key), DES (56 bit), 3DES (168 bit) IPSec
  28. 28. Application I/O Acceleration • Distance impacts performance of disk replication and tape backups • Latency due to distance is compounded by multiple round trips per command 28 I/O Accelerator (IOA) disk and tape over FC or FCIP Write Acceleration disk over FCIP (FCIP-WA) Tape Acceleration tape over FCIP (FCIP-TA) Solution MDS 9250i24/10 SAN Extension Module (SEM)
  29. 29. FCIP Data Compression • Compression increases link data capacity or reduce consumed bandwidth • Two compression modes available • Auto – Optimizes Compression based on bandwidth and data rate (Recommended) • Mode2 – Deflate based compression algorithm • Data Compressibility is data stream dependent • All nulls or ones → high compression (>30:1) • Random data (e.g., encrypted) → low compression (~1:1) • “Typical” rate is 4:1 (MDS 9250i and 24/10 SEM), but may vary considerably • Application throughput is the most important factor Supported in HW on MDS 9250i and 24/10 SEM without any additional latency
  30. 30. SAN Extension – Expert Advice by Mark Allen, Manager, Storage Technical Marketing
  31. 31. SAN Extension Best Practices Do not leave FCIP configuration to default values 0 5 10 15 20 25 30 35 40 45 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Exponential “Slow Start” (increase 2x pkts per RTT) LossLoss # Round Trip Times (RTT) PacketsSentperRoundTrip Low Throughput During This Period Linear “Congestion Avoidance” (MDS +2/cwnd per ack) (TCP +1/cwnd per ack) Congestion Window Halved on Packet Loss; Retransmission Signals Congestion…Slow Start Threshold Adjusted Traditional TCP MDS TCP implementation
  32. 32. SAN Extension Best Practices • In built Shaper sends at a rate consumable by the downstream path • Immediately sends at “minimum-bandwidth” rate (avoids early stages of traditional slow start) • Ramps up to “maximum-bandwidth” rate (using usual slow start and congestion avoidance methods) Configure TCP max and min bandwidth Dedicated link Shared link with other FCIP tunnel Shared link with non-storage traffic max= path bandwidth (BW) max = allocated for this tunnel max = (link BW – other traffic BW) min = 95% of max min = 80-95% of max min = 80-95% of max switch(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 900 round-trip-time-ms 10
  33. 33. SAN Extension Best Practices • TCP window size is directly proportional to RTT • Set correct RTT to fully utilize WAN pipe • Do not leave to default (1 ms) • Use ‘ping’ • OR ‘ips measure-rtt’ (preferred) to determine RTT Configure correct value for Round Trip Time (RTT) MDS9000# ping 192.168.20.2 PING 192.168.20.2 (192.168.20.2) 56(84) bytes of data. 64 bytes from 192.168.20.2: icmp_seq=1 ttl=254 time=0.740 ms 64 bytes from 192.168.20.2: icmp_seq=2 ttl=254 time=0.621 ms 64 bytes from 192.168.20.2: icmp_seq=3 ttl=254 time=17.8 ms MDS9000# ips measure-rtt 192.168.20.2 interface ipStorage 1/2 Round trip time is 111 micro seconds (0.11 milli seconds)
  34. 34. SAN Extension Best Practices • Set proper IP DSCP values to prioritize FCIP traffic Apply proper QoS policies when sharing link between storage and other traffic
  35. 35. FCIP Capacity Planning FCIP Link IP MAN / WAN SiteA SiteB MDS MDS • Multiple parameters must be tuned to keep the WAN pipe full • TCP Parameters (Window size, max BW, Round trip time, SACK…) • Outstanding I/Os • Transfer size • Standard traffic generating tools (like IOmeter) can be used • Requires test hosts and target
  36. 36. SAN Extension Tuner (SET) • Lightweight tool (Only SCSI Read and Writes) integrated with NxOS on MDS • Allows you to configure an unused iSCSI interface as a FC Initiator and Target • Generates custom traffic and reports parameters • I/O per second, Throughput, Round Trip Time, Compression ratio • Configured by CLI or GUI (Cisco DCNM aka Fabric Manager) FCIP Link IP MAN / WAN SiteA SiteB IPStorage1/1 IPStorage1/2 IPStorage1/2 IPStorage1/1 MDS MDS
  37. 37. Top 3 pitfalls to avoid 1. Do not over-complicate the design 2. Understand the QoS policy. Storage traffic must be subjected to tcp-max-bw, not the typical traffic policing 3. Understand when to use Inter-VSAN Routing
  38. 38. What Cisco has done really well Investment Protection 48 x 16G FC 48 x 10Gbps FCoE 24 x 40 Gbps FCoE 24 x 16G FC, 8 x 1/10 GE & 2 x 40 GE • All current (and future) modules can be used in any slot without any restrictions on MDS 9700 • Full FCIP backward interoperability is maintained • 24/10 SEM module can be connected to MDS 9250i, MDS 9222i and SSN-16 module (for MDS 9500) • Protects your investment for more than a decade • Not mandatory to upgrade remote locations just because you upgraded the primary location MDS 9250i 16 port GigE Storage Services Node (SSN-16) MDS 9222i FCIP
  39. 39. Summary
  40. 40. Advanced Features Extended Credits FCIP WACompression Encryption FCIP TA IOA QoS SET FCIP Tuning TCP max and min bandwidth Round Trip Time (RTT) SAN Extension for Business Continuity and Disaster Recovery

×