Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB) Case Studies TechBook
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB) Case Studies TechBook

on

  • 6,127 views

This EMC Engineering TechBook focuses on case studies and provides a brief overview of FCoE, the supported configurations and features, and case studies to show how to incorporate EMC Connectrix B ...

This EMC Engineering TechBook focuses on case studies and provides a brief overview of FCoE, the supported configurations and features, and case studies to show how to incorporate EMC Connectrix B switches, Cisco Nexus switches, Blade Servers and Juniper QFX3500 switches, and Brocade VDX switches utilizing FCoE into an existing data center.

Statistics

Views

Total Views
6,127
Views on SlideShare
5,979
Embed Views
148

Actions

Likes
0
Downloads
509
Comments
0

10 Embeds 148

http://rrnamb.blogspot.in 110
http://rrnamb.blogspot.com 26
http://rrnamb.blogspot.de 2
http://rrnamb.blogspot.kr 2
http://rrnamb.blogspot.nl 2
http://rrnamb.blogspot.ie 2
http://www.rrnamb.blogspot.de 1
http://rrnamb.blogspot.co.uk 1
http://rrnamb.blogspot.ru 1
http://rrnamb.blogspot.com.ar 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB) Case Studies TechBook Document Transcript

  • 1. Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB) Case Studies Version 5.0 • EMC Connectrix B Setup Examples • Cisco Nexus Series, vPC Setup Examples • FCoE-Based Blade Server Solutions • Juniper QFX3500 Setup Examples • Brocade VDX Switches Setup Examples Mark Lippitt Erik Smith Erik Paine Aditya Nadkarni Benjamin Heydary
  • 2. Fibre Channel over Ethernet (FCoE) Case Studies TechBook2 Copyright © 2008–2014 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United State and other countries. All other trademarks used herein are the property of their respective owners. For the most up-to-date regulator document for your product line, go to EMC Online Support (https://support.emc.com). Part number H10524.5
  • 3. Fibre Channel over Ethernet (FCoE) Case Studies TechBook 3 Preface Chapter 1 Introduction to Fibre Channel over Ethernet Introduction ....................................................................................... 26 Benefits ............................................................................................... 28 Terminology....................................................................................... 29 Management tools............................................................................. 31 Cable management recommendations .......................................... 32 Protocols............................................................................................. 33 Supported connectivity options...................................................... 34 Configuration rules ...................................................................34 Connectivity options ..................................................................36 Chapter 2 EMC Connectrix B Setup Examples MP-8000B ........................................................................................... 46 MP-8000B overview....................................................................46 Management................................................................................48 POST and boot specifications....................................................48 MP-8000B supported features and topologies.............................. 49 FCoE interface .............................................................................50 Ethernet uplink interface...........................................................50 FC interface..................................................................................52 Ethernet end devices ..................................................................52 MP-8000B direct-connect topology................................................. 53 Existing topology .......................................................................54 Target topology...........................................................................56 Prerequisites ................................................................................57 Configuring the MP-8000B........................................................58 Contents
  • 4. Fibre Channel over Ethernet (FCoE) Case Studies TechBook4 Contents MP-8000B firmware upgrade procedure....................................... 90 Assumptions ...............................................................................90 Upgrading firmware..................................................................90 Troubleshooting the MP-8000B....................................................... 93 Fibre Channel FOS......................................................................93 CMSH...........................................................................................99 Connectivity troubleshooting.................................................102 VLAN troubleshooting............................................................102 DCBX troubleshooting.............................................................102 FCoE troubleshooting ..............................................................103 QoS troubleshooting ................................................................103 Link Aggregation (LAG) troubleshooting ............................103 ED-DCX-B........................................................................................ 104 Topology....................................................................................105 Compatibility ............................................................................105 Supported topologies...............................................................106 Installing the PB-DCX-24FCoE...............................................106 Configuring the PB-DCX-24FCoE..........................................107 Useful commands.....................................................................109 Chapter 3 Nexus Series Switches Setup Examples Nexus Series switches .................................................................... 113 Nexus 6004.................................................................................113 Nexus 6001.................................................................................114 Nexus 2248PQ Fabric Extender (FEX)...................................115 Nexus 5596.................................................................................116 Nexus 5548.................................................................................116 Nexus 5020.................................................................................117 Nexus 5010.................................................................................118 Nexus 4000.................................................................................118 Nexus 2232PP............................................................................119 Management..............................................................................121 VPN Routing and Forwarding (VRF)....................................122 Nexus 6004 supported features and topologies ......................... 123 Interfaces....................................................................................123 Features......................................................................................125 Nexus 6001 supported features and topologies ......................... 128 Interfaces....................................................................................128 Features......................................................................................130 Nexus 2248PQ FEX supported features and topologies ........... 133 Interfaces....................................................................................133 Features......................................................................................134
  • 5. 5Fibre Channel over Ethernet (FCoE) Case Studies TechBook Contents Nexus 5000 supported features and topologies.......................... 136 FCoE interface ...........................................................................137 Ethernet uplink interface .........................................................138 FC interface ................................................................................139 Ethernet end-device interface..................................................140 Nexus 2000 supported features and topologies.......................... 141 FCoE interface ...........................................................................141 Ethernet uplink interface .........................................................142 Ethernet end-device interface..................................................142 Nexus 6000 Series and Fabric Extenders topologies.................. 143 Multihop FCoE..........................................................................143 Leaf and Spine (Layer-2/Layer-3) fabric...............................144 High Performance Compute (HPC) and High Frequency Trading (HFT)........................................................145 Scalable data center access and aggregation with Fabric Extenders....................................................................................146 Fabric Extenders (FEX).............................................................147 Configuring 40GbE FCoE connectivity with Nexus 6000 Series ................................................................................................. 149 Prerequisites...............................................................................149 Summary of configuration steps.............................................150 Individual reference topologies..............................................152 Nexus 5000 direct-connect topology ............................................ 173 Summary of configuration steps.............................................173 Existing topology ......................................................................174 Target topology .........................................................................176 Prerequisites...............................................................................177 Configuring the virtual interfaces on Nexus 5020 ...............178 Etherchannel configuration on a Nexus 7000 switch...........203 Spanning tree configuration and verification on Nexus 5020..............................................................................................206 Nexus 7000, Nexus 5000, and MDS 9500 series topology ......... 210 Summary of configuration steps.............................................210 Target topology .........................................................................212 Prerequisites...............................................................................214 Configuring the Nexus 7000....................................................214 Configuring the Nexus 5548....................................................231 Configuring the MDS 9513 ......................................................238 Nexus 5000/Nexus 2232PP topology........................................... 242 Summary of configuration steps.............................................242 Existing topology example ......................................................243 Target topology .........................................................................244 Prerequisites...............................................................................245
  • 6. Fibre Channel over Ethernet (FCoE) Case Studies TechBook6 Contents Configuring the Nexus 5000 ...................................................246 Configuring Cisco VE_Ports ......................................................... 272 Configuring Cisco FCoE NPV ports ............................................ 274 Complex FCoE topologies............................................................. 277 Fixed port chassis .....................................................................277 Modular chassis........................................................................278 Nexus Series switches firmware upgrade procedure................ 280 Assumptions .............................................................................280 Upgrading firmware................................................................280 Troubleshooting the Nexus Series switches................................ 284 Show tech-support selections .................................................284 General connectivity troubleshooting ...................................285 Fibre Channel troubleshooting...............................................286 Hardware-related errors..........................................................288 High CPU issue troubleshooting............................................288 Gatos Port ASIC troubleshooting...........................................289 Altos Fabric ASIC troubleshooting........................................289 FCoE troubleshooting ..............................................................289 DCBX troubleshooting.............................................................290 Verify zoning configuration and that CNA WWPN is zoned active...............................................................................290 Verify CNA WWPN is registered in nameserver in your VSAN..........................................................................................290 Ethernet PortChannel troubleshooting .................................290 Verify PortChannels.................................................................291 Spanning-tree troubleshooting...............................................291 Multicast troubleshooting .......................................................292 Virtual PortChannel........................................................................ 293 Introduction...............................................................................293 vPC overview............................................................................299 vPC topologies overview ........................................................306 vPC topology considerations..................................................309 vPC case studies .......................................................................324 Chapter 4 Blade Server Solutions Setup Examples Introduction..................................................................................... 420 Cisco UCS ..................................................................................421 IBM BladeCenter.......................................................................423 Cisco UCS (B-Series)....................................................................... 426 Cisco UCS supported features and topology .......................426 UCS fabric interconnect initial configuration.......................432
  • 7. 7Fibre Channel over Ethernet (FCoE) Case Studies TechBook Contents Cisco UCS Fibre Channel Switch Mode configuration example.......................................................................................434 Cisco UCS Fibre Channel End Host mode configuration example.......................................................................................452 Storage provisioning.................................................................470 Directly attaching EMC storage to Cisco UCS Fabric Interconnect ...............................................................................477 Setting up and implementing VE_Port on Cisco UCS.........508 Setting up and implementing VF_Port on Cisco UCS.........534 Configuring UCS 10G iSCSI using VNX-2 arrays................563 IBM BladeCenter ............................................................................. 588 Brocade Converged 10 GbE Switch Module supported features and topologies ............................................................588 Brocade Converged 10 GbE configuration example............592 Cisco Nexus 4000 supported features and topologies.........609 Cisco Nexus 4001I configuration example ............................611 HP Virtual Connect FlexFabric and Flex-10 ................................ 643 Virtual Connect FlexFabric solution.......................................643 Flex-10 technology ....................................................................647 FlexFabric components and data center communication....648 Using FlexHBAs for FC SAN connectivity............................655 Creating Server Profiles ...........................................................657 Case study..................................................................................660 References ..................................................................................676 Chapter 5 Juniper QFX3500 Switches Setup Examples Juniper QFX3500 overview............................................................ 678 Juniper QFX3500 supported features and topologies................ 680 FCoE Interface ...........................................................................681 Ethernet uplink Interface .........................................................681 FC Interface................................................................................682 Ethernet end device Interface..................................................682 QFX3500 Gateway and Transit mode topology.......................... 683 Summary of configuration steps.............................................683 Existing topology ......................................................................685 Target topology .........................................................................685 Configuration steps......................................................................... 687 Configure Gateway mode switch (54.80) ..............................687 Configure transit mode switch (54.81)...................................694 Useful JUNOS commands.............................................................. 699
  • 8. Fibre Channel over Ethernet (FCoE) Case Studies TechBook8 Contents Chapter 6 Brocade VDX Switches Setup Examples Brocade VDX data center switches .............................................. 702 Ethernet fabrics .........................................................................702 VDX 6710 ...................................................................................703 VDX 6720 ...................................................................................703 VDX 6730 ...................................................................................703 Modes of operation ..................................................................704 Brocade VCS fabric technology .................................................... 706 Introduction...............................................................................706 Terminology..............................................................................708 VCS Ethernet fabric..................................................................708 Brocade VDX switches supported and unsupported features ............................................................................................. 710 Supported features and configurations.................................710 Unsupported features and configurations............................712 Brocade VDX switches in a VCS fabric case study .................... 714 Summary of configuration steps ............................................714 VCS LAN/SAN Convergence case study.............................714 References ........................................................................................ 729 Appendix A Installation Notes Installing the Nexus 5000............................................................... 732 Installing the MP-8000B................................................................. 737 Prerequisites ............................................................................. 737 Installing power to the MP-8000B......................................... 739 Interpreting POST results....................................................... 739 Installing the MLX.......................................................................... 746 Glossary
  • 9. Fibre Channel over Ethernet (FCoE) Case Studies TechBook 9 Title Page 1 Typical topology versus FCoE example using Cisco Nexus 5000........... 27 2 Nexus 2000 supported connectivity options.............................................. 36 3 Nexus 2248PQ supported connectivity options ........................................ 37 4 Nexus 4001i supported connectivity options............................................. 37 5 Nexus 5000 supported connectivity options.............................................. 38 6 Nexus 6000 supported connectivity options.............................................. 39 7 Nexus 61x0 supported connectivity options.............................................. 40 8 Nexus 7000 supported connectivity options.............................................. 41 9 MDS 9000 supported connectivity options ................................................ 42 10 Brocade 8000/DCX supported connectivity options................................ 43 11 Juniper QFX3500/3600 supported connectivity options.......................... 44 12 Switch layout example................................................................................... 47 13 Topology example.......................................................................................... 49 14 Existing topology example............................................................................ 54 15 Target topology .............................................................................................. 56 16 switchshow output......................................................................................... 96 17 ED-DCX-4S-B switch and PB-DCX-24FCoE module topology example............................................................................................................105 18 Straight-through configuration example .................................................. 120 19 Cross-connected configuration example................................................... 120 20 Topology example........................................................................................ 136 21 Nexus 2000 supported features and topologies....................................... 141 22 Cisco Nexus 6004/6001 40GbE Multihop FCoE topology example ..... 144 23 Cisco Nexus 6004/6001 Leaf and Spine topology example................... 145 24 Cisco Nexus 6004/6001 HPC and HFT topology example.................... 146 25 Scalable data center access and aggregation with Fabric Extenders example............................................................................................................147 26 Cisco Nexus 2000 Series Fabric Extenders design scenarios.................. 148 27 Reference topologies.................................................................................... 152 28 Existing topology example.......................................................................... 174 Figures
  • 10. Fibre Channel over Ethernet (FCoE) Case Studies TechBook10 Figures 29 Target topology example ............................................................................ 176 30 Virtual FC interface...................................................................................... 179 31 Target topology example ............................................................................ 180 32 Target topology example ............................................................................ 212 33 Existing topology example ......................................................................... 243 34 Target topology example ............................................................................ 244 35 VE_Ports topology ....................................................................................... 272 36 VN_Ports topology ...................................................................................... 274 37 Fixed port chassis configuration example................................................ 278 38 Modular chassis configuration example................................................... 279 39 STP example.................................................................................................. 295 40 PortChannel example .................................................................................. 297 41 vPC example................................................................................................. 300 42 Switch vPC/straight-through topology example ................................... 307 43 Switch vPC/cross-connect topology example......................................... 308 44 Host vPC direct-connect topology example............................................. 309 45 Switch vPC/straight-through Nexus 5000 to Nexus 2000 configuration...................................................................................................311 46 Supported FCoE topology example .......................................................... 313 47 Non-supported topology for FCoE example ........................................... 314 48 Switch vPC/straight-through Nexus 5000 to Nexus 4000 configuration...................................................................................................315 49 Switch vPC/cross-connect Nexus 5000 to Nexus 2000 configuration...................................................................................................318 50 Switch vPC/cross-connect Nexus 5000 to Nexus 4000 configuration...................................................................................................320 51 Supported FCoE topology .......................................................................... 322 52 Straight-through topology example.......................................................... 326 53 Existing topology example ......................................................................... 328 54 Target topology example ............................................................................ 329 55 Ethernet connectivity with vPC and FEX Straight-Through topology example...........................................................................................336 56 SAN connectivity with vPC and FEX Straight-Through topology example............................................................................................................348 57 Cross-connect topology example............................................................... 358 58 Existing topology example ......................................................................... 361 59 Target topology example ............................................................................ 362 60 Direct-connect topology example.............................................................. 370 61 Existing topology example ......................................................................... 372 62 iSCSI target topology................................................................................... 374 63 FCoE vPC direct connect target topology ................................................ 378 64 Unisphere navigation and user interface.................................................. 393 65 Unisphere navigation and user interface.................................................. 410
  • 11. 11Fibre Channel over Ethernet (FCoE) Case Studies TechBook Figures 66 Cisco Unified Computer System (UCS) .................................................... 421 67 Brocade Converged 10 GbE Switch Module (Brocade 8470) ................. 423 68 IBM H series chassis, rear view .................................................................. 424 69 Target topology example............................................................................. 427 70 Connection between Fabric Extender and Fabric Interconnect............. 428 71 Existing topology.......................................................................................... 436 72 Target topology............................................................................................. 437 73 Change the mode.......................................................................................... 438 74 Enable port channeling and Trunking....................................................... 439 75 Create VSAN ................................................................................................. 440 76 Disable default zoning................................................................................. 441 77 Create Storage VSAN................................................................................... 442 78 SAN tab.......................................................................................................... 443 79 Add Ports screen........................................................................................... 444 80 Enable Port-Channel .................................................................................... 445 81 Assign VSAN to vHBAs .............................................................................. 446 82 Assign VSAN to storage FC and FCoE interfaces.................................... 447 83 Existing topology.......................................................................................... 455 84 Target topology............................................................................................. 456 85 Set chassis discovery policy ........................................................................ 457 86 Change connectivity policy......................................................................... 458 87 Create VSAN in SAN cloud ........................................................................ 459 88 Create VSAN ................................................................................................. 459 89 Assign VSAN to vHBA................................................................................ 460 90 Assign uplink FC interface.......................................................................... 461 91 Enable port channel and trunking ............................................................. 462 92 Create FC port channels .............................................................................. 463 93 Add Ports....................................................................................................... 463 94 Assign VSAN to port channel..................................................................... 464 95 CLARiiON Connectivity Status window.................................................. 471 96 Register Initiator Record dialog box.......................................................... 471 97 EMC Symmetrix Manager Console, Tasks ............................................... 473 98 Masking Wizard Summary window ......................................................... 475 99 Existing topology example.......................................................................... 478 100 Target topology example............................................................................. 479 101 Target topology............................................................................................. 509 102 Target topology............................................................................................. 535 103 Configuring UCS 10G iSCSI using VNX-2 arrays example.................... 564 104 Topology example........................................................................................ 589 105 Existing topology.......................................................................................... 593 106 Target topology, example 1......................................................................... 594 107 Target topology, example 2......................................................................... 596 108 AMM I/O Module Power/Restart window ............................................ 598
  • 12. Fibre Channel over Ethernet (FCoE) Case Studies TechBook12 Figures 109 I/O Module Configuration window......................................................... 599 110 Nexus 4000 topology example ................................................................... 609 111 Existing topology ......................................................................................... 613 112 Target topology ............................................................................................ 614 113 AMM I/O Module Admin/Power/Restart window............................. 618 114 AMM I/O Module Configuration window ............................................. 619 115 FlexFabric interconnect module................................................................. 644 116 FlexFabric adapters...................................................................................... 646 117 FlexNIC.......................................................................................................... 647 118 Communication between FlexFabric technology in HP BladeServer enclosure and the data center.......................................................................649 119 Mapping mode ............................................................................................. 651 120 Mapping Mode used for Shared Uplinks example................................. 652 121 Tunneling mode ........................................................................................... 653 122 Tunneling Mode used for Dedicated Uplinks example ......................... 654 123 High-level target topology example.......................................................... 660 124 Specific target topology example............................................................... 661 125 Edit SAN Fabric screen for defining B300_b5X1X2 fabric ..................... 663 126 Edit SAN Fabric screen for defining B300_b6X1X2 fabric ..................... 664 127 Edit SAN Fabric screen for defining C9148_b5X3X4 fabric................... 665 128 Edit SAN Fabric screen for defining C9148_b6X3X4 fabric................... 666 129 All SAN fabrics and link status.................................................................. 667 130 Defining the Server Profile for Blade 2 ..................................................... 668 131 Configuring the Server Profile for Blade 2 screen................................... 669 132 Defining the Server Profile for Blade 15 screen ....................................... 670 133 Configuring the Server Profile for Blade 15 screen................................. 670 134 Status of internal/external ports on FlexFabric module, Bay 5 ............ 671 135 Status of internal/external ports on FlexFabric module, Bay 6 ............ 672 136 Status between FlexFabric adapter ports in Server 2 and the external Brocade SAN switch.......................................................................673 137 Status between FlexFabric adapter ports in Server 15 and the external Cisco SAN switch............................................................................674 138 Brocade Name Server FlexHBA port logins, Server 2 ............................ 675 139 Cisco Name Server FlexHBA port logins, Server 15............................... 675 140 Juniper QXF3500 overview......................................................................... 680 141 Existing topology ......................................................................................... 685 142 Target topology ............................................................................................ 686 143 Classic Ethernet and corresponding VCS Fabric architecture............... 707 144 Source topology example............................................................................ 715 145 Target topology ............................................................................................ 716 146 Detailed topology......................................................................................... 718 147 Attaching a front rackmount bracket to a Nexus 5020 switch .............. 733 148 Installing the slider rails.............................................................................. 734
  • 13. 13Fibre Channel over Ethernet (FCoE) Case Studies TechBook Figures 149 Sliding the chassis into the rack.................................................................. 735 150 Attaching the switch to the rack................................................................. 736 151 LEDs on the port side of MP-8000-B.......................................................... 742 152 LEDs on the non-port side of MP-8000-B.................................................. 742
  • 14. Fibre Channel over Ethernet (FCoE) Case Studies TechBook14 Figures
  • 15. Fibre Channel over Ethernet (FCoE) Case Studies TechBook 15 Title Page 1 Acronyms .........................................................................................................29 2 IP address and Domain ID assignment ......................................................213 3 Host 1 information ........................................................................................213 4 Storage 1 information ...................................................................................213 5 Port groups .....................................................................................................216 6 Port groups .....................................................................................................224 7 LACP compatibility ......................................................................................299 8 Parameters ......................................................................................................304 9 Default values for keepalive messages ......................................................306 10 Supported protocols ......................................................................................325 11 Supported protocols ......................................................................................357 12 Supported protocols ......................................................................................369 13 Supported switch modes and port types ...................................................422 14 Brocade and IBM Converged Module part numbers ...............................423 15 Mapping and Tunneling mode comparison .............................................655 16 VCS terminology ...........................................................................................708 17 Nexus 5020 switch rackmount kit ...............................................................732 18 Power supply specifications ........................................................................738 19 LED patterns ..................................................................................................743 20 LEDs on non-port side of switch .................................................................744 Tables
  • 16. Fibre Channel over Ethernet (FCoE) Case Studies TechBook16 Tables
  • 17. Fibre Channel over Ethernet (FCoE) Case Studies TechBook 17 Preface The former EMC Engineering Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB) Concepts and Protocol TechBook has been divided into two TechBooks. This TechBook focuses on case studies and provides a brief overview of Fibre Channel over Ethernet (FCoE), the supported configurations and features, and case studies to show how to incorporate EMC Connectrix B switches, Cisco Nexus switches, and Blade Servers utilizing FCoE into an existing data center. The current Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook provides basic information to better understand the various aspects and protocols involved with a typical Ethernet environment. FCoE connectivity and storage in an FCoE environment are discussed. Information on RecoverPoint and Celerra MPFS as solutions in an FCoE environment is included. Basic troubleshooting techniques are also provided. This TechBook can be found at http://elabnavigator.EMC.com, under the Topology Resource Center tab. E-Lab would like to thank all the contributors to this document, including EMC engineers, EMC field personnel, and partners. Your contributions are invaluable. As part of an effort to improve and enhance the performance and capabilities of its product lines, EMC periodically releases revisions of its hardware and software. Therefore, some functions described in this document may not be supported by all versions of the software or hardware currently in use. For the most up-to-date information on product features, refer to your product release notes. If a product does not function properly or does not function as described in this document, please contact your EMC representative.
  • 18. 18 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Preface Note: This document was accurate at publication time. New versions of this document might be released on EMC Online Support at https://support.emc.com. Check to ensure that you are using the latest version of this document. Audience This TechBook is intended for EMC field personnel, including technology consultants, and for the storage architect, administrator, and operator involved in acquiring, managing, operating, or designing a networked storage environment that contains EMC and host devices. EMC Support Matrix and E-Lab Interoperability Navigator For the most up-to-date information, always consult the EMC Support Matrix (ESM), available through E-Lab Interoperability Navigator at http://elabnavigator.EMC.com. Included in the ELN are: ◆ The EMC Support Matrix, a complete guide to interoperable, and supportable, configurations. ◆ Subset matrices for specific storage families, server families, operating systems or software products. ◆ Host connectivity guides for complete, authoritative information on how to configure hosts effectively for various storage environments. Consult the Internet Protocol pdf for EMC's policies and requirements for the EMC Support Matrix. Related documentation ◆ The following documents, including this one, are available through the E-Lab Interoperability Navigator at http://elabnavigator.EMC.com. These documents are also available at the following location: http://www.emc.com/products/interoperability/topology-resource-center.htm • Backup and Recovery in a SAN TechBook • Building Secure SANs TechBook • Extended Distance Technologies TechBook • Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB) Concepts and Protocols TechBook • Fibre Channel SAN Topologies TechBook • iSCSI SAN Topologies TechBook • Networked Storage Concepts and Protocols TechBook • Networking for Storage Virtualization and RecoverPoint TechBook • WAN Optimization Controller Technologies TechBook
  • 19. Fibre Channel over Ethernet (FCoE) Case Studies TechBook 19 Preface • EMC Connectrix SAN Products Data Reference Manual • Legacy SAN Technologies Reference Manual • Non-EMC SAN Products Data Reference Manual ◆ EMC Support Matrix, available through E-Lab Interoperability Navigator at http://elabnavigator.EMC.com ◆ RSA security solutions documentation, which can be found at http://RSA.com > Content Library All of the following documentation and release notes can be found at EMC Online Support at https://support.emc.com. Hardware documents and release notes include those on: ◆ Connectrix B series ◆ Connectrix M series ◆ Connectrix MDS (release notes only) ◆ Symmetrix ◆ VNX series ◆ CLARiiON ◆ Celerra Software documents include those on: ◆ EMC Ionix ControlCenter ◆ RecoverPoint ◆ Invista ◆ TimeFinder ◆ PowerPath The following E-Lab documentation is also available: ◆ Host Connectivity Guides ◆ HBA Guides For Cisco, Brocade, and HP documentation, refer to the vendor’s website. ◆ http://cisco.com ◆ http://brocade.com ◆ http://hp.com
  • 20. 20 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Preface Authors of this TechBook This TechBook was authored by Mark Lippitt, Erik Smith, Erik Paine, and Shreedhan Nikam, with contributions from the following EMC employees: Kieran Desmond, Ger Halligan, and Ron Stern, along with other EMC engineers, EMC field personnel, and partners. Mark Lippitt is a Technical Director in EMC E-Lab with over 30 years experience in the storage industry, including Engineering and Marketing roles at Data General, Tandem Computers, and EMC. Mark initiated and led the Stampede project in 1997, which became EMC's first Connectrix offering. Mark is an active T11 participant, a committee within the InterNational Committee for Information Technology Standards, responsible for Fibre Channel Interfaces. Erik Smith is a Consulting Technologist for the Connectrix business unit within EMC Engineering. Over the past 13 years, Erik has held various technical roles in both EMC Engineering and Technical Support. Erik has authored and coauthored several EMC TechBooks. Erik is also a member of T11. Erik Paine is a Principal Systems Integration Engineer and has been with EMC for over 12 years. Erik transferred to E-Lab with a strong networking background spanning over 20 years, including time spent at BBN Inc., Tufts University, and numerous engineering roles within EMC. Erik is using his networking and Ethernet knowledge to help qualify and integrate the emerging storage technologies utilizing Ethernet as a medium. Aditya Nadkarni is a Senior Systems Integration Engineer and has been working with EMC's E-Lab for over 7 years. Aditya qualifies new Brocade-based FC switch hardware, firmware, and management application. In addition, Adi is involved in multi-vendor switch interoperability and blade server embedded module qualification projects which cover FC switch modules, pass-through modules, NPIV gateways and, more recently, the FCoE-based convergence modules. Benjamin Heydary is a Principal Systems Integration Engineer with over 30 years of experience in the storage industry. Ben has been with EMC for over 16 years, 14 with EMC E-Lab. Ben has worked on many different storage technologies integrations and has authored numerous White Papers, Technical Notes, and TechBooks.
  • 21. Fibre Channel over Ethernet (FCoE) Case Studies TechBook 21 Preface Conventions used in this document EMC uses the following conventions for special notices: IMPORTANT An important notice contains information essential to software or hardware operation. Note: A note presents information that is important, but not hazard-related. Typographical conventions EMC uses the following type style conventions in this document. Normal Used in running (nonprocedural) text for: • Names of interface elements, such as names of windows, dialog boxes, buttons, fields, and menus • Names of resources, attributes, pools, Boolean expressions, buttons, DQL statements, keywords, clauses, environment variables, functions, and utilities • URLs, pathnames, filenames, directory names, computer names, links, groups, service keys, file systems, and notifications Bold Used in running (nonprocedural) text for names of commands, daemons, options, programs, processes, services, applications, utilities, kernels, notifications, system calls, and man pages Used in procedures for: • Names of interface elements, such as names of windows, dialog boxes, buttons, fields, and menus • What the user specifically selects, clicks, presses, or types Italic Used in all text (including procedures) for: • Full titles of publications referenced in text • Emphasis, for example, a new term • Variables Courier Used for: • System output, such as an error message or script • URLs, complete paths, filenames, prompts, and syntax when shown outside of running text Courier bold Used for specific user input, such as commands
  • 22. 22 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Preface Where to get help EMC support, product, and licensing information can be obtained on the EMC Online Support site as described next. Note: To open a service request through the EMC Online Support site, you must have a valid support agreement. Contact your EMC sales representative for details about obtaining a valid support agreement or to answer any questions about your account. Product information For documentation, release notes, software updates, or for information about EMC products, licensing, and service, go to the EMC Online Support site (registration required) at: https://support.EMC.com Technical support EMC offers a variety of support options. Support by Product — EMC offers consolidated, product-specific information on the Web at: https://support.EMC.com/products The Support by Product web pages offer quick links to Documentation, White Papers, Advisories (such as frequently used Knowledgebase articles), and Downloads, as well as more dynamic content, such as presentations, discussion, relevant Customer Support Forum entries, and a link to EMC Live Chat. EMC Live Chat — Open a Chat or instant message session with an EMC Support Engineer. Courier italic Used in procedures for: • Variables on the command line • User input variables < > Angle brackets enclose parameter or variable values supplied by the user [ ] Square brackets enclose optional values | Vertical bar indicates alternate selections — the bar means “or” { } Braces enclose content that the user must specify, such as x or y or z ... Ellipses indicate nonessential information omitted from the example
  • 23. Fibre Channel over Ethernet (FCoE) Case Studies TechBook 23 Preface eLicensing support To activate your entitlements and obtain your Symmetrix license files, visit the Service Center on https://support.EMC.com, as directed on your License Authorization Code (LAC) letter e-mailed to you. For help with missing or incorrect entitlements after activation (that is, expected functionality remains unavailable because it is not licensed), contact your EMC Account Representative or Authorized Reseller. For help with any errors applying license files through Solutions Enabler, contact the EMC Customer Support Center. If you are missing a LAC letter, or require further instructions on activating your licenses through the Online Support site, contact EMC's worldwide Licensing team at licensing@emc.com or call: ◆ North America, Latin America, APJK, Australia, New Zealand: SVC4EMC (800-782-4362) and follow the voice prompts. ◆ EMEA: +353 (0) 21 4879862 and follow the voice prompts. We'd like to hear from you! Your suggestions will help us continue to improve the accuracy, organization, and overall quality of the user publications. Send your opinions of this document to: techpubcomments@emc.com Your feedback on our TechBooks is important to us! We want our books to be as helpful and relevant as possible. Send us your comments, opinions, and thoughts on this or any other TechBook to: TechBooks@emc.com
  • 24. 24 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Preface
  • 25. Introduction to Fibre Channel over Ethernet 25 1 This chapter provides an introduction to Fibre Channel over Ethernet (FCoE) and includes the following information: ◆ Introduction ........................................................................................ 26 ◆ Benefits ................................................................................................ 28 ◆ Terminology........................................................................................ 29 ◆ Management tools.............................................................................. 31 ◆ Cable management recommendations............................................ 32 ◆ Protocols .............................................................................................. 33 ◆ Supported connectivity options....................................................... 34 Note: More detailed Fibre Channel over Ethernet information can be found in the Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook at http://elabnavigator.EMC.com, under the Topology Resource Center tab. Information in this TechBook includes FCoE and Ethernet basics, EMC storage, RecoverPoint and Celerra MPFS as solutions in an FCoE environment, and troubleshooting basic FCoE and CEE problems. Introduction to Fibre Channel over Ethernet
  • 26. 26 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Introduction to Fibre Channel over Ethernet Introduction I/O consolidation has been long sought by the IT industry to unify the multiple transport protocols in the data center. This section provides a basic introduction to Fibre Channel over Ethernet (FCoE), which is an approach to I/O consolidation that has been defined by the FC-BB-5 T11 work group. I/O consolidation, simply defined, is the ability to carry different types of traffic, having different traffic characteristics and handling requirements, over the same physical media. I/O consolidation’s most difficult challenge is to satisfy the requirements of different traffic classes within a single network. Since Fibre Channel is the dominant storage protocol in the data center, any viable I/O consolidation solution for storage must allow for the FC model to be seamlessly integrated. FCoE meets this requirement in part by encapsulating each Fibre Channel frame inside an Ethernet frame. The goal of FCoE is to provide I/O consolidation over Ethernet, allowing Fibre Channel and Ethernet networks to share a single, integrated infrastructure, thereby reducing network complexities in the data center. An example is shown in Figure 1 on page 27. FCoE consolidates both SANs and Ethernet traffic onto one Converged Network Adapter (CNA), eliminating the need for using separate Host Bus Adapters (HBAs) and Network Interface Cards (NICs).
  • 27. Introduction 27 Introduction to Fibre Channel over Ethernet . Figure 1 Typical topology versus FCoE example using Cisco Nexus 5000 Note: More detailed Fibre Channel over Ethernet information can be found in the Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook at http://elabnavigator.EMC.com, under the Topology Resource Center tab. LAN SAN A Today SAN B LAN Data Centre Ethernet and FCoE Ethernet SAN A I/O Consolidation with FCoE SAN B FC GEN-001008
  • 28. 28 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Introduction to Fibre Channel over Ethernet Benefits The Fibre Channel portion of FCoE appears as normal Fibre Channel to a host or a switch, and therefore to a user. It is based completely on the FC model, which makes it easy to understand, manage, and troubleshoot. A major value is that FCoE uses Ethernet hardware to deliver an enterprise storage solution, while also using the existing FC management infrastructure. The benefits of FCoE include: ◆ Becomes part of the Fibre Channel architecture, allowing for: • Seamless integration with existing FC SANs • Uses existing FC SAN admin tools and workflows ◆ Requires no gateway • Since the FC frame is untouched, the operation is completely stateless ◆ Provides the following current functions and services, allowing for a smooth transition: • Zoning • dNS (distributed Name Server) • RSCN (Registered State Change Notification) • FSPF (Fibre Channel Shortest Path First) • Management tools • Storage and server virtualization Further benefits include: ◆ Fewer cables, simplifying cable management ◆ Fewer adapters and switch ports, saving in power, equipment, and cooling costs
  • 29. Terminology 29 Introduction to Fibre Channel over Ethernet Terminology Table 1 provides commonly used acronyms. Table 1 Acronyms (page 1 of 2) Acronym Definition ACL access control list CEE Converged Enhanced Ethernet (Deprecated, see “DCB”) CNA Converged Network Adapter CRC Cyclical Redundancy Check DA Destination MAC Address DCB Data Center Bridge DCBX Data Center Bridging Capability eXchange Protocol DCFM Data Center Fabric Manager ETHv2 Ethernet Version 2 EOF End of Frame ETS Enhanced Transmission Selection FCF FCoE Forwarder FCoE Fibre Channel over Ethernet FPMA Fabric Provided MAC Address IP Internet Protocol LACP Link Aggregation Control Protocol LAG Link Aggregation Group LAN Local Area Network LLDP Link Layer Discovery Protocol MAC Media Access Control MSTP Multiple Spanning Tree Protocol NPV N_Port virtualization
  • 30. 30 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Introduction to Fibre Channel over Ethernet NPIV N_Port ID virtualization NIC Network Interface Card PFC Priority Flow Control QoS Quality of Service SA Source MAC Address SAN Storage Area Network SPMA Server Provided MAC Address STP Spanning Tree Protocol RSTP Rapid Spanning Tree Protocol VE Virtual E port VF Virtual Fabric port VFC Virtual Fibre Channel VLAN Virtual LAN (ID) WAN Wide Area Network Table 1 Acronyms (page 2 of 2) Acronym Definition
  • 31. Management tools 31 Introduction to Fibre Channel over Ethernet Management tools The management tools used to manage FCoE and Fibre Channel environments are similar. CNA management tools include: ◆ Emulex — HBAnywhere/OneCommand Manager ◆ QLogic — SANSurfer ◆ Brocade — HCM Switch management tools include: ◆ Brocade • Fabric OS CLI – For configuration of Fibre Channel features • CMSH (CEE management shell) CLI – For configuration of Converged Enhanced Ethernet (CEE) features • CMCNE - Connectrix Manager Converged Network Edition ◆ Cisco • Fabric Manager – For configuration of Fibre Channel features • NX-OS CLI – For configuration of CEE features • Data Center Network Manager (DCNM)
  • 32. 32 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Introduction to Fibre Channel over Ethernet Cable management recommendations Consider the following recommendations for cable management. The minimum bend radius for a 50 micron cable is 2 inches under full tensile load and 1.2 inches with no tensile load. Cables can be organized and managed in a variety of ways, for example, using cable channels on the sides of the cabinet or patch panels to minimize cable management. Following is a list of recommendations: Note: You should not use tie wraps with optical cables because they are easily overtightened and can damage the optic fibers. ◆ Plan for rack space required for cable management before installing the switch. ◆ Leave at least 1 m (3.28 ft) of slack for each port cable. This provides room to remove and replace the switch, allows for inadvertent movement of the rack, and helps prevent the cables from being bent to less than the minimum bend radius. ◆ If you are using Brocade ISL Trunking, consider grouping cables by trunking groups. The cables used in trunking groups must meet specific requirements, as described in the Fabric OS Administrator’s Guide. ◆ For easier maintenance, label the fiber optic cables and record the devices to which they are connected. ◆ Keep LEDs visible by routing port cables and other cables away from the LEDs. ◆ Use hook and loop style straps to secure and organize fiber optic cables.
  • 33. Protocols 33 Introduction to Fibre Channel over Ethernet Protocols FCoE is two different protocols: ◆ FCoE — Data plane protocol This protocol is data intensive and requires lossless Ethernet. It is typically implemented in hardware and is used to carry most of the FC frames and all the SCSI traffic. ◆ FIP (FCoE Initialization Protocol) — Control plane protocol This is not data intensive. It is typically implemented in software, and used to discover FCoE capable devices connected to an Ethernet network and to negotiate capabilities. For more information on FIP, refer to the “FCoE Initialization Protocol (FIP)” section, "Introduction to Fibre Channel over Ethernet" chapter, in the Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB) Concepts and Protocols TechBook at http://elabnavigator.EMC.com, under the Topology Resource Center tab. The following list contains a few of the technologies and protocols required to enable I/O consolidation: ◆ Converged Network Adapters ◆ Fibre Channel Forwarders ◆ Priority Flow Control and PAUSE ◆ Data Center Bridging (DCB) ◆ Data Center Bridging Capability eXchange Protocol (DCBX) Each of these are discussed further in the "Introduction to Fibre Channel over Ethernet" chapter in the Fibre Channel over Ethernet (FCoE) Data Center Bridging (DCB) Concepts and ProtocolsTechBook at http://elabnavigator.EMC.com, under the Topology Resource Center tab. This document also provides information on encapsulation and connectivity options.
  • 34. 34 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Introduction to Fibre Channel over Ethernet Supported connectivity options This section provides the current configuration rules as well as graphical illustrations of the connectivity options currently supported by EMC: ◆ “Configuration rules ” on page 34 ◆ “Connectivity options” on page 36 Configuration rules The following configuration rules were used to create the topology diagrams used in this TechBook. If there is a discrepancy between the diagrams and the configuration rules, the configuration rules take precedence. Note: These rules are numbered for referencing convenience only. Cisco Nexus and MDS switches 1. The Nexus 2232 requires a connection to a Nexus 5000 in order to function. 2. The Nexus 2232 does not support native Fibre Channel ports. 3. The Nexus 4001i requires a connection to a Nexus 5000 in order to support FCoE. 4. The Nexus 4001i does not support native Fibre Channel ports. 5. The Nexus 4001i does not support external VN_Port connections. 6. The Fibre Channel ports on the Nexus 5000 cannot be used for end device connectivity while the switch is in NPV mode. 7. Storage ports cannot be connected to the Nexus 5000 while it is running in NPV mode. 8. The Nexus 7000 does not support native Fibre Channel ports. 9. As of November 1, 2011, FCoE is supported on the MDS 95xx and Nexus 7000 platforms. 10. All host connectivity to the UCS 6100 is via FCoE. 11. The Fibre Channel ports on the UCS 6100 cannot be used for end device connectivity while the switch is in NPV mode.
  • 35. Supported connectivity options 35 Introduction to Fibre Channel over Ethernet 12. The UCS 6100 does not currently provide an FC zoning interface. As a result, when the 6100 is running in FC-SW mode, it must be connected to another Cisco switch that is also running in switch mode. 13. UCS does not currently support VE_Ports. 14. The Nexus 2248PQ requires an upstream connection to a parent switch such as the Nexus 6000 series. 15. The Nexus 6000 series does not support native Fibre Channel ports. Brocade 8000 and DCX switches 16. The Brocade 8000 and DCX do not support VE_Ports. 17. The Fibre Channel ports on the Brocade 8000 cannot be used for end device connectivity while the switch is in NPV mode. Brocade/Cisco interoperability 18. FC-SW interoperability between Brocade and Cisco switches is not supported. When attaching a Brocade switch to a Cisco FCF and vice versa, either NPV mode must be used on the Cisco switch or AG mode must be used on the Brocade switch. 19. FCoE NPV uplinks between Cisco and Brocade switches are not supported. 20. Cascading switches that are running in NPV/AG mode is not supported. Juniper QFX3500/3600 21. Juniper QFX3500 and QFX3600 switches that are running in Gateway mode must be connected to either a Brocade or Cisco switch that supports the NPIV protocol. 22. Juniper QFX3500 and QFX3600 switches that are running in Transit switch mode must be connected to either a QFX3500 or QFX3600 that is running in Gateway mode or to an FCoE-capable Cisco switch. 23. From a supported topologies perspective, a Juniper QFabric is equivalent to a Juniper QFX3500 or QFX3600. 24. EMC Storage may not be directly connected to Juniper QFX3500 or QFX3600 switches running in either Gateway or Transit switch modes.
  • 36. 36 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Introduction to Fibre Channel over Ethernet Connectivity options This section contains information on the connectivity options for the following connectivity components: ◆ “Nexus 2000” on page 36 ◆ “Nexus 2248PQ” on page 37 ◆ “Nexus 4001i” on page 37 ◆ “Nexus 5000” on page 38 ◆ “Nexus 6000” on page 39 ◆ “Nexus 61x0” on page 40 ◆ “Nexus 7000” on page 41 ◆ “MDS 9000” on page 42 ◆ “Brocade 8000/DCX” on page 43 ◆ “Juniper QFX3500/3600” on page 44 Nexus 2000 Figure 2 shows configurations specific to the Nexus 2000. Figure 2 Nexus 2000 supported connectivity options Host Storage FCoE Host Storage FCoE FCoE FCNexus 2232 FCoE Nexus 5000 (NPV)FCoE Nexus 2232 Host Storage FCoE Nexus 5000 (NPV)FCoE Nexus 2232 Host Storage FCoE Nexus 5000 (NPV)FCoE Nexus 2232 Host Storage FCoE FCoE FCNexus 5000 (NPV)FCoE Nexus 2232 FCoE FCoE FCoE FCNexus 5000 (FC-SW) FC FCMDS (FC-SW) FC FCoE FCBrocade 8000/DCX FC FCBrocade FOS/EOS Nexus 5000 (FC-SW) EMC EMC EMC EMC EMC
  • 37. Supported connectivity options 37 Introduction to Fibre Channel over Ethernet Nexus 2248PQ Figure 3 shows configurations specific to the Nexus 2248PQ. Figure 3 Nexus 2248PQ supported connectivity options Nexus 4001i Figure 4 shows configurations specific to the Nexus 4001i. Figure 4 Nexus 4001i supported connectivity options Host FCoE FCoEFCoE Nexus 2248PQ Nexus 6004 Storage EMC Host FCoE FCoEFCoE Nexus 2248PQ Nexus 6001 Storage EMC GEN-002129 Host Storage FCoE Host Storage FCoE FCoE FCNexus 4001i FCoE Nexus 5000 (NPV)FCoE Nexus 4001i Host Storage FCoE Nexus 5000 (NPV)FCoE Nexus 4001i Host Storage FCoE Nexus 5000 (NPV)FCoE Nexus 4001i Host Storage FCoE FCoE FCoE FCNexus 5000 (NPV)FCoE Nexus 4001i FCoE FCoE FCNexus 5000 (FC-SW) FC FCMDS (FC-SW) FC FCoE FCBrocade 8000/DCX FC FCBrocade FOS/EOS Nexus 5000 (FC-SW) EMC EMC EMC EMC EMC
  • 38. 38 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Introduction to Fibre Channel over Ethernet Nexus 5000 Figure 5 shows configurations specific to the Nexus 5000. Figure 5 Nexus 5000 supported connectivity options
  • 39. Supported connectivity options 39 Introduction to Fibre Channel over Ethernet Nexus 6000 Figure 6 shows configurations specific to the Nexus 6000. Figure 6 Nexus 6000 supported connectivity options Host FCoE FCoE Nexus 6000 Storage EMC Host FCoE FCoEFCoE Nexus 5000 Nexus 6000 Storage EMC Host FCoE FCoEFCoE Nexus 6000 Nexus 6000 Storage EMC Host FCoE FCoEFCoE Nexus 6000 Nexus 5000 Storage EMC GEN-0
  • 40. 40 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Introduction to Fibre Channel over Ethernet Nexus 61x0 Figure 7 shows configurations specific to the Nexus 61x0. Figure 7 Nexus 61x0 supported connectivity options Host Storage Nexus 61x0 (NPV)FCoE FC Host Storage FCoE Host Storage FCoE Host Storage Nexus 61x0 (FC-SW) Nexus 61x0 (FC-SW) Nexus 61x0 (NPV)FCoE FC FCoE FCNexus 5000 (FC-SW) FC FCMDS (FC-SW) FC FCoE FCBrocade 8000/DCX Host Storage Nexus 61x0 (NPV)FCoE FC FCBrocade FOS/EOS FCoE FCNexus 5000 (FC-SW) Host Storage FCoE Nexus 61x0 (NPV) FC FCMDS (FC-SW) Host Storage FCoE Nexus 5000 (FC-SW) FC FCoE FCNexus 61x0 (FC-SW) Host Storage FCoE Nexus 5000 (FC-SW) FC FCoE Nexus 61x0 (NPV) Host Storage Nexus 61x0 (NPV)FCoE FCoE This configuration requires that that 61x0 be connected to another switch running in FC-SW mode. Host Storage FCoE Nexus 61x0 (FC-SW) FC FCoE Nexus 5000 (NPV) EMC EMC EMC EMC EMC EMC EMC EMC EMC EMC
  • 41. Supported connectivity options 41 Introduction to Fibre Channel over Ethernet Nexus 7000 Figure 8 shows configurations specific to the Nexus 7000. Figure 8 Nexus 7000 supported connectivity options Host Storage Nexus 7000 Host Storage FCoE FCoE FC FCoE FCoE FCoE Host Storage FC FCoE FCoE FCoE Host Storage FC FCoE MDS FCoE Host Storage FCoE FCoE FCoE Nexus 7000 FCoE EMC EMC EMC EMC EMC Nexus 7000 Nexus 5000 Nexus 7000 Nexus 7000 Nexus 7000 (FC-SW) Nexus 5000 (NPV)
  • 42. 42 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Introduction to Fibre Channel over Ethernet MDS 9000 Figure 9 shows configurations specific to the MDS 9000 series. Figure 9 MDS 9000 supported connectivity options Host Storage MDS FCFC Host Storage MDS Nexus 5000 (FC-SW) FCFC Host Storage MDS FCFC Host Storage MDS MDS FC Host Storage MDS Nexus 5000 (NPV) FC FCoE FC Nexus 61x0 (FC-SW) FC FC FC FC Host Storage MDS Nexus 5000 (FC-SW) FC FCoE FC FC Host Storage MDS FCFC Nexus 61x0 (NPV) FCoE FCoE FCoE EMC EMC EMC EMC EMC EMC EMC FCoE FCoE FCoE FCoE FCoE FCoE FCoE FCoE FCoE FCoE FCoE FCoE Host Storage MDS Nexus 7000 FC FCoE EMC FCoE FCoE FCoE FCoE
  • 43. Supported connectivity options 43 Introduction to Fibre Channel over Ethernet Brocade 8000/DCX Figure 10 shows configurations specific to the Brocade 8000/DCX. Figure 10 Brocade 8000/DCX supported connectivity options
  • 44. 44 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Introduction to Fibre Channel over Ethernet Juniper QFX3500/3600 Figure 11 shows configurations specific to the Juniper QFX3500/3600. Figure 11 Juniper QFX3500/3600 supported connectivity options
  • 45. EMC Connectrix B Setup Examples 45 2 This chapter provides basic information, supported features, topologies, and detailed setup steps to create the configuration shown in Figure 15, “Target topology,” on page 56. Information on the ED-DCX-B topology, Figure 17 on page 105, is also included. ◆ MP-8000B............................................................................................. 46 ◆ MP-8000B supported features and topologies............................... 49 ◆ MP-8000B direct-connect topology.................................................. 53 ◆ MP-8000B firmware upgrade procedure ........................................ 90 ◆ Troubleshooting the MP-8000B ........................................................ 93 ◆ ED-DCX-B ......................................................................................... 104 Note: More detailed Fibre Channel over Ethernet information can be found in the Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook at http://elabnavigator.EMC.com, under the Topology Resource Center tab. Information in this TechBook includes FCoE and Ethernet basics, EMC storage, RecoverPoint and Celerra MPFS as solutions in an FCoE environement, andtroubleshooting basic FCoE and CEE problems. EMC Connectrix B Setup Examples
  • 46. 46 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples MP-8000B The MP-8000B is a 24-port 10 GbE line-rate, low latency, lossless Converged Enhanced Ethernet (CEE) and an 8-port auto-sensing 1, 2, 4, or 8 Gb/s Fibre Channel switch that delivers the latest Brocade ASIC technology and architecture for Fibre Channel Storage Area Networks (SANs). The MP-8000B enables the Fibre Channel over Ethernet (FCoE) protocol and is a high performance 8 Gb/s Fibre Channel switch designed for the needs of enterprise environments. The MP-8000B supplies Reliability, Availability, and Serviceability (RAS) performance and scalability requirements of an enterprise switch along with the interoperability and ease-of-use advantages found only in the EMC® Connectrix® product family. This section contains the following information: ◆ “MP-8000B overview” on page 46 ◆ “Management” on page 48 ◆ “POST and boot specifications” on page 48 MP-8000B overview The MP-8000B internal architecture consists of a Fibre Channel switch and an Ethernet switch, with an FCoE Gateway in between them. This means that there are separate processors, ASICs, and configuration files for the Fibre Channel and Ethernet portions of the switch. Each time you access the switch, whether it is through a serial cable or through a telnet/SSH session you will be entering the switch through the Fibre Channel portion of the switch. As a result of the partitions within the switch, there are two separate operating systems that need to be configured: the Fibre Channel side of the switch (FOS) and the Ethernet side of the switch (CMSH), each with its own type of operating system, as noted below: ◆ FOS – (Fabric Operating System) for Fibre Channel switch ◆ CMSH – (CEE Management Shell) for Ethernet switch
  • 47. MP-8000B 47 EMC Connectrix B Setup Examples Figure 12 is a high level diagram showing the switch layout. Notice the designation of the FC switch and the enhanced Ethernet switch with the FCoE gateway in the middle. Figure 12 Switch layout example The following configuration steps will show how to configure both sides of the switch as well as the initial configuration to get the switch ready for remote access. IMPORTANT Follow these steps in order to ensure a smooth installation. After the initial configuration is completed and remote access is tested, all the physical connections will need to be connected and verified. FC Switch Condor Chipset FCoE Gateway Zeus Chipset Ethernet Switch Anvil Chipset Ethernet Interfaces 0 - 23 Fibre Channel Interfaces 0 - 7 CEE Port 0 CEE Port 23 SYM-002069 FC Port 0 FC Port 7
  • 48. 48 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples Management The MP-8000B provides two different operating environments, Fibre Channel and Ethernet. Both need to be configured. The operating environments and how to configure each are: ◆ Command Line Interface (CLI) • FOS – (Fabric Operating System) for Fibre Channel switch • CMSH – (CEE Management Shell) for Ethernet switch ◆ HTTP - (Web Tools) • FOS • CMSH – Minimal configurations POST and boot specifications When the switch is turned on or rebooted, the switch performs a POST (Power-on self test). Total boot time with POST can be several minutes. POST can be omitted after subsequent reboots by using the fastboot command or entering the diagDisablePost command to persistently disable POST. For more information about these commands, refer to the Brocade Converged Enhanced Ethernet Command Reference manual. POST The success or failure results of the diagnostic tests that run during POST can be monitored through LED activity, the error log, or the command line interface. POST includes the following tasks: ◆ Conducts preliminary POST diagnostics. ◆ Initializes the operating system. ◆ Initializes hardware. ◆ Runs diagnostic tests on several functions, including circuitry, port functionality, memory, statistics counters, and serialization. Refer to “Interpreting POST results” on page 739 for more information.
  • 49. MP-8000B supported features and topologies 49 EMC Connectrix B Setup Examples MP-8000B supported features and topologies The supported features and topologies have been broken down into the following four areas, shown in Figure 13, and discussed further in this section. ◆ “FCoE interface” on page 50 ◆ “Ethernet uplink interface” on page 50 ◆ “FC interface” on page 52 ◆ “Ethernet end devices ” on page 52 Figure 13 is for clarification purposes only. The preferred method of incorporating the MP-8000B into an existing topology can be found in the target topology. Figure 13 Topology example
  • 50. 50 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples FCoE interface The following interfaces are supported: ◆ CNAs directly connected via physical media to the MP-8000B Note: Connecting the CNA to the MP-8000B via a stand-alone Top of Rack (TOR) or End Of Row (EOR) intermediate 10G Ethernet Bridge (either DCB capable or not) is not supported unless stated otherwise below. ◆ Physical connectivity • EMC direct-attached native FCoE storage devices • Fabric A /Fabric B connectivity is supported • An example of Fabric A /Fabric B connectivity is two MP-8000Bs, each connected to one of the two sides of a mirrored fabric • SFP+ optical (short reach only) • SFP+ cu (Twinax - Molex) • 1 m, 3 m, and 5 m are available ◆ Hardware-based FCoE CNAs from Brocade, Emulex, QLogic Note: Refer to the EMC Support Matrix for the most up-to-date support information. ◆ Software-based FCoE CNAs from Intel and Broadcom ◆ Native FCoE Storage: • EMC VNX® • EMC CLARiiON® CX4 • EMC Symmetrix® VMAX® Ethernet uplink interface The following distribution switches are preferred: ◆ Brocade BIGIron series ◆ Brocade NETIRON series ◆ Cisco Catalyst 6500
  • 51. MP-8000B supported features and topologies 51 EMC Connectrix B Setup Examples ◆ Cisco Nexus 7000 Note: The MP-8000B does not support 1 GbE, 10 GbE only. ◆ Any 10 GbE switch that supports 802.3ae-2002 (10 GbE optical, 10GBASE-SR and 10GBASE-LR) and 802.3ak-2004 (SFP+ Direct Attach, 10GSFP+Cu, Twinax copper connections) and is also IEEE 802.1ad and 802.1Q compliant Layer 2 features The following Layer 2 features are supported: ◆ PAUSE 802.3x will not be required. ◆ Etherchannels for Link Aggregation (LAG) ◆ Link Aggregation Control Protocol (LACP) ◆ Active mode – initiates LACP ◆ Passive mode – responds to LACP, does not initiate ◆ On – forces the LAG on, bypasses LACP ◆ Spanning Tree ◆ 802.1d - STP ◆ 802.1.s - MSTP ◆ 802.1w - RSTP, one instance for all VLANs ◆ Ethernet Edge Ports ◆ PAUSE frames ◆ Spanning Tree Edge ◆ Spanning Tree Auto Edge ◆ Spanning Tree Link Type ◆ Spanning Tree Root Guard ◆ Spanning Tree BPDU Guard ◆ Network Interface Card (NIC) teaming • FCoE interface except with EMC iSCSI ◆ Jumbo frames up to 9208 bytes
  • 52. 52 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples FC interface The following Fibre Channel features are supported: ◆ For connectivity to Connextrix B-Series switches, the MP-8000B can be running in FC-SW mode (E_Ports) or in Access Gateway (AG) mode ◆ For connectivity to Connectrix MDS-Series switches, the MP-8000B can only be running in AG mode ◆ Access Gateway (NPV) ◆ NPIV support ◆ 8 FC ports with 1/2/4/8G FC ◆ Universal E, F, and FL ports ◆ Builds on top of FOS 6.1+ release ◆ Hot-pluggable, Brocade branded SFP+ ◆ Short-Wave Laser (SWL), Long-Wave Laser (LWL) ◆ Default Licenses • Full Fabric • 8Gb • Enhanced Group Management • FCoE ◆ Optional licenses • Fabric Watch (FW) • Advanced Performance Monitoring (APM) • ISL Trunking (TRK) Ethernet end devices The following are default licenses: ◆ Generic 10 GbE NICs that conform to media requirements ◆ Celerra 10 GbE (NAS+iSCSI+MPFS) ◆ Symmetrix, VNX series, or CLARiiON iSCSI interfaces ◆ NIC Teaming
  • 53. MP-8000B direct-connect topology 53 EMC Connectrix B Setup Examples MP-8000B direct-connect topology This section contains the following information: ◆ “Existing topology ” on page 54 ◆ “Target topology” on page 56 ◆ “Prerequisites” on page 57 ◆ “Configuring the MP-8000B” on page 58 Refer to “Installing the MP-8000B” on page 737 for information on installing the MP-8000B.
  • 54. 54 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples Existing topology It is assumed that the customer has an existing LAN and mirrored SAN, as shown in Figure 14, and is adding the MP-8000B to both. Figure 14 Existing topology example Existing LAN The existing LAN has the following features/configuration parameters: ◆ Etherchannel (802.1q) ◆ LACP (Link Aggregation Control Protocol) ◆ RSTP (Rapid Spanning Tree Protocol) ◆ MST (Multiple Spanning Tree) ◆ IGMP Snooping ◆ VLANs 2/11/1 1/1 2/1 Distribution-1 Distribution-2 1/2 1/3 1/4 1/5 1/2 1/3 1/4 1/5 Fabric A Fabric B Storage 1 2 3 4 1/1 1/1 DCX_1 Domain 1 172.23.185.24 DCX_2 Domain 2 172.23.185.25 1/1 1/2 1/3 1/4 1/5 1/1 1/2 1/3 1/4 1/5 MDS 9513_1 VSAN 100 Domain 1 172.23.185.22 MDS 9513_2 VSAN 200 Domain 2 172.23.185.23 CNA_1 CNA_2 CNA_1 CNA_2 Host_1 (Windows) Host_2 (Linux) CNA_1: QLogic WWPN = 21:01:00:1b:32:2a:c0:b8 Storage = Storage port 2 CNA_1: Emulex WWPN = 10:00:00:00:c9:3c:f8:3c Storage = Storage port 1 CNA_2: Emulex WWPN = 10:00:00:00:c9:3c:f7:c0 Storage = Storage port 3 CNA_2: QLogic WWPN = 21:00:00:1b:32:0a:c0:b8 Storage = Storage port 4 Storage port 1: 50:06:04:82:D5:2E:69:79 Storage port 2: 50:06:04:82:D5:2E:69:49 Storage port 3: 50:06:04:8A:D5:2E:69:79 Storage port 4: 50:06:04:82:D5:2E:69:59 SYM-002072 10 GbE 4 Gb/s FC
  • 55. MP-8000B direct-connect topology 55 EMC Connectrix B Setup Examples Existing SAN The existing LAN has the following features/configuration parameters: Fabric A ◆ DCX_1 • Domain ID 1 ◆ DCX_2 • Domain ID 2 Fabric B ◆ 9513_1 • Domain ID 1 ◆ 9513_2 • Domain ID 2
  • 56. 56 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples Target topology Figure 15 shows the target topology. Figure 15 Target topology
  • 57. MP-8000B direct-connect topology 57 EMC Connectrix B Setup Examples Prerequisites Note the following prerequisites: FCF prerequisites The MP-8000B requires: ◆ 2u of rack space, preferably at the top of the rack where the servers that will be connecting to the MP-8000B are located ◆ Two power drops of either 110V or 220V ◆ An IP address for management purposes and a network port to which to connect the 8000 management port ◆ A customer-supplied password for the admin user Laptop prerequisites Note: The laptop used in this example is needed only to initially configure the MP-8000B and will be disconnected after the installation is complete. Laptop prerequisites include: ◆ Communications port 1 with a DB-9 connector ◆ Download and install PuTTY Note: Installing PuTTY is recommended but not mandatory. ◆ Ensure that HyperTerminal is installed Note: The operating system used for this example was Windows XP professional and HyperTerminal was installed by default. Some of the later Windows-based operating systems do not have HyperTerminal installed. This will need to be done before attempting to configure the MP-8000Bs. See the help feature for the particular version of Windows you are using for more information on installing HyperTerminal. MLX considerations Before connecting the MP-8000Bs to the MLX switches, ensure they are properly configured. Refer to “ED-DCX-B” on page 104 for more information on completing link aggregation configurations on the Brocade NetIron MLX-16 switching router.
  • 58. 58 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples Configuring the MP-8000B This section contains the following information: ◆ “Configuring MP-8000B-1” on page 58 ◆ “Configuring MP-8000B-2” on page 73 ◆ “Configuring the Windows host” on page 88 ◆ “Configuring the Linux host” on page 88 ◆ “Configuring the VMware host” on page 89 ◆ “Configuring EMC storage” on page 89 Configuring MP-8000B-1 Initial configuration The initial configuration will be from the Fibre Channel (FOS) side of the switch. You can enter the help command while within FOS, which will show all available configuration commands. You can also use the help command to see detailed descriptions on a specific configuration parameter. For example: MP_8000B:admin> help This command shows all available configuration commands. MP_8000B:admin> help ipaddrset This command shows a detailed description for the ipaddrset command. To complete the initial configuration you will need to connect to the MP-8000B using the supplied serial cable. 1. Connect the rj45 connector into the console port on the front of the switch. 2. Attach the db9 connector to COM port 1 of the laptop. 3. Open up the Hyperterminal. Select Start > Programs > Accessories > Communications > Hyperterminal. The New Connection dialog is opened with the entry name dialog highlighted. 4. Enter a name such as MP-8000B, and then click OK. The Connect to dialog box displays.
  • 59. MP-8000B direct-connect topology 59 EMC Connectrix B Setup Examples 5. Select COM 1 under the Connect using pull-down list, and then click OK. 6. In the COM 1 Properties dialog box, use the following settings: • Bits per second: = 9600 • Data bits: = 8 • Parity: = none • Stop bits: = 1 • Flow control: = None Note: The first time that you boot up the MP- 8000B, you will need to enter some information that will give the switch its basic configuration needed to access it remotely. Prior to the initial boot sequence the following information is needed to complete the process. 7. At the prompt enter "admin". MP_8000B:admin> admin 8. Enter the default password, which is "password". 9. Enter the following information, which is required: • Ip address Ipaddrset – This command initiates the script to walk you through setting the ip address, subnet mask, and default gateway. At the FOS switch prompt, enter the command “ipaddrset” and you will be prompted for the below information. MP_8000B:admin> ipaddrset Ethernet IP Address 172.23.185.114 Ethernet Subnetmask 255.255.255.0 Gateway IP Address 172.23.185.2 DHCP [Off]: - leave off • Subnet mask • Default gateway • Switch name • Time Zone information 10. Enter the following information: • Switch name switchname – Add name of switch after the command
  • 60. 60 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples MP_8000B:admin>switchname MP-8000B-1 • Time zone tsTimeZone – Time zone you are in MP_8000B:admin> tsTimeZone --interactive Note: This will guide you through an interactive menu to complete the time zone configuration. • Date date – Date and time The date value of 0610123009 has a format of mmddHHMMyy, where: – mm is the month; valid values are 01 through 12. – dd is the date; valid values are 01 through 31. – HH is the hour; valid values are 00 through 23. – MM is minutes; valid values are 00 through 59. – yy is the year; valid values are 00 through 99 (values greater than 69 are interpreted as 1970 through 1999, and values less than 70 are interpreted as 2000-2069). MP_8000B:admin>date 0610123009 Help command Within the Ethernet side of the switch (CMSH) you can enter the ? command to see all available configuration commands. The following are example descriptions from the MP-8000B: ◆ Full help is available when you are ready to enter a command argument (e.g., show ?) and describes each possible argument. ◆ Partial help is provided when an abbreviated argument is entered and you want to know what arguments match the input (e.g., show ve?). Configuration mode This section describes how to navigate the configuration mode. Configuration mode is the name of the process in which you configure the Ethernet portion of the MP-8000B, whether configuring the MP-8000B directly through a serial connection or remotely through a telnet session. In either case, you will enter the MP-8000B through the FOS first. To get the CMSH, enter the cmsh command. To make any type of configuration changes within the CMSH you must enter configuration mode. Enter the configure terminal
  • 61. MP-8000B direct-connect topology 61 EMC Connectrix B Setup Examples command, which will bring you to the configuration prompt. The following is an example of the output. switch# configure terminal Enter configuration commands, one per line. End with CNTL/Z. switch(config)# From this prompt you can enter the ? command to see available options. The running-configuration is the active configuration on the device. Any changes made to the switch while in the CMSH are made to the help command. The configuration stored in nonvolatile memory (nvram) is called the startup-configuration. To copy the running-configuration to nvram, enter the command copy running-config startup-config. switch# copy running-config startup-config Overwrite the startup config file (y/n): y Global configuration refers to anything within the running-configuration that is system wide to the Ethernet switch. The following is a truncated example of the running configuration output and how the different configuration modes are broken down. Interfaces TenGigabitEthernet 0/2 through TenGigabitEthernet 0/22 have been left out to keep the example short. All of the same properties apply to these interfaces. switch# sho run ! hostname switch ------ Global Configuration begins here----- ! vlan classifier rule 1 proto fcoe encap ethv2 vlan classifier rule 2 proto fip encap ethv2 vlan classifier group 1 add rule 1 vlan classifier group 1 add rule 2 no protocol spanning-tree ! cee-map default priority-group-table 1 weight 40 pfc priority-group-table 2 weight 60 priority-table 2 2 2 1 2 2 2 2 ! ------ Global Configuration ends here----- ! ! ------ Interface Configuration begins here----- interface Vlan 1 ! interface Vlan 1002
  • 62. 62 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples fcf forward ! interface TenGigabitEthernet 0/0 shutdown ! interface TenGigabitEthernet 0/1 switchport switchport mode access no shutdown cee default | | interface TenGigabitEthernet 0/23 shutdown ! ------ Interface Configuration ends here----- ! ! ------ Global Configuration resumes here----- protocol lldp advertise dcbx-fcoe-app-tlv advertise dcbx-fcoe-logical-link-tlv ! line console 0 login line vty 0 31 login ! end CMSH configuration 1. Enter the CMSH to begin configuring the Ethernet portion of the switch. MP-8000B-1:admin> cmsh Note: MP-8000Bs sold through EMC may have a default configuration already loaded. The default configuration parameters included are: Spanning -tree, VLAN Classifiers, VLAN Classifier Group, CEE Maps, and LLDP. To view these default configurations or to verify the parameters are included, enter the show running-config command. 2. Enter the show running-config command to check the configuration. The following shows a a truncated output of the show running-config command with the default parameters present. If the parameters are not included in the configuration, then Step 3 through Step 7 will need to be applied to the running-configuration. MP-8000B-1# show running-config !
  • 63. MP-8000B direct-connect topology 63 EMC Connectrix B Setup Examples protocol spanning-tree mstp <– Spanning-tree ! ! vlan classifier rule 1 proto fcoe encap ethv2 <– Vlan Classifier rule 1 vlan classifier rule 2 proto fip encap ethv2 <– Vlan Classifier rule 2 vlan classifier group 1 add rule 1 <– Vlan Classifier group 1 vlan classifier group 1 add rule 2 <– Vlan Classifier group 1 ! cee-map default <– CEE Map priority-group-table 1 weight 40 pfc priority-group-table 2 weight 60 priority-table 2 2 2 1 2 2 2 2 ! . NOTE: output truncated . protocol lldp <– LLDP advertise dcbx-fcoe-app-tlv advertise dcbx-fcoe-logical-link-tlv 3. Configure Spanning Tree. Note: By default, Spanning Tree should be enabled and MSTP should be configured. The Spanning Tree Protocol (STP) is an algorithm that works to prevent Layer 2 loops in the network by ensuring there is only one active forwarding port between Ethernet switches at a time. MP-8000B-1# config t MP-8000B-1(config)# protocol spanning-tree mstp MP-8000B-1(conf-mstp)# end Verify MSTP is configured MP-8000B-1# sho spanning-tree brief Spanning-tree Mode: Multiple Spanning Tree Protocol ← Ensure MSTP is configured 4. Configure the VLAN Classifier. The VLAN Classifier is used to apply the FCoE VLAN ID to the inbound frames from the CNA that are either the FIP or FCoE protocol. Inbound packets that are “untagged” (Ethernet frames that do not have a VLAN ID assigned) and match either rule #1 or rule #2 will be assigned the correct FCoE VLAN at the interface configuration. The interface configuration steps are covered later in the document.
  • 64. 64 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples The steps that follow will classify any FCoE traffic encapsulated within Ethernet with rule #1 and any FIP traffic encapsulated within Ethernet with rule #2. Note: This is an MP-8000B configuration only, and is done to classify the FCoE traffic to ensure that both the FCoE and FIP protocols will be guaranteed at least 40% of the bandwidth in the event of congestion on the interface. MP-8000B-1# config t MP-8000B-1(config)# vlan classifier rule 1 proto fcoe encap ethv2 MP-8000B-1(config)# vlan classifier rule 2 proto fip encap ethv2 Show configured classifier rules MP-8000B-1(config)# do sho vlan classifier rule vlan classifier rule 1 proto fcoe encap ethv2 vlan classifier rule 2 proto fip encap ethv2 The commands provided above have classified the two types of traffic, FCoE and FIP, both encapsulated within Ethernet Version 2. 5. Configure the VLAN Classifier Group. Use the following configuration steps to put both rule #1 and rule #2 into the same VLAN classifier group - 1. MP-8000B-1(config)# vlan classifier group 1 add rule 1 MP-8000B-1(config)# vlan classifier group 1 add rule 2 Show configured classifier groups. MP-8000B-1(config)# do sho vlan classifier group vlan classifier group 1 add rule 1 vlan classifier group 1 add rule 2 At this point we have classified FCoE and FIP traffic encapsulated within Ethernet Version 2 and assigned them to Group- 1. 6. Configure CEE-MAP. MP-8000B-1(config)# cee-map default MP-8000B-1(conf-ceemap)# priority-group-table 1 weight 40 pfc MP-8000B-1(conf-ceemap)# priority-group-table 2 weight 60 MP-8000B-1(conf-ceemap)# priority-table 2 2 2 1 2 2 2 2 MP-8000B-1(conf-ceemap)# exit Show configured cee maps MP-8000B-1(conf-ceemap)# do sho cee maps
  • 65. MP-8000B direct-connect topology 65 EMC Connectrix B Setup Examples CEE Map default Precedence 1 Priority Group Table 1: Weight 40, PFC Enabled, TrafficClass 3, BW% 40 2: Weight 60, PFC Disabled, TrafficClass 4, BW% 60 15.0: PFC Disabled 15.1: PFC Disabled 15.2: PFC Disabled 15.3: PFC Disabled 15.4: PFC Disabled 15.5: PFC Disabled 15.6: PFC Disabled 15.7: PFC Disabled Priority Table CoS: 0 1 2 3 4 5 6 7 --------------------------------------------- PGID: 2 2 2 1 2 2 2 2 Enabled on the following interfaces CEE Map test Precedence 1 Enabled on the following interfaces 7. Configure LLDP. The LLDP portion of the configuration is at the bottom of the running-configuration. The LLDP configuration is needed so the switch will advertise the appropriate Type Length Values (TLVs). MP-8000B-1(config)# protocol lldp MP-8000B-1(conf-lldp)# advertise dcbx-fcoe-app-tlv MP-8000B-1(conf-lldp)# advertise dcbx-fcoe-logical-link-tlv Show global LLDP information MP-8000B-1(conf-lldp)# do sho lldp LLDP Global Information system-name: MP-8000B-1 system-description: CEE Switch description: State: Enabled Mode: Receive/Transmit Advertise transmitted: 30 seconds Hold time for advertise: 120 seconds Re-init Delay Timer: 2 seconds Tx Delay Timer: 1 seconds Transmit TLVs: Chassis ID Port ID TTL IEEE DCBx DCBx FCoE App DCBx FCoE Logical Link Link Prim Brocade Link
  • 66. 66 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples DCBx FCoE Priority Bits: 0x8 DCBx iSCSI Priority Bits: 0x10 Show interface specific LLDP information MP-8000B-1(conf-lldp)# do sho lldp int t 0/1 LLDP information for Te 0/1 State: Enabled Mode: Receive/Transmit Advertise Transmitted: 30 seconds Hold time for advertise: 120 seconds Re-init Delay Timer: 2 seconds Tx Delay Timer: 1 seconds DCBX Version : CEE Auto-Sense : Yes Transmit TLVs: Chassis ID Port ID TTL IEEE DCBx DCBx FCoE App DCBx FCoE Logical Link Link Prim Brocade Link 8. Configure VLAN. On the MP-8000B switch you can create VLANs with VLAN IDs up to 3583. You can only configure one VLAN acting as the Fibre Channel Forwarder (FCF). VLANs are created to segment broadcast domains and different types of user traffic. A VLAN allows for a more dynamic environment while providing the segmentation of a physical LAN. This is accomplished by configuring VLAN assignments through software rather than moving physical interfaces on the switch. Note: VLAN 1 is the default VLAN on the MP-8000Bs. Currently, it is not possible to change the value of the default VLAN from 1. MP-8000B-1(conf-lldp)# int vlan 100 9. Configure FCoE VLAN. There can only be one VLAN configured as the FCF on a MP-8000B. The following example shows VLAN 1002 as the FCF forwarder. MP-8000B-1(conf-if-vl-100)# int vlan 1002 MP-8000B-1(conf-if-vl-1002)#fcf forward
  • 67. MP-8000B direct-connect topology 67 EMC Connectrix B Setup Examples MP-8000B-1 Interface configurations The following example shows how to configure configured interface TenGigabitEthernet 0/0. Perform Step 1 through Step 7 on all interfaces that will use FCoE. This example uses interfaces 0/0 and 0/4. 1. Run the switchport command to enable the layer 2 switching capability. MP-8000B-1(conf-if-vl-1002)# int t 0/0 MP-8000B-1(conf-if-te-0/0)# switchport 2. Configure the interfaces so they can handle both types of Ethernet traffic. Brocade has created the “converged” mode which will forward both the untagged (Ethernet) and tagged (FCoE) packets. MP-8000B-1(conf-if-te-0/0)# switchport mode converged 3. Assign the default VLAN for the interfaces. This configuration uses VLAN 100 as the default VLAN. This is the VLAN used for Ethernet traffic, not FCoE. MP-8000B-1(conf-if-te-0/0)# switchport converged vlan 100 Note: By default, VLAN 1 is used as the default VLAN if there is not one configured. 4. Configure the interfaces to handle the FCoE traffic. For this configuration step it is necessary to apply the VLAN classifier created in the Global Configuration and apply it to the FCoE VLAN, 1002. MP-8000B-1(conf-if-te-0/0)# vlan classifier activate group 1 vlan 1002 5. Apply STP configurations to the interfaces. These commands work in conjunction with the Global Configuration command – protocol spanning-tree RSTP. The following commands will configure the ports as an edgeport. This will allow the ports to begin forwarding traffic immediately after a STP event as well as guard it against incoming BPDUs. MP-8000B-1(conf-if-te-0/0)# spanning-tree edgeport MP-8000B-1(conf-if-te-0/0)# spanning-tree edgeport bpdu-guard
  • 68. 68 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples 6. Perform the next configuration step to apply the default CEE-MAP to the interface that you created earlier. The syntax for this command is cee <name>. The default CEE-MAP is named “default”. The command is: MP-8000B-1(conf-if-te-0/0)# cee default 7. Enable the interfaces so that they will be ready to pass traffic. MP-8000B-1(conf-if-te-0/0)# no shut Note: The no shut command will need to be completed on any of the Ethernet interfaces being used. By default all interfaces are in the “shutdown” state and will not pass any traffic. 8. Repeat Step 1 through Step 7 for other FCoE interfaces. 9. Perform the following to verify the configuration steps have been completed successfully: MP-8000B-1(conf-if-te-0/0)# do sho run ! hostname MP-8000B-1 ! protocol spanning-tree rstp ! vlan classifier rule 1 proto fcoe encap ethv2 vlan classifier rule 2 proto fip encap ethv2 vlan classifier group 1 add rule 1 vlan classifier group 1 add rule 2 ! cee-map default priority-group-table 1 weight 40 pfc priority-group-table 2 weight 60 priority-table 2 2 2 1 2 2 2 2 ! interface Vlan 1 ! interface Vlan 100 ! interface Vlan 1002 fcf forward ! interface TenGigabitEthernet 0/0 switchport switchport mode converged switchport converged vlan 100 vlan classifier activate group 1 vlan 1002 no shutdown
  • 69. MP-8000B direct-connect topology 69 EMC Connectrix B Setup Examples spanning-tree edgeport spanning-tree edgeport bpdu-guard cee default ! interface TenGigabitEthernet 0/1 shutdown ! interface TenGigabitEthernet 0/2 shutdown ! interface TenGigabitEthernet 0/3 shutdown ! interface TenGigabitEthernet 0/4 switchport switchport mode converged switchport converged vlan 100 vlan classifier activate group 1 vlan 1002 spanning-tree edgeport spanning-tree edgeport bpdu-guard no shutdown |. . . . NOTE: output truncated . . interface TenGigabitEthernet 0/23 shutdown ! protocol lldp advertise dcbx-fcoe-app-tlv advertise dcbx-fcoe-logical-link-tlv ! line console 0 login line vty 0 31 login ! end 10. Perform Save configuration – copy running-config to startup-config. Note: At this point, you will need to exit out of configuration mode. You can either enter "exit" to back out one level at a time, or "end" to exit completely out of configuration mode. MP-8000B-1(conf-if-te-0/0)# end MP-8000B-1# copy run start Overwrite the startup config file (y/n): y Building configuration...
  • 70. 70 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples 11. Create the port channel interface. The port channel interface is used to configure a generic configuration that can be applied to the appropriate interfaces for link aggregation (LAG). LAG, or Ether-Channel is a way in which to combine the interfaces into a single virtual link, therefore combining its bandwidth. MP-8000B-1(config)# interface port-channel 1 MP-8000B-1(conf-if-po-1)# switchport MP-8000B-1(conf-if-po-1)# switchport mode trunk MP-8000B-1(conf-if-po-1)# switchport trunk allowed vlan add 100 MP-8000B-1(conf-if-po-1)# no shut MP-8000B-1(conf-if-po-1)# exit 12. Configure Uplink interfaces t 0/15 and t 0/16 as LAG ports. MP-8000B-1(config)# interface t 0/15 MP-8000B-1 (conf-if-te-0/15)# channel-group 1 mode on type standard MP-8000B-1 (conf-if-te-0/15)# no shut MP-8000B-1(config)# interface t 0/16 MP-8000B-1 (conf-if-te-0/16)# channel-group 1 mode on type standard MP-8000B-1 (conf-if-te-0/16)# no shut 13. Verify the port channel interface is configured. MP-8000B-1(conf-if-te-0/16)#do sh run int port-channel 1 ! interface Port-channel 1 switchport switchport mode trunk no shutdown 14. Perform Save configuration – copy running-config to startup-config. MP-8000B-1(conf-if-te-0/16)# end MP-8000B-1# copy runn start Overwrite the startup config file (y/n): y Building configuration... MP-8000B-1 Zoning configuration After the configuration steps for the CMSH portion of the switch have been completed and you have saved the configuration you will need to complete the zoning configurations. Note: At this point, the traditional GUI-based tools should be able to access the MP-8000B to complete the zoning configuration, but for this example, we will complete the zoning configurations using the FOS CLI. At the Ethernet prompt enter the command exit to return to the FOS. MP-8000B-1#exit
  • 71. MP-8000B direct-connect topology 71 EMC Connectrix B Setup Examples You will notice that the prompt has changed and you are now ready to configure the Fibre Channel portion. MP-8000B-1:admin> The Fibre Channel portion of the configuration will include the Fibre Channel attached storage devices. At this point the CNAs are installed into the Servers with the correct drivers loaded, and physically connected to the MP-8000B. After you plug the storage devices into the MP-8000B, they should log in to the fabric. EMC allows only one Fibre Channel Forwarder (FCF) VLAN per switch so any zone configurations within the FOS are going to be to the same Fabric. The steps required to complete the Fibre Channel portion of the configuration are: • Identify WWPNs • Create zones • Create zone configuration • Enable the zone configuration • Save the zone configuration These steps are further detailed as follows: 1. Identify the WWPNs associated to the CNAs. Print local Name Server information MP-8000B-1:admin> fcoe --loginshow ============================================================================= Port Te port Device WWN Device MAC Session MAC ============================================================================= 8 Te 0/0 10:00:00:00:c9:3c:f8:3c 00:00:c9:3c:f8:3c 0e:fc:00:01:14:01 10 Te 0/2 21:01:00:1b:32:2a:c0:b8 00:1b:32:2a:c0:b8 0e:fc:00:01:18:01 2. Create applicable zones. a. Create each zone by using the following command: zonecreate "<zone name a>", "WWPN1; WWPN2; .... WWPNn" b. Create a zone named Host1_CNA1 with the above WWPNs using the zonecreate command.
  • 72. 72 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples Note: In the following examples the direct attached CNAs are in the output from the nsshow command because they were directly attached to the switch. The second WWPN in the examples are from Figure 15 on page 56. MP-8000B-1:admin> zonecreate "Host1_CNA1","10:00:00:00:c9:3c:f8:3c;50:06:04:82:d5:2e:69:79" c. Create a zone named Host1_CNA2 with the above WWNs using the zonecreate command. MP-8000B-1:admin> zonecreate "Host1_CNA2", "10:00:00:00:c9:3c:f7:c0;0:06:04:82:d5:2e:69:49" 3. Create the zone configuration. a. Create the zone configuration by using the following command: cfgcreate "<cfg name>", "<zone name a>; <zone name b>; .... <zone name n>". b. Create a zone configuration using the newly created zones, Host1_CNA1 and Host1_CNA2. MP-8000B-1:admin> configcreate "Host1_FCoE", "Host1_CNA1; Host1_CNA2" 4. Enable the zone configuration. a. Enable the zone config by using by using the following command: cfgenable "<cfg name>". b. Enable the new zone configuration. MP-8000B-1:admin> cfgenable “Host1_FCoE” 5. Save the configuration by using the cfgsave command: MP-8000B-1:admin> cfgsave 6. Verify the zone configuration. Show the zone configuration by using the cfgshow command. MP-8000B-1:admin> cfgshow Defined configuration: cfg: Host1_FCoE Host1_CNA1; Host1_CNA2 zone: Host1_CNA1 10:00:00:00:c9:3c:f8:3c; 50:06:04:82:d5:2e:69:79
  • 73. MP-8000B direct-connect topology 73 EMC Connectrix B Setup Examples zone: Host1_CNA2 10:00:00:00:c9:3c:f7:c0; 50:06:04:82:d5:2e:69:49 Effective configuration: cfg: Host1_FCoE zone: Host1_CNA1 10:00:00:00:c9:3c:f8:3c 50:06:04:82:d5:2e:69:79 zone: Host1_CNA2 10:00:00:00:c9:3c:f7:c0 50:06:04:82:d5:2e:69:49 Configuring MP-8000B-2 Initial configuration The initial configuration will be from the Fibre Channel (FOS) side of the switch. You can enter the help command while within FOS, which will show all available configuration commands. You can also use the help command to see detailed descriptions on a specific configuration parameter. For example: MP_8000B:admin> help This command shows all available configuration commands. MP_8000B:admin> help ipaddrset This command shows a detailed description for the ipaddrset command. To complete the initial configuration, you will need to connect to the MP-8000B using the supplied serial cable by following the next steps: 1. Connect the rj45 connector into the console port on the front of the switch. 2. Connect the db9 connector to COM port 1 of the laptop. 3. Open up the Hyperterminal. Select Start > Programs > Accessories > Communications > Hyperterminal. The New Connection dialog box is opened with the entry name dialog highlighted. 4. Enter a name, such as MP-8000B, and then click OK. The Connect to dialog box displays. 5. Select COM 1 under the Connect using pull-down list, and then click OK. 6. In the COM 1 Properties dialog box, use the following settings:
  • 74. 74 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples • Bits per second: = 9600 • Data bits: = 8 • Parity: = none • Stop bits: = 1 • Flow control: = None Note: The first time that you boot up the MP-8000B you will need to enter some information that will give the switch its basic configuration needed to access it remotely. You will need the below information prior to the initial boot sequence to complete the process. 7. At the prompt, enter admin. MP_8000B:admin> admin 8. Enter the default password, which is password. 9. Enter the following information, which is required: • Ip address The Ipaddrset command initiates the script to walk you through setting the ip address, subnet mask, and default gateway. At the FOS switch prompt, enter the ipaddrset command and you will be prompted for the below information. MP_8000B:admin> ipaddrset Ethernet IP Address 172.23.185.115 Ethernet Subnetmask 255.255.255.0 Fibre Channel IP Address [none] - leave blank Fibre Channel Subnetmask [none]: - leave blank Gateway IP Address 172.23.185.2 DHCP [Off]: - leave off • Subnet mask • Default gateway • Switch name • Time Zone information 10. Enter the following information: • Switch name switchname – Add name of switch after the command MP_8000B:admin>switchname MP-8000B-2
  • 75. MP-8000B direct-connect topology 75 EMC Connectrix B Setup Examples • Time zone tsTimeZone – Time zone you are in MP_8000B:admin> tsTimeZone US/Eastern (case sensitive) • Date date – Date and time The date value of 0610123009 has a format of mmddHHMMyy, where: – mm is the month; valid values are 01 through 12. – dd is the date; valid values are 01 through 31. – HH is the hour; valid values are 00 through 23. – MM is minutes; valid values are 00 through 59. – yy is the year; valid values are 00 through 99 (values greater than 69 are interpreted as 1970 through 1999, and values less than 70 are interpreted as 2000-2069). MP_8000B:admin>date 0610123009 Enabling Access Gateway (MP-8000B-2 only) For details on Access Gateway (AG), refer to the “Brocade Access Gateway” section in the Non-EMC SAN Products Data Reference Manual, available through the E-Lab Interoperability Navigator, Topology Resource Center tab, at http://elabnavigator.EMC.com. To enabling Access Gateway, complete the following steps: 1. Disable the switch. The switch must be disabled before you can enable AG mode. MP_8000B:admin> switchdisable 2. Enable AG mode. MP_8000B:admin> ag --modeenable Warning: Access Gateway mode changes the standard behavior of the switch. Please check the Access Gateway Administrator's Guide before proceeding. Enabling agmode will remove all the configuration data on the switch, including zoning configuration and security database. Back up your configuration usingthe configupload command. This operation will reboot the switch. Do you want to continue? (yes, y, no, n): [no] y Access Gateway mode was enabled successfully Switch is being rebooted...
  • 76. 76 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples IMPORTANT If you are connected to the switch with a console cable, you will need to wait until the switch comes back online. If you are logging into the switch remotely, you will need to establish a new telnet session after the switch comes back online. 3. Verify the switch is in AG mode. After the switch has come back online, enter the following command to verify that the switch is now in AG mode: MP_8000B:admin> ag --modeshow Access Gateway mode is enabled. AG commands The following are useful AG commands: ag --modeshow ag --show ag --help Configuration mode This section describes how to navigate the configuration mode. Configuration mode is the name of the process in which you configure the Ethernet portion of the MP-8000B, whether configuring the MP-8000B directly through a serial connection or remotely through a telnet session. In either case, you will enter the MP-8000B through the FOS first. To get the CMSH, enter the cmsh command. Within the Ethernet side of the switch (CMSH) you can enter the ? command to see all available configuration commands. The following are example descriptions from the MP-8000B: ◆ Full help is available when you are ready to enter a command argument (e.g., show ?) and describes each possible argument. ◆ Partial help is provided when an abbreviated argument is entered and you want to know what arguments match the input (e.g., show ve?).
  • 77. MP-8000B direct-connect topology 77 EMC Connectrix B Setup Examples To make any type of configuration changes within the CMSH, you must enter configuration mode. Enter the configure terminal command, which will bring you to the configuration prompt. The following is an example of the output. switch# configure terminal Enter configuration commands, one per line. End with CNTL/Z. switch(config)# From this prompt you can enter ? to see available options. The running-configuration is the active configuration on the device. Any changes made to the switch while in the CMSH are made to the running-configuration. The configuration stored in Non volatile memory (nvram) is called the startup-configuration. To copy the running-configuration to nvram, enter the command copy running-config startup-config. switch# copy running-config startup-config Overwrite the startup config file (y/n): y Global configuration refers to anything within the running configuration that is system wide to the Ethernet switch. The following is a truncated example of the running configuration output and how the different configuration modes are broken down. Interfaces TenGigabitEthernet 0/2 through TenGigabitEthernet 0/22 have been left out to keep the example short. All of the same properties apply to these interfaces. switch# sho run ! hostname switch ------ Global Configuration begins here----- ! vlan classifier rule 1 proto fcoe encap ethv2 vlan classifier rule 2 proto fip encap ethv2 vlan classifier group 1 add rule 1 vlan classifier group 1 add rule 2 no protocol spanning-tree ! cee-map default priority-group-table 1 weight 40 pfc priority-group-table 2 weight 60 priority-table 2 2 2 1 2 2 2 2 ! ------ Global Configuration ends here----- ! ! ------ Interface Configuration begins here----- interface Vlan 1 !
  • 78. 78 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples interface Vlan 1002 fcf forward ! interface TenGigabitEthernet 0/0 shutdown ! interface TenGigabitEthernet 0/1 switchport switchport mode access no shutdown cee default | | interface TenGigabitEthernet 0/23 shutdown ! ------ Interface Configuration ends here----- ! ! ------ Global Configuration resumes here----- protocol lldp advertise dcbx-fcoe-app-tlv advertise dcbx-fcoe-logical-link-tlv ! line console 0 login line vty 0 31 login ! end CMSH configuration 1. Enter the CMSH to begin configuring the Ethernet portion of the switch. MP-8000B-1:admin> cmsh Note: MP-8000Bs sold through EMC may have a default configuration already loaded. The default configuration parameters included are: Spanning -tree, VLAN Classifiers, VLAN Classifier Group, CEE Maps, and LLDP. To view these default configurations or to verify the parameters are included, enter the show running-config command. 2. Enter the show running-config command to check the configuration. The following shows a a truncated output of the show running-config command with the default parameters present. If the parameters are not included in the configuration, then Step 3 through Step 8 will need to be applied to the running-configuration.
  • 79. MP-8000B direct-connect topology 79 EMC Connectrix B Setup Examples MP-8000B-1# show running-config hostname MP8000B-1 ! protocol spanning-tree mstp <– Spanning-tree ! ! vlan classifier rule 1 proto fcoe encap ethv2 <– Vlan Classifier rule 1 vlan classifier rule 2 proto fip encap ethv2 <– Vlan Classifier rule 2 vlan classifier group 1 add rule 1 <– Vlan Classifier group 1 vlan classifier group 1 add rule 2 <– Vlan Classifier group 1 ! cee-map default <– CEE Map priority-group-table 1 weight 40 pfc priority-group-table 2 weight 60 priority-table 2 2 2 1 2 2 2 2 ! . NOTE: output truncated . protocol lldp <– LLDP advertise dcbx-fcoe-app-tlv advertise dcbx-fcoe-logical-link-tlv 3. Configure Spanning Tree. Note: By default, Spanning Tree should be enabled and MSTP should be configured. The Spanning Tree Protocol (STP) is an algorithm that works to prevent Layer 2 loops in the network by ensuring there is only one active forwarding port between Ethernet switches at a time. If necessary, enable the STP: MP-8000B-1# config t MP-8000B-1(config)# protocol spanning-tree mstp MP-8000B-1(conf-mstp)# end Verify MSTP is configured MP-8000B-2# sho spanning-tree brief Spanning-tree Mode: Multiple Spanning Tree Protocol Root ID Priority 32768 Address 0005.1e76.81a0 Hello Time 2, Max Age 20, Forward Delay 15 Bridge ID Priority 32768 Address 0005.1e76.81a0 Hello Time 2, Max Age 20, Forward Delay 15, Tx-HoldCount 6 Migrate Time 3 sec
  • 80. 80 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples 4. Configure the VLAN Classifier. The VLAN Classifier is used to apply the FCoE VLAN ID to the inbound frames from the CNA that are either the FIP or FCoE protocol. Inbound packets that are “untagged” (Ethernet frames that do not have a VLAN ID assigned) and match either rule #1 or rule #2 will be assigned the correct FCoE VLAN at the interface configuration. The interface configuration steps are covered later in the document. The steps that follow will classify any FCoE traffic encapsulated within Ethernet with rule #1 and any FIP traffic encapsulated within Ethernet with rule #2. Note: This is an MP-8000B configuration only, and is done to classify the FCoE traffic to ensure that both the FCoE and FIP protocols will be guaranteed at least 40% of the bandwidth in the event of congestion on the interface. Show configured classifier rules MP-8000B-2(config)# do sho vlan classifier rule vlan classifier rule 1 proto fcoe encap ethv2 vlan classifier rule 2 proto fip encap ethv2 The commands provided above have classified the two types of traffic, FCoE and FIP, both encapsulated within Ethernet Version 2. 5. Configure the VLAN Group. VLAN Group groups multiple VLAN Classifiers into a single group so they can be applied in a simpler fashion to an interface. Now that the traffic is classified, you need to apply this to a group for further configuration. In the next configuration steps, we are going to put both rule #1 and rule #2 into the same VLAN classifier group - 1. Show configured classifier groups MP-8000B-2(conf-lldp)# do sho vlan classifier group vlan classifier group 1 add rule 1 vlan classifier group 1 add rule 2 At this point we have classified FCoE and FIP traffic encapsulated within Ethernet Version 2 and assigned them to Group 1. 6. Configure the CEE-MAP.
  • 81. MP-8000B direct-connect topology 81 EMC Connectrix B Setup Examples The CEE-Map prioritizes the Ethernet traffic. This is done by putting the Ethernet interface into a “trusted” state so it will honor any priority bits within the Ethernet frames. A frame coming into the interface that has a priority within it will be “trusted”. A frame coming into the Ethernet interface that does not contain a priority will be given a default value of 0, which translates to a Priority Group ID (PGID) of 2 in the default CEE-MAP configuration. Any FCoE traffic will have a priority within the frame and the priority is trusted. Next is an example of a CEE-Map when all FCoE traffic is given a priority of 1 while any other type of traffic is given a priority of 2. This will guarantee that any FCoE traffic will get 40% of the available bandwidth. MP-8000B-2(config)# cee-map default MP-8000B-2(conf-ceemap)# priority-group-table 1 weight 40 pfc MP-8000B-2(conf-ceemap)# priority-group-table 2 weight 60 MP-8000B-2(conf-ceemap)# priority-table 2 2 2 1 2 2 2 2 Show configured cee maps MP-8000B-2(conf-ceemap)# do sho cee maps CEE Map default Precedence 1 Priority Group Table 1: Weight 40, PFC Enabled, TrafficClass 3, BW% 40 2: Weight 60, PFC Disabled, TrafficClass 4, BW% 60 15.0: PFC Disabled 15.1: PFC Disabled 15.2: PFC Disabled 15.3: PFC Disabled 15.4: PFC Disabled 15.5: PFC Disabled 15.6: PFC Disabled 15.7: PFC Disabled Priority Table CoS: 0 1 2 3 4 5 6 7 --------------------------------------------- PGID: 2 2 2 1 2 2 2 2 FCoE CoS: None Enabled on the following interfaces MP-8000B-2(conf-ceemap)# exit 7. Configure the LLDP. The LLDP portion of the configuration is at the bottom of the running-configuration. The LLDP configuration is needed so the switch will advertise the appropriate Type Length Values (TLVs).
  • 82. 82 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples MP-8000B-2(config)# protocol lldp MP-8000B-2(conf-lldp)# advertise dcbx-fcoe-app-tlv MP-8000B-2(conf-lldp)# advertise dcbx-fcoe-logical-link-tlv Show global lldp information MP-8000B-2(conf-lldp)# do sho lldp LLDP Global Information system-name: MP-8000B-1 system-description: CEE Switch description: State: Enabled Mode: Receive/Transmit Advertise transmitted: 30 seconds Hold time for advertise: 120 seconds Re-init Delay Timer: 2 seconds Tx Delay Timer: 1 seconds Transmit TLVs: Chassis ID Port ID TTL IEEE DCBx DCBx FCoE App DCBx FCoE Logical Link Link Prim Brocade Link DCBx FCoE Priority Bits: 0x8 DCBx iSCSI Priority Bits: 0x10 Show interface specific LLDP information MP-8000B-2(conf-lldp)# sho lldp int t 0/1 LLDP Interface statistics for Te 0/1 Frames transmitted: 25652 Frames Aged out: 0 Frames Discarded: 0 Frames with Error: 0 Frames Recieved: 25159 TLVs discarded: 0 TLVs unrecognized: 0 8. Configure the VLAN configuration. On the MP-8000B switch, you can create VLANs with VLAN IDs up to 3583, but you can only configure one VLAN acting as the Fibre Channel Forwarder (FCF). IMPORTANT VLAN 1 is the default VLAN on the MP-8000Bs and should not be altered MP-8000B-2(conf-lldp)# exit MP-8000B-2(config)# int vlan 100
  • 83. MP-8000B direct-connect topology 83 EMC Connectrix B Setup Examples 9. Configure the FCoE VLAN. There can only be one VLAN configured as the FCF per MP-8000B. The next example uses VLAN 1002 as the FCF forwarder. MP-8000B-2(conf-if-vl-1002)# int vlan 1002 MP-8000B-2(conf-if-vl-1002)# fcf forward MP-8000B-2 Interface configurations The following example shows how to configure configured interface TenGigabitEthernet 0/0. Note: These same steps will need to be performed on interface TenGigaibitEthernet 0/4 to complete this installation. 1. Run the switchport command to enable the layer 2 switching capability. MP-8000B-2(conf-if-vl-1002)# int t 0/0 MP-8000B-2(conf-if-te-0/0)# switchport 2. Configure the interfaces so they can handle both types of Ethernet traffic. Brocade has created the “converged” mode which will forward both the untagged (Ethernet) and tagged (FCoE) packets. MP-8000B-2(conf-if-te-0/0)# switchport mode converged 3. Assign the default VLAN for the interfaces. This configuration uses VLAN 100 as the default VLAN. This is the VLAN used for Ethernet traffic, not FCoE. MP-8000B-2(conf-if-te-0/0)# switchport converged vlan 100 Note: By default, VLAN 1 is used as the default VLAN if there is not one configured. 4. Configure the interfaces to handle the FCoE traffic. For this configuration step it is necessary to apply the VLAN classifier created in the Global Configuration and apply it to the FCoE VLAN, 1002. MP-8000B-2(conf-if-te-0/0)# vlan classifier activate group 1 vlan 1002 5. Apply STP configurations to the interfaces.
  • 84. 84 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples These commands work in conjunction with the Global Configuration command – protocol spanning-tree RSTP. The following commands will configure the ports as an edgeport. This will allow the ports to begin forwarding traffic immediately after a STP event as well as guard it against incoming BPDUs. MP-8000B-2(conf-if-te-0/0)# spanning-tree edgeport MP-8000B-2(conf-if-te-0/0)# spanning-tree edgeport bpdu-guard 6. Perform the next configuration step to apply the default CEE-MAP to the interface. The syntax for this command is cee <name>. The default CEE-MAP is named “default”. The command is: MP-8000B-2(conf-if-te-0/0)# cee default 7. Enable the interfaces so that they will be ready to pass traffic. MP-8000B-2(conf-if-te-0/0)# no shut Note: The no shut command will need to be completed on any of the Ethernet interfaces being used. By default all interfaces are in the “shutdown” state and will not pass any traffic. 8. Perform the following to verify the configuration steps have been completed successfully: MP-8000B-2(conf-if-te-0/0)# do sho run ! hostname MP-8000B-2 ! protocol spanning-tree mstp ! vlan classifier rule 1 proto fcoe encap ethv2 vlan classifier rule 2 proto fip encap ethv2 vlan classifier group 1 add rule 1 vlan classifier group 1 add rule 2 ! cee-map default priority-group-table 1 weight 40 pfc priority-group-table 2 weight 60 priority-table 2 2 2 1 2 2 2 2 ! interface Vlan 1 ! interface Vlan 100 ! interface Vlan 1002 fcf forward !
  • 85. MP-8000B direct-connect topology 85 EMC Connectrix B Setup Examples interface TenGigabitEthernet 0/0 switchport switchport mode converged switchport converged vlan 100 vlan classifier activate group 1 vlan 1002 spanning-tree edgeport spanning-tree edgeport bpdu-guard no shutdown ! interface TenGigabitEthernet 0/1 shutdown ! interface TenGigabitEthernet 0/2 shutdown ! interface TenGigabitEthernet 0/3 shutdown ! interface TenGigabitEthernet 0/4 switchport switchport mode converged switchport converged vlan 100 vlan classifier activate group 1 vlan 1002 spanning-tree edgeport spanning-tree edgeport bpdu-guard no shutdown |. . . . NOTE: output truncated . . interface TenGigabitEthernet 0/23 shutdown ! protocol lldp advertise dcbx-fcoe-app-tlv advertise dcbx-fcoe-logical-link-tlv ! line console 0 login line vty 0 31 login ! end 9. Perform Save configuration – copy running-config to startup-config. MP-8000B-2(conf-if-te-0/0)# end MP-8000B-2 # copy run start Overwrite the startup config file (y/n): y Building configuration...
  • 86. 86 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples 10. Create the port channel interface. The port channel interface is used to configure a generic configuration that can then be applied to the appropriate interfaces for link aggregation (LAG). MP-8000B-2(config)# interface port-channel 1 MP-8000B-2(conf-if-po-1)# switchport MP-8000B-2(conf-if-po-1)# switchport mode trunk MP-8000B-2(conf-if-po-1)#switchport trunk allowed vlan 100 MP-8000B-2(conf-if-po-1)# no shut 11. Configure interfaces t 0/15 and t 0/16 as LAG ports. MP-8000B-2(config)# interface t 0/15 MP-8000B-2(conf-if-te-0/15)# channel-group 1 mode on type standard MP-8000B-2(conf-if-te-0/15)# no shut MP-8000B-2(config)# interface t 0/16 MP-8000B-2(conf-if-te-0/16)# channel-group 1 mode on type standard MP-8000B-2(conf-if-te-0/16)# no shut 12. Verify the port channel interface is configured. MP-8000B-2(conf-if-te-0/16)# do sh run int port-channel 1 ! interface Port-channel 1 switchport switchport mode trunk no shutdown 13. Perform the save configuration by issuing the copy running-config startup-config command. MP-8000B-2# copy run start Overwrite the startup config file (y/n): y Building configuration... MP-8000B-2 Zoning configuration After the configuration steps for the CMSH portion of the switch have been completed and you have saved the configuration you will need to complete the zoning configurations. At this point, the CNAs are installed into the Servers with the correct drivers loaded and physically connected to the MP-8000B. Since MP-8000B-2 is running in AG mode, all zoning will need to be performed from the MDS-9513-2. It is strongly recommended you cut and paste the CLI to be used for zoning to minimize transcription types of errors when manually typing in the WWPNs. Plug the correct storage devices into the corresponding DCX, at which point they should log in to the switch.
  • 87. MP-8000B direct-connect topology 87 EMC Connectrix B Setup Examples The steps required to complete the zoning of the CNAs are: 1. Log in to the switch. 2. Create applicable zones. a. zone name <ZONE_NAME> vsan <VSAN ID> b. member pwwn <PWWN_1> c. member pwwn <PWWN_2> 3. Create zoneset. a. zone set name <ZONE_SET_NAME> vsan <VSAN ID> b. member <ZONE_NAME> 4. Active zoneset. a. zoneset activate name <ZONE_SET_NAME> vsan <VSAN ID> 5. Distribute zoneset. a. zoneset distribute full vsan <VSAN> 6. Confirm zoneset activation. a. show zoneset active vsan <VSAN> 7. Save configuration. a. Copy running startup Each of these steps are further detailed as follows: 1. Log in to the switch 172.23.185.22. 2. Create applicable zones: MDS-9513-1# config terminal MDS-9513-1 (config)# zone name host1_CNA2 vsan 100 MDS-9513-1 (config-zone)# member pwwn 10:00:00:00:c9:3c:f7:c0 MDS-9513-1 (config-zone)# member pwwn 50:06:04:8A:D5:2E:69:79 MDS-9513-1(config)# zone name host2_CNA2 vsan 100 MDS-9513-1 (config-zone)# member pwwn 21:00:00:1B:32:0A:C0:B8 MDS-9513-1 (config-zone)# member pwwn 50:06:04:82:D5:2E:69:59 3. Create zoneset: MDS-9513-1 (config)# zoneset name FCoE_100 vsan 100 MDS-9513-1 (config-zoneset)# member host1_CNA2 MDS-9513-1 (config-zoneset)# member host2_CNA2
  • 88. 88 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples 4. Activate zoneset: MDS-9513-1 (config-zoneset)# zoneset activate name FCoE_100 vsan 100 Zoneset activation initiated. check zone status MDS-9513-1(config)# exit 5. Distribute zoneset: MDS-9513-1 (config)# zoneset distribute full vsan 100 6. Confirm zoneset activation: MDS-9513-1(config)# show zoneset active vsan 100 zoneset name FCoE_100 vsan 100 zone name host1_cna2 vsan 100 *fcid 0x920006 [pwwn 10:00:00:00:c9:3c:f7:c0] *fcid 0x100400 [pwwn 50:06:04:8a:d5:2e:69:79] zone name host2_cna2 vsan 100 * fcid 0x920033 [pwwn 21:00:00:1b:32:0a:c0:b8] * fcid 0x100300 [pwwn 50:06:04:82:d5:2e:69:59] 7. Save configuration: MDS-9513-1(config)# exit MDS-9513-1# copy running startup-config Configuring the Windows host To configure the Windows host, refer to the following documentation, available at EMC Online Support at https://support.emc.com: ◆ QLogic: EMC Host Connectivity with QLogic Fibre Channel and iSCSI Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) in the Windows Environment P/N 300--001-164 ◆ Emulex: EMC Host Connectivity with Emulex Fibre Channel Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) in the Windows Environment P/N 300-001-157 ◆ EMC Host Connectivity Guide for Windows, P/N 300-000-603 Configuring the Linux host To configure the Linux host, refer to the following documentation, available at EMC Online Support at https://support.emc.com: ◆ QLogic: EMC Host Connectivity with QLogic Fibre Channel and iSCSI Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) for the Linux Environment, P/N 300-002-803
  • 89. MP-8000B direct-connect topology 89 EMC Connectrix B Setup Examples ◆ Emulex: EMC Host Connectivity with Emulex Fibre Channel Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) for the Linux v2.6.x Kernel Environment and the v8.x-Series Driver, P/N 300-002-583 ◆ EMC Host Connectivity Guide for Linux, P/N 300-003-805 Configuring the VMware host To configure the VMware host, refer to the following documentation, available at EMC Online Support at https://support.emc.com: ◆ EMC Host Connectivity Guide for VMware ESX Server, P/N 300-002-304 Configuring EMC storage Once the switch, hosts, and adapters have been configured, the storage will need to be configured to all each initiator to access a number of LUNs.
  • 90. 90 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples MP-8000B firmware upgrade procedure Downloading firmware to the MP-8000B from a remote FTP server is done from the Fibre Channel (FOS) side of the switch. This process is used to upgrade the firmware on the MP-8000B. This procedure is completed only from the FOS and will upgrade both the FOS and CMSH portions of the switch. The following is an example of how to complete this operation. IMPORTANT This procedure will reloaded the switch, during which time all traffic will be disrupted for 3-10 minutes. Assumptions This procedure assumes the following: ◆ The firmware files are stored in the outgoing/firmware directory of an FTP server at the IP address of 168.159.216.19 ◆ The name of the image is v6.1.2_cee1 Upgrading firmware Note: The text in bold is input by the user. To upgrade firmware, complete the following steps: 1. From the CMSH, create a backup of your existing configuration file. (Ensure the running configuration is correct.) Note: This command needs to be run from the CMSH only; the FOS will save any configuration changes automatically. MP-800B-1# copy running-config startup-config 2. Back up the existing configuration file. This can be done through the ftp server using the following format: copy startup-config ftp://[username[:password]@server/path]
  • 91. MP-8000B firmware upgrade procedure 91 EMC Connectrix B Setup Examples MP-800B-1# copy startup-config ftp://anonymous:anonymous@company.com@168.159.216.19/MP-8000B-1_startup-config Building configuration... 3. Download and install new firmware. MP-800B-1# firmwaredownload Server Name or IP Address: 168.159.216.19 User Name: anonymous File Name: v6.1.2_cee1 Network Protocol(1-auto-select, 2-FTP, 3-SCP) [1]: 2 Password: anonymous@company.com Checking system settings for firmwaredownload... Server IP: <ip address of FTP sever>, Protocol IPv4 System settings check passed. You can run firmwaredownloadstatus to get the status of this command. This command will cause a cold/disruptive reboot and will require that existing telnet, secure telnet or SSH sessions be restarted. Do you want to continue [Y]: y Firmware is being downloaded to the switch. This step may take up to 30 minutes. Preparing for firmwaredownload... Start to install packages... dir ################################################## ldconfig ################################################## glibc ################################################## glibc-linuxthreads ################################################## bash ################################################## NOTE: output truncated prom-8548 ################################################## Please avoid powering off the system during prom update. awk ################################################## Removing unneeded files, please wait ... Finished removing unneeded files. All packages have been downloaded successfully. Firmware has been downloaded to the secondary partition of the switch. HA Reboot not supported Rebooting system now.. Broadcast message from root (pts/0) Tue Aug 25 17:55:19 2009... The system is going down for reboot NOW !! 4. Reconnect to the switch after the reload has completed. Verify the status of Firmware download. MP-8000B-1# firmwaredownloadstatus [1]: Tue Aug 25 17:50:53 2009
  • 92. 92 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples Firmware is being downloaded to the switch. This step may take up to 30 minutes. [2]: Tue Aug 25 17:55:18 2009 Firmware has been downloaded to the secondary partition of the switch. [3]: Tue Aug 25 17:56:07 2009 The firmware commit operation has started. This may take up to 10 minutes. [4]: Tue Aug 25 17:56:48 2009 The commit operation has completed successfully. [5]: Tue Aug 25 17:56:48 2009 Firmwaredownload command has completed successfully. Use firmwareshow to verify the firmware versions. 5. Verify the status of Firmware download. MP-800B-1# firmwareshow Appl Primary/Secondary Versions ------------------------------------------ FOS v6.1.2_cee1 v6.1.2_cee1
  • 93. Troubleshooting the MP-8000B 93 EMC Connectrix B Setup Examples Troubleshooting the MP-8000B There are two sides to the MP-8000B, each briefly discussed in this section: ◆ The Fibre Channel side of the switch that uses the Fibre Channel OS (FOS). Commands and information are included in “Fibre Channel FOS” on page 93. ◆ The Ethernet side of the switch that uses the CMSH (CEE Management Shell). Commands and information are included in “CMSH” on page 99. Troubleshooting information is provided for the following: ◆ “Connectivity troubleshooting” on page 102 ◆ “VLAN troubleshooting” on page 102 ◆ “DCBX troubleshooting” on page 102 ◆ “FCoE troubleshooting” on page 103 ◆ “QoS troubleshooting” on page 103 ◆ “Link Aggregation (LAG) troubleshooting” on page 103 Fibre Channel FOS This section includes command information for the Fibre Channel FOS. help This command, without a parameter, lists all available commands within the FOS where help is available. With a parameter, it displays help about the specified command only. The following is an example of how this command works. MP-8000B:admin> help fcoe Administrative commands include: fcoe(1m) NAME fcoe - Manages and displays FCoE configuration. SYNOPSIS fcoe --cfgshow [port] fcoe --disable port
  • 94. 94 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples fcoe --enable port fcoe --loginshow port fcoe --fcmapset -vlan vid fcmapid fcoe --fcmapunset -vlan vid fcoe --fipcfg -advintvl intvl fcoe --fipcfgshow fcoe --resetlogin [-teport slot/port | -device wwn ] fcoe --help supportsave This command retrieves information on the switch involving zoning, performance, reboots, CP failovers, fabric-wide problems, etc. If no operands are provided, then the command is interactive. Note: The Supportsave command is a minimum step in troubleshooting and is normally required with any type of escalation process. MP-8000B-1:admin> supportsave This command collects RASLOG, TRACE, supportShow, core file, FFDC data and other support information and then transfer it to a FTP/SCP server or a USB device. This operation can take several minutes. Note: SupportSave transfers existing trace dump file first, then automatically generates and transfers the latest one. There will be two trace dump files transferred after this command. OK to proceed? (yes, y, no, n): [no] yes Host IP or Host Name: [x.x.x.x] User Name: [ftp username] Password: [ftp password] Protocol (ftp or scp): ftp Remote Directory: [ftp directory of server] Below is an example of the output: Saving support information for switch:MP-8000B-1, module:CONSOLE0... ...rtSave_files/MP-8000B-1-S0-200908111724-CONSOLE0.gz: 6.05 kB 180.38 kB/s Saving support information for switch:MP-8000B-1, module:RASLOG... ...tSave_files/MP-8000B-1-S0-200908111724-RASLOG.ss.gz: 41.34 kB 1.04 MB/s Saving support information for switch:MP-8000B-1, module:TRACE_OLD... ...les/MP-8000B-1-S0-200908111725-old-tracedump.dmp.gz: 1.60 MB 10.26 MB/s
  • 95. Troubleshooting the MP-8000B 95 EMC Connectrix B Setup Examples Saving support information for switch:MP-8000B-1, module:TRACE_NEW... ...les/MP-8000B-1-S0-200908111725-new-tracedump.dmp.gz: 1.69 MB 10.29 MB/s Saving support information for switch:MP-8000B-1, module:ZONE_LOG... ...ave_files/MP-8000B-1-S0-200908111725-ZONE_LOG.ss.gz: 26.71 kB 846.44 kB/s Saving support information for switch:MP-8000B-1, module:RCS_LOG... ...Save_files/MP-8000B-1-S0-200908111725-RCS_LOG.ss.gz: 873.00 B 23.39 kB/s (output truncated) switchshow This command prints switch and port status. This gives output of the FC interfaces and Virtual Fibre Channel interfaces. MP-8000B-1:admin> switchshow switchName: MP-8000B-1 switchType: 76.7 switchState: Online switchMode: Native switchRole: Subordinate switchDomain: 1 switchId: fffc01 switchWwn: 10:00:00:05:1e:76:77:00 zoning: ON (FCoE) switchBeacon: OFF Area Port Media Speed State Proto ===================================== 0 0 id N4 Online FC F-Port 50:06:04:82:D5:2E:69:79 1 1 id N4 Online FC F-Port 50:06:04:82:D5:2E:69:49 2 2 id N2 Online FC F-Port 50:06:04:8A:D5:2E:69:79 3 3 id N8 No_Light FC 4 4 id N2 No_Light FC 5 5 id N2 Online FC F-Port 50:06:04:82:D5:2E:69:59 6 6 id AN No_Sync FC 7 7 id N4 No_Light FC E-Port 10:00:00:05:1e:04:31:67 (upstream)(Trunk master) 8 8 -- 10 Online FCoE F-Port 2 NPIV public} 9 9 -- 10 Online FCoE F-Port 2 NPIV public} 10 10 -- 10 Online FCoE F-Port 2 NPIV public} FCoE Ports 11 11 -- 10 Online FCoE F-Port 3 NPIV public} (VF_PORTS) 12 12 -- 10 Online FCoE F-Port 2 NPIV public} 13 13 -- 10 Online FCoE F-Port 3 NPIV public} Note: In the above output from the switchshow command, ports 0 through 7 are the Fibre Channel interfaces. Ports 8 through 13 are the six 10gig FCoE_Ports connected between the Fibre Channel switch and the FCoE Gateway. Each one of the 6 FCoE-Ports translate to 4 ports on the Ethernet switch. See Figure 16 on page 96.
  • 96. 96 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples Figure 16 switchshow output nsshow Use this command to display local Name Server information about devices connected to this switch. The output from this command will include all Fibre Channel and FCoE devices logged in to the switch. This command allows you to correlate the WWPN and FCoE-Port, which you can use to find the Ethernet interface bound to a particular FCoE device. The following is an example of the nsshow command. MP-8000B-1:admin>nsshow { Type Pid COS PortName NodeName TTL(sec) N 010000; 3; 50:06:04:82:D5:2E:69:79; 50:06:04:82:D5:2E:a4:00; na FC4s: FCP PortSymb: [94] "SYMMETRIX::000192601257::SAF- 8fB::FC::5874_173+::EMUL B80F0000 2C1385CD 647680 07.29.09 13:22" NodeSymb: [38] "SYMMETRIX::000192601257::FC::5874_173+" Fabric Port Name: 20:00:00:05:1e:76:77:00 Permanent Port Name: 50:06:04:82:D5:2E:69:79 Ethernet Switch CMSH FCoE Gateway FC Switch FOS Interfaces t 0/0 - 0/3 Interfaces t 0/4 - 0/7 Interfaces t 0/8 - 0/11 Interfaces t 0/12 - 0/15 Interfaces t 0/16 - 0/19 Interfaces t 0/20 - 0/23 0 8 x 8 G FC FE Ports SYM-002070 24 x 10 GE FE Ports 6 x 10 G FCoE-Ports (VF_Ports) 7 8 9 10 11 12 13
  • 97. Troubleshooting the MP-8000B 97 EMC Connectrix B Setup Examples Port Index: 0 Share Area: No Device Shared in Other AD: No Redirect: No N 010100; 3; 50:06:04:82:D5:2E:69:49;50:06:04:82:D5:2E:ac:00; na FC4s: FCP PortSymb: [94] "SYMMETRIX::000192601259::SAF- 8gB::FC::5874_173+::EMUL B80F0000 2C1385CD 647680 07.29.09 13:22" NodeSymb: [38] "SYMMETRIX::000192601259::FC::5874_173+" Fabric Port Name: 20:01:00:05:1e:76:77:00 Permanent Port Name: 50:06:04:82:D5:2E:69:49 Port Index: 1 Share Area: No Device Shared in Other AD: No Redirect: No N 010804; 3;10 10:00:00:00:c9:3c:f8:3c;20:00:00:05:1e:9a:95:14; na FC4s: FCP PortSymb: [89] "BR-1020 | 2.1.0.005 | WIN-9I2KM0KQSSS | Windows Server (R) 2008 Standard | Service Pack 1" Fabric Port Name: 20:08:00:05:1e:76:77:00 Permanent Port Name: 20:08:00:05:1e:76:77:00 Port Index: 8 Share Area: No Device Shared in Other AD: No Redirect: No N 010902; 3; 10:00:00:00:c9:3c:f7:c0; 20:00:00:00:c9:3c:f7:c0; na FC4s: FCP PortSymb: [89] "BR-1020 | 2.1.0.005 | WIN-9I2KM0KQSSS | Windows Server (R) 2008 Standard | Service Pack 1" Fabric Port Name: 20:09:00:05:1e:76:77:00 Permanent Port Name: 20:09:00:05:1e:76:77:00 Port Index: 9 Share Area: No Device Shared in Other AD: No Redirect: No In the above output, each device successfully logged in to the switch as shown. The first two devices are logged in with Port Indexes of 0 and 1, which translates to Fibre Channel ports of 0 and 1. The second two devices are Brocade CNAs (BR-1020) and are showing Port Indexes of 8 and 9, which are FCoE ports bound to Ethernet interfaces. To see what Ethernet interfaces each device is bound to, complete the fcoe --loginshow [port] command, explained next.
  • 98. 98 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples fcoe --loginshow [port] The [port] is the Port Index portion of the nsshow command explained previously. This is only valid for the FCoE-Ports (8 - 13). If you try to complete this command using a Fibre Channel port you will receive the following error: error: loginshow failed Invalid slot/port number MP-8000B-1:admin> fcoe -loginshow 8 Number of connected devices: 1 ================================================================================ Peer Type Connect Info Device WWN Device MAC Session MAC FCoE Port MAC Te port ================================================================================ FCOE_DEVICE Direct 10:00:00:00:c9:3c:f8:3c 00:00:c9:3c:f8:3c 0e:fc:00:01:08:04 00:05:1e:76:77:00 Te 0/0 MP-8000B-1:admin> fcoe --loginshow 9 Number of connected devices: 1 ================================================================================ Peer Type Connect Info Device WWN Device MAC Session MAC FCoE Port MAC Te port ================================================================================ FCOE_DEVICE Direct 10:00:00:00:c9:3c:f7:c0 00:00:c9:3c:f7:c0 0e:fc:00:01:09:02 00:05:1e:76:77:06 Te 0/4 (output truncated) Note: The output from this command is showing only one successful login to each of the FCoE ports, 8 and 9. Te 0/0 is bound to FCoE-Port 8 and Te 0/4 is bound to FCoE-Port 9. fcoe--cfgshow [port] — Displays the configuration of a specified embedded FCoE port. If a port is not specified, the command displays all port configurations. This command gives a quick representation of how many devices are logged in to a specific FCoE port. MP-8000B-1:admin> fcoe --cfgshow User Port Status Port WWN DeviceCount Port Type Peer MAC ================================================================================ 8 ENABLED 20:08:00:05:1e:76:77:00 1 FCOE F-Port 9 ENABLED 20:09:00:05:1e:76:77:00 1 FCOE F-Port 10 ENABLED 20:0a:00:05:1e:76:77:00 0 FCOE F-Port 11 ENABLED 20:0b:00:05:1e:76:77:00 0 FCOE F-Port 12 ENABLED 20:0c:00:05:1e:76:77:00 0 FCOE F-Port 13 ENABLED 20:0d:00:05:1e:76:77:00 0 FCOE F-Port
  • 99. Troubleshooting the MP-8000B 99 EMC Connectrix B Setup Examples Note: In the above command there was no FCoE port specified, therefore the output is showing all of the FCoE ports. In this output, ‘User Port’ is the FCoE port, the ‘Port WWN’ is the address associated to the FCoE port, and the ‘DeviceCount’ is the number of devices logged into the FCoE port. In this case you can see that the ‘User Ports’ 8 and 9 each have one successful login, which correlates to the previous fcoe–loginshow command. fcoe – enable/disable <port> — Enable or disable the specified FCoE port. MP-8000B-1:admin> fcoe -disable 13 The FCoE port 13 is now disabled. If a FCoE port is disabled it will not perform any of the FCoE port functions and will show up in the above outputs as disabled, as shown in the below example using the fcoe-cfgshow command: MP-8000B-1:admin> fcoe --cfgshow User Port Status Port WWN DeviceCount Port Type Peer MAC ================================================================================ 8 ENABLED 20:08:00:05:1e:76:77:00 1 FCOE F-Port 9 ENABLED 20:09:00:05:1e:76:77:00 1 FCOE F-Port 10 ENABLED 20:0a:00:05:1e:76:77:00 0 FCOE F-Port 11 ENABLED 20:0b:00:05:1e:76:77:00 0 FCOE F-Port 12 ENABLED 20:0c:00:05:1e:76:77:00 0 FCOE F-Port 13 DISABLED 20:0d:00:05:1e:76:77:00 0 FCOE F-Port Note: The above output is an example only and is not relevant to the configurations used within this document. CMSH To help in the troubleshooting process of the CEE Management Shell (CMSH), you need to have a basic understanding of how this shell actually works. This section contains information on some of the key features and commands that will help you when working with the different modes within the CMSH. Key features of the CMSH include: ◆ Commands are NOT case sensitive. ◆ Supports abbreviated commands. you need to enter only enough characters for the switch to recognize the command as unique. • MP-8000B-1(config)# show interface brief - sho int bri
  • 100. 100 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples • MP-8000B-1(config)# configure terminal - config t ◆ Configuration commands show errors only if command fails or if there is an error in syntax. ◆ Configuration commands have a [no] option to negate or remove configuration. • MP-8000B-1(config)# no int VLAN 100 Key commands include the following: config t Enter configuration mode. exit Go to previous mode, back one level from current mode. end Exit completely from configuration mode. do Execute a command from configuration mode. MP-8000B-1(config)# do sho run fos Execute a FOS command from within the CMSH MP8000B-1# fos nsshow To execute a fos command from within the configuration mode you will also need to enter the do command. MP-8000B-1(config)# do fos nsshow ? Provides a list of available commands from the current mode. This can also be used to help finish a command string. show The following are show command options: MP8000B-1# show ? calendar Show the calendar information cee QoS Converged Enhanced Ethernet (CEE) clock Show clock information cores Display Core dump information debug Debugging functions (see also 'undebug') environment Show environment information file Displays the contents of the file history Display the session command history
  • 101. Troubleshooting the MP-8000B 101 EMC Connectrix B Setup Examples interface The layer2 interfaces ip Internet Protocol (IP) lacp LACP commands line Display information about terminal lines lldp Link Layer Discovery Protocol logging Display raslog messages mac MAC Access-list information mac-address-table Display Mac Address table information port Port commands port-channel LACP port-channel power Show power information privilege Show current privilege level processes Show processes information qos Quality of Service (QoS) rmon Remote Monitoring Protocol (RMON) running-config Current Operating configuration spanning-tree Display spanning tree information startup-config Contents of startup configuration statistics Accounting information system Show system information tech-support Show system information for Tech-Support users Display information about terminal lines version Display FOS version vlan Vlan commands zlog Debugging functions (see also 'undebug')
  • 102. 102 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples Connectivity troubleshooting The following commands help troubleshoot connectivity issues. VLAN troubleshooting The following commands help troubleshoot VLAN issues. DCBX troubleshooting The following commands help troubleshoot DCBX issues. MP8000B-1# show logging Show logging messages MP8000B-1# sho running-config Show current configuration MP8000B-1# sho system Show system information MP8000B-1# sho ip interface brief Summary of IP status and configuration MP8000B-1# sho interface switchport Display the modes of the Layer2 interfaces MP8000B-1# sho interface [x/y] Display specific Layer2 interface configuration and statistics MP8000B-1# sho mac-address-table Show entire Mac Address table MP8000B-1# sho mac-address-table int [x/y] Show Mac Address for specific interface MP8000B-1# sho mac-address-table vlan [x] Show Mac Address table for specific vlan MP8000B-1# sho vlan [x] Show specific vlan MP8000B-1# sho vlan brief Show all vlan information MP8000B-1# sho vlan fcoe Show FCoE Vlan and active ports MP8000B-1# sho lldp interface t [x/y] Show interface status and configuration MP8000B-1# sho lldp neighbors Show neighbor information MP8000B-1# sho lldp neighbors detail Show neighbor detail information by interface
  • 103. Troubleshooting the MP-8000B 103 EMC Connectrix B Setup Examples FCoE troubleshooting The following commands help troubleshoot FCoE issues. QoS troubleshooting The following commands help troubleshoot QoS issues. Link Aggregation (LAG) troubleshooting The following commands help troubleshoot Etherchannel (LAG) issues. MP8000B-1# sho vlan classifier group Show vlan classifier group information MP8000B-1# sho vlan classifier rule Show vlan classifier rule information MP8000B-1# sho vlan classifier interface group Show vlan classifier group interfaces MP8000B-1# sho cee maps Show active CEE Map information MP8000B-1# sho qos flowcontrol interface t [x/y] Show Per Priority Pause Flow MP8000B-1# sho qos queue interface t [x/y] Show queueing information MP8000B-1# sho port-channel Show Portchannel information MP8000B-1# sho lacp counter ShowLACP counters information
  • 104. 104 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples ED-DCX-B The topology discussed in this section, as illustrated in Figure 17 on page 105, shows the PB-DCX-24FCoE module plugged into a Connectrix ED-DCX-B switch, enabling FCoE. With FOS version v6.4.1_fcoe the DCX/DCX-4s (model numbers ED-DCX-B and ED-DCX-4S-B) are capable of running FCoE through the FCoE-capable line cards (PB-DCX-24FCoE). The line cards are 24 port FCoE 10 Gb/s Ethernet modules that plug directly into the DCX switches. In either switch, up to four modules are usable within a single chassis. Although there are other versions of FOS that will run FCoE, v6.4.1_fcoe is the only one capable of High Availability between CP failovers and is the only current version that supports ISSU (In Service Software upgrade) for FCoE traffic. This section contains the following information: ◆ “Topology” on page 105 ◆ “Compatibility” on page 105 ◆ “Supported topologies” on page 106 ◆ “Installing the PB-DCX-24FCoE” on page 106 ◆ “Configuring the PB-DCX-24FCoE” on page 107 ◆ “Useful commands” on page 109
  • 105. ED-DCX-B 105 EMC Connectrix B Setup Examples Topology The topology illustrated in Figure 17 shows the PB-DCX-24FCoE module plugged into an ED-DCX-4S-B switch, enabling FCoE. Figure 17 ED-DCX-4S-B switch and PB-DCX-24FCoE module topology example Compatibility The DCX chassis is not capable of supporting other intelligent application blades while running the PB-DCX-24FCoE modules. These include: ◆ Fibre Channel application blade PB-48K-AP4-18 ◆ Fibre Channel router blade PB-48K-18i ◆ Fibre Channel over IP blade PB-DCX-FX8-24 ◆ Storage encryption blade PB-DCX-16EB
  • 106. 106 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples Supported topologies In effect, the PB-DCX-24FCoE module is the MP-8000B stand alone switch with the sheet metal removed. Once installed into a DCX chassis, it has all of the same properties of the MP-8000B and is configured in the same manner. The FCoE module supports all of the same topologies as the MP-8000B and will perform the same functions as the MP-8000B. For more information on the supported topologies, refer to “MP-8000B supported features and topologies” on page 49. Installing the PB-DCX-24FCoE To install the FCoE module(s) into a DCX chassis, complete the following steps. 1. Install the module(s) into the DXC chassis. 2. Enable the Ethernet service. DCX_4s_admin> fosconfig -enable ethsw WARNING: This operation will enable the Ethernet Switch Service on this switch. Would you like to continue [Y/N]: y Enabling the Ethernet Switch Service. Please wait ... The Ethernet switch service has been enabled. Issue the slotpoweroff and slotpoweron commands on all of the blades with ID 74 to complete the process. 3. Verify the switch modules. In the following example there are FCoE modules in slot 1 and slot 8. DCX_4s_admin> slotshow DCX_4s_Edison:FID128:admin> slotshow Slot Blade Type ID Status ----------------------------------- 1 AP BLADE 74 FAULTY (9) 2 SW BLADE 51 ENABLED 3 CORE BLADE 46 ENABLED 4 CP BLADE 50 ENABLED 5 CP BLADE 50 ENABLED 6 CORE BLADE 46 ENABLED 7 SW BLADE 55 ENABLED 8 AP BLADE 74 FAULTY (9)
  • 107. ED-DCX-B 107 EMC Connectrix B Setup Examples 4. Issue the slotpoweroff for all new FCoE modules. DCX_4s_admin> slotpoweroff 1 Slot 1 is being powered off DCX_4s_admin> slotpoweroff 8 Slot 8 is being powered off 5. Issue the slotpoweron command for the FCoE modules. DCX_4s_admin> slotpoweron 1 Powering on slot 1 DCX_4s_admin> slotpoweron 8 Powering on slot 8 6. Verify the installation of FCoE modules was successful. Note: It will take several minutes for the modules to come up and complete the POST diagnostics. DCX_4s_admin> slotshow Slot Blade Type ID Status ----------------------------------- 1 AP BLADE 74 ENABLED 2 SW BLADE 51 ENABLED 3 CORE BLADE 46 ENABLED 4 CP BLADE 50 ENABLED 5 CP BLADE 50 ENABLED 6 CORE BLADE 46 ENABLED 7 SW BLADE 55 ENABLED 8 AP BLADE 74 ENABLED The FCoE modules are now installed and ready to configure. Configuring the PB-DCX-24FCoE The configuration of the FCoE modules is identical to that of the MP-8000B. You need to enter the cmsh through the FOS side of the switch. This will bring you to the Ethernet side of the switch where you will complete all Ethernet and FCoE configurations. DCX_4s_admin> cmsh DCX_4s_admin#
  • 108. 108 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples Enhanced FCoE configuration With v.6.4.1_fcoe, there are some provisioning steps that have been optimized and require fewer steps for interface configurations. The following parameters are now enabled by default and will not need to be configured manually. ◆ FCoE VLAN • VLAN 1002 is now preconfigured. This can be changed but only one FCoE vlan is allowed. ◆ CEE map • cee-map default This is now configured by default and cannot be deleted ◆ VLAN classifiers • FCoE/FIP VLAN classifier These parameters are now part of the default fcoe-map. The fcoe-map applies the FCoE-specific configurations to one place holder. With the fcoe-map, a user now only has to apply it to an interface using the fcoeport command, rather than configure each of the parameters separately. ◆ LLDP • LLDP - configured by default – FCoE TLVs Interface configuration With the new parameters applied by default, the configuration steps on an interface have been greatly reduced. Complete the following steps to apply the FCoE-specific information to an interface using the new fcoeport command. 1. Enter the configuration mode. DCX_4s_admin# config t DCX_4s_admin(config)# int t 1/0 DCX_4s_admin(conf-if-te-1/0)# fcoeport DCX_4s_admin(conf-if-te-1/0)# no shut 2. Verify the new configurations. DCX_4s_admin(conf-if-te-1/0)# do sho run int t 1/0 ! interface TenGigabitEthernet 1/0 fcoeport no shutdown ! The above interface is now ready for FCoE traffic.
  • 109. ED-DCX-B 109 EMC Connectrix B Setup Examples Additional configuruations For more detailed steps and configuration options, refer to “Configuring the MP-8000B” on page 58. Useful commands The following are some useful commands and applicable output. Show running-config DCX_4s_admin# sho run ! no protocol spanning-tree <- this will need to be configured accordingly ! cee-map default <- default cee-map priority-group-table 1 weight 40 pfc priority-group-table 2 weight 60 priority-table 2 2 2 1 2 2 2 2 ! fcoe-map default <- fcoe map fcoe-vlan 1002 NOTE: output truncated ! interface TenGigabitEthernet 1/0 fcoeport <- command to apply fcoe-map no shutdown NOTE: output truncated protocol lldp <- default lldp parameters advertise dcbx-fcoe-app-tlv advertise dcbx-fcoe-logical-link-tlv Show cee maps DCX_4s_admin# sho cee maps CEE Map default Precedence 1 Priority Group Table 1: Weight 40, PFC Enabled, TrafficClass 3, BW% 40 2: Weight 60, PFC Disabled, TrafficClass 6, BW% 60 15.0: PFC Disabled 15.1: PFC Disabled 15.2: PFC Disabled 15.3: PFC Disabled 15.4: PFC Disabled 15.5: PFC Disabled 15.6: PFC Disabled 15.7: PFC Disabled Priority Table CoS: 0 1 2 3 4 5 6 7 --------------------------------------------- PGID: 2 2 2 1 2 2 2 2
  • 110. 110 Fibre Channel over Ethernet (FCoE) Case Studies TechBook EMC Connectrix B Setup Examples Enabled on the following interfaces Te 1/0 Show fcoe maps DCX_4s_admin# sho fcoe-map fcoe-map default cee-map : default fcoe-vlan : 1002 interface(s) : Te 1/0
  • 111. Nexus Series Switches Setup Examples 111 3 This chapter provides basic information, supported features, topologies, and detailed setup steps for Cisco Nexus Series switches. Model numbers and troubleshooting information are also provided. Note: This section provides information for the Nexus 5596, 5548, 5020, and 5010 switches. When the information presented is applicable to only specific switches, the model number is used. Nexus 5000 is used when information is relevant to all switches. Note: The terms Nexus 4000 and Nexus 4001I are used interchangeably. ◆ Nexus Series switches............................................................................ 113 ◆ Nexus 6004 supported features and topologies................................. 123 ◆ Nexus 6001 supported features and topologies................................. 128 ◆ Nexus 2248PQ FEX supported features and topologies................... 133 ◆ Nexus 5000 supported features and topologies................................. 136 ◆ Nexus 2000 supported features and topologies................................. 141 ◆ Configuring 40GbE FCoE connectivity with Nexus 6000 Series..... 149 ◆ Nexus 5000 direct-connect topology ................................................... 173 ◆ Nexus 7000, Nexus 5000, and MDS 9500 series topology ................ 210 ◆ Nexus 5000/Nexus 2232PP topology.................................................. 242 ◆ Configuring Cisco VE_Ports................................................................. 272 ◆ Configuring Cisco FCoE NPV ports.................................................... 274 ◆ Complex FCoE topologies .................................................................... 277 ◆ Nexus Series switches firmware upgrade procedure ....................... 280 ◆ Troubleshooting the Nexus Series switches ....................................... 284 ◆ Virtual PortChannel............................................................................... 293 Nexus Series Switches Setup Examples
  • 112. 112 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Note: More detailed Fibre Channel over Ethernet information can be found in the Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook at http://elabnavigator.EMC.com, under the Topology Resource Center tab. Information in this TechBook includes FCoE and Ethernet basics, EMC storage, RecoverPoint and Celerra MPFS as solutions in an FCoE environment, and troubleshooting basic FCoE and CEE problems. Note: More details on Cisco Nexus switches can be found in the Non-EMC Connectrix Products Data Reference Manual, located on the E-Lab Navigator, in the Topology Resource Center tab.
  • 113. Nexus Series switches 113 Nexus Series Switches Setup Examples Nexus Series switches This section provides basic information for the following Nexus switches, along with information on management and VPN Routing and Forwarding (VRF): ◆ “Nexus 6004” on page 113 ◆ “Nexus 6001” on page 114 ◆ “Nexus 2248PQ Fabric Extender (FEX)” on page 115 ◆ “Nexus 5596” on page 116 ◆ “Nexus 5548” on page 116 ◆ “Nexus 5020” on page 117 ◆ “Nexus 5010” on page 118 ◆ “Nexus 2232PP” on page 119 ◆ “Management” on page 121 ◆ “VPN Routing and Forwarding (VRF)” on page 122 Note: Some of the information for the Nexus Series switches comes from Cisco documentation. For more detailed information about any Cisco product, refer to http://www.cisco.com. Nexus 6004 The Cisco Nexus 6004 is a 4RU 10 and 40 Gigabit Ethernet switch offering wire-speed performance for up to ninety-six 40 Gigabit Ethernet ports or three hundred eighty-four 10 Gigabit Ethernet ports (using QSFP breakout cables) for Ethernet and FCoE traffic with an overall throughput of 7.68 Tb/s. The Cisco Nexus 6004 offers 48 fixed ports of 40 Gigabit Ethernet with the base of the chassis and 4 line-card expansion module (LEM) slots. Although the base chassis has 48 fixed 40 Gigabit Ethernet ports, customers can purchase a twenty-four 40 Gigabit Ethernet port entry option to address the small-scale deployments. With two additional 12-port 40 Gigabit Ethernet software licenses, the remaining 24 ports can be enabled to use all 48 ports of 40 Gigabit Ethernet on the base chassis. Each LEM supports 12 ports of 40 Gigabit Ethernet in a QSFP form factor. For example, by adding four LEMs to the base chassis, 48 additional 40 Gigabit Ethernet ports can
  • 114. 114 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples be achieved. Each 40 Gigabit Ethernet port can be split into four 10 Gigabit Ethernet ports using QSFP breakout cables. The Cisco Nexus 6004 Switch as a high-density 10 and 40 Gigabit Ethernet and FCoE platform can be deployed in multiple scenarios: direct-attach 10 and 40 Gigabit Ethernet server-access and high-density FEX aggregation deployments, leaf and spine architectures, or compact aggregation to build a scalable Cisco Unified Fabric across a diverse set of physical and virtual server environments in data centers. The Cisco Nexus 6004 supports up to 4 additional expansion modules of the following offering: N6K-C6004-M12: 12 ports of 40-Gb/s Gigabit Ethernet and FCoE ports using a QSFP interface. These ports can also be configured as breakout ports consisting of four 10GbE ports per physical QSFP (up to 48 10GbE ports per LEM). The supported features and topologies for the Nexus 6004 are listed in “Nexus 6004 supported features and topologies” on page 123. For additional information regarding the Cisco Nexus 6004, refer to the Cisco Nexus 6000 Series Switches on the Cisco website at http://www.cisco.com. Nexus 6001 The Cisco Nexus 6001 is a 1RU 10 and 40 Gigabit Ethernet switch offering wire-speed performance for up to sixty-four 10 Gigabit Ethernet ports (using Quad Small Form-Factor Pluggable [QSFP] breakout cables) for Ethernet and FCoE traffic, with an overall throughput of 1.28Tbp/s. The Cisco Nexus 6001 offers 48 fixed 10 Gigabit Ethernet Enhanced Small Form-Factor Pluggable (SFP+) ports and four 40 Gigabit Ethernet QSFP+ ports. Each 40 Gigabit Ethernet port can be split into four 10 Gigabit Ethernet ports using a QSFP breakout cable. The Cisco Nexus 6001 delivers low port-to-port latency of approximately 1 microsecond and low jitter independent of packet size using cut-through switching architecture and with features enabled. The Cisco Nexus 6001 can be deployed in multiple scenarios - direct-attach 10 and 40 Gigabit Ethernet server access and high-density fabric extender aggregation deployments, leaf and spine designs, and compact aggregation - to build scalable Cisco Unified
  • 115. Nexus Series switches 115 Nexus Series Switches Setup Examples Fabric across a diverse set of physical and virtual server environments in the data center. The supported features and topologies for the Nexus 6004 are listed in “Nexus 6001 supported features and topologies” on page 128. For additional information regarding the Cisco Nexus 6001, refer to the Cisco Nexus 6000 Series Switches on the Cisco website at http://www.cisco.com. Nexus 2248PQ Fabric Extender (FEX) The Cisco Nexus 2248PQ is a 1RU 10 Gigabit Ethernet Fabric Extender. It supports high-density 10 Gigabit Ethernet environments with 48 1/10 Gigabit Ethernet SFP+ host ports and 4 QSFP+ fabric ports (16 x 10 GE fabric ports). The Cisco Nexus 2248PQ 10GE Fabric Extender also supports FCoE and a set of network technologies known collectively as Data Center Bridging (DCB). These features allow support for multiple traffic classes over a lossless Ethernet fabric, thus enabling consolidation of LAN, storage area network (SAN), and cluster environments. The Cisco Nexus 2248PQ Fabric Extenders connect to a parent Cisco Nexus switch through their fabric links and behave like remote line cards for a parent Cisco Nexus 5000, Nexus 6000, or Nexus 7000 Series Switch. Working in conjunction with Cisco Nexus switches, the Cisco Nexus 2248PQ Fabric Extenders extend the capabilities and benefits offered by the parent Cisco Nexus switch. Like other Cisco Nexus Fabric Extenders (FEX), the Nexus 2248PQ does not switch any frames locally. All frames are forwarded to the parent switch through its uplink ports and forwarded from there to the Destination MAC Address. Because of this, the Nexus 2248PQ must remain connected to at least one Nexus series parent switch in order to function. In addition to providing all switching, the parent Nexus switch also downloads firmware to the Nexus 2248PQ. All management functions for the Nexus 2248PQ are performed on the supported parent Nexus switch. The supported features and topologies for the Nexus 2248PQ ESX are listed in “Nexus 2248PQ FEX supported features and topologies” on page 133. For additional information regarding the Cisco Nexus 2248PQ Fabric Extender (FEX), refer to the Cisco Nexus 2000 Series Fabric Extenders on the Cisco website at http://www.cisco.com.
  • 116. 116 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Nexus 5596 The Cisco Nexus 5596UP switch is a 2RU 10 Gigabit Ethernet, Fibre Channel, and FCoE switch offering up to 1920 Gb/s of throughput and up to 96 ports. The switch has 48 unified ports and three expansion slots. Unified ports support traditional Ethernet, Fibre Channel, and Fibre Channel over Ethernet. Connectivity options include 1 Gb Ethernet and 10 Gig Ethernet with FCoe, and 1/2/4/8 Gb Native Fibre Channel. The Cisco Nexus 5596UP supports three expansion modules from the following offerings: ◆ Ethernet module that provides sixteen 1 and 10 Gigabit Ethernet and FCoE ports using the SFP+ interface. ◆ Fibre Channel plus Ethernet module that provides eight 1 and 10 Gigabit Ethernet and FCoE ports using the SFP+ interface, and eight ports of 8/4/2/1-Gb/s native Fibre Channel connectivity using the SFP+/SFP interface. ◆ A unified port module that provides up to sixteen 1 and 10 Gigabit Ethernet and FCoE ports using the SFP+ interface or up to sixteen ports of 8/4/2/1-Gb/s native Fibre Channel connectivity using the SFP+ and SFP interface. The use of 1 and 10 Gigabit Ethernet or 8/4/2/1-Gbps Fibre Channel on a port is mutually exclusive but selectable for any of the 16 physical ports per module. ◆ A Layer 3 module provides up to 160 Gb/s of Layer 3 forwarding capability (240 mpps) that can be shared by all the I/O ports in the chassis. The supported features and topologies for the Nexus 5548 are listed in “Nexus 5000 supported features and topologies” on page 136. Nexus 5548 The Nexus 5548 is a one rack-unit (1U), 10 GbE, Cisco Data Center Ethernet, and FCoE 1/2/4/8 Gb/s Fibre Channel switch built to provide 960 Gb/s throughput with very low latency. The Nexus 5548 has 32 fixed 1 GbE and 10 GbE ports, capable of supporting Cisco Data Center Ethernet and FCoE Small Form Factor Pluggable Plus (SFP+).
  • 117. Nexus Series switches 117 Nexus Series Switches Setup Examples Note: Expansion modules used in the Nexus 5020 and Nexus 5010 switches are not compatible with the Nexus 5548 switch. The switch has a serial console port and an out-of-band 10/100/1000 Mb/s Ethernet management port. Two N+N redundant, hot-pluggable power supplies and two N+N redundant, hot-pluggable fan modules provide highly reliable front-to-back cooling. Nexus 5548 delivers unified fabric at the network access layer, or edge, where servers connect to the LAN, SAN, and server clusters. To integrate with existing infrastructure, the Nexus 5548 Series provides native Fibre Channel uplinks to facilitate connection with installed SANs and available SAN switches. The supported features and topologies for the Nexus 5548 are listed in “Nexus 5000 supported features and topologies” on page 136. Nexus 5548UP The Nexus 5548UP is the same as a Nexus 5548 switch, except that all of the interfaces are unified ports (UPs). The Nexus 5548UP is a 1 rack-unit chassis that provides 32 fixed unified ports on the base chassis along with 1 expansion slot, scaling up to a total of 48 ports. Nexus 5020 The Nexus 5020 is a two rack-unit (2U), 10 GbE, Cisco Data Center Ethernet, and FCoE 1/2/4/8 Gb/s Fibre Channel switch built to provide 1.04 Tb/s (terabits) throughput with very low latency. The Nexus 5020 has 40 fixed 10 GbE ports capable of supporting Cisco Data Center Ethernet, and FCoE Small Form Factor Pluggable Plus (SFP+). The switch has a serial console port and an out-of-band 10/100/1000 Mb/s Ethernet management port. The switch is powered by 1+1 redundant, hot-pluggable power supplies, and 4+1 redundant, hot-pluggable fan modules to provide highly reliable front-to-back cooling. Nexus 5020 delivers unified fabric at the network access layer, or edge, where servers connect to the LAN, SAN, and server clusters. To integrate with existing infrastructure, the Nexus 5020 Series provides native Fibre Channel uplinks to facilitate connection with installed SANs and available SAN switches.
  • 118. 118 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples The supported features and topologies for the Nexus 5020 are listed in “Nexus 5000 supported features and topologies” on page 136. Nexus 5010 The Nexus 5010 is a one rack-unit (1U), 10 GbE, Cisco Data Center Ethernet, and FCoE 11/2/4/8 Gb/s Fibre Channel switch built to provide 520 Gb/s throughput with very low latency. I/O consolidation using the Nexus 5010 is shown in Figure 20 on page 136. It has 20 fixed 10 GbE ports capable of supporting Cisco Data Center Ethernet, and FCoE Small Form Factor Pluggable Plus (SFP+). The switch has a serial console port and an out-of-band 10/100/1000 Mb/s Ethernet management port. The switch is powered by 1+1 redundant, hot-pluggable power supplies, and 4+1 redundant, hot-pluggable fan modules to provide highly reliable front-to-back cooling. Nexus 5010 delivers unified fabric at the network access layer, or edge, where servers connect to the LAN, SAN, and server clusters. To integrate with existing infrastructure, the Nexus 5010 Series provides native Fibre Channel uplinks to facilitate connection with installed SANs and available SAN switches. The supported features and topologies for the Nexus 5010 are listed in “Nexus 5000 supported features and topologies” on page 136. Nexus 4000 The Cisco Nexus 4000 is a 20-port, 10 G FCoE aware Ethernet switch module intended for use in IBM Blade Servers and is fully compliant with the IEEE 802.1 Data Center Bridging (DCB) specification. This blade switch is also referred to as the Nexus 4001I Switch Module. The term FCoE aware indicates that the switch is capable of performing FIP snooping and supports the creation of dynamic ACLs. It also indicates that the Nexus 4000 does not contain a Fibre Channel Forwarder (FCF) and as a result FCoE frames received from CNAs connected to the Nexus 4000 that are destined to a native FC device must be forwarded to a switch that does contain an FCF before the FC frame can be de-encapsulated and forwarded onto the SAN. Because the CNAs are not directly connected to a switch that contains an FCF, the resulting topology is actually referred to as a DCB Cloud.
  • 119. Nexus Series switches 119 Nexus Series Switches Setup Examples Due to the nature of a DCB Cloud topology, some of the setup steps will differ from steps used in a direct connect environment. Note: For more information on direct connect and DCB Cloud, refer to the “FCoE Initialization Protocol (FIP)” section in the Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook at http://elabnavigator.EMC.com, under the Topology Resource Center tab. Fourteen of the 10 G ports are used to connect to the server blades and six of the ports are available to connect to the Nexus 5020. The six external interfaces can all be bundled into a port channel. The Nexus 4000 can be managed via Telnet. Nexus 2232PP The Cisco Nexus 2232PP is effectively an expansion blade for the Nexus 5000 series switches. The Nexus 2232PP does not switch any frames locally. All frames are forwarded to the parent 2232(s) through its uplink ports and forwarded from there to the Destination MAC Address. Because of this, the Nexus 2232PP must remain connected to at least one Nexus 5000 series switch in order to function. In addition to providing all switching, the parent Nexus 5000 also downloads firmware to the Nexus 2232. All management functions for the Nexus 2232PP are performed on the parent Nexus 5000. The Cisco Nexus 2232PP provides 32 10 Gb Ethernet and Fibre Channel Over Ethernet (FCoE) Small Form-Factor Pluggable Plus (SFP+) server ports and eight 10 Gb Ethernet and FCoE SFP+ uplink ports in a one rack unit (1RU) form factor. The Nexus 2232PP can be connected either in a straight-through configuration, as shown in Figure 18 on page 120, or in a cross-connected configuration, as shown in Figure 19 on page 120.
  • 120. 120 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Figure 18 Straight-through configuration example Figure 19 Cross-connected configuration example The obvious difference between the two configurations is the number of parent switches. N5k N4k / N2k N5k N4k / N2k Host ICO-IMG-000854 N5k N4k / N2k N5k N4k / N2k Host ICO-IMG-000855
  • 121. Nexus Series switches 121 Nexus Series Switches Setup Examples ◆ In the straight-through configuration, each Nexus 2232PP is only connected to a single parent Nexus 5000 switch. ◆ In the cross-connected configuration, each Nexus 2232PP is connected to two parent Nexus 5000 switches. Another difference between the two configurations is the use of the Virtual Port Channel (vPC) feature on the Nexus 5000 switches. ◆ In the straight-through topology, vPC can be configured but is not required. ◆ In the cross-connected topology, vPC must be configured. Note: Refer to “Virtual PortChannel” on page 293 for more information on vPC. IMPORTANT FCoE is not currently supported and will not function properly in a cross-connected topology. For additional information on the Nexus 5000/2232 refer to the Data Center Access Design with Cisco Nexus 5000 Series Switches and 2000 Series Fabric Extenders and Virtual Port Channels document, located at http://www.cisco.com. Management The management tools used to manage FCoE and non-FCoE environments are similar. In all cases, the FC portion of the CNA appears as a FC adapter to the OS, and the IP portion appears as a 10 GbE adapter. Other management tools include: ◆ Fabric Manager (FM) • FM will properly display – FCoE interface – FC interface – CNA logins ◆ EMC Ionix™ Network Configuration Manager (NCM) • Discover/Identify • Configuration management • Device access • Credential management
  • 122. 122 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples ◆ Ethernet Uplink Interface will be shown in Data Center Network Manager (DCNM) ◆ Device Manager ◆ CLI ◆ SNMP VPN Routing and Forwarding (VRF) VPN Routing and Forwarding (VRF) is used on the management interface of the Nexus 5000. VRF increases functionality by allowing multiple instances of a routing table to co-exist within the same Nexus 5000 at the same time. It also increases functionality by allowing network paths to be segmented without using multiple devices. VRF acts like a logical router, but while a logical router may include many routing tables, a VRF instance uses only a single routing table. If you are familiar with SAN-OS, the presence of VRF in NX-OS may require that you slightly alter the commands you are used to when performing actions involving the Management interface such as copying files and pinging other IP addresses. Refer to “Nexus Series switches firmware upgrade procedure” on page 280 for an example of using VRF management.
  • 123. Nexus 6004 supported features and topologies 123 Nexus Series Switches Setup Examples Nexus 6004 supported features and topologies The supported interfaces, features and topologies have been broken down into multiple areas, and are discussed in this section. Actual reference topologies are provided in “Nexus 6000 Series and Fabric Extenders topologies” on page 143. Interfaces This section contains information on the following interfaces: ◆ “Physical interfaces” on page 123 ◆ “FCoE interfaces” on page 124 ◆ “Ethernet Fabric Extension interfaces and technologies” on page 124 ◆ “Ethernet end-device interfaces” on page 124 ◆ “FC interfaces” on page 125 Physical interfaces The following interfaces speed configurations are supported: ◆ 40GbE (True 40GbE) ◆ 40GbE (4x10GbE) ◆ 10GbE The following optic types are supported: ◆ QSFP-40G-SR4 (MMF – 850nm) ◆ QSFP-40G-CSR4 (MMF – 850nm) ◆ QSFP-40G-LR4 (SMF – 1310nm) The following cable types are supported: ◆ 40G QSFP+ to 4x 10G SFP+ fibre breakout (1xMTP-to-4xLC) ◆ 40G QSFP+ to 4x 10G SFP+ copper breakout ◆ 40G QSFP+ direct-attach copper (Twinax CU 1M, 3M, 5M - passive) ◆ 40G QSFP+ direct-attach copper (Twinax CU 7M, 10M - active) Refer to the Cisco documentation on the Cisco website at http://www.cisco.com for a current and fully updated list of supported optics and cables.
  • 124. 124 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples FCoE interfaces The following interfaces are supported: ◆ Native FCoE CNA directly connected to Cisco Nexus 6004 QSFP+ via breakout cable (4x10GbE) ◆ Native FCoE Storage directly connected to Cisco Nexus 6004 QSFP+ via breakout cable (4x10GbE) • EMC VNX Series • EMC VMAX Series ◆ FC Enhanced port types: VE, TE and VF Note: Refer to the EMC Support Matrix available through E-Lab Interoperability Navigator at http://elabnavigator.EMC.com, under the PDFs and Guides tab for a current list of supported 10GbE iSCSI, FCoE and CNA adapters. Ethernet Fabric Extension interfaces and technologies The following are supported: ◆ Cisco Adapter FEX ◆ Cisco Data Center Virtual Machine FEX (VM-FEX) ◆ Support for up to 24 fabric extenders (Layer 2) with each Cisco Nexus 6004 switch ◆ Cisco 40GbE Line Expansion Modules (LEM) Note: Refer to the EMC Support Matrix available through E-Lab Interoperability Navigator at http://elabnavigator.EMC.com, under the PDFs and Guides tab for a current list of supported fabric extenders (FEX). Ethernet end-device interfaces The following interfaces are supported: ◆ 10GbE CNA directly connected to Cisco Nexus 6004 QSFP+ via breakout cable (4x10GbE) ◆ 10GbE iSCSI directly connected to Cisco Nexus 6004 QSFP+ via breakout cable (4x10GbE) ◆ 10GbE FCoE and iSCSI targets directly connected to Cisco Nexus 6004 QSFP+ via breakout cable (4x10GbE) • EMC VNX Series • EMC VMAX Series
  • 125. Nexus 6004 supported features and topologies 125 Nexus Series Switches Setup Examples ◆ 10GbE NICs, which adhere to industry standards, via QSFP+ breakout cable (4x10GbE) ◆ PortChannels (PC) ◆ Virtual Port Channels (vPC) ◆ NIC Teaming FC interfaces The following interfaces are supported: ◆ Direct connect FC Interfaces are not supported on the Nexus 6004 ◆ FC Enhanced port types: VE, TE and VF Features This section lists the information on the following features. ◆ “Data Center Bridging (DCB)” on page 125 ◆ “FCoE ” on page 125 ◆ “Ethernet (Layer-2)” on page 126 ◆ “Ethernet (Layer-3)” on page 127 Data Center Bridging (DCB) ◆ CEE- and IEEE-compliant PFC (per-priority Pause frame support) ◆ PFC link distance support: 300m ◆ CEE-compliant DCBX Protocol ◆ CEE- and IEEE-compliant Enhanced Transmission Selection FCoE ◆ T11 standards-compliant FCoE (FC-BB-5) ◆ T11 FCoE Initialization Protocol (FIP) (FC-BB-5) ◆ Any 10 or 40 Gigabit Ethernet port configurable as FCoE ◆ Fibre Channel forwarding (FCF) ◆ Fibre Channel enhanced port types: VE, TE, and VF ◆ Direct attachment of FCoE targets ◆ Up to 32 virtual SANs (VSANs) per switch ◆ Fabric Device Management Interface (FDMI)
  • 126. 126 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples ◆ Fibre Channel ID (FCID) persistence ◆ In-order delivery ◆ Port tracking ◆ Cisco FCoE_NPV technology ◆ N-port identifier virtualization (NPIV) ◆ Fabric services: Name server, registered state change notification (RSCN), login services, and name-server zoning ◆ Per-VSAN fabric services ◆ Cisco Fabric Services ◆ Fabric Shortest Path First (FSPF) ◆ Storage Management Initiative Specification (SMI-S) ◆ Boot from SAN over vPC and Enhanced vPC (EvPC) ◆ FCP ◆ Native Interop Mode 2 & 3 ◆ VSAN trunking ◆ Cisco NPV technology Ethernet (Layer-2) ◆ Layer 2 switch ports and VLAN trunks ◆ IEEE 802.1Q VLAN encapsulation ◆ Rapid Per-VLAN Spanning Tree Plus (PVRST+) (IEEE 802.1w compatible) ◆ Multiple Spanning Tree Protocol (IEEE 802.1s) instances: 64 instances ◆ Spanning Tree PortFast ◆ Spanning Tree root guard ◆ Spanning Tree Bridge Assurance ◆ Cisco EtherChannel technology (up to 16 ports per EtherChannel) ◆ Cisco vPC technology ◆ vPC configuration synchronization ◆ Link Aggregation Control Protocol (LACP): IEEE 802.3ad ◆ Jumbo Frames on all ports (up to 9216 bytes) ◆ Pause frames (IEEE 802.3x)
  • 127. Nexus 6004 supported features and topologies 127 Nexus Series Switches Setup Examples ◆ Storm control (unicast, multicast, and broadcast) ◆ Private VLANs ◆ Private VLAN over trunks (isolated and promiscuous) ◆ Private VLANs over vPC and EtherChannels ◆ VLAN Remapping ◆ Cisco FabricPath ◆ PC and vPC+ with FabricPath ◆ Cisco Adapter FEX ◆ Cisco Data Center Virtual Machine FEX (VM-FEX) ◆ Support for up to 24 fabric extenders (Layer 2) with each Cisco Nexus 6004 platform Ethernet (Layer-3) ◆ Layer 3 interfaces: Routed ports on Cisco Nexus 6004 platform interfaces, switched virtual interface (SVI), PortChannels, subinterfaces, and PortChannel subinterfaces ◆ Sixteen-way Equal-Cost Multipathing (ECMP)* ◆ Routing protocols: Static, RIPv2, EIGRP, OSPFv2, and BGP ◆ IPv6 routing protocols: Static, OPFv3, BGPv6, and EIGRPv6 ◆ IPv6 VRF-Lite ◆ HSRP and VRRP ◆ ACL: Routed ACL with Layer 3 and 4 options to match ingress and egress ACL ◆ Multicast: PIMv2 sparse mode, Source Specific Multicast (SSM), BiDir PIM, Multicast Source Discovery Protocol (MSDP), IGMP v2 and v3, and Multicast VLAN Registration (MVR) ◆ VRF: VRF-Lite (IP VPN); VRF-aware unicast; and BGP-, OSPF-, RIP-, and VRF-aware multicast ◆ URFP with ACL; strict and loose modes ◆ Jumbo Frame support (up to 9216 bytes) ◆ Support for up to 24 fabric extenders on each Cisco Nexus 6004
  • 128. 128 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Nexus 6001 supported features and topologies The supported interfaces, features and topologies have been broken down into multiple areas, and are discussed in this section. Actual reference topologies are provided in “Nexus 6000 Series and Fabric Extenders topologies” on page 143. Interfaces This section contains information on the following interfaces: ◆ “Physical interfaces” on page 128 ◆ “FCoE interfaces” on page 129 ◆ “Ethernet Fabric Extension interfaces and technologies” on page 129 ◆ “Ethernet end-device interfaces” on page 129 ◆ “FC interfaces” on page 130 Physical interfaces The following interfaces speed configurations are supported: ◆ 40GbE (True 40GbE) ◆ 40GbE (4x10GbE) ◆ 10GbE ◆ 1GbE The following optic types are supported: ◆ QSFP-40G-SR4 (MMF – 850nm) ◆ QSFP-40G-CSR4 (MMF – 850nm) ◆ SFP-10G-SR4 (MMF - 850nm) ◆ SFP-10G-LR4 (SMF) ◆ SFP-10G-ER4 (SMF) The following cable types are supported: ◆ 40G QSFP+ to 4x 10G SFP+ fibre breakout (1xMTP-to-4xLC) ◆ 40G QSFP+ to 4x 10G SFP+ copper breakout ◆ 40G QSFP+ direct-attach copper (Twinax CU 1M, 3M, 5M - passive) ◆ 40G QSFP+ direct-attach copper (Twinax CU 7M, 10M - active)
  • 129. Nexus 6001 supported features and topologies 129 Nexus Series Switches Setup Examples ◆ 10G SFP+ direct-attach copper (Twinax CU 1M, 3M, 5M - passive) ◆ 10G SFP+ direct-attach copper (Twinax CU 7M, 10M - active) ◆ Standard OM3 / OM4 Fibre with SFP+ Type LC connectors Refer to the Cisco documentation on the Cisco website at http://www.cisco.com for a current and fully updated list of supported optics and cables. FCoE interfaces The following interfaces are supported: ◆ Native FCoE CNA directly connected to Cisco Nexus 6001 SFP+ ◆ Native FCoE Storage directly connected to Cisco Nexus 6001 SFP+ • EMC VNX Series • EMC VMAX Series ◆ FC Enhanced port types: VE, TE, and VF Note: Refer to the EMC Support Matrix available through E-Lab Interoperability Navigator at http://elabnavigator.EMC.com, under the PDFs and Guides tab for a current list of supported 10GbE iSCSI, FCoE and CNA adapters. Ethernet Fabric Extension interfaces and technologies The following are supported: ◆ Cisco Adapter FEX ◆ Cisco Data Center Virtual Machine FEX (VM-FEX) ◆ Support for up to 24 fabric extenders (Layer 2) with each Cisco Nexus 6001 switch ◆ Cisco 40GbE Line Expansion Modules (LEM) Note: Refer to the EMC Support Matrix available through E-Lab Interoperability Navigator at http://elabnavigator.EMC.com, under the PDFs and Guides tab for a current list of supported fabric extenders (FEX). Ethernet end-device interfaces The following interfaces are supported: ◆ 10GbE CNA directly connected to Cisco Nexus 6001 SFP+ via breakout cable (4x10GbE) ◆ 10GbE iSCSI directly connected to Cisco Nexus 6001 SFP+ via breakout cable (4x10GbE)
  • 130. 130 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples ◆ 10GbE FCoE and iSCSI targets directly connected to Cisco Nexus 6001 SFP+ via breakout cable (4x10GbE) • EMC VNX Series • EMC VMAX Series ◆ 10GbE NICs, which adhere to industry standards, via SFP+ breakout cable (4x10GbE) ◆ PortChannels (PC) ◆ Virtual Port Channels (vPC) ◆ NIC Teaming FC interfaces The following interfaces are supported: ◆ Direct connect FC Interfaces are not supported on the Nexus 6004 ◆ FC Enhanced port types: VE, TE, and VF Features This section lists the information on the following features. ◆ “Data Center Bridging (DCB)” on page 130 ◆ “FCoE ” on page 130 ◆ “Ethernet (Layer-2)” on page 131 ◆ “Ethernet (Layer-3)” on page 132 Data Center Bridging (DCB) ◆ CEE- and IEEE-compliant PFC (per-priority Pause frame support) ◆ PFC link distance support: 300m ◆ CEE-compliant DCBX Protocol ◆ CEE- and IEEE-compliant Enhanced Transmission Selection FCoE ◆ T11 standards-compliant FCoE (FC-BB-5) ◆ T11 FCoE Initialization Protocol (FIP) (FC-BB-5) ◆ Any 10 or 40 Gigabit Ethernet port configurable as FCoE ◆ Fibre Channel forwarding (FCF) ◆ Fibre Channel enhanced port types: VE, TE, and VF ◆ Direct attachment of FCoE targets ◆ Up to 32 virtual SANs (VSANs) per switch
  • 131. Nexus 6001 supported features and topologies 131 Nexus Series Switches Setup Examples ◆ Fabric Device Management Interface (FDMI) ◆ Fibre Channel ID (FCID) persistence ◆ In-order delivery ◆ Port tracking ◆ Cisco FCoE_NPV technology ◆ N-port identifier virtualization (NPIV) ◆ Fabric services: Name server, registered state change notification (RSCN), login services, and name-server zoning ◆ Per-VSAN fabric services ◆ Cisco Fabric Services ◆ Fabric Shortest Path First (FSPF) ◆ Storage Management Initiative Specification (SMI-S) ◆ Boot from SAN over vPC and Enhanced vPC (EvPC) ◆ FCP ◆ Native Interop Mode 2 & 3 ◆ VSAN trunking ◆ Cisco NPV technology Ethernet (Layer-2) ◆ Layer 2 switch ports and VLAN trunks ◆ IEEE 802.1Q VLAN encapsulation ◆ Rapid Per-VLAN Spanning Tree Plus (PVRST+) (IEEE 802.1w compatible) ◆ Multiple Spanning Tree Protocol (IEEE 802.1s) instances: 64 instances ◆ Spanning Tree PortFast ◆ Spanning Tree root guard ◆ Spanning Tree Bridge Assurance ◆ Cisco EtherChannel technology (up to 16 ports per EtherChannel) ◆ Cisco vPC technology ◆ vPC configuration synchronization ◆ Link Aggregation Control Protocol (LACP): IEEE 802.3ad ◆ Jumbo Frames on all ports (up to 9216 bytes)
  • 132. 132 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples ◆ Pause frames (IEEE 802.3x) ◆ Storm control (unicast, multicast, and broadcast) ◆ Private VLANs ◆ Private VLAN over trunks (isolated and promiscuous) ◆ Private VLANs over vPC and EtherChannels ◆ VLAN Remapping ◆ Cisco FabricPath ◆ PC and vPC+ with FabricPath ◆ Cisco Adapter FEX ◆ Cisco Data Center Virtual Machine FEX (VM-FEX) ◆ Support for up to 24 fabric extenders (Layer 2) with each Cisco Nexus 6001 platform Ethernet (Layer-3) ◆ Layer 3 interfaces: Routed ports on Cisco Nexus 6004 platform interfaces, switched virtual interface (SVI), PortChannels, subinterfaces, and PortChannel subinterfaces ◆ Sixteen-way Equal-Cost Multipathing (ECMP)* ◆ Routing protocols: Static, RIPv2, EIGRP, OSPFv2, and BGP ◆ IPv6 routing protocols: Static, OPFv3, BGPv6, and EIGRPv6 ◆ IPv6 VRF-Lite ◆ HSRP and VRRP ◆ ACL: Routed ACL with Layer 3 and 4 options to match ingress and egress ACL ◆ Multicast: PIMv2 sparse mode, Source Specific Multicast (SSM), BiDir PIM, Multicast Source Discovery Protocol (MSDP), IGMP v2 and v3, and Multicast VLAN Registration (MVR) ◆ VRF: VRF-Lite (IP VPN); VRF-aware unicast; and BGP-, OSPF-, RIP-, and VRF-aware multicast ◆ URFP with ACL; strict and loose modes ◆ Jumbo Frame support (up to 9216 bytes) ◆ Support for up to 24 fabric extenders on each Cisco Nexus 6001
  • 133. Nexus 2248PQ FEX supported features and topologies 133 Nexus Series Switches Setup Examples Nexus 2248PQ FEX supported features and topologies The supported interfaces, features and topologies have been broken down into multiple areas, and are discussed in this section. Actual reference topologies are provided in “Nexus 6000 Series and Fabric Extenders topologies” on page 143. Interfaces This section contains information on the following interfaces: ◆ “Physical interfaces” on page 133 ◆ “FCoE interfaces” on page 134 ◆ “Ethernet end-device interfaces” on page 134 Physical interfaces The following interfaces speed configurations are supported: ◆ 40GbE (4x10GbE) ◆ 10GbE ◆ 1GbE The following optic types are supported: ◆ QSFP-40G-SR4 (MMF – 850nm) ◆ QSFP-40G-CSR4 (MMF – 850nm) ◆ SFP-10G-SR4 (MMF - 850nm) ◆ SFP-10G-LR4 (SMF) The following cable types are supported: ◆ 40G QSFP+ to 4x 10G SFP+ fibre breakout (1xMTP-to-4xLC) ◆ 40G QSFP+ to 4x 10G SFP+ copper breakout ◆ 40G QSFP+ direct-attach copper (Twinax CU 1M, 3M, 5M - passive) ◆ 40G QSFP+ direct-attach copper (Twinax CU 7M, 10M - active) ◆ 10G SFP+ direct-attach copper (Twinax CU 1M, 3M, 5M - passive) ◆ 10G SFP+ direct-attach copper (Twinax CU 7M, 10M - active) ◆ Standard OM3 / OM4 Fibre with SFP+ Type LC connectors
  • 134. 134 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Refer to the Cisco documentation on the Cisco website at http://www.cisco.com for a current and fully updated list of supported optics and cables. FCoE interfaces The following interface is supported: ◆ Native 10GbE FCoE CNA directly connected to Cisco Nexus 2248PQ SFP+ Note: Refer to the EMC Support Matrix available through E-Lab Interoperability Navigator at http://elabnavigator.EMC.com, under the PDFs and Guides tab for a current list of supported 10GbE iSCSI, FCoE and CNA adapters. Ethernet end-device interfaces The following interfaces are supported: ◆ 10GbE CNA directly connected to Cisco Nexus 2248PQ ◆ 10GbE iSCSI directly connected to Cisco Nexus 2248PQ ◆ 10GbE FCoE and iSCSI targets directly connected to Cisco Nexus 2248PQ ◆ 10GbE NICs, which adhere to industry standards ◆ PortChannels (PC) ◆ Virtual Port Channels (vPC) ◆ NIC Teaming Features This section lists the information on the following features. ◆ “Data Center Bridging (DCB)” on page 130 ◆ “Ethernet (Layer-2)” on page 131 Data Center Bridging (DCB) ◆ DCB - Data Center Bridging Ethernet (Layer-2) ◆ IEEE 802.1p: CoS prioritization ◆ IEEE 802.1Q: VLAN tagging ◆ IEEE 802.3: Ethernet ◆ IEEE 802.3ae: 10 Gigabit Ethernet Layer ◆ Layer 2 VLAN trunks
  • 135. Nexus 2248PQ FEX supported features and topologies 135 Nexus Series Switches Setup Examples ◆ IEEE 802.1Q VLAN encapsulation ◆ Cisco EtherChannel technology on uplinks ◆ PortChannel on server ports on Cisco Nexus 2200 platforms ◆ Advanced PortChannel hashing ◆ Jumbo Frames on all ports (up to 9216 bytes) ◆ Pause frames (priority flow control [PFC] and IEEE 802.3x) ◆ Private VLANs (promiscuous only on uplinks)
  • 136. 136 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Nexus 5000 supported features and topologies The supported features and topologies have been broken down into four areas, as shown in Figure 20, and are discussed in this section. ◆ “FCoE interface” on page 137 ◆ “Ethernet uplink interface” on page 138 ◆ “FC interface” on page 139 ◆ “Ethernet end-device interface” on page 140 Figure 20 is for clarification purposes only. Actual reference topologies are provided in the following sections. Figure 20 Topology example MDS Brocade FC Interface Ethernet uplink Interface Ethernet end device Interface FCoE Interface Nexus 5000 Ethernet Bridge (Switch) 10 GbE NIC ServerServer CNA Storage (e.g., VMAX, VNX, CX4) Storage (e.g., VMAX, VNX, CX4) Storage (e.g., VNXe) GEN-001009a
  • 137. Nexus 5000 supported features and topologies 137 Nexus Series Switches Setup Examples FCoE interface The following interfaces are supported: ◆ CNAs directly connected via physical media to the Nexus 5000 Note: Connecting the CNA to the Nexus via a stand-alone Top of Rack (TOR) or End Of Row (EOR) intermediate 10G Ethernet Bridge (either DCB capable or not) is not supported unless stated otherwise in this section. ◆ IBM Blade Server Nexus 4001I FIP Snooping connections ◆ VE_Ports are supported starting with NX-OS 5.0(2)N2(1) • VE_Ports allow for FCoE ISLs to be created and enable multi-hop FCoE ◆ Physical connectivity • Fabric A /Fabric B connectivity is supported • An example of Fabric A /Fabric B connectivity is two Nexus 5020s, each connected to one of the two sides of a mirrored fabric. • SFP+ optical (short reach only) • SFP+ cu (Twinax - Molex) • 1 m, 3 m, 5 m, 7 m, and 10 m are available Note: Refer to "Enabling technologies" section, "Introduction to Fibre Channel over Ethernet" chapter in the Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook at http://elabnavigator.EMC.com, under the Topology Resource Center tab. Note: Refer to the “Twinax interoperability constraints” table, "Introduction to Fibre Channel over Ethernet" chapter in the Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook at http://elabnavigator.EMC.com, under the Topology Resource Center tab, for information on interoperability constraints. ◆ Hardware-based FCoE CNAs from Brocade, Emulex, QLogic Note: Refer to the EMC Support Matrix for the most up-to-date support information.
  • 138. 138 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples ◆ Software-based FCoE CNAs from Intel and Broadcom ◆ Native FCoE storage • VNX • CX4 • VMAX Ethernet uplink interface The following are supported: ◆ Preferred Ethernet connectivity to Catalyst 6k or Nexus 7k ◆ Supported UNH interop approved Ethernet for 10 GbE • 1 GigE is supported on the first eight ports of the Nexus 5010 and the first 16 ports of the Nexus 5020 when 1 Gb/s optical/Cu interface SFPx modules are used ◆ PAUSE 802.3x is supported but will not be required on Nexus 5020 uplink ◆ Etherchannels for link aggregation • Support up to 256 VLANs ◆ 802.1w, Per VLAN Rapid Spanning Tree (PVRST+) • Preferred, may be better if available ◆ 802.1s Multiservice Transport Platform (MSTP) • Most widely deployed ◆ Spanning Tree PortFast and PortFast Guard ◆ Spanning Tree UplinkFast and BackboneFast ◆ Bridge Protocol Data Unit (BPDU) Guard ◆ Spanning Tree Root Guard ◆ Internet Group Management Protocol (IGMP) snooping ◆ IGMP snooping querier ◆ Protocol Independent Multicast (PIM) ◆ Cisco Etherchannel • 8 members per channel • 12 channels per chassis
  • 139. Nexus 5000 supported features and topologies 139 Nexus Series Switches Setup Examples • VSAN truck pruning done through configuration; no VTP support on Nexus 5000 ◆ Link Aggregation Control Protocol (LACP) ◆ Advanced port channel hashing ◆ Jumbo frames up to 9216 bytes ◆ Storm control ◆ Private VLANs / Private VLANs over trunks ◆ Virtual PortChannel (vPC) is supported • See “Virtual PortChannel” on page 293 for configuration information ◆ VE_Ports are supported FC interface Note: The Nexus 5000 can be configured to run in NPV mode to connect to either Brocade (FOS or EOS) or MDS products, but not at the same time. ◆ NPV — Brocade and Cisco • Minimum number of NP_Ports per Nexus 5000 is 1 • Maximum number of NP_Ports per Nexus 5000 is 16 • Maximum distance is shortwave – Longwave is currently not supported ◆ FC-SW — Cisco MDS only • Maximum number of ISLs per Nexus 5000 is 16 • Max domains is per the EMC Support Matrix • Maximum distance is shortwave – Longwave is not currently supported • Max hops is per the EMC Support Matrix • Interopmode is native • Port channels (link aggregation) is supported • Trunking (multiple VSANs) is supported • Scalability – Same as MDS (except VSAN up to 32)
  • 140. 140 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples • HBA and storage direct connect to the Nexus FC ports are supported starting with the 4.0(1a)N1(1a) firmware release. • The supported HBA and storage ports for the Nexus are the same as those for the MDS. Note: For the most up-to-date support information, refer to the EMC Support Matrix. Ethernet end-device interface The following are supported: ◆ Generic 10 GbE NICs that conform to media requirements ◆ Celerra® 10 GbE (NAS+iSCSI+MPFS) ◆ Symmetrix, VNX series, or CLARiiON iSCSI interfaces ◆ NIC Teaming ◆ Virtual PortChannels (vPC)
  • 141. Nexus 2000 supported features and topologies 141 Nexus Series Switches Setup Examples Nexus 2000 supported features and topologies The supported features and topologies have been broken down into three areas as shown in Figure 21, each discussed further: ◆ “FCoE interface” on page 141 ◆ “Ethernet uplink interface” on page 142 ◆ “Ethernet end-device interface” on page 142 Figure 21 is for clarification purposes only. Actual reference topologies are provided in the following sections. Figure 21 Nexus 2000 supported features and topologies FCoE interface The following interfaces are supported: ◆ CNAs directly connected via physical media to the Nexus 5000 Note: Connecting the CNA to the Nexus via a stand-alone Top of Rack (TOR) or End Of Row (EOR) intermediate 10G Ethernet Bridge (either DCB capable or not) is not supported unless stated otherwise in this section. Ethernet uplink Interface Ethernet end device Interface FCoE Interface Nexus 2232PP Nexus 5000 Parent switch Celerra 10 GbE NIC 10 GbE NIC ServerServer CNA CNA Server ICO-IMG-000853
  • 142. 142 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples ◆ IBM Blade Server Nexus 4001I FIP Snooping connections ◆ Physical connectivity • SFP+ optical (short reach only) • SFP+ cu (Twinax - Molex) • 1 m, 3 m, and 5 m are available ◆ Hardware-based FCoE CNAs from Brocade, Emulex, QLogic Note: Refer to the EMC Support Matrix for the most up-to-date support information. ◆ Software-based FCoE CNAs from Intel and Broadcom Ethernet uplink interface The Nexus 2232PP must be directly connected to a Nexus 5000. The following are supported: ◆ Features • Static Pinning Fabric Interface Connections • EtherChannel Fabric Interface Connections • Virtual Port Channel (VPC) ◆ Physical connectivity • Fabric Extender Transceivers (FET) • SFP+ optical (short reach only) • SFP+ cu (Twinax - Molex) • 1 m, 3 m, and 5 m are available Ethernet end-device interface The following are supported: ◆ Generic 10 GbE NICs that conform to media requirements ◆ Celerra 10 GbE (NAS+iSCSI+MPFS) ◆ Symmetrix, VNX series, or CLARiiON iSCSI interfaces ◆ NIC Teaming
  • 143. Nexus 6000 Series and Fabric Extenders topologies 143 Nexus Series Switches Setup Examples Nexus 6000 Series and Fabric Extenders topologies This section contains information for the following Nexus 6000 series topologies. ◆ “Multihop FCoE” on page 143 ◆ “Leaf and Spine (Layer-2/Layer-3) fabric” on page 144 ◆ “High Performance Compute (HPC) and High Frequency Trading (HFT)” on page 145 ◆ “Scalable data center access and aggregation with Fabric Extenders” on page 146 ◆ “Fabric Extenders (FEX)” on page 147 Multihop FCoE The Cisco Nexus 6004 can support end-to-end data center convergence from server to storage by delivering multihop FCoE capability in the data center. The FCoE function complements the existing FCoE function on the Cisco Nexus 5500 Switch platform. With the addition of the Nexus 6000 series switches, Cisco Unified Fabric supports both the access and core network layers, supporting all storage traffic (FCoE, Small Computer System Interface over IP [iSCSI], and network attached storage (NAS) over simplified infrastructure based on lossless 10 and 40 Gigabit Ethernet.
  • 144. 144 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Figure 22 shows an example of a Nexus 6004/6001 40GbE multihop FCoE topology. Figure 22 Cisco Nexus 6004/6001 40GbE Multihop FCoE topology example Leaf and Spine (Layer-2/Layer-3) fabric Note: Some information in this section comes from Cisco documentation. For more detailed information, refer to http://www.cisco.com. The Cisco Nexus 6004 can be deployed as a leaf or spine node in a Layer 2 or Layer 3 fabric design. Leaf and spine layer designs using high-density and low-latency switches lead to flatter network architectures, allowing connections scaling from hundreds to more than 10,000 servers with large bisectional bandwidth. The Leaf and Spine design helps ensure low-latency fabric with minimal hop count. ◆ The spine switches create a non-blocking, low-latency fabric, forwarding packets between leaves. ◆ The leaf switches provide connectivity to servers.
  • 145. Nexus 6000 Series and Fabric Extenders topologies 145 Nexus Series Switches Setup Examples Using a highly-meshed architecture helps ensure the highest possible network availability with minimal effect on traffic if a failure occurs. Cisco Nexus 6004 can be deployed as a Layer 2 or Layer 3 spine or leaf switch, providing a high degree of design flexibility. Figure 23 shows an example of a Leaf and Spine topology with Nexus 6004/6001 switches. Figure 23 Cisco Nexus 6004/6001 Leaf and Spine topology example High Performance Compute (HPC) and High Frequency Trading (HFT) Note: Some information in this section comes from Cisco documentation. For more detailed information, refer to http://www.cisco.com. The Cisco Nexus 6004 can be deployed as a high-density access layer switch to consolidate a large number of 10 and 40 Gigabit Ethernet servers where the deployment calls for a minimal number of hops from the server to the upstream network to minimize latency. The Cisco Nexus 6004 has a high density of 10 and 40 Gigabit Ethernet ports per rack unit, ~1 microsecond latency port to port for any packet size and integrated line-rate Layer 2 and Layer 3 features with scale.
  • 146. 146 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Figure 24 shows an example of a High Performance Compute (HPC) and High Frequency Trading (HFT) topology with Nexus 6004/6001 switches. Figure 24 Cisco Nexus 6004/6001 HPC and HFT topology example Scalable data center access and aggregation with Fabric Extenders Note: Some information in this section comes from Cisco documentation. For more detailed information, refer to http://www.cisco.com. Data center designs are increasingly moving toward less oversubscription starting from the server access layer. Moreover, growth of 10 Gigabit Ethernet server deployments is accelerating the need for top-of-rack (ToR) solutions that can provide 40 Gigabit Ethernet uplink capabilities to meet the oversubscription requirement. The Cisco Nexus 6004 with high-density 40 Gigabit Ethernet ports can be utilized as a compact aggregation platform for the ToR switches with 40 Gigabit Ethernet uplinks.
  • 147. Nexus 6000 Series and Fabric Extenders topologies 147 Nexus Series Switches Setup Examples Figure 25 shows an example of scalable data center access and aggregation using Fabric Extenders. Figure 25 Scalable data center access and aggregation with Fabric Extenders example Fabric Extenders (FEX) Note: Some information in this section comes from Cisco documentation. For more detailed information, refer to http://www.cisco.com. The Cisco Nexus 2000 Series can be used in conjunction with a Cisco Nexus parent switch in two main design scenarios (shown in Figure 22 on page 144). Note the following deployment scenarios: ◆ Cisco Nexus 2000 Series Fabric Extenders single-connected to one upstream Cisco Nexus 5000, Nexus 6000, or Nexus 7000 Series Switch In this deployment scenario, access-layer redundancy is achieved through redundant server connections to two upstream distributed modular systems using vPC (Cisco Nexus 5000 or Nexus 6000 Series) or server NIC teaming to two Cisco Nexus 2000 Series Fabric Extenders. ◆ Cisco Nexus 2000 Series Fabric Extenders dual-connected to two upstream Cisco Nexus 5000 or Nexus 6000 Series Switches (vPC)
  • 148. 148 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples In this deployment scenario, access-layer redundancy is achieved through a combination of Cisco Nexus 2000 Series Fabric Extenders dual-connected to an upstream parent switch and server NIC teaming. ◆ Enhanced vPC (EvPC): In this deployment scenario, access layer redundancy is achieved in two ways: through redundant connections between the Cisco Nexus 2000 Fabric Extenders and the Cisco Nexus parent switches using vPC, and through redundant server connections to two fabric extenders using vPC and active-active server NIC teaming. ◆ vPC+ In this deployment scenario, access-layer redundancy is achieved through server vPC, FEX vPC, and EvPC. In addition, a vPC+ domain allows the Cisco Nexus parent switch and the fabric extenders to be viewed as a single virtual switch in a Cisco FabricPath network. Figure 26 shows an example of these four scenarios. Figure 26 Cisco Nexus 2000 Series Fabric Extenders design scenarios
  • 149. Configuring 40GbE FCoE connectivity with Nexus 6000 Series 149 Nexus Series Switches Setup Examples Configuring 40GbE FCoE connectivity with Nexus 6000 Series This section describes the setup steps required to implement 40GbE FCoE connectivity between the Nexus 6004 and the three different 40GbE supported options (switch, fabric extender, and LEM). Note: Cisco Nexus 6000 switches can be used in place of the Cisco Nexus 5000 switches used in the other case studies defined in this TechBook. This section focuses only on the connectivity options being introduced by the Cisco Nexus 6000. This section is not intended to address a fully high-availability functional fabric, rather 40GbE connectivity. Refer to Figure 27 on page 152 for the configuration used in this section. This section includes the following information: ◆ “Prerequisites” on page 149 ◆ “Summary of configuration steps” on page 150 ◆ “Individual reference topologies” on page 152 Prerequisites This section contains prerequisites for the Cisco Nexus 6004, 6001, 5596UP, and 2248PQ FEX switches and fabric extender. Nexus 6004 ◆ Cisco NX-OS 6.0(2)N1(2a) minimum release ◆ Appropriate cable for 40GbE ISL connectivity (Ex. QSFP-H40G-CU1M, etc.) ◆ Appropriate QSFP+ optics for breakout QSFP+ to 4x10GbE cable for host or storage 10GbE connectivity (Ex. QSFP-40G-SR4) ◆ Appropriate QSFP-to-4x10GbE breakout cable ◆ 12 port 40GbE line expansion modules (LEM), if needed ◆ All required license keys are installed Nexus 6001 ◆ Cisco NX-OS 6.0(2)N1(2a) minimum release ◆ Appropriate cable for 40GbE ISL connectivity (Ex. QSFP-H40G-CU1M, etc.)
  • 150. 150 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples ◆ All required license keys are installed Nexus 5596UP ◆ Cisco NX-OS 6.0(2)N1(2a) minimum release ◆ Appropriate cable for 40GbE ISL connectivity (Ex. QSFP-H40G-CU1M, etc.) ◆ Cisco 40GbE expansion module: N55-M4Q ◆ All required license keys are installed Nexus 2248PQ FEX ◆ Cisco NX-OS 6.0(2)N1(2a) minimum release (installed via parent switch) ◆ Appropriate cable for 40GbE ISL connectivity (Ex. QSFP-H40G-CU1M, etc.) Summary of configuration steps This section summarizes the configuration steps detailed in the following sections: ◆ “Connecting and configuring a Cisco Nexus 6004 to another Cisco Nexus 6004” on page 152 ◆ “Connecting and configuring a Cisco Nexus 6004 to Cisco Nexus 6001” on page 157 ◆ “Connecting and configuring a Cisco Nexus 6004 to a Cisco Nexus 5596UP with LEM N55-M4Q” on page 162 ◆ “Connecting and configuring a Cisco Nexus 6004 to a Cisco Nexus 2248PQ FEX” on page 168 Connecting and configuring a Cisco Nexus 6004 to another Cisco Nexus 6004 1. Enable features. 2. Create VLAN/VSAN. 3. Configure 40GbE PC. 4. Create vfcs. 5. Configure 40GbE (4x10GbE) breakout ports for storage/hosts 10GbE. 6. Create vfcs. 7. Connect cables.
  • 151. Configuring 40GbE FCoE connectivity with Nexus 6000 Series 151 Nexus Series Switches Setup Examples Connecting and configuring a Cisco Nexus 6004 to Cisco Nexus 6001 1. Enable features. 2. Create VLAN/VSAN. 3. Configure 40GbE PC. 4. Create vfcs. 5. Configure 40GbE (4x10GbE) breakout ports for storage/hosts 10GbE. 6. Create vfcs. 7. Connect cables. Connecting and configuring a Cisco Nexus 6004 to a Cisco Nexus 5596UP with LEM N55-M4Q 1. Enable features. 2. Create VLAN/VSAN. 3. Configure 40GbE breakout ports for PC. 4. Configure 40GbE (4x10GbE) PC. 5. Create vfcs. 6. Configure 40GbE (4x10GbE) breakout ports for storage/hosts 10GbE. 7. Create vfcs. 8. Connect cables. Connecting and configuring a Cisco Nexus 6004 to a Cisco Nexus 2248PQ FEX 1. Enable features. 2. Create VLAN/VSAN. 3. Configure 40GbE (4x10GbE) PC. 4. Create and associate FEX. 5. Configure 40GbE (4x10GbE) breakout ports for storage/hosts 10GbE. 6. Create vfcs. 7. Connect cables.
  • 152. 152 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Individual reference topologies Figure 27 shows the reference topologies used in the following four configurations discussed in this section. ◆ “Connecting and configuring a Cisco Nexus 6004 to another Cisco Nexus 6004” on page 152 ◆ “Connecting and configuring a Cisco Nexus 6004 to Cisco Nexus 6001” on page 157 ◆ “Connecting and configuring a Cisco Nexus 6004 to a Cisco Nexus 5596UP with LEM N55-M4Q” on page 162 ◆ “Connecting and configuring a Cisco Nexus 6004 to a Cisco Nexus 2248PQ FEX” on page 168 Figure 27 Reference topologies Connecting and configuring a Cisco Nexus 6004 to another Cisco Nexus 6004 On both Cisco Nexus 6004 switches Complete the following steps: 1. Enable the FCoE feature. a. feature fcoe 2. Enable LACP. a. feature lacp
  • 153. Configuring 40GbE FCoE connectivity with Nexus 6000 Series 153 Nexus Series Switches Setup Examples 3. Enable LLDP. a. feature lldp 4. Enable vPC. a. feature vpc 5. Create VLAN/VSAN and map FCoE LAN to VSAN. a. vsan database b. vsan 2 c. vlan 2 d. fcoe vsan 2 e. exit 6. Create Port-Channel. a. interface port-channel 4 b. switchport mode trunk c. switchport trunk allowed vlan 2,100,200 d. no shutdown e. exit 7. Configure interfaces for port-channel 4. a. interface ethernet 2/1-2 b. switchport mode trunk c. switchport trunk allowed vlan 2,100,200 d. channel-group 4 e. exit 8. Create vFC for port-channel. a. interface vfc 104 b. bind interface port-channel4 c. switchport mode E d. switchport trunk allowed vsan 2 e. no shutdown f. exit
  • 154. 154 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 9. Associate vFC of port-channel to VSAN. a. vsan database b. vsan 2 interface vfc 104 c. exit 10. Validate. a. show port-channel database b. show interface vfc 103 c. show interface ethernet 2/1-2 d. show vsan membership e. show vsan 2 f. show fcoe database g. show vlan fcoe h. show interface brief i. show running-config 11. Save the configuration. a. copy running-config startup-config 12. Breakout True 40GbE interface ports into 4x10GbE for 10GbE storage connectivity, if 40GbE storage ports not available. a. show interface brief b. show module c. interface breakout slot 1 port 7-9 map 10g-4x d. poweroff module 1 e. show module f. no poweroff module 1 g. show module h. show interface brief 13. Configure breakout 10GbE port for storage connectivity. a. interface ethernet 1/7/1 b. spanning-tree port type edge trunk
  • 155. Configuring 40GbE FCoE connectivity with Nexus 6000 Series 155 Nexus Series Switches Setup Examples c. switchport mode trunk d. switchport trunk allowed vlan 2 e. no shutdown f. exit 14. Create vFC for storage interface. a. interface vfc 1 b. bind interface ethernet 1/7/1 c. no shutdown d. exit 15. Associate storage vFC to VSAN. a. vsan database b. vsan 2 interface vfc 1 c. exit 16. Validate. a. show interface vfc 1 b. show fcoe database c. show interface fcoe d. show flogi database e. show interface Ethernet 1/7/1 f. show vsan membership g. show vlan h. show interface brief 17. Save the configuration. a. copy running-config startup-config 18. Configure 10GbE interface port for host CNA connectivity. a. interface ethernet 1/7/2 b. spanning-tree port type edge trunk c. switchport mode trunk d. switchport trunk allowed vlan 2
  • 156. 156 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples e. no shutdown f. exit 19. Create vFC for storage interface. a. interface vfc 1 b. bind interface ethernet 1/7/2 c. no shutdown d. exit 20. Associate storage vFC to VSAN. a. vsan database b. vsan 2 interface vfc 1 c. exit 21. Validate. a. show interface vfc 1 b. show fcoe database c. show interface fcoe d. show flogi database e. show interface Ethernet 1/7/2 f. show vsan membership g. show vlan h. show interface brief 22. Save the configuration. a. copy running-config startup-config 23. Create zones, zoneset and activate on FCoE VSAN. Refer to Cisco zoning documentation 24. Verify. a. VSAN zone mode (basic, enhanced) matches across switches – show zone status b. zoneset is active on FCoE VSAN – show zoneset active vsan 2
  • 157. Configuring 40GbE FCoE connectivity with Nexus 6000 Series 157 Nexus Series Switches Setup Examples c. Verify port-channel and vfc are up – show port-channel database – show interface port-channel 3 – show interface vfc 103 d. Verify switches are merged and zoneset is active on appropriate VSAN across all switches in the fabric – show fcdomain – show fcs database – show fcns database On the storage array 1. Check Storage array to ensure that host initiator is logged into array. 2. Configure device masking. On the host 1. Rescan host bus. 2. Ensure host discovers LUNs. Connecting and configuring a Cisco Nexus 6004 to Cisco Nexus 6001 On both Nexus 6004 and 6001 Complete the following steps: 1. Enable the FCoE feature. a. feature fcoe 2. Enable LACP. a. feature lacp 3. Enable LLDP. a. feature lldp 4. Enable vPC. a. feature vpc 5. Create VLAN/VSAN and map FCoE LAN to VSAN. a. vsan database b. vsan 2 c. vlan 2 d. fcoe vsan 2 e. exit
  • 158. 158 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 6. Create Port-Channel. a. interface port-channel 3 b. switchport mode trunk c. switchport trunk allowed vlan 2,100,200 d. no shutdown e. exit 7. Configure interfaces for port-channel 3. a. interface ethernet 3/1-2 b. switchport mode trunk c. switchport trunk allowed vlan 2,100,200 d. channel-group 3 e. exit 8. Create vFC for port-channel. a. interface vfc 103 b. bind interface port-channel 3 c. switchport mode E d. switchport trunk allowed vsan 2 e. no shutdown f. exit 9. Associate vFC of port-channel to VSAN. a. vsan database b. vsan 2 interface vfc 103 c. exit 10. Validate. a. show port-channel database b. show interface vfc 103 c. show interface ethernet 3/1-2 d. show vsan membership e. show vsan 2
  • 159. Configuring 40GbE FCoE connectivity with Nexus 6000 Series 159 Nexus Series Switches Setup Examples f. show fcoe database g. show vlan fcoe h. show interface brief i. show running-config 11. Save the configuration. a. copy running-config startup-config On Nexus 6004 only 1. Breakout True 40GbE interface ports into 4x10GbE for 10GbE storage connectivity, if 40GbE storage ports not available. a. show interface brief b. show module c. interface breakout slot 1 port 7-9 map 10g-4x d. poweroff module 1 e. show module f. no poweroff module 1 g. show module h. show interface brief 2. Configure breakout 10GbE port for storage connectivity. a. interface ethernet 1/7/1 b. spanning-tree port type edge trunk c. switchport mode trunk d. switchport trunk allowed vlan 2 e. no shutdown f. exit 3. Create vFC for storage interface. a. interface vfc 1 b. bind interface ethernet 1/7/1 c. no shutdown d. exit 4. Associate storage vFC to VSAN.
  • 160. 160 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples a. vsan database b. vsan 2 interface vfc 1 c. exit 5. Validate. a. show interface vfc 1 b. show fcoe database c. show interface fcoe d. show flogi database e. show interface Ethernet 1/7/1 f. show vsan membership g. show vlan h. show interface brief 6. Save the configuration. a. copy running-config startup-config On Nexus 6001 only 1. Configure 10GbE interface port for host CNA connectivity. a. interface ethernet 1/7/2 b. spanning-tree port type edge trunk c. switchport mode trunk d. switchport trunk allowed vlan 2 e. no shutdown f. exit 2. Create vFC for storage interface. a. interface vfc 1 b. bind interface ethernet 1/7/2 c. no shutdown d. exit 3. Associate storage vFC to VSAN. a. vsan database b. vsan 2 interface vfc 1
  • 161. Configuring 40GbE FCoE connectivity with Nexus 6000 Series 161 Nexus Series Switches Setup Examples c. exit 4. Validate. a. show interface vfc 1 b. show fcoe database c. show interface fcoe d. show flogi database e. show interface Ethernet 1/7/1 f. show vsan membership g. show vlan h. show interface brief 5. Save the configuration. a. copy running-config startup-config On Nexus 6004 only 1. Create zones, zoneset and activate on FCoE VSAN. Refer to Cisco zoning documentation. On Nexus 6004 and 6001 1. Verify. a. VSAN zone mode (basic, enhanced) matches across switches – show zone status b. zoneset is active on FCoE VSAN – show zoneset active vsan 2 c. Verify port-channel and vfc are up – show port-channel database – show interface port-channel 3 – show interface vfc 103 d. Verify switches are merged and zoneset is active on appropriate VSAN across all switches in the fabric – show fcdomain – show fcs database – show fcns database On the storage array 1. Check Storage array to ensure that host initiator is logged into array.
  • 162. 162 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 2. Configure device masking. On the host 1. Rescan host bus. 2. Ensure host discovers LUNs. Connecting and configuring a Cisco Nexus 6004 to a Cisco Nexus 5596UP with LEM N55-M4Q On both Nexus 6004 and 5596 Complete the following steps. 1. Enable the FCoE feature. a. feature fcoe 2. Enable LACP. a. feature lacp 3. Enable LLDP. a. feature lldp 4. Enable vPC. a. feature vpc 5. Create VLAN/VSAN and map FCoE LAN to VSAN. a. vsan database b. vsan 2 c. vlan 2 d. fcoe vsan 2 e. exit On Nexus 6004 only 1. Breakout True 40GbE interface ports into 4x10GbE for connectivity to the Nexus 5596 N55-M4Q 40GbE line expansion module. a. show interface brief b. show module c. interface breakout slot 1 port 1-3 map 10g-4x d. poweroff module 1 e. show module f. no poweroff module 1
  • 163. Configuring 40GbE FCoE connectivity with Nexus 6000 Series 163 Nexus Series Switches Setup Examples g. show module h. show interface brief 2. Create Port-Channel. a. interface port-channel 1 b. switchport mode trunk c. switchport trunk allowed vlan 2,100,200 d. no shutdown e. exit 3. Configure interfaces for port-channel 1. a. interface ethernet 1/1/1-4 b. switchport mode trunk c. switchport trunk allowed vlan 2,100,200 d. channel-group 1 e. exit 4. Create vFC for port-channel. a. interface vfc 100 b. bind interface port-channel 1 c. switchport mode E d. switchport trunk allowed vsan 2 e. no shutdown f. exit 5. Associate vFC of port-channel to VSAN. a. vsan database b. vsan 2 interface vfc 100 c. exit 6. Validate. a. show port-channel database b. show interface vfc 100 c. show interface ethernet 1/1/1-4
  • 164. 164 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples d. show vsan membership e. show vsan 2 f. show fcoe database g. show vlan fcoe h. show interface brief i. show running-config 7. Save the configuration. a. copy running-config startup-config On Nexus 5596 only 1. Create Port-Channel. a. interface port-channel 1 b. switchport mode trunk c. switchport trunk allowed vlan 2,100,200 d. no shutdown e. exit 2. Configure interfaces for port-channel 1. a. interface ethernet 4/1/1-4 b. switchport mode trunk c. switchport trunk allowed vlan 2,100,200 d. channel-group 1 e. exit 3. Create vFC for port-channel. a. interface vfc 100 b. bind interface port-channel 1 c. switchport mode E d. switchport trunk allowed vsan 2 e. no shutdown f. exit 4. Associate vFC of port-channel to VSAN. a. vsan database
  • 165. Configuring 40GbE FCoE connectivity with Nexus 6000 Series 165 Nexus Series Switches Setup Examples b. vsan 2 interface vfc 100 c. exit 5. Validate. a. show port-channel database b. show interface vfc 100 c. show interface ethernet 4/1/1-4 d. show vsan membership e. show vsan 2 f. show fcoe database g. show vlan fcoe h. show interface brief i. show running-config 6. Save the configuration. a. copy running-config startup-config On Nexus 6004 only 1. Breakout True 40GbE interface ports into 4x10GbE for 10GbE storage connectivity, if 40GbE storage ports not available. a. show interface brief b. show module c. interface breakout slot 1 port 7-9 map 10g-4x d. poweroff module 1 e. show module f. no poweroff module 1 g. show module h. show interface brief 2. Configure breakout 10GbE port for storage connectivity. a. interface ethernet 1/7/1 b. spanning-tree port type edge trunk c. switchport mode trunk d. switchport trunk allowed vlan 2
  • 166. 166 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples e. no shutdown f. exit 3. Create vFC for storage interface. a. interface vfc 1 b. bind interface ethernet 1/7/1 c. no shutdown d. exit 4. Associate storage vFC to VSAN. a. vsan database b. vsan 2 interface vfc 1 c. exit 5. Validate. a. show interface vfc 1 b. show fcoe database c. show interface fcoe d. show flogi database e. show interface Ethernet 1/7/1 f. show vsan membership g. show vlan h. show interface brief 6. Save the configuration. a. copy running-config startup-config On Nexus 5596 only 1. Configure breakout 10GbE interface port for host CNA connectivity. a. interface ethernet 1/31 b. spanning-tree port type edge trunk c. switchport mode trunk d. switchport trunk allowed vlan 2 e. no shutdown
  • 167. Configuring 40GbE FCoE connectivity with Nexus 6000 Series 167 Nexus Series Switches Setup Examples f. exit 2. Create vFC for storage interface. a. interface vfc 1 b. bind interface ethernet 1/31 c. no shutdown d. exit 3. Associate storage vFC to VSAN. a. vsan database b. vsan 2 interface vfc 1 c. exit 4. Validate. a. show interface vfc 1 b. show fcoe database c. show interface fcoe d. show flogi database e. show interface Ethernet 1/31 f. show vsan membership g. show vlan h. show interface brief 5. Save the configuration. a. copy running-config startup-config On Nexus 6004 only 1. Create zones, zoneset and activate on FCoE VSAN. Refer to Cisco zoning documentation. On Nexus 6004 and 5596 1. Verify. a. VSAN zone mode (basic, enhanced) matches across switches – show zone status b. zoneset is active on FCoE VSAN – show zoneset active vsan 2 c. Verify port-channel and vfc are up
  • 168. 168 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples – show port-channel database – show interface port-channel 3 – show interface vfc 103 d. Verify switches are merged and zoneset is active on appropriate VSAN across all switches in the fabric – show fcdomain – show fcs database – show fcns database On the storage array 1. Check Storage array to ensure that host initiator is logged into array. 2. Configure device masking. On the host 1. Rescan host bus. Ensure host discovers LUNs. Connecting and configuring a Cisco Nexus 6004 to a Cisco Nexus 2248PQ FEX On Nexus 6004 with 2248PQ FEX Complete the following steps: 1. Enable the FCoE feature. a. feature fcoe 2. Enable LACP. a. feature lacp 3. Enable LLDP. a. feature lldp 4. Enable vPC. a. feature vpc 5. Enable FEX b. feature fex 6. Create VLAN/VSAN and map FCoE LAN to VSAN. a. vsan database b. vsan 2 c. vlan 2 d. fcoe vsan 2
  • 169. Configuring 40GbE FCoE connectivity with Nexus 6000 Series 169 Nexus Series Switches Setup Examples e. exit 7. Create Port-Channel. a. interface port-channel 2 b. switchport c. switchport mode fex-fabric d. switchport trunk allowed vlan 2 e. fex associate 100 f. no shutdown g. exit 8. Configure interfaces for port-channel 2. a. interface ethernet 4/1/1-4 b. switchport mode fex fabric c. switchport trunk allowed vlan 2 d. fex associate 100 e. channel-group 2 f. no shutdown g. exit 9. Validate. a. show interface brief b. show fex detail c. show module fex all d. show fex e. show fex 100 f. show fex 100 detail g. show interface fex-fabric h. show port-channel database i. show interface port-channel 2 j. show interface ethernet 4/1/1-4 k. show running-config
  • 170. 170 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 10. Save the configuration. a. copy running-config startup-config 11. Breakout True 40GbE interface ports into 4x10GbE for 10GbE storage connectivity, if 40GbE storage ports not available. a. show interface brief b. show module c. interface breakout slot 1 port 7-9 map 10g-4x d. poweroff module 1 e. show module f. no poweroff module 1 g. show module h. show interface brief 12. Configure breakout 10GbE port for storage connectivity. a. interface ethernet 1/7/1 b. spanning-tree port type edge trunk c. switchport mode trunk d. switchport trunk allowed vlan 2 e. no shutdown f. exit 13. Create vFC for storage interface. a. interface vfc 1 b. bind interface ethernet 1/7/1 c. no shutdown d. exit 14. Associate storage vFC to VSAN. a. vsan database b. vsan 2 interface vfc 1 c. exit 15. Validate.
  • 171. Configuring 40GbE FCoE connectivity with Nexus 6000 Series 171 Nexus Series Switches Setup Examples a. show interface vfc 1 b. show fcoe database c. show interface fcoe d. show flogi database e. show interface Ethernet 1/7/1 f. show vsan membership g. show vlan h. show interface brief 16. Save the configuration. a. copy running-config startup-config On Nexus 6004 and 22488PQ Complete the following steps. 1. Configure breakout 10GbE port for host CNA connectivity. a. interface ethernet 100/1/1 b. spanning-tree port type edge trunk c. switchport mode trunk d. switchport trunk allowed vlan 2 e. no shutdown f. exit 2. Create vFC for storage interface. a. interface vfc 5 b. bind interface ethernet 100/1/1 c. no shutdown d. exit 3. Associate storage vFC to VSAN. a. vsan database b. vsan 2 interface vfc 5 c. exit 4. Validate. a. show interface vfc 5
  • 172. 172 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples b. show fcoe database c. show interface fcoe d. show flogi database e. show interface Ethernet 100/1/1 f. show vsan membership g. show vlan h. show interface brief 5. Save the configuration. a. copy running-config startup-config On Nexus 6004 and 22488PQ 1. Create zones, zoneset and activate on FCoE VSAN. Refer to Cisco zoning documentation. 2. Verify. a. VSAN zone mode (basic, enhanced) matches across switches – show zone status b. zoneset is active on FCoE VSAN – show zoneset active vsan 2 c. Verify port-channel and vfc are up – show port-channel database – show interface port-channel 3 – show interface vfc 103 d. Verify switches are merged and zoneset is active on appropriate VSAN across all switches in the fabric – show fcdomain – show fcs database – show fcns database On the storage array 1. Check Storage array to ensure that host initiator is logged into array. 2. Configure device masking. On the host 1. Rescan host bus. Ensure host discovers LUNs.
  • 173. Nexus 5000 direct-connect topology 173 Nexus Series Switches Setup Examples Nexus 5000 direct-connect topology This section details the setup steps required to create the configuration shown in Figure 29 on page 176. It includes the following information: ◆ “Summary of configuration steps” on page 173 ◆ “Existing topology” on page 174 ◆ “Target topology” on page 176 ◆ “Prerequisites” on page 177 ◆ “Configuring the virtual interfaces on Nexus 5020” on page 178 ◆ “Etherchannel configuration on a Nexus 7000 switch” on page 203 ◆ “Spanning tree configuration and verification on Nexus 5020” on page 206 Refer to “Installing the Nexus 5000” on page 732 for information on installing the Nexus switch. Summary of configuration steps This section provides a summary and examples of the steps to create the configuration shown in the target topology, Figure 29 on page 176. To create a vFC (virtual) in Nexus FCoE switch, complete the following steps: 1. Create Virtual FC interfaces. 2. Create a VSAN. 3. Create a VLAN to map to a VSAN. 4. Configure the physical Ethernet port as a trunk port and allow the VLAN.
  • 174. 174 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Existing topology It is assumed that the customer has an existing LAN and mirrored SAN, as shown in Figure 28, and is adding the Nexus 5020 to both. Figure 28 Existing topology example Core-N7K-1 CNA_1 Host_1 (Windows) CNA_1: Emulex VSAN = 100 WWPN= 10:00:00:00:c9:3c:f8:3c Storage = Storage port 1 CNA_2: Emulex VSAN = 200 WWPN = 10:00:00:00:c9:3c:f7:c0 Storage = Storage port 3 Storage port 4: 50:06:04:82:D5:2E:69:59 VSAN = 400 Storage port 3: 50:06:04:8A:D5:2E:69:79 VSAN = 200 Storage port 2: 50:06:04:82:Da5:2E:69:49 Storage port 1: 50:06:04:82:D5:2E:69:79 CNA_2 Core-N7K-2 1/1 2/1 1/1 2/1 1/2 1/3 1/4 1/5 Storage Fabric BFabric A 1 2 3 4 1/2 1/2 1/3 1/4 1/5 1/2 1/2 1/1 1/3 1/4 1/5 1/2 1/1 1/3 1/4 1/5 CNA_1 Host_1 (Linux) VSAN 100 VSAN 300 VSAN 200 VSAN 400 4Gb/s FC GEN-001013 CNA_1: QLogic VSAN = 300 WWPN = 21:01:00:1b:32:2a:c0:b8 Storage = Storage port 2 CNA_2: QLogic VSAN = 400 WWPN = 21:00:00:1b:32:0a:c0:b8 Storage = Storage port 4 CNA_2 DCX_1 Domain 1 172.23.185.24 DCX_2 Domain 2 172.23.185.25 MDS-9513-1 VSAN 400 Domain 1 172.23.185.22 MDS-9513-2 VSAN 200 Domain 2 172.23.185.23
  • 175. Nexus 5000 direct-connect topology 175 Nexus Series Switches Setup Examples Existing LAN The existing LAN has the following features/configuration parameters: ◆ Etherchannel (802.1q) ◆ Link Aggregation Control Protocol (LACP) ◆ Rapid Spanning Tree Protocol (RSTP) ◆ Multiple Spanning Tree (MST) ◆ IGMP Snooping ◆ VLANs Existing SAN The existing SAN has the following features/configuration parameters: ◆ DCX_1 • Domain ID 1 ◆ DCX_2 • Domain ID 2 ◆ 9513_1 • Domain ID 1 • VSAN 400 ◆ 9513_2 • Domain ID 2 • VSAN 200
  • 176. 176 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Target topology Figure 29 shows the target topology. Figure 29 Target topology example 2/11/1 1/1 2/1 Core-N7K-1 Core-N7K-2 Port-channel 1 Port-channel 2 1/2 1/3 1/4 1/5 1/2 1/3 1/4 1/5 Nexus-5020-1 Nexus-5020-2 1/21/1 1/21/1 SAN A SAN B Storage 1 2 3 4 1/1 1/1 DCX_1 Domain 1 172.23.185.24 DCX_2 Domain 2 172.23.185.25 SAN Port Channels 1/1 1/2 1/3 1/4 1/5 1/1 1/2 1/3 1/4 1/5 MDS 9513_1 VSAN 100 Domain 1 172.23.185.22 MDS 9513_2 VSAN 200 Domain 2 172.23.185.23 CNA_1 CNA_2 CNA_1 CNA_2 Host_1 (Windows) Host_2 (Linux) VLAN / VSAN 100 VLAN / VSAN 200 VLAN / VSAN 300 VLAN / VSAN 400 10 GbE 4 Gb/s FC IP = 172.23.185.112 Domain = N/A (NPV mode) CNA_1: QLogic VSAN = 300 WWPN = 21:01:00:1b:32:2a:c0:b8 Storage = Storage port 2 CNA_1: Emulex VSAN = 100 WWPN = 10:00:00:00:c9:3c:f8:3c Storage = Storage port 1 CNA_2: Nexus-5020-1 IP = 172.23.185.113 Domain = 3 Nexus-5020-2 Emulex VSAN = 200 WWPN = 10:00:00:00:c9:3c:f7:c0 Storage = Storage port 3 CNA_2: QLogic VSAN = 400 WWPN = 21:00:00:1b:32:0a:c0:b8 Storage = Storage port 4 Storage port 1: 50:06:04:82:D5:2E:69:79 Storage port 2: 50:06:04:82:D5:2E:69:49 Storage port 3: 50:06:04:8A:D5:2E:69:79 VSAN = 200 Storage port 4: 50:06:04:82:D5:2E:69:59 VSAN = 400 GEN-001122
  • 177. Nexus 5000 direct-connect topology 177 Nexus Series Switches Setup Examples Prerequisites FCoE requires the use of specialized host adapters, referred to as a Converged Network Adapter (CNA), and specialized switch hardware known as a Fibre Channel Forwarder (FCF). CNAs are installed in the host and then attached to an Ethernet port that provides FCF functionality. The connection between the CNA and Ethernet switch port can be either optical or copper based. CNA prerequisites The first generation of CNAs requires a server that has a PCIe slot and can contain a full length and height PCIe expansion card. Refer to the EMC Support Matrix for information on the supported host platforms. FCF prerequisites The Nexus 5020 requires: ◆ 2 U of rack space, preferably at the top of the rack where the servers that will be connecting to the Nexus 5020 are located ◆ Two power drops of either 110 V or 220 V ◆ An IP address for management purposes and a network port to which to connect the Nexus 5020 management port ◆ A customer-supplied password for the admin user ◆ An available Domain ID in the fabric to which the Nexus 5020 will be connected Note: If the Nexus 5020 will be carved into multiple VSANs, a Domain ID will need to be reserved for each VSAN. ◆ For each Nexus 5020, at least one expansion module containing 4 G FC ports will need to be installed Laptop prerequisites Laptop prerequisites include: Note: The laptop used in this example is needed only to initially configure the Nexus 5020 and will be disconnected after the installation is complete. ◆ Com port with a DB-9 connector ◆ Download and install PuTTY Note: Installing PuTTY is recommended but not mandatory. ◆ Ensure that HyperTerminal is installed
  • 178. 178 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Note: The operating system used for this example was Windows 2000 Professional and HyperTerminal was installed by default. Some of the later Windows-based operating systems do not have HyperTerminal installed. This will need to be done before attempting to configure the Nexus 5020s. See the help feature for the particular version of Windows you are using for more information on installing HyperTerminal. Configuring the virtual interfaces on Nexus 5020 This section contains the following information: ◆ “Configuring Nexus-5020-1” on page 180 ◆ “Configuring Nexus-5020-2” on page 191 ◆ “Configuring the Windows host” on page 202 ◆ “Configuring the Linux host” on page 203 ◆ “Configuring the VMware host” on page 203 ◆ “Configuring EMC storage” on page 203 ◆ “Configuring network parameters” on page 203 By default, the Ethernet interfaces on the Nexus 5020 do not forward Ethernet frames with an Ethertype of FCoE. To allow the forwarding of FCoE frames, a Virtual Fabric Channel (VFC) interface will need to be created and bound to the port where the FCoE frames will be received. An example of a Virtual FC interface is shown in Figure 30 on page 179.
  • 179. Nexus 5000 direct-connect topology 179 Nexus Series Switches Setup Examples Figure 30 Virtual FC interface The basic steps to configure a switch port to support FCoE are: 1. Create a VLAN for FCoE traffic. 2. Associate a VSAN with the FCoE VLAN. 3. Create the Virtual Fibre Channel interface. 4. Configure the Ethernet port settings. Once these steps have been completed, the CNA should be able to perform Fabric Login (FLOGI) with the FCF functionality within the Nexus 5020. As shown in Figure 31 on page 180, each Nexus 5020 has two CNAs connected to it, one on interface 1/1 and the other on interface 1/2. Due to the port-to-ASIC layout on the Nexus 5020, it is recommended that you connect the first CNA to interface 1/1, then connect subsequent CNAs by starting at interface 1/2, and then proceeding in order up to interface 1/40. This ensures that each ASIC remains equally loaded. PCIs Links 10Gb 10Gb Fibre Channel Ethernet LOGIC Eth1/1 vfc1 Eth1/29 vfc29 Virtual FC interface: Maps FC vHBA on server GEN-001022Converged Network Adapter (CNA)
  • 180. 180 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Figure 31 Target topology example Configuring Nexus-5020-1 Before configuring the Nexus-5020-1, consider the following: The configuration of Nexus-5020-1 will be slightly different from Nexus-5020-2 for the following reasons: ◆ Nexus-5020-1 will be attached to a Brocade fabric consisting of two Brocade ED-DCX-B switches. Since the Nexus 5020 does not currently support FC-SW interop to an EMC ED-DCX-B, Nexus-5020-1 will need to run in NPV mode. 2/11/1 1/1 2/1 Core-N7K-1 Core-N7K-2 Port-channel 1 Port-channel 2 1/2 1/3 1/4 1/5 1/2 1/3 1/4 1/5 Nexus-5020-1 Nexus-5020-2 1/21/1 1/21/1 SAN A SAN B Storage 1 2 3 4 1/1 1/1 DCX_1 Domain 1 172.23.185.24 DCX_2 Domain 2 172.23.185.25 SAN Port Channels 1/1 1/2 1/3 1/4 1/5 1/1 1/2 1/3 1/4 1/5 MDS 9513_1 VSAN 100 Domain 1 172.23.185.22 MDS 9513_2 VSAN 200 Domain 2 172.23.185.23 CNA_1 CNA_2 CNA_1 CNA_2 Host_1 (Windows) Host_2 (Linux) VLAN / VSAN 100 VLAN / VSAN 200 VLAN / VSAN 300 VLAN / VSAN 400 10 GbE 4 Gb/s FC IP = 172.23.185.112 Domain = N/A (NPV mode) CNA_1: QLogic VSAN = 300 WWPN = 21:01:00:1b:32:2a:c0:b8 Storage = Storage port 2 CNA_1: Emulex VSAN = 100 WWPN = 10:00:00:00:c9:3c:f8:3c Storage = Storage port 1 CNA_2: Nexus-5020-1 IP = 172.23.185.113 Domain = 3 Nexus-5020-2 Emulex VSAN = 200 WWPN = 10:00:00:00:c9:3c:f7:c0 Storage = Storage port 3 CNA_2: QLogic VSAN = 400 WWPN = 21:00:00:1b:32:0a:c0:b8 Storage = Storage port 4 Storage port 1: 50:06:04:82:D5:2E:69:79 Storage port 2: 50:06:04:82:D5:2E:69:49 Storage port 3: 50:06:04:8A:D5:2E:69:79 VSAN = 200 Storage port 4: 50:06:04:82:D5:2E:69:59 VSAN = 400 GEN-001122
  • 181. Nexus 5000 direct-connect topology 181 Nexus Series Switches Setup Examples ◆ NPV mode allows the Nexus 5020 to act as an N_Port and pass login requests directly from the CNA to the Fabric by using the NPIV protocol. For more information on NPV and NPIV, refer to the “NPIV” section in the Networked Storage Concepts and Protocols TechBook available on the E-Lab Interoperability Navigator, Topology Resource Center tab, at http://elabnavigator.EMC.com. Further information on how to configure NPV mode and the supported features with NPV is available at http://www.cisco.com. Since Nexus-5020-1 will be connected to two different ED-DCX-Bs, VSAN assignments to the Nexus 5020 VFC and FC interfaces will be used to allow the CNA to attach to the proper fabric. To configure Nexus-5020-1: 1. Connect to the console port of Nexus-5020-1. a. Attach the DB-9 connector of the serial cable to COM port 1 on the laptop. b. Attach the RJ-45 connector of the serial cable to the console port on the Nexus-5020-1. c. Select Start > Programs > Accessories > Communications > Hyperterminal. The new connection dialog box is opened with the entry name dialog highlighted. d. Type a name such as Nexus, and then click OK. The Connect to dialog box displays. e. Select COM 1 in the Connect using pull-down list, and then click OK. The COM1 Properties dialog box displays. f. In the COM 1 Properties dialog box, use the following settings: – Bits per second: = 9600 – Data bits: = 8 – Parity: = none – Stop bits: = 1 – Flow control: = None g. Click OK. You will be connected and should see the following displayed:
  • 182. 182 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Enter the password for "admin": Note: You may need to press Enter several times before any text is displayed. h. Type the customer-supplied password for admin, which in this case is nbv12345. Note: It may be helpful to log the output by using the capture text feature under the Transfer: pull-down list. i. Re-type the password for the admin user. The Basic System Configuration dialog box displays. 2. Complete the basic configuration: a. Answer the prompts as follows: Would you like to enter the basic configuration dialog (yes/no):Y Note: If the installer does not use the install startup script, management vrf requires manual installation. Create another login account (yes/no) [n]: N Configure read-only SNMP community string (yes/no) [n]: N Configure read-write SNMP community string (yes/no) [n]: N Enter the switch name: Nexus-5020-1 Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: Y Mgmt0 IPv4 address: 172.23.185.112 Mgmt0 IPv4 netmask: 255.255.255.0 Configure the default gateway? (yes/no) [y]: Y IPv4 address of the default gateway: 172.23.185.2 Enable the telnet service? (yes/no) [y]: Y Enable the ssh service? (yes/no) [n]: Y Type of ssh key you would like to generate (dsa/rsa): dsa Number of key bits <768-2048>: 1024 Configure the ntp server? (yes/no) [n]: N Enter basic FC configurations (yes/no) [n]: Y Configure default physical FC switchport interface state (shut/noshut) [shut]: shut Configure default physical FC switchport trunk mode (on/off/auto) [on]: on Configure default zone policy (permit/deny) [deny]: deny Enable full zoneset distribution? (yes/no) [n]: y A summary of the configuration displays: The following configuration will be applied: switchname Nexus-5020-1
  • 183. Nexus 5000 direct-connect topology 183 Nexus Series Switches Setup Examples interface mgmt0 ip address 172.23.185.112 255.255.255.0 no shutdown exit vrf context management ip route 0.0.0.0/0 172.23.185.2 exit telnet server enable ssh key dsa 1024 force ssh server enable b. Answer the next questions as follows: Would you like to edit the configuration? (yes/no) [n]:N Use this configuration and save it? (yes/no) [y]: Y The configuration will be saved. c. Once the configuration is completed, you will be prompted to log in as follows: Nexus-5020-1 login: admin Password: nbv12345 d. Connect the network management cable. e. Use the show int brief command to show the interfaces on the switch. You will notice between 40 and 52 Ethernet interfaces. Note: If no FC interfaces are displayed and one or both of the GEMs contain FC ports, ensure that the FCoE feature has been enabled using the feature FCoE command. Nexus-5020-1# show int brief ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- fc2/1 1 auto on down swl -- -- fc2/2 1 auto on down swl -- -- fc2/3 1 auto on down swl -- -- . . . fc3/4 1 auto on down swl -- -- --------------------------------------------------------------------------------
  • 184. 184 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth1/1 1 eth access down Link not connected 10G(D) -- Eth1/2 1 eth access down Link not connected 10G(D) -- Eth1/3 1 eth access down Link not connected 10G(D) -- . . . Eth1/40 1 eth access down Link not connected 10G(D) -- -------------------------------------------------------------------------------- Port VRF Status IP Address Speed MTU -------------------------------------------------------------------------------- mgmt0 -- up 172.23.185.113 1000 1500 Nexus-5020-1# f. If there are no FC interfaces displayed, enable the FCoE feature as follows: Nexus-5020-1# config terminal Nexus-5020-1(config)# feature fcoe Note: It may be necessary to press Enter if the prompt is not displayed after entering the feature FCoE command. g. Save the configuration. Nexus-5020-1(config)# copy running startup-config h. Enable NPV mode. Although NPV mode is supported when connecting to either Connectrix MDS or non-Connectrix MDS switches, it only needs to be enabled when connecting a Nexus 5020 to a non-Connectrix MDS switch, as is the case with this example. If you are connecting the Nexus 5020 to a Connectrix MDS switch, and do not want to use NPV mode, see “Configuring Nexus-5020-2” on page 191. Nexus-5020-1(config)# npv enable Verify that boot variables are set and the changes are saved. Changing to npv mode erases the current configuration and reboots the switch in npv mode. Do you want to continue? (y/n): Y Note: It is recommended that you leave the console connection up while the reboot is taking place. i. Open an SSH session to the Nexus 5020 with an IP address of 172.23.185.12.
  • 185. Nexus 5000 direct-connect topology 185 Nexus Series Switches Setup Examples 3. Enable disruptive load-balancing. Disruptive load-balancing allows the Nexus to evenly distribute CNA logins across all NP ports while running in NPV mode. To enable disruptive load-balancing, use the npv auto-load-balance disruptive command, as shown next. Nexus-5020-1# config terminal Nexus-5020-1(config)# npv auto-load-balance disruptive 4. If you are configuring a Nexus 5548 or Nexus 5596, you must manually enable the QoS class-fcoe. To do this, attach the following QoS policies under system qos: switch (config-sys-qos)# service-policy type qos input fcoe-default-in-policy switch (config-sys-qos)# service-policy type queuing input fcoe-default-in-policy switch (config-sys-qos)# service-policy type queuing output fcoe-default-out-policy switch (config-sys-qos)# service-policy type network-qos fcoe-default-nq-policy 5. Create the VSANs. Nexus-5020-1# config Nexus-5020-1(config)# vsan database Nexus-5020-1(config-vsan-db)# vsan 100 Nexus-5020-1(config-vsan-db)# vsan 300 Nexus-5020-1(config-vsan-db)# exit Nexus-5020-1(config)# 6. Create a VLAN for FCoE Traffic and associate with a VSAN. Note: Due to the one-to-one relationship between a VSAN and VLAN, if multiple VSANs will be configured (as is the case in this example) on the Nexus 5020, then multiple FCoE VLANs will also need to be configured. Nexus-5020-1(config)# vlan 100 Nexus-5020-1(config-vlan)# fcoe vsan 100 Nexus-5020-1(config-vlan)# vlan 300 Nexus-5020-1(config-vlan)# fcoe vsan 300 7. Use show FCoE VLAN to verify whether the configuration was successful. Nexus-5020-1(config-vlan)# show vlan fcoe VLAN VSAN Status -------- -------- -------- 100 100 Non-operational 300 300 Non-operational
  • 186. 186 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 8. Create the VFC interfaces. Nexus-5020-1(config-vlan)# interface vfc 1 Nexus-5020-1(config-if)# bind interface ethernet 1/1 Nexus-5020-1(config-if)# switchport trunk allowed vsan 100 Nexus-5020-1(config-if)# no shut Nexus-5020-1(config-if)# interface vfc 2 Nexus-5020-1(config-if)# bind interface ethernet 1/2 Nexus-5020-1(config-if)# switchport trunk allowed vsan 300 Nexus-5020-1(config-if)# no shut 9. Configure the Ethernet Interface settings. Nexus-5020-1(config-if)# interface ethernet 1/1 Nexus-5020-1(config-if)# switchport mode trunk Nexus-5020-1(config-if)# switchport trunk native vlan 1 Nexus-5020-1(config-if)# switchport trunk allowed vlan 1, 100 Nexus-5020-1(config-if)# spanning-tree bpduguard enable Nexus-5020-1(config-if)# spanning-tree port type edge trunk Nexus-5020-1(config-if)# interface ethernet 1/2 Nexus-5020-1(config-if)# switchport mode trunk Nexus-5020-1(config-if)# switchport trunk native vlan 1 Nexus-5020-1(config-if)# switchport trunk allowed vlan 1, 300 Nexus-5020-1(config-if)# spanning-tree bpduguard enable Nexus-5020-1(config-if)# spanning-tree port type edge trunk 10. Add the vfc interfaces to the proper VSAN. Nexus-5020-1(config-if)# vsan database Nexus-5020-1(config-vsan-db)# vsan 100 interface vfc1 Nexus-5020-1(config-vsan-db)# vsan 300 interface vfc2 Nexus-5020-1(config-vsan-db)# exit Nexus-5020-1(config-if)# The output of show int brief displays the following: Nexus-5020-1 (config-if)# show int brief ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) -----------------------------------------------------------------------------fc2 /1 1 NP -- down swl -- -- fc2/2 1 NP -- down swl -- -- fc2/3 1 NP -- down swl -- -- fc2/4 1 NP -- down swl -- -- fc3/1 1 NP -- down swl -- -- fc3/2 1 NP -- down swl -- -- fc3/3 1 NP -- down swl -- -- fc3/4 1 NP -- down swl -- -- -----------------------------------------------------------------------------
  • 187. Nexus 5000 direct-connect topology 187 Nexus Series Switches Setup Examples Interface Status IP Address Speed MTU Port Channel ----------------------------------------------------------------------------- Ethernet1/1 notConnect -- -- 1500 -- Ethernet1/2 notConnect -- -- 1500 -- Ethernet1/3 notConnect -- -- 1500 -- . . Ethernet1/40 notConnect -- -- 1500 -- . . ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ----------------------------------------------------------------------------- vfc1 100 F -- down -- -- -- vfc2 300 F -- down -- -- -- Nexus-5020-1# The output from this show int brief command differs from the output shown in Step e. Note that the vfc interfaces that were created in the previous steps are now displayed. Note: In this example, the Cisco Nexus-5020-1 is running in NPV mode and will be connected to two different ED-DCX-B Directors. Because of this, we will be using VSANs to statically map one CNA to DCX_1 and the other CNA to DCX_2. 11. Add FC interfaces to the proper VSANs. As shown in Figure 29 on page 176, Nexus-5020-1 is connected to two different fabrics. In order to ensure that each CNA attached to Nexus-5020-1 logs in to the same fabric as its storage port, each CNA will need to be statically mapped to a particular VSAN. This is accomplished by adding the VFC Interface as well as the physical FC interfaces to the appropriate VSAN. To summarize the steps: a. Enter the vsan database configuration mode by using vsan database. b. Add the fc interfaces on 5020_1 to the appropriate VSAN using vsan <VSAN ID> interface vfc <vfc number>. Nexus-5020-1(config-if)# vsan database Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 2/1 Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 2/3
  • 188. 188 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 3/1 Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 3/3 Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 2/2 Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 2/4 Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 3/2 Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 3/4 Nexus-5020-1(config-vsan-db)# exit Nexus-5020-1(config)# c. Enable all of the fc interfaces by specifying the interface by using the interface <interface number> command, and then the no shut command. Nexus-5020-1(config)# interface fc 2/1-4 Nexus-5020-1(config-if)# no shut Nexus-5020-1(config)# interface fc 3/1-4 Nexus-5020-1(config-if)# no shut d. Validate the configuration by using the show int brief command. The following is a partial output of this command. Nexus-5020-1(config)# show int brief ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ----------------------------------------------------------------------------- fc2/1 100 NP -- down swl -- -- fc2/2 300 NP -- down swl -- -- fc2/3 100 NP -- down swl -- -- fc2/4 300 NP -- down swl -- -- fc3/1 100 NP -- down swl -- -- fc3/2 300 NP -- down swl -- -- fc3/3 100 NP -- down swl -- -- fc3/4 300 NP -- down swl -- -- . . . ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ----------------------------------------------------------------------------- vfc1 100 F -- down -- -- -- vfc2 300 F -- down -- -- -- Nexus-5020-1(config-if)# 12. Connect the cables.
  • 189. Nexus 5000 direct-connect topology 189 Nexus Series Switches Setup Examples a. Once the interface configuration has been verified, connect the cables between the CNAs and Nexus-5020-1 as well as from the FC switches to Nexus-5020-1. b. Use show int brief to validate the interfaces are initialized. Note: With Step 12 a and b, assume that the CNA installation steps have been completed (such as drivers, firmware, and CEE firmware). If they have not been completely installed, the CNA will not be shown in the show int brief output. If this is the case, once the CNA has been completely installed, re-run the show int brief command and verify that it is shown as being logged in. Nexus-5020-1(config-if)# show int brief ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ----------------------------------------------------------------------------- fc2/1 100 NP -- up swl NP 2 -- fc2/2 100 NP -- up swl NP 2 -- fc2/3 300 NP -- up swl NP 2 -- fc2/4 300 NP -- up swl NP 2 -- fc3/1 100 NP -- up swl NP 2 -- fc3/2 100 NP -- up swl NP 2 -- fc3/3 300 NP -- up swl NP 2 -- fc3/4 300 NP -- up swl NP 2 -- ----------------------------------------------------------------------------- Interface Status IP Address Speed MTU Port Channel ----------------------------------------------------------------------------- Ethernet1/1 up -- 10000 1500 -- Ethernet1/2 up -- 10000 1500 -- Ethernet1/3 notConnect -- -- 1500 -- . . . Ethernet3/4 sfpIsAbsen -- -- 1500 -- . . . ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ----------------------------------------------------------------------------- vfc1 100 F -- up -- F auto -- vfc2 300 F -- up -- F auto -- Nexus-5020-1#
  • 190. 190 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples c. Use show npv flogi-table vsan <vsan ID> to verify that each CNA is logged in to the appropriate vsan. Nexus-5020-1(config-if)# show npv flogi-table vsan 100 ----------------------------------------------------------------------------- SERVER EXTERNAL INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE ----------------------------------------------------------------------------- vfc1 100 0x680101 10:00:00:00:c9:3c:f8:3c 20:00:00:00:c9:3c:f8:3c fc2/1 Total number of flogi = 1. Nexus-5020-1(config-if)# show npv flogi table vsan 300 ----------------------------------------------------------------------------- SERVER EXTERNAL INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE ----------------------------------------------------------------------------- vfc2 300 0x690101 21:01:00:1b:32:2a:c0:b8 20:01:00:1b:32:2a:c0:b8 fc2/3 Total number of flogi = 1. Nexus-5020-1(config-if)# The fc interfaces are now online and running in NPV mode. The vfc interfaces come online as soon as one of the fc interfaces in the same vsan comes online. 13. Save the configuration: Nexus-5020-1(config-if) # copy running startup-config 14. Zone the CNAs. Since Nexus-5020-1 is running in NPV mode, all zoning will need to be performed from the ED-DCX-B Director to which it is attached. It is strongly recommended that if the CLI is to be used for zoning, then cut and paste should be utilized to minimize transcription types of errors when manually typing in the WWPNs. The set of steps required to create the appropriate zones are: a. Create each zone by using zonecreate "<zone name a>", "WWPN1; WWPN2; .... WWPNn". b. Create the zone config by using cfgcreate "<cfg name>", "<zone name a>; <zone name b>; .... <zone name n>".
  • 191. Nexus 5000 direct-connect topology 191 Nexus Series Switches Setup Examples c. Enable the zone config by using cfgenable "<cfg name>". d. Save the configuration by using cfgsave. The exact steps that would be performed in this example are as follows: a. Zone the CNA logged in to 172.23.185.25 to access its storage port. – Telnet into the switch with an IP address of 172.23.185.24. DCX_1:admin> zonecreate “host1_CNA1”, “10:00:00:00:c9:3c:f8:3c; 50:06:04:82:D5:2E:69:79” DCX_1:admin> cfgcreate “FCoE_1”, “host1_CNA1” DCX_1:admin> cfgenable “FCoE_1” DCX_1:admin> cfgshow Default Zone: OFF Safe Zone: OFF Defined configuration: Effective configuration: cfg: FCoE_1 zone: host1_CNA1 10:00:00:00:c9:3c:f8:3c 50:06:04:82:D5:2E:69:79 DCX_1:admin> – Log out of the switch with an IP address of 172.23.185.24. b. Zone the CNA logged in to 172.23.185.25 to access its Storage ports. – Telnet into the switch with an IP address of 172.23.185.25. DCX_2:admin> zonecreate “host2_CNA1”, “21:01:00:1b:32:2a:c0:b8; 50:06:04:82:D5:2E:69:49” DCX_2:admin> cfgcreate “FCoE_2”, “host2_CNA1” DCX_2:admin> cfgenable “FCoE_2” DCX_2:admin> cfgshow Default Zone: OFF Safe Zone: OFF Defined configuration: Effective configuration: cfg: FCoE_2 zone: host2_CNA1 21:01:00:1b:32:2a:c0:b8 50:06:04:82:D5:2E:69:49 DCX_2:admin> – Log out of the switch with an IP address of 172.23.185.25. Configuring Nexus-5020-2 Nexus-5020-2 has two CNAs connected to it, one on interface 1/1 and the other on interface 1/2. This example will configure the virtual interfaces on interface 1/1 first and then interface 1/2.
  • 192. 192 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples To configure the Nexus-5020-2: 1. Connect to the console port of Nexus-5020-2. a. Attach the DB-9 connector of the serial cable to COM port 1 on the laptop. b. Attach the RJ-45 connector of the serial cable to the console port on the Nexus-5020-2. c. Select Start > Programs > Accessories > Communications > Hyperterminal. The new connection dialog box is opened with the entry name dialog highlighted. d. Enter a name such as Nexus, and then click OK. The Connect to dialog box displays. e. Select COM 1 in the Connect using pull-down list, and then click OK. The COM1 Properties dialog box displays. f. In the COM 1 Properties dialog box, use the following settings: – Bits per second: = 9600 – Data bits: = 8 – Parity: = none – Stop bits: = 1 – Flow control: = None g. Click OK. You will be connected and should see the following displayed: Enter the password for "admin": Note: You may need to press Enter several times before any text is displayed. h. Enter the customer-supplied password for admin, which in this case is nbv12345. Note: It may be helpful to log the output by using the capture text feature under the Transfer: pull-down menu. i. Re-enter the password for the admin user.
  • 193. Nexus 5000 direct-connect topology 193 Nexus Series Switches Setup Examples The Basic System Configuration dialog box displays. 2. Complete the basic configuration as follows: a. Answer the prompts as follows: Would you like to enter the basic configuration dialog (yes/no):Y Note: If the installer does not use install startup script, management VRF requires manual installation. Create another login account (yes/no) [n]: N Configure read-only SNMP community string (yes/no) [n]: N Configure read-write SNMP community string (yes/no) [n]: N Enter the switch name: Nexus-5020-2 Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: Y Mgmt0 IPv4 address: 10.1.1.113 Mgmt0 IPv4 netmask: 255.255.255.0 Configure the default gateway? (yes/no) [y]: Y IPv4 address of the default gateway: 172.23.185.2 Enable the telnet service? (yes/no) [y]: Y Enable the ssh service? (yes/no) [n]: Y Type of ssh key you would like to generate (dsa/rsa): dsa Number of key bits <768-2048>: 1024 Configure the ntp server? (yes/no) [n]: N Enter basic FC configurations (yes/no) [n]: Y Configure default physical FC switchport interface state (shut/noshut) [shut]: shut Configure default physical FC switchport trunk mode (on/off/auto) [on]: on Configure default zone policy (permit/deny) [deny]: deny Enable full zoneset distribution? (yes/no) [n]: y A summary of the configuration displays: The following configuration will be applied: switchname Nexus-5020-2 interface mgmt0 ip address 10.1.1.113 255.255.255.0 no shutdown exit vrf context management ip route 0.0.0.0/0 172.23.185.2 exit telnet server enable ssh key dsa 1024 force ssh server enable b. Answer the questions as follows: Would you like to edit the configuration? (yes/no) [n]:N Use this configuration and save it? (yes/no) [y]: Y
  • 194. 194 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples The configuration will be saved. c. Once this is completed, you will be prompted to log in as follows: Nexus-5020-2 login: admin Password: nbv12345 d. Connect the network management cable. e. Use the show int brief command to show the interfaces on the switch. You will notice between 40 and 52 Ethernet interfaces. Note: If no FC interfaces are displayed and one or both of the GEMs contain FC ports, ensure that the FCoE feature has been enabled using the feature FCoE command. Nexus-5020-2# show int brief ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- fc2/1 1 auto on down swl -- -- fc2/2 1 auto on down swl -- -- fc2/3 1 auto on down swl -- -- . . . fc3/4 1 auto on down swl -- -- -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth1/1 1 eth access down Link not connected 10G(D) -- Eth1/2 1 eth access down Link not connected 10G(D) -- Eth1/3 1 eth access down Link not connected 10G(D) -- . . . Eth1/40 1 eth access down Link not connected 10G(D) -- -------------------------------------------------------------------------------- Port VRF Status IP Address Speed MTU -------------------------------------------------------------------------------- mgmt0 -- up 172.23.185.113 1000 1500 Nexus-5020-2#
  • 195. Nexus 5000 direct-connect topology 195 Nexus Series Switches Setup Examples f. If there are no FC interfaces displayed, enable the FCoE feature as follows: Nexus-5020-2# config terminal Nexus-5020-2(config)# feature fcoe Note: It may be necessary to press Enter if the prompt is not displayed after entering the feature FCoE command. g. Save the configuration. Nexus-5020-2(config)# copy running startup-config h. Open an SSH session to the Nexus 5020 with an IP address of 10.1.1.113. 3. Create the VSANs. Nexus-5020-2# config Nexus-5020-2(config)# vsan database Nexus-5020-2(config-vsan-db)# vsan 200 Nexus-5020-2(config-vsan-db)# vsan 400 Nexus-5020-2(config-vsan-db)# exit Nexus-5020-2(config)# 4. If you are configuring a Nexus 5548 or Nexus 5596, you must manually enable the QoS class-fcoe. To do this, attach the following QoS policies under system qos: switch (config-sys-qos)# service-policy type qos input fcoe-default-in-policy switch (config-sys-qos)# service-policy type queuing input fcoe-default-in-policy switch (config-sys-qos)# service-policy type queuing output fcoe-default-out-policy switch (config-sys-qos)# service-policy type network-qos fcoe-default-nq-policy 5. Create a VLAN for FCoE Traffic and associate with a VSAN. Note: Due to the one-to-one relationship between a VSAN and VLAN, if multiple VSANs will be configured (as is the case in this example) on the Nexus 5020, then multiple FCoE VLANs will also need to be configured. Nexus-5020-2(config)# vlan 200 Nexus-5020-2(config-vlan)# fcoe vsan 200 Nexus-5020-2(config-vlan)# vlan 400 Nexus-5020-2(config-vlan)# fcoe vsan 400
  • 196. 196 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 6. Use show FCoE VLAN to verify the configuration was successful. Nexus-5020-2(config-vlan)# show vlan fcoe VLAN VSAN Status -------- -------- -------- 200 200 Non-operational 400 400 Non-operational 7. Create the VFC interfaces. Nexus-5020-2(config-vlan)# interface vfc 1 Nexus-5020-2(config-if)# bind interface ethernet 1/1 Nexus-5020-2(config-if)# switchport trunk allowed vsan 200 Nexus-5020-2(config-if)# no shut Nexus-5020-2(config-if)# interface vfc 2 Nexus-5020-2(config-if)# bind interface ethernet 1/2 Nexus-5020-2(config-if)# switchport trunk allowed vsan 400 Nexus-5020-2(config-if)# no shut 8. Configure the Ethernet Interface settings. Nexus-5020-2(config-if)# interface ethernet 1/1 Nexus-5020-2(config-if)# switchport mode trunk Nexus-5020-2(config-if)# switchport trunk native vlan 1 Nexus-5020-1(config-if)# spanning-tree bpduguard enable Nexus-5020-1(config-if)# spanning-tree port type edge trunk Nexus-5020-2(config-if)# switchport trunk allowed vlan 1, 200 Nexus-5020-1(config-if)# spanning-tree bpduguard enable Nexus-5020-1(config-if)# spanning-tree port type edge trunk Nexus-5020-2(config-if)# interface ethernet 1/2 Nexus-5020-2(config-if)# switchport mode trunk Nexus-5020-2(config-if)# switchport trunk native vlan 1 Nexus-5020-1(config-if)# switchport trunk allowed vlan 1, 400 9. Add the vfc interfaces to the proper VSAN. Nexus-5020-2(config-if)# vsan database Nexus-5020-2(config-vsan-db)# vsan 100 interface vfc1 Nexus-5020-2(config-vsan-db)# vsan 300 interface vfc2 Nexus-5020-2(config-vsan-db)# exit Nexus-5020-2(config-if)# The output of show int brief displays the following: Nexus-5020-2(config-if)# show int brief ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ------------------------------------------------------------------------------- fc2/1 1 auto on trunking swl TE 4 --
  • 197. Nexus 5000 direct-connect topology 197 Nexus Series Switches Setup Examples fc2/2 1 auto on trunking swl TE 4 -- fc2/3 1 auto on trunking swl TE 4 -- fc2/4 1 auto on trunking swl TE 4 -- fc3/1 1 auto on trunking swl TE 4 -- fc3/2 1 auto on trunking swl TE 4 -- fc3/3 1 auto on trunking swl TE 4 -- fc3/4 1 auto on trunking swl TE 4 -- ----------------------------------------------------------------------------- Interface Status IP Address Speed MTU Port Channel ----------------------------------------------------------------------------- Ethernet1/1 up -- 10000 1500 -- Ethernet1/2 up -- 10000 1500 -- Ethernet1/3 sfpIsAbsen -- -- 1500 -- . . . Ethernet3/4 sfpIsAbsen -- -- 1500 -- . . . ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ----------------------------------------------------------------------------- vfc1 200 F -- up -- F auto -- vfc2 400 F -- up -- F auto -- Nexus-5020-2(config-if)# The output from this show int brief command differs from the display shown in Step e. Note that the vfc interfaces that were created in the previous steps are now displayed. Note: In this example, Nexus-5020-2 is connected to two different MDS switches and the interfaces on each MDS are in different VSANs. One VSAN will be used for the server using an Emulex CNA and the other VSAN will be used for the server using a QLogic CNA. Enable all of the fc interfaces. Nexus-5020-2(config-if)# interface fc2/1-4 Nexus-5020-2(config-if)# no shut Nexus-5020-2(config-if)# interface fc3/1-4 Nexus-5020-2(config-if)# no shut i. Connect the cables from the CNAs to Nexus-5020-2 as shown in Figure 29 on page 176.
  • 198. 198 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples j. Validate the configuration using the show int brief command. The following is a partial output of the show int brief command: Nexus-5020-2(config)# show int brief ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ------------------------------------------------------------------------------- fc2/1 1 auto on notConnected swl -- -- fc2/2 1 auto on notConnected swl -- -- fc2/3 1 auto on notConnected swl -- -- fc2/4 1 auto on notConnected swl -- -- fc3/1 1 auto on notConnected swl -- -- fc3/2 1 auto on notConnected swl -- -- fc3/3 1 auto on notConnected swl -- -- fc3/4 1 auto on notConnected swl -- -- ----------------------------------------------------------------------------- Interface Status IP Address Speed MTU Port Channel ----------------------------------------------------------------------------- Ethernet1/1 up -- 10000 1500 -- Ethernet1/2 up -- 10000 1500 -- Ethernet1/3 sfpIsAbsen -- -- 1500 -- . . . Ethernet3/4 sfpIsAbsen -- -- 1500 -- . . . ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ----------------------------------------------------------------------------- vfc1 200 F -- up -- F auto -- vfc2 400 F -- up -- F auto -- Nexus-5020-2(config-if)# 10. Configure the FC interfaces on the Cisco MDS 9513s.
  • 199. Nexus 5000 direct-connect topology 199 Nexus Series Switches Setup Examples Note: The storage ports attached to each MDS will need to be added to the proper VSAN in order for the CNAs to discover them. In addition, the Allowed VSANs list for TE-ports (used for Cisco trunking of VSANs) connecting the Nexus 5020 and the MDS will need to be checked to ensure that they can carry VSANs 200 and 400. The basic steps for configuring the FC interfaces on the MDS are: a. Telnet in to the MDS switch. b. Create the appropriate vsan on the MDS switch using the command vsan <VSAN ID>. c. Add the appropriate interfaces on the MDS switches to the appropriate VSAN using the vsan <VSAN ID> interface fc <slot/port> command. The actual commands used for this example are as follows: • For MDS-9513-1: MDS-9513-1# config terminal Enter configuration commands, one per line. End with CNTL/Z. MDS-9513-1(config)# vsan database MDS-9513-1(config-vsan-db)# vsan 400 MDS-9513-1(config-vsan-db)# vsan 400 interface fc 2/14 Traffic on fc2/14 may be impacted. Do you want to continue? (y/n) y MDS-9513-1(config-vsan-db)# exit MDS-9513-1(config) # copy running startup-config MDS-9513-1(config)# • For MDS-9513-2: MDS-9513-2# config terminal Enter configuration commands, one per line. End with CNTL/Z. MDS-9513-2(config)# vsan database MDS-9513-2(config-vsan-db)# vsan 200 MDS-9513-2(config-vsan-db)# vsan 200 interface fc 2/13 Traffic on fc2/13 may be impacted. Do you want to continue? (y/n) y MDS-9513-2(config-vsan-db)# exit MDS-9513-2(config) # copy running startup-config MDS-9513-2(config)# 11. Connect cables from CNAs to Nexus-5020-2. a. Once the interface configuration has been verified, connect the cables between the CNAs and Nexus-5020-2 as well as from the FC switches to Nexus-5020-2. b. Use show int brief to validate that the interfaces initialized as expected.
  • 200. 200 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Nexus-5020-2(config-if)# show int brief ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ----------------------------------------------------------------------------- fc2/1 1 auto on trunking swl TE 4 -- fc2/2 1 auto on trunking swl TE 4 -- fc2/3 1 auto on trunking swl TE 4 -- fc2/4 1 auto on trunking swl TE 4 -- fc3/1 1 auto on trunking swl TE 4 -- fc3/2 1 auto on trunking swl TE 4 -- fc3/3 1 auto on trunking swl TE 4 -- fc3/4 1 auto on trunking swl TE 4 -- ----------------------------------------------------------------------------- Interface Status IP Address Speed MTU Port Channel ----------------------------------------------------------------------------- Ethernet1/1 up -- 10000 1500 -- Ethernet1/2 up -- 10000 1500 -- Ethernet1/3 sfpIsAbsen -- -- 1500 -- . . . Ethernet3/4 sfpIsAbsen -- -- 1500 -- . . . ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ----------------------------------------------------------------------------- vfc1 200 F -- up -- F auto -- vfc2 400 F -- up -- F auto -- Nexus-5020-2(config-if)# 12. Verify that the appropriate VSANs are in the TE ports allowed list and that they are shown as up. To do this, use the show interface trunk vsan command, as shown next. Nexus-5020-2# show interface trunk vsan fc2/1 is trunking Vsan 1 is up (None), FCID is 0x9a0000 Vsan 400 is up (None), FCID is 0x4e0001 fc2/2 is trunking Vsan 1 is up (None), FCID is 0x9a0001 Vsan 200 is up (None), FCID is 0xd00001 fc2/3 is trunking Vsan 1 is up (None), FCID is 0x9a0002 Vsan 400 is up (None), FCID is 0x4e0002
  • 201. Nexus 5000 direct-connect topology 201 Nexus Series Switches Setup Examples fc2/4 is trunking Vsan 1 is up (None), FCID is 0x9a0003 Vsan 200 is up (None), FCID is 0xd00002 fc3/1 is trunking Vsan 1 is up (None), FCID is 0x9a0004 Vsan 400 is up (None), FCID is 0x4e0003 fc3/2 is trunking Vsan 1 is up (None), FCID is 0x9a0005 Vsan 200 is up (None), FCID is 0xd00003 fc3/3 is trunking Vsan 1 is up (None), FCID is 0x9a0006 Vsan 400 is up (None), FCID is 0x4e0004 fc3/4 is trunking Vsan 1 is up (None), FCID is 0x9a0007 Vsan 200 is up (None), FCID is 0xd00004 c. Use show flogi database vsan <vsan ID> to verify that each CNA is logged in to the appropriate vsan. Nexus-5020-2(config-if)# show flogi database vsan 200 --------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------- vfc1 200 0xd00000 10:00:00:00:c9:3c:f7:c0 20:00:00:00:c9:3c:f7:c0 Total number of flogi = 1. Nexus-5020-2(config-if)# show flogi database vsan 400 --------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------- vfc2 400 0x4e0000 21:00:00:1b:32:0a:c0:b8 20:00:00:1b:32:0a:c0:b8 Total number of flogi = 1. Nexus-5020-2(config-if)# 13. Zone the CNAs. It is strongly recommended that if the CLI is to be used for zoning, that cut and paste be utilized to minimize transcription types of errors when manually typing in the WWPNs. The set of steps required to create the appropriate zones are: a. zone name <ZONE_NAME> vsan <VSAN ID>. b. member pwwn <PWWN_1>. c. member pwwn <PWWN_2>. d. zone set name <ZONE_SET_NAME> vsan <VSAN ID>.
  • 202. 202 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples e. member <ZONE_NAME>. f. zoneset activate name <ZONE_SET_NAME> vsan <VSAN ID>. g. Copy running startup. 14. Telnet into Nexus-5020-2 by completing the following steps: Create zones for the VSAN 200: Nexus-5020-2# config terminal Nexus-5020-2(config)# zone name host1_CNA2 vsan 200 Nexus-5020-2(config-zone)# member pwwn 10:00:00:00:c9:3c:f7:c0 Nexus-5020-2(config-zone)# member pwwn 50:06:04:8A:D5:2E:69:79 Nexus-5020-2(config-zone)# exit Nexus-5020-2(config)# zoneset name FCoE_200 vsan 200 Nexus-5020-2(config-zoneset)# member host1_CNA2 Nexus-5020-2(config-zoneset)# zoneset activate name FCoE_200 vsan 200 Zoneset activation initiated. check zone status Nexus-5020-2(config)# zoneset distribute full vsan 200 Nexus-5020-2(config)# show zoneset active vsan 200 zoneset name FCoE_200 vsan 200 zone name host1_CNA2 vsan 200 * fcid 0xd00000 [pwwn 10:00:00:00:c9:3c:f7:c0] * fcid 0x6f0000 [pwwn 50:06:04:8a:d5:2e:69:79] Nexus-5020-2(config)# Create zones for the VSAN 400: Nexus-5020-2(config)# zone name host2_CNA2 vsan 400 Nexus-5020-2(config-zone)# member pwwn 21:00:00:1B:32:0A:C0:B8 Nexus-5020-2(config-zone)# member pwwn 50:06:04:82:D5:2E:69:59 Nexus-5020-2(config-zone)# exit Nexus-5020-2(config)# zoneset name FCoE_400 vsan 400 Nexus-5020-2(config-zoneset)# member host2_CNA2 Nexus-5020-2(config-zoneset)# zoneset activate name FCoE_400 vsan 400 Zoneset activation initiated. check zone status Nexus-5020-2(config)# zoneset distribute full vsan 400 Nexus-5020-2(config)# 15. Save the configuration: Nexus-5020-2(config)# copy running startup-config Configuring the Windows host To configure the Windows host, refer to the following documentation, available at EMC Online Support at https://support.emc.com: ◆ QLogic: EMC Host Connectivity with QLogic Fibre Channel and iSCSI Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) in the Windows Environment
  • 203. Nexus 5000 direct-connect topology 203 Nexus Series Switches Setup Examples ◆ Emulex: EMC Host Connectivity with Emulex Fibre Channel Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) in the Windows Environment ◆ EMC Host Connectivity Guide for Windows Configuring the Linux host To configure the Linux host, refer to the following documentation, available at EMC Online Support at https://support.emc.com: ◆ QLogic: EMC Host Connectivity with QLogic Fibre Channel and iSCSI Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) for the Linux Environment ◆ Emulex: EMC Host Connectivity with Emulex Fibre Channel Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) for the Linux v2.6.x Kernel Environment and the v8.x-Series Driver ◆ EMC Host Connectivity Guide for Linux Configuring the VMware host To configure the VMware host, refer to the following document available EMC Online Support at https://support.emc.com: ◆ EMC Host Connectivity Guide for VMware ESX Server Configuring EMC storage Once the switch, hosts, and adapters have been configured, the storage will need to be configured to each initiator to access a number of LUNs. Configuring network parameters Using the network topology shown in Figure 31 on page 180, display configuration parameters and then follow the steps needed to deploy this Layer 2 network. Etherchannel configuration on a Nexus 7000 switch The example shown next is the creation of the proper Etherchannel setup on a Cisco Catalyst or Cisco Nexus 7000 switch to connect successfully to the Nexus 5020 switch using Ethernet port channels. An Etherchannel is a port channel for L2 VLAN connectivity. Do not confuse an Ethernet channel with SAN-port channels, which are used for VSAN connectivity across bundled SAN E_Ports.
  • 204. 204 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Create the port channel on Core-N7K-1 A switchport, when a part of the Ethernet, is a layer-2 switching port. interface Port-channel switchport switchport trunk encapsulation dot1q1 switchport trunk allowed vlan 2002 switchport mode trunk no ip address 10 GigE interfaces 1/1 and 2/1 on Core-N7K-1 are the members used to make up this port channel: interface TenGigabitEthernet1/1 switchport switchport trunk encapsulation dot1q switchport trunk allowed vlan 200 switchport mode trunk no ip address channel-group 1 mode active3 interface TenGigabitEthernet2/1 switchport switchport trunk encapsulation dot1q switchport trunk allowed vlan 200 switchport mode trunk no ip address channel-group 1 mode active Spanning tree mode for vlan 200 is set to run RPVST versus MST: spanning-tree mode rapid-pvst spanning-tree vlan 200 priority 4096 Create the port channel on Core-N7K-2 Perform the steps from “Create the port channel on Core-N7K-1” on page 204 on Core-N7K-2. VLAN and Etherchannel configuration on Nexus 5020 This section shows how to configure the Nexus-5020-1 and Nexus-5020-2. 1. 802.1q is the only supported type for connection to N5K. 2. Only VLAN 200 is allowed across the port channel. 3. Interface is assigned to port channel 1, active keyword turns on LACP protocol for this link.
  • 205. Nexus 5000 direct-connect topology 205 Nexus Series Switches Setup Examples Configure Nexus-5020-1 IMPORTANT It is important to configure the port channels in this order for "easy on" implementation. Nexus-5020-1(config)# vlan 2001 Nexus-5020-1(config-vlan)# exit Nexus-5020-1(config)# feature lacp ß Link aggregation protocol enabled on Nexus-5020-1 Nexus-5020-1(config)# interface port-channel 1 Nexus-5020-1(config-if-range)# exit Nexus-5020-1(config)# interface ethernet 2/1, 3/12 Nexus-5020-1(config-if-range)# channel-group 1 mode active3 Message***Ethernet1/3 Ethernet1/4 added to port-channel1*** Nexus-5020-1(config-if-range)# interface port-channel 1 ß places you back into the port-channel interface Nexus-5020-1(config-if)# switchport mode trunk Nexus-5020-1(config-if)# switchport trunk allowed vlan 2004 Nexus-5020-1(config-if)# no shut Nexus-5020-1(config-if)# copy running startup-config Configure Nexus-5020-2 Repeat the steps in “Configure Nexus-5020-1” on page 205. Common show commands to display configuration results for port-channel operation: Nexus-5020-1# sh port-channel summary Flags: D - down U - up in port-channel I - Individual S - suspended H - Hot-standby (LACP only) R - Module-removed --------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel --------------------------------------------------------------- 1 Po1(U) Eth LACP Eth1/3(U) Eth1/4(U)5 1. VLAN 200 is configured on the Nexus 5020, matching the VLAN on the core switch. 2. Assigns interfaces 2/1 & 3/1 to the port channel, you can also use hyphens to assign a range of interfaces, such as 1/3-4. 3. Enables LACP. 4. Allows only VLAN 200 across this port-channel. 5. Up and connected using LACP.
  • 206. 206 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Spanning tree configuration and verification on Nexus 5020 You need to know what spanning-tree protocol the VLAN will operate in. Default spanning-tree protocol on the Nexus 5020 is RPVST+. The following example assumes that the core Ethernet switch is running RSTP. Nexus-5020-1# show spanning-tree vlan 200 VLAN0200 Spanning tree enabled protocol rstp Root ID Priority 32768 Address 001d.a1e5.ec58 Cost 1 Port 4097 (port-channel2) Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 32984 (priority 32768 sys-id-ext 200) Address 000d.ec6d.7b41 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Po1 Root FWD 1 128.4097 P2p Nexus-5020-1# sho spanning-tree active detail VLAN0200 is executing the rstp compatible Spanning Tree protocol Bridge Identifier has priority 32768, sysid 211, address 000d.ec6d.7b41 Configured hello time 2, max age 20, forward delay 15 Current root has priority 32768, address 001d.a1e5.ec58 Root port is 4097 (port-channel1), cost of root path is 1 Topology change flag not set, detected flag not set Number of topology changes 1 last change occurred 0:13:54 ago from port-channel1 Times: hold 1, topology change 35, notification 2 hello 2, max age 20, forward delay 15 Timers: hello 0, topology change 0, notification 0 Nexus-5020-1# show spanning-tree summary Switch is in rapid-pvst mode Root bridge for: none Port Type Default is disable Edge Port [PortFast] BPDU Guard Default is disabled Edge Port [PortFast] BPDU Filter Default is disabled Bridge Assurance is disabled Loopguard Default is disabled Pathcost method used is short Name Blocking Listening Learning Forwarding STP Active ---------------------- -------- --------- -------- ---------- ---------- VLAN0200 0 0 0 1 1 ---------------------- -------- --------- -------- ---------- ---------- 1 vlan 0 0 0 1 1
  • 207. Nexus 5000 direct-connect topology 207 Nexus Series Switches Setup Examples Configuration of Multiple Spanning Tree for a VLAN Multiple Spanning Tree (MST), which is the IEEE 802.1s standard, allows you to assign two or more VLANs to a spanning-tree instance. MST is not the default spanning-tree mode. This example shows how to enter MST configuration mode, map VLANs 2 and 4 to MST instance 1, name the region region1, and set the configuration revision to 1. Nexus-5020-1(config)# spanning-tree mode mst1 Nexus-5020-1(config)# spanning-tree mst configuration2 Nexus-5020-1 (config-mst)# instance 1 vlan 2,4 Nexus-5020-1 (config-mst)# name region1 Nexus-5020-1 (config-mst)# revision 1 Nexus-5020-1 (config-mst)# exit3 Multicast on Nexus 5020 By default, the Nexus 5020 switch is enabled for multicast support. IGMP snooping is enabled by default. Internet Group Management Protocol (IGMP) is used to manage the membership of Internet Protocol multicast groups. IGMP is used by IP hosts and adjacent multicast routers to establish multicast group memberships. IGMP Snooping is the process of listening to IGMP traffic. IGMP snooping, as implied by the name, is a feature that allows the switch to "listen in" on the IGMP conversation between hosts and routers by processing the layer 3 IGMP packets sent in a multicast network. (IGMP Snooping is "on" by default for the Nexus 5020.) To display proper operation of the multicast environment and to see enabled igmp options, use <show ip igmp snooping>. The output here shows all defaults. This example shows default VLAN 1 and VLAN 5. VLAN 5 is running multicast. Nexus-5020-1# show ip igmp snooping Global IGMP Snooping Information: IGMP Snooping enabled IGMP Snooping information for vlan 1 IGMP snooping enabled IGMP querier none Switch-querier disabled Explicit tracking enabled 1. Enables MST feature. 2. Enters config mode for MST. 3. Commits all the changes and exits MST configuration.
  • 208. 208 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Fast leave disabled Report suppression enabled Router port detection using PIM Hellos, IGMP Queries Number of router-ports: 0 Number of groups: 0 IGMP Snooping information for vlan 5 IGMP snooping enabled IGMP querier present, address: 172.16.24.1, version: 3 Querier interval: 125 secs Querier last member query interval: 10 secs Querier robustness: 2 Switch-querier enabled, address 172.16.24.1, currently running Explicit tracking enabled Fast leave enabled Report suppression enabled Router port detection using PIM Hellos, IGMP Queries Number of router-ports: 1 Number of groups: 1 Other useful commands to verify proper operation on the core switch, in this case a Nexus 7000: Note: The configuration commands listed below will work for both the Nexus 7000 and the Cat6500. show vlan show port-channel summary show port-channel database show port-channel traffic show port-channel compatibility-parameters show port-channel load-balance show port-channel usage sho interface port-channel 2 switchport sho interface port-channel 2 trunk show lacp neighbor sho interface port-channel 2 counters sho interface port-channel 2 counters errors
  • 209. Nexus 5000 direct-connect topology 209 Nexus Series Switches Setup Examples N7K-1# show spanning-tree VLAN0200 Spanning tree enabled protocol rstp Root ID Priority 4096 Address 0011.bc52.20d3 This bridge is the root Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority 4096 Address 0011.bc52.20d3 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Po1 Desg FWD 1 128.1665 P2p
  • 210. 210 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Nexus 7000, Nexus 5000, and MDS 9500 series topology This section details the setup steps required to create the configuration shown in Figure 32 on page 212, the target topology. It includes the following information: ◆ “Summary of configuration steps” on page 210 ◆ “Target topology” on page 212 ◆ “Prerequisites” on page 214 ◆ “Configuring the Nexus 7000” on page 214 ◆ “Configuring the Nexus 5548” on page 231 ◆ “Configuring the MDS 9513” on page 238 Summary of configuration steps “Configuring N7K-1,” discussed further on page 214. 1. Create storage VDC. 2. Create VLAN/VSAN. 3. Create vfcs. 4. Configure vPC. 5. Connect cables. “Configuring N7K-2,” discussed further on page 223. 1. Create storage VDC. 2. Create VLAN/VSAN. 3. Create vfcs. 4. Configure vPC. 5. Connect cables. “Configuring Nexus-5548-1,” discussed further on page 231. 1. Create VLAN/VSAN. 2. Create vfcs. 3. Configure vPC. 4. Connect cables.
  • 211. Nexus 7000, Nexus 5000, and MDS 9500 series topology 211 Nexus Series Switches Setup Examples “Configuring Nexus-5548-2,” discussed further on page 234. 1. Create VLAN/VSAN. 2. Create vfcs. 3. Configure vPC. 4. Connect cables. “Configuring MDS-9513-1,” discussed further on page 238. 1. Create VLAN/VSAN. 2. Create vfcs. 3. Connect cables. “Configuring MDS-9513-2,” discussed further on page 240. 1. Create VLAN/VSAN. 2. Create vfcs. 3. Connect cables. The process of configuring vPC is covered in “Virtual PortChannel” on page 293. As a result, the vPC configuration steps will not be repeated in this case study.
  • 212. 212 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Target topology Figure 32 shows the target topology. Figure 32 Target topology example 8 7 6 5 4 3 M1 F12 1 8 7 6 5 4 3 2 1 Host_1 CNA_1 CNA_2 Nexus-5548-1 (FC-SW) Nexus-5548-2 (NPV) 10GbE (no FCoE) N7K-1 N7K-2 1 2 3 4 5 6 M1 F1 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 2 1 1 1 0 9 8 7 6 5 4 3 2 1 7 8 1 2 3 4 5 6 7 8 1 2 3 4 FCoE 1 2 3 4 5 MDS-9513-1 FCoE5 4 3 MDS-9513-2 VDC 1 VDC Storage VDC 1 VDC Storage SAN A SAN B vPC vPC 10GbE (FCoE only) 10GbE (Converged) VLAN 100 VLAN/ VSAN 200 Po403 Po402 Po401 Po200 Po200 Po100 Po100 Po301 Po302 Po303 vPC200 vPC100 5 4 3 2 1 FC Native FC VMAX/ VNX ICO-IMG-001031
  • 213. Nexus 7000, Nexus 5000, and MDS 9500 series topology 213 Nexus Series Switches Setup Examples Table 2 lists the Target topology IP address and Domain ID assignments. Table 3 lists the Target topology host information. Table 4 lists the Target topology storage information. Table 2 IP address and Domain ID assignment Target topology IP address Domain ID assignment N7K-1 10.1.1.10 N/A N7K-1-Storage VDC 10.1.1.11 1 MDS-9513-1 10.1.1.12 3 Nexus-5548-1 10.1.1.13 5 N7K-2 10.2.1.10 N/A N7K-2-Storage VDC 10.2.1.11 2 MDS-9513-2 10.2.1.12 4 Nexus-5548-2 10.2.1.13 6 Table 3 Host 1 information CNA Vendor CNA Port # WWPN Storage Ports Emulex 0 10:00:00:00:c9:3c:f8:3c 1 and 2 Emulex 0 10:00:00:00:c9:3c:f7:c0 3 and 4 Table 4 Storage 1 information Name Storage WWPN Host WWPN Storage 1 50:06:04:82:D5:2E:69:79 10:00:00:00:c9:3c:f8:3c Storage 2 50:06:04:82:D5:2E:69:49 10:00:00:00:c9:3c:f8:3c Storage 3 50:06:04:8A:D5:2E:69:79 10:00:00:00:c9:3c:f7:c0 Storage 4 50:06:04:8A:D5:2E:69:49 10:00:00:00:c9:3c:f7:c0
  • 214. 214 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Prerequisites Before configuring the Nexus or MDS switches, confirm the following: ◆ All switch chassis are powered up and have had IP Addresses assigned to them. ◆ In the Nexus 7000, slot 1 contains an M1 linecard and slot 2 contains an F1 line card. ◆ Nexus-5548-2 has the full storage services license installed. ◆ In MDS-9513-1, slot 1 contains an FCoE line card. ◆ In MDS-9513-2, slot 1 contains an FCoE line card and slot 2 contains a line card that supports FC. Configuring the Nexus 7000 This section contains the following information: ◆ “Configuring N7K-1” on page 214 ◆ “Configuring N7K-2” on page 223 Configuring N7K-1 Note: This section assumes that the N7K chassis is powered up and has had an IP Address assigned to it. It also assumes that slot 1 contains an M1 line card and slot 2 contains an F1 line card. Nexus 7000 FCoE requirements ◆ Minimum release of NX-OS 5.2(1) ◆ FCoE module N7K-F132XP-15 ◆ FCoE license FCOE-N7K-F132XP ◆ FCoE will run on a Storage VDC (virtual device context) Note: The Nexus 7000 does not have any native FC capabilities and does not operate in NPV mode. Create the Storage VDC After logging in to the Nexus 7000, you will be in the default VDC where you will complete the following configuration steps. These steps will install the FCoE license and build the Storage VDC.
  • 215. Nexus 7000, Nexus 5000, and MDS 9500 series topology 215 Nexus Series Switches Setup Examples Note: Make sure the FCoE module is physically installed in the chassis before trying to enable the FCoE license. 1. Enable the FCoE feature. N7K-1# config t Enter configuration commands, one per line. End with CNTL/Z. N7K-1(config)# install feature-set fcoe 2. Update QoS settings. N7K-1(config)# system qos N7K-1(config-sys-qos)# service-policy type network-qos default-nq-7e-policy N7K-1(config-sys-qos)# exit N7K-1(config)# 3. Enable LACP and LLDP. N7K-1(config)# feature lacp N7K-1(config)# feature lldp N7K-1(config)# 4. Create the Storage VDC. Note: The vdc name "StorageVDC" is used for example purposes only and is not a requirement. N7K-1(config)# vdc StorageVDC type storage Note: Creating VDC, one moment please ... N7K-1(config-vdc)# 5. Associate the FCoE license to the FCoE module. N7K-1(config-vdc)# license fcoe module 2 N7K-1(config)# 6. Verify the license is installed. N7K-1 (config)# show license usage Feature Ins Lic Status Expiry Date Comments Count -------------------------------------------------------------------------------- FCOE-N7K-F132XP Yes 1 In use Never - Note: output truncated -------------------------------------------------------------------------------- 7. Verify that the FCoE feature is installed. N7K-1(config)# show feature-set fcoe Feature Set Name ID State -------------------- -------- -------- fcoe 1 enabled
  • 216. 216 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 8. Allocate Ethernet interface to Storage VDC. In this example we have the FCoE module inserted into slot 2. When allocating ports to the storage VDC, keep in mind that both members of each port group will need to be assigned to the same VDC. For the FCoE "F1" blade, the port groups are as shown in Table 5. N7K-1(config)# vdc StorageVDC N7K-1(config-vdc)# allocate interface ethernet 2/1-8 Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the ports (y/n)? [yes] y N7K-1(config-vdc)# 9. Allocate the VLANS to be used for FCoE. Note: By default the range is from 1 - 3967. N7K-1 (config-vdc)# allocate fcoe-vlan-range 200 N7K-1 (config-vdc)# Table 5 Port groups Port Group Port Number Group 1 1 and 2 Group 2 3 and 4 Group 3 5 and 6 Group 4 7 and 8 Group 5 9 and 10 Group 6 11 and 12 Group 7 13 and 14 Group 8 15 and 16 Group 9 17 and 18 Group 10 19 and 20 Group 11 21 and 22 Group 12 23 and 24 Group 13 25 and 26 Group 14 27 and 28 Group 15 29 and 30 Group 16 31 and 32
  • 217. Nexus 7000, Nexus 5000, and MDS 9500 series topology 217 Nexus Series Switches Setup Examples 10. Switch to the newly created VDC and complete the VDC setup. Note: The answers to the following prompts are provided for the sake of this example only and do not necessarily represent best practices. N7K-1 (config-vdc)# switchto vdc StorageVDC ---- System Admin Account Setup ---- Do you want to enforce secure password standard (yes/no) [y]: n Enter the password for "admin": <Enter password> Confirm the password for "admin": <Re-enter password> ---- Basic System Configuration Dialog VDC: 4 ---- This setup utility will guide you through the basic configuration of the system. Setup configures only enough connectivity for management of the system. Please register Cisco Nexus7000 Family devices promptly with your supplier. Failure to register may affect response times for initial service calls. Nexus7000 devices must be registered to receive entitled support services. Press Enter at anytime to skip a dialog. Use ctrl-c at anytime to skip the remaining dialogs. Would you like to enter the basic configuration dialog (yes/no): y Create another login account (yes/no) [n]: n Configure read-only SNMP community string (yes/no) [n]: n Configure read-write SNMP community string (yes/no) [n]: n Enter the switch name : StorageVDC Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: y Mgmt0 IPv4 address : 10.1.1.11 Mgmt0 IPv4 netmask : 255.255.255.0 Configure the default gateway? (yes/no) [y]: y IPv4 address of the default gateway : 10.1.1.1 Configure advanced IP options? (yes/no) [n]: n
  • 218. 218 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Enable the telnet service? (yes/no) [n]: n Enable the ssh service? (yes/no) [y]: y Type of ssh key you would like to generate (dsa/rsa) [rsa]: rsa Number of rsa key bits <1024-2048> [1024]: 1024 Configure default interface layer (L3/L2) [L3]: L3 Configure default switchport interface state (shut/noshut) [shut]: shut The following configuration will be applied: no password strength-check switchname StorageVDC interface mgmt0 ip address 10.1.1.11 255.255.255.0 no shutdown vrf context management ip route 0.0.0.0/0 10.1.1.1 exit no feature telnet ssh key rsa 1024 force feature ssh no system default switchport system default switchport shutdown Would you like to edit the configuration? (yes/no) [n]: n Use this configuration and save it? (yes/no) [y]: y Configuration update aborted: This vdc has had a global configuration change since the last saved config. Please save config in default vdc before proceeding Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php N7K-1-StorageVDC#
  • 219. Nexus 7000, Nexus 5000, and MDS 9500 series topology 219 Nexus Series Switches Setup Examples 11. Ensure that the appropriate interfaces have been allocated to the VDC. N7K-1-StorageVDC# show interface brief ------------------------------------------------------------------------------- Interface Status Speed (Gbps) ------------------------------------------------------------------------------- sup-fc0 up 1 -------------------------------------------------------------------------------- Port VRF Status IP Address Speed MTU -------------------------------------------------------------------------------- mgmt0 -- up 10.1.1.11 1000 1500 -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth2/1 1 eth access down Administratively down 10G(D) -- Eth2/2 1 eth access down Administratively down 10G(D) -- Eth2/3 1 eth access down Administratively down 10G(D) -- Eth2/4 1 eth access down Administratively down 10G(D) -- Eth2/5 1 eth access down Administratively down 10G(D) -- Eth2/6 1 eth access down Administratively down 10G(D) -- Eth2/7 1 eth access down Administratively down 10G(D) -- Eth2/8 1 eth access down Administratively down 10G(D) -- N7K-1-StorageVDC# 12. Enable needed features within Storage VDC. Note: Enabling these features within the Storage VDC does not enable them in the default VDC. N7K-1-StorageVDC# config Enter configuration commands, one per line. End with CNTL/Z. N7K-1-StorageVDC(config)# feature lldp N7K-1-StorageVDC(config)# feature lacp 13. Save the configuration. N7K-1-StorageVDC(config)# exit N7K-1-StorageVDC# exit N7K-1(config-vdc)# copy running-config startup-config [########################################] 100% Copy complete, now saving to disk (please wait)... N7K-1(config-vdc)#
  • 220. 220 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 14. Verify the Storage VDC. N7K-1(config-vdc)# show vdc StorageVDC vdc_id vdc_name state mac type lc ------ -------- ----- ---------- --------- ------------------------------------ 1 N7K-1 active 00:24:f7:17:05:c1 Ethernet m1 f1 m1xl 2 StorageVDC active 00:24:f7:17:05:c2 Storage f1 Configure FCoE settings on the Storage VDC 1. Switch to the Storage VDC. N7K-1# switchto vdc StorageVDC N7K-1-StorageVDC# config N7K-1-StorageVDC(config)# 2. Create the FCoE VLAN. Note: VLANs created in the storage VDC must fall within the range of VLAN IDs specified in the "allocate fcoe-vlan-range <VLAN RANGE>" configuration parameter. See step 8 on page 216 for additional information. N7K-1-StorageVDC(config)# vlan 100 N7K-1-StorageVDC(config-vlan)# fcoe vsan 100 N7K-1-StorageVDC(config-vlan)# 3. Create the FCoE VSAN and add the vfc for the storage port. N7K-1-StorageVDC(config-vlan)# vsan database N7K-1-StorageVDC(config-vsan-db)# vsan 100 N7K-1-StorageVDC(config-vsan-db)# exit N7K-1-StorageVDC(config)# 4. Configure the PortChannels. Note: As shown in the target topology diagram, Figure 32 on page 212, in this example there are 5 physical interfaces being used in the storage VDC. Two of these physical interfaces will be added to PortChannel 401, two others will be added to PortChannel 403. Each of these PortChannels will be configured in this section.
  • 221. Nexus 7000, Nexus 5000, and MDS 9500 series topology 221 Nexus Series Switches Setup Examples IMPORTANT One of the PortChannels that will be configured in this section will be attached to an MDS. When configuring a PortChannel interface on an MDS, the PortChannel ID must be greater than 256 to avoid a potential overlap with a SAN-PortChannel ID instance. This restriction on PortChannel IDs does not exist on the Nexus 7000. As a result, if a PortChannel ID less than or equal to 256 is used on the Nexus 7000, it will not be possible to use the same ID on the MDS. Although it is not a requirement for the IDs to match on both sides of the PortChannel, it is considered a best practice to have them match to eliminate confusion in the future should the need to troubleshoot arise. a. Configure PortChannel 401. N7K-1-StorageVDC(config)# interface port-channel 401 N7K-1-StorageVDC(config-if)# switchport N7K-1-StorageVDC(config-if)# switchport mode trunk N7K-1-StorageVDC(config-if)# switchport trunk allowed vlan 100 N7K-1-StorageVDC(config-if)# no shut b. Configure PortChannel 403. N7K-1-StorageVDC(config)# interface port-channel 403 N7K-1-StorageVDC(config-if)# switchport N7K-1-StorageVDC(config-if)# switchport mode trunk N7K-1-StorageVDC(config-if)# switchport trunk allowed vlan 100 N7K-1-StorageVDC(config-if)# no shut 5. Configure the physical interfaces. Note: As shown in the target topology diagram, Figure 32 on page 212, there are 5 physical interfaces in this example being used in the storage VDC. Two of these physical interfaces will be added to PortChannel 401, two others will be added to PortChannel 403 and the final interface will be used to connect to port 2 on the Storage array. Each of these interfaces will be configured in this section. a. Configure interface Ethernet 2/1-2. N7K-1-StorageVDC(config)# interface ethernet 2/1-2 N7K-1-StorageVDC(config-if-range)# switchport mode trunk N7K-1-StorageVDC(config-if-range)# switchport trunk allowed vlan 100 N7K-1-StorageVDC(config-if-range)# channel-group 403 mode active N7K-1-StorageVDC(config-if-range)# no shut N7K-1-StorageVDC(config-if-range)# exit N7K-1-StorageVDC(config)#
  • 222. 222 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples b. Configure interface Ethernet 2/3-4. N7K-1-StorageVDC(config)# interface ethernet 2/3-4 N7K-1-StorageVDC(config-if-range)# switchport mode trunk N7K-1-StorageVDC(config-if-range)# switchport trunk allowed vlan 100 N7K-1-StorageVDC(config-if-range)# channel-group 401 mode active N7K-1-StorageVDC(config-if-range)# no shut N7K-1-StorageVDC(config-if-range)# exit N7K-1-StorageVDC(config)# c. Configure interface Ethernet 2/8. N7K-1-StorageVDC(config)# interface ethernet 2/8 N7K-1-StorageVDC(config-if)# switchport mode trunk N7K-1-StorageVDC(config-if)# switchport trunk allowed vlan 100 N7K-1-StorageVDC(config-if)# no shut N7K-1-StorageVDC(config-if)# exit N7K-1-StorageVDC(config)# 6. Create the VFC interfaces. Note: As shown in the target topology diagram, Figure 32 on page 212, there are 5 physical interfaces in this example being used in the storage VDC. Two of these physical interfaces will be added to PortChannel 401, two others will be added to PortChannel 403 and the final interface will be used to connect to port 2 on the Storage array. Each of these interfaces will be configured in this section a. Configure interface vfc-port-channel 401. N7K-1-StorageVDC(config)# interface vfc-port-channel 401 N7K-1-StorageVDC(config-if)# switchport mode e N7K-1-StorageVDC(config-if)# switchport trunk allowed vsan 100 N7K-1-StorageVDC(config-if)# no shut b. Configure interface vfc-port-channel 403. N7K-1-StorageVDC(config)# interface vfc-port-channel 403 N7K-1-StorageVDC(config-if)# switchport mode e N7K-1-StorageVDC(config-if)# switchport trunk allowed vsan 100 N7K-1-StorageVDC(config-if)# no shut c. Configure the interface for the storage port on vfc 8. N7K-1-StorageVDC(config)# interface vfc 8 N7K-1-StorageVDC(config-if)# switchport mode f N7K-1-StorageVDC(config-if)# switchport trunk allowed vsan 100 N7K-1-StorageVDC(config-if)# bind int e 2/8 N7K-1-StorageVDC(config-if)# no shut
  • 223. Nexus 7000, Nexus 5000, and MDS 9500 series topology 223 Nexus Series Switches Setup Examples 7. Add the vfc for the storage port. N7K-1-StorageVDC(config-if)# vsan database N7K-1-StorageVDC(config-vsan-db)# vsan 100 interface vfc 8 N7K-1-StorageVDC(config-vsan-db)# exit N7K-1-StorageVDC(config)# Configuring N7K-2 Note: This section assumes that the N7K chassis is powered up and has had an IP Address assigned to it. It also assumes that slot 1 contains an M1 line card and slot 2 contains an F1 line card. Nexus 7000 FCoE requirements ◆ Minimum release of NX-OS 5.2(1) ◆ FCoE module N7K-F132XP-15 ◆ FCoE license FCOE-N7K-F132XP ◆ FCoE will run on a Storage VDC (virtual device context) Note: The Nexus 7000 does not have any native FC capabilities and does not operate in NPV mode. Create the Storage VDC After logging in to the Nexus 7000, you will be in the default VDC where you will complete the following configuration steps. These steps will install the FCoE license and build the Storage VDC. Note: Make sure the FCoE module is physically installed in the chassis before trying to enable the FCoE license. 1. Enable the FCoE feature. N7K-2# config t Enter configuration commands, one per line. End with CNTL/Z. N7K-2(config)# install feature-set fcoe 2. Update QoS settings. N7K-2(config)# system qos N7K-2(config-sys-qos)# service-policy type network-qos default-nq-7e-policy N7K-2(config-sys-qos)# exit N7K-2(config)# 3. Enable LACP and LLDP. N7K-2(config)# feature lacp N7K-2(config)# feature lldp N7K-2(config)#
  • 224. 224 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 4. Create the Storage VDC. Note: The vdc name "StorageVDC" is used for example purposes only and is not a requirement. N7K-2(config)# vdc StorageVDC type storage Note: Creating VDC, one moment please ... N7K-2(config-vdc)# 5. Associate the FCoE license to the FCoE module. N7K-2(config-vdc)# license fcoe module 2 N7K-2(config)# 6. Verify the license is installed. N7K-2 (config)# show license usage Feature Ins Lic Status Expiry Date Comments Count -------------------------------------------------------------------------------- FCOE-N7K-F132XP Yes 1 In use Never - Note: output truncated -------------------------------------------------------------------------------- 7. Verify that the FCoE feature is installed. N7K-2(config)# show feature-set fcoe Feature Set Name ID State -------------------- -------- -------- fcoe 1 enabled 8. Allocate Ethernet interface to Storage VDC. In this example we have the FCoE module inserted into slot 2. When allocating ports to the storage VDC, keep in mind that both members of each port group will need to be assigned to the same VDC. For the FCoE "F1" blade, the port groups are as shown in Table 6. Table 6 Port groups (page 1 of 2) Port Group Port Number Group 1 1 and 2 Group 2 3 and 4 Group 3 5 and 6 Group 4 7 and 8 Group 5 9 and 10 Group 6 11 and 12 Group 7 13 and 14
  • 225. Nexus 7000, Nexus 5000, and MDS 9500 series topology 225 Nexus Series Switches Setup Examples N7K-2(config)# vdc StorageVDC N7K-2(config-vdc)# allocate interface ethernet 2/1-8 Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the ports (y/n)? [yes] y N7K-2(config-vdc)# 9. Allocate the VLANS to be used for FCoE. Note: By default the range is from 1 - 3967. N7K-2 (config-vdc)# allocate fcoe-vlan-range 200 N7K-2 (config-vdc)# 10. .Switch to the newly created VDC and complete the VDC setup. Note: The answers to the following prompts are provided for the sake of this example only and do not necessarily represent best practices. N7K-2 (config-vdc)# switchto vdc StorageVDC ---- System Admin Account Setup ---- Do you want to enforce secure password standard (yes/no) [y]: n Enter the password for "admin": <Enter password> Confirm the password for "admin": <Re-enter password> ---- Basic System Configuration Dialog VDC: 4 ---- This setup utility will guide you through the basic configuration of the system. Setup configures only enough connectivity for management of the system. Please register Cisco Nexus7000 Family devices promptly with your supplier. Failure to register may affect response times for initial Group 8 15 and 16 Group 9 17 and 18 Group 10 19 and 20 Group 11 21 and 22 Group 12 23 and 24 Group 13 25 and 26 Group 14 27 and 28 Group 15 29 and 30 Group 16 31 and 32 Table 6 Port groups (page 2 of 2)
  • 226. 226 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples service calls. Nexus7000 devices must be registered to receive entitled support services. Press Enter at anytime to skip a dialog. Use ctrl-c at anytime to skip the remaining dialogs. Would you like to enter the basic configuration dialog (yes/no): y Create another login account (yes/no) [n]: n Configure read-only SNMP community string (yes/no) [n]: n Configure read-write SNMP community string (yes/no) [n]: n Enter the switch name : StorageVDC Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: y Mgmt0 IPv4 address : 10.1.2.11 Mgmt0 IPv4 netmask : 255.255.255.0 Configure the default gateway? (yes/no) [y]: y IPv4 address of the default gateway : 10.1.2.1 Configure advanced IP options? (yes/no) [n]: n Enable the telnet service? (yes/no) [n]: n Enable the ssh service? (yes/no) [y]: y Type of ssh key you would like to generate (dsa/rsa) [rsa]: rsa Number of rsa key bits <1024-2048> [1024]: 1024 Configure default interface layer (L3/L2) [L3]: L3 Configure default switchport interface state (shut/noshut) [shut]: shut The following configuration will be applied: no password strength-check switchname StorageVDC interface mgmt0 ip address 10.1.2.11 255.255.255.0 no shutdown vrf context management ip route 0.0.0.0/0 10.1.2.1 exit no feature telnet ssh key rsa 1024 force feature ssh
  • 227. Nexus 7000, Nexus 5000, and MDS 9500 series topology 227 Nexus Series Switches Setup Examples no system default switchport system default switchport shutdown Would you like to edit the configuration? (yes/no) [n]: n Use this configuration and save it? (yes/no) [y]: y Configuration update aborted: This vdc has had a global configuration change since the last saved config. Please save config in default vdc before proceeding Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php N7K-2-StorageVDC# 11. Ensure that the appropriate interfaces have been allocated to the VDC. N7K-2-StorageVDC# show interface brief ------------------------------------------------------------------------------- Interface Status Speed (Gbps) ------------------------------------------------------------------------------- sup-fc0 up 1 -------------------------------------------------------------------------------- Port VRF Status IP Address Speed MTU -------------------------------------------------------------------------------- mgmt0 -- up 10.1.1.11 1000 1500 -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth2/1 1 eth access down Administratively down 10G(D) -- Eth2/2 1 eth access down Administratively down 10G(D) -- Eth2/3 1 eth access down Administratively down 10G(D) -- Eth2/4 1 eth access down Administratively down 10G(D) -- Eth2/5 1 eth access down Administratively down 10G(D) -- Eth2/6 1 eth access down Administratively down 10G(D) -- Eth2/7 1 eth access down Administratively down 10G(D) -- Eth2/8 1 eth access down Administratively down 10G(D) -- N7K-2-StorageVDC#
  • 228. 228 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 12. Enable needed features within Storage VDC. Note: Enabling these features within the Storage VDC does not enable them in the Default VDC. N7K-2-StorageVDC# config Enter configuration commands, one per line. End with CNTL/Z. N7K-2-StorageVDC(config)# feature lldp N7K-2-StorageVDC(config)# feature lacp N7K-2-StorageVDC(config)# feature npiv 13. Save the configuration. N7K-2-StorageVDC(config)# exit N7K-2-StorageVDC# exit N7K-2(config-vdc)# copy running-config startup-config [########################################] 100% Copy complete, now saving to disk (please wait)... N7K-2(config-vdc)# 14. Verify the Storage VDC. N7K-2(config-vdc)# show vdc StorageVDC vdc_id vdc_name state mac type lc ------ -------- ----- ---------- --------- ------------------------------------ 1 N7K-2 active 00:24:f7:17:05:c1 Ethernet m1 f1 m1xl 2 StorageVDC active 00:24:f7:17:05:c2 Storage f1 Configure FCoE settings on the Storage VDC 1. Switch to the Storage VDC. N7K-2# switchto vdc StorageVDC N7K-2-StorageVDC# config N7K-2-StorageVDC(config)# 2. Create the FCoE VLAN. Note: VLANs created in the storage VDC must fall within the range of VLAN IDs specified in the "allocate fcoe-vlan-range <VLAN RANGE>" configuration parameter. See step 8 on page 224 for additional information. N7K-2-StorageVDC(config)# vlan 200 N7K-2-StorageVDC(config-vlan)# fcoe vsan 200 N7K-2-StorageVDC(config-vlan)# 3. Create the FCoE VSAN and add the vfc for the storage port. N7K-2-StorageVDC(config-vlan)# vsan database N7K-2-StorageVDC(config-vsan-db)# vsan 200
  • 229. Nexus 7000, Nexus 5000, and MDS 9500 series topology 229 Nexus Series Switches Setup Examples N7K-2-StorageVDC(config-vsan-db)# exit N7K-2-StorageVDC(config)# 4. Configure the PortChannels. Note: As shown in the target topology diagram, Figure 32 on page 212, in this example there are 5 physical interfaces being used in the storage VDC. Two of these physical interfaces will be added to PortChannel 401, two others will be added to PortChannel 403. Each of these PortChannels will be configured in this section. IMPORTANT One of the PortChannels that will be configured in this section will be attached to an MDS. When configuring a PortChannel interface on an MDS, the PortChannel ID must be greater than 256 to avoid a potential overlap with a SAN-PortChannel ID instance. This restriction on PortChannel IDs does not exist on the Nexus 7000. As a result, if a PortChannel ID less than or equal to 256 is used on the Nexus 7000, it will not be possible to use the same ID on the MDS. Although it is not a requirement for the IDs to match on both sides of the PortChannel, it is considered a best practice to have them match to eliminate confusion in the future should the need to troubleshoot arise. a. Configure PortChannel 301. N7K-2-StorageVDC(config)# interface port-channel 301 N7K-2-StorageVDC(config-if)# switchport N7K-2-StorageVDC(config-if)# switchport mode trunk N7K-2-StorageVDC(config-if)# switchport trunk allowed vlan 200 N7K-2-StorageVDC(config-if)# no shut b. Configure PortChannel 303. N7K-2-StorageVDC(config)# interface port-channel 303 N7K-2-StorageVDC(config-if)# switchport N7K-2-StorageVDC(config-if)# switchport mode trunk N7K-2-StorageVDC(config-if)# switchport trunk allowed vlan 200 N7K-2-StorageVDC(config-if)# no shut 5. Configure the physical interfaces.
  • 230. 230 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Note: As shown in the target topology diagram, Figure 32 on page 212, there are 5 physical interfaces in this example being used in the storage VDC. Two of these physical interfaces will be added to PortChannel 401, two others will be added to PortChannel 403 and the final interface will be used to connect to port 2 on the Storage array. Each of these interfaces will be configured in this section. a. Configure interface Ethernet 2/1-2. N7K-2-StorageVDC(config)# interface ethernet 2/1-2 N7K-2-StorageVDC(config-if-range)# switchport mode trunk N7K-2-StorageVDC(config-if-range)# switchport trunk allowed vlan 200 N7K-2-StorageVDC(config-if-range)# channel-group 403 mode active N7K-2-StorageVDC(config-if-range)# no shut N7K-2-StorageVDC(config-if-range)# exit N7K-2-StorageVDC(config)# b. Configure interface Ethernet 2/3-4. N7K-2-StorageVDC(config)# interface ethernet 2/3-4 N7K-2-StorageVDC(config-if-range)# switchport mode trunk N7K-2-StorageVDC(config-if-range)# switchport trunk allowed vlan 200 N7K-2-StorageVDC(config-if-range)# channel-group 401 mode active N7K-2-StorageVDC(config-if-range)# no shut N7K-2-StorageVDC(config-if-range)# exit N7K-2-StorageVDC(config)# c. Configure interface Ethernet 2/8. N7K-2-StorageVDC(config)# interface ethernet 2/8 N7K-2-StorageVDC(config-if)# switchport mode trunk N7K-2-StorageVDC(config-if)# switchport trunk allowed vlan 200 N7K-2-StorageVDC(config-if)# no shut N7K-2-StorageVDC(config-if)# exit N7K-2-StorageVDC(config)# 6. Create the VFC interfaces. Note: As shown in the target topology diagram, Figure 32 on page 212, there are 5 physical interfaces in this example being used in the storage VDC. Two of these physical interfaces will be added to PortChannel 401, two others will be added to PortChannel 403 and the final interface will be used to connect to port 2 on the Storage array. Each of these interfaces will be configured in this section a. Configure interface vfc-port-channel 301. N7K-2-StorageVDC(config)# interface vfc-port-channel 301 N7K-2-StorageVDC(config-if)# switchport mode e
  • 231. Nexus 7000, Nexus 5000, and MDS 9500 series topology 231 Nexus Series Switches Setup Examples N7K-2-StorageVDC(config-if)# switchport trunk allowed vsan 200 N7K-2-StorageVDC(config-if)# no shut b. Configure interface vfc-port-channel 303. N7K-2-StorageVDC(config)# interface vfc-port-channel 303 N7K-2-StorageVDC(config-if)# switchport mode e N7K-2-StorageVDC(config-if)# switchport trunk allowed vsan 200 N7K-2-StorageVDC(config-if)# no shut c. Configure the interface for the storage port on vfc 8. N7K-2-StorageVDC(config)# interface vfc 8 N7K-2-StorageVDC(config-if)# switchport mode f N7K-2-StorageVDC(config-if)# switchport trunk allowed vsan 200 N7K-2-StorageVDC(config-if)# bind int e 2/8 N7K-2-StorageVDC(config-if)# no shut 7. Add the vfc for the storage port. N7K-2-StorageVDC(config-if)# vsan database N7K-2-StorageVDC(config-vsan-db)# vsan 200 interface vfc 8 N7K-2-StorageVDC(config-vsan-db)# exit N7K-2-StorageVDC(config)# Configuring the Nexus 5548 This section contains the following information: ◆ “Configuring Nexus-5548-1” on page 231 ◆ “Configuring Nexus-5548-2” on page 234 Configuring Nexus-5548-1 Note: This section assumes that the Nexus 5548 chassis is powered up and has had an IP Address assigned to it. 1. Enable the FCoE and LACP features. Nexus-5548-1# config t Enter configuration commands, one per line. End with CNTL/Z. Nexus-5548-1 (config)# feature fcoe Nexus-5548-1 (config)# feature lacp 2. Configure QOS parameters. Nexus-5548-1 (config)# system qos Nexus-5548-1 (config-sys-qos)# service-policy type qos input fcoe default-in-policy Nexus-5548-1 (config-sys-qos)# service-policy type queuing input fcoe-default-in-policy Nexus-5548-1 (config-sys-qos)# service-policy type queuing output fcoe-default-out-policy
  • 232. 232 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Nexus-5548-1 (config-sys-qos)# service-policy type network-qos fcoedefault-nq-policy 3. Configure the VLAN and VSAN. Nexus-5548-1 (config)# vlan 100 Nexus-5548-1 (config)# fcoe vsan 100 Nexus-5548-1 (config)# vsan database Nexus-5548-1 (config-vsan-db)# vsan 100 Nexus-5548-1 (config-vsan-db)# vsan 100 interface vfc 10 Nexus-5548-1 (config-vsan-db)# exit Nexus-5548-1 (config)# 4. Configure the PortChannels. Note: As shown in the target topology diagram, Figure 32 on page 212, there are 5 physical interfaces in this example being used for FCoE on Nexus-5548-1; they are 1/1-4 and 1/10. Two of these physical interfaces will be added to PortChannel 401, two others will be added to PortChannel 402, and the remainder will be used for the host to attach to. Each of the PortChannels will be configured in this section. IMPORTANT One of the PortChannels that will be configured in this section will be attached to an MDS. When configuring a PortChannel interface on an MDS, the PortChannel ID must be greater than 256 to avoid a potential overlap with a SAN-PortChannel ID instance. This restriction on PortChannel IDs does not exist on the Nexus 5548. As a result, if a PortChannel ID less than or equal to 256 is used on the Nexus 5548, it will not be possible to use the same ID on the MDS. Although it is not a requirement for the IDs to match on both sides of the PortChannel, it is generally considered a best practice to have them match to eliminate confusion in the future should the need to troubleshoot arise. a. Configure PortChannel 401. Note: PortChannel 401 is being connected to a Nexus 7k. As a result, the link between the two will actually be an FCoE ISL. Nexus-5548-1 (config)# interface port-channel 401 Nexus-5548-1 (config-if)# switchport Nexus-5548-1 (config-if)# switchport mode trunk Nexus-5548-1 (config-if)# switchport trunk allowed vlan 100 Nexus-5548-1 (config-if)# no shut
  • 233. Nexus 7000, Nexus 5000, and MDS 9500 series topology 233 Nexus Series Switches Setup Examples b. Configure PortChannel 402. Nexus-5548-1 (config)# interface port-channel 402 Nexus-5548-1 (config-if)# switchport Nexus-5548-1 (config-if)# switchport mode trunk Nexus-5548-1 (config-if)# switchport trunk allowed vlan 100 Nexus-5548-1 (config-if)# no shut Nexus-5548-1 (config-if)# exit Nexus-5548-1 (config)# 5. Configure load balancing. Note: According to Cisco documentation, on the N5K the default load balance mechanism on the LACP port channel is source-dest-ip. If left in this mode, all FCoE traffic will take the same link in the port channel when the N5K is forwarding FCoE frames to the N7K and MDS, as this provides SID-DID load balancing for FCoE frames. To enable the N5K to load balance using exchange ID, you must configure the N5K for source-dest-port load balancing. Nexus-5548-1 (config)# port-channel load-balance source-dest-port Nexus-5548-1 (config)# 6. Configure the physical interfaces. Note: As shown in the target topology diagram, Figure 32 on page 212, there are 5 physical interfaces in this example being used for FCoE on Nexus-5548-1; they are 1/1-4 and 1/10. Two of these physical interfaces will be added to PortChannel 401, two others will be added to PortChannel 402, and the remainder will be used for the host to attach to. Each of the PortChannels will be configured in this section. a. Configure interface Ethernet 1/1-2. Nexus-5548-1 (config)# interface ethernet 1/1-2 Nexus-5548-1 (config-if-range)# switchport mode trunk Nexus-5548-1 (config-if-range)# switchport trunk allowed vlan 100 Nexus-5548-1 (config-if-range)# channel-group 402 mode active Nexus-5548-1 (config-if-range)# no shut Nexus-5548-1 (config-if-range)# exit Nexus-5548-1 (config)# b. Configure interface Ethernet 1/3-4. Nexus-5548-1 (config)# interface ethernet 1/3-4 Nexus-5548-1 (config-if-range)# switchport mode trunk Nexus-5548-1 (config-if-range)# switchport trunk allowed vlan 100 Nexus-5548-1 (config-if-range)# channel-group 401 mode active Nexus-5548-1 (config-if-range)# no shut Nexus-5548-1 (config-if-range)# exit Nexus-5548-1 (config)#
  • 234. 234 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples c. Configure interface Ethernet 1/10. Nexus-5548-1 (config)# interface ethernet 1/10 Nexus-5548-1 (config-if)# switchport mode trunk Nexus-5548-1 (config-if)# switchport trunk allowed vlan 100 Nexus-5548-1 (config-if)# no shut Nexus-5548-1 (config-if)# exit Nexus-5548-1 (config)# 7. Create the VFC interfaces. Note: As shown in the target topology diagram, Figure 32 on page 212, there are 5 physical interfaces in this example being used for FCoE on Nexus-5548-1; they are 1/1-4 and 1/10. Two of these physical interfaces will be added to PortChannel 401, two others will be added to PortChannel 402, and the remainder will be used for the host to attach to. Each of the PortChannels will be configured in this section. a. Configure interface vfc-port-channel 401. Nexus-5548-1 (config)# interface vfc-port-channel 401 Nexus-5548-1 (config-if)# switchport mode e Nexus-5548-1 (config-if)# switchport trunk allowed vsan 100 Nexus-5548-1 (config-if)# no shut b. Configure interface vfc-port-channel 402. Nexus-5548-1 (config)# interface vfc-port-channel 402 Nexus-5548-1 (config-if)# switchport mode e Nexus-5548-1 (config-if)# switchport trunk allowed vsan 100 Nexus-5548-1 (config-if)# no shut c. Configure the interface for the storage port on vfc 10. Nexus-5548-1 (config)# interface vfc 10 Nexus-5548-1 (config-if)# switchport mode f Nexus-5548-1 (config-if)# switchport trunk allowed vsan 100 Nexus-5548-1 (config-if)# no shut Configuring Nexus-5548-2 In this section, a Nexus 5548 that is running in NPV mode will be connected to both a Nexus 7k and an MDS switch at the same time. This configuration presents a problem since currently only a single FCoE NPV link can be configured on the Nexus 5548. As a result, the uplink to the Nexus 7k will be an FCoE NPV uplink and the other will be an FC NPV uplink to the MDS. If the Nexus 7k was not a part of this configuration, then the uplink to the MDS switch could be an FCoE NPV uplink.
  • 235. Nexus 7000, Nexus 5000, and MDS 9500 series topology 235 Nexus Series Switches Setup Examples 1. Enable the FCoE and NPV features. Nexus-5548-2# config t Enter configuration commands, one per line. End with CNTL/Z. Nexus-5548-2 (config)# feature fcoe Nexus-5548-2 (config)# feature npv Verify that boot variables are set and the changes are saved. Changing to npv mode erases the current configuration and reboots the switch in npv mode. Do you want to continue? (y/n):y Note: The switch will reboot at this point. Once the switch comes back up, continue on with the next steps. 2. Enable the LACP feature. Nexus-5548-2# config t Enter configuration commands, one per line. End with CNTL/Z. Nexus-5548-2 (config)# feature lacp 3. Configure QOS parameters. Nexus-5548-2 (config)# system qos Nexus-5548-2 (config-sys-qos)# service-policy type qos input fcoe default-in-policy Nexus-5548-2 (config-sys-qos)# service-policy type queuing input fcoe-default-in-policy Nexus-5548-2 (config-sys-qos)# service-policy type queuing output fcoe-default-out-policy Nexus-5548-2 (config-sys-qos)# service-policy type network-qos fcoedefault-nq-policy 4. Configure the VLAN and VSAN. Nexus-5548-2 (config)# vlan 200 Nexus-5548-2 (config)# fcoe vsan 200 Nexus-5548-2 (config)# vsan database Nexus-5548-2 (config-vsan-db)# vsan 200 Nexus-5548-2 (config-vsan-db)# vsan 200 interface vfc 10 Nexus-5548-2 (config-vsan-db)# exit Nexus-5548-2 (config)# 5. Configure the PortChannels. Note: As shown in the target topology diagram, Figure 32 on page 212, there are 5 physical interfaces in this example being used for FCoE on Nexus-5548-2; they are e1/3-4, e1/10 and fc2/1-2. Two of these physical interfaces will be added to PortChannel 301, two others will be added to PortChannel 302, and the remainder will be used for the host to attach to. Each of the PortChannels will be configured in this section.
  • 236. 236 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples IMPORTANT One of the PortChannels that will be configured in this section will be attached to an MDS. When configuring a PortChannel interface on an MDS, the PortChannel ID must be greater than 256 to avoid a potential overlap with a SAN-PortChannel ID instance. This restriction on PortChannel IDs does not exist on the Nexus 5548. As a result, if a PortChannel ID less than or equal to 256 is used on the Nexus 5548, it will not be possible to use the same ID on the MDS. Although it is not a requirement for the IDs to match on both sides of the PortChannel, it is considered a best practice to have them match to eliminate confusion in the future should the need to troubleshoot arise. a. Configure PortChannel 301. Note: Since Nexus-5548-2 is running in NPV mode, the link between the two will actually be an FCoE NPV link. Nexus-5548-2 (config)# interface port-channel 301 Nexus-5548-2 (config-if)# switchport Nexus-5548-2 (config-if)# switchport mode trunk Nexus-5548-2 (config-if)# switchport trunk allowed vlan 200 Nexus-5548-2 (config-if)# no shut b. Configure PortChannel 302. Nexus-5548-2 (config)# interface san-port-channel 302 Nexus-5548-2 (config-if)# channel mode active Nexus-5548-2 (config-if)# switchport mode np Nexus-5548-2 (config-if)# switchport trunk allowed vsan 100 Nexus-5548-2 (config-if)# switchport trunk mode on Nexus-5548-2 (config-if)# no shut Nexus-5548-2 (config-if)# exit Nexus-5548-2 (config)# 6. Configure load balancing. Note: According to Cisco documentation, on the N5K the default load balance mechanism on the LACP port channel is source-dest-ip. If left in this mode, all FCoE traffic will take the same link in the port channel when the N5K is forwarding FCoE frames to the N7K and MDS, as this provides SID-DID load balancing for FCoE frames. To enable the N5K to load balance using exchange ID, you must configure the N5K for source-dest-port load balancing.
  • 237. Nexus 7000, Nexus 5000, and MDS 9500 series topology 237 Nexus Series Switches Setup Examples Nexus-5548-2 (config)# port-channel load-balance source-dest-port Nexus-5548-2 (config)# 7. Configure the physical interfaces. Note: As shown in the target topology diagram, Figure 32 on page 212, there are 5 physical interfaces in this example being used for FCoE on Nexus-5548-2. Two of these physical interfaces will be added to PortChannel 401, two others will be added to PortChannel 402, and the remainder will be used for the host to attach to. Each of the PortChannels will be configured in this section. a. Configure interface Ethernet 2/1-2. Nexus-5548-2 (config)# interface fc 2/1-2 Nexus-5548-2 (config-if-range)# switchport trunk mode on Nexus-5548-2 (config-if-range)# channel-group 302 force Nexus-5548-2 (config-if-range)# no shut Nexus-5548-2 (config-if-range)# exit Nexus-5548-2 (config)# b. Configure interface Ethernet 1/3-4. Nexus-5548-2 (config)# interface ethernet 1/3-4 Nexus-5548-2 (config-if-range)# switchport mode trunk Nexus-5548-2 (config-if-range)# switchport trunk allowed vlan 200 Nexus-5548-2 (config-if-range)# channel-group 301 mode active Nexus-5548-2 (config-if-range)# no shut Nexus-5548-2 (config-if-range)# exit Nexus-5548-2 (config)# c. Configure interface Ethernet 1/10. Nexus-5548-2 (config)# interface ethernet 1/10 Nexus-5548-2 (config-if)# switchport mode trunk Nexus-5548-2 (config-if)# switchport trunk allowed vlan 200 Nexus-5548-2 (config-if)# spanning-tree port type edge trunk Nexus-5548-2 (config-if)# spanning-tree bpduguard enable Nexus-5548-2 (config-if)# no shut Nexus-5548-2 (config-if)# exit Nexus-5548-2 (config)# 8. Create the VFC interfaces. Note: As shown in the target topology diagram, Figure 32 on page 212, there are 5 physical interfaces in this example being used for FCoE on Nexus-5548-2. Two of these physical interfaces will be added to PortChannel 301, two others will be added to PortChannel 302, and the remainder will be used for the host to attach to. Each of the PortChannels will be configured in this section.
  • 238. 238 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples a. Configure interface vfc-port-channel 301. Nexus-5548-2 (config)# interface vfc-port-channel 301 Nexus-5548-2 (config-if)# switchport mode np Nexus-5548-2 (config-if)# switchport trunk allowed vsan 200 Nexus-5548-2 (config-if)# no shut b. Configure interface vfc-port-channel 302. Nexus-5548-2 (config)# interface vfc-port-channel 302 Nexus-5548-2 (config-if)# switchport mode np Nexus-5548-2 (config-if)# switchport trunk allowed vsan 200 Nexus-5548-2 (config-if)# no shut c. Configure the interface for the storage port on vfc 10. Nexus-5548-2 (config)# interface vfc 10 Nexus-5548-2 (config-if)# switchport mode f Nexus-5548-2 (config-if)# switchport trunk allowed vsan 200 Nexus-5548-2 (config-if)# no shut Configuring the MDS 9513 This section contains the following information: ◆ “Configuring MDS-9513-1” on page 238 ◆ “Configuring MDS-9513-2” on page 240 Configuring MDS-9513-1 According to the Cisco documentation, there is no need to enable FCoE explicitly on the MDS switch. The following features will be enabled once an FCoE capable line card is detected: ◆ install feature-set fcoe ◆ feature-set fcoe ◆ feature lldp ◆ feature vlan-vsan-mapping You do need to enable LACP as you will be building an LACP port channel. 1. Enable the LACP feature. MDS-9513-1# config t MDS-9513-1(config) # feature lacp 2. Set the appropriate system QoS settings. MDS-9513-1(config) # system qos MDS-9513-1(config-sys-qos) # service-policy type network-qos default-nq-7e-policy
  • 239. Nexus 7000, Nexus 5000, and MDS 9500 series topology 239 Nexus Series Switches Setup Examples 3. Create the required VLAN and VSAN. MDS-9513-1(config) # vsan database MDS-9513-1(config-vsan-db) # vsan 100 MDS-9513-1(config-vsan-db) # vlan 100 MDS-9513-1(config-vlan) # fcoe vsan 100 MDS-9513-1(config-vlan) # exit MDS-9513-1(config) # 4. Create PortChannels. a. Create PortChannel 402. MDS-9513-1(config) # interface ethernet 1/1-2 MDS-9513-1(config-if-range) # switchport mode trunk MDS-9513-1(config-if-range) # switchport trunk allowed vlan 100 MDS-9513-1(config-if-range) # channel-group 402 force mode active MDS-9513-1(config-if-range) # no shut MDS-9513-1(config-if-range) # interface ethernet-port-channel 402 MDS-9513-1(config-if) # no shut MDS-9513-1(config-if) # int vfc-port-channel 402 MDS-9513-1(config-if) # switchport mode e MDS-9513-1(config-if) # switchport mode trunk allowed vsan 100 MDS-9513-1(config-if) # no shut MDS-9513-1(config-if) # end MDS-9513-1 # b. Create PortChannel 403. MDS-9513-1(config) # interface ethernet 1/3-4 MDS-9513-1(config-if-range) # switchport mode trunk MDS-9513-1(config-if-range) # switchport trunk allowed vlan 100 MDS-9513-1(config-if-range) # channel-group 403 force mode active MDS-9513-1(config-if-range) # no shut MDS-9513-1(config-if-range) # interface ethernet-port-channel 403 MDS-9513-1(config-if) # no shut MDS-9513-1(config-if) # int vfc-port-channel 403 MDS-9513-1(config-if) # switchport mode e MDS-9513-1(config-if) # switchport mode trunk allowed vsan 100 MDS-9513-1(config-if) # no shut MDS-9513-1(config-if) # exit MDS-9513-1 (config) # 5. Create the vfc for the FCoE storage port to connect to on interface 1/5. MDS-9513-1(config) # interface vfc 5 MDS-9513-1(config-if) # bind interface ethernet 1/5 MDS-9513-1(config-if) # no shut 6. Add the newly created vfc to the vsan database. MDS-9513-1(config) # vsan database MDS-9513-1(config-vsan-db) # vsan 100 interface vfc 5
  • 240. 240 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples MDS-9513-1(config-vsan-db) # exit MDS-9513-1(config) # 7. Configure the physical interface for the storage port vfc. MDS-9513-1(config) # interface e1/5 MDS-9513-1(config-if) # switchport mode trunk MDS-9513-1(config-if) # switchport trunk allowed vlan 100 MDS-9513-1(config-if) # no shut Configuring MDS-9513-2 According to the Cisco documentation, there is no need to enable FCoE explicitly on the MDS switch. The following features will be enabled once an FCoE capable line card is detected: ◆ install feature-set fcoe ◆ feature-set fcoe ◆ feature lldp ◆ feature vlan-vsan-mapping You do need to enable LACP as you will be building an LACP PortChannel. 1. Enable the LACP and NPIV features. MDS-9513-2# config t MDS-9513-2(config) # feature lacp MDS-9513-2(config) # feature npiv 2. Set the appropriate system QoS settings. MDS-9513-2(config) # system qos MDS-9513-2(config-sys-qos) # service-policy type network-qos default-nq-7e-policy 3. Create the required VLAN and VSAN. MDS-9513-2(config) # vsan database MDS-9513-2(config-vsan-db) # vsan 200 MDS-9513-2(config-vsan-db) # vlan 200 MDS-9513-2(config-vlan) # fcoe vsan 200 MDS-9513-2(config-vlan) # exit MDS-9513-2(config) # 4. Create PortChannels. a. Create PortChannel 302. Note: PortChannel 302 will make use of FC ports 2/1-2. MDS-9513-2(config) # interface ethernet 2/1-2 MDS-9513-2(config-if-range) # switchport mode f
  • 241. Nexus 7000, Nexus 5000, and MDS 9500 series topology 241 Nexus Series Switches Setup Examples MDS-9513-2(config-if-range) # switchport rate-mode dedicated MDS-9513-2(config-if-range) # channel-group 302 force MDS-9513-2(config-if-range) # no shut MDS-9513-2(config-if-range) # interface ethernet-port-channel 302 MDS-9513-2(config-if) # no shut MDS-9513-2(config-if) # int vfc-port-channel 302 MDS-9513-2(config-if) # switchport mode f MDS-9513-2(config-if) # switchport mode trunk allowed vsan 200 MDS-9513-2(config-if) # no shut MDS-9513-2(config-if) # end MDS-9513-2 # b. Create PortChannel 303. MDS-9513-2(config) # interface ethernet 1/3-4 MDS-9513-2(config-if-range) # switchport mode trunk MDS-9513-2(config-if-range) # switchport trunk allowed vlan 200 MDS-9513-2(config-if-range) # channel-group 303 force mode active MDS-9513-2(config-if-range) # no shut MDS-9513-2(config-if-range) # interface ethernet-port-channel 303 MDS-9513-2(config-if) # no shut MDS-9513-2(config-if) # int vfc-port-channel 303 MDS-9513-2(config-if) # switchport mode e MDS-9513-2(config-if) # switchport mode trunk allowed vsan 200 MDS-9513-2(config-if) # no shut MDS-9513-2(config-if) # exit MDS-9513-2 (config) # 5. Create the vfc for the FCoE storage port to connect to on interface 1/5. MDS-9513-2(config) # interface vfc 5 MDS-9513-2(config-if) # bind interface ethernet 1/5 MDS-9513-2(config-if) # no shut 6. Add the newly created vfc to the vsan database. MDS-9513-2(config) # vsan database MDS-9513-2(config-vsan-db) # vsan 200 interface vfc 5 MDS-9513-2(config-vsan-db) # exit MDS-9513-2(config) # 7. Configure the physical interface for the storage port vfc. MDS-9513-2(config) # interface e1/5 MDS-9513-2(config-if) # switchport mode trunk MDS-9513-2(config-if) # switchport trunk allowed vlan 200 MDS-9513-2(config-if) # no shut
  • 242. 242 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Nexus 5000/Nexus 2232PP topology This section details the setup steps required to create the configuration shown in Figure 34 on page 244, the target topology. It includes the following information: ◆ “Summary of configuration steps” on page 242 ◆ “Existing topology example” on page 243 ◆ “Target topology” on page 244 ◆ “Prerequisites” on page 245 ◆ “Configuring the Nexus 5000” on page 246 Summary of configuration steps Nexus-5020-1 This section provides a summary and examples of the steps to create the configuration shown in the target topology, as shown in Figure 34 on page 244. 1. Configure the switch management interface, basic settings and password. 2. Enable the FCoE, LACP, FEX, LLDP and NPV (if applicable) features. 3. Create the appropriate VSANs. 4. Create the appropriate VLANs. 5. Create and configure a FEX instance. 6. Create and configure the FEX EtherChannel. 7. Create VFC interfaces and assign to the appropriate VSAN. 8. Configure Ethernet uplink interface settings. 9. Add FC interfaces to the appropriate VSAN. 10. Connect cables and verify login. 11. Save the configuration. 12. Perform fabric zoning. 13. Provision storage.
  • 243. Nexus 5000/Nexus 2232PP topology 243 Nexus Series Switches Setup Examples Existing topology example It is assumed that the customer has an existing LAN and mirrored SAN as shown in Figure 33 and is adding the Nexus 5000/Nexus 2232 to both. Figure 33 Existing topology example Core-N7K-1 CNA_1 Host_1 (Windows) CNA_1: Emulex VSAN = 100 WWPN= 10:00:00:00:c9:3c:f8:3c Storage = Storage port 1 CNA_2: Emulex VSAN = 200 WWPN = 10:00:00:00:c9:3c:f7:c0 Storage = Storage port 3 Storage port 4: 50:06:04:82:D5:2E:69:59 VSAN = 400 Storage port 3: 50:06:04:8A:D5:2E:69:79 VSAN = 200 Storage port 2: 50:06:04:82:Da5:2E:69:49 Storage port 1: 50:06:04:82:D5:2E:69:79 CNA_2 Core-N7K-2 1/1 2/1 1/1 2/1 1/2 1/3 1/4 1/5 Storage Fabric BFabric A 1 2 3 4 1/2 1/2 1/3 1/4 1/5 1/2 1/2 1/1 1/3 1/4 1/5 1/2 1/1 1/3 1/4 1/5 CNA_1 Host_1 (Linux) VSAN 100 VSAN 300 VSAN 200 VSAN 400 4Gb/s FC CNA_1: QLogic VSAN = 300 WWPN = 21:01:00:1b:32:2a:c0:b8 Storage = Storage port 2 CNA_2: QLogic VSAN = 400 WWPN = 21:00:00:1b:32:0a:c0:b8 Storage = Storage port 4 CNA_2 DCX_1 Domain 1 172.23.185.24 DCX_2 Domain 2 172.23.185.25 MDS-9513-1 VSAN 400 Domain 1 172.23.185.22 MDS-9513-2 VSAN 200 Domain 2 172.23.185.23
  • 244. 244 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Target topology Figure 34 shows the target topology. Figure 34 Target topology example Host_1 (Windows) CNA_1 CNA_2 DCX_1 Domain 1 172.23.185.24 DCX_2 Domain 2 172.23.185.25 MDS-9513-1 VSAN 400 Domain 1 172.23.185.22 MDS-9513-2 VSAN 200 Domain 2 172.23.185.23 Storage SAN BSAN A Nexus-5020-1 IP = 172.23.185.112 Domain = N/A (NPV mode) Nexus-5020-2 IP = 172.23.185.113 Domain = 3 1/2 1/3 1/4 1/5 1/2 1/3 1/4 1/5 1/2 1/3 1/4 1/5 1/2 1/3 1/4 1/5 1/11/1 1/11/1 Host_2 (Linux) CNA_1 CNA_2 CNA_1: QLogic VSAN = 300 WWPN = 21:01:00:1b:32:2a:c0:b8 ENode MAC = 00:1b:32:2a:c0:b8 IP Address = 10.246.53.107 Storage = Storage port 2 CNA_2: QLogic VSAN = 400 WWPN = 21:00:00:1b:32:0a:c0:b8 ENode MAC = 00:1b:32:0a:c0:b9 IP Address = 10.246.54.107 Storage = Storage port 4 1 2 3 4 Storage port 1: 50:06:04:82:D5:2E:69:79 Storage port 2: 50:06:04:82:D5:2E:69:49 Storage port 3: 50:06:04:8A:D5:2E:69:79 VSAN = 200 Storage port 4: 50:06:04:82:D5:2E:69:59 VSAN = 400 Nexus- 5020-1 Nexus- 5020-2 CNA_1: Emulex VSAN = 100 WWPN= 10:00:00:00:c9:3c:f8:3c ENode MAC = 00:00:c9:3c:f8:3c IP Address = 10.246.53.106 Storage = Storage port 1 CNA_2: Emulex VSAN = 200 WWPN = 10:00:00:00:c9:3c:f7:c0 ENode MAC = 00:00:c9:3c:f7:c0 IP Address = 10.246.54.106 Storage = Storage port 3 VLAN/VSAN 100 VLAN/VSAN 200 VLAN/VSAN 300 VLAN/VSAN 400 10GbE 4Gb/s FC Core-N7K-1 Port- channel 1 Port- channel 2 1/1 2/1 2/11/1 Core-N7K-2 SAN Port Channels Nexus-2232-1 Nexus-2232-2 1/1 1/2 1/3 1/4 1/5 1/6 1/1 1/2 1/3 1/4 1/5 1/6 Port-Channel 3 Port-Channel 4 Host 1 Host 2 PO3 PO4 FEX instance 102FEX instance 101 ICO-IMG-000850_Straight Thru
  • 245. Nexus 5000/Nexus 2232PP topology 245 Nexus Series Switches Setup Examples Prerequisites FCoE requires the use of specialized host adapters, referred to as a Converged Network Adapter (CNA), and specialized switch hardware known as a Fibre Channel Forwarder (FCF). CNAs are installed in the host and then attached to an Ethernet port that provides the FCF functionality. The connection between the CNA and Ethernet switch port can be either optical or copper based. FCF prerequisites The Nexus 5020 requires: ◆ 2 U of rack space, preferably at the top of the rack where the servers that will be connecting to the Nexus 5020 are located ◆ Two power drops of either 110 V or 220 V ◆ An IP address for management purposes and a network port to which to connect the Nexus 5020 management port ◆ A customer-supplied password for the admin user ◆ An available Domain ID in the fabric to which the Nexus 5020 will be connected Note: If the Nexus 5020 will be carved into multiple VSANs, a Domain ID will need to be reserved for each VSAN. ◆ For each Nexus 5020, at least one expansion module containing 4 G FC ports will need to be installed ◆ An FCoE license ◆ Fabric Extender Transceivers (FET) Note: FETs are optional, but are used in this example to connect each Nexus 5020 to its Nexus 2232PP. Laptop prerequisites Laptop prerequisites include: Note: The laptop used in this example is needed only to initially configure the Nexus 5020 and will be disconnected after the installation is complete. ◆ Com port with a DB-9 connector ◆ Download and install PuTTY Note: Installing PuTTY is recommended but not mandatory. ◆ Ensure that HyperTerminal is installed
  • 246. 246 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Note: The operating system used for this example was Windows 2000 Professional and HyperTerminal was installed by default. Some of the later Windows-based operating systems do not have HyperTerminal installed. This will need to be done before attempting to configure the Nexus 5020s. See the help feature for the particular version of Windows you are using for more information on installing HyperTerminal. Configuring the Nexus 5000 This section contains the following information: ◆ “Configuring Nexus-5020-1” on page 246 ◆ “Configuring Nexus-5020-2” on page 259 ◆ “Configuring the Windows host” on page 270 ◆ “Configuring the Linux host” on page 271 ◆ “Configuring the VMware host” on page 271 ◆ “Configuring EMC storage” on page 271 ◆ “Configuring network parameters” on page 271 Configuring Nexus-5020-1 Before configuring the Nexus-5020-1, consider the following: The configuration of Nexus-5020-1 will be slightly different from Nexus-5020-2 for the following reasons: ◆ Nexus-5020-1 will be attached to a Brocade fabric consisting of two Brocade ED-DCX-B switches. Since the Nexus 5020 does not currently support FC-SW interop to an EMC ED-DCX-B, Nexus-5020-1 will need to run in NPV mode. ◆ NPV mode allows the Nexus 5020 to act as an N_Port and pass login requests directly from the CNA to the Fabric by using the NPIV protocol. For more information on NPV and NPIV, refer to the “NPIV” section in the Networked Storage Concepts and Protocols TechBook at http://elabnavigator.EMC.com, under the Topology Resource Center tab. Further information on how to configure NPV mode and the supported features with NPV is available at http://www.cisco.com.
  • 247. Nexus 5000/Nexus 2232PP topology 247 Nexus Series Switches Setup Examples ◆ Since Nexus-5020-1 will be connected to two different ED-DCX-Bs, VSAN assignments to the Nexus 5020 VFC and FC interfaces will be used to allow the CNA to attach to the proper fabric. To configure Nexus-5020-1: 1. Connect to the console port of Nexus-5020-1. a. Attach the DB-9 connector of the serial cable to COM port 1 on the laptop. b. Attach the RJ-45 connector of the serial cable to the console port on the Nexus-5020-1. c. Select Start > Programs > Accessories > Communications > Hyperterminal. The new connection dialog box is opened with the entry name dialog box highlighted. d. Enter a name such as Nexus, and then click OK. The Connect to dialog box displays. e. Select COM 1 in the Connect using pull-down list, and then click OK. The COM1 Properties dialog box displays. f. In the COM 1 Properties dialog box, use the following settings: – Bits per second: = 9600 – Data bits: = 8 – Parity: = none – Stop bits: = 1 – Flow control: = None g. Click OK. You will be connected and should see the following displayed: Do you want to enforce secure password standard (yes/no): Y Note: The password should contain characters from at least three of the following classes: lowercase letters, uppercase letters, digits, and special characters.
  • 248. 248 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples h. Enter and then confirm the customer-supplied password for admin, which in this case is Seabiscuit!. Note: It may be helpful to log the output by using the capture text feature under the Transfer: pull-down menu. Enter the password for "admin": Confirm the password for "admin": 2. Complete the basic configuration: a. Answer the prompts as follows: Would you like to enter the basic configuration dialog (yes/no):Y Note: If the installer does not use the install startup script, management vrf requires manual installation. Create another login account (yes/no) [n]: N Configure read-only SNMP community string (yes/no) [n]: N Configure read-write SNMP community string (yes/no) [n]: N Enter the switch name: Nexus-5020-1 Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: Y Mgmt0 IPv4 address: 172.23.185.112 Mgmt0 IPv4 netmask: 255.255.255.0 Configure the default gateway? (yes/no) [y]: Y IPv4 address of the default gateway: 172.23.185.2 Enable the telnet service? (yes/no) [y]: Y Enable the http-server? (yes/no) [y]: Y Enable the ssh service? (yes/no) [n]: Y Type of ssh key you would like to generate (dsa/rsa): dsa Number of key bits <768-2048>: 1024 Configure the ntp server? (yes/no) [n]: N Enter basic FC configurations (yes/no) [n]: Y Configure default physical FC switchport interface state (shut/noshut) [shut]: shut Configure default physical FC switchport trunk mode (on/off/auto) [on]: on Configure default zone policy (permit/deny) [deny]: deny Enable full zoneset distribution? (yes/no) [n]: y A summary of the configuration displays: The following configuration will be applied: switchname Nexus-5020-1 interface mgmt0 ip address 172.23.185.112 255.255.255.0 no shutdown exit vrf context management
  • 249. Nexus 5000/Nexus 2232PP topology 249 Nexus Series Switches Setup Examples ip route 0.0.0.0/0 172.23.185.2 exit telnet server enable ssh key dsa 1024 force ssh server enable b. Answer the next questions as follows: Would you like to edit the configuration? (yes/no) [n]:N Use this configuration and save it? (yes/no) [y]: Y The configuration will be saved. c. Once the configuration is completed, you will be prompted to log in as follows: Nexus-5020-1 login: admin Password: Seabiscuit! d. Connect the network management cable. e. Use the show int brief command to show the interfaces on the switch. You will notice between 40 and 52 Ethernet interfaces as well as between 4 and 16 FC interfaces if the GEM module contains FC ports. Note: If no FC interfaces are displayed and one or both of the GEMs contain FC ports, ensure that the FCoE feature has been enabled using the feature FCoE command. Nexus-5020-1# show int brief ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- fc2/1 1 auto on down swl -- -- fc2/2 1 auto on down swl -- -- fc2/3 1 auto on down swl -- -- . . . fc3/4 1 auto on down swl -- -- -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth1/1 1 eth access down Link not connected 10G(D) -- Eth1/2 1 eth access down Link not connected 10G(D) --
  • 250. 250 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Eth1/3 1 eth access down Link not connected 10G(D) -- . . . Eth1/40 1 eth access down Link not connected 10G(D) -- -------------------------------------------------------------------------------- Port VRF Status IP Address Speed MTU -------------------------------------------------------------------------------- mgmt0 -- up 172.23.185.112 1000 1500 Nexus-5020-1# f. Enable NPV mode. Although NPV mode is supported when connecting to either Connectrix MDS or non-Connectrix MDS switches, it only needs to be enabled when connecting a Nexus 5020 to a non-Connectrix MDS switch, as is the case with this example. If you are connecting the Nexus 5020 to a Connectrix MDS switch, and do not want to use NPV mode, see “Configuring Nexus-5020-2” on page 191. Nexus-5020-1(config)# npv enable Verify that boot variables are set and the changes are saved. Changing to npv mode erases the current configuration and reboots the switch in npv mode. Do you want to continue? (y/n): Y Note: It is recommended that you leave the console connection up while the reboot is taking place. g. Open an SSH session to the Nexus 5020 with an IP address of 172.23.185.112. 3. Enable disruptive load-balancing. Disruptive load-balancing allows the Nexus to evenly distribute CNA logins across all NP ports while running in NPV mode. To enable disruptive load-balancing, use the npv auto-load-balance disruptive command, as shown next. Nexus-5020-1# npv auto-load-balance disruptive Enabling this feature may flap the server interfaces whenever load is not in a balanced state. This process may result in traffic disruption. Do you want to proceed? (y/n): Y Nexus-5020-1(config)#
  • 251. Nexus 5000/Nexus 2232PP topology 251 Nexus Series Switches Setup Examples 4. If you are configuring a Nexus 5548 or Nexus 5596, you must manually enable the QoS class-fcoe. To do this, attach the following QoS policies under system qos: switch (config-sys-qos)# service-policy type qos input fcoe-default-in-policy switch (config-sys-qos)# service-policy type queuing input fcoe-default-in-policy switch (config-sys-qos)# service-policy type queuing output fcoe-default-out-policy switch (config-sys-qos)# service-policy type network-qos fcoe-default-nq-policy 5. Create the VSANs. Nexus-5020-1(config)# vsan database Nexus-5020-1(config-vsan-db)# vsan 100 Nexus-5020-1(config-vsan-db)# vsan 300 Nexus-5020-1(config-vsan-db)# exit Nexus-5020-1(config)# 6. Create a VLAN for FCoE Traffic and associate with a VSAN. Note: Due to the one-to-one relationship between a VSAN and VLAN, if multiple VSANs will be configured (as is the case in this example) on the Nexus 5020, then multiple FCoE VLANs will also need to be configured. Nexus-5020-1(config)# vlan 100 Nexus-5020-1(config-vlan)# fcoe vsan 100 Nexus-5020-1(config-vlan)# vlan 300 Nexus-5020-1(config-vlan)# fcoe vsan 300 7. Create the VLAN that will be used as the default VLAN on the FCoE interfaces. Nexus-5020-1(config-vlan)# vlan 700 Note: This step is only required if you are changing the default VLAN on ethernet interface. In later examples, VLAN 700 will be used for iSCSI. 8. Use show FCoE VLAN to verify whether the configuration was successful. Nexus-5020-1(config-vlan)# show vlan fcoe VLAN VSAN Status -------- -------- -------- 100 100 Operational 300 300 Operational
  • 252. 252 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 9. Enable the Fabric Extender feature. Nexus-5020-1(config-vlan)# feature fex 10. Create and configure a FEX instance. Note: The pinning max-links parameter does not need to be set since the Nexus-2232PP will be connected to Nexus-5020-1 via an EtherChannel. Nexus-5020-1(config)# fex 101 Nexus-5020-1(config-fex)# description FEX0101 11. Create an EtherChannel that will be used to connect to Nexus-2232-1. Nexus-5020-1(config-fex)# interface port-channel 3 12. Configure the EtherChannel as a FEX fabric interface. Nexus-5020-1(config-if)# switchport mode fex-fabric 13. Associate the FEX instance with the EtherChannel. Nexus-5020-1(config-if)# fex associate 101 14. Configure and associate the PortChannel with the appropriate Ethernet interfaces. Nexus-5020-1(config-if)# interface ethernet 1/1-4 Nexus-5020-1(config-if)# switchport mode fex-fabric Nexus-5020-1(config-if)# fex associate 101 Nexus-5020-1(config-if-range)# channel-group 3 15. Using any of the supported media types, connect Nexus-5020-1 to Nexus 2232-1 as shown in the target topology diagram shown in Figure 34 on page 244. 16. Using any of the supported media types, connect Host_1 / CNA_1 and Host_2 / CNA_1 to Nexus 2232-1 as shown in the target topology diagram shown in Figure 34 on page 244. 17. Display the switch interfaces using show interface brief: Nexus-5020-1(config-if-range)# show int brief ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- fc2/1 1 NP off down swl -- -- fc2/2 1 NP off down swl -- --
  • 253. Nexus 5000/Nexus 2232PP topology 253 Nexus Series Switches Setup Examples fc2/3 1 NP off down swl -- -- . . . fc3/4 1 NP off down swl -- -- -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth1/1 1 eth fabric up none 10G(D) 3 Eth1/2 1 eth fabric up none 10G(D) 3 Eth1/3 1 eth fabric up none 10G(D) 3 Eth1/4 1 eth fabric up none 10G(D) 3 Eth1/5 1 eth access down Link not connected 10G(D) -- . . . Eth1/40 1 eth access down Link not connected 10G(D) -- -------------------------------------------------------------------------------- Port-channel VLAN Type Mode Status Reason Speed Protocol Interface -------------------------------------------------------------------------------- Po3 1 eth fabric up none a-10G(D) none -------------------------------------------------------------------------------- Port VRF Status IP Address Speed MTU -------------------------------------------------------------------------------- mgmt0 -- up 172.23.185.112 1000 1500 -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth101/1/1 1 eth access up none 10G(D) -- Eth101/1/2 1 eth access up none 10G(D) -- . . . Eth101/1/32 1 eth access down SFP not inserted 10G(D) -- Nexus-5020-1(config-if-range)# 18. Display the EtherChannel interface. Nexus-5020-1(config-if)# show interface port-channel 3 fex-intf Fabric FEX Interface Interfaces --------------------------------------------------------------------- Po3 Eth101/1/1 Eth101/1/2 Eth101/1/3 Eth101/1/4
  • 254. 254 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 19. Create the VFC interfaces and bind them to an interface on the 2232PP. Nexus-5020-1(config-if-range)# interface vfc 1011 Nexus-5020-1(config-if)# bind interface ethernet 101/1/1 Nexus-5020-1(config-if)# switchport trunk allowed vsan 100 Nexus-5020-1(config-if)# no shut Nexus-5020-1(config-if)# interface vfc 1012 Nexus-5020-1(config-if)# bind interface ethernet 101/1/2 Nexus-5020-1(config-if)# switchport trunk allowed vsan 300 Nexus-5020-1(config-if)# no shut 20. Assign each VFC to the appropriate VSAN as shown in the target topology diagram, Figure 34 on page 244. Nexus-5020-1(config-if)# vsan database Nexus-5020-1(config-vsan-db)# vsan 100 interface vfc1011 Nexus-5020-1(config-vsan-db)# vsan 300 interface vfc1012 Nexus-5020-1(config-vsan-db)# exit Nexus-5020-1(config)# The output of the show interface brief shows the following: Nexus-5020-1(config)# show int brief ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- fc2/1 1 NP off down swl -- -- fc2/2 1 NP off down swl -- -- . . . fc3/4 1 NP off down swl -- -- -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth1/1 1 eth fabric up none 10G(D) 3 Eth1/2 1 eth fabric up none 10G(D) 3 Eth1/3 1 eth fabric up none 10G(D) 3 Eth1/4 1 eth fabric up none 10G(D) 3 . . . Eth1/40 1 eth access down Link not connected 10G(D) -- --------------------------------------------------------------------------------
  • 255. Nexus 5000/Nexus 2232PP topology 255 Nexus Series Switches Setup Examples Port-channel VLAN Type Mode Status Reason Speed Protocol Interface -------------------------------------------------------------------------------- Po3 1 eth fabric up none a-10G(D) none -------------------------------------------------------------------------------- Port VRF Status IP Address Speed MTU -------------------------------------------------------------------------------- mgmt0 -- up 172.22.185.112 1000 1500 ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- vfc1011 100 F off down -- -- -- vfc1012 300 F off down -- -- -- -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth101/1/1 1 eth access up none 10G(D) -- Eth101/1/2 1 eth access up none 10G(D) -- Eth101/1/3 1 eth access up none 10G(D) -- Eth101/1/4 1 eth access up none 10G(D) -- Eth101/1/5 1 eth access down SFP not inserted 10G(D) -- . . . Eth101/1/32 1 eth access down SFP not inserted 10G(D) -- Nexus-5020-1(config)# 21. Configure the Ethernet port settings. Each interface will be set to trunk mode to allow it to handle multiple VLANs and 802.1q tagged frames. The native VLAN is set to 700 in this case to allow for all non-FCoE untagged frames to be forwarded onto VLAN 700. The allowed VLAN list is set to allow the default VLAN as well as the FCoE VLAN. Nexus-5020-1(config-if)# interface ethernet 101/1/1 Nexus-5020-1(config-if)# switchport mode trunk Nexus-5020-1(config-if)# switchport trunk native vlan 700 Nexus-5020-1(config-if)# switchport trunk allowed vlan 100,700 Nexus-5020-1(config-if)# spanning-tree port type edge trunk Nexus-5020-1(config-if)# interface ethernet 101/1/2 Nexus-5020-1(config-if)# switchport mode trunk Nexus-5020-1(config-if)# switchport trunk native vlan 700 Nexus-5020-1(config-if)# switchport trunk allowed vlan 300,700 Nexus-5020-1(config-if)# spanning-tree port type edge trunk
  • 256. 256 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 22. Add FC interfaces to the proper VSANs. As shown in Figure 29 on page 176, Nexus-5020-1 is connected to two different fabrics. In order to ensure that each CNA attached to Nexus-5020-1 logs in to the same fabric as its storage port, each CNA will need to be statically mapped to a particular VSAN. This is accomplished by adding the VFC Interface as well as the physical FC interfaces to the appropriate VSAN. To summarize the steps: a. Enter the vsan database configuration mode by using vsan database. b. Add the fc interfaces on 5020_1 to the appropriate VSAN using vsan <VSAN ID> interface vfc <vfc number>. Nexus-5020-1(config-if)# vsan database Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 2/1 Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 2/3 Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 2/2 Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 2/4 Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 3/1 Nexus-5020-1(config-vsan-db)# vsan 100 interface fc 3/3 Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 3/2 Nexus-5020-1(config-vsan-db)# vsan 300 interface fc 3/4 Nexus-5020-1(config-vsan-db)# exit Nexus-5020-1(config)# c. Enable all of the fc interfaces by specifying the interface by using the interface <interface number> command, and then the no shut command. Nexus-5020-1(config)# interface fc 2/1-4 Nexus-5020-1(config-if)# no shut Nexus-5020-1(config)# interface fc 3/1-4 Nexus-5020-1(config-if)# no shut 23. Attach the fiber cables between Nexus-5020-1 and the Brocade DCX Directors as shown in the target topology diagram in Figure 34 on page 244. 24. Validate the configuration by using the show int brief command. The following is a partial output of this command.
  • 257. Nexus 5000/Nexus 2232PP topology 257 Nexus Series Switches Setup Examples Nexus-5020-1(config)# show int brief ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ----------------------------------------------------------------------------- fc2/1 100 NP -- down swl -- -- fc2/2 300 NP -- down swl -- -- fc2/3 100 NP -- down swl -- -- fc2/4 300 NP -- down swl -- -- fc3/1 100 NP -- down swl -- -- fc3/2 300 NP -- down swl -- -- fc3/3 100 NP -- down swl -- -- fc3/4 300 NP -- down swl -- -- . . . ----------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gb/s) ----------------------------------------------------------------------------- vfc1 100 F -- down -- -- -- vfc2 300 F -- down -- -- -- Nexus-5020-1(config-if)# 25. Use show npv flogi-table vsan <vsan ID> to verify that each CNA is logged in to the appropriate vsan. Nexus-5020-1(config-if)# show npv flogi-table vsan 100 ----------------------------------------------------------------------------- SERVER EXTERNAL INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE ----------------------------------------------------------------------------- vfc1 100 0x680101 10:00:00:00:c9:3c:f8:3c 20:00:00:00:c9:3c:f8:3c fc2/1 Total number of flogi = 1. Nexus-5020-1(config-if)# show npv flogi table vsan 300 ----------------------------------------------------------------------------- SERVER EXTERNAL INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE ----------------------------------------------------------------------------- vfc2 300 0x690101 21:01:00:1b:32:2a:c0:b8 20:01:00:1b:32:2a:c0:b8 fc2/3 Total number of flogi = 1. Nexus-5020-1(config-if)#
  • 258. 258 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples The fc interfaces are now online and running in NPV mode. The vfc interfaces come online as soon as one of the fc interfaces in the same vsan comes online. 26. Save the configuration: Nexus-5020-1(config-if) # copy running startup-config 27. Zone the CNAs. Since Nexus-5020-1 is running in NPV mode, all zoning will need to be performed from the ED-DCX-B Director to which it is attached. It is strongly recommended that if the CLI is to be used for zoning, then cut and paste be utilized to minimize transcription types of errors when manually typing the WWPNs. The set of steps required to create the appropriate zones are: a. Create each zone by using zonecreate "<zone name a>", "WWPN1; WWPN2; .... WWPNn". b. Create the zone config by using cfgcreate "<cfg name>", "<zone name a>; <zone name b>; .... <zone name n>". c. Enable the zone config by using cfgenable "<cfg name>". d. Save the configuration by using cfgsave. The exact steps that would be performed in this example are as follows: a. Zone the CNA logged in to 172.23.185.25 to access its storage port. – Telnet into the switch with an IP address of 172.23.185.24. DCX_1:admin> zonecreate “host1_CNA1”, “10:00:00:00:c9:3c:f8:3c; 50:06:04:82:D5:2E:69:79” DCX_1:admin> cfgcreate “FCoE_1”, “host1_CNA1” DCX_1:admin> cfgenable “FCoE_1” DCX_1:admin> cfgshow Default Zone: OFF Safe Zone: OFF Defined configuration: Effective configuration: cfg: FCoE_1 zone: host1_CNA1 10:00:00:00:c9:3c:f8:3c 50:06:04:82:D5:2E:69:79 DCX_1:admin> – Log out of the switch with an IP address of 172.23.185.24.
  • 259. Nexus 5000/Nexus 2232PP topology 259 Nexus Series Switches Setup Examples b. Zone the CNA logged in to 172.23.185.25 to access its Storage ports. – Telnet into the switch with an IP address of 172.23.185.25. DCX_2:admin> zonecreate “host2_CNA1”, “21:01:00:1b:32:2a:c0:b8; 50:06:04:82:D5:2E:69:49” DCX_2:admin> cfgcreate “FCoE_2”, “host2_CNA1” DCX_2:admin> cfgenable “FCoE_2” DCX_2:admin> cfgshow Default Zone: OFF Safe Zone: OFF Defined configuration: Effective configuration: cfg: FCoE_2 zone: host2_CNA1 21:01:00:1b:32:2a:c0:b8 50:06:04:82:D5:2E:69:49 DCX_2:admin> – Log out of the switch with an IP address of 172.23.185.25. Configuring Nexus-5020-2 Before configuring the Nexus-5020-2, consider the following: The configuration of Nexus-5020-2 will be slightly different from Nexus-5020-1 for the following reasons: ◆ Nexus-5020-2 will be attached to a Cisco fabric consisting of two MDS-9513 switches. Since the Nexus 5020 is being connected to another Cisco switch, FC-SW mode is supported and Nexus-5020-2 will not need to run in NPV mode. ◆ Since Nexus-5020-2 will be connected to two different MDS-9513s via TE_Ports, VSAN assignments are only required on the Nexus 5020 VFC interfaces. To configure Nexus-5020-2: 1. Connect to the console port of Nexus-5020-2. a. Attach the DB-9 connector of the serial cable to COM port 1 on the laptop. b. Attach the RJ-45 connector of the serial cable to the console port on the Nexus-5020-2. c. Select Start > Programs > Accessories > Communications > Hyperterminal. The new connection dialog box is opened with the entry name dialog highlighted.
  • 260. 260 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples d. Enter a name such as Nexus, and then click OK. The Connect to dialog box displays. e. Select COM 1 in the Connect using pull-down list, and then click OK. The COM1 Properties dialog box displays. f. In the COM 1 Properties dialog box, use the following settings: – Bits per second: = 9600 – Data bits: = 8 – Parity: = none – Stop bits: = 1 – Flow control: = None g. Click OK. You will be connected and should see the following displayed: Do you want to enforce secure password standard (yes/no): Y Note: The password should contain characters from at least three of the following classes: lowercase letters, uppercase letters, digits, and special characters. h. Enter and then confirm the customer-supplied password for admin, which in this case is Seabiscuit!. Note: It may be helpful to log the output by using the capture text feature under the Transfer: pull-down menu. Enter the password for "admin": Confirm the password for "admin": 2. Complete the basic configuration: a. Answer the prompts as follows: Would you like to enter the basic configuration dialog (yes/no):Y Note: If the installer does not use the install startup script, management vrf requires manual installation. Create another login account (yes/no) [n]: N Configure read-only SNMP community string (yes/no) [n]: N Configure read-write SNMP community string (yes/no) [n]: N
  • 261. Nexus 5000/Nexus 2232PP topology 261 Nexus Series Switches Setup Examples Enter the switch name: Nexus-5020-2 Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: Y Mgmt0 IPv4 address: 172.23.185.113 Mgmt0 IPv4 netmask: 255.255.255.0 Configure the default gateway? (yes/no) [y]: Y IPv4 address of the default gateway: 172.23.185.2 Enable the telnet service? (yes/no) [y]: Y Enable the http-server? (yes/no) [y]: Y Enable the ssh service? (yes/no) [n]: Y Type of ssh key you would like to generate (dsa/rsa): dsa Number of key bits <768-2048>: 1024 Configure the ntp server? (yes/no) [n]: N 172.23.185.113 Enter basic FC configurations (yes/no) [n]: Y Configure default physical FC switchport interface state (shut/noshut) [shut]: shut Configure default physical FC switchport trunk mode (on/off/auto) [on]: on Configure default zone policy (permit/deny) [deny]: deny Enable full zoneset distribution? (yes/no) [n]: y A summary of the configuration displays: The following configuration will be applied: switchname Nexus-5020-2 interface mgmt0 ip address 172.23.185.113 255.255.255.0 no shutdown exit vrf context management ip route 0.0.0.0/0 172.23.185.2 exit telnet server enable ssh key dsa 1024 force ssh server enable b. Answer the next questions as follows: Would you like to edit the configuration? (yes/no) [n]:N Use this configuration and save it? (yes/no) [y]: Y The configuration will be saved. c. Once the configuration is completed, you will be prompted to log in as follows: Nexus-5020-2 login: admin Password: Seabiscuit! d. Connect the network management cable. e. Use the show int brief command to show the interfaces on the switch.
  • 262. 262 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples You will notice between 40 and 52 Ethernet interfaces as well as between 4 and 16 FC interfaces if the GEM module contains FC ports. Note: If no FC interfaces are displayed and one or both of the GEMs contain FC ports, ensure that the FCoE feature has been enabled using the feature FCoE command. Nexus-5020-2# show int brief ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- fc2/1 1 auto on down swl -- -- fc2/2 1 auto on down swl -- -- fc2/3 1 auto on down swl -- -- . . . fc3/4 1 auto on down swl -- -- -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth1/1 1 eth access down Link not connected 10G(D) -- Eth1/2 1 eth access down Link not connected 10G(D) -- Eth1/3 1 eth access down Link not connected 10G(D) -- . . . Eth1/40 1 eth access down Link not connected 10G(D) -- -------------------------------------------------------------------------------- Port VRF Status IP Address Speed MTU -------------------------------------------------------------------------------- mgmt0 -- up 172.23.185.113 1000 1500 Nexus-5020-2# f. Open an SSH session to the Nexus 5020 with an IP address of 172.23.185.113. 3. If you are configuring a Nexus 5548 or Nexus 5596, you must manually enable the QoS class-fcoe. To do this, attach the following QoS policies under system qos: switch (config-sys-qos)# service-policy type qos input fcoe-default-in-policy switch (config-sys-qos)# service-policy type queuing input
  • 263. Nexus 5000/Nexus 2232PP topology 263 Nexus Series Switches Setup Examples fcoe-default-in-policy switch (config-sys-qos)# service-policy type queuing output fcoe-default-out-policy switch (config-sys-qos)# service-policy type network-qos fcoe-default-nq-policy 4. Create the VSANs. Nexus-5020-2(config)# vsan database Nexus-5020-2(config-vsan-db)# vsan 200 Nexus-5020-2(config-vsan-db)# vsan 400 Nexus-5020-2(config-vsan-db)# exit Nexus-5020-2(config)# 5. Create a VLAN for FCoE Traffic and associate with a VSAN. Note: Due to the one-to-one relationship between a VSAN and VLAN, if multiple VSANs will be configured (as is the case in this example) on the Nexus 5020, then multiple FCoE VLANs will also need to be configured. Nexus-5020-2(config)# vlan 200 Nexus-5020-2(config-vlan)# fcoe vsan 200 Nexus-5020-2(config-vlan)# vlan 400 Nexus-5020-2(config-vlan)# fcoe vsan 400 6. Create the VLAN that will be used as the default VLAN on the FCoE interfaces. Nexus-5020-2(config-vlan)# vlan 701 Note: This step is only required if you are changing the default VLAN on ethernet interface. In later examples, VLAN 700 will be used for iSCSI. 7. Use show FCoE VLAN to verify the configuration was successful. Nexus-5020-2(config-vlan)# show vlan fcoe VLAN VSAN Status -------- -------- -------- 200 200 Operational 400 400 Operational 8. Enable the Fabric Extender feature. Nexus-5020-2(config-vlan)# feature fex 9. Create and configure a FEX instance.
  • 264. 264 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Note: The pinning max-links parameter does not need to be set since the Nexus-2232PP will be connected to Nexus-5020-2 via an EtherChannel. Nexus-5020-2(config)# fex 102 Nexus-5020-2(config-fex)# description FEX0102 10. Create an EtherChannel that will be used to connect to Nexus-2232-2. Nexus-5020-2(config-fex)# interface port-channnel 4 11. Configure the EtherChannel as a FEX fabric interface. Nexus-5020-2(config-if)# switchport mode fex-fabric 12. Associate the FEX instance with the EtherChannel. Nexus-5020-2(config-if)# fex associate 102 13. Configure and associate the PortChannel with the appropriate ethernet interfaces. Nexus-5020-2(config-if)# interface ethernet 1/1-4 Nexus-5020-2(config-if)# switchport mode fex-fabric Nexus-5020-2(config-if)# fex associate 102 Nexus-5020-2(config-if-range)# channel-group 4 14. Using any of the supported media types, connect Nexus-5020-2 to Nexus 2232-2 as shown in the target topology diagram shown in Figure 34 on page 244. 15. Using any of the supported media types, connect Host_1 / CNA_2 and Host_2 / CNA_2 to Nexus 2232-2 as shown in the target topology diagram shown in Figure 34 on page 244. 16. Display the switch interfaces using show interface brief: Nexus-5020-2(config-if-range)# show int brief ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- fc2/1 1 auto on down swl -- -- fc2/2 1 auto on down swl -- -- fc2/3 1 auto on swl -- -- . . . fc3/4 1 auto on down swl -- --
  • 265. Nexus 5000/Nexus 2232PP topology 265 Nexus Series Switches Setup Examples -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth1/1 1 eth fabric up none 10G(D) 4 Eth1/2 1 eth fabric up none 10G(D) 4 Eth1/3 1 eth fabric up none 10G(D) 4 Eth1/4 1 eth fabric up none 10G(D) 4 Eth1/5 1 eth access down Link not connected 10G(D) -- . . . Eth1/40 1 eth access down Link not connected 10G(D) -- -------------------------------------------------------------------------------- Port-channel VLAN Type Mode Status Reason Speed Protocol Interface -------------------------------------------------------------------------------- Po4 1 eth fabric up none a-10G(D) none -------------------------------------------------------------------------------- Port VRF Status IP Address Speed MTU -------------------------------------------------------------------------------- mgmt0 -- up 172.23.185.113 1000 1500 -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth102/1/1 1 eth access up none 10G(D) -- Eth102/1/2 1 eth access up none 10G(D) -- . . . Eth102/1/32 1 eth access down SFP not inserted 10G(D) -- Nexus-5020-2(config-if-range)# 17. Display the EtherChannel interface. Nexus-5020-2(config-if)# show interface port-channel 4 fex-intf Fabric FEX Interface Interfaces --------------------------------------------------------------------- Po4 Eth102/1/1 Eth102/1/2 Eth102/1/3 Eth102/1/4 18. Create the VFC interfaces and bind them to an interface on the 2232PP. Nexus-5020-2(config-if-range)# interface vfc 1021 Nexus-5020-2(config-if)# bind interface ethernet 102/1/1 Nexus-5020-2(config-if)# switchport trunk allowed vsan 200
  • 266. 266 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Nexus-5020-2(config-if)# no shut Nexus-5020-2(config-if)# interface vfc 1022 Nexus-5020-2(config-if)# bind interface ethernet 102/1/2 Nexus-5020-2(config-if)# switchport trunk allowed vsan 400 Nexus-5020-2(config-if)# no shut 19. Assign each VFC to the appropriate VSAN as shown in the target topology diagram, Figure 34 on page 244. Nexus-5020-2(config-if)# vsan database Nexus-5020-2(config-vsan-db)# vsan 200 interface vfc1021 Nexus-5020-2(config-vsan-db)# vsan 400 interface vfc1022 Nexus-5020-2(config-vsan-db)# exit Nexus-5020-2(config)# The output of the show interface brief shows the following: Nexus-5020-2(config)# show int brief ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- fc2/1 1 auto on down swl -- -- fc2/2 1 auto on down swl -- -- . . . fc3/4 1 auto on down swl -- -- -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth1/1 1 eth fabric up none 10G(D) 3 Eth1/2 1 eth fabric up none 10G(D) 3 Eth1/3 1 eth fabric up none 10G(D) 3 Eth1/4 1 eth fabric up none 10G(D) 3 . . . Eth1/40 1 eth access down Link not connected 10G(D) -- -------------------------------------------------------------------------------- Port-channel VLAN Type Mode Status Reason Speed Protocol Interface -------------------------------------------------------------------------------- Po3 1 eth fabric up none a-10G(D) none -------------------------------------------------------------------------------- Port VRF Status IP Address Speed MTU
  • 267. Nexus 5000/Nexus 2232PP topology 267 Nexus Series Switches Setup Examples -------------------------------------------------------------------------------- mgmt0 -- up 172.22.185.113 1000 1500 ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- vfc1021 200 F off down -- -- -- vfc1022 400 F off down -- -- -- -------------------------------------------------------------------------------- Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------- Eth102/1/1 1 eth access up none 10G(D) -- Eth102/1/2 1 eth access up none 10G(D) -- Eth102/1/3 1 eth access up none 10G(D) -- Eth102/1/4 1 eth access up none 10G(D) -- Eth102/1/5 1 eth access down SFP not inserted 10G(D) -- . . . Eth102/1/32 1 eth access down SFP not inserted 10G(D) -- Nexus-5020-2(config)# 20. Configure the Ethernet port settings. Each interface will be set to trunk mode to allow it to handle multiple VLANs and 802.1q tagged frames. The native VLAN is set to 700 in this case to allow for all non-FCoE untagged frames to be forwarded onto VLAN 700. The allowed VLAN list is set to allow the default VLAN as well as the FCoE VLAN. Nexus-5020-2(config-if)# interface ethernet 102/1/1 Nexus-5020-2(config-if)# switchport mode trunk Nexus-5020-2(config-if)# switchport trunk native vlan 700 Nexus-5020-2(config-if)# switchport trunk allowed vlan 100, 700 Nexus-5020-2(config-if)# spanning-tree port type edge trunk Nexus-5020-2(config-if)# interface ethernet 102/1/2 Nexus-5020-2(config-if)# switchport mode trunk Nexus-5020-2(config-if)# switchport trunk native vlan 700 Nexus-5020-2(config-if)# switchport trunk allowed vlan 300, 700 Nexus-5020-2(config-if)# spanning-tree port type edge trunk 21. Add FC interfaces to the proper VSANs.
  • 268. 268 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples As shown in Figure 29 on page 176, Nexus-5020-2 is connected to two different fabrics. In order to ensure that each CNA attached to Nexus-5020-2 logs in to the same fabric as its storage port, each CNA will need to be statically mapped to a particular VSAN. This is accomplished by adding the VFC Interface to the appropriate VSAN. To summarize the steps: a. Enter the vsan database configuration mode by using vsan database. b. Add the fc interfaces on 5020_1 to the appropriate VSAN using vsan <VSAN ID> interface vfc <vfc number>. Nexus-5020-2(config-if)# vsan database Nexus-5020-2(config-vsan-db)# vsan 200 interface fc 2/1 Nexus-5020-2(config-vsan-db)# vsan 200 interface fc 2/3 Nexus-5020-2(config-vsan-db)# vsan 400 interface fc 2/2 Nexus-5020-2(config-vsan-db)# vsan 400 interface fc 2/4 Nexus-5020-2(config-vsan-db)# vsan 200 interface fc 3/1 Nexus-5020-2(config-vsan-db)# vsan 200 interface fc 3/3 Nexus-5020-2(config-vsan-db)# vsan 400 interface fc 3/2 Nexus-5020-2(config-vsan-db)# vsan 400 interface fc 3/4 Nexus-5020-2(config-vsan-db)# exit Nexus-5020-2(config)# c. Enable all of the fc interfaces by specifying the interface by using the interface <interface number> command, and then the no shut command. Nexus-5020-2(config)# interface fc 2/1-4 Nexus-5020-2(config-if)# no shut Nexus-5020-2(config)# interface fc 3/1-4 Nexus-5020-2(config-if)# no shut 22. Attach the fiber cables between Nexus-5020-2 and the Brocade DCX Directors as shown in the target topology diagram in Figure 34 on page 244. 23. Validate the configuration by using the show int brief command. The following is a partial output of this command. Nexus-5020-2(config-if)# show int brief ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- fc2/1 200 auto on trunking swl TE 4 -- fc2/2 400 auto on trunking swl TE 4 --
  • 269. Nexus 5000/Nexus 2232PP topology 269 Nexus Series Switches Setup Examples fc2/3 200 auto on trunking swl TE 4 -- fc2/4 400 auto on trunking swl TE 4 -- fc3/1 200 auto on trunking swl TE 4 -- fc3/2 400 auto on trunking swl TE 4 -- fc3/3 200 auto on trunking swl TE 4 -- fc3/4 400 auto on trunking swl TE 4 -- . . . ------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Mode Trunk Mode Speed Channel Mode (Gbps) ------------------------------------------------------------------------------- vfc1021 200 F on trunking -- TF auto -- vfc1022 400 F on trunking -- TF auto -- Nexus-5020-2 (config-if)# 24. Use show npv flogi-table vsan <vsan ID> to verify that each CNA is logged in to the appropriate vsan. Nexus-5020-2 (config-if)# show flogi da -------------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME -------------------------------------------------------------------------------- vfc1021 500 0x930033 10:00:00:00:c9:3c:f7:c0 20:00:00:00:c9:3c:f7:c0 vfc1022 500 0x930134 21:00:00:1b:32:0a:c0:b8 21:00:00:1b:32:0a:c0:b8 Total number of flogi = 2. Nexus-5020-2# The fc interfaces are now online and running in FC-SW mode. The vfc interfaces come online as soon as one of the fc interfaces in the same vsan comes online. 25. Save the configuration: Nexus-5020-2(config-if) # copy running startup-config 26. Zone the CNAs. Since Nexus-5020-2 is running in FC-SW mode, zoning can be performed from either the MDS-9513 Director to which it is attached or from the Nexus 5000 itself. It is strongly recommended that if the CLI is to be used for zoning, then cut and paste be utilized to minimize transcription types of errors when manually typing in the WWPNs.
  • 270. 270 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples The set of steps required to create the appropriate zones are: a. zone name <ZONE_NAME> vsan <VSAN ID>. b. member pwwn <PWWN_1>. c. member pwwn <PWWN_2>. d. zone set name <ZONE_SET_NAME> vsan <VSAN ID>. e. member <ZONE_NAME>. f. zoneset activate name <ZONE_SET_NAME> vsan <VSAN ID>. g. Copy running startup. 27. Telnet into Nexus-5020-2 by completing the following steps: Create zones for the VSAN 200: Nexus-5020-2# config terminal Nexus-5020-2(config)# zone name host1_CNA2 vsan 200 Nexus-5020-2(config-zone)# member pwwn 10:00:00:00:c9:3c:f7:c0 Nexus-5020-2(config-zone)# member pwwn 50:06:04:8A:D5:2E:69:79 Nexus-5020-2(config-zone)# exit Nexus-5020-2(config)# zoneset name FCoE_200 vsan 200 Nexus-5020-2(config-zoneset)# member host1_CNA2 Nexus-5020-2(config-zoneset)# zoneset activate name FCoE_200 vsan 200 Zoneset activation initiated. check zone status Nexus-5020-2(config)# zoneset distribute full vsan 200 Nexus-5020-2(config)# show zoneset active vsan 200 zoneset name FCoE_200 vsan 200 zone name host1_CNA2 vsan 200 * fcid 0xd00000 [pwwn 10:00:00:00:c9:3c:f7:c0] * fcid 0x6f0000 [pwwn 50:06:04:8a:d5:2e:69:79] Nexus-5020-2(config)# Create zones for the VSAN 400: Nexus-5020-2(config)# zone name host2_CNA2 vsan 400 Nexus-5020-2(config-zone)# member pwwn 21:00:00:1B:32:0A:C0:B8 Nexus-5020-2(config-zone)# member pwwn 50:06:04:82:D5:2E:69:59 Nexus-5020-2(config-zone)# exit Nexus-5020-2(config)# zoneset name FCoE_400 vsan 400 Nexus-5020-2(config-zoneset)# member host2_CNA2 Nexus-5020-2(config-zoneset)# zoneset activate name FCoE_400 vsan 400 Zoneset activation initiated. check zone status Nexus-5020-2(config)# zoneset distribute full vsan 400 Nexus-5020-2(config)# Log out of the switch with an IP address of 172.23.185.25. Configuring the Windows host To configure the Windows host, refer to the following documentation, available at EMC Online Support at https://support.emc.com:
  • 271. Nexus 5000/Nexus 2232PP topology 271 Nexus Series Switches Setup Examples ◆ QLogic: EMC Host Connectivity with QLogic Fibre Channel and iSCSI Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) in the Windows Environment ◆ Emulex: EMC Host Connectivity with Emulex Fibre Channel Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) in the Windows Environment ◆ EMC Host Connectivity Guide for Windows Configuring the Linux host To configure the Linux host, refer to the following documentation, available at EMC Online Support at https://support.emc.com: ◆ QLogic: EMC Host Connectivity with QLogic Fibre Channel and iSCSI Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) for the Linux Environment ◆ Emulex: EMC Host Connectivity with Emulex Fibre Channel Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs) for the Linux v2.6.x Kernel Environment and the v8.x-Series Driver ◆ EMC Host Connectivity Guide for Linux Configuring the VMware host To configure the VMware host, refer to the following document available at EMC Online Support at https://support.emc.com: ◆ EMC Host Connectivity Guide for VMware ESX Server Configuring EMC storage Once the switch, hosts, and adapters have been configured, the storage will need to be configured to each initiator to access a number of LUNs. Configuring network parameters Using the network topology shown in Figure 31 on page 180, display configuration parameters and then follow the steps needed to deploy this Layer 2 network.
  • 272. 272 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Configuring Cisco VE_Ports VE_Ports allow for two Cisco Nexus 5000 products to be connected together and merged into the same fabric using FCoE links. VE_Ports can be created as either individual links or as shown in Figure 35, configured to use a PortChannel. The topology used in this example is shown in Figure 35. Figure 35 VE_Ports topology Note: To complete the following steps, feature lacp must be enabled. Setup steps On Nexus 5000 A: 1. Create the appropriate VSANs. NEXUS5000A# config t Enter configuration commands, one per line. End with CNTL/Z. IOP054071(config)# vsan database IOP054071(config-vsan-db)# vsan 200 IOP054071(config-vsan-db)# vsan 300 IOP054071(config-vsan-db)# vsan 400 IOP054071(config-vsan-db)# exit IOP054071(config)# 2. Create the appropriate VLAN and associate them with the appropriate VSAN. IOP054071(config)# vlan 200 IOP054071(config-vlan)# fcoe vsan 200 IOP054071(config-vlan)# vlan 300 IOP054071(config-vlan)# fcoe vsan 300 IOP054071(config-vlan)# vlan 400 IOP054071(config-vlan)# fcoe vsan 400 IOP054071(config-vlan)# exit IOP054071(config)# VLAN/VSAN 100 VLAN/VSAN 200 VLAN/VSAN 300 VLAN/VSAN 100 VLAN/VSAN 200 VLAN/VSAN 300 Nexus 5000 A Nexus 5000 B e1/2 DCB Cloud Po 777 e1/1 e1/2 e1/1 ICO-IMGo000974
  • 273. Configuring Cisco VE_Ports 273 Nexus Series Switches Setup Examples 3. Create the port-channel that will be used to connect the two switches. IOP054071(config)# interface port-channel 777 IOP054071(config-if)# switchport mode trunk IOP054071(config-if)# exit IOP054071(config)# 4. Create the vfc interface that will be associated with the port-channel, specify it as an E_Port and allow the appropriate VSANs. IOP054071(config)# interface vfc777 IOP054071(config-if)# bind interface port-channel777 IOP054071(config-if)# switchport mode E IOP054071(config-if)# no shutdown IOP054071(config-if)# exit IOP054071(config)# 5. Add the Ethernet interfaces to the port-channel. IOP054071(config)# int e1/1-2 IOP054071(config-if)# switchport mode trunk IOP054071(config-if-range)# channel-group 777 mode active IOP054071(config-if-range)# exit IOP054071(config)# At this point the VE_Port should initialize and can be viewed using the show fcoe da command. IOP054071(config)# show fcoe database ------------------------------------------------------------------------------- INTERFACE FCID PORT NAME MAC ADDRESS ------------------------------------------------------------------------------- VE Ports: ------------------------------------------------------------------------------- INTERFACE MAC ADDRESS VSAN ------------------------------------------------------------------------------- vfc777 00:05:73:af:13:68 200 vfc777 00:05:73:af:13:68 300 vfc777 00:05:73:af:13:68 400 IOP054071(config)#
  • 274. 274 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Configuring Cisco FCoE NPV ports FCoE NPV allows for traditional Fibre Channel NPV functionality to run over FCoE links. FCoE NPV ports can be created as either individual links or configured to use a PortChannel. The topology used in this example is shown in Figure 36. Figure 36 VN_Ports topology Note: In this configuration, the upstream switch cannot be in NPV mode unless it is connected to another upstream switch. Setup steps On Nexus 5596-1: 1. Configure the needed features for this configuration. Note: If the environment is a strictly FCoE environment with no FC interfaces, then the FCoE-NPV feature can be used to replace the separate FCoE and NPV features. This does not require a switch reboot. Nexus-5596-1# config t Enter configuration commands, one per line. End with CNTL/Z. Nexus-5596-1(config)# feature lacp Nexus-5596-1(config)# feature fcoe Nexus-5596-1(config)# feature npv Verify that boot variables are set and the changes are saved. Changing to npv mode erases the current configuration and reboots the switch in npv mode. Do you want to continue? (y/n):y 2. Configure needed QoS policies. Nexus-5596-1# config t Enter configuration commands, one per line. End with CNTL/Z.
  • 275. Configuring Cisco FCoE NPV ports 275 Nexus Series Switches Setup Examples Nexus-5596-1(config)# service-policy type qos input fcoe-default-in-policy Nexus-5596-1(config)# service-policy type queuing input fcoe-default-in-policy Nexus-5596-1(config)# service-policy type queuing output fcoe-default-out-policy Nexus-5596-1(config)# service-policy type network-qos fcoe-default-nq-policy 3. Create the appropriate VLAN and associate them with the corresponding VSAN. Nexus-5596-1 (config)# vlan 200 Nexus-5596-1 (config-vlan)# fcoe vsan 200 Nexus-5596-1 (config-vlan)# vlan 300 Nexus-5596-1 (config-vlan)# fcoe vsan 300 Nexus-5596-1 (config-vlan)# vlan 400 Nexus-5596-1 (config-vlan)# fcoe vsan 400 Nexus-5596-1 (config-vlan)# exit Nexus-5596-1 (config)# 4. Create the port-channel that will be used to connect the two switches. Nexus-5596-1 (config)# interface port-channel 300 Nexus-5596-1 (config-if)# switchport mode trunk Nexus-5596-1 (config-if)# exit Nexus-5596-1 (config)# 5. Create the vfc interface that will be associated with the port-channel, specify it as an NP_Port. By default the vfc interface will trunk all created VSAN'S. Nexus-5596-1 (config)# interface vfc300 Nexus-5596-1 (config-if)# bind interface port-channel300 Nexus-5596-1 (config-if)# switchport mode NP Nexus-5596-1 (config-if)# no shutdown Nexus-5596-1 (config-if)# exit Nexus-5596-1 (config)# 6. Add the Ethernet interfaces to the port-channel. Nexus-5596-1(config-if)# switchport mode trunk Nexus-5596-1(config-if-range)# channel-group 300 mode active Nexus-5596-1(config-if-range)# exit Nexus-5596-1(config)# 7. Verify the vfc interface is working properly by looking at the interface. Nexus-5596-1(config)# sho int vfc300 vfc300 is trunking (Not all VSANs UP on the trunk) Bound interface is port-channel300 Hardware is Virtual Fibre Channel Port WWN is 21:2b:00:05:73:b1:7a:3f Admin port mode is NP, trunk mode is on
  • 276. 276 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples snmp link state traps are enabled Port mode is TNP Port vsan is 1 Trunk vsans (admin allowed and active) (1,100,200,300,600,800,1002) Trunk vsans (up) (300) Trunk vsans (isolated) () Trunk vsans (initializing) (1,100,200,600,800,1002) 1 minute input rate 15773704 bits/sec, 1971713 bytes/sec, 2825 frames/sec 1 minute output rate 13929328 bits/sec, 1741166 bytes/sec, 2111 frames/sec 22280246234 frames input, 36599930008008 bytes 0 discards, 0 errors 9395517636 frames output, 12524663141656 bytes 3 discards, 0 errors last clearing of "show interface" counters never Interface last changed at Tue Aug 23 14:01:23 2011
  • 277. Complex FCoE topologies 277 Nexus Series Switches Setup Examples Complex FCoE topologies Previous section in this chapter discuss the process of configuring the Nexus switches in various basic topologies. These basic topologies are provided for example purposes and can be combined to create more complex topologies. This section describes two complex topologies based on the following: ◆ “Fixed port chassis” on page 277 ◆ “Modular chassis” on page 278 Note: The Nexus 7000 used in the modular case study is not currently supported by EMC for FCoE connectivity. Fixed port chassis The fixed port chassis configuration is based on the combination of the Nexus 5596 and the Nexus 2232PF (FEX) module. The basic building block of this configuration is a pair of racks (pod). Each rack in the pod has a single Nexus 2232 at the top of the rack and contains 16 servers. The servers in each rack are cross-connected so that all SAN A connections from both racks go to the top of one cabinet; the SAN B connection from both cabinets are all connected to the top of the other rack. Figure 37 on page 278 shows an example of a fixed port chassis configuration.
  • 278. 278 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Figure 37 Fixed port chassis configuration example In this example, these racks are part of a row that contains 30 racks for a total of 480 servers or 960 ports. Also shown are also a total of 96 storage ports evenly distributed among the 5596s. Since there are 960 server ports, this will equate to a 10:1 server to storage port ratio. Modular chassis The modular chassis configuration is based on the Nexus 7000. In this example, there are 24 racks and 384 servers. There are also two Nexus 7018s and each Nexus contains (14) 32 port 10G "F1" line cards for a total of 448 ports. Figure 38 on page 279 shows an example of a modular chassis configuration.
  • 279. Complex FCoE topologies 279 Nexus Series Switches Setup Examples Figure 38 Modular chassis configuration example Currently, this configuration is slightly oversubscribed since the server facing side allows for 320 Gb/s of connectivity while the backplane can only handle 230 Gb/s per slot. However, a fully non-blocking solution is planned for a future release. The configuration also has 768 OM3 fiber cables to allow for connectivity from the servers to the switch. There is a total of 78 storage ports connected to the Nexus 7000s.
  • 280. 280 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Nexus Series switches firmware upgrade procedure This procedure transfers files from an FTP server to the bootflash: directory on the Nexus Series switches and then uses install all to actually install the new version of firmware. It requires that the new firmware files already be uploaded to an FTP server that is accessible from the Management port of the Nexus Series switches. IMPORTANT This procedure does require the switch to be reloaded, during which time all traffic will be disrupted for 10-15 minutes. This section contains the following information: ◆ “Assumptions” on page 280 ◆ “Upgrading firmware” on page 280 Assumptions This procedure assumes the following: ◆ The firmware files are stored in the outgoing/firmware directory of an FTP server at the IP address of 168.159.216.19 ◆ The name of the kickstart image is n5000-uk9-kickstart.4.0.1a.N1.1a.bin ◆ The name of the system image is n5000-uk9.4.0.1a.N1.1a.bin Upgrading firmware Note: The text in bold is input by the user. To upgrade firmware: 1. Create a backup of your existing configuration file (make sure the running configuration is correct). Nexus-5020-2# copy running-config startup-config 2. Back up the existing configuration file. You can do this through the ftp server using the following format:
  • 281. Nexus Series switches firmware upgrade procedure 281 Nexus Series Switches Setup Examples copy startup-config ftp://<server IP address>/startup-config_<switch name>_<date> vrf management Nexus-5020-2# copy startup-config ftp://168.159.216.19/startup-config_Nexus5020-2_102308 vrf management Enter username: anonymous Password: email@company.com ***** Transfer of file Completed Successfully ***** 3. Verify the status of the Supervisor is "active" and each module is "ok". Nexus-5020-2# show module Mod Ports Model-type Model Status --- ----- ------------------------------- ------------------ --------- 1 40 40x10GE/Supervisor N5K-C5020P-BF active * 2 8 4x10GE + 4x1/2/4G FC Module N5K-M1404 ok 3 8 4x10GE + 4x1/2/4G FC Module N5K-M1404 ok Mod Sw Hw World-Wide-Name(s) (WWN) --- --------------------------------------------------------------------- 1 4.1(3)N1(1) 1.0 -- 2 4.1(3)N1(1) 1.0 20:41:00:0d:ec:a3:28:80 to 20:44:00:0d:ec:a3:28:80 3 4.1(3)N1(1) 0.100 20:81:00:0d:ec:a3:28:80 to 20:84:00:0d:ec:a3:28:80 Mod Mac Address(es) Serial-Num --- ---------------------------------- --------------- 1 000d.eca3.2888 to 000d.eca3.28af JAB122800VP 2 000d.eca3.28b0 to 000d.eca3.28b7 JAB1231022S 3 000d.eca3.28b8 to 000d.eca3.28bf JAB1231023U 4. Transfer the kickstart image to the bootflash: directory on the Nexus 5020. Nexus-5020-2# copy ftp://168.159.216.19/outgoing/firmware/n5000-uk9-kickstart.4.1.3.N2.1.bin bootflash:/n5000-uk9-kickstart.4.1.3.N2.1.bin vrf management Enter username: anonymous Password: email@company.com ***** Transfer of file Completed Successfully ***** 5. Transfer the system image to the bootflash: directory on the Nexus 5020. Nexus-5020-2# copy ftp://168.159.216.19/outgoing/firmware/n5000-uk9.4.1.3.N2.1.bin bootflash:/n5000-uk9.4.1.3.N2.1.bin vrf management Enter username: anonymous Password: email@company.com ***** Transfer of file Completed Successfully *****
  • 282. 282 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples 6. Verify that the file was copied in the required directory. Nexus-5020-2# cd bootflash: Nexus-5020-2# dir 20217856 Apr 17 10:05:59 2009 n5000-uk9-kickstart.4.0.1a.N1.1a.bin 85704356 Apr 17 10:06:40 2008 n5000-uk9.4.0.1a.N1.1a.bin 7. Ensure that the software images are not damaged or corrupted in the saved bootflash: When copying a new image to your switch, confirm that the image was not corrupted during the copy process. Use the show version image command to verify that the required image was copied successfully. Nexus-5020-2# show version image bootflash:n5000-uk9.4.1.3.N2.1.bin image name: n5000-uk9.4.1.3.N2.1.bin system: version 4.1(3)N2(1) compiled: 11/6/2009 11:00:00 [11/06/2009 20:40:29] 8. Use the install all command to install the firmware. Nexus-5020-2# install all kickstart bootflash:/n5000-uk9-kickstart.4.1.3.N2.1.bin system bootflash:/n5000-uk9.4.1.3.N2.1.bin Verifying image bootflash:/n5000-uk9-kickstart.4.0.1a.N1.1a.bin for boot variable "kickstart". [# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS Verifying image bootflash:/n5000-uk9.4.0.1a.N1.1a.bin for boot variable "system". [# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS Verifying image package type. [# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS Extracting "system" version from image bootflash:/n5000-uk9.4.0.1a.N1.1a.bin. [# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS Extracting "kickstart" version from image bootflash:/n5000-uk9-kickstart.4.0.1a.N1.1a.bin. [# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS Extracting "bios" version from image bootflash:/n5000-uk9.4.0.1a.N1.1a.bin. [# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS Collecting "running" plugin(s) information. [# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS Collecting plugin(s) information from "new" image. [# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS Compatibility check is done:
  • 283. Nexus Series switches firmware upgrade procedure 283 Nexus Series Switches Setup Examples Module bootable Impact Install-type Reason ------ -------- --------- ---------------- ------ 1 yes disruptive reset Reset due to single supervisor Images will be upgraded according to following table: Module Image Running-Version New-Version Upg-Required ------ ---------- ---------------------- ---------------------- 1 system 4.1(3)N1(1) 4.1(3)N2(1) yes 1 kickstart 4.1(3)N1(1) 4.1(3)N2(1) yes 1 bios v1.3.0(09/08/09) no Do you want to continue with the installation (y/n)? [n] Y Install is in progress, please wait. Setting boot variables. [# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS Performing configuration copy. [# # # # # # # # # # # # # # # # # # # # ] 100% -- SUCCESS Install has been successful. 9. Reconnect to the switch after the reload has completed, if applicable. Verify that the status of the Supervisor is "active" and each module is "ok". Nexus-5020-2# show module Mod Ports Model-type Model Status --- ----- ------------------------------- ------------------ ----------- 1 40 40x10GE/Supervisor N5K-C5020P-BF active * 2 8 4x10GE + 4x1/2/4G FC Module N5K-M1404 ok 3 8 4x10GE + 4x1/2/4G FC Module N5K-M1404 ok Mod Sw Hw World-Wide-Name(s) (WWN) --- -------------------------- -------------------------------------- 1 4.1(3)N2(1) 1.0 -- 2 4.1(3)N2(1) 1.0 20:41:00:0d:ec:a3:28:80 to 20:44:00:0d:ec:a3:28:80 3 4.1(3)N2(1) 0.100 20:81:00:0d:ec:a3:28:80 to 20:84:00:0d:ec:a3:28:80 Mod Mac Address(es) Serial-Num --- ---------------------------------- ------------ 1 000d.eca3.2888 to 000d.eca3.28af JAB122800VP 2 000d.eca3.28b0 to 000d.eca3.28b7 JAB1231022S 3 000d.eca3.28b8 to 000d.eca3.28bf JAB1231023U
  • 284. 284 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Troubleshooting the Nexus Series switches This section explains which troubleshooting outputs to capture from Nexus Series switches, including: ◆ “Show tech-support selections” on page 284 ◆ “General connectivity troubleshooting” on page 285 ◆ “Fibre Channel troubleshooting” on page 286 ◆ “Hardware-related errors” on page 288 ◆ “High CPU issue troubleshooting” on page 288 ◆ “Gatos Port ASIC troubleshooting” on page 289 ◆ “Altos Fabric ASIC troubleshooting” on page 289 ◆ “FCoE troubleshooting” on page 289 ◆ “DCBX troubleshooting” on page 290 ◆ “Verify zoning configuration and that CNA WWPN is zoned active” on page 290 ◆ “Verify CNA WWPN is registered in nameserver in your VSAN” on page 290 ◆ “Ethernet PortChannel troubleshooting” on page 290 ◆ “Verify PortChannels” on page 291 ◆ “Spanning-tree troubleshooting” on page 291 ◆ “Multicast troubleshooting” on page 292 Show tech-support selections Capturing show tech-support information is useful when gathering data for troubleshooting and escalations. ◆ show tech-support details is used to collect everything on the switch. ◆ sh tech-support commands lists out what will be displayed. To dump output of a show tech-support details to bootflash: on the switch for retrieval with FTP use tac-pac bootflash:/// dir bootflash: 73482 Oct 06 18:26:34 2008 show_tech_out.gz  output is zipped
  • 285. Troubleshooting the Nexus Series switches 285 Nexus Series Switches Setup Examples > Redirect it to a file aaa Display aaa information aclmgr ACL commands adjmgr Display Adjmgr information arp Display ARP information bootvar Gather detailed information for bootvar troubleshooting brief Display the switch summary cdp Gather information for CDP trouble shooting cfs Gather detailed information for cfs troubleshooting commands Show commands executed as part of show tech-support commands details Gather detailed information for troubleshooting device-alias Show device-alias technical support information ethpm Gather detailed information for ETHPM troubleshooting fc Get fibre channel related information fcdomain Gather detailed information for fcdomain troubleshooting fcns Show information for fcns technical support flogi Gather detailed information for flogi troubleshooting fspf Show information for fspf technical support im Gather detailed information for IM troubleshooting interface-vlan Gather detailed information for interface-vlan troubleshooting internal Gather internal info for troubleshooting ip Display IP information l3vm Display VRF information license Gather detailed information for license troubleshooting logging Show information on logging for technical support staff module Gather info related to a module monitor Gather detailed information for monitor troubleshooting platform Get platform related information port-channel Gather detailed information for port channel troubleshooting private-vlan Gather detailed information for private-vlan troubleshooting radius Display radius information routing Display routing information rscn Show information for rscn technical support security show tech support information for security snmp Gather info related to snmp stp Gather detailed information for STP troubleshooting sysmgr Gather detailed information for sysmgr troubleshooting vdc Gather detailed information for VDC troubleshooting vlan Gather detailed information for VLAN troubleshooting vsan Gather info related to an vsan zone Show information for zoneserver technical support | Pipe command output to filter General connectivity troubleshooting show logging SFP issues - show system internal ethpm event-history errors show interface ethx/x sh tech-support ethpm (this can be large output, need to capture) show cdp neighbors show mac
  • 286. 286 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples ethernet: show interface ethernet {x/y} counters brief ---> 5min in/out rate show interface ethernet {x/y} priority-flow-control show interface ethernet {x/y} status show interface ethernet {x/y} status err-disable show policy-map interface [ethernet {x/y}] ---> qos queue drops vfc: show interface vfc {x} counters brief show interface vfc {x} brief Access-lists: show platform afm info attachment brief ---> look for the interface with the acl in question and note group# & ASIC# show platform afm info group-config id {group# } asic {asic# } ---> group# and asic# are from previous command Fibre Channel troubleshooting show tech fc This output will capture the following shows when the switch is set in switch mode: `show interface brief` `show interface` `show port internal info all` `show port internal event-history lock` `show port internal event-history msgs` `show port internal event-history errors` `show port internal mem-stats detail` `show san-port-channel internal event-history all` `show san-port-channel internal event-history errors` `show san-port-channel internal event-history msgs` `show san-port-channel internal event-history lock` `show san-port-channel internal mem-stats detail` `show san-port-channel usage` `show san-port-channel summary` `show san-port-channel consistency detail` `show tech-support device-alias` `show device-alias database` `show device-alias pending` `show device-alias pending-diff` `show device-alias status` `show device-alias session status` `show device-alias merge status` `show device-alias statistics` `show device-alias internal info`
  • 287. Troubleshooting the Nexus Series switches 287 Nexus Series Switches Setup Examples `show device-alias internal peer-info` `show device-alias internal validation-info` `show device-alias internal event-history session-fsm` `show device-alias internal errors` `show device-alias internal mem-stats detail` `show device-alias internal msgs` `show fcdomain domain-list` `show tech-support fcns` `show fcns database vsan 1-4093` `show fcns database detail vsan 1-4093` `show fcns database local vsan 1-4093` `show fcns database local detail vsan 1-4093` `show fcns statistics vsan 1-4093` `show fcns statistics detail vsan 1-4093` `show fcns internal info vsan 1-4093` `show fcns internal event-history` `show fcns internal event-log` `show fcroute unicast` `show fcs database` `show fcs ie` `show fctimer` `show flogi database` `show flogi internal info` `show fspf` `show fspf database` `show tech-support rscn` `show rscn internal vsan 1-4093` `show rscn internal event-history` `show rscn internal mem-stats detail` `show rscn internal session-history vsan 1-4093` `show rscn internal merge-history vsan 1-4093` `show rscn statistics vsan 1-4093` `show rscn scr-table vsan 1-4093` `show rscn session status vsan 1-4093` `show vsan` `show vsan membership` `show tech-support zone` `show zone status vsan 1-4093` `show zoneset active vsan 1-4093` `show zoneset vsan 1-4093` `show zone vsan 1-4093` `show fcalias vsan 1-4093` `show zone policy vsan 1-4093` `show zoneset pending active vsan 1-4093` `show zoneset pending vsan 1-4093` `show zone pending vsan 1-4093` `show zone pending active vsan 1-4093` `show fcalias pending vsan 1-4093` `show zone policy pending vsan 1-4093` `show zone pending-diff vsan 1-4093` `show zone analysis active vsan 1-4093` `show zone analysis vsan 1-4093`
  • 288. 288 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples `show zone ess vsan 1-4093` `show zone internal vsan 1-4093` `show zone internal change event-history vsan 1-4093` `show zone internal ifindex-table vsan 1-4093` `show zone internal merge event-history vsan 1-4093` `show zone internal event-history` `show zone internal event-history errors` `show zone internal tcam event-history vsan 1-4093` `show zone statistics vsan 1-4093` `show system default zone` `show zone internal ddas-table` `show zone internal sdv-table vsan 1-4093` `show zone internal mem-stats` `show zone internal mem-stats detail` `show zone internal transit-table received vsan 1-4093` `show zone internal transit-table forwarded vsan 1-4093` `show zone internal transit-table rejected vsan 1-4093` This output will capture the following shows when switch is set in npv mode: `show interface brief` `show interface` `show port internal info all` `show port internal event-history lock` `show port internal event-history msgs` `show port internal event-history errors` `show port internal mem-stats detail` `show vsan` `show vsan membership` Hardware-related errors show hardware internal fc-mac all-ports <<< Link Issues - ports flapping show hardware internal plog errors Gatos port statistics for FC : If there is any one port having issues show hardware internal gatos port fc x/y show hardware internal gatos port fc x/y {counters|error|rx|tx|interrupt} High CPU issue troubleshooting show platform fwm info pif all ---> gives physical interface (pif) details show platform fwm info lif all ---> gives logical interface (lif) details show ip traffic
  • 289. Troubleshooting the Nexus Series switches 289 Nexus Series Switches Setup Examples Gatos Port ASIC troubleshooting show hardware internal gatos port ethernet {x/y} show hardware internal gatos port ethernet {x/y} counters error show hardware internal gatos port ethernet {x/y} counters rx show hardware internal gatos port ethernet {x/y} counters tx show hardware internal gatos port ethernet {x/y} interrupt show platform fwm info pif ethernet {x/y} ---> note the gatos number for the port for next command show platform fwm info gatos-errors {gatos# } ---> gives reason for drops in previous command show hardware internal gatos event-history errors | grep error_interrupt Altos Fabric ASIC troubleshooting show hardware internal altos counters interrupts all ---> shows non-zero interupts FCoE troubleshooting show tech-support flogi show int ethernet x/x priority-flow-control LLDP Transmit and Receive should be enabled on the interface for DCBX negotiation to be successful switch# show lldp interface ethernet 2/1 | grep tx_en tx_enabled: TRUE switch# show lldp interface ethernet 2/1 | grep rx_en rx_enabled: TRUE Verify ethernet interface is UP - sh int ethernet 28/1 Verify vfc interface is connected - show int vfc 1
  • 290. 290 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples DCBX troubleshooting show lldp neighbors show lldp interface ethernet {x/y} | grep tx_en ---> LLDP Tx should be enabled for DCBX negotiation to be successful show lldp interface ethernet {x/y} | grep rx_en ---> LLDP Rx should be enabled for DCBX negotiation to be successful show lldp interface ethernet {x/y} | grep "No remote peers exist" ---> check that peer supports LLDP show platform software dcbx internal errors | grep "Received bad LLDP Packet" ---> check LLDP rxed by peer is good show platform software dcbx internal info interface ethernet {x/y} | grep "DCX TLV" ---> If DCBX TLV's are not shown, peer doesn't support DCBX TLV show platform software dcbx internal info interface ethernet {x/y} ---> for every feature negotiation result: "error=0" and an "Operating Config" should exist Can use ethanalyzer to see packets going from/to CNA Typical Menlo based CNA trace ethanalyzer local interface inbound-hi brief display-filter eth.addr==00:c0:dd:0a:ad:1d (find the eth addr from show int ethernet x/y and show int vfc x/y ) Verify zoning configuration and that CNA WWPN is zoned active sh zoneset active zoneset name ZS_V11 vsan 11 zone name Z_V11 vsan 11 * fcid 0x2e0001 [pwwn 21:01:00:1b:32:36:a2:ed] * fcid 0x5c0400 [pwwn 50:06:0b:00:00:1e:8a:a9] Verify CNA WWPN is registered in nameserver in your VSAN show fcns database vsan 11 Ethernet PortChannel troubleshooting When you add an interface to a channel group, the Cisco NX-OS checks certain interface attributes to ensure that the interface is compatible with the channel group. The Cisco NX-OS also checks a number of operational attributes for an interface before allowing that interface to participate in the port-channel aggregation.
  • 291. Troubleshooting the Nexus Series switches 291 Nexus Series Switches Setup Examples The compatibility check includes the following operational attributes: ◆ Port mode ◆ Access VLAN ◆ Trunk native VLAN ◆ Allowed VLAN list ◆ Flow control Use the show port-channel compatibility-parameters command to see the full list of compatibility checks that the Cisco NX-OS uses. Verify PortChannels sh spanning-tree show port-channel summary show port-channel database show port-channel traffic show port-channel internal info all show platform fwm info pc port-channel {# } show lacp neighbor Spanning-tree troubleshooting show tech-support stp show spanning-tree <option> <CR> > Redirect it to a file active Report on active interfaces only blockedports Show blocked ports bridge Status and configuration of this bridge brief Brief summary of interface information detail Detailed information inconsistentports Show inconsistent ports interface Spanning Tree interface status and configuration internal Show internal STP information mst Multiple spanning trees pathcost Show Spanning pathcost options root Status and configuration of the root bridge summary Summary of port states vlan VLAN Switch Spanning Trees | Pipe command output to filter
  • 292. 292 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples Multicast troubleshooting show ip igmp snooping <option> explicit-tracking Display explicit-tracking database for IGMPv3 groups Display snooping information for group address mrouter Display multicast routers detected querier Display snooping querier information statistics Display packet/error counter statistics vlan Display Vlan IGMP snooping membership information | Pipe command output to filter sh mac-address-table multicast <option> count Display only the count of MAC entries igmp-snooping Display Entries learned from IGMP Snooping user Display User Entered Entries lan VLAN sh interface ethernet x/x counters storm-control
  • 293. Virtual PortChannel 293 Nexus Series Switches Setup Examples Virtual PortChannel This section provides information on Cisco virtual PortChannel (vPC) and includes topology examples and Nexus case studies. ◆ “Introduction” on page 293 ◆ “vPC overview” on page 299 ◆ “vPC topologies overview” on page 306 ◆ “vPC topology considerations” on page 309 ◆ “vPC case studies” on page 324 Introduction A virtual PortChannel (vPC) allows PortChannel links that are physically connected to two different Cisco Nexus products (7000, 6000, or 5000) to appear as a single virtual PortChannel to a third device. This technology allows for Layer 2 multipathing while maintaining a loop free topology. The purpose of this chapter is to explain the advent of vPC and provide detailed case studies that describe how to configure several of the supported vPC topologies. The case studies are based on configurations that have been set up within the EMC E-Lab™ and focus primarily on the storage protocols FCoE, iSCSI, and NAS. This section explains the following: ◆ “vPC and the storage protocols” on page 293 ◆ “Underlying protocols” on page 294 vPC and the storage protocols This section briefly describes how the new vPC protocol interacts with the three major storage protocols, each discussed further in this section. ◆ “FCoE” on page 293 ◆ “iSCSI” on page 294 ◆ “NAS” on page 294 FCoE Like Fibre Channel, FCoE gains its path failover and load balancing features from host-based multipathing products, such as EMC PowerPath or MPIO. FCoE does not operate over vPC because a host
  • 294. 294 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples must use NIC teaming/bonding to operate with vPC, which inherently breaks the native Fibre Channel multipathing software. However, FCoE runs successfully with CNAs participating in vPCs, but the FCoE traffic will not take advantage of the vPC protocol itself. iSCSI iSCSI is like FCoE in that it gains its path failover and load-balancing features from host-based multipathing products, such as EMC PowerPath and MPIO. iSCSI will operate in a vPC environment, but does not take advantage of the vPC protocol. iSCSI relies on the upper layer applications to provide failover capabilities. NAS NAS, along with any other type of Ethernet traffic capable of running over a NIC teaming/bonding configuration within a server or host, will function over the vPC protocol. Underlying protocols To better understand how vPC works, it is important to have a working knowledge of the two underlying protocols that have had an impact on its development, each discussed further in this section: ◆ “Spanning Tree Protocol (STP)” on page 294 ◆ “Link Aggregation” on page 296 Spanning Tree Protocol (STP) The Spanning Tree Protocol (STP) is a Link Layer network protocol that was developed to guarantee a loop-free topology within a switched or bridged (Layer 2) network topology. STP prevents loops in a Layer 2 network by placing redundant links into a blocking, or non-forwarding, state. While this allows for large-scale Layer 2 networks to be built, it does have its drawbacks, such as: ◆ In any type of redundant topology, only one link can be forwarding from one Layer 2 device to another Layer 2 device at a time. ◆ In the event of a failover from an active link to a redundant link, the STP protocol will cause the interfaces to stop forwarding all traffic for a period of time. The length of time depends on the STP version in use. Both of these limitations are necessary, do work well, and allow for large and reliable Layer 2 networks. However, in today's data centers there is a new trend to merge traditional Ethernet traffic with newer storage protocols, which has driven the need to re-think the way in which a Layer 2 network should behave.
  • 295. Virtual PortChannel 295 Nexus Series Switches Setup Examples For more detailed information on the Spanning Tree Protocol, refer to the Spanning Tree Protocol (STP)” section in the "Ethernet Basics" chapter in the Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook at http://elabnavigator.EMC.com, under the Topology Resource Center tab. Figure 39 shows a high-level example of how STP works. Figure 39 STP example In Figure 39, STP puts the redundant links to the Root Bridge into a blocking state to eliminate any loops created by the topology. Although this is the way STP is designed to work, the downside is that the redundant link sits in an idle state until the primary link is brought down. This effectively cuts the available bandwidth to the upstream switches in half. Also, when a failover occurs, the switches will stop forwarding all traffic until the topology is stable again. Disruption times vary depending on the size of the Layer 2 network, the complexity of the network, the version of STP in use, and where a failure occurs. The benefits and limitations of Spanning Tree are as follows: ◆ Benefits include: • Scalable • Resilient X X Root bridge SW 1 SW 2 SW 3 SW 4 X = Blocking port ICO-IMG-000877
  • 296. 296 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples ◆ Limitations include: • Limits bandwidth • Convergence times Link Aggregation This section provides information on the following: ◆ “Link Aggregation” on page 296 ◆ “Link Aggregation Control Protocol (LACP)” on page 298 Link Aggregation Link Aggregation is a method of bundling multiple physical interfaces into a single virtual interface within a single piece of hardware, such as a switch or router. Servers with multiple Network Interface Cards (NICs) can also take advantage of this through the use of NIC Teaming (for Windows Operating Systems) or NIC Bonding (for Linux Operating Systems). Link Aggregation, generically known in Cisco terminology as a PortChannel, can also be called an EtherChannel or LAG, depending on which vendor is being referenced. Link aggregation groups (LAG) appear as one interface to a switch and therefore are treated as one instance to Spanning-Tree. This allows all ports within a LAG to be active and forward traffic simultaneously.
  • 297. Virtual PortChannel 297 Nexus Series Switches Setup Examples As shown in Figure 40, PortChannels are created between switches 1–3 and 2–4. The total theoretical bandwidth of each PortChannel is 40 Gb/s. Figure 40 PortChannel example When a LAG is used, two things happen: ◆ Overall bandwidth increases ◆ STP sees the virtual interface as a single entity As a result, STP will forward through the virtual link rather than put one or more of the redundant links into a blocking state. If one of the physical links within a PortChannel fails, then any traffic flows currently using that link would failover to one of the other links within the bundle, bypassing STP. This failover time is within milliseconds and is transparent to most applications. A PortChannel can be configured manually or by using the standards based Link Aggregation Control Protocol (LACP). If LACP is not used, the configurations on both sides of a PortChannel must be configured identically or the link will not come up. For LACP compatibility, refer to Table 7 on page 299. The benefits and limitations of Link Aggregation are as follows: ◆ Benefits include: • Bypass STP • Increased bandwidth • Quick failover times 10 Gig 10 Gig 10 Gig Port Channel 4 x 10 Gig = 40 Gig Port Channel 4 x 10 Gig = 40 Gig SW 1 SW 2 SW 3 SW 4 ICO-IMG-000878
  • 298. 298 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples ◆ Limitations include: • Point-to-point only • Not scalable For more detailed information on Link Aggregation, refer to “Link Aggregation” section in the "Ethernet Basics" chapter in the Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB) Concepts and Protocols TechBook at http://elabnavigator.EMC.com, under the Topology Resource Center tab. IMPORTANT It is recommended that LACP be used to configure the PortChannel interfaces. Link Aggregation Control Protocol (LACP) LACP is part of the IEEE specification (802.3ad) and was developed to allow for the dynamic formation of Link Aggregations between two participating peer devices. This is accomplished by devices on both ends of a link participating in the protocol by transmitting Link Aggregation Control Protocol Data Units (LACPDUs). If a sending device finds a LACP-enabled device on the other end, then a communication process begins in which both ends of the link will forward configuration information. All links participating in LACP will be able to determine if they belong to the same Link Aggregation and will form the bundle as configured. For example, if you were to build a four-link bundle using LACP between two end devices (switches, in this case) you would first need to configure the switches to tell the participating interfaces that they all belong to the same PortChannel with LACP enabled. Once the devices are connected, the interfaces will communicate and form the PortChannel dynamically, based on the pre-configuration of the interfaces. LACP can be configured in two ways, active or passive. ◆ Active will continually send out frames hoping to reach another LACP-enabled interface ◆ Passive will not send out any LACP frames but will listen for them and form a link only after it has received frames from an Active link. If you configure a PortChannel as ON, then the LACP protocol will not be invoked and manual configurations will be used. Interfaces
  • 299. Virtual PortChannel 299 Nexus Series Switches Setup Examples configured as ON will not participate in the LACP protocol and therefore will not work with interfaces running LACP. For this link to form, both ends of the link need to be ON and the configurations must match. Table 7 lists LACP compatibility. vPC overview Cisco's virtual PortChannel (vPC) feature was created to overcome both Spanning Tree and Link Aggregation limitations. vPC overcomes these limitations by allowing a multi-chassis PortChannel to be created while bypassing STP. More details are provided in the following Cisco documentation, located at www.cisco.com: ◆ Virtual PortChannel Quick Configuration Guide ◆ Cisco Nexus 7000 Series NX-OS Interfaces Configuration Guide ◆ Cisco Nexus 5000 Series Layer 2 Switching Configuration Guide A vPC allows links that are physically connected to two different Cisco Nexus products (7000 or 5000) to appear as a single PortChannel to a third device. The third device can be a switch, server, or any other networking device capable of running Link Aggregation protocol. For an example, refer to Figure 41 on page 300. The links from the server are physically connected to SW 3 and SW 4. These links appear as a single PortChannel on the server. Table 7 LACP compatibility Switch Switch/Host Comments active active Recommended. active passive Link aggregation occurs if negotiation is successful. on on Link aggregation occurs without LACP. Note: Although this works, it is not recommended. on active Will not work. on passive Will not work.
  • 300. 300 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples A vPC provides multipathing, which allows you to create redundancy by enabling multiple parallel paths between devices and also allows for traffic to be load balanced where alternative paths exist. The third device sees the multi-chassis vPC as a single entity. This allows for all paths to be actively forwarding because STP is no longer putting the redundant links into a blocking state. The following sections provide more information on vPC: ◆ “vPC example” on page 300 ◆ “Benefits” on page 301 ◆ “Concepts” on page 301 vPC example Figure 41 shows an example of vPC. Figure 41 vPC example Note: Figure 41 shows that a vPC can be created to a host. This is not applicable in all topologies. vPC x vPC z vPC y vPC Device vPC Device Nexus 5000 Nexus 5000 SW 1 SW 2 SW 3 SW 4 ICO-IMG-000879 vPC Domain Peer-keepalive link vPC Peer link
  • 301. Virtual PortChannel 301 Nexus Series Switches Setup Examples Benefits The benefits of vPC include: ◆ Allows a single device to create a PortChannel across two upstream devices ◆ Eliminates Spanning Tree Protocol blocked ports ◆ Provides a loop-free topology ◆ Uses all available uplink bandwidth ◆ Provides fast convergence if either the link or a device fails ◆ Provides link-level resiliency ◆ Helps ensure high availability ◆ Hashing algorithm by default is the same as used in traditional load-balancing. • Layer 3 – Source-destination ip addresses • Layer 2 – Source-destination mac addresses Concepts To better understand vPC, consider the following vPC-specific terminology, as described in the Virtual PortChannel Quick Configuration Guide, located at www.cisco.com. vPC The combined EtherChannel between the vPC peer devices and the downstream device. vPC domain This domain includes both vPC peer devices, the vPC peer-keepalive link, and all of the port channels in the vPC connected to the downstream devices. It is also associated to the configuration mode that you must use to assign vPC global parameters. The vPC Domain ID must be the same on both switches. Refer to “vPC domain” on page 302 for more information. vPC Domain ID A unique numeric value in the range of 1-1000 assigned to a vPC Domain. vPC peer switch One of a pair of devices that are connected with the special EtherChannel known as the vPC peer link.
  • 302. 302 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples vPC peer link The link used to synchronize states between the vPC peer devices. The vPC peer link carries control traffic between two vPC switches as well as multicast, broadcast data traffic. In some link failure scenarios, it also carries unicast traffic. You should have at least two 10 Gigabit Ethernet interfaces for peer links. Refer to “vPC peer link” on page 303 for more information. vPC peer-keepalive link The peer-keepalive link monitors the vitality of a vPC peer Cisco Nexus 5000 Series device. The peer-keepalive link sends configurable, periodic keepalive messages between vPC peer devices. The vPC peer-keepalive link can be a management interface or switched virtual interface (SVI) No data or synchronization traffic moves over the vPC peer-keepalive link; the only traffic on this link is a message that indicates that the originating switch is operating and running vPCs. Refer to “peer-keepalive link and messages” on page 305 for more information. vPC member port Interfaces that belong to the vPCs. Host vPC port vPC between a host running NIC teaming or bonding to a set of upstream switches. Switch vPC vPC running between two sets of switches. This is accomplished by configuring vPCs within a domain to connect to a set of downstream switches. vPC domain A vPC domain is used to identify the vPC switches that will participate together within a vPC configuration. Any two switches needing to share vPC configurations will need to share the same vPC Domain ID, a value from 1 to 1000. A Nexus 5000 can be part of only one vPC domain at a time. Cisco switches capable of running Virtual Device Context (VDC) can have a different vPC Domain ID per context. To configure the vPC Domain ID, assign it through Global Configuration Mode. Note: The vPC Domain ID is completely separate and has nothing to do with the FC Domain ID. After the ID is assigned, you will be placed into the vPC domain configuration mode where you will finish any configurations needed within the domain. Switches that are members of the same vPC domain are referred to as vPC peer devices.
  • 303. Virtual PortChannel 303 Nexus Series Switches Setup Examples Some of the configuration options within the vPC domain configuration are: ◆ Peer-keepalive—This is a mandatory configuration parameter that is needed to identify the vPC destination peer switch by IP address. The peer switches use this link to communicate by sending periodic keepalive messages. You must have Layer 3 connectivity between the peer switches to transmit these messages. The system cannot bring up the vPC peer link unless a peer-keepalive link is already up and running. Note: It is recommend that you configure the vPC peer-keepalive link on the Cisco Nexus 5000 Series switches to run in the management VRF using the mgmt 0 interfaces. ◆ Role priority—Optional configuration to explicitly set vPC priority (1 to 65636; default value is 32677). The peer device with the lower priority becomes the vPC primary switch. ◆ System priority—Optional configuration to explicitly set the LACP priority within a vPC domain (1 to 65636; default value is 32677). Note: It is recommended that the vPC Peer switches be set as the primary devices. This priority will need to be the same on both vPC switches or the vPC will not come up. ◆ System mac—Optional configuration. The switch will automatically assign a virtual mac address to the vPC Domain. The Cisco Nexus 5000 Series Switches Virtual PortChannel Quick Configuration Guide, located at www.cisco.com, explains vPC peer link in more detail. vPC peer link vPC peer links are the physical links connecting two vPC switches. These are used to carry the synchronization and control data between peer switches. In the event of some link failure scenarios, they can also carry normal user traffic. These links do not carry the peer-keepalive messages. Note: For redundancy, it is recommended that at least two 10 Gb links be used to form an EtherChannel between two vPC switches. If one of the interfaces in the vPC peer link fails, the switch will automatically failover to the other interface.
  • 304. 304 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples You can have only two switches as vPC peers. Each switch can serve as a vPC peer to only one other vPC peer. The vPC peer switches can also have non-vPC links to other switches. To make a valid configuration, you configure an EtherChannel on each switch and then configure the vPC domain. You assign the EtherChannel on each switch as a peer link. Since the vPC configuration involves two separate switches, there are certain configurations that must match on both switches or the vPC will not come up. Table 8 provides a list of the parameters that must match using the sho vpc consistency-parameters global command. Table 8 Parameters (page 1 of 2) Name Type Local value Peer value QoS 2 ([], [3], [], [], [], []) ([], [3], [], [], [], []) Network QoS (MTU) 2 (1538, 2240, 0, 0, 0, 0) (1538, 2240, 0, 0, 0, 0) Network Qos (Pause) 2 (F, T, F, F, F, F) (F, T, F, F, F, F) Input Queuing (Bandwidth) 2 (50, 50, 0, 0, 0, 0) (50, 50, 0, 0, 0, 0) Input Queuing (Absolute 2 (F, F, F, F, F, F) (F, F, F, F, F, F) Priority) Output Queuing (Bandwidth) 2 (50, 50, 0, 0, 0, 0) (50, 50, 0, 0, 0, 0) Output Queuing (Absolute Priority) 2 (F, F, F, F, F, F) (F, F, F, F, F, F) STP Mode 1 Rapid-PVST Rapid-PVST STP Disabled 1 None None STP MST Region Name 1 "" "" STP MST Region Revision 1 0 0 STP MST Region Instance to 1 VLAN Mapping STP Loopguard 1 Disabled Disabled STP Bridge Assurance 1 Enabled Enabled
  • 305. Virtual PortChannel 305 Nexus Series Switches Setup Examples The Cisco Nexus 5000 Series Switches Virtual PortChannel Quick Configuration Guide, located at www.cisco.com, explains vPC peer link in more detail. peer-keepalive link and messages The peer-keepalive link between vPC peer switches is actually a Layer 3 connection used to transmit periodic keepalive messages. This peer-keepalive link does not use the vPC peer link connections used to carry the vPC traffic between vPC peers. Note: On Nexus 5000 switches, it is recommended that you configure the mgmt 0 interface to carry the keepalive Hello packets. You must have Layer 3 connectivity between the peer switches to transmit these messages. The system cannot bring up the vPC peer link unless a peer-keepalive link is already up and running. A keepalive message is nothing more than a frame being transmitted by peer switches indicating that they are alive and running vPC. No data or synchronization information is sent over the keepalive link. STP Port Type, Edge 1 Normal, Disabled, Normal, Disabled, BPDUFilter, Edge BPDUGuard Disabled Disabled STP MST Simulate PVST 1 Enabled Enabled Allowed VLANs - 1,100, 300, 402, 700-701, 800, 900-910 1, 400-402, 700-701, 800, 900-910 Local suspended VLANs - 100, 200, 300, 400-401, 500, 1000 - Local suspended VLANs - 100, 200, 300, 400-401 - Table 8 Parameters (page 2 of 2) Name Type Local value Peer value
  • 306. 306 Fibre Channel over Ethernet (FCoE) Case Studies TechBook Nexus Series Switches Setup Examples The keepalive messages are configurable through the vPC Domain configuration. Table 9 lists the default values used for the keepalive messages. The Cisco Nexus 5000 Series Switches Virtual PortChannel Quick Configuration Guide, located at www.cisco.com, explains peer-keepalive link and messages in more detail. vPC topologies overview This section provides basic examples of the following vPC topologies: ◆ “Switch vPC/Straight-through topology” on page 306 ◆ “Switch vPC/cross-connect topology” on page 307 ◆ “Host vPC/direct-connect topology” on page 309 For more details on deploying the different topologies mentioned in this section, refer to “vPC topology considerations” on page 309. For case studies of each of the topologies, refer to “vPC case studies” on page 324. Switch vPC/Strai