The document provides an overview of Cisco Nexus 2000 Fabric Extender architectures, features, and topologies. It begins with introductions to various Cisco Nexus parent switch platforms that support Nexus 2000 FEX devices, including the Nexus 5000, 6000, 7000, 7700, 9300, and 9500 series. It then discusses the Nexus 2000 platform architectures, supported topologies and features. The agenda outlines a deep dive into features, designs, configurations, and connectivity options for Nexus 2000 FEX deployments.

2. In-depth and personal with the Cisco Nexus
2000 Fabric Extender Architectures, Features,
and Topologies
Pavan S Kaushik
Technical Marketing Engineer
BRKARC-3454

3. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Never heard of FEX before ??
No Problem !!
FEX Gurus in the house ??
No Problem !!
BRKARC-3454 3

4. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Goals
• Brief Nexus 5000/6000/7000/9000 Parent platform introduction.
• Discuss Nexus 2000 platform architectures, supported parent switches,
actual designs and best practices.
• Nexus 2000 topologies, features and implementation.
• vPC/FCoE/FC/FP/QoS/ACI Deep dive and troubleshooting will not be
covered in this session.
BRKARC-3454 4

5. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
House Rules
• We have Nexus 2300 switches in the Room !! Do take a look.
• Keep the session interactive.
• Highly detail-oriented questions will be taken offline(ex. Optics/Model
compatibility etc).
5BRKARC-3454

6. Agenda
• Introduction
• Parent switches overview
• Platform overview
• Features deep-dive and packet-walk
• Design and topologies
• Configuration and Provisioning
• Connectivity options and Optics
• Summary

7. Introduction
Nexus 2000 / FEX

8. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Modular Switch
In-chassis Linecards
Cisco Nexus 2000 Fabric EXtenders

Parent Switch*
With Remote Linecards (FEX)
* - Parent switch can be Fixed or Modular
BRKARC-3454 8

9. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FEX: Virtualized Access Switching
• Flexible and simplified top-of-rack (ToR) designs
• Simplified Management, Centralized configuration
• Multiple Parent Switch support – N5K, N6K, N7K, N9K
• Unified Ports provide flexible LAN and SAN deployment
Extending the boundaries of Data Center Fabric
Virtualized Switch
. . .
BRKARC-3454 9

10. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 Solution
Low OPEX
Low CAPEX
Lower Devices to Manage
Central Point of Management
Traditional Architecture
Agnostic(ACI, FA)
Enhanced Buffers
Low over subscription
High Port Density
Enhanced
Performance
Low TCO
Simplified
Operation
Nexus 2000 Architecture with Single Point
of Management
Architecture
Flexibility
BRKARC-3454 10

11. Parent switches overview :
Nexus 5000/6000/7000/9000

12. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fixed Switching Platforms Modular Switching Platforms
Application Centric Infrastructure (ACI)
Nexus 9000 Standalone
Nexus 2200/
Nexus 2300
Nexus 6000 Nexus 7000
Nexus 5000/5500
Nexus 5600
Nexus 2000 - Parents Switches Family
Nexus 7700
Nexus 9000
BRKARC-3454 12

13. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
NEXUS 5600 SERIES
Nexus 5672UP
7.0(1)N1(1)
Nexus 56128P
7.0(2)N1(1)
Nexus 5624Q
7.1(0)N1(1a)
Nexus 5648Q
7.1(1)N1(1)
Nexus 5696Q
7.0(4)N1(1)
Switch Fabric
Throughput
1.44 Tbps 2.56 Tbps 1.92 Tbps 3.84 Tbps 7.68 Tbps
Port-to-Port Latency ~1.0 us
Layer 3 Capability Integrated Line-Rate
Switch Footprint 1RU 2RU 1RU 2RU 4RU
10 Gigabit Ethernet
Port
72 128 96 192 384
40 Gigabit Ethernet
Port
6 8 24 48 96
Unified Ports 16 48 N/A N/A 48
1/10 Gigabit Ethernet
With FEX
1152 1152 1152 1152 2304
Packet Buffer 25MB per 3x40G (or 12x10G)
FEX Support Yes* Yes* Yes* Yes* Yes*
SCALEDENSITYPERFORMANCE
BRKARC-3454 13
* Verify model-specific compatibility through DataSheets / ReleaseNotes / Hardware Installation Guides

14. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 7000 Product Family
Shipping up to 384x40G ports and 192x100G ports
192 x 10-Gbps Ports or
96 x 40-Gbps Ports or
48 x 100-Gbps Ports
384 x 10-Gbps Ports or
192 x 40-Gbps Ports or
96 x 100-Gbps Ports
768 x 10-Gbps Ports or
384 x 40-Gbps Ports or
192 x 100-Gbps Ports768 x 10-Gbps Ports* or
192 x 40-Gbps Ports
384 x 10-Gbps Ports* or
96 x 40-Gbps Ports
336 x 10-Gbps Ports *
84 x 40-Gbps Ports
96 x 10-Gbps Ports *
24 x 40-Gbps Ports
Nexus 7000 Family
F3 Modules
Nexus 7706
Nexus 7710
Nexus 7718
Nexus 7018Nexus 7010
Nexus 7009
Nexus 7004
48x 10-Gbps Ports or
24x 40-Gbps Ports or
Nexus 7002
BRKARC-3454 14

15. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 7000 / 7700 Line Card Modules
M1 1G/10G M2 10G / 40G / 100G
F1 10G F2 10G
F2E 10G F3 40G
F2E 10G
F3 10G / 40G / 100G
F3 closes the F/M feature gap!
M3 1G / 10G / 40G
BRKARC-3454 15

16. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 7000/7700 Supervisor FEX support
Nexus 7000 SUP2 Nexus 7000 SUP2E Nexus 7700 SUP2E
CPU Quad-Core Xeon Dual Quad-Core Xeon Dual Quad-Core Xeon
Speed (GHz) 2.13 2.13 2.13
Memory (GB) 12 32 32
Max VDCs 4 (user) +1 (admin) 8 (user) +1 (admin) 8 (user) +1 (admin)
Max VLANs per FEX 2000 2000 2000
Max VLANs per FEX HIF 75 75 75
Max. no. of FEX/Ports 32 FEX /1536 Ports 64 FEX /3072 Ports 64 FEX /3072 Ports
BRKARC-3454 16

17. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
NEXUS 7000 Line Card FEX support
M2-24XP M2-06FQ F2-48XP F2E-48XP F3-12FQ F2E-48XP F3-48XP F3-24FQ
Chassis
Supported
Nexus 7000 Nexus 7000 Nexus 7000 Nexus 7000 Nexus 7000 Nexus 7700 Nexus 7700 Nexus 7700
Ports and
Speed
24x10GE 6x40GE 48x1/10GE 48x1/10GE 12x40GE 48X1/10GE 48x1/10GE 24x40GE
Interfaces SFP+ QSFP+ SFP/SFP+ SFP/SFP+ QSFP+ SFP/SFP+ SFP/SFP+ QSFP+
Max Fabric
Cards and BW
(GBps)
3
240
3
240
5
480
5
480
3
480
3
480
3
480
5
960
Performance
(Mpps) 120
120 720 720 720 720 720 1440
vPC (Support) Yes Yes Yes Yes Yes Yes Yes Yes
FCoE, FP
Support
No Yes Yes Yes Yes Yes Yes Yes
Max. Power (W) 795 795 450 450 500 450 450 650
FEX Support Yes* Yes* Yes* Yes* Yes* Yes* Yes* Yes*
BRKARC-3454 17
* Verify model-specific compatibility through DataSheets / Release Notes / Hardware Installation Guides

18. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Nexus 9500 Platform Switches
Density in DC Optimized Footprint
21RU
8-Slot
Cisco Nexus® 9500
4-Slot
16
57.6 Tbps
512
1024
576
2304/2048
8
28.8 Tbps
256
512
288
1152/1024
4
14.4 Tbps
128
256
144
576/512
Nexus 9516
16-Slot
Nexus 9508 Nexus 9504
Payload Slots
Capacity
100G Density
50G Density
40G Density
10/25G Density
13RU
7RU
Fabric Bandwidth:
3.2 Tbps/ slot
Cloud Deployment Options:
ACI Multi-Site, Telemetry,
Cloud Scale
Multi-Generation
Investment Protection:
No Mid-plane, Power Supply Headroom
for 100/400G and integrated optics/
encryption
BRKARC-3454 18

19. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 9500 Supervisor FEX support
Supervisor A Supervisor B
CPU 4 Core, 1.8GHz 6 Core, 2.2GHz
Memory (GB) 16 GB 24GB
Storage (SSD) 64 GB 256GB
Max. Power (W) 80 90
FEX Support 32 FEX /1536 Ports 32 FEX / 1536 Ports
VLANs per FEX 2000 2000
VLANs per FEX HIF 75 75
BRKARC-3454 19

20. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
NEXUS 9500 Line Card FEX support
X9636PQ X9536PQ X9564PX X9432PQ X9464PX
Chassis Supported Nexus 9504/9508 Nexus 9500 Nexus 9500 Nexus 9500 Nexus 9500
Ports and Speed 36x40GB 36x40GB
48x1/10GB,
4x40GB
32x40GB
48x1/10GB,
4x40GB
Interfaces QSFP+ QSFP+
SFP/SFP+
QSFP
QSFP+
SFP/SFP+
QSFP+
Performance
(Mpps)
120 720 720 720 720
vPC (Support) Yes Yes Yes Yes Yes
Buffer 36 MB 104 MB 104 MB 24 MB 12 MB
Max. Power (W) 795 450 450 310 500
FEX Support Yes* Yes* Yes* Yes* Yes*
BRKARC-3454 20
* Verify model-specific compatibility through DataSheets / Release Notes / Hardware Installation Guides

21. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
+
Nexus 9300 Fixed Series
ACI Leaf and NX-OS Switches
9396 SFP+(48x10+12x40G)
9396 Copper (48x10+12x40G)
93128 Copper (96x10+8x40G)
Modular Uplink | 2HCY‘13
• Analytics
• Quad-core CPU
• 64 GB SSD
• 24 GB Memory
93120 Copper (96x10+6x40G)
9372(E) SFP+ (48x10+6x40G)
9332 QSFP+ (32x40G)
9372 (E) Copper (48x10+6x40G)
Integrated Uplink | CY’14-15
93180YC-EX (48x25G+6x100G)
93108TC-EX (48x10GT+6x100G)
32p QSFP Switch
28p 40G + 4p 100G
24p 40G + 6p 100G
NEW
Q2’CY16
Q4CY16
21

22. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 9300 Switches FEX support
Nexus 9332PQ Nexus 9372PX Nexus 9396PX
Form Factor (RU) 1 1 2
Throughput(Tbps) 2.56 1.44 1.44
1/10GB Ports X 48 48
40 GB Ports 32 6 12
Latency (us) < 2 < 2 < 2
Buffer 25 MB 25 MB 50 MB
Max. Power (W) 650 650 650
Applications ACI Leaf ACI Leaf ACI Leaf
FEX Support Yes Yes Yes
BRKARC-3454 22
* Verify model-specific compatibility through DataSheets / Release Notes / Hardware Installation Guides

23. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FEX Scalability
+
16-64
FEX
Cisco Parent Switch
Cisco Nexus® 2000 FEX
Supports 3000+ ports per single point of management
Nexus
5500
Nexus
5600
Nexus
5696
Nexus
7000
Nexus
7700
Nexus
9300
Nexus
9500
L3 16 24 32 64 64 16 32
L2 24 24 48 64 64 16 32
23

24. Platform Overview:
Nexus 2000 / FEX

25. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 / FEX – Naming Convention
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
N2K-C2232PP-10GE
1–3: Cisco Nexus 2000 Series Identifier
4: Dash
5: Chassis
6: Series Number
7: Series Generation
8–9: Number of Ports
10: Host Physical Interface Type (P = Requires SFP+, T = Copper, UP = Unified Port, F = Includes FET)
11: Uplink Physical Interface Type (P = Requires SFP+, Q = QSFP, T = Copper, F = Includes FET, M = Module)
12: Dash
13–16: Host Connection Speed, Enhanced model or Bundle identifier (-BUN)
BRKARC-3454 25

26. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
History of Cisco Nexus 2000 Series
Simplified Operations with Revolutionary Scale
N2K-C2248TP-E-1GE
(Enhanced Buffers)
N2K-C2232PP-10GE
N2K-C2232TM-E-10GE
N2K-C2248PQ – 10GE
(QSFP Uplink)
Blade FEX
SFP/SFP+ FEX
B22 Dell FEX
1/10G FEX for
Dell Blade servers
B22 FTS FEX
1/10G FEX for
Fujitsu Blade serversB22 HP FEX
1/10G FEX for
HP Blade servers
1/10G FEX for
IBM Blade servers
B22 IBM
N2K-C2348UPQ-1/10GE
(40GE Uplink, Enhanced Buffers)
N2K-C2332TQ-1/10/40GE
(40G Uplink, Enhanced Buffers)
BRKARC-3454 26
N2K-C2348TQ / TQ-E
(40G Uplink, Enhanced Buffers)
Base-T FEX

27. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 100M/1GBASE-T
Nexus 2224TP Nexus 2248TP Nexus 2248TP-E
Form Factor (RU) 1 1 1
Network Interfaces 2 (SFP+) 4 (SFP+) 4 (SFP+)
Host Interfaces 24 48 48
Performance 88Gbps 176Gbps 176Gbps
FCoE Support No No No
Buffer 8 MB 8 MB 32 MB
Max. Power (W) 80 W 95 W 95 W
Oversubscription 1.2:1 1.2:1 1.2:1
Nexus Parent Switches Nexus 5000, 6000, 7000, 9000 Nexus 5000, 6000, 7000, 9000 Nexus 5000, 6000, 7000, 9000
BRKARC-3454 27

28. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 1/10GBASE-T
Nexus 2232TM Nexus 2232TM-E Nexus 2332TQ Nexus 2348TQ/E*
Form Factor (RU) 1 1 1 1
Network
Interfaces
8 (SFP+) 8 (SFP+) 4 (QSFP+) 6 (QSFP+)
Host Interfaces 32 32 32 48
Performance 560Gbps 560Gbps 1.2Tbps 1.2Tbps
FCoE Support No No No Yes (FCoE)
Buffer 8 MB 8 MB 32 MB 32 MB
Max. Power (W) 350 W 240 W 350 W 350 W
Oversubscription 4:1 4:1 2:1 2:1
Nexus Parent
Switches
Nexus 5000, 6000, 7000,
9000
Nexus 5000, 6000, 7000,
9000
Nexus 5000, 6000,
7000, 9000
Nexus 5000, 6000, 7000,
9000
* Some N2348TQ-E items are on the roadmap
BRKARC-3454 28

29. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 10G (SFP+)
Nexus 2232PP Nexus 2248PQ Nexus 2348UPQ
Form Factor (RU) 1 1 1
Network Interfaces 8 (SFP+) 4 (QSFP+) 6 (QSFP+)
Host Interfaces 32 48 48
Performance 560Gbps 960Gbps 1.2Tbps
FCoE Support Yes Yes Yes
Buffer 8 MB 32 MB N2H, 8 MB H2N 32 MB
Max. Power (W) 210 W 175 W 350 W
Oversubscription 4:1 3:1 2:1
Nexus Parent Switches
Nexus 5000, 6000, 7000, 9000
UCS FI
Nexus 5000, 6000, 7000, 9000 Nexus 5000, 6000, 7000, 9000
BRKARC-3454 29

30. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FANSIDEOFFEXPORTSIDEOFFEX
Nexus 2348UPQ Overview
• 6x40G QSFP+ Ports
• 48x1/10GSFP+ Ports
Choice of Port Side
Intake/ExhaustAirflow
Redundant1+1
AC/DC Power Supplies Unit
RedundantN+1 Fan
Modules
• 48 Fixed 1/10G SFP+ Unified Capable Ports.
• Traditional Ethernet or FCoE
• 48 Unified Ports provide up to 16GFC
• 32M of Bufferwith Lower Oversubscriptionratio of 2:1
• Parent Switch – N5k, N6k, N7k, N9k.
BRKARC-3454 30

31. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2348UPQ Buffer Allocation
• Total 32MB shared by 6 NIF(Network
Interface) and 48 HIF(Host Interface)
• Dedicated buffer pool and shared buffer
pool Each of the NIF or HIF can manage
all 32MB buffer
• Queue-limits can be used to control
buffer-usage by bursty flows.
NIF 1-6x40GE
HIF 1-48
dedicated
shared
32MB
BRKARC-3454 31

32. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 Internal Architecture
Nexus 2248PQ vs. Nexus 2348UPQ
QSFP QSFP QSFP QSFP
Uplink/BufferUplink/Buffer
Port 1 - 24
(Ethernet/FCoE)
ASIC A
Port 25 - 48
(Ethernet/FCoE)
ASIC B
4x 40Gbe QSFP+
(4x10G-Mode only)
Port 1 - 48
(Unified Ports: Ethernet/FCoE/FC)
QSFP QSFPQSFP QSFP QSFP QSFP
3rd Generation ASIC
6x 40Gbe QSFP+
(40G or 4x10G-Mode)
BRKARC-3454 32
* For all N23XX : max. 8 x 10GE Uplink Port-Channel bundle

33. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Nexus 2348UPQ
Compared with other Nexus 2000 1/10GE Fabric Extender
Model Nexus 2232PP Nexus 2248PQ Nexus 2348UPQ
Host Ports 32x SFP/SFP+ (1/10G) 48 x SFP/SFP+ (1/10G)
48x SFP/SFP+ (1/10G)
(true 40G or 4x 10G-Mode)
Uplink Ports 8x 10GbE SFP+
4x 40Gbe QSFP+
(4x 10G-Mode only)
6x 40Gbe QSFP+*
(true 40G or 4x 10G-Mode)
FCoE ✔ ✔ ✔
Buffer 157.5KB per port
157.5KB per port
2x 16M Shared Buffer (Uplink)
32MB Shared Buffer
BRKARC-3454 33
* For all N23XX : max. 8 x 10GE Uplink Port-Channel bundle

34. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric Extender 100M/1G/10G Copper
Nexus 2348TQ-E
48 port 3rd Generation Fabric Extender
• Cost optimized version of N2K-C2348TQ.
• Identical to N2K-C2348TQ in features.
• 48 x 100M/1G/10G host/downlink interfaces
• Large 32MB Buffer
• Supports FCoE for LAN/SAN convergence.
48 100/1000 RJ45 Downlinks
BRKARC-3454 34
Nexus 23XX FEX Uplink support
Max. 6 x 40GE Uplinks
or
Max. 8 x 10GE Uplinks
(throught breakout cables)

35. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric Extender 100M/1GE
Nexus 2248TP-E-GE Overview
• 48 ports 100M/1G-BaseT
Enhanced Fabric Extender
• 48x 100/1000M host/downlink
interfaces.
• 4x 10GE on network/uplink interfaces
• Upstream N5K supports various FEX
(mix-and-match)
• 6 user-defined QoS queues (8 QoS
queues)
• Design scenario:
• High density 100M/1G access
• Cost effective 100M/1G solution
• Buffer optimized for specific Data Center
workloads such as Big Data, Hadoop and
Distributed Storage
48 100/1000 RJ45 Downlinks
Key differences from N2248TP:
• Optimized Buffers
• Enhanced Debug counters
BRKARC-3454 35

36. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2248TP-E Buffer Allocation
• Total 32MB shared by 4 NIF(Network
Interface) and 48 HIF(Host Interface)
• Dedicated buffer pool and shared buffer
pool. Size of each pool is configurable by
software
• Current 128KB/port dedicated at NIF.
Rest (around 26MB) is shared
• Each of the NIF or HIF can manage all
32MB buffer
NIF 1-4x10GE
HIF 1-48
dedicated
shared
32MB
BRKARC-3454 36

37. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Bigdata and Hadoop
Cisco Unified Fabric
NIF 1-4x10GE
HIF 1-48
dedicated
shared
N2248TP-E
Cloudera
certified !!
N2248-TP-E
http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/white_paper_c11-690561.html
32MB Buffer
BRKARC-3454 37

38. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
32 ports 1/10G Fabric Extender
• 32x 1/10GE SFP+ host interfaces
• 8x SFP network interfaces
Design Scenario
• 1/10G Server Connectivity (SFP+ ToR)
• Connectivity Flexibility
• Predictable low latency
Nexus 2232PP-10GE Fabric Extender
8 SFP+ Uplinks
32 1G/10GE SFP+ Downlinks
BRKARC-3454 38

39. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2232TM-E
What is same as 2232TM?
• Form Factor, ports, power supplies, etc.
• Cabling options and distance supported
• Port to port latency
What is different?
• A newer and better 10GBase-T PHY (40nm)
• Better BER characteristics expected – qualification underway
• Reduction in power consumption
32 1G/10GT Downlinks
8 x 10GE SFP+ Uplinks
BRKARC-3454 39

40. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• IEEE 802.3.an standard
• Structured Copper Cabling with RJ45 connectors
• Supports 10Gbps throughput for up to 100 meters or 330 feet
• Leverages Category 6, Category 6A, and Category 7 type cabling
Cable Transceiver
Latency
PHY Power
(each side)
DistanceTechnology
Cat6a/7
Cat6a/7
~3ms
~3ms
~6W
~4W
100m
30m
10GBASE-T
– 65nm
Cat6a/7
Cat6a/7
~2.5ms
~2.5ms
~5W
~3W
100m
30m
10GBASE-T
– 40nm
Nexus 2232TM-E 10GBASE-T
BRKARC-3454 40

41. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
48 ports 10/1G Fabric Extender
• 4x QSFP (16x10GE SFP+) on network interfaces
• Feature parity with Nexus 2232PP
• Additional uplink buffers (2x16MB)
Design Scenario
• High Density 10GE SFP+ ToR
• Connectivity Flexibility
• Predictable low latency
48 1G/10GE SFP+ Downlinks
4 QSFP+ Uplinks
Nexus 2248PQ-10GE Fabric Extender
BRKARC-3454 41

42. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 Overview
2348UPQ/2348TQ PPT
• Redundant 1+1 AC/DC Power Supplies
• Redundant N+1 Fan Modules
• Color coding for both power supplies and Fan for port side exhaust
and port side intake airflow
Port Side Exhaust
Port Side Intake
BRKARC-3454 42

43. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 Airflow
Port Side Exhaust
(Hot Aisle-Server Rack Airflow Alignment)
Port Side Intake
(Cold Aisle-Network Rack Airflow Alignment)
BRKARC-3454 43

44. Features deep-dive
and Packet Walk

45. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 5000/6000 and 2000
Packet Forwarding Overview
Unified
Crossbar
Fabric
Ingress
UPC
Egress
UPC
Nexus 2000
FEX ASIC
1. Frame received on
N2K HIF port
1
2 3 4 5
6
2. Nexus 2000
appends VNTag and
forwards frame to
fabric uplink
3. Nexus 5000 UPC
performs ingress
forwarding and
queuing
4. If required
egress queuing
and flow control
5. Nexus 5000 UPC
appends destination
VNTag and forwards
frame on fabric link
6. VNTag
stripped and
frame forwarded
out on N2K HIF
port
Nexus 5000/6000 Nexus 2000Nexus 2000
Nexus 2000
FEX ASIC
BRKARC-3454 45

46. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 Fabric Extender (FEX) VNTag
Logical Interface
(LIF) on the
ingress UPC is
used to forward
the packet
N2K ASIC maps
specific VNTag to
HIF interface
Packet is forwarded
over fabric link using
a specific VNTag
• Nexus 2000 Fabric Extender operates as a remote line card and
does not support local switching
• All forwarding is performed on the Nexus 5500/5600/6000 UPC
VNTag is a Network Interface Virtualization (NIV) technology that
‘extends (Logical Interface = LIF) to the Nexus 2000 VIF referred
to as a Host Interface (HIF)
• VNTag is added/removed to the packet between Fabric
Extender and Nexus 5500/5600/6000/7000/9000
• VNTAG is a 6Bytes Virtual NIC Tag used for forwarding
decisions HIF
LIF
VNTAG Ethertype
source virtual interface
destination virtual interfaced p
l Frame
Payload
CRC[4]
VNTAG[6]
SA[6]
DA[6]
802.1Q[4]
BRKARC-3454 46

47. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 Port Channels
Nexus 2348/2248 Port Channels
• Nexus 2000 series FEX support local port channels
• A local port channel on the N2K is still seen as a single
extended port
• Extended ports are each mapped to a specific VNTag
• HW hashing occurs on the N2K ASIC
• Number of ‘local’ port channels on each N2K is based on
the local ASIC
• 22xx – Support up to 24 local port channels of up to 8
interfaces each as well as vPC (total of 2 x 8 = 16 ports)
• 23xx – Support up to 24 local port channels of up to 8
interfaces each as well as vPC (total of 2 x 8 = 16 ports)
1. Packet is
received and
lookup forwards
out a LIF (N2K)
interface
3. N2K ASIC
hashes locally and
transmits packet
on one HIF
interface
VMKVM
2. Packet is
forwarded over
fabric link using a
specific VNTag for
the destination
N2K LIF (port
channel)
BRKARC-3454 47

48. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Pinning a HIF to a Port Channel

Port Channel
• All server ports are pinned To the
Port Channel
• Automatic load balancing
in case of a link failure
• Definite traffic flow path
visibility and dedicated
bandwidth
BRKARC-3454 48

49. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FCoE and Unified Ports
• Fibre Channel over Ethernet
(FCoE) allows encapsulation and
transport of Fibre Channel traffic
over a shared Ethernet network
• SAN “A” / “B” isolation is maintained
across the network
• Unified Ports may be configured to
support either native Fibre Channel
or Ethernet
Nexus 2000 flexible definition of port function
Fibre
Channel
Traffic
Ethernet
F
C
Fibre Channel
Disk Array
Servers with Converged
Network Adapters (CNAs)
Nexus 5500 Ethernet and
Fibre Channel Switches
SAN
A
SAN
B
FCoE Links
BRKARC-3454 49

50. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 SPAN
• FEX Host ports CAN be SPAN source
ports
(Except with F2 Line card in Nexus 7000
parent switch)
• FEX Fabric ports CAN be SPAN source
ports with all parent switches
• FEX Host ports CANNOT be SPAN
destination port with any parent switch
……
Sender Host
Receiver Host
Receiver Host
✔ ✖
✔
BRKARC-3454 50

51. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 QoS
Tuning the Port Buffers
• Each Fabric Extender (FEX) has local port buffers (FEX leverages
a shared memory model)
• You can control the queue limit for a specified Fabric Extender for
egress direction (from the network to the host)
• You can use a lower queue limit value on the Fabric Extender to
prevent one blocked receiver from affecting traffic that is sent to
other non-congested receivers ("head-of-line blocking”)
Gen 2 UPC
Unified Crossbar Fabric
Gen 2 UPC
Nexus 2000
FEX ASIC
40G Fabric
1G Sink
10G Source (NFS)
# Disabling the per port tail drop threshold
N5672UP-1(config)# system qos
N5672UP-1(config-sys-qos)# no fex queue-limit
N5672UP-1(config-sys-qos)#
# Tuning of the queue limit per FEX HIF port
N5672UP-1(config)# fex 100
N5672UP-1(config-fex)# hardware N2248T queue-limit 356000
N5672UP-1(config-fex)# hardware N2248T queue-limit rx/tx ?
<CR>
<2560-652800> Queue limit in bytes
10GNFS
BRKARC-3454 51

52. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Speed mismatch between 10G NAS and 1G server requires
QoS tuning
• 32MB shared buffer to handle larger traffic bursts
• Hadoop, AVID are examples of bursty applications
• You can control the queue limit for a specified Fabric
Extender for egress direction (from the network to the host)
N5672UP-1(config-fex)# hardware N2348UPQ queue-limit 4000000 rx
N5672UP-1(config-fex)# hardware N2348UPQ queue-limit 4000000 tx
N5672UP-1(config)#interface e110/1/1
N5672UP-1(config-if)# hardware N2348UPQ queue-limit 4096000
VM
#4
VM
#3
VM
#2
NAS
iSCSI
10G Attached Source (NAS Array)
1G Attached Server
Nexus 2348UPQ
32MB Shared Buffer
Tune 2348UPQ to support a extremely large
burst (Hadoop, AVID, …)
10G
NFS
BRKARC-3454 52

53. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 Offload
CDP
LLDP
LACP
DCBX
QoS
• Efficient utilization of Nexus parent switch CPU
• TCAM entries are installed on corresponding FEX
ASICs
• QoS offload is disabled by default
N5672UP-1(config)# fex 100
N5672UP-1(config-fex)# hardware n2348uPQ qos-policy-offload
BRKARC-3454 53

54. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 QoS Offload Supported
• N2224TP Fabric Extender 24x1G 2x10G SFP+ Module
• N2232PP Fabric Extender 32x10G SFP+ 8x10G SFP+ Module
• N2232TM Fabric Extender 32x10GBase-T 8x10G SFP+ Module
• N2248T Fabric Extender 48x1G 4x10G SFP+ Module
• N2248TP E Fabric Extender 48x1G 4x10G SFP+ Module
• N2248PQ Fabric Extender 48x10G SFP+ 16x10G SFP+ Module
• N2232TM-E Fabric Extender 32x10GBase-T 8x10G SFP+ Module
• N23xx Fabric Extender 48x10G SFP+ 6x40G QSFP Module
BRKARC-3454 54

55. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 STP Design Considerations
N5672UP-1#show spanning-tree int eth100/1/1 detail
Port 1025 (Ethernet100/1/1, vPC) of VLAN0190 is designated forwarding
Port path cost 1, Port priority 128, Port Identifier 128.1025
Snip->>>
The port type is edge by port type edge trunk configuration
Link type is point-to-point by default
Bpdu guard is enabled
Bpdu filter is enabled by default
BPDU: sent 11, received 0
• N2K HIF ports have BPDU Guard enabled by default (it is not
possible to disable currently)
• If a BPDU is received port will transition to err-disable
state
• Global BPDU Filter compliments BPDU Guard
• On link up port will send 11 BPDUs and then stop (in order to
reduce CPU load)
EE E
1. X-Connected
patch cable
3. BPDU Guard
err-disables
edge port and
prevents loop
2. BPDU Sent on
Link-Up
4. BPDU are not
sent once link is
up and active
BRKARC-3454 55

56. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 : Routing Protocol Neighborships
• Routing Protocol neighborship across FEX requires Priority
queueing for Control Plane traffic at FEX HIF, among others.
• Currently this feature is only Supported on Nexus 7k starting
6.2(2), except for N2248PQ.
Note:
Nexus 5000/6000/9000 Series switches do NOT support this.
It is not recommended to connect high performance / critical
servers to a FEX, rather use the Parent Port.
On the N7k-FEX, this option may be used for non-critical low
bandwidth devices that need routing neighborships.
BRKARC-3454 56
HIF
Router
Routing-Protocol
Neighborship

57. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FCoE Deployment
FEX as Unified TOR
Nexus 5K
FC
Ethernet
Converged
FCoE
LAN
FC/FCoE SAN
Nexus 2K
B22 Dell FEX
1/10G FEX for
Dell Blade servers
B22 HP FEX
1/10G FEX for
HP Blade servers
1/10G FEX for
FTS Blade servers
B22F FEX
B22 IBM FEX
1/10G FEX for
IBM Blade servers
• The fabric extenders (FEX) can
be used at the ToR
• ToR FEX connect to the end or
row switches which manages
the FEX
• FEX work like external line cards
to the end or row switches
• The EoR switches splits the
Ethernet and FC traffic
BRKARC-3454 57

58. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 7K - FEX FCoE Features Update
NXOS 7.3(0)N1(1) introduces
support for:
• FCoE over FEX with F3 (N2K-
C2348UPQ-10GE, B22HP, N2K-
C2232PP-10GE)
• F3 FCoE support for FEX with
physical port vPC and vPC+
BRKARC-3454 58

59. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FC Ports on Nexus 2348UPQ UP FEX
• 6x40G QSFP+ Ports
• 4x10G SFP+ Ports Flexibility
• 48x1/10GSFP+ Ports
or 16G FC
• 48 Fixed 1/10G SFP+ Unified Capable Ports supporting Ethernet or
FCoE or FC
• Unified Ports provide up to 24x 16GFC ports or 48x 8GFC ports
• 32M of Bufferwith Lower Oversubscriptionratio of 2:1
• Parent Switch for Ethernet/FCoE:Nexus 5K, 6K, 7K and 9K
• Parent Switch for FC: Nexus 5600
• Shipped with Ethernet/FCoE support in July 2014
with 7.0(3)N1(1)
• Enabled FC support in March 2016 with 7.3(0)N1(1)
• Fibre Channel FEX supported number (ID) range is
from 100 to 130
• Support native Fibre Channel speed 2/4/8/16G
• Fibre Channel ports must be configured in groups of 4
ports and starts from beginning
First FEX Solution Designed for All Storage Connectivity
• Only F port is supported on FC-FEX
• NP Port type is not supported so NPV device cannot
connect to the FC FEX directly
• F-port trunking and F-port channel are not supported
on FC-FEX ports
• The default BB credits on each FC interface are 32;
and can be set to max 128
• Scale per parent switch: 24 FEX, MAX 96 FC ports
The FC ports on Nexus 2348UPQ are supported only on the Cisco Nexus 5600 switches.

60. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
 To support 16G FC, the fabric mode on the parent Nexus 5600 should be
in 40G mode.
 Only odd-numbered HIF ports can be configured as the 16G FC.
 When the odd-numbered HIF port is configured as 16G FC, the adjacent
even-numbered ports must be changed to the out-of-service state. Those
adjacent even-numbered ports next to it cannot be used.
 When the speed is set to Auto, the 16G host port comes up at 8G speed.
For the ports to come up at 16G speed, use the switchport speed 16000
command to explicitly set the speed to 16G.
Configure 16G FC Ports on UP FEX
stg-5672-16G-1(config-vsan-db)# interface fc121/1/11
stg-5672-16G-1(config-if)# switchport speed 16000
fc121/1/11: (error) Config not allowed - required to change the mode of the next port to out-of-service
stg-5672-16G-1(config-if)#
BRKARC-3454 60

61. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Extending ACI Policy Based Forwarding into Existing Data
Center Networks
N5K N2K N6K
AVS OVS
Extended ACI Fabric
AVSvSwitch
ACI Enabled
Remote N9K
• Extend Policy Based Forwarding
• Extend Visibility, Fault and Audit
• Automated Device Management for extended Fabric nodes
vSwitch HyperV
BRKARC-3454 61

62. Design and Topologies

63. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Single Homed Nexus, Single Homed FEX Attached Host
Nexus 7000
Nexus 5000
Nexus 2000
Nexus 7000/5000 Port Channel
• Simple and Easy to Configure
End to end
• Traffic is automatically re-distributed
across remaining NIFs upon failure
• Simple L2/L3 Flow visibility HIF/NIF to
next hop nexus switch
• Single link to host: no redundancy at
Server or FEX level.
UCS / Hosts
BRKARC-3454 63

64. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 7000
Nexus 5000
Nexus 2000
Nexus 7000/5000 Virtual Port Channel
• Easy to Configure FEX
• Network Resiliency and High availability
between N7k & N5k with vPC / double-sided
vPC.
• Simple L2/L3 visibility from HIF-NIF to N5k
• Single link to host: no redundancy at Server
or FEX level.
UCS / Hosts
Dual Homed Nexus, Single Homed FEX Attached Host
BRKARC-3454 64

65. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 7000
Nexus 5000
Nexus 2000
Nexus 7000/5000/2000 Virtual Port Channel
• High Availability and network
resiliency between N7k & N5k& FEX
• No Blocking ports in STP when using
vPC.
• L2/L3 Flows can be based on primary
or secondary vPC switch.
UCS / Hosts
Dual Homed Nexus/FEX,
Single Homed FEX Attached Host
BRKARC-3454 65

66. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Network Resiliency and High
availability across network and
compute
• Increased Network Bandwidth
• Active NIC teaming can also be
used for further options.
Nexus 7000
Nexus 5600
Nexus 2000
Nexus 7000/5600/2000 Virtual Port Channel
Active
Dual Homed Nexus/FEX, Dual Homed FEX AA-Host
UCS / Hosts
BRKARC-3454 66

67. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Dual Homed Nexus/FEX, Dual Homed FEX AS-Host
Nexus 7000
Nexus 5000
Nexus 2000
Nexus 7000/5000/2000 Virtual Port Channel
Active
Standby
• Network Resiliency and High
Availability across network and
compute
• Increased Network Bandwidth
• Active/Standby NIC teaming can
be dependent on CNA capabilities
UCS / Hosts
BRKARC-3454 67

68. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 (FEX) Straight-Through Deployment with VPC
• Port-channel connectivity from the server
• Two Nexus switches bundled into a vPC
pair
• Suited for servers with Dual NIC and
capable of running Port-Channel
Fex 100
Fabric Links
Fex 101
VPC
HIF HIF
S1 S2
68BRKARC-3454

69. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 (FEX)Active-Active Deployment with VPC
Fabric Links
HIFHIF
Fex 100
Fex 101
S1 S2
• Fabric Extender connected to two Nexus
5X00 / 6000 or Nexus 7000/7700
• Suited for servers with Single NIC or
Dual NIC not having port-channel
capability.
• Scale implications of less FEX per
system and less VPC
Note :
• This design is currently not supported on Nexus 9X00
69BRKARC-3454

70. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• N7X00 can support up to 64 FEXs
• N7X00 supports max. 15 Active-Active FEX in 7.2(0)D1(1)
• N7X00 supports max. 32 Active-Active FEX in 7.2(0)D1(1)
• Straight-Through FEX and Active-Active FEX cannot exist on the
same ASIC instance
• Layer 3 HIF ports are not supported with Active-Active FEX
• Active-Active FEX is not supported with vPC+
Nexus 2000 (FEX) Active-Active Scale &
Limitations (N7X00)
BRKARC-3454 70

71. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 (FEX) - Enhanced VPC
Fabric Links
Fex 100 Fex 101
S1 S2
• Port-channel connectivity to dual-homed
FEXs
• From the server perspective a single access
switch with port-channel support – each line
card supported by redundant supervisors
• Ideal design for a combination of single
NIC and Dual NIC servers with port-
channel capability
• Scale implications of less FEX per system
and less VPC
Note :
This design is currently not supported on N7000 / N7700 and N9X00
HIFHIF
71BRKARC-3454

72. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Physical Port vPC
• vPC configuration on a physical Layer 2 port as opposed to a port-channel
• Front panel ports and FEX ports connected to F2/F2e/F3 only
• Improves scaling as separate PC interface not created for single-link VPC leg
• Key benefit: more than 1000 host facing VPCs with FEX
vPC domain
FEX101 FEX102
e101/1/1 e102/1/1Po1
Port-channel vPC
VPC1 VPC1
Po1
vPC domain
FEX101 FEX102
e101/1/1 e102/1/1
Physical port vPC
VPC1 VPC1
interface e101/1/1
switchport
vpc 1
lacp mode active
72BRKARC-3454

73. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2000 (FEX) Active-Active (Unsupported)
Reference : Supported and Unsupported FEX Topologies: N5000/6000/7000/9000
http://www.cisco.com/c/en/us/support/docs/switches/nexus-2000-series-fabric-extenders/200363-Nexus-2000-Fabric-Extenders-Supported-Un.html
BRKARC-3454 73

74. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN Termination on TOR (Nexus 5600)
Supported Combinations
vtep vtep vtep
✔ ✔✔ ✔
*Switch Local Port-Channel (non VPC) is supported
- TOR to Hypervisor
- TOR to FEX
- FEX to Hypervisor
- TOR to FEX to Hypervisor
vtep vtep vtep
BRKARC-3454 74

75. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN Termination on TOR (Nexus 5600)
Supported Combinations
vtep vtep vtep vtep
✔ ✔✔ ✔
vtep vtep vtep
BRKARC-3454 75

76. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN Termination on TOR (Nexus 5600)
Supported Combinations
vtep
vtep
vtep vtep
vtep vtep vtep
✔ ✔✔ ✔
*Switch Local Port-Channel (non VPC) is supported
- TOR to Hypervisor
- TOR to FEX
- FEX to Hypervisor
- TOR to FEX to Hypervisor
vtep vtep vtep
BRKARC-3454 76

77. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN Termination on TOR (Nexus 5600)
Supported Combinations
vtep vtep vtep vtep
✔ ✔✔ ✔
BRKARC-3454 77

78. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN
Hypervisor
VXLAN Termination on TOR (Nexus 5600)
Unupported Combinations
vtep vtep
vtep
vtep vtep
vtep vtep vtep
✖ ✖✖ ✖
*FEX connectivity with combination of Hypervisor
(Host originated VXLAN) and Bare-Metal Servers
is not supported
vtep vtep vtep
BRKARC-3454 78

79. Configuration and Provisioning
Nexus 2000

80. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
N5672UP-1(config)# feature fex
N5672UP-1(config-if)# int port-channel100
N5672UP-1(config-if)# switchport mode fex-fabric
N5672UP-1(config-if)# fex associate 100
N5672UP-1(config-if)# vpc 100
N5672UP-1(config-if)# no shut
Configuration
Configuration
N5672UP-1(config)#show fex 100
FEX: 100 Description: FEX0100 state: Online
FEX version: 7.1(0)N1(1a) [Switch version: 7.1(0)N1(1a)]
Extender Serial: FOC1818R2M9
Extender Model: N2K-C2348UPQ, Part No: 73-15489-03
Pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth2/5
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po100 - Interface Up. State: Active
Eth2/5 - Interface Up. State: Active
Eth2/6 - Interface Up. State: Active
Verification
BRKARC-3454 80

81. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
N5672UP-1(config)# slot 101
N5672UP-1(config-slot)# provision model N2K-C2348UPQ
Pre-Provisioning Configuration
N5672-1(config-slot)# show run int eth101/1/1
!Command: show running-config interface Ethernet103/1/1
!Time: Mon Mar 23 10:58:01 2015
version 7.1(0)N1(1a)
interface Ethernet101/1/1
N5672-1(config-if)# int eth101/1/1
N5672-1(config-if)# switchport access vlan 190
N5672-1(config-if)# no shut
Provision
Configuration & Verification
BRKARC-3454 81

82. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
N5672UP-1(config-if)# sh queuing interface eth100/1/1
Ethernet110/1/1 queuing information:
Input buffer allocation:
Qos-group: 0
frh: 2
drop-type: drop
cos: 0 1 2 3 4 5 6
xon xoff buffer-size
---------+---------+-----------
19200 24320 48640
Queueing:
queue qos-group cos priority bandwidth mtu
--------+------------+--------------+---------+---------+----
2 0 0 1 2 3 4 5 6 WRR 100 1600
Queue limit: 66560 bytes
Queue Statistics:
queue rx tx
------+---------------+---------------
2 24434 3066978
Port Statistics:
rx drop rx mcast drop rx error tx drop mux ovflow
---------------+---------------+---------------+---------------+--------------
0 0 0 0 InActive
Ingress queue limit(Configurable)
Egress queue limit(Configurable)
Egress queues:
CoS to queue mapping
Bandwidth allocation
MTU
Drop counters
Nexus 2248TP-E Enhanced Counters
BRKARC-3454 82

83. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
N5672UP-1(config-if)# show queuing interface ethernet 100/1/1
if_slot 32, ifidx 0x1f630000
Ethernet100/1/1 queuing information:
Input buffer allocation:
Qos-group: 0
frh: 2
drop-type: drop
cos: 0 1 2 3 4 5 6
xon xoff buffer-size
---------+---------+-----------
5120 15360 71680
Queueing:
queue qos-group cos priority bandwidth mtu
--------+------------+--------------+---------+---------+----
2 0 0 1 2 3 4 5 6 WRR 100 1600
Queue limit: 2088960 bytes
Queue Statistics:
queue rx tx
------+---------------+---------------
2 126 49
Port Statistics:
rx drop rx mcast drop rx error tx drop mux ovflow
---------------+---------------+---------------+---------------+--------------
0 0 0 0 InActive
Priority-flow-control enabled: no
Ingress queue limit
Egress queue limit
Egress queues:
CoS to queue mapping
Bandwidth allocation
MTU
Drop counters
Nexus 2348UPQ Enhanced Counters
BRKARC-3454 83

84. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
N5672UP-1(config-if)# show interface ethernet 100/1/1 flowcontrol
--------------------------------------------------------------------------------
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
---------------------------------------------------------------------
Eth100/1/1 on off off off 0 0
Ethernet Flow Control (802.3x) Configuration
N5672UP-1(config-if)# interface ethernet 100/1/1
N5672UP-1(config-if)# flowcontrol send on/off
N5672UP-1(config-if)# flowcontrol receive on/off
• Why do I care to honor flow control?
BRKARC-3454 84

85. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
N5672UP-1(config)# feature fcoe (If FCoE not enabled)
FC license checked out successfully
fc_plugin extracted successfully
FC plugin loaded successfully
FCoE manager enabled successfully
FC enabled on all modules successfully
Enabled FCoE QoS policies successfully
N5672UP-1(config)# vlan 100
N5672UP-1(config)# fcoe vsan 100
N5672UP-1(config)# vsan database
N5672UP-1(config)# vsan 100
N5672UP-1(config)# fex 100
N5672UP-1(config)# fcoe
N5672UP-1(config)# interface vfc 1011
N5672UP-1(config)# bind interface ethernet 100/1/1
N5672UP-1(config)# switchport trunk allowed vsan 100
N5672UP-1(config)# no shut
FCoE configuration
• Is it not Fairly easy to configure FCoE?
BRKARC-3454 85

86. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
N5k-1#show fex 100 detail
FEX: 100 Description: FEX0100 state: Online
FEX version: 7.1(0)N1(1a) [Switch version: 7.1(0)N1(1a)]
FEX Interim version: 7.1(0)N1(1a)
Switch Interim version: 7.1(0)N1(1a)
Extender Serial: FOC1818R2M9
Extender Model: N2K-C2348UPQ, Part No: 73-15489-03
Snip->>>>>>
Logs:
03/18/2015 09:52:21.289192: Module register received
03/23/2015 09:18:18.516214: Module Offline
03/23/2015 09:18:18.561004: Deleting route to FEX
03/23/2015 09:18:18.568532: Module disconnected
03/23/2015 09:21:53.581415: Module register received
03/23/2015 09:21:53.583664: Registration response sent
03/23/2015 09:21:54.634249: create module inserted event.
03/23/2015 09:21:54.635044: Module Online Sequence
03/23/2015 09:22:03.006536: Module Online
BRKARC-3454 86
Configuration Verification

87. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKARC-3454 87
* - Verify model specific FEX support with ACI
Configuring FEX with ACI

88. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKARC-3454 88
* - Verify model specific FEX support with ACI
Configuring FEX with ACI

89. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKARC-3454 89
* - Verify model specific FEX support with ACI
Configuring FEX with ACI

90. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKARC-3454 90
* - Verify model specific FEX support with ACI
Configuring FEX with ACI

91. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKARC-3454 91
* - Verify model specific FEX support with ACI
Configuring FEX with ACI

92. Connectivity Options
Nexus 2000

93. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
10G SFP+ Connectivity 40G QSFP+ Connectivity
Nexus 2300 Connectivity Options
Nexus Parent Switch Nexus Parent Switch
Nexus 2300 Nexus 2300
BRKARC-3454 93

94. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus 2300 Connectivity Options
Network Interface (NIF)
• The Network Interface Fabric Ports can operate 10G/40G mode
• FET-40G, QSFP-Bidi, QSFP+, Active Twinax, including AoC cables can be used to connect to a parent switch
while operating in 40G mode
Host Interface (HIF)
• SFP+ Host Interface will support SR, LR, GLC-T, GLC-SX-MM, AoC
• Supported Twinax: 1, 3, 5M (passive) and 7, 10M (active)
*Check the parent switch documentation for the latest supported cable options Twinax Passive cable is
Not supported on NIF
BRKARC-3454 94

95. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
OM3-MM Fiber Cable
For QSFP-40G-SR4 and
QSFP-40G-CSR4
12-Fiber MPO Connector
• MPO/MTP connector with 12
pins
• Use 4 fiber pairs
− 4 TX and 4 RX allows for 40 GE
− 4 unused fibers in the center
BRKARC-3454 95

96. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
QSFP-40G-SR4
40G-SR4 40G-SR4
40G-SR4
10G-SR
10G-SR
10G-SR
10G-SR
MPO-12 MPO-12
MPO-12  LC Breakout
 Support 100M with OM3 MMF
and 150M with OM4 MMF
BRKARC-3454 96

97. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
QSFP 40G Bidirectional Transceiver – 40G
deployment with 10G cabling
 Cisco’s innovation has created a new QSFP pluggable 40Gbit/s
Bidirectional (BiDi) Transceiver that works with existing infrastructure:
Multi-mode fiber
– Using the existing 10Gbit/s Multimode infrastructure
– QSFP-40G-SR-BD has dual LC Connector
– 40Gbit/s BiDi transceiver has two 20Gbit/s channels each transmitted and
received simultaneously on two wavelengths
BRKARC-3454 97

98. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
QSFP-40G-CSR4
 Supports 300M with OM3 MMF
and 400M with OM4 MMF
 Compatible with Cisco QSFP-
40G-SR4
 Compatible with 10G SFP-SR
40G-CSR4 40G-SR4/CSR4
40G-CSR4
10G-SR
10G-SR
10G-SR
10G-SR
MPO-12 MPO-12
MPO-12  LC Breakout
BRKARC-3454 98

99. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
QSFP-40G-SR-BD
40G-BD 40G-BD
40G-BD
10G-SR
10G-SR
10G-SR
10G-SR
 Supports 100m with OM3 with
MMF and 150m with OM4
(OM4+) MMF
 LC duplex connectors
 Wavelength: ~850nm and
~900nm
 Pull-Tab Color: grey
Multimode
LC Duplex
Multimode
LC Duplex
BRKARC-3454 99

100. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Physical Connection Choices
QSFP-SR4/CSR4 QSFP-SR4/CSR4
1M, 3M, 5M Passive
7M, 10M Active
QSFP-SR4/CSR4
QSFP-4x10G
40G QSFP+ 40G QSFP+
QSFP-LR4 QSFP-LR4
40G QSFP+ 40G QSFP+
QSFP-SR-BD QSFP-SR-BD
40G QSFP+
40G QSFP+
40G QSFP+
40G QSFP+
40G QSFP+
40G QSFP+
10G SFP+
10G SFP+
BRKARC-3454 100

101. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FET-40G (aka QSFP+ FET)
• Low-cost QSFP optical transceiver connecting FEX to Cisco Nexus 6004
• Supported on Cisco Nexus 5500/N5600/6000/N7000/N9000 and Nexus 2248PQ-10G/Nexus 2300
• Interoperable with FET-10G
• Support for 100m distance with OM3
Cisco Nexus 5600/6000 Cisco Nexus 5600/6000
Cisco Nexus 2232PP and 2232TM-E 2232TM
Cisco Nexus 2248TP-E 2248TP
Cisco Nexus 2248PQ-10G/N2300
FET-40G
FET-40G
FET-10G
FET-40G
BRKARC-3454 101

102. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Convert 40GE Interface to Four 10 GE Interfaces
Convert the QSFP Interfaces to Four 10 GE Interfaces When Connecting to a SFP+ Port or to the QSFP Interfaces That
Operate in Four 10 GE Mode, such as the QSFP Uplink of N2248PQ-10G or the Cisco Nexus® 5500 QSFP GEM.
Apply global CLI to change interface types
to 10 GE
• Every three contiguous QSFP interfaces resides
on one Port ASIC.
• The port range specified in the CLI has to
include all ports on the ASIC
Power off the affected modules
• Every group of 12 QSFP interfaces is managed
as one module, even for the fixed interfaces
Power on the affected modules
1
N6004(config)# interface breakout slot
1 port 1-6 map 10g-4x
N6004(config)# poweroff module ?
<1-8> Please enter module number
N6004(config)# poweroff module 1
N6004(config)# no poweroff module 1
N6004(config)#
2
3
BRKARC-3454 102

103. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
QSA-QSFP+ to SFP Adapter
• Convert one QSFP port to one SFP+ port.
• Provides 1G/10G transceiver support
• Long reach connection between for QSFP+ devices and SFP+ devices
• Following 1G/10G transceivers are qualified with QSA in release
• 10G-LR
• 1G GLC-T, 1G SX, LX
• Requires a mode change on QSFP+ port to 4x10G
• Only first 10G interface can be used. Remaining three 10G shown as “Link Not Connected”
QSFP+ Adapter
1G SFP or 10G SFP+
Transceiver
BRKARC-3454 103

104. Summary
Nexus 2000

105. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Understand FEX Architectures & Capabilities
 Realize the benefits offered by a FEX design
 Evaluate FEX for YOUR network requirements.
Key Takeaways
Lower CAPEX
OPEX
Easy to Provision, manage, troubleshoot.
Lowest Cost option for connectivity
Unified
Network Flexible Ethernet, FCoE and FC solutions with FEX
Flexible and Scalable ToR design provides Large number of Ports,
with low number of management points.
Simplify your
Network
BRKARC-3454 105

106. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Acronyms and Abbreviations
• HIF – Host Interface
• NIF – Network Interface
• CIF – CPU Interface
• VNTag – Virtual NIC Tag
• H2N – Host to Network
• N2H – Network to Host
• ASIC – Application-Specific Integrated Circuit
• vPC – Virtual Port-Channel
• FEX – Fabric Extender
• VXLAN – Virtual Extensible LAN
• ACI – Application Centric Infrastructure
• APIC – Application Policy Infrastructure Controller
• OTV – Overlay Transport Virtualization
• STP – Spanning-Tree
• LPM – Longest Prefix Match
• SoC – System on Chip / Switch on Chip
• FC – Fiber Channel
• FCoE – Fiber Channel over Ethernet
• UP – Unified Port
• UCF – Unified Crossbar Fabric
• UPC – Unified Port Controller
• IGMP – Internet Group Management Protocol
• ALE – Application Leaf Engine
• NFE – Network Forwarding Engine
• TCAM – Ternary Content Addressable Memory
• FIB – Forwarding Information Base
• HPC – High-Performance Computing
• Mroute – Multicast Route
• LC – Line Card
• EOBC – Ethernet Out-of-Band Channel
• CPU – Central Processing Unit
BRKARC-3454 106

107. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
• Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.
BRKARC-3454 107

108. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
BRKARC-3454 108

109. Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016
11:30 am - 12:30 pm, In the Oceanside A room
What to expect from this innovation talk
• Insights on market trends and forecasts
• Preview of key technologies and capabilities
• Innovative demonstrations of the latest and greatest products
• Better understanding of how Cisco can help you succeed
Register to attend the session live now or
watch the broadcast on cisco.com

110. Thank you

112. Appendix

113. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Host to
Network
1
10G MAC Rx 10G MAC Tx
2
VNTag
Forwarding
802.1q
Rewrite
VNTag
Rewrite
Shared Packet Buffer 32MB
37 write
37 read
3
Input port
Accounting
Flow Cntr
4
Read Mux Tx Arb
Output Queue Time Stamp
5
6
Svif Check
and
ACL Data
HIF
7
10G MAC Tx
VNTag Rewr 802.1q Rewr
8
9
HIF
BRKARC-3454 113
Reference Slides

114. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Reference Slides1
Shared Packet Buffer 32MB
37 write
37 read
5
Input port
Accounting
Flow Cntr
4
3
Unicast FW
Multicast FW
HIF
10G MAC Rx 10G MAC Tx
VNTag
Forwarding
802.1q
Rewrite
VNTag
Rewrite
2
Read Mux Tx Arb
Output Queue
6
NIF
10G MAC Tx
VNTag Rewr 802.1q Rewr
8
7
9
BRKARC-3454 114
Network to
Host

115. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Typical UP FEX Topologies
Active-Active FEX
VP
C
VSAN A VSAN B
Nexus
5600
Nexus
2348UPQ
• FC-HIF ports configuration must be the same in both the
primary and the secondary VPC switches.
• For the Active-Active FEX, the port-conversion CLI should
be configured on both the vPC peer switches.
FPC FPC
Straight-Through FEX
Nexus
5600
Nexus
2348UPQ
VSAN A VSAN B
VP
C
• The Active-Active FEX is supported, but only one FC FEX
can be associated with a parent switch in the pair.
switch1(config)# fex 101
switch1(config-fex)# fcoe
40G FCoE 40GE
40G FCoE
10GE NIC + FC HBA
8/16G FC
10GE
10GE NIC + FC HBA
8/16G FC
10GE
10G FCoE CNA
BRKARC-3454 115
Reference Slides

116. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
 To support 16G FC, the fabric mode on the parent Nexus 5600 should be
in 40G mode.
 Only odd-numbered HIF ports can be configured as the 16G FC.
 When the odd-numbered HIF port is configured as 16G FC, the adjacent
even-numbered ports must be changed to the out-of-service state. Those
adjacent even-numbered ports next to it cannot be used.
 When the speed is set to Auto, the 16G host port comes up at 8G speed.
For the ports to come up at 16G speed, use the switchport speed 16000
command to explicitly set the speed to 16G.
Configuration of 16G FC Ports on UP FEX
stg-5672-16G-1(config-vsan-db)# interface fc121/1/11
stg-5672-16G-1(config-if)# switchport speed 16000
fc121/1/11: (error) Config not allowed - required to change the mode of the next port to out-of-service
stg-5672-16G-1(config-if)#
BRKARC-3454 116
Reference Slides

117. © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VFC Creation on 4-Port (HIF) VPC
N5600-1 N5600-2
Host
CNA
vPC
vfc1 vfc2 vfc3 vfc4
Switch(config)# int port-channel 500
Switch(config-if)# fcoe multi-vfc  enable the feature
N5600-1 N5600-2
FEX-1 FEX-2
Host
CNA
vPC
vfc1 vfc2 vfc3 vfc4
Supported from Cisco NX-OS release 7.2(0)N1(1)
Straight-Through FEX
• Only supported on N5600/N6000 platforms (Bigsur ASIC)
• Only supported on Nexus 2300 FEX (Tiburon ASIC)
• Not supported for NPV mode
BRKARC-3454 117
Reference Slides