Brkdct 3101

Nexus9000(Standalone)
Architecture And
Troubleshooting
Shridhar V. Dhodapkar –Technical Leader (Services)
CCIE 6367 (Routing & Switching)
BRKDCT-3101

Session Abstract
This session presents briefly the architecture of the latest generation
of Nexus 9000 Series Modular switches. Topics include supervisors,
fabrics, I/O modules, forwarding engines, and physical design elements, as
well as the Top of the Rack Nexus9300 Switches.
The session will also cover how to monitor the health of the system.
We will walk you through in depth troubleshooting Tools and Techniques.

Session Goal
• To provide an overall understanding of the Nexus 9000 switching
architecture, supervisor, fabric, and I/O module design, packet flows, and
key forwarding engine functions
• This session will introduce System Telemetry, Troubleshooting tool Kits
and troubleshooting case scenarios
• This session will not examine NX-OS software architecture or other Nexus
platform architectures

Related Sessions
BRKARC-2222 - Cisco Nexus 9000 architecture
BRKARC-3471 - Cisco NX-OS Software Architecture
BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches

• Introduction
• Architecture
• System Health check Telemetry
• Troubleshooting Toolkit
• Nexus 9000 Troubleshooting
• Common Link Layer Issues-L1
• Fabric Connectivity and
• In band
• L2/L3 Packet Forwarding
• vPC
• Nexus9000 Specific Limitation and Goodies
Agenda

Introduction-What is Nexus9000 Family ?
Nexus 9500 Series Switches Nexus 9300 Series Switches
Nexus9504/Nexus9508/Nexus9516 N9K-C9332PQ N9K-C9372PX N9K-C9372TX N9K-C9396

9500 Field Upgradeable Units (FRU)
• 9500 has the following modular components which can upgraded or
replaced in the field
• Supervisor
• Fabric Module
• Line Card
• System Controller
• Fan Tray
• Power Supply
• The Supervisor, System controller ,Fabric Module and LC have OBFL
(On-Board Failure Logging) for failure analysis
Nexus® 9508 Front View Nexus® 9508 Rear View

Nexus 9500 Platform FRU
Supervisor Module-What it is Role
• Redundant Half-width supervisor engine
• Common for 4-, 8-, and 16- slot chassis
• External Clock Input (PTP)
• Responsible for control-plane functions
System Controller-What it is Role
• Offload supervisor from internal device management tasks
• Central Point of Chassis Control
• EOBC Switch (Ethernet Out of Band Channel)
• EPC Switch (Ethernet Protocol Channel)
• Power Supplies via SMB (System Management Bus)
• Fan Trays

Nexus 9500 Platform Line Card
• I/O module with Merchant and
Merchant+ ASIC
• Have Various Forwarding Tables
• L2 Mac Table And L3 Host Table
• ACL and Buffers for Queuing
F
P
1
F
P
2
F
P
3
F
P
4
F
P
5
F
P
6
F
P
7
F
P
8
F
P
9
F
P
1
0
F
P
1
1
F
P
1
2
F
P
1
3
F
P
1
4
F
P
1
5
F
P
1
6
F
P
1
7
F
P
1
8
F
P
1
9
F
P
2
0
F
P
2
1
F
P
2
2
F
P
2
3
F
P
2
4
F
P
2
5
F
P
2
6
F
P
2
7
F
P
2
8
F
P
2
9
F
P
3
0
F
P
3
1
F
P
3
2
F
P
3
3
F
P
3
4
F
P
3
5
F
P
3
6
F
P
3
7
F
P
3
8
F
P
3
9
F
P
4
0
F
P
4
1
F
P
4
2
F
P
4
3
F
P
4
4
F
P
4
5
F
P
4
6
F
P
4
7
F
P
4
8
Note: Internal ports are called as Hi-Gig/HG ports
10G SFP+ Ports
40G
QSFP
HG
MUX1
HG
MUX3
FP
49
FP
50
FP
51
FP
52
Northstar 1
Warpcor
e
MF Port
7
-
5
2
-
0
3
1-
2
9
2
6-
2
4 T2
7
-
5
2
6-
2
4
0
-
2
3
-
5
6-
8
9-
1
1
FM4 FM3FM5FM6 FM2 FM1
HG
MUX4
HG
MUX2
HG
MUX5
HG
MUX6
MN Port
0 1
2 3
4 5
6 7
8 9
10
11
Northstar 2
MF Port
0
-
2
9-
1
1
MN Port
0 1
2 3
4 5
6 7
8 9
10
11
T2
7
-
5
2
-
0
3
1-
2
9
2
6-
2
4
HG
Line Cards
ASIC Name
NFE=Network Forwarding Engine-Trident 2(T2)
ALE=Application Leaf Engine-North Star(NS)
-Donner N9K-X9564PQ

Nexus 9500 Fabric Module
• Interconnect Line Card slots
• Installed at the rear of the chassis
• Leverages Broadcom Trident II ASICs
• Max 1.92 Tbps per line card slot (6 Fabric Cards)
• 960 Gbps per line card slot (3 Fabric Cards)
• All Fabric Cards are active and carry traffic
• Fan Tray requires Fabric Card to be present in even slot
Trident II
ASIC-NFE
Trident II
ASIC-NFE
32 x 40G
Hi-Gig2
32 x 40G
Hi-Gig2
Fabrics Modules

T2
Fabric 1
T2
320 Gbps
(8x 40Gbps)
T2
Fabric 2
T2
320 Gbps
(8x 40Gbps)
T2
Fabric 3
T2
320 Gbps
(8x 40Gbps)
T2
Fabric 4
T2
320 Gbps
(8x 40Gbps)
T2
Fabric 5
T2
320 Gbps
(8x 40Gbps)
T2
Fabric 6
T2
320 Gbps
(8x 40Gbps)
Line Card Slot
1.92 Tbps
• An 8-Slot chassis fabric module can provide up to 320Gbps to each Line Card slot
• With 6 fabric modules, each Line Card slot can have up to 1.92Tbps duplex
forwarding bandwidth
Data Plane Scaling for 8-Slot Chassis
1.60 Tbps
1.28 Tbps
960 Gbps
640 Gbps
320 Gbps
Nexus 9500 Fabric Module

NFE
ALE ALE
NFE
Fabric 1
Nx NFE
Fabric 2
Nx NFE
Fabric 3
Nx NFE
Fabric 4
Nx NFE
Fabric 5
Nx NFE
Fabric 6
Nx NFE
2 x 42 Gbps
2 x 42 Gbps
12 x 42 Gbps 12 x 42 Gbps
N = 1 for N9504
N = 2 for N9508
N = 4 for N9516
NFE
ALE ALE
NFE
12 x 42 Gbps 12 x 42 Gbps
Note: Internal ports are called as Hi-Gig/HG ports
Distributed Data Plane of Nexus 9500 Series Switches

Nexus9500 Series Line Card Summary
Information X9600 Series Line
Cards
X9500 Series Line
Cards
X9400 Series
ASIC Technology Merchant only
N9K-X9636PQ
Merchant+
N9K-X9564PX
N9K-X9564TX
N9K-X9536PQ
Merchant only
• N9K-X9432PQ
• N9K-X9464PX
• N9K-X9464TX
Number of ASIC 3 T2 2 T2 + 2 NS
2 T2 + 2 NS
2 T2 + 2 NS
2 T2 40 gig 32 Ports
1 T2 48 1/10 gig , 4
QSFP
Non Blocking Non Blocking Line rate > 200 byte
packet
Buffer Size 36 MB 104 MB 12 MB with one T2
24 MB with two T2

High Level Block Diagram-N9500
All PSU, SC, SUP, FM, and
LC plug into the same
Power Supply Interface

N9K-C9300 Series
• Fixed Chassis
• Port QSFP+ Uplink Module
• 1 RU or 2RU or 3RU
• AC/DC Power Supply
• Front-to-Back & Back-to-Front Airflow
• Latency: 1-2 usec
• Wire-Speed L2/L3 Forwarding
• Switch will not boot up without GEM
Expansion Module

Nexus 9300 Series Switch Summary
N9396TX/PX N93128TX N9372TX N9372TX N9372PX
NFE (BCM T2) 1 1 2 1 1
ALE ( NorthStar)/GEM GEM-1 NS GEM-1 NS No GEM-1
Donner
No GEM -1
Donner
No GEM- 1
Donner
Oversubscribed No 1.5:1 No No No
Line Rate Yes Yes (packets >
194-Bytes)
Yes Yes
QoS Classes 8 4 8 4 4
Buffer (MB) 36 (12*3) 104 (12*2+40*2) 24 (12*2) 104 (12*2+40*2) 104 (12*2+40*2)

High Level Block Diagram-N9300
Northstar
Egress
(12+12)x12
Ingress
(12+12)x12
BRCM Trident2
48 10G x 12 40G
CPU
2C 1.5GHz
DDR3
DIMM2
16GBTotal
PCIe
Trident II
ASIC
NorthStar
ASIC 1
Network Interfaces
12 x 40G
Hi-Gig2
12 x 40G
Ethernet
Front Panel 48x 1GE/10GE Ports
GEM 4x 40GE QSFP+ Uplinks
1000BaseT
Mgmt Port
2 USB
Ports
eUSB
Boot Flash
12C
• The last 2/3 numbers stand for
total bandwidth in Gigabits
• 93128 – 128G (96 x 10G + 8 x 40G)
• 9396 – 96G (48 x 10G + 12 x 40G)
• 9372 – 72G ( 48 x 10G + 8 x 40 G)

T2-NFE
Parser
L2/L3
Lookup &
forwarding
I-ACL
Traffic
Classification
& Remarking
Ingress
Accounting &
Policing
Packet
Modification
E-ACL
Output Q
& Shaping
EoQALE-NS
Network Interface
Fabric Module
L3 LPM Lookup & Forwarding
T2-NFE
Parser
L2/L3
Lookup &
forwarding
I-ACL
Traffic
Classification
& Remarking
Ingress
Accounting &
Policing
Packet
Modification
E-ACL
Output Q
& Shaping
EoQALE-NS
Network Interface
Ingress Line Card Egress Line Card
Nexus9500 Unicast Packet Flow
Parse the first 128
Byte and extract
header info
L2/L3 Lookup in
MAC Table and IP
Host Table
Classify traffic
based on 802.1q
COS, IP Pres,
DSCP &ACL
Remark if needed
Egress Line card
sends packet to
egress port based
on DMOD/DPORT
Class-based output
queues. Support 6
classes including
control traffic class
Additional buffer is
available for
extended out put
Ques EoQ
Fabric Module
Performs L3 LPM
lookup and resolves
Egress port and
next-hop
OOBFC
Signaling
OOBFC
Signaling

N9K-C9300 High Level Block Diagram
HiGiG2 Interface on T2
MACF ports on the GEM and to MACN ports
(16 x 10G) x 3 =
480G FP Bandwidth
(12 x 40G) = 480G
Bandwidth to GEM
Module
(12 x 40G) = 480G FP
Bandwidth Uplink Ports
MACN ports.
(16 x 10G) x 3 =
480G FP Bandwidth
(12 x 40G) = 480G
Bandwidth to GEM
Module

Main Features of Trident2 1280Gbps Switch ASIC
Packet
Buffer
Content aware Engine
DCB Engine
L2
MAC
L3
Route
L2/L3
Multicast
128 Integrated SerDes
Dynamic Memory
Manager
L2/L3
Processing
Host IF
Counters
128 SERDES@10Gbps
OR
32 SERDES@40Gbps
Features Information
Maximum IO and Core bandwidth 1280G
MAC(L2) Entries 32K min -288K max
L3 Hosts IPv4:16K min-112Kmax
IPv6:8K min-56 max
L3 Multicast Group 8K
Virtual Ports 16K
Maximum number of Physical ports 104

North Star
Features Information
Support Mixed Speed but in Fixed
configuration.
Network Interface:12 Ports Fabric
Interface: 12 40 Gig
Forwarding 720Mpps lookup rate on Ingress
Datapath
720Mpps lookup rate at Egress
Datapath
Shared Memory Subsystem
Ingress Path Buffer
Egress Path Buffer
10 Mbytes
30 Mbytes
Maximum number of Physical ports 24

Broadcom Unified Forwarding Table
SUPPORTED COMBINATIONS
T2 has the following Unified Forwarding Table:
Mode L2 L3 Host LPM
0 288K 16K 16K
1 224K 56K 16K
2 160K 90K 16K
3 98K 122K 16K
4 32K 16K 128K

Routing Mode for Nexus9300
LPM Routing Mode Broadcom T2
Mode
CLI Command
Default system routing mode 3
ALPM Routing mode 4 System routing max-mode l3
N93K#show system routing mode
Configured System Routing Mode: Hierarchical
Applied System Routing Mode: Hierarchical (Default)
N93K#show hardware internal forwarding table utilization module 1
Max Host Route Entries (shared v4/v6): 124928
Max LPM Table Entries : 16384

Routing Mode for Nexus9500
show hardware internal forwarding table utilization mod 1
Max Host Route Entries (shared v4/v6):16384
Max LPM Table Entries : 131072
show hardware internal forwarding table utilization mod 21
Max Host Route Entries (shared v4/v6): 0
Max LPM Table Entries :0
LPM Routing Mode Broadcom T2 Mode Cli Command
Default System routing mode 3 (For Line card)
4 (For Fabric Module)
Max-host routing mode 2--Line Card- V6 in LPM
3--For Fabric Module
System routing max-mode host
Nonhierarchical routing mode 3--For Line Card
4--With max-l3-mode option
For Line card
No Routes on Fabric Module
System routing non-hierarchical
Option [max-l3-mode]
64-bit ALPM routing mode Sub mode of mod 4 for
Fabric modules
System routing mode hierarchical
64b-alpm
Non hierarchical
routing mod

ACL TCAM TABLE
Characteristic
• Ingress ACL: 4K TCAM entries - 4x 512 banks + 8x 256 banks
• Egress ACL: 1K TCAM entries - 4x 256 banks
• Each ACL type needs its own dedicated bank/banks
• IPv4, IPv6 or MAC each needs dedicated bank/banks
• MAC-ACL IPv6 & any QOS needs double-width entries, which means needs at least 2 banks
• VACL is programmed symmetrically in both egress and ingress ACL
Interface
Type
Ingress ACL Egress ACL
SVI TCAM Shared TCAM Not shared
L3 TCAM Shared TCAM Shared

ACL Characteristics
• Atomic/hitless update of existing applied ACL while modified
• Temporary label swap (no use of default-result)
• Two acl copies in tcam, if there is no enough space, process fails
• ACL TCAM banks chaining not supported
• L4OPs/LOUs only used for expansion beyond 5 lines, configurable
• 10 L4op per acl limit
• Specific applications (dhcp, bfd) may install their own ACLs which must merge
with user configured racl, vacl, pacl

TCAM Carving for Nexus 9000
TCAM Region-N9500 Size Per Region
IPV4 RACL 1536
IPv4 L3 QOS 256
Ingress System 256
SPAN 256
Ingress CoPP 256
Redirect 256
vPC Convergence 512
Egress IPv4
RACL
768
Egress System 256
256
Ingress
Egress
3X512
256
256
256
256
3X256
512
256
TCAM Region-N9300 Size Per Region
IPv4 PACL 512
IPV4 VACL 512
IPV4 RACL 512
IPv4 Port QOS 256
Ingress System 256
SPAN 256
Ingress CoPP 256
Redirect 256
vPC Convergence 512
Egress IPv4 RACL 256
Egress IPv4 VACL 512
Egress System 256
256
Ingress
Egress
512
256
512
256
256
512
256
512
512
256
256

ACL TCAM Default Region and Carving
• TCAM Banks will first get assigned to Feature which has largest region.
• Next TCAM Bank will get assigned to Feature which need double Width.
• TCAM Carving requires Line Card/TOR reload to take effect
• To read current TCAM allocation
N9K#Show system internal access-lists global
• To reconfigure TCAM Region
N9K(config)hardware access-list tcam <feature name> <size>

Buffer And Queuing-T2
Shared
Buffer
12 MB
Control
Default
OOBFC
• T2 has 12 Mbytes of
Buffer shared by all
ports for all Traffic
Shared
Buffer
12 MB
Control
Default
Module with T2 only Module with T2 And NS
OOBFC: Out of band flow control unicast service pool
• Shared buffer divided Into
Control and default service
pool if module is T2 only
• Shared buffer divided
into Control, default and
OOBFC service Pool if
Module is T2 and NS
based

Buffer And Queuing-North Star
Trident II
ASIC
NorthStar
ASIC 1
12 x 40G
Hi-Gig2
12 x 40G
Ethernet
Shared
Buffer
Control
Default
SPAN
• North Star has 40 Mbytes of Buffer
• Divided in to Three Pool
• Control , SPAN , Default
10 MB
Buffer
20 MB
Buffer
10 MB
Buffer

Buffer Boost Function with T2 and NS
• Buffer boost is function which allow T2 to use extra
buffer of NS
• When Buffer boost is enabled on a port , T2 Local switch
traffic is Sent to NS for extra buffer space-
• When Buffer boost is disabled on a port, T2 local traffic
to this port remains local on this NFE
• Buffer Boost is enabled by default and can be disabled
on a per port basis
1/10GE 1/10GE 1/10GE
ALE-NS
NFE
T2
Network Interface
10 MB
Buffer 10 MB
Buffer
20 MB
Buffer
12 MB Buffer Shared by all
ports
Fabric Module
1/10GE

Most Common System Health Check
• What is the Best Recommended NX-OS Release
• CPU & Memory usage
• Inter Process Messaging usage-MTS
• Traffic Stats/Drop To CPU
• CoPP/Hardware Rate Limiter Drops
• Ethernet Out of Band Drops/Error
• Instant Buffer usage Stats
• FATAL System Errors
• Interface Errors for STP/Error disable
• Inter ASIC Utilization
• Hardware Capacity Check
• Consistency Checkers –Various Tables
• GOLD Diagnostic Checks
• Sev1/2 Syslog

Platform Series Minimum Release Recommended Release
Cisco Nexus 9500 6.1(2)I2(2b) 6.1(2)I3(4a)
Cisco Nexus 9300 6.1(2)I2(2b) 6.1(2)I3(4a)
General Recommendation for New and Existing
Deployments
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/recommend
ed_releaseb_Minimum_and_Recommended_Cisco_NXOS_Releases_for_Cisco_Nex
us_9000_Series_Switches.html
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-
x/scalability/guide_34/b_Cisco_Nexus_9000_Series_NXOS_Verified_Scalability_Guide_61
2I34/b_Cisco_Nexus_9000_Series_NXOS_Verified_Scalability_Guide_612I34_chapter_01.
html
• Software Recommendation
• Verified Scale limits for different features and protocol for each release

CPU & Memory Usage
N9K#show system resources
Load average: 1 minute: 0.00 5 minutes: 0.03 15 minutes: 0.05
Processes : 432 total, 1 running
CPU states : 2.76% user, 0.75% kernel, 96.48% idle
CPU0 states : 0.00% user, 0.00% kernel, 100.00% idle
Memory usage: 16402328K total,3443588K used, 12958740K free
Current memory status: OK
N9K#show system internal memory-usage-per-module in-KB
Slot 01:Used:1647420 Kbytes,Free:425680 Kbytes,Total:2073100 Kbytes
N9K#show system internal memory-alerts-log Make sure log is clean
CPU
D
R
A
M
D
R
A
M

CPU & Memory Usage
show processes cpu sort | head lines 12
PID Runtime(ms) Invoked uSecs 1Sec Process
----- ----------- -------- ----- ------ -----------
3357 220 3100 7099 45.50% adjmgr
5853 31655 10181 3109 0.50% ipqosmgr
5859 9489 52308 181 2.00% diag_port_lb
3477 672 3107 216 0.50% netstack
3478 268 175 1535 0.50% ospf
Possibly ARP Table Churn
Provides top process using CPU cycle

N9K#run bash
bash-4.2$ top
top - 11:13:32 up 9 days, 3:34, 4 users, load average: 0.11, 0.11, 0.08
Tasks: 226 total, 1 running, 220 sleeping, 0 stopped, 5 zombie
Cpu(s): 0.8%us, 0.2%sy, 0.0%ni, 98.5%id, 0.0%wa, 0.1%hi, 0.3%si, 0.0%st
Mem: 16402328k total, 3445044k used, 12957284k free, 72676k buffers
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 2152 620 556 S 0 0.0 0:08.05 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:00.58 ksoftirqd/0
Top Command-display top CPU processes
Auto update
“top” provides an ongoing look at processor activity in real time

N9K#sh system internal mts buffers sum | diff
node sapno recv_q pers_q npers_q log_q
sup 320 0 0 4592 0
sup 284 0 19 0 0
sup 250 2 0 0 0
N9K#sh sockets client detail | inc pim|drops|Errors
select drops: 10
Errors:
select drops: 0
Errors:
select drops: 0
Errors:
Inter Process Messaging Usage
For SAP 320 own
by “OSPF” npers_q
increasing
Make Sure Drops/Errors not incrementing
Message and transaction service-MTS

N9K#show hardware internal cpu-mac inband stats
eth2 stats:
RMON counters Rx Tx
total packets 601163425 318962431
Per Queue Stats
Queue Idx Packet Count Bytes Drops Csum Errors Allocation Failure
Queue 0 17677525 111822449180 0 0 0
- - - - - - - - - - - - - - -SNIP- - - - - - - - - - - - - - - - - - - - - - - -
Queue 7 17677525 111822449180 0 0 0
Interrupt Counters
Rx overrun 0
Error counters
Inband Driver Statistics-CPU Drops
Rate statistics
Rx packet rate (current/peak) 717 / 80695 pps
Tx packet rate (current/peak) 360 / 1338 pps
CRC errors/Collisions/late Collisions
Alignment errors
Symbol errors
Sequence errors/Rx jabbers
RX errors/Rx length errors

N9K# show system internal frame traffic | in drops
Global input drops: bad-interface 0, bad-encap 0, failed-decap 0,
Global output drops:
eth_output_err 0, gre_err 0 otv_err 0 span_drop_en: 0 span_drops: 0
Crossbar down drops : 0 Flood_to_core LTL: Hits: 0 Misses: 0
Traffic Stats/Drops to CPU— (Cont’d)
N9K# show system inband queuing statistics | in drop
bpdu: recv 68, drop 0, congested 0 rcvbuf 2097152, sndbuf 262142 no drop 0
(q0): recv 1249377, drop 0, congested 0 rcvbuf 2097152, sndbuf 262142 no drop
(q1): recv 4138154, drop 0, congested 0 rcvbuf 2097152, sndbuf 262142 no drop
Drops From PKTmgr

Instant Buffer Utilization For CPU Port
show hardware internal buffer info pkt-stats cpu
[Q00-07] 0 0 0 0 0 0 0 0
[Q08-15] 0 0 0 0 0 0 0 0
[Q16-23] 0 0 0 0 0 0 0 0
[Q24-31] 0 0 0 0 0 0 0 0
[Q32-39] 0 0 0 0 0 0 0 0
[Q40-47] 0 0 0 0 0 0 0 0
• Total 48 Queues
• Each Line Display Cell utilized
for 8 queues
• One Cell represent approximately
208 Bytes
Congestion encountered if Counters
keep incrementing

Ethernet Out Of Band Drops/Errors
N9K#show hardware internal eobc stats | inc dropped
RX packets:248308217 errors:0 dropped:0 overruns:0 frame:0
TX packets:71554006 errors:0 dropped:0 overruns:0 carrier:0
N9K# show system internal emon stats
EMON MOD ONLINE BMP: 37f00067
FSM ID: 0 EOBCMON
=======================================
HB tx_req 186396
module 1:
rx_req 176410
rx_resp 176426
rx_miss 7
tx_resp 176410
Provides Stats for all Modules
including Fabric module
Heart bit miss

Instant Buffer Usage Stats
N9K#show hardware internal buffer info pkt-stats mod 1
INSTANCE: 0
----------------------------------------------------------
Output Shared Service Pool Buffer Utilization (in cells)
SP-0 SP-1 SP-2 SP-3
-----------------------------------------------------------
Total Instant Usage 4474 0 89 2939
Remaining Instant
Usage 25466 0 14255 3405
Peak/Max Cells Used 4821 0 327 3060
Switch Cell Count 29940 0 14344 6344
----------------------------------------------------------
show hardware internal ns buffer info pkt-stats
• Instant Buffer utilization per queue
per port
• One cell represents 208 bytes
Show hardware internal buffer info pkt-
stats input mod 1
• SP-3-Dedicted resource for Control
Traffic
• SP-0-Resource for Locally Switched
Unicast ,Multicast and SPAN
• SP-2 Extended Output queue for
Unicast using buffers from North
Star
Buffer polling interval for 7.0
Release is 500msecs

INSTANCE: 0
SP-0 SP-1 SP-2 SP-3
-------------------------------------------------------------------------
Remaining Instant Usage 25466 0 14255 3405
-------------------------------------------------------------------------
ASIC Port Q3 Q2 Q1 Q0 CPU SPAN
[13]
UC(OOBFC)->0 0 0 0
UC-> 0 0 0 1249 332 0
MC-> 0 0 0 3247 1996 0
Only printed if there is congestion
• SP-3 Started filling
the Queue
• CPU buffer filling
up
Port 13 onwards are Front Panel Port
Instant Buffer Usage Stats - With Buffer Usage

CoPP Drops
We recommend that you use the strict default CoPP policy initially and then later modify the CoPP
policies based on the data center and application requirements.
Parameters Default
Default policy Strict
Default Policy 9 policy entries
N9K# show policy-map interface control-plane mod 1 | in dropped
dropped 0 packets;
dropped 0 packets;
dropped 0 packets;
dropped 0 packets;
dropped 7800 packets; Drops Seen for Default-Class at minimal rate is normal

CoPP Drops-Exception drops
class-map copp-system-p-class-l3uc-data (match-any)
match exception glean
class-map copp-system-p-class-redirect (match-any)
match access-group name copp-system-p-acl-ptp
class-map copp-system-p-class-exception (match-any)
match exception ip option
match exception ip icmp unreachable
match exception ipv6 option
match exception ipv6 icmp unreachable
class-map copp-system-p-class-exception-diag (match-any)
match exception ttl-failure
match exception mtu-failure
Goal is to Classify all Traffic Using CoPP

Hardware Rate Limiter
N9K# show hardware rate-limiter mod 1
Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters
Module: 1
R-L Class Config Allowed Dropped Total
+----------+-----+------------+------------+-------------+
L3 glean 100 0 0 0
L3 mcast loc-grp 3000 0 0 0
access-list-log 100 0 0 0
bfd 10000 1352890 0 1352890
fex 3000 0 0 0
span 50 0 0 0

FATAL System Errors
N9K#show logging onboard mod 1 exception-log | incl FATAL prev 15
------------------------------------------------------------------------
Date (mm/dd/yy)=01/15/15 Time (hs:mn:sec): 00:16:58
OBFL Exception log data for THIS SUP Module:0
********* Exception info for module 0 ********
exception information --- exception instance 1 ----
Device Name : System Manager
Device Errorcode : 0x0000023a
ErrNum (devInfo) : 58 (0x3a)
System Errorcode : 0x401e0089 Service in VDC has had a hap-reset
Error Type : FATAL error

Common Interface Error counters and Status
N9K# show interface counters errors mod 4
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------
Eth4/1 0 100 0 581 0 0
N9K# show interface status err-disabled
Port Name Status Reason
--------------------------------------------------------------------------
Eth4/1 err-disable link-flap

Interface Queuing Stats
N9K#show queuing interface 4/18
Egress Queuing for Ethernet4/18 [System]
QoS-Group# Bandwidth% PrioLevel Shape Qlimit
Min Max Units
3 1 - - - 6(D)
-------------------------SNIP--------------------------
0 100 - - 6(D)
----------------------------------------------------
QOS GROUP 0
Unicast | OOBFC Unicast | Multicast
Dropped Pkts | 0| 0| 0|
------------------------------------------------------------
QOS GROUP 7
Unicast | OOBFC Unicast | Multicast
Dropped Pkts | 0| 0| 0|

N9K#show system internal interface counters mod 1
Internal Port Counters (150 secs rate) for Slot: 1
====================================================
Interface ASIC ASIC BCM TxBitRate(BwUtil) TxPktRate RxBitRate(BwUtil) RxPktRate
Port Inst Port (bps) (pps) (bps) (pps)
-----------------------------------------------------------------------------------------
ii1/1/1 HG0 0 1 170512 (0.00) 0 0(0.00) 0
-------------------------------------------Snip------------------------------------------
ii1/1/14 HG1 1 2 0( 0.00) 0 1129882872(2.51) 960753
ii1/1/25 HG0 1 1 1790648 (0.00) 1043 22864(0.00) 20
Inter ASIC Utilization-HG Ports
T2 #0 T2 #1
T2 #0 T2 #1 T2 #2
HG00HG00
Line Card
Fabric Module

Verify Consistency Between Software and Hardware
Table
Table CLI
Physical Interface show consistency-checker link-state
Port-Channel
Membership
show consistency-checker membership port-channels
Mac Address Table show consistency-checker l2
Vlan Membership show consistency-checker membership vlan
L3 interface-LIF
programming
L3 interface-LIF programming –Logical Interface for Routing
For RIB and FIB show consistency-checker forwarding ipv4 unicast

Consistency Checkers-Link and STP state
N9K#show consistency-checker link-state mod 1
Link State Checks: Link state only
Consistency Check: PASSED
No inconsistencies found for:
Ethernet1/1
2015 Mar 24 03:23:27 N9508a-SJ %$ VDC-1 %$ vshd: CC_LINK_STATE: Consistency
Check: PASSED
N9K# show consistency-checker stp-state vlan 18
Checks: Spanning tree state
2015 Mar 24 03:25:21 N9508a-SJ %$ VDC-1 %$ vshd: CC_VLAN_STP_STATE:

Consistency Checkers-Port Channel-Vlan Membership
N9K# show consistency-checker membership vlan 18
Checks: Port membership of Vlan in vlan and egr_vlan table
Ports configured as "switchport monitor” will be skipped
Vlan:18, Hardware state consistent for:
Ethernet2/49
2015 Mar 24 03:28:31 N95a%$ VDC-1 %$ vshd: CC_VLAN_MEMBERSHIP: Consistency
Check: PASSED
N9K#show consistency-checker membership port-channels
Checks: Trunk group and trunk membership table.
Consistency Check: Failed
Inconsistency found for port-channel1:
Module:1, Unit: ['Ethernet3/49', 'Ethernet2/49']
Module:26, Unit: ['Ethernet3/49', 'Ethernet2/49’]

Consistency Checkers-Mac address Table
N9K# show consistency-checker l2 module 1
Consistency check: PASSED
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen, + - primary entry using vPC Peer-Link,
(T) - True, (F) - False
Missing entries in the HW MAC Table
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
Extra and Discrepant entries in the HW MAC Table
---------+-----------------+--------+---------+------+----+------------------

Consistency Checkers-L3 Interface
N9K# show consistency-checker l3 mod 1
L3 LIF Checks: L3 Vlan, CML Flags, IPv4 Enable
No inconsistencies found for:
Ethernet1/1
Ethernet1/2
Ethernet1/3
2015 Mar 24 04:07:27 N9508a-SJ %$ VDC-1 %$ vshd: CC_L3_LIF: Consistency Check:
PASSED

Consistency Checker –Unicast Forwarding
N9K#test consistency-checker forwarding
Consistency check started.
N9K# show consistency-checker forwarding ipv4 unicast module 1
IPV4 Consistency check (in progress): table_id(0x1) slot(1)
Elapsed time : 8257 ms
N9K# show consistency-checker forwarding ipv4 unicast module 1
IPV4 Consistency check : table_id(0x1) slot(1)
Execution time : 13244 ms ()
No inconsistent adjacencies.
No inconsistent routes.
Consistency-Checker: PASS for 1

Gold Diagnostic Checks
N9K# show diagnostic result mod 2
Module 2: 48x1/10G-T 4x40G Ethernet Module
Test results:(.=Pass, F=Fail,I=Incomplete,U=Untested,A=Abort,E=Error disabled)
1) ASICRegisterCheck------------> .
2) PrimaryBootROM---------------> .
3) SecondaryBootROM-------------> .
4) OBFL-------------------------> .
6) BootFlash--------------------> .
7) AsicMemory-------------------> .
8) FpgaRegTest---------------- -> .
9) PortLoopback:--------------- > .
Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
-----------------------------------------------------
U U U . U U U U . . U U U . U U
RewriteEngineLoopback
On Demand Diagnostic can be executed

Sev1/2 Syslog
show logging logfile | incl -1-|-2-
2015 Feb 25 10:30:17 N9508a-SJ %PLATFORM-2-MOD_PWRUP: Module 26 powered up
(Serial number SAL1738D37W)
2015 Feb 25 10:32:37 N9508a-SJ %XBAR-2-XBAR_HGLINK_NOT_UP: fabric link 1 on
module 2 unit 0 connected to fabric module 26 unit:0 is not up during module
bring up
2015 Feb 25 10:32:39 N9508a-SJ %MODULE-2-MOD_FAIL: Initialization of module 26
(Serial number: SAL1738D37W) failed
2015 Feb 25 10:32:39 N9508a-SJ %PLATFORM-2-MOD_PWRDN: Module 26 powered down
(Serial number SAL1738D37W)

Troubleshooting Toolkit
• Ethanalyzer
• TCP Dump
• ELAM
• Packet Tracer
• Flex Counter
• ERSPAN
• Consistency Checkers

Ethanalyzer-When To Use it
• To Analyze the traffic sent and received by CPU
• It uses wireshark’s code (an open source software)
• Troubleshooting High CPU
• Troubleshoot Control Plane issues Ex. OSPF , PIM , STP
Flap.
SUP
Netstack
NIC-ETH2
Pseudo Inband
Note: Ethanalyzer does not allow capturing of hardware switched traffic between data
ports of the switch

Ethanalyzer-CLI
N9K# ethanalyzer local interface inband capture-filter "pim” detail
Capturing on inband
Frame 1 (60 bytes on wire, 60 bytes captured)
Arrival Time: Mar 24, 2015 10:01:10.018889000
-------Snip------------------
[Protocols in frame: eth:ip:pim]
N9K#ethanalyzer local interface inband display-filter "ospf” detail
Capturing on inband
Frame 1 (82 bytes on wire, 82 bytes captured)
Arrival Time: Mar 24, 2015 10:04:11.425523000
-------------------Snip--------------------
[Frame is marked: False]
[Protocols in frame: eth:ip:ospf]
Some Available Options
autostop :Autostop
decode-internal :Internal
header decoding
limit-captured-frames :Maximum
number of

TCP Dump
• Tcpdump command works on most flavors of Linux operating system
• Helps to prints out a description of the contents of packets on a network interface
• Tcpdump will, if not run with the -c flag, continue capturing packets until it is
interrupted by a SIGINT signal –CTRL-C
• Tcpdump output can be saved to file for further reference
• More info at http://www.tcpdump.org/

Tcpdump -syntax
Syntax: tcpdump -h
tcpdump version 4.1.1
libpcap version 1.2.1
Usage: tcpdump [-aAbdDefIKlLnNOpqRStuUvxX] [ -B size ] [ -c count]
[ -C file_size ] [ -E algo:secret ] [ -F file ] [ -G seconds]
[ -i interface ] [ -M secret ] [ -r file ]
[ -s snaplen ] [ -T type ] [ -w file ] [ -W filecount ]
[ -y datalinktype ] [ -z command ] [ -Z user ]
[ expression ]
bash-4.2#
N9K# show feature | in bash
Feature Name Instance State
bash-shell 1 enabled
N9K# run bash
bash-4.2# sudo su
Password:******
bash-4.2# whoami
root
bash-4.2# tcpdump –c 10 –I ps-inb

Tcpdump-Examples-
bash-4.2# tcpdump -c 100 -w tcpdump.pcap -vvvv -i ps-inb
tcpdump: WARNING: ps-inb: no IPv4 address assigned
tcpdump: listening on ps-inb, link-type EN10MB (Ethernet), capture size 65535
bytes
100 packets captured
102 packets received by filter
bash-4.2#cd /bootflash
bash-4.2# tcpdump -tttt -r tcpdump.pcap | more
reading from file tcpdump.pcap, link-type EN10MB (Ethernet)
2015-04-26 03:21:31.309350 00:0e:ee:01:1b:01 (oui Unknown) > 00:00:00:ff:ff:01
(oui Ethernet), ethertype Unknown (0x8833), length 160:
0x0000: 0000 fc08 0b00 0000 0000 0800 0000 0ffd ...............
-------------------------------------more---------------------------------
Capturing 100 packets And
writing to file
Reading captured file

tshark
bash-4.2$ tshark -i ps-inb
Capturing on inband
0.000000 00:0e:ee:01:1b:01 -> 00:00:00:ff:ff:01 0x8833 Ethernet II
12.328377 00:0e:ee:01:1b:01 -> 00:00:00:ff:ff:01 0x8833 Ethernet II
^C2 packets captured
bash-4.2$

Elam-Embedded Logic Analyzer Module-NS
• Elam Allows to capture single packet based on Trigger
• Triggers are configured using Packet information
• Only Supported on North Star Based(ALE) Line Cards and GEMs
• Use with TAC Supervision
• Help to Answer following Questions
• Was the Packet indeed Received by device on given Line card?
• How did the Packet Look like?
• How was the packet rewritten based on forwarding Decision made by T2?
• Was the Packet correctly forwarded or Dropped?

ELAM Configuration
1. Init
2. Config
5. Reset
3. Arm
4. Read
Trigger
• Init – Initialize the ELAM – select the Asic instance, pipeline and
select lines
module-1# debug platform internal ns elam asic
module-1(NS-elam)# trigger init ingress in-select 3 out-select 5
• Config – Configure the trigger based on different fields in the packet
module-1(NS-elam-insel3)# set outer ipv4 src_ip 13.13.13.10
• Arm – Arm the trigger by setting the fields to match in hardware
module-1(NS-elam-insel3)# start
• Read – Once the trigger is triggered, read the report
module-1(NS-elam-insel3)# report
• Reset – Once the process is complete, reset the trigger to restart
the process
module-1(NS-elam-insel3)# reset

Elam Ingress & Egress Direction-TOR
Trident II
ASIC
NorthStar
ASIC 1
Network Interfaces
12 x 40G
Hi-Gig2
12 x 40G
Ethernet
IP.Add=13.13.13.10
• Traffic entering GEM ports which has NS and
exiting T2 is Egress Pipeline
Ex. trigger init egress in-select 3 out-select 5
set outer ipv4 dst_ip 13.13.13.10
• Traffic Entering T2 and Exiting GEM ports is
Ingress Pipeline
Ex. trigger init ingress in-select 3 out-select 5
set outer ipv4 src_ip 13.13.13.10
IngressEgress

Elam Ingress & Egress Direction-EOR
13.13.13.10
Trident II
ASIC
North Star ASIC
Network Interfaces
12 x 40G
Hi-Gig2
12 x 40G
Ethernet
N
FE
Fabric 1
N
FE
Fabric 3
Line Card
• Traffic entering from Fabric Module in to NS of
Line Card is Egress Pipeline
Ex. trigger init egress in-select 3 out-select 5
set outer ipv4 dst_ip 13.13.13.10
• Traffic Entering NS and exiting towards Fabric
Module is Ingress Pipeline
Ex. trigger init ingress in-select 3 out-select 5
set outer ipv4 src_ip 13.13.13.10
IngressEgress

ELAM Sample Configuration & Key Info
N9K# attach mod 6
module-6# debug platform internal ns elam asic 1
module-6(NS-elam)# trigger init egress in-select 3 out-select 5
module-6(NS-elam-insel3)# set outer ipv4 dst_ip 13.13.13.10
module-6(NS-elam-insel3)# start
module-6(NS-elam-insel3)# status
module-6(NS-elam-insel3)# report
Eth5/1 Eth6/52
Nexus9508 with N9K-X9564TX
13.13.13.1/30
N9K-X9564TX 4 40Gig Port On NS 40 1/10 Gig On T2
13.13.13.10/30
If Packet Captured
Status: Triggered

Important ELAM Fields
GBL_C++: [MSG] - sideband is complete
GBL_C++: [INFO] ovector: 000FFF
GBL_C++ [INFO] hg2_srcmod: 0E
GBL_C++ [INFO] hg2_srcpid: 0D
GBL_C++ [INFO] hg2_dstmod: 11
GBL_C++ [INFO] hg2_dstpid: 0A
GBL_C++ [INFO] ip_da: 000000000000D0D0D0A
GBL_C++ [INFO] ip_sa: 000000000000D0D0D01
N9K# show interface hardware-mappings
-------------------------------------------
----------------------------
Name Ifindex Smod Unit HPort FPort NPort VPort
------------------------------------------
Eth5/2 1a280000 14 0 13 255 0 -1
Eth6/52 1a286600 17 1 10 255 51 -1
Information
is in Hex
Convert to
Dec.
Sideband is the result where
packet will be sprayed.
Should never be “0”

Packet Tracer-T2
• Helps to Trace the packet inside Switch.
• Only packets in the direction of the flow are traced
• Two Acls are installed for each filter on each Line card
• One ACL for Front Panel Port Group
• Second ACL for traffic exiting Fabric Module and ingressing Line
card
Trident II
ASIC
Network Interfaces
FM Mod

Packet Tracer Configuration
13.13.13.10/30
Configure Filter
Start Tracer
Clear/Remove-all
Stop Tracer
Check Counter
Filter
rt
test packet-tracer dst-ip 13.13.13.10 detail-fp
test packet-tracer dst-ip 13.13.13.10 detail-hg
test packet-tracer start
test packet-tracer stop
test packet-tracer show
test packet-tracer clear remove

Sample Configuration & Identify Front Port-LC
13.13.13.10/30
N9K#test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-fp
N9K#test packet-tracer show filter 1 non-zero Packet-tracer stats
Module 6:
Filter 1 installed: src-ip 13.13.13.1 dst-ip 13.13.13.10 detail-fp
Module 21:
Module 26:
Eth6/52Eth6/1
13.13.13.1/30 13.13.13.10/30

Packet Tracer Sample Configuration & Key Info
N9K# test packet-tracer start filter 1
N9K# test packet-tracer show filter 1 mod 6 non-zero
Packet-tracer stats
Module 6:
Filter1 installed: src-ip 13.13.13.1 dst-ip
13.13.13.10 detail-fp
ASIC instance 0:
Entry 1: id = 7426, count = 5, active, fp, port 13
N9K# show interface hardware-mappings | grep 6/1
Name Ifindex Smod Unit Hport FPort Nport VPort
Eth6/1 1a280000 16 0 13 255 0 -1
13.13.13.10/30
Eth6/52Eth6/1
13.13.13.1/30 13.13.13.10/30

Sample Configuration Identify Fabric Port LC From FM
N9K# test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-hg
N9K# test packet-tracer start filter 1
N9K# test packet-tracer show mod 6 non-zero
Module 6:
Filter 1 installed: src-ip 13.13.13.1 dst-ip 13.13.13.10 detail-hg
ASIC instance 0:
Entry 0: id = 7425, count = 68, stopped, fp,
ASIC instance 1:
Entry 1: id = 7426, count = 13, stopped, hg, port 1
Entry 2: id = 7427, count = 11, stopped, hg, port 2
13.13.13.10/30Eth6/52Eth8/1
13.13.13.1/30 13.13.13.10/30

Flex Counters –Adjacency Statistics
• Flex counters used to count Next hop Adjacency stats
• One can attach Stats to multiple Adjacency at same time
• One Stat Counter per adjacency
• Total Flex Counters are 16K per Switch

How To Configure Flex Counters
N9K# sh ip route 13.13.13.10
IP Route Table for VRF "default"
‘'%<string>' in via output denotes VRF <string>
13.13.13.8/30, ubest/mbest: 1/0
*via 13.13.13.6, Eth6/52, [110/41], 00:33:14, ospf-10, intra
N9K# test hardware internal adjacency statistics nexthop ipv4 13.13.13.6
interface ethernet 6/52 (enable |disable | show)
13.13.13.10/30
Eth6/52Eth6/1
13.13.13.1/30 13.13.13.10/30

Sample Configuration
13.13.13.10/30
interface ethernet 6/52 show
Module:21 Unit:0
------------------
Adjacency counters for nhip 13.13.13.6 if Ethernet6/52:
Ucast: Packets 738 Bytes 90036
Mcast: Packets 0 Bytes 0
Module:22 Unit:1
------------------
Adjacency counters for nhip 13.13.13.6 if Ethernet6/52:
Ucast: Packets 946 Bytes 115412
Mcast: Packets 0 Bytes 0
Eth6/52Eth6/1
13.13.13.1/30 13.13.13.10/30

SPAN & ERSPAN
• Switch Port Analyzer”
• Provides efficient, high-performance traffic monitoring service
• Duplicates network traffic to one or more monitor interfaces
• Types Of SPAN
• Local SPAN
• Encapsulated Remote SPAN(ERSPAN)
• Applications:
• Troubleshooting connectivity issues
• Base lining network utilization/performance
• Detecting anomalous traffic flows
• On Nexus9000 Span Traffic uses dedicated queue
• Queue carrying SPAN traffic has low Priority over other queue’s
during congestion

SPAN QOS Queue
N9K# show queuing interface ethernet 4/18 | begin SPAN
| SPAN QOS GROUP |
+-----------------------------------------------------------------+
| | Unicast | OOBFC Unicast | Multicast |
+------------------------------------------------------------------+
| Tx Pkts | 0| 0| 0|
| Tx Byts | 0| 0| 0|
| Dropped Pkts | 0| 0| 0|
| Dropped Byts | 0| 0| 0|
| Q Depth Byts | 0| 0| 0|

SPAN Configuration
N9K(config)# monitor session 1
N9K(config-monitor)# source interface sup-eth 0 both
N9K(config-monitor)# source interface ethernet 6/1
N9K(config-monitor)# destination interface ethernet 6/2
N9K(config-monitor)# No Shut
N9K(config-monitor)# show monitor
Session State Reason Description
--- ----- ------------ --------------------
1 up The session is up Local SPAN Session
e6/1 e6/2
Local SPAN
LocalSup-eth
N9K(config)#int et 6/2
N9K(config-if)# switchport monitor

ERSPAN Configuration
N9K(config)# monitor erspan origin ip-address 13.13.13.2
global
N9K(config)# monitor session 1 type erspan-source
N9K(config-erspan-src)# header-type 3
N9K(config-erspan-src)# source interface ethernet 6/1
N9K(config-erspan-src)# erspan-id 1
N9K(config-erspan-src)# ip ttl 16
N9K(config-erspan-src)# vrf default
N9K(config-erspan-src)# destination ip 9.1.1.2
N9K(config-erspan-src)# marker-packet-2
N9K(config-erspan-src)# no shut
Layer 3
e6/1
ERSPAN
e6/2
L3
Only Supports Source ERSPAN
Type-3 Header 32-bit Timestamp
Supports on Nexus9300 only
Marker packet carry original UTC time
stamp to over come 32-bit wrapper
issue

Consistency Checkers-Summary
• Show consistency-checker stp-state vlan
• Show consistency-checker link-state
• Show consistency-checker membership vlan
• Show consistency-checker membership port-channels
• Show consistency-checker membership port-channels
• Show consistency-checker l2
• Show consistency-checker l3
• Show consistency-checker forwarding ipv4 unicast

Understanding T2 interfaces-Xe0/hg
N9K# bcm-shell mod 1 "show unit"
Unit 0 chip BCM56852_A2 (current)
Unit 1 chip BCM56852_A2
N9K#bcm-shell mod 1 “0:ps”
ena/ speed/ link auto STP lrn inter max loop
port link duplex scan neg? state pause discrd ops face frame back
hg0 up 42G FD HW No Forward None FA XGMII 16360
hg2 up 42G FD HW No Forward None FA XGMII 16360
--------------------------------Snip----------------------------------
Hg11 up 42G FD HW No Forward None FA XGMII 16360
Xe0 !ena 40G FD HW No Disable None FA XGMII 1582
xe1 up 40G FD HW No Disable None FA XGMII 1582
--------------------------------Snip----------------------------------
Xe11 !ena 40G FD HW No Disable None FA XGMII 1582
Hg=Internal Ports
Xe=Front Panel Port
QSPF
Ports
QSPF
Ports
F
P
01
F
P
02
F
P
03
F
P
04
F
P
05
F
P
06
F
P
07
F
P
08
F
P
09
F
P
10
F
P
11
F
P
12
F
P
13
F
P
14
F
P
15
F
P
16
F
P
17
F
P
18
F
P
19
F
P
20
F
P
21
F
P
22
F
P
23
F
P
24
T2
Instance 0
T2
Instance 1
Eth1/1 Eth1/24
Xe0 Xe0
hg0 hg11
Xe11
Eth1/12
Xe11
Eth1/13
hg0 hg11

Layer -1 Issues- Transceiver Not Recognized
N9K# show interface ethernet 4/18 transceiver details
Ethernet4/18
transceiver is not present
module-4# show hardware internal bcm-usd event-history xcvr 18
1) Event:E_STRING, length:135, at 220346 usecs after Thu Apr 16 20:50:17 2015
bcm_usd_xcvr_fcot_notify_default(941): [unit=0 nxosport=18 bcmport=30]
fcot_state:0x2 fcot_type:0 sent MTS_OPC_FCOT_EVENT_INFO, rc 0x0
2) Event:E_STRING, length:93, at 647132 usecs after Thu Apr 16 20:50:14 2015
bcm_usd_xcvr_fcot_scan_sfp(3003): [unit=0 nxosport=18 bcmport=30]
FCOT not supported err=-1

Interface MTU/Speed/Flow Control Verification
N9K# show interface Ethernet 4/18
Ethernet4/18 is up
admin state is up, Dedicated Interface Belongs to Po10
Hardware: 10000/40000 Ethernet, address: 7c69.f66e.d860 (bia 7c69.f66e.d860)
MTU 9216 bytes, BW 40000000 Kbit, DLY 10 usec
N9K# bcm-shell module 4 ” 1: ps Xe17"
xe17 up 40G FD HW No Disable None FA SR4 9298

Interface Flow Control Check
N9K#Show interface ethernet 1/1 flowcontrol
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
-----------------------------------------------------------------------------
Eth1/1 off off off off
0 0
N9K#bcm-shell module 1 "ps" Wrong programming
ena/ speed/ link auto TP lrn inter max loop
xe0 up 10G FD HW No Disable TX RX None FA SFI 9298

Interface Input Drops
N9K#bcm-shell mod1 “ cstat xe29”
+------------------Programmable Statistics Counters[Port xe29]------+
| Type | No. | Value | Enabled For |
+----------------------------------------------------------------- -+
| RX | 0(R)| 19163028| RIPD4 RIPD6 RDISC RPORTD |
| | | | PDISC VLANDR |
| | 1(R)| 28744286| IMBP |
| | 4 | 993820| RPORTD FcmPortClass3RxDiscards |
| | 6 | 19163407| RFILDR FcmPortClass2RxDiscards |
| | 7 | 19163048| RDROP |
| | 8 | 18169208| VLANDR |
+-------------------------------------------------------------------+
| | 3(R)| 14704| TPKTD |
| | 4(R)| 968303| TGIP4 TGIP6 FcmPortClass3TxFrames|
| | 6 | 968303| TGIP4 FcmPortClass3TxFrames |
+-------------------------------------------------------------------+
Ethernet1/30 is up
Hardware: 1000/10000 Ethernet, address: 7426.acea.ceb9 (bia 7426.acea.ceb9)
EtherType is 0x8100
0 input with dribble 1316 input discard
bcm-shell mod 6 "cstat info" | gre VLANDR
VLANDR Rx VLAN drops

Fabric Connectivity and Troubleshooting
• In an 4-slot chassis N9K-C9504-FM has 1 T2 per module
• FMs provides redundancy for internal data flow, the loss of FMs just increases
the oversubscription factor.
T2
T2 T2
T2T2
T2
N9K-C9508-FM-8 N9K-C9516-FM-16
T2
N9K-C9508-FM-4

Full-Rate Mode(FRM) V/S Oversubscribed Mode(OSM)
• Each T2 have 32 40Gigport with total capacity of 1.2Tbps with “2” switching
mode
OSM(Default) - Uses all 32 40 Gig ports Line Rate achieved for packets > 200 Bytes
FRM - Uses only 24 40 Gig ports Line rate achieved for > 64 Bytes
Configuration Knob to Change the mode.
N9K(config)# system fabric-mode full-rate
Configuration effective after Reboot
N9K#show system fabric-mode
Applied System Fabric Mode:Full rate mode
Use FRM mode to achieve line rate for 64 byte packets on 9636PQ , 9564PQ ,
9564TX cards
All other 94xx line cards will not be powered up in this mode

RTAG7 and DLB
• Two Packet Hashing algorithm available from LC to FM
• RTAG7-To Select HG Port use Packet Header.
• For a flow same HG Link is used
• DLB-Dynamic Load Balancing- Default algorithm
• Initial Hash same as RTAG7
• Based on Link Quality pick up optimum HG Port
• Better utilization of all HG links
• N9K(config)# port-channel load-balance internal [dlb/rtg7]
• N9K# show port-channel load-balance internal algorithm
• HighGig port-channel load balance algorithm: dlb
LC1 LC2
FM-2FM1
FM6
HG-
Ports
HG-
Ports

Higig Link Failures – Fabric Module Policy
• For any single Higig link failure between FM and LC
Bring down the FM, if there is more than one FM
Else bring down LC
• Multiple Higig links failures for a Single LC going to Multiple FM - Bring down
the LC module.
• Multiple Higig links failures on LC to one of the FM - Bring down the LC module

4/8 slot Chassis – Fabric Connectivity
N9K-X9536PQ
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T2
T
2
T
2
T2
40 Gig Link
• 9500/9600 Series Line Card’s T2
have connectivity to all 6 Fabric
Module’s T2
• 9400 series Line cards connects to
all T2 but use only 4 Fabric Modules
-No Connection to Slot 21 & 25
• Traffic between 9500/9600 Series
Line Card and 9400 Line card will
use subset Hi Gig links .

16 slot Chassis – Fabric Connectivity
N9K-C9516-FM
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
N9K-X9536PQ
T
2
T
2
T2
T
2
T
2
T2
• 9500 Series Line Card’s T2 will have
connection to all 6 Fabric Module but to
only 2 T2’s from each Fabric Module
• 9500 series Line Card’s T2 will have
connection to all 4 T2’s of Fabric module if
there are only 3 Fabric module present
• 9400 series cards connects to all T2 but
use only 4 FM-No Connection to Slot 21 &
25
• Traffic between 9500 Series Line Card and
9400 will use subset Hi gig links.
• N9K-X9636PQ line card module is not
supported in 16 slot chassis
40 Gig Link

• With 3 FM configuration All 4 T2 units in
each FM are connected to 9500 series LC
modules' T2 units
• Each blue line represents one 40 Gig link
16 slot Chassis – Fabric Connectivity
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
T
2
N9K-X9536PQ
T2 T2

HG
MUX1
HG
MUX3
Northstar 1
Warpcore
MF Port
7-
5
2-
0
31-
29
26-
24
T2
7-
5
26-
24
0-
2
3-
5
6-8 9-
11
HG
MUX4
HG
MUX2
HG
MUX5
HG
MUX6
MN Port
0 1 2
3
4 5 6
7
8 9 10
11
Northstar 2
MF Port
0-
2
9-
11
MN Port
0 1 2
3
4 5 6
7
8 9 10
11
T2
7-
5
2-
0
31-
29
26-
24
• Line cards N9K-X9464PX/TX ,
N9K-X9564PQ/TX have Mux
• Mux used for connecting HiG Link
from Line Cards to multiple Fabric
Module
• Mux available only for Half of the
HiG interface of LC
• By Default Mux Link Active to Odd
number of Fabric Module
Line Card
Fabric Module’s
Active Mux Link
Standby Mux Link
Line Cards With Mux to FM

FM Connectivity For N9K-X9564PX –With MUX
show system internal fabric connectivity mod 5 | in B
HiGIG Link-info Linecard slot:5
LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink
5 0 HG02 1B 25 0 HG12
5 0 HG03 1B 25 1 HG12
show system internal fabric connectivity mod 5
HiGIG Link-info Fabriccard slot:5
5 0 HG02 1A 26 0 HG14
5 0 HG03 1A 26 1 HG14
With FM from Slot 25 Down FM-26
T2-0 T2-1
HG014
LC
T2-0 T2-1
FM-25
T2-0 T2-1
MUX
HG02 HG03
HG012HG012 HG014
AB
• Line cards N9K-X9464PX/TX ,
N9K-X9564PQ/TX have Mux
• Mux used for connecting HiG Link
from Line Cards to multiple Fabric
Module
• Mux available only for Half of the
HiG interface of LC
• By Default Mux Link Active to Odd
number of Fabric Module

FM Connectivity For N9K-X9564PX –With MUX
show system internal fabric connectivity mod 5 | in B
5 0 HG02 1B 25 0 HG12
5 0 HG03 1B 25 1 HG12
5 0 HG02 1A 26 0 HG14
5 0 HG03 1A 26 1 HG14
With FM from Slot 25 Down
LC
T2-0 T2-1
FM-25
T2-0 T2-1
FM-26
T2-0 T2-1
MUX
HG02 HG03
HG012HG012 HG014 HG014
LC
T2-0 T2-1
FM-25
T2-0 T2-1
FM-26
T2-0 T2-1
MUX
HG02 HG03
HG012HG012 HG014 HG014
A
A
B
B

Fabric Troubleshooting commands
1 0 HG00 - 21 0 HG00
1 0 HG01 - 21 1 HG00
FM-Slot FM-Unit FM-HGLink LC-Slot LC-Unit LC-HGLink MUX
21 0 HG00 1 0 HG00
21 1 HG00 1 0 HG01
T2
#0
T2
#1
T2
#0
T2
#1
T2
#2
Line Card Slot-1
Fabric Module Slot-21
HG00HG00

Fabric Port Drops and Link Status
N9K# bcm-shell mod 21 "ps” | inc hg0
hg0 up 42 FD HW No Forward None FA XGMII 16360
N9K# show hardware internal fabric interface asic counters mod 21
Counters for Fabric Ports:
FabricInterface Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx
RxDrops TxDrops RxDrops TxDrops Drops Drops
0 / 1 / HG0 0 0 0 0 0 0
1 / 1 / HG0 0 0 1 0 0 0 0

Fabric Port STP State HW point of View
N9K# sh vlan id 100
VLAN Name Status Ports
---- ------------------ --------- 100
VLAN0100 active Po1, Eth1/1
show sys internal xbar event-history {trace|errors|msgs|sw}
show sys internal xbar-client event-history {trace|errors|msgs|sw}
show tech-support xbar
N9K# bcm mod 21 " stg show”
STG 5: contains 1 VLAN (100)
Forward: hg

Path of the Packet -Inband
CPU
NIC-Eth2
Netstack
NIC-Eth3
System Controller-SC1
FabricModule
FabricModule
FabricModule
Line Card
Mod21
Mod26
Mod29
Mod23
OSPF Hello
Eth6/1
• Traffic from all ingress Line Card
to Supervisor will hash to one
Fabric module
• Traffic from Supervisor Card to
Egress Line cad will hash on one
FM. May not be same
• CoPP is operational on all LC.
However aggregate CoPP is on
FM

Check for Drops/Errors-Line Card
N9K#show hardware internal interface ethernet 6/1 asic counters
Important Counters/Drops
--------------- --------- --------- --------- --------- --------- ---------
Interface Name Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx
--------------- --------- --------- --------- --------- --------- ---------
Ethernet6/1 870 0 100 0 0 0
--------------- --------- --------- --------- --------- --------- ---------
Forward Rx Drops = [ RDBGC0 RDBGC4 RDBGC6 RDBGC7 RDBGC8 ]
Forward Tx Drops = [ TDBGC1 TDBGC3 TDBGC5 (excludes expected Multicast drops)]
ErrorPkt Rx Drops= [ IUNHGI IUNKOPC RFCS RALN RFLR RERPKT RJBR RSCHCRC RUND RMTUE]
ErrorPkt Tx Drops= [ TJBR TFCS TRPKT RMTUE TUFL TPCE ]
QOS Rx Drops = [ RDISC DROP_PKT_ING DROP_PKT_IMTR DROP_PKT_YEL DROP_PKT_RED ]
QOS Tx Drops = [ MCQ_DROP_PKT(0) MCQ_DROP_PKT(1) MCQ_DROP_PKT(2)
Use slot <#> show hardware internal interface indiscard-stats instance <#>
N9K#bcm-shell mod 6 "listreg RALN"| grep Description
Description: Receive Alignment Error Frame Counter
Trident II
ASIC
North Star ASIC
Network Interfaces
Line Card
RDBGC0

Instant Buffer Usage Stats-With Buffer Usage
INSTANCE: 0
SP-0 SP-1 SP-2 SP-3
-------------------------------------------------------------------------
Remaining Instant Usage 25466 0 14255 3405
------------------------------------------------------------------------
ASIC Port Q3 Q2 Q1 Q0 CPU SPAN
[13]
UC(OOBFC)-> 0 0 0 0
UC-> 0 0 0 1249 332 0
MC-> 0 0 0 3247 1996 0
Only printed if there is congestion
• SP-3 Started filling
the Queue
• CPU buffer filling
up

CoPP Drops on Line Card
N9K# show policy-map interface control-plane mod 6 class copp-system-p-class-
critical | in ospf|trans|dropped
match access-group name copp-system-p-acl-ospf
transmitted 21898 packets;
dropped 0 packets;
Trident II
ASIC
North Star ASIC
Network Interfaces
Line Card

Identify FM -Check CoPP Drops
N9K# show hardware internal cpu-mac inband active-fm traffic-to-sup
Active FM Module for traffic to sup:
0x00000015 Fabric Module in Slot 21 carry all traffic to Sup
N9K# show policy-map interface control-plane mod 21 class copp-system-p-class-
critical | in ospf|trans|dropped
match access-group name copp-system-p-acl-ospf
match access-group name copp-system-p-acl-ospf6
transmitted 21898 packets;
dropped 0 packets;

Check for Drops/Errors-Fabric Module
N9K# show system internal fabric connectivity mod 6 | grep 21 Identify HG Port on LC and FM
6 0 HG10 3B 21 0 HG15
N9K# sh hardware internal fabric interface asic counters module 6 instance 0 asic-port 11
Important Counters/Drops Verify Drops/Error on HG port on LC
FabricInterface Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx
0 / 11 / HG10 0 0 0 0 0 0
N9K# sh hardware internal fabric interface asic counters mod 21 in 0 asic-port 16
0 / 11 / HG15 0 0 0 0 0 0

Verify Drops Between FM and SC
module-21# show mvdxn internal port-status
Switch type: Marvell 98DXN11 - 10 port switch Fabric Module in Slot 21
Port Descr Enable Status ANeg Speed Mode InByte OutByte InPkts OutPkts
3 SC1EPCswitch Yes UP No 2 6 109548011 117051401 274144 587285
module-29# show mvdxn internal port-status
Switch type: Marvell 98DXN11 - 10 port switch System Controller in Slot 29
Port Descr Enable Status ANeg Speed Mode InByte OutByte InPkts OutPkts
7 FM1EPCswitch Yes UP No 2 6 746159513 60543666 620863 269592
10 port switch on System
controller and Fabric
module connect SC to FM
FABRIC CARD
System Controller
MVDXN-SW
MVDXN-SW

Drops/Errors On Supervisor
N9K#show hardware internal cpu-mac inband counters in eth|ps-
inb|dro
eth2 Link encap:Ethernet HWaddr 00:00:00:01:1b:01
eth3 Link encap:Ethernet HWaddr 00:00:00:01:1b:01
ps-inb Link encap:Ethernet HWaddr 00:00:00:01:1b:01
Netstack
NIC-Eth2
Pseudo Inband
NIC-Eth3
Supervisor Card

Drops/Errors On Supervisor-Cont.
N9K#show hardware internal cpu-mac inband stats | in errors|rate|Queue
Queue Idx Packet Count Bytes Drops Csum Errors Allocation Failure
Queue 0 65429 580195964 2 0 0
Queue 7 65429 580195964 0 0 0
CRC errors ...................... 0
Alignment errors ................ 0
Symbol errors ................... 0
Carrier extension errors .........0
Rx packet rate (current/peak) 812 / 1097 pps
Tx packet rate (current/peak) 454 / 741 pps
Related show tech(s)
Nexus9500# sh tech-support inband
counters
Nexus9500# show tech-support pktmgr
Nexus9500# show tech-support <service>

L2 Mac And Vlan Table Verification
N9K# sh mac address-table dynamic vlan 100
Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) -
True, (F) - False
* 100 547f.ee1c.06fc dynamic 0 F F Eth6/1
N9K# bcm-shell mod 6 " l2 show" | in Hit
mac=54:7f:ee:1c:06:fc vlan=100 GPORT=0x800800d modid=16 port=13/xe0 Hit
N9K# bcm-shell mod 6 "vlan show 100”
vlan 100 ports xe0,hg ....... untagged xe0
interface Ethernet6/1
switchport
switchport access vlan 100
no shutdown
Eth6/1
Mac=547f:ee1c.06fc

Spanning Tree Verification
N9K# sh spanning-tree interface ethernet 6/1
Vlan Role Sts Cost Prio.Nbr Type
VLAN0100 Desg FWD 4 128.1537 P2p
N9K# bcm-shell mod 6 "dump vlan 100”
VLAN.ipipe0[100]: <VP_GROUP_BITMAP=0x00000……STG=0X67
FID_ID=0x64
N9K# bcm-shell mod 6 "stg stp 103”
STG 103:
Block: xe1-xe47
Forward: xe0,hg
interface Ethernet6/1
switchport
switchport access vlan 100
no shutdown
Eth6/1
Mac=547f:ee1c.06fc
N9K# Dec 0x67=103
STG= STP Group ID
N9K# Dec 0x64=100
FID_ID=Vlan ID.

Unicast L3 Forwarding
• T2 has combination of dedicated TCAM table space and shared hash table
memory known as Unified Forwarding Table (UFT)
• The UFT is partitioned into three forwarding tables
• MAC Address Table
• IP Host Table
• Longest Prefix Match-LPM Table
• To maximize the system-wide forwarding scalability UFT tables on line
cards and fabric modules for different forwarding lookup functions
FM
LC
Feature Scale
L3 Host Table
And L2/L3
Multicast
120K
L2 Mac Table 96K
Feature Scale
L3 LPM
Table
128K

Unicast L3 Forwarding- Component Information
Hardware-T2
uFDM
Supervisor
AM
uRIB
OSPF ARP
FIB Manager
Forwarding Hardware
Theory of Operation
Software/Hardware Programming
• OSPF communicates with uRIB to build the
routing table
• AM builds the next-hop adjacency entry
• uFDM distributes the information to the line
cards
• IP FIB (running on the line cards) programs the
ASIC components with the forwarding and
adjacency information.
Remember: Software forwarding by the SUP is only
used for control and exception packets

L3 Unicast Troubleshooting Flow
HW Programming
On LC/FM
Use BCM commands
Next-Hop
Check the routing table
Checking Route on
RIB And FIB.
ARP/MAC
Check the ARP Table
Check Forwarding Route
Show ip route [ipv4] [<prefix>]
Show ip arp [ipv4]
show ip adjacency (Ipv4]
show forwarding adjacency platform [ipv4]
module <mod>
show forwarding [ipv4] route module <mod>
bcm-shell mod 22 "l3 defip show"

Unicast L3 Forwarding- Two Possible Scenarios
Case 1: If incoming packet hit /32 host route on LC, forwarding decision is made on LC
Case 2: If incoming packet miss /32 host route on LC. Now for Longest Prefix
match (LPM) the packet get forwarded to FM
• Install a default route 0.0.0.0/0 on Line Cards using the virtual MOD ID for Fabric Module
as the DMOD to force Line Cards to forward LPM packets to Fabric Modules
• Fabric Modules perform LPM lookup and forward packets to the resolved Destination
MOD/Destination PORT
Also will verify How to Check ECMP Route

Network Diagram-Problem Definition
13.13.13.0/30
13.13.13.12/30
.1
13.13.13.8/30
.2 .17 .9 .10
N9K#
Nexus3064Q-ESC#
N9508d-SJ#
N9508c-SJ#
Nexus3064Q-ESC# ping 13.13.13.10
PING 13.13.13.10 (13.13.13.10): 56 data bytes
Request 0 timed out
Nexus3064Q-ESC# traceroute 13.13.13.10
traceroute to 13.13.13.10 (13.13.13.10), 30 hops max, 40 byte
packets
1 13.13.13.2 (13.13.13.2) 1.124 ms 0.911 ms 0.752 ms
2 * * *
.18
.13 .14
13.13.13.16/30

Router MAC Programming Check
• Router Mac address must be programmed in Hardware
N9K1#show interface ethernet 6/1 | grep address
Hardware: 100/1000/10000 Ethernet, address: 003a.99fc.dd7f
N9K1# bcm-shell mod 6 "0:d chg my_station_tcam" | grep dd7f
MY_STATION_TCAM.ipipe0[0]: <VALID=1,------snip----MAC_ADDR=0x003a99fcdd7f,

Verify /32 Host Route on Line card-Case 1
N9K1#show ip route 13.13.13.14
13.13.13.14/32, ubest/mbest: 1/0, attached
*via 13.13.13.14, Eth6/33, [250/0], 00:37:24, am
N9K1#bcm-shell mod 6 "0:l3 l3table show" | grep 13.13.13.14
Entry VRF IP address Mac Address INTF MOD PORT CLASS HIT
10 1 13.13.13.14 00:00:00:00:00:00 100010 0 0 0 y
N9K1#bcm-shell mod 6 "0:l3 egress show"| grep 100010
Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop
100010 88:f0:31:bf:ad:17 4095 4432 45 16 -1 no no
N9K1#show system internal ethpm info interface ethernet 6/33 | grep -i STATIC
IF_STATIC_INFO: port_name=Ethernet6/33,if_index:0x1a284000,ltl=40875,slot=5,
nxos_port=32,dmod=16,dpid=45,
/32 Host Entry

Next Hop Reached via L3-Port Channel
N9K1#show ip route 10.164.112.22
10.164.112.22/32, ubest/mbest: 1/0
*via 13.13.13.14, Po200, [110/3], 00:09:33, ospf-10, intra
N9K1#bcm-shell mod 6 "0:l3 l3table show" | grep 10.164.112.22
175660 1 10.164.112.22 00:00:00:00:00:00 100012 0 0 0 y
N9K1#bcm-shell mod 6 "0:l3 egress show"| grep 100012
100010 88:f0:31:bf:ad:17 665 4761 3t 0 -1 no no
N9K1#show system internal ethpm info interface port-channel 200 |grep –I STATIC
IF_STATIC_INFO: port_name=port-channel200,if_index:0x160000c7,ltl=2597,slot=0,
nxos_port=02,dmod=0,dpid=3,
/32 Host Entry

Verify HW-Programming on LC or FM ? Case 2
N9K# show ip route 13.13.13.10
IP Route Table for VRF "default”
13.13.13.8/30, ubest/mbest: 1/0
*via 13.13.13.6, Eth6/52, [110/41],
00:22:29, ospf-10, intra
N9K# show forwarding route 13.13.13.10 module 21
IPv4 routes for table default/base
Prefix | Next-hop Interface | Labels
13.13.13.8/30 13.13.13.6 Ethernet6/52
This is not /32 host Route.
Packet forwarding decision
responsibility is of the Fabric
Module
ALL FM will be programmed
with this Route

Line Card Punting Packets to Fabric For LPM ?
N9K# show hardware internal forwarding adjacency statistics default-route mod 6
Module:6 Unit:0
Traffic matched adjacency for default route (destined to FM):
Unicast: Packets 148 Bytes 13382
N9K# bcm-shell mod 6 "0:l3 defip show"
Unit 0, Total Number of DEFIP entries: 12288
# VRF Net addr Next Hop Mac INTF MODID PORT PRIO CLASS HIT
VLAN
3072Override 0.0.0.0/0 00:00:00:00:00:00 149149 0 0 0 0 y
N9K# bcm-shell mod 6 "l3 egress show" | inc 149149
149149 00:12:12:12:12:12 4095 8189 1 100 -1 no no
Mod 100 is assign to Fabric Module

Longest Prefix Match on Fabric Module
N9K# bcm-shell mod 22 "l3 defip show" | grep 13.13.13.8
# VRF Net addr Next Hop Mac INTF MODID PORT PRIO CLASS HIT VLAN
196620 1 13.13.13.8/30 00:00:00:00:00:00 100008 0 0 0 0 n
N9K# bcm-shell mod 22 "l3 egress show" | grep 100008
100008 88:f0:31:bf:ad:17 4095 4520 10 17 -1 no no
N9K# show system internal ethpm info interface eth 6/52 | grep dmod
IF_STATIC_INFO:
port_name=Ethernet6/52,if_index0x1a286600,ltl=40856,slot=5,nxos_port=51,
dmod=17,dpid=10,unit=1,
Mac add used for rewrite

ECMP Route Validation
N9K#show ip route 10.164.112.22
10.164.112.22/32, ubest/mbest: 2/0
N9K#sh routing hash 13.13.13.2 10.164.112.22 mod 6
Hashing to path *13.13.13.18
Out Interface: Eth6/34
N9K#bcm-shell mod 6 "0:l3 l3table show" | grep 10.164.112.22
17 1 10.164.112.22 00:00:00:00:00:00 200256 0 0 0 n (ECMP)
N9K#bcm-shell mod 6 "l3
multipath show"
Multipath Egress Object 200256
Interfaces: 100008 100010
Follow same steps demonstrated for /32 Host entry to learn about Interface in multipath show cli
Multi-Path

Use Tools From Toolkit
• ELAM- IF Line Card has North Star
module-6# debug platform internal ns elam asic 1
module-6(NS-elam)# trigger init egress in-select 3 out-select 5
module-6(NS-elam-insel3)# set outer ipv4 dst_ip 13.13.13.10
• Packet Tracer- For All FM and LC having T2
N9K# test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-fp
• Flex Counter- Check Adjacency hit counter
interface ethernet 6/52 enable
• Consistency Checker
show consistency-checker forwarding ipv4 unicast
show tech-support forwarding l3 unicast
show tech-support adjmgr
show tech routing unicast

Virtual Port-Channel-vPC
• Allow a single device to use a port channel across
two upstream switches
• Eliminate STP blocked ports
• Dual-homed server operate in active-active mode
• HSRP-Both active and standby peers forward
packets-ARP response by Active
• Configuration steps Same as other Nexus
Products
Logical Topology with vPC

Case:1 All vPC Leg UP
MCT-1/1, 4/1
N9k1 N9k2
vPC20vPC10
Eth4/18
Eth6/20
Keep Alive
Eth4/18
Eth6/20
SVI10
10.10.10.1/24
SVI-Mac 78da.6e71.9a3f
Standby 10.10.10.3
HSRP-Mac 0000.0c07.ac0a
SVI20
SVI-mac 78da.6e71.9a3f
10.10.20.1/24
Standby 10.10.20.3
HSRP-Mac 0000.0c07.ac14
Switch-A Switch-B
Vlan-10 Vlan-20
10.10.10.x/24 20.20.20.x/24
HOST-A HOST-B
SVI10
10.10.10.2/24
SVI-mac 003a.99fc.dd7f
Standby 10.10.10.3
HSRP-Mac 0000.0c07.ac0a
SVI20
SVI-mac 003a.99fc.dd7f
10.10.20.2/24
Standby 10.10.20.3
HSRP-Mac 0000.0c07.ac14
Scenario: Traffic of a Host in Vlan 10 connected to Switch-A hash to N9K1 to reach Host in Vlan 20
connected to Switch-B
PC1-PeerLink
vPC Peer Link =Eth1/1,4/1

vPC-Router MAC Programming Check
• Both Active and Standby Peer responsible for L3 switching
• Virtual Mac address must be programmed in Hardware on Both peers
Interface Grp Prio P State Active addr Standby addr Group addr
Vlan10 10 100 Active 10.10.10.2 local 10.10.10.3
N9K1# bcm-shell mod 4 "0:d chg my_station_tcam" | grep
VLAN_ID=0xa
VLAN_ID=0xa,VALID=1, MAC_ADDR=0xc07ac0a,
Interface Grp Prio P State Active addr Standby addr Group addr
Vlan10 10 100 Standby 10.10.10.2 local 10.10.10.3
N9K2# bcm-shell mod 4 "0:d chg my_station_tcam" | grep
VLAN_ID=0xa
VLAN_ID=0xa,VALID=1, MAC_ADDR=0xc07ac0a,

vPC Peer Gateway Programming Check
• Are N9K’s Configured with Peer-Gateway
N9K1-SJ# show mac address-table vlan 10 | in G
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
G 10 0000.0c07.ac0a static - F F vPC Peer-Link(R)
G 10 003a.99fc.dd7f static - F F sup-eth1(R) N9K2 SVI MAC
G 10 78da.6e71.9a3f static - F F vPC Peer-Link®
N9K# bcm-shell mod 4 "0:d chg my_station_tcam" | egrep 0x003a99fcdd7f
MY_STATION_TCAM.ipipe0[0]:
<VALID=1,MAC_ADDR_MASK=0xffffffffffff,MAC_ADDR=0x003a99fcdd7f,KEY=0x00000000003
a99fcdd7f,IPV6_TERMINATION_ALLOWED=1,IPV4_TERMINATION_ALLOWED=1,DATA=0x38,ARP_R
ARP_TERMINATION_ALLOWED=1>

vPC Check For Traffic Ingressing Peer Link
Egress Block Mask
• vPC Check-Traffic from Peer Link should Not L2/L3 Switch with local and remote
Legs up
N9K1# show vpc brief | grep Po
id Port Status Active vlans
1 Po1 up 10-20
id Port Status Consistency Reason Activevlans
10 Po10 up success success 10-20
N9K2# show vpc brief | grep Po
id Port Status Active vlans
1 Po1 up 10-20
id Port Status Consistency Reason Activevlans
MCT-1/1, 4/1
N9k1 N9k2
vPC20vPC10
Eth4/18
Eth6/20
Keep Alive
Eth4/18
Eth6/20
Switch-A Switch-B
PC1-PeerLink

MCT-1/1, 4/1
N9k1 N9k2
vPC20vPC10
Eth4/18
Eth6/20
Keep Alive
Eth4/18
Eth6/20
Switch-A Switch-B
PC1-PeerLink
vPC Check for Traffic Ingressing Peer Link (Cont’d)
N9K1#show port-ch summary | in Po
Group Port- Type Protocol Member Ports
1 Po1(SU) Eth LACP Eth1/1(P)Eth4/1(P)
10 Po10(SU)Eth LACP Eth4/18(P)
20 Po20(SU)Eth LACP Eth6/20(P)
N9K1# show system internal vpcm info mask
module 6
Masked ports for Module 6, Unit 0:
[Src Port None]: Eth6/20
[Src Port Eth1/1]: Eth6/20
[Src Port Eth4/1]: Eth6/20
Masked ports for Module 6, Unit 1:
Traffic Ingressing on Eth1/1 and
Eth4/1 will not exit Eth 6/20

ACL redirect logic for routed packets-vPC Leg Down
• Redirect ACL installed to redirect routed packets for the
vPC for which local interface goes down
• Mac address learned from vPC points virtual port
MCT-1/1, 4/1
N9k1 N9k2
vPC20vPC10
Eth4/18
Eth6/20
Keep Alive
Eth4/18
Eth6/20
Switch-A Switch-B
PC1-PeerLink
Link Down
N9K1# show hardware access-list tcam region | grep vpc
VPC Convergence [vpc-convergence] size = 512
N9K1# sh mac address-table address30f7.0d9b.d401
20 30f7.0d9b.d401 dynamic 0 F F vPC Peer-Link

ACL redirect logic for routed packets-vPC Leg Down
• On N9K1 traffic entering Eth6/20 after L3 switch
should egress Peer Link
• N9K2 Should not drop traffic entering Peer link and
forward traffic out to Eth 4/8 MCT-1/1, 4/1
N9k1 N9k2
vPC20vPC10
Eth4/18
Eth6/20
Keep Alive
Eth4/18
Eth6/20
Switch-A Switch-B
PC1-PeerLink
Ln Down
N9K# bcm-shell module 6 "fp show group 57”
InPorts->L3Routable
DstTrunk
Offset: 213 Width: 16
DATA=0x00008003
action={act=RedirectTrunk, param0=1(0x1) Trunk-id of vPC Peerlink
Trunk-id of “3” Down vPC

ACL redirect logic for routed packets-Verify TrunkID
N9Ka# show system internal ethpm info int port-channel1 | grep dpid
IF_STATIC_INFO: port_name=port-channel1,if_index:0x16000000,ltl=2595,slot=95
dpid=1,unit=0,queue=0,xbar_unitbmp=0x0 ns_pid=0
N9508a-SJ# show system internal ethpm info int port-channel10 | grep dpid
IF_STATIC_INFO: port_name=port-channel10,if_index:0x16000000,ltl=2595,slot=95
dpid=3,unit=0,queue=0,xbar_unitbmp=0x0 ns_pid=0
show tech-support vPC
show tech-support cfs
show tech-support port-channel
Some important info to capture

ACL redirect logic for routed packets-Verify TrunkID
NX-OS -7.0(3)I1(2)
N9508a-SJ# show system internal access-list vpc-convergence mod 6
------------------------------------------------------------
VPC Convergence Entries
------------------------------------------------------------
Instance: 0
==========
Ingress:
----------
Entry-ID DstTrunk-GID RedirectTrunk-GID Packet-Count
------------------------------------------------------------------------
1539 3 1 6082015
Trunk-id of “3” Down vPC
Trunk-id of vPC Peerlink

Nexus9000 Specific
Limitation and Goodies

Email from Nexus9000 To Cisco SR
• Commands output directly sent to email address
• Information from Nexus9000 Can be directly attached to Service Request.
• Information is sent as body to email- not as attachment
N9K(config)# email
N9K(config-email)# smtp
N9K(config-email)# smtp-host 173.37.37.37
N9K(config-email)# from N9508a-sj@cisco.com
N9K(config-email)# smtp-port 25
show run | email subject <SR-number> attach@cisco.com

Bash Support !!!!
• Goes beyond what standard CLI can provide
• Customers demand more capabilities/freedom Creativity
• Feature: bash-shell
• User Role: dev-ops or network-admin or vdc-admin*
• Strongly recommended: Some experience with shell/Linux-Use with
extreme care

Broadcom ASIC shell access on the Nexus 9000 !!!
• The Nexus 9000 is based largely on the Broadcom Trident II ASIC-Known as T2
• The modular unit Fabric Modules (FM) and Line Cards (LC) each contain multiple
instances of the T2 ASIC, as well as the TOR (top of rack) units
• Access is provided to each and every instance of the T2 ASIC
• No additional license is required to access the bcm-shell
• Permitted by default role network-admin
• Role based access control (RBAC) can be used to limit user access
• Accounting log available for BCM activity

BCM Access some Examples
N9K# bcm-shell mod 6 "show unit"
Unit 0 chip BCM56852_A2 (current)
Unit 1 chip BCM56852_A2
N9K# bcm-shell mod 6 "ps" | in 19
xe19 up 1G FD SW Yes Disable None FA XGMII 1582
N9K# show accounting log | last 2
Mon Apr 20 08:31:52 2015:type=update:id=console0:user=admin:cmd=bcm-shell
module 6 "show unit" (SUCCESS)
Mon Apr 20 08:32:14 2015:type=update:id=console0:user=admin:cmd=bcm-shell
module 6 "ps" | in 19 (SUCCESS)
QSPF
Ports
QSPF
Ports
F
P
01
F
P
02
F
P
03
F
P
04
F
P
05
F
P
06
F
P
07
F
P
08
F
P
09
F
P
10
F
P
11
F
P
12
F
P
13
F
P
14
F
P
15
F
P
16
F
P
17
F
P
18
F
P
19
F
P
20
F
P
21
F
P
22
F
P
23
F
P
24
T2
Instance 0
T2
Instance 1
Eth1/1 Eth1/24
Xe0 Xe0
hg0 hg11
Xe11
Eth1/12
Xe11
Eth1/13
hg0 hg11

BCM Access some Examples (Cont’d)
N9K# bcm-shell mod 21 "config show l3"
l3_alpm_enable=2
l3_max_ecmp_mode=1
l3_mem_entries=16384
N9K# bcm-shell mod 4 "config show l2 ”
l2xmsg_hostbuf_size=16384
l2_mem_entries=98304

Python !!!!
• Python is - Established, Modern and Powerful, Clean, lots of libraries, liberal
license
• Perl is available in gdb images only – not available in final images
• Tcl is there but no one uses it in NX-OS
• The license that Python has (GPL-Like with very few restrictions on modification,
distribution and commercial use) make it very attractive to embed and distribute
• On the box applications that can currently use Python scripts
• Embedded Event Manager
• Power On Auto Provisioning (POAP)
• Create your own scripts that are like “Super commands”
• Create your own command modifiers – the things that act on commands applied with a
pipe “|”

Python-Continued
• There are two Python environments on the N9000
• One executed from VSH
• One executed from Bash
• Both run in their own forked process
• The main differences comes from the environment that they get initialized into
• These differences between them should be minimal
• There is a sandbox that should primarily contain lower privileged users
• Network-admin users get basically a “pure” 2.7.5 python environment
• That sandbox mostly applies to lower privileged users, they may be prevented from doing certain things
in python
• Also prevents file operations on files outside of bootflash

Python-Example
N9K# python
Python 2.7.5 (default, Oct 8 2013, 23:59:43)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or
"license" for more information.
>>>
N9K# run bash python
Python 2.7.5 (default, Oct 8 2013, 23:59:43)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or
"license" for more information.
>>>
switch between VSH and the
Interpreter (Bash 1)
switching between VSH and Python

Why Patching?
Begin Code Test &
Qualification Cycle
Target Deployment
Bug Found, Diagnose, Root
Cause
Defect Resolved, integrated
into Maint.
Maint. Released
Restart Qual Cycle Actual Deployment
6 Months
10 Months
Many customers spend extensive time and effort to test and qualify software prior to deployment. In today’s
environments, if a defect is found, effectively root-caused, and integrated, since it is rolled out through a
maintenance release, customers would need to restart their qualification cycle, wasting time, and pushing out
deployment dates…

NX-OS Image Patching
Begin Code Test &
Qualification Cycle
Target Deployment
Bug Found, Diagnose, Root
Cause
Defect Resolved, Patch
Released
Continue Qual
With additional tests Actual Deployment
6 Months
7 Months
The Nexus9000 Standalone platforms introduces new patching capabilities that allows specific defects to be
rolled out in an independent package that can be applied to existing base software binaries. This will help
reduce customer code certification times, leading to greater customer satisfaction.

Patching Overview
• NXOS platforms release major versions when introducing new features and engineering
special builds to provide bug fixes.
• The new goal will be to allow customers to deploy patches for specific fixes only without
affecting the data plane of the device.
• The patching architecture comes from IOS XR (SMU – Software Maintenance Upgrade)
used to deliver Quick, Effective and Focused patches for specific sections of code.
• Both binaries and libraries can be patched.
• Supervisors and Line Card services can be patched.
• Software patching will leverage process restart/reload or ISSU

Patch Uninstall Workflow - Detailed
• User invokes “install deactivate <patch_name>”
• System manager gracefully shuts down each impacted process
• Softlinks are changed from active SMU to one in backup folder (if present).
• Relevant SMU is removed from the /var/installer/activated/SMU directory.
• System Manager triggers restart of impacted processes
• (Optional) “install remove” deletes the patch from the local repository

CLI Commands – Patch Install
Command Syntax Function Notes
Install add install add <uri> [activate] Download patch from URI and add
patch to repository.
Only one patch can be added at
a time. Optionally can activate
patch in this step.
Install remove install remove [<package> |
inactive]
User can remove only non-
activated patches
Confirmation y/n will be prompted
Install activate install activate <package> [test] Installs a patch from the local
repository. If not present, an error
will be returned.
Only one patch can be activated
at a time. No show commands
permitted during operation.
Install deactivate install deactivate <package> Uninstall patch and move it to non-
activated repository
Only one patch can be
deactivated at a time. **Patches
must have no other patch
dependencies
Install commit install commit Preserves all activated patches
across reloads.
Activated patches are committed
to a list kept in the patch
repository

CLI Commands – Show Commands
Command Function Sample
show install request Shows current install operation along
with time stamp, package name,
initiating user and % complete.
Fri May 10 09:06:55.921 UTC
Install operation 13 '(admin) ‘install activate n9000-dk.6.0.2.U1.1.CSCuf08219.bin’
Started by user 'cisco' via CLI at 09:06:48 UTC Fri May 10 2013
The operation is 10% complete
show install log [id | detail
| from | last | reverse]
Shows user information on previous
installation operations. Optional [detail]
command for verbose information.
Install operation 1 by user ‘admin’ at Tue Sep 28 01:37:02 2004:
install commit
Operation completed successfully
Install operation 2 by user ‘admin’ at Mon Oct 18 17:26:36 2004:
install add tftp://10.52.241.252/bcarter/n3000-uk9.6.0.2.U1.1.CSCuf08219.bin
Operation completed successfully
Install operation 7 by user ‘lab’ at Mon Oct 18 17:31:13 2004:
install activate n3000-uk9.6.0.2.U1.1.CSCuf08219’
Operation failed because service failed to come up.
show install active [on-
reload]
Displays boot images and active or
committed patches
switch# show install active
Boot Images:
Kickstart Image: bootflash:/n9000-dk.6.1.234.gbin
System Image: package:/isanboot/bin/images/sys
Active Packages:
n9000-dk.6.1.1.CSCui56298.bin

CLI Commands – Show Commands (Cont’d)
Command Function Sample
show install inactive [on-
reload]
Shows patches in the repository not
yet activated
switch# show install inactive
Boot Images:
Image: bootflash:/inseor.6.1.1.234.gbin
System Image: package:/isanboot/bin/images/sys
Inactive Packages:
switch#
show install pkg-info
<package>
Shows details of a specific patch.
Requires that patch has been added
using ‘install add’ first.
switch# show install pkg-info n9000-dk.6.1.1.CSCui56298.bin
Contents of Package file 'n9000-dk.6.1.1.CSCui56298.bin':
Expiry date : Jan 19, 2015 02:55:56 UTC
Uncompressed size : 17892613
Vendor : Cisco Systems
Desc : Bug Fix for CDET: CSCui56298
Build : Built on Wed May 10 08:04:58 UTC 2013
Source : By n9k-infra-bld
Platform: Nexus-9000.
Supersedes: n9000-uk9.6.1.1.U1.1.CSCuf09119, n9000-uk9.6.1.1.U1.1.CSCuf02229
Pre-requisite: n9000-uk9.6.1.1.U1.1.CSCuf09219
Restart information: BGP process restart.

Sample Patch Install – Copy Patch to Switch
N9K# copy
scp://sdn@172.18.217.42/home/sdn/n9k/inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
bootflash:
Enter vrf (If no input, current vrf 'default' is considered): management
sdn@172.18.217.42's password:
inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin 100% 233KB
232.7KB/s 00:01
Copy complete, now saving to disk (please wait)...
N9508#
N9508# dir | grep .gbin
238230 Jan 15 10:52:31 2014inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
N9508#

Sample Patch Install – Add patch to repository & verify
N9K# install add bootflash:inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
Install operation 19 completed successfully at Wed Jan 15 10:55:14 2014
N9508#
N9K# show install packages
-----------------------------------------------------------
inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin inactive-commit
Modules
Module #27: inactive-commit
Module #28: inactive-commit
-----------------------------------------------------------
N9K# show install inactive
Inactive Packages:
inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
N9K#

Important Limitations
• For every Feature please review Guidelines and Limitations
• Cisco Nexus 9000 Series NX-OS Verified Scalability Guide
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-
x/scalability/guide_34/b_Cisco_Nexus_9000_Series_NX-OS_Verified_Scalability_Guide_612I34.html
• Only one software image (called nx-os) is required to load the Cisco NX-OS operating system.
• EPLD Upgrade are recommended but are not mandatory
• User Configured MAC address for SVI- Packets will not be flooded if Layer 2 Adjacency is missing
• Diagnostic-The Port Loop back and Boot up Port Loop back tests are not supported
• ASIC Memory-NS test is applicable only for the N9K-X9564PX and N9K-X9564TX line cards.
• Priority flow control (PFC) is supported on Cisco Nexus 9500 Series switches with the N9K-
X9636PQ line card.
• FEX is supported only on the Cisco Nexus 9372PX and 9396PX switches.
• Cisco Nexus 9500 Series Switch can run in 8-queue mode only if all of its line cards are capable of
running 8-queue mode.

Participate in the “My Favorite Speaker” Contest
• Promote your favorite speaker through Twitter and you could win $200 of Cisco
Press products (@CiscoPress)
• Send a tweet and include
• Your favorite speaker’s Twitter handle <Speaker—enter your Twitter handle here>
• Two hashtags: #CLUS #MyFavoriteSpeaker
• You can submit an entry for more than one of your “favorite” speakers
• Don’t forget to follow @CiscoLive and @CiscoPress
• View the official rules at http://bit.ly/CLUSwin
Promote Your Favorite Speaker and You Could Be a Winner

Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner
will receive a $750 Amazon
gift card.
• Complete your session surveys
though the Cisco Live mobile
app or your computer on
Cisco Live Connect.

Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings
• Related sessions

Backup Slides
Backup Slides !!!!

Fabric Module
NFE
Fabric Module for Nexus 9504
NFE NFE
NFE NFE NFE NFE
Chassis Type Nexus 9504 Nexus 9508 Nexus 9516
NFEs per Fabric Module 1 2 4

Nexus 9500 Platform FRU- Line Card
Connect to Fabric Modules
Connect to Hosts or
Network
NFE 1
ALE 1
12 x 42 Gbps
Network
Interfaces
12 x 42
Gbps
18x 40
Gbps
Ethern
et
18x 40Gbps
NFE NFE
12 x 40 Gbps12 x 40 Gbps
NFE
12 x 40 Gbps
N
F
E
Fabric 1
N
F
E
N
F
E
Fabric 2
N
F
E
N
F
E
Fabric 3
N
F
E
N
F
E
Fabric 4
N
F
E
N
F
E
Fabric 5
N
F
E
N
F
E
Fabric 6
N
F
E
1 x 42
Gbps
1 x 42
Gbps

N9K-X9636PQ
HG Ports HG Ports HG Ports
QSPF PortsQSPF PortsQSPF Ports
FP
01
FP
02
FP
03
FP
04
FP
05
FP
06
FP
07
FP
08
FP
09
FP
10
FP
11
FP
12
FP
13
FP
14
FP
15
FP
16
FP
17
FP
18
FP
19
FP
20
FP
21
FP
22
FP
23
FP
24
FP
25
FP
26
FP
27
FP
28
FP
29
FP
30
FP
31
FP
32
FP
33
FP
34
FP
35
FP
36
T2
Instance 0
T2
Instance 1
T2
Instance 2

N9K-X9464PX
HG Ports HG Ports
10G SFP+ Ports 40G QSFP
FM3FM2 FM4 FM6
MUX1-2 MUX3-4
T2
FP
1
FP
2
FP
3
FP
4
FP
5
FP
6
FP
7
FP
8
FP
9
FP
10
FP
11
FP
12
FP
13
FP
14
FP
15
FP
16
FP
17
FP
18
FP
19
FP
20
FP
21
FP
22
FP
23
FP
24
FP
25
FP
26
FP
27
FP
28
FP
29
FP
30
FP
31
FP
32
FP
33
FP
34
FP
35
FP
36
FP
37
FP
38
FP
39
FP
40
FP
41
FP
42
FP
43
FP
44
FP
45
FP
46
FP
47
FP
48
FP
49
FP
50
FP
51
FP
52

N9K-X9464TX
HG Ports HG Ports
100/1000/10000 T Ports 40G QSFP
FM3FM2 FM4 FM6
MUX1-2 MUX3-4
T2
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
FP
49
FP
50
FP
51
FP
52
FP
1
FP
2
FP
3
FP
4
FP
5
FP
6
FP
7
FP
8
FP
9
FP
10
FP
11
FP
12
FP
13
FP
14
FP
15
FP
16
FP
17
FP
18
FP
19
FP
20
FP
21
FP
22
FP
23
FP
24
FP
25
FP
26
FP
27
FP
28
FP
29
FP
30
FP
31
FP
32
FP
33
FP
34
FP
35
FP
36
FP
37
FP
38
FP
39
FP
40
FP
41
FP
42
FP
43
FP
44
FP
45
FP
46
FP
47
FP
48

N9K-X9432PQ
QSPF PortsQSPF Ports
HG Ports HG Ports
FP
01
FP
02
FP
03
FP
04
FP
05
FP
06
FP
07
FP
08
FP
09
FP
10
FP
11
FP
12
FP
21
FP
22
FP
23
FP
24
FP
25
FP
26
FP
27
FP
28
FP
29
FP
30
FP
31
FP
32
FM3 FM4FM2 FM6
T2
Instance 0
T2
Instance 2
FP
13
FP
14
FP
15
FP
16
FP
17
FP
18
FP
19
FP
20

N9K-X9564PQ
10G SFP+ Ports
40G QSFP
HG MUX1 HG MUX3
FP
49
FP
50
FP
51
FP
52
FP
1
FP
2
FP
3
FP
4
FP
5
FP
6
FP
7
FP
8
FP
9
FP
10
FP
11
FP
12
FP
13
FP
14
FP
15
FP
16
FP
17
FP
18
FP
19
FP
20
FP
21
FP
22
FP
23
FP
24
FP
25
FP
26
FP
27
FP
28
FP
29
FP
30
FP
31
FP
32
FP
33
FP
34
FP
35
FP
36
FP
37
FP
38
FP
39
FP
40
FP
41
FP
42
FP
43
FP
44
FP
45
FP
46
FP
47
FP
48
Northstar 1
Warpcore
MF Port
7-5 2-0 31-29 26-24
T2
7-5 26-24
0-2 3-5 6-8 9-11
HG MUX4 HG MUX2 HG MUX5 HG MUX6
MN Port
0 1 2 3 4 5 6 7 8 9 10 11
Northstar 2
MF Port
0-2 9-11
MN Port
0 1 2 3 4 5 6 7 8 9 10 11
T2
7-5 2-0 31-29 26-24

N9K-X9564TX
100/1000/10000 T Ports
40G QSFP
HG MUX1 HG MUX3
T2
FP
49
FP
50
FP
51
FP
52
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
10G
PHY
FP
1
FP
2
FP
3
FP
4
FP
5
FP
6
FP
7
FP
8
FP
9
FP
10
FP
11
FP
12
FP
13
FP
14
FP
15
FP
16
FP
17
FP
18
FP
19
FP
20
FP
21
FP
22
FP
23
FP
24
FP
25
FP
26
FP
27
FP
28
FP
29
FP
30
FP
31
FP
32
FP
33
FP
34
FP
35
FP
36
FP
37
FP
38
FP
39
FP
40
FP
41
FP
42
FP
43
FP
44
FP
45
FP
46
FP
47
FP
48
Northstar 1
MF Port
7-5 2-0 31-29 26-24
T2
7-5 26-24
0-2 3-5 6-8 9-11
HG MUX4 HG MUX2 HG MUX5 HG MUX6
MN Port
0 1 2 3 4 5 6 7 8 9 10 11
Northstar 2
MF Port
0-2 9-11
MN Port
0 1 2 3 4 5 6 7 8 9 10 11

Multicast L3 Forwarding
• Before hardware can forward any Multicast packets,
forwarding information has to propagate from Sup to the LC
• Several layers are to be verified:
MRIB (control-plane is created here)
MFDM PI /PD (platform independent & forwarding
information)
• MFIB-IPFIB
• IP FIB process programs hardware:
FIB Table contains (*,G) and (S,G) forwarding entries and RPF
information
GROUP table contains forwarding and pointers replication
information (pointers to MC VLAN)
MC VLAN tables contain replication information (~OIF lists)
Hardware (packets are forwarded here) & SDK
Supervisor
MRIB
MF DM
IP FIB
IGMPPIM MSDP
T2
FIB Table MC VLAN Table
IPMC_GR
Line Card

L2/L3 Multicast Packet Walk
Fabric Module
Trident II
Parser
Network Interfaces
L2/L3
Lookup &
pkt rewrite
10GE 40GE
EACL
Egress Q
Trident II
Parser
L2/L3
Lookup &
Pkt rewrite
EACL
Egress Q
Trident II
IACL
Traffic
Classification
& Remarking
IACL
Traffic
Classification&
Remarking
Network Interfaces
10GE 40GE
Lkup in Host Table
& L2 Table
Lookup to resolve egr.
modules;
Sends one copy to each
egr. module;
Examines ingress
packet. Get packet
headers for
processing.
Lookup for local
receiving ports;
replicate pkts onto
those ports.L2/L3 mcast lookup;
Replicate pckts to local
receiving ports;
Send 1 copy to fabric
module;

Multicast L3 Forwarding-MRIB
N9K# show ip mroute 239.10.10.10 shared-tree
IP Multicast Routing Table for VRF "default”
(*, 239.10.10.10/32), uptime: 00:23:32, ip pim
Incoming interface: Ethernet6/1, RPF nbr:
13.13.13.1
Outgoing interface list: (count: 1)
Ethernet6/52, uptime: 00:22:42, pim
Supervisor
MRIB
MF DM
IP FIB
PIM MSDPIGMP

Multicast L3 Forwarding-mFDM PI-Supervisor
N9K# show forwarding distribution multicast outgoing-
interface-list l3 1
Outgoing Interface List Index: 1
Reference Count: 4
Platform Index: 0xb00001
Number of Outgoing Interfaces: 1 t6/52
N9K# show forwarding distribution ip multicast route group 239.10.10.10 source
13.13.13.14 | in 13|Index
(13.13.13.14/32, 239.10.10.10/32), RPF Interface: Ethernet6/1, flags:
Supervisor
MRIB
MF DM
IP FIB
PIM MSDPIGMP

Multicast L3 Forwarding IPFIB-Line card
N9K# show forwarding ip multicast route group 239.10.10.10 source
13.13.13.14 mod 6 | inc 239|Eth
(13.13.13.14/32, 239.10.10.10/32), RPF Interface: Ethernet6/1, flags:
Outgoing Interface List Index: 0x1
Ethernet6/52
T2
IPMC_GR
Line Card
Mod 6 is N9K-X9564TX
To reach Ethernet 6/52 which is on NS from front port of T2,Packets need to
cross Fabric module

Multicast L3 Forwarding Entries on LC –BCM Shell
N9K# bcm-shell mod 6 "ipmc table show"
SRC IP ADDRESS MC IP ADDRESS MC GROUP VID VRF COS HWIDX CLASS HIT
13.13.13.14 239.10.10.10 0x2000007 0 1 0 75680 1 no
0.0.0.0 239.10.10.10 0x2000007 0 1 0 86578 2 no
N9K#bcm-shell module 6 "mc show group=0x2000007"
Group 0x2000007 (L3)
port hg0, encap id -1
-------snip------------
port hg11, encap id -1
T2
FIB Table MC VLAN
Table IPMC_GR
Line Card
Traffic spared to Hig towards Fabric

Multicast L3 Forwarding Entries on LC –BCM Shell
N9K# bcm-shell mod 6 " search l3_entry_ipv4_multicast group_ip_addr=0xef0a0a0a
source_ip_addr=0x0d0d0d0e”
L3_ENTRY_IPV4_MULTICAST.ipipe0[75680]:
SOURCE_IP_ADDR=0xd0d0d0e,
GROUP_IP_ADDR=0xef0a0a0a,
L3MC_INDEX=7
N9K# bcm-shell mod 6 " dum chg l3_entry_ipv4_multicast 75680”
IPV4MC:EXPECTED_L3_IIF=0x112e,
N9K# show system internal eltm info interface ethernet 6/1 | in LIF
cr_flags = INTF LIF , LIF = 4398 (0x112e), LTL = 40959 (0x9fff) (S 0x0, P 0x0)
T2
IPMC_GR
Line Card
show tech-support multicast`
show tech-support forwarding multicast

IGM Snooping
Forwarding programming in vPC Scenario
• IGMP Process Provides both Layer 3 IGMP Processing , and Layer 2 IGMP snooping functionality
• Receivers use IGMP (Internet Group Management Protocol) to report their multicast group
Membership to router
• Layer 2 IGMP Snooping functions of IGMP process include processing snooped multicast router
Packets Including IGMP reports and leaves sent by receiver
• Once the group membership is learned , the Supervisor Engine informs I/O modules , which
program Hardware
• This will Constrain data-plane multicast packets to only those ports with multicast routeror interested
receivers in HW

IGMP Snooping continued…
• BCM on FM are in Mode 4. This will have L2 Table size of 32K & L3 Host Table 16K
• L3 Host table will be used to program (*,G) /(S,G) entry. This will will accommodate
maximum of 8K entry.
• MFDM sends two OIF List information to MFIB. One for LC (S,G) OIF List and other for
FM ( Mac, Group) OIF List in PIM disable Vlan.
• MFIB will use (S,G) OIF list to program LC and Mac Group to Program FM in 32K L2 Table.
• If PIM is enable FM can accommodate 8K(VRF, S,G) and will program Hardware.
• Address aliasing is possible because on FM we use L2 table to program Mac Group information

IGMP Snooping (Cont’d)
• With vPC IGMP will have knowledge of multi chassis Ether Channel trunk (MCT) interface.
• When one of the vPC peer receives IGMP join , it will sync up this with peer over MCT link
using cFS-Cisco Fabric Services over Ethernet .
• Duplication of traffic crossing MCT is avoided using Port block Mask
• VPC Support PIM-SM Only
• For source in VPC domain – dual Forwarders are used
• For Source in Layer 3 Cloud , Unicast best metric determines active forwarder
• VPC Operational Primary in case of tie. CFS used to negotiate active Forwarder role

Configuration-IGMP Snooping enable by default
Nexus9508-13# sh ip igm snooping vlan 103
IGMP Snooping information for vlan 103
IGMP snooping enabled
Lookup mode: IP
Optimised Multicast Flood (OMF) enabled
IGMP querier present, address: 10.10.103.5, version: 2, i/f Po30
Nexus9508-13# sh ip igm snooping vlan 100
IGMP Snooping information for vlan 100
IGMP snooping enabled
Lookup mode: IP
Optimised Multicast Flood (OMF) enabled
IGMP querier present, address: 192.168.100.2, version: 2, i/f Vlan100
Querier interval: 125 secs
Querier last member query interval: 1 secs

Reference Topology for Troubleshooting
N35K
Eth 1/17,Eth 1/19 , Eth 1/33-34
N9508-12 N9508-13
N93k
vPC 35vPC30
vPC Keep Alive
vPC Peer Link PO-10
Ixia 10/2-Source Ixia 10/1-Receiver
Eth1/3/1-4
Eth 6/9/1-4
Eth 1/48
Eth1/48
Eth 3/1-2 Eth 3/1-2
Eth 1/17-18 ,Eth 1/33-34

IGMP Snooping Troubleshooting
• Stream will enter one of the VPC-Peer , Which will get forwarded across Peer link to other VPC Peer
• Both boxes will have (S ,G)
• Upon Creation of (S,G) , VPC Peers negotiate best metric
• Both realize source is VPC-Connected
• Install Entry as Win-Force
• If either peer gets a PIM/IGMP Join for the given source , they both add Interface to OIF
Nexus9508-12(config)# sh ip pim internal vpc rpf-source
PIM vPC RPF-Source Cache for Context "default" - Chassis
Role Primary
Source: 192.168.100.10
Pref/Metric: 0/0
Source role: primary
Forwarding state: Win-force (forwarding)
MRIB Forwarding state: forwarding
Nexus9508-13# sh ip pim internal vpc rpf-source
PIM vPC RPF-Source Cache for Context "default" - Chassis
Role Secondary
Source: 192.168.100.10
Pref/Metric: 0/0
Source role: secondary
Forwarding state: Win-force (forwarding)
MRIB Forwarding state: forwarding

• IGMP Join from one of the receiver enter one of the VPC Pee.
• This Peer encapsulates IGMP in CFS , sends to other Peer
• Both Peer have identical State
• Both Peer install OIF
• Data traffic flows down to Receiver, also forwarded to other Peer on Peer Link
• Other Peer drop the packet either by PORT BLOCK MASK blocking or no OIF
Nexus9508-ESC-12# sh ip mroute 239.10.10.10 192.168.100.10
IP Multicast Routing Table for VRF "default"
(192.168.100.10/32, 239.10.10.10/32), uptime: 01:00:09, ip pim
mrib
Incoming interface: Vlan100, RPF nbr: 192.168.100.10, uptime:
01:00:09, internal
Vlan101, uptime: 00:59:40, mrib
Nexus9508-ESC-12#
Nexus9508-ESC-13# sh ip mroute 239.10.10.10 192.168.100.10
IP Multicast Routing Table for VRF "default"
(192.168.100.10/32, 239.10.10.10/32), uptime: 04:25:36, ip pim
mrib
Incoming interface: Vlan100, RPF nbr: 192.168.100.10, uptime:
04:25:36
Vlan101, uptime: 02:04:41, mrib
Nexus9508-ESC-13#
vPC Peer receiving Join

Step to verify PI On Supervisor. Verify on Both Peers
Nexus9508-ESC-12# sh ip igmp groups 239.10.10.10
IGMP Connected Group Membership for VRF "default" - matching Group "239.10.10.10"
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
Group Address Type Interface Uptime Expires Last Reporter
239.10.10.10 D Vlan101 00:01:23 00:02:56 192.168.101.13
Nexus9508-ESC-12#
Nexus9508-ESC-13# sh ip igmp groups 239.10.10.10
IGMP Connected Group Membership for VRF "default" - matching Group "239.10.10.1
0"
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
Group Address Type Interface Uptime Expires Last Reporter
239.10.10.10 D Vlan101 00:01:18 00:03:01 192.168.101.13
Nexus9508-ESC-13#

CFS Provide info
Nexus9508-ESC-12# sh ip igmp snooping groups vlan 101 detail
IGMP Snooping group membership for vlan 101
Group addr: 239.10.10.10
Group ver: v2 [old-host-timer: not running]
Last reporter: 192.168.101.10
IGMPv2 member ports:
IGMPv1/v2 memb ports:
Po35 [1 GQ missed], cfs:false, native:true
vPC grp peer-link flag: exclude
M2RIB vPC grp peer-link flag: exclude
Nexus9508-ESC-12#
Nexus9508-ESC-13# sh ip igm snooping groups vlan
101 det
IGMP Snooping group membership for vlan 101
Group addr: 239.10.10.10
Group ver: v2 [old-host-timer: not running]
Last reporter: 192.168.101.10
IGMPv2 member ports:
IGMPv1/v2 memb ports:
Po35 [0 GQ missed], cfs:true, native:false
vPC grp peer-link flag: exclude
M2RIB vPC grp peer-link flag: exclude
Nexus9508-ESC-13#

Verifying Multicast forwarding Distribution Module
Platform Independent On Supervisor
Nexus9508-ESC-12# sh forwarding distribution multicast route group 239.10.10.10
source 192.168.100.10
(192.168.100.10/32, 239.10.10.10/32), RPF Interface: Vlan100, flags:
Received Packets: 1073 Bytes: 36977
Number of Outgoing Interfaces: 2
Vlan100
( Mem L2 Ports: port-channel10 )
Vlan101
( Mem L2 Ports: port-channel35 )
Note: On shutting down local vpc only, igmp does not send update to mfdm/ipfib to update the mroute state.
That is why you did not see mfdm/ipfib removing local vpc. So if local leg of vPC is down we will still PC in the above output.
Not showing PC 10 for Vlan 101 because of
exclude flag seen while checking igmp
snooping stats.

Platform Independent On Supervisor-(Cont’d)
Nexus9508-12# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 1
Number of next hops: 2
Vlan: 101
port-channel35
bridged Vlan
port-channel10
Hardware Outgoing Interface List Index: 33554443

Platform Independent On Supervisor-IGMP-Snooping
Nexus9508-12# sh forwarding distribution ip igmp snooping vlan 101 group 239.10.10.10 det
Vlan: 101, Group: 239.10.10.10, Source: 0.0.0.0
Reference Count: 1
Platform Index: 0xa00004
Vpc peer link exclude flag set
port-channel10
port-channel35
Nexus9508-13# sh forwarding distribution ip igmp snooping vlan 101 group
239.10.10.10 det
Vlan: 101, Group: 239.10.10.10, Source: 0.0.0.0
Reference Count: 1
port-channel10
port-channel35

Verifying Multicast Forwarding Distribution Module
Platform Independent On Supervisor-Snooping Group.
Nexus9508-12# sh forwarding distribution l2 multicast mac-based vlan
101
Vlan: 101, Group: 0100.5e0a.0a0a, Source: 0000.0000.0000
Reference Count: 1
port-channel10
port-channel35
Nexus9508-13# sh forwarding distribution l2 multicast mac-based vlan 101
Vlan: 101, Group: 0100.5e0a.0a0a, Source: 0000.0000.0000
Reference Count: 1
port-channel10
port-channel35

IPFIB on LC for IGMP Snooping programming.
Nexus9508--12# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 1
port-channel30 (Vlan: 101)
port-channel10 (bridged)
Nexus9508-13# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 6
port-channel30 (Vlan: 101)
port-channel10 (bridged)

Nexus9508--12# bcm-shell mod 1 "mc show group=33554441"
Executing mc show group=33554441 on bcm shell on module 1
Group 0x2000009 (L3)
port hg0, encap id 400005
port xe10, encap id 21
port xe11, encap id 21
Verifying Hardware Programming
Nexus9508-12# bcm-shell mod 3 "mc show group=33554441"
Executing mc show group=33554441 on bcm shell on module 3
Group 0x2000009 (L3)
port xe0, encap id -1
port xe1, encap id -1
Nexus9508-12# sh system internal eltm info interface vlan 101 | in LIF
cr_flags = INTF VLAN , LIF = 21 (0x15), LTL = -1 (0xffffffff) (S 0x0, P 0x0)
Nexus9508-ESC-12#
If we see encap id a positive #
then it is LIF
If we see encap id = -1 then it is
L2 bridge copy.

Nexus9508-12# bcm-shell module 1 "l2 show" | in MCast
mac=01:00:5e:0a:0a:0a vlan=101 GPORT=0x0 modid=0 port=0 Static Hit MCast=33554435
mac=01:00:5e:0a:0a:14 vlan=100 GPORT=0x0 modid=0 port=0 Static MCast=33554435
Nexus9508-12# sh ip igmp gr vlan 100
From BCM to check what is HW index for given Group
• Static entry of Mcast group
• Hit Bit indicate flow is present
• Mcast Index is where the traffic need to bridge
show tech-support ip igmp snooping
show tech-support ip multicast

Brkdct 3101

More Related Content

What's hot

Similar to Brkdct 3101

Recently uploaded

Brkdct 3101