Uploaded byKalkiNarayana
45 views

N7K Hardware Architecture and it's components

The document provides a comprehensive overview of the architecture and design of Cisco's Nexus 7000 and 7700 data center switches, focusing on components such as supervisors, fabrics, and I/O modules. It aims to enhance understanding of key functionalities, including packet flows and forwarding engines while outlining the differences and similarities between the two switch families. The presentation does not cover the NX-OS software architecture or other Nexus platform architectures.

Embed presentation

Download to read offline
Cisco Nexus 7000 Switch Architecture BRKARC-3470 Ron Fuller, CCIE#5851 (R&S/Storage) Technical Marketing Engineer
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 3 Session Abstract This session presents an in-depth study of the architecture of the latest generation of Nexus 7000 and Nexus 7700 data centre switches. Topics include supervisors, fabrics, I/O modules, forwarding engines, and physical design elements, as well as a discussion of key hardware-enabled features that combine to implement high-performance data centre network services.
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 4 Session Goal  To provide a thorough understanding of the Nexus 7000 / Nexus 7700 switching architecture, supervisor, fabric, and I/O module design, packet flows, and key forwarding engine functions  This session will examine the Nexus 7700 system, as well as the latest additions to the Nexus 7000  This session will not examine NX-OS software architecture or other Nexus platform architectures 4
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 5 What Is Nexus 7000? Data-centre class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection Nexus 7000 designed for general-purpose Data Centre deployments, focused on 10G density plus 40G/100G I/O Modules Supervisor Engines Fabrics Chassis
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 6 What Is Nexus 7700? Data-centre class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection Nexus 7700 designed for SP and MSDC Data Centre deployments, focused on high- density 40G/100G I/O Modules Supervisor Engine Fabrics Chassis
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 7 Nexus 7000 General purpose DC switching w/10/40/100G Nexus 7700 Targeted at Densest 40G/100G deployments Common Foundation • Same release vehicles, versioning, feature-sets • Common configuration model • Common operational model • Common fabric ASICs (Fab2) and architecture • Same central arbitration model • Same VOQ/QoS model • Identical forwarding ASICs (F2E, F3) • Consistent hardware feature sets • Parallel evolution of hardware capability/scale Nexus 7000 / Nexus 7700 – Common Foundation
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 8 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 9 Nexus 7000 Chassis Family Front Rear 21RU N7K-C7010 25RU Front Rear N7K-C7018 Front Rear N7K-C7009 14RU NX-OS 4.1(2) and later NX-OS 5.2(1) and later Nexus 7010 Nexus 7018 Nexus 7009 Front N7K-C7004 7RU NX-OS 6.1(2) and later Rear Nexus 7004 Front Back Side Side Side Side Side Back
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 10 Nexus 7700 Chassis Family Front Rear 26RU N77-C7718 Nexus 7718 Front Rear 14RU N77-C7710 Nexus 7710 Front Rear 9RU N77-C7706 Nexus 7706 NX-OS 6.2(6) and later NX-OS 6.2(2) and later NX-OS 6.2(2) and later Front Back Front Back Front Back
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 11 Key Chassis Components Nexus 7000  Common components: – Supervisor engines – I/O modules – Power supplies (except 7004)  Chassis-specific components: – Fabric modules – Fan trays Nexus 7700  Common components: – Supervisor engines – I/O modules – Power supplies  Chassis-specific components: – Fabric modules – Fan trays Common hardware components between Nexus 7000 and Nexus 7700: NONE No interchangeable hardware components between Nexus 7000 and Nexus 7700
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 12 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 13  Next generation supervisors providing control plane and management functions  Connects to fabric via 1G inband interface  Interfaces with I/O modules via 1G switched EOBC  Second-generation dedicated central arbiter ASIC – Controls access to fabric bandwidth via dedicated arbitration path to I/O modules Supervisor Engine 2 / 2E Console Port Management Ethernet N7K-SUP2/N7K-SUP2E USB Host Ports ID and Status LEDs Supervisor Engine 2 (Nexus 7000) Supervisor Engine 2E (Nexus 7000 / Nexus 7700) Base performance High performance One quad-core 2.1GHz CPU with 12GB DRAM Two quad-core 2.1GHz CPU with 32GB DRAM USB Log Flash USB Expansion Flash N77-SUP2E ID and Status LEDs Console Port Management Ethernet USB Expansion Flash
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 14 Nexus 7000 / 7700 I/O Module Families M1 1G and 10G M2 10G / 40G / 100G F1 10G F2 10G F2E 10G F3 40G F2E 10G F3 10G / 40G / 100G F3 closes the F/M feature gap!
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 15  10G / 40G / 100G M2 I/O modules  Share common hardware architecture  Two integrated forwarding engines (120Mpps) – Support for “XL” forwarding tables (licensed)  Distributed L3 multicast replication  802.1AE LinkSec on all ports N7K-M224XP-23L Nexus 7000 M2 I/O Modules N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L Supported in NX-OS release 6.1(1) and later N7K-M206FQ-23L N7K-M202CF-22L Module Port Density Optics Bandwidth M2 10G 24 x 10G (plus Nexus 2000 FEX support) SFP+ 240G M2 40G 6 x 40G (or up to 24 x 10G via breakout) QSFP+ 240G M2 100G 2 x 100G CFP 200G
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 16 Nexus 7000 M2 I/O Module Architecture N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L LinkSec + 12 X 10G MAC -or- 3 X 40G MAC -or- 1 X 100G MAC Forwarding Engine VOQs Fabric 2 ASIC To Fabric Modules Replication Engine Replication Engine Front Panel Ports LC CPU EOBC VOQs LinkSec + 12 X 10G MAC -or- 3 X 40G MAC -or- 1 X 100G MAC Forwarding Engine VOQs Replication Engine Replication Engine VOQs To Central Arbiters Arbitration Aggregator …
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 18 Nexus 7000 / 7700 F2E I/O Modules N7K-F248XP-25E / N7K-F248XT-25E / N77-F248XP-23E 7000: Supported in NX-OS release 6.1(2) and later 7700: Supported in NX-OS release 6.2(2) and later N7K-F248XP-25E N7K-F248XT-25E  48-port 1G/10G with SFP/SFP+ transceivers  480G full-duplex fabric connectivity  System-on-chip (SoC) forwarding engine design – 12 independent SoC ASICs  Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS)  Interoperability with M1/M2, in Layer 2 mode on Nexus 7000 – Proxy routing for inter-VLAN/L3 traffic  LinkSec support* – Last 8 ports (SFP+) – All 48 ports (Copper)  Supports Nexus 2000 (FEX) connections * Roadmap item N77-F248XP-23E
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 19 Nexus 7000 F2E Module Architecture N7K-F248XP-25E / N7K-F248XT-25E 4 X 10G SoC Front Panel Ports To Fabric Modules Fabric 2 2 4 LC CPU EOBC To Central Arbiters Arbitration Aggregator … 4 X 10G SoC 6 8 4 X 10G SoC 10 12 4 X 10G SoC 14 16 4 X 10G SoC 18 20 4 X 10G SoC 22 24 4 X 10G SoC 26 28 4 X 10G SoC 30 32 4 X 10G SoC 34 36 4 X 10G SoC 38 40 4 X 10G SoC 42 44 4 X 10G SoC 46 48 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 LinkSec-capable (F2E fibre) LinkSec-capable (F2E copper)
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 20 Nexus 7700 F2E Module Architecture N77-F248XP-23E 4 X 10G SoC Front Panel Ports To Fabric Modules Fabric 2 2 4 LC CPU EOBC To Central Arbiters Arbitration Aggregator … 4 X 10G SoC 6 8 4 X 10G SoC 10 12 4 X 10G SoC 14 16 4 X 10G SoC 18 20 4 X 10G SoC 22 24 4 X 10G SoC 26 28 4 X 10G SoC 30 32 4 X 10G SoC 34 36 4 X 10G SoC 38 40 4 X 10G SoC 42 44 4 X 10G SoC 46 48 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 LinkSec-capable Fabric 2 To Fabric Modules
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 22 Nexus 7000 F3 40G Module  12-port 40G QSFP+ module  480G full-duplex fabric connectivity  SoC forwarding engine design – 6 independent SoC ASICs  Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features  Fabric Services Accelerator (FSA) CPU  Breakout cable support  Requires Supervisor Engine 2 / 2E N7K-F312FQ-25
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 23 Nexus 7000 12-Port 40G Module Architecture 1 Front Panel Ports (QSFP+) To Fabric Modules FSA CPU EOBC To Central Arbiters Arbitration Aggregator 2 X 40G SoC 1 2 X 40G SoC 2 2 X 40G SoC 3 2 X 40G SoC 4 2 X 40G SoC 5 2 X 40G SoC 6 Fabric ASIC LC Inband 2 3 4 5 6 7 8 9 10 11 12 … x 6 to FSA CPU to ARB x 6 1G switch x 6 …
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 24 FSA CPU Fabric Services Accelerator (FSA)  High-performance module CPU with on-board acceleration engines – 6Gbps inband connectivity from SOCs to FSA – Multi-Mpps packet processing – 2GB dedicated DRAM  Performance/scale boost for distributed fabric services, including BFD and sampled NetFlow (roadmap)  Other potential applications include distributed ARP/ping processing, data plane packet analysis (wireshark), network probing, etc. 6 x 1Gbps Module Inband I/O 2GB DRAM Dual-Core LC CPU Acceleration Engines 2GB DRAM EOBC
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 25 Nexus 7700 F3 48-Port 1G/10G Module  48-port 1G/10G with SFP/SFP+ transceivers  480G full-duplex fabric connectivity  SoC-based forwarding engine design – 6 independent SoC ASICs  Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features  Fabric Services Accelerator (FSA) CPU  LinkSec support (last 8 ports)*  Supports Nexus 2000 (FEX) connections N77-F348XP-23 * Roadmap item
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 26 8 X 10G SoC 1 Nexus 7700 F3 48-Port 1G/10G Module Architecture To Fabric Modules To Central Arbiters Arbitration Aggregator 8 X 10G SoC 2 8 X 10G SoC 3 8 X 10G SoC 4 8 X 10G SoC 5 8 X 10G SoC 6 Fabric ASIC Fabric ASIC … x 6 1 Front Panel Ports (SFP/SFP+) 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 LinkSec-capable to FSA CPU to ARB FSA CPU EOBC LC Inband x 6 1G switch x 6 …
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 27 Nexus 7700 F3 40G and 100G Modules  24-port 40G QSFP+ module / 12-port 100G CPAK module  960G/1.2T full-duplex fabric connectivity  SoC forwarding engine design – 12 independent SoC ASICs  Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features  Fabric Services Accelerator (FSA) CPU  40G breakout cable support* N77-F324FQ-25 N77-F312CK-26 * Roadmap item
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 28 2 X 40G SoC 1 Nexus 7700 F3 24-Port 40G Module Architecture 1 Front Panel Ports (QSFP+) To Fabric Modules FSA CPU EOBC To Central Arbiters Arbitration Aggregator 2 X 40G SoC 2 2 X 40G SoC 3 2 X 40G SoC 4 2 X 40G SoC 5 2 X 40G SoC 6 2 X 40G SoC 7 2 X 40G SoC 8 2 X 40G SoC 9 2 X 40G SoC 10 2 X 40G SoC 11 2 X 40G SoC 12 Fabric ASIC Fabric ASIC LC Inband 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1G switch … … x 12 to FSA CPU to ARB x 12 x 6
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 29 Nexus 7700 F3 12-Port 100G Module Architecture Front Panel Ports (CPAK) To Fabric Modules To Central Arbiters Arbitration Aggregator 1 X 100G SoC 2 2 1 X 100G SoC 3 3 1 X 100G SoC 4 4 1 X 100G SoC 5 5 1 X 100G SoC 6 6 1 X 100G SoC 7 1 X 100G SoC 8 1 X 100G SoC 9 1 X 100G SoC 10 1 X 100G SoC 11 Fabric ASIC Fabric ASIC 7 8 9 10 11 1 X 100G SoC 12 12 1 X 100G SoC 1 1 FSA CPU EOBC LC Inband 1G switch … … x 12 to FSA CPU to ARB x 12 x 6
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 30 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 31 M-Series Forwarding Engine Hardware  Two hardware forwarding engines integrated on every M2 I/O module  120Mpps (60Mpps per forwarding engine) Layer 2 bridging with hardware MAC learning  120 Mpps (60Mpps per forwarding engine) Layer 3 IPv4  60Mpps (30Mpps per forwarding engine) Layer 3 IPv6 unicast  Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir)  MPLS/VPLS/EoMPLS  OTV  RACL/VACL/PACL  QoS remarking and policing policies  Policy-based routing (PBR)  Unicast RPF check and IP source guard  IGMP snooping  Ingress and egress NetFlow (full and sampled) Hardware Table M-Series Modules without Scale License M-Series Modules with Scale License MAC Address Table 128K 128K FIB TCAM 128K IPv4 / 64K IPv6 900K IPv4 / 350K IPv6 Classification TCAM (ACL/QoS) 64K 128K NetFlow Table 1M 1M
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 32 From I/O Module Replication Engines To I/O Module Replication Engines M-Series Forwarding Engine Architecture L2 Engine Ingress Parser MAC Table L2 Lookup (pre-L3) L2 Lookup (post-L3) Final Results L3 Engine Classification (ACL/QoS) NetFlow Layer 3 FIB Policing FIB TCAM/ ADJ CL TCAM FE Daughter Card Ingress lookup pipeline Egress lookup pipeline  Egress NetFlow collection  Ingress MAC table lookups  Port-channel hash result  Ingress IGMP snooping lookups  FIB TCAM and adjacency table lookups for Layer 3 forwarding  ECMP hashing  Multicast RPF check  Ingress policing  Egress MAC lookups  Egress IGMP snooping lookups PKT HDR  Egress ACL/QoS classification  Ingress NetFlow collection  Ingress ACL/QoS classification  Egress policing
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 33 F2E Forwarding Engine Hardware  Each SoC forwarding engine services 4 front-panel 10G ports (12 SoCs per module)  60Mpps per SoC Layer 2 bridging with hardware MAC learning  60Mpps per forwarding engine Layer 3 IPv4/ IPv6 unicast  Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir*)  RACL/VACL/PACL  QoS remarking and policing policies  Policy-based routing (PBR)  Unicast RPF check and IP source guard  IGMP snooping  FabricPath forwarding  FCoE (with Sup2 / Sup2E) – Roadmap on Nexus 7700  Ingress sampled NetFlow Hardware Table Per F2E SoC Per F2E Module MAC Address Table 16K 192K* FIB TCAM 32K IPv4/16K IPv6 32K IPv4/16K IPv6 Classification TCAM (ACL/QoS) 16K 192K* * Assumes specific configuration to scale SoC resources * Roadmap item
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 34 F3 Forwarding Engine Hardware  Each SoC forwarding engine services: – 8 front-panel 10G ports – 2 front-panel 40G ports – 1 front-panel 100G port  148Mpps per SoC Layer 2 bridging with hardware MAC learning  148Mpps per forwarding engine Layer 3 IPv4/ IPv6 unicast  Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir*)  RACL/VACL/PACL  QoS remarking and policing policies  Policy-based routing (PBR)  Unicast RPF check and IP source guard  IGMP snooping  FabricPath forwarding  Overlay Transport Virtualisation (OTV)  MPLS/VPLS/EoMPLS, LISP, VXLAN, GRE, FCoE*  Ingress/egress* sampled NetFlow Hardware Table Per F3 SoC Per F3 Module MAC Address Table 64K 384K/768K** FIB TCAM 64K IPv4/32K IPv6 64K IPv4/32K IPv6 Classification TCAM (ACL/QoS) 16K 96K/192K** ** Assumes specific configuration to scale SoC resources * Roadmap items
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 35 F3 Forwarding Engine Decision Engine Layer 3 Lookups QoS / ACL Ingress Parser MAC Table FIB/ADJ CL L2 Lookup (post-L3) Front-panel To/From Central Arbiter To Fabric From Fabric Ingress Buffer (VOQ) Virtual output queues L2 Lookup (pre-L3) Egress Parser F3 SoC Ingress and egress forwarding decisions (L2/L3 lookups, ACL/QoS, features etc.) 8 x 1/10G OR 2 x 40G OR 1 x 100G per ASIC Forwarding tables 1G / 10G / 40G / 100G 1G / 10G / 40G / 100G capable interface MAC Egress Buffer Egress fabric receive buffer HDR PKT HDR PKT PKT HDR
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 36 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 37 Crossbar Switch Fabric Modules  Provide interconnection of I/O modules  Each installed fabric increases available per-payload slot bandwidth  Nexus 7000 and Nexus 7700 fabrics based on Fabric 2 ASIC  Different I/O modules leverage different amount of available fabric bandwidth  Access to fabric bandwidth controlled using QoS-aware central arbitration with VOQ N7K-C7018-FAB-2 N7K-C7010-FAB-2 N7K-C7009-FAB-2 Fabric Module Supported Chassis Per-fabric module bandwidth Max fabric modules Total bandwidth per slot Nexus 7000 Fabric 2 7009 / 7010 / 7018 110Gbps per slot 5 550Gbps per slot Nexus 7700 Fabric 2 7706 / 7710 / 7718 220Gbps per slot 6 1.32Tbps per slot N77-C7718-FAB-2 N77-C7710-FAB-2 N77-C7706-FAB-2
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 38 110G (2 x 55G) Ingress Module Egress Module Multistage Crossbar Nexus 7000 / Nexus 7700 implement 3-stage crossbar switch fabric  Stages 1 and 3 on I/O modules  Stage 2 on fabric modules 1st stage Egress Module 2nd stage Ingress Module 3rd stage Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric Modules Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC 1 Fabric ASIC 2 3 4 5 Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC 6 Fabric ASIC 1.32T 1st stage 3rd stage 550G 110G (2 x 55G) 1 Fabric ASIC 2 3 4 5 Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric Modules Nexus 7000 Nexus 7700
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 39 110Gbps 220Gbps 330Gbps 440Gbps 550Gbps Local Fabric 2 (480G) Local Fabric 2 (240G) I/O Module Capacity – Nexus 7000 One fabric:  Any port can pass traffic to any other port in VDC Three fabrics:  240G M2 module has maximum bandwidth Five fabrics:  480G F2E/F3 module has maximum bandwidth Fabric 2 Modules 1 Fabric 2 ASIC 2 Fabric 2 ASIC 3 Fabric 2 ASIC 4 Fabric 2 ASIC 5 Fabric 2 ASIC per slot bandwidth
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 40 What About Nexus 7004?  Nexus 7004 has no fabric modules  I/O modules have local fabric with 10 available fabric channels – I/O modules connect “back-to-back” via 8 fabric channels – Two fabric channels “borrowed” to connect supervisor engines Sup Slot 2 Sup Slot 1 M2/F2E/F3 Module 4 M2/F2E/F3 Module 3 Fabric ASIC Fabric 2 ASIC Fabric 2 ASIC Fabric ASIC 2 * 55G fabric channels 8 * 55G local fabric channels interconnect I/O modules (440G)
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 41 220Gbps 440Gbps 660Gbps 880Gbps 1100Gbps 1320Gbps Local Fab2 #1 (480G) Local Fab2 #1 (960G) Local Fab2 #1 (1.2T) Fab2 #2 Fab2 #2 Fab2 #2 I/O Module Capacity – Nexus 7700 One fabric:  Any port can pass traffic to any other port in VDC Three fabrics:  480G F2E/F3 10G module has maximum bandwidth Five fabrics:  960G F3 40G module has maximum bandwidth Six fabrics:  1.2T F3 100G module has maximum bandwidth per slot bandwidth Fabric 2 Modules 1 Fabric 2 ASICs 2 Fabric 2 ASICs 3 Fabric 2 ASICs 4 Fabric 2 ASICs 5 Fabric 2 ASICs 6 Fabric 2 ASICs
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 42 Fabric, VOQ, and Arbitration  Crossbar fabric – Provides dedicated, high-bandwidth interconnects between ingress and egress I/O modules  Virtual Output Queues (VOQs) – Provide buffering and queuing for ingress- buffered switch architecture  Central arbitration – Controls scheduling of traffic into fabric based on fairness, priority, and bandwidth availability at egress ports  Fabric, VOQ, and arbitration combine to provide all necessary infrastructure for packet transport inside switch
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 43 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 44 Buffering, Queuing, and Scheduling  Buffering – storing packets in memory – Needed to absorb bursts, manage congestion  Queuing – buffering packets according to traffic class – Provides dedicated buffer for packets of different priority  Scheduling – controlling the order of transmission of buffered packets – Ensures preferential treatment for packets of higher priority and fair treatment for packets of equal priority  Nexus 7000 / Nexus 7700 use queuing policies and network-QoS policies to define buffering, queuing, and scheduling behaviour  Default queuing and network-QoS policies always in effect in absence of any user configuration
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 45 I/O Module Buffering Models  Buffering model varies by I/O module family – M-series modules: hybrid model combining ingress VOQ-buffered architecture with egress port-buffered architecture – F-series modules: pure ingress VOQ-buffered architecture
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 46 Egress Module Ingress Module VOQ 0 Port ASIC 0 M2 – Hybrid Ingress/Egress Buffered DWRR VOQ 1 RE 1 RE 0 … DWRR Port 1 Port 12 FABRIC VOQ 0 VOQ 1 RE 1 RE 0 Port ASIC 0 Port 1 Port 12 Ingress port buffer – Manages congestion of ingress forwarding/replication engines, and congestion toward egress destinations (VQIs) Buffering / queuing / scheduling INGRESS QUEUING POLICIES 10G module used as example Diagram represents half of each I/O module 8 ingress queues
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 47 Egress Module Ingress Module VOQ 0 1 2 3 4 5 6 SP 2 3 4 VOQ Buffer Source Priority Port ASIC 0 M2 – Hybrid Ingress/Egress Buffered DWRR VOQ 1 RE 1 RE 0 … DWRR Port 1 Port 12 FABRIC VOQ 0 VQI 1 DWRR SP … VQI 6 DWRR SP VOQ 1 Sources 7-12 VQIs 7-12 RE 1 RE 0 Port ASIC 0 Port 1 Port 12 Ingress port buffer – Manages congestion of ingress forwarding/replication engines, and congestion toward egress destinations (VQIs) Buffering / queuing / scheduling Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling FABRIC-QOS POLICY 10G module used as example Diagram represents half of each I/O module Shared buffer carved by source and priority 4 priority levels
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 48 Egress Module Ingress Module VOQ 0 1 2 3 4 5 6 SP 2 3 4 VOQ Buffer Source Priority Port ASIC 0 DWRR VOQ 1 RE 1 RE 0 … DWRR Port 1 Port 12 FABRIC VOQ 0 VQI 1 DWRR SP … VQI 6 DWRR SP VOQ 1 Sources 7-12 VQIs 7-12 RE 1 RE 0 Port ASIC 0 Port 1 Port 12 DWRR SP … DWRR SP Ingress port buffer – Manages congestion of ingress forwarding/replication engines, and congestion toward egress destinations (VQIs) Buffering / queuing / scheduling Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling Egress port buffer – Manages congestion at egress physical interface Buffering / queuing / scheduling EGRESS QUEUING POLICIES 10G module used as example Diagram represents half of each I/O module 8 egress queues M2 – Hybrid Ingress/Egress Buffered
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 49 Egress SOC Ingress SOC Ingress VOQ F2E – Ingress Buffered (Nexus 7000) FABRIC Egress VOQ VQI 1 DWRR PQ … VQI 4 DWRR PQ Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling 1 2 3 4 hi VOQ Buffer lo hi lo hi lo hi lo 10G Port 1 10G Port 2 10G Port 3 10G Port 4 10G Port 1 10G Port 2 10G Port 3 10G Port 4 Diagram represents one SoC on each I/O module INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES 2 or 4 ingress queues per port 4 priority levels
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 50 Egress SOC Ingress SOC Ingress VOQ F3 10G – Ingress Buffered (Nexus 7700) FABRIC Egress VOQ VQI 1 … VQI 8 Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling 2 4 6 8 VOQ Buffer hi lo 1 3 5 7 10G Port 1 10G Port 3 10G Port 5 10G Port 2 Diagram represents one SoC on each I/O module INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES 10G Port 4 10G Port 6 10G Port 7 10G Port 8 hi lo hi lo hi lo hi lo hi lo hi lo hi lo 10G Port 1 10G Port 2 10G Port 3 10G Port 4 10G Port 5 10G Port 6 10G Port 7 10G Port 8 DWRR PQ DWRR PQ 2 or 4 ingress queues per port 8 priority levels
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 51 Egress SOC Ingress SOC Ingress VOQ F3 40G – Ingress Buffered (Nexus 7000) FABRIC Egress VOQ VQI 1 DWRR PQ VQI 2 DWRR PQ Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling 1 2 VOQ Buffer lo hi lo hi 40G Port 1 40G Port 2 40G Port 1 Diagram represents one SoC on each I/O module INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES 40G Port 2 2 or 4 ingress queues per port 4 priority levels
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 52 Egress SOC Ingress SOC Ingress VOQ F3 40G – Ingress Buffered (Nexus 7700) FABRIC Egress VOQ VQI 1 VQI 2 Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling 40G Port 1 Diagram represents one SoC on each I/O module INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES 40G Port 2 DWRR PQ DWRR PQ 1 2 VOQ Buffer lo hi lo hi 40G Port 1 40G Port 2 2 or 4 ingress queues per port 8 priority levels
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 53 Egress SOC Ingress SOC Ingress VOQ F3 100G – Ingress Buffered (Nexus 7700) FABRIC Egress VOQ VQI 1 Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling 1 VOQ Buffer hi lo 100G Port 1 Diagram represents one SoC on each I/O module INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES DWRR PQ 100G Port 1 2 or 4 ingress queues per port 8 priority levels
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 54 FAQ: What Is a VQI?  VQI = Virtual Queuing Index  “A Destination Across the Fabric”  For M2 / F2E / F3 10G modules, VQI == 10G interface  For M2 40/100G ports, uses multiple 10G VQIs  For F3 40/100G ports, uses single 40/100G VQI
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 55 40G Port Ingress Modules 10G 10G 40G 40G 100G Spines Spines Spines Spines Fabrics M2 Module 40G and 100G Flow Limits  Each Virtual Queuing Index (VQI) sustains 10G traffic flow  All packets in given 5-tuple flow hash to single VQI  Single-flow limit is 10G  Packets split into 66-bit “code words”  Four code words transmitted in parallel, one on each physical Tx fibre  No per-flow limit imposed – splitting occurs at physical layer Egress Interfaces Destination VQIs 1 VQI 1 VQI 4 VQIs 4 VQIs 10 VQIs Internal to Nexus 7000 System n … 4 3 2 1 64 bits 1 packet On the Wire (40G) Tx 1 Tx 2 Tx 3 Tx 4 66 bits 1 5 2 3 4 6 … 64/66B Encoding
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 56 Ingress Modules 10G 10G 40G 40G 100G Spines Spines Spines Spines Fabrics F3 Module 40G and 100G Flow Limits  Virtual Queuing Index (VQI) sustains 10G, 40G, or 100G traffic flow based on destination interface type  No single-flow limit – full 40G/100G flow support Egress Interfaces Destination VQIs 1 VQI 1 VQI 1 VQI 1 VQI 1 VQI Internal to Nexus 7000 / 7700 System
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 57 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 58 Hardware Layer 2 Forwarding Process Layer 2 forwarding – traffic steering based on destination MAC address  MAC table lookup drives Layer 2 forwarding  Source MAC and destination MAC lookups performed for each frame, based on {VLAN,MAC} pairs  Source MAC lookup drives new learns and refreshes aging timers  Destination MAC lookup dictates outgoing switchport
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 59 Module 1 Fabric Module 1 Fabric ASIC Fabric Module 2 Fabric ASIC Fabric Module 3 Fabric ASIC Supervisor Engine Central Arbiter Fabric 2 ASIC 10G/40G/100G MAC / LinkSec VOQs Replication Engine Replication Engine VOQs e1/1 Layer 2 Engine Layer 3 Engine Forwarding Engine Module 2 Fabric 2 ASIC 10G/40G/100G MAC / LinkSec VOQs Replication Engine Replication Engine VOQs e2/2 Layer 2 Engine Layer 3 Engine Forwarding Engine M2 L2 Packet Flow  Receive packet from wire  LinkSec decryption  Ingress port QoS  Submit packet headers for lookup  ACL/QoS/ NetFlow lookups  VOQ arbitration and queuing  Round-robin transmit to fabric  Receive from fabric  Return buffer credit  Return credit to pool  Transmit packet on wire  Return result – destination + hash result  Credit grant for fabric access  Egress port QoS  LinkSec encryption  Static or hash- based RE uplink selection  Hash-based uplink and VQI selection  Round-robin transmit to VQI  Static downlink selection  L2 SMAC/ DMAC lookups  Port-channel hash result HDR = Packet Headers DATA = Packet Data = Internal Signalling CTRL
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 60 SoC VOQ SoC DE F2E / F3 L2 Packet Flow Module 2 Fabric ASIC e2/2 Module 1 Fabric ASIC e1/1 Fabric Module 1 Fabric ASIC Fabric Module 2 Fabric ASIC Fabric Module 3 Fabric ASIC Supervisor Engine Central Arbiter  VOQ arbitration  Credit grant for fabric access  Receive from fabric  Return credit to pool  Transmit packet on wire Fabric Module 4 Fabric ASIC Fabric Module 5 Fabric ASIC  Transmit to fabric VOQ  Receive packet from wire  Ingress port QoS (VOQ)  Ingress L2 SMAC/ DMAC lookups, ACL/QoS lookups, NetFlow sampling  Return result – destination  Submit packet headers for lookup  Egress port QoS (Scheduling)  Return buffer credit HDR = Packet Headers DATA = Packet Data = Internal Signalling CTRL
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 61 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 62 Layer 3 Forwarding  Nexus 7000 decouples control plane and data plane  Forwarding tables built on control plane using routing protocols or static configuration – OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing  Tables downloaded to forwarding engine hardware for data plane forwarding – FIB TCAM contains IP prefixes – Adjacency table contains next-hop information
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 63 Hardware Layer 3 Forwarding Process  FIB TCAM lookup based on longest-match destination prefix comparison  FIB “hit” returns adjacency, adjacency contains rewrite information (next-hop)  Pipelined forwarding engine architecture also performs ACL, QoS, and NetFlow lookups, affecting final forwarding result
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 64 10.1.1.2 10.1.1.3 10.10.0.10 10.10.0.100 10.10.0.33 10.1.1.4 10.1.2.xx 10.1.3.xx 10.1.1.xx 10.100.1.xx 10.10.0.xx 10.100.1.xx 10.10.100.xx IP FIB TCAM Lookup FIB TCAM Generate Lookup Key 10.1.1.10 Generate TCAM lookup key (destination IP address) Forwarding Engine FIB DRAM Load-Sharing Hash Adjacency Table Next-hop 4 (IF, MAC) Next-hop 6 (IF, MAC) Next-hop 7 (IF, MAC) Next-hop 5 (IF, MAC) Next-hop 3 (IF, MAC) Next-hop 1 (IF, MAC) Next-hop 2 (IF, MAC) 10.1.1.xx Ingress unicast IP packet header Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Hit in FIB returns result in FIB DRAM Adjacency index identifies ADJ block to use Modulo function selects exact next hop entry to use Offset Compare lookup key Return lookup result # next- hops Flow Data Result HIT! Adj Index mod
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 65 Module 1 Fabric Module 1 Fabric ASIC Fabric Module 2 Fabric ASIC Fabric Module 3 Fabric ASIC Supervisor Engine Central Arbiter Fabric 2 ASIC 10G/40G/100G MAC / LinkSec VOQs Replication Engine Replication Engine VOQs e1/1 Layer 2 Engine Layer 3 Engine Forwarding Engine Module 2 Fabric 2 ASIC 10G/40G/100G MAC / LinkSec VOQs Replication Engine Replication Engine VOQs e2/2 Layer 2 Engine Layer 3 Engine Forwarding Engine M2 L3 Packet Flow  Receive packet from wire  LinkSec decryption  Ingress port QoS  Submit packet headers for lookup  L3 FIB/ADJ lookup  Ingress and egress ACL/QoS/NetFlow lookups  VOQ arbitration and queuing  Round-robin transmit to fabric  Receive from fabric  Return buffer credit  Return credit to pool  Transmit packet on wire  Return result – destination + hash result  Credit grant for fabric access  Egress port QoS  LinkSec encryption  Static or Hash-based uplink selection  Hash-based uplink (and VQI) selection  Round-robin transmit to VOQ  Static RE downlink selection  L2 ingress and egress SMAC/ DMAC lookups  Port-channel hash result HDR = Packet Headers DATA = Packet Data = Internal Signalling CTRL
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 66 SoC VOQ SoC DE Module 2 Fabric ASIC e2/2 Module 1 Fabric ASIC e1/1 Fabric Module 1 Fabric ASIC Fabric Module 2 Fabric ASIC Fabric Module 3 Fabric ASIC Supervisor Engine Central Arbiter Fabric Module 4 Fabric ASIC Fabric Module 5 Fabric ASIC VOQ F2E / F3 L3 Packet Flow HDR = Packet Headers DATA = Packet Data = Internal Signalling CTRL  VOQ arbitration  Credit grant for fabric access  Return credit to pool  Transmit packet on wire  Transmit to fabric  Receive packet from wire  Ingress port QoS (VOQ)  Return result – destination  Submit packet headers for lookup  L2 ingress and egress SMAC/ DMAC lookups  L3 FIB/ADJ lookup  Ingress and egress ACL/QoS lookups, NetFlow sampling  Receive from fabric  Egress port QoS (Scheduling)  Return buffer credit
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 67 Layer 3 Forwarding – Module Interoperability Models Two interoperability models for L3 forwarding:  “Proxy Forwarding”  “Ingress Forwarding” with Lowest Common Denominator
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 68  From F1/F2E perspective, Router MAC reachable through giant port-channel  All packets destined to Router MAC forwarded through fabric toward one “member port” in that channel Proxy Forwarding Model – Conceptual All F1/F2E modules All M1/M2 modules Up to 128 “links” 10.1.10.100 vlan 10 10.1.20.100 vlan 20 interface vlan 10 ip address 10.1.10.1/24 ! interface vlan 20 ip address 10.1.20.1/24
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 69 Proxy Forwarding Model – Actual 10.1.10.100 vlan 10 e1/1 Fabric F1/F2E SoC FE e2/1 Fabric F1/F2E SoC FE 10.1.20.100 vlan 20 Replication Engine e3/1 e3/2 M1/M2 Replication Engine Replication Engine Replication Engine VOQs VOQs FE FE Fabric e3/3 e3/4 e3/5 e3/6 e3/7 e3/8 Replication Engine e4/1 e4/2 M1/M2 Replication Engine Replication Engine Replication Engine VOQs VOQs FE FE Fabric e4/3 e4/4 e4/5 e4/6 e4/7 e4/8 Fabric Fabric Modules Fabric … VLAN DMAC Dest Port 10 router_mac → internal_channel (e3/1-8,e4/1-8) EtherChannel Hash Function hash_input (from packet) → select_member_port Ingress MAC: VLAN DMAC Dest Port 10 router_mac → L3_lookup Routing: DIP Next Hop 10.1.20.100 → server_2_mac (v20) Egress MAC: VLAN DMAC Dest Port 20 server_2_mac → e2/1 1 2 3 4 6 5 7 8 9 10 Programming of all M1/M2 forwarding engines Programming of all F1/F2E forwarding engines interface vlan 10 ip address 10.1.10.1/24 ! interface vlan 20 ip address 10.1.20.1/24 Can be up to 128 M1/M2 VQIs Mod 1 Mod 2 Mod 4 Mod 3
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 71 Ingress Forwarding with Lowest Common Denominator Model  F3 module interoperability always Ingress Forwarding – NO proxy forwarding with F3 – Essentially equivalent to current M1 + M2 interoperability model – The ingress module makes all the forwarding decisions  Supported feature set based on Lowest Common Denominator – Feature available if all modules support the feature VDC Type Layer 2 Layer 3 vPC Fabric Path VXLAN FEX MPLS OTV LISP FCoE Table Sizes F3 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ F3 size M2 + F3 ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✗ ✗ F3 size F2/F2E + F3 ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✓ F2E size Not all features supported by software today
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 73 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 74 What Is Classification?  Matching packets – Layer 2, Layer 3, and/or Layer 4 information  Used to decide whether to apply a particular policy to a packet – Enforce security, QoS, or other policies  Some examples: – Match TCP/UDP source/destination port numbers to enforce security policy – Match destination IP addresses to apply policy-based routing (PBR) – Match 5-tuple to apply marking policy – Match protocol-type to apply Control Plane Policing (CoPP) – etc.
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 75 CL TCAM Lookup – ACL ip access-list example permit ip any host 10.1.2.100 deny ip any host 10.1.68.44 deny ip any host 10.33.2.25 permit tcp any any eq 22 deny tcp any any eq 23 deny udp any any eq 514 permit tcp any any eq 80 permit udp any any eq 161 xxxxxxx | 10.1.2.100 | xx | xxx | xxx xxxxxxx | 10.1.68.44 | xx | xxx | xxx xxxxxxx | 10.33.2.25 | xx | xxx | xxx xxxxxxx | xxxxxxx | tcp | xxx | 22 xxxxxxx | xxxxxxx | tcp | xxx | 23 xxxxxxx | xxxxxxx | tcp | xxx | 80 xxxxxxx | xxxxxxx | udp | xxx | 161 xxxxxxx | xxxxxxx | udp | xxx | 514 Packet header: SIP: 10.1.1.1 DIP: 10.2.2.2 Protocol: TCP SPORT: 33992 DPORT: 80 CL TCAM Generate Lookup Key Generate TCAM lookup key CL SRAM 10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80 xxxxxxx | 10.2.2.2 | xx | xxx | xxx xxxxxxx | xxxxxxx | tcp | xxx | 80 SIP | DIP | Pr | SP | DP Compare lookup key to CL TCAM entries Comparisons (X = “Mask”) Hit in CL TCAM returns result in CL SRAM Security ACL Forwarding Engine Result Return lookup result Result affects final packet handling Permit Permit Permit Permit Deny Deny Deny Deny HIT! Results SIP | DIP | Pr | SP | DP
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 76 Packet header: SIP: 10.1.1.1 DIP: 10.2.2.2 Protocol: TCP SPORT: 33992 DPORT: 80 Result affects final packet handling Generate Lookup Key Forwarding Engine xxxxxxx | 10.3.3.xx | xx | xxx | xxx xxxxxxx | 10.4.12.xx | xx | xxx | xxx 10.1.1.xx | xxxxxxx | udp | xxx | xxx 10.1.1.xx | xxxxxxx | tcp | xxx | xxx xxxxxxx | 10.5.5.xx| tcp | xxx | 23 CL TCAM Lookup – QoS ip access-list police permit ip any 10.3.3.0/24 permit ip any 10.4.12.0/24 ip access-list remark-dscp-32 permit udp 10.1.1.0/24 any ip access-list remark-dscp-40 permit tcp 10.1.1.0/24 any ip access-list remark-prec-3 permit tcp any 10.5.5.0/24 eq 23 CL TCAM 10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80 xxxxxxx | 10.2.2.xx | xx | xxx | xxx 10.1.1.xx | xxxxxxx | tcp | xxx| xxx HIT! CL SRAM QoS Classification ACLs Generate TCAM lookup key SIP | DIP | Pr | SP | DP Compare lookup key Hit in CL TCAM returns result in CL SRAM Result Return lookup result Policer ID 1 Policer ID 1 Remark DSCP 32 Remark DSCP 40 Remark IP Prec 3 SIP | DIP | Pr | SP | DP Comparisons (X = “Mask”) Results
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 77 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 78 NetFlow  NetFlow collects flow data for packets traversing the switch  Each module maintains independent NetFlow table M2 F2E / F3 Per-interface NetFlow Yes Yes NetFlow direction Ingress/Egress Ingress only Full NetFlow Yes No Sampled NetFlow Yes Yes FSA Assist for Sampled NetFlow No F3 only (future) Bridged NetFlow Yes Yes Hardware Cache Yes No Software Cache No Yes Hardware Cache Size 512K entries per forwarding engine N/A NDE (v5/v9) Yes Yes
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 79 Full vs. Sampled NetFlow  NetFlow collects full or sampled flow data  Full NetFlow: Accounts for every packet of every flow on interface – Available on M-Series modules only – Flow data collection up to capacity of hardware NetFlow table  Sampled NetFlow: Accounts for M in N packets on interface – Available on both M2 (ingress/egress) and F2E/F3 (ingress only) – M2: Flow data collection up to capacity of hardware NetFlow table – F2E/F3: Flow data collection for up to ~1000pps per module – F3 (future): Increased per-module sampling rate leveraging on-board Fabric Services Accelerator (FSA) complex
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 81 NetFlow on M2 Modules Fabric ASIC VOQs Mgmt Enet Supervisor Engine Forwarding Engine LC CPU NetFlow Table M2 Module Forwarding Engine LC CPU NetFlow Table M2 Module Forwarding Engine LC CPU NetFlow Table M2 Module Hardware Flow Creation Hardware Flow Creation Hardware Flow Creation Aged Flow Info Aged Flow Info Aged Flow Info Generate NetFlow v5 or v9 export packets Main CPU To NetFlow Collector To NetFlow Collector Switched EOBC via Supervisor Inband via mgmt0
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 82 Sampled NetFlow on F2E/F3 Modules F3 Module FSA CPU SoC Decision Engine DRAM NetFlow Cache F3 Module Fabric ASIC VOQs Mgmt Enet Supervisor Engine FSA CPU SoC Decision Engine Main CPU To NetFlow Collector To NetFlow Collector Switched EOBC via Supervisor Inband via mgmt0 DRAM NetFlow Cache Populate cache based on received samples Age flows and generate NetFlow v5 or v9 export packets F2E Module LC CPU SoC Decision Engine DRAM NetFlow Cache Data Flow Data Flow Data Flow via Module Inband via Module Inband via Module Inband Sampled Packets Sampled Packets Sampled Packets Aged Flows Aged Flows Aged Flows
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 83 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 84 Nexus 7000 / Nexus 7700 Architecture Summary I/O Modules Supervisor Engines Fabrics Chassis
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 85 Conclusion  You should now have a thorough understanding of the Nexus 7000 / Nexus 7700 switching architecture, I/O module design, packet flows, and key forwarding engine functions…  Any questions? 85
Q & A
© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 88 Complete Your Online Session Evaluation Give us your feedback and receive a Cisco Live 2014 Polo Shirt! Complete your Overall Event Survey and 5 Session Evaluations.  Directly from your mobile device on the Cisco Live Mobile App  By visiting the Cisco Live Mobile Site www.ciscoliveaustralia.com/mobile  Visit any Cisco Live Internet Station located throughout the venue Polo Shirts can be collected in the World of Solutions on Friday 21 March 12:00pm - 2:00pm Learn online with Cisco Live! Visit us online after the conference for full access to session videos and presentations. www.CiscoLiveAPAC.com
N7K Hardware Architecture and it's components

More Related Content

PDF
Brkarc 3470 - cisco nexus 7000-7700 switch architecture (2016 las vegas) - 2 ...
bykds850
 
PDF
BRKARC-3470 - Nexus 7000 and 7700 Architecture.pdf
byssuser1d1e521
 
PDF
Brkarc 3454
byNguyen Van Linh
 
PDF
Brkarc 3454 - in-depth and personal with the cisco nexus 2000 fabric extender...
bykds850
 
PDF
Brkarc 3601
byNguyen Van Linh
 
PPTX
Training Nexus 7 data & operation 2024.pptx
byrudyhermawan37
 
PPTX
Cisco nexus series
byAnwesh Dixit
 
PPT
Cisco nexus 7009 overview
byHamza Al-Qudah
 
Brkarc 3470 - cisco nexus 7000-7700 switch architecture (2016 las vegas) - 2 ...
bykds850
 
BRKARC-3470 - Nexus 7000 and 7700 Architecture.pdf
byssuser1d1e521
 
Brkarc 3454
byNguyen Van Linh
 
Brkarc 3454 - in-depth and personal with the cisco nexus 2000 fabric extender...
bykds850
 
Brkarc 3601
byNguyen Van Linh
 
Training Nexus 7 data & operation 2024.pptx
byrudyhermawan37
 
Cisco nexus series
byAnwesh Dixit
 
Cisco nexus 7009 overview
byHamza Al-Qudah
 

Similar to N7K Hardware Architecture and it's components

PDF
Data Centre Portfolio Update
byCisco Canada
 
DOCX
Cisco nexus 7000 and nexus 7700
byIT Tech
 
PDF
Cisco Connect Vancouver 2017 - Gain insight and programmability with Cisco DC...
byCisco Canada
 
PPT
Dcna technology update
byRamana Rongala
 
PDF
Gain Insight and Programmability with Cisco DC Networking
byCisco Canada
 
PDF
Cisco DC Networking: Gain Insight and Programmability with
byCisco Canada
 
PDF
Gain Insight and Programmability with Cisco DC Networking
byCisco Canada
 
PPTX
Nexus 7000 Series Innovations: M3 Module, DCI, Scale
byTony Antony
 
PDF
cisco-n2k-c2348tq-e-datasheet.pdf
byHi-Network.com
 
PDF
Next Generation Nexus 9000 Architecture
byCisco Canada
 
PPTX
Nexus 9000 Series - 400G and beyond.pptx
byITMall
 
PDF
Cisco Catalyst 6500 Technical Deep Dive.pdf
byjuergenJaeckel
 
PPT
transforming_datacenter_core_with_dce_cisco_nexus.ppt
byBalanjaneyaPrasad
 
PDF
Cisco connect winnipeg 2018 gain insight and programmability with cisco dc ...
byCisco Canada
 
PPTX
01-05-N9K Hardware.pptx
byFeizhang41
 
PDF
Data sheet c78-707001
byMauricio Jimenez
 
PDF
Gain Insight and Programmability with Cisco DC Networking
byCisco Canada
 
PDF
Data Centre Design for Canadian Small & Medium Sized Businesses
byCisco Canada
 
PDF
Cisco usNIC: how it works, how it is used in Open MPI
byJeff Squyres
 
PDF
cisco-n2k-c2348upq-datasheet.pdf
byHi-Network.com
 
Data Centre Portfolio Update
byCisco Canada
 
Cisco nexus 7000 and nexus 7700
byIT Tech
 
Cisco Connect Vancouver 2017 - Gain insight and programmability with Cisco DC...
byCisco Canada
 
Dcna technology update
byRamana Rongala
 
Gain Insight and Programmability with Cisco DC Networking
byCisco Canada
 
Cisco DC Networking: Gain Insight and Programmability with
byCisco Canada
 
Gain Insight and Programmability with Cisco DC Networking
byCisco Canada
 
Nexus 7000 Series Innovations: M3 Module, DCI, Scale
byTony Antony
 
cisco-n2k-c2348tq-e-datasheet.pdf
byHi-Network.com
 
Next Generation Nexus 9000 Architecture
byCisco Canada
 
Nexus 9000 Series - 400G and beyond.pptx
byITMall
 
Cisco Catalyst 6500 Technical Deep Dive.pdf
byjuergenJaeckel
 
transforming_datacenter_core_with_dce_cisco_nexus.ppt
byBalanjaneyaPrasad
 
Cisco connect winnipeg 2018 gain insight and programmability with cisco dc ...
byCisco Canada
 
01-05-N9K Hardware.pptx
byFeizhang41
 
Data sheet c78-707001
byMauricio Jimenez
 
Gain Insight and Programmability with Cisco DC Networking
byCisco Canada
 
Data Centre Design for Canadian Small & Medium Sized Businesses
byCisco Canada
 
Cisco usNIC: how it works, how it is used in Open MPI
byJeff Squyres
 
cisco-n2k-c2348upq-datasheet.pdf
byHi-Network.com
 

Recently uploaded

PPTX
Software Analysis &Design ethiopia chap-2.pptx
byAbbas484771
 
PDF
CompTIA Cybersecurity Analyst (CySA+) CS0-003: Unit 5
byVICTOR MAESTRE RAMIREZ
 
PDF
Our Digital Tribe_ Cultivating Connection and Growth in Our Slack Community 🌿...
bysanjeetmishra30
 
PDF
Real-Time Data Insight Using Microsoft Forms for Business
byElevate
 
PDF
TrustArc Webinar - Looking Ahead: The 2026 Privacy Landscape
byTrustArc
 
PDF
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
PDF
What Is a Private LLM and Why Enterprises Need It
byAivada
 
PDF
Day 4 - Access, Deployments, and Monitoring - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
PDF
The State of the Gen AI economy - 2025 - The Meliora Company
byClive Dickens
 
PDF
Cross-Cultural Agile Development -Challenges and Strategies for Overcoming Them-
byTakashi Makino
 
PDF
API-First Architecture in Financial Systems
bycyy111112
 
PDF
Six Shifts For 2026 (And The Next Six Years)
byDavid Armano
 
PPTX
UiPath Autonomous Agents | Building and Orchestrating Agents End-to-End
byUiPathCommunity
 
PPTX
Building Cyber Resilience for 2026: Best Practices for a Secure, AI-Driven Bu...
byYasir Naveed Riaz
 
PPTX
Emancipatory Information Retrieval: Radically Reorienting Information Retriev...
byBhaskar Mitra
 
PDF
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
PPTX
DYNAMICALLY.pptx good for the teachers or students to do seminars and for tea...
bympoorvika031
 
PDF
Day 5 - Red Team + Blue Team in the Cloud - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
PDF
HCSP-Presales-Campus Network Planning and Design V2.0
byVeronica916344
 
PPTX
Building Agents in Microsoft Agent Framework.pptx
byUdaiappa Ramachandran
 
Software Analysis &Design ethiopia chap-2.pptx
byAbbas484771
 
CompTIA Cybersecurity Analyst (CySA+) CS0-003: Unit 5
byVICTOR MAESTRE RAMIREZ
 
Our Digital Tribe_ Cultivating Connection and Growth in Our Slack Community 🌿...
bysanjeetmishra30
 
Real-Time Data Insight Using Microsoft Forms for Business
byElevate
 
TrustArc Webinar - Looking Ahead: The 2026 Privacy Landscape
byTrustArc
 
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
What Is a Private LLM and Why Enterprises Need It
byAivada
 
Day 4 - Access, Deployments, and Monitoring - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
The State of the Gen AI economy - 2025 - The Meliora Company
byClive Dickens
 
Cross-Cultural Agile Development -Challenges and Strategies for Overcoming Them-
byTakashi Makino
 
API-First Architecture in Financial Systems
bycyy111112
 
Six Shifts For 2026 (And The Next Six Years)
byDavid Armano
 
UiPath Autonomous Agents | Building and Orchestrating Agents End-to-End
byUiPathCommunity
 
Building Cyber Resilience for 2026: Best Practices for a Secure, AI-Driven Bu...
byYasir Naveed Riaz
 
Emancipatory Information Retrieval: Radically Reorienting Information Retriev...
byBhaskar Mitra
 
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
DYNAMICALLY.pptx good for the teachers or students to do seminars and for tea...
bympoorvika031
 
Day 5 - Red Team + Blue Team in the Cloud - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
HCSP-Presales-Campus Network Planning and Design V2.0
byVeronica916344
 
Building Agents in Microsoft Agent Framework.pptx
byUdaiappa Ramachandran
 

N7K Hardware Architecture and it's components

  • 2.
    Cisco Nexus 7000Switch Architecture BRKARC-3470 Ron Fuller, CCIE#5851 (R&S/Storage) Technical Marketing Engineer
  • 3.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 3 Session Abstract This session presents an in-depth study of the architecture of the latest generation of Nexus 7000 and Nexus 7700 data centre switches. Topics include supervisors, fabrics, I/O modules, forwarding engines, and physical design elements, as well as a discussion of key hardware-enabled features that combine to implement high-performance data centre network services.
  • 4.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 4 Session Goal  To provide a thorough understanding of the Nexus 7000 / Nexus 7700 switching architecture, supervisor, fabric, and I/O module design, packet flows, and key forwarding engine functions  This session will examine the Nexus 7700 system, as well as the latest additions to the Nexus 7000  This session will not examine NX-OS software architecture or other Nexus platform architectures 4
  • 5.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 5 What Is Nexus 7000? Data-centre class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection Nexus 7000 designed for general-purpose Data Centre deployments, focused on 10G density plus 40G/100G I/O Modules Supervisor Engines Fabrics Chassis
  • 6.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 6 What Is Nexus 7700? Data-centre class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection Nexus 7700 designed for SP and MSDC Data Centre deployments, focused on high- density 40G/100G I/O Modules Supervisor Engine Fabrics Chassis
  • 7.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 7 Nexus 7000 General purpose DC switching w/10/40/100G Nexus 7700 Targeted at Densest 40G/100G deployments Common Foundation • Same release vehicles, versioning, feature-sets • Common configuration model • Common operational model • Common fabric ASICs (Fab2) and architecture • Same central arbitration model • Same VOQ/QoS model • Identical forwarding ASICs (F2E, F3) • Consistent hardware feature sets • Parallel evolution of hardware capability/scale Nexus 7000 / Nexus 7700 – Common Foundation
  • 8.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 8 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
  • 9.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 9 Nexus 7000 Chassis Family Front Rear 21RU N7K-C7010 25RU Front Rear N7K-C7018 Front Rear N7K-C7009 14RU NX-OS 4.1(2) and later NX-OS 5.2(1) and later Nexus 7010 Nexus 7018 Nexus 7009 Front N7K-C7004 7RU NX-OS 6.1(2) and later Rear Nexus 7004 Front Back Side Side Side Side Side Back
  • 10.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 10 Nexus 7700 Chassis Family Front Rear 26RU N77-C7718 Nexus 7718 Front Rear 14RU N77-C7710 Nexus 7710 Front Rear 9RU N77-C7706 Nexus 7706 NX-OS 6.2(6) and later NX-OS 6.2(2) and later NX-OS 6.2(2) and later Front Back Front Back Front Back
  • 11.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 11 Key Chassis Components Nexus 7000  Common components: – Supervisor engines – I/O modules – Power supplies (except 7004)  Chassis-specific components: – Fabric modules – Fan trays Nexus 7700  Common components: – Supervisor engines – I/O modules – Power supplies  Chassis-specific components: – Fabric modules – Fan trays Common hardware components between Nexus 7000 and Nexus 7700: NONE No interchangeable hardware components between Nexus 7000 and Nexus 7700
  • 12.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 12 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
  • 13.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 13  Next generation supervisors providing control plane and management functions  Connects to fabric via 1G inband interface  Interfaces with I/O modules via 1G switched EOBC  Second-generation dedicated central arbiter ASIC – Controls access to fabric bandwidth via dedicated arbitration path to I/O modules Supervisor Engine 2 / 2E Console Port Management Ethernet N7K-SUP2/N7K-SUP2E USB Host Ports ID and Status LEDs Supervisor Engine 2 (Nexus 7000) Supervisor Engine 2E (Nexus 7000 / Nexus 7700) Base performance High performance One quad-core 2.1GHz CPU with 12GB DRAM Two quad-core 2.1GHz CPU with 32GB DRAM USB Log Flash USB Expansion Flash N77-SUP2E ID and Status LEDs Console Port Management Ethernet USB Expansion Flash
  • 14.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 14 Nexus 7000 / 7700 I/O Module Families M1 1G and 10G M2 10G / 40G / 100G F1 10G F2 10G F2E 10G F3 40G F2E 10G F3 10G / 40G / 100G F3 closes the F/M feature gap!
  • 15.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 15  10G / 40G / 100G M2 I/O modules  Share common hardware architecture  Two integrated forwarding engines (120Mpps) – Support for “XL” forwarding tables (licensed)  Distributed L3 multicast replication  802.1AE LinkSec on all ports N7K-M224XP-23L Nexus 7000 M2 I/O Modules N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L Supported in NX-OS release 6.1(1) and later N7K-M206FQ-23L N7K-M202CF-22L Module Port Density Optics Bandwidth M2 10G 24 x 10G (plus Nexus 2000 FEX support) SFP+ 240G M2 40G 6 x 40G (or up to 24 x 10G via breakout) QSFP+ 240G M2 100G 2 x 100G CFP 200G
  • 16.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 16 Nexus 7000 M2 I/O Module Architecture N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L LinkSec + 12 X 10G MAC -or- 3 X 40G MAC -or- 1 X 100G MAC Forwarding Engine VOQs Fabric 2 ASIC To Fabric Modules Replication Engine Replication Engine Front Panel Ports LC CPU EOBC VOQs LinkSec + 12 X 10G MAC -or- 3 X 40G MAC -or- 1 X 100G MAC Forwarding Engine VOQs Replication Engine Replication Engine VOQs To Central Arbiters Arbitration Aggregator …
  • 17.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 18 Nexus 7000 / 7700 F2E I/O Modules N7K-F248XP-25E / N7K-F248XT-25E / N77-F248XP-23E 7000: Supported in NX-OS release 6.1(2) and later 7700: Supported in NX-OS release 6.2(2) and later N7K-F248XP-25E N7K-F248XT-25E  48-port 1G/10G with SFP/SFP+ transceivers  480G full-duplex fabric connectivity  System-on-chip (SoC) forwarding engine design – 12 independent SoC ASICs  Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS)  Interoperability with M1/M2, in Layer 2 mode on Nexus 7000 – Proxy routing for inter-VLAN/L3 traffic  LinkSec support* – Last 8 ports (SFP+) – All 48 ports (Copper)  Supports Nexus 2000 (FEX) connections * Roadmap item N77-F248XP-23E
  • 18.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 19 Nexus 7000 F2E Module Architecture N7K-F248XP-25E / N7K-F248XT-25E 4 X 10G SoC Front Panel Ports To Fabric Modules Fabric 2 2 4 LC CPU EOBC To Central Arbiters Arbitration Aggregator … 4 X 10G SoC 6 8 4 X 10G SoC 10 12 4 X 10G SoC 14 16 4 X 10G SoC 18 20 4 X 10G SoC 22 24 4 X 10G SoC 26 28 4 X 10G SoC 30 32 4 X 10G SoC 34 36 4 X 10G SoC 38 40 4 X 10G SoC 42 44 4 X 10G SoC 46 48 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 LinkSec-capable (F2E fibre) LinkSec-capable (F2E copper)
  • 19.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 20 Nexus 7700 F2E Module Architecture N77-F248XP-23E 4 X 10G SoC Front Panel Ports To Fabric Modules Fabric 2 2 4 LC CPU EOBC To Central Arbiters Arbitration Aggregator … 4 X 10G SoC 6 8 4 X 10G SoC 10 12 4 X 10G SoC 14 16 4 X 10G SoC 18 20 4 X 10G SoC 22 24 4 X 10G SoC 26 28 4 X 10G SoC 30 32 4 X 10G SoC 34 36 4 X 10G SoC 38 40 4 X 10G SoC 42 44 4 X 10G SoC 46 48 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 LinkSec-capable Fabric 2 To Fabric Modules
  • 20.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 22 Nexus 7000 F3 40G Module  12-port 40G QSFP+ module  480G full-duplex fabric connectivity  SoC forwarding engine design – 6 independent SoC ASICs  Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features  Fabric Services Accelerator (FSA) CPU  Breakout cable support  Requires Supervisor Engine 2 / 2E N7K-F312FQ-25
  • 21.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 23 Nexus 7000 12-Port 40G Module Architecture 1 Front Panel Ports (QSFP+) To Fabric Modules FSA CPU EOBC To Central Arbiters Arbitration Aggregator 2 X 40G SoC 1 2 X 40G SoC 2 2 X 40G SoC 3 2 X 40G SoC 4 2 X 40G SoC 5 2 X 40G SoC 6 Fabric ASIC LC Inband 2 3 4 5 6 7 8 9 10 11 12 … x 6 to FSA CPU to ARB x 6 1G switch x 6 …
  • 22.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 24 FSA CPU Fabric Services Accelerator (FSA)  High-performance module CPU with on-board acceleration engines – 6Gbps inband connectivity from SOCs to FSA – Multi-Mpps packet processing – 2GB dedicated DRAM  Performance/scale boost for distributed fabric services, including BFD and sampled NetFlow (roadmap)  Other potential applications include distributed ARP/ping processing, data plane packet analysis (wireshark), network probing, etc. 6 x 1Gbps Module Inband I/O 2GB DRAM Dual-Core LC CPU Acceleration Engines 2GB DRAM EOBC
  • 23.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 25 Nexus 7700 F3 48-Port 1G/10G Module  48-port 1G/10G with SFP/SFP+ transceivers  480G full-duplex fabric connectivity  SoC-based forwarding engine design – 6 independent SoC ASICs  Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features  Fabric Services Accelerator (FSA) CPU  LinkSec support (last 8 ports)*  Supports Nexus 2000 (FEX) connections N77-F348XP-23 * Roadmap item
  • 24.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 26 8 X 10G SoC 1 Nexus 7700 F3 48-Port 1G/10G Module Architecture To Fabric Modules To Central Arbiters Arbitration Aggregator 8 X 10G SoC 2 8 X 10G SoC 3 8 X 10G SoC 4 8 X 10G SoC 5 8 X 10G SoC 6 Fabric ASIC Fabric ASIC … x 6 1 Front Panel Ports (SFP/SFP+) 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 LinkSec-capable to FSA CPU to ARB FSA CPU EOBC LC Inband x 6 1G switch x 6 …
  • 25.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 27 Nexus 7700 F3 40G and 100G Modules  24-port 40G QSFP+ module / 12-port 100G CPAK module  960G/1.2T full-duplex fabric connectivity  SoC forwarding engine design – 12 independent SoC ASICs  Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features  Fabric Services Accelerator (FSA) CPU  40G breakout cable support* N77-F324FQ-25 N77-F312CK-26 * Roadmap item
  • 26.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 28 2 X 40G SoC 1 Nexus 7700 F3 24-Port 40G Module Architecture 1 Front Panel Ports (QSFP+) To Fabric Modules FSA CPU EOBC To Central Arbiters Arbitration Aggregator 2 X 40G SoC 2 2 X 40G SoC 3 2 X 40G SoC 4 2 X 40G SoC 5 2 X 40G SoC 6 2 X 40G SoC 7 2 X 40G SoC 8 2 X 40G SoC 9 2 X 40G SoC 10 2 X 40G SoC 11 2 X 40G SoC 12 Fabric ASIC Fabric ASIC LC Inband 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1G switch … … x 12 to FSA CPU to ARB x 12 x 6
  • 27.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 29 Nexus 7700 F3 12-Port 100G Module Architecture Front Panel Ports (CPAK) To Fabric Modules To Central Arbiters Arbitration Aggregator 1 X 100G SoC 2 2 1 X 100G SoC 3 3 1 X 100G SoC 4 4 1 X 100G SoC 5 5 1 X 100G SoC 6 6 1 X 100G SoC 7 1 X 100G SoC 8 1 X 100G SoC 9 1 X 100G SoC 10 1 X 100G SoC 11 Fabric ASIC Fabric ASIC 7 8 9 10 11 1 X 100G SoC 12 12 1 X 100G SoC 1 1 FSA CPU EOBC LC Inband 1G switch … … x 12 to FSA CPU to ARB x 12 x 6
  • 28.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 30 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
  • 29.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 31 M-Series Forwarding Engine Hardware  Two hardware forwarding engines integrated on every M2 I/O module  120Mpps (60Mpps per forwarding engine) Layer 2 bridging with hardware MAC learning  120 Mpps (60Mpps per forwarding engine) Layer 3 IPv4  60Mpps (30Mpps per forwarding engine) Layer 3 IPv6 unicast  Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir)  MPLS/VPLS/EoMPLS  OTV  RACL/VACL/PACL  QoS remarking and policing policies  Policy-based routing (PBR)  Unicast RPF check and IP source guard  IGMP snooping  Ingress and egress NetFlow (full and sampled) Hardware Table M-Series Modules without Scale License M-Series Modules with Scale License MAC Address Table 128K 128K FIB TCAM 128K IPv4 / 64K IPv6 900K IPv4 / 350K IPv6 Classification TCAM (ACL/QoS) 64K 128K NetFlow Table 1M 1M
  • 30.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 32 From I/O Module Replication Engines To I/O Module Replication Engines M-Series Forwarding Engine Architecture L2 Engine Ingress Parser MAC Table L2 Lookup (pre-L3) L2 Lookup (post-L3) Final Results L3 Engine Classification (ACL/QoS) NetFlow Layer 3 FIB Policing FIB TCAM/ ADJ CL TCAM FE Daughter Card Ingress lookup pipeline Egress lookup pipeline  Egress NetFlow collection  Ingress MAC table lookups  Port-channel hash result  Ingress IGMP snooping lookups  FIB TCAM and adjacency table lookups for Layer 3 forwarding  ECMP hashing  Multicast RPF check  Ingress policing  Egress MAC lookups  Egress IGMP snooping lookups PKT HDR  Egress ACL/QoS classification  Ingress NetFlow collection  Ingress ACL/QoS classification  Egress policing
  • 31.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 33 F2E Forwarding Engine Hardware  Each SoC forwarding engine services 4 front-panel 10G ports (12 SoCs per module)  60Mpps per SoC Layer 2 bridging with hardware MAC learning  60Mpps per forwarding engine Layer 3 IPv4/ IPv6 unicast  Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir*)  RACL/VACL/PACL  QoS remarking and policing policies  Policy-based routing (PBR)  Unicast RPF check and IP source guard  IGMP snooping  FabricPath forwarding  FCoE (with Sup2 / Sup2E) – Roadmap on Nexus 7700  Ingress sampled NetFlow Hardware Table Per F2E SoC Per F2E Module MAC Address Table 16K 192K* FIB TCAM 32K IPv4/16K IPv6 32K IPv4/16K IPv6 Classification TCAM (ACL/QoS) 16K 192K* * Assumes specific configuration to scale SoC resources * Roadmap item
  • 32.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 34 F3 Forwarding Engine Hardware  Each SoC forwarding engine services: – 8 front-panel 10G ports – 2 front-panel 40G ports – 1 front-panel 100G port  148Mpps per SoC Layer 2 bridging with hardware MAC learning  148Mpps per forwarding engine Layer 3 IPv4/ IPv6 unicast  Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir*)  RACL/VACL/PACL  QoS remarking and policing policies  Policy-based routing (PBR)  Unicast RPF check and IP source guard  IGMP snooping  FabricPath forwarding  Overlay Transport Virtualisation (OTV)  MPLS/VPLS/EoMPLS, LISP, VXLAN, GRE, FCoE*  Ingress/egress* sampled NetFlow Hardware Table Per F3 SoC Per F3 Module MAC Address Table 64K 384K/768K** FIB TCAM 64K IPv4/32K IPv6 64K IPv4/32K IPv6 Classification TCAM (ACL/QoS) 16K 96K/192K** ** Assumes specific configuration to scale SoC resources * Roadmap items
  • 33.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 35 F3 Forwarding Engine Decision Engine Layer 3 Lookups QoS / ACL Ingress Parser MAC Table FIB/ADJ CL L2 Lookup (post-L3) Front-panel To/From Central Arbiter To Fabric From Fabric Ingress Buffer (VOQ) Virtual output queues L2 Lookup (pre-L3) Egress Parser F3 SoC Ingress and egress forwarding decisions (L2/L3 lookups, ACL/QoS, features etc.) 8 x 1/10G OR 2 x 40G OR 1 x 100G per ASIC Forwarding tables 1G / 10G / 40G / 100G 1G / 10G / 40G / 100G capable interface MAC Egress Buffer Egress fabric receive buffer HDR PKT HDR PKT PKT HDR
  • 34.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 36 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
  • 35.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 37 Crossbar Switch Fabric Modules  Provide interconnection of I/O modules  Each installed fabric increases available per-payload slot bandwidth  Nexus 7000 and Nexus 7700 fabrics based on Fabric 2 ASIC  Different I/O modules leverage different amount of available fabric bandwidth  Access to fabric bandwidth controlled using QoS-aware central arbitration with VOQ N7K-C7018-FAB-2 N7K-C7010-FAB-2 N7K-C7009-FAB-2 Fabric Module Supported Chassis Per-fabric module bandwidth Max fabric modules Total bandwidth per slot Nexus 7000 Fabric 2 7009 / 7010 / 7018 110Gbps per slot 5 550Gbps per slot Nexus 7700 Fabric 2 7706 / 7710 / 7718 220Gbps per slot 6 1.32Tbps per slot N77-C7718-FAB-2 N77-C7710-FAB-2 N77-C7706-FAB-2
  • 36.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 38 110G (2 x 55G) Ingress Module Egress Module Multistage Crossbar Nexus 7000 / Nexus 7700 implement 3-stage crossbar switch fabric  Stages 1 and 3 on I/O modules  Stage 2 on fabric modules 1st stage Egress Module 2nd stage Ingress Module 3rd stage Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric Modules Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC 1 Fabric ASIC 2 3 4 5 Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC 6 Fabric ASIC 1.32T 1st stage 3rd stage 550G 110G (2 x 55G) 1 Fabric ASIC 2 3 4 5 Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric Modules Nexus 7000 Nexus 7700
  • 37.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 39 110Gbps 220Gbps 330Gbps 440Gbps 550Gbps Local Fabric 2 (480G) Local Fabric 2 (240G) I/O Module Capacity – Nexus 7000 One fabric:  Any port can pass traffic to any other port in VDC Three fabrics:  240G M2 module has maximum bandwidth Five fabrics:  480G F2E/F3 module has maximum bandwidth Fabric 2 Modules 1 Fabric 2 ASIC 2 Fabric 2 ASIC 3 Fabric 2 ASIC 4 Fabric 2 ASIC 5 Fabric 2 ASIC per slot bandwidth
  • 38.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 40 What About Nexus 7004?  Nexus 7004 has no fabric modules  I/O modules have local fabric with 10 available fabric channels – I/O modules connect “back-to-back” via 8 fabric channels – Two fabric channels “borrowed” to connect supervisor engines Sup Slot 2 Sup Slot 1 M2/F2E/F3 Module 4 M2/F2E/F3 Module 3 Fabric ASIC Fabric 2 ASIC Fabric 2 ASIC Fabric ASIC 2 * 55G fabric channels 8 * 55G local fabric channels interconnect I/O modules (440G)
  • 39.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 41 220Gbps 440Gbps 660Gbps 880Gbps 1100Gbps 1320Gbps Local Fab2 #1 (480G) Local Fab2 #1 (960G) Local Fab2 #1 (1.2T) Fab2 #2 Fab2 #2 Fab2 #2 I/O Module Capacity – Nexus 7700 One fabric:  Any port can pass traffic to any other port in VDC Three fabrics:  480G F2E/F3 10G module has maximum bandwidth Five fabrics:  960G F3 40G module has maximum bandwidth Six fabrics:  1.2T F3 100G module has maximum bandwidth per slot bandwidth Fabric 2 Modules 1 Fabric 2 ASICs 2 Fabric 2 ASICs 3 Fabric 2 ASICs 4 Fabric 2 ASICs 5 Fabric 2 ASICs 6 Fabric 2 ASICs
  • 40.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 42 Fabric, VOQ, and Arbitration  Crossbar fabric – Provides dedicated, high-bandwidth interconnects between ingress and egress I/O modules  Virtual Output Queues (VOQs) – Provide buffering and queuing for ingress- buffered switch architecture  Central arbitration – Controls scheduling of traffic into fabric based on fairness, priority, and bandwidth availability at egress ports  Fabric, VOQ, and arbitration combine to provide all necessary infrastructure for packet transport inside switch
  • 41.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 43 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
  • 42.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 44 Buffering, Queuing, and Scheduling  Buffering – storing packets in memory – Needed to absorb bursts, manage congestion  Queuing – buffering packets according to traffic class – Provides dedicated buffer for packets of different priority  Scheduling – controlling the order of transmission of buffered packets – Ensures preferential treatment for packets of higher priority and fair treatment for packets of equal priority  Nexus 7000 / Nexus 7700 use queuing policies and network-QoS policies to define buffering, queuing, and scheduling behaviour  Default queuing and network-QoS policies always in effect in absence of any user configuration
  • 43.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 45 I/O Module Buffering Models  Buffering model varies by I/O module family – M-series modules: hybrid model combining ingress VOQ-buffered architecture with egress port-buffered architecture – F-series modules: pure ingress VOQ-buffered architecture
  • 44.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 46 Egress Module Ingress Module VOQ 0 Port ASIC 0 M2 – Hybrid Ingress/Egress Buffered DWRR VOQ 1 RE 1 RE 0 … DWRR Port 1 Port 12 FABRIC VOQ 0 VOQ 1 RE 1 RE 0 Port ASIC 0 Port 1 Port 12 Ingress port buffer – Manages congestion of ingress forwarding/replication engines, and congestion toward egress destinations (VQIs) Buffering / queuing / scheduling INGRESS QUEUING POLICIES 10G module used as example Diagram represents half of each I/O module 8 ingress queues
  • 45.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 47 Egress Module Ingress Module VOQ 0 1 2 3 4 5 6 SP 2 3 4 VOQ Buffer Source Priority Port ASIC 0 M2 – Hybrid Ingress/Egress Buffered DWRR VOQ 1 RE 1 RE 0 … DWRR Port 1 Port 12 FABRIC VOQ 0 VQI 1 DWRR SP … VQI 6 DWRR SP VOQ 1 Sources 7-12 VQIs 7-12 RE 1 RE 0 Port ASIC 0 Port 1 Port 12 Ingress port buffer – Manages congestion of ingress forwarding/replication engines, and congestion toward egress destinations (VQIs) Buffering / queuing / scheduling Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling FABRIC-QOS POLICY 10G module used as example Diagram represents half of each I/O module Shared buffer carved by source and priority 4 priority levels
  • 46.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 48 Egress Module Ingress Module VOQ 0 1 2 3 4 5 6 SP 2 3 4 VOQ Buffer Source Priority Port ASIC 0 DWRR VOQ 1 RE 1 RE 0 … DWRR Port 1 Port 12 FABRIC VOQ 0 VQI 1 DWRR SP … VQI 6 DWRR SP VOQ 1 Sources 7-12 VQIs 7-12 RE 1 RE 0 Port ASIC 0 Port 1 Port 12 DWRR SP … DWRR SP Ingress port buffer – Manages congestion of ingress forwarding/replication engines, and congestion toward egress destinations (VQIs) Buffering / queuing / scheduling Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling Egress port buffer – Manages congestion at egress physical interface Buffering / queuing / scheduling EGRESS QUEUING POLICIES 10G module used as example Diagram represents half of each I/O module 8 egress queues M2 – Hybrid Ingress/Egress Buffered
  • 47.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 49 Egress SOC Ingress SOC Ingress VOQ F2E – Ingress Buffered (Nexus 7000) FABRIC Egress VOQ VQI 1 DWRR PQ … VQI 4 DWRR PQ Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling 1 2 3 4 hi VOQ Buffer lo hi lo hi lo hi lo 10G Port 1 10G Port 2 10G Port 3 10G Port 4 10G Port 1 10G Port 2 10G Port 3 10G Port 4 Diagram represents one SoC on each I/O module INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES 2 or 4 ingress queues per port 4 priority levels
  • 48.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 50 Egress SOC Ingress SOC Ingress VOQ F3 10G – Ingress Buffered (Nexus 7700) FABRIC Egress VOQ VQI 1 … VQI 8 Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling 2 4 6 8 VOQ Buffer hi lo 1 3 5 7 10G Port 1 10G Port 3 10G Port 5 10G Port 2 Diagram represents one SoC on each I/O module INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES 10G Port 4 10G Port 6 10G Port 7 10G Port 8 hi lo hi lo hi lo hi lo hi lo hi lo hi lo 10G Port 1 10G Port 2 10G Port 3 10G Port 4 10G Port 5 10G Port 6 10G Port 7 10G Port 8 DWRR PQ DWRR PQ 2 or 4 ingress queues per port 8 priority levels
  • 49.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 51 Egress SOC Ingress SOC Ingress VOQ F3 40G – Ingress Buffered (Nexus 7000) FABRIC Egress VOQ VQI 1 DWRR PQ VQI 2 DWRR PQ Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling 1 2 VOQ Buffer lo hi lo hi 40G Port 1 40G Port 2 40G Port 1 Diagram represents one SoC on each I/O module INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES 40G Port 2 2 or 4 ingress queues per port 4 priority levels
  • 50.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 52 Egress SOC Ingress SOC Ingress VOQ F3 40G – Ingress Buffered (Nexus 7700) FABRIC Egress VOQ VQI 1 VQI 2 Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling 40G Port 1 Diagram represents one SoC on each I/O module INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES 40G Port 2 DWRR PQ DWRR PQ 1 2 VOQ Buffer lo hi lo hi 40G Port 1 40G Port 2 2 or 4 ingress queues per port 8 priority levels
  • 51.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 53 Egress SOC Ingress SOC Ingress VOQ F3 100G – Ingress Buffered (Nexus 7700) FABRIC Egress VOQ VQI 1 Ingress VOQ buffer – Manages congestion toward egress destinations (VQIs) Buffering / queuing Egress VOQ buffer – Receives frames from fabric Scheduling 1 VOQ Buffer hi lo 100G Port 1 Diagram represents one SoC on each I/O module INGRESS QUEUING POLICIES EGRESS QUEUING POLICIES DWRR PQ 100G Port 1 2 or 4 ingress queues per port 8 priority levels
  • 52.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 54 FAQ: What Is a VQI?  VQI = Virtual Queuing Index  “A Destination Across the Fabric”  For M2 / F2E / F3 10G modules, VQI == 10G interface  For M2 40/100G ports, uses multiple 10G VQIs  For F3 40/100G ports, uses single 40/100G VQI
  • 53.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 55 40G Port Ingress Modules 10G 10G 40G 40G 100G Spines Spines Spines Spines Fabrics M2 Module 40G and 100G Flow Limits  Each Virtual Queuing Index (VQI) sustains 10G traffic flow  All packets in given 5-tuple flow hash to single VQI  Single-flow limit is 10G  Packets split into 66-bit “code words”  Four code words transmitted in parallel, one on each physical Tx fibre  No per-flow limit imposed – splitting occurs at physical layer Egress Interfaces Destination VQIs 1 VQI 1 VQI 4 VQIs 4 VQIs 10 VQIs Internal to Nexus 7000 System n … 4 3 2 1 64 bits 1 packet On the Wire (40G) Tx 1 Tx 2 Tx 3 Tx 4 66 bits 1 5 2 3 4 6 … 64/66B Encoding
  • 54.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 56 Ingress Modules 10G 10G 40G 40G 100G Spines Spines Spines Spines Fabrics F3 Module 40G and 100G Flow Limits  Virtual Queuing Index (VQI) sustains 10G, 40G, or 100G traffic flow based on destination interface type  No single-flow limit – full 40G/100G flow support Egress Interfaces Destination VQIs 1 VQI 1 VQI 1 VQI 1 VQI 1 VQI Internal to Nexus 7000 / 7700 System
  • 55.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 57 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
  • 56.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 58 Hardware Layer 2 Forwarding Process Layer 2 forwarding – traffic steering based on destination MAC address  MAC table lookup drives Layer 2 forwarding  Source MAC and destination MAC lookups performed for each frame, based on {VLAN,MAC} pairs  Source MAC lookup drives new learns and refreshes aging timers  Destination MAC lookup dictates outgoing switchport
  • 57.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 59 Module 1 Fabric Module 1 Fabric ASIC Fabric Module 2 Fabric ASIC Fabric Module 3 Fabric ASIC Supervisor Engine Central Arbiter Fabric 2 ASIC 10G/40G/100G MAC / LinkSec VOQs Replication Engine Replication Engine VOQs e1/1 Layer 2 Engine Layer 3 Engine Forwarding Engine Module 2 Fabric 2 ASIC 10G/40G/100G MAC / LinkSec VOQs Replication Engine Replication Engine VOQs e2/2 Layer 2 Engine Layer 3 Engine Forwarding Engine M2 L2 Packet Flow  Receive packet from wire  LinkSec decryption  Ingress port QoS  Submit packet headers for lookup  ACL/QoS/ NetFlow lookups  VOQ arbitration and queuing  Round-robin transmit to fabric  Receive from fabric  Return buffer credit  Return credit to pool  Transmit packet on wire  Return result – destination + hash result  Credit grant for fabric access  Egress port QoS  LinkSec encryption  Static or hash- based RE uplink selection  Hash-based uplink and VQI selection  Round-robin transmit to VQI  Static downlink selection  L2 SMAC/ DMAC lookups  Port-channel hash result HDR = Packet Headers DATA = Packet Data = Internal Signalling CTRL
  • 58.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 60 SoC VOQ SoC DE F2E / F3 L2 Packet Flow Module 2 Fabric ASIC e2/2 Module 1 Fabric ASIC e1/1 Fabric Module 1 Fabric ASIC Fabric Module 2 Fabric ASIC Fabric Module 3 Fabric ASIC Supervisor Engine Central Arbiter  VOQ arbitration  Credit grant for fabric access  Receive from fabric  Return credit to pool  Transmit packet on wire Fabric Module 4 Fabric ASIC Fabric Module 5 Fabric ASIC  Transmit to fabric VOQ  Receive packet from wire  Ingress port QoS (VOQ)  Ingress L2 SMAC/ DMAC lookups, ACL/QoS lookups, NetFlow sampling  Return result – destination  Submit packet headers for lookup  Egress port QoS (Scheduling)  Return buffer credit HDR = Packet Headers DATA = Packet Data = Internal Signalling CTRL
  • 59.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 61 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
  • 60.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 62 Layer 3 Forwarding  Nexus 7000 decouples control plane and data plane  Forwarding tables built on control plane using routing protocols or static configuration – OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing  Tables downloaded to forwarding engine hardware for data plane forwarding – FIB TCAM contains IP prefixes – Adjacency table contains next-hop information
  • 61.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 63 Hardware Layer 3 Forwarding Process  FIB TCAM lookup based on longest-match destination prefix comparison  FIB “hit” returns adjacency, adjacency contains rewrite information (next-hop)  Pipelined forwarding engine architecture also performs ACL, QoS, and NetFlow lookups, affecting final forwarding result
  • 62.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 64 10.1.1.2 10.1.1.3 10.10.0.10 10.10.0.100 10.10.0.33 10.1.1.4 10.1.2.xx 10.1.3.xx 10.1.1.xx 10.100.1.xx 10.10.0.xx 10.100.1.xx 10.10.100.xx IP FIB TCAM Lookup FIB TCAM Generate Lookup Key 10.1.1.10 Generate TCAM lookup key (destination IP address) Forwarding Engine FIB DRAM Load-Sharing Hash Adjacency Table Next-hop 4 (IF, MAC) Next-hop 6 (IF, MAC) Next-hop 7 (IF, MAC) Next-hop 5 (IF, MAC) Next-hop 3 (IF, MAC) Next-hop 1 (IF, MAC) Next-hop 2 (IF, MAC) 10.1.1.xx Ingress unicast IP packet header Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Hit in FIB returns result in FIB DRAM Adjacency index identifies ADJ block to use Modulo function selects exact next hop entry to use Offset Compare lookup key Return lookup result # next- hops Flow Data Result HIT! Adj Index mod
  • 63.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 65 Module 1 Fabric Module 1 Fabric ASIC Fabric Module 2 Fabric ASIC Fabric Module 3 Fabric ASIC Supervisor Engine Central Arbiter Fabric 2 ASIC 10G/40G/100G MAC / LinkSec VOQs Replication Engine Replication Engine VOQs e1/1 Layer 2 Engine Layer 3 Engine Forwarding Engine Module 2 Fabric 2 ASIC 10G/40G/100G MAC / LinkSec VOQs Replication Engine Replication Engine VOQs e2/2 Layer 2 Engine Layer 3 Engine Forwarding Engine M2 L3 Packet Flow  Receive packet from wire  LinkSec decryption  Ingress port QoS  Submit packet headers for lookup  L3 FIB/ADJ lookup  Ingress and egress ACL/QoS/NetFlow lookups  VOQ arbitration and queuing  Round-robin transmit to fabric  Receive from fabric  Return buffer credit  Return credit to pool  Transmit packet on wire  Return result – destination + hash result  Credit grant for fabric access  Egress port QoS  LinkSec encryption  Static or Hash-based uplink selection  Hash-based uplink (and VQI) selection  Round-robin transmit to VOQ  Static RE downlink selection  L2 ingress and egress SMAC/ DMAC lookups  Port-channel hash result HDR = Packet Headers DATA = Packet Data = Internal Signalling CTRL
  • 64.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 66 SoC VOQ SoC DE Module 2 Fabric ASIC e2/2 Module 1 Fabric ASIC e1/1 Fabric Module 1 Fabric ASIC Fabric Module 2 Fabric ASIC Fabric Module 3 Fabric ASIC Supervisor Engine Central Arbiter Fabric Module 4 Fabric ASIC Fabric Module 5 Fabric ASIC VOQ F2E / F3 L3 Packet Flow HDR = Packet Headers DATA = Packet Data = Internal Signalling CTRL  VOQ arbitration  Credit grant for fabric access  Return credit to pool  Transmit packet on wire  Transmit to fabric  Receive packet from wire  Ingress port QoS (VOQ)  Return result – destination  Submit packet headers for lookup  L2 ingress and egress SMAC/ DMAC lookups  L3 FIB/ADJ lookup  Ingress and egress ACL/QoS lookups, NetFlow sampling  Receive from fabric  Egress port QoS (Scheduling)  Return buffer credit
  • 65.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 67 Layer 3 Forwarding – Module Interoperability Models Two interoperability models for L3 forwarding:  “Proxy Forwarding”  “Ingress Forwarding” with Lowest Common Denominator
  • 66.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 68  From F1/F2E perspective, Router MAC reachable through giant port-channel  All packets destined to Router MAC forwarded through fabric toward one “member port” in that channel Proxy Forwarding Model – Conceptual All F1/F2E modules All M1/M2 modules Up to 128 “links” 10.1.10.100 vlan 10 10.1.20.100 vlan 20 interface vlan 10 ip address 10.1.10.1/24 ! interface vlan 20 ip address 10.1.20.1/24
  • 67.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 69 Proxy Forwarding Model – Actual 10.1.10.100 vlan 10 e1/1 Fabric F1/F2E SoC FE e2/1 Fabric F1/F2E SoC FE 10.1.20.100 vlan 20 Replication Engine e3/1 e3/2 M1/M2 Replication Engine Replication Engine Replication Engine VOQs VOQs FE FE Fabric e3/3 e3/4 e3/5 e3/6 e3/7 e3/8 Replication Engine e4/1 e4/2 M1/M2 Replication Engine Replication Engine Replication Engine VOQs VOQs FE FE Fabric e4/3 e4/4 e4/5 e4/6 e4/7 e4/8 Fabric Fabric Modules Fabric … VLAN DMAC Dest Port 10 router_mac → internal_channel (e3/1-8,e4/1-8) EtherChannel Hash Function hash_input (from packet) → select_member_port Ingress MAC: VLAN DMAC Dest Port 10 router_mac → L3_lookup Routing: DIP Next Hop 10.1.20.100 → server_2_mac (v20) Egress MAC: VLAN DMAC Dest Port 20 server_2_mac → e2/1 1 2 3 4 6 5 7 8 9 10 Programming of all M1/M2 forwarding engines Programming of all F1/F2E forwarding engines interface vlan 10 ip address 10.1.10.1/24 ! interface vlan 20 ip address 10.1.20.1/24 Can be up to 128 M1/M2 VQIs Mod 1 Mod 2 Mod 4 Mod 3
  • 68.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 71 Ingress Forwarding with Lowest Common Denominator Model  F3 module interoperability always Ingress Forwarding – NO proxy forwarding with F3 – Essentially equivalent to current M1 + M2 interoperability model – The ingress module makes all the forwarding decisions  Supported feature set based on Lowest Common Denominator – Feature available if all modules support the feature VDC Type Layer 2 Layer 3 vPC Fabric Path VXLAN FEX MPLS OTV LISP FCoE Table Sizes F3 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ F3 size M2 + F3 ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✗ ✗ F3 size F2/F2E + F3 ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✓ F2E size Not all features supported by software today
  • 69.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 73 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
  • 70.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 74 What Is Classification?  Matching packets – Layer 2, Layer 3, and/or Layer 4 information  Used to decide whether to apply a particular policy to a packet – Enforce security, QoS, or other policies  Some examples: – Match TCP/UDP source/destination port numbers to enforce security policy – Match destination IP addresses to apply policy-based routing (PBR) – Match 5-tuple to apply marking policy – Match protocol-type to apply Control Plane Policing (CoPP) – etc.
  • 71.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 75 CL TCAM Lookup – ACL ip access-list example permit ip any host 10.1.2.100 deny ip any host 10.1.68.44 deny ip any host 10.33.2.25 permit tcp any any eq 22 deny tcp any any eq 23 deny udp any any eq 514 permit tcp any any eq 80 permit udp any any eq 161 xxxxxxx | 10.1.2.100 | xx | xxx | xxx xxxxxxx | 10.1.68.44 | xx | xxx | xxx xxxxxxx | 10.33.2.25 | xx | xxx | xxx xxxxxxx | xxxxxxx | tcp | xxx | 22 xxxxxxx | xxxxxxx | tcp | xxx | 23 xxxxxxx | xxxxxxx | tcp | xxx | 80 xxxxxxx | xxxxxxx | udp | xxx | 161 xxxxxxx | xxxxxxx | udp | xxx | 514 Packet header: SIP: 10.1.1.1 DIP: 10.2.2.2 Protocol: TCP SPORT: 33992 DPORT: 80 CL TCAM Generate Lookup Key Generate TCAM lookup key CL SRAM 10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80 xxxxxxx | 10.2.2.2 | xx | xxx | xxx xxxxxxx | xxxxxxx | tcp | xxx | 80 SIP | DIP | Pr | SP | DP Compare lookup key to CL TCAM entries Comparisons (X = “Mask”) Hit in CL TCAM returns result in CL SRAM Security ACL Forwarding Engine Result Return lookup result Result affects final packet handling Permit Permit Permit Permit Deny Deny Deny Deny HIT! Results SIP | DIP | Pr | SP | DP
  • 72.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 76 Packet header: SIP: 10.1.1.1 DIP: 10.2.2.2 Protocol: TCP SPORT: 33992 DPORT: 80 Result affects final packet handling Generate Lookup Key Forwarding Engine xxxxxxx | 10.3.3.xx | xx | xxx | xxx xxxxxxx | 10.4.12.xx | xx | xxx | xxx 10.1.1.xx | xxxxxxx | udp | xxx | xxx 10.1.1.xx | xxxxxxx | tcp | xxx | xxx xxxxxxx | 10.5.5.xx| tcp | xxx | 23 CL TCAM Lookup – QoS ip access-list police permit ip any 10.3.3.0/24 permit ip any 10.4.12.0/24 ip access-list remark-dscp-32 permit udp 10.1.1.0/24 any ip access-list remark-dscp-40 permit tcp 10.1.1.0/24 any ip access-list remark-prec-3 permit tcp any 10.5.5.0/24 eq 23 CL TCAM 10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80 xxxxxxx | 10.2.2.xx | xx | xxx | xxx 10.1.1.xx | xxxxxxx | tcp | xxx| xxx HIT! CL SRAM QoS Classification ACLs Generate TCAM lookup key SIP | DIP | Pr | SP | DP Compare lookup key Hit in CL TCAM returns result in CL SRAM Result Return lookup result Policer ID 1 Policer ID 1 Remark DSCP 32 Remark DSCP 40 Remark IP Prec 3 SIP | DIP | Pr | SP | DP Comparisons (X = “Mask”) Results
  • 73.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 77 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
  • 74.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 78 NetFlow  NetFlow collects flow data for packets traversing the switch  Each module maintains independent NetFlow table M2 F2E / F3 Per-interface NetFlow Yes Yes NetFlow direction Ingress/Egress Ingress only Full NetFlow Yes No Sampled NetFlow Yes Yes FSA Assist for Sampled NetFlow No F3 only (future) Bridged NetFlow Yes Yes Hardware Cache Yes No Software Cache No Yes Hardware Cache Size 512K entries per forwarding engine N/A NDE (v5/v9) Yes Yes
  • 75.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 79 Full vs. Sampled NetFlow  NetFlow collects full or sampled flow data  Full NetFlow: Accounts for every packet of every flow on interface – Available on M-Series modules only – Flow data collection up to capacity of hardware NetFlow table  Sampled NetFlow: Accounts for M in N packets on interface – Available on both M2 (ingress/egress) and F2E/F3 (ingress only) – M2: Flow data collection up to capacity of hardware NetFlow table – F2E/F3: Flow data collection for up to ~1000pps per module – F3 (future): Increased per-module sampling rate leveraging on-board Fabric Services Accelerator (FSA) complex
  • 76.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 81 NetFlow on M2 Modules Fabric ASIC VOQs Mgmt Enet Supervisor Engine Forwarding Engine LC CPU NetFlow Table M2 Module Forwarding Engine LC CPU NetFlow Table M2 Module Forwarding Engine LC CPU NetFlow Table M2 Module Hardware Flow Creation Hardware Flow Creation Hardware Flow Creation Aged Flow Info Aged Flow Info Aged Flow Info Generate NetFlow v5 or v9 export packets Main CPU To NetFlow Collector To NetFlow Collector Switched EOBC via Supervisor Inband via mgmt0
  • 77.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 82 Sampled NetFlow on F2E/F3 Modules F3 Module FSA CPU SoC Decision Engine DRAM NetFlow Cache F3 Module Fabric ASIC VOQs Mgmt Enet Supervisor Engine FSA CPU SoC Decision Engine Main CPU To NetFlow Collector To NetFlow Collector Switched EOBC via Supervisor Inband via mgmt0 DRAM NetFlow Cache Populate cache based on received samples Age flows and generate NetFlow v5 or v9 export packets F2E Module LC CPU SoC Decision Engine DRAM NetFlow Cache Data Flow Data Flow Data Flow via Module Inband via Module Inband via Module Inband Sampled Packets Sampled Packets Sampled Packets Aged Flows Aged Flows Aged Flows
  • 78.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 83 Agenda  Chassis Architecture  Supervisor Engine and I/O Module Architecture  Forwarding Engine Architecture  Fabric Architecture  I/O Module Queuing  Layer 2 Forwarding  Layer 3 Forwarding  Classification  NetFlow  Conclusion
  • 79.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 84 Nexus 7000 / Nexus 7700 Architecture Summary I/O Modules Supervisor Engines Fabrics Chassis
  • 80.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 85 Conclusion  You should now have a thorough understanding of the Nexus 7000 / Nexus 7700 switching architecture, I/O module design, packet flows, and key forwarding engine functions…  Any questions? 85
  • 81.
  • 82.
    © 2014 Ciscoand/or its affiliates. All rights reserved. BRKARC-3470 Cisco Public 88 Complete Your Online Session Evaluation Give us your feedback and receive a Cisco Live 2014 Polo Shirt! Complete your Overall Event Survey and 5 Session Evaluations.  Directly from your mobile device on the Cisco Live Mobile App  By visiting the Cisco Live Mobile Site www.ciscoliveaustralia.com/mobile  Visit any Cisco Live Internet Station located throughout the venue Polo Shirts can be collected in the World of Solutions on Friday 21 March 12:00pm - 2:00pm Learn online with Cisco Live! Visit us online after the conference for full access to session videos and presentations. www.CiscoLiveAPAC.com