Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Argentina, Chile, Mexico, Puerto Rico, Brasil      May-Ago/2012© 2011 Cisco and/or its affiliates. All rights reserved.   ...
Cisco Unified Data Center                          Architecture & Evolution   Carlos Pereira   Distinguished Systems Engin...
Need for better high availability (HA) and lower fate sharingNeed to achieve higher scalability (L2 Multipath, L3, MAC, VM...
Denser Server Cabinets, Denser PODs & Denser X-connectsLonger cable distances, diverse connector typesServer Migration to ...
Cisco UnifiedData Center Fabric                     5
client-to-server                             client-to-server        Aggregation                                          ...
Hypervisor based server virtualization and the             Data Center Row 1     associated capabilities (VM Mobility, ) a...
Server, Storage, Application and     Facilities are driving Layer 2     Scalability requirements                  Server V...
Ethernet is the network for Data Center!© 2011 Cisco and/or its affiliates. All rights reserved.                          ...
• IEEE 802            Evolution of Ethernet                 10 GE, 40 GE, 100 GE, copper and fiber            Evolution of...
© 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Confidential   11
“Scaling UP” the Network Pod and Scaling ‘OUT’ the Fabric     Scaling ‘Up’ of the building blocks (High Density 10G, Unifi...
Traditional                  Scalable PODs                   Scalable Fabric                Topological Approach          ...
2010© 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Confidential   14                                    ...
Network Planes of Operation                                                           The business glue of the network. Ru...
Fully specifies a Port Extender (FEX Equivalent)                                                                          ...
Distributed Modular System to the ToR, Server, and Virtual Machine                                                        ...
Distributed Modular System to the ToR, Server, and Virtual Machine                                                        ...
Fabric Extender Evolution     Distributed Modular System to the ToR, Server, and Virtual Machine                          ...
Distributed Modular System to the ToR, Server, and Virtual Machine                                                        ...
1                                                                                       2                                 ...
Nexus 2000 Fabric Extender (FEX)                                                                                        Ci...
MultiChassis EtherChannel (MCEC)• vPC is a Port-channeling concept      extending link aggregation to two separate      ph...
Co-existence of LAN and SAN   LAN and SAN utilize different High Availability Models   SAN is dual fabric, LAN is fully me...
2                                                           Rack 13   Rack 14   Rack 24© 2011 Cisco and/or its affiliates....
• Cisco Nexus 5x00 and 2200        represent a virtual access        switch POD                                           ...
Nexus 7000                                                             vPC+                    Unified Computing          ...
2010© 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Confidential   28                                    ...
Large Scale Web 2.0 Environments                                                           L3                             ...
Data Center Fabric Topologies & Attributes - Trending                           L3 Cloud                             Large...
The protocol choices allow like topologies to be build            Equal Cost Multi Path (ECMP) over L2 or L3            Pl...
L3                                                           L3/L2   L3/L2                                                ...
WHY should I and                                        HOW can I                            leverage this evolution on my...
Focused on Stability Features                                                                                             ...
Focused on Scalability Features, with a pair of Aggregation boxes.                                                        ...
• MAC addresses encode no location or network hierarchy    • Default forwarding behavior in bridged network is flood    • ...
MAC v.s. IP                                                                                       Network Address         ...
What Can Be Improved?  • Network Address Scheme: Flat                            Hierarchical               Additional hea...
Cisco FabricPath                                Data Plane Innovation                  Control Plane Innovation           ...
Multi-Domain – Silos                      FabricPath – Any App, Anywhere!                                                 ...
Externally, a Fabric looks like a single switch                           Internally, a protocol adds Fabric-wide intellig...
Enabling Network Fabrics                                                                                         FabricPat...
Example 1: Classical POD Migration                                                           Q: Why migrate traditional Ac...
Q: How to interconnect DC PODs in order to have VLANs anywhere?   A:   • Provide server/host connection to any edge port i...
Cisco FabricPath                    Spanning-Tree                                 vPC           FabricPath                ...
Pod                        Spine                                                                          Tier 2          ...
Terminology                                                       Interface connected to another FabricPath device        ...
A New Data Plane                                            • The association MAC address/Switch ID is                    ...
Conversational MAC Learning                                                               S10     S20         S30         ...
Conversational MAC Learning                                                               S10       S20              S30  ...
It’s a Routed Network                                             • Describes shortest (best) paths to each Switch ID     ...
(1) Broadcast ARP Request                                                                                                 ...
MAC Address Table after the first ARP frame                             •       S100:                                     ...
(2) Broadcast ARP Reply                                                                                         Root for  ...
MAC Address Table after the first ARP frame                           •       S100:                                     S1...
Unicast Data - Routed                                      FabricPath Routing                                      Table o...
Unicast forwarding                           •      S100:                                    S100# sh mac address-table dy...
Unicast ForwardingS100# sh fabricpath routeFabricPath Unicast Route Tablea/b/c denotes ftag/switch-id/subswitch-id[x/y] de...
S3                                      → FabricPath                                                                      ...
Driven by multiple vendors, including Cisco                    TRILL is now an IETF proposed standard                    F...
HSRP                                                                HSRP                                                  ...
© 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Confidential   62
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution
Upcoming SlideShare
Loading in …5
×

Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution

6,050 views

Published on

Published in: Technology, Education

Cisco Unified Computing and Virtualization: Architecture, Design and Deployment - Architecture and Evolution

  1. 1. Argentina, Chile, Mexico, Puerto Rico, Brasil May-Ago/2012© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1
  2. 2. Cisco Unified Data Center Architecture & Evolution Carlos Pereira Distinguished Systems Engineer II – Data Center May/2012© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
  3. 3. Need for better high availability (HA) and lower fate sharingNeed to achieve higher scalability (L2 Multipath, L3, MAC, VMs, etc.)Need to accommodate diverse workloads concurrentlyNeed to further simplify operational modelsNeed better network visibilityNeed to be prepared for: These require Cisco to address: Intel next-gen CPUs Increase feature, function and scale without increasing PCIe 3.0 complexity 10G LOMs Continue to leverage/develop standards protocols to 10G-T support open and interoperable environments 40G Uplinks Increase visibility, instrumentation and manageability 100G Interconnects QSFP+ optics Evolutionary Steps prevent disruption to operational models 3
  4. 4. Denser Server Cabinets, Denser PODs & Denser X-connectsLonger cable distances, diverse connector typesServer Migration to 10G uplinks to higher density 10G or 40GEarly 40G server adoption – specific workloadsHigher 10G and 40G switch density per RU & Denser 2nd switch tierFlexible L2/L3 Boundary Placement & Redundancy from 1+1 to N+1Virtual Machine & bare metal mobility: within and across DCsApplication logical isolation at scaleApplication processing closer to the wire 4
  5. 5. Cisco UnifiedData Center Fabric 5
  6. 6. client-to-server client-to-server Aggregation Aggregation L3 Service Service processing srv-to-srv or vm-to-vm processing Access Access L2 srv-to-srv srv-to-srv or vm-to-vm srv-to-srv Virtual Access Virtual Access L2 VM-to-VM srv-to-srv or vm-to-vm srv-to-srv VM-to-VM srv-to-srv srv-to-srv srv-to-srv srv-to-srv Srv-to-client Srv-to-client srv-to-srvTraffic Patterns Changing:• More server to server traffic, and more L2 server to server traffic. Apps such as VM mobility, clustering, intra-Tier and larger subnets• Client to server traffic to same subnet instances across DCs: increase of /32 from DC out• Virtual Server Environments could perform a fair degree of local switching• Server Roll-outs and workload movement require physical and network infrastructure coordination © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
  7. 7. Hypervisor based server virtualization and the Data Center Row 1 associated capabilities (VM Mobility, ) are changing multiple aspects of the Data Center design Where is the server now? Where is the access port? Where does the VLAN exist? Any VLAN Anywhere? How large do we need to scale Layer 2? Data Center Row 2 What are the capacity planning requirements for flexible workloads? Where are the policy boundaries with flexible workload (Security, QoS, WAN acceleration, )?© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
  8. 8. Server, Storage, Application and Facilities are driving Layer 2 Scalability requirements Server Virtualization and Clustering driving the need for every / any VLAN everywhere based design Facilities requirements defining the network topology “No watt shall be left behind”• VM requirements along with Data Storage growth mandating a need for more efficient and pervasive network based storage Technology changes will impact any cabling plant design Migration to 10GE as the default LoM technology © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
  9. 9. Ethernet is the network for Data Center!© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
  10. 10. • IEEE 802 Evolution of Ethernet 10 GE, 40 GE, 100 GE, copper and fiber Evolution of switching DCB: Data Center Bridging 802.1BR: Bridge Port Extension• INCITS/T11 Evolution of Fibre Channel FCoE (Fibre Channel over Ethernet)• IBTA (Infiniband Trade Association) RoCE (RDMA over converged Ethernet), aka IBoE or RoE• IETF Layer 2 Multi-Path (L2MP) TRILL (Transparent Interconnection of Lots of Links)© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
  11. 11. © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
  12. 12. “Scaling UP” the Network Pod and Scaling ‘OUT’ the Fabric Scaling ‘Up’ of the building blocks (High Density 10G, Unified IO, FEX, Adapter-FEX, vPC, FabricPath) Scaling ‘Out’ of the Fabric (FabricPath, OTV, OSPF/EIGRP/ISIS/BGP, MPLS) Scaling ‘OUT’ the Fabric Scaling ‘UP’ the aggregation block VM VM VM (POD) VM VM VM #2 #3 #4 VM VM VM #2 #3 #4 #2 #3 #4© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
  13. 13. Traditional Scalable PODs Scalable Fabric Topological Approach FEX and switch scaling Multipathing Strategic Investments L2 / L3 • 10/40 GbE Leadership • Fabric Scale (L2 & L3) • Manageability and Programmability Enterprise, SMB • Location Independence Cloud providers, large DC Enterprise, SMB, HPC / Mobility 100s – 1,000s • Control Plane Intelligence 100s - 10,000s 10,000s - 100,000s servers per POD Servers/VMs per POD Servers/VMs per POD • Simplicity of Management© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
  14. 14. 2010© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14 14
  15. 15. Network Planes of Operation The business glue of the network. Rules execution, decision Policy making, Service Manager and all the other components to make a Plane productize service. Services Overlay “Layer 7” application flow built on the foundation of the Plane other layers. Dependent on the other layers. Management The management plane is the logical path of all traffic related to Plane the system management of the platform. Control It’s the brain of any networking platform and the technical glue of the network. The control plane is where all routing, switching, Plane other protocols and control information are exchanged The data plane receives, processes, and transmits network data Data Plane between network elements, and represents the bulk of network traffic that passes to and through the gear.© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
  16. 16. Fully specifies a Port Extender (FEX Equivalent) Extended Bridge Extends ports of a switch to lower entities in a network Port Extenders are not individually managed Controllin g Their ports become ports of the controlling switch PE Bridge Cascading Port Extenders Allows one to choose the appropriate controlling switch PE PE Frame replication supported for efficient multicast / flooding Bridge PE Traffic from each “Extended Port” is reliably segregated to an E-channel and identified by a tag containing an E-channel identifier (ECID) Does not require prior knowledge of MAC addresses; switch performs standard learning functions Server PE Works with all devices including VEBs, VEPAs, individual VMs, ECID physical services, and devices providing transparent services VM vF 1 W Controlling Bridge + PE = Extended Bridge Single Point of Management PE Port Extender© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
  17. 17. Distributed Modular System to the ToR, Server, and Virtual Machine One Network Parent Switch to Top of Rack FEX Architecture Network Administrator IEEE 802.1 BR* Consolidates network management FEX managed as line card of parentMany applications FEX switch require Uses Pre-standard IEEE 802.1Qbhmultiple interfaces Legacy *IEEE 802.1QR Pre-Standard© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
  18. 18. Distributed Modular System to the ToR, Server, and Virtual Machine One Network Parent Switch to Adapter Network Administrator IEEE 802.1 BR* Adapter FEX FEX Consolidates multiple 1Gb interfaceMany applications into a single 10Gb interface requiremultiple interfaces Extends network into server Uses Pre-standard IEEE 802.1Qbh IEEE 802.1 Qbh* Legacy Adapter FEX *IEEE 802.1QR Pre-Standard© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
  19. 19. Fabric Extender Evolution Distributed Modular System to the ToR, Server, and Virtual Machine One Network Virtual Same As Physical Network Administrator IEEE 802.1 BR* FEX VM-FEX Consolidates virtual and physical network Each VM gets a dedicated port on IEEE 802.1 Qbh* IEEE 802.1 Qbh* switch Uses Pre-standard IEEE 802.1Qbh Hypervisor VM network managed by Server administrator Legacy Adapter FEX VM-FEX *IEEE 802.1QR Pre-Standard© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
  20. 20. Distributed Modular System to the ToR, Server, and Virtual Machine One Network Parent Switch to Application Single Point of Management Network Administrator FEX Architecture IEEE 802.1 BR*Manage network all Consolidates network management FEX the way to FEX managed as line card of parentthe OS interface – switch Physical and Adapter FEX Virtual Consolidates multiple 1Gb interface IEEE 802.1 Qbh* IEEE 802.1 Qbh* into a single 10Gb interface Extends network into server VM-FEX Consolidates virtual and physical Hypervisor network Each VM gets a dedicated port on switch Legacy Adapter FEX VM FEX *IEEE 802.1QR Pre-Standard© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
  21. 21. 1 2 3 4 5 7 8 6 Switch Nexus 5500 Eth Eth 1 2 3 4 5 Port Extension 802.1BR PE Tag 802.1BR Port Extender 1 Nexus 2200 (FEX) 1 2 3 PE Tag Server 802.1BR Adapter Port 0 Port 1 … Port n Hypervisor NIV Capable Adapter VM VM VM VM VM VM vNIC vNIC vNIC vNIC vNIC 1 2 3 4 5 IEEE Bridge Port Extender = Cisco FEX (Fabric Extender)© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
  22. 22. Nexus 2000 Fabric Extender (FEX) Cisco Nexus® 7000 Cisco Nexus® 5500 + + Distributed High Density Edge Switching System Cisco Nexus® 2000 FEX Cisco Nexus® 2000 FEX © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
  23. 23. MultiChassis EtherChannel (MCEC)• vPC is a Port-channeling concept extending link aggregation to two separate physical switches• Allows the creation of resilient L2 topologies based on Link Aggregation. Physical Topology Logical Topology Eliminates the need for STP in the Virtual Port Channel access-distribution L2• Provides increased bandwidth Si Si All links are actively forwarding• vPC maintains independent control plane Non-vPC vPC Increased BW with vPC© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
  24. 24. Co-existence of LAN and SAN LAN and SAN utilize different High Availability Models SAN is dual fabric, LAN is fully meshed fabric vPC enables ‘both’ architectures at the edge (single device models not acceptable to SAN customers) WAN Core FC Core Core L3 Aggregation L2 Access Edge© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
  25. 25. 2 Rack 13 Rack 14 Rack 24© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
  26. 26. • Cisco Nexus 5x00 and 2200 represent a virtual access switch POD VPC pair• Nexus 7000 at Aggregation Layer NO Loop Nexus 5x00/2200 Virtualized Access Switch PODs ... NO STP© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
  27. 27. Nexus 7000 vPC+ Unified Computing System (UCS) Nexus 5000 / 5500 + 2200 Virtual Access Switch POD Nexus 7000 + 2200 Virtual Blade Switching Virtual Access (VBS) Switch POD© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
  28. 28. 2010© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28 28
  29. 29. Large Scale Web 2.0 Environments L3 1. Soft L3 on access - per ToR VLANs L3 2. East-west traffic: 2 tiers – large scale L3 3. Limited VLAN extension – overlays (ex.: OTV) L2 L3 Enterprise Environments 1. VPC / STP used for L2 – restricted VLAN Scale L3 L2 2. Pod traffic: 2 tiers – limited scale 3. Cross-pod East-West traffic = 3 tiers L2 L3 L2 Large Scale SPDC: Hosting & Cloud 1. L2 on access and aggregation L2 2. VLAN Scale limited – high fate sharing 3. East-west traffic: 2 or 3 tier L2© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
  30. 30. Data Center Fabric Topologies & Attributes - Trending L3 Cloud Large Scale Web 2.0 Environments 1. L3 on access - per ToR VLANs L3 2. Migration to 10G: 3 tiers – very large scale 3. Broad VLAN extension through overlays L3 L2 L3 Fabrics Commonalities between Enterprise & SPDC 1. L2MP (FabricPath) used for L2 – increase VLAN spread L3 L3 Cloud 2. 2-tier east to west traffic L3 3. N-way tier 2 (spine) L2 Differences between Enterprise & SPDC L2 1. Host Density L3/L2 2. VLAN Scale – virtualization scale Fabrics 3. Public-cloud vs private cloud© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
  31. 31. The protocol choices allow like topologies to be build Equal Cost Multi Path (ECMP) over L2 or L3 Plug and Play Nature of L2 Protocols Redundancy, stability and scale of Layer 3 Protocols High Availability models become similar for L2 and L3: N+1 redundancyDifferent Environments Have: A preferred placement for the L2/L3 boundary A High-Availability, Scale, and functional targetsThese Converge given the flexibility offered by the forthcoming protocols Choice of L2 or L3 Protocols does not prevent redundancy and multi-pathing requirements Location of L2/L3 boundary does not prevent adjacency or redundancy options L2/L3 Boundary becomes less relevant CLOS Topologies dominate new implementations High Availability models shift Server Edge becomes more intelligent Data Center Fabric becomes more scalable© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
  32. 32. L3 L3/L2 L3/L2 L2 L2 East-West traffic – Fate Sharing Domain Larger POD East-West Traffic – Fate Sharing Domain STP has been the protocol of choice N+1 redundancy 1+1 redundancy – limited forwarding paths IS-IS is the protocol of choice Replicated Stateful Services per Pod Broad forwarding paths Broader Adjacency Support East-West across L3 boundaries Service Insertion is not replicated OSPF/EIGRP are protocols of choice N+1 redundancy – Broad forwarding Paths Same number of physical boxes and links North-South traffic Protocol behavior is L3-like OSPF/EIGRP are protocols of choice Multi-pathing over L2 and L3 N+1 redundancy – Broad forwarding paths More flexible L2 adjacency, better scale capacity Better latency consistency within POD© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32
  33. 33. WHY should I and HOW can I leverage this evolution on my own Data Center infrastructure© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33
  34. 34. Focused on Stability Features N Network port (Bridge Assurance) E Edge port - Normal port type Data Center B BPDUguard Core R Rootguard L Loopguard F Global BPDU filter HSRP HSRP ACTIVE STANDBY Layer 3 Aggregation N N Backup Root Root Layer 2 (STP + Bridge Assurance) N N N - N N N - R R R R R R R R Layer 2 (STP + BA + Rootguard) N N Access N N N N L L E F E F E F E F E F B B B B B Layer 2 (STP + BPDUguard)© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
  35. 35. Focused on Scalability Features, with a pair of Aggregation boxes. N Network port (Bridge Assurance) E Edge port - Normal port type Data Center B BPDUguard Core R Rootguard L Loopguard F Global BPDU filter VPC HSRP domain HSRP ACTIVE ACTIVE Layer 3 Aggregation N N Root Root Layer 2 (STP + Bridge Assurance) - - - - - - - - R R R R R R R R Layer 2 (STP + Rootguard) - Access - - L E E E E E F F F F F B B B B B Layer 2 (STP + BPDUguard) © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
  36. 36. • MAC addresses encode no location or network hierarchy • Default forwarding behavior in bridged network is flood • MAC filtering database limits scope of flooding • Ultimately, may not scale well as every switch learns every MAC MAC Table MAC Table A A Layer 2 Domain MAC Table MAC Table MAC Table A MAC Table A A A© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
  37. 37. MAC v.s. IP Network Address 10.0.0.0/24 0011.1111.1111 10.0.0.10 /24 Non-hierarchical Host Address Address 10.0.0.10 0011.1111.1111 0011.1111.1111 10.0.0.0/16 20.0.0.0/16 0011.1111.1111 10.0.0.0/24 20.0.0.0/24 0011.1111.1111 0011.1111.1111 10.0.0.10 20.0.0.20 L2 Forwarding (Bridging) L3 Forwarding (Routing) Data-plane learning Control-plane learning Flat address space and forwarding Hierarchical address space and table (MAC everywhere !!!) forwarding Flooding required for unknown unicast Only forwarding to destination destination addresses with matching routes in the Destination MACs need to be known table for all switches in the same network to Flooding is isolated within subnets avoid flooding No dependence on data-plane for maintaining forwarding table© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
  38. 38. What Can Be Improved? • Network Address Scheme: Flat Hierarchical Additional header is required to allow L2 “Routing” instead of “Bridging”. “Switch ID” comes to the picture. Provide additional loop-prevention mechanism like TTL • Address Learning: Data Plane Control Plane Eliminate the needs to program all MACs on every switches to avoid flooding • Control Plane: Distance-Vector Link-State Improve scalability, minimize convergence time, and allow multipathing inherently The ultimate solution needs to take both control and data plane into consideration this time!!!© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
  39. 39. Cisco FabricPath Data Plane Innovation Control Plane Innovation FabricPath encapsulation Plug-n-Play Layer 2 IS-IS Conversation Learning Support unicast and multicast Routing, not bridging Fast, efficient, and scalable Built-in loop-mitigation Equal Cost Multipathing Time-to-Live (TTL) (ECMP) RPF Check VLAN and Multicast Pruning Cisco NX-OS Cisco Nexus Platform© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
  40. 40. Multi-Domain – Silos FabricPath – Any App, Anywhere! Fabric Web Servers App Servers New Apps Web Servers App Servers Silo 1 Silo 2 Silo 3 New Apps • Benefits server team by providing a network Fabric that looks like a single switch → Breaks down silos, permits workload mobility, provides maximum flexibility • Lowers OPEX by simplifying server team operation → Reduces dependency on/interaction with network team© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40
  41. 41. Externally, a Fabric looks like a single switch Internally, a protocol adds Fabric-wide intelligence and ties the elements together. This protocol provides in a plug-and-play fashion: Optimal, low latency connectivity any to any High bandwidth, high resiliency Open management and troubleshooting Cisco FabricPath provides additional capabilities in term of scalability and L3 integration FabricPath FabricPath© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
  42. 42. Enabling Network Fabrics FabricPath • Connect a group of switches using an arbitrary topology • With a simple CLI, aggregate them into a Fabric: N7K(config)# interface ethernet 1/1 N7K(config-if)# switchport mode fabricpath An open protocol based on L3 technology provides Fabric- wide intelligence and ties the elements together© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42
  43. 43. Example 1: Classical POD Migration Q: Why migrate traditional Access/ Aggregation building block to FabricPath? A: • No STP – No STP sync, no topology changes, no blocked ports, no risk of loops • Simple configuration • Total flexibility in design and cabling • Enables organic bandwidth growth – Grow where and whenever needed with minimal impact© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43
  44. 44. Q: How to interconnect DC PODs in order to have VLANs anywhere? A: • Provide server/host connection to any edge port in network, regardless of physical location • Physical/rack/distribution pair location of host irrelevant with respect to IP subnet and Layer 2 adjacency with other hosts • Gateway placement options include GLBP, MHSRP and “leaf-attached” gateways, so far. POD 1 POD 2 POD 3 PODS 1-3 VLANs 100-199 VLANs 200-299 VLANs 300-399 VLANs 100-399© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44
  45. 45. Cisco FabricPath Spanning-Tree vPC FabricPath 16 SwitchesActive Paths Single Dual 16 WayPod Up to 10 Tbps Up to 20 Tbps Up to 160 TbpsBandwidth © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45
  46. 46. Pod Spine Tier 2 Leaf Tier 1 • • • • • • • •© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 46
  47. 47. Terminology Interface connected to another FabricPath device Sends/receives traffic with FabricPath header Does not run spanning tree Does not perform MAC learning! Exchanges topology info through L2 ISIS adjacency Forwarding based on ‘Switch ID Table’ FP Core Ports S10 S20 S30 S40 Spine Switch FabricPath (FP) S100 S200 S300 Leaf Switch 1/1 1/2 Classical Ethernet (CE) A B CE Edge Ports Interface connected to traditional network device Sends/receives traffic in standard 802.3 Ethernet frame format Participates in STP domain Forwarding based on MAC table© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 47
  48. 48. A New Data Plane • The association MAC address/Switch ID is maintained at the edge S10 S20 S30 S40 Switch ID space: S300: FabricPath Routing Routing Table decisions are A B S100 S300 made based on Switch IF the FabricPath routing table FabricPath (FP) S100 S200 S300 S100 L1, L2, L3, L4 MAC address 1/1 1/2 S300: CE MAC space: Address Table Switching based Classical Ethernet (CE) MAC IF on MAC address A B B 1/2 tables A S100 • Traffic is encapsulated across the Fabric© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 48
  49. 49. Conversational MAC Learning S10 S20 S30 S40 A B S100 M FabricPath Lookup B: Hit S100 S200 S300 Learn source A Lookup B: Miss Flood Lookup B: Miss Don’t learn S100: CE MAC 1/1 S200: CE MAC 1/2 S300: CE MAC Address Table Address Table Address Table MAC IF MAC IF MAC IF A B A 1/1 B 1/2 A S100 Classical Ethernet© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 49
  50. 50. Conversational MAC Learning S10 S20 S30 S40 S300: FabricPath Routing Table B A S300 S100 Lookup A: Hit Lookup A: Hit Switch IF Learn source B FabricPath Send to S100 S100 S200 S300 S100 L1, L2, L3, L4 S100: CE MAC 1/1 S200: CE MAC 1/2 S300: CE MAC Address Table Address Table Address Table MAC IF MAC IF MAC IF A B A 1/1 B 1/2 B S300 A S100 Classical Ethernet Conversational Learning© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 50
  51. 51. It’s a Routed Network • Describes shortest (best) paths to each Switch ID based on link metrics • Equal-cost paths supported between FabricPath switches S10 S20 S30 S40 FabricPath Routing Table on S100 Switch IF One ‘best’ path S10 L1 to S10 (via L1) S20 L2 S30 L3 S40 L4 S200 L1, L2, L3, L4 S100 S200 FabricPath S300 Four equal-cost S300 L1, L2, L3, L4 paths to S300© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 51
  52. 52. (1) Broadcast ARP Request Root for Root for Multi-destination Tree 1 Tree 2 Trees on Switch 10 S10 S20 S30 S40 Tree 4 IFFtag → 1 po100,po200,po300 po300 2 po100 DA→FF po100 po200 Ftag→1 SA→100.0.12 Multidestination DA→FF DMAC→FF Ftag→1 Trees on Switch 100 po10 po20 po30 po40 SMAC→A po10 po20 po30 po40 SA→100.0.12 Tree 3 IF S300 Payload DMAC→FFS100 Broadcast → 1 po10 S200 SMAC→A 2 po10,po20,po30,po40 Multidestination Payload Trees on Switch 300 5 6 FabricPath e1/13 Tree IF e2/29 Payload MAC Table on S100 DMAC→FF SMAC→A SMAC→A Ftag → 1 po10,po20,po30,po40 MAC IF/SID DMAC→FF 2 po40 A e1/13 (local) 2 Payload MAC A 1 MAC B FabricPath MAC Table on S200 MAC IF/SID Don’t learn MACs from Learn MACs of directly-connected frames flood devices unconditionally © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 52
  53. 53. MAC Address Table after the first ARP frame • S100: S100# sh mac address-table dynamic Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------ * 10 0000.0000.000a dynamic 0 F F Eth1/13 MAC A learned as local entry on e1/13 • S10 (and S20, S30, S40, S200): S10# sh mac address-table dynamic Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------ • S300: MAC A not S300# sh mac address-table dynamic learned on other switches Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 53 53
  54. 54. (2) Broadcast ARP Reply Root for Root for Multidestination Tree 1 Tree 2 S10 S20 S30 S40 Trees on Switch 10 10 Tree IF Ftag → 1 po100,po200,po300 po300 2 po100 DA→MC1 Ftag→1 po100 po200 SA→300.0.64 DA→MC1 Ftag→1 DMAC→A Multidestination SA→300.0.64 Trees on Switch 100 SMAC→B po10 po20 po30 po40 DMAC→A po10 po20 po30 Payload Tree 11 IF po40 S300 SMAC→B Ftag → 1 po10 S200 Multidestination Payload 2 po10,po20,po30,po40 Trees on Switch 300 9 Tree IF 7 FabricPath e1/13 MAC Table on S100 Payload Unknown → 1 po10,po20,po30,po40 e2/29 DMAC→A 2 po40 SMAC→B MAC IF/SID SMAC→B Payload A e1/13 (local) 12 DMAC→A MAC A FabricPath B 300.0.64 (remote) MAC Table on S300 MAC B 8 MAC MAC IF/SID IF/SID A → MISS B e2/29 (local) If DMAC is known, then learn remote MAC *MC1 = 01:0f:ff:c1:01:c0© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 54
  55. 55. MAC Address Table after the first ARP frame • S100: S100# sh mac address-table dynamic Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------ * 10 0000.0000.000a dynamic 90 F F Eth1/13 10 0000.0000.000b dynamic 60 F F 300.0.64 S100 learns MAC B as remote entry reached through S100# S300 • S300: S300# sh mac address-table dynamic Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------ * 10 0000.0000.000b dynamic 0 F F Eth2/29 MAC B learned as local entry on S300# e2/29© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 55 55
  56. 56. Unicast Data - Routed FabricPath Routing Table on S30 S10 S20 S30 S40 Switch IF S300 → S300 po300 16 po300 DA→300.0.64 DA→300.0.64 FabricPath Routing Ftag→1 Ftag→1 Table on S100 SA→100.0.12 SA→100.0.12 Switch IF DMAC→B DMAC→B S10 po10 SMAC→A SMAC→A po10 po20 po30 po40 S20 po20 po10 po20 po30 Payload Payload Hash po40 S30 po30 S300 S40 po40 S200 FabricPath Routing S100 po10, po20, Table on S300 S200 po30, po40 17 15 Switch IF po10, po20, e1/13 S300 PayloadS300 → po30, po40 e2/29 S300 → S300 Use LID (64) SMAC→A DMAC→B FabricPath DMAC→B MAC Table on S100 SMAC→A FabricPath MAC A MAC IF/SID Payload MAC Table on S300 MAC B A e1/13 (local) 14 13 MAC IF/SID 18 A S100.0.12 (remote) B→ B 300.0.64 (remote) If DMAC is known, then learn remote MAC B e2/29 (local)© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 56
  57. 57. Unicast forwarding • S100: S100# sh mac address-table dynamic Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------ * 10 0000.0000.000a dynamic 90 F F Eth1/13 10 0000.0000.000b dynamic 60 F F 300.0.64 S100# • S300: S300# sh mac address-table dynamic Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID ---------+-----------------+--------+---------+------+----+------------------ S100 learns MAC A 10 0000.0000.000a dynamic 30 F F 100.0.12 as remote entry reached through S100 * 10 0000.0000.000b dynamic 90 F F Eth2/29 S300#© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 57 57
  58. 58. Unicast ForwardingS100# sh fabricpath routeFabricPath Unicast Route Tablea/b/c denotes ftag/switch-id/subswitch-id[x/y] denotes [admin distance/metric] Topology (ftag),ftag 0 is local ftag Switch ID, Sub-subswitch-id 0 is default subswitch-id Switch ID AdministrativeFabricPath Unicast Route Table for Topology-Default distance, routing0/100/0, number of next-hops: 0 metric via ---- , [60/0], 0 day/s 04:43:51, local1/10/0, number of next-hops: 1 via Po10, [115/20], 0 day/s 02:24:02, isis_fabricpath-default Route age1/20/0, number of next-hops: 1 via Po20, [115/20], 0 day/s 04:43:25, isis_fabricpath-default1/30/0, number of next-hops: 1 via Po30, [115/20], 0 day/s 04:43:25, isis_fabricpath-default1/40/0, number of next-hops: 1 Client protocol via Po40, [115/20], 0 day/s 04:43:25, isis_fabricpath-default1/200/0, number of next-hops: 4 via Po10, [115/40], 0 day/s 02:24:02, isis_fabricpath-default via Po20, [115/40], 0 day/s 04:43:06, isis_fabricpath-default Next-hop via Po30, [115/40], 0 day/s 04:43:06, isis_fabricpath-default interface(s) FabricPath via Po40, [115/40], 0 day/s 04:43:06, isis_fabricpath-default1/300/0, number of next-hops: 4 S10 S20 S30 S40 via Po10, [115/40], 0 day/s 02:24:02, isis_fabricpath-default via Po20, [115/40], 0 day/s 04:43:25, isis_fabricpath-default via Po30, [115/40], 0 day/s 04:43:25, isis_fabricpath-default via Po40, [115/40], 0 day/s 04:43:25, isis_fabricpath-defaultS100# po10 po20 po30 po40 S100 S200 S300 A B C© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 58 58
  59. 59. S3 → FabricPath → CE L1 L2Introducing vPC+ F1 VPC+ F1 S1 F1 F1 S2 • Allows dual-homed connections from edge ports into FabricPath domain with active/active forwarding F1 F1 • Can also provide active/active HSRP • Configuration virtually identical to standard VPC Physical Host A • Supported on both N7K (F1/F2) and 5500 • VPC+ peer switches share a “virtual” FabricPath switch ID Logical S3 Host A→S4→L1,L2 • MAC addresses behind VPC+ port-channels appear as “connected” to the virtual switch, not the VPC+ physical L1 L2 peer switches F1 VPC+ F1 • Allows load-balancing within FabricPath domain toward the VPC+ virtual switch S1 F1 F1 S2 • VPC+ requires F1/F2 modules on N7K with FabricPath F1 F1 enabled in the VDC or FabricPath enabled on the Nexus 5500. S4 Virtual “Switch 4” becomes egress switch for Host A in FabricPath domain Host A© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 59
  60. 60. Driven by multiple vendors, including Cisco TRILL is now an IETF proposed standard FabricPath will provide a TRILL mode with a software upgrade (Cisco hardware is already TRILL capable) Cisco is pushing FabricPath specific enhancements to TRILL FabricPath TRILL Frame routing Yes Yes (ECMP, TTL, RPFC etc ) vPC+ Yes No FHRP active/active Yes No Multiple topologies Yes No Conversational learning Yes No Inter-switch links Point-to-point only Point-to-point OR shared http://datatracker.ietf.org/wg/trill/© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 60
  61. 61. HSRP HSRP L3 B L3 B Active/Active Active/Standby FabricPath G1 G2 TRILL G1 G2 Multipathing Multipathing a1 a2 a5 a6 a1 a2 a5 a6 VPC+ TRILL Active/Active Active/Standby A C A C • End-to-end multipathing (L2 edge, Fabric, L3 edge) provides resiliency and fast convergence© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 61
  62. 62. © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 62

×