The document discusses Cisco's FabricPath technology, focusing on its control plane and data plane architecture for Layer 2 networking in data centers. It explains elements like routing tables, multidestination trees, and MAC address handling, highlighting how FabricPath efficiently manages unicast and multicast traffic across a network by using switch IDs instead of traditional MAC addresses. Additionally, it covers configuration, load balancing, and the interaction of FabricPath with VLANs and STP in a network environment.

1. FABRICPATH

3. Why layer 2 in DC?

4. Typical DC Design

5. End to End L2

6. Limitation of Traditional L2

7. Cisco FabricPath Goal

8. Why FabricPath?

10. Control Plane
Key FabricPath control plane elements:
•Routing table – FabricPath IS-IS learns switch IDs (SIDs)
and builds routing table
•Multidestination trees – FabricPath IS-IS elects roots and
builds multidestination forwarding trees
•Mroute table – IGMP snooping learns group membership
at the edge, FabricPath IS-IS floods group-membership
LSPs (GM-LSPs) into the fabric

11. MAC-Based Routing ?
•NO!
•Routing information consists of Switch IDs
•Forwarding in fabric based on Switch IDs, not MAC addresses

12. FabricPath Routing Table
•Contains shortest path(s) to each SID, based on link metrics / path cost
•Equal-cost multipath (ECMP) supported on up to 16 next-hop interfaces

13. FabricPath Routing Table

14. ECMP Load-Sharing
•ECMP path chosen based on hash function
•Hash uses SIP/DIP + L4 + VLAN by default
•Use show fabricpath load-balance unicast to
determine ECMP path for a given packet

15. Mutlidestination TREE

16. MDT Root Selection
•FabricPath network elects a primary root switch for the first
multidestination tree in the topology
•Switch with highest priority value becomes root for the tree
–Tie break: root priority → highest system ID → highest SID
•Primary root determines roots of additional trees and
announces them in Router Capability TLV
–Roots spread among available switches to balance load

17. Root?Tree?..Is it STP??
•NO! – More like IP multicast routing
•Trees do NOT dictate forwarding path of
unicast frames, only multidestination frames
•Multiple trees allow load-sharing for any
multidestination frames
•Control plane state further constrains IP
multicast forwarding (based on mrouter and
receiver activity)

18. Data Plane
Key FabricPath data plane elements:
•MAC table – Hardware performs MAC lookups at CE/FabricPath edge only
•Switch table – Hardware performs destination SID lookups to forward
unicast frames to other switches
•Multidestination table – Hash function selects tree*, multidestination table
identifies on which interfaces to flood based on selected tree

19. FP Mac Table
•Edge switches perform MAC table lookups on ingress frames
•Lookup result identifies output interface or destination FabricPath switch

20. Encapsulation

21. SwitchID
•Every FabricPath switch automatically assigned a Switch ID
–Optionally, network administrator can manually configure SIDs
•FabricPath network automatically detects conflicting SIDs and
prevents data path initialization on violating switch
•Encoded in “Outer MAC addresses” of FabricPath MAC-in-MAC
frames
•Enables deterministic numbering schemes, e.g.:
–Spine switches assigned two-digit SIDs
–Leaf switches assigned three-digit SIDs
–VPC+ virtual SIDs assigned four-digit SIDs
–etc.

22. More about SID

23. F Tag
•Forwarding tag – Unique 10-bit number encoded in FabricPath header
•Overloaded field that identifies FabricPath topology or multidestination
tree
•For unicast packets, identifies which FabricPath IS-IS topology to use
•For multidestination packets (broadcast, multicast, unknown unicast),
identifies which multidestination tree to use
•FTAG: (Forwarding TAG) Used for multidestination traffic; carries the ID of
the tree chosen at the FabricPath ingress switch. DRAP is responsible to
keep FTAGs unique/consistent. For known unicast, FTAG carries topology
ID

24. Terminology• Classical Ethernet (CE)
– Regular Ethernet with regular flooding,
regular STP, etc.
• Leaf Switch
– Connects CE domain to FP domain
• Spine Switch
– FP backbone switch with all ports in
the FP domain only
• FP Core Ports
– Links on Leaf up to Spine, or Spine to
Spine
– i.e. the switchport mode fabricpath
links
• CE Edge Ports
– Links on Leaf connecting to regular
Classical Ethernet domain
– i.e. not the switchport mode
fabricpath links

25. Fabricpath Support

26. Configuration

27. More in Encapsulation
Outer SA

28. More in Encapsulation
Outer DA

29. Conflict Resolution

30. Fabricpath Tree

31. Forwarding Tree + VLAN

32. Root Election /Tree Construction

33. Other Encapsulation

34. Reverse Path Forwarding Check

35. Topologies
•Routing table & Trees (FTAGs) are per topology
•Switch ID is shared across all topologies
•FP interface may belong to several topologies
•N7K: up to 8 topologies support starting in 6.2
•N5K/N6K: 2 topologies supported since 5.2.1; main use is to permit separate L2 pods to use same
local vlan set
Config:---------------------------------

36. FabricPath Software Architecture &
Hardware tables
on the Supervisor Engine:
•FabricPath IS-IS routing protocol process that forms the core of the
FabricPath control plane
•DRAP Dynamic Resource Allocation Protocol, ensures network-wide
unique and consistent Switch IDs and FTAGs
–Resolves switch id conflicts
•IGMP Provides IGMP snooping support for building multicast forwarding
database
•M2RIB Multicast Layer 2 RIB, contains the multicast Layer 2 routing
information
•U2RIB Unicast Layer 2 RIB, containing the “best” unicast Layer 2 routing
information
•L2FM Layer 2 forwarding manager, controls MAC address table
•MFDM Multicast forwarding distribution manager, connects platform-
independent control-plane processes and platform-specific processes on
I/O modules
on the Linecards:
•U2FIB – Unicast Layer 2 FIB, managing the hardware unicast routing
table
•MTM – MAC Table Manager, managing the hardware MAC address table
•M2FIB – Multicast Layer 2 FIB, manages the hardware multicast routing table

38. FabricPath: Forwarding Tables
FabricPath uses 3 tables to forward frames
•MAC address table
VLAN, MAC Address, Port (local or remote), FTAG (for
non-unicast)
•Switch-ID table
remote switch-ID, local next-hop interfaces (up to 16)
•Multidestination tree table
Per Tree: remote switch-ID, local next-hop/RPF interface
Tree#1 (broadcast, unknown unicast, IP multicast)
Tree#2 (IP multicast)

39. Forwarding: unicast CEFP

40. Unicast: Known Destination MAC

41. Forwarding: broadcast/multicast
CEFP

42. Multidestination (broadcast, multicast,
unicast flood)

43. Forwarding: FP->FP or FP->CE
• Multicast lookups are
done using VLAN,
FTAG, and ODA (each
multicast mac
appears twice)
• SubSwitchID lookups
are omitted here
• Remember about
special LIDs (Sup,
Flood, …)
• FF frames are
forwarded out of CE
ports only when DA is
locally learned

44. Load-balancing
•Symmetric: idea is to make ab
and ba flows take same path
by sorting addresses, before
feeding them to hash
•Rotate: polarization avoidance;
hash result is rotated by
specified number of bytes.
Number is derived from unique
system MAC

45. Reducing impact of forwarding loops
•Transient loops might occur during
convergence (as with L3 routing)
•To contain impact of these loops FabricPath
uses TTL. Starting in 6.2(2), can set the initial
TTL via fabricpath [multicast | unicast] ttl
•For Multidestination Trees Reverse Path
Forwarding check performed on source
switch ID

46. MAC Address Learning
•Learning MAC addresses is not required in FabricPath
Core as switching is based on Switch ID
•FP Edge switches learn local MAC addresses (behind
edge ports) conventionally
•FP Edge devices learn remote addresses (behind Core-
facing ports) using conversational learning
o For packets arriving from FP, source MAC (not outer SA!) is
learned when destination MAC of the frame is already known
on any Edge port of this switch
•No learning from broadcasts (though existing entries
will be updated)
•Normal Learning from multicasts (example: HSRP
address)

47. Conversational MAC Address Learning

48. FabricPath Multicast Control Plane
•IGMP/IGMP snooping tracks connected hosts/routers
interest in receiving multicast
•ISIS distributes information from igmp snooping to
other FP nodes using GM-LSPs. Intermediate nodes
flood GM-LSPs
•A pruned subtree is created for each group (+flood,
OMF) per vlan per FTAG

49. STP & FabricPath
• No STP inside FP network
• BPDUs do not traverse FP network (dropped at FP edge, with the exception of TCNs)
• FP network pretends to be 1 switch from STP point of view: all FP edge switches send
BPDUs with the same Bridge ID c84c.75fa.60xx (xx is domain ID in hex, default 00)
• Before FP ports are up, switch will use its own Bridge ID (like STP without FP would
do)
• Ports inside FP cannot be blocked, FP edge switches will always want to have STP
designated role, if superior BPDU is received such port will be blocked as L2GW
inconsistent

50. STP, FabricPath & TCNs
• When CE STP domains are connected to multiple FP
switches STP TCN handling might be needed to
maintain accuracy of MAC address tables inside CE
• Example if link CE1-CE2 goes down, link CE2-CE3 will
become forwarding. Now to reach MAC B, switches
inside FP need to send traffic to S5 instead of S4…
• To achieve this, FP switches when receiving a TCN from
CE will propagate it to all FP switches in the network
(via ISIS)
• Each FP switch will flush all remote MAC addresses
learned from switches in the same STP domain as
domain originating the TCN
• In addition, if FP switch is also part of the same STP
domain, it will propagate TCN to the CE domain
• TCNs are not propagated to CE in domain 0 (default
domain)

51. Control Plane Protection
•Both N7K, N6K, and N5K recognize and protect FP ISIS traffic at COPP level
•COPP needs to be updated when deploying FabricPath; standard profiles are FP-aware as of 5.2(1)
•In case of complex CE-side STP topologies (with blocking ports), usual STP safeguards are recommended (Bridge
Assurance & Dispute / UDLD)
•On N7K-F1 cards: rate-limiters allow up to 4500 PPS worth of control plane FabricPath packets

53. VPC+: Why, What and How
•Goal: provide redundant, active-active L2 links to separate FP
switches with active-active HSRP
•Challenge : depending on the path the packet AB takes,
switch S3 will learn MAC A behind S1 or S2 (or MAC will be
moving)
•Solution: introduce Emulated Switch S100 to represent devices
behind VPCs: MAC A will appear behind S100 in S3 MAC
address table. HSRP MAC is advertised with emulated switch
as a source – taking advantage of VPC+ multipathing

54. VPC VPC+
•To enable VPC+ an Emulated Switch ID must be configured in
VPC domain on both peers (must be the same on both peers
and globally unique). ES represents ALL VPC+ channels of
the domain
•Peer-link and VPC+ ports must be fabric-path capable
•Peer-link is FP interface (no STP, only FP vlans are carried, VPC
check is no more).
VPC+ channels are CE
•VPC+ domain must be the root for CE STP, otherwise VPC+
channels will be blocked as L2GW inconsistent
•FP switches use same STP bridge ID so peer-switch is implicit

55. VPC+: Prevention of Duplicate Packets
•How is packet received from VPC+ and flooded on S1 prevented from
being flooded on S2 to same VPC+ again?
•N7K-F1 linecards:
Each VPC+ will have its own sub-switch ID. Mac addresses will be
learned behind <es_id>.<subsw_id>.<lid>, for example 100.11.65535
(emulated switch 100, sub-switch 11, LID 65535). S2 will recognize ES
+ SubSwitch tuple as its own port and will not flood the frame back to
VPC
•N7K-F2, N7K-F3 linecards & N5K, N6K:
By default same as above, as below with ‘fabricpath multicast load-
balance’
Each VPC+ peer will be forwarding only for 1 FTAG and traffic coming
from other peer will have different FTAG. For example flooded packet
coming from S1 will have FTAG1, but S2 will only flood FTAG2 packets
out of the VPC

56. VPC+ Failover
•VPC+ member link goes down
–Traffic diverted over Peer-Link
•Peer-Link goes down (but Peer-Keepalive up)
–Primary: No action
–Secondary: Bring down VPC+ channels
–Stop advertising reachability to Emulated Switch
•Dual active is much less likely than with normal VPC: if Peer-Link and Peer-
Keepalive go down, but peer is reachable via FP – secondary will not
become primary

57. FabricPath: What command comes
from where
MAC

151. Question ???