In the second of our two-part series on EVPN, Cumulus Networks Chief Scientist Dinesh Dutt dives into more technical details of network routing, EVPN use cases, and best practices for operationalizing EVPN in the data center. To view the recording of this webinar, visit http://go.cumulusnetworks.com/l/32472/2017-09-23/95t7xh

1. 1
Nov 1, 2017
Dinesh G Dutt, Vivek Venkataraman | Cumulus Networks
Part 2: Routing, Deployment Use Cases & Best Practices
Operationalizing EVPN in the DC

2. 2Cumulus Networks
EVPN Summary
Routing Models
Configuring Routing
Troubleshooting EVPN
Deployment Models and Recommendations
Agenda

3. 3Cumulus Networks
Key Takeaways
• EVPN supports routing as well as bridging
• Since L2 is no longer behind a single rack, multiple routing
models are possible
▪ VRF is supported in all models
• Pick right routing model based on use case
• FRR/Cumulus continues the simple configuration model
even with EVPN routing

4. 4Cumulus Networks
The Story So Far
• Designed to address the twin issues of:
▪ Multi-tenancy over an L3 network
▪ Allow disjointed L2 segments over an L3 network
• Dataplane:
▪ Supports multiple encapsulations: MPLS, VxLAN, NVGRE…
▪ VxLAN is the common choice within the data center
• Control plane is BGP
• Standards-based
▪ IETF original draft for MPLS: RFC 7432
▪ IETF draft for support with VxLAN: draft-ietf-bess-evpn-overlay

5. 5Cumulus Networks
Why Now ?
• Adoption of leaf-spine based IP fabrics to build data centers
• Rise of switching silicon that supports VxLAN routing
• Multi-vendor support for EVPN
▪ Lack of widespread adoption of controller-based overlays

6. 6Cumulus Networks
The Next Chapter
• EVPN is more than just multi-tenancy L2:
▪ supports routing, multicast handling, MAC/VM mobility etc.
• This part will cover these other aspects
• Plus, deployment models

7. 7Cumulus Networks
VXLAN Summary
• UDP/IP based encapsulation carrying L2 payloads
▪ RFC 7438
• Source port hashing allows fine-grained traffic spreading of
overlay traffic without requiring deep packet parsing
• 24-bit Virtual Network Identifier (VNI) identifies the VPN
• Tunnel ingress and egress are called VTEP (VXLAN Tunnel
Endpoint)

8. 8Cumulus Networks
• Protocol aspects based on BGP-based MPLS VPNs:
▪ Routes of a tenant kept separate with Route Distinguisher (RD)
▪ Routes contain Route Targets (RTs) to identify the VPN (L2 and/or L3 )
▪ Uses MP-BGP AFI L2VPN (25) SAFI EVPN (70)
▪ Various new BGP attributes (extended communities) - MAC Mobility,
Default Gateway, Encapsulation, Router MAC etc.
• Multiple pieces of information exchanged in EVPN:
▪ Another level of encoding, called route types, to identify the information
carried
EVPN Summary: Protocol

9. 9Cumulus Networks
EVPN Summary - key route types
Route
Type
Name Usage
RT-2 MAC/IP Advertisement Route Advertise MACs and/or MACIPs
RT-3 Inclusive Multicast Ethernet Tag
Route
Advertise VNI membership (primarily to prune
recipients of BUM traffic)
RT-5 IP Prefix Route Advertise routes to subnet prefixes
RT-1 Ethernet AutoDiscovery (A-D) Route For multi-homing, used to let remote VTEPs
know about connectivity to an Ethernet
Segment and VLANs reachable on it.
RT-4 Ethernet Segment Route For designated forwarder (DF) election for
BUM traffic handling in multi-homing scenarios.
RT-6 Selective Multicast Ethernet Tag
Route
To carry IGMP multicast group membership
information for a tenant using EVPN.
Route/VNI
info
Dual attach
support
Multicast
Info

10. 10Cumulus Networks
H11 -> H41: VXLAN Bridging (Packet Forwarding Level Set)
50.1.1.11 (VL 100) 50.1.1.41 (VL 100)
L1
L2 L3
L4
S1 S2
H11 H41
Unencapsulated
packet: DMAC is H41
Encapsulated packet:
Routed from
L1 ->
S1 Encapsulated
packet:
Routed
from
S1
->
L4
Unencapsulated
packet: DMAC is H41

11. 11Cumulus Networks
H11 -> H41: VXLAN Bridging (Packet Forwarding Level Set)
50.1.1.11 50.1.1.41
L1
L2 L3
L4
S1 S2
H11 H41
Unencapsulated
packet: DMAC is H41
Encapsulated packet:
Routed from
L1 ->
S1 Encapsulated
packet:
Routed
from
S1
->
L4
● Spines use only the VXLAN Header to route the packet
● Inner packet is carried practically unmodified
● L1 maps brown VLAN to brown VNI, L4 does the opposite
Unencapsulated
packet: DMAC is H42
DMAC: H41
SMAC: H11
DstP: H41
SrcIP: H11
DMAC: H41
SMAC: H11
DstP: H41
SrcIP: H11
DMAC: H41
SMAC: H11
DstP: H41
SrcIP: H11
DMAC: H41
SMAC: H11
DstP: H41
SrcIP: H11
Data
Data
Data
Data
DMAC: S1
SMAC: L1
DstIP: L4
SrcIP: L1
VNI: Brown
DMAC: S1
SMAC: L1
DstIP: L4
SrcIP: L1
VNI: Brown
VXLAN
Header

12. 12
Routing Models

13. 13Cumulus Networks
Regular Routing (H11 -> H12), No VxLAN: Case 1
50.1.1.11 (VLAN 100) 50.1.2.22 (VLAN 110)
L1
L2 L3
L4
S1 S2
H11 H12
1. H11 bridges to L1, default
gateway
2. L1:
a. routes to Blue subnet
b. L1 identifies Blue subnet
as being local
c. L1 does neighbor lookup
on H12
3. L1 bridges to H12
1
2

14. 14Cumulus Networks
Regular Routing (H11 -> H42), No VxLAN: Case 2
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1
L2 L3
L4
S1 S2
H11 H42
L1 and L4 have exchanged
subnet routes
1. H11 bridges to L1, default
gateway
2. L1 routes to next hop S1 (or
S2)
3. S1 (or S2) routes to L4
4. On L4, destination is on a
local subnet. L4 does
neighbor lookup and
bridges to H42
1
2 3
4

15. 15Cumulus Networks
Routing (H11 -> H42) with VxLAN
• Where is H11’s (and H42’s)
default router ?
• If L1 is the default router, what
happens after initial routing?
▪ Bridge to H42 (case 1) ?
▪ Routing at next hop L4 (case
2)?
• L1 and L4 always encapsulate
and decapsulate VXLAN
packet
• Spines only route encapsulated
packets
50.1.1.11 (VL 100) 50.1.2.42 (VL 110)
L1
L2 L3
L4
S1 S2
H11 H42

16. 16Cumulus Networks
The Rise of the Routing Models
• Where is H11’s (and H42’s) default router ?
▪ Specific per-VNI (or all VNI) gateways (Centralized routing)
▪ All ingress VTEPs are gateways (Distributed routing)
• So, what happens after the initial routing ?
▪ Bridge (case 1): Asymmetric Routing
▪ Route (case 2): Symmetric Routing

17. 17Cumulus Networks
Asymmetric vs Symmetric: Observations
• Asymmetric Model assumes all subnets are locally attached
• Symmetric model assumes all subnets are NOT locally
attached
• This choice plays a role in what’s suitable for what
deployment

18. 18Cumulus Networks
Asymmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1
L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1
a. DMAC = L1’s MAC, DIP = H42’s IP
1

19. 19Cumulus Networks
Asymmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1
L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1
a. DMAC = L1’s MAC, DIP = H42’s IP
2. L1:
a. routes the packet (in tenant’s VRF) to
blue subnet
b. identifies it is a local subnet and does a
neighbor lookup to get H42’s MAC*
c. Determines H42’s MAC is behind L4
d. L1 encapsulates the packet with VxLAN
header:
i. Payload: DMAC = H42’s MAC.
SMAC = L1’s MAC
ii. DIP = L4’s VTEP, SIP = L1’s
VTEP, VNI = Blue
iii. DMAC = S1’s MAC, SMAC = L1’s
MAC
1
2

20. 20Cumulus Networks
Asymmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1
L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1
a. DMAC = L1’s MAC, DIP = H42’s IP
2. L1:
a. routes the packet (in tenant’s VRF) to
blue subnet
b. identifies it is a local subnet and does a
neighbor lookup to get H42’s MAC*
c. Determines H42’s MAC is behind L4
d. L1 encapsulates the packet with VxLAN
header:
i. Payload: DMAC = H42’s MAC.
SMAC = L1’s MAC
ii. DIP = L4’s VTEP, SIP = L1’s
VTEP, VNI = Blue
iii. DMAC = S1’s MAC, SMAC = L1’s
MAC
3. S1 routes to L4
4. L4:
a. decapsulates the packet; VNI = Blue
b. Looks up DMAC of H42 on
corresponding VLAN, bridges out port
1
2 3
4

21. 21Cumulus Networks
Asymmetric Routing: Putting It All Together
1. Host sends packet to gateway router
2. Ingress VTEP (GW):
a. Routes
b. Bridges
c. Encapsulates
3. Spine switches (underlay) route
4. Egress VTEP:
a. Decapsulates
b. Bridges to end host
Packets are transported through the fabric in the final
destination VNI
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1
L2 L3
L4
S1 S2
H11 H42
1
2 3
4

22. 22Cumulus Networks
Symmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1
L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1
a. DMAC = L1’s MAC, DstIP = H42
1

23. 23Cumulus Networks
Symmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1
L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1
a. DMAC = L1’s MAC, DIP = H42’s IP
2. L1:
a. routes the packet (/32 route) to next hop
L4 - DMAC is L4’s Router MAC
b. L1 encapsulates the packet with VxLAN
header:
i. Payload: DMAC = L4’s Router
MAC. SMAC = L1’s MAC
ii. DIP = L4’s VTEP, SIP = L1’s
VTEP, VNI = ??
iii. DMAC = S1’s MAC, SMAC = L1’s
MAC
1
2

24. 24Cumulus Networks
Symmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1
L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1
a. DMAC = L1’s MAC, DIP = H42’s IP
2. L1:
a. routes the packet (/32 route) to next hop
L4 - DMAC is L4’s Router MAC
b. L1 encapsulates the packet with VxLAN
header:
i. Payload: DMAC = L4’s Router
MAC. SMAC = L1’s MAC
ii. DIP = L4’s VTEP, SIP = L1’s
VTEP, VNI = ??
iii. DMAC = S1’s MAC, SMAC = L1’s
MAC
Question: What VNI to use to transport the frame to
L4 ?
1. Brown (ingress VNI)
2. Blue (egress VNI, but how do I know ?)
3. Some other VNI
1
2

25. 25Cumulus Networks
Symmetric Routing H11 -> H42: Step by Step
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1
L2 L3
L4
S1 S2
H11 H42
1. H11 sends unencapsulated to L1
a. DMAC = L1’s MAC, DIP = H42’s IP
2. L1:
a. routes the packet (/32 route) to next hop
L4 - DMAC is L4’s Router MAC
b. L1 encapsulates the packet with VxLAN
header:
i. Payload: DMAC = L4’s Router
MAC. SMAC = L1’s MAC
ii. DIP = L4’s VTEP, SIP = L1’s
VTEP, VNI = per-tenant L3
transport VNI
iii. DMAC = S1’s MAC, SMAC = L1’s
MAC
3. S1 routes to L4
4. L4:
a. decapsulates the packet. VNI is the L3
VNI - identifies the VRF.
b. Looks up the DIP in VRF and routes to
local subnet
c. Looks up neighbor table for H42
d. Bridges to H42
1
2 3
4

26. 26Cumulus Networks
Symmetric Routing: Putting It All Together
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1
L2 L3
L4
S1 S2
H11 H42
1. Host sends packet to gateway router
2. Ingress VTEP (GW):
a. Routes to egress VTEP
b. Encapsulates
3. Spine switches (underlay) route
4. Egress VTEP:
a. Decapsulates
b. Routes to local subnet
c. Bridges to end host
Packets are transported through the fabric in a
per-tenant L3 VNI.
1
2 3
4

27. 27Cumulus Networks
• L3 VNI - configured and exchanged in control plane and carried in
routed packets.
▪ Additional configuration
▪ Corresponds to VRF associated with the L2 VNI(s)
▪ Different number space from L2 VNI
• Router MAC - Automatically derived (in Cumulus Linux/FRR) and
exchanged in the control plane. Used in routed packets to indicate
packet should be routed by egress VTEP (next hop)
Symmetric routing - L3 Transport VNI and Router MAC

28. 28Cumulus Networks
Asymmetric vs Symmetric: Packet Header View
50.1.1.11 50.1.2.42
L1
L2 L3
L4
S1 S2
H11 H42
DMAC: L1
SMAC: H11
DstP: H42
SrcIP: H11
Data
DMAC: H42
SMAC: L1
DstP: H42
SrcIP: H11
Data
DMAC: S1
SMAC: L1
DstIP: L4
SrcIP: L1
VNI: L3 VNI
DMAC: L4
SMAC: S1
DstIP: L4
SrcIP: L1
VNI: L3 VNI
DMAC: L4
SMAC: L1
DstP: H42
SrcIP: H11
Data
DMAC: L4
SMAC: L1
DstP: H42
SrcIP: H11
Data
DMAC: H42
SMAC: L1
DstP: H42
SrcIP: H11
Data
DMAC: S1
SMAC: L1
DstIP: L4
SrcIP: L1
VNI: Blue
DMAC: L4
SMAC: S1
DstIP: L4
SrcIP: L1
VNI: Blue
ASYMM SYMM ASYMMSYMM
DMAC: H42
SMAC: L1
DstP: H42
SrcIP: H11

29. 29Cumulus Networks
Asymmetric vs Symmetric: Forwarding Tables View
Asymmetric Symmetric
MAC Table All end stations End stations in all locally known
subnets plus remote VTEPs
Neighbor Table All end stations End stations in all locally known
subnets*
plus remote VTEPs
Route Table Locally attached prefixes All end stations plus local subnets
VNIs All VNIs in fabric Locally attached VNIs plus L3
transport VNIs
* - Needed for ARP Suppression

30. 30Cumulus Networks
Asymmetric vs Symmetric: Configuration View
Asymmetric Symmetric
Uniform configuration Yes No, since not all VNIs are
present everywhere
Need Orchestrator No Most likely, since VNIs and their
VLAN mappings will need to be
configured or torn down as
hosts/VMs move
Scaling Yes, breaking mobility up
into pods
Yes
Miscellaneous Need configuring and mapping
additional L3 transport VNIs

31. 31Cumulus Networks
Asymmetric vs Symmetric: Vendor Interop View
Aymmetric Symmetric
Arista X
Cisco X
Juniper X
Cumulus/FRR X X*
* - Supported in upcoming 3.5 release of Cumulus Linux

32. 32Cumulus Networks
Distributed Routing Model
• Since end station IP/MAC is spread throughout the network,
no specific first hop router can be first hop router
• Distributed model assumes every ToR switch is the first hop
router for all locally attached subnets
▪ Anycast IP and anycast MAC model
▪ Similar to VRR used today (VARP in Arista lingo)
• Most common deployed: when used to replace existing
VLAN-based access-agg-core networks with VXLAN-based
Clos networks

33. 33Cumulus Networks
Centralized Routing Model
• Encapsulated packets bridged to a designated first hop
router
• Packets are routed by this router
• Encapsulated packets bridged to final destination by this
router
• Primary switching silicon requirement:
▪ To decapsulate, route, bridge, encapsulate, route on underlay
header
• Most commonly deployed: when EVPN is used for
multi-tenancy in cloud-like environments

34. 34Cumulus Networks
Centralized Routing H11 -> H42: Sample Packet Flow
50.1.1.11 (VLAN 100) 50.1.2.42 (VLAN 110)
L1
L2 L3
L4
S1 S2
H11 H42
1. Host sends packet to gateway router (L2)
2. Ingress VTEP (GW):
a. Bridges to egress VTEP/router L2
b. Encapsulates packet & sends out
3. Spine switches (underlay) route
4. Gateway VTEP:
a. Decapsulates
b. Routes to local subnet
c. Bridges to end host
d. Encapsulates packet & sends out
5. Spine switches (underlay) route
6. Egress VTEP:
a. Decapsulates
b. Bridges to end host
Packets are transported through the fabric in the
bridge VNI.
1
2 3 4 5
6

35. 35Cumulus Networks
How do I talk to the outside world?
• Routing/Packet Forwarding was all based on /32 routes or neighbor
entries.
• To route to external networks, we need to route to prefixes.
▪ Enter EVPN type-5 routes (RT-5).
• RT-5 allows an IP prefix to be advertised, not just MAC+IP.
▪ For the common scenario of connecting to another subnet or external
network, the advertising VTEP is itself the next hop. RT-5 contains the
Router MAC of this VTEP.
▪ Specified in draft-ietf-bess-evpn-prefix-advertisement

36. 36Cumulus Networks
Control Plane Illustration for External Routing
L1
L2 L3
L4
S1 S2
● Per-tenant VRF peering
between Border Leaf BL1 and
WAN edge router R1
● R1 advertises prefixes relevant
to a tenant (e.g., default route)
on corresponding peering.
● BLs are typically deployed in
pairs for redundancy.
● For internal destinations to be
reachable, BLs will advertise
corresponding subnets to R1.
BL1
R1
WAN

37. 37Cumulus Networks
Control Plane Illustration for External Routing
L1
L2 L3
L4
S1 S2
● Per-tenant VRF peering
between Border Leaf BL1 and
WAN edge router R1
● R1 advertises prefixes relevant
to a tenant (e.g., default route)
on corresponding peering.
● BLs are typically deployed in
pairs for redundancy.
● For internal destinations to be
reachable, BLs will advertise
corresponding subnets to R1.
BL1
● BL1 installs routes in VRF
routing table
● BL1 exports these routes into
EVPN as RT-5.
● RT-5 advertised to other VTEPs
with L3 VNI of associated VRF.
Next hop is BL1..
R1
WAN

38. 38Cumulus Networks
Control Plane Illustration for External Routing
L1
L2 L3
L4
S1 S2
Receiving VTEPs (L1, …) install
routes into VRF routing table -
next hop is BL1, MAC is BL1’s
RMAC
● Per-tenant VRF peering
between Border Leaf BL1 and
WAN edge router R1
● R1 advertises prefixes relevant
to a tenant (e.g., default route)
on corresponding peering.
● BLs are typically deployed in
pairs for redundancy.
● For internal destinations to be
reachable, BLs will advertise
corresponding subnets to R1.
BL1
● BL1 installs routes in VRF
routing table
● BL1 exports these routes into
EVPN as RT-5.
● RT-5 advertised to other VTEPs
with L3 VNI of associated VRF.
Next hop is BL1..
R1
WAN
Note: This is for illustration
purposes, a real deployment is
likely to have NAT, FW etc.

39. 39Cumulus Networks
External Routing: Packet Flow
L1
L2 L3
L4
S1 S2
BL1
R1
WAN
50.1.1.11 (VL 100)
H11
201.11.1.45
H100
H11 sends the packet
for H100 to L1 - its
default GW

40. 40Cumulus Networks
External Routing: Packet Flow
L1
L2 L3
L4
S1 S2
BL1
R1
WAN
50.1.1.11 (VL 100)
H11
201.11.1.45
H100
H11 sends the packet
for H100 to L1 - its
default GW
● L1 matches packet against external
route (default or prefix advertised
by BL1) and routes to next hop
VTEP BL1.
● Packet routed over core with DMAC
= BL1’s Router MAC. VNI is the L3
VNI for this VRF.

41. 41Cumulus Networks
External Routing: Packet Flow
L1
L2 L3
L4
S1 S2
BL1 terminates the VxLAN tunnel and
routes the packet in the tenant VRF -
on to R1.
.
BL1
R1
WAN
50.1.1.11 (VL 100)
H11
201.11.1.45
H100
H11 sends the packet
for H100 to L1 - its
default GW
● L1 matches packet against external
route (default or prefix advertised
by BL1) and routes to next hop
VTEP BL1.
● Packet routed over core with DMAC
= BL1’s Router MAC. VNI is the L3
VNI for this VRF.

42. 42Cumulus Networks
Wait...Is RT-5 used only for external connectivity?
• No! RT-5 can also be used for inter-POD and inter-DC
communication.
• It really depends on how the subnets have been provisioned i.e.,
contained within a POD or DC.
• Cumulus Linux (and FRR) supports RT-5 for external and
inter-POD/inter-DC communication - available in upcoming release.

43. 43
Configuration Example

44. 44Cumulus Networks
Configuration Steps: Asymmetric Routing
• Provision VLANs and VNIs on all leaves
• Provision subnets for all relevant VLANs (SVIs)
• Map SVIs to appropriate VRF
• Configure eBGP between leaf and spine
• Activate and advertise information about all locally active
VNIs

45. 45Cumulus Networks
Configuration Steps: Symmetric Routing
• Provision relevant locally attached VLANs and VNIs on the
leaves (dynamic, non-uniform compared to asymmetric)
• Provision subnets for all locally attached VLANs (SVIs)
• Map SVIs to appropriate VRF
• For each VRF, provision an L3 VNI (additional step
compared to asymmetric)
• Configure eBGP between leaf and spine
• Activate and advertise information about all locally active
VNIs

46. 46Cumulus Networks
Asymmetric vs Symmetric Routing: FRR Configuration
# BGP/EVPN configuration
router bgp 65456
bgp router-id 110.0.0.1
neighbor fabric peer-group
neighbor fabric remote-as external
neighbor uplink-1 interface peer-group fabric
neighbor uplink-2 interface peer-group fabric
address-family ipv4 unicast
neighbor fabric activate
redistribute connected
address-family l2vpn evpn
neighbor fabric activate
advertise-all-vni
# L3 VNI configuration for tenant VRF
vrf vrf-tenant1
vni 104001
# BGP/EVPN configuration
router bgp 65456
bgp router-id 110.0.0.1
neighbor fabric peer-group
neighbor fabric remote-as external
neighbor uplink-1 interface peer-group fabric
neighbor uplink-2 interface peer-group fabric
address-family ipv4 unicast
neighbor fabric activate
redistribute connected
address-family l2vpn evpn
neighbor fabric activate
advertise-all-vni

47. 47Cumulus Networks
Centralized routing
• Fundamental configuration on
Gateway VTEP(s) is same as
in the distributed case.
• Gateway VTEP(s) need to be
configured to advertise their
own MACIP.
# BGP/EVPN configuration
router bgp 65456
bgp router-id 110.0.0.5
neighbor fabric peer-group
neighbor fabric remote-as external
neighbor uplink-1 interface peer-group
fabric
neighbor uplink-2 interface peer-group
fabric
address-family ipv4 unicast
neighbor fabric activate
redistribute connected
address-family l2vpn evpn
neighbor fabric activate
advertise-all-vni
advertise-default-gw

48. 48Cumulus Networks
Switching Silicon Support
• Considering only native, single-pass support for VxLAN
routing
• Cavium and Barefoot chipsets are supposed to have
support for all modes
T2 T2+ T3 Tomahawk
family
Spectrum/
A0
Spectrum
/A1
Spectrum2
Asymmetric - X X - X X X
Symmetric - X X - X X X
Centralized - X X - - X X

49. 49
What about multicast?

50. 50Cumulus Networks
The jury is still out
• Multicast routing in EVPN is still evolving.
• There are at least two key aspects:
▪ Optimized intra-subnet multicast (only to VTEPs behind which
interested receivers are present)
▪ Optimized inter-subnet multicast - local/distributed routing wherever
possible
• There are multiple proposals being discussed - including leveraging
MVPN and VPLS Multicast.
• Stay tuned for a future update on this topic!

51. 51Cumulus Networks
Summary
• EVPN supports routing besides bridging
• Due to the distributed nature of L2 in EVPN, several routing
models are possible
• Choose the right model based on deployment use case
▪ Choose wisely
• Cumulus/FRR supports (or will shortly support) all of the
routing models, including interop with other vendors
▪ Most other vendors support only a subset of these
• Cumulus/FRR provides a radically simplified config for
EVPN routing

52. 52
Thank you!
Visit us at cumulusnetworks.com or follow us @cumulusnetworks or
slack.cumulusnetworks.com
© 2017 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus
Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark
Linux®
is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.

53. 53Cumulus Networks
Flood multicast only where there are receivers
• Basic BUM handling will flood to all remote VTEPs.
• What if there is real multicast traffic (i.e., non link-local) for a tenant
- e.g., system monitoring, discovery, data dissemination using
Pub/Sub etc? Receivers may be dispersed in the DC.
▪ Enter Selective Multicast and EVPN Type-6 (RT-6) routes
▪ IGMP/MLD state on attachment circuits (ACs) conveyed using EVPN
RT-6 to remote VTEPs
▪ Receiving VTEPs generate proxy reports on their ACs
▪ Receiving VTEPs also build state indicating which VTEPs need traffic
for a particular (C-*, C-G) or (C-S, C-G)

54. 54Cumulus Networks
Distributed multicast routing
• When multicast sources and receivers are on different subnets, the
(inter-subnet) multicast routing can get hairy:
▪ Only one VTEP can be the Designated Router (DR) on a subnet, so
even for local receivers on a different subnet from source, packet may
have to be routed by a remote VTEP.
▪ A VTEP could get multiple copies of the packet, one for each subnet
• Distributed multicast routing is the solution. In one proposal:
▪ Each VTEP routes to local receivers on all subnets.
▪ Only one copy sent to remote VTEPs - on source subnet
▪ Receivers will receive on a special broadcast domain if they don’t have
the source subnet.

55. 55Cumulus Networks
Symmetric routing - sample topology
50.1.1.11 (VL 100)
50.1.4.44 (VL 130)
L1
L2 L3
L4
S1 S2
H11
50.1.2.12 (VL 110)
H12
50.1.3.43 (VL 120)
H43
VL 130
H44
● Tenant has 4 VLANs:
○ VL 100 - 50.1.1.x/24
○ VL 110 - 50.1.2.x/24
○ VL 120 - 50.1.3.x/24
○ VL 130 - 50.1.4.x/24
● VLANs 100 and 110 (and
corresponding SVIs) are
provisioned on {L1, L2} and
VLANs 120 and 130 on {L3, L4}
● Anycast GW IP is 50.1.x.250 -
provisioned on all Leafs.
● VLAN - VNI mappings:
○ VL 100 - VNI 10100
○ VL 110 - VNI 10110
○ VL 120 - VNI 10120
○ VL 130 - VNI 10130
● L3 VLAN and VNI for tenant are
4001 and 104001 respectively

56. 56Cumulus Networks
Symmetric routing - sample interface configuration (L1)
# VxLAN interfaces and VLAN-VNI mappings (local ones)
auto vxlan100
iface vxlan100
vxlan-id 10100
vxlan-local-tunnelip 110.0.0.1
bridge-learning off
bridge-access 100
bridge-arp-nd-suppress on
# VxLAN interface and VLAN-VNI mapping for the L3VNI
auto vxlan4001
iface vxlan4001
vxlan-id 104001
vxlan-local-tunnelip 110.0.0.1
bridge-learning off
bridge-access 4001
# Bridge with member ports (VLAN-aware)
auto br0
iface br0
bridge-vlan-aware yes
bridge-ports swp3 swp4 swp5 swp6 vxlan100 vxlan110
vxlan4001
bridge-stp on
bridge-vids 100 110 4001
# Tenant VRF configuration - if multiple tenants exist
auto vrf-tenant1
iface vrf-tenant1
vrf-table auto
# SVI with anycast GW IP (for local tenant subnets)
auto vlan100
iface vlan100
address 50.1.1.1/24
vlan-id 100
vlan-raw-device br0
address-virtual 00:00:5e:00:01:01 50.1.1.250/24
vrf vrf-tenant1
# L3 VLAN interface per tenant (for L3 VNI)
auto vlan4001
iface vlan4001
vlan-id 4001
vlan-raw-device br0
vrf vrf-tenant1

57. 57Cumulus Networks
Symmetric routing - sample FRR configuration (L1)
# L3 VNI configuration for tenant VRF
vrf vrf-tenant1
vni 104001
# BGP/EVPN configuration
router bgp 65456
bgp router-id 110.0.0.1
neighbor fabric peer-group
neighbor fabric remote-as external
neighbor uplink-1 interface peer-group fabric
neighbor uplink-2 interface peer-group fabric
address-family ipv4 unicast
neighbor fabric activate
redistribute connected
address-family l2vpn evpn
neighbor fabric activate
advertise-all-vni