brocade-bgp-evpn-based-dci-bvd

BROCADE VALIDATED DESIGN
BGP-EVPN-Based Data Center Interconnect
53-1004313-03
9 December 2016

© 2016, Brocade Communications Systems, Inc. All Rights Reserved.
Brocade, the B-wing symbol, and MyBrocade are registered trademarks of Brocade Communications Systems, Inc., in the United States and in other
countries. Other brands, product names, or service names mentioned of Brocade Communications Systems, Inc. are listed at www.brocade.com/en/legal/
brocade-Legal-intellectual-property/brocade-legal-trademarks.html. Other marks may belong to third parties.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment,
equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without
notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade
sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the
United States government.
The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to the accuracy of this
document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs that accompany it.
The product described by this document may contain open source software covered by the GNU General Public License or other open source license
agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, and
obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.
2 53-1004313-03

Contents
Preface...................................................................................................................................................................................................................................5
Brocade Validated Designs....................................................................................................................................................................................................................5
Purpose of the Document......................................................................................................................................................................................................................5
Target Audience..........................................................................................................................................................................................................................................5
About the Authors......................................................................................................................................................................................................................................5
Document History......................................................................................................................................................................................................................................6
About Brocade............................................................................................................................................................................................................................................ 6
Terminology..........................................................................................................................................................................................................................7
Introduction.......................................................................................................................................................................................................................... 9
Brocade EVPN-Based DCI Deployment Model—Overview................................................................................................................................... 11
EVPN DCI Deployment Model 1—BGP-EVPN-Based L2 and L3 Extension ...................................................................................................13
IP Fabric DC Component Review.................................................................................................................................................................................................... 13
BGP-EVPN-Based L2 and L3 Extension—Extending the BGP Control Plane.......................................................................................................... 15
EVPN DCI Deployment Model 2—BGP-EVPN-Based L2 Extension..................................................................................................................21
BGP-EVPN-Based L2 Extension—DCI Tier to DCI Tier.......................................................................................................................................................22
Validated Design—EVPN DCI with BGP-EVPN-Based L2 and L3 Extension...................................................................................................25
Topology Description............................................................................................................................................................................................................................ 26
Configuration Steps................................................................................................................................................................................................................................27
Configuration—Border Leaf to Spine Layer 3.............................................................................................................................................................................27
Configuration—Border Leaf to WAN Edge Layer 3................................................................................................................................................................. 30
Configuration—Border Leaf eBGP Multihop...............................................................................................................................................................................34
Example 1—DCI L2 Extension......................................................................................................................................................................................................... 35
Example 2—DCI L2 Extension and L2 Multitenancy..............................................................................................................................................................46
Multitenancy Across DCS Using Different TORs.............................................................................................................................................................46
Multitenancy Across DCS Using the Same TOR.............................................................................................................................................................54
Example 3—DCI VLAN Routing...................................................................................................................................................................................................... 64
Asymmetric Routing ....................................................................................................................................................................................................................64
Symmetric Routing.......................................................................................................................................................................................................................70
Example 4—Adding Services to Border-Leaf Nodes..............................................................................................................................................................78
Example 5—Extending a Tenant VRF to the WAN Edge.......................................................................................................................................................87
Example 6—Providing Internet Route Reachability for Tenant VRFs at TORs Through Public VRF at Border-Leaf............................... 100
Validated Design—EVPN DCI with BGP-EVPN-Based L2 Extension.............................................................................................................. 113
Topology Description......................................................................................................................................................................................................................... 113
Data Center Interconnect Tier......................................................................................................................................................................................................... 114
Configuration Steps.............................................................................................................................................................................................................................116
Configuration: DCI Tier to WAN Edge.........................................................................................................................................................................................123
Example 1—Layer 2 Extension......................................................................................................................................................................................................124
Example 1a: L2 Extension Between DCI Tier – DCI Tier..........................................................................................................................................124
Example 1b: L2 Extension Between DCI Tier - Leaf Node..................................................................................................................................... 140
Example 2—VLAN Routing.............................................................................................................................................................................................................154
Example 2a: Symmetric VLAN Routing Between Two Flexible Type Data Centers .................................................................................... 154
Example 2b: Symmetric VLAN Routing between flexible type and IP Fabric Data Centers......................................................................165
Example 3—Providing Internet Route Reachability for Tenant VRFs at DCI Tier Through Public VRF .........................................................176
53-1004313-03 3

Example 4a—DCI L2 Extension....................................................................................................................................................................................................187
Example 4b—VLAN Asymmetric Routing................................................................................................................................................................................210
Validated Design: EVPN DCI with BGP-EVPN-Based L2 and L3 Extension through Spines.................................................................... 225
Topology Description......................................................................................................................................................................................................................... 225
Hardware/Software Matrix ...............................................................................................................................................................................................................226
Configuration Steps.............................................................................................................................................................................................................................226
Configuration: Spine to Spine Layer 3........................................................................................................................................................................................226
Example 1—DCI L2 Extension...................................................................................................................................................................................................... 228
Example 2—DCI VLAN Routing ..................................................................................................................................................................................................241
Inter VLAN traffic........................................................................................................................................................................................................................241
Symmetric Routing ...................................................................................................................................................................................................................249
Design Considerations..................................................................................................................................................................................................261
Tunnel Scale...........................................................................................................................................................................................................................................261
Tunnels * VLANs..................................................................................................................................................................................................................................261
BGP-EVPN-Based L2 and L3 Extension Validated Scale............................................................................................................................................... 261
BGP-EVPN-Based L2 Extension Validated Scale................................................................................................................................................................262
References.......................................................................................................................................................................................................................263
4 53-1004313-03

Preface
• Brocade Validated Designs.............................................................................................................................................................................. 5
• Purpose of the Document.................................................................................................................................................................................5
• Target Audience.....................................................................................................................................................................................................5
• About the Authors................................................................................................................................................................................................ 5
• Document History................................................................................................................................................................................................6
• About Brocade.......................................................................................................................................................................................................6
Brocade Validated Designs
Brocade validated designs are reference architectures that are created and validated by Brocade engineers to address various customer
deployment scenarios and use cases. These validated designs provide a well-defined and standardized architecture for each deployment
scenario, and they incorporate a broad set of technologies and feature sets across Brocade's product range that address customer-
unique requirements. These designs are comprehensively validated end-to-end so that the design solutions and configurations can be
deployed more quickly, more reliably, and more predictably. Brocade validated designs are continuously validated using a test
automation framework to ensure that once a design has been validated, it remains validated on new software releases and products.
Purpose of the Document
This Brocade validated design provides guidance for implementing EVPN-based Data Center Interconnect (DCI) using Brocade
hardware and software. It details the Brocade reference architecture for two unique deployment models:
• BGP-EVPN-based L2 extension
• BGP-EVPN-based L2 and L3 extension
It should be noted that not all features, such as automation practices, zero-touch provisioning, and monitoring of the Brocade IP fabric,
are included in this document. Future versions of this document are planned to include these aspects of the Brocade IP fabric solution.
The design practices documented here follow the best-practice recommendations, but there are variations to the design that are
supported as well.
Target Audience
This document is written for Brocade system engineers and network architects who design, implement, and support data center
networks. This document is intended for experienced data center architects and network administrators/engineers. The reader must have
a good understanding of data center switching and routing features and Multi-Protocol BGP/MPLS VPN for understanding multitenancy
in VXLAN EVPN networks.
About the Authors
Jayanthi Jayaraman is a Senior Solution Architect on the IP SQA team at Brocade. She has over a decade of experience in the
networking industry, both as a software developer and as the lead engineer for verification of service provider, enterprise, and data center
solutions. At Brocade, her focus is on developing, building, and validating reference architectures and end-to-end customer network
solutions and on creating deployment guides.
53-1004313-03 5

Vedraj Cheela is a Software Test Engineer on the IP SQA team at Brocade. He has significant network solution experience and data
center virtualization expertise. At Brocade, his focus is on developing, building, and validating reference architectures and end-to-end
customer network solutions and on creating deployment guides.
Anuj Dewangan is the lead Technical Marketing Engineer (TME) for Brocade's data center switching products. He holds a CCIE in
Routing and Switching and has several years of experience in the networking industry with roles in software development, solution
validation, and technical marketing. At Brocade, his focus is creating reference architectures, working with customers and account teams
to address their challenges with data center networks, and creating product and solution collateral. He speaks at industry events and has
authored several white papers on data center networking.
The authors would like to acknowledge the following Brocadians for their technical guidance in developing this validated design:
• Mangesh Shingane: Principal Engineer
• Syed Hasan Raza Naqvi: Technical Leader
Document History
Date Part Number Description
April 2016 53-1004313-01 Initial version.
September 13, 2016 53-1004313-02 Multitenancy across DCS using the same TORs.
Providing Internet route reachability for tenant VRFs at TORs through public VRF at border leaf.
Layer 2 extension in EVPN DCI with BGP-EVPN-based L2 extension.
VLAN routing in EVPN DCI with BGP-EVPN-based L2 extension.
Providing Internet route reachability for tenant VRFs at DCI tier through public VRF.
Design considerations.
December 2016 53-1004313-03 EVPN DCI with BGP-EVPN-based L2 and L3 extension through Spines.
About Brocade
Brocade® (NASDAQ: BRCD) networking solutions help the world's leading organizations transition smoothly to a world where
applications and information reside anywhere. This vision is designed to deliver key business benefits such as unmatched simplicity,
non-stop networking, application optimization, and investment protection.
Innovative Ethernet and storage networking solutions for data center, campus, and service provider networks help reduce complexity and
cost while enabling virtualization and cloud computing to increase business agility.
To help ensure a complete solution, Brocade partners with world-class IT companies and provides comprehensive education, support,
and professional services offerings (www.brocade.com).
Document History
6 53-1004313-03

Terminology
Terms Description
Active-Active vLAG Active-Active Virtual Link Aggregation Group
AF Address Family
ARP Address Resolution Protocol
ASN Autonomous System Number
BGP Border Gateway Protocol
BL Border Leaf
BUM Broadcast, Unicast, and Multicast
CLI Command-Line Interface
DC Data Center
DCI Data Center Interconnect
DCS Data Center Site
eBGP Exterior Border Gateway Protocol
ECMP Equal Cost Multi-Path
EVPN Ethernet Virtual Private Network
iBGP Interior Border Gateway Protocol
IMR Inclusive Multicast Route
IP Internet Protocol
IRB Integrated Routing and Bridging
LAG Link Aggregation Group
LDP Label Distribution Protocol
LSP Label Switched Path
MAC Media Access Control
MH Multihop
BGP Border Gateway Protocol
MPLS Multi-Protocol Label Switching
ND Neighbor Discovery
Overlay GW Overlay Gateway
PoD Point of Delivery
RD Route Distinguisher
RT Route Target
ToR Top of Rack
UDP User Datagram Protocol
VCS Virtual Cluster Switching
vLAG Virtual Link Aggregation Group
vLAG pair Virtual Link Aggregation Group pair
VLAN Virtual Local Area Network
VM Virtual Machine
VNI Virtual Network Identifier
53-1004313-03 7

Terms Description
VPN Virtual Private Network
VRF Virtual Routing and Forwarding
VTEP VXLAN Tunnel End Point
VXLAN Virtual Extensible LAN
WAN Wide Area Network
8 53-1004313-03

Introduction
Based on the principles of the New IP, Brocade is building on the proven success of the VDX platform by expanding our cloud-optimized
network and network virtualization architectures to meet customer demand for higher levels of scale, agility, and operational efficiency.
This document describes network designs for interconnecting data center sites leveraging BGP EVPN. The intention of this Brocade
validated design document is to provide reference configurations and document the best practices for interconnecting data centers using
Brocade VDX switches with BGP EVPN.
This document describes the following architectures:
It is highly recommended to review the data center fabric architectures described in the Brocade Data Center Fabric Architectures white
paper for a detailed discussion on data center architectures for building data center sites.
53-1004313-03 9

10 53-1004313-03

Brocade EVPN-Based DCI Deployment
Model—Overview
Many data center deployments are required to span multiple geographically separated sites for availability and performance. Availability
in this context comes from site/tenant-level backup and redundancy to safeguard against infrastructure failures and provide increased
application and service reliability. The requirement of the data center network to span multiple sites may include extending the Layer 3
(and, in many cases, the Layer 2) reachability between sites.
There are two EVPN-based DCI deployment models detailed in this document:
Both of these models leverage VXLAN for efficient tunneling of traffic across a core network between data centers; they are differentiated
by how each data center "hands off" traffic to the core network, i.e., either at Layer 2 or at Layer 3.
The BGP-EVPN-based L2 and L3 model is targeted at interconnecting EVPN-based IP fabric data centers; whereas the EVPN-based
L2 model provides a more generic DCI solution with L2 VLAN extension from any type of data center deployment, e.g., Brocade VCS or
a BGP EVPN IP fabric. There are multiple design considerations for each; a brief summary is given in the following table, and details are
discussed further in the upcoming sections.
TABLE 1 BGP-EVPN-Based DCI Model Comparison
DCI Requirement BGP-EVPN-Based L2 & L3 Extension (DCI for
EVPN-Based IP Fabric DCs)
BGP-EVPN-Based L2 Extension (DCI for
Flexible DC Type)
Layer 2 extension Yes
(L2 EVPN control-plane learning between DCs)
Yes
(Data-plane learning between border leaf and
DCI tier)
Inter-VLAN routing Yes
(Asymmetric or symmetric routing with L3 VNI)
Yes
(Asymmetric or symmetric routing at DCI tier)
VLAN re-use
(VLAN re-use between tenants and leafs)
Yes
(VLAN-to-VNI mapping at DC leaf only)
Limited
(VLANs converge at the DCI tier and DC edge,
e.g. border leaf of EVPN-based IP fabric)
Control-plane segmentation
(Demarcation between DCs and the DCI)
Not segmented
The control plane is extended via the WAN and
is shared between data centers.
Segmented
The control plane is extended via the WAN
between DCI tiers, but is not shared between
data centers. Segmentation can be avoided with
control-plane extension from the DCI tier to the
leaf node.
VXLAN tunnel scale Tunnels span between leafs of EVPN-based IP
fabric DCs (tunnel scale: many to many)
Scale consideration at DC leaf (VXLAN tunnel
from a given leaf to each remote leaf sharing a
common VNI).
DCI tier to DCI tier tunnel scale (dependency
on number of remote sites)
Tunnels contained inside EVPN-based IP
fabric DC (tunnel scale: many to one, i.e. N
leafs to border leaf)
Scale consideration at border leaf (BL will have a
tunnel to each leaf requiring VLAN extension).
VLANs extended over VXLAN Scale consideration at leaf (many to many)
High VLAN and VNI scale between data centers
is possible with distributed scale across N leafs.
Scale consideration at DCI tier and BL nodes
for IP fabric EVPN DC (many to one)
53-1004313-03 11

12 53-1004313-03

EVPN DCI Deployment Model 1—BGP-
EVPN-Based L2 and L3 Extension
• IP Fabric DC Component Review...............................................................................................................................................................13
• BGP-EVPN-Based L2 and L3 Extension—Extending the BGP Control Plane.....................................................................15
The BGP-EVPN-based L2 and L3 extension DCI deployment model is designed for interconnecting BGP-EVPN-based (IP fabric) data
centers by extending the control plane between sites. With a common control plane, the interconnected sites behave as a single logical
data center, enabling efficient traffic patterns across an IP interconnect network. This deployment model provides the following key
benefits:
• Layer 2 extension and Layer 3 VRF host routing
• Dynamic VXLAN tunnel discovery and establishment
• BUM reduction with MAC address reachability exchange and ARP/ND suppression
• Conversational ARP/ND
• VM mobility support
• VXLAN head-end replication and single-pass efficient VXLAN routing
• Open standards and interoperability
IP Fabric DC Component Review
Before jumping into interconnecting EVPN-based IP fabric data centers, let's review the basic IP fabric design and its key elements. The
design is based on a leaf-spine multistage (e.g., 3 or 5) folded Clos topology that leverages Layer 3 ECMP between the leaf and spine
nodes. An example topology is shown in the following figure:
53-1004313-03 13

FIGURE 1 A 3-Stage Folded Clos Topology with Border Leaf
The basic IP fabric topology consists of the following elements:
• Spine layer
• Leaf layer
• Border leaf
Spine Layer
The role of the spine is to provide interconnectivity between the leafs. Network endpoints do not connect to the spines. Since most policy
is implemented at the leafs, the major role of the spine is to participate in the control-plane and data-plane operations for traffic
forwarding between leaf switches. Some differentiating characteristics of spine nodes include:
• Individual nodes have Layer 3 connectivity to each physical leaf switch.
• Spine nodes are not physically or logically connected to each other.
Leaf Layer
The role of the leaf switch is to provide connectivity to the endpoints in the network. These endpoints include compute servers and
storage devices, as well as other networking devices like routers and switches, load balancers, firewalls, and any other networking
endpoint—physical or virtual. For network efficiency, policy enforcement, including security, traffic path selection, Quality of Service (QoS)
marking, traffic policing, shaping, and traffic redirection, is implemented on leaf switches. Some differentiating characteristics at the leaf
layer are:
• Server VLANs terminate at the leaf switches (Layer 2 from devices to leaf).
• Leaf switches can be deployed individually as a top-of-rack device or as a pair providing switch-level redundancy with active-
active vLAG connections to servers.
IP Fabric DC Component Review
14 53-1004313-03

• L3 connectivity exists between the spine and leaf switches using L3 physical ports.
• Routing underlay: BGP is used to propagate IPv4/IPv6 routes with BGP neighbors formed from each leaf switch to each spine.
• Load balancing is achieved with L3 ECMP.
• Leaf-to-spine inter-switch point-to-point L3 links configured as “IP Unnumbered” or /31 subnets to conserve IP addresses
and optimize hardware resources (best practice).
Border Leaf
The role of the border leaf switches in the network is to provide external connectivity to the data center site and access to associated
access services like firewalls, load balancers, and edge VPN routers. The border leaf switches together with the edge racks housing these
common services form the edge services PoD. Since all North-South traffic will pass through the border leaf switches, it is important to
account for the bandwidth requirements for both:
• Internet traffic (external access to/from the data center)
• Data Center Interconnect (DCI) traffic (traffic passing between interconnected data centers, e.g., backup)
The ratio of the aggregate bandwidth of the uplinks connecting to the spines (two-tier case) or super-spines (three-tier case) to the
aggregate bandwidth of the uplink connecting to the WAN edge routers determines the over-subscription ratio for traffic exiting the data
center site.
The figure above shows the positioning and connectivity of a border leaf switch pair in a two-tier topology: that is, border leaf switches are
connected to all spines in the DC PoD (same as standard leaf switches) and also have external-facing connections to the WAN edge. In
the case of a three-tier fabric topology, border leaf switches would be connected to the super-spines (third tier), providing external
connectivity for N data center PoDs. The border-leaf to spine/super-spine connections are strictly Layer 3 with a BGP EVPN underlay;
whereas the border-leaf to WAN connections can be either Layer 2 or Layer 3 or a combination of both depending on the requirements
and the DCI deployment model. The upcoming sections will focus on the DCI deployment model details.
BGP-EVPN-Based L2 and L3 Extension—Extending
the BGP Control Plane
In the case of the BGP-EVPN-based L2 and L3 extension deployment model, WAN edge routers and IP/MPLS network are providing
only IP reachability and transport between data centers. Border leaf nodes in each data center learn how to reach each other from their
respective WAN edge routers. This is achieved by border leaf nodes advertising their peering address (e.g., local loopback) to the local
WAN edge router, which will, in turn, share the routing information with the remote WAN edge routers and remote border leaf nodes.
Once border leaf nodes have IP reachability to each other, an eBGP (multihop) session can be established. The following figure shows an
example of multihop eBGP peering between border leaf nodes in DC 1 and border leaf nodes in DC 2.
BGP-EVPN-Based L2 and L3 Extension—Extending the BGP Control Plane
53-1004313-03 15

FIGURE 2 DC1-DC2 eBGP Multihop Peering Between Border Leaf Nodes
To extend the EVPN control plane between sites, the EVPN address family is enabled for the eBGP multihop peering between border
leaf nodes. Continuing the example above and enabling the EVPN address family, the border leaf nodes will send EVPN routes from
their respective data centers to the remote data center; e.g., the border leaf from DC1 sends EVPN routes from DC1 to DC2 and vice
versa. The border leaf nodes then propagate the routes into their local data center. Depicted in the figure below, both data centers now
dynamically share routing information (i.e., IPv4 for VTEP reachability and EVPN) by extending the BGP control plane between sites.
While the control plane is extended over a separate network (e.g. third-party service provider), the internal EVPN routes are not
exchanged with the network providing the extension. That is, by establishing the BGP peering directly between border leaf nodes, BGP
update messages are exchanged directly between border leaf nodes only and not with the WAN edge routers. The WAN edge routers will
route the BGP control traffic only across the transport network. The route information exchanged between the border leaf and the WAN
edge is limited to the following:
• Border leaf router ID: For establishing eBGP MH neighborship.
• Leaf switch VTEP IPs: Forwarding across the IP core network is based on the destination VTEP IP.
16 53-1004313-03

FIGURE 3 DC1-DC2 DCI with Extended Control Plane (EVPN)
Behavior/Core Functions
Multiple data center sites sharing a common BGP-EVPN control plane will behave as a single logical IP fabric data center, enabling L2
VLAN extension and routing between VLANs between leaf switches at different sites.
Layer 2 Extension
Through the exchange of EVPN routes that contain VXLAN tunnel endpoint (VTEP) IP addresses between sites, leaf switches discover
remote leaf switch VTEP IP addresses (automatic VTEP discovery via EVPN Type 3 IMR). Leaf switches that share common VNIs will
dynamically create VXLAN tunnels between them using the discovered VTEP IP addresses.
The figure below shows an example of tunnel formation from a leaf switch in DC1 to a leaf switch in DC2, providing Layer 2 VLAN
extension. Layer 2 traffic is "tunneled" by encapsulating it into an IP User Datagram Protocol packet with an additional VXLAN header.
The outer IP source and destination for tunneled traffic are the source and destination VXLAN tunnel endpoint (VTEP) IP addresses in
this case, the leaf switches in DC1 and DC2 respectively. All transit routers forward the encapsulated Layer 2 traffic based on the outer IP
header, and only the router configured with the destination VTEP de-capsulates the packet to expose the inner Layer 2 frame . With the
Layer 3 handoff deployment model, the border leaf nodes provide both control-plane extension through the exchange of BGP EVPN
routes and data-plane forwarding for IP traffic (including tunneled VXLAN traffic) between sites.
The figure below shows an example of tunnel formation between leaf switches in DC1 and DC2 over an IP/MPLS network. After VXLAN
tunnel formation between leaf switches, Layer 2 traffic will be tunneled between sites. A 5-step example for L2 forwarding is shown:
53-1004313-03 17

FIGURE 4 DC1-to-DC2 VXLAN Tunnel Formation
1. A host in data center 1 forwards Ethernet traffic to its directly attached leaf switch (e.g. known unicast or BUM traffic).
2. Leaf switch in data center 1 receives the L2 traffic, learns or refreshes the source MAC address (data-plane learning), looks up
the destination MAC address, and encapsulates the received Ethernet frame into an IP User Datagram Protocol packet in which
the IP source/destination will be equal to the VTEP source/destination IP addresses plus a VXLAN header using automatic
(1:1) or user-defined VNI mapping and forwards the traffic to the spine layer.
NOTE
The source MAC address learned by the leaf switch is shared within the data center using BGP EVPN update
messages. The border leaf exchanges the BGP update messages with remote DC2 via its border leaf nodes (control-
plane learning). BGP updates are shared directly between border leaf nodes via eBGP multihop peering; i.e., updates
are not shared or leaked from the border leaf to the WAN edge.
3. The following nodes in this example all perform forwarding based on the destination VTEP IP address of the encapsulated
VXLAN packet (from Step 2):
• DC 1 spine
• DC 1 border leaf
• WAN edge and IP/MPLS core
• DC2 border leaf
• DC2 spine
4. The DC2 destination leaf switch receives traffic with a destination IP address matching the local VTEP address, performs
decapsulation revealing the inner Ethernet frame, and forwards traffic in the destination VLAN over the L2 interface toward the
target host.
5. The destination host in DC2 receives L2 traffic from its directly attached leaf switch.
Inter-VLAN Routing
18 53-1004313-03

The Layer 3 deployment model supports both asymmetric and symmetric routing for inter-VLAN traffic. Symmetric routing is the
recommended approach for the L3 DCI deployment model to simplify the configuration requirements and efficiently use the resources
at the leaf layer.
• Asymmetric routing—Both source and destination VLANs and associated gateways are configured on ingress and egress leaf
switches. Traffic is routed between the source and destination VLAN by the ingress leaf and is then tunneled to the remote leaf
using the VNI that is mapped to the destination VLAN. The inner L2 frame is then decapsulated at the remote egress leaf and
forwarded in the destination VLAN.
• Symmetric routing—The destination VLAN and gateway are not configured on the ingress leaf switch, and a common VNI is
used for extension between racks. Remote prefixes are advertised within the BGP EVPN address family as reachable with a
next hop equal to the remote leaf VTEP IP address and a VNI shared between to be used for tunneling traffic between local and
remote racks. When the same VLAN extension is not configured between two leaf nodes, leaf switches will not exchange
inclusive multicast routes (Type 3 routes). In the symmetric case, the leaf switches exchange L3 prefixes (Type 5 routes used for
automatic VTEP discovery), which will form a VXLAN tunnel between the leaf switches using a common VNI. A simplified
example is given in the following figure to illustrate the high-level steps for symmetric routing.
– The ingress leaf in DC 1 receives traffic from the VLAN 204 subnet and performs L3 lookup for the destination subnet
VLAN 201, and it resolves the NH to a remote VTEP in DC 2 with a VNI 2001 to be used for transport (associated with
the source and destination leaf switches).
– VXLAN-encapsulated traffic is routed between DC1 and DC2, and the destination IP address is the DC2 leaf VTEP.
– The egress leaf in DC2 de-capsulates the VXLAN traffic, performs L3 lookup for the destination subnet, and via the
destination VLAN GW, resolves the destination ARP and forwards traffic accordingly at L2 to the target host in VLAN 201.
FIGURE 5 VLAN Reuse Between Tenants in DC1 to DC2
The control-plane capability of the border leaf is unique within the IP fabric since it will not filter BGP-EVPN routes based on route
targets; i.e., it passes on advertisement of all routes to its neighbors similar to a spine node and also has the capability of initiating and
terminating tunnels as standard leaf switches. The specific configuration requirements are detailed in the validated design sections that
follow.
Shared Control Plane
53-1004313-03 19

One of the requirements for the BGP-EVPN-based L2 and L3 extension model is that the control plane is shared between sites. This
model is best suited for deployments where the operational/administrative control is centralized between sites to allow for effective
control and configuration, e.g., ensuring consistent VLAN-to-VNI mapping in local and remote data centers.
20 53-1004313-03

EVPN DCI Deployment Model 2—BGP-
EVPN-Based L2 Extension
• BGP-EVPN-Based L2 Extension—DCI Tier to DCI Tier................................................................................................................. 22
The BGP-EVPN-based L2 extension Data Center Interconnect (DCI) deployment model is designed to provide interconnection between
data centers at Layer 2 regardless of the data center type, e.g. VDX VCS, IP fabric. This deployment model introduces a new layer
referred to as the DCI tier, which connects to WAN edge routers at Layer 3 and the data center at Layer 2. The Layer 2 connection to the
data center refers to untagged or tagged Ethernet (802.3/802.1Q), and for redundancy, the recommended topology is to use two DCI
tier nodes connected as a vLAG pair. The following figure illustrates the DCI tier placement and connectivity to the WAN edge.
FIGURE 6 DCI Tier Network Placement
The DCI tier leverages the same underlying concepts described for the border leaf nodes in the Layer 3 handoff model; that is, DCI tier
nodes share a common extended control plane between sites. The differentiator is that ingress traffic to the border leaf is strictly Layer 2,
and the DCI tier nodes perform VTEP functions for inter-site traffic. The use of a shared EVPN control plane between DCI tiers enables
efficient forwarding across an IP interconnect network in addition to the following:
• Layer 2 extension and Layer 3 VRF host routing
• Dynamic VXLAN tunnel discovery and establishment (between DCI tier nodes)
• BUM reduction with MAC address reachability exchange and ARP/ND suppression
• Conversational ARP/ND
• VXLAN head-end replication and single-pass efficient VXLAN routing
• Open standards and interoperability
53-1004313-03 21

BGP-EVPN-Based L2 Extension—DCI Tier to DCI
Tier
As with the BGP-EVPN-based L2 and L3 extension deployment model, the L2 extension model uses eBGP multihop for peering with
EVPN for extending the control plane between sites and the WAN edge routers and an IP/MPLS network only to provide IP reachability.
The difference with the BGP-EVPN-based L2 model is that the peering between sites is between DCI tier nodes, and the interface to the
local data center is Layer 2. DCI tier nodes in each data center learn how to reach each other from their respective WAN edge routers.
This is achieved by DCI tier nodes advertising their peering address (e.g. local loopback) to the local WAN edge router, which, in turn, will
share the routing information with the remote WAN edge routers and remote DCI tier nodes. Once DCI tier nodes have IP reachability to
each other, an eBGP (multihop) session can be established. The following figure shows an example of multihop eBGP peering between
DCI tier nodes in DC 1 and border leaf nodes in DC 2.
FIGURE 7 DC1-DC2 DCI Tier eBGP Multihop Peering
Layer 2 Extension
Through the exchange of EVPN routes between DCI tier nodes, automatic VTEP discovery occurs (updates contain VTEP IP addresses).
DCI tier nodes sharing common VNIs will dynamically create VXLAN tunnels between them using the discovered VTEP IP addresses.
The following figure shows an example of tunnel formation between DCI tier nodes over an IP/MPLS network. After VXLAN tunnel
formation between DCI tier nodes, Layer 2 traffic will be tunneled between sites. A 5-step example for L2 forwarding is shown:
BGP-EVPN-Based L2 Extension—DCI Tier to DCI Tier
22 53-1004313-03

FIGURE 8 Packet Path Between Two Data Center Sites
1. Data center 1 forwards an Ethernet frame to its local DCI tier node (e.g. known unicast or BUM traffic).
2. The DCI tier at data center 1 receives the L2 traffic and learns or refreshes the source MAC address (data-plane learning), looks
up the destination MAC address, and encapsulates the received Ethernet frame into an IP UDP packet in which the IP source/
destination will be equal to the VTEP source/destination IP addresses plus a VXLAN header using automatic (1:1) or user-
defined VNI mapping, and it forwards the traffic to the spine layer and forwards the traffic to the WAN edge.
NOTE
The source MAC address learned by the DCI tier is shared using MP BGP-EVPN routes with remote DCI tier nodes
(control-plane learning), and BGP updates are shared directly between DCI tier nodes via eBGP multihop peering (i.e.,
updates are not shared or leaked to the WAN edge).
3. The WAN edge receives encapsulated traffic and performs forwarding based on the outer IP header (e.g., simple L3 forwarding
or MPLS depending on the core network).
4. Traffic received at the remote DCI tier with a destination IP address matching the local VTEP address is decapsulated, revealing
the inner Ethernet frame, and is forwarded in the destination VLAN over the L2 interface connected to data center 2.
5. Data center 2 receives the Ethernet traffic from the DCI tier as L2 traffic and adds or refreshes the source MAC address in its
table (data-plane learning).
In short, DCI tier nodes perform data-plane learning over their local L2 interfaces and control-plane learning over their L3 interfaces for
remote MAC addresses, ARP, etc. The result is efficient forwarding by DCI tier nodes because remote MAC addresses and ARPs are
shared with remote DCI tier nodes, reducing the amount of BUM traffic over the core network.
Inter-VLAN Routing
53-1004313-03 23

The BGP-EVPN-based L2 extension deployment model is targeted at extending Layer 2 VLANs across a shared core network. For
cases where routing between VLANs is required, there are two ways to achieve it- Asymmetric and Symmetric routing. In Asymmetric
routing the packet is routed first inside the DC, then switched to destination. Symmetric routing achieves routing at the gateway level
using common L3 VNI extension. When the individual data center control planes are separated by an L2 boundary (i.e., DC to DCI tier),
inter-VLAN traffic will be routed asymmetrically. The DCI tier nodes then receive and transport traffic in a single VLAN to the remote site.
When data center control planes are extended across without a boundary, Symmetric routing is efficient.
VLAN Scoping/Multitenancy
Traffic between sites is tunneled using VXLAN encapsulation as described in the example above, and the VLAN to VXLAN VNI mapping
is configured at the DCI tier nodes. For traffic between sites, the separation is based on the VNI. That is, inter-site forwarding with this
deployment model will only occur for cases where the VNI is common between local and remote DCI tier nodes. Therefore, different
tenants at different sites can use overlapping VLANs provided they use unique VNIs for transport across the core network.
Separated Control Plane/Administrative Control
For the BGP-EVPN-based L2 extension deployment model, the BGP EVPN control plane between DCI tier nodes is shared between
sites; whereas, the control plane between data centers is separated and relies on data-plane learning between the DC and DCI tier. The
separation between remote data centers may be well suited to operational scenarios where administrative control for data centers at
different sites is split between different teams in an enterprise, allowing each to manage and configure their data centers independently
and only hand off a VLAN to the DCI tier for extension; versus the BGP-EVPN-based L2 and L3 extension deployment model, where
administrative control over interconnected data centers would be better suited to a single team since the control plane between DCs is
shared.
24 53-1004313-03

Validated Design—EVPN DCI with BGP-
EVPN-Based L2 and L3 Extension
• Topology Description.......................................................................................................................................................................................26
• Configuration Steps..........................................................................................................................................................................................27
• Configuration—Border Leaf to Spine Layer 3....................................................................................................................................... 27
• Configuration—Border Leaf to WAN Edge Layer 3............................................................................................................................30
• Configuration—Border Leaf eBGP Multihop..........................................................................................................................................34
• Example 1—DCI L2 Extension....................................................................................................................................................................35
• Example 2—DCI L2 Extension and L2 Multitenancy........................................................................................................................ 46
• Example 3—DCI VLAN Routing.................................................................................................................................................................64
• Example 4—Adding Services to Border-Leaf Nodes.........................................................................................................................78
• Example 5—Extending a Tenant VRF to the WAN Edge................................................................................................................. 87
• Example 6—Providing Internet Route Reachability for Tenant VRFs at TORs Through Public VRF at
Border-Leaf.......................................................................................................................................................................................................100
This section provides step-by-step configuration examples for the BGP-EVPN-based L2 and L3 extension deployment model based
on a test topology, and it walks through common use cases with selected show commands to demonstrate intended functions.
FIGURE 9 Topology
53-1004313-03 25

Topology Description
• In Data Center Site1, all leaf nodes are connected to four spine nodes (with IPV4 addresses configured on interfaces in /31
subnet) using IPv4 eBGP adjacency with all four spine nodes in the same AS 64610. Leaf 1 and Leaf 2 are single, and Leaf3-
Leaf4, Leaf5-Leaf6, Border-Leaf1-Border-Leaf2 are a vLAG-pair. Leaf 1 is in AS 64630, Leaf 2 is in AS 64650, Leaf 3-Leaf
4 are in AS 64640, Leaf5-Leaf 6 are in AS 64670, and Border-Leaf1-Border-Leaf2 are in AS 64680. ECMP is achieved
using multipath eBGP.
• In Data Center Site2, all leaf nodes are connected to four spine nodes (with IPV4 addresses configured on interfaces in /31
subnet) using IPv4 iBGP adjacency with spine nodes being route-reflectors. All nodes are in AS 64620. Peer group is
configured to establish the BGP adjacency. ECMP is achieved using BGP add-path capability. Border-Leaf3-Border-Leaf4 are
a vLAG pair and all other leaf nodes Leaf 7, Leaf 8, Leaf 9, and Leaf 10 are single nodes.
• Leaf-spine adjacencies are activated under L2VPN EVPN address-family on all leaf and spine switches. Leaf-spine adjacencies
are configured with next-hop-unchanged to advertise routes from EVPN peers to other EVPN peers without changing the next
hop.
• In spine switches, retain route-target all is configured under EVPN address-family. This is to prevent stripping of RTs when
passing routes from one hop to another hop. Leaf switches compare RTs before installing routes with import RT under local
EVPN instance, RT advertised by each leaf node should be maintained before reflecting to other leaf nodes.
• VTEP addresses (Loopbacks) are advertised using the network command. Next-hop-recursion is used for next-hop-
reachability on Data Center Site2 since it is iBGP and redistribute connected is used on all spine nodes to provide next-hop
reachability.
• Border-Leaf1 and Border-Leaf2 are connected to WAN edge1 and WAN edge2 respectively using 4-10G port ECMP and
LAG. Border-Leaf3 and Border-Leaf4 are connected to WAN edge3 and WAN edge4 respectively using 4-10G port ECMP
and LAG. Border-Leaf node pairs are connected to respective WAN edge node pairs (with IPv4 address configured on LAG
interfaces in /31 subnet) using IPv4 eBGP adjacency with all WAN edge nodes in same AS 30614.
• L3 MPLS VPN adjacency is established between Site1 and Site2 WAN edge nodes.
• eBGP multihop session is established between Border-Leaf pair on Data Center Site1 and Border-Leaf pair on Data Center
Site2. Multihop BGP adjacency between Border-Leaf pairs on DCS1 and DCS2 are activated under EVPN address-family.
• Leaf to Host interfaces are configured as an active-active vLAG (aggregation of multiple physical links across multiple switches
from a single fabric forming single logical interface). The interfaces can be in access or trunk VLANs with IPV4, IPV6 any cast
address configured to allow VM mobility within or across DCS.
• Overlay gateway is configured in global context on all leaf nodes (applies to both nodes in case of two node vLAG pair) with
type of overlay to be used, respective VLAN VNI mapping, VTEP membership, switches membership, and VXLAN monitoring
like VLAN stats and SFLOW.
• EVPN instance is configured under rbridge mode for each leaf with RD, RT, VNIs to be extended.
• The retain route-target all command is configured on border-leaf nodes in order to advertise EVPN routes between data center
sites without stripping RT to form tunnel between leaf nodes from Site 1 and Site 2. In this approach, overlay gateway and
EVPN instance configurations can be avoided on border-leaf nodes. In case of symmetric routing, VRF configuration is not
needed on border-leaf nodes. Hence, border-leaf nodes will not form tunnels to other leaf nodes.
• If services have to be added on border-leaf nodes, they have to have tunnels. For this, needed VLAN-VNI mapping should be
added under overlay-gateway configuration with EVPN instance on border leaf nodes.
Hardware/Software Matrix
Role of Node Chassis Name (Possible Chassis Types) Minimum Software Version Required
Leaf BR-VDX6940-36Q
BR-VDX6940-144S
Network OS 7.0 and later
Topology Description
26 53-1004313-03

Role of Node Chassis Name (Possible Chassis Types) Minimum Software Version Required
BR-VDX6740T
BR-VDX6740
Border leaf BR-VDX6940-36Q
BR-VDX6940-144S
Spine BR-VDX8770-4/8
BR-VDX6940-36Q
BR-VDX6940-144S
DCI tier BR-VDX6940-36Q
BR-VDX6940-144S
BR-VDX6740
WAN edge MLXe-4/8/16/32 NetIron 5.9.00
Configuration Steps
The BGP-EVPN-based L2 and L3 extension deployment model is characterized by the following:
• Use of Layer 3 interfaces between the border leaf nodes and the WAN edge routers
• Layer 3 reachability between border leaf nodes in different data centers via the WAN edge routers (IP transport)
• BGP neighborship between border leaf nodes in different data centers (eBGP multihop) with EVPN AF enabled
Configuration—Border Leaf to Spine Layer 3
Interface configuration on Border-Leaf1 to Spine A (similar configuration is needed on interfaces to other spines from Border-Leaf1 and
on interfaces from Border-Leaf2 to spines).
BGP Configuration on Border-Leaf1 to spines (similar configuration is on Border-Leaf2 to spines with respective IP addresses).
53-1004313-03 27

Verify that eBGP neighborship is established on Border Leaf 1 to Spine A.
Verify that EVPN neighborship is established on Border Leaf 1 to Spine A.
28 53-1004313-03

Interface Configuration on Border-Leaf3 to Spine I (similar configuration is needed on interfaces to other spines from Border-Leaf3 and
on interfaces from Border-Leaf4 to Spines).
BGP Configuration on Border-Leaf3 to Spines (similar configuration is on Border-Leaf2 to Spines with respective IP addresses).
Verification can be done similar to verification of Border-Leaf1 to Spine.
53-1004313-03 29

Configuration—Border Leaf to WAN Edge Layer 3
• It is recommended to use two-node vLAG pair on border leaf to avoid a single point of failure.
• It is recommended to have full-mesh eBGP adjacency between each border-leaf node to WAN edge nodes.
• It is recommended to use LAG between the border-leaf node and the WAN edge.
• The network command is used to advertise the loopback interface to border-leaf nodes.
• eBGP adjacency between the border leaf and the WAN edge is not activated under EVPN address-family.
• Tracking the links between WAN edge and border leaf can help isolate a potential traffic black hole when all the links from one
BL to WAN Edge fail. This can be achieved by configuring link-tracking feature on the border leaf nodes.
TABLE 2 Border Leaf to WAN Edge Interface Connections
Connection Between Nodes Type of Connection
Border-Leaf1 to WAN Edge 1 4 - path ECMP (10G each)
Border-Leaf2 to WAN Edge 2 4 - 10G port LAG
Border-Leaf3 to WAN Edge 3 4 - path ECMP (10G each)
Border-Leaf4 to WAN Edge 4 4 - 10G port LAG
Interface Configuration on Border-Leaf1 to WAN Edge1 (a similar configuration is needed on other ECMP ports and on ECMP ports
used in Border-Leaf3).
Interface Configuration on Border Leaf 2 to WAN Edge2 (a similar configuration is needed on Border-Leaf4).
30 53-1004313-03

BGP configuration on Border-Leaf1 to WAN Edge1 ( a similar configuration is needed on other border-leaf nodes too).
Verification of eBGP neighborship from Border Leaf 1 to WAN Edge 1 can be done using the show ip bgp summary command as in
Border-Leaf1 to Spine verification.
WAN Edge 1 to Border Leaf 1 Interface Configuration (a similar configuration is needed on other interfaces connected to border-leaf
nodes and on other WAN edges too).
53-1004313-03 31

WAN Edge 1 to MPLS Core Interface Configuration (a similar configuration is needed on other WAN edges too. This interface will be
added into MPLS configuration).
BGP configuration on WAN Edge 1 (a similar configuration is needed on other WAN edges).
32 53-1004313-03

Verify that eBGP neighborship is established from Border Leaf 1 to WAN edge 1.
53-1004313-03 33

Configuration—Border Leaf eBGP Multihop
• Full-mesh eBGP multihop adjacency is established between each border-leaf node from a DCS to two border-leaf nodes in
another DCS and are activated under EVPN address-family. (No peering is established between local border-leaf nodes that are
a vLAG pair.)
• The retain route-target all command is configured under EVPN address-family of border-leaf nodes to advertise EVPN routes
to peers without stripping the route target. In this approach, the user doesn’t have to configure an overlay-gateway with VLAN-
to-VNI mapping, EVPN instance, or VRF configurations for symmetric routing on border-leaf nodes for exchanging EVPN
routes with other DCS.
• Border-leaf nodes will not form VXLAN tunnels to other leaf nodes.
Full-mesh eBGP multihop configuration to Border-Leaf1 (site 1) to Border-Leaf3(site 2) and Border-Leaf4 (site2) (a similar
configuration is needed on other border-leaf nodes).
Verification can be done similar to Border-Leaf1 BGP and BGP EVPN verification.
Configuration—Border Leaf eBGP Multihop
34 53-1004313-03

Example 1—DCI L2 Extension
In the following example for the BGP-EVPN-based L2 and L3 extension deployment model:
• VLAN 203 is extended between Data Center Site1 (Leaf 5 and 6) and Data Center Site2 (Leaf 8) with VNI 20003.
• Traffic between Leaf 5 and 6 and Leaf 8 is verified using traceroute from servers attached to the leaf nodes.
• Configuration examples of servers, interfaces, BGP, overlay-gateway, and EVPN instance on leaf nodes are discussed in the
section that follows.
• Refer to sections "Configuration: Border Leaf to Spine Layer3", "Border Leaf to WAN Edge Layer3", and "Border Leaf eBGP
Multihop for Border-Leaf and DCI Configurations".
FIGURE 10 DCI L2 Extension
Server Configurations
Server 1 Bond interface configuration for CentOS server attached to Leaf 5 and Leaf 6 of Data Center Site1.
Server 2 interface configuration for CentOS VM attached to Leaf 8 of Data Center Site2.
53-1004313-03 35

Leaf Node Configurations on DC1
Port-channel interface configuration to Server 1 on Leaf 5 and 6.
VLAN interface configuration on Leaf 5 and 6.
VE interface configuration to Server 1 on Leaf 5 and 6.
36 53-1004313-03

Loopback interface configuration (VTEP address) on Leaf 5 and 6.
BGP Configurations on Leaf 5.
53-1004313-03 37

BGP Configuration on Leaf 6.
38 53-1004313-03

Overlay gateway configuration on Leaf 5 and Leaf 6.
NOTE
VLAN-to-VNI mapping can be done manually or automatically. If automatic mapping is enabled, the VNI-to-VLAN mapping is
1:1, i.e. VLAN 201 maps to VNI 201.
EVPN instance configuration on Leaf 5 and Leaf 6 (per rbridge).
53-1004313-03 39

Port-channel verification on Leaf 5 and Leaf 6.
BGP and EVPN verification on Leaf 5 can be done similar to Border-Leaf1 to spine (Leaf 6 can be verified using the same command).
vLAG-pair verification on Leaf 5 (Leaf 6 can be verified using the same command).
Anycast gateway verification on Leaf 5 (the same command can be used to verify on other leaf nodes).
Leaf Node Configurations on DCS2
Interface configuration to Server 2 on Leaf 8.
40 53-1004313-03

VLAN interface configuration on Leaf 8.
VE interface configuration to Server 1 on Leaf 8.
Loopback interface configuration (VTEP address) on Leaf 8.
BGP Configuration on Leaf 8.
53-1004313-03 41

Overlay gateway configuration on Leaf 8 (under config mode).
EVPN instance configuration on Leaf 8 (under rbridge mode)
Inclusive multicast route verification on Leaf 5 for VNI associated with VLAN 203 (the same command can be used to verify on other
nodes).
42 53-1004313-03

Tunnel status verification on Leaf 5 (the same command can be used to verify on other nodes).
Individual tunnel verification on Leaf 5 (the same command can be used to verify on other nodes).
53-1004313-03 43

VLAN verification on Leaf 5 and Leaf 6 for 203 (the same command can be used to verify on other nodes).
Server 1 attached to Leaf 5 and 6 issuing ARP.
44 53-1004313-03

ARP verification on Leaf 5 (locally learnt ARP entries can be verified using this command).
ARP suppression verification on Leaf 8 (remote ARP learnt via BGP EVPN can be verified using show ip arp suppression-cache).
Server 2 attached to Leaf 8 issuing ARP.
Local and remote MAC verification on Leaf 5 for VLAN 203 ( the same command can be used to verify on other leaf nodes).
53-1004313-03 45

Server 1 to Server 2 traceroute traffic
DC1 Border-Leaf1 to DC2 Leaf 8 traceroute traffic
Example 2—DCI L2 Extension and L2 Multitenancy
Multitenancy Across DCS Using Different TORs
• VLAN 203 is extended between Data Center Site1 (Leaf 5 and 6) and Data Center Site2 (Leaf 8) using VNI 20003.
• Traffic between Leaf 5 and 6 and Leaf 8 is verified using traceroute from servers attached to the leaf nodes (using VNI 20003).
• Traffic between Leaf 3 and 4 and Leaf 7 is verified using traceroute from servers attached to the leaf nodes (using VNI 30003).
• Configuration examples of servers, interfaces, BGP, overlay-gateway, and EVPN instance on leaf nodes (Leaf 3 and 4, and Leaf
7) are discussed in the following section.
• Refer to Example 1 for the configurations and verifications of Servers, VLAN 203, corresponding port-channel in that VLAN,
Overlay-Gateway, EVPN instance, BGP and tunnel for Leaf 5 and 6 and Leaf 8.
• Refer to sections "Configuration: Border Leaf to Spine Layer 3", "Border Leaf to WAN edge Layer 3", and "Border Leaf eBGP
46 53-1004313-03

FIGURE 11 DCI L2 Extension and L2 Multitenancy
Server 5 interface configuration for Windows VM attached to Leaf 3 and Leaf 4 of Data Center Site1.
53-1004313-03 47

Server 6 interface configuration for Windows VM attached to Leaf 7 of Data Center Site2.
48 53-1004313-03

Loopback interface configuration (VTEP address) on Leaf 3 and 4.
Overlay gateway configuration on Leaf 3 and 4 (under config mode).
53-1004313-03 49

EVPN instance configuration on Leaf 3 and 4 (under rbridge mode).
50 53-1004313-03

Loopback interface configuration (VTEP address) on Leaf 7.
EVPN instance configuration on Leaf 7 (under rbridge mode).
Inclusive multicast route verification on Leaf 3 for VNI associated with VLAN 203 (the same command can be used to verify on Leaf 4
and Leaf 7).
53-1004313-03 51

and Leaf 8).
52 53-1004313-03

VLAN verification on Leaf 3 for 203 (the same command can be used to verify on other nodes).
Tunnel status verification on Leaf 3 (the same command can be used to verify on other nodes).
ARP verification on Leaf 3 (Locally learnt ARP entries can be verified using this command).
53-1004313-03 53

Server 5 to Server 6 traceroute traffic (connected to Leaf 3 and 4 and Leaf 7 with extended VNI 30003).
Server 1 to Server 2 traceroute traffic (connected to Leaf 5 and 6 and Leaf 8 with extended VNI 20003).
Multitenancy Across DCS Using the Same TOR
• VLAN 203 is extended between Data Center Site1 (Leaf 5 & 6) and Data Center Site2 (Leaf 8) using Virtual Fabric VLANs
7000 & 7001 and VNIs 7000 & 7001 respectively.
• VE interface 7000 & 7001 is configured under a VRF vrf3 & vrf4 on Leaf 5 & 6 (DCS1) and Leaf 8 (DCS2) respectively.
• VE interfaces 7000 and 7001 are configured with overlapping subnets (172.17.18.0)
• Traffic between Leaf 5 & 6 and Leaf 8 is verified using traceroute from servers attached to the leaf nodes (using VNI 7000 and
VNI 7001).
• Configuration examples of servers, interfaces, overlay-gateway, and EVPN instance on leaf nodes (Leaf 5 & 6, and Leaf 8) are
discussed in the below section.
• Refer Example 1 for the configurations and verifications of BGP and tunnel for Leaf 5 & 6 and Leaf 8.
54 53-1004313-03

• Refer to sections "Configuration: Border Leaf to Spine Layer 3", "Border Leaf to WAN edge Layer 3", and "Border Leaf eBGP
Server 1 Bond interface configuration for CentOS server attached to Leaf 5 and Leaf 6 of Data Center Site1
Server 2 interface configuration for CentOS VM attached to Leaf 8 of Data Center Site2
53-1004313-03 55

Server 4 interface configuration for Windows VM attached to Leaf 8 of Data Center Site2
56 53-1004313-03

Interface configuration to Server 1 on Leaf 5 & 6
Interface configuration to Server 3 on Leaf 5 & 6
VLAN interface configuration on Leaf 5 & 6
53-1004313-03 57

VE interface configuration to Server 1 on Leaf 5 & 6
VE interface configuration to Server 3 on Leaf 5 & 6
Overlay Gateway configuration on Leaf 5 & 6 (under config mode)
58 53-1004313-03

EVPN instance configuration on Leaf 5 & 6 (under rbridge mode)
Interface configuration to Server 2 on Leaf 8
Interface configuration to Server 4 on Leaf 8
VLAN interface configuration on Leaf 8
53-1004313-03 59

VE interface configuration to Server 2 on Leaf 8
VE interface configuration to Server 6 on Leaf 8
Overlay Gateway configuration on Leaf 8 (under config mode)
EVPN instance configuration on Leaf 8 (under rbridge mode)
60 53-1004313-03

Inclusive-multicast route verification on Leaf 5 for VNI associated with VLAN 7000 (same command can be used to verify on Leaf 6
and Leaf 8)
53-1004313-03 61

VLAN verification on Leaf 5 for 7000 (same command can be used to verify on other nodes and for VLAN 7001)
ARP verification on Leaf 5 in VRF vrf4 (Locally learnt ARP entries can be verified using this command)
ARP suppression verification on Leaf 5 for VLAN 7001 (Remote ARP learnt via BGP EVPN can be verified using show ip arp
suppression-cache)
62 53-1004313-03

ARP verification on Leaf 5 in VRF vrf3 (Locally learnt ARP entries can be verified using this command)
ARP suppression verification on Leaf 5 for VLAN 7000 (Remote ARP learnt via BGP EVPN can be verified using show ip arp
suppression-cache)
MAC verification on Leaf 5 for VLAN 7001
Server 1 to Server 2 traceroute traffic (Connected to Leaf 5 & 6 and Leaf 8 with extended VNI 7001)
Server 3 to Server 4 traceroute traffic (Connected to Leaf 5 & 6 and Leaf 8 with extended VNI 7000)
53-1004313-03 63

Example 3—DCI VLAN Routing
Asymmetric Routing
• VLAN 203 and 204 is extended between Data Center Site1 (Leaf 5 and 6) and Data Center Site2 (Leaf 8) with VNIs 20003
and 20004 respectively.
• Traffic between Leaf 5 and 6 (Site1) and Leaf 8 (Site2) is verified using traceroute from servers attached to the leaf nodes
(between VLAN 203 and 204).
• Configuration examples of servers, interfaces, overlay-gateway, and EVPN instance on leaf nodes are discussed in the below
section.
• Refer to Example 1 for the configurations and verifications of VLAN 203, corresponding port-channel in that VLAN, Overlay-
gateway, EVPN instance, BGP and tunnel.
• Refer to sections Configuration: "Border Leaf to Spine Layer3", "Border Leaf to WAN Edge Layer3", and Border Leaf eBGP
Multihop for Border-leaf and DCI Configurations".
FIGURE 12 DCI VLAN Routing—Asymmetric Routing
64 53-1004313-03

Server 1 bond interface configuration for CentOS server attached to Leaf 5 and Leaf 6 of Data Center Site1.
53-1004313-03 65

Overlay gateway configuration on Leaf 5 and Leaf 6 under config mode).
EVPN instance configuration on Leaf 5 and Leaf 6 (under rbridge mode).
Port-channel verification on Leaf 5 and Leaf 6.
66 53-1004313-03

53-1004313-03 67

and Leaf 8).
VLAN verification on Leaf 5 and Leaf 6 for 204 (the same command can be used to verify on Leaf 8).
Server 1 Attached to Leaf 5 and 6 issuing ARP.
68 53-1004313-03

Local and remote MAC verification on Leaf 5 for VLAN 204 (the same command can be used to verify on other leaf nodes).
53-1004313-03 69

Server 1 to Server 3 traceroute traffic.
Conversational ARP verification on Leaf 5 by sending continuous traffic between Server 1 and Server 3.
Symmetric Routing
• VLAN 203 is configured on Data center Site1 (Leaf 5 and 6) and VLAN 204 is configured on Data Center Site2 (Leaf 8).
• VRF vpn1 is configured on Leaf 5, Leaf 6 and Leaf8 with respective import, export route-targets and with common L3 VNI
2005.
• This VNI 2005 is not needed to add under EVPN instance. But VLAN-to-VNI mapping is needed under overlay-gateway
configuration.
• VE interfaces of 203, 204, and VNI VLAN VE will be configured under VRF vpn1.
• VRF address-family must be enabled under BGP configuration to advertise EVPN type 5 routes.
• Traffic between Leaf 5 and 6 and Leaf 8 is verified using traceroute from servers attached to the leaf nodes (between VLAN
203 and 204).
• Configuration examples of servers, interfaces, VRF, overlay-gateway, and EVPN instance on leaf nodes are discussed in the
following section.
• Refer to Example 1 for tunnel, port-channel, and VLAN verifications.
• Refer to sections Configuration: "Border Leaf to Spine Layer3", "Border Leaf to WAN Edge Layer 3", and "Border Leaf eBGP
70 53-1004313-03

FIGURE 13 DCI VLAN Routing—Symmetric Routing
53-1004313-03 71

VRF configuration on Leaf 5 and 6.
VRF VNI VLAN and VE interface configuration on Leaf 5 and 6.
72 53-1004313-03

VE interface configuration to server on Leaf 5 and 6.
Overlay gateway configuration on Leaf 5 and Leaf 6 (under config mode).
BGP configuration on Leaf 5 (similar configuration is needed on Leaf 6) .
53-1004313-03 73

VRF configuration on Leaf 8.
VRF VNI VLAN and VE interface configuration on Leaf 8.
74 53-1004313-03

VE interface configuration toward server on Leaf 8.
BGP configuration on Leaf 8.
53-1004313-03 75

VRF VNI verification on Leaf 5 (the same command can be used to verify on other leaf nodes).
L3 prefixes (type 5 routes) verification on Leaf 5 (the same command can be used to verify on other leaf nodes).
76 53-1004313-03

VRF route verification on Leaf 5 (the same command can be used to verify on other leaf nodes).
Server 1 attached to Leaf 5 and 6 issuing ARP.
53-1004313-03 77

Example 4—Adding Services to Border-Leaf Nodes
In some situations, border-leaf nodes can hold some services and need to extend some VLANs with other leaf nodes or to perform
routing in VRF. To achieve this, overlay-gateway and EVPN instance or VRF with common L3 VNI configuration will be added on
border-leaf nodes for some VLANs. Retain route-target all will also be configured under EVPN address-family on border-leaf nodes to
pass the EVPN routes between DC sites for the VNIs that are not configured on border-leaf nodes.
• VLAN 203 and 204 is extended between Data Center Site1 (Leaf 5 and 6) and Data Center Site2 (Leaf 8) with VNIs 20003
and 20004 respectively.
• In addition to retain route-target all under BGP on border-leaf nodes in DCS1 and DCS2, VTEP address, overlay-gateway and
EPVN-instance with VNI mapping 20003 and 20004 will be configured. Border-leaf nodes will form tunnels with leaf nodes
that extended the VNIs 20003 and 20004. Also, they will forward the EVPN routes for VNI 30003.
• Traffic between Leaf 5 and 6 and Leaf 8 is verified using traceroute from servers attached to the leaf nodes (for VNI 20003,
20004).
• Traffic between Leaf 3 and 4 and Leaf 7 is verified using traceroute from servers attached to the leaf nodes (for VNI 30003).
• Configuration examples of servers, interfaces, overlay-gateway, and EVPN instance on border-leaf nodes are discussed in the
following section.
• Refer to Use Case 1, Use Case 2, and Use Case 3 for the configurations and verifications of VLAN 203, VLAN 204,
corresponding port-channel in that VLAN, Overlay-Gateway, EVPN instance, BGP and tunnel.
• Refer to sections "Configuration—Border Leaf to Spine Layer 3", "Border Leaf to WAN Edge Layer 3", and "Border Leaf eBGP
78 53-1004313-03

FIGURE 14 Adding Services to Border Leaf Nodes
53-1004313-03 79

Server 5 interface configuration for Windows VM attached to Leaf 3 and Leaf 4 of Data Center Site1.
Server 6 interface configuration for Windows VM attached to Leaf 7 of Data Center Site2.
Border-leaf Node Configurations on DCS1
VLAN interface configuration on Border-Leaf1 and Border-Leaf2 (similar configuration is needed on Border-Leaf3 and Border-Leaf4).
80 53-1004313-03

VE interface configuration on Border-Leaf1 and Border-Leaf2 (similar configuration is needed on Border-Leaf3 and Border-Leaf4).
Loopback interface configuration (VTEP address) on Border-Leaf1 and Border-Leaf2.
Unique loopback interface on Border-Leaf1 and Border-Leaf2 to establish eBGP multihop session with Border-Leaf3 and Border-
Leaf4.
53-1004313-03 81

Loopback interface configuration (VTEP address) on Border-Leaf3 and Border-Leaf4.
Unique loopback interface on Border-Leaf3 and Border-Leaf4 to establish eBGP multihop session with Border-Leaf1 and Border-
Leaf2.
BGP configuration on Border-Leaf1.
82 53-1004313-03

Overlay gateway configuration on Border-Leaf1 and Border-Leaf2 (under config mode, similar configuration is needed on Border-Leaf3
and Border-Leaf4).
EVPN instance configuration on Leaf 3 and 4 (under rbridge mode, similar configuration is needed on Border-Leaf3 and Border-Leaf4).
Inclusive multicast route verification on Border-Leaf1 for VNI 20003 (the same command can be used to verify on other border-leaf
nodes).
53-1004313-03 83

Inclusive multicast route verification on Border-Leaf1 for VNI 30003 (the same command can be used to verify on other border-leaf
nodes).
84 53-1004313-03

Tunnel status verification on Border-Leaf1 (the same command can be used to verify on other border-leaf nodes).
53-1004313-03 85

ARP verification on Leaf 5 after issuing ARP ping from all servers (locally learnt ARP entries can be verified using this command).
86 53-1004313-03

Example 5—Extending a Tenant VRF to the WAN
Edge
• VLAN 204 is configured in Data center Site1 (Leaf 5 and 6) and Data Center Site2 (Leaf 8).
• VRF vpn1 is configured on Leaf 5, Leaf 6, Border-Leaf1, Border-Leaf2, and Leaf8 with respective import, export route-targets
and with common L3 VNI 2005.
• This VNI 2005 is not needed to add under EVPN instance on all nodes. But VLAN-to-VNI mapping is needed under overlay-
gateway configuration.
• VE interfaces of 204 and VNI VLAN VE (2005) will be configured under VRF.
• VRF address-family must be enabled under BGP configuration to advertise Type 5 routes.
• Since VE 204 on both DCS1 and DCS2 is in same subnet, VNI mapping for VLAN 204 can be added under overlay-gateway
and respective VNI can be added under EVPN instance on Site1 Leaf 5 and 6 and Site2 Leaf 8. This makes the inclusive-
multicast route exchange between leaf nodes (Leaf 5 and 6 and Leaf 8).
• To extend the VRF to WAN edge, one of the connections between Border-Leaf1 and WAN Edge1 is configured in two different
VLANs (800, 802) with VE 800 in default VRF (to support DCI interconnect and to have multihop eBGP adjacency to border-
leaf nodes on DCS2) and VE 802 in VRF vpn1. Similar changes made on WAN edge 1.
• A link from Border-leaf2 to WAN edge1 is enabled in VLAN 801 with VE 801 in VRF vpn1 on both sides.
• eBGP adjacency between border-leaf nodes (BL1 and BL2) and WAN edge1 is established under VRF vpn1 address-family
with redistribute connected.
Example 5—Extending a Tenant VRF to the WAN Edge
53-1004313-03 87

• This makes route exchange between Leaf 5 and 6, Leaf 8 in VLAN 204, and WAN-edge1 in VLAN 801 and 802. (Type 5
route-exchange)
FIGURE 15 Extending Tenant VRF to WAN Edge
88 53-1004313-03

VRF configuration on Leaf 5 and 6.
53-1004313-03 89

VRF VNI VLAN and VE interface configuration on Leaf 5 and 6.
VE interface configuration to server 4 on Leaf 5 and 6.
90 53-1004313-03

Overlay gateway configuration on Leaf 5 and Leaf 6 (under config mode).
EVPN instance configuration on Leaf 5 and Leaf 6 (under rbridge mode).
BGP has to be configured similar to Example 5 on Leaf 5 and 6.
53-1004313-03 91

VRF configuration on Leaf 8.
VRF VNI VLAN and VE interface configuration on Leaf 8.
VE interface configuration toward server on Leaf 8.
92 53-1004313-03

Leaf Node Configurations on Border-Leaf1 and Border-Leaf2.
Interface configuration on Border-Leaf1 to WAN edge1.
Interface configuration on Border-Leaf2 to WAN edge1.
VLAN interface configuration on Border-Leaf1 and Border-Leaf2.
VRF configuration on Border-Leaf1 and Border-Leaf2.
53-1004313-03 93

VRF VNI VLAN and VE interface configuration on Border-Leaf1 and Border-Leaf2.
VE interface configuration to WAN Edge1 on Border-Leaf1 (Similar configuration is needed on WAN edg1).
94 53-1004313-03

VE interface configuration to WAN Edge1 on Border-Leaf2 (Similar configuration is needed on WAN edg1).
Overlay gateway configuration on Border-Leaf1 and Border-Leaf2 (under config mode).
EVPN instance configuration on Border-Leaf1 and Border-Leaf2 (under rbridge mode).
53-1004313-03 95

BGP configuration on WAN Edge 1 (similar configuration is needed on other WAN edges).
96 53-1004313-03

Inclusive multicast route verification on Border-Leaf1 for VNI 20004 (since VNI for VLAN 204 defined in VRF vpn1 is extended, both
Leaf 5 and 6, and Leaf 8 advertises the IMR route).
53-1004313-03 97

Local and remote L3 prefixes (Type 5 routes) in Border-Leaf1,
98 53-1004313-03

VRF route verification on Border-Leaf1.
VRF route verification on Leaf 5 (the same command can be used to verify on Leaf 6 and Leaf 8).
53-1004313-03 99

VRF route verification on WAN Edge1.
Server 3 to WAN edge1 ping traffic.
Example 6—Providing Internet Route Reachability for
Tenant VRFs at TORs Through Public VRF at Border-
Leaf
• Virtual Fabric VLAN 7000 is configured with customer-tag 203 in Data center Site1 (Leaf5 & 6) and VLAN 530 is configured
Data Center Site2 (Leaf 8).
• VRF tenant-vrf is configured on Leaf 5 & 6 (DCS1) and on Leaf 8 (DCS2) with a common L3 VNI 5060 with respective import
and export route-targets.
• There is no need to add the L3 VNI 5060 under EVPN instance but VLAN VNI mapping is needed under overlay-gateway
configuration.
• VE interfaces 7000 on Leaf 5&6 (DCS1) and 203 on Leaf 8 (DCS2) will be configured under VRF tenant-vrf.
• VE interface corresponding to L3 VNI 5060 must be enabled under tenant-vrf on Leaf 5 & 6 (DCS1) and on Leaf 8 (DCS2).
Example 6—Providing Internet Route Reachability for Tenant VRFs at TORs Through Public VRF at Border-Leaf
100 53-1004313-03

• VRF tenant-vrf address-family must be enabled under BGP configuration to advertise Type 5 routes.
• ISP is connected to BL1 & BL2 on DCS1 and BL3 & BL4 on DCS2.
• To extend the tenant-vrf from Leaf 5&6 and Leaf 8 to ISP one of the connections between Border-Leaf1 and WAN Edge1 is
configured in two different VLANs (800, 802) with VE 800 in default VRF (to support DCI interconnect and to have multi-hop
eBGP adjacency to border-leaf nodes on DCS2) and VE 802 in VRF public-vrf. Similar changes made on WAN edge 1.
• Similar configurations are needed between Border-Leaf2 and WAN Edge2 in VLANs 30 (default-vrf) and 31 (public-vrf),
Border-Leaf3 and WAN Edge3 in VLANs 850 (Default VRF) and 851 (public-vrf), and Border-Leaf4 and WAN Edge4 in
VLANs 40 (Default VRF) and 41 (public-vrf).
• EBGP adjacency between Border-Leaf nodes (BL1 & BL2, BL3 & BL4) with respective WAN Edges (WE1 & WE2, WE3 &
WE4) is established using VE interfaces mentioned above in respective VRFs.
• WAN edge is configured to advertise only default routes to respective Border-Leaf nodes in public-vrf.
• Route leak is configured between BL and individual leaf nodes in respective DCS with import and export route-targets under
VNIs.
• VLAN to VNI mapping for the VNI added under EVPN instance for route leak must be added under overlay-gateway on Leaf 5
& 6 and BL1 & BL2. Similar configuration is needed on Leaf 8 and BL3 and BL4.
• VE interface corresponding to the VNI for route-leak must be enabled on both Leaf 5 & 6 and on Border-Leaf1 and Border-
Leaf2. Similar configuration is needed on Leaf 8 and BL3 and BL4.
• This makes route exchange between tenant-vrfs of Leaf 5 & 6 with public-vrfs of BL1 & BL2, tenant-vrf of Leaf 8 with public-
vrfs of BL3 & BL4.
• Traffic to internet route is verified from a server attached to Leaf 5 & 6 with ping.
NOTE
In this example, private IPV4 addresses are used from TOR to ISP. This can be modified to public IPV4 addresses with NAT
placed either at WAN edge or at ISP.
53-1004313-03 101

Interface configuration to Server 3 on Leaf 5 & 6 (Similar configuration is needed on Leaf 8)
102 53-1004313-03

VLAN interface configuration on Leaf 5 & 6
VRF configuration on Leaf 5 & 6 (Similar configuration is needed on Leaf 8)
VE interface configuration to Server 3 on Leaf 5 & 6 (Similar configuration is needed on Leaf 8)
53-1004313-03 103

EVPN instance configuration on Leaf 5 and Leaf 6 (under rbridge mode) (Similar configuration is needed on Leaf 8. Leaf 8 will import
default route to reach internet routes from BL3 & BL4.)
VLAN interface configuration on Leaf 5 & 6 for L3 VNI VLANs (Similar configuration is needed on Leaf 8
104 53-1004313-03

VE interface configuration on Leaf 5 & 6 for L3 VNI VLANs (Similar configuration is needed on Leaf 8)
BGP configuration on Leaf 5 (Similar configuration is needed on Leaf 6 and Leaf 8)
Interface configuration on Border-Leaf1 to WAN edge1
Interface configuration on Border-Leaf2 to WAN edge2
53-1004313-03 105

VLAN interface configuration on Border-Leaf1 & Border-Leaf2
VRF configuration on Border-Leaf1 and Border-Leaf2 (similar configuration needed on BL3 & BL4)
VRF VNI VLAN and VE interface configuration on Border-Leaf1 and Border-Leaf2
106 53-1004313-03

VE interface configuration to WAN Edge1 on Border-Leaf1 (Similar configuration is needed on WAN edg1)
VE interface configuration to WAN Edge2 on Border-Leaf2 (Similar configuration is needed on WAN edg2)
Overlay Gateway configuration on Border-Leaf1 and Border-Leaf2 (under config mode)
53-1004313-03 107

brocade-bgp-evpn-based-dci-bvd

brocade-bgp-evpn-based-dci-bvd

Recommended

Recommended

More Related Content

Similar to brocade-bgp-evpn-based-dci-bvd

Similar to brocade-bgp-evpn-based-dci-bvd (20)

brocade-bgp-evpn-based-dci-bvd