Ethernet VPN (EVPN)
EVerything Provider Needs
by Anton Karneliuk and Pau Nadeu
Ethernet services are evolving, standards too.
New services, more bandwidth.
(even your grandma it’s also using TikTok)
Ethernet
Invented
IEEE
802.3
IEEE
802.3u
(FE)
IEEE
802.1Q
Ethernet
Over
MPLS
IEEE
802.1ad
(PB)
IEEE
802.1ah
(PBB)
TRILL
NVO3
EVPN
Segment
Routing
Ethernet
II
Ethernet
Switch-
ing
IEEE
802.3z
(GE)
IEEE
802.3ad
(LAG)
IEEE
802.3ae
(10 GE)
VPLS
IEEE
802.3ba
(40 GE,
100 GE)
IEEE
802.1aq
(SPB)
1982
1973 1983 1989 1995 1998 1998 2000 ~2001 2003 2005 2007 2008 2010 2011 2012 2013 2018+
SR Arch.
SR-MPLS
SRv6
Why Everything
Provider
Needs?
01
It never rains, but it pours.
(why we need more technologies?)
MPLS/VPLS and PBB are
proven technologies, but:
- Control plane approach isn't evolving.
- Uses flood and learn to build Layer 2
FDB (Forwarding Database).
01 EVPN ease delivery on simple
and complex services.
- Takes profit from a lot of experiences of years of
VPLS usage on production networks.
- Abstract and Separate Control Plane and data plane:
BGP carries MAC/IP information, choice your favorite
data plane encapsulation.
- Flexibility for service deliveries over Layer3 networks.
02
Fast enabler for engineers to meet needs what peoples want.
- Use as overlay technology to simplify other scenarios and remove redundant protocols from
the network.
- Integrated Layer 2 and Layer 3 VPN Services (also with IRB).
- Cloud and Virtualization services
- Datacenter interconnect (DCI)
- FTTx wholesale services (Bitstream)
- VM mobility across WAN-connected datacenters.
03
Sounds too cool, but…
(give me more )
Unified Services
- Have similarities with L3VPN in terms of
control and stability.
- Layer2 and Layer3 over same interface,
802.1Q and VPN services.
Designed as Flexible
- You can choose between IP or MPLS.
- NVO’s encapsulations allows services
over a simple IP network.
Have the power
- Consistency between control and data plane,
BGP signaled on control plane, flood-and learn
on data plane.
- Proxy ARP/ND allows PEs to respond ARP/ND
requests enabling you to have a clean network.
- MAC/IP provisioning allows you have a
methodical network control.
Squeezing efficiency
- Multihoming in all-active forwarding enable
loadbalancing across all PEs.
- Optimize BUM frames delivery.
- More efficiency delivering hybrid services over
logical or physical interfaces.
Let’s compare
apples to
apples.
02
VPLS PBB-EVPN EVPN
PE Auto-discovery
Access multi-homing (single active)
All-active multi-homing (flow load balancing)
Fast convergence on failure
MAC mobility
Control plane scaling of large number of MACs
MAC-IP based policying (flexible topologies)
Avoid flooding of unknown-unicast and ARP
Fast convergence based on local rep. on failures
Optimized VM (MAC-IP) mobility & inter-subnet
routing
Simplify configuration and O&M
Per service Class-of-Service
Yes
No
No
CP
Current EVPN
status.
03
EVPN use and evolution.
New but mature
- RFC 7209 (requirements for
EVPN) was released on 2014.
- RFC 7432 (BGP EVPN-MPLS)
was released on 2015.
- A lot of improvements and
new features are introduced
year to year.
Used by big SP / CSPs.
Telefonica, AT&T, Verizon, Apple,
Google, Facebook… are actively
using, supporting and
contributing on EVPN evolution.
Well-known vendors
EVPN it’s supported by almost all
well-known vendors: Nokia (ALU),
Cisco, Juniper, Huawei, FRRouting,
Arista…
Holded by multiple groups.
There are new Multiple work groups
are working together to standardize
new extensions and EVPN functions in
“IETF Internet L2 Working group”.
03
01
02 04
Getting deeper.
04
EVPN nomenclatures.
EVI EVPN instance, extended along all participants PEs.
MAC-VRF EVI routing table
ESI
Network segment connected to PEs that implement EVI.
Ethernet Tag Identifies broadcast domain (i.e. BD
or VLAN)
S-A/ A-A modes
Single-Active redundancy
All-Active redundancy
Single-Homed Device/Network (SHD/SHN) -> ESI = 0
Multi-Homed Device/Network (MHD/MHN)
Control and data plane in EVPN
- MAC/IP tuple routing it’s introduced using MP-BGP.
- Flood-and-learn method are kept between PE-CE (in data plane).
- MP-BGP use between PEs enable routing and MAC policy, TE criteria's, etc...
- EVPN it’s using MP-BGP AFI 25 (VPLS) and SAFI 70 (EVPN), and includes Route
Distinguisher (RD), Route Targets (RT), BGP communities, etc…
BGP Routes
Route Types:
[1] Ethernet Auto-Discovery (AD) Route
[2] MAC Advertisement Route
[3] Inclusive Multicast Route
[4] Ethernet Segment Route
[5] IP Prefix Route
BGP Route Attributes
Extended Communities:
- ESI-MPLS Label
- ES-Import RT
- MAC Mobility
- Default Gateway
Service Interfaces
- There are three service interface ”flavours”: VLAN Based Interface, VLAN Bundle
Interface, VLAN Aware Bundle Interface.
- There is another, and obvious, service interface type: Port Based Interface.
VLAN Based VLAN Bundle VLAN Aware Bundle
VLAN per EVI 1:1 N:1 N:1
Tenant ID per EVI Single Multiple Multiple
BD (Bridge-Domain) per EVI Single Single Multiple
MAC/VLAN overlapping
VLAN Translation (Mapping)
Yes
No
EVPN flavours...
- There are… three EVPN variants depending on data plane choice.
- All of this are based on route learning between PE’s using BGP.
- EVPN uses MPLS/IP, as RFC 7209 and RFC 7432 requires.
- PBB E-VPN combines 802.1ah PBB with EVPN in order to achieve MAC scalability stacking C-
MACs over B-MACs.
- What EVPN-VXLAN uses in data plane? Obviously, VXLAN.
Data Plane
Control Plane
EVPN MP-BGP
draft-ietf-l2vpn-evpn
Multiprotocol
Label
Switching
(MPLS)
draft-ietf-l2vpn-evpn
Provider
Backbone
Bridges (PBB)
draft-ietf-l2vpn-pbb-evpn
Network
Virtualization Overlay
(NVO)
draft-sd-l2vpn-evpn-
overlay
MAC Mobility & Duplication
VM migration it’s frequent between CPDs.
01
This implies that a MAC address that at a given moment is advertised from a
network PE/ESI (old and incorrect route) changes its location and passes and
advertises itself from another PE / ESI of the EVI (new and correct route).
02
The method used to identify the valid path from among the
existing in the network for each MAC is based on a counter
associated with the “MAC mobility” community, which is added to each
route type 2 "MAC Advertisement"
03
Route with highest sequence number it’s the valid route.
04
A-A Multihoming and DF election
PE’s connected to same ESI are auto-discovered using “Ethernet Segment”
route.
01
Each PE have automatically enabled filters to import type 4 routes with
community ES-Import.
02
One of PE connected to same ESI are choosed as “DF” (Designated
Forwarder): it sends BUM traffic received from core to ESI-connected CEs,
other PE’s filter the traffic to CE).
03
DF election in a site can be done in multiple ways:
- A multiple DF can coexist to have load balancing.
- One DF it’s choose for ESI or Ethernet Tag.
04
A-A Multihoming and Split Horizon
PE’s announces ESI MPLS Label associated to each ES with multihoming
topology.
01
This ”Split Horizon Label” it’s sent associated with type 1 routes (Auto-
Discovery) in a BGP Community.
02
Split-Horizon as Loop avoiding mechanism: don’t forward traffic on interface
through which it is received.
03
When PE sends BUM traffic to core, it replicates the frame to all EVI
participant PEs, and add ESI MPLS Label to identify ESI where it’s originated.
04
PEs connected to the same network segment identify the labels and filters
BUM traffic towards CEs connected to the ESI.
05
Unkown Unicast Management and
ARP/ND
The PEs participating in the EVI perform the functions of Proxy-
ARP/ND based on the EVPN route tables of each instance.
01
“ARP/ND/DHCP snooping” optimizes and reduces traffic of the type
"Unknown Unicast" between sites of an EVI.
02
Static provisioning of the MAC addresses of an ESI you can further reduce
(or even eliminate) “unknown unicast” traffic.
03
A-A Multihoming & Aliasing
Type 2 "MAC Advertisement" routes include information about
the ESI of the EVI where the MAC of each host resides.
01
Type 1 "Ethernet A-D" routes include information indicating
the PEs connected to each ESI.
02
Combining this information, remote PEs can identify
the MAC addresses found on each ESI, regardless of the PE that advertised
those MACs.
03
S-A (Single-Active) Multihoming mode: allows the use of backup paths,
improving network convergence in the event of link / node failures.
A-A (Active-Active) Multihoming mode: allows load balancing by flow
towards servers connected to remote PEs.
04
Convergence optimization on failures.
PEs learn using BGP the MACs on each ESI (Type-2 Route), and PEs connected
to each ESI (Type-1 Route)
01
When PE detect a failure on a link that’s connecting to an ESI, all learned
MACs learned using this path are considered as inactive.
02
To indicate this to remote PEs, the local PE simply has to remove the
“Ethernet A-D” route for failed ESI. Remote PEs will massively remove all
MAC/ESI routes reached through the originating PE.
03
Automate
EVPN.
05
Building blocks
EVI/VTEP
01
Route Targets (RT) ad Route Distinguisher (RD)
02
ESI
03
Ethernet tag
04
Encapsulation type (VXLAN, SR)
05
Built-in automation
- Auto-derived RD
- Auto-derived RT
- RFC 7432 and 8365
Benefits Drawbacks
- Simpler provisioning
- Certain logic
- In multivendor networks interop
shall be checked
- Not suitable for complex
deployments with route leaking or
asymmetric VRFs
Automation toolkit
Source of Truth
Documentation of all the
information (EVI/VTEP,
RD/RT, ESI, etc) – NetBox
Standard templates and
scripts – GitLab/GitHub
Interfaces
NETCONF/YANG,
RESTCONF/YANG,
GNMI/YANG
Tools
- Python
- Python w/ Nornir
- Ansible
- Go
- Code-less automation
Workflow managers
- Ansible Automation
Controller (AWX)
- StackStorm
- Apache Airflow
03
01
02 04
How does that work?
Executor
Orchestrator
Data Source 2
Data Source 1
Data Visualisation
Data Collector
CLI/SSH NETCONF/YANG RESTCONF/YANG
gNMI/YANG
SNMP
SYSLOG
GRPC
Thank you for
attention!
Pau Nadeu / Anton Karneliuk

Ethernet VPN (EVPN) EVerything Provider Needs

  • 1.
    Ethernet VPN (EVPN) EVerythingProvider Needs by Anton Karneliuk and Pau Nadeu
  • 2.
    Ethernet services areevolving, standards too. New services, more bandwidth. (even your grandma it’s also using TikTok) Ethernet Invented IEEE 802.3 IEEE 802.3u (FE) IEEE 802.1Q Ethernet Over MPLS IEEE 802.1ad (PB) IEEE 802.1ah (PBB) TRILL NVO3 EVPN Segment Routing Ethernet II Ethernet Switch- ing IEEE 802.3z (GE) IEEE 802.3ad (LAG) IEEE 802.3ae (10 GE) VPLS IEEE 802.3ba (40 GE, 100 GE) IEEE 802.1aq (SPB) 1982 1973 1983 1989 1995 1998 1998 2000 ~2001 2003 2005 2007 2008 2010 2011 2012 2013 2018+ SR Arch. SR-MPLS SRv6
  • 3.
  • 4.
    It never rains,but it pours. (why we need more technologies?) MPLS/VPLS and PBB are proven technologies, but: - Control plane approach isn't evolving. - Uses flood and learn to build Layer 2 FDB (Forwarding Database). 01 EVPN ease delivery on simple and complex services. - Takes profit from a lot of experiences of years of VPLS usage on production networks. - Abstract and Separate Control Plane and data plane: BGP carries MAC/IP information, choice your favorite data plane encapsulation. - Flexibility for service deliveries over Layer3 networks. 02 Fast enabler for engineers to meet needs what peoples want. - Use as overlay technology to simplify other scenarios and remove redundant protocols from the network. - Integrated Layer 2 and Layer 3 VPN Services (also with IRB). - Cloud and Virtualization services - Datacenter interconnect (DCI) - FTTx wholesale services (Bitstream) - VM mobility across WAN-connected datacenters. 03
  • 5.
    Sounds too cool,but… (give me more ) Unified Services - Have similarities with L3VPN in terms of control and stability. - Layer2 and Layer3 over same interface, 802.1Q and VPN services. Designed as Flexible - You can choose between IP or MPLS. - NVO’s encapsulations allows services over a simple IP network. Have the power - Consistency between control and data plane, BGP signaled on control plane, flood-and learn on data plane. - Proxy ARP/ND allows PEs to respond ARP/ND requests enabling you to have a clean network. - MAC/IP provisioning allows you have a methodical network control. Squeezing efficiency - Multihoming in all-active forwarding enable loadbalancing across all PEs. - Optimize BUM frames delivery. - More efficiency delivering hybrid services over logical or physical interfaces.
  • 6.
  • 7.
    VPLS PBB-EVPN EVPN PEAuto-discovery Access multi-homing (single active) All-active multi-homing (flow load balancing) Fast convergence on failure MAC mobility Control plane scaling of large number of MACs MAC-IP based policying (flexible topologies) Avoid flooding of unknown-unicast and ARP Fast convergence based on local rep. on failures Optimized VM (MAC-IP) mobility & inter-subnet routing Simplify configuration and O&M Per service Class-of-Service Yes No No CP
  • 8.
  • 9.
    EVPN use andevolution. New but mature - RFC 7209 (requirements for EVPN) was released on 2014. - RFC 7432 (BGP EVPN-MPLS) was released on 2015. - A lot of improvements and new features are introduced year to year. Used by big SP / CSPs. Telefonica, AT&T, Verizon, Apple, Google, Facebook… are actively using, supporting and contributing on EVPN evolution. Well-known vendors EVPN it’s supported by almost all well-known vendors: Nokia (ALU), Cisco, Juniper, Huawei, FRRouting, Arista… Holded by multiple groups. There are new Multiple work groups are working together to standardize new extensions and EVPN functions in “IETF Internet L2 Working group”. 03 01 02 04
  • 10.
  • 11.
    EVPN nomenclatures. EVI EVPNinstance, extended along all participants PEs. MAC-VRF EVI routing table ESI Network segment connected to PEs that implement EVI. Ethernet Tag Identifies broadcast domain (i.e. BD or VLAN) S-A/ A-A modes Single-Active redundancy All-Active redundancy Single-Homed Device/Network (SHD/SHN) -> ESI = 0 Multi-Homed Device/Network (MHD/MHN)
  • 12.
    Control and dataplane in EVPN - MAC/IP tuple routing it’s introduced using MP-BGP. - Flood-and-learn method are kept between PE-CE (in data plane). - MP-BGP use between PEs enable routing and MAC policy, TE criteria's, etc... - EVPN it’s using MP-BGP AFI 25 (VPLS) and SAFI 70 (EVPN), and includes Route Distinguisher (RD), Route Targets (RT), BGP communities, etc… BGP Routes Route Types: [1] Ethernet Auto-Discovery (AD) Route [2] MAC Advertisement Route [3] Inclusive Multicast Route [4] Ethernet Segment Route [5] IP Prefix Route BGP Route Attributes Extended Communities: - ESI-MPLS Label - ES-Import RT - MAC Mobility - Default Gateway
  • 13.
    Service Interfaces - Thereare three service interface ”flavours”: VLAN Based Interface, VLAN Bundle Interface, VLAN Aware Bundle Interface. - There is another, and obvious, service interface type: Port Based Interface. VLAN Based VLAN Bundle VLAN Aware Bundle VLAN per EVI 1:1 N:1 N:1 Tenant ID per EVI Single Multiple Multiple BD (Bridge-Domain) per EVI Single Single Multiple MAC/VLAN overlapping VLAN Translation (Mapping) Yes No
  • 14.
    EVPN flavours... - Thereare… three EVPN variants depending on data plane choice. - All of this are based on route learning between PE’s using BGP. - EVPN uses MPLS/IP, as RFC 7209 and RFC 7432 requires. - PBB E-VPN combines 802.1ah PBB with EVPN in order to achieve MAC scalability stacking C- MACs over B-MACs. - What EVPN-VXLAN uses in data plane? Obviously, VXLAN. Data Plane Control Plane EVPN MP-BGP draft-ietf-l2vpn-evpn Multiprotocol Label Switching (MPLS) draft-ietf-l2vpn-evpn Provider Backbone Bridges (PBB) draft-ietf-l2vpn-pbb-evpn Network Virtualization Overlay (NVO) draft-sd-l2vpn-evpn- overlay
  • 15.
    MAC Mobility &Duplication VM migration it’s frequent between CPDs. 01 This implies that a MAC address that at a given moment is advertised from a network PE/ESI (old and incorrect route) changes its location and passes and advertises itself from another PE / ESI of the EVI (new and correct route). 02 The method used to identify the valid path from among the existing in the network for each MAC is based on a counter associated with the “MAC mobility” community, which is added to each route type 2 "MAC Advertisement" 03 Route with highest sequence number it’s the valid route. 04
  • 16.
    A-A Multihoming andDF election PE’s connected to same ESI are auto-discovered using “Ethernet Segment” route. 01 Each PE have automatically enabled filters to import type 4 routes with community ES-Import. 02 One of PE connected to same ESI are choosed as “DF” (Designated Forwarder): it sends BUM traffic received from core to ESI-connected CEs, other PE’s filter the traffic to CE). 03 DF election in a site can be done in multiple ways: - A multiple DF can coexist to have load balancing. - One DF it’s choose for ESI or Ethernet Tag. 04
  • 17.
    A-A Multihoming andSplit Horizon PE’s announces ESI MPLS Label associated to each ES with multihoming topology. 01 This ”Split Horizon Label” it’s sent associated with type 1 routes (Auto- Discovery) in a BGP Community. 02 Split-Horizon as Loop avoiding mechanism: don’t forward traffic on interface through which it is received. 03 When PE sends BUM traffic to core, it replicates the frame to all EVI participant PEs, and add ESI MPLS Label to identify ESI where it’s originated. 04 PEs connected to the same network segment identify the labels and filters BUM traffic towards CEs connected to the ESI. 05
  • 18.
    Unkown Unicast Managementand ARP/ND The PEs participating in the EVI perform the functions of Proxy- ARP/ND based on the EVPN route tables of each instance. 01 “ARP/ND/DHCP snooping” optimizes and reduces traffic of the type "Unknown Unicast" between sites of an EVI. 02 Static provisioning of the MAC addresses of an ESI you can further reduce (or even eliminate) “unknown unicast” traffic. 03
  • 19.
    A-A Multihoming &Aliasing Type 2 "MAC Advertisement" routes include information about the ESI of the EVI where the MAC of each host resides. 01 Type 1 "Ethernet A-D" routes include information indicating the PEs connected to each ESI. 02 Combining this information, remote PEs can identify the MAC addresses found on each ESI, regardless of the PE that advertised those MACs. 03 S-A (Single-Active) Multihoming mode: allows the use of backup paths, improving network convergence in the event of link / node failures. A-A (Active-Active) Multihoming mode: allows load balancing by flow towards servers connected to remote PEs. 04
  • 20.
    Convergence optimization onfailures. PEs learn using BGP the MACs on each ESI (Type-2 Route), and PEs connected to each ESI (Type-1 Route) 01 When PE detect a failure on a link that’s connecting to an ESI, all learned MACs learned using this path are considered as inactive. 02 To indicate this to remote PEs, the local PE simply has to remove the “Ethernet A-D” route for failed ESI. Remote PEs will massively remove all MAC/ESI routes reached through the originating PE. 03
  • 21.
  • 22.
    Building blocks EVI/VTEP 01 Route Targets(RT) ad Route Distinguisher (RD) 02 ESI 03 Ethernet tag 04 Encapsulation type (VXLAN, SR) 05
  • 23.
    Built-in automation - Auto-derivedRD - Auto-derived RT - RFC 7432 and 8365 Benefits Drawbacks - Simpler provisioning - Certain logic - In multivendor networks interop shall be checked - Not suitable for complex deployments with route leaking or asymmetric VRFs
  • 24.
    Automation toolkit Source ofTruth Documentation of all the information (EVI/VTEP, RD/RT, ESI, etc) – NetBox Standard templates and scripts – GitLab/GitHub Interfaces NETCONF/YANG, RESTCONF/YANG, GNMI/YANG Tools - Python - Python w/ Nornir - Ansible - Go - Code-less automation Workflow managers - Ansible Automation Controller (AWX) - StackStorm - Apache Airflow 03 01 02 04
  • 25.
    How does thatwork? Executor Orchestrator Data Source 2 Data Source 1 Data Visualisation Data Collector CLI/SSH NETCONF/YANG RESTCONF/YANG gNMI/YANG SNMP SYSLOG GRPC
  • 26.
    Thank you for attention! PauNadeu / Anton Karneliuk