Overlay Networks & IP Fabrics
FrOSCon 13 Network Track
Falk Stern, Maximilian Wilhelm
1 / 27

Agenda
1. Who's who
2. Encapsulate me one more time
3. Tunnel technologies
4. IP-Fabrics
5. Real-World-examples
2 / 27

Who's who Falk Stern
Full Stack Infrastructure Engineer
IPv6 fanboy
Runs his own Kubernetes cluster in his basement
Consultant @ Profi Engineering Systems AG
Contact
@wrf42
falk@fourecks.de
3 / 27

Who's who Maximilian Wilhelm
Networker
OpenSource Hacker
Fanboy of
(Debian) Linux
ifupdown2
Occupation:
By day: Senior Infrastructure Architect, Uni Paderborn
By night: Infrastructure Archmage, Freifunk Hochstift
In between: Freelance Solution Architect for hire
Contact
@BarbarossaTM
max@sdn.clinic
4 / 27

Who's who
Encaps/Decaps
Encapsulation
Wrapping frames or packets into other packets
Not always a good idea
A bit like Christmas, you're never sure what you get
5 / 27

Who's who
Encaps/Decaps
6 / 27

Who's who
Encaps/Decaps
Tunnel
IPSec
Authenticates and/or encrypts IP packets
Authenticated Header, Encrypted Security Payload
IP protocol 50, 51 RFC4301
Transport / Tunnelmode
Transport inserts header into packet, Tunnel encapsulates
Dynamic Keying through IKEv1, IKEv2
IKE is based on UDP, Port 500
Complex protocol, many options
NAT unfriendly, NAT-Traversal is negotiated, UDP port 4500
7 / 27

Who's who
Encaps/Decaps
Tunnel
IPSec
Phase 1
Exchange of encryption proposals
Both ends exchange session keys through Diffie-Hellman key exchange
Pre-Shared-Key or certificate exchange encrypted with session key
Security Associations are exchanged
Phase 2
Diffie-Hellman key exchange
Periodic key changes for perfect forward secrecy
Only traffic matching Security Associations is encrypted
8 / 27

Who's who
Encaps/Decaps
Tunnel
GRE - Generic Routing Encapsulation
Developed by Cisco in 1994, now RFC2784 and RFC2890
Encapsulates IP, IPX, AppleTalk in IP
IP protocol 47
Adds a 4 byte GRE header, total overhead 20 bytes
Used in
PPTP VPN (encapsulates IP in PPP in GRE)
IPv6 in IPv4
Tunnel between IPSEC endpoints
Low overhead tunnel between everything
9 / 27

Who's who
Encaps/Decaps
Tunnel
L2TP - Layer 2 Tunneling Protocol
L2TPv2 developed to tunnel PPP - RFC2661
L2TPv3 as alternative to MPLS - RFC3931
Based on UDP
NAT-friendly
L2TPv2 VPNs encapsulate L2TP frames in IPSEC - RFC3193
10 / 27

Who's who
Encaps/Decaps
Tunnel
OpenVPN
Flexible SSL VPN
TCP/UDP based
Tunnels IPv4, IPv6 or even Ethernet frames
X.509 Certificate based Authentication
Username/Passwort with 2FA possible
11 / 27

Who's who
Encaps/Decaps
Tunnel
Multi Protocol Label Switching / MPLS
Developed to enable fast switching in core routers
Switching packets based on IP required TCAM, was expensive
Label lookup is faster, no need for longest match
Layer 2.5, requires IP to work, RFC3031
Enables predefined paths through a network
Allows Traffic-Engineering
Used by service providers
12 / 27

Who's who
Encaps/Decaps
Tunnel
Virtual eXtensible LAN / VXLAN
Poor man's MPLS
Developed by Cisco, Arista Networks, VMware, now RFC7348
Broad industry backing
24 bit VXLAN identifier (VNI) instead of 12 bit VLAN ID
16M vs. 4096
Encapsulates Ethernet frames in UDP packets
40 bytes overhead over IPv4, 60 bytes over IPv6
Endpoints are called Virtual Tunnel EndPoint (VTEP)
13 / 27

Who's who
Encaps/Decaps
Tunnel
Virtual eXtensible LAN / VXLAN
Unicast mode
VTEPs are statically defined
Point-to-Point
Can be used for data center interconnects
Multicast mode
All VTEPs listen on a specified multicast address
Broadcasts and unknown Unicasts (BUM) are mapped to Multicast Groups
Dynamic learning of endpoints through listening
14 / 27

Who's who
Encaps/Decaps
Tunnel
Virtual eXtensible LAN / VXLAN
Controller based
An external controller programs VTEP endpoints and MAC mappings
BUM traffic gets replicated to all VTEPs
Ideally there is no BUM traffic
Commercially available
Cisco APIC
VMware NSX through OVSDB
Cumulus vxfld (https://github.com/CumulusNetworks/vxfld)
15 / 27

Who's who
Encaps/Decaps
Tunnel
IP Fabrics IP Fabrics
16 / 27

Who's who
Encaps/Decaps
Tunnel
IP Fabrics
Clos Fabrics
Invented for the telephone network
Formalized by Charles Clos in 1952
Far fewer connections required than with a single switch
17 / 27

Who's who
Encaps/Decaps
Tunnel
IP Fabrics
What's wrong with Layer 2?
Spanning-Tree needs blocked paths
L2 only has a single path
IP can use multiple concurrent paths
MCLAG and LACP are possible solutions
But way too complex, limited to 2 upstream devices
Does not scale
18 / 27

Who's who
Encaps/Decaps
Tunnel
IP Fabrics
IP Fabrics
Predictable latency through the whole fabric
Scalable
Predictable bandwidth through the whole fabric
Perfect underlay for overlay networks
Typical design includes Leaf and Spine
Bisectional bandwidth can be scaled with number of Spines
19 / 27

Who's who
Encaps/Decaps
Tunnel
IP Fabrics
BGP to the rescue!
BGP has always defined multiple paths between AS
Is able to carry all necessary routes
Through VPNv4 AFI, can carry MAC addresses
One AS per Rack
iBGP would need Route Reflectors or full mesh
20 / 27

Who's who
Encaps/Decaps
Tunnel
IP Fabrics
BGP to the rescue!
Packetflow in an IP Fabric
21 / 27

Who's who
Encaps/Decaps
Tunnel
IP Fabrics
Tying it all together
VXLAN* can be used as an overlay protocol in the fabric
BGP carries all MAC adresses with next-hop of VTEP
Suddenly
All links in use
More than 4096 VLANs available
"A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN)"
RFC8365, March 2018
* To be fair MPLS can be used as data plane too 22 / 27

Who's who
Encaps/Decaps
Tunnel
IP Fabrics
VXLAN Tunnel via Unicast between LO-
IPs of
dr-01
cr-D
VTEPs bridged into
Chaos network on dr-01
eth1 NIC on cr-D
AS13020
AS39225
Core
Distribution
Border
br-01 cr-E cr-A
cr-D
cr-B
cr-C
dr-01
Access
sw-01 ap-04
ap-03ap-02ap-01
dr-02
VXLAN
Tunneled Chaos Ethernet
23 / 27

Who's who
Encaps/Decaps
Tunnel
IP Fabrics
Tunneled Chaos Ethernet
Set up VTEPs
dr-01# ip link add vx_chaos type vxlan id 31337 local 94.45.224.0 remote 194.107.207.4
dr-01# ip l s dev vx_chaos up
cr-D# ip link add vx_chaos type vxlan id 31337 remote 94.45.224.0 local 194.107.207.4
cr-D# ip l s dev vx_chaos up
Join VTEP into precon gured bridge br_chaos
# ip l vx_chaos set master br_chaos
24 / 27

Who's who
Encaps/Decaps
Tunnel
IP Fabrics
Real World Examples
Cisco ACI
BGP control-plane with VXLAN overlay
VMware NSX
Controller based control-plane with VXLAN overlay
OpenNebula / Apache Cloudstack
Network virtualization with mcast-VXLAN
25 / 27

Who's who
Encaps/Decaps
Tunnel
IP Fabrics
Links
Further Reading
Cumulus BGP im DC
cumulus EVPN
26 / 27

Who's who
Encaps/Decaps
Tunnel
IP Fabrics
Links
Questions
Questions?
27 / 27