This document provides an overview of networking concepts in Linux including layer 2 and layer 3 topics. It discusses link aggregation (LAGs), VLANs, bridges, routing tables, policy-based routing (PBR), VRFs, and network namespaces (NetNS). Key points covered include using LACP for LAGs, VLAN tagging formats, the purpose of bridges, routing tables other than the default main table, and how VRFs and NetNS provide layer 3 and layer 1 separation respectively. Real-world applications of tunnels and VPNs with VRFs are also highlighted.

1. L2/L3 für Fortgeschrittene
Helle und dunkle Magie imLinux-Netzwerkstack
FrOSCon 13 Network Track
Falk Stern, Maximilian Wilhelm
1 / 36

2. Agenda
1. Who are we
2. Layer 2
1. Link Aggregation
2. VLANs
3. Bridges
3. Layer 3
1. Policy based routing
2. VRFs
3. NetNS
2 / 36

3. Who's who Falk Stern
Full Stack Infrastructure Engineer
IPv6 fanboy
Runs his own Kubernetes cluster in his basement
Consultant @ Profi Engineering Systems AG
Contact
@wrf42
falk@fourecks.de
3 / 36

4. Who's who Maximilian Wilhelm
Networker
OpenSource Hacker
Fanboy of
(Debian) Linux
ifupdown2
Occupation:
By day: Senior Infrastructure Architect, Uni Paderborn
By night: Infrastructure Archmage, Freifunk Hochstift
In between: Freelance Solution Architect for hire
Contact
@BarbarossaTM
max@sdn.clinic
4 / 36

5. Who's who
Layer 2
Layer 2 / LAGs
5 / 36

6. Who's who
Layer 2
LAGs
Link Aggregation
Combine one or more physical links between two peers to one virtual link, to
increase over-all bandwidth
create a redundant Layer 2 link
both
Also know as:
LAG
Bonding (Linux)
Aggregated Ethernet (Juniper)
Port-Channel (Cisco)
Trunk (3Com, HP?)
NIC-Teaming
6 / 36

7. Who's who
Layer 2
LAGs
Link Aggregation - Simple Linux bonding
Just use multiple links and hope the peer does, too.
Drawbacks:
If media converters are involved a link-down event may not propagate
No way to tell it the peer is configured the same way
7 / 36

8. Who's who
Layer 2
LAGs
Link Aggregation - LACP
Link Aggregation Control Protocol (802.3ad / 802.1AX)
De-facto standard within networking world
Use LACP signalling to set up LAG with peer
Maximum of 8 interface per LAG
Keep alive every 1s (fast) or every 30s (slow)
An interface can be on one of two modes:
active: send out LACP packets to activly form the LAG
passive: wait for and only then reply to LACP packets
8 / 36

9. Who's who
Layer 2
LAGs
Multi-Chassis Link Aggregation Groups
Link Aggregation between more than two peers
At least on peer as to do magic to make this work
Also know as:
MC-LAG
MLAG
Virtual Port-Channel (vPC)
Source: Wikipedia
9 / 36

10. Who's who
Layer 2
LAGs
Loadbalancing Tra c over LAGs
Round-Robin
One packet on link 1, one on link 2, ..., and repeat
Hashing of header elds
Layer 2 (src MAC + dst MAC)
Only useful if communication is to multiple stations within local subnet
Layer 2+3 (src MAC + dst MAC + src IP + dst IP)
Might be more useful for communication without local subnet
Layer 3+4 (src IP + dst IP + src Port + dst Port)
Probably most useful when communicating with multiple peers
10 / 36

11. Who's who
Layer 2
LAGs
Layer 2 / VLANs
11 / 36

12. Who's who
Layer 2
LAGs
VLANs
Virtual Local Area Networks (VLANs)
Used to separate broadcast domains in LANs
VLAN transport between switches standardized as IEEE 802.1q
after proprietary standards from Cisco, 3COM
12 bit VLAN Identifier
only 4096 possible VLANs, ~100 reserved for internal switch functions
1 2 3 4 5 6
Destination MAC
1 2 3 4 5 6
Source MAC
1 2 3 4
802.1Q Header
TPID=0x8100 PCP/DEI/VID
1 . . . n
Payload
1 2
EtherType/
Size
1 2 3 4
CRC / FCS
1 2 3 4 5 6
Inter Frame Gap
7 8 9 10 11 12
1 2 3 4 5 6
Destination MAC
1 2 3 4 5 6
Source MAC
1 . . . n
Payload
1 2
EtherType/
Size
1 2 3 4
CRC / FCS
1 2 3 4 5 6
Inter Frame Gap
7 8 9 10 11 12
n = 42–1500
n = 46–1500
1 2 3 4 5 6
Preamble
7 8
SFD
1 2 3 4 5 6
Preamble
7 8
SFD
12 / 36

13. Who's who
Layer 2
LAGs
VLANs
QinQ - 802.1ad Double Tag
Of course we can put VLANs into VLANs
1 2 3 4 5 6
Destination MAC
1 2 3 4 5 6
Source MAC
1 2 3 4
802.1Q Header
TPID=0x 8100 PCP/DEI/VID
1 . . . n
Payload
1 2
EtherType/
Size
1 2 3 4
CRC / FCS
1 2 3 4 5 6
Inter Frame Gap
7 8 9 10 11 12
1 2 3 4 5 6
Destination MAC
1 2 3 4 5 6
Source MAC
1 . . . n
Payload
1 2
EtherType/
Size
1 2 3 4
CRC / FCS
1 2 3 4 5 6
Inter Frame Gap
7 8 9 10 11 12
n = 42–1500
n = 46–1500
1 2 3 4 5 6
Preamble
7 8
SFD
1 2 3 4 5 6
Preamble
7 8
SFD
1 2 3 4 5 6
Destination MAC
1 2 3 4 5 6
Source MAC
1 2 3 4
802.1Q Header
TPID=0x 88A8 PCP/DEI/VID
1 . . . n
Payload
1 2
EtherType/
Size
1 2 3 4
CRC / FCS
1 2 3 4 5 6
Inter Frame Gap
7 8 9 10 11 12
n = 38–1500
1 2 3 4 5 6
Preamble
7 8
SFD
1 2 3 4
802.1Q Header
TPID=0x 8100 PCP/DEI/VID
13 / 36

14. Who's who
Layer 2
LAGs
VLANs
Bridges
Layer 2 / Bridges
14 / 36

15. Who's who
Layer 2
LAGs
VLANs
Bridges
Bridges
The switch(es) within your Linux box
Usage: ... bridge [ forward_delay FORWARD_DELAY ]
[ hello_time HELLO_TIME ]
[ max_age MAX_AGE ]
[ ageing_time AGEING_TIME ]
[ stp_state STP_STATE ]
[ vlan_filtering VLAN_FILTERING ]
[ vlan_default_pvid VLAN_DEFAULT_PVID ]
[ mcast_snooping MULTICAST_SNOOPING ]
[...]
[ nf_call_iptables NF_CALL_IPTABLES ]
[ nf_call_ip6tables NF_CALL_IP6TABLES ]
[ nf_call_arptables NF_CALL_ARPTABLES ]
ip link add br0 type bridge
ip link set br0 up
ip link set eth0 master br0
15 / 36

16. Who's who
Layer 2
LAGs
VLANs
Bridges
VLANs and Bridges
Two options, both suck
External trunk as bridge member
External interface is part of the bridge
All VLANs transported within the bridge
All VLANs forwarded on any port
External trunk with many bridges
One interface per VLAN on trunk (e.g. bond0.2342)
One bridge per VLAN (e.g. br2342)
16 / 36

17. Who's who
Layer 2
LAGs
VLANs
Bridges
VXLAN and Bridges
One bridge per VNI
Possibly multiple physical or virtual NICs within bridge, too
VLAN interfaces
VM interfaces (e.g. on KVM host)
17 / 36

18. Who's who
Layer 2
LAGs
VLANs
Bridges
Vlan-aware Bridges
VLANs and bridges have been a challenge
That ain't true no more
Now it's a “regular switch”
Configured with bridge utility from iproute
Real World Use Case:
Simple KVM/Qemu hook for VLAN assignment
https://github.com/FreifunkHochstift/ffho-salt-public/blob/master/kvm/qemu-hook
18 / 36

19. Who's who
Layer 2
LAGs
VLANs
Bridges
Vlan-aware Bridges
Port VLAN management
bridge vlan { add | del }
vid VLAN_ID dev DEV
[ pvid ] [ untagged ]
[ self ] [ master ]
bridge vlan show [ dev DEV ]
[ vid VLAN_ID ]
Forwarding database
bridge fdb [...]
19 / 36

20. Who's who
Layer 2
LAGs
VLANs
Bridges
Vlan-aware Bridges with ifupdown2
iface br0
bridge-ports bond0
bridge-vlan-aware yes
bridge-vids 1013 4002
iface bond0
bridge-vids 100 101 200 201 1013 2000 [..]
iface cr02_eth1
bridge-vids 1013 2000 2004 2006 3002
iface br0.1013
address 10.132.252.22/28
20 / 36

21. Who's who
Layer 2
LAGs
VLANs
Bridges
Vlan-aware Bridges and VXLAN
Extending 24 bit to 36 bit
21 / 36

22. Who's who
Layer 2
Layer 3
Layer 3 / Routing tables
22 / 36

23. Who's who
Layer 2
Layer 3
Routing tables
Every Linux box has a number of routing tables
$ ip route help
Usage: ip route { list | flush } SELECTOR
...
SELECTOR := ... [ table TABLE_ID ]
...
TABLE_ID := [ local | main | default | all | NUMBER ]
By default routing table main is used
So ip route show and ip route show table main show the same thing
23 / 36

24. Who's who
Layer 2
Layer 3
Routing tables
Table local
Contains all routes to
Locally connected IPs
Broadcast addresses
Table main
Contains "usual" routes
Locally connected subnets
Routes to remote subnets
Table default
Usually empty
24 / 36

25. Who's who
Layer 2
Layer 3
PBR
Policy based routing
Available since Linux 2.2 (1999)
Defaut routing policy on every Linux box:
$ ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
Drawbacks
No mechanism for persitancy available
Beware to close every loophole
Rule for IPv4
Rule for IPv6
Rule for incoming interface
25 / 36

26. Who's who
Layer 2
Layer 3
PBR
VRFs
Virtual Routing and Forwarding (VRFs)
Independent routing instances
L3-VPNs
Usually in combination with MPLS
Layer 3 separation
VRF interface is master for “real” interfaces
Defines routing table for VRF
Since Kernel 4.[345] (use >= 4.9)
26 / 36

27. Who's who
Layer 2
Layer 3
PBR
VRFs
Virtual Routing and Forwarding (VRFs)
By foot
ip link add vrf_external type vrf table 1023
ip link set eth0 master vrf_external
ifupdown2
auto eth0
iface eth0
address 2002:db8:23:42::2/64
gateway 2001:db8:23:42::1/64
vrf vrf_external
auto vrf_external
iface vrf_external
vrf-table 1023
Device routes move from table main and local to table 1023
27 / 36

28. Who's who
Layer 2
Layer 3
PBR
VRFs
Connecting VRFs
Requires vEth pair
Like a virtual network cable within the box
A end in main VRF, Z end in VRF “foo”
Usual routing
Static
Bird talking BGP to itself
28 / 36

29. Who's who
Layer 2
Layer 3
PBR
VRFs
Connecting VRFs
By foot
ip link add VETH_END1 type veth
peer name VETH_END2
ifupdown2*
iface veth_ext2int
link-type veth
veth-peer-name veth_int2ext
vrf vrf_external
iface veth_int2ext
link-type veth
veth-peer-name veth_ext2int
* veth-peer-name not merged upstream yet (PR25) 29 / 36

30. Who's who
Layer 2
Layer 3
PBR
VRFs
Real World Applications for VRFs
External interface in VRF
External interface is part of vrf_external
GRE / OpenVPN tunnel sent / receive encapsulated packets over VRF
Local tunnel endpoint is in main VRF
Helpful sysctl
/proc/sys/net/ipv4/tcp_l3mdev_accept
l3mdev == Layer3 Master Device
VRF info is added to socket
Replies send out in VRF where request originated
30 / 36

31. Who's who
Layer 2
Layer 3
PBR
VRFs
Real World Applications - Tunnels / GRE
Outer and/or inner side of tunnel can be part of a VRF
Send
ip link add DEVICE type gre remote ADDR local ADDR dev PHYS_DEV
If PHYS_DEV is within a VRF, all encapsulated packets are send/received in VRF
That's how your internet access is built right now :)
Pushing the inner side of a tunnel into a VRF is equally simple:
ip link set DEVICE master VRF
31 / 36

32. Who's who
Layer 2
Layer 3
PBR
VRFs
Real World Applications - Tunnel / OpenVPN
Pushing the inner side of an OpenVPN tunnel into a VRF is a simple as before.
Sending/receiving encapsulated packets into/from a VRF is more complicated
But there's a patch since October 2016
https://github.com/OpenVPN/openvpn/pull/65
Used to glue remote POPs from Freifunk Hochstift together
openvpn --config your_config.cfg --bind-dev VRF
Now go and motivate Gert - Hi Gert! - to merge it, so we all can us it :)
32 / 36

33. Who's who
Layer 2
Layer 3
PBR
VRFs
NetNS
Network Namespaces (NetNS)
Layer 1 separation
Since Kernel 2.6.29
Own set of routing tables
VRFs and PBR available within NetNS
Own set of netfilter rules
A process can be run in a special NetNS
Two NetNS can be connected by vETH, too.
33 / 36

34. Who's who
Layer 2
Layer 3
Takeaways
Key takeaways
Linux networking has evolved A LOT
Linux today is a first class citizen wrt networking
Vlan-aware bridges are great for virtualization hosts
VRFs can help separte layer 3 domains nicely
Tunneling technologies integrate accordingly
34 / 36

35. Who's who
Layer 2
Layer 3
Takeaways
Links
Further Reading
Contemporary Linux Networking - DENOG9 (2017)
https://www.slideshare.net/BarbarossaTM/contemporary-linux-networking
VRFs
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/
networking/vrf.txt
https://cumulusnetworks.com/blog/vrf-for-linux/
https://de.slideshare.net/CumulusNetworks/operationalizing-vrf-in-the-data-center
35 / 36

36. Who's who
Layer 2
Layer 3
Takeaways
Links
Questions
Question?
36 / 36