These are the days of VMs, containers, and service meshes. The network, for a long time the sysadmin’s mysterious domain, is now at the fore-front: providing overlays, security features, and headaches. It’s vital to be able to understand what’s going on under the hood of a cloud-native platform if you ever hope to debug it, but do you know a TAP from a TUN, let alone an ipvlanL3? This talk will take you through all the network interface types on modern linux, from good old eth0 to the vEths used by Docker and the tunnels used by Calico.

1. Software Networking and Interfaces @mt165
Software Networking
and Interfaces
Matt Turner
SRECon, Singapore | June 2019
on Linux
@mt165 | mt165.co.uk

2. Software Networking and Interfaces @mt165

3. Software Networking and Interfaces @mt165
Outline
● Networking 101
● Interfaces
● Bridges
● Emergent Systems

4. Software Networking and Interfaces @mt165
This is not about
● IP, TCP, addressing
● SDN

5. Software Networking and Interfaces @mt165
Networking 101

6. Software Networking and Interfaces @mt165
Ethernet and ARP
Ethernet - L2 protocol
MAC Address - Media Access Control Address - Ethernet address, eg
c0:ff:ee:be:ef:69
ARP - Address Resolution Protocol - DNS for ethernet: IP -> MAC
$ arp
Address HWtype HWaddress Flags Mask Iface
192.168.0.239 ether 48:3b:38:01:6a:23 C enp2s0
172.28.0.13 ether 02:42:ac:1c:00:0d C br-de368312f566

7. Software Networking and Interfaces @mt165
vLANs
Virtual LANs
IEEE 802.1q
Simulates multiple networks using one set of cables and switches
Each vLAN has a short numeric ID
Nested vLANs - IEEE 802.1ad, aka “q in q”

8. Software Networking and Interfaces @mt165
iptables
Linux kernel subsystem
Can do lots of things to packets as they pass through a system
Including: packet manipulation

9. Software Networking and Interfaces @mt165
NAT vs Routing
123.1.2.1
123.1.2.3
123.1.2.4
192.168.0.10
192.168.0.11
234.1.2.3
234.1.2.4
10.0.0.10
10.0.0.11
123.1.2.3 234.1.2.3
234.1.2.1
��
��
��
��

10. Software Networking and Interfaces @mt165
Route tables
Gives the next hop for a destination subnet.
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default moon.lan 0.0.0.0 UG 3 0 0 enp2s0
loopback localhost 255.0.0.0 UG 0 0 0 lo
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0

11. Software Networking and Interfaces @mt165
Classful Routing
Class A 8 bit 0.0.0.0 127.0.0.1
Class B 16 bit 128.0.0.0 191.255.0.0
Class C 24 bit 192.0.0.0 223.255.255.0
Class D - multicast 224.0.0.0 239.255.255.255
Class E - reserved 240.0.0.0 255.255.255.255

12. Software Networking and Interfaces @mt165
Private Address Ranges
The “24-bit block” 8 bit prefix 10.0.0.0 –
10.255.255.255
1 class A
The “20-bit block” 12 bit prefix 172.16.0.0 –
172.31.255.255
16 class Bs
The “16-bit block “ 16 bit prefix 192.168.0.0 –
192.168.255.255
256 class Cs
Loopback 8 bit prefix 127.0.0.0 -
127.255.255.255
1 class A

13. Software Networking and Interfaces @mt165
Classless Routing and CIDRs
Classless Inter-Domain Routing
Classful was too rigid and wasteful
Arbitrary ranges of addresses, notated as start address and size (as prefix mask)
CIDR notation: 192.168.42.0/24; 10.0.0.0/8

14. Software Networking and Interfaces @mt165
Address all, some, one host
● All hosts in a subnet
● Some hosts, which have opted in
● One arbitrary host from a set
Broadcast, Multicast, Anycast

15. Software Networking and Interfaces @mt165
DHCP
Asks a central server to allocate you an IP address
Based on an ethernet broadcast

16. Software Networking and Interfaces @mt165
Interfaces and Bridges

17. Software Networking and Interfaces @mt165

18. Software Networking and Interfaces @mt165
eth0
enp0s2

19. Software Networking and Interfaces @mt165
eth0
enp0s2
192.168.0.10

20. Software Networking and Interfaces @mt165
eth0
enp0s2
192.168.0.10
ping
connect(“google.com”);

21. Software Networking and Interfaces @mt165
eth0
enp0s2
192.168.0.10
nginx
bind(“*:80”);

22. Software Networking and Interfaces @mt165
eth0
192.168.0.10
nginx
172.16.0.10
bind(“*:80”);

23. Software Networking and Interfaces @mt165
eth0
nginx
bind(“192.168.0.10:80”);
192.168.0.10
172.16.0.10

24. Software Networking and Interfaces @mt165
eth0
192.168.0.10
nginx
eth1
172.16.0.10
bind(“192.168.0.10:80”);

25. Software Networking and Interfaces @mt165
eth0
192.168.0.10
nginx
bind(“172.16.0.10:80”);
eth1
172.16.0.10

26. Software Networking and Interfaces @mt165
eth0
192.168.0.10e1000

27. Software Networking and Interfaces @mt165
e1000
eth0
192.168.0.10
hardware software

28. Software Networking and Interfaces @mt165
eth0
192.168.0.10e1000
foo0

29. Software Networking and Interfaces @mt165
eth0
192.168.0.10e1000
foo0

30. Software Networking and Interfaces @mt165
eth0
192.168.0.10e1000
foo0
packetd

31. Software Networking and Interfaces @mt165
eth0
192.168.0.10e1000
tun0
packetd172.16.0.10

32. Software Networking and Interfaces @mt165
eth0
192.168.0.10e1000
ping
tun0
packetd172.16.0.10

33. Software Networking and Interfaces @mt165
e1000
eth0
192.168.0.10
tun0
packetd172.16.0.10

34. Software Networking and Interfaces @mt165
eth0
192.168.0.10e1000
tun0
packetd172.16.0.10
NAT

35. Software Networking and Interfaces @mt165
eth0
192.168.0.10e1000
tun0
packetd172.16.0.10
ip_forward = 1

36. Software Networking and Interfaces @mt165
eth0
192.168.0.10e1000
tap0
packetd
172.16.0.10
DE:AD:BE:EF:00:00

37. Software Networking and Interfaces @mt165
eth0
e1000
tap0
packetd
br0

38. Software Networking and Interfaces @mt165
br0
eth0
e1000
tap0
packetd

39. Software Networking and Interfaces @mt165
br0
eth0
e1000
tap0
packetd
br0 192.168.0.10

40. Software Networking and Interfaces @mt165
ovs0
eth0
e1000
tap0
packetd
ovs0 192.168.0.10

41. Software Networking and Interfaces @mt165
br0
eth0
e1000
tap0
br0
192.168.0.10
qemu [+ kvm]

42. Software Networking and Interfaces @mt165
br0
eth0
e1000
tap0
br0
192.168.0.10
e1000
eth0
qemu
192.168.0.11
C0:FF:EE:C0:FF:EE

43. Software Networking and Interfaces @mt165
br0
eth0
e1000
tap0
br0
192.168.0.10
virtio_
net
eth0
qemu
192.168.0.11
C0:FF:EE:C0:FF:EE

44. Software Networking and Interfaces @mt165
br0
eth0
e1000
br0
192.168.0.10

45. Software Networking and Interfaces @mt165
br0
eth0
e1000
br0
192.168.0.10

46. Software Networking and Interfaces @mt165
br0
eth0
e1000
br0
192.168.0.10
nginx
ping

47. Software Networking and Interfaces @mt165
br0
eth0
e1000
br0
192.168.0.10
nginx
ping
veth0 veth1

48. Software Networking and Interfaces @mt165
br0
eth0
e1000
br0
192.168.0.10
nginx
ping
veth0 veth1
172.16.0.1 172.16.0.2

49. Software Networking and Interfaces @mt165
br0
eth0
e1000
br0
192.168.0.10
nginx
ping
veth0
veth1 172.16.0.2

50. Software Networking and Interfaces @mt165
br0
eth0
e1000
br0
192.168.0.10
nginx
ping
veth0
veth1
172.16.0.2
ftpd

51. Software Networking and Interfaces @mt165
br0
eth0
e1000
br0
192.168.0.10
nginx
ping
veth0
eth0
172.16.0.2
ftpd

52. Software Networking and Interfaces @mt165
br0
eth0
e1000
br0
192.168.0.10
nginx
ping
eth0
172.16.0.2
ftpd

53. Software Networking and Interfaces @mt165
eth0
e1000 192.168.0.10
eth0.42
172.16.0.2
ftpd
vlan : 42
ftpd

54. Software Networking and Interfaces @mt165
eth0
e1000 192.168.0.10
nginx
eth0.42
172.16.0.2
ftpd
vlan : 42

55. Software Networking and Interfaces @mt165
eth0e1000
C0:FF:EE
192.168.0.10
eth1
192.168.0.11
ftpd
eth2
192.168.0.12

56. Software Networking and Interfaces @mt165
eth0e1000
F0:00:01 192.168.0.10
eth1
192.168.0.11
ftpd
macvlan
F0:0B:4R
eth2
192.168.0.12
macvlan
F0:07:07

57. Software Networking and Interfaces @mt165
eth0e1000
F0:00:01 192.168.0.10
eth1
192.168.0.11
ftpd
macvlan
F0:0B:4R
eth2
192.168.0.12
macvlan
F0:07:07

58. Software Networking and Interfaces @mt165
eth0e1000
F0:00:01 192.168.0.10
eth1
192.168.0.11
ftpd
macvlan
F0:0B:4R
eth2
192.168.0.12
macvlan
F0:07:07

59. Software Networking and Interfaces @mt165
eth0e1000
F0:00:01 192.168.0.10
eth1
192.168.0.11
ftpd
macvlan
F0:0B:4R
eth2
192.168.0.12
macvlan
F0:07:07

60. Software Networking and Interfaces @mt165
br0
eth0e1000
F0:00:01 192.168.0.10
macvlan0
ftpd
macvlan
F0:0B:4R
macvtap
F0:07:07
qemu
tap0
qemu
macvtap0

61. Software Networking and Interfaces @mt165
eth0e1000
C0:FF:EE 192.168.0.10
eth1
192.168.0.11
ftpd
ipvlan
C0:FF:EE
eth2
192.168.0.12
ipvlan
C0:FF:EE

62. Software Networking and Interfaces @mt165
eth0e1000
C0:FF:EE 192.168.0.10
eth1
192.168.1.10
ftpd
ipvlan L3
eth2
192.168.2.10ipvlan L3

63. Software Networking and Interfaces @mt165
eth0e1000
C0:FF:EE 192.168.0.10
192.168.1.10ipvlan L3
ipvlan L3
BIRD
Calico
192.168.1.10 nodejs

64. Software Networking and Interfaces @mt165
K8s etcd
K8s API
plugin
Calico etcd
calico-node
felix
confd
BIRD
eth0 (192.168.0.1)
vEth-xxxx (no IP)
vEth pair
L2 broadcast domain (routing by ARP),
Or, L3 network (routing by BGP peering)
eth0 eth0
iptables (outside namespace)
Linux kernel
ip_forwardin
g
Kernel
route
table
BGP
(mesh up to 100 nodes)
workload
workload
i.e. no in-host
broadcast
domain switch
No multi-host
broadcast
domain
overlay

65. Software Networking and Interfaces @mt165
Recap - Interface Types
● Loopback
● Dummy
● “real”
● (multiple L3 addresses)
● TAP / TUN
● vEth
● Vlan
● Macvlan / macvtap
● Ipvlan / ipvtap L2
● Ipvlan / ipvtap L3

66. Software Networking and Interfaces @mt165
Recap - Bridge Types
● Linux Bridge
● Open vSwitch (inc DPDK)
● Macvtap
● (netmap / VALE)
● (snabbswitch)
● (SR-IOV NIC, Cisco vNIC)

67. Software Networking and Interfaces @mt165
Thanks!
@mt165
Slides
Videos
Demo code
mt165.co.uk