Ziglu runs kubernetes and Istio in production, and it adds up to a complicated system. In order to debug deeply one day, Matt had to dive into the inner workings of Docker networking, virtual switches, and sidecar containers. This talk will share what he learned about off these systems, in the hope it’ll save other people time!

1. Networks, Linux,
Containers, and Pods
Matt Turner
Docker Meetup, London UK | January 2020 @mt165 | mt165.co.uk

2. Interfaces and Bridges

4. eth0
enp0s2

5. eth0
enp0s2
192.168.0.10

6. eth0
enp0s2
192.168.0.10
ping
connect(“google.com”);

7. eth0
enp0s2
192.168.0.10
nginx
bind(“*:80”);

8. eth0
192.168.0.10
nginx
eth1
172.16.0.10
bind(“192.168.0.10:80”);

9. eth0
192.168.0.10
nginx
bind(“172.16.0.10:80”);
eth1
172.16.0.10

10. eth0
192.168.0.10e1000

11. e1000
eth0
192.168.0.10
hardware software

12. eth0
192.168.0.10e1000
foo0

13. eth0
192.168.0.10e1000
foo0

14. eth0
192.168.0.10e1000
foo0
packetd

15. eth0
192.168.0.10e1000
tun0
packetd172.16.0.10

16. eth0
192.168.0.10e1000
ping
tun0
packetd172.16.0.10

17. eth0
192.168.0.10e1000
tap0
packetd
172.16.0.10
DE:AD:BE:EF:00:00

18. eth0
e1000
tap0
packetd
br0

19. br0
eth0
e1000
tap0
packetd

20. br0
eth0
e1000
tap0
packetd
br0 192.168.0.10

21. br0
eth0
e1000
tap0
br0
192.168.0.10
qemu [+ kvm]

22. br0
eth0
e1000
tap0
br0
192.168.0.10
e1000
eth0
qemu
192.168.0.11
C0:FF:EE:C0:FF:EE

23. br0
eth0
e1000
br0
192.168.0.10

24. br0
eth0
e1000
br0
192.168.0.10
nginx
ping

25. br0
eth0
e1000
br0
192.168.0.10
nginx
ping

26. br0
eth0
e1000
br0
192.168.0.10
nginx
ping
veth0 veth1

27. br0
eth0
e1000
br0
192.168.0.10
nginx
ping
veth0 veth1
172.16.0.1 172.16.0.2

28. br0
eth0
e1000
br0
192.168.0.10
nginx
ping
veth0
veth1 172.16.0.2

29. br0
eth0
e1000
br0
192.168.0.10
nginx
ping
veth0
veth1
172.16.0.2
ftpd

30. br0
eth0
e1000
br0
192.168.0.10
nginx
ping
veth0
eth0
172.16.0.2
ftpd

31. br0
eth0
e1000
br0
192.168.0.10
nginx
ping
eth0
172.16.0.2
ftpd

32. “Containers”
nginx
nginx
net

33. “Containers”
nginx
nginx
mnt
pid
uts
user
ipc
net

34. Kubernetes Pods
nginx
nginx
mnt
pid
uts
user
ipc
net
proxy
envoy
mnt
pid

35. Kubernetes Pods
nginx
nginx
mnt
uts
user
ipc
net
proxy
envoy
mnt
pid

36. Kubernetes Pods
nginx
nginx
mnt
uts
user
ipc
net
proxy
envoy
mnt
192.168.0.42
eth0
lo
sockets
iptables
routes
pid

37. Kubernetes Pods
nginx
nginx
mnt
uts
user
ipc
net
proxy
envoy
mnt
192.168.0.42
eth0
lo
sockets
iptables
routes
:8080/tcp
pid

38. Kubernetes Pods
nginx
nginx
mnt
uts
user
ipc
net
proxy
envoy
mnt
192.168.0.42
eth0
lo
sockets
iptables
routes
:8080/tcp
pid

39. Sidecar Injection
uts
user
ipc
net
192.168.0.42
eth0
lo
sockets
iptables
routes
pid

40. Sidecar Injection
uts
user
ipc
net
192.168.0.42
eth0
lo
sockets
iptables
routes
docker.io/istio/proxyv2
/istio-iptables -p 15001 -u 1337 ...
pid
mnt

41. pid
Sidecar Injection
nginx
nginx
mnt
uts
user
ipc
net
docker.io/istio/proxyv2
envoy
mnt
192.168.0.42
eth0
lo
sockets
iptables
routes
:15001/tcp

45. foofoobarfoo
ContainerPort

46. foofoobarfoo
ContainerPort

47. foofoobarfoo
Cluster IP
ContainerPort
Port

48. foofoobarfoo
kube-proxy kube-proxy
Cluster IP
ContainerPort
Port
etcd
iptables

49. foofoobarfoo
kube-proxy kube-proxy
Cluster IP
iptables
TargetPort
ContainerPort
Port
etcd

50. foofoobarfoo
kube-proxy kube-proxy
Cluster IP
iptables
TargetPort
ContainerPort
Port
etcd
ipset

51. Thanks!
@mt165
Slides
Videos
Demo code
mt165.co.uk