Kubernetes networking allows pods to communicate across nodes using IP-in-IP tunnels. When a pod makes a request to a service IP, the request is distributed to backend pods using iptables rules and connection tracking. DNS queries in pods are resolved by CoreDNS, which returns service IPs and pod IPs configured in Kubernetes.

1. The Journey to the
Kubernetes Networking

2. TOC
• Network setting
• Pod IP routing (same node)
• Pod IP routing (different node)
• Service IP routing
• DNS lookup

3. Prerequisite
• Container and Kubernetes fundamental
• Basic network routing and TCP/IP

4. NOT include
• CNI detail
• Calico detail
• Ingress
• Network policy
• Source code tracing

5. Environment
• RHEL 7.8
• Kernel 3.10.0-1127
• Kubernetes v1.18.2
• Docker 19.3.13
• Containerd 1.3.7-3.1
• Etcd v3.3.12
• Calico v3.14.0

6. Scenario
• 1 namespace, 3 pods
• 2 pods are nginx server,
the alpine pod is client

7. Network setting

8. Container basis
• In Linux, container = namespace + cgroup
• Namespace: environment isolation (E.g., network & pid namespace)
• Cgroup: resource constraint (E.g., CPU & memory limit)

12. Container network
• In early years, container or pod IP is managed by container runtime
• Recently, pod IP is managed by CNI plugin in k8s.
• Different pods have different network namespace, different container
share the same namespace in same pod.

13. Kubernetes Network Model
• Pods can communicate with all other pods on any other node without
NAT
• Agents on a node (e.g. system daemons, kubelet) can communicate
with all pods on that node
https://kubernetes.io/docs/concepts/services-networking/

14. CNI and CNI plugins
• The CNI protocol is based on execution of binaries invoked by the
container runtime. CNI defines the protocol between the plugin
binary and the runtime.
• A CNI plugin is responsible for configuring a container’s network
interface in some manner.
https://www.cni.dev/docs/spec/#section-2-execution-protocol

15. K8s pod & service IP range
CIDR Range for Pods in cluster
CIDR Range for Services in cluster
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/

16. CNI
IPAM: IP Address Management

17. Log
Kubelet log
Calico log

18. IP address in pod & host
IP interface in pod
IP interface on host

19. Network namespace of container
container PID

20. Pod IP Routing (same node)

21. on w05t query host 10.233.112.167

23. Determine the routing in namespace

24. Determine the routing by enabling proxy_arp
https://wiki.debian.org/BridgeNetworkConnectionsProxyArp
https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.bridging.proxy-arp.html

25. Determine the routing on host

26. Pod IP Routing (different node)

27. on w05t query host 10.233.112.167

29. IP in IP tunnel

30. on w04t query host 10.240.205.39 and not port 53

31. on w04t query host 10.240.205.39 and not port 53

33. Service IP routing

34. Service IP allocation
• Kubernetes assigns this Service an IP address which is used by the
Service proxies (E.g., kube-proxy or CNI plugin).
• Kube-proxy in iptables mode install the iptables rule and continuously
update the corresponding endpoints on every worker node.
https://kubernetes.io/docs/concepts/services-networking/service/

35. Scenario
• Create 1 deployment with 3 replicas and expose it as a service

36. NAT and conntrack when query service IP
on w05t query host 10.233.112.167
DNAT
SNAT & conntrack

37. Iptables load-balance probability
• 1st endpoint:
1
3
• 2nd endpoint:
2
3
×
1
2
=
1
3
• 3rd endpoint:
2
3
×
1
2
× 1 =
1
3
https://github.com/kubernetes/kubernetes/issues/37932

38. DNS lookup

39. DNS resolution for service and pod
• DNS queries may be expanded using the Pod's /etc/resolv.conf.
Kubelet sets this file for each Pod and passes DNS resolver (CoreDNS)
information to each container.
• CoreDNS will reply to DNS queries based on IP of the services and
pods of Kubernetes.
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
CoreDNS config

40. on w05t query host 10.233.99.52 or host 10.233.88.44

41. DNS resolution for others
• Any queries that are not within the cluster domain of Kubernetes will
be forwarded to predefined resolvers (/etc/resolv.conf).
https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/
CoreDNS config

42. DNS cache
• NodeLocal DNSCache improves Cluster DNS performance by running
a DNS caching agent on cluster nodes as a DaemonSet.
• Pods will reach out to the DNS caching agent running on the same
node, thereby avoiding iptables DNAT rules and connection tracking.
https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/
nodelocaldns config

43. on w05t query host 169.254.25.10 and port 53

44. Proxy ARP Reference
• https://www.tkng.io/cni/calico/
• https://projectcalico.docs.tigera.io/reference/faq#why-does-my-
container-have-a-route-to-16925411
• https://www.sobyte.net/post/2022-06/proxy-arp-calico/
• https://developpaper.com/disclosure-of-calico-network-
communication-principle/
• https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-
2-a07f5bf0ff14