Kubernetes at Datadog Scale - Ara Pulido
•
0 likes•60 views
Kubernetes networking is complex with several components including pod networking, service networking, and DNS. Pod networking requires assigning each pod a unique IP and enabling communication between pods using CNI plugins like Flannel or Calico which implement overlay networking. Service networking is done via kube-proxy using either iptables or IPVS mode, with IPVS being more scalable. DNS lookups are handled by CoreDNS to map service names to cluster IPs. Overall, Kubernetes networking takes work to understand but the ecosystem adapts quickly to issues.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Kubernetes at Datadog Scale - Ara Pulido
Similar to Kubernetes at Datadog Scale - Ara Pulido (20)
Recently uploaded
Recently uploaded (20)
Kubernetes at Datadog Scale - Ara Pulido
- 1. Kubernetes at Datadog scale Ara Pulido @arapulido
- 4. 17 major releases since 2015 +80k commits +2k contributors
- 5. Kubernetes popularity past 5 years
- 6. Kubernetes popularity past 5 years
- 7. Kubernetes v.s Mariah Carey
- 8. ● 2 years full time Kubernetes ● CKA; part of the team that created CKAD ● Writing apps for Kubernetes ● User of Kubernetes (15-25 nodes) ● Amateur youtuber (really!) https://dtdg.co/k8s_es My journey with Kubernetes
- 11. Dozens of clusters +1K nodes / cluster Multi-cloud
- 12. Survey: number of nodes / cluster Source: https://github.com/MagalixCorp/kubernetes-autoscale-survey
- 13. Let’s talk about the elephant in the room
- 15. POD NETWORKING DNS SERVICE NETWORKING Kubernetes networking 101
- 16. Pod 172.17.0.2 Pod 172.17.0.3 GET EVERY POD A UNIQUE IP MAKE SURE EVERY POD CAN TALK TO ANY OTHER POD IN THE CLUSTER Pod networking
- 17. Client 172.17.0.6 Pod 172.17.0.4 Pod 172.17.0.5 Service 10.107.122.52 kube-proxy Service networking
- 19. Pod networking
- 20. Pod 172.17.0.2 Pod 172.17.0.3 GET EVERY POD A UNIQUE IP MAKE SURE EVERY POD CAN TALK TO ANY OTHER POD IN THE CLUSTER
- 21. Pod 172.17.0.2 Pod 172.17.0.3 CNI Container Network Interface
- 23. Pod 10.244.2.1 10.244.1.0/24 10.244.0.0/16 10.244.2.0/24 10.244.0.0/16 cni0 flannel0 cni0 flannel0 Pod 10.244.1.1 Overlays (flannel, weavenet) I WAS HERE 192.168.0.1 192.168.0.2
- 25. 10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.0 10.244.0.0/16 dev flannel proto kernel scope link src 10.244.1.1 Overlays Src: 192.168.0.1 Dst: 192.168.0.2 Src: 10.244.1.1 Dst: 10.244.2.1 Pod 10.244.2.1 10.244.1.0/24 10.244.0.0/16 10.244.2.0/24 10.244.0.0/16 cni0 flannel0 cni0 flannel0 Pod 10.244.1.1 192.168.0.1 192.168.0.2
- 26. Pod 10.244.162.131/32 10.244.143.129/32calixxxx calixxxx Pod 10.244.162.131 192.168.0.1 192.168.0.2 BGP BGP BGP (Calico) 10.244.143.129
- 27. 10.244.162.131 dev cali12d4a061371 scope link 10.244.143.129 via 192.168.0.2 dev eth0 Calico (BGP) 10.244.143.129 dev cali12d4bf881 scope link 10.244.162.131 via 192.168.0.1 dev eth0 Pod 10.244.162.131/32 10.244.143.129/32calixxxx calixxxx Pod 10.244.162.131 192.168.0.1 192.168.0.2 BGP BGP 10.244.143.129
- 28. Direct Pod Routing (EKS, Lyft) Pod 10.0.0.3 Pod 10.0.0.4 SDN
- 29. Direct Pod Routing (EKS, Lyft) Pod 10.0.0.3 Pod 10.0.0.4 SDN DATADOG
- 30. Direct Pod Routing (EKS, Lyft) Pod 10.0.0.3 Pod 10.0.0.4 SDN
- 31. Be ready to become an “expert” in the CNI plugin of your choice (yay!!)
- 34. Client 172.17.0.6 Pod 1 172.17.0.4 Pod 2 172.17.0.5 Iptables 10.107.122.52 kube-proxy API Watch svc, endpoints iptables mode (default)
- 35. Client 172.17.0.6 Pod 1 172.17.0.4 Pod 2 172.17.0.5 Iptables 10.107.122.52 kube-proxy API Watch svc, endpoints iptables mode (default) I WAS HERE
- 36. Client 172.17.0.6 Pod 1 172.17.0.4 Pod 2 172.17.0.5 Iptables 10.107.122.52 kube-proxy API Watch svc, endpoints iptables mode (default)
- 37. Chain KUBE-SERVICES (2 references) KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- anywhere 10.107.122.52 /* default/nginx: cluster IP */ tcp dpt:www Chain KUBE-SVC-4N57TFCL4MD7ZTDA (1 references) target prot opt source destination KUBE-SEP-UZXILYFQQ2IZUWN5 all -- anywhere anywhere statistic mode random probability 0.50000000000 KUBE-SEP-43IWXJI557JKCKCF all -- anywhere anywhere Chain KUBE-SEP-43IWXJI557JKCKCF (1 references) target prot opt source destination KUBE-MARK-MASQ all -- 172.17.0.5 anywhere DNAT tcp -- anywhere anywhere tcp to:172.17.0.5:80 iptables mode (default)
- 38. GROWS LINEARLY WITH PODS AND SERVICES THERE IS NO ROOM FOR MORE FEATURES FOR EVERY CHANGE, THE WHOLE TABLE NEEDS TO BE RESYNC iptables mode (default)
- 39. IPVS mode GA SINCE 1.11
- 40. GA SINCE 1.11 IPVS mode DATADOG
- 41. ATOMIC CHANGES (NO NEED TO RESYNC EVERY RULE) O(1) COMPLEXITY DIFFERENT LOAD BALANCING ALGORITHMS (RR, LEAST CONN, etc.) ipvsadm -ln --tcp-service 10.107.122.52:80 Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.107.122.52:80 rr -> 172.17.0.4:80 Masq 1 0 37 -> 172.17.0.5:80 Masq 1 0 44 GA SINCE 1.11 IPVS mode
- 42. Source: https://www.projectcalico.org/comparing-kube-proxy-modes-iptables-or-ipvs/ iptables v.s. IPVS
- 43. Source: https://www.projectcalico.org/comparing-kube-proxy-modes-iptables-or-ipvs/ iptables v.s. IPVS
- 44. Many large deployments are already moving to IPVS and newer solutions are being developed (eBPF based)
- 45. DNS
- 49. ` Client DNS Cache CoreDNS DNS Local Cache
- 50. ` Client DNS Cache CoreDNS DNS Local Cache DATADOG
- 51. ` Client DNS Cache CoreDNS DNS Local Cache
- 52. NodeLocal DNSCache is now part of Kubernetes (Beta 1.15)
- 53. Take aways
- 54. The bad
- 55. KUBERNETES IS FLEXIBLE FOR DEVS, STILL COMPLEX FOR OPS YOU WILL HIT BUGS CNI PLUGINS WORK *VERY* DIFFERENTLY. LEARN YOURS.
- 56. The good
- 57. THE ECOSYSTEM PICKS UP QUICKLY DEVELOPMENT EXPERIENCE DOESN’T CHANGE (MUCH) CHANGES DON’T HAVE TO HAPPEN FOR ALL WORKLOADS DIRECTLY
- 59. Thank you! Ara Pulido @arapulido WE ARE HIRING