Kubernetes the Very Hard Way. Velocity Berlin 2019
•
5 likes•1,532 views
Kubernetes the Very Hard Way discusses making a Kubernetes cluster production-ready by addressing challenges that arise after initial setup. This includes making the control plane resilient and scalable through techniques like having separate etcd nodes and load balancing the API servers. Securing the control plane requires careful certificate management as certificates are used everywhere but can cause issues if not rotated. Efficient networking involves techniques like giving pods their own IP addresses and routing them natively instead of using overlays, to avoid performance overhead. Ingresses must also be set up to allow external access to clusters and applications in a scalable way.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Kubernetes the Very Hard Way. Velocity Berlin 2019
Similar to Kubernetes the Very Hard Way. Velocity Berlin 2019 (20)
More from Laurent Bernaille
More from Laurent Bernaille (7)
Recently uploaded
Recently uploaded (20)
Kubernetes the Very Hard Way. Velocity Berlin 2019
- 1. Kubernetes the Very Hard Way Laurent Bernaille Staff Engineer, Infrastructure @lbernail
- 2. lbernail Datadog Over 350 integrations Over 1,200 employees Over 8,000 customers Runs on millions of hosts Trillions of data points per day 10000s hosts in our infra 10s of k8s clusters with 50-2500 nodes Multi-cloud Very fast growth
- 3. lbernail Why Kubernetes? Dogfooding Improve k8s integrations Immutable Move from Chef Multi Cloud Common API Community Large and Dynamic
- 4. The very hard way?
- 5. It was much harder
- 6. lbernail This talk is about the fine print “Of course, you will need a HA master setup” “Oh, and yes, you will have to manage your certificates” “By the way, networking is slightly more complicated, look into CNI / ingress controllers”
- 7. lbernail What happens after “Kube 101” 1. Resilient and Scalable Control Plane 2. Securing the Control Plane a. Kubernetes and Certificates b. Exceptions? c. Impact of Certificate Rotation 3. Efficient networking a. Giving pod IPs and routing them b. Ingresses: Getting data in the cluster
- 8. lbernail What happens after “Kube 101” 1. Resilient and Scalable Control Plane 2. Securing the Control Plane a. Kubernetes and Certificates b. Exceptions? c. Impact of Certificate Rotation 3. Efficient networking a. Giving pod IPs and routing them b. Ingresses: Getting data in the cluster
- 9. Resilient and Scalable Control Plane
- 10. lbernail Kube 101 Control Plane kubelet kubectl etcd apiserver controllersscheduler Master in-cluster apps Service
- 11. lbernail Making it resilient etcd apiserver controllersscheduler kubelet kubectl Master etcd apiserver controllersscheduler Master etcd apiserver controllersscheduler Master LoadBalancer in-cluster apps Service
- 12. lbernail Kube 101 Control Plane kubelet kubectl etcd apiserver controllersscheduler Master in-cluster apps Service
- 14. lbernail apiserver controllersscheduler kubelet kubectl Master apiserver controllersscheduler Master apiserver controllersscheduler Master LoadBalancer in-cluster apps Service Single active Controller/scheduler etcd etcd
- 15. lbernail apiserver controllers kubelet kubectl apiserver apiserver LoadBalancer in-cluster apps Service Split scheduler/controllers controllers schedulers schedulers etcd
- 16. lbernail apiserver controllers kubelet kubectl apiserver apiserver LoadBalancer in-cluster apps Service Split etcd controllers schedulers schedulers etcd etcd events
- 17. lbernail apiserver controllers kubelet kubectl apiserver apiserver LoadBalancer in-cluster apps Service Sizing the control plane controllers schedulers schedulers 2x (3 or 5 nodes) disk + net ios X nodes RAM + net ios 2 nodes CPU 2 nodes CPU etcd etcd events
- 18. lbernail 1. Resilient and Scalable Control Plane 2. Securing the Control Plane a. Kubernetes and Certificates b. Exceptions? c. Impact of Certificate Rotation 3. Efficient networking a. Giving pod IPs and routing them b. Ingresses: Getting data in the cluster What happens after “Kube 101”
- 20. lbernail From “the hard way”
- 21. lbernail “Our cluster broke after ~1y”
- 22. lbernail Certificates in Kubernetes ● Kubernetes uses certificates everywhere ● Very common source of incidents ● Our Strategy: Rotate all certificates daily
- 23. lbernail Certificate management etcd apiserver Vault etcd PKIPeer/Server cert Etcd Client cert
- 24. lbernail Certificate management etcd apiserver controllers scheduler Vault etcd PKIPeer/Server cert Etcd Client cert kube PKI Apiserver/kubelet client cert Controller client cert Scheduler client cert kubelet Kubelet client/server cert
- 25. lbernail Certificate management etcd apiserver controllers scheduler Vault etcd PKIPeer/Server cert Etcd Client cert kube PKI Apiserver/kubelet client cert kube kv SA public key SA private key Controller client cert Scheduler client cert In-cluster app SA token kubelet Kubelet client/server cert
- 26. lbernail Certificate management etcd apiserver controllers scheduler apiservice webhook... Vault etcd PKIPeer/Server cert Etcd Client cert apiservice PKI Apiservice cert (proxy/webhooks) kube PKI Apiserver/kubelet client cert kube kv SA public key SA private key Controller client cert Scheduler client cert In-cluster app SA token kubelet Kubelet client/server cert
- 27. lbernail Certificate management etcd apiserver controllers scheduler apiservice webhook... Vault etcd PKIPeer/Server cert Etcd Client cert apiservice PKI Apiservice cert (proxy/webhooks) kube PKI Apiserver/kubelet client cert kube kv SA public key SA private key Controller client cert Scheduler client cert OIDC provider kubectl OIDC auth In-cluster app SA token kubelet Kubelet client/server cert
- 29. lbernail Kubelet: TLS Bootstrap apiserver controllers Vault kube PKI kube kv3- Get signing key admin 1- Create Bootstrap token 2- Add Bootstrap token to vault
- 30. lbernail Kubelet: TLS Bootstrap apiserver controllers Vault kube PKI kube kv kubelet 5- Verify RBAC for CSR creator 6- Sign certificate 1- Get Bootstrap token 2- Authenticate with token 4- Create CSR 7- Download certificate 8- Authenticate with cert 9- Register node 3- Verify Token and map groups
- 31. lbernail Kubelet certificate issue 1. One day, some Kubelets were failing to start or took 10s of minutes 2. Nothing in logs 3. Everything looked good but they could not get a cert 4. Turns out we had a lot of CSRs in flight 5. Signing controller was having a hard time evaluating them all CSR resources in the cluster Lower is better!
- 32. lbernail Why? Kubelet Authentication ● Initial creation: bootstrap token, mapped to group “system:bootstrappers” ● Renewal: use current node certificate, mapped to group “system:nodes“ Required RBAC permissions ● CSR creation ● CSR auto-approval CSR creation CSR auto-approval system:bootstrappers OK OK system:nodes OK
- 34. lbernail Temporary solution apiserver webhook Vault kube kv Get cert and key admin Create webhook with self-signed cert as CA Add self-signed cert + key to Vault One day, after ~1 year ● Creation of resources started failing (luckily only a Custom Resource) ● Cert had expired...
- 35. lbernail Take-away ● Rotate server/client certificates ● Not easy But, “If it’s hard, do it often” > no expiration issues anymore
- 38. lbernail Impact on etcdapiserver restarts etcd slow queries etcd traffic We have multiple apiservers We restart each daily Significant etcd network impact (caches are repopulated) Significant impact on etcd performances
- 39. Impact on Load-balancers apiserver restarts ELB surge queue Significant impact on LB as connections are reestablished Mitigation: increase queues on apiservers net.ipv4.tcp_max_syn_backlog net.core.somaxconn
- 40. lbernail Impact on apiserver clients apiserver restarts coredns memory usage ● Apiserver restarts ● clients reconnect and refresh their cache > Memory spike for impacted apps No real mitigation today
- 41. lbernail Impact on traffic balance Number of connections / traffic very unbalanced Because connections are very long-lived More clients => Bigger impact clusterwide 15MB/s 2.5MB/s 2300 connections 300 connections
- 42. lbernail Why? Simple simulation Simulation for 48h ● 5 apiservers ● 10000 connections (4 x 2500 nodes) ● Every 4h, one apiserver restarts ● Reconnections evenly dispatched Cause ● Cloud TCP load-balancers use round-robin ● Long-lived connections ● No rebalancing
- 44. Pod graceful termination apiserver kubelet containerd admin or controller Delete pod Stop Container with timeout “terminationGracePeriodSeconds” container Send SIGTERM After timeout, send SIGKILL
- 45. Restarts impact graceful termination apiserver containerd admin or controller Delete pod container Send SIGTERM After timeout, or Context Cancelled send SIGKILL Kubelet restarts end graceful termination Fixed upstream “Do not SIGKILL container if container stop is cancelled” https://github.com/containerd/cri/pull/1099 kubelet
- 46. Impact on pod readiness Issue upstream “pod with readinessProbe will be not ready when kubelet restart” https://github.com/kubernetes/kubernetes/issues/78733 kubelet restarts on “system” nodes (coredns + other services) coredns endpoints NotReady On kubelet restart ● Readiness probes marked as failed ● Pods removed from service endpoints ● Requires readiness to succeed again
- 47. lbernail Take-away Restarting components is not transparent It would be great if ○ Components could transparently reload certs (server & client) ○ Clients could wait 0-Xs to reconnect to avoid thundering herd ○ Reconnections did not trigger memory spikes ○ Cloud TCP load-balancers supported least-conn algorithm ○ Connections were rebalanced (kill them after a while?)
- 48. lbernail What happens after “Kube 101” 1. Resilient and Scalable Control Plane 2. Securing the Control Plane a. Kubernetes and Certificates b. Exceptions? c. Impact of Certificate Rotation 3. Efficient networking a. Giving pod IPs and routing them b. Ingresses: Getting data in the cluster
- 50. lbernail Throughput Trillions of data points daily Scale 1000-2000 nodes clusters Network challenges Latency End-to-end pipeline Topology Multiple clusters Access from standard VMs
- 51. Giving pods IPs & Routing them
- 52. lbernail From “the Hard Way” node IP Pod CIDR for this node
- 53. lbernail Small cluster? Static routes Node 1 IP: 192.168.0.1 Pod CIDR: 10.0.1.0/24 Routes (local or cloud provider) 10.0.1.0/24 => 192.168.0.1 10.0.2.0/24 => 192.168.0.2 Node 2 IP: 192.168.0.2 Pod CIDR: 10.0.2.0/24 Limits local: nodes must be in the same subnet cloud provider: number of routes
- 54. lbernail Mid-size cluster? Overlay Limits Overhead of the overlay Scaling route distribution (control plane) Node 1 IP: 192.168.0.1 Pod CIDR: 10.0.1.0/24 Node 2 IP: 192.168.0.2 Pod CIDR: 10.0.2.0/24 VXLAN VXLAN Tunnel traffic between hosts Examples: Calico, Flannel
- 55. lbernail Large cluster with a lot of traffic? Native pod routing Performance Datapath: no overhead Control plane: simpler Addressing Pod IPs are accessible from ● Other clusters ● VMs
- 56. lbernail In practice On premise BGP Calico Kube-router Macvlan AWS Additional IPs on ENIs AWS EKS CNI plugin Lyft CNI plugin Cilium ENI IPAM GCP IP aliases
- 57. lbernail How it works on AWS eth1 agent Pod 1 Pod 2 kubelet cni containerd CRI CNI eth0 Attach ENI Allocate IPs Create veth ip 1 ip 2 ip 3 Routing rule “From IP1, use eth1” Routing eth0 ip 1
- 58. lbernail Address space planning Pod Cidr: /24 ● /24 leads to inefficient address usage ● sig-network: remove contiguous range requirement for CIDR allocation ● But also ○ Address space for node IPs (another /20 per cluster for 4096 nodes) ○ Service IP range (/20 would make sense for such a cluster) ● Total: 1 /15 for pods, 2 /20 for nodes and service! pod cidr 8bitsnode prefix: 12bits10. (8bits) 4bits Up to 255 pods per node Simple addressing Up to 4096 nodes 4 bits available Up to 16 clusters
- 59. lbernail Take-away ● Native pod routing has worked very well at scale ● A bit more complex to debug ● Much more efficient datapath ● Topic is still dynamic (Cilium introduced ENI recently) ● Great relationship with Lyft / Cilium ● Plan your address space early
- 60. Ingresses
- 61. lbernail Ingress: cross-clusters, VM to clusters A A A B B B C C D D Cluster 1 Cluster 2Classic (VM) C? C? B?
- 62. lbernail Master Kubernetes default: LB service External Client Load-Balancer pod pod pod kube-proxy kube-proxy kube-proxy NP NP NP Healthchecker data path health checks configuration (from watching ingresses on apiservers) service-controller
- 63. lbernail Master Inefficient Datapath & cross-application impacts Web traffic Load-Balancer web-1 web-2 web-3 kube-proxy kube-proxy kube-proxy NP NP NP Healthchecker data path health checks configuration (from watching ingresses on apiservers) service-controller kafka
- 64. lbernail Master ExternalTrafficPolicy: Local? Web traffic Load-Balancer web-1 web-2 web-3 kube-proxy kube-proxy kube-proxy NP NP NP Healthchecker data path health checks configuration (from watching ingresses on apiservers) service-controller kafka
- 65. lbernail L7-proxy ingress controller data path health checks configuration from watching ingresses/endpoints on apiservers (ingress-controller) from watching LoadBalancer services (service-controller) External Client Load-Balancer l7proxy l7proxy kube-proxy kube-proxy kube-proxy NP NP NP Heathchecker ingress-controller pod pod pod pod Create l7proxy deployments Update backends using service endpoints Master service-controller
- 66. lbernail Limits All nodes as backends (1000+) Inefficient datapath Cross-application impacts Alternatives? ExternalTrafficPolicy: Local? > Number of nodes remains the same > Issues with some CNI plugins K8s ingress > Still load-balancer based > Need to scale ingress pods > Still inefficient datapath Challenges
- 67. lbernail Our target: native routing External Client ALB pod pod pod Healthchecker data path health checks alb-ingress-controller configuration (from watching ingresses/endpoints on apiservers)
- 68. lbernail Limited to HTTP ingresses No support for TCP/UDP Ingress v2 should address this Remaining challenges Registration delay Slow registration with LB Pod rolling-updates much faster Mitigations - MinReadySeconds - Pod ReadinessGates
- 69. lbernail Workaround External Client Load-Balancer l7proxy l7proxy Heathchecker pod pod pod pod Not managed by k8s Dedicated nodes Pods in host network TCP / Registration delay not manageable > Dedicated gateways
- 70. lbernail Take-away ● Ingress solutions are not great at scale yet ● May require workarounds ● Definitely a very important topic for us ● The community is working on v2 Ingresses
- 71. Conclusion
- 72. lbernail A lot of other topics ● Accessing services (kube-proxy) ● DNS (it’s always DNS!) ● Challenges with Stateful applications ● How to DDOS <insert ~anything> with Daemonsets ● Node Lifecycle / Cluster Lifecycle ● Deploying applications ● ...
- 73. lbernail Getting started? “Deep Dive into Kubernetes Internals for Builders and Operators” Jérôme Petazzoni, Lisa 2019 https://lisa-2019-10.container.training/talk.yml.html Minimal cluster, showing interactions between main components “Kubernetes the Hard Way” Kelsey Hightower https://github.com/kelseyhightower/kubernetes-the-hard-way HA control plane with encryption
- 74. lbernail You like horror stories? “Kubernetes the very hard way at Datadog” https://www.youtube.com/watch?v=2dsCwp_j0yQ “10 ways to shoot yourself in the foot with Kubernetes” https://www.youtube.com/watch?v=QKI-JRs2RIE “Kubernetes Failure Stories” https://k8s.af
- 75. lbernail Key lessons Self-managed Kubernetes is hard > If you can, use a managed service Networking is not easy (especially at scale) The main challenge is not technical > Build a team > Transforming practices and training users is very important