10 ways to shoot yourself in the foot with kubernetes, #9 will surprise you!
•
2 likes•986 views
Kubernetes is a very powerful and complicated system, and many users don’t understand the underlying systems. Come learn how your users can abuse container runtimes, overwhelm your control plane, and cause outages - it’s actually quite easy! In the last year, we have containerized hundreds of applications and deployed them in large scale clusters (more than 1000 nodes). The journey was eventful and we learned a lot along the way. We’ll share stories of our ten favorite Kubernetes foot guns, including the dangers of cargo culting, rolling updates gone wrong, the pitfalls of initContainers, and nightmarish daemonset upgrades. The talk will present solutions we adopted to avoid or work around some these problems and will finally show several improvements we plan deploy in the future.
Recommended
More Related Content
More from Laurent Bernaille
More from Laurent Bernaille (16)
Recently uploaded
Recently uploaded (20)
10 ways to shoot yourself in the foot with kubernetes, #9 will surprise you!
- 14. @lbernail @roboll_ Pod A Pod B Pod C App
- 15. @lbernail @roboll_ Pod B Pod C App Pod D
- 16. @lbernail @roboll_ Pod B Pod C App Pod D
- 17. @lbernail @roboll_ Pod B Pod C App Pod D
- 20. @lbernail @roboll_ 52 Completed 1 ErrImagePull 155 Evicted 7 ImagePullBackOff 7 Init:ErrImagePull 4 Init:Error 49 Init:ImagePullBackOff 16 Pending 1267 Running 2 Terminating
- 21. @lbernail @roboll_ 52 Completed 1 ErrImagePull 155 Evicted 7 ImagePullBackOff 7 Init:ErrImagePull 4 Init:Error 49 Init:ImagePullBackOff 16 Pending 1267 Running 2 Terminating
- 44. @lbernail @roboll_ • • • nodegroup.yaml --- instanceType: c5.4xlarge deployment.yaml --- resources: requests: cpu: 15 memory: 30Gi limits: cpu: 15 memory: 30Gi
- 53. @lbernail @roboll_ apiVersion: apps/v1 kind: Deployment spec: replicas: 60
- 54. @lbernail @roboll_ apiVersion: apps/v1 kind: Deployment spec: replicas: 60 apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: myapp minReplicas: 1 maxReplicas: 10 targetCPUUtilizationPercentage: 50
- 55. @lbernail @roboll_ apiVersion: apps/v1 kind: Deployment spec: replicas: 60 apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: myapp minReplicas: 1 maxReplicas: 10 targetCPUUtilizationPercentage: 50
- 56. @lbernail @roboll_ apiVersion: apps/v1 kind: Deployment spec: replicas: 60 apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: myapp minReplicas: 1 maxReplicas: 10 targetCPUUtilizationPercentage: 50
- 59. @lbernail @roboll_ ● 100+ nodes Cassandra cluster ● Deployed fine ● Broken the following morning
- 60. @lbernail @roboll_ ● 100+ nodes Cassandra cluster ● Deployed fine ● Broken the following morning > 25% pods pending: “Volume affinity issue”
- 61. @lbernail @roboll_ ● 100+ nodes Cassandra cluster ● Deployed fine ● Broken the following morning > 25% pods pending: “Volume affinity issue” > 25% nodes had been deleted + local volumes bound to deleted nodes
- 62. @lbernail @roboll_ ● 100+ nodes Cassandra cluster ● Deployed fine ● Broken the following morning > 25% pods pending: “Volume affinity issue” > 25% nodes had been deleted + local volumes bound to deleted nodes
- 70. @lbernail @roboll_ ps auxf | grep -c defunct 16018
- 71. @lbernail @roboll_ ps auxf | grep -c defunct 16018
- 75. @lbernail @roboll_ Symptoms ● Deployment slow ● Pod took 1+mn to start
- 76. @lbernail @roboll_ Symptoms ● Deployment slow ● Pod took 1+mn to start Cause ● Load on nodes running deployment high
- 77. @lbernail @roboll_ Symptoms ● Deployment slow ● Pod took 1+mn to start Cause ● Load on nodes running deployment high ● Because IOs are being rate limited
- 79. @lbernail @roboll_ ● Nodes were running coredns pods ● An app started doing 5000+ dns queries per second ● coredns logs filled the disk
- 87. @lbernail @roboll_ ips = $(aws ec2 describe-instances --filter=consul) kubectl update endpoints consul $ips
- 88. @lbernail @roboll_ ips = $(aws ec2 describe-instances --filter=consul) kubectl update endpoints consul $ips