Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Recently uploaded
Recently uploaded (20)
Kubernetes Resource Mis-management - What not to do
- 2. About me Robusta.dev co-founder Developer Low level stuff Marketing fun Natan Yellin aantn
- 3. What we'll talk about 3 reasons scheduling is hard 2 warm-up questions 4 misconceptions and mistakes Natan Yellin aantn
- 5. Natan Yellin aantn pods nodes
- 6. Natan Yellin aantn pods nodes
- 7. Natan Yellin aantn pods nodes
- 8. Natan Yellin aantn pods nodes 2 cpu
- 9. Natan Yellin aantn pods nodes 2 cpu 1 cpu
- 11. Natan Yellin aantn pods nodes 2 cpu 1 cpu
- 12. Natan Yellin aantn pods nodes 2 cpu 1 cpu a request is an estimate of the resources a pod will need
- 13. Natan Yellin aantn if only things were that simple...
- 14. Natan Yellin aantn challenge 1 inaccurate predictions
- 15. Natan Yellin aantn our estimated cpu usage nodes A
- 16. Natan Yellin aantn actual cpu usage nodes A
- 17. Natan Yellin aantn challenge 2 high switching costs
- 18. Natan Yellin aantn switching costs nodes B A
- 19. Natan Yellin aantn spin up a new node nodes B
- 20. Natan Yellin aantn terminate existing pod nodes B
- 22. Natan Yellin aantn start new pod nodes B
- 23. Natan Yellin aantn challenge 3 its not just cpu
- 24. Natan Yellin aantn cpu request pods nodes 2 cpu 1 cpu
- 25. Natan Yellin aantn cpu request memory request pods nodes 2 cpu 1 cpu 1 gb 4 gb
- 26. Natan Yellin aantn cpu request memory request other resources (gpu, huge pages, etc) pods nodes 2 cpu 1 cpu 1 gb 4 gb 0.5 gpu 1 gpu
- 27. Natan Yellin aantn the real world
- 28. Natan Yellin aantn the real world
- 29. Natan Yellin aantn 2 warm up questions
- 30. A Natan Yellin aantn Pod A requested 1 CPU but is actually using 2 CPU actual usage = 2 CPU request = 1 CPU
- 31. B A Natan Yellin aantn Pod A requested 1 CPU but is actually using 2 CPU Pod B is created and requests 1 CPU actual usage = 2 CPU request = 1 CPU request = 1 CPU
- 32. B A Natan Yellin aantn Will pod B be scheduled? actual usage = 2 CPU request = 1 CPU request = 1 CPU
- 33. A Natan Yellin aantn Pod B will be scheduled! kubernetes does not look at actual usage when scheduling B
- 34. B A Natan Yellin aantn nodes After scheduling, Pod A tries to use all the CPUs on the node.
- 35. B A Natan Yellin aantn nodes Will Pod B get the 1 CPU it requested?
- 36. B A Natan Yellin aantn nodes Requests are enforced at runtime! Pods are always guaranteed their request!
- 37. B A zz Natan Yellin aantn nodes If B isn't doing anything...
- 38. B A zz Natan Yellin aantn nodes If B isn't doing anything... we'll let A get away with it. No harm done!
- 39. A Natan Yellin aantn nodes Assuming this doesn't break A's limits
- 40. Natan Yellin aantn 4 misconceptions and mistakes
- 41. A pod will not be CPU throttled if CPU is under the limit Natan Yellin aantn
- 44. Natan Yellin aantn cpu limits 1 second
- 45. Natan Yellin aantn cpu limits 100 ms 100 ms 100 ms 100 ms 100 ms 100 ms
- 46. Natan Yellin aantn cpu limits 100 ms 100 ms 100 ms 100 ms 100 ms 100 ms
- 47. Natan Yellin aantn cpu limits 100 ms 100 ms 100 ms 100 ms 100 ms 100 ms 100 ms
- 48. Natan Yellin aantn cpu limits 100 ms 100 ms 100 ms 100 ms 100 ms 100 ms 50 ms
- 49. Natan Yellin aantn cpu limits 100 ms 100 ms 100 ms 100 ms 100 ms 100 ms 50 ms
- 50. Natan Yellin aantn cpu limits 100 ms 100 ms 100 ms 100 ms 100 ms 100 ms
- 51. Natan Yellin aantn cpu limits 100 ms 100 ms 100 ms 100 ms 100 ms 100 ms throttling
- 52. Learn more Natan Yellin aantn link
- 53. If my node has 4 CPUs then I can fill up 4 CPUs with requests Natan Yellin aantn
- 57. When you go over a memory limit you get OOMKilled Natan Yellin aantn
- 60. Natan Yellin aantn cpu memory
- 61. Natan Yellin aantn a pod is using more memory than expected it doesn't have a limit the node runs out of memory what pod gets killed?
- 62. Natan Yellin aantn there is no simple answer! *but in the future this will get better: https://kubernetes.io/blog/2021/11/26/qos-memory-resources/
- 63. Natan Yellin aantn always set memory limits set request=limit so pods will only be killed for their own behaviour
- 64. One pod can't cause another to stop running Natan Yellin aantn
- 66. Natan Yellin aantn A A B B comes along and evicts A
- 67. Natan Yellin aantn A A B Having enough capacity in the cluster doesn't mean everything can run! e.g. anti-affinities
- 68. Natan Yellin aantn A A B
- 70. Other recent talks Natan Yellin aantn Building a runbook automation system for Prometheus and Kubernetes
- 71. Thank you! Follow on LinkedIn for more content Natan Yellin Natan Yellin aantn