Sarah Christoff - Systems Engineer, Cloudreach Moustapha Gueye - Solutions Architect, Docker Distributed infrastructure is necessary with the growing demands in our everyday applications. But with distributed infrastructure comes distributed problems, and the additional complexity these systems add make our systems teams want to pull their hair out. This talk covers the evolving approaches to operations for container infrastructure. By examining real-world examples from Swarm and Kubernetes, we'll dive into outages that others have had and identify the learning opportunities that we can apply to our own systems going forward.

1. Tales of Woe
Wisdom from Ops-Failures

2. Cloud Systems Engineer
Cloudreach
puponalaptop.com
Sarah Christoff

3. Solutions Architect
Docker
Moustapha Gueye

4. - Discovering an infra-failure
- Troubleshooting an Opps-liciation
- Finding your piece
Lesson Plan

5. Discovering an infra-failure

6. Case One: Master Your Infrastructure

7. Case One: Alerts
Sam
e !
Con er
C U o d A ag

9. Case One: Alerts

11. Suspect: Resources
Case One: Troubleshooting

12. Suspect:
Build Nodes
Case One: Troubleshooting

13. Suspect: Docker Enterprise
Case One: Troubleshooting

15. Suspect:
CPU Ready
Case One: Troubleshooting

16. Suspect: CPU Ready
Case One: Troubleshooting

17. Suspect: CPU Ready
Case One: Troubleshooting
root@ucpleader# uptime .
16:59:30 up 7 days, 19:55, 2 users, load average: 16.57, 16.51, 15.47

18. Suspect: CPU Ready
Case One: Troubleshooting

19. Suspect: CPU Ready
Case One: Troubleshooting

21. Suspect: vMotion
Case One: Troubleshooting

22. Suspect: vMotion & CPU Ready
Abettor: Oversubscription
Case One: Troubleshooting
Node Node
Node

23. Guilty: CPU Ready & vMotion
Abettor: Overloaded Hosts
Case One: Result

24. Guilty: CPU Ready & vMotion
Abettor: Overloaded Hosts
Case One: Result

25. Case One:
Solution

26. - Use the devops concept "pet vs cattle".
Case One: Lessons Learned

27. - Don’t oversubscribe your infrastructure.
Case One: Lessons Learned

29. Troubleshooting an Oops-lication

30. Case Two: Alerts

31. Case Two: Alerts

32. Suspect: Resources
Case Two: Troubleshooting

33. Suspect: Java
Case Two: Troubleshooting

34. Case Two: Troubleshooting

35. Case Two: Troubleshooting
Suspect: Resource Limitations

36. Case Two:
Solution

37. Case Two: Solution
Guilty: Resource Limitations

38. Case Two: Troubleshooting
Suspect: Resource Limitations

39. resources:
limits:
cpus: '1’'
memory: 1g
reservations:
cpus: '0.5'
memory: 1g
containers:
name: foo
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 500m
memory: 1Gi

41. - Perform tests to understand the memory
requirements of your application before
placing it into production.
Case Two: Lessons Learned

42. - Limit the amount of resources your
container can use
Case Two: Lessons Learned

43. - Use service-level constraints and node
labels to ensure that the application runs
only on hosts with enough resources
Case Two: Lessons Learned

44. Networking Nightmares

45. Case Three: Networking Nightmare

46. Case Three: Alerts

47. Case Three: Alerts

48. Suspect: Availability Zones
Case Three: Troubleshooting

49. Suspect: Allocation Conflict
Case Three: Troubleshooting

50. Case Three:
Solution

51. Guilty :
Allocation Conflict
Case Three: Solution

52. - docker0 specificity of bridge network range
is one thing, dtr overlay network is also
something to watch out for!
Case Three: Lessons Learned

53. - Keep as little pets as possible
- Set resource limitations
- Double check with your network engineers
before deploying something
Summary

55. Thank you!