Slides used at DevNexus2020

1. Ana Maria Mihalceanu
Afterlife tales
Troubleshooting
Containerized Applications
#DevNexus2020 @ammbra1508

2. I am Ana
Technical Lead @ IBM
Co-founder of Bucharest Software Craftsmanship
Community
HELLO!
#DevNexus2020 @ammbra1508

3. DevNexus2020 @ammbra1508
DEV PROD

4. Pod Phases
#DevNexus2020 @ammbra1508
PENDING RUNNING SUCCEEDED FAILED UNKNOWN

5. Deployment-ReplicaSet-Pod
#DevNexus2020 @ammbra1508
YAML
Deployment ReplicaSet
Selector
Template
spec.replicas = 2
POD
POD
APPLY
Selector
Pod Template
replicas = 2

6. DevNexus2020 @ammbra1508

7. Why am I forbidden access?
#DevNexus2020 @ammbra1508

8. Role Base Access Control (RBAC)
#DevNexus2020 @ammbra1508
Resources
(Deployments, Pods,
Secrets, Configmaps
etc)
Actions (get, create,
list, watch etc)
API versions (core,
apps, extensions)
ClusterRole ClusterRoleBinding
Role RoleBinding
namespace
Cluster-wide
Subjects
User
ServiceAccount
Group

9. DevNexus2020 @ammbra1508

10. Authorization issues (403)
#DevNexus2020 @ammbra1508
How to validate permissions for a user?

11. Authorization issues - Recommendations
#DevNexus2020 @ammbra1508
Administrators check API access with user impersonation
kubectl auth can-i list deployment
--as=<user_name> --namespace=<name>
How to validate permissions for a user?

12. Authorization issues - Recommendations
#DevNexus2020 @ammbra1508
Check the audit logs
Grant proper role or cluster role
kubectl create clusterrolebinding <binding-name>
--clusterrole=<role_name> --user=<user_name>
[--group=<group_name>]
How to validate permissions for a user?

13. What if my Pod is not feeling well?
#DevNexus2020 @ammbra1508

14. Health checks – The good
#DevNexus2020 @ammbra1508
Readiness probe – when
the container is available
for accepting traffic
Liveness probe – when to
restart a container

15. Health checks – The (not so) good
#DevNexus2020 @ammbra1508
What happens if a
readiness probe
gives up ?
no traffic routed to
the pod
What happens if a
liveness probe
gives up ?
pod will be
restarted
Readiness probe Liveness probe

16. Readiness probe – The not so good
#DevNexus2020 @ammbra1508
What if there is a small temporary increase in
latency to one dependent service?

17. #DevNexus2020 @ammbra1508
Readiness probe – Recommendation
Set the timeout longer than the maximum response time for
that dependency.
What if there is a small temporary increase in
latency to one dependent service?

18. #DevNexus2020 @ammbra1508
Readiness probe – The not so good
Is frequency of the readiness probe impacting
the failureThreshold count?

19. #DevNexus2020 @ammbra1508
Readiness probe – Recommendation
Depending on the periodSeconds parameter you may want to
increase the failureThreshold count.
Is frequency of the readiness probe impacting
the failureThreshold count?

20. #DevNexus2020 @ammbra1508
Liveness probe – The not so good
How can I know that the liveness probe
verifies the responsiveness of an application?

21. #DevNexus2020 @ammbra1508
Liveness probe – Recommendation
Regularly (exercise and) evaluate deployments, restarts, and
failures as part of running an application.
How can I know that the liveness probe
verifies the responsiveness of an application?

22. #DevNexus2020 @ammbra1508
Liveness probe – The not so good
What if the container startup-latency changes
over time?

23. #DevNexus2020 @ammbra1508
Liveness probe – Recommendation
Set initialDelaySeconds and timeout conservatively.
What if the container startup-latency changes
over time?

24. How you manage disruptions in
Kubernetes?
#DevNexus2020 @ammbra1508

25. Involuntary Disruptions- Recommendations
#DevNexus2020 @ammbra1508
Replicate your application if you need higher availability, use a
multi-zone cluster and/or check pod requests and resources.
How to mitigate involuntary disruptions?

26. Voluntary Disruptions- Recommendations
#DevNexus2020 @ammbra1508
Using PodDisruptionBudget.
How to mitigate voluntary disruptions?

27. DevNexus2020 @ammbra1508

28. Somehow my Pod is running...something
#DevNexus2020 @ammbra1508

29. Recommendations
#DevNexus2020 @ammbra1508
Get the pod’s YAML
kubectl get pod <pod> --namespace=<name>
-o yaml > <resource_def>.yaml
How to validate what is running inside a pod?

30. Recommendations
#DevNexus2020 @ammbra1508
Get the pod’s YAML
Validate it locally
kubectl apply –f <resource_def>.yaml --validate
How to validate what is running inside a pod?

31. Recommendations
#DevNexus2020 @ammbra1508
kubectl exec –it <pod> --namespace=<name>
-- /bin/bash
What if I want to see what is inside my pod?

32. DevNexus2020 @ammbra1508

33. How can I replace images?
#DevNexus2020 @ammbra1508

34. Recommendations
#DevNexus2020 @ammbra1508
If Image tag has changed but your deployment
uses the old tag

35. Recommendations
#DevNexus2020 @ammbra1508
kubectl set image deployment/<deployment_name>
image=image:new_tag --record --namespace=<name>
If Image tag has changed but your deployment
uses the old tag

36. Recommendations
#DevNexus2020 @ammbra1508
Add your own label and patch the deployment.
If Image tag has not changed

37. DevNexus2020 @ammbra1508

38. Is a perfect Job ?!
#DevNexus2020 @ammbra1508

39. Recommendations
#DevNexus2020 @ammbra1508
Suspend the job
Set a backofflimit to 0 in combination with restartPolicy: Never and
concurrencyPolicy: Forbid
Stop the cronjob

40. A positive attitude
can change
anything!
#DevNexus2020 @ammbra1508

41. Samples location :
https://github.com/ammbra/troubleshooting
Thank YOU!
#DevNexus2020 @ammbra1508