Kubernetes Best Practices (Application Developer's View)

1. Kubernetes Best Practices
Akash Agrawal
Twitter: @akkiagrawal29
LinkedIn: www.linkedin.com/in/akash-agrawal-58a97813

2. 2
Kubernetes Best Practices
● Container Best Practices
● Kubernetes Best Practices
● Application Design & Architecture Best
Practices/Considerations

3. Container Best Practices

4. 4
Building Container Images
● Keep your Container Images Small & Secure
● Why:
Performance (Time to Build/Push/Pull Container Images)
Security (Less Attack Surface Area for Vulnerabilities)
● How:
Use Small Base Images
Follow Docker Best Practices for writing Docker Files
Don't blindly use Arbitrary Images from Docker Hub
Builder Pattern/Docker Multi Stage Builds
Vulnerability Scanning
● Impact:
Think about Micro Service Environment, CI/CD Pipelines, Build
Frequencies (Daily/Hourly) etc.

5. 5
Container Internals
● Use a Non-Root User Inside the Container
● Why:
Extra Layer of Security, avoiding disaster in case of Container
Escape
● How:
PodSpec ---> SecurityContext ---> runAsNonRoot
● Impact:
Extra Layer of Security
● Explore:
https://medium.com/@mccode/processes-in-containers-should-not-
run-as-root-2feae3f0df3b

6. Kubernetes Best Practices

7. 7
Organizing Kubernetes Workloads
● Organize K8S workloads with Namespaces
● Why:
Organizing workloads
Helpful in Resource Utilization
Security
● How:
Designed to keep different project environments like
Dev/QA/UAT/Build/Prod etc
Consider Namespace Granularity based on (Small Team/Rapidly
Growing Team/Large Company/Large Enterprises/Number of
Projects/Project Environments/Services/MicroServices/Resource
Quota/Access Control)
● Impact:
Think about a Project with multiple environments separated in
different namespaces (K8S Artifects Isolation/Resource
Isolation/Access Control & Policies)

8. 8
Setting up Health Checks
● Health check is the simple way, to let the
system know, that instance of your app is
working or not.
● Goal: Send the traffic to only those instances,
who are live & ready.
● Using Liveness & Readiness Probes

9. 9
Setting up Health Checks .. cont.
● Why:
Think about your Application takes minute to warm up
Think about the Use-Cases Pod Re-scheduling, Scaling Out/In etc.
● How:
3 Types of Probes: HTTP/TCP/Command
Ways to Configure Probes: initialDelaySeconds, periodSeconds,
timeoutSeconds etc.
● Impact:
Designing Robust & Smart System
● Explore:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-
liveness-readiness-probes/

10. 10
Setting Resource "Requests" &
"Limits"
● Control Node Resource Allocation (like
CPU/Memory) to the Containers
● Why:
For Scheduling Requirements
● How:
define this at Container/Pod Level & Namespace Level
requests.cpu, requests.memory, limits.cpu, limits.memory
also explore Resource Quota & Limit Range

11. 11
Setting Resource "Requests" &
"Limits"
● Impact:
Tuning the Node & Cluster Resource Allocation for Applications
● Explore:
https://kubernetes.io/docs/concepts/configuration/manage-compute-
resources-container/
https://kubernetes.io/docs/concepts/policy/resource-quotas/

12. 12
Graceful Termination of Applications
● Handling Graceful Termination (e.g. Closing
Connections/Writing Data etc) is Key for any
distributed system.
● Why:
K8S terminates pods in various scenarios (e.g. in case of Rolling
Updates, Node Draining, Resource Outage on the Worker Node,
Scaling In etc.) ……. so make sure your application terminates
gracefully.
● How:
terminationGracePeriodSeconds
preStop Hook

13. 13
Graceful Termination of Applications
● Impact:
Graceful Termination of Applications
● Explore:
https://pracucci.com/graceful-shutdown-of-kubernetes-pods.html

14. 14
Kubernetes Cluster Environment
● Create Administrative Boundaries
Namespaces
AccessControl / RBAC
Resource Quota & Limit Range
Pod Security Policy / Pod Security Context
Network Policy
● Secrets:
Pay attention (if Secrets are stored or accessible in plain text/who
can access which secrets/expiry or rotation of secrets etc.)
● Access:
Limit SSH access to K8S Nodes
Limit Access/Audit to K8S Dashboard / Kubectl / K8S API Access

15. Application Design & Architecture
Best Practices/Considerations

16. 16
Application Design & Architecture
● Follow “One Process Per Container” Principle
● Multi Container Pods (e.g. for Watchers/Proxy
etc.)
● Use Labels & Selectors Carefully
● Follow "Least Privileges" Principle
● Don't focus on "application restart" in case of
failures, instead focus on "Crashing it Clean"

17. 17
Application Design & Architecture
● Don't use ServiceType=LoadBalancer blindly,
in most cases "Ingress" is good enough.
● Use “Init Containers” for bootstrapping instead
of Sidecar design pattern
● Use the “record” option for easier rollbacks
● Don’t use :latest or no tag for images
● Always use Deployments instead bare pods

18. 18
Application Design & Architecture
● Explore:
https://matthewpalmer.net/kubernetes-app-
developer/articles/multi-container-pod-design-
patterns.html

19. 19
Others
● In Cloud – look for Regional & Zonal Clusters
● Look for Service Mesh in MicroService
Environment (e.g. Istio)
● Look for Managed K8S Platforms features
● Explore CNCF Projects e.g.
FluentD/OpenTracing/Promotheous etc.

20. Thanks