Kubernetes has become the de facto Container orchestration platform over the last few years. For all the power and the scale that Kubernetes provides, it's still a complex platform to configure and manage. Managed Kubernetes service like GKE takes a lot of the pain out in managing Kubernetes. When we are using Kubernetes in production, it makes sense to follow the best practises around distributed system development targeted towards Kubernetes. In this session, I will talk about Kubernetes and GKE best practices around infrastructure, security and applications with specific focus on day-2 operations.

1. Best practises for using
Kubernetes in production
Sreenivas Makam
August 2, 2020

2. About myself
● Application modernization specialist at Google Cloud. Previously at Cisco
and few startups
● Interest areas - Containers, Kubernetes, Networking, Cloud native
technologies
● Author of “Mastering CoreOS”, published 2016. Reviewed many technology
books
● Docker Captain from Oct 15 - Mar 18
● Active blogger and Community speaker

3. Agenda
● What is Kubernetes?
● What makes Kubernetes unique?
● Kubernetes day 2 operations
○ Cluster Management
○ Application Design
○ Security
● Kubernetes ecosystem and tools

4. What is Kubernetes?

5. What is Kubernetes
Kubernetes is a portable, extensible,
open-source platform for managing
containerized workloads and services,
that facilitates both declarative
configuration and automation
https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

6. Kubernetes market share
https://sysdig.com/blog/sysdig-2019-container-usage-report/

7. https://kubernetes.io/docs/concepts/overview/components/
Kubernetes Architecture

8. What makes Kubernetes unique?

9. Kubernetes Uniqueness
● Declarative rather than imperative
● Extensible - custom resource, controllers, schedulers
● Meet the user where they are - (eg) read config, secrets from
applications
● Decouple distributed system application development
● Open source ecosystem friendly

10. Borrowed from:
https://www.digitalocean.com/communit
y/tutorials/imperative-vs-declarative-ku
bernetes-management-a-digitalocean-c
omic

11. Custom
authorization
, admission
control
Custom
resource(operator
pattern. eg: etcd,
prometheus)
Custom
scheduler
Custom
controller, works
with custom
resource
Network
plugin
Storage
plugin
Extend
kubectl
Kubernetes extensions

12. Controller(Standard,
custom)
(Standard eg: Replica set
controller, Deployment
controller)
API server
Standard Kubernetes
resources
Desired
State
Current
State
Kubernetes controller

13. Kubernetes operator
(eg: prometheus, etcd,
Spark, Airflow)
API server
Custom resources
Desired
State
Current
State
Operators manage the lifecycle of the custom application
Extensions -Kubernetes Operator

14. https://coreos.com/blog/the-prometheus-operator.html
Prometheus Operator

15. apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: mypod
image: redis
volumeMounts:
- name: foo
mountPath: "/etc/foo"
readOnly: true
volumes:
- name: foo
secret:
secretName: mysecret
apiVersion: v1
kind: Pod
metadata:
name: secret-env-pod
spec:
containers:
- name: mycontainer
image: redis
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: mysecret
key: username
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: password
restartPolicy: Never
Use Mount paths Use environment variables
Applications can consume config/secrets without knowledge of Kubernetes
Meet user where they are -Consume secrets in App

16. https://itnext.io/tutorial-basics-of-kubernetes-volumes-part-2-b2ea6f397402
Kubernetes provides portability
by decoupling
infrastructure(Storage,
networking) from the application
manifest
Decouple distributed system -Storage provisioning

17. Init container
(Clone git repo and
generate config)
App container
(Web server)
Pod
Execution sequence
Specialized containers that runs to completion before application containers in a pod can
get started. This enforces sequence.
Pod patterns -Init Containers

18. Pod patterns -Sidecar
Sidecar containers extend and enhance the “main” container
Other examples:
Istio envoy proxy
Monitoring
Database config

19. Pod patterns -Adapter
Adapter containers standardize and normalize output so that external services can access
interface in a standard way(eg: Prometheus adapter)

20. Pod patterns - Ambassador
Ambassador containers proxies a local connection to the world and hides the complexity to
access external service.
Examples:
Accessing different kinds of
cache based on environment
Client side service discovery
using different mechanisms

21. Single app defined using Dockerfile and
multiple apps done using deployment
Config map and secrets
Service abstraction
and discovery
Stateless containers, stateful
dataset where needed
Services provides
different options for
port bindings
Autoscaler support
is comprehensive
Centralized log management with
third party integrations possible
Autohealing
Many ways to create and
manage clusters(cloud
provider, kops, kubeadm)
Map Twelve factor apps to Kubernetes

22. Kubernetes day 2 Operations

23. Kubernetes Day 2 operations -Best practises
● Cluster management
○ Multi-tenant design(clusters/namespaces, multi-cluster handling, zonal/regional), Upgrade
policy(node and containers, pod disruption budget), Ingress(load balancers), External service
access policy(db, cache etc)
● Application design
○ Pod design(using pod design patterns), Lifecycle(health check, graceful termination),
Scaling(resource request, autoscaling), Application types(stateful/stateless/batch/Big
Data/ML), Service mesh
● Security
○ Access control(rbac), Image validation(binary authorization, vulnerability scanning), Secure
clusters(private cluster, firewall)

24. Kubernetes day 2 Operations
Cluster management

25. Use Namespaces and RBAC for isolation
https://cloud.google.com/solutions/prep-kubernetes-engine-for-prod

26. Multicluster handling
Need for multiple clusters - Different applications, teams, environments, regions
Central policy management using
Anthos config management(ACM)
Proximity based cluster routing

27. 4 wayAutoscaling in GKE
HPA
Autoscales pool of workers on
custom metrics
VPA
Recommends podspec
Actuates the adjustment
CA NAP+
Scale Nodepools
Create right nodes for the job
Gate changes by HPA + NAP
Workload
Infrastructure

28. Kubernetes day 2 Operations
Application design

29. Readiness and Liveness probe
https://cloud.google.com/blog/products/gcp/k
ubernetes-best-practices-setting-up-health-ch
ecks-with-readiness-and-liveness-probes

30. Graceful shutdown handling
Best practises
● Have handler for Prestop hooks or
SIGTERM and handle shutdown
gracefully
● Keep readiness check interval
aggressive
● Have client retry failed requests
https://dzone.com/articles/kubernetes-lifecycle-of-a-pod

31. Big Data on Kubernetes
● Kubernetes as a replacement for YARN for Big data workloads
● Spark and Flink operators for Kubernetes are available as beta
● Dataproc on GKE is available as beta
● Advantages
○ Single orchestrator for applications and Big Data
○ Better use of cluster resources
○ Big Data application dependency handled through containers
○ Use Kubernetes ecosystem for Big Data

32. MLworkloads with Kubeflow
● Deploying and managing ML models at scale using Kubernetes
● Build, train and serve models
● Components - Notebooks, UI, training, Serving, Pipelines
● Multiple frameworks supported for training as well as serving
● Advantages
○ Portable ML pipelines
○ Best of Kubernetes features used for Machine learning

33. Kubernetes day 2 Operations
Security

34. Private Clusters in GKE

35. Secure Image pipeline

36. Kubernetes Network policies
Topology Defined with Network Policy API:
● A first-class Kubernetes API
● Defines allowed traffic patterns
How does it work:
● K8s defines the API.
● User applies a policy.
● Network policy agents watch and enforce.
● Restricts pod to pod traffic.
K8sMaster
etcd
policy.yaml
spec:
podSelector:
matchLabels:
run: nginx
ingress:
- from:
- podSelector:
matchLabels:
run:
access
Frontend
Service A
Service B
Policy Agent
K8sNode

37. Kubernetes ecosystem and Tools

38. Kubernetes ecosystem
CI/CD
(Tekton, Argo)
Monitoring
(Prometheus)
Logging
(Fluentd)
Service Mesh
(Istio, Linkerd)
Packaging
(Helm, kpt)
Infra
(Network, storage plugin)
Service Discovery
(CoreDNS)
Serverless
(knative, Virtual kubelet)
ML
(Kubeflow)
Registry
(Harbor)
Security
(Falco, Open policy)
VM
(Kubevirt, Config connector)

39. Helpers
Kubectx
kubens
Config Mgmt
Kustomize
Pkg Mgmt
Helm
Build
Dockerfile
Kaniko
Jib
CI/CD
Skaffold
IDE
Cloud
code for
VSCODE
Kubernetes tools(my favorites)

40. Practical Experiences
● Networking and security for clusters has to be pre-planned. These cannot be
changed later.
● Plan IP addresses before-hand. Kubernetes needs lot of addresses(Node, Pod,
Service)
● Use managed services when possible
● Keep separate environments for Dev, staging and production
● Isolate helper applications(CI/CD, Monitoring) from primary workloads
● Start with stateless workloads and then expand to stateful, big data and ML
● Invest in monitoring/logging/secret management solution
● Backup and DR is important for Kubernetes
● Make sure that every container has resource requests

41. References
● Kubernetes design principles video
● Kubernetes patterns video
● Kubernetes patterns slides
● Building Cloud native applications with Kubernetes and Istio - Kelsey
● Designing cloud native applications
● Extending Kubernetes