Slides from Altinity CEO, Robert Hodges, presentation during Kubecon + CloudNativeCon EU Virtual 2020. August 18th, 2020.

1. Five Great Ways to Lose
Data on Kubernetes
(And How to Avoid Them)
Robert Hodges - KubeCon Europe 2020
1

2. Presenter and Company Bio
www.altinity.com
Enterprise provider for ClickHouse, a
popular, open source data warehouse.
Implementors of ClickHouse
Kubernetes operator.
Robert Hodges - Altinity CEO
30+ years on DBMS plus
virtualization and security. Using
Kubernetes since 2018.
2

3. Two boring
definitions
3

4. What do we mean by “data”?
Rep SKU Date City Customer Units Price ...
25 #556 25 Nov SFO Wells Fargo 378 $25.00
36 #556 25 Nov SEA Boeing 259 $25.50
55 #558 28 Nov BOS Au Bon Pain 100 $29.33
... ... ... ... ... ... ...
Which products
have the best
gross margins
over time?
Do some SKUs
sell better on
different days
of the week?
Which kinds of
companies are
most likely to
buy SKU 556? 4

5. And what do we mean by “lose”?
The data loss “arrow of evil”
Temporarily
unavailable
All of it gone,
forever
Can’t process
transactions
Unable to see
market
conditions
Business stops
functioning
5

6. The single copy
catastrophe
6

7. Starting a database is easy in Kubernetes
(Install helm)
kubectl create ns mysql
helm install --namespace mysql
--name mysql-server stable/mysql
7

8. Kubernetes Node
A delicate flower is our database
Container
“mysql”
Storage
Delete pod
Delete volume
Delete node
8

9. Kubernetes Node
Traditional database solution: replicas
Storage
Kubernetes Node
Storage
Replication
Container
“mysql”
Backup DBMS
Copy
DBMS
Copy
DBMS
Backup
Static copies
Live replica
Primary Replica
Container
“mysql”
9

10. Simplest K8s path to replicas: use an operator
Kubernetes
Operator
Single specification
Best practice
deployment
Custom
Resource
Definition
Kubernetes API
Native
Controller
Native
Controller
Native
Controllers
Etcd
Pod Service
StatefulSet
10

11. Complex systems made simple[r]
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "ch01"
spec:
configuration:
clusters:
- name: replicated
layout:
shardsCount: 2
replicasCount: 2
zookeeper:
nodes:
- host: zookeeper.zk
Name to identify resource
Definition of cluster
Name of service we depend on
11

12. Blast-radius
blues
12

13. What is “blast radius?”
HostKubernetes
Data Center
Region
13

14. Host 172.20.49.116
Affinity vs. anti-affinity
zookeeper-0
Host 172.20.71.114
zookeeper-1zookeeper-1
Anti-
Affinity
Affinity
14

15. Pod anti-affinity covers host failures
apiVersion: v1
kind: Pod
spec:
affinity:
"podAntiAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": [
{
"labelSelector": {
"matchLabels": {
"clickhouse.altinity.com/chi": "zdist1",
}
},
"topologyKey": "kubernetes.io/hostname" . . .
15

16. Node affinity + failure domain covers AZ failure
apiVersion: v1
kind: Pod
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- us-west-2a
16

17. Protect replicas using distance
CH PodCH Pod
Zookeeper
Pod
CH PodCH Pod
Zookeeper
Pod
us-west-2a
us-west-2b
CH PodCH Pod
Zookeeper
Pod
us-west-2c
DBMS
Copy
DBMS
CopyBackup
Host
Object
Storage
17

18. Blast protection across regions or K8s
(It’s complicated)
18

19. Affinity
afflictions
19

20. Where are my pods running?
kubectl get pod
-o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nod
eName -n zk
NAME STATUS NODE
zookeeper-0 Running ip-172-20-49-116.us-west-2.compute.internal
zookeeper-1 Running ip-172-20-71-114.us-west-2.compute.internal
zookeeper-2 Running ip-172-20-49-116.us-west-2.compute.internal
20

21. Which hosts are in which AZs?
kubectl get node
-o=custom-columns=NODE:.metadata.name,ZONE:.metadata.labels.'failure-doma
in.beta.kubernetes.io/zone'
NODE ZONE
ip-172-20-47-4.us-west-2.compute.internal us-west-2a
ip-172-20-49-116.us-west-2.compute.internal us-west-2a
ip-172-20-52-196.us-west-2.compute.internal us-west-2a
ip-172-20-70-184.us-west-2.compute.internal us-west-2b
ip-172-20-71-114.us-west-2.compute.internal us-west-2b
21

22. 172.20.49.116172.20.47.4
Affinity is useless if you don’t actually use it
zookeeper-0 zookeeper-2
172.20.70.184 172.20.71.114
zookeeper-1
us-west-2a
us-west-2b
22

23. Searching quickly for missing affinity rules
kubectl get pods -o json -n zk |
jq -r "[.items[] | {name: .metadata.name,
affinity: .spec.affinity}]"
[
{
"name": "zookeeper-0",
"affinity": null
},
. . .
]
23

24. The persistent
volume that
wasn’t
24

25. Kubernetes Node
Ephemeral storage is a feature, not a bug!
“bad”
pod
Kubernetes Node
“good”
pod
Network
Storage
Application Application
25

26. Things to look for in database storage
kubectl get pvc -o=custom-columns=NAME:.metadata.name
,SIZE:.spec.resources.request.storage,
CLASS:.spec.storageClassName,VOLUME:.spec.volumeName
...
NAME SIZE CLASS VOLUME
storage...0-0-0 10Gi kops-ssd-1-17 pvc-8f86...7a8
kubectl get storageclass/kops-ssd-1-17
...
NAME PROVISIONER RECLAIMPOLICY..
kops-ssd-1-17 (default) kubernetes.io/aws-ebs Delete
26

27. PVs are not enough if you don’t use them!
Kubernetes Node
EBS
Storage
Application
“zookeeper” pod
/data /datalog /var/lib/zk/data
Ephemeral Storage
27

28. Testing to the point of abuse is the solution
Check Kubernetes resource definitions
Inspect file system mounts
Kill pods
Kill nodes
Kill and restart all pods in replicated databases
Delete volumes
Test with large amounts of data
28

29. Fat fingers of
fate
29

30. The best way to lose data: do it yourself
helm delete mysql-server --purge
30

31. Fix PV reclaim policy to prevent vaporization
kubectl get pv
-o=custom-columns=PV:.metadata.name,NAME:.spec.claimRef.name,
POLICY:.spec.persistentVolumeReclaimPolicy
PV NAME POLICY
pvc-3969c6e7-1e79-424e-afdb-be050ba47f08 mysql-server Delete
kubectl patch pv pvc-3969c6e7-1e79-424e-afdb-be050ba47f08
-p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
31

32. Or use a different storage provider...
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: kops-ssd-1-17-retain
parameters:
encrypted: "true"
type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
32

33. “Orphan” PV can now be reclaimed
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-server
namespace: mysql
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
storageClassName: "kops-ssd-1-17"
volumeMode: Filesystem
volumeName: pvc-3969c6e7-1e79-424e-afdb-be050ba47f08
PVC definition must
exactly match volume!
33

34. Steps to reclaim -- Automation anyone?
Remove claimRef tag from PV using ‘kubectl edit’
kubectl apply -f reclaim-pvc.yaml
helm install --namespace mysql --name mysql-server
--set persistence.existingClaim=mysql-server,mysqlRootPassword=secret
stable/mysql
34

35. Wrap-up
35

36. How to avoid losing data on Kubernetes
Replicas!
Indulg i
Paranoi !
Distance!
Testing!
Affinity
Rules!
Reclaim
Policies!
Use operators if available
36

37. Thank you!
We’re hiring
Email:
rhodges@altinity.com
Code:
https://github.com/Altinity/click
house-operator
Company:
https://www.altinity.com
37