Of all services, databases are perhaps the most resistant to the cloud native cattle-not-pets model. So what does it really mean to run a large database on Kubernetes? Using ClickHouse as an example, Robert Hodges discusses operators as the key enabler of data on Kubernetes, then digs into some challenges to making complex databases really behave like cattle.

1. © 2022 Altinity, Inc.
Life in the Herd -
Cloud Native ClickHouse
Robert Hodges, Altinity
DoK San Francisco Meetup - 20 July
1

2. © 2022 Altinity, Inc.
Personal and company introductions
ClickHouse support and services including Altinity.Cloud
Authors of Altinity Kubernetes Operator for ClickHouse
and other open source projects
Robert Hodges
Database geek with 30+ years
on DBMS systems. Day job:
Altinity CEO
Altinity Engineering
Database geeks with centuries
of experience in DBMS and
applications
2

3. © 2022 Altinity, Inc.
© 2022 Altinity, Inc.
The Kubernetes
Vision: Databases
as cattle, not pets
3

4. © 2022 Altinity, Inc.
Introducing ClickHouse. It’s complicated.
ClickHouse Server
shard1
Analytic
application
Zookeeper Server
ClickHouse Server
shard1
ClickHouse Server
shard1
ClickHouse Server
shard2
ClickHouse Server
shard2
ClickHouse Server
shard2
Zookeeper Server Zookeeper Server
Availability Zone Availability Zone Availability Zone
4

5. © 2022 Altinity, Inc.
kube-system namespace
Operators make complex databases work on Kubernetes
Altinity
ClickHouse
Operator
your-favorite namespace
Apache 2.0 source,
distributed as Docker
image
kubectl -f apply
my-cluster.yaml
Best practice deployment
“Adjust reality”
ClickHouse
Resource
Definition
5

6. © 2022 Altinity, Inc.
Operators make complex configurations simple
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "demo-01"
spec:
configuration:
zookeeper:
nodes:
- host: zookeeper.zoo1ns
port: 2181
clusters:
- name: "cl"
layout:
shardsCount: 1
replicasCount: 2
. . .
6
Shards and replicas
Where is Zookeeper?
More stuff like
storage and
versions

7. © 2022 Altinity, Inc.
Well-written operators make your life better
7
Rolling
Upgrade
Prometheus
Metrics
Export
Vertical
Scaling
Horizontal
Scaling
Multi-AZ
Deployment
Node
Anti-
Affinity

8. © 2022 Altinity, Inc.
© 2022 Altinity, Inc.
But…
Cattle are
individuals, too!
8

9. © 2022 Altinity, Inc.
A typical distributed service
Load
Balancer
Service
#1
Service
#3
Service
#2
Storage
Storage
Storage
Traffic

10. © 2022 Altinity, Inc.
Created using Kubernetes stateful set
Pod
“svc-1”
Persistent
Volume
Service
“svc”
Stateful
Set
Persistent
Volume
Claim
Persistent
Volume
Persistent
Volume
Pod
“svc-2”
Pod
“svc-3”
Persistent
Volume
Claim
Persistent
Volume
Claim
Config
Maps
Config
Maps

11. © 2022 Altinity, Inc.
Database replicas are asymmetric
DBMS
Replica 1
DBMS
Replica 3
DBMS
Replica 2
Different software versions
Different resources
R/W vs. R/O Different AZs

12. © 2022 Altinity, Inc.
Databases need a more sophisticated replica model
SS/Pod
“svc-1”
Persistent
Volume
Service
“svc”
ClickHouse
Installation
Persistent
Volume
Claim
Persistent
Volume
Persistent
Volume
SS/Pod
“svc-2”
SS/Pod
“svc-3”
Persistent
Volume
Claim
Persistent
Volume
Claim
Config
Maps
Config
Maps

13. © 2022 Altinity, Inc.
© 2022 Altinity, Inc.
What if your calf
gets sick?
13

14. © 2022 Altinity, Inc.
$ kubectl apply -f crash.yaml
clickhouseinstallation.clickhouse.altinity.com/crash-demo configured
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
chi-crash-demo-ch-0-0-0 0/1 CrashLoopBackOff 1 (12s ago) 15s
14
Every [Kubernetes] SRE’s least favorite error

15. © 2022 Altinity, Inc.
Traditional cure for database crashes
15
Database
DBA

16. © 2022 Altinity, Inc.
Malign effects of EBS hardware failure in AWS
SS/Pod
“svc-1”
Persistent
Volume
Service
“svc”
ClickHouse
Installation
Persistent
Volume
Claim
Persistent
Volume
Persistent
Volume
SS/Pod
“svc-2”
SS/Pod
“svc-3”
Persistent
Volume
Claim
Persistent
Volume
Claim
Config
Maps
Config
Maps

17. © 2022 Altinity, Inc.
Operators need adaptation to handle DBMS failures
17
Logs accessible
outside pod
Change CRD and
apply
Take DBMS out of
rotation & fix online
Robust node
replace protocol
kubectl exec
Volume
corruption
Metadata
corruption
Misconfiguration
Bad upgrade
Unfulfilled resources
Accidental
volume deletion
Built-in locks on
PVC and/or PV
Slow startup

18. © 2022 Altinity, Inc.
© 2022 Altinity, Inc.
How can cattle
adapt to the
herd?
18

19. © 2022 Altinity, Inc.
File-based vs. network management
Server
Storage
Logs &
Metric
Data
Config
Files
Direct access to
file system
required
Configuration Backup & Restore Metrics export
Lots of sidecars!

20. © 2022 Altinity, Inc.
File-based vs. network management
Server
Storage
Logs &
Metric
Data
Config
Files
Direct access to
file system
required
Server
Storage
Logs &
Metric
Data
Config
Files
All
management
via network
Configuration
Metrics export
Backup & Restore
Configuration Backup & Restore Metrics export
Lots of sidecars! Just one pod!

21. © 2022 Altinity, Inc.
Attached local storage vs. network only with local cache
Server
NVMe
SSD
Server
NVMe
SSD
Server
NVMe
SSD
Pods pinned to host!
(But low latency)

22. © 2022 Altinity, Inc.
Attached local storage vs. network only with local cache
Server
NVMe
SSD
Server
NVMe
SSD
Server
NVMe
SSD
Server
SSD
Cache
Server
SSD
Cache
Server
SSD
Cache
Shared
Storage
Layer
Pods pinned to host!
(But low latency)
Pods scheduled on any host!
(But higher latency)

23. © 2022 Altinity, Inc.
Shared nothing versus separated compute/storage
Server
Dedicated
Storage
Server
Dedicated
Storage
Server
Dedicated
Storage
Server
Server
Server
Server
Server
Shared
Storage
Layer
Increasing volume of data, increasing elasticity

24. © 2022 Altinity, Inc.
© 2022 Altinity, Inc.
Final thoughts
on life in the
herd
24

25. © 2022 Altinity, Inc.
Database happiness on Kubernetes
● Build a replica model that permits variation
○ Not stateful sets
● Fix things without resort to ‘kubectl exec’
● Access management and storage through the network
25

26. © 2022 Altinity, Inc.
Thank you!
Questions?
https://altinity.com
rhodges at altinity.com
26
Altinity.Cloud
Altinity
Kubernetes
Operator for
ClickHouse
We’re hiring!