Over the last few years Kubernetes has transitioned from an object of curiosity and fear to a robust platform for big data. Watch this webinar and you will learn how the Altinity Kubernetes Operator for ClickHouse enables users to run high performance analytics on ClickHouse. You will see a simple installation and teach you how to scale it into a cluster that can analyze 100s of terabytes of data. Along the way we’ll share our lessons for ClickHouse on Kubernetes in Altinity.Cloud. We built it on Kubernetes using the Altinity Operator and now run hundreds of clusters in the cloud. You can too!

1. Cloud Native ClickHouse at Scale
Using the Altinity Kubernetes
Operator
Robert Hodges and Altinity Engineering
1

2. Let’s make some introductions
ClickHouse support and services including Altinity.Cloud
Authors of Altinity Kubernetes Operator for ClickHouse
and other open source projects
Robert Hodges
Database geek with 30+ years
on DBMS systems. Day job:
Altinity CEO
Altinity Engineering
Database geeks with centuries
of experience in DBMS and
applications
2

3. Understands SQL
Runs on bare metal to cloud
Shared nothing architecture
Stores data in columns
Parallel and vectorized execution
Scales to many petabytes
Is Open source (Apache 2.0)
ClickHouse is a real-time analytic database
It’s a popular engine for
real-time analytics
ClickHouse
Event
Streams
ELT
Object
Storage
Interactive
Graphics
Dashboards
APIs
3

4. Kubernetes,
Operators, and
ClickHouse
4

5. VM
Kubernetes manages container-based applications
5
ClickHouseC
ontainer
Block
Storage

6. Kubernetes maps resource definitions to infrastructure
6
Stateful
Set
Persistent
Volume
Pod
Persistent
Volume
Claim
Kubernetes
Resources
Container
AWS EBS
Storage
Runtime
Abstractions Volume
Process
running on
host
Physical
Infrastructure

7. 7
General control loop for Kubernetes resources
Actual
State
Act Observe
Control
Loop
Analyze
Desired
State

8. ClickHouse is usually a little more complicated!
ClickHouse Server
shard1
Analytic
application
Zookeeper Server
ClickHouse Server
shard1
ClickHouse Server
shard2
ClickHouse Server
shard2
Zookeeper Server Zookeeper Server
Availability Zone Availability Zone Availability Zone
8

9. …Which means lots of Kubernetes resources
Zookeeper
Services
Zookeeper-0
Zookeeper-2
Zookeeper-1
Shard 1 Replica 1
Replica
Service
Load
Balancer
Service
Shard 1 Replica 2
Shard 2 Replica 1
Shard 2 Replica 2
Replica
Service
Replica
Service
Replica
Service
Configd Config Map
Stateful
Set
Pod
Persistent
Volume
Claim
Persistent
Volume
Per-replica Config Map
. . . 9
User Config Map
Stateful
Set
Pod
Persistent
Volume
Claim
Persistent
Volume
Per-replica Config Map

10. 10
Custom
Resource
Definition
Operators implement and manage “custom resources”
ClickHouseInstallation
Custom Resource
Definition
Change
Events
Tracking
Operator
Reconciliation
Error handling
Desired
State(s)
Desired
State(s)
Desired
State(s)
Desired
State(s)
Adjust
Apply

11. kube-system namespace
Result: Operators make databases work on Kubernetes
Altinity
ClickHouse
Operator
your-favorite namespace
Apache 2.0 source,
distributed as Docker
image
kubectl -f apply
demo.yaml
Best practice deployment
“Adjust reality”
ClickHouse
Installation CRD
11

12. Setting up your
first ClickHouse
cluster in
Kubernetes
12

13. kubectl apply -f
https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/dep
loy/operator/clickhouse-operator-install-bundle.yaml
Step 1: Install ClickHouse operator from GitHub
Defines the
ClickHouse CRD
Installs operator
in kube-system
And other good
stuff

14. Get Zookeeper stateful set definition:
wget
https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/dep
loy/zookeeper/quick-start-persistent-volume/zookeeper-1-node.yaml
Install Zookeeper.
kubectl create ns zoo1ns
kubectl apply -f zookeeper-1-node.yaml -n zoo1ns
Step 2: Set up ZooKeeper
14
Dev only

15. Step 3: Define your cluster (cluster configuration)
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "demo"
spec:
configuration:
clusters:
- name: "cl"
layout:
shardsCount: 1
replicasCount: 2
templates:
podTemplate: server
volumeClaimTemplate: storage
zookeeper:
nodes:
- host: zookeeper.zoo1ns
port: 2181
15
Shards and replicas
Definitions for pods and storage
Where is Zookeeper?

16. templates:
podTemplates:
- name: server
spec:
containers:
- name: clickhouse
image: altinity/clickhouse-server:22.3.15.34.altinitystable
16
Step 3: Define your cluster (pod definition)
Server version

17. volumeClaimTemplates:
- name: storage
# Do not delete PVC if installation is dropped.
reclaimPolicy: Retain
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
17
Protect storage from deletion
Storage size
Step 3: Define your cluster (pod definition)

18. 18
DEMO TIME!

19. Accessing our creations
Safety first...Check storage!
$ kubectl get pvc; kubectl get pv
Access ClickHouse
$ kubectl exec -it pod/chi-demo-cl-0-0-0 -- clickhouse-client
Forward port to external network.
$ kubectl port-forward service/clickhouse-demo 8123 > /dev/null &
$ curl http://localhost:8123
Ok.
19

20. Adding a user
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
. . .
spec:
configuration:
clusters: . . .
zookeepers: . . .
users:
root/password_sha256_hex: 2bb80……7a25b
root/networks/ip:
- ::1
- 127.0.0.1
root/quota: default
root/access_management: 1
20
User definition
Enable RBAC

21. Scaling up to
production on
Kubernetes
21

22. Choosing a Kubernetes distribution
22
*
* Used for demo/test only

23. Your Kubernetes setup needs to get a few things right
23
Master Node Worker Node
Worker Node
Pod
Pod
Pod
Pod
ClickHouse Pod ClickHouse Pod
Provision a cluster
Add/remove nodes as
needed (auto-scaling)
Manage cloud block
storage for data

24. …Provided you have the right magic configured
Pod Persistent
Volume
Persistent
Volume
Claim
Karpenter
Provisioner
Storage
Class
EBS Volume
Worker
Node VMs
Worker
Node VMs
Worker
Node VMs
EBS Volume
EBS Volume
Creates PVs
in response
to claims
Adjusts
VMs to match
Allocates
storage to
match PVs
Watches pod
resource requests

25. We can control VM type, zone, and storage type
Network-Attached
Block Storage
CPU
RAM
VM
m5.large
us-west-2a
/data1
50Gb gp2
IOPS

26. Use pod templates to place replicas in different zones
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"metadata:
name: "prod"
spec:
configuration:
clusters:
- name: "ch"
layout:
replicas:
- templates:
podTemplate: clickhouse-zone-2a
- templates:
podTemplate: clickhouse-zone-2b
shardsCount: 1
templates:
volumeClaimTemplate: storage
26
Separate template for
each availability zone

27. Node selectors and instance types force pods to nodes
podTemplates:
- name: clickhouse-zone-2a
spec:
containers:
- name: clickhouse
image: altinity/clickhouse-server:22.3.15.34.altinitystable
resources:
limits:
cpu: 1800m
memory: 7Gi
requests:
cpu: "1"
memory: 6452Mi
nodeSelector:
node.kubernetes.io/instance-type: m5.large
zone:
key: topology.kubernetes.io/zone
values:
- us-west-2a
27
Requires a node with
m5.large VM type
Requires a node in zone
us-west-2a
Reserves resources on
VM

28. Volume claim templates allocate storage for pods
volumeClaimTemplates:
- name: storage
# Do not delete PVC if installation is dropped.
reclaimPolicy: Retain
spec:
storageClassName: gp2
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
28
Amount of storage
requested
Set up storage classes for
the storage types that
you want

29. Let’s scale up vertically!
Network-Attached
Block Storage
CPU
RAM
VM
m5.xlarge
us-west-2a
/data1
IOPS
100Gb gp2

30. Scale pods using nodeSelector and resources
podTemplates:
- name: clickhouse-zone-2a
spec:
containers:
- name: clickhouse
image: altinity/clickhouse-server:22.3.15.34.altinitystable
resources:
limits:
cpu: 3600m
memory: 15Gi
requests:
cpu: "2"
memory: 13Gi
nodeSelector:
node.kubernetes.io/instance-type: m5.xlarge
zone:
key: topology.kubernetes.io/zone
values:
- us-west-2a
30
Requires an m5.xlarge
now
Reserves resources on
VM; max the values to
reserve it for ClickHouse

31. Storage templates define persistent volume claims (PVCs)
volumeClaimTemplates:
- name: storage
# Do not delete PVC if installation is dropped.
reclaimPolicy: Retain
spec:
storageClassName: gp2
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
31
You can increase storage but
cannot decrease it
Future versions of the
operator will extend storage
without requiring a restart
Will use default storage class
if this is omitted

32. Know your storage class(es)!
$ kubectl get storageclass gp2 -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
storageclass.kubernetes.io/is-default-class: "true"
name: gp2
parameters:
fsType: ext4
type: gp2
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
32
This storage class will be
picked if you don’t specify
anything else
Make sure your storage class
supports volume expansion

33. How does the operator manage adding replicas?
VM VM VM VM
Update cluster
metadata
Add stateful set
Copy schema
to new replica Data replication
(ClickHouse does this)
Provisioner adds VM

34. Safety tips for happy Kubernetes operation
34
Tip 1: Never run this command while there are active ClickHouse clusters. It
deletes the ClickHouseInstallation CRD definition. Kubernetes will then delete your
clusters.
kubectl delete -f clickhouse-operator-install-bundle.yaml
Tip 2: Use the “reclaimPolicy: Retain” setting to keep storage from being deleted if
you accidentally delete a cluster
Tip 3: Move data off shards before deactivating them. The Altinity Operator will not
do it automatically.

35. What’s next?
35

36. More things to learn about
● External network access
○ Check out service deployment examples in https://github.com/Altinity/clickhouse-operator
○ Altinity Operator can configure external and internal load balancers
● Backup
○ We use clickhouse-backup running as a sidecar to ClickHouse
● ZooKeeper 3 node ensemble
○ See setup recommendations on GitHub
● Security
○ Check out the write-up in the Altinity Operator hardening guide.
● Monitoring and alerting
36

37. Typical Day 2 monitoring from Altinity.Cloud
37

38. Looking for an easier way? Check out Altinity.Cloud.
38

39. Final thoughts
39

40. How to get started with ClickHouse on Kubernetes
● The Altinity Kubernetes Operator manages ClickHouse clusters
● Try it out on Minikube or other dev versions of ClickHouse
● Most people use managed Kubernetes services for production systems
○ But OpenShift, Rancher, KOPS are OK too…
● Map ClickHouse nodes to VMs using a provisioner or node groups
● Check the docs for advanced topics
40

41. More information!
● Altinity Kubernetes Operator for ClickHouse on GitHub
○ https://github.com/Altinity/clickhouse-operator
● Altinity documention (https://docs.altinity.com)
● Altinity blog (https://altinity.com/blog)
● Kubernetes docs (https://kubernetes.io/docs/home/)
● EKS and GKE documentation
○ Including eksctl
● Karpenter documentation (https://karpenter.sh/)
41

42. Help make the operator better!!!
https://github.com/Altinity/clickhouse-operator
Tell your friends!
Log issues!
Send us pull requests!
42

43. 43
Thank you!
Questions?
https://altinity.com
rhodges at altinity.com
alz at altinity.com
Altinity.Cloud
Altinity
Kubernetes
Operator for
ClickHouse
We’re hiring!