Is It Safe? Security Hardening for Databases Using Kubernetes Operators - Robert Hodges, Altinity Thanks to the Operator Pattern, Kubernetes is now an outstanding platform to run databases. But to quote Marathon Man, "is it safe?" This talk is a top-level review of the database security problem in Kubernetes, standard ways that operators can mitigate threats, and a wallet-sized checklist of security features you should look for in any operator you use. Our talk is practical and focused on needs of Kubernetes developers. Join us!

1. © 2023 Altinity, Inc.
Robert Hodges - Altinity
DoK Day North America 2023
1
Is It Safe?
Security Hardening for
Databases using
Kubernetes Operators

2. © 2023 Altinity, Inc.
A brief message from our sponsor…
ClickHouse software and services: Altinity.Cloud and Altinity Stable Builds
Authors of Altinity Kubernetes Operator for ClickHouse
Robert Hodges
Database geek with 30+ years
on DBMS. Kubernaut since
2018. Day job: Altinity CEO
Altinity Engineering
Database geeks with centuries
of experience in DBMS and
applications
2

3. © 2023 Altinity, Inc.
Kubernetes orchestrates container-based applications
3
Kubernetes
Resources
AWS EBS
Storage
Process
Physical
Infrastructure
Logical
Design
Block
Storage
ClickHouse
Server
Mapping
Mapping
Stateful
Set
Persistent
Volume
Pod
Persistent
Volume
Claim

4. © 2023 Altinity, Inc.
Kubernetes is a great platform for databases!
4
But every silver lining has a cloud…
Database
Stateful
Set
Persistent
Volume
Pod
Persistent
Volume
Claim

5. © 2023 Altinity, Inc.
Protecting data at the ground level
ClickHouse
Pod
Object Storage
EBS Volume
Unencrypted
data
Service
Persistent
Volume
ClickHouse
Pod EBS Volume
Persistent
Volume
Client
App
Backup
Backup
Exposed
public
endpoint
Unprotected
credentials
Exposed object storage
ConfigMap
Unencrypted
connection
Unsecured logins

6. © 2023 Altinity, Inc. 6
Yikes!
Databases are
complicated

7. © 2023 Altinity, Inc.
We can split database protection into three parts
7
Database
Stateful
Set
Persistent
Volume
Pod
Persistent
Volume
Claim
External
Data
Storage
Protect the
database
Protect Kubernetes Protect external data

8. © 2023 Altinity, Inc.
Operators reduce databases to a single resource
8
Bit o’
Yaml
ClickHouse
Pod
Persistent
Volume
Kubernetes
Operator
Custom Resource Definition
Aka “CRD”
kubectl apply -f db.yaml

9. © 2023 Altinity, Inc.
9
Custom
Resource
Definition
Operators translate CRDs to a best practice deployment
Custom
Resource
Definition
Change
Events
Tracking
Operator
Reconciliation
Error handling
Desired
State(s)
Desired
State(s)
Desired
State(s)
Desired
State(s)
Adjust
Apply

10. © 2023 Altinity, Inc.
That’s a big win for humans when databases get complex
10
Lots o’
Yaml
Operator
kubectl apply -f db.yaml

11. © 2023 Altinity, Inc.
Good operators come with built-in security features
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"metadata:
name: "prod"
spec:
configuration:
users:
default/password_sha256_hex: 716b...e448
clusters:
- name: "default"
secret:
valueFrom:
secretKeyRef:
name: "secure-inter-cluster-communications"
key: "secret"
11
Secure comms with
other databases
Eliminate empty
password for default
user
✔
✔
Restrict default user
to localhost and
cluster IPs
✔

12. © 2023 Altinity, Inc. 12
Wait.
What about
credentials?
Operators

13. © 2023 Altinity, Inc.
Credentials are everywhere!!
13
ClickHouse
Pod
Passwords for
database logins
ClickHouse
Pod
Credentials
for other
databases
Event queue
credentials
Object storage
keys

14. © 2023 Altinity, Inc.
Secrets transfer safely credentials to pods
14
Passwords for
database logins
Credentials
for other
databases
Event queue
credentials
Object storage
keys
Secret
Secret
Secret
Secret
ClickHouse
Pod
ClickHouse
Pod

15. © 2023 Altinity, Inc.
Most operators understand Kubernetes secrets
15
apiVersion: v1
kind: Secret
metadata:
name: db-passwords
type: Opaque
data:
root_login: NTgt...

16. © 2023 Altinity, Inc.
The Altinity operator has built-in syntax for passwords
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"metadata:
name: "prod"
spec:
configuration:
users:
default/password_sha256_hex: db-passwords/root_login
clusters:
- name: "default"
secret:
valueFrom:
secretKeyRef:
. . .
16
Pod-specific way to
set password
securely
✔
apiVersion: v1
kind: Secret
metadata:
name: db-passwords
type: Opaque
data:
root_login: NTgt...

17. © 2023 Altinity, Inc.
Kubernetes also has general ways to apply secret values
spec:
containers:
- name: clickhouse
image: altinity/clickhouse-server:23.3.8.22.altinitystable
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: s3-credentials
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: s3-credentials
key: AWS_SECRET_ACCESS_KEY
. . .
17
Assign to environmental variable
✔
apiVersion: v1
kind: Secret
metadata:
name: s3-credentials
type: Opaque
data:
AWS_SECRET_ACCESS_KEY: QUtJ...
AWS_ACCESS_KEY_ID: b00r...
Simple way to set S3 credentials using secrets

18. © 2023 Altinity, Inc. 18
Let’s protect
connections to
the database
Secrets
Operators

19. © 2023 Altinity, Inc.
TLS encrypts data on connections to/from databases
19
ClickHouse
Pod
ClickHouse
Pod
Client
App
Certificate
Private Key
Certificate
Private Key
CA Cert
CA Cert Optional

20. © 2023 Altinity, Inc.
Look for ways to configure ports and TLS
spec:
configuration:
clusters:
- name: "ch"
secure: "yes"
secret:
auto: "yes"
. . .
settings:
tcp_port: 9000 # keep for localhost
tcp_port_secure: 9440
https_port: 8443
files:
openssl.xml: |
<clickhouse>
<openSSL>
<server>
. . .
20
Use TLS-encrypted ports ✔
Specify ports to use ✔
Supply openSSL settings ✔

21. © 2023 Altinity, Inc.
Make sure you can also manage certificates
spec:
containers:
- name: clickhouse
image: altinity/clickhouse-server:23.3.8.22.altinitystable
volumeMounts:
- name: server-crt-volume
mountPath: "/opt/certs/server.crt"
subPath: server.crt
- name: server-crt-volume
mountPath: "/opt/certs/server.key"
subPath: server.key
. . .
volumes:
- name: server-crt-volume
secret:
secretName: server-certs
. . .
21
apiVersion: v1
kind: Secret
metadata:
name: server-certs
stringData:
server.crt: |-
-----BEGIN CERTIFICATE-----
...
server.key: |-
-----BEGIN PRIVATE KEy-----
...
Files mounted automatically ✔

22. © 2023 Altinity, Inc. 22
Stored data is
obviously
important
Secrets
Operators
Secrets

23. © 2023 Altinity, Inc.
Side bar: How Kubernetes “makes”storage
Stateful
Set
Persistent
Volume
Persistent
Volume
Claim
Storage
Class
EBS Volume
EBS Volume
EBS Volume
Creates PVs
in response
to claims
Allocates
storage to
match PVs
Pod

24. © 2023 Altinity, Inc.
We can make a custom storage class that encrypts data
24
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: encrypted-gp3
provisioner: ebs.csi.aws.com
parameters:
type: gp3
fsType: ext4
encrypted: "true"
allowVolumeExpansion: true

25. © 2023 Altinity, Inc.
Operators should leverage the power of storage classes
volumeClaimTemplates:
- name: storage
# Do not delete PVC if installation is dropped.
reclaimPolicy: Retain
spec:
storageClassName: encrypted-gp3
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
25
Picking class
encrypts data
automatically
✔

26. © 2023 Altinity, Inc. 26
What about container
security and
configuration control?
Storage
Classes
Secrets
Operators
Secrets

27. © 2023 Altinity, Inc.
Look for operators that have good development hygiene
27
Container Registry
Docker
Image
Docker
Image
Trivy
Container scanning
from command line
Docker Scout
Scanning in Docker
repo(s)
Quick CVE Checking
✔

28. © 2023 Altinity, Inc.
Use GitOps to deploy operators and database resources
28
Kubernetes
Kubernetes
Manifests
Kustomize +
Manifests
Helm Charts
App Resources
App Resources
App Resources
ArgoCD

29. © 2023 Altinity, Inc.
Don’t forget data outside the database…
29
Database
Stateful
Set
Persistent
Volume
Pod
Persistent
Volume
Claim
Backups
Object Storage
Database logs
Table
Data
Table
Data
Beware of sensitive
messages
Access policies
Secure storage
Encryption

30. © 2023 Altinity, Inc.
Security features to look for in database operators
30
Data
At-rest volume encryption
File system permissions
Secure logs / event data
Backup encryption
Networking
X509 certificate management
Application TLS configuration
Intra-cluster TLS configuration
Disable insecure ports
Public Cloud Integration
Private network load balancing
Encrypted object / block storage
Cloud IAM account integration
User Management
Secure `default` accounts
Strong password configuration
Use secrets to pass credentials
Network access restrictions
Kubernetes
Minimal ClusterRole privileges
Integration with cluster monitoring
Software Supply Chain
Signed, scanned containers
CVE reporting and fixes
Dependency management

31. © 2023 Altinity, Inc.
Can’t remember all that? We have you covered!
DoKC Operator Security and Hardening Guide
A guide to security best practices for database operators
https://github.com/dokc/sig-operator
Interested in helping? Get involved!
Join the #sig-operator channel in DoKC Slack Workspace
31

32. © 2023 Altinity, Inc.
Thank you and good luck!
Any questions?
Robert Hodges - Altinity CEO
● rhodges at altinity dot com
● LinkedIn
● DoKC Slack Workspace
32