PGConf.ASIA 2019 Bali - 9 September 2019 Speaker: Takahiro Kobayashi Room: SQL Title: Building PostgreSQL as a Service with Kubernetes

1. Building PostgreSQL as a Service
with Kubernetes
PGConf.Asia 2019
2019/9/9
@tzkb

2. 3
My Actitvities
PGConf.Asia 2018 @Tokyo
A guide of PostgreSQL on Kuberntes
- In terms of Storage -
CloudNativeDays Tokyo 2019
The Future of Database on Kubernetes
- What run with Cloud Native Storage -
 ＋ ＝∞

3. 4
Agenda
1. Recap: What is Kubernetes?
2. The Issues for Database on Kubernetes
3. How to run your PostgreSQL on K8s
4. Kubernetes becomes The Platform

4. 6
1. Recap: What is Kubernetes?

5. 7
What is Kubernetes?
Pod Pod
Pod
Pod Pod
• Kubernetes(K8s) is the orchestrator tool for containers.
It has 3 features below.
• Declarative config
• Auto-healing
• Immutable
Database is
not Immutable.

6. 8
Better to handle the database system by Kubernetes?
Node Node Node
Master Slave
Replicate
• The database usually has a state that is not easy to maintain
by Kubernetes.
<Disadvantage for DB>
• Necessary to startup in
turn.
• Must never lose their
data.
• Handle the database as
pets.

7. 9
Example of Database on Kubernetes: Vitess
VTtablet
VTtablet
VTtablet
VTgate
app
app
app
SQL
SQL
SQL
• Vitess that used on YouTube is the CNCF incubating project.
• Vitess provides MySQL
sharding in K8s.
• VTgate and VTtablet
can scale by K8s.
• When terminating a
component abnormally,
Kubernetes repair it
automatically.

8. 10
The choice: How to manage your database
Compute
Storage
Managed
Amazon Aurora
Amazon Redshift
Amazon RDS
on Cloud on Kubernetes
• You can choose to manage the database by yourself or else.

9. 17
2. The Issues for Database on Kubernetes

10. 18
Kubernetes is the Distributed Systems
• Developed as following a distributed architecture.
• When doesn’t a node reply
– Network partition?
– Process failure?
– Node failure?
• If the disk resource attached,
harder to determine.
FailOver?

11. 20
Database Architects are familiar with Clustering
 “If you don’t know the status, it’s okay.
We act on the premise of failsafe.”
 “No need to share resources. Right?”
 “Both have long been known for database
clustering.”

12. 21
Basic: Database Clustering
HA
(Active/Standby)
１
Sharding
Replication
(Active/Active)
２or
more
Instances Redundancy
2 or
more
Shared
Disk
Log
Shipping
---
×
Scaleout?
Read
Read/
Write
Failover
(Fencing)
Availability
Promotion
(Election)
---
• There are differences to build a DB cluster with some nodes.

13. 22
Clustering #1: HA
• With Linux-HA
• Use high-available shared
storage
<Worst Case>
• Multiple writes to storage
<Solution>
• Fencing
VIP
Linux-HA
Controller Controller
• It's been used since before Linux but helpful.

14. 23
Note: Fencing
VIP
Linux-HA
Controller Controller
< When Detecting Node Failure >
1. Forced node power off
i. Definite processes stop
ii. Unmount storage
iii. Detach virtual IP
2. PostgreSQL starts to run on
the standby node.
• Failed node is isolated from resources = Fencing

15. 24
Clustering #2: Replication
WAL
• The master can Read/Write,
Slaves are Read-Only.
• Data synchronization by WAL
transmission
<Worst Case>
• 2 or more Masters
<Solution>
• Leader Election
• Redundancy built into PostgreSQL = Streaming Replication
Master
SlaveSlave

16. 25
Note: Leader Election
WAL
Be promoted as
a master,
The other is still
a slave.
• Always one master
• The former master joins as a
slave.
<Master in unknown state>
1. The remaining one slave is
elected as the leader
2. The leader is promoted as a
master.
• Algorithms such as Paxos and Raft are used.
MasterSlave

17. 26
Clustering #3 Sharding
• Divide data between nodes
and operates as one DB.
• Dispatches queries to relevant
nodes.
• Basically no availability.
• Problems with the transaction.
• For rather scalability than availability.
Coordinator

18. 27
3. How to Run your on Kubernetes

19. 28
Implemetation Overview : on Kubernetes
# Category OSS used Description
ⅰ
HA
• Use Rook/Ceph as Shared
Storage.
ⅱ
• Use LINSTOR/DRBD as
Shared Storage.
ⅲ Replication
• Use Streaming Replication,
without Shared Storage.
ⅳ Operator
• Building and Operating
Replication automatically.
• We can see following four patterns.

20. 29
• K8s manages
everything(DB,storage)
• Shared-Storage: Ceph
• Fenced by kube-fencing
< Disadvantage >
• Complicated
• Insufficient IO
HA (i)：
Replicas:1
• is deployed as StatefulSet using Rook/Ceph.
kube-fencing

21. 30
Note: Without Fencing
Replicas:1
• When a node goes down, never failover.
• To avoid network
partition.
• It is by design.

22. 31
Note: What is
• Rook is Kubernetes Operator managing Ceph or others.
operator
agent/discover agent/discover agent/discover
osd osd osd
mon mon mon
CSI
csi-provisioner
csi-rbdplugin csi-rbdplugin csi-rbdplugin
Rook
• Rook makes easy to
build Ceph cluster.
• Also easy to deploy
CSI modules.
• CSI: Containar
Storage Interface

23. 32
HA (ii)：
Replicas:1
kube-fencing
• LINSTOR is Software-Defined Storage based on DRBD.
• K8s manages
everything(DB,storage)
• Redundancy: DRBD
• Simple, Read IO
without Network
< Disadvantage >
• Limited to Scale

24. 33
Benchmark Results
Single(with EBS) Rook/Ceph DRBD
1nodes 5nodes 2nodes
100
37.8
77.1
• Measured by pgbench for 3 patterns.
TPS

25. 34
Replication :
proxy proxy proxy
keeper keeper keeper
sentinel sentinel sentinel
• Builds Streaming Replication on top of Kubernetes.
• 3 types of processes
have different roles
• Without Shared-
Resources
< Disadvantage >
• Not builtin Read Off-
loading

26. 36
Operator :
• KubeDB operates not only but also others.
kubedb-operator
-0 -1 -2
postgres snapshotdormantdabases
• Database Operator for
– PostgreSQL
– MySQL
– Redis
• Kubedb-operator
builds SR.
• Able to get/restore
snapshot easily.

27. 37
Example : PostgreSQL Configration by KubeDB
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
name: ha-postgres
namespace: demo
spec:
version: “10.6-v2"
replicas: 3
storageType: Durable
storage:
storageClassName: "standard"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
 spec.version
– Choose PostgreSQL version.
 spec.replicas
– The number of Instances.
 spec.storage
– Define storage type/size, etc.
• Allows to define Streaming Replication with a simple YAML.

28. 38
Example : Snapshot by KubeDB
apiVersion: kubedb.com/v1alpha1
kind: Snapshot
metadata:
name: snapshot-to-s3
labels:
kubedb.com/kind: Postgres
spec:
databaseName: ha-postgres
storageSecretName: s3-secret
s3:
endpoint: 's3.amazonaws.com'
bucket: kubedb-qa
prefix: demo
• Write declarative Snapshot settings by YAML.
• Simple backup that applies only
this YAML.
• You can select storage,
– S3
– Swift
– Kubernetes Persistent Volume

29. 39
Note : Backup with PostgreSQL + Ceph
$ kubectl exec -it -n rook-ceph rook-ceph-tools-seq -- rbd -p replicapool ls
pvc-bdbc6e53-f6e9-11e8-b0d9-02f062df6b48
$ kubectl exec -it pg-rook-sf-0 -- psql -h localhost -U postgres -c "SELECT pg_start_backup(now()::text);"
pg_start_backup
-----------------
0/C000028
(1 row)
$ kubectl exec -it -n rook-ceph rook-ceph-tools-seq -- rbd snap create replicapool/img@snap
$ kubectl exec -it pg-rook-sf-0 -- psql -h localhost -U postgres -c "SELECT pg_stop_backup();"
NOTICE: pg_stop_backup complete, all required WAL segments have been archived
pg_stop_backup
----------------
0/D000050
(1 row)
• Need to know PostgreSQL & Ceph Commands.

30. 41
4. Kubernetes becomes The Platform

31. 42
To Recap
 The components of database clustering with
Kubernetes Native are already available.
 You can see some operators for DBA task
automation.
 However, it is not over yet.
Cloud Native Storage ＋ ＋ ＝ ???

32. 43
The Signs
I. Pluggable Storage
 Optimized Storage system for DB on K8s?
II. Forked and Cloud-Oriented PostgreSQL
 AWS Aurora, Azure Hyperscale

33. 44
THE LOG IS THE DATABASE.
SQL
Transactions
Caching
Storage
Logging
Storage
Logging
Storage
Logging
CPU
Memory
Cache(SSD)
Page
Cache(SSD) Log
AWS Aurora(PostgreSQL) Azure Hyperscale
• Both divide RDBMS functions and are extended by each cloud.

34. 45
As the platform for PostgreSQL as a Service
DBaaS by Kubernetes
STaaS by Kubernetes
What we got for DBaaS
• HA
• Streaming Replication
• DB Operator
Also for STaaS
• Simple Redundancy
• Distributed Storage
• Interoperable IF(CSI)
• Kubernetes will be "The Platform for Platforms."

35. 46
Questions?
@tzkb
@tzkoba

36. 47
Appendix