This slide was delivered at the Bay Area NewSQL meetup in California on how TiDB, an open source NewSQL distributed database, is deployed and managed on any Kubernetes-enabled cloud environment by applying the Operator pattern.

1. “Smooth” Operator
Kevin Xu (@kevinsxu; kevin@pingcap.com)

2. Agenda
● History and Community
● Technical Walkthrough
● Operator to Run Stateful App on Kubernetes

3. A little about PingCAP
● Founded in April 2015 by 3 infrastructure engineers
● Created and maintains TiDB, TiKV (and a few other things…)
● Offices throughout North America and China

4. PingCAP.com
Community
Stars
● TiDB: 17,000+
● TiKV: 4,500+
Contributors
● TiDB: 230+
● TiKV: 120+
TiDB DevCon:
● 700+ devs

5. PingCAP.com
Who’s Using TiDB?
300+
Deployments

6. Use Cases
1. Approaching the maximum size for MySQL
on a single server. Debating whether or not
to shard.
2. Already sharded MySQL, but having a hard
time doing analytics on up-to-date data.

7. PingCAP.com
Mobike + TiDB
● 200 million users
● 200 cities
● 9 million smart bikes
● ~30 TB / day

8. Technical Walkthrough

9. Technical Inspiration
TiDB is a NewSQL database that speaks the MySQL protocol
It is not based on the MySQL source code
It is an ACID/strongly consistent database
Inspiration came from Google Spanner + F1
It separates SQL processing and atorage into separate layers
Both of them are independently scalable
The SQL processing layer is stateless
It is designed for both Transaction and Analytical Processing (HTAP)

10. TiDB
TiDB
Region 1 L
TiKV Node 1
Region 2
Region 3
Region 4
Region 2 L
TiKVNode 3
Region 3
Region 4 L
Region 1
Region 4
TiKV Node 2
Region 3 L
Region 2
Region 1
PD Cluster

11. PingCAP.com
Cloud Native Ecosystem
● Prometheus
○ (maintains Rust implementation:
https://github.com/pingcap/rust-prometheus)
● gRPC
○ (maintains Rust implementation: https://github.com/pingcap/grpc-rs)
● etcd
○ we contributed the “Learner” feature

12. Row + Column Storage (Announced Jan 2019)
Spark Cluster
TiDB
TiDB
Region 1
TiKV Node 1
Region 2
Region 3
Region 4
Region 2
TiKV Node 3
Region 3
Region 4
Region 1
Region 4
TiKV Node 2
Region 3
Region 2
Region 1
TiFlash Node 2
TiFlash Extension Cluster TiKV Cluster
TiSpark
Worker
TiSpark
Worker
TiFlash Node 1

13. Operator

14. Operator History
● Operator pattern pioneered by CoreOS...now
Redhat...now IBM
○ etcd operator, Prometheus operator
● Introduced in 2016, Operator Framework in 2018
● TiDB Operator (2017); Predated Operator
Framework

15. Why Should We Care?
● Manages stateful applications:
○ databases, caches, monitoring system, etc.
● Encodes application domain knowledge
○ Extension of your SRE team
● Kubernetes-enabled Hybrid / Multi-Cloud
● Growing popularity in database community:
○ https://thenewstack.io/databases-operators-bring-stateful-workloads-to-kubernet
es/

16. PingCAP.com
TiDB Operator Architecture
API ServerController ManagerScheduler
Kubernetes Core
TiDB Controller Manager
TiDB Cluster Controller
PD Controller
TiKV
Controller
TiDB
Controller
TiDB Scheduler:
TiDB Cloud Manager
API Gateway
Control Plane
Cost Controller
Kube Scheduler
TiDB Scheduler

17. PingCAP.com
High Level
Resources => Controllers => Clusters

18. Resources -- CRD
● Custom Resource Definition (CRD):
○ An application-specific YAML file
○ End user writes the domain operation logic in CRD
○ Simple to implement and deploy
● (There is another way):
○ API Aggregation:
■ More control, more powerful but…
■ Hard to deploy, not well-supported by k8s engines

19. Cluster State -- StatefulSet (a controller)
Unlike Deployment, StatefulSet...
● Guarantees ordering and uniqueness of pods
○ pd -> tikv -> tidb
● Gives “sticky” identity -- network and storage
● *No* interchangeable pods (always map the volume back
to the same pod)
● Stable since k8s 1.9 (we are at 1.13 now)

20. How TiDB manages state -- Custom Controller
Spec:
component:
image:
replicas:
...
Status:
image
replicas
state
CRD
(provided by user)
Custom
Controller
Cluster State

21. How TiDB manages state
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
name: demo
spec:
pd:
image: pingcap/pd:v2.1.0
replicas: 3
requests:
cpu: “4”
memory: “8Gi”
…
tikv:
image: pingcap/tikv:v2.1.0
…
status:
tikv:
stores:
“5”:
podName: demo-tikv-2
state: Up
...
type Manager interface {
Sync(*TidbCluster) error
}

22. PingCAP.com
TiDB Operator Architecture
API ServerController ManagerScheduler
Kubernetes Core
TiDB Controller Manager
TiDB Cluster Controller
PD Controller
TiKV
Controller
TiDB
Controller
TiDB Scheduler:
TiDB Cloud Manager
API Gateway
Control Plane
Cost Controller
Kube Scheduler
TiDB Scheduler

23. PingCAP.com
All Open Sourced
https://github.com/pingcap/tidb-operator

24. PingCAP.com
On GCP Marketplace

25. PingCAP.com
TiDB, Managed As A Cloud Service
Early Access: https://www.pingcap.com/tidb-cloud/

26. PingCAP.com
Operator Resources
● Operator Developer Guide (RedHat):
https://operators.gitbook.io/operator-developer-guide-for-red-hat-partners/
○ They also have a Slack channel...who doesn’t these days?
● Operator Framework:
○ https://github.com/operator-framework
● Brandon Philips’s blog posts:
○ https://coreos.com/blog/introducing-operators.html
○ https://www.redhat.com/en/blog/introducing-operator-framework-building-apps-kubernete
s

27. Thank You
@kevinsxu; kevin@pingcap.com