This document discusses PingCAP's Kubernetes operator for TiDB, an open source distributed SQL database. It provides a brief history of PingCAP and the TiDB community. It then gives a technical overview of TiDB's architecture before explaining how the TiDB operator works. The operator allows users to deploy and manage TiDB clusters on Kubernetes through custom resources that are controlled by custom controllers. This provides capabilities like automated scaling, updates, and failover for stateful applications running on Kubernetes. The operator is open source and TiDB is also available as a managed service on GCP Marketplace.

1. “Smooth” Operator
Kevin Xu (@kevinsxu; kevin@pingcap.com)

2. Agenda
● History and Community
● Technical Walkthrough
● Kubernetes Operator
○ to run cloud-native stateful app

3. A little about PingCAP
● Founded in April 2015 by 3
infrastructure engineers
● Created and maintains TiDB, TiKV
● Offices throughout North America
and China

4. PingCAP.com
Community
Stars
● TiDB: 17,000+
● TiKV: 4,500+
Contributors
● TiDB: 245+
● TiKV: 130+
TiDB DevCon:
● 700+

5. PingCAP.com
Who’s Using TiDB?
300+
Deployments

6. Use Cases
1. Approaching the maximum size for MySQL
on a single server. Debating whether or not
to shard.
2. Already sharded MySQL, but having a hard
time doing analytics on up-to-date data.

7. PingCAP.com
Mobike + TiDB
● 200 million users
● 200 cities
● 9 million smart bikes
● ~35 TB in TiDB

8. Technical Walkthrough

9. Technical Inspiration
TiDB is a NewSQL database that speaks the MySQL protocol
It is not based on the MySQL source code
It is an ACID/strongly consistent database
The inspiration is Google Spanner + F1
It separates SQL processing and storage into separate components
Both of them are independently scalable
The SQL processing layer is stateless
It is designed for both Transaction and Analytical Processing (HTAP)

10. TiDB
TiDB
Region 1 L
TiKV Node 1
Region 2
Region 3
Region 4
Region 2 L
TiKVNode 3
Region 3
Region 4 L
Region 1
Region 4
TiKV Node 2
Region 3 L
Region 2
Region 1
TiKV Cluster
PD Cluster

11. Row + Column Storage (Announced Jan 2019)
Spark Cluster
TiDB
TiDB
Region 1
TiKV Node 1
Region 2
Region 3
Region 4
Region 2
TiKV Node 3
Region 3
Region 4
Region 1
Region 4
TiKV Node 2
Region 3
Region 2
Region 1
TiFlash Node 2
TiFlash Extension Cluster TiKV Cluster
TiSpark
Worker
TiSpark
Worker
TiFlash Node 1

12. Kubernetes -- Operator

13. Kubernetes on 1 slide
● Pod: Smallest unit of work (analogous to a node, an instance)
○ can hold 1 or more containers
○ it dies? it dies. just get a new pod (cattle v. pet)
● Deployment: API for a Pod
○ great for stateless apps, like RESTful API
○ use pod resources, declare # of replicas, rolling deployment, etc.
● StatefulSet: Another API for a Pod, designed for stateful apps
○ maintain order, “sticky” identity, no interchangeability, etc.

15. Operator History
● Operator pattern pioneered by CoreOS...now
Redhat...now IBM
● Introduced in 2016, Operator Framework in 2018
○ First 2: etcd operator, Prometheus operator
● TiDB Operator (2017); Predated Operator
Framework
● OperatorHub.io

16. Why Do We (As in TiDB) Care?
● Manage multiple clusters (multi-tenancy)
● Safe scaling (up or down, in or out)
● Use different types of Network or Local Storage (different
performance)
● Automatic monitoring
● Rolling updates
● Automatic failover
● *Multi-Cloud* (as long as it has k8s)

17. Why Should YOU Care?
● Manages *stateful* applications:
○ databases, caches, monitoring system, etc.
● Encodes application domain knowledge
○ Extension of your SRE team
● Kubernetes-enabled Hybrid / Multi-Cloud
● Growing popularity in database community:
○ https://thenewstack.io/databases-operators-bring-stateful-workloads-to-kubernet
es/

18. PingCAP.com
High Level
Resources => Controllers => Clusters

19. Resources -- CRD
● Custom Resource Definition (CRD):
○ An application-specific YAML file
○ End user writes the domain operation logic in CRD
○ Simple to implement and deploy
● (There is another way):
○ API Aggregation:
■ More control, more powerful but…
■ Hard to deploy, not well-supported by k8s engines

20. Cluster State -- StatefulSet
StatefulSet...
● Guarantees ordering and uniqueness of pods
○ pd -> tikv -> tidb
● Gives “sticky” identity -- network and storage
● *No* interchangeable pods
○ always map the same volume to the same pod
● Stable since k8s 1.9 (we are at 1.13 1.14 now)

21. How TiDB manages state -- Custom Controller
Spec:
component:
image:
replicas:
...
Status:
image
replicas
state
CRD
(provided by user)
Custom
Controller
Cluster State

22. How TiDB manages state
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
name: demo
spec:
pd:
image: pingcap/pd:v2.1.0
replicas: 3
requests:
cpu: “4”
memory: “8Gi”
…
tikv:
image: pingcap/tikv:v2.1.0
…
status:
tikv:
stores:
“5”:
podName: demo-tikv-2
state: Up
...
type Manager interface {
Sync(*TidbCluster) error
}

23. “Smooth” Operator

24. “Smooth” Operator

25. PingCAP.com
All Open Sourced
https://github.com/pingcap/tidb-operator/blob/
master/docs/user-guide.md

26. PingCAP.com
On GCP Marketplace

27. PingCAP.com
TiDB, Managed As A Cloud Service
Early Access: https://www.pingcap.com/tidb-cloud/

28. Thank You
@kevinsxu; kevin@pingcap.com