Database in Kubernetes: Diagnostics and Monitoring

Database in Kubernetes:
Diagnostics and Monitoring
Sveta Smirnova
Principal Support Engineering Coordinator

• MySQL Support Engineer
• Author
MySQL Troubleshooting
MySQL Cookbook, 4th Edition
• JSON UDF functions
• FILTER clause for MySQL
• Speaker
• Percona Live, OOW, Fosdem,
DevConf, HighLoad...
Sveta Smirnova

Database Performance Influencers
3
© Copyright 2023 Percona ® LLC. All rights reserved

• Standard database tools: EXPLAIN
• Interactive tools
• Log files
Query Performance Diagnostics
4

• Interactive tools
• Same as on bare metal
Concurrency Diagnostics
5

• Managed by Kubernetes
• Limits, set by Kubernetes
• DBA does not have access to tools,
available on bare hardware
Hardware
6

• Supported
PMM
7

• Supported
• All Percona operators are shipped with
PMM client
PMM
8

• Supported
PMM client
• PMM server needs to be installed
separately
• Install via Helm is available
PMM
9

• Supported
PMM client
• PMM server needs to be installed
separately
• Install via Helm is available
• Add pmm.enabled=true into
deploy/cr.yaml, configure PMM server
host and API key, then apply
PMM
10

Database Performance Troubleshooting
11

Database Performance Troubleshooting
12

• Global buffers to cache "hot" data
• InnoDB Buffer Pool
• WiredTiger Cache
• Shared buffers in PostgreSQL
• Operation and thread-specific buffers
• "Internal" buffers not under user control
How Memory Used by Databases?
13

• Physically available for nodes
• Limited by the Operator
resources:
requests:
memory: 32G
...
# limits:
# memory: 32G
Memory Limits
14

• kubectl top pods
• kubectl top nodes
• kubectl describe nodes
Memory Diagnostics
15

• Database-specific instruments
• Performance Schema in MySQL
Memory Diagnostics
16

• Table data
• Indexes
• Log files
• Temporary data
How Disk Used by Databases?
17

volumeSpec:
persistentVolumeClaim:
resources:
requests:
storage: 2G
Disk Limitations
18

• kubectl get persistentvolumeclaims
Disk Diagnostics
19

Disk Diagnostics
20

• Error logs of the database
2018-11-28T00:03:26.322318+06:00 0 [Note] InnoDB: page_cleaner:
1000ms intended loop took 4188ms. The settings might not be optimal.
(flushed=692 and evicted=0, during the time.)
Disk Diagnostics
21

• Error logs of the database
2018-11-28T00:03:26.322318+06:00 0 [Note] InnoDB: page_cleaner:
1000ms intended loop took 4188ms. The settings might not be optimal.
(flushed=692 and evicted=0, during the time.)
• Counters in the database
Disk Diagnostics
22

• Handle connections
• Threads in MySQL, MongoDB
• Sub-processes in PostgreSQL
• Perform calculations
• Perform IO operations
How CPU Used by Databases?
23

resources:
requests:
cpu: "16"
limits:
cpu: "32"
CPU Limits
24

CPU Diagnostics
25

• Counters in the database
CPU Diagnostics
26

• Communication between server and
client
• Regular client
• Application
• Replicas
• Synchronous nodes
How Databases Use Network?
27

Network Diagnostics
28

• Logs and statuses of the namespace
kube-system and your CNI
Network Diagnostics
29

• Logs and statuses of the namespace
kube-system and your CNI
• Latency in databases
Network Diagnostics
30

• By default, log files are gone after pod
failure
• Critical in such situations as database
crash
Log Files Rotation
31

• By default, log files are gone after pod
failure
• Critical in such situations as database
crash
• Solution
• PXC Operator: separate pod to store logs
• Other operators: K8-recommended
solutions such as Grafana Loki
Log Files Rotation
32

• Created for Support team to automate
K8 troubleshooting collections
• Owned by Support team
• In active progress
• New featurs each version
pt-k8s-debug-collector
33

pt-stalk
• Default for bare
metal
• Bare hardware
• Such tools as
• iostat
• vmstat
• dmesg
• top
• Default for K8
• kubectl
• Such commands
as
• get pods
• get
storageclasses
• get replicasets
• get logs
Batch Data Collection
34

• Basic batch collections per databases
• MySQL and PXC
pt-summary
pt-mysql-summary
Log files
• MongoDB
pt-mongodb-summary
• PostgreSQL
pg_gather
Features in pt-k8s-debug-collector
35

• MySQL and PXC
pt-summary
pt-mysql-summary
Log files
• MongoDB
pt-mongodb-summary
• PostgreSQL
pg_gather
• kubectl get
36

• MySQL and PXC
pt-summary
pt-mysql-summary
Log files
• MongoDB
pt-mongodb-summary
• PostgreSQL
pg_gather
• kubectl get
• kubectl logs
37

• pt-stalk
• More kubectl commands
• Extended MongoDB support
• Nicer output format for summaries
Future of the pt-k8s-debug-collector
38

• Mind limits in cr.yaml
• Work with K8 administrators for setting
up policies, allowing you to view
hardware usage
• Setup Loki or similar solution to save the
log files after restart
• Enable online database
instrumentation
• Use PMM or similar monitoring solution
• Plan in advance
Conclusion
39

Percona K8 Operators
PMM Demo
pt-stalk
pg_gather
Grafana Loki
Tools Mentioned
40

Thank you!
/svetsmirnova /svetasmirnova
/svetsmirnova /SvetaSmirnova

Database in Kubernetes: Diagnostics and Monitoring

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Database in Kubernetes: Diagnostics and Monitoring

Similar to Database in Kubernetes: Diagnostics and Monitoring (20)

More from Sveta Smirnova

More from Sveta Smirnova (20)

Recently uploaded

Recently uploaded (20)

Database in Kubernetes: Diagnostics and Monitoring