More Related Content Similar to Database in Kubernetes: Diagnostics and Monitoring (20) More from Sveta Smirnova (20) Database in Kubernetes: Diagnostics and Monitoring2. • MySQL Support Engineer
• Author
MySQL Troubleshooting
MySQL Cookbook, 4th Edition
• JSON UDF functions
• FILTER clause for MySQL
• Speaker
• Percona Live, OOW, Fosdem,
DevConf, HighLoad...
Sveta Smirnova
4. • Standard database tools: EXPLAIN
• Interactive tools
• Log files
Query Performance Diagnostics
4
© Copyright 2023 Percona ® LLC. All rights reserved
5. • Interactive tools
• Same as on bare metal
Concurrency Diagnostics
5
© Copyright 2023 Percona ® LLC. All rights reserved
6. • Managed by Kubernetes
• Limits, set by Kubernetes
• DBA does not have access to tools,
available on bare hardware
Hardware
6
© Copyright 2023 Percona ® LLC. All rights reserved
8. • Supported
• All Percona operators are shipped with
PMM client
PMM
8
© Copyright 2023 Percona ® LLC. All rights reserved
9. • Supported
• All Percona operators are shipped with
PMM client
• PMM server needs to be installed
separately
• Install via Helm is available
PMM
9
© Copyright 2023 Percona ® LLC. All rights reserved
10. • Supported
• All Percona operators are shipped with
PMM client
• PMM server needs to be installed
separately
• Install via Helm is available
• Add pmm.enabled=true into
deploy/cr.yaml, configure PMM server
host and API key, then apply
PMM
10
© Copyright 2023 Percona ® LLC. All rights reserved
13. • Global buffers to cache "hot" data
• InnoDB Buffer Pool
• WiredTiger Cache
• Shared buffers in PostgreSQL
• Operation and thread-specific buffers
• "Internal" buffers not under user control
How Memory Used by Databases?
13
© Copyright 2023 Percona ® LLC. All rights reserved
14. • Physically available for nodes
• Limited by the Operator
resources:
requests:
memory: 32G
...
# limits:
# memory: 32G
Memory Limits
14
© Copyright 2023 Percona ® LLC. All rights reserved
15. • kubectl top pods
• kubectl top nodes
• kubectl describe nodes
Memory Diagnostics
15
© Copyright 2023 Percona ® LLC. All rights reserved
16. • kubectl top pods
• kubectl top nodes
• kubectl describe nodes
• Database-specific instruments
• Performance Schema in MySQL
Memory Diagnostics
16
© Copyright 2023 Percona ® LLC. All rights reserved
17. • Table data
• Indexes
• Log files
• Temporary data
How Disk Used by Databases?
17
© Copyright 2023 Percona ® LLC. All rights reserved
19. • kubectl get persistentvolumeclaims
Disk Diagnostics
19
© Copyright 2023 Percona ® LLC. All rights reserved
20. • kubectl get persistentvolumeclaims
• kubectl describe nodes
Disk Diagnostics
20
© Copyright 2023 Percona ® LLC. All rights reserved
21. • kubectl get persistentvolumeclaims
• kubectl describe nodes
• Error logs of the database
2018-11-28T00:03:26.322318+06:00 0 [Note] InnoDB: page_cleaner:
1000ms intended loop took 4188ms. The settings might not be optimal.
(flushed=692 and evicted=0, during the time.)
Disk Diagnostics
21
© Copyright 2023 Percona ® LLC. All rights reserved
22. • kubectl get persistentvolumeclaims
• kubectl describe nodes
• Error logs of the database
2018-11-28T00:03:26.322318+06:00 0 [Note] InnoDB: page_cleaner:
1000ms intended loop took 4188ms. The settings might not be optimal.
(flushed=692 and evicted=0, during the time.)
• Counters in the database
Disk Diagnostics
22
© Copyright 2023 Percona ® LLC. All rights reserved
23. • Handle connections
• Threads in MySQL, MongoDB
• Sub-processes in PostgreSQL
• Perform calculations
• Perform IO operations
How CPU Used by Databases?
23
© Copyright 2023 Percona ® LLC. All rights reserved
25. • kubectl top pods
• kubectl top nodes
• kubectl describe nodes
CPU Diagnostics
25
© Copyright 2023 Percona ® LLC. All rights reserved
26. • kubectl top pods
• kubectl top nodes
• kubectl describe nodes
• Counters in the database
CPU Diagnostics
26
© Copyright 2023 Percona ® LLC. All rights reserved
27. • Communication between server and
client
• Regular client
• Application
• Replicas
• Synchronous nodes
How Databases Use Network?
27
© Copyright 2023 Percona ® LLC. All rights reserved
28. • kubectl describe nodes
Network Diagnostics
28
© Copyright 2023 Percona ® LLC. All rights reserved
29. • kubectl describe nodes
• Logs and statuses of the namespace
kube-system and your CNI
Network Diagnostics
29
© Copyright 2023 Percona ® LLC. All rights reserved
30. • kubectl describe nodes
• Logs and statuses of the namespace
kube-system and your CNI
• Latency in databases
Network Diagnostics
30
© Copyright 2023 Percona ® LLC. All rights reserved
31. • By default, log files are gone after pod
failure
• Critical in such situations as database
crash
Log Files Rotation
31
© Copyright 2023 Percona ® LLC. All rights reserved
32. • By default, log files are gone after pod
failure
• Critical in such situations as database
crash
• Solution
• PXC Operator: separate pod to store logs
• Other operators: K8-recommended
solutions such as Grafana Loki
Log Files Rotation
32
© Copyright 2023 Percona ® LLC. All rights reserved
33. • Created for Support team to automate
K8 troubleshooting collections
• Owned by Support team
• In active progress
• New featurs each version
pt-k8s-debug-collector
33
© Copyright 2023 Percona ® LLC. All rights reserved
34. pt-stalk
• Default for bare
metal
• Bare hardware
• Such tools as
• iostat
• vmstat
• dmesg
• top
pt-k8s-debug-collector
• Default for K8
• kubectl
• Such commands
as
• get pods
• get
storageclasses
• get replicasets
• get logs
Batch Data Collection
34
© Copyright 2023 Percona ® LLC. All rights reserved
35. • Basic batch collections per databases
• MySQL and PXC
pt-summary
pt-mysql-summary
Log files
• MongoDB
pt-mongodb-summary
• PostgreSQL
pg_gather
Features in pt-k8s-debug-collector
35
© Copyright 2023 Percona ® LLC. All rights reserved
36. • Basic batch collections per databases
• MySQL and PXC
pt-summary
pt-mysql-summary
Log files
• MongoDB
pt-mongodb-summary
• PostgreSQL
pg_gather
• kubectl get
Features in pt-k8s-debug-collector
36
© Copyright 2023 Percona ® LLC. All rights reserved
37. • Basic batch collections per databases
• MySQL and PXC
pt-summary
pt-mysql-summary
Log files
• MongoDB
pt-mongodb-summary
• PostgreSQL
pg_gather
• kubectl get
• kubectl logs
Features in pt-k8s-debug-collector
37
© Copyright 2023 Percona ® LLC. All rights reserved
38. • pt-stalk
• More kubectl commands
• Extended MongoDB support
• Nicer output format for summaries
Future of the pt-k8s-debug-collector
38
© Copyright 2023 Percona ® LLC. All rights reserved
39. • Mind limits in cr.yaml
• Work with K8 administrators for setting
up policies, allowing you to view
hardware usage
• Setup Loki or similar solution to save the
log files after restart
• Enable online database
instrumentation
• Use PMM or similar monitoring solution
• Plan in advance
Conclusion
39
© Copyright 2023 Percona ® LLC. All rights reserved
40. Percona K8 Operators
PMM Demo
pt-k8s-debug-collector
pt-stalk
pg_gather
Grafana Loki
Tools Mentioned
40
© Copyright 2023 Percona ® LLC. All rights reserved