The document discusses the architecture and components of Kubernetes monitoring, specifically focusing on the metrics pipeline. It outlines the roles of individual components like metrics-server, kubelet, and the container runtime interface in collecting and exposing resource metrics. The conclusion emphasizes the structured approach in which each component communicates via APIs to effectively manage Kubernetes resource metrics.

1. The Journey to the Kubernetes Metrics
智金處 智能系統部 黃晨懿
2021/12/22
1

2. Outline
• Scenario
• Kubernetes Monitoring Architecture
• Components details
• Conclusion
2

3. Scenario
• Monitoring cluster status at Grafana
• Check CPU and memory usage by kubectl top
3
example

4. Kubernetes Monitoring Architecture
4

5. source: https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/monitoring_architecture.md
K8s architecture
5

6. source: https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/monitoring_architecture.md
K8s architecture
6
Core metrics pipeline
Monitoring pipeline

7. Component details
7

8. kubectl top
• Check node and pod CPU and memory usage through CLI tool
provided by k8s
8
example

9. REST client: kubectl top -> API server
9
kubernetes/vendor/k8s.io/metrics/pkg/client/clientset/versioned/typed/metrics/v1beta1/nodemetrics.go

10. 10
kubernetes/vendor/k8s.io/metrics/pkg/apis/metrics/types.go

11. API server
• Core component to communicate with k8s in control plane
• Expose Resource Metrics API to the following components
• kubectl top
• kube-scheduler
• HPA
• …
source: https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/resource-metrics-api.md
11

12. metrics-server
• Store locally only latest values
• Expose the master metrics API
source: https://github.com/kubernetes-sigs/metrics-server
12

13. REST server: API server -> metrics-server
13
metrics-server/pkg/storage/node.go
metrics-server/pkg/storage/types.go

14. 14
metrics-server/pkg/server/server.go

15. 15
metrics-server/pkg/server/server.go

16. REST client: metrics-server -> kubelet
16
metrics-server/pkg/scraper/client/resource/client.go

17. Kubelet
• The daemon that runs on every kubernetes node and controls pod
and container lifecycle, among many other things.
• Combine container-level metrics form cAdvisor/CRI with the kubelets
knowledge of kubernetes constructs to produce the kubelet Summary
statistics.
source: https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/core-metrics-pipeline.md
17

18. 18
kubernetes/pkg/kubelet/kubelet.go

19. 19
kubernetes/pkg/kubelet/stats/cri_stats_provider.go

20. gRPC client: kubelet -> CRI
20
kubernetes/pkg/kubelet/cri/remote/remote_runtime.go

21. Container Runtime Interface (CRI)
• The Container Runtime Interface designed to provide an abstraction
over runtimes (docker, rkt, etc).
• Consist of specs, protobuf API and libraries for container runtimes to
integrate with kubelet on a node
source:https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/core-metrics-pipeline.md
21

22. gRPC client: kubelet -> CRI
22
kubernetes/pkg/kubelet/cri/remote/remote_runtime.go

23. 23
containerd/vendor/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto

24. gRPC server: kubelet -> containerd CRI
24
containerd/pkg/cri/server/container_stats_list.go

25. and it goes to Stats method…
25
containerd/runtime/v2/runc/v1/service.go

26. 26
containerd/vendor/github.com/containerd/cgroups/cpuacct.go

27. Conclusion
27

28. source: https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/monitoring_architecture.md
K8s architecture
28
Core metrics pipeline
Monitoring pipeline

29. • In Kubernetes Monitoring Architecture,
• Each component is responsible for producing, scraping and leveraging the
metrics
• Each component is communicated with API, such as RESTful API and gRPC
29

30. Backups
30

31. K8s architecture
31

32. 32

33. cgroup
• cgroup core organizes processes hierarchically
• cgroup controller distributes system resources along the hierarchy in
a controlled and configurable manner.
• cgroup controllers:
• cpu
• memory
• …
source:https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt
33
source:https://www.kernel.org/doc/Documentation/cgroup-v1/cpuacct.txt

34. 34
container = namespace + cgroup
example

35. 35
container ID = cgroup ID
example

36. 36
example