This document summarizes concepts and tools for Kubernetes monitoring. It discusses generating metrics with Dropwizard Metrics, collecting and aggregating metrics with Prometheus, and visualizing metrics with Grafana. Key steps include annotating Pods and Services to expose metrics, deploying Prometheus with the Prometheus Operator, and integrating Node Exporters and Kubernetes metrics to expose system and resource metrics. The document concludes with a demo of the end monitoring setup.

1. CrashcourseinKubernetesmonitoring
RobertMunteanu,Adobe
Slides revision: 20191006-0bbb891
#Devoxx #KubeMonitoring @rombert
1

2. Aboutme
2

3. Outline
Monitoring concepts
Generating metrics
Collecting and aggregating metrics
Publishing metrics
Kubernetes integration
Demo
3

4. Monitoringconcepts
4

5. MonitoringforsimplesystemsMonitoringforsimplesystemsMonitoringforsimplesystemsMonitoringforsimplesystemsMonitoringforsimplesystems
5

6. MonitoringforcomplexsystemsMonitoringforcomplexsystemsMonitoringforcomplexsystemsMonitoringforcomplexsystemsMonitoringforcomplexsystems
6

7. Whataremetrics?
Information about the system at a given point in time:
running: yes
running on the 5th of October 2019 at 18:00? yes
running on the 5th of October 2019 at 18:00 on
host002.acme.com? yes
Optional, but recommended: atomic
7

8. Monitoringarchitecture
8

9. Generatingmetrics
9

10. DropWizardMetrics
A library for recording and publishing metrics.
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-servlets</artifactId>
<version>${metrics.version}</version>
</dependency>
10

11. DropWizardMetrics-Recording
public class Handler {
private final MetricRegistry metrics;
private final Counter requests =⏎
metrics.counter("requests");
public Handler(MetricsRegistry metrics) {
this.metrics = metrics;
}
public void handle(Request req, Response resp) {
requests.inc();
// actually handle the request
}
}
11

12. DropWizardMetrics-Publishing
public class Reporter {
private final MetricsRegistry registry;
public Reporter(MetricsRegistry metrics) {
this.metrics = metrics;
}
public void start() {
ConsoleRegistry.forRegistry(metrics)
.build()
.start(5, TimeUnit.SECONDS);
}
}
12

13. Collectingandaggregatingmetrics
13

14. Prometheus
Open-Source monitoring solution (d'oh)
graduated project
Time-series database backend
14

15. GettingdataintoPrometheus-metrics
endpoint
# HELP requests_total Total number of requests served
# TYPE requests_total counter
requests_total 42.0
requests_total{status="200",method="GET"} 22.0
# TYPE free_worker_threads gauge
free_worker_threads 15
# TYPE http_request_duration_seconds histogram
request_duration_seconds_bucket{le="0.5"} 0
request_duration_seconds_bucket{le="1"} 1
request_duration_seconds_bucket{le="2"} 2
request_duration_seconds_bucket{le="+Inf"} 3
request_duration_seconds_sum 6
request_duration_seconds_count 3
15

16. GettingdataintoPrometheus
16

17. Prometheusqueries
17

18. Prometheusqueries
18

19. Prometheusqueries
19

20. Prometheusqueries
20

21. Prometheusqueries
21

22. Publishingmetrics
22

23. Prometheusalerting
23

24. Prometheusalerting
24

25. Grafana
Open source analytics & monitoring solution
Large number of data sources: Graphite, InfluxDB, Prometheus,
ElasticSearch, etc
Excellent graphing and dashboarding capabilities
25

26. Grafanapanels
26

27. Grafanapanels
27

28. Grafanapanels
28

29. Grafanapanels
29

30. Grafanadashboards
30

31. Kubernetesintegration
31

32. Annotations
# partial
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/custom-metrics"
labels:
app: sling
spec:
containers:
name: sling
image: registry.homelab.lmn.ro/apache/sling-cloud-native
32

33. ServiceMonitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: example-app
labels:
team: frontend
spec:
selector:
matchLabels:
app: example-app
endpoints:
- port: web
33

34. NodeExporter
Exposes hardware and OS metrics for *nix kernels.
Typically runs on all nodes as a DaemonSet
Typical metrics: cpu, cpufreq, diskstats, entropy,
meminfo, etc.
34

35. Kubestatemetrics
Exposes metrics based on the states of kubernetes resources
Based on the Kubernetes API server
Typical metrics:
kube_pod_container_resource_requests_cpu_cores,
kube_pod_init_container_status_running,
kube_deployment_status_replicas,
kube_hpa_status_desired_replicas,
kube_hpa_status_current_replicas, etc.
35

36. Prometheusexporterssummary
36

37. BatteriesBatteriesBatteriesBatteriesBatteriesKubernetesnotincludedKubernetesnotincludedKubernetesnotincludedKubernetesnotincludedKubernetesnotincluded
37

38. Monitoringsetup
$ helm install --name prometheus --namespace monitoring ⏎
stable/prometheus
$ helm install --name grafana --namespace monitoring ⏎
--set persistence.enabled=true stable/grafana --values grafana-values.yaml
# fragment
dashboards:
default:
kube-dash:
gnetId: 6663
revision: 1
datasource: Prometheus
38

39. Demo
39

40. Endresult
40

41. Resources
https://sling.apache.org
https://github.com/rombert/sling-cloud-native
https://metrics.dropwizard.io/
https://prometheus.io/
https://github.com/coreos/prometheus-operator
https://grafana.com/
https://kubernetes.io/
41