Monitoring Kubernetes Clusters with Prometheus is state of the art. The difficulty is to find the significant metrics from the vast amount of available metrics. This talk shows a Monitoring Cockpit defined to get a quick overview of the cluster health and usage. It uses the Standard Metrics available for Kubernetes/OpenShift Clusters and their standard services. The monitoring solution is based on Prometheus, using InfluxDB for central long term storage and Grafana.

1. Monitoring Cockpit
for Kubernetes Clusters
Ulrike Klusik
5.11.2019

2. Monitoring Cockpit 2
Our Customers Kubernetes Implementation: OpenShift
• OpenShift is a commercial
Kubernetes implementation (OKD its
community version)
• From Monitoring perspective:
• Nodes as compute resources
• central service URLs with high
availability and performance SLAs,
• infrastructure Pods implementing
the services which can be
dynamically changing.
• The API already provides meta data
about the cluster components =>
used to determine the metric targets from https://docs.okd.io/3.11/architecture/index.html

3. Monitoring Cockpit 3
Monitoring with Prometheus
• Prometheus is good integrated in Kubernetes
ecosystem.
• Idea of Monitoring with Prometheus
• Monitored targets must provide their specific
metrics via http(s) endpoints
• Targets are typically determined dynamically
via service discovery and regularly scraped
• Alert rules are defined as conditions on metrics
• Alertmanager deduplicates alerts
and routes them to the incident handling tools
from https://prometheus.io/assets/architecture.png

4. Monitoring Cockpit 4
State of the Art Kubernetes Monitoring with Prometheus
• Prometheus Monitoring Mixin for Kubernetes
(https://github.com/kubernetes-monitoring/kubernetes-mixin)
• Provides for the standard Kubernetes services:
• Alert rules
• Dashboards
• Redhat’s OpenShift (a commercial Kubernetes implementation) includes an immutable
Prometheus Monitoring solution (https://github.com/openshift/cluster-monitoring-operator)
with fix alerts and dashboards. Also bases on the Mixins, plus some OpenShift specific
additions.
• What we were missing in these solutions:
• End user / application experience of cluster services
• Metrics volume too large for longer metric retention
• Cluster Overview over service and node availability

5. Monitoring Cockpit 5
namespace
Nodes
host
NODE-
EXPORTER
OMD server
INFLUXDB
ALERTMGR
(cluster possible)
Container
OMD-Service
Grafana
Monitoring Architecture
Kubelet +
cAdvisor
Openshift
metric target
HAProxy(Router)
infrastructure projects
remote write
(selected metrics)
Incident Mananagent
systems (e.g. Remedy,
Service Now)
custom webhook
api-servers
kube controllers
EFK Logging
(via Pods)
GlusterFS (via
Heketi-Route)
Project prometheus-infra-mon
PROMETHEUS
KSM/OSM
Kubernetes/OpenShift Cluster
• Kube-State-Metrics(KSM)/
OpenShift-State-Metrics(OSM):
metrics over objects and their
states
• Node-Exporter for operation
system metrics
• Blackbox Exporter: for test calls
to Service URLs
Blackbox
Exporter
Kubernetes
metric target

6. Monitoring Cockpit 6
Dashboards (Demo)

7. Monitoring Cockpit 7
Conclusion
• Special design decisions:
• remote write: The key metrics are stored in an external database for longer retention
• blackbox exporter: for active service availability tests
• Grafana and its plugins (especially polystat-panel) are an awesome tool to visualize
metrics in a compact way.

8. Thank You!

9. ConSol
Consulting & Solutions Software GmbH
St.-Cajetan-Straße 43
D-81669 München
Tel.: +49-89-45841-100
info@consol.de
www.consol.de
Twitter: @consol_de