This document introduces Kubernetes monitoring. It discusses monitoring tools like Datadog and Prometheus, how they work, and how to set them up. It provides an overview of monitoring concepts like observability and the monitoring landscape. Example metrics are shown for Datadog and Prometheus, and demos of Prometheus and Grafana are presented.

1. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 1/31
Kubernetes Monitoring
Introduction
1 / 31

2. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 2/31
Kyohei Mizumoto(@kyohmizu)
C# Software Engineer
Interests
Docker/Kubernetes
Go
Security
whoami
2 / 31

3. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 3/31
Required
Kubernetesの概要程度の知識
Targets
Kubernetes Monitoring未経験者
3 / 31

4. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 4/31
Contents
1. Monitoring Overview
2. Datadog
3. Prometheus
4. Demo
4 / 31

5. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 5/31
Monitoring Overview
5 / 31

6. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 6/31
Monitoring
Problem Detection
ダッシュボード、アラート
Problem Resolution
根本原因の特定、トラブルシュート
Continuous Improvement
キャパシティ、コスト最適化
6 / 31

7. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 7/31
Observability
A measure of how well internal states of a system can be
inferred from knowledge of its external outputs
7 / 31

8. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 8/31
Landscape
8 / 31

9. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 9/31
Tools
Datadog
Prometheus
Managed Service
Azure Monitor for containers
Stackdriver(for GKE)
9 / 31

10. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 10/31
Tools
Datadog
Prometheus
Managed Service
Azure Monitor for containers
Stackdriver(for GKE)
10 / 31

11. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 11/31
Datadog
11 / 31

12. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 12/31
SaaS型の監視＆解析ツール
課⾦制
様々なサービスのメトリクスをシー
ムレスに集約
Datadog
12 / 31

13. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 13/31
Dashboard
13 / 31

14. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 14/31
Metrics
ホストのCPU使⽤率、ディスク使⽤率
ノード上のコンテナのCPU使⽤率
Deployment のPod数、要求起動数
Jobの成功数、失敗数
etc...
14 / 31

15. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 15/31
Prometheus
15 / 31

16. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 16/31
CNCFがホストするOSSの監視ツー
ル
時系列でメトリクスを収集
Reliability
Prometheus
16 / 31

17. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 17/31
Architecture
pull metrics
HDD / SSD
Pushgateway
Short-lived jobs
Jobs / Exporters
StorageRetrieval PromQL
Prometheus Server
Node
Service Discovery
find
targets
Prometheus Server
Alertmanager
push alerts
Web UI
Grafana
API clients
PagerDuty Email
DNS
Kubernetes
Consul
...
Custom integration
notify
...
17 / 31

18. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 18/31
Architecture
Prometheus Server
メトリクスの収集(Pull型)、保存
Alert Manager
Exporter
要求に応じてメトリクスを送信
Push Gateway
メトリクスをPushしておく
18 / 31

19. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 19/31
Get Started
19 / 31

20. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 20/31
Use Helm
Helmを使⽤してマニフェストを作成
公式サイト︓https://helm.sh/
# Chartのダウンロード
$ helm fetch stable/prometheus --version 8.8.0
# Chartを元にマニフェストを作成
$ helm template --name sample-prometheus
prometheus-8.8.0.tgz
> sample-prometheus.yaml
20 / 31

21. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 21/31
Values
LBとして外部に公開する場合は、valuesファイルを⽤意
しておく
alertmanager:
service:
type: LoadBalancer
pushgateway:
service:
type: LoadBalancer
server:
service:
type: LoadBalancer
21 / 31

22. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 22/31
Deploy
サンプルをデプロイ
https://github.com/Kyohei-M/slide-k8s-monitoring/blob/master/sample-
prometheus.yaml
# Prometheusを起動
$ kubectl apply -f sample-prometheus.yaml
22 / 31

23. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 23/31
Demo
23 / 31

24. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 24/31
Prometheus Server
24 / 31

25. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 25/31
Prometheus Server
25 / 31

26. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 26/31
PromQL
Prometheus Query Language
# Example
# apiserverのhttpリクエスト合計
http_requests_total{job="apiserver"}
# 直近5分のhttpリクエストの増加率
rate(http_requests_total[5m])[30m:1m]
# 直近1時間での空きメモリ量の差
delta(node_memory_MemFree_bytes[1h])
26 / 31

27. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 27/31
Grafana
OSSのデータ可視化ツール
27 / 31

28. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 28/31
Grafana
helmでインストール
$ helm install stable/grafana
28 / 31

29. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 29/31
Books
29 / 31

30. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 30/31
Links
Monitoring and Observability
https://thenewstack.io/monitoring-and-observability-whats-the-
difference-and-why-does-it-matter/
Datadog
https://www.datadoghq.com/
Prometheus
https://prometheus.io/
Grafana
https://grafana.com/
30 / 31

31. 2019/3/28 Kubernetes Monitoring Introduction
127.0.0.1:5500/index.html#2 31/31
Thank you!
31 / 31