1. Making sure your containers
aren’t on fire
Monitoring microservices with Prometheus
Brice Fernandes
@fractallambda

3. Getting started with Kubernetes1
2
3
4
The monitoring maturity ladder
Whitebox vs blackbox monitoring
Monitoring with Prometheus
Using PromQL5

4. Getting started with Kubernetes1
2
3
4
The monitoring maturity ladder
Whitebox vs blackbox monitoring
Monitoring with Prometheus
Using PromQL5

5. How
I.T.
Was
OS
App

6. How
I.T.
Was
OS
App
Foo v1.1.0

7. How
I.T.
Was
OS
App
Foo v1.1.0 Foo v1.5.0

8. How
I.T.
Was
OS
App
Foo v1.1.0 Foo v1.5.0
?

10. How
I.T.
Was
Reproducible Deployment
Continuous Deployment
Fault Recovery
Memory & CPU allocation
Managing VMs?
?
?
?
?
?
?

11. The
New
Hotness
OS
Manager
Container
App

12. The
New
Hotness
OS
Manager
Container
App
Somebody Else’s Problem (SEP)™

13. The
New
Hotness
OS
Manager
Container
App
Somebody Else’s Problem (SEP)™

14. Reproducible deployments
Fault recovery
Continuous deployment
Don’t care about machine virtualisation
Memory & CPU multiplexing
Buzzword compliance
The
New
Hotness

15. But…

16. But…

17. But…

18. But…

19. But…

20. Mo’ containers
Mo’ problems

21. Kubernetes
–
Greek for Helmsman or Pilot

24. Master
kube-apiserver
kube-controller-manager
kube-scheduler

26. Node
kubelet
kube-proxy

28. This is what I want

29. This is what I want

30. This is what I want

31. This is what I want

32. This is what I want
xxx.xxx.xxx.xxx:30003

33. This is what I want
xxx.xxx.xxx.xxx:30003

34. xxx.xxx.xxx.xxx:30003

35. Set up
Kubernetes

36. ➤ minikube start

37. ➤ minikube start
Starting local Kubernetes v1.7.5 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components…
Kubectl is now configured to use the cluster.

38. ➤ minikube start
Starting local Kubernetes v1.7.5 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components…
Kubectl is now configured to use the cluster.
Start a local cluster

39. ➤ minikube start
Starting local Kubernetes v1.7.5 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components…
Kubectl is now configured to use the cluster.
Set up the kubernetes tools
to point to our cluster

40. ➤ kubectl get all

41. ➤ kubectl get all
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kubernetes 10.0.0.1 <none> 443/TCP 5m

42. ➤ kubectl get all
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/kubernetes 10.0.0.1 <none> 443/TCP 5m
Default kubernetes service

43. HOST

44. HOST
VM

45. HOST
VM
Kubernetes

46. Deploy an app

47. ➤ kubectl apply -f
https://tinyurl.com/kube-prom-demo-v1

48. ➤ kubectl apply -f
https://tinyurl.com/kube-prom-demo-v1
Our definition manifest

50. Where to find the image

51. How many to run

52. Which port to expose externally

53. ➤ kubectl apply -f
https://tinyurl.com/kube-prom-demo-v1
deployment "mighty-fine-fe" created
service "mighty-fine-fe" created
Creates our pods

54. ➤ kubectl apply -f
https://tinyurl.com/kube-prom-demo-v1
deployment "mighty-fine-fe" created
service "mighty-fine-fe" created
Exposes a service

55. ➤ kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S)
kubernetes 10.0.0.1 <none> 443/TCP
mighty-fine-fe 10.0.0.223 <nodes> 3000:30001/TCP
Port 3000 of app is visible
on port 30001 of cluster

56. ➤ open http://$(minikube ip):30001

57. ➤ open http://$(minikube ip):30001

58. Getting started with Kubernetes1
2
3
4
The monitoring maturity ladder
Whitebox vs blackbox monitoring
Monitoring with Prometheus
Using PromQL5

59. Why monitor?

60. Quality
Assurance
Continuous
Improvement

61. NOT about collecting data
Why vs How

62. IS relevant outside of IT

63. Q:What’s the
most important
metric?

64. A: What’s the
purpose of your
organisation?

65. Maybe:
Educational goals
# People reached
# Papers published

66. Probably £/$/€

67. Metrics come from
purpose.
Monitor your goals

68. Ignorance
Availability
Collection
Aggregation
0
Analysis
1
Learning
Automation
Proactivity
2
3
4
5
6
7
The
Monitoring
Ladder

69. Ignorance0
The
Monitoring
Ladder
Availability
Collection
Aggregation
Analysis
1
Learning
Automation
Proactivity
2
3
4
5
6
7
You don’t know
what’s going on.

70. Ignorance0
The
Monitoring
Ladder
Availability
Collection
Aggregation
Analysis
1
Learning
Automation
Proactivity
2
3
4
5
6
7
You know whether your
systems are available.
You may have alerts

71. Ignorance0
The
Monitoring
Ladder
Availability
Logging
Aggregation
Analysis
1
Learning
Automation
Proactivity
2
3
4
5
6
7
You collect logs.
Forensics is possible .
Alerts

72. Ignorance0
The
Monitoring
Ladder
Availability
Collection
Aggregation
Analysis
1
Learning
Automation
Proactivity
2
3
4
5
6
7
You aggregate and persist
data in a central place.
Correlation is possible.
Alerts
Logs
Forensics

73. Ignorance0
The
Monitoring
Ladder
Availability
Collection
Analysis
1
Learning
Automation
Proactivity
2
4
5
6
7
You actually analyse
the aggregated and
correlated data.
Use it to fix issues. Alerts
Logs
Forensics
Aggregation3Persistence

74. Ignorance0
The
Monitoring
Ladder
Availability
Collection
Analysis
1
Learning
Automation
Proactivity
2
4
5
6
7
Root cause analysis.
Strengthening fixes.
Antifragile.
Still responsive. Alerts
Logs
Forensics
Aggregation3Persistence

75. Ignorance0
The
Monitoring
Ladder
Availability
Collection
Analysis
1
Learning
Automation
Proactivity
2
4
5
6
7
Automated remedial actions.
Data collection for analysis.
No customer impact.
Alerts
Logs
Forensics
Aggregation3Persistence
Antifragile

76. Ignorance0
The
Monitoring
Ladder
Availability
Collection
Analysis
1
Learning
Automation
Proactivity
2
4
5
6
7
Automated remedial actions.
Data collection for analysis.
No customer impact.
Alerts
Logs
Forensics
Aggregation3Persistence
Antifragile

77. Ignorance0
The
Monitoring
Ladder
Availability
Collection
Analysis
1
Learning
Automation
Proactivity
2
4
5
6
7
Active strengthening
by attacking
production systems.
Alerts
Logs
Forensics
Aggregation3Persistence
Antifragile
0-Impact

78. Ignorance0
The
Monitoring
Ladder
Availability
Collection
Analysis
1
Learning
Automation
Proactivity
2
4
5
6
7
Alerts
Logs
Forensics
Aggregation3Persistence
Antifragile
0-Impact
Monitoring is
a broad topic

79. Getting started with Kubernetes1
2
3
4
The monitoring maturity ladder
Whitebox vs blackbox monitoring
Monitoring with Prometheus
Using PromQL5

80. Whitebox vs Blackbox

81. Whitebox vs Blackbox

82. Push vs Pull

83. Push vs Pull

84. Realtime vs Historic

85. Realtime vs Historic

86. Which one is right?
Pull or Push?
Whitebox or Blackbox?

87. Which one is right?
Pull or Push?
Whitebox or Blackbox?
Both

88. Pull based
Whitebox
Historic

89. Getting started with Kubernetes1
2
3
4
The monitoring maturity ladder
Whitebox vs blackbox monitoring
Monitoring with Prometheus
Using PromQL5

90. Monitoring infrastructure
Key metrics

91. ➤ kubectl apply -f
https://tinyurl.com/kube-prom-monitoring
deployment "prometheus" created
service "prometheus" created
service "internal-prometheus" created
deployment "grafana" created
service "grafana" created
configmap "prometheus-configmap" created
Create and expose
Prometheus

92. ➤ kubectl apply -f
https://tinyurl.com/kube-prom-monitoring
deployment "prometheus" created
service "prometheus" created
service "internal-prometheus" created
deployment "grafana" created
service "grafana" created
configmap "prometheus-configmap" created Create and expose
Grafana

93. ➤ kubectl apply -f
https://tinyurl.com/kube-prom-monitoring
deployment "prometheus" created
service "prometheus" created
service "internal-prometheus" created
deployment "grafana" created
service "grafana" created
configmap "prometheus-configmap" created
Configure Prometheus
Using a ConfigMap

94. ➤ kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S)
grafana 10.0.0.120 <nodes> 3000:30002/TCP
internal-prometheus 10.0.0.39 <none> 9090/TCP
kubernetes 10.0.0.1 <none> 443/TCP
mighty-fine-fe 10.0.0.223 <nodes> 3000:30001/TCP
prometheus 10.0.0.112 <nodes> 9090:30003/TCP

95. ➤ kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S)
grafana 10.0.0.120 <nodes> 3000:30002/TCP
internal-prometheus 10.0.0.39 <none> 9090/TCP
kubernetes 10.0.0.1 <none> 443/TCP
mighty-fine-fe 10.0.0.223 <nodes> 3000:30001/TCP
prometheus 10.0.0.112 <nodes> 9090:30003/TCP
Prometheus internal IP

96. ➤ kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S)
grafana 10.0.0.120 <nodes> 3000:30002/TCP
internal-prometheus 10.0.0.39 <none> 9090/TCP
kubernetes 10.0.0.1 <none> 443/TCP
mighty-fine-fe 10.0.0.223 <nodes> 3000:30001/TCP
prometheus 10.0.0.112 <nodes> 9090:30003/TCP
Prometheus external port

97. ➤ open http://$(minikube ip):30003

98. ➤ open http://$(minikube ip):30003

99. ➤ open http://$(minikube ip):30002

100. ➤ open http://$(minikube ip):30002

101. Built-in
Prometheus
provider

102. Internal
Prometheus
IP
and port

103. Proxy
instead
of data from
browser

104. PromQL
Query

105. But…
Aggregation3
Ignorance
Availability
Collection
0
Analysis
1
Learning
Automation
Proactivity
2
4
5
6
7
What about persistence?

106. Using
Weave Cloud’s
Hosted Prometheus

107. Name your cluster

108. Pick
your
platform

109. Choose
your
environment

110. Run
Command

111. ➤ kubectl apply
-n kube-system
-f “<some_url>&t=<some_token>”
serviceaccount "weave-flux" created
clusterrole "weave-flux" created
clusterrolebinding "weave-flux" created
secret "flux-git-deploy" created
deployment "weave-flux-memcached" created
service "weave-flux-memcached" created
deployment "weave-flux-agent" created
serviceaccount "weave-scope" created
clusterrole "weave-scope" created
clusterrolebinding "weave-scope" created
daemonset "weave-scope-agent" created
serviceaccount "weave-cortex" created
clusterrole "weave-cortex" created
clusterrolebinding "weave-cortex" created
deployment "weave-cortex-agent" created
service "weave-cortex-agent" created
daemonset "weave-cortex-node-exporter" created
configmap "weave-cortex-agent-config" created

112. ➤ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS
kube-addon-manager-minikube 1/1 Running 1
kube-dns-910330662-bv35c 3/3 Running 3
kubernetes-dashboard-zj028 1/1 Running 1
weave-cortex-agent-815474457-5q0rg 1/1 Running 0
weave-cortex-node-exporter-5tf88 1/1 Running 0
weave-flux-agent-1731903026-d0gw8 1/1 Running 0
weave-flux-memcached-2601059440-f31vp 1/1 Running 0
weave-scope-agent-6fq0b 1/1 Running 0

113. Go to monitoring

115. Monitoring infrastructure
Key metrics

116. Adding the
Prometheus
Agent to our app

117. ➤ npm install —save epimetheus

118. ➤ npm install —save epimetheus
Client libraries in: Go, Java, Python,
Ruby, Bash, C++, Common Lisp,
Elixir, Erlang, Haskell, Lua, .NET,
PHP, Rust…

120. Very straight
forward
in most languages

121. …
Omitted for brevity:
Pushing new image to registry
Creating new manifest
…

122. ➤ kubectl apply -f
https://tinyurl.com/kube-prom-demo-v2

123. ➤ kubectl apply -f
https://tinyurl.com/kube-prom-demo-v2
deployment "mighty-fine-fe" configured
service "mighty-fine-fe" configured

124. ➤ open http://$(minikube ip):30001/metrics

125. ➤ open http://$(minikube ip):30001/metrics

126. Getting
Prometheus
to scrape our app

128. ➤ kubctl apply -f
https://tinyurl.com/kube-prom-monitoring-v2

129. ➤ kubctl apply -f
https://tinyurl.com/kube-prom-monitoring-v2
deployment "prometheus" configured
service "prometheus" configured
service "internal-prometheus" configured
deployment "grafana" configured
service "grafana" configured
configmap "prometheus-configmap" configured

130. ➤ curl -X POST
http://$(minikube ip):30001/-/reload
Tell Prometheus to
reload its config

131. NodeJS
Metrics

132. Weave
discovers
the new
Metrics
too

133. Defining a
custom metric

135. …
Omitted for brevity:
Pushing new image to registry
Creating new manifest
…

136. ➤ kubctl apply -f
https://tinyurl.com/kube-prom-monitoring-v3
deployment "prometheus" configured
service "prometheus" configured
service "internal-prometheus" configured
deployment "grafana" configured
service "grafana" configured
configmap "prometheus-configmap" configured

137. ➤ open http://$(minikube ip):30001

138. ➤ open http://$(minikube ip):30001

139. ➤ open http://$(minikube ip):30001

140. Custom
Metric

141. Monitoring infrastructure
Key metrics

142. Getting started with Kubernetes1
2
3
4
The monitoring maturity ladder
Whitebox vs blackbox monitoring
Monitoring with Prometheus
Using PromQL5

143. Joel York’s SaaS Metrics
http://chaotic-flow.com

144. Worked
Example:
Churn rate
C × Δt
Churn Ratemonth = ΔCcancel

145. Worked
Example:
Churn rate
C × Δt
Number of
cancellations
In interval
Churn Ratemonth = ΔCcancel

146. Worked
Example:
Churn rate
C × Δt
Number of
cancellations
In interval
Churn Ratemonth = ΔCcancel
Number of customers
(at start of interval)

147. Worked
Example:
Churn rate
Time interval
Number of customers
(at start of interval)
Number of
cancellations
In interval
Churn Ratemonth = ΔCcancel
C × Δt

148. Worked
Example:
Churn rate
Churn Ratemonth = ΔCcancel
C × Δt

149. Worked
Example:
Churn rate
Assumed metrics:
total_signups (counter)
total_cancels (counter)
Churn Ratemonth = ΔCcancel
C × Δt

150. Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Churn Ratemonth = ΔCcancel
C × Δt

151. Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Base metric
(scalar)
Churn Ratemonth = ΔCcancel
C × Δt

152. Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Base metric
(scalar)
Churn Ratemonth = ΔCcancel
C × Δt
t0, t1, t2, t3, t4, t5, t6

153. Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Base metric
(scalar) Data window
Churn Ratemonth = ΔCcancel
C × Δt

154. Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Base metric
(scalar) Data window
Data range
(vector)
Churn Ratemonth = ΔCcancel
C × Δt

155. Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Base metric
(scalar) Data window
Data range
(vector)
Churn Ratemonth = ΔCcancel
C × Δt
t0, t1, t2, t3, t4, t5, t6
0
2
4
7
9
11
…
0
2
4
7
9
11
…
0
2
4
7
9
11
…
0
2
4
7
9
11
…
0
2
4
7
9
11
…
0
2
4
7
9
11
…
0
2
4
7
9
11
…

156. Worked
Example:
Churn rate
ΔCcancel = rate(total_cancels[1m])
Base metric
(scalar) Data window
Data range
(vector)
Built-in rate function
Churn Ratemonth = ΔCcancel
C × Δt

157. Worked
Example:
Churn rate
C = (total_signups offset 1m) -
(total_cancels offset 1m)
Churn Ratemonth = ΔCcancel
C × Δt

158. Worked
Example:
Churn rate
C = (total_signups offset 1m) -
(total_cancels offset 1m)
Churn Ratemonth = ΔCcancel
C × Δt

159. Worked
Example:
Churn rate
C = (total_signups offset 1m) -
(total_cancels offset 1m)
One month ago
Churn Ratemonth = ΔCcancel
C × Δt

160. Worked
Example:
Churn rate
Churn Ratemonth =
rate(total_cancels[1m]) /
((total_signups offset 1m) - (total_cancels offset 1m))
Churn Ratemonth = ΔCcancel
C × Δt

161. Worked
Example:
Churn rate
Churn Ratemonth =
rate(total_cancels[1m]) /
((total_signups offset 1m) - (total_cancels offset 1m))
Churn Ratemonth = ΔCcancel
C × Δt

162. Getting started with Kubernetes1
2
3
4
The monitoring maturity ladder
Whitebox vs blackbox monitoring
Monitoring with Prometheus
Using PromQL5
Review

163. References & useful links
- https://landing.google.com/sre/book/chapters/monitoring-distributed-systems.html
- http://www.ncsysadmin.org/meetings/1010/Monitoring_and_Alerting.pdf
- https://www.oreilly.com/ideas/monitoring-distributed-systems
- https://www.slideshare.net/brianbrazil/monitoring-what-matters-the-prometheus-approach-to-
whitebox-monitoring-berlin-ops-summit-2016
Thank You!
Brice Fernandes
@fractallambda
@weaveworks
Slides: https://tinyurl.com/prometheus-kubernetes-slides
Code: https://tinyurl.com/prometheus-kubernetes-code
Video: https://tinyurl.com/cloud-native-2017
https://weave.works