Monitoring cloud native apps creates a whole new set of challenges that traditional monitoring systems struggle with. In this Workshop, Brice Fernandes from Weaveworks will take you through installing and using the open source Prometheus monitoring toolkit to monitor a Kubernetes cluster. You do not need any previous experience with Kubernetes or Prometheus. After this talk, you'll be able to set up a development kubernetes cluster and use Prometheus to monitor your microservices. We'll cover: An introduction to Kubernetes to manage containers; The monitoring maturity model; An overview of whitebox and blackbox monitoring; Monitoring with Prometheus; Using PromQL (the Prometheus Query Language) to monitor your app in a dynamic system.

1. Making sure your cluster isn't on fire
Hands-on monitoring with Prometheus
Brice Fernandes – brice@weave.works – February 2018
1

2. Q: Who uses docker?
Q: Who uses Kubernetes?
Q: Who uses Prometheus?
Hello Amsterdam 👋
2

3. Deploy and test a service
Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
3
&
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Log in to cloud 9 instance
Instrument the application
Inspect business metrics
BREAK
Q & A
💻

4. Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
4
&
Log in to cloud 9 instance
💻
Deploy and test a service
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Instrument the application
Inspect business metrics
BREAK
Q & A

5. Navigate to http://INSTANCE-NAME.training.weave.works
Type your username and password
Setting up your environment
5

6. 6

7. 7
Application code

8. 8
Terminal to cluster

9. 9
Log in to cloud 9 instance 💻
1. Log in to your instance
2. Take a look at the demo application in demo-
app.
3. Find out what’s running on the cluster with
➤ docker ps

10. Logging into
your test
environment

11. Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
11
&
Log in to cloud 9 instance
💻
Deploy and test a service
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Instrument the application
Inspect business metrics
BREAK
Q & A

12. Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
12
&
Log in to cloud 9 instance
💻
Deploy and test a service
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Instrument the application
Inspect business metrics
BREAK
Q & A

13. 13
First…
… a fast and inaccurate
history lesson

14. 14
App
OS

15. 15
App
OS
Foo v1.1.0

16. 16
App
OS
Foo v1.1.0
Foo v2.5.0

17. 17
App
OS?
Foo v1.1.0
Foo v2.5.0

18. 18

19. 19
?
Reproducible deployments
Managing VMs
Fault Recovery
Resource Allocation
Continuous Deployments
?
?
?
?
?
?

20. 20
App
OS
Manager
Container

21. 21
App
OS
Manager
Container
Someone Else’s Problem (SEP)™

22. 22
App
OS
Manager
Container
Someone Else’s Problem (SEP)™

23. 23
Reproducible deployments
Fault recovery
Continuous deployment
Don’t care about machine virtualization
Memory & CPU multiplexing
Buzzword compliance

24. 24

25. 25

26. 26

27. 27

28. 28

29. 29

30. 30
Mo’ containers
Mo’ problems

31. 31
Kubernetes
–
Greek for Helmsman or Pilot

32. 32

33. 33
kube-apiserver
kube-controller-manager
kube-scheduler
Master

34. 34

35. 35
kubelet
kube-proxy
Node

36. 36
This is what I want

37. 37
This is what I want

38. 38
This is what I want

39. 39
This is what I want
xxx.xxx.xxx.xxx

40. 40
This is what I want
xxx.xxx.xxx.xxx

41. 41
xxx.xxx.xxx.xxx

42. 42
Build the docker image 💻
1. Create the service image
➤ cd demo-app
➤ docker build -t mighty-fine:v1 .
2.Check it exists
➤ docker images

43. Building a
Docker
image

44. ➔Node App
➔Dockerfile

45. 45
Deploy the service using kubectl 💻
1. Edit the service definition file manifests/app.yaml to
refer to your version
2. Deploy the service
➤ kubectl apply -f manifests/app.yaml
3. Deploy the ingress
➤ kubectl apply -f manifests/ingress.yaml
4. Navigate to .../mighty-fine to check the service runs

46. Deploying a
service to a
kubernetes
cluster

47. ➔Deployment
➔Service
➔Ingress

49. ➔kubectl get
➔kubectl describe
➔kubectl logs

50. Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
50
&
Log in to cloud 9 instance
💻
Deploy and test a service
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Instrument the application
Inspect business metrics
BREAK
Q & A

51. Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
51
&
Log in to cloud 9 instance
💻
Deploy and test a service
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Instrument the application
Inspect business metrics
BREAK
Q & A

52. 52

53. 53
Prometheus
Display dataGather & Store
Query & Alerts
Grafana

54. 54
*.training.weave.works/prometheus

55. 55
PromQL query. Try

56. 56
Prometheus
Display dataGather & Store
Query & Alerts
Grafana

57. 57
*.training.weave.works/grafana

58. 58
Same query (go_memstats_alloc_bytes).
Persistent graphs, Dashboards

59. 59
Prometheus
Display dataGather & Store
Query & Alerts
Grafana

60. 60
How?

61. 61
demo-app/manifests/monitoring.yaml

62. Use the dashboard definition in
demo-app/resources/docker-monitoring_rev1.json
Create a dashboard using Grafana’s import functionality.
62
Inspect the system metrics 💻

63. Importing a
default
dashboard

64. ➔JSON all the way
➔Dashboards as data
➔Can start with
dashboards

65. Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
65
&
Log in to cloud 9 instance
💻
Deploy and test a service
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Instrument the application
Inspect business metrics
BREAK
Q & A

66. Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
66
&
Log in to cloud 9 instance
💻
Deploy and test a service
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Instrument the application
Inspect business metrics
BREAK
Q & A

67. Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
67
&
Log in to cloud 9 instance
💻
Deploy and test a service
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Instrument the application
Inspect business metrics
BREAK
Q & A

68. 68
Why monitor?

69. 69
Quality
Assurance
Continuous
Improvement

70. 70
NOT about collecting data
Why vs How

71. 71
IS relevant outside of IT

72. 72
Q: What’s the
most important metric?

73. 73
Q: What’s the purpose of
your organisation?

74. 74
Maybe:
Educational goals
# People reached
# Papers published

75. 75
Probably £/$/€

76. 76
Metrics come from
purpose.
Monitor your goals

77. 77
1 Availability
2 Collection
0 Ignorance
3 Aggregation
4 Analysis
5 Learning
6 Automation
7 ProactivityThe
Monitoring
Ladder

78. 78
1 Availability
2 Collection
3 Aggregation
4 Analysis
5 Learning
6 Automation
7 ProactivityThe
Monitoring
Ladder
0 Ignorance
You don’t know
what’s going on

79. 79
1 Availability
2 Collection
3 Aggregation
4 Analysis
5 Learning
6 Automation
7 ProactivityThe
Monitoring
Ladder
You know whether your
systems are available.
You may have alerts
0 Ignorance
1 Availability
0 Ignorance

80. 80
1 Availability
2 Collection
3 Aggregation
4 Analysis
5 Learning
6 Automation
7 ProactivityThe
Monitoring
Ladder
You collect logs.
Forensics is possible.
0 Ignorance
2 Collection
1 Availability
0 Ignorance
Alerts

81. 81
1 Availability
2 Collection
3 Aggregation
4 Analysis
5 Learning
6 Automation
7 ProactivityThe
Monitoring
Ladder
You aggregate and persist
data in a central place.
Correlation is possible.
0 Ignorance
2 Collection
1 Availability
0 Ignorance
Alerts
Logs
Forensics
3 Aggregation

82. 82
1 Availability
2 Collection
3 Aggregation
4 Analysis
5 Learning
6 Automation
7 ProactivityThe
Monitoring
Ladder
You actually analyse
the aggregated and
correlated data.
Use it to fix issues.
0 Ignorance
Alerts
3 Aggregation
Logs
Forensics
1 Availability
2 Collection
0 Ignorance
4 Analysis
Persistence

83. 83
1 Availability
2 Collection
3 Aggregation
4 Analysis
5 Learning
6 Automation
7 ProactivityThe
Monitoring
Ladder
Root cause analysis.
Strengthening fixes.
Antigfragile.
Still responsive.
0 Ignorance
Alerts
3 Aggregation
Logs
Forensics
1 Availability
2 Collection
0 Ignorance
4 Analysis
Persistence
5 Learning

84. 84
1 Availability
2 Collection
3 Aggregation
4 Analysis
5 Learning
6 Automation
7 ProactivityThe
Monitoring
Ladder
Automated remediations.
Incident report creation.
No customer impact.
0 Ignorance
Alerts
3 Aggregation
Logs
Forensics
1 Availability
2 Collection
0 Ignorance
4 Analysis
Persistence
5 Learning
6 Automation
Antifragile

85. 85
1 Availability
2 Collection
3 Aggregation
4 Analysis
5 Learning
6 Automation
7 ProactivityThe
Monitoring
Ladder
0 Ignorance
Alerts
3 Aggregation
Logs
Forensics
1 Availability
2 Collection
0 Ignorance
4 Analysis
Persistence
5 Learning
6 Automation
Antifragile
0-impact

86. 86
1 Availability
2 Collection
3 Aggregation
4 Analysis
5 Learning
6 Automation
7 ProactivityThe
Monitoring
Ladder
0 Ignorance
Alerts
3 Aggregation
Logs
Forensics
1 Availability
2 Collection
0 Ignorance
4 Analysis
Persistence
5 Learning
6 Automation
Antifragile
0-impact
Active strengthening
by attacking
production systems.
“Chaos Engineering”
7 Proactivity

87. 87
1 Availability
2 Collection
3 Aggregation
4 Analysis
5 Learning
6 Automation
7 ProactivityThe
Monitoring
Ladder
0 Ignorance
Alerts
3 Aggregation
Logs
Forensics
1 Availability
2 Collection
0 Ignorance
4 Analysis
Persistence
5 Learning
6 Automation
Antifragile
0-impact
7 ProactivityChaos Engineering
Monitoring is a
broad topic

88. 88
1 Availability
2 Collection
3 Aggregation
4 Analysis
5 Learning
6 Automation
7 ProactivityThe
Monitoring
Ladder
0 Ignorance
Alerts
3 Aggregation
Logs
Forensics
1 Availability
2 Collection
0 Ignorance
4 Analysis
Persistence
5 Learning
6 Automation
Antifragile
0-impact
7 ProactivityChaos Engineering
Boundaries
aren’t so
clear cut.

89. 89
Whitebox vs Blackbox

90. 90
Whitebox vs Blackbox

91. 91
Pull vs Push

92. 92
Pull vs Push

93. 93
Realtime vs Historic

94. 94
Realtime vs Historic

95. 95
Which one is right?
Pull or Push?
Whitebox or Blackbox?

96. 96
Which one is right?
Pull or Push?
Whitebox or Blackbox?
Both

97. 97
Pull based
Whitebox
Historic

98. Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
98
&
Log in to cloud 9 instance
💻
Deploy and test a service
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Instrument the application
Inspect business metrics
BREAK
Q & A

99. Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
99
&
Log in to cloud 9 instance
💻
Deploy and test a service
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Instrument the application
Inspect business metrics
BREAK
Q & A

100. 100
Maslow’s
Pyramid of
Metrics (?)

101. 10
1
Maslow’s
Pyramid of
Metrics (?)
Platform
CPU, Memory, GC, JVM,
Ruby, threads and bytes.
Domain - Computers

102. 10
2
Maslow’s
Pyramid of
Metrics (?)
Platform
Requests, HTTP Errors,
Queue stats.
Domain - Protocols
Middleware

103. 10
3
Maslow’s
Pyramid of
Metrics (?)
Platform
Users, widgets,
teams, products.
Domain - Application Middleware
App

104. 10
4
Maslow’s
Pyramid of
Metrics (?)
Platform
Money, subscriptions, growth.
Domain - Business
Middleware
App
Business

105. 10
5
Let’s start with
platform metrics
Platform

106. 10
6
Let’s start with
platform metrics
… there already 😊

107. 10
7
Let’s move to
middleware
metrics
Platform
Middleware

108. 1. Checkout v2-default-metrics
2. Rebuild the image with a new tag
➤ docker build -t mighty-fine:v2 .
3. Change the manifests/app.yaml to refer to the new
image, and use kubectl to apply the manifest again.
4. Look in prometheus for nodejs* metrics
10
8
Instrument the app to get middleware metrics 💻

109. Instrumenting
with
middleware
metrics

110. ➔Straight forward
➔Client libraries
available for lots of
languages and
platforms

111. 11
1
Application
Metrics
Platform
Middleware
App

112. 1. Checkout v3-custom-metrics
2. Rebuild the image with a new tag
➤ docker build -t mighty-fine:v3 .
3. Change the manifests/app.yaml to refer to the new
image, and use kubectl to apply the manifest again.
4. Look in prometheus for sales* metrics
11
2
Instrument the app to get middleware metrics 💻

113. Instrumenting
with
application
metrics

114. ➔Also straightforward
➔Counters, Gauges,
Histograms

115. 11
5
Business
Metrics
Platform
Middleware
App
Business

116. 11
6
http://chaotic-flow.com/media/saas-metrics-guide-to-saas-financial-performance.pdf
Joel York’s SaaS Metrics
http://chaotic-flow.com

117. 11
7

118. 11
8

119. 11
9

120. 12
0

121. 12
1
We’ll need
subscribe_count
unsubscribe_count
(Both counters)

122. 1. Checkout v4-subscription-service
2. Rebuild the image with a new tag
➤ docker build -t mighty-fine:v4 .
3. Change the manifests/app.yaml to refer to the new
image, and use kubectl to apply the manifest again.
4. Look in prometheus for un/subscribe_count metrics
12
2
Instrument the app to get business metrics 💻

123. Instrumenting
with business
metrics

124. 12
4
ΔCcancel= rate(unsubscribe_count[1m])

125. 12
5
ΔCcancel= rate(unsubscribe_count[1m])

126. 12
6
ΔCcancel= rate(unsubscribe_count[1m])

127. 12
7
ΔCcancel= rate(unsubscribe_count[1m])

128. 12
8
ΔCcancel= rate(unsubscribe_count[1m])

129. 12
9
ΔCcancel= rate(unsubscribe_count[1m])

130. 13
0
ΔCcancel= rate(unsubscribe_count[1m])

131. 13
1
C = (total_signups offset 1m) -
(total_cancels offset 1m)

132. 13
2
ChurnRatemonth =
rate(total_cancels[1m]) /
((total_signups offset 1m) - (total_cancels offset 1m))

133. 13
3
ChurnRatemonth =
rate(total_cancels[1m]) /
((total_signups offset 1m) - (total_cancels offset 1m))

134. Live demo of
changing
metrics

135. ➔Simple metrics can
give deep insights
➔A little math goes a
long way

136. Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
13
&
Log in to cloud 9 instance
💻
Deploy and test a service
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Instrument the application
Inspect business metrics
BREAK
Q & A

137. Setting up an environment
Starting out with Kubernetes
- Introduction to Kubernetes
- Deploying to Kubernetes using kubectl
Monitoring in practice
- Setting up Prometheus
- Setting up Grafana
- Inspecting system metrics
Understanding Monitoring
- Monitoring: Why and How?
- Monitoring maturity
- The prometheus model
Monitoring in depth
- Using different types of metrics
- Using PromQL
- Instrumenting your code
13
&
Log in to cloud 9 instance
💻
Deploy and test a service
Inspect system metrics
Instrument the platform
Inspect middleware metrics
Instrument the application
Inspect business metrics
BREAK
Q & A

138. Q & A
13
8

139. THANK YOU!
13
9
Brice Fernandes
@fractallambda
@weaveworks
https://weave.works
Slides: https://tinyurl.com/prometheus-kubernetes-slides
Code: https://tinyurl.com/prometheus-kubernetes-code
References:
● Joel York’s SaaS metrics http://chaotic-
flow.com/saas-metrics/

140. 14
0