Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

3

Share

Download to read offline

Hands-on monitoring with Prometheus

Download to read offline

Monitoring cloud native apps creates a whole new set of challenges that traditional monitoring systems struggle with. In this Workshop, Brice Fernandes from Weaveworks will take you through installing and using the open source Prometheus monitoring toolkit to monitor a Kubernetes cluster. You do not need any previous experience with Kubernetes or Prometheus. After this talk, you'll be able to set up a development kubernetes cluster and use Prometheus to monitor your microservices. We'll cover: An introduction to Kubernetes to manage containers; The monitoring maturity model; An overview of whitebox and blackbox monitoring; Monitoring with Prometheus; Using PromQL (the Prometheus Query Language) to monitor your app in a dynamic system.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Hands-on monitoring with Prometheus

  1. 1. Making sure your cluster isn't on fire Hands-on monitoring with Prometheus Brice Fernandes – brice@weave.works – February 2018 1
  2. 2. Q: Who uses docker? Q: Who uses Kubernetes? Q: Who uses Prometheus? Hello Amsterdam 👋 2
  3. 3. Deploy and test a service Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 3 & Inspect system metrics Instrument the platform Inspect middleware metrics Log in to cloud 9 instance Instrument the application Inspect business metrics BREAK Q & A 💻
  4. 4. Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 4 & Log in to cloud 9 instance 💻 Deploy and test a service Inspect system metrics Instrument the platform Inspect middleware metrics Instrument the application Inspect business metrics BREAK Q & A
  5. 5. Navigate to http://INSTANCE-NAME.training.weave.works Type your username and password Setting up your environment 5
  6. 6. 6
  7. 7. 7 Application code
  8. 8. 8 Terminal to cluster
  9. 9. 9 Log in to cloud 9 instance 💻 1. Log in to your instance 2. Take a look at the demo application in demo- app. 3. Find out what’s running on the cluster with ➤ docker ps
  10. 10. Logging into your test environment
  11. 11. Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 11 & Log in to cloud 9 instance 💻 Deploy and test a service Inspect system metrics Instrument the platform Inspect middleware metrics Instrument the application Inspect business metrics BREAK Q & A
  12. 12. Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 12 & Log in to cloud 9 instance 💻 Deploy and test a service Inspect system metrics Instrument the platform Inspect middleware metrics Instrument the application Inspect business metrics BREAK Q & A
  13. 13. 13 First… … a fast and inaccurate history lesson
  14. 14. 14 App OS
  15. 15. 15 App OS Foo v1.1.0
  16. 16. 16 App OS Foo v1.1.0 Foo v2.5.0
  17. 17. 17 App OS? Foo v1.1.0 Foo v2.5.0
  18. 18. 18
  19. 19. 19 ? Reproducible deployments Managing VMs Fault Recovery Resource Allocation Continuous Deployments ? ? ? ? ? ?
  20. 20. 20 App OS Manager Container
  21. 21. 21 App OS Manager Container Someone Else’s Problem (SEP)™
  22. 22. 22 App OS Manager Container Someone Else’s Problem (SEP)™
  23. 23. 23 Reproducible deployments Fault recovery Continuous deployment Don’t care about machine virtualization Memory & CPU multiplexing Buzzword compliance
  24. 24. 24
  25. 25. 25
  26. 26. 26
  27. 27. 27
  28. 28. 28
  29. 29. 29
  30. 30. 30 Mo’ containers Mo’ problems
  31. 31. 31 Kubernetes – Greek for Helmsman or Pilot
  32. 32. 32
  33. 33. 33 kube-apiserver kube-controller-manager kube-scheduler Master
  34. 34. 34
  35. 35. 35 kubelet kube-proxy Node
  36. 36. 36 This is what I want
  37. 37. 37 This is what I want
  38. 38. 38 This is what I want
  39. 39. 39 This is what I want xxx.xxx.xxx.xxx
  40. 40. 40 This is what I want xxx.xxx.xxx.xxx
  41. 41. 41 xxx.xxx.xxx.xxx
  42. 42. 42 Build the docker image 💻 1. Create the service image ➤ cd demo-app ➤ docker build -t mighty-fine:v1 . 2.Check it exists ➤ docker images
  43. 43. Building a Docker image
  44. 44. ➔Node App ➔Dockerfile
  45. 45. 45 Deploy the service using kubectl 💻 1. Edit the service definition file manifests/app.yaml to refer to your version 2. Deploy the service ➤ kubectl apply -f manifests/app.yaml 3. Deploy the ingress ➤ kubectl apply -f manifests/ingress.yaml 4. Navigate to .../mighty-fine to check the service runs
  46. 46. Deploying a service to a kubernetes cluster
  47. 47. ➔Deployment ➔Service ➔Ingress
  48. 48. ➔kubectl get ➔kubectl describe ➔kubectl logs
  49. 49. Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 50 & Log in to cloud 9 instance 💻 Deploy and test a service Inspect system metrics Instrument the platform Inspect middleware metrics Instrument the application Inspect business metrics BREAK Q & A
  50. 50. Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 51 & Log in to cloud 9 instance 💻 Deploy and test a service Inspect system metrics Instrument the platform Inspect middleware metrics Instrument the application Inspect business metrics BREAK Q & A
  51. 51. 52
  52. 52. 53 Prometheus Display dataGather & Store Query & Alerts Grafana
  53. 53. 54 *.training.weave.works/prometheus
  54. 54. 55 PromQL query. Try
  55. 55. 56 Prometheus Display dataGather & Store Query & Alerts Grafana
  56. 56. 57 *.training.weave.works/grafana
  57. 57. 58 Same query (go_memstats_alloc_bytes). Persistent graphs, Dashboards
  58. 58. 59 Prometheus Display dataGather & Store Query & Alerts Grafana
  59. 59. 60 How?
  60. 60. 61 demo-app/manifests/monitoring.yaml
  61. 61. Use the dashboard definition in demo-app/resources/docker-monitoring_rev1.json Create a dashboard using Grafana’s import functionality. 62 Inspect the system metrics 💻
  62. 62. Importing a default dashboard
  63. 63. ➔JSON all the way ➔Dashboards as data ➔Can start with dashboards
  64. 64. Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 65 & Log in to cloud 9 instance 💻 Deploy and test a service Inspect system metrics Instrument the platform Inspect middleware metrics Instrument the application Inspect business metrics BREAK Q & A
  65. 65. Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 66 & Log in to cloud 9 instance 💻 Deploy and test a service Inspect system metrics Instrument the platform Inspect middleware metrics Instrument the application Inspect business metrics BREAK Q & A
  66. 66. Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 67 & Log in to cloud 9 instance 💻 Deploy and test a service Inspect system metrics Instrument the platform Inspect middleware metrics Instrument the application Inspect business metrics BREAK Q & A
  67. 67. 68 Why monitor?
  68. 68. 69 Quality Assurance Continuous Improvement
  69. 69. 70 NOT about collecting data Why vs How
  70. 70. 71 IS relevant outside of IT
  71. 71. 72 Q: What’s the most important metric?
  72. 72. 73 Q: What’s the purpose of your organisation?
  73. 73. 74 Maybe: Educational goals # People reached # Papers published
  74. 74. 75 Probably £/$/€
  75. 75. 76 Metrics come from purpose. Monitor your goals
  76. 76. 77 1 Availability 2 Collection 0 Ignorance 3 Aggregation 4 Analysis 5 Learning 6 Automation 7 ProactivityThe Monitoring Ladder
  77. 77. 78 1 Availability 2 Collection 3 Aggregation 4 Analysis 5 Learning 6 Automation 7 ProactivityThe Monitoring Ladder 0 Ignorance You don’t know what’s going on
  78. 78. 79 1 Availability 2 Collection 3 Aggregation 4 Analysis 5 Learning 6 Automation 7 ProactivityThe Monitoring Ladder You know whether your systems are available. You may have alerts 0 Ignorance 1 Availability 0 Ignorance
  79. 79. 80 1 Availability 2 Collection 3 Aggregation 4 Analysis 5 Learning 6 Automation 7 ProactivityThe Monitoring Ladder You collect logs. Forensics is possible. 0 Ignorance 2 Collection 1 Availability 0 Ignorance Alerts
  80. 80. 81 1 Availability 2 Collection 3 Aggregation 4 Analysis 5 Learning 6 Automation 7 ProactivityThe Monitoring Ladder You aggregate and persist data in a central place. Correlation is possible. 0 Ignorance 2 Collection 1 Availability 0 Ignorance Alerts Logs Forensics 3 Aggregation
  81. 81. 82 1 Availability 2 Collection 3 Aggregation 4 Analysis 5 Learning 6 Automation 7 ProactivityThe Monitoring Ladder You actually analyse the aggregated and correlated data. Use it to fix issues. 0 Ignorance Alerts 3 Aggregation Logs Forensics 1 Availability 2 Collection 0 Ignorance 4 Analysis Persistence
  82. 82. 83 1 Availability 2 Collection 3 Aggregation 4 Analysis 5 Learning 6 Automation 7 ProactivityThe Monitoring Ladder Root cause analysis. Strengthening fixes. Antigfragile. Still responsive. 0 Ignorance Alerts 3 Aggregation Logs Forensics 1 Availability 2 Collection 0 Ignorance 4 Analysis Persistence 5 Learning
  83. 83. 84 1 Availability 2 Collection 3 Aggregation 4 Analysis 5 Learning 6 Automation 7 ProactivityThe Monitoring Ladder Automated remediations. Incident report creation. No customer impact. 0 Ignorance Alerts 3 Aggregation Logs Forensics 1 Availability 2 Collection 0 Ignorance 4 Analysis Persistence 5 Learning 6 Automation Antifragile
  84. 84. 85 1 Availability 2 Collection 3 Aggregation 4 Analysis 5 Learning 6 Automation 7 ProactivityThe Monitoring Ladder 0 Ignorance Alerts 3 Aggregation Logs Forensics 1 Availability 2 Collection 0 Ignorance 4 Analysis Persistence 5 Learning 6 Automation Antifragile 0-impact
  85. 85. 86 1 Availability 2 Collection 3 Aggregation 4 Analysis 5 Learning 6 Automation 7 ProactivityThe Monitoring Ladder 0 Ignorance Alerts 3 Aggregation Logs Forensics 1 Availability 2 Collection 0 Ignorance 4 Analysis Persistence 5 Learning 6 Automation Antifragile 0-impact Active strengthening by attacking production systems. “Chaos Engineering” 7 Proactivity
  86. 86. 87 1 Availability 2 Collection 3 Aggregation 4 Analysis 5 Learning 6 Automation 7 ProactivityThe Monitoring Ladder 0 Ignorance Alerts 3 Aggregation Logs Forensics 1 Availability 2 Collection 0 Ignorance 4 Analysis Persistence 5 Learning 6 Automation Antifragile 0-impact 7 ProactivityChaos Engineering Monitoring is a broad topic
  87. 87. 88 1 Availability 2 Collection 3 Aggregation 4 Analysis 5 Learning 6 Automation 7 ProactivityThe Monitoring Ladder 0 Ignorance Alerts 3 Aggregation Logs Forensics 1 Availability 2 Collection 0 Ignorance 4 Analysis Persistence 5 Learning 6 Automation Antifragile 0-impact 7 ProactivityChaos Engineering Boundaries aren’t so clear cut.
  88. 88. 89 Whitebox vs Blackbox
  89. 89. 90 Whitebox vs Blackbox
  90. 90. 91 Pull vs Push
  91. 91. 92 Pull vs Push
  92. 92. 93 Realtime vs Historic
  93. 93. 94 Realtime vs Historic
  94. 94. 95 Which one is right? Pull or Push? Whitebox or Blackbox?
  95. 95. 96 Which one is right? Pull or Push? Whitebox or Blackbox? Both
  96. 96. 97 Pull based Whitebox Historic
  97. 97. Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 98 & Log in to cloud 9 instance 💻 Deploy and test a service Inspect system metrics Instrument the platform Inspect middleware metrics Instrument the application Inspect business metrics BREAK Q & A
  98. 98. Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 99 & Log in to cloud 9 instance 💻 Deploy and test a service Inspect system metrics Instrument the platform Inspect middleware metrics Instrument the application Inspect business metrics BREAK Q & A
  99. 99. 100 Maslow’s Pyramid of Metrics (?)
  100. 100. 10 1 Maslow’s Pyramid of Metrics (?) Platform CPU, Memory, GC, JVM, Ruby, threads and bytes. Domain - Computers
  101. 101. 10 2 Maslow’s Pyramid of Metrics (?) Platform Requests, HTTP Errors, Queue stats. Domain - Protocols Middleware
  102. 102. 10 3 Maslow’s Pyramid of Metrics (?) Platform Users, widgets, teams, products. Domain - Application Middleware App
  103. 103. 10 4 Maslow’s Pyramid of Metrics (?) Platform Money, subscriptions, growth. Domain - Business Middleware App Business
  104. 104. 10 5 Let’s start with platform metrics Platform
  105. 105. 10 6 Let’s start with platform metrics … there already 😊
  106. 106. 10 7 Let’s move to middleware metrics Platform Middleware
  107. 107. 1. Checkout v2-default-metrics 2. Rebuild the image with a new tag ➤ docker build -t mighty-fine:v2 . 3. Change the manifests/app.yaml to refer to the new image, and use kubectl to apply the manifest again. 4. Look in prometheus for nodejs* metrics 10 8 Instrument the app to get middleware metrics 💻
  108. 108. Instrumenting with middleware metrics
  109. 109. ➔Straight forward ➔Client libraries available for lots of languages and platforms
  110. 110. 11 1 Application Metrics Platform Middleware App
  111. 111. 1. Checkout v3-custom-metrics 2. Rebuild the image with a new tag ➤ docker build -t mighty-fine:v3 . 3. Change the manifests/app.yaml to refer to the new image, and use kubectl to apply the manifest again. 4. Look in prometheus for sales* metrics 11 2 Instrument the app to get middleware metrics 💻
  112. 112. Instrumenting with application metrics
  113. 113. ➔Also straightforward ➔Counters, Gauges, Histograms
  114. 114. 11 5 Business Metrics Platform Middleware App Business
  115. 115. 11 6 http://chaotic-flow.com/media/saas-metrics-guide-to-saas-financial-performance.pdf Joel York’s SaaS Metrics http://chaotic-flow.com
  116. 116. 11 7
  117. 117. 11 8
  118. 118. 11 9
  119. 119. 12 0
  120. 120. 12 1 We’ll need subscribe_count unsubscribe_count (Both counters)
  121. 121. 1. Checkout v4-subscription-service 2. Rebuild the image with a new tag ➤ docker build -t mighty-fine:v4 . 3. Change the manifests/app.yaml to refer to the new image, and use kubectl to apply the manifest again. 4. Look in prometheus for un/subscribe_count metrics 12 2 Instrument the app to get business metrics 💻
  122. 122. Instrumenting with business metrics
  123. 123. 12 4 ΔCcancel= rate(unsubscribe_count[1m])
  124. 124. 12 5 ΔCcancel= rate(unsubscribe_count[1m])
  125. 125. 12 6 ΔCcancel= rate(unsubscribe_count[1m])
  126. 126. 12 7 ΔCcancel= rate(unsubscribe_count[1m])
  127. 127. 12 8 ΔCcancel= rate(unsubscribe_count[1m])
  128. 128. 12 9 ΔCcancel= rate(unsubscribe_count[1m])
  129. 129. 13 0 ΔCcancel= rate(unsubscribe_count[1m])
  130. 130. 13 1 C = (total_signups offset 1m) - (total_cancels offset 1m)
  131. 131. 13 2 ChurnRatemonth = rate(total_cancels[1m]) / ((total_signups offset 1m) - (total_cancels offset 1m))
  132. 132. 13 3 ChurnRatemonth = rate(total_cancels[1m]) / ((total_signups offset 1m) - (total_cancels offset 1m))
  133. 133. Live demo of changing metrics
  134. 134. ➔Simple metrics can give deep insights ➔A little math goes a long way
  135. 135. Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 13 & Log in to cloud 9 instance 💻 Deploy and test a service Inspect system metrics Instrument the platform Inspect middleware metrics Instrument the application Inspect business metrics BREAK Q & A
  136. 136. Setting up an environment Starting out with Kubernetes - Introduction to Kubernetes - Deploying to Kubernetes using kubectl Monitoring in practice - Setting up Prometheus - Setting up Grafana - Inspecting system metrics Understanding Monitoring - Monitoring: Why and How? - Monitoring maturity - The prometheus model Monitoring in depth - Using different types of metrics - Using PromQL - Instrumenting your code 13 & Log in to cloud 9 instance 💻 Deploy and test a service Inspect system metrics Instrument the platform Inspect middleware metrics Instrument the application Inspect business metrics BREAK Q & A
  137. 137. Q & A 13 8
  138. 138. THANK YOU! 13 9 Brice Fernandes @fractallambda @weaveworks https://weave.works Slides: https://tinyurl.com/prometheus-kubernetes-slides Code: https://tinyurl.com/prometheus-kubernetes-code References: ● Joel York’s SaaS metrics http://chaotic- flow.com/saas-metrics/
  139. 139. 14 0
  • hteixe

    Jun. 19, 2019
  • TimothyIsrael

    Apr. 11, 2018
  • songjiayang

    Mar. 4, 2018

Monitoring cloud native apps creates a whole new set of challenges that traditional monitoring systems struggle with. In this Workshop, Brice Fernandes from Weaveworks will take you through installing and using the open source Prometheus monitoring toolkit to monitor a Kubernetes cluster. You do not need any previous experience with Kubernetes or Prometheus. After this talk, you'll be able to set up a development kubernetes cluster and use Prometheus to monitor your microservices. We'll cover: An introduction to Kubernetes to manage containers; The monitoring maturity model; An overview of whitebox and blackbox monitoring; Monitoring with Prometheus; Using PromQL (the Prometheus Query Language) to monitor your app in a dynamic system.

Views

Total views

805

On Slideshare

0

From embeds

0

Number of embeds

3

Actions

Downloads

63

Shares

0

Comments

0

Likes

3

×