Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Monitor your CentOS stack with Prometheus

191 views

Published on

his talk will introduce you to the Prometheus monitoring solution and how you can use it to monitor yous CentOS servers, and the applications that run on top of them. It will provide tips about the setup and show some great, real life example.

A small demo involving OpenShift will also be produced, to demonstrate how Prometheus can work with dynamic environments.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Monitor your CentOS stack with Prometheus

  1. 1. Monitor your CentOS stack with Prometheus Julien Pivotto (@roidelapluie) CentOS Dojo Brussels February 1st, 2019
  2. 2. user{name="roidelapluie"} 1 I like Open Source I like monitoring I like automation ... and all of that is my daily job at inuits
  3. 3. Traditional sysadmin-style monitoring Disk is full No zombie processes not 100 user connected ssh works swap is empty
  4. 4. Questions that come after: It's up but it is performant? It's down but for everyone? Its is degraded but are the users impacted? Is it even relevant?
  5. 5. Metrics Monitoring e.g. traditionally graphite Gather fine grained data at frequent interval Make them useful by labelling them ; store them Analyze them to understand what is going on
  6. 6. Metrics ARE PART OF monitoring Do not maintain a metrics + a "traditional monitoring" stack Alert from metrics directly!
  7. 7. Monitoring source Metrics then Logs then Spans (opentracing)
  8. 8. We are in the cloud era. Here are some buzzwords for you cloud, API, openstack, devops, docker, bimodal, stateless, kubernetes, orchestration, automation, serverless, docker, humanops, ansible, continuous deployment, cri-o, jenkins, agile, docker, red hat, containers, virtualization, provisionning, monitoring, observability...
  9. 9. What is the cloud Scale Velocity Change
  10. 10. On Premise looks like the cloud Nowadays you have no choice. Scale Velocity Change
  11. 11. What are the needs Automation Scalability
  12. 12. Bye bye all-in-one tools tools that don't scale tools you can not automate
  13. 13. We need deserve better tools Our customers ask us to respond fast, in seconds We make hundreds of operations per second What is your monitoring frequency... 5 minutes?
  14. 14. Time to get better tools / protocols
  15. 15. Prometheus https://prometheus.io/
  16. 16. Cloud Native Easy to configure, deploy, maintain Designed in multiple services Container ready Orchestration ready (dynamic config) Fuzziness
  17. 17. Data Centric A Metric in Prometheus has metadata: myql_global_status_handlers_total{handler="tmp_write"} 1122 And lots of function to filter, change, remove... those metadata while fetching them. => OpenMetrics.io
  18. 18. Open Source Apache 2.0 Go Support for multiple OS Many "exporters": https://github.com/prometheus/prometheus/wiki/Default- port-allocations
  19. 19. Simple 1 service = 1 things Takes care of its db (time based retention and/or disk space based retention)
  20. 20. RPM https://github.com/openshift/prometheus/blob/ master/golang-github-prometheus- prometheus.spec (I have not tested it) https://github.com/lest/prometheus-rpm
  21. 21. How does it work?
  22. 22. How does it work?
  23. 23. How does it work?
  24. 24. How does it work?
  25. 25. How does it work?
  26. 26. Exporters Exporters expose metrics with an HTTP API Bindings available for many languages Exporters do not save data ; they are not "proxies" and don't "cache" anything
  27. 27. Exploring Metrics
  28. 28. Exploring Metrics
  29. 29. Exploring Metrics
  30. 30. Exploring Metrics
  31. 31. PromQL mysql_global_status_commands_total
  32. 32. PromQL mysql_global_status_commands_total{command="select"}
  33. 33. PromQL mysql_global_status_commands_total {command=~"select|set_options"}
  34. 34. PromQL mysql_global_status_commands_total{command=~"select|se t_options"}
  35. 35. PromQL deriv(mysql_global_status_connections[5m])
  36. 36. PromQL {__name__=~".+innodb.+cache.*"} predict_linear(mysql_heartbeat_lag_seconds[5m], 60*2) sum(rate(mysql_global_status_commands_total{command=~" (commit|rollback)"}[5m])) without (command)
  37. 37. Demo prometheus node_exporter + file collector pushgateway within openshift
  38. 38. Alerting Prometheus has recording rules Record frequent queries or alerting Alert base on any PromQL queries.
  39. 39. One tool does one job... Prometheus collects data Exporters expose data Grafana graphes data Alertmanager dispatches alerts
  40. 40. Alertmanager Open Source Same developers as Prometheus Second service
  41. 41. What is the Alertmanager doing? Receives alerts Group them Inhibits them Dispatches them Deals with HA
  42. 42. How to alerts? Email Some vendors: Slack, Hipchat, VictorOps, pagerduty, ... Generic Webhook -> Plug in anything you want
  43. 43. High Availability 2 prometheus servers do exact the same job They send alerts to Alertmanagers Alertmanagers are clustered not to send the same notification twice
  44. 44. Grafana Open Source (Apache 2.0) Web app Specialized in visualization Pluggable Multiple datasources: prometheus, graphite, influxdb... Has an API!
  45. 45. History of Grafana Grafana is a fork of Kibana 3 ; used to be JS- Driven. Now fully featured, requires a database, multi- projects/users support, etc...
  46. 46. Grafana and Prometheus Prometheus shipped its own consoles Now it recommends Grafana and deprecated its own consoles
  47. 47. Grafana Dashboards
  48. 48. Grafana Dashboards
  49. 49. Time Picker
  50. 50. Configure Prometheus in Grafana
  51. 51. Configure Prometheus in Grafana
  52. 52. Prometheus Dashboard
  53. 53. Creating Grafana Dashboards Takes time Requires deep knowledge of the tools Improved over time Easy to share (json + online library) Try grafonnet-lib!
  54. 54. Conclusion Lots of data that can be explored in many ways (subqueries are coming) Trends and deviations are easy to calculate Can monitor both business and technical Very convenient to monitor any kind of stack
  55. 55. Conclustion Open Source View on technical + business Multiple small pieces Rich ecosystem For Cloud and On Premise
  56. 56. Julien Pivotto roidelapluie roidelapluie@inuits.eu Inuits https://inuits.eu info@inuits.eu Contact I will give a prometheus workshop at OSMC.de! - - SOLD� OUT

×