Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Sensu and Kubernetes 1.x


Published on

In this Sensu Summit 2018 talk, Lee Briggs, Senior Infrastructure Engineer at Apptio, discusses how to monitor Kubernetes components and applications using the classic sensu components.

He covers some of the tricks you can use when monitoring Kubernetes resources and cluster components. We’ll cover the kind of things you should and shouldn't monitor with sensu at this stage, as well as some of the lessons learned along the way.

Published in: Technology
  • Be the first to comment

Sensu and Kubernetes 1.x

  1. 1. Sensu 1.x & Kubernetes Lee Briggs Senior Infrastructure Engineer 22/08/2018
  2. 2. © 2016 Apptio, All rights reserved (v2.5)2 $(whoami)  Based in Seattle (formerly London & Yorkshire)  Work for Apptio  We are hiring!  Github:  Twitter:  Blog:
  3. 3. © 2016 Apptio, All rights reserved (v2.5)3 Kubernetes/Sensu Background  Using Sensu since 2012  Over 7000 sensu clients in dev & prod  Running/operating Kubernetes since 2016  Actually attempted to run k8s v1.0  Operating multiple clusters  Across several regions/environments/cloud providers  Run a hybrid monitoring stack  Prometheus for container based workloads  Sensu for all other workloads  Very excited about Sensu 2.0!
  4. 4. © 2016 Apptio, All rights reserved (v2.5)4
  5. 5. © 2016 Apptio, All rights reserved (v2.5)5 Target Audience  You’re familiar with/running sensu 1.x  You know Sensu 1.x fairly well  You probably have a cloud footprint  Used to ephemeral clients  Just getting started using container based workloads  Investigating a orchestration tool  Strongly recommend Kubernetes!
  6. 6. © 2016 Apptio, All rights reserved (v2.5)6 Before I begin  A lot of the concepts here were inspired by another talk:  Tom Wilkie did an amazing talk at GrafanaCon EU  services/
  7. 7. Monitoring Models
  8. 8. © 2016 Apptio, All rights reserved (v2.5)8 Host Based Monitoring  A lot of traditional monitoring systems are host based  Nagios  Sensu  Zabbix  This was fine for a long time, because most of our operational workload was host based!
  9. 9. © 2016 Apptio, All rights reserved (v2.5)9 USE Method  Popularized by Brendan Gregg:  For every resource, monitor:  Utilization (% time that the resource was busy)  Saturation (amount of work resource has to do, often queue length)  Errors (count of error events)  This can be hard to apply to some resources  How does sensu 1.x handle this?  You have to set arbitrary thresholds, which might not always be best
  10. 10. © 2016 Apptio, All rights reserved (v2.5)10 Service Based Monitoring  The hosts running the service no longer matter  As long as the service is healthy, we don’t care about the underlying constructs  With containers and many cloud platforms, this is the desired end state  Examples:  Prometheus  TICK (Telegraf, InfluxDB, Kapacitor, Chronograf)
  11. 11. © 2016 Apptio, All rights reserved (v2.5)11 RED Method  Popularized by Tom Wilkie himself:  For every resource, monitor:  Rate (the number of requests per second)  Errors (the number of those requests that are failing)  Duration (the amount of time those requests take)
  12. 12. © 2016 Apptio, All rights reserved (v2.5)12 4 Golden Rules  From the Google SRE Book:  For each service, monitor:  Latency (time taken to serve a request)  Traffic (how much demand is placed on your system)  Errors (rate of requests that are failing)  Saturation (how “full” your service is)
  13. 13. Kubernetes
  14. 14. © 2016 Apptio, All rights reserved (v2.5)14 Kubernetes Monitoring Requirements  Kubernetes applications require service based monitoring  Too many ephemeral components  Metric based monitoring is highly desirable  Monitor on trends, not thresholds  However, Kubernetes still needs hosts to run  The controlplane, workers, etcd all runs on a host  The USE method still applies here
  15. 15. © 2016 Apptio, All rights reserved (v2.5)15 Transitioning to Containers  Most enterprises can’t go “all-in” with containers  Usually you’d move some workloads to containers first  You already have an awesome monitoring solution!  And implementing it was a lot of work  The hard truth is that a lot of companies use cloud providers like traditional datacenters  So your application(s) might not be designed for ephemeral workloads
  16. 16. © 2016 Apptio, All rights reserved (v2.5)16 Finding a middle ground  Use Sensu 1.x for the more static components in your stack  Think of the USE method  Number of replicas is an example  There are some tricks to make Sensu service based  Using source attributes  API based monitoring  Monitor the kubernetes components that are relatively static  # replicas  Controlplane  Application health
  17. 17. Sensu on Kubernetes
  18. 18. © 2016 Apptio, All rights reserved (v2.5)18 Running Sensu on Kubernetes  Running Sensu on Kubernetes kinda makes sense  Components:  Redis -  No RabbitMQ  There’s not currently an operator for RabbitMQ  Uncomfortable running stateful workloads without some kind of operator pattern  You could run it outside the cluster but adds overhead  API  Standard Sensu Deployment  Server  Standard Sensu Deployment  Client  Daemonset  Sidecar containers
  19. 19. © 2016 Apptio, All rights reserved (v2.5)19 Example Repo  Contains:  Deployment manifests for sensu/redis  Example application with healthcheck monitoring  Sock shop microservices demo  
  20. 20. Quick Demo
  21. 21. Going further
  22. 22. © 2016 Apptio, All rights reserved (v2.5)22 Extensions  If you’re comfortable with Sensu, the journey doesn’t have to end here  Prometheus & Sensu together: collector-972c441d45e  Sens8 – Sensu Operator:  Sensu 2.0!  More in depth example of k8s & Sensu together: journey-implementing-sensu-to-monitor-kubernetes-in-production-5764aff2dd50
  23. 23. © 2016 Apptio, All rights reserved (v2.5)23 Summary  Monitoring is a journey  It moves and migrates with your infrastructure  Sensu’s flexibility allows it to be used in a variety of hosting & monitoring models  Sensu 2.x continues with this flexibility  Use existing tools as you make transitions!
  24. 24. THANK YOU