Sensu 1.x & kubernetes

Sensu 1.x & Kubernetes
Lee Briggs
Senior Infrastructure Engineer
22/08/2018

© 2016 Apptio, All rights reserved (v2.5)2
$(whoami)
 Based in Seattle (formerly London & Yorkshire)
 Work for Apptio
 We are hiring!
 Github:
https://github.com/jaxxstorm
 Twitter:
https://twitter.com/briggsl
 Blog:
https://www.leebriggs.co.uk

Kubernetes/Sensu Background
 Using Sensu since 2012
 Over 7000 sensu clients in dev & prod
 Running/operating Kubernetes since 2016
 Actually attempted to run k8s v1.0
 Operating multiple clusters
 Across several regions/environments/cloud providers
 Run a hybrid monitoring stack
 Prometheus for container based workloads
 Sensu for all other workloads
 Very excited about Sensu 2.0!

Target Audience
 You’re familiar with/running sensu 1.x
 You know Sensu 1.x fairly well
 You probably have a cloud footprint
 Used to ephemeral clients
 Just getting started using container based workloads
 Investigating a orchestration tool
 Strongly recommend Kubernetes!

Before I begin
 A lot of the concepts here were inspired by another talk:
 Tom Wilkie did an amazing talk at GrafanaCon EU
 https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-
services/

Host Based Monitoring
 A lot of traditional monitoring systems are host based
 Nagios
 Sensu
 Zabbix
 This was fine for a long time, because most of our operational workload
was host based!

USE Method
 Popularized by Brendan Gregg:
http://www.brendangregg.com/usemethod.html
 For every resource, monitor:
 Utilization (% time that the resource was busy)
 Saturation (amount of work resource has to do, often queue length)
 Errors (count of error events)
 This can be hard to apply to some resources
 How does sensu 1.x handle this?
 You have to set arbitrary thresholds, which might not always be best

Service Based Monitoring
 The hosts running the service no longer matter
 As long as the service is healthy, we don’t care about the underlying
constructs
 With containers and many cloud platforms, this is the desired end state
 Examples:
 Prometheus
 TICK (Telegraf, InfluxDB, Kapacitor, Chronograf)

RED Method
 Popularized by Tom Wilkie himself:
 For every resource, monitor:
 Rate (the number of requests per second)
 Errors (the number of those requests that are failing)
 Duration (the amount of time those requests take)

4 Golden Rules
 From the Google SRE Book: https://landing.google.com/sre/book.html
 For each service, monitor:
 Latency (time taken to serve a request)
 Traffic (how much demand is placed on your system)
 Errors (rate of requests that are failing)
 Saturation (how “full” your service is)

Kubernetes Monitoring Requirements
 Kubernetes applications require service based monitoring
 Too many ephemeral components
 Metric based monitoring is highly desirable
 Monitor on trends, not thresholds
 However, Kubernetes still needs hosts to run
 The controlplane, workers, etcd all runs on a host
 The USE method still applies here

Transitioning to Containers
 Most enterprises can’t go “all-in” with containers
 Usually you’d move some workloads to containers first
 You already have an awesome monitoring solution!
 And implementing it was a lot of work
 The hard truth is that a lot of companies use cloud providers like
traditional datacenters
 So your application(s) might not be designed for ephemeral workloads

Finding a middle ground
 Use Sensu 1.x for the more static components in your stack
 Think of the USE method
 Number of replicas is an example
 There are some tricks to make Sensu service based
 Using source attributes
 API based monitoring
 Monitor the kubernetes components that are relatively static
 # replicas
 Controlplane
 Application health

Running Sensu on Kubernetes
 Running Sensu on Kubernetes kinda makes sense
 Components:
 Redis - https://github.com/spotahome/redis-operator
 No RabbitMQ
 There’s not currently an operator for RabbitMQ
 Uncomfortable running stateful workloads without some kind of operator pattern
 You could run it outside the cluster but adds overhead
 API
 Standard Sensu Deployment
 Server
 Standard Sensu Deployment
 Client
 Daemonset
 Sidecar containers

Example Repo
 Contains:
 Deployment manifests for sensu/redis
 Example application with healthcheck monitoring
 Sock shop microservices demo
 https://github.com/jaxxstorm/sensu-on-kubernetes
 https://github.com/jaxxstorm/docker-sensu-kubernetes

Extensions
 If you’re comfortable with Sensu, the journey doesn’t have to end here
 Prometheus & Sensu together: https://blog.sensuapp.org/the-sensu-prometheus-
collector-972c441d45e
 Sens8 – Sensu Operator: https://github.com/hootsuite/sens8
 Sensu 2.0! https://github.com/portertech/sensu-demo
 More in depth example of k8s & Sensu together: https://blog.sensuapp.org/our-
journey-implementing-sensu-to-monitor-kubernetes-in-production-5764aff2dd50

Summary
 Monitoring is a journey
 It moves and migrates with your infrastructure
 Sensu’s flexibility allows it to be used in a variety of hosting & monitoring
models
 Sensu 2.x continues with this flexibility
 Use existing tools as you make transitions!

Sensu 1.x & kubernetes

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Sensu 1.x & kubernetes

Similar to Sensu 1.x & kubernetes (20)

Recently uploaded

Recently uploaded (20)

Sensu 1.x & kubernetes

Editor's Notes