Sensu and Kubernetes 1.x

Sensu 1.x & Kubernetes
Lee Briggs
Senior Infrastructure Engineer
22/08/2018

© 2016 Apptio, All rights reserved (v2.5)2
$(whoami)
 Based in Seattle (formerly London & Yorkshire)
 Work for Apptio
 We are hiring!
 Github:
https://github.com/jaxxstorm
 Twitter:
https://twitter.com/briggsl
 Blog:
https://www.leebriggs.co.uk

Kubernetes/Sensu Background
 Using Sensu since 2012
 Over 7000 sensu clients in dev & prod
 Running/operating Kubernetes since 2016
 Actually attempted to run k8s v1.0
 Operating multiple clusters
 Across several regions/environments/cloud providers
 Run a hybrid monitoring stack
 Prometheus for container based workloads
 Sensu for all other workloads
 Very excited about Sensu 2.0!

Target Audience
 You’re familiar with/running sensu 1.x
 You know Sensu 1.x fairly well
 You probably have a cloud footprint
 Used to ephemeral clients
 Just getting started using container based workloads
 Investigating a orchestration tool
 Strongly recommend Kubernetes!

Before I begin
 A lot of the concepts here were inspired by another talk:
 Tom Wilkie did an amazing talk at GrafanaCon EU
 https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-
services/

Host Based Monitoring
 A lot of traditional monitoring systems are host based
 Nagios
 Sensu
 Zabbix
 This was fine for a long time, because most of our operational workload
was host based!

USE Method
 Popularized by Brendan Gregg:
http://www.brendangregg.com/usemethod.html
 For every resource, monitor:
 Utilization (% time that the resource was busy)
 Saturation (amount of work resource has to do, often queue length)
 Errors (count of error events)
 This can be hard to apply to some resources
 How does sensu 1.x handle this?
 You have to set arbitrary thresholds, which might not always be best

Service Based Monitoring
 The hosts running the service no longer matter
 As long as the service is healthy, we don’t care about the underlying
constructs
 With containers and many cloud platforms, this is the desired end state
 Examples:
 Prometheus
 TICK (Telegraf, InfluxDB, Kapacitor, Chronograf)

RED Method
 Popularized by Tom Wilkie himself:
 For every resource, monitor:
 Rate (the number of requests per second)
 Errors (the number of those requests that are failing)
 Duration (the amount of time those requests take)

4 Golden Rules
 From the Google SRE Book: https://landing.google.com/sre/book.html
 For each service, monitor:
 Latency (time taken to serve a request)
 Traffic (how much demand is placed on your system)
 Errors (rate of requests that are failing)
 Saturation (how “full” your service is)

Kubernetes Monitoring Requirements
 Kubernetes applications require service based monitoring
 Too many ephemeral components
 Metric based monitoring is highly desirable
 Monitor on trends, not thresholds
 However, Kubernetes still needs hosts to run
 The controlplane, workers, etcd all runs on a host
 The USE method still applies here

Transitioning to Containers
 Most enterprises can’t go “all-in” with containers
 Usually you’d move some workloads to containers first
 You already have an awesome monitoring solution!
 And implementing it was a lot of work
 The hard truth is that a lot of companies use cloud providers like
traditional datacenters
 So your application(s) might not be designed for ephemeral workloads

Finding a middle ground
 Use Sensu 1.x for the more static components in your stack
 Think of the USE method
 Number of replicas is an example
 There are some tricks to make Sensu service based
 Using source attributes
 API based monitoring
 Monitor the kubernetes components that are relatively static
 # replicas
 Controlplane
 Application health

Running Sensu on Kubernetes
 Running Sensu on Kubernetes kinda makes sense
 Components:
 Redis - https://github.com/spotahome/redis-operator
 No RabbitMQ
 There’s not currently an operator for RabbitMQ
 Uncomfortable running stateful workloads without some kind of operator pattern
 You could run it outside the cluster but adds overhead
 API
 Standard Sensu Deployment
 Server
 Standard Sensu Deployment
 Client
 Daemonset
 Sidecar containers

Example Repo
 Contains:
 Deployment manifests for sensu/redis
 Example application with healthcheck monitoring
 Sock shop microservices demo
 https://github.com/jaxxstorm/sensu-on-kubernetes
 https://github.com/jaxxstorm/docker-sensu-kubernetes

Extensions
 If you’re comfortable with Sensu, the journey doesn’t have to end here
 Prometheus & Sensu together: https://blog.sensuapp.org/the-sensu-prometheus-
collector-972c441d45e
 Sens8 – Sensu Operator: https://github.com/hootsuite/sens8
 Sensu 2.0! https://github.com/portertech/sensu-demo
 More in depth example of k8s & Sensu together: https://blog.sensuapp.org/our-
journey-implementing-sensu-to-monitor-kubernetes-in-production-5764aff2dd50

Summary
 Monitoring is a journey
 It moves and migrates with your infrastructure
 Sensu’s flexibility allows it to be used in a variety of hosting & monitoring
models
 Sensu 2.x continues with this flexibility
 Use existing tools as you make transitions!

Sensu and Kubernetes 1.x

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Sensu and Kubernetes 1.x

Similar to Sensu and Kubernetes 1.x (20)

More from Sensu Inc.

More from Sensu Inc. (20)

Recently uploaded

Recently uploaded (20)

Sensu and Kubernetes 1.x