Monitoring using Sensu

R.I.Pienaar
Malta DevOps December 2016
Monitoring using Sensu

R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Who am I?
• Malta since December 2015
• Consultant for 20+ years
• Government, Finance, Health, Social Media,
Fortune 50, Startups
• DevOps, Automation, Architect,
Development
• Open Source @ github.com/ripienaar
• Linux since Kernel 99 alpha p11

Tool
vs
Framework

Checks
API
MetricsDynamic
External InputsSecure
Events
Plugins
JSON

Architecture - Overview

Architecture - Schedule

Architecture - Event

Architecture - Integrate

{
"checks": {
"procs_rhel7": {
"command": "/usr/lib64/nagios/plugins/check_procs -w 50 -c 70 -k",
"interval": 300,
"subscribers": [
"RedHat-7"
]
}
}
}
Conﬁg - Checks

{
"checks": {
"procs_rhel7": {
"interval": 300,
"subscribers": [
"RedHat-7"
]
}
}
}
Conﬁg - Checks
Name
Command to run
How often
Where

{
"client": {
"name": "dev3.devco.net",
"address": "139.162.163.118",
"subscriptions": [
"all",
"linode_fra",
"de",
"RedHat-7"
],
"redact": [
],
"socket": {
"bind": "127.0.0.1",
"port": 3030
},
"procs": {
"total": {
"warn": 80,
"crit": 90
}
}
}
}
Conﬁg - Clients
Local event sync
What am I?
Node Parameters

{
"client": {
…,
"procs": {
"total": {
"warn": 80,
"crit": 90
}
}
}
}
Conﬁg - Clients
Node Parameters
{
"checks": {
"procs_rhel7": {
"interval": 300,
"subscribers": [
"RedHat-7"
]
}
}
}

{
"client": {
…,
"procs": {
"total": {
"warn": 80,
"crit": 90
}
}
}
}
Conﬁg - Clients
Node Parameters
{
"checks": {
"procs_rhel7": {
"command": “…/check_procs -w :::procs.total.warn|50::: -c :::procs.total.crit|70::: -k",
"interval": 300,
"subscribers": [
"RedHat-7"
]
}
}
}
Check Tokens

require ‘sensu-plugin/check/cli'
class CheckFile < Sensu::Plugin::Check::CLI
option :file,
long: “—file FILE”,
description: “The file to check”,
default: nil
def run
unknown(“no file specified, please use —file”) unless config[:file]
if File.exist?(config[:file])
ok(“file %s exist” % [config[:file]])
else
critical(“file %s does not exist” % [config[:file]])
end
end
end
Checks - Writing in Ruby

require “sensu-plugin/check/cli”
class CheckFile < Sensu::Plugin::Check::CLI
option :file,
long: “—file FILE”,
description: “The file to check”,
default: nil
def run
unknown(“no file specified, please use —file”) unless config[:file]
if File.exist?(config[:file])
ok(“file %s exist” % [config[:file]])
else
critical(“file %s does not exist” % [config[:file]])
end
end
end
Checks - Writing in Ruby

# yum install nagios-plugins-all
# /usr/lib64/nagios/plugin/check_file_age —help
…
Usage:
check_file_age [-w <secs>] [-c <secs>] [-W <size>] [-C <size>] [-i] -f <file>
check_file_age [-h | --help]
check_file_age [-V | --version]
# /usr/lib64/nagios/plugins/check_file_age -f /keepalive; echo $?
FILE_AGE UNKNOWN: No file specified
3
FILE_AGE CRITICAL: File not found - /keepalive
2
FILE_AGE WARNING: /keepalive is 353 seconds old and 0 bytes
1
FILE_AGE OK: keepalive is 4 seconds old and 0 bytes
0
Checks - Reuse

Checks - Reuse
http://sensu-plugins.io/

$ cat <<EOF > /dev/tcp/localhost/3030
{
"name":"my_awesome_check",
"output":"everything is fine",
"ttl":600,
“status”:0
}
EOF
Checks -Your own App

{
"mailer": {
"mail_from": "sensu@example.com",
"mail_to": "monitor@example.com",
"smtp_address": "smtp.example.org",
"smtp_port": "25",
"smtp_domain": "example.org",
"template": “/etc/sensu/templates/mail.erb”,
"subscriptions": {
"subscription_name": {
"mail_to": "teamemail@example.com"
}
}
}
}
Handlers - Email
https://github.com/sensu-plugins/sensu-plugins-mailer
/etc/sensu/handlers/mailer.json
Custom mail body

<%= output %>
Admin GUI: <%= admin_gui %>
Host: <%= @event['client']['name'] %>
Timestamp: <%= Time.at(@event['check']['issued']) %>
Address: <%= @event['client']['address'] %>
Check Name: <%= @event['check']['name'] %>
Command: <%= command %>
Status: <%= status_to_string %>
Occurrences: <%= @event['occurrences'] %>
Handlers - Email
https://github.com/sensu-plugins/sensu-plugins-mailer
/etc/sensu/templates/mail.erb

Handlers - Thirdparties

events = JSON.parse(
RestClient.get(“http://example:4567/events").body
)
statusmap = {0 => "OK", 1 => "WARNING", 2 => "CRITICAL", 3 => "UNKNOWN"}
puts "Found %d events" % [events.size]
events.each do |event|
puts "%30s: %-10s %s" % [
event["client"]["name"],
statusmap[event["check"]["status"]],
event[“check"]["output"]
]
end
API - JSON
Found 2 events
puppet1.example.net: CRITICAL DISK CRITICAL - /run/docker/netns/default is not accessible:
Permission denied
dev1.example.net: WARNING PROCS WARNING: 183 processes

API - Uchiwa
https://uchiwa.io/

Links
https://sensuapp.org/
https://uchiwa.io/
https://graphiteapp.org/
http://grafana.org/
http://sensu-plugins.io/

Questions?
twitter: @ripienaar
email: rip@devco.net
blog: www.devco.net
github: ripienaar
freenode: Volcane
slack.puppet.com: ripienaar
https://www.devco.net/

Monitoring using Sensu

Recommended

Recommended

More Related Content

What's hot

What's hot (6)

Similar to Monitoring using Sensu

Similar to Monitoring using Sensu (20)

Recently uploaded

Recently uploaded (20)

Monitoring using Sensu