Creating aggregated Sensu monitors, or how to monitor a group of monitors
1. Creating aggregated
Sensu monitors
Or, how to check a group of Sensu checks
Maggie Moreno
Production Engineer at Tubular Labs
maggie@tubularlabs.com
August 15, 2017
2. What is an aggregated Sensu monitor?
And why do I care?
3.
4. How to create aggregated Sensu checks
1. Create queriable logging and metrics store
2. Send Sensu check logs and outputs to the central
logging/metrics store
3. Create a tool let a Sensu check alert on a query
against those logs
4. Party because now you have all the tools to create an
uptime monitor!
5. 1. Create queriable logging and metrics store
2. Send Sensu check logs and outputs to the central
logging/metrics store
3. Create a tool let a Sensu check alert on a query
against those logs
4. Party because now you have all the tools to create an
uptime monitor!
How to create aggregated Sensu checks
7. 1. Central log/metrics store: Elastic Stack
At Tubular, we make heavy use of Elasticsearch, so we
have a lot of expertise in that database and related
technologies.
8.
9. Once you can query logs from one source, it makes
sense to want to send your Sensu event logs there as
well.
With Elastic Stack, that was pretty simple-- we just told
Filebeat to collect Sensu event logs and added a
Logstash filter to parse them.
2. Send Sensu check logs and outputs to the central log/metrics
10.
11. 3. Sensu check to query and alert on logging events
The tool that I created for my team is
called senseuss.
Senseuss lets the user create Sensu
checks that define an arbitrary
Elasticsearch query and expressions
to alert on the results of that query.
Image Source: http://vignette1.wikia.nocookie.net/seuss/images/a/ad/Cat-in-the-hat.gif/revision/latest?cb=20121112031951
12.
13. Why should I care?
Sensu checks execute a command and expect a non-arbitrary output
Senseuss let's the user
• Construct an arbitrary JSON Elasticsearch query from command line
arguments
• Create expressions to define the way the results of that query should be
processed
• Add query result into to the alert message for informative check outputs
Because our Sensu check output is stored in Elasticsearch with our other
logs, we can use this simple tool to query for and alert on any logging
event, including the results of other Sensu checks
Senseuss
14. /usr/local/bin/senseuss time_diff
--index-prefix 'elasticbeat'
--query '_exists_:cluster_stats.Indices.Docs.count AND
cluster_stats.cluster_name:falcon'
--metric 'cluster_stats.Indices.Docs.count'
--aggregation-type 'max'
--timeframe 'now-5m'
--critical-expression 'result["latest"]["value"] - result["previous"]["value"]
<= 0'
--warning-expression 'result["latest"]["value"] - result["previous"]["value"] <
10**6'
--alert-message 'Document count on ES metrics cluster falcon has fallen below
normal levels'
--message-expression '"New doc count at {:.4G}".format(result["latest"]["value"]
- result["previous"]["value"])'
Senseuss
17. Recap: How to monitor your monitors
1. Create queriable logging and metrics store
• E.g., Elastic stack
2. Send Sensu check logs and outputs to the central logging/metrics
store
• We used filebeat
3. Create a tool let a Sensu check alert on a query against those logs
• Senseuss!
4. Party because now you have all the tools to create an uptime
monitor!
