Anders Haal's presentation on using adaptive thresholds with Nagios. The presentation was given during the Nagios World Conference North America held Sept 25-28th, 2012 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/nwcna

1. Why
dynamic & adaptive
thresholds
matters

2. anders håål, ingenjörsbyn ab
anders.haal@ingby.com
@thenodon

3. Bischeck ­
dynamic & adaptive
thresholds
for Nagios
www.bischeck.org

4. Threshold

5. What is the limitation with static threshold?
✗ Not static
✗ Load varies throughout the day, week
✗ To many or to few alarms
✗ Collecting and thresholding in the
same context
✗ Based on the current measurement
✗ Do not consider dependency to other services

6. How to make thresholds
dynamic & adaptive?

7. {example 1}
“Database table size should not be bigger then 5 % of
yesterdays max size “

8. {example 1}
“Database table size should not be bigger then 5 % of max
size yesterday“
200000
180000
160000
140000 table size < max(yesterday)*1.05
120000
100000
Table size
80000
60000
40000
20000
0
7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Yesterday Today

9. {example 2}
“Number of on­line users should not be more then 10 %
higher then the average number of on­line users for the
last 10 data points”

10. {example 2}
“Number of on­line users should not be more then 10 %
higher then the average number of on­line users for the
last 10 data points”
users < avg(X0+X1+.....+X9)*1.1
Where X is the historical on-line users
data points

11. {example 3}
“The number of orders with errors should be lower then 5%
of the total number of registered orders”

12. {example 3}
“The number of orders with errors should be lower then 5%
of the total number of registered orders”
Total #orders
#orders with errors < 0.05 * (Total #orders)

13. {example 4}
“Message queue size should be above the defined Friday
threshold profile”
count
time of the day

14. {example 4}
“Message queue size should be above the defined Friday
threshold profile”
count
count
time of the day
time of the day

15. How to make thresholds
dynamic & adaptive?
✗ Time profiles
✗ Historical data points
✗ Math and statistical operations

16. We did not want a check_XYZ hack
We wanted a tool

17. Collecting

18. Collecting
Separation
Thresholding

19. Collecting
Historical data

20. Collecting
Logic
Historical data

21. Collecting
Day profile
Logic
Historical data

22. Collecting
Day profile
Calender Logic
Historical data

23. Collecting
Day profile
Calender Logic
Historical data
Nagios

24. Collecting
Scheduling
Day profile
Calender Logic
Historical data
Server
Interface
OpenTSDB Nagios XYZ

27. bischeck basics
● Configuration like Nagios – host, service but
also service item
● Host is just a container of the rest
● Service specify the connection and scheduling
● Service item specify the “query” and the threshold
class to use
● Host and service name must be the same as in
the Nagios configuration

28. Threshold – 24 hour day profile
● Divide the day in 24 hour points, where every
point can be:
● Static value
● Dynamic value
– Math expression on single value or range of data from
the cache
– Based on cached data points retrieved by
● Index – single value or index range
● Time – single value (closest) or time range (between)

29. ....
<!­­ 12:00 Static ­­>
<hour>7000</hour>
....

30. ....
<!­­ 12:00 Static ­­>
<hour>7000</hour>
<!­­ 13:00 Adaptive ­­>
<hour>erpserver­orders­ediOrders[0] / 3</hour>
....

31. ....
<!­­ 12:00 Static ­­>
<hour>7000</hour>
<!­­ 13:00 Adaptive ­­>
<hour>erpserver­orders­ediOrders[0] / 3</hour>
<!­­ 14:00 Adaptive with math function ­­>
<hour>avg(erpserver­orders­ediOrders[­30M:­60M]) / 2</hour>
....

32. Threshold – 24 hour day profile
Between every “full” hour a linear equation is
calculated
Day profile
60000
50000
40000
30000
20000
10000
0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Hour

33. Threshold – 24 hour day profile
● Connect calender to the day profile and evaluate
according to the following order:
1. Month and day of month
2.Week and day of week
3.Day in month
4.Day in the week
5.Month
6.Week
● Holiday – exception days

34. And more....
● Multi­threaded and multi­scheduling schema per service
● interval
● cron
● Data collection – jdbc, livestatus, internal cache
● Virtual services
● Date macros in execution statements
● Customize
● connection (service classes)
● execution (service item classes)
● thresholds (threshold classes)
● server integration (server classes)
● XML configuration supported with WEBui (beta)
● GPL 2 license

35. Future
● Improved time series database
● Patterns/baselines
● More statistic functions
● “Sensors” ­ alarms on multiple/aggregated data points
● Any ideas?

36. Infrastructure monitoring
Application performance monitoring [APM]
Business activity monitoring [BAM]
Operational Business intelligence [OBI]

37. Questions & Feedback
Pictures – Creative Commons
www.flickr.com/photos/loneprimate/4017405677
www.flickr.com/photos/catatronic/2397319483
www.flickr.com/photos/dtrimarchi/6815004766
www.flickr.com/photos/bikeracer/6740232