Leland Lammert's presentation on Distributed Heirarchical Nagios. The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference

1. Hierarchical/Distributed Nagios
Leland V. Lammert, PhD
Omnitec Corporation
St. Louis, MO
lvl@omnitec.net
@lvlammert

2. Introduction & Agenda
• The problem:
• Multiple Remote Sites
• System Status/Local Nagios
• Central Dispatch for problems
• The solution:
• SRM Hardware
• Software Build
• Network Issues
• Nagios Configuration

3. Background
● Specialists in Open Source Solutions
● Core IT Infrastructure
● Multi-platform Nagios
● Leland V. Lammert, PhD
Chief Scientist

4. SRM Hardware

5. SRM Hardware
● Raspberry Pi B
or
● Raspberry Pi B+
● R-Pi Camera
● Temperature Probe
● Cyntech case w/SD cover

6. SRM Hardware

7. SRM Hardware
● Raspberry Pi Camera

8. SRM Hardware
Temperature Probe Options
DHT11
Temperature and Humidity
DS18B20
OneWire
Temperature Only

9. SRM Hardware
SD Card
Camera Temperature
Sensor
Status
LEDs
Server Room Monitor [SRM]

10. Software Build

11. Software Build
● Debian for Pi - Raspian
http://www.raspbian.org/
● Download from Raspberry Pi Project
http://www.raspberrypi.org/downloads/
● Or, purchase a pre-installed SD
● First task, update: aptitude update & upgrade

12. Software Build
● Add packages for Nagios & admin:
– php5, php5-commin, php5-cli
– (apache2-mpm conflicts, will be replaced)
– chkconfig
– telnet
– libgd2-xpm-dev
– heirloom-mailx (MUA)
– ssmtp (MTA)

13. Software Build
● Setup GPIO for the DS18B20
– Add to /etc/modules:
w1_gpio
w1_therm
– Use modprobe to install manually
modprobe [w1_gpio, w1_therm]

14. Software Build
● Check the ID of the installed device:
● ll /sys/bus/w1/devices/
/sys/bus/w1/devices/28-00000489b929/w1_slave
^^^^^^^^^^^^^^^^^^
● ID on this device 28-00000489b929

15. Software Build
● Create startup script - /root/SRMStart
# Start the camera
raspistill -t 0 -tl 1500 -o /usr/local/nagios/share/tmp/snapshot.jpg &>/dev/null
&
# Start the NSCA daemon
/usr/local/nagios/libexec/nsca -c /usr/local/nagios/etc/nsca.cfg –daemon
# Setup tunnel if required
#/usr/bin/autossh -a -M 9005:1005 -N -R <remote>:127.0.0.1:<local>
link@nagios.omnitec.net &

16. Software Build
● Add SRMStart to /etc/rc.local:
##
Start the SRM components
#
/root/SRMstart

17. Software Build
● Install Nagios – NagiosCore
● Download and build from Source
– ./configure -with-gd-lib=/usr/local/lib/, make, make install
– make install-init - Install the init script in /etc/init.d
– make install-commandmode - Install and sets
permissions on the directory for external commands
– make install-config - Install sample config files in
/usr/local/nagios/etc

18. Software Build
● Install NSCA
– ./configure && make
– nsca deamon, send_nsca client
– Copy binaries to /usr/local/nagios/libexec
– Configs to usr/local/nagios/etc
● Create password and add to both configs
● Initial encryption XOR, select as appropriate

19. Software Build
● Build a local copy of plugins
– ./configure, make, make install
– Installs to /usr/local/nagios/libexec
● Verify operation
– cd /usr/local/nagios/libexec
– ./check_

20. Software Build
● Other system plugins
– Build in that environment
● Linux
● Windows
● Mac
● Verify operation
– ./check_ [in that environment]

21. Network Issues

22. Network Issues
Typical network configuration
?

23. Network Issues
● An ssh tunnel would normally be
required to traverse the firewall for
each remote host which is definitely
not ideal
● Solution – use NSCA!
● By forwarding data via NSCA, firewall
changes are eliminated, .. greatly
reducing load on central system!

24. Network Issues
● How does NSCA allow one Nagios system
communicate with another?
● By forwarding notifications to a remote system!
– Notifications for Services
– Notifications for Hosts
– Using a specific Contact
● Only requires one open port, at the Central site!

25. Network Issues
Remote Site(s) Central Dispatch
One open
port
required!

26. Nagios Configuration

27. Nagios Configuration
Nagios system communications
• Central system has NSCA Server which listens for events, normally in
5667
• Remote systems have NSCA Client which sends events to the Server
on 5667
• Notifications are encrypted according to the NSCA configuration

28. Nagios Configuration
● send_nsca Configuration
– password=fV5P0V8ggaFtNE
– encryption_method=1
● nsca daemon Configuration
– password=fV5P0V8ggaFtNE
– encryption_method=1
1 = Simple XOR (Just obfuscation, but very fast)

29. Nagios Configuration
● Encryption Options
0 = None 1 = Simple XOR 2 = DES
3 = 3DES (Triple DES) 4 = CAST-128 5 = CAST-256
6 = xTEA 7 = 3WAY 8 = BLOWFISH
9 = TWOFISH 10 = LOKI97 11 = RC2
12 = ARCFOUR 14 = RIJNDAEL-128
15 = RIJNDAEL-192 16 = RIJNDAEL-256
19 = WAKE 20 = SERPENT
22 = ENIGMA (Unix crypt) 23 = GOST 24 = SAFER64
25 = SAFER128 26 = SAFER+

30. Nagios Configuration
define command {
command_name notify-service-by-nsca
command_line /usr/bin/printf
"%st%st%st%sn"
"$HOSTNAME$" "$SERVICEDESC$"
$SERVICESTATEID$" "$SERVICEOUTPUT$ |
$SERVICEPERFDATA$" | tee -a
/tmp/service_alert.log | /usr/local/nagios/libexec/send_nsca -H
$CONTACTADDRESS1$ -c
/usr/local/nagios/etc/send_nsca.cfg
}

31. Nagios Configuration
define command {
command_name notify-host-by-nsca
command_line /usr/bin/printf "%st%st%sn"
"$HOSTNAME$" "$HOSTSTATEID$"
"$HOSTOUTPUT$" | /usr/sbin/send_nsca -H
$CONTACTADDRESS1$ -c
/usr/local/nagios/etc/send_nsca.cfg
}

32. Nagios Configuration
define contact {
contact_name Central_Dispatch
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,u,c,r,f,s
host_notification_options d,u,r,f,s
service_notification_commands
notify-service-by-nsca
host_notification_commands
notify-host-by-nsca
address1 nagios.omnitec.net
}

33. Nagios Configuration
● Central Site
– Set the service definition or template to passive
passive_check_enabled=1, and active_checks_enabled=0.
– Note: For any service to be monitored upstream, the
hostname and service description must match exactly the
data being sent via send_nsca!

34. Nagios Configuration
● Test service template on Central Server
define service {
name passive-service
use generic-service
check_freshness 1
passive_checks_enabled 1
active_checks_enabled 0
is_volatile 0
flap_detection_enabled 0
notification_options w,u,c,s
freshness_threshold 57600 ;12hr
}

35. Nagios Configuration
● Test service on Central Server
define service {
use passive-service
host_name localhost
service_description test
check_command check_dummy!3!"No Data Received"
}

36. Nagios Configuration
● Test service on Remote SRM
#!/usr/bin/perl
#############################################################
# RETURN CODES:
# 0-OK, 1-WARNING, 2-CRITICAL, 3-UNKNOWN
#############################################################
#CONFIG FILES
#$debug=1;
$config="/usr/local/nagios/etc/send_nsca.cfg";
# LOCAL SYSTEM CONFIG OPTIONS
$nsca_host="srm.omnitec.net";
$host="DevelSRM";
$service="test_service";

37. Nagios Configuration
● Test service on Remote SRM (cont'd)
# DEFAULT RETURNS
$code=3;
$result="What's going on";
# COMMAND LINE
$send_nsca="/usr/local/nagios/bin/send_nsca -c $config -H $nsca_host";
# Start
# INSERT YOUR FUN CODE HERE, Setting a $code and $result value
# End
if ($debug) {print "SENDING: $hostt$servicet$codet$resultn";}
open(SEND,"|$send_nsca") || die "Could not run $send_nsca: $!n";
print SEND "$hostt$servicet$codet$resultn";
close SEND;

38. Nagios Configuration
● System troubleshooting tips
– Check logs
● /usr/local/nagios/logs
● /var/log/apache2/error.log
– Verify UI user permissions
● /usr/local/nagios/etc/htpssword.users

39. Distributed/Hierarchical
Architecture
• Low cost
• Local Nagios Instance
• Central monitoring
and dispatch
Conclusion

40. Ongoing work, ..
● Motion Video
– B+ hardware
– OpenGL accssible (non-proprietary)
● Configuration UI
– Adagios
– Not fully NagiosCore 4 compatible yet

41. Questions?
Thank you!

42. The End
Lee Lammert
lvl@omnitec.net
@lvlammert