Mike Weber's presentation on using Nagios and preventing disasters in your configuration. The presentation was given during the Nagios World Conference North America held Sept 25-28th, 2012 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/nwcna

1. 10 Quick Steps To Disaster
Mike Weber
mweber@spidertools.com

2. Inheriting Aberrations
with Objects

3. Where are those settings coming from?
Object Inheritance
Object Priorities
Object Chaining
Incomplete Objects
Canceling Inheritance
Additive Inheritance
2012 3

4. Object Inheritance
2012 4

5. Object Inheritance: Templates
2012 5

6. Object Inheritance: No Hostgroups?
2012 6

7. Object Inheritance: From Hostgroup
2012 7

8. Object Inheritance: Info Option
2012 8

9. Object Priorities: Local then Inheritance
2012 9

10. Object Priorities: Order in List (Chaining)
2012 10

11. Incomplete Object: Only Lists One Image
2012 11

12. Canceling Inheritance: Object Contains Parents
2012 12

13. Canceling Inheritance: Wrong Parents
2012 13

14. Canceling Inheritance: Cancel Parents
2012 14

15. Canceling Inheritance: Canceled Parents
2012 15

16. Additive Inheritance: Append Object Contents
2012 16

17. Additive Inheritance: Append Object Contents
2012 17

18. Hoping BAD Things
Won't Happen

19. Real BAD Things Will Happen
Backups
Updates
Dependencies
2012 19

20. XI: Automated Backup
/etc/cron.d/nagiosxi
0 7 * * * root /root/scripts/automysqlbackup
0 8 * * * root /root/scripts/autopostgresqlbackup
/store/backups/mysql
daily
weekly
monthly
/store/backups/postgresql
daily
weekly
monthly
2012 20

21. XI: Upgrade Backup
#!/bin/bash
##### BackUp Of Nagios Before Upgrade #####
# Timestamp Backups
TIMESTAMP=$(date +%Y%m%d_%H%M);
echo $TIMESTAMP
service nagiosxi stop
service npcd stop
service ndo2db stop
service nagios stop
mkdir /bk/upgrade_$TIMESTAMP
tar cjf /bk/upgrade_$TIMESTAMP/nagios_$TIMESTAMP.tar.bz2 /usr/local/nagios
tar cjf /bk/upgrade_$TIMESTAMP/nagiosxi_$TIMESTAMP.tar.bz2 /usr/local/nagiosxi
pg_dump -U nagiosxi -c -F p nagiosxi | bzip2 -c >
/bk/upgrade_$TIMESTAMP/pg_nagiosxi_$TIMESTAMP.sql.bz2
mysqldump -u root -pnagiosxi nagios | bzip2 -c >
/bk/upgrade_$TIMESTAMP/my_nagios_$TIMESTAMP.sql.bz2
mysqldump -u root -pnagiosxi nagiosql | bzip2 -c >
/bk/upgrade_$TIMESTAMP/my_nagiosql_$TIMESTAMP.sql.bz2
service nagios start
service ndo2db start
service npcd start
service nagiosxi start
2012 21

22. Core: Backup
#!/bin/sh
# Timestamped Back Up
TIMESTAMP=`date +%Y%m%d_%H%M%S`;
echo $TIMESTAMP
tar czvf /bk/nagios_dir_$TIMESTAMP.tar.gz /usr/local/nagios
tar czvf /bk/pnp4nagios_dir_$TIMESTAMP.tar.gz /usr/local/pnp4nagios
2012 22

23. Ignoring/Encouraging
System Warnings

24. Configuration Errors: Service Checks
2012 24

25. Solution: Service Template Management
2012 25

26. Service Template: Check Settings
2012 26

27. Service Template: Alert Settings
2012 27

28. Service Template: Add Hostgroup
2012 28

29. Solution: Service Template Management
2012 29

30. Max Concurrent Service Checks
2012 30

31. Maximum Concurrent Checks
Edit nagios.cfg to avoid latency issues.
max_concurrent_checks=0
2012 31

32. Mangling Users and Contacts

33. Managing Users and Contacts
Users (access to the web interface)
Contacts (notifications)
2012 33

34. Creating Users: Web Interface
2012 34

35. Creating Users: Web Interface
2012 35

36. Creating Users: Restricted
2012 36

37. Creating Users: Restricted
2012 37

38. Managing Administrators: Full Access
2012 38

39. Managing Administrators: Full Access
2012 39

40. Core: cgi.cfg
authorized_for_system_information=nagiosadmin,john,sue,mark,tom,mary,ralph
authorized_for_configuration_information=nagiosadmin,john,sue,mark,tom,mary,ralph
authorized_for_system_commands=nagiosadmin,john,sue,mark,tom,mary,ralph
authorized_for_all_services=nagiosadmin,management,john,sue,mark,tom,mary,ralph
authorized_for_all_hosts=nagiosadmin,management,john,sue,mark,tom,mary,ralph
authorized_for_all_service_commands=nagiosadmin,john,sue,mark,tom,mary,ralph
authorized_for_all_host_commands=nagiosadmin,john,sue,mark,tom,mary,ralph
authorized_for_read_only=management
2012 40

41. Contacts
2012 41

42. Monitoring Non-Existent Ports
on Switches

43. Save Resources
Use AdminDown on Ports
* Administratively set unused ports as AdminDown
* Modify ifoperstatus
Turn Off Monitoring on Used Ports
Remove the Checks
2012 43

44. Unused Switch Ports: Wasting Resources
* check port status
* check bandwidth
* send notifications
* ignore notifications
2012 44

45. Modify check_ifoperstatus
Here is the code the affects output. You need to modify the line:
if ( not defined $adminWarn or $adminWarn eq "w" ) {
$state = 'WARNING';
to
$state = 'OK';
It is highlighted in the example.
##
if ( not ($response­>{$snmpIfAdminStatus} == 1) ) {
$answer = "Interface $name (index $snmpkey) is administratively
down.";
if ( not defined $adminWarn or $adminWarn eq "w" ) {
$state = 'OK';
} elsif ( $adminWarn eq "i" ) {
$state = 'OK';
} elsif ( $adminWarn eq "c" ) {
$state = 'CRITICAL';
} else { # If wrong value for ­a, say warning
$state = 'WARNING';
}
2012 45

46. Administratively Down Ports
2012 46

47. Disable Port Checks
* 790 port checks disabled
* 1.5 GB of RAM saved
* 18% reduction in max service check execution time
2012 47

48. Encouraging Non-Accountability
for Changes

49. Who Makes Changes on Your Nagios?
Limit Admin Access
Require Training
Create Policy for Changes
Use a Test Server
2012 49

50. Audit Log
2012 50

51. Abusing Nagios XI Wizards

52. Wizard or Manual Creation: Assessment
Installation
Which method provides the most efficient installation?
Example: Using a wizard for a switch is most efficient.
Example: Manually creating a service check to be used on 100
servers is most efficient.
Visibility
Will it provide access to view the grouping of devices?
Example: Can effective reports be created from visible devices?
Management
Does it make management easier in the long run?
Example: The use of templates is an efficient method to manage
multiple devices that are similar.
2012 52

53. Template Management
2012 53

54. Disregarding Network
Relationships

55. Reachability
2012 55

56. Host: Manage Parents
2012 56

57. Host: Manage Parents
2012 57

58. Network Relationships: Parents
2012 58

59. Importing Infectious Diseases

60. GUI Infection: Lack of Command Line Skills
Backups
* cron jobs
* manual backups
* verification
Analysis
* disk space
* logs
Troubleshooting
* finding stuff
* processes
* permissions
Edit Files
* learning vi or nano
2012 60

61. Short Cut Infection: Auto-Discovery
2012 61

62. Overestimating Human
Intelligence

63. Some of the Things We Do as
Humans Defies Logic