Mike Weber's presentation on Mike Weber. The presentation was given during the Nagios World Conference North America held Sept 20-Oct 2nd, 2013 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/nwcna

1. SNMP Extensions
Mike Weber
mweber@spidertools.com

2. 2013 2
SNMP: 9000 Puzzle Pieces

3. 2013 3
Building the SNMP Puzzle
Opening the Box
* dumping the pieces
* stand up box to see the goal
Random Puzzle Pieces
* examining the connections
* searching for straight pieces
* organizing the colors
Fitting the Puzzle Pieces Together
* clicking the pieces together
* making sense of the mess

4. 2013 4
SNMP: 9000 Puzzle Pieces

5. 2013 5
SNMP: The Challenging Puzzle
Actual Quote From Amazon.com
I have to be truthful. I have not finished the 9000 piece Ravensburger
Puzzle entitled Underwater Paradise. Despite my valiant attempt at this
immense and challenging endeavor, I simply don't think that I'm smart
enough or patient enough to do so. I have been bested by a puzzle
and I will readily admit it to anyone that asks.

6. 2013 6
SNMP extend Scripts
extend dfcheck /bin/df ­h
extend http_event "/bin/sh /usr/local/nagios/libexec/http_event"
extend provides multi­line output from the command and is indicated by a name not a
number
Creating New Options with SNMP
Edit /etc/snmp/snmpd.conf

7. 2013 7
SNMP extend Scripts: Windows
It is possible to edit the Windows registry to allow SNMP Extensions.
http://support.microsoft.com/kb/128729
Possible but not probable.

8. 2013 8
SNMP Monitoring: Limitations

9. 2013 9
SNMP Data Collection:
Static Data
snmpwalk -v2c -c public 192.168.5.190 hrSWRunName.9909
HOST-RESOURCES-MIB::hrSWRunName.9909 = STRING: "httpd"
Dynamic Data
snmpget -v2c -c public 192.168.5.190 sysUpTimeInstance
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (574870)
1:35:48.70

10. 2013 10
Solution: NRPE or NRDS

11. 2013 11
SNMP Monitoring: No Limitations

12. 2013 12
SNMP Customized Data Collection:
Static Data
snmpwalk -v2c -c public 192.168.5.190 hrSWRunName.9909
HOST-RESOURCES-MIB::hrSWRunName.9909 = STRING: "httpd"
Dynamic Data
snmpget -v2c -c public 192.168.5.190 sysUpTimeInstance
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (574870)
1:35:48.70
Customized Data: Executing Commands/Scripts
extend dfcheck /bin/df ­h
Creating commands and arguments
Leverage Nagios Plugins
Reactions to Events
extend http_event /bin/sh
/usr/local/nagios/libexec/http_event

13. 2013 13
Why Make the Effort?
Complete Flexibility for Monitoring
capture data
capture data created
leverage Nagios plugins and scripts
Ability to Create Responses to Server Situations
* event handlers
* security responses
Total SNMP Server Monitoring
* avoid other methods of monitoring
Using check_snmp a Compiled Plugin
* saves on resources

14. 2013 14
Compiled Plugin Savings!!!
Compiled NSCA NSClient++ SSH Perl
0
2
4
6
8
10
12
RAM

15. 2013 15
SNMP Extensions:
C Modules
SNMP agent for PostgresSQL
dlmod Modules
mod-apache-snmp, HP Insight Management Agents, Software RAID
AgentX
VMWare ESX Server SNMP Agent, Net-SNMP Subagent for NFSv3
Perl sub-agent
MySQL Monitoring,Bind9 Statistics, APC UPS Monitoring
pass Scripts
Bacula Subagent
exec/extend Scripts
Bind9 Statistics, Linux Disk IO, Extending Net-SNMP
Proxy
Squid SNMP Support

16. 2013 16
Standard MIBs
AGENTX­MIB.txt IPV6­TC.txt SNMP­NOTIFICATION­MIB.txt
DISMAN­EVENT­MIB.txt IPV6­UDP­MIB.txt SNMP­PROXY­MIB.txt
DISMAN­SCHEDULE­MIB.txt LM­SENSORS­MIB.txt SNMP­TARGET­MIB.txt
DISMAN­SCRIPT­MIB.txt MTA­MIB.txt SNMP­USER­BASED­SM­MIB.txt
EtherLike­MIB.txt NET­SNMP­AGENT­MIB.txt SNMP­USM­AES­MIB.txt
HCNUM­TC.txt NET­SNMP­EXAMPLES­MIB.txt SNMP­USM­DH­OBJECTS­MIB.txt
HOST­RESOURCES­MIB.txt NET­SNMP­EXTEND­MIB.txt SNMPv2­CONF.txt
HOST­RESOURCES­TYPES.txt NET­SNMP­MIB.txt SNMPv2­MIB.txt
IANA­ADDRESS­FAMILY­NUMBERS­MIB.txt NET­SNMP­TC.txt SNMPv2­SMI.txt
IANAifType­MIB.txt NETWORK­SERVICES­MIB.txt SNMPv2­TC.txt
IANA­LANGUAGE­MIB.txt NOTIFICATION­LOG­MIB.txt SNMPv2­TM.txt
IANA­RTPROTO­MIB.txt RFC1155­SMI.txt SNMP­VIEW­BASED­ACM­MIB.txt
IF­INVERTED­STACK­MIB.txt RFC1213­MIB.txt TCP­MIB.txt
IF­MIB.txt RFC­1215.txt TRANSPORT­ADDRESS­MIB.txt
INET­ADDRESS­MIB.txt RMON­MIB.txt UCD­DEMO­MIB.txt
IP­FORWARD­MIB.txt SCTP­MIB.txt UCD­DISKIO­MIB.txt
IP­MIB.txt SMUX­MIB.txt UCD­DLMOD­MIB.txt
IPV6­ICMP­MIB.txt SNMP­COMMUNITY­MIB.txt UCD­IPFWACC­MIB.txt
IPV6­MIB.txt SNMP­FRAMEWORK­MIB.txt UCD­SNMP­MIB.txt
IPV6­TCP­MIB.txt SNMP­MPD­MIB.txt UDP­MIB.txt

17. 2013 17
netSnmpExtendMIB
Here is the tree of the netSnmpExtendMib
1.3.6.1.4.1.8072.1.3.1
1 = iso
3 = org
6 = dod
1 = internet
4 = private
1 = enterprises
8072 = netSNMP
1 = nsExtensions
3 = nsSnmpExtendMIB

18. 2013 18
Extend MIB
netSnmpExtendMIB 1.3.6.1.4.1.8072.1.3.1
nsExtendObjects 1.3.6.1.4.1.8072.1.3.2
nsExtendNumEntries 1.3.6.1.4.1.8072.1.3.2.1
nsExtendConfigTable 1.3.6.1.4.1.8072.1.3.2.2
nsExtendConfigEntry 1.3.6.1.4.1.8072.1.3.2.2.1
nsExtendToken 1.3.6.1.4.1.8072.1.3.2.2.1.1
nsExtendCommand 1.3.6.1.4.1.8072.1.3.2.2.1.2
nsExtendStorage 1.3.6.1.4.1.8072.1.3.2.2.1.20
nsExtendStatus 1.3.6.1.4.1.8072.1.3.2.2.1.21
nsExtendArgs 1.3.6.1.4.1.8072.1.3.2.2.1.3
nsExtendInput 1.3.6.1.4.1.8072.1.3.2.2.1.4
nsExtendCacheTime 1.3.6.1.4.1.8072.1.3.2.2.1.5
nsExtendExecType 1.3.6.1.4.1.8072.1.3.2.2.1.6
nsExtendRunType 1.3.6.1.4.1.8072.1.3.2.2.1.7
nsExtendOutput1Table 1.3.6.1.4.1.8072.1.3.2.3
nsExtendOutput1Entry 1.3.6.1.4.1.8072.1.3.2.3.1
nsExtendOutput1Line 1.3.6.1.4.1.8072.1.3.2.3.1.1
nsExtendOutputFull 1.3.6.1.4.1.8072.1.3.2.3.1.2
nsExtendOutNumLines 1.3.6.1.4.1.8072.1.3.2.3.1.3
nsExtendResult 1.3.6.1.4.1.8072.1.3.2.3.1.4
nsExtendOutput2Table 1.3.6.1.4.1.8072.1.3.2.4
nsExtendOutput2Entry 1.3.6.1.4.1.8072.1.3.2.4.1
nsExtendLineIndex 1.3.6.1.4.1.8072.1.3.2.4.1.1
nsExtendOutLine 1.3.6.1.4.1.8072.1.3.2.4.1.2
nsExtendGroups 1.3.6.1.4.1.8072.1.3.3
nsExtendConfigGroup 1.3.6.1.4.1.8072.1.3.3.1
nsExtendOutputGroup 1.3.6.1.4.1.8072.1.3.3.2

19. 2013 19
Extend MIB
snmpwalk ­v1 ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2
NET­SNMP­EXTEND­MIB::nsExtendNumEntries.0 = INTEGER: 2
NET­SNMP­EXTEND­MIB::nsExtendCommand."dfcheck" = STRING: /bin/df
NET­SNMP­EXTEND­MIB::nsExtendCommand."http_event" = STRING: /bin/sh /usr/local/nagios/libexec/http_event
NET­SNMP­EXTEND­MIB::nsExtendArgs."dfcheck" = STRING: ­h
NET­SNMP­EXTEND­MIB::nsExtendArgs."http_event" = STRING:
NET­SNMP­EXTEND­MIB::nsExtendInput."dfcheck" = STRING:
NET­SNMP­EXTEND­MIB::nsExtendInput."http_event" = STRING:
NET­SNMP­EXTEND­MIB::nsExtendCacheTime."dfcheck" = INTEGER: 5
NET­SNMP­EXTEND­MIB::nsExtendCacheTime."http_event" = INTEGER: 5
NET­SNMP­EXTEND­MIB::nsExtendExecType."dfcheck" = INTEGER: exec(1)
NET­SNMP­EXTEND­MIB::nsExtendExecType."http_event" = INTEGER: exec(1)
NET­SNMP­EXTEND­MIB::nsExtendRunType."dfcheck" = INTEGER: run­on­read(1)
NET­SNMP­EXTEND­MIB::nsExtendRunType."http_event" = INTEGER: run­on­read(1)
NET­SNMP­EXTEND­MIB::nsExtendStorage."dfcheck" = INTEGER: permanent(4)
NET­SNMP­EXTEND­MIB::nsExtendStorage."http_event" = INTEGER: permanent(4)
NET­SNMP­EXTEND­MIB::nsExtendStatus."dfcheck" = INTEGER: active(1)
NET­SNMP­EXTEND­MIB::nsExtendStatus."http_event" = INTEGER: active(1)
NET­SNMP­EXTEND­MIB::nsExtendOutput1Line."dfcheck" = STRING: Filesystem Size Used Avail Use% Mounted on
NET­SNMP­EXTEND­MIB::nsExtendOutput1Line."http_event" = STRING: Stopping httpd: [ OK ]
NET­SNMP­EXTEND­MIB::nsExtendOutputFull."dfcheck" = STRING: Filesystem Size Used Avail Use% Mounted on
/dev/simfs 4.0G 3.1G 844M 79% /
none 303M 4.0K 303M 1% /dev
NET­SNMP­EXTEND­MIB::nsExtendOutputFull."http_event" = STRING: Stopping httpd: [ OK ]
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.5.190 for
ServerName
[ OK ]
NET­SNMP­EXTEND­MIB::nsExtendOutNumLines."dfcheck" = INTEGER: 3
NET­SNMP­EXTEND­MIB::nsExtendOutNumLines."http_event" = INTEGER: 3
NET­SNMP­EXTEND­MIB::nsExtendResult."dfcheck" = INTEGER: 0
NET­SNMP­EXTEND­MIB::nsExtendResult."http_event" = INTEGER: 0
NET­SNMP­EXTEND­MIB::nsExtendOutLine."dfcheck".1 = STRING: Filesystem Size Used Avail Use% Mounted on
NET­SNMP­EXTEND­MIB::nsExtendOutLine."dfcheck".2 = STRING: /dev/simfs 4.0G 3.1G 844M 79% /
NET­SNMP­EXTEND­MIB::nsExtendOutLine."dfcheck".3 = STRING: none 303M 4.0K 303M 1% /dev
NET­SNMP­EXTEND­MIB::nsExtendOutLine."http_event".1 = STRING: Stopping httpd: [ OK ]
NET­SNMP­EXTEND­MIB::nsExtendOutLine."http_event".2 = STRING: Starting httpd: httpd: Could not reliably determine the
server's fully qualified domain name, using 192.168.5.190 for ServerName
NET­SNMP­EXTEND­MIB::nsExtendOutLine."http_event".3 = STRING: [ OK ]

20. 2013 20
Convert to Numerical Values
snmpwalk ­v1 ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2 ­On
.1.3.6.1.4.1.8072.1.3.2.1.0 = INTEGER: 2
.1.3.6.1.4.1.8072.1.3.2.2.1.2.7.100.102.99.104.101.99.107 = STRING: /bin/df
.1.3.6.1.4.1.8072.1.3.2.2.1.2.10.104.116.116.112.95.101.118.101.110.116 = STRING: /bin/sh
/usr/local/nagios/libexec/http_event
.1.3.6.1.4.1.8072.1.3.2.2.1.3.7.100.102.99.104.101.99.107 = STRING: ­h
.1.3.6.1.4.1.8072.1.3.2.2.1.3.10.104.116.116.112.95.101.118.101.110.116 = STRING:
.1.3.6.1.4.1.8072.1.3.2.2.1.4.7.100.102.99.104.101.99.107 = STRING:
.1.3.6.1.4.1.8072.1.3.2.2.1.4.10.104.116.116.112.95.101.118.101.110.116 = STRING:
.1.3.6.1.4.1.8072.1.3.2.2.1.5.7.100.102.99.104.101.99.107 = INTEGER: 5
.1.3.6.1.4.1.8072.1.3.2.2.1.5.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: 5
.1.3.6.1.4.1.8072.1.3.2.2.1.6.7.100.102.99.104.101.99.107 = INTEGER: exec(1)
.1.3.6.1.4.1.8072.1.3.2.2.1.6.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: exec(1)
.1.3.6.1.4.1.8072.1.3.2.2.1.7.7.100.102.99.104.101.99.107 = INTEGER: run­on­read(1)
.1.3.6.1.4.1.8072.1.3.2.2.1.7.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: run­on­read(1)
.1.3.6.1.4.1.8072.1.3.2.2.1.20.7.100.102.99.104.101.99.107 = INTEGER: permanent(4)
.1.3.6.1.4.1.8072.1.3.2.2.1.20.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: permanent(4)
.1.3.6.1.4.1.8072.1.3.2.2.1.21.7.100.102.99.104.101.99.107 = INTEGER: active(1)
.1.3.6.1.4.1.8072.1.3.2.2.1.21.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: active(1)
.1.3.6.1.4.1.8072.1.3.2.3.1.1.7.100.102.99.104.101.99.107 = STRING: Filesystem Size Used Avail Use% Mounted on
.1.3.6.1.4.1.8072.1.3.2.3.1.1.10.104.116.116.112.95.101.118.101.110.116 = STRING: Stopping httpd: [ OK ]
.1.3.6.1.4.1.8072.1.3.2.3.1.2.7.100.102.99.104.101.99.107 = STRING: Filesystem Size Used Avail Use% Mounted on
/dev/simfs 4.0G 3.1G 843M 79% /
none 303M 4.0K 303M 1% /dev
.1.3.6.1.4.1.8072.1.3.2.3.1.2.10.104.116.116.112.95.101.118.101.110.116 = STRING: Stopping httpd: [ OK ]
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.5.190 for
ServerName
[ OK ]
.1.3.6.1.4.1.8072.1.3.2.3.1.3.7.100.102.99.104.101.99.107 = INTEGER: 3
.1.3.6.1.4.1.8072.1.3.2.3.1.3.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: 3
.1.3.6.1.4.1.8072.1.3.2.3.1.4.7.100.102.99.104.101.99.107 = INTEGER: 0
.1.3.6.1.4.1.8072.1.3.2.3.1.4.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: 0
.1.3.6.1.4.1.8072.1.3.2.4.1.2.7.100.102.99.104.101.99.107.1 = STRING: Filesystem Size Used Avail Use% Mounted on
.1.3.6.1.4.1.8072.1.3.2.4.1.2.7.100.102.99.104.101.99.107.2 = STRING: /dev/simfs 4.0G 3.1G 843M 79% /
.1.3.6.1.4.1.8072.1.3.2.4.1.2.7.100.102.99.104.101.99.107.3 = STRING: none 303M 4.0K 303M 1% /dev
.1.3.6.1.4.1.8072.1.3.2.4.1.2.10.104.116.116.112.95.101.118.101.110.116.1 = STRING: Stopping httpd: [ OK ]
.1.3.6.1.4.1.8072.1.3.2.4.1.2.10.104.116.116.112.95.101.118.101.110.116.2 = STRING: Starting httpd: httpd: Could not
reliably determine the server's fully qualified domain name, using 192.168.5.190 for ServerName
.1.3.6.1.4.1.8072.1.3.2.4.1.2.10.104.116.116.112.95.101.118.101.110.116.3 = STRING: [ OK ]

21. 2013 21
SNMP Extend: Puzzle Pieces

22. 2013 22
nsExtendNumEntries
nsExtendNumEntries 1.3.6.1.4.1.8072.1.3.2.1
This will be an integer indicating the number of rows in the nsExtendConfigTable. This is
the number of extend commands that are available. With three extend commands you will get a
return of "3".
snmpwalk ­v2c ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2.1
NET­SNMP­EXTEND­MIB::nsExtendNumEntries.0 = INTEGER: 3
Here is current list of extend scripts found in the /etc/snmp/snmpd.conf.
extend http_event "/bin/sh /usr/local/nagios/libexec/http_event"
extend dfcheck /bin/df ­h
extend who /usr/bin/who

23. 2013 23
nsExtendCommand
nsExtendCommand 1.3.6.1.4.1.8072.1.3.2.2.1.2
The full path of the command to run.
snmpwalk ­v2c ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2.2.1.2
NET­SNMP­EXTEND­MIB::nsExtendCommand."who" = STRING: /usr/bin/who
NET­SNMP­EXTEND­MIB::nsExtendCommand."dfcheck" = STRING: /bin/df
NET­SNMP­EXTEND­MIB::nsExtendCommand."http_event" = STRING: /bin/sh
/usr/local/nagios/libexec/http_event
/etc/snmp/snmpd.conf
extend http_event "/bin/sh /usr/local/nagios/libexec/http_event"
extend dfcheck /bin/df ­h
extend who /usr/bin/who

24. 2013 24
nsExtendStatus
nsExtendStatus 1.3.6.1.4.1.8072.1.3.2.2.1.21
snmpwalk ­v2c ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2.2.1.21
NET­SNMP­EXTEND­MIB::nsExtendStatus."who" = INTEGER: active(1)
NET­SNMP­EXTEND­MIB::nsExtendStatus."dfcheck" = INTEGER: active(1)
NET­SNMP­EXTEND­MIB::nsExtendStatus."http_event" = INTEGER: active(1)

25. 2013 25
nsExtendArgs
nsExtendArgs 1.3.6.1.4.1.8072.1.3.2.2.1.3
Command line arguments for the command.
snmpwalk ­v2c ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2.2.1.3
NET­SNMP­EXTEND­MIB::nsExtendArgs."who" = STRING:
NET­SNMP­EXTEND­MIB::nsExtendArgs."dfcheck" = STRING: ­h
NET­SNMP­EXTEND­MIB::nsExtendArgs."http_event" = STRING:
/etc/snmp/snmpd.conf
extend http_event "/bin/sh /usr/local/nagios/libexec/http_event"
extend dfcheck /bin/df ­h
extend who /usr/bin/who

26. 2013 26
nsExtendCacheTime
nsExtendCacheTime 1.3.6.1.4.1.8072.1.3.2.2.1.5
The length of time the output for the command will be cached. If an additional command is
received during the cache time, the command will not be rerun to obtain the value. If the
value of ­1 is used the cache will not be used.
snmpwalk ­v2c ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2.2.1.5
NET­SNMP­EXTEND­MIB::nsExtendCacheTime."who" = INTEGER: 5
NET­SNMP­EXTEND­MIB::nsExtendCacheTime."dfcheck" = INTEGER: 5
NET­SNMP­EXTEND­MIB::nsExtendCacheTime."http_event" = INTEGER: 5

27. 2013 27
nsExtendCacheTime
nsExtendCacheTime 1.3.6.1.4.1.8072.1.3.2.2.1.5
Extend cache time
snmpwalk ­v2c ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2.2.1.5
NET­SNMP­EXTEND­MIB::nsExtendCacheTime."who" = INTEGER: 5
NET­SNMP­EXTEND­MIB::nsExtendCacheTime."dfcheck" = INTEGER: 5
NET­SNMP­EXTEND­MIB::nsExtendCacheTime."http_event" = INTEGER: 5
snmpset ­v2c ­c nagios 192.168.5.190 'NET­SNMP­EXTEND­MIB::nsExtendCacheTime."dfcheck"' i 20
snmpwalk ­v2c ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2.2.1.5
NET­SNMP­EXTEND­MIB::nsExtendCacheTime."who" = INTEGER: 5
NET­SNMP­EXTEND­MIB::nsExtendCacheTime."dfcheck" = INTEGER: 20
NET­SNMP­EXTEND­MIB::nsExtendCacheTime."http_event" = INTEGER: 5
The value is reset to the default when snmpd is restarted.

28. 2013 28
nsExtendExecType
nsExtendExecType 1.3.6.1.4.1.8072.1.3.2.2.1.6
There are two options; exec and shell. This is the mechanism used to invoke the command.
snmpwalk ­v2c ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2.2.1.6
NET­SNMP­EXTEND­MIB::nsExtendExecType."who" = INTEGER: exec(1)
NET­SNMP­EXTEND­MIB::nsExtendExecType."dfcheck" = INTEGER: exec(1)
NET­SNMP­EXTEND­MIB::nsExtendExecType."http_event" = INTEGER: exec(1)

29. 2013 29
nsExtendRunType
nsExtendRunType 1.3.6.1.4.1.8072.1.3.2.2.1.7
There are two options here; run­on­read which immediately invokes the command or run­on­set
which will only be invoked on the receipt of a SET assignment.
snmpwalk ­v2c ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2.2.1.7
NET­SNMP­EXTEND­MIB::nsExtendRunType."who" = INTEGER: run­on­read(1)
NET­SNMP­EXTEND­MIB::nsExtendRunType."dfcheck" = INTEGER: run­on­read(1)
NET­SNMP­EXTEND­MIB::nsExtendRunType."http_event" = INTEGER: run­on­read(1)

30. 2013 30
nsExtendOutputFull
nsExtendOutputFull 1.3.6.1.4.1.8072.1.3.2.3.1.2
Full output of command as a string.
snmpwalk ­v2c ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2.3
NET­SNMP­EXTEND­MIB::nsExtendOutput1Line."who" = STRING: root pts/0 2013­08­31
23:19 (192.168.5.103)
NET­SNMP­EXTEND­MIB::nsExtendOutput1Line."dfcheck" = STRING: Filesystem Size
Used Avail Use% Mounted on
NET­SNMP­EXTEND­MIB::nsExtendOutput1Line."http_event" = STRING: Stopping httpd: [ OK ]
NET­SNMP­EXTEND­MIB::nsExtendOutputFull."who" = STRING: root pts/0 2013­08­31
23:19 (192.168.5.103)
NET­SNMP­EXTEND­MIB::nsExtendOutputFull."dfcheck" = STRING: Filesystem Size Used
Avail Use% Mounted on
/dev/simfs 4.0G 3.4G 526M 87% /
none 303M 4.0K 303M 1% /dev
NET­SNMP­EXTEND­MIB::nsExtendOutputFull."http_event" = STRING: Stopping httpd: [ OK ]
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain
name, using 192.168.5.190 for ServerName
[ OK ]
NET­SNMP­EXTEND­MIB::nsExtendOutNumLines."who" = INTEGER: 1
NET­SNMP­EXTEND­MIB::nsExtendOutNumLines."dfcheck" = INTEGER: 3
NET­SNMP­EXTEND­MIB::nsExtendOutNumLines."http_event" = INTEGER: 3
NET­SNMP­EXTEND­MIB::nsExtendResult."who" = INTEGER: 0
NET­SNMP­EXTEND­MIB::nsExtendResult."dfcheck" = INTEGER: 0
NET­SNMP­EXTEND­MIB::nsExtendResult."http_event" = INTEGER: 0

31. 2013 31
Recognizing Command OIDs: 107
Here is an extend script found in the /etc/snmp/snmpd.conf.
extend dfcheck /bin/df ­h
.1.3.6.1.4.1.8072.1.3.2.2.1.2.7.100.102.99.104.101.99.107 = STRING: /bin/df
.1.3.6.1.4.1.8072.1.3.2.2.1.3.7.100.102.99.104.101.99.107 = STRING: ­h
.1.3.6.1.4.1.8072.1.3.2.2.1.4.7.100.102.99.104.101.99.107 = STRING:
.1.3.6.1.4.1.8072.1.3.2.2.1.5.7.100.102.99.104.101.99.107 = INTEGER: 5
.1.3.6.1.4.1.8072.1.3.2.2.1.6.7.100.102.99.104.101.99.107 = INTEGER: exec(1)
.1.3.6.1.4.1.8072.1.3.2.2.1.7.7.100.102.99.104.101.99.107 = INTEGER: run­on­read(1)
.1.3.6.1.4.1.8072.1.3.2.2.1.20.7.100.102.99.104.101.99.107 = INTEGER: permanent(4)
.1.3.6.1.4.1.8072.1.3.2.2.1.21.7.100.102.99.104.101.99.107 = INTEGER: active(1)
.1.3.6.1.4.1.8072.1.3.2.3.1.1.7.100.102.99.104.101.99.107 = STRING: Filesystem Size Used Avail Use% Mounted on
.1.3.6.1.4.1.8072.1.3.2.3.1.2.7.100.102.99.104.101.99.107 = STRING: Filesystem Size Used Avail Use% Mounted on
/dev/simfs 4.0G 3.1G 843M 79% /
none 303M 4.0K 303M 1% /dev
.1.3.6.1.4.1.8072.1.3.2.3.1.3.7.100.102.99.104.101.99.107 = INTEGER: 3
.1.3.6.1.4.1.8072.1.3.2.3.1.4.7.100.102.99.104.101.99.107 = INTEGER: 0
.1.3.6.1.4.1.8072.1.3.2.4.1.2.7.100.102.99.104.101.99.107.1 = STRING: Filesystem Size Used Avail Use% Mounted on
.1.3.6.1.4.1.8072.1.3.2.4.1.2.7.100.102.99.104.101.99.107.2 = STRING: /dev/simfs 4.0G 3.1G 843M 79% /
.1.3.6.1.4.1.8072.1.3.2.4.1.2.7.100.102.99.104.101.99.107.3 = STRING: none 303M 4.0K 303M 1% /dev

32. 2013 32
Recognizing Command OIDs: 116
Here is an extend script found in the /etc/snmp/snmpd.conf.
extend http_event "/bin/sh /usr/local/nagios/libexec/http_event"
.1.3.6.1.4.1.8072.1.3.2.2.1.2.10.104.116.116.112.95.101.118.101.110.116 = STRING: /bin/sh
/usr/local/nagios/libexec/http_event
.1.3.6.1.4.1.8072.1.3.2.2.1.3.10.104.116.116.112.95.101.118.101.110.116 = STRING:
.1.3.6.1.4.1.8072.1.3.2.2.1.4.10.104.116.116.112.95.101.118.101.110.116 = STRING:
.1.3.6.1.4.1.8072.1.3.2.2.1.5.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: 5
.1.3.6.1.4.1.8072.1.3.2.2.1.6.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: exec(1)
.1.3.6.1.4.1.8072.1.3.2.2.1.7.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: run­on­read(1)
.1.3.6.1.4.1.8072.1.3.2.2.1.20.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: permanent(4)
.1.3.6.1.4.1.8072.1.3.2.2.1.21.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: active(1)
.1.3.6.1.4.1.8072.1.3.2.3.1.1.10.104.116.116.112.95.101.118.101.110.116 = STRING: Stopping httpd: [ OK ]
.1.3.6.1.4.1.8072.1.3.2.3.1.2.7.100.102.99.104.101.99.107 = STRING: Filesystem Size Used Avail Use% Mounted on
/dev/simfs 4.0G 3.1G 843M 79% /
none 303M 4.0K 303M 1% /dev
.1.3.6.1.4.1.8072.1.3.2.3.1.2.10.104.116.116.112.95.101.118.101.110.116 = STRING: Stopping httpd: [ OK ]
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.5.190 for
ServerName
[ OK ]
.1.3.6.1.4.1.8072.1.3.2.3.1.3.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: 3
.1.3.6.1.4.1.8072.1.3.2.3.1.4.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: 0
.1.3.6.1.4.1.8072.1.3.2.4.1.2.10.104.116.116.112.95.101.118.101.110.116.1 = STRING: Stopping httpd: [ OK ]
.1.3.6.1.4.1.8072.1.3.2.4.1.2.10.104.116.116.112.95.101.118.101.110.116.2 = STRING: Starting httpd: httpd: Could not
reliably determine the server's fully qualified domain name, using 192.168.5.190 for ServerName
.1.3.6.1.4.1.8072.1.3.2.4.1.2.10.104.116.116.112.95.101.118.101.110.116.3 = STRING: [ OK ]

33. 2013 33
Multiple Systems: OIDs the Same
It is important that the OIDs are the same, if the command is exactly the same, on
multiple machines.
CentOS 6
.1.3.6.1.4.1.8072.1.3.2.2.1.2.7.100.102.99.104.101.99.107 = STRING: /bin/df
.1.3.6.1.4.1.8072.1.3.2.2.1.2.10.104.116.116.112.95.101.118.101.110.116 = STRING:
/bin/sh /usr/local/nagios/libexec/http_event
Ubuntu 12.04
.1.3.6.1.4.1.8072.1.3.2.2.1.2.7.100.102.99.104.101.99.107 = STRING: /bin/df
.1.3.6.1.4.1.8072.1.3.2.2.1.2.10.104.116.116.112.95.101.118.101.110.116 = STRING:
/bin/sh /usr/local/nagios/libexec/http_event

34. 2013 34
OIDs May Produce Different Outcome
In this example the OID is the same, however the output is different because the
filesystem is different.
CentOS 6.3
.1.3.6.1.4.1.8072.1.3.2.3.1.2.7.100.102.99.104.101.99.107 = STRING: Filesystem
Size Used Avail Use% Mounted on
/dev/simfs 4.0G 3.1G 843M 79% /
none 303M 4.0K 303M 1% /dev
Ubuntu 12.04
.1.3.6.1.4.1.8072.1.3.2.3.1.2.7.100.102.99.104.101.99.107 = STRING: Filesystem
Size Used Avail Use% Mounted on
/dev/vda 20G 1.3G 18G 7% /
none 243M 148K 243M 1% /dev
none 247M 0 247M 0% /dev/shm
none 247M 84K 247M 1% /var/run
none 247M 0 247M 0% /var/lock
none 247M 0 247M 0% /lib/init/rw
none 20G 1.3G 18G 7% /var/lib/ureadahead/debugfs

35. Fitting the Pieces TogetherFitting the Pieces Together

36. 2013 36
Edit /etc/snmp/snmpd.conf
extend dfcheck /bin/df ­h
extend http_event "/bin/sh /usr/local/nagios/libexec/http_event"

37. 2013 37
Example: extend dfcheck /bin/df -h
Problem Develops:
Limiting to OID That Lists Partitions
snmpwalk ­v1 ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2.3.1.2.7.100.102.99.104.101.99.107
NET­SNMP­EXTEND­MIB::nsExtendOutputFull."dfcheck" = STRING: Filesystem Size Used Avail
Use% Mounted on
/dev/simfs 4.0G 3.1G 843M 79% /
none 303M 4.0K 303M 1% /dev
Limit with grep for / Partition
snmpwalk ­v1 ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2.3.1.2.7.100.102.99.104.101.99.107 |
grep "/$"
/dev/simfs 4.0G 3.1G 843M 79% /
Problem: Test a Plugin
check_snmp ­H 192.168.5.190 ­C public ­o .1.3.6.1.4.1.8072.1.3.2.3.1.2.7.100.102.99.104.101.99.107
SNMP OK ­ "Filesystem Size Used Avail Use% Mounted on/dev/simfs 4.0G 3.1G
843M 79% /
none 303M 4.0K 303M 1% /dev"

38. 2013 38
Creating the Service Check
Scan the tree for an OID for the Command
snmpwalk ­v1 ­c public 192.168.5.190 .1.3.6.1.4.1.8072.1.3.2
.1.3.6.1.4.1.8072.1.3.2.4.1.2.7.100.102.99.104.101.99.107.2 = STRING: /dev/simfs 4.0G
3.1G 843M 79% /
Text String for Service Check
­C public ­o .1.3.6.1.4.1.8072.1.3.2.4.1.2.7.100.102.99.104.101.99.107.2

39. 2013 39
XI Service Check

40. 2013 40
Core Service Check
define command{
command_name check_snmp
command_line $USER1$/check_snmp ­H $HOSTADDRESS$ $ARG1$
}
define service{
use generic­service
host_name centos
service_description extend: / Partition
check_command check_snmp!­C public ­o .
1.3.6.1.4.1.8072.1.3.2.4.1.2.7.100.102.99.104.101.99.107.2
}

41. 2013 41
XI Service Check: Results

42. 2013 42
SNMP: Event Handler

43. 2013 43
sudo: Event Handler
extend http_event “/bin/sh /usr/local/nagios/libexec/http_event”
sudo Permissions for nagios User
User_Alias NAGIOS = nagios, nagiocmd
Cmnd_Alias NAGIOSCOM = /sbin/service, /etc/rc.d/init.d/httpd, /usr/sbin/httpd
Defaults:NAGIOS !requiretty
NAGIOS ALL=(ALL) NOPASSWD: NAGIOSCOM
Test sudo commands
su ­ nagios
sudo /sbin/service httpd restart

44. 2013 44
ACLs: Logs
Using ACLs
getfacl /var/log/messages
# file: var/log/messages
# owner: root
# group: root
user::rw­
group::­­­
other::­­­
setfacl ­m u:nagios:r /var/log/messages
# file: var/log/messages
# owner: root
# group: root
user::rw­
user:nagios:r­­
group::­­­
mask::r­­
Other::­­­
­rw­r­­­­­+ 1 root root 464963 Oct 2 06:51 /var/log/messages

45. 2013 45
Example: Event Handler
List the OID to Execute
snmpwalk ­v2c 192.168.5.190 ­c public 1.3.6.1.4.1.8072.1.3.2.3.1.4
NET­SNMP­EXTEND­MIB::nsExtendResult."http_event" = INTEGER: 0
Retrieve the Numerical Values
snmpwalk ­v2c 192.168.5.190 ­c public 1.3.6.1.4.1.8072.1.3.2.3.1.4 ­On
.1.3.6.1.4.1.8072.1.3.2.3.1.4.10.104.116.116.112.95.101.118.101.110.116 = INTEGER: 0
SNMP Executes the script when the extend name (http_event) is called

46. 2013 46
Example: Event Handler
Create a Script on Nagios to Monitor and Execute
#!/bin/sh
host=$1
string="NAGIOS TEST PAGE"
output=$(/usr/local/nagios/libexec/check_http $host ­s $string|grep "HTTP OK HTTP/1.1
200 OK "|wc ­l)
if [ $output ­eq 1 ]
then
echo "Web Page is OK"
exit 0
else
echo "Web Page is Being Restarted"
snmpwalk ­v2c $host ­c public .
1.3.6.1.4.1.8072.1.3.2.3.1.4.10.104.116.116.112.95.101.118.101.110.116
exit 2
fi

47. 2013 47
Event Handler: Service Check

48. 2013 48
Event Handler: Service Check

49. Questions?Questions?