Slides for the March 2018 pfSense Hangout video

1. Monitoring pfSense 2.4
with SNMP
March 2018 Hangout
Jim Pingle

2. About this Hangout
● Project News
● What is SNMP?
● What is an MIB?
● Network Monitoring Systems
● SNMP Security Concerns
● SNMP support in pfSense
● bsnmpd vs NET-SNMP
● Configuring bsnmpd
● NET-SNMP Basics
● NET-SNMP Users (USM)
● NET-SNMP TLS
● NET-SNMP Custom
Commands

3. Project News
● 2.4.3-RELEASE is out
– Security, bug fixes, a few new features
– Kernel PTI mitigations for Meltdown
– IBRS mitigation for Spectre V2 (requires updated CPU microcode)
– https://www.netgate.com/blog/pfsense-2-4-3-release-now-available.html
– https://doc.pfsense.org/index.php/2.4.3_New_Features_and_Changes
● XG-7100 1U device shipping in approximately two weeks
● ACME v2 is live
– The latest version of the ACME package can be used to generate wildcard certificates using the new
ACMEv2 production servers
– Requires a DNS update method
– Info on the forum and pfSense subreddit, doc updates coming soon
● Hangouts older than 12 months will now appear on the Netgate YouTube channel
– https://www.youtube.com/c/NetgateOfficial

4. What is SNMP?
●
Simple Network Management Protocol
●
Used to retrieve, and in some cases set, device data for network management
●
Most common use is for monitoring
– Interface traffic, errors, CPU usage, memory usage, disk usage, many other stats
●
Stateless protocol with no concept of sessions
– Each query and reply is handled in an isolated manner
●
Common ports include:
– UDP/161
– UDP/162 for traps
– TCP/10161 for TLS
– UDP/10161 for DTLS
– UDP is generally the best protocol to minimize overhead and increase efficiency, but TCP can be more reliable, especially for TLS
●
Data is organized in a hierarchy of numeric object identifiers (OIDs)
– Example: The table of interface information is at .1.3.6.1.2.1.2.2.1.1

5. What is SNMP?
●
Three common versions:
– SNMP v1
●
Poor security, community-based authentication which is essentially a password sent in the clear
– SNMP v2c
●
Improved, but incompatible, version of v1 with speed and efficiency improvements
●
v2c continues using the community-based authentication of v1, which is not ideal
– Some implementations (e.g. some Cisco devices) MD5 hashing of the community
– SNMP v3
●
Essentially v2c with added security
●
Verifies the identity of its peers to control access
●
Authentication via…
– User-based security (User Security Model, USM)
– Transport-based security (Transport Security Model, TSM) utilizing TLS (certificates)
●
Privacy via encrypted communications with either USM or TSM
●
Integrity checking to ensure the packets have not been modified

6. What is an MIB?
● Management Information Base
● Defines a model for data in SNMP, including names for items, data types, and in some
cases how to interpret responses
● Also allows SNMP software to translate these names to numeric OIDs, which is much
more user friendly (like DNS)
– What is easier to remember “.1.3.6.1.4.1.2021.10.1.100.1” or “UCD-SNMP-MIB::laErrorFlag”
which can be shortened to “laErrorFlag”?
– See accompanying mib-example.txt for a break-down of how an MIB helps SNMP software
translate names to an OID number and lets SNMP interpret results
● MIBs for pfSense/FreeBSD can be found in /usr/share/snmp/mibs/ and these can be
added to a Network Monitoring System as needed, but the PF-specific pieces only work
with bsnmpd at this time

7. Network Monitoring Systems
● A Network Monitoring System (NMS) is a piece of software that collects data from
devices and performs actions using that data
● Typical capabilities include graphing and alerting/notifications, among others
● A wide variety of NMS software exists
– Some free, some commercial
– Zabbix, Cacti, Nagios, Icinga, Check MK, LibreNMS, Pandora FMS, OpenNMS, Zenoss,
PRTG, many others
– https://en.wikipedia.org/wiki/Comparison_of_network_monitoring_systems
● Research the capabilities of the software to ensure it meets your requirements
– Ex: If you want to use SNMPv3, check if the software supports SNMPv3 user auth and/or TLS,
it may not support SNMPv3 at all, and may not support TLS for instance

8. SNMP Security Concerns
● SNMP exposes a HUGE amount of data about your device, so it can be dangerous!
● Never run SNMP over the Internet or any untrusted network!
– SNMP v1 and v2c both transmit the community and other data in the clear
– SNMPv3 may have encrypted transport capabilities but the service itself is still weak in some ways (e.g.
brute forcing USM), so it is best not to expose it to the Internet if possible
● Use multiple layers and tactics to protect the service and the data
– Firewall rules to control access to the daemon
– Encrypted transport such as a VPN, TLS, or at least an isolated management network
– Access controls in the SNMP daemon, if available
– A complex/long community name if forced to use SNMPv1 or v2c
– Strong passwords and encryption keys for SNMPv3 USM
– Strong certificates (e.g. Key length >= 2048, SHA256) for SNMPv3 TSM

9. SNMP support in pfSense
●
Two ways to use SNMP in pfSense:
– The built-in lightweight bsnmp daemon
– The more capable, but heavier, NET-SNMP package
●
The bsnmpd package can provide some pf specific data that the NET-SNMP package cannot, but it is not widely
used
– State table size, table info, ALTQ info, packet processing stats, and more
– See /usr/share/snmp/mibs/BEGEMOT-PF-MIB.txt on the firewall for details
– Alternately, snmpwalk through .1.3.6.1.4.1.12325.1.200.1 or BEGEMOT-PF-MIB::begemotPfObjects to see the entire set of
data
●
Otherwise, standard UNIX and/or FreeBSD system templates apply, depending on the NMS
●
Note that some common requested items cannot be obtained via SNMP out of the box, such as:
– HA/CARP status, gateway status, VPN status, captive portal stats, and other pfSense-specific data
– These can be accommodated via NET-SNMP extend scripts in most cases, covered later
– Per-IP-address throughput cannot be obtained via SNMP at all, use netflow instead

10. bsnmpd vs NET-SNMP
● bsnmpd is built-in, NET-SNMP is available as an add-on package
● bsnmpd
– Small/light binary that consumes little memory and CPU
– Supports SNMPv1 and v2c on pfSense
– Community-based authentication only
– No transport encryption or integrity checking
– Supports IPv4 UDP
– Port can be customized but defaults to 161
– Can be bound to one or more interfaces or VIPs
– No viable way to execute custom commands in the pfSense GUI
– Has been known to have stability issues in the past on certain platforms
●
All known issues have been resolved, but some may still exist
●
Consider using the Service Watchdog package to monitor the bsnmpd daemon

11. bsnmpd vs NET-SNMP
●
NET-SNMP Package
– Consumes more memory/CPU/resources than bsnmpd, but has many more features
– Supports SNMPv1, v2c, and v3
– TCP and UDP
– IPv4 and IPv6
– Custom service and port bindings, can use multiple ports and protocols at the same time
– User-based authentication (USM) with privacy protection (encryption)
– Transport-based authentication (TSM) with TLS/certificates
– Community access for v1/v2c can be limited
●
Multiple communities can be made with different access settings
●
Access can be by controlled source address
●
A Base OID can be set to limit what a community can see
– SNMP over TLS over TCP support
●
The package also has SNMP over DTLS over UDP support, but some operating systems have problems using it, so use TCP/TLS instead
– Custom shell commands can be used to return data over SNMP via “extend” entries
– Can generate and receive SNMP traps, but we won’t have time to cover that today
– Can set bulk request limits to prevent runaway queries from causing a DoS

12. Configuring bsnmpd
● Services > SNMP
● Check enable to turn on the SNMP service
● The polling port should probably be left at 161, the default
● The system location and contact are free text fields that can help
identify this device automatically via SNMP
● The Read Community String is required and should be set to a
strong value so it is not easily guessed
● Traps are optional, not covered today due to time constraints

13. Configuring bsnmpd
● Modules are optional but provide useful additional information
– Each module loaded will cause bsnmpd to use a little more memory and cpu to handle requests
– MIB II
●
Covers networking information and interfaces
●
Status, hardware, IP addresses, the amount of data transmitted and received, & more
– Netgraph
●
Netgraph node names and statuses, hook peers, and errors
– PF
●
PF-related information, details were mentioned on a previous slide
– Host Resources (requires MIB II)
● Information about the host itself
●
Uptime, load average, processes, storage types/usage, system devices, installed software, & more
– UCD (UC Davis MIB)
● Memory usage, disk usage, running programs, and more
– Regex (not used)
●
Not hooked into the GUI yet, but can creating SNMP counters from logs or other text files

14. Configuring bsnmpd
● Interface binding sets how bsnmpd listens on interfaces or virtual IP addresses
– When set to ‘all’, the SNMP daemon will respond from the closest IP address to the
destination
● This may not match where the request was sent, which can cause communication issues with an NMS
– Avoid binding to ‘all’ or to a WAN if possible for security reasons
– When using SNMP over a VPN, bind to an address that can communicate across the VPN
● Ex: LAN subnet inside an IPsec P2, bind only to LAN, send queries to LAN
●
Allow access to the SNMP daemon in firewall rules on your local management
interface, VPN, or however the daemon will be reached
● Click Save and that’s all!

15. NET-SNMP Basics
● Disable bsnmpd if it is enabled (Services > SNMP)
● Install the NET-SNMP package (2.4 and above only)
● Visit Services > SNMP (NET-SNMP)
● The NET-SNMP GUI has a wealth of help info and examples, click the (i) in various places to see detailed info and
examples for snmpwalk/snmpget command usage, configuration files, and more
● Check enable
● Enter Interface Binding entries
– If none are entered, the default of udp/161 will be used
– If one or more are entered, keep a manual entry for udp/161 to retain that binding
– To use TLS, add an entry set to TLS/TCP with a port of 10161
– For IPv6, set an entry for UDP6 or TCP6
– Enter an IP address to bind to a specific address or leave blank to bind to any/all
– Allow access to the SNMP daemon in firewall rules on your local management interface, VPN, or however the daemon will be
reached

16. NET-SNMP Basics
● Identification
– Most commonly, the engine ID is left as “Firewall IPv4 Address” but it can be an arbitrary string as well
– For TLS
● Set an appropriate CA, use “self-signed” if the server certificate is self-signed, or leave at “None” if TLS will not be active
● Set an appropriate server certificate, or leave at “None” if TLS will not be active
● Best practice is to use a CA structure similar to openvpn (CA created on pfSense, server and clients signed by that CA)
– Use Transport Prefix: Leave unchecked unless you know it’s necessary
● Rate Limits
– Usually left at 0/0 but can be set as needed
– Careful not to set too low or an NMS may fail to retrieve needed data
– “too low” is all relative to the NMS and its typical requests, consult NMS docs for details or analyze traffic captures of
its requests
● Advanced options can pass through custom directives to snmpd.conf to enable features not present in the
package GUI

17. NET-SNMP Basics
● Host Information tab
●
General Info
– Free text fields to provide information which uniquely identifies this device
● Interface Handling
– Provides some tweaks that can help handle dynamic interfaces that appear and disappear (ppp,
tun, etc)
– Generally left at the default but can be raised higher if interface index values change too often
●
Disk Handling
– Percentage of disk space on a partition at which point the dskErrorFlag value is true
● Load Monitoring
– Load average threshold values above which laErrorFlag will be true

18. NET-SNMP Custom Commands
● Host Monitoring tab, Extended Command section
● Runs a shell command and returns the output via SNMP
● Can be used with scripts to retrieve status values not available directly in
SNMP
● Entries are indexed by name, for convenience, rather than relying on a
specific order
● Use snmpwalk against nsExtendOutput1 to get output from all commands
● To reference a single value by name, use
nsExtendOutputFull.”commandname”

19. NET-SNMP Custom Commands
●
Each entry has several values to customize:
– Type
●
The type of extend command, either extend or extendfix, only covering extend today
– MIB OID
●
The OID where the extend command will be rooted, leave blank for the default
●
This does not override the name indexing, only changes the root
– Name
●
The short name of the command used to retrieve the output
– Program
●
The full path to the program binary or script that will be called
●
Note: You cannot use pipes to exec multiple programs, use a shell script instead if that is necessary
– Arguments
●
Command line arguments passed to the Program
● These arguments are all passed to Program, and you cannot use pipes, redirects, etc.

20. NET-SNMP Custom Commands
● Example: Retrieve current pf state table usage
– Extend, <blank>, curstates, /root/snmp_curstates.sh, <blank>
– Script contains:
#!/bin/sh
/sbin/pfctl -si | /usr/bin/grep -A1 'State Table' | /usr/bin/tail -1 | /usr/bin/awk '{print $3;}'
– Make sure script is executable (+x)
– Output:
$ snmpget bill nsExtendOutputFull."curstates"
NET-SNMP-EXTEND-MIB::nsExtendOutputFull."curstates" = STRING: 105

21. NET-SNMP Communities
● Communities are used for SNMPv1 and v2c
● Communities tab, click + to add entry
● Use a strong/long name for the community
● Set Read Only or Read/Write
– Usually only Read access is necessary, and it is more secure
● Address Family controls the AF for this community
– Pick either IPv4 or IPv6
– Or select “Both” to allow the community to work over IPv4 or IPv6
● Source Restriction is an IP address or subnet from which this community can be accessed
– Leave blank to allow from anywhere
– Enter “default” if you need to set a Base OID without a source restriction
● Base OID sets the root of the OID tree under which this community can read data

22. NET-SNMP Users (USM)
● Users tab
● There will be a default manager user with a randomly generated password and passphrase
– This is for internal use by the SNMP daemon and should be left alone
– You can change the password and passphrase but the default randomized entries are already strong!
● Click + To add a new entry
● Enter a username, e.g. myuser
● Set the entry type to “User Entry (USM)” or “Both” to allow user authentication
● The description is optional and may be used to note the purpose of the account
● Access Control
– Can restrict the user to read only, or allow read/write, and can set a base OID to restrict access to data

23. NET-SNMP Users (USM)
● USM User Configuration
– Values used here may depend on your NMS and what it supports!
– Authentication Type: Hash to use against the password
● SHA or MD5, preferably SHA
– Password: The password for this user
● Minimum of 8 characters, preferably much longer!
– Privacy Protocol: Encryption to use to protect data transmission
●
Can be AES, DES, or None, preferably AES
● May not be supported by all SNMP clients/NMS software
– Passphrase: Encryption Key used with the Privacy Protocol
●
As random and long as possible!
– Minumum USM Security Level
●
The lowest level to allow for this user
●
Private is the best, since it requires authentication and encryption

24. NET-SNMP TLS (TSM)
●
Similar to USM, but fill in the Certificate Mapping Information
●
Create or import a user certificate in the user manager first (System > Cert Manager)
●
Set Entry Type to “Certificate Mapping (TSM)”, or “Both”
●
Priority
– Used for choosing a match when there are multiple mappings for the same certificate but with different fields
– Required, but normally left at a simple value like 10 and not used since multiple mappings are rarely necessary
●
User Certificate
– The certificate for this user in the Certificate Manager
●
Certificate Field
– The field of the certificate which matches the username for this user
– Typically set to Common Name, since most people set the username as the common name for user certificates
– Can also be set to a variety of other SAN types (email, IP address, hostname), or set to match ANY SAN in the certificate
●
Security Name
– Used with the “Manual Security Name “ setting of Certificate Field to manually set the username to match
●
Minimum TSM Security Level
– Lowest level at which this user can operate
– When configured with TSM the user is only matched when the certificate is valid, so ‘Private’ makes the most sense, since that is the only effective choice

25. Conclusion
● Questions?
● Ideas for hangout topics? Post on forum, comment on the blog
posts, Reddit, etc