SNMP – protocol, WMI – set of extensions to the Windows Driver Model that provides an interface which the instrumented components provide information and notification.
Cisco uses an access list to allow SNMP traffic to and from a specific host via Read only or Read-Write community strings
Time intervals – like if an threshold is tripped 10 times over 20 minutes, then produce an alert.
You can have many monitoring tools run a script, restart a service, send a notification, etc. in response to an alert. In the case of event log monitors, you can set it to perform what is called a ‘looping list’, which resets itself after every event log scan. This allows it to scan event log dates and remember the last place it scanned before running another check.
Network Monitoring Basics
• WAN links between sites• Links between core network devices• Important devices like servers and core appliances• Websites
• Drive space, CPU and memory utilization• Log files (for errors or other text)• Network utilization and bandwidth• Important services and processes• Internal or External website availability
“My Internet is slow” Measure bandwidth or CPU of firewall, outbound connections (virus?)“I can’t get any email, is the server down?” Check Exchange services, monitor outbound mail traffic per sec.“We are paying $900 per month to connect our satellite office witha high speed connection. Is it worth it?” Watch for peak bandwidth usage during day, week, month.“Everyone here can’t print. You did something, didn’t you?” Monitor spooler service, watch for errors in system log regarding printers
A good monitoring system will query a device for a specificset of statistics, retain this data and report to an appropriateadministrator if those statistics exceed an acceptablethreshold… …if a drive is 90% full, let the IT administrator know via email so they can begin to remedy the situation.
What do monitoring systems use to get their data?SNMP – Linux, Network Hardware, WindowsWMI – WindowsPerformance Counters – WindowsSSH – Linux
SNMP Usually requires MIB (management information base) files to monitor advanced system statisticsWMI Typically available by default, but highly security conscious network admins may have this locked downPerformance Counters If you can view it in Windows Perfmon, you can track it in some monitoring toolsSSHSSH Requires root access to run commands
• Monitor threshold – at what point does something trigger an alert?• Alert – When a threshold is met for a period of time, go into „Alert‟ status.• Action – Send an email, SMS, restart a service, run a script, etc.
• Historical trending and reporting• Maintenance windows• Multiple notification methods• Ability to perform action in response to an alert• NOC (Network Operation Center) view• Large variety of monitor types that support WMI, SNMP, etc.• Ability to produce alerts based a defined span of time
• Company shared drive size and availability• Ensure Exchange service and Accounting system DB is accessible after backups• Make sure outgoing Internet connection is not saturated• Keep invalid domain logon attempts at bay• Watch for system errors
• How long until something is considered an emergency?• Will the condition return to normal without your intervention?• How do you want to be notified – email, SMS, page, IM, Net Send?• Do you want the monitoring tool attempt to remedy the situation automatically?
• Configure your monitors with high thresholds while you determine what is “normal”• Watch these monitors over time to get an idea of normal peaks and valleys of performance stats• Tweak your monitors according to trending and growth patterns
1. The Death Star depends on tractor beam2. The IT Admin sets up a monitor to watch service: “tractor_beam”3. He then configures the alert to “Email” Darth Vader when the tractor beam goes down4. Obi-Wan disables the tractor beam5. 5 minutes later, the Millenium Falcon escapes6. Tractor beam is down for an additionl 5 minutes, then monitoring system sends email7. Vader is busy choking one of his employees, and has his BlackBerry set on “vibrate”… “…probably should have set the monitoring system to restart the service before Han got awa-aaacccchhdhhshhpfffft” – IT admin speaking with Darth Vader
=====================================Time: 2010/10/05 20:34:22Object: DC-ROA-01(DC-ROA-01)Monitor: Security events=====================================Status: AlarmMessage: Found matching eventlog record This is an example of an event logEvent id: 529Computer: DC-ROA-01 report when a user attempted to logSource: Security in with an invalid password.User: SYSTEMTime Generated: 2010/10/05 20:06:27Message: Logon Type: 8 means the password was passed usingLogon Failure: ClearTextReason: Unknown user name or bad passwordUser Name: email@example.com Caller Process is the PID of the executable on the serverDomain: mydomain.com processing the logon attempt.Logon Type: 8Logon Process: Advapi Source network address is the user‟s Comcast IP.Authentication Package: NegotiateWorkstation Name: DC-ROA-01Caller User Name: DC-ROA-01$Caller Domain: MYDOMAINCaller Logon ID: (0x0,0x3E7)Caller Process ID: 7708Transited Services: -Source Network Address: 220.127.116.11Source Port: 56049
It is important to be able to keep a history oftrending, especially with storage devices andservice outages. This will help determine futureneeds for backup and DR processes.You can get an idea of heavily usedvolumes/resources, allowing you to organizeplanned downtime when moving them.
Windows based• Total Network Monitorhttp://www.softinventive.com/products/total-network-monitor/• MicroTik‟s “The Dude”http://www.mikrotik.com/thedude.php• Hyperic HQ Open Sourcehttp://www.hyperic.com/products/open-source-systems-monitoring• Spotlight on Windows (realtime monitoring only)http://www.quest.com/spotlight-on-windows/ - free registration required• Splunk (logfile indexing)http://www.splunk.org• Spiceworks (general activity monitoring)http://community.spiceworks.comLinux based• Zenosshttp://www.zenoss.com/• Nagioshttp://www.nagios.org
This presentation will be available from www.ninp.org (viaSlideShare)Rob Dunn: firstname.lastname@example.org