Building a Large Scale SolarWinds Installation

Building a Large-Scale SolarWinds® Installation
With Leon Adato, Monitoring Architect at Cardinal Health®
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

Enterprise – Bigger Than a Breadbox
Leon’s not-so-humble-opinion:
» Over 2,000 devices
» More than 20 locations
» 10,000 elements

Most Projects Fail at the Top
The Real OSI Model
Politics
Legal/Compliance
Finance
Application
Presentation
Session
Transport
Network
Data© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

What I really want…
» A complete, accurate, meaningful inventory.
» Network connectivity to all devices part 1: ping
» Network connectivity part 2: Firewall
» A domain or local account that has permission to pull WMI/RPC
» SNMP enabled with a known RO string on all devices
– And your polling engines PERMITTED to SNMP poll

SNMP WTF?

ALERTS: The Four Questions
» What will you do when you get this alert?
» Do you have a documented procedure?
» Can you make this problem happen on purpose (for testing)
» What will happen (what is the cost) if you don’t get this alert?

Standard Enterprise Design

Kick It Up a Notch

Portocalypse Now*
For SNMP and RPC, you will need:
» ICMP bi-directional
» TCP 135
» TCP 139
» TCP 445
» TCP 161
» UDP 137
» UDP 138
» For WMI? All ports over 1024, or go read “WMI Portocalypse” on thwack
*Tip of the Thwack-cap to aLTeReGo

DMZ? Make sure these are open
» ICMP bi-directionally
– And that DNS works in both directions
» TCP 17777
» TCP 17778
» TCP 80 (or whatever your SolarWinds portal runs on)
» TCP 1433 (or whatever your database listens on)
– Named instances will not work here.

Account Permissions
» RPC – local or domain user that is a member of:
– Performance Monitor Users group
– Distributed COM users group
– And has the “Allow logon locally” permission
– (NOT a member of allow login through remote desktop)
» WMI – local or domain user that is a member of:
– Performance Monitor Users group
– Distributed COM users group
– Remote desktop users group
– Has “Allow logon locally” permission
– Has been added to the “root” and “CIMv2” branches of the WMI settings.
» Or just give it local admin already!

Security Hates Me*
So you can’t get them to permit WMI
» Most SAM items that use WMI can be obtained via RPC
– Most processes can be monitored via SNMP with CPU and RAM stats
– Windows eventlog can be gathered via RPC
– Windows process and/or services
– Perfmon has always used RPC
What you are giving up (ie: How To Make Your Case):
» Virtual Mount Points
» Restarting a service as an alert action
» Realtime process/service explorer
» Windows® script monitoring (VBScript™ or PowerShell®)
» WMI Monitors (duh!)
*Or: “I have to monitor through a NAT-ed VPN”

Server Spec’s
» Separate the Database and primary poller.
» Database is physical. “Need More Power” should be your request. Mine has 12 CPU
and 128Gb RAM.
» Pollers can be virtual. I tend to run with 8 CPU, 12Gb ram to start.
» Disk is a big deal. You want lots of spindles. You want RAID 10 or you want the SAN
team to tell you they can load your entire database into the memory of the storage
array.
» Everything has to be in the same time zone.
» Primary poller and database must be in the same location

Element Counts – The Real story
» Sure, you have elements (~10,000)*
– Ping-able IP’s
– Interfaces
– Disks
» But you also have UnDP’s
» And SAM components
» And the number of DOWN items is going to affect how much you can handle
» Upshot: There is no formula.
*Ultimate limit is ~110,000 per Solarwinds instance

Know Thy Pollers!

Processes You Have to Nail Down
» Device Lifecycle
– Who, how, where, when devices are added
– Ditto for elements
– Ditto for SAM items
» Devices (and volumes, and interfaces) missing key custom property information
» Decom devices
» Down Devices
» Devices not SNMP polling
» Applications in “unknown” status
» Duplicate Nodes
» Bad application report

Device Down Report

Applications with Problems

Handy Documents to Have on Hand
SolarWinds Run Book
» Overall design
» Accounts used for installation, connectivity (Poller to db) etc.
» Additional software installed (Perl®, ImageMagik®, PuTTY™, etc)
» How to install Windows patches
» How to install SolarWinds patches
» Common troubleshooting techniques
» Common validation/error-checking techniques
» How to open a ticket with SolarWinds (with your account ID)

Gotchas
» Event correlation
» Where does scalability end?
» No DBA? That’s gonna hurt.
» Time zone

Thank You!
The SOLARWINDS and SOLARWINDS & Design marks are the exclusive property of SolarWinds
Worldwide, LLC, are registered with the U.S. Patent and Trademark Office, and may be registered or
pending registration in other countries. All other SolarWinds trademarks, service marks, and logos
may be common law marks, registered or pending registration in the United States or in other
countries. All other trademarks mentioned herein are used for identification purposes only and
may be or are trademarks or registered trademarks of their respective companies.

Building a Large Scale SolarWinds Installation

More Related Content

What's hot

Viewers also liked

Similar to Building a Large Scale SolarWinds Installation

More from SolarWinds

Recently uploaded

Building a Large Scale SolarWinds Installation