Nagios Conference 2011 - David Thomas - Know Its Broke Before Your Customers Do

Know it’s Broke Before Your Customers Do. David Thomas [email_address]

Acknowledgements
Nagios World Conference
Ethan Galstad
And of Course all of you!
2011

A little about me, and what I do. David Thomas Director of Information Technology and Emergency Communications Sandoval County, State of New Mexico Over 20 years in the IT field, certified ITIL, Six Sigma, HDI and a bunch of other stuff.

Sandoval County
Sandoval County a pretty unique place.
2011
The county has a total area of 3,714 square miles.
5 square miles of water.
Population of 131, 561
12 Indian reservations (pueblos), The second most of any county in the United States.
Bandelier National Monument
Cibola National Forest
Tent Rocks National Monument
Santa Fe National Forest
Valles Caldera National Preserve
5 National Protected Areas The county is also flanked by two national laboratories
Sandia National laboratory to the south
Los Alamos National laboratory to the north
Numerous nuclear waste and weapons transportation routes run through the Sandoval County as well.

Sandoval County
A Unique place with Unique Issues:
Day-to-day operations
Over 70 Buildings throughout the 3,714 square miles of Sandoval County
County Sherriff facilities including 60 sheriff vehicles (PC and radio functions)
County Fire including 24 Fire Stations 80 vehicles (PC and radio functions)
County Detention Center 500+ inmate facility
County dispatch center
And all the other functions of county government; Treasure, Assessor, public works, ext.
County Heath Clinic including dental, dietary, indigent populations operation
Tribal governments of the 12 pueblos
2011

Sandoval County Customers
So this presentation is called
So Who we talking about?
EVERYONE!
Population of 131, 561 our constituents
All County Government
All Municipality Governments
All 12 Indian Pueblos and their governments
2011 Know it’s Broke Before Your Customers Do.

Sandoval County Network 2011 Pretty typical midsize network

Here is where it get’s interesting! 2011 Talk about terrain we start on a hill shoot across the river valley back over the river valley up to a mountain peak to get to Cuba, NM 75 miles from Bernalillo, NM where we started. Not much of a chose if we are to get high speed networks to the far reaches of the county.

So Much for the background.
Now for something completely different.
It’s still about customers!
And in this case when it broke it broke bad!
2011

Real emergencies in need of real solutions
In 2011 Sandoval County had to activate it’s Emergency Operations Center Twice.
February 2011 – Regional wide natural gas outage during a record breaking -40 degree temperature outbreak
June & July 2011 – Los Conches Fire - Largest wildfire fire in New Mexico History
2011 Nagios played a pivotal role in the successful emergency operations of Sandoval County during these events!

Regional wide natural gas outage
What happened:
2011 During a record breaking cold spell demand was such that gas companies made the decision to cut gas to less populated areas so urban areas would not be affected. The result was a sudden lost of gas for many areas across the desert southwest. What did Sandoval County Information Technology and Emergency Communications do?
Activation of County Emergency Operations Center and information center
Code Red emergency notification to citizens activated
24 hour information line set up manned by IT
Ham radio coordination with county shelters and red cross
Updated information on County web sites 24/7
FEMA and NIMS forms compliance
Operations lasted 120 hours

Los Conches Fire
What happened:
2011 Down power line turns into largest wildfire in New Mexico history. What did Sandoval County Information Technology and Emergency Communications do?
Activation of County Emergency Operations Center and information center
Code Red emergency notification to citizens activated
Operations lasted 336 hours
Mobile emergency radio communications and command unit activated IT manned
COM/L activation on site communication coordination
Updated information on County web sites 24/7
FEMA and NIMS forms compliance
Massive evacuations including the city of Los Alamos and LANL
Ham radio coordination with county shelters and red cross
Monsoon rains now major flooding.

Smoke plumes from my backyard. 2011 This image showing the Las Conchas wildfire raging in the Jemez Mountains of the Santa Fe National Forest in north-central New Mexico the was taken by a crew member aboard the ISS on June 27 Residences: 63 destroyed Outbuildings: 49 destroyed; 2 damaged Injuries to Date: 15 Size: 156,593 acres

The Role Nagios Played and it’s a big role
In both situations when the County EOC had to be activated Nagios turned into a very useful tool.
2011
Nagios was constantly being monitored by EOC staff
We have remote devices in remote locations that where in the fire
All fire vehicles and sheriffs Vehicles could be monitored by VPN’ed laptops
Thanks to IP based radios and repeaters those were monitored too
Generator power was monitored on remote microwave shelters
Mobile command truck could monitor and we could monitor it from the EOC
Don’t forget standard uses for Nagios even more important in an emergency
All our front line people have access to view including dispatch center
All of this has to be documented for FEMA, and best of all it can be used as your change management database for ITIL functions. Nagois has all that built in.
We use Cacti too!

Nagios played a vital roll: 2011 Know it’s Broke Before Your Customers Do. Or If it’s frozen or burning down Use advanced monitoring tools to help Save lives and property. David Thomas [email_address] Nagios played a vital roll and will be a tool we use time and again in emergency situations and the day-to-day operations of Sandoval County, NM

Nagios Conference 2011 - David Thomas - Know Its Broke Before Your Customers Do

by Nagios

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 33

Statistics

Nagios Conference 2011 - David Thomas - Know Its Broke Before Your Customers Do Presentation Transcript