ACUTA 2009 Conf Presentation

Lessons Learned From the Texas Gulf Coast ACUTA 2009 Palm Springs, California

About Your Speakers Texas State University System Dr. Michael E. Dobe, Sr., Associate Vice President, Chief Information Officer & Adjunct Professor at Lamar University (Beaumont, Texas) http://www.linkedin.com/in/mdobe Texas A & M System Mr. Steven Conway, Director of Computing and Info Services at Texas A&M University at Galveston (Galveston, Texas) http://www.linkedin.com/pub/5/b7/24a

About the Checklist Steven Conway Texas A&M University at Galveston (TAMUG)

ACUTA/ATIS Hurricane Preparedness Project Formed Committee May 2007 Reviewed and Modified ATIS document Published ACUTA Hurricane Checklist Oct 2007

Hurricane Preparedness Checklist Hurricane Katrina, August 2005, Terra MODIS True Color Image Courtesy of Louisiana State University ACUTA ‐ The Association for Information Communications Technology Professionals in Higher Education Developed in cooperation with ATIS – The Alliance for Telecommunications Industry Solutions http://www.acuta.org/?2026

Committee Amy Phillips, Louisiana State University Health Sciences Center Frank O'Quinn, Louisiana State University Sandra Thompson, University of West Florida James Shea, Boston University Jeanne Spinosa, Johnson & Wales University Garret Yoshimi, University of Hawaii George Denbow, University of Texas at Austin Michele Narcavage, University of Pennsylvania Steve Ellis, University of Notre Dame Steve Conway, Texas A&M University at Galveston Walt Magnussen, Texas A&M University ‐ Task Force Chair Veronica Lancaster, ATIS Brad Ruark, ATIS /AT&T Jeri Semer, ACUTA Executive Director Aaron Fuehrer, ACUTA Staff Liaison

ACUTA Hurricane Preparedness Checklist

Using the Checklist As An After‐Action Review Tool Perspectives From Beaumont Dr. Michael Dobe, Sr. Lamar University

About Lamar University Located in Beaumont, TX (Approx 40 Minutes from Louisiana Border) Central Data Center in Beaumont Serves Three Campuses: Lamar University Lamar Institute of Technology Lamar State College‐ Orange Total Students Served – Approximately 16,000

Proximity of Beaumont and Galveston 119 mi – about 1 hour 57 mins

Rita, Edouard, Gustav, Ike Hurricane Activity in SE Texas Rita – September 2005 (CAT 4 With Major Damage to Beaumont) 2006 and 2007 Atlantic Hurricane Seasons Quiet in SE Texas Edouard – August 2008 (Missed SE Texas) Gustav – September 1, 2008 (Minor Impact in SE Texas) Ike Hit SE Texas Less Than Two Weeks After Gustav…

Hurricane Ike’s Path http://en.wikipedia.org/wiki/File:Ike_2008_track.png

Hurricane Ike Timeline Lamar evacuated campus 10 Sept 2008 Hurricane Ike landfall Evening of 12‐13 Sept 2008 Low Lying Areas of Beaumont Flooded, Extensive Flooding in Orange County, Bridge City Hardest Hit Lamar University Campus in Beaumont Did Not Flood Library, Family and Consumer Sciences, and Montaigne Center Buildings Had Roof Damage 24 Sept 2008 Essential Staff Returned to Campus 25 Sept 2008 Classes Resumed at Lamar University

Assessing Services During Ike LU Lessons Learned Storm Facts Inventory Contact Lists Physical Assets Emergency Sites and Lodging Power and Fuel Administration, Network Service Communications, and Main Computer Center and Manpower Key Telecommunications Safety, Security and Training Hubs Long‐Term Preparations Post‐Storm Activities Damage Assessment and Post‐Storm Activities

Storm Facts NOAA Website Major News Media: CNN, Weather Channel and Storm Pulse (http://www.stormpulse.com/) Importance of Local News Media: KFDM and Entergy Government Contacts: Jefferson County and FEMA Local Contacts: State Police and Beaumont Police

Contact Lists Updated and Distributed Internal Contacts For Gustav Leadership Team Phone List Pocket Card for Satellite Phone Numbers ITS Contact List External Contact Lists Maintained By Staff Sections

Emergency Sites and Lodging Emergency Control Site Remained on Campus in Beaumont Temporary Lodging and Shelter Arranged by Individuals Disaster Recovery/Business Continuity Site at San Marcos State University

Texas State University System http://www.tsus.edu/ TSUS Institutions (East Texas Locations in Bold) Lamar University Sam Houston State University Sul Ross State University Sul Ross State University Rio Grande College Texas State University‐San Marcos Lamar Institute of Technology Lamar State College‐Orange Lamar State College‐Port Arthur

Sam Houston State in Hunstville 153 mi – about 2 hours 29 mins TOO CLOSE FOR COMFORT

Stephen F. Austin State 128 mi – about 2 hours 36 NOT MEMBER OF TSUS

San Marcos State University 251 mi – about 4 hours 15 mins During Ike – Approx. 16 hrs LAMAR UNIVERSITY BC Site

Administration, Communications and Manpower Importance of Clear Chain of Command and Clear Delineation of Responsibilities ITS Set Up Two Teams: Offsite B/C Team and Onsite Command Center Team Communication with Employees via LU Website and ConnectED (Now Blackboard Connect) to E‐Mail and Phones. Communications Methods: Cell Phones, Satellite Phones, Land Lines and SIP Phones for Key Employees

ACUTA NOTE: Effective Leadership Was Key to Success

ACUTA Note: Importance of Coordination of Operations Functions

ACUTA NOTE: Importance of Coordination of the Academic Response

Administration, Communications and Manpower ITS Hunkered Down With LUPD in Beaumont Daily Meetings for Coordination in Beaumont on Campus Reporting and Notification Lead By President and Leadership Team Conference Bridges For Coordination With Remote Staff Managed by Network Services Public Coordination Handled by LU PD Managed Through Office of VP for Finance and Amin Supplemental Personnel Accountability and Time Reporting Costs

LU IT Services Staff Disposition Local Core Team Stayed on Campus Throughout Remote Core Team Relocated to San Marcos Area – Worked on Ordering Replacement Systems Remaining Staff Dispersed With Families AT&T Conference Bridge to Maintain Contact With Dispersed Staff

Onsite IT Service Lead: John Genuardi, Customer Support Director

Onsite Network Support: Patrick Stewart, Manager of Network Services

Onsite Data Center Lead: Srinivas “Sri” Vandaraj, Senior Security Analyst

Safety, Security and Training President’s Weekly Security Meeting Initiated as Outcome of Rita System Access Managed By Onsite Core ITS Team (Issues With Passwords: Need One‐Time Password System) Physical Re‐Entry Credentials Provided by Chief of Police ITS and PD Coordinated Security Cameras Shutdown and Start‐up Training for ConnectedED

Inventory and Physical Assets Impact of Key Staff Resignation Planning: AVP (IT Infrastructure and Security) Responsible for Availability of Emergency Supplies and Work Supplies AVP Had Resigned Between Gustav and Ike! Single Point of Staff Failure – Transitioned Responsibilities Quickly … Equipment Inventory, To Include Network and Data Center Backup and Recovery Plans Needed to Be Transitioned ON THE FLY! (New Role of Onsite Lead) Note on Vehicle Planning: Golf Carts Proved Critical After Storm

Power and Fuel: Handled By Facilities Management Staff Continuity Maintained ‐ Gustav to Ike Communication with Local Fuel Vendors Communication and Coordination With Power Companies Engine/Genset (portable and permanent) Fuel Tanks Batteries Rectifiers Staging and Deployment

Network Service Preventative Measures: No Redundant Routes for Network (Big Exposure) LEARN Connectivity Initiative Q 1 of 2009 Half‐Completed VoIP Migration (Half Cisco/Half Avaya) Priority Areas: Network Services Spread Between Windows Services Group (DHCP) and Unix Group (DNS). Failure to Standardize Led to Confusion and LAN Service Outages During the Storm (Premature AD Shutdown). Failure to Document All Services Creates Risk

Significant Telecomm Events GOT LUCKY ‐‐ LU OC‐3 Circuit to Houston Stayed Up Throughout Event – NO INTERNET CONNECTIVITY OUTAGE (Gigaman to Orange Also Stayed Up) Onsite Team Kept Campus Services Running Until Night of Landfall (Sept. 12‐13) Throughout Storm VoIP access via WAN using SIP Phones (Ordered More Broadband Cards After Storm) Cell Service Unreliable Beaumont to Remote Sites Residential lines out for over several weeks Satellite Phones Proved Difficult to Use

Main Computer Center and Key Telecommunications Hubs Success Story: No Systems Shutdown Sequence in Place For Data Centers. Planning Initiated in Gustav Continued Into Ike Lead Up Downside to Not Running Your Own Power: Power Failure Exposure Due to Unmet Power Requirements in Data Center Concerns re: Lack of Fuel Led Local Team to Bring Down Systems After Event Tape Backups: Contract for Off Site Had Just Lapsed! Can You Sing “Do the Hustle”? Wind and High Water: Failure to Stock Pile Sandbags

Long Term Preparations Maintain Year‐Round Example of Successful Planning: Replication of Website and Content Management Service Criticality of ERP Disaster Recovery/BC Site at San Marcos Hardware In Place in San Marcos Software to Be Configured With Data Replication (Tested) By Hurricane Season 2009 Long‐Term Plans/Prevention Move to the Cloud: Portal and Mail Outsourcing Initiatives Importance of Service Continuity: Redesign of Network Services (Standardization Under Network Services Group)

Post‐Storm Activities Led By Onsite Team Lead Employee Safety Contacts Conference Bridges Costs Temporary Lodging/Shelter Extra Work Force Safety Emergency Supply Availability Power and Fuel

Flooding in Low-Lying Areas

Damage to People’s Homes

Damage to Local Businesses

Damage Assessment and Service Restoration Announcement Via ConnectED on Return of Staff to Campus (September 24th), Return of Students to Campus (September 25th) Class Schedule Readjusted by Office of the Provost, With Resumption of Classes on September 25th President’s Faculty/Staff Convocation on September 25th

Perspectives From Galveston Steven Conway Texas A&M University at Galveston (TAMUG)

Hurricane Ike TAMUG evacuated campus 10 Sept 2008 Hurricane Ike landfall 12 Sept 2008 75% of Galveston was flooded TAMUG did not flood, 2 buildings had roof damage TAMUG restarted classes at TAMU in College Station TX Wed 24 Sept 91% of the students returned 20 Jan 2008 Classes resume in Galveston

Significant Telecomm Events Cell service stayed up throughout storm‐Cat 2 UTMB Gigaman Circuit to Houston stayed up TAMUG‐UTMB Gigaman available 24 Sept when power returned on campus T1’s out for over a month Residential lines out for over 6 weeks VoIP access to College Station via WAN

Hurricane Ike Flooding Galveston A house burns uncontrolled in a flooded neighborhood as Hurricane Ike approaches the Texas coast, Friday, Sept. 12, 2008, in Galveston, Texas. (AP Photo/Matt Slocum)

TAMUG Boat on Causeway

ACUTA 2009 Conf Presentation

by Michael Dobe on Jan 24, 2009

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 3

Statistics

ACUTA 2009 Conf Presentation — Presentation Transcript