Your SlideShare is downloading. ×
LU - ACUTA 2009 Conference Presentation
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

LU - ACUTA 2009 Conference Presentation


Published on

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Lessons Learned From the Texas Gulf Coast ACUTA 2009 Palm Springs, California
  • 2. About Your SpeakersTexas State University System Dr. Michael E. Dobe, Sr., Associate Vice President, Chief Information Officer & Adjunct Professor at Lamar University (Beaumont, Texas)Texas A & M System Mr. Steven Conway, Director of Computing and Info Services at Texas A&M University at Galveston (Galveston, Texas)
  • 3. About the Checklist Steven ConwayTexas A&M University at Galveston (TAMUG)
  • 4. ACUTA/ATIS Hurricane Preparedness ProjectFormed Committee May 2007Reviewed and Modified ATIS documentPublished ACUTA Hurricane Checklist Oct 2007
  • 5. Hurricane Preparedness Checklist Hurricane Katrina, August 2005, Terra MODIS True Color Image Courtesy of Louisiana State UniversityACUTA - The Association for Information Communications Technology Professionals in Higher Education Developed in cooperation with ATIS – The Alliance for Telecommunications Industry Solutions
  • 6. Committee Amy Phillips, Louisiana State University Health Sciences Center Frank OQuinn, Louisiana State University Sandra Thompson, University of West Florida James Shea, Boston University Jeanne Spinosa, Johnson & Wales University Garret Yoshimi, University of Hawaii George Denbow, University of Texas at Austin Michele Narcavage, University of Pennsylvania Steve Ellis, University of Notre Dame Steve Conway, Texas A&M University at Galveston Walt Magnussen, Texas A&M University - Task Force Chair Veronica Lancaster, ATIS Brad Ruark, ATIS /AT&T Jeri Semer, ACUTA Executive Director Aaron Fuehrer, ACUTA Staff Liaison
  • 7. ACUTA Hurricane Preparedness Checklist
  • 8. Using the Checklist As An After-Action Review Tool Perspectives From Beaumont Dr. Michael Dobe, Sr. Lamar University
  • 9. About Lamar UniversityLocated in Beaumont, TX (Approx 40 Minutes from Louisiana Border)Central Data Center in Beaumont Serves Three Campuses: Lamar University Lamar Institute of Technology Lamar State College- OrangeTotal Students Served – Approximately 16,000
  • 10. Proximity of Beaumont and Galveston119 mi – about 1 hour 57 mins
  • 11. Rita, Edouard, Gustav, Ike Hurricane Activity in SE TexasRita – September 2005 (CAT 4 With Major Damage to Beaumont)2006 and 2007 Atlantic Hurricane Seasons Quiet in SE TexasEdouard – August 2008 (Missed SE Texas)Gustav – September 1, 2008 (Minor Impact in SE Texas)Ike Hit SE Texas Less Than Two Weeks After Gustav…
  • 12. Hurricane Ike’s Path
  • 13. Hurricane Ike TimelineLamar evacuated campus 10 Sept 2008Hurricane Ike landfall Evening of 12-13 Sept 2008Low Lying Areas of Beaumont Flooded, Extensive Flooding in Orange County, Bridge City Hardest HitLamar University Campus in Beaumont Did Not FloodLibrary, Family and Consumer Sciences, and Montaigne Center Buildings Had Roof Damage24 Sept 2008 Essential Staff Returned to Campus25 Sept 2008 Classes Resumed at Lamar University
  • 14. Assessing Services During IkeLU Lessons LearnedStorm Facts InventoryContact Lists Physical AssetsEmergency Sites and Lodging Power and FuelAdministration, Network Service Communications, and Main Computer Center and Manpower Key TelecommunicationsSafety, Security and Training Hubs Long-Term Preparations Post-Storm Activities Damage Assessment and Post-Storm Activities
  • 15. Storm FactsNOAA WebsiteMajor News Media: CNN, Weather Channel and Storm Pulse (Importance of Local News Media: KFDM and EntergyGovernment Contacts: Jefferson County and FEMALocal Contacts: State Police and Beaumont Police
  • 16. Contact ListsUpdated and Distributed Internal Contacts For Gustav Leadership Team Phone List Pocket Card for Satellite Phone Numbers ITS Contact ListExternal Contact Lists Maintained By Staff Sections
  • 17. Emergency Sites and LodgingEmergency Control Site Remained on Campus in BeaumontTemporary Lodging and Shelter Arranged by IndividualsDisaster Recovery/Business Continuity Site at San Marcos State University
  • 18. Texas State University System Institutions (East Texas Locations in Bold)  Lamar University  Sam Houston State University  Sul Ross State University  Sul Ross State University Rio Grande College  Texas State University-San Marcos  Lamar Institute of Technology  Lamar State College-Orange  Lamar State College-Port Arthur
  • 19. Sam Houston State in Hunstville153 mi – about 2 hours 29 minsTOO CLOSE FOR COMFORT
  • 20. Stephen F. Austin State128 mi – about 2 hours 36NOT MEMBER OF TSUS
  • 21. San Marcos State University251 mi – about 4 hours 15 minsDuring Ike – Approx. 16 hrsLAMAR UNIVERSITY BC Site
  • 22. Administration, Communications andManpowerImportance of Clear Chain of Command and Clear Delineation of ResponsibilitiesITS Set Up Two Teams: Offsite B/C Team and Onsite Command Center TeamCommunication with Employees via LU Website and ConnectED (Now Blackboard Connect) to E-Mail and Phones.Communications Methods: Cell Phones, Satellite Phones, Land Lines and SIP Phones for Key Employees
  • 23. ACUTA NOTE: Effective Leadership Was Key to Success
  • 24. ACUTA Note: Importance of Coordination of Operations Functions
  • 25. ACUTA NOTE:Importance ofCoordination of theAcademic Response
  • 26. Administration, Communications andManpowerITS Hunkered Down With LUPD in Beaumont  Daily Meetings for Coordination in Beaumont on Campus  Reporting and Notification Lead By President and Leadership Team  Conference Bridges For Coordination With Remote Staff Managed by Network Services  Public Coordination Handled by LU PDManaged Through Office of VP for Finance and Amin  Supplemental Personnel  Accountability and Time Reporting
  • 27. LU IT Services Staff DispositionLocal Core Team Stayed on Campus ThroughoutRemote Core Team Relocated to San Marcos Area – Worked on Ordering Replacement SystemsRemaining Staff Dispersed With FamiliesAT&T Conference Bridge to Maintain Contact With Dispersed Staff
  • 28. Onsite IT Service Lead: John Genuardi, Customer Support Director
  • 29. Onsite Network Support: Patrick Stewart, Manager of Network Services
  • 30. Onsite Data Center Lead: Srinivas “Sri” Vandaraj, Senior Security Analyst
  • 31. Safety, Security and TrainingPresident’s Weekly Security Meeting Initiated as Outcome of RitaSystem Access Managed By Onsite Core ITS Team (Issues With Passwords: Need One-Time Password System)Physical Re-Entry Credentials Provided by Chief of PoliceITS and PD Coordinated  Security Cameras Shutdown and Start-up  Training for ConnectedED
  • 32. Inventory and Physical AssetsImpact of Key Staff ResignationPlanning: AVP (IT Infrastructure and Security) Responsible for Availability of Emergency Supplies and Work SuppliesAVP Had Resigned Between Gustav and Ike!Single Point of Staff Failure – Transitioned Responsibilities Quickly …Equipment Inventory, To Include Network and Data Center Backup and Recovery Plans Needed to Be Transitioned ON THE FLY! (New Role of Onsite Lead)Note on Vehicle Planning: Golf Carts Proved Critical After Storm
  • 33. Power and Fuel: Handled ByFacilities ManagementStaff Continuity Maintained - Gustav to IkeCommunication with Local Fuel VendorsCommunication and Coordination With Power CompaniesEngine/Genset (portable and permanent)Fuel TanksBatteriesRectifiersStaging and Deployment
  • 34. Network ServicePreventative Measures:  No Redundant Routes for Network (Big Exposure)  LEARN Connectivity Initiative Q 1 of 2009  Half-Completed VoIP Migration (Half Cisco/Half Avaya)Priority Areas:  Network Services Spread Between Windows Services Group (DHCP) and Unix Group (DNS).  Failure to Standardize Led to Confusion and LAN Service Outages During the Storm (Premature AD Shutdown).
  • 35. Significant Telecomm EventsGOT LUCKY -- LU OC-3 Circuit to Houston Stayed Up Throughout Event – NO INTERNET CONNECTIVITY OUTAGE (Gigaman to Orange Also Stayed Up)Onsite Team Kept Campus Services Running Until Night of Landfall (Sept. 12-13)Throughout Storm VoIP access via WAN using SIP Phones (Ordered More Broadband Cards After Storm)Cell Service Unreliable Beaumont to Remote SitesResidential lines out for over several weeksSatellite Phones Proved Difficult to Use
  • 36. Main Computer Center and KeyTelecommunications HubsSuccess Story: No Systems Shutdown Sequence in Place For Data Centers. Planning Initiated in Gustav Continued Into Ike Lead UpDownside to Not Running Your Own Power: Power Failure Exposure Due to Unmet Power Requirements in Data Center Concerns re: Lack of Fuel Led Local Team to Bring Down Systems After EventTape Backups: Contract for Off Site Had Just Lapsed! Can You Sing “Do the Hustle”?Wind and High Water: Failure to Stock Pile Sandbags
  • 37. Long Term PreparationsMaintain Year-Round  Example of Successful Planning: Replication of Website and Content Management Service  Criticality of ERP Disaster Recovery/BC Site at San Marcos  Hardware In Place in San Marcos  Software to Be Configured With Data Replication (Tested) By Hurricane Season 2009  Long-Term Plans/Prevention  Move to the Cloud: Portal and Mail Outsourcing Initiatives  Importance of Service Continuity: Redesign of Network Services (Standardization Under Network Services Group)
  • 38. Post-Storm Activities Led ByOnsite Team LeadEmployee SafetyContactsConference BridgesCostsTemporary Lodging/ShelterExtra Work ForceSafetyEmergency Supply AvailabilityPower and Fuel
  • 39. Flooding in Low-Lying Areas
  • 40. Damage to People’s Homes
  • 41. Damage to Local Businesses
  • 42. Damage Assessment and ServiceRestorationAnnouncement Via ConnectED on Return of Staff to Campus (September 24th), Return of Students to Campus (September 25th)Class Schedule Readjusted by Office of the Provost, With Resumption of Classes on September 25thPresident’s Faculty/Staff Convocation on September 25th
  • 43. Perspectives From Galveston Steven ConwayTexas A&M University at Galveston (TAMUG)
  • 44. Hurricane IkeTAMUG evacuated campus 10 Sept 2008Hurricane Ike landfall 12 Sept 200875% of Galveston was floodedTAMUG did not flood, 2 buildings had roof damageTAMUG restarted classes at TAMU in College Station TX Wed 24 Sept91% of the students returned20 Jan 2008 Classes resume in Galveston
  • 45. Significant Telecomm EventsCell service stayed up throughout storm-Cat 2UTMB Gigaman Circuit to Houston stayed upTAMUG-UTMB Gigaman available 24 Sept when power returned on campusT1’s out for over a monthResidential lines out for over 6 weeksVoIP access to College Station via WAN
  • 46. Hurricane Ike Flooding Galveston A house burns uncontrolled in a flooded neighborhood as Hurricane Ike approaches the Texas coast, Friday, Sept. 12, 2008, in Galveston, Texas. (AP Photo/Matt Slocum)
  • 47. TAMUG Boat on Causeway