LU - ACUTA 2009 Conference Presentation


Published on

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

LU - ACUTA 2009 Conference Presentation

  1. 1. Lessons Learned From the Texas Gulf Coast ACUTA 2009 Palm Springs, California
  2. 2. About Your SpeakersTexas State University System Dr. Michael E. Dobe, Sr., Associate Vice President, Chief Information Officer & Adjunct Professor at Lamar University (Beaumont, Texas)Texas A & M System Mr. Steven Conway, Director of Computing and Info Services at Texas A&M University at Galveston (Galveston, Texas)
  3. 3. About the Checklist Steven ConwayTexas A&M University at Galveston (TAMUG)
  4. 4. ACUTA/ATIS Hurricane Preparedness ProjectFormed Committee May 2007Reviewed and Modified ATIS documentPublished ACUTA Hurricane Checklist Oct 2007
  5. 5. Hurricane Preparedness Checklist Hurricane Katrina, August 2005, Terra MODIS True Color Image Courtesy of Louisiana State UniversityACUTA - The Association for Information Communications Technology Professionals in Higher Education Developed in cooperation with ATIS – The Alliance for Telecommunications Industry Solutions
  6. 6. Committee Amy Phillips, Louisiana State University Health Sciences Center Frank OQuinn, Louisiana State University Sandra Thompson, University of West Florida James Shea, Boston University Jeanne Spinosa, Johnson & Wales University Garret Yoshimi, University of Hawaii George Denbow, University of Texas at Austin Michele Narcavage, University of Pennsylvania Steve Ellis, University of Notre Dame Steve Conway, Texas A&M University at Galveston Walt Magnussen, Texas A&M University - Task Force Chair Veronica Lancaster, ATIS Brad Ruark, ATIS /AT&T Jeri Semer, ACUTA Executive Director Aaron Fuehrer, ACUTA Staff Liaison
  7. 7. ACUTA Hurricane Preparedness Checklist
  8. 8. Using the Checklist As An After-Action Review Tool Perspectives From Beaumont Dr. Michael Dobe, Sr. Lamar University
  9. 9. About Lamar UniversityLocated in Beaumont, TX (Approx 40 Minutes from Louisiana Border)Central Data Center in Beaumont Serves Three Campuses: Lamar University Lamar Institute of Technology Lamar State College- OrangeTotal Students Served – Approximately 16,000
  10. 10. Proximity of Beaumont and Galveston119 mi – about 1 hour 57 mins
  11. 11. Rita, Edouard, Gustav, Ike Hurricane Activity in SE TexasRita – September 2005 (CAT 4 With Major Damage to Beaumont)2006 and 2007 Atlantic Hurricane Seasons Quiet in SE TexasEdouard – August 2008 (Missed SE Texas)Gustav – September 1, 2008 (Minor Impact in SE Texas)Ike Hit SE Texas Less Than Two Weeks After Gustav…
  12. 12. Hurricane Ike’s Path
  13. 13. Hurricane Ike TimelineLamar evacuated campus 10 Sept 2008Hurricane Ike landfall Evening of 12-13 Sept 2008Low Lying Areas of Beaumont Flooded, Extensive Flooding in Orange County, Bridge City Hardest HitLamar University Campus in Beaumont Did Not FloodLibrary, Family and Consumer Sciences, and Montaigne Center Buildings Had Roof Damage24 Sept 2008 Essential Staff Returned to Campus25 Sept 2008 Classes Resumed at Lamar University
  14. 14. Assessing Services During IkeLU Lessons LearnedStorm Facts InventoryContact Lists Physical AssetsEmergency Sites and Lodging Power and FuelAdministration, Network Service Communications, and Main Computer Center and Manpower Key TelecommunicationsSafety, Security and Training Hubs Long-Term Preparations Post-Storm Activities Damage Assessment and Post-Storm Activities
  15. 15. Storm FactsNOAA WebsiteMajor News Media: CNN, Weather Channel and Storm Pulse (Importance of Local News Media: KFDM and EntergyGovernment Contacts: Jefferson County and FEMALocal Contacts: State Police and Beaumont Police
  16. 16. Contact ListsUpdated and Distributed Internal Contacts For Gustav Leadership Team Phone List Pocket Card for Satellite Phone Numbers ITS Contact ListExternal Contact Lists Maintained By Staff Sections
  17. 17. Emergency Sites and LodgingEmergency Control Site Remained on Campus in BeaumontTemporary Lodging and Shelter Arranged by IndividualsDisaster Recovery/Business Continuity Site at San Marcos State University
  18. 18. Texas State University System Institutions (East Texas Locations in Bold)  Lamar University  Sam Houston State University  Sul Ross State University  Sul Ross State University Rio Grande College  Texas State University-San Marcos  Lamar Institute of Technology  Lamar State College-Orange  Lamar State College-Port Arthur
  19. 19. Sam Houston State in Hunstville153 mi – about 2 hours 29 minsTOO CLOSE FOR COMFORT
  20. 20. Stephen F. Austin State128 mi – about 2 hours 36NOT MEMBER OF TSUS
  21. 21. San Marcos State University251 mi – about 4 hours 15 minsDuring Ike – Approx. 16 hrsLAMAR UNIVERSITY BC Site
  22. 22. Administration, Communications andManpowerImportance of Clear Chain of Command and Clear Delineation of ResponsibilitiesITS Set Up Two Teams: Offsite B/C Team and Onsite Command Center TeamCommunication with Employees via LU Website and ConnectED (Now Blackboard Connect) to E-Mail and Phones.Communications Methods: Cell Phones, Satellite Phones, Land Lines and SIP Phones for Key Employees
  23. 23. ACUTA NOTE: Effective Leadership Was Key to Success
  24. 24. ACUTA Note: Importance of Coordination of Operations Functions
  25. 25. ACUTA NOTE:Importance ofCoordination of theAcademic Response
  26. 26. Administration, Communications andManpowerITS Hunkered Down With LUPD in Beaumont  Daily Meetings for Coordination in Beaumont on Campus  Reporting and Notification Lead By President and Leadership Team  Conference Bridges For Coordination With Remote Staff Managed by Network Services  Public Coordination Handled by LU PDManaged Through Office of VP for Finance and Amin  Supplemental Personnel  Accountability and Time Reporting
  27. 27. LU IT Services Staff DispositionLocal Core Team Stayed on Campus ThroughoutRemote Core Team Relocated to San Marcos Area – Worked on Ordering Replacement SystemsRemaining Staff Dispersed With FamiliesAT&T Conference Bridge to Maintain Contact With Dispersed Staff
  28. 28. Onsite IT Service Lead: John Genuardi, Customer Support Director
  29. 29. Onsite Network Support: Patrick Stewart, Manager of Network Services
  30. 30. Onsite Data Center Lead: Srinivas “Sri” Vandaraj, Senior Security Analyst
  31. 31. Safety, Security and TrainingPresident’s Weekly Security Meeting Initiated as Outcome of RitaSystem Access Managed By Onsite Core ITS Team (Issues With Passwords: Need One-Time Password System)Physical Re-Entry Credentials Provided by Chief of PoliceITS and PD Coordinated  Security Cameras Shutdown and Start-up  Training for ConnectedED
  32. 32. Inventory and Physical AssetsImpact of Key Staff ResignationPlanning: AVP (IT Infrastructure and Security) Responsible for Availability of Emergency Supplies and Work SuppliesAVP Had Resigned Between Gustav and Ike!Single Point of Staff Failure – Transitioned Responsibilities Quickly …Equipment Inventory, To Include Network and Data Center Backup and Recovery Plans Needed to Be Transitioned ON THE FLY! (New Role of Onsite Lead)Note on Vehicle Planning: Golf Carts Proved Critical After Storm
  33. 33. Power and Fuel: Handled ByFacilities ManagementStaff Continuity Maintained - Gustav to IkeCommunication with Local Fuel VendorsCommunication and Coordination With Power CompaniesEngine/Genset (portable and permanent)Fuel TanksBatteriesRectifiersStaging and Deployment
  34. 34. Network ServicePreventative Measures:  No Redundant Routes for Network (Big Exposure)  LEARN Connectivity Initiative Q 1 of 2009  Half-Completed VoIP Migration (Half Cisco/Half Avaya)Priority Areas:  Network Services Spread Between Windows Services Group (DHCP) and Unix Group (DNS).  Failure to Standardize Led to Confusion and LAN Service Outages During the Storm (Premature AD Shutdown).
  35. 35. Significant Telecomm EventsGOT LUCKY -- LU OC-3 Circuit to Houston Stayed Up Throughout Event – NO INTERNET CONNECTIVITY OUTAGE (Gigaman to Orange Also Stayed Up)Onsite Team Kept Campus Services Running Until Night of Landfall (Sept. 12-13)Throughout Storm VoIP access via WAN using SIP Phones (Ordered More Broadband Cards After Storm)Cell Service Unreliable Beaumont to Remote SitesResidential lines out for over several weeksSatellite Phones Proved Difficult to Use
  36. 36. Main Computer Center and KeyTelecommunications HubsSuccess Story: No Systems Shutdown Sequence in Place For Data Centers. Planning Initiated in Gustav Continued Into Ike Lead UpDownside to Not Running Your Own Power: Power Failure Exposure Due to Unmet Power Requirements in Data Center Concerns re: Lack of Fuel Led Local Team to Bring Down Systems After EventTape Backups: Contract for Off Site Had Just Lapsed! Can You Sing “Do the Hustle”?Wind and High Water: Failure to Stock Pile Sandbags
  37. 37. Long Term PreparationsMaintain Year-Round  Example of Successful Planning: Replication of Website and Content Management Service  Criticality of ERP Disaster Recovery/BC Site at San Marcos  Hardware In Place in San Marcos  Software to Be Configured With Data Replication (Tested) By Hurricane Season 2009  Long-Term Plans/Prevention  Move to the Cloud: Portal and Mail Outsourcing Initiatives  Importance of Service Continuity: Redesign of Network Services (Standardization Under Network Services Group)
  38. 38. Post-Storm Activities Led ByOnsite Team LeadEmployee SafetyContactsConference BridgesCostsTemporary Lodging/ShelterExtra Work ForceSafetyEmergency Supply AvailabilityPower and Fuel
  39. 39. Flooding in Low-Lying Areas
  40. 40. Damage to People’s Homes
  41. 41. Damage to Local Businesses
  42. 42. Damage Assessment and ServiceRestorationAnnouncement Via ConnectED on Return of Staff to Campus (September 24th), Return of Students to Campus (September 25th)Class Schedule Readjusted by Office of the Provost, With Resumption of Classes on September 25thPresident’s Faculty/Staff Convocation on September 25th
  43. 43. Perspectives From Galveston Steven ConwayTexas A&M University at Galveston (TAMUG)
  44. 44. Hurricane IkeTAMUG evacuated campus 10 Sept 2008Hurricane Ike landfall 12 Sept 200875% of Galveston was floodedTAMUG did not flood, 2 buildings had roof damageTAMUG restarted classes at TAMU in College Station TX Wed 24 Sept91% of the students returned20 Jan 2008 Classes resume in Galveston
  45. 45. Significant Telecomm EventsCell service stayed up throughout storm-Cat 2UTMB Gigaman Circuit to Houston stayed upTAMUG-UTMB Gigaman available 24 Sept when power returned on campusT1’s out for over a monthResidential lines out for over 6 weeksVoIP access to College Station via WAN
  46. 46. Hurricane Ike Flooding Galveston A house burns uncontrolled in a flooded neighborhood as Hurricane Ike approaches the Texas coast, Friday, Sept. 12, 2008, in Galveston, Texas. (AP Photo/Matt Slocum)
  47. 47. TAMUG Boat on Causeway