Your SlideShare is downloading. ×
LU - ACUTA 2009 Conference Presentation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

LU - ACUTA 2009 Conference Presentation

107

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
107
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Lessons Learned From the Texas Gulf Coast ACUTA 2009 Palm Springs, California
  • 2. About Your SpeakersTexas State University System Dr. Michael E. Dobe, Sr., Associate Vice President, Chief Information Officer & Adjunct Professor at Lamar University (Beaumont, Texas) http://www.linkedin.com/in/mdobeTexas A & M System Mr. Steven Conway, Director of Computing and Info Services at Texas A&M University at Galveston (Galveston, Texas) http://www.linkedin.com/pub/5/b7/24a
  • 3. About the Checklist Steven ConwayTexas A&M University at Galveston (TAMUG)
  • 4. ACUTA/ATIS Hurricane Preparedness ProjectFormed Committee May 2007Reviewed and Modified ATIS documentPublished ACUTA Hurricane Checklist Oct 2007
  • 5. Hurricane Preparedness Checklist Hurricane Katrina, August 2005, Terra MODIS True Color Image Courtesy of Louisiana State UniversityACUTA - The Association for Information Communications Technology Professionals in Higher Education Developed in cooperation with ATIS – The Alliance for Telecommunications Industry Solutions http://www.acuta.org/?2026
  • 6. Committee Amy Phillips, Louisiana State University Health Sciences Center Frank OQuinn, Louisiana State University Sandra Thompson, University of West Florida James Shea, Boston University Jeanne Spinosa, Johnson & Wales University Garret Yoshimi, University of Hawaii George Denbow, University of Texas at Austin Michele Narcavage, University of Pennsylvania Steve Ellis, University of Notre Dame Steve Conway, Texas A&M University at Galveston Walt Magnussen, Texas A&M University - Task Force Chair Veronica Lancaster, ATIS Brad Ruark, ATIS /AT&T Jeri Semer, ACUTA Executive Director Aaron Fuehrer, ACUTA Staff Liaison
  • 7. ACUTA Hurricane Preparedness Checklist
  • 8. Using the Checklist As An After-Action Review Tool Perspectives From Beaumont Dr. Michael Dobe, Sr. Lamar University
  • 9. About Lamar UniversityLocated in Beaumont, TX (Approx 40 Minutes from Louisiana Border)Central Data Center in Beaumont Serves Three Campuses: Lamar University Lamar Institute of Technology Lamar State College- OrangeTotal Students Served – Approximately 16,000
  • 10. Proximity of Beaumont and Galveston119 mi – about 1 hour 57 mins
  • 11. Rita, Edouard, Gustav, Ike Hurricane Activity in SE TexasRita – September 2005 (CAT 4 With Major Damage to Beaumont)2006 and 2007 Atlantic Hurricane Seasons Quiet in SE TexasEdouard – August 2008 (Missed SE Texas)Gustav – September 1, 2008 (Minor Impact in SE Texas)Ike Hit SE Texas Less Than Two Weeks After Gustav…
  • 12. Hurricane Ike’s Path http://en.wikipedia.org/wiki/File:Ike_2008_track.png
  • 13. Hurricane Ike TimelineLamar evacuated campus 10 Sept 2008Hurricane Ike landfall Evening of 12-13 Sept 2008Low Lying Areas of Beaumont Flooded, Extensive Flooding in Orange County, Bridge City Hardest HitLamar University Campus in Beaumont Did Not FloodLibrary, Family and Consumer Sciences, and Montaigne Center Buildings Had Roof Damage24 Sept 2008 Essential Staff Returned to Campus25 Sept 2008 Classes Resumed at Lamar University
  • 14. Assessing Services During IkeLU Lessons LearnedStorm Facts InventoryContact Lists Physical AssetsEmergency Sites and Lodging Power and FuelAdministration, Network Service Communications, and Main Computer Center and Manpower Key TelecommunicationsSafety, Security and Training Hubs Long-Term Preparations Post-Storm Activities Damage Assessment and Post-Storm Activities
  • 15. Storm FactsNOAA WebsiteMajor News Media: CNN, Weather Channel and Storm Pulse (http://www.stormpulse.com/)Importance of Local News Media: KFDM and EntergyGovernment Contacts: Jefferson County and FEMALocal Contacts: State Police and Beaumont Police
  • 16. Contact ListsUpdated and Distributed Internal Contacts For Gustav Leadership Team Phone List Pocket Card for Satellite Phone Numbers ITS Contact ListExternal Contact Lists Maintained By Staff Sections
  • 17. Emergency Sites and LodgingEmergency Control Site Remained on Campus in BeaumontTemporary Lodging and Shelter Arranged by IndividualsDisaster Recovery/Business Continuity Site at San Marcos State University
  • 18. Texas State University System http://www.tsus.edu/TSUS Institutions (East Texas Locations in Bold)  Lamar University  Sam Houston State University  Sul Ross State University  Sul Ross State University Rio Grande College  Texas State University-San Marcos  Lamar Institute of Technology  Lamar State College-Orange  Lamar State College-Port Arthur
  • 19. Sam Houston State in Hunstville153 mi – about 2 hours 29 minsTOO CLOSE FOR COMFORT
  • 20. Stephen F. Austin State128 mi – about 2 hours 36NOT MEMBER OF TSUS
  • 21. San Marcos State University251 mi – about 4 hours 15 minsDuring Ike – Approx. 16 hrsLAMAR UNIVERSITY BC Site
  • 22. Administration, Communications andManpowerImportance of Clear Chain of Command and Clear Delineation of ResponsibilitiesITS Set Up Two Teams: Offsite B/C Team and Onsite Command Center TeamCommunication with Employees via LU Website and ConnectED (Now Blackboard Connect) to E-Mail and Phones.Communications Methods: Cell Phones, Satellite Phones, Land Lines and SIP Phones for Key Employees
  • 23. ACUTA NOTE: Effective Leadership Was Key to Success
  • 24. ACUTA Note: Importance of Coordination of Operations Functions
  • 25. ACUTA NOTE:Importance ofCoordination of theAcademic Response
  • 26. Administration, Communications andManpowerITS Hunkered Down With LUPD in Beaumont  Daily Meetings for Coordination in Beaumont on Campus  Reporting and Notification Lead By President and Leadership Team  Conference Bridges For Coordination With Remote Staff Managed by Network Services  Public Coordination Handled by LU PDManaged Through Office of VP for Finance and Amin  Supplemental Personnel  Accountability and Time Reporting
  • 27. LU IT Services Staff DispositionLocal Core Team Stayed on Campus ThroughoutRemote Core Team Relocated to San Marcos Area – Worked on Ordering Replacement SystemsRemaining Staff Dispersed With FamiliesAT&T Conference Bridge to Maintain Contact With Dispersed Staff
  • 28. Onsite IT Service Lead: John Genuardi, Customer Support Director
  • 29. Onsite Network Support: Patrick Stewart, Manager of Network Services
  • 30. Onsite Data Center Lead: Srinivas “Sri” Vandaraj, Senior Security Analyst
  • 31. Safety, Security and TrainingPresident’s Weekly Security Meeting Initiated as Outcome of RitaSystem Access Managed By Onsite Core ITS Team (Issues With Passwords: Need One-Time Password System)Physical Re-Entry Credentials Provided by Chief of PoliceITS and PD Coordinated  Security Cameras Shutdown and Start-up  Training for ConnectedED
  • 32. Inventory and Physical AssetsImpact of Key Staff ResignationPlanning: AVP (IT Infrastructure and Security) Responsible for Availability of Emergency Supplies and Work SuppliesAVP Had Resigned Between Gustav and Ike!Single Point of Staff Failure – Transitioned Responsibilities Quickly …Equipment Inventory, To Include Network and Data Center Backup and Recovery Plans Needed to Be Transitioned ON THE FLY! (New Role of Onsite Lead)Note on Vehicle Planning: Golf Carts Proved Critical After Storm
  • 33. Power and Fuel: Handled ByFacilities ManagementStaff Continuity Maintained - Gustav to IkeCommunication with Local Fuel VendorsCommunication and Coordination With Power CompaniesEngine/Genset (portable and permanent)Fuel TanksBatteriesRectifiersStaging and Deployment
  • 34. Network ServicePreventative Measures:  No Redundant Routes for Network (Big Exposure)  LEARN Connectivity Initiative Q 1 of 2009  Half-Completed VoIP Migration (Half Cisco/Half Avaya)Priority Areas:  Network Services Spread Between Windows Services Group (DHCP) and Unix Group (DNS).  Failure to Standardize Led to Confusion and LAN Service Outages During the Storm (Premature AD Shutdown).
  • 35. Significant Telecomm EventsGOT LUCKY -- LU OC-3 Circuit to Houston Stayed Up Throughout Event – NO INTERNET CONNECTIVITY OUTAGE (Gigaman to Orange Also Stayed Up)Onsite Team Kept Campus Services Running Until Night of Landfall (Sept. 12-13)Throughout Storm VoIP access via WAN using SIP Phones (Ordered More Broadband Cards After Storm)Cell Service Unreliable Beaumont to Remote SitesResidential lines out for over several weeksSatellite Phones Proved Difficult to Use
  • 36. Main Computer Center and KeyTelecommunications HubsSuccess Story: No Systems Shutdown Sequence in Place For Data Centers. Planning Initiated in Gustav Continued Into Ike Lead UpDownside to Not Running Your Own Power: Power Failure Exposure Due to Unmet Power Requirements in Data Center Concerns re: Lack of Fuel Led Local Team to Bring Down Systems After EventTape Backups: Contract for Off Site Had Just Lapsed! Can You Sing “Do the Hustle”?Wind and High Water: Failure to Stock Pile Sandbags
  • 37. Long Term PreparationsMaintain Year-Round  Example of Successful Planning: Replication of Website and Content Management Service  Criticality of ERP Disaster Recovery/BC Site at San Marcos  Hardware In Place in San Marcos  Software to Be Configured With Data Replication (Tested) By Hurricane Season 2009  Long-Term Plans/Prevention  Move to the Cloud: Portal and Mail Outsourcing Initiatives  Importance of Service Continuity: Redesign of Network Services (Standardization Under Network Services Group)
  • 38. Post-Storm Activities Led ByOnsite Team LeadEmployee SafetyContactsConference BridgesCostsTemporary Lodging/ShelterExtra Work ForceSafetyEmergency Supply AvailabilityPower and Fuel
  • 39. Flooding in Low-Lying Areas
  • 40. Damage to People’s Homes
  • 41. Damage to Local Businesses
  • 42. Damage Assessment and ServiceRestorationAnnouncement Via ConnectED on Return of Staff to Campus (September 24th), Return of Students to Campus (September 25th)Class Schedule Readjusted by Office of the Provost, With Resumption of Classes on September 25thPresident’s Faculty/Staff Convocation on September 25th
  • 43. Perspectives From Galveston Steven ConwayTexas A&M University at Galveston (TAMUG)
  • 44. Hurricane IkeTAMUG evacuated campus 10 Sept 2008Hurricane Ike landfall 12 Sept 200875% of Galveston was floodedTAMUG did not flood, 2 buildings had roof damageTAMUG restarted classes at TAMU in College Station TX Wed 24 Sept91% of the students returned20 Jan 2008 Classes resume in Galveston
  • 45. Significant Telecomm EventsCell service stayed up throughout storm-Cat 2UTMB Gigaman Circuit to Houston stayed upTAMUG-UTMB Gigaman available 24 Sept when power returned on campusT1’s out for over a monthResidential lines out for over 6 weeksVoIP access to College Station via WAN
  • 46. Hurricane Ike Flooding Galveston A house burns uncontrolled in a flooded neighborhood as Hurricane Ike approaches the Texas coast, Friday, Sept. 12, 2008, in Galveston, Texas. (AP Photo/Matt Slocum)
  • 47. TAMUG Boat on Causeway

×