Successfully reported this slideshow.

DR luncheon presentation

186 views

Published on

Disaster Recovery Planning..a 9/11 CIO story with lessons learned

  • Be the first to comment

  • Be the first to like this

DR luncheon presentation

  1. 1. Disaster Recovery White Feather Solutions Disaster Recovery Planning
  2. 2. Summary First Six Months As CIO of organization in close proximity to twin towers What occurred during and after 9/11 Post Mortem Planning for a Disaster BCP recommendations RIO proposed roadmap of DR facilities and services
  3. 3. First Six Months As CIOwith Boutique Financial Firm left three ominous letters from the previous CIO 1. Evaluate departmental requirements and assess overhead 2. Cut Unnecessary Overhead 3. Cut Extraneous Servers and Systems 4. Cut Personnel
  4. 4. First Six Months As CIO At Financial Firm Discovered report from the previous CIO who was in place during the 1993 bombing of the World Trade Center  Enterprise was shutdown for 10 days  Losses totaled over 100M  No insurance  No Business Continuity Plan
  5. 5. New Mission Developed disaster recovery plan  Had expertise with this type of firm prior to becoming CIO and wrote several BCP’s for similar firms engaged in Y2K projects  Utilized a BCP I developed which was accepted by Deloitte auditors for Y2K and updated it to reflect the present IT procedures
  6. 6. New Mission Interviewed all the departments in order to add their segments of the enterprise to the BCP with the latest business practices Queried department heads as to what were their most critical functions and what needed to be protected and running during a disaster Ascertained what functional interdependencies currently existed between the departments and if they needed to be protected and running
  7. 7. Presentation of the DR Plan to Management Reviewed the outcome of the 1993 bombing and the significant financial impact it had on the organization which, like many in NYC, had been caught unprepared on many levels. Outlined the Business Continuity Plan Reviewed what was needed in terms of resources for BCP Demonstrated graphically how each process would be addressed during an actual disaster Reviewed SLAs negotiated with the vendors Proposed new processes to be incorporated in the Enterprise Reviewed the costs involved in the implementation of the plan, approximated $100,000.
  8. 8. Results Management balked at the business processes and their associated costs. A common roadblock to BCP success: management did not buy into the ongoing cost for backup servers that, as they saw it, were not being used. BCP presentation took place on February 9th, 2001 With zero budget, over the next 5 months I created, implemented and tested a DR plan in my own department. I shared the plan with all department heads. A few of the back office departments implemented the plan. THEN CAME 9/11...
  9. 9. Workday visuals This is what many NYC residents who walked to work everyday routinely saw
  10. 10. September 11, 2001 reports came into the office that a small plane was off course and had hit one of the twin towers. Office talk was buzzing as to whether business should close 15 minutes later the second plane hit . A call came in from management asking for direction. As CIO, every facet of the business touched the data network whose operation was my responsibility. My response was clear. The business had to shut down. Background as Marine fighter pilot post Viet Nam served myself and my company well.
  11. 11. September 11 2001 About an hour or so after that the collapse:
  12. 12. September 11 2001 And finally the aftermath:
  13. 13. September 11 2001
  14. 14. Spring in to action Our communications plan was immediately implemented.  Emailed a standard evacuation list to department heads and instructed them to vacate. For those departments who had the plan invoke it and vacate.  SMT (Survival Management Teams – 5 two person teams) went through 5 floors of offices making sure everyone was leaving and giving instruction on where to meet, well away from the disaster, which had been predetermined. Ordered a shutdown of all systems and the evacuation of all personnel, and collection of all software backups. Attempted to call vendors, however land and wireless lines were all tied up. I sent emails ordering lines, systems, software to be brought to a second facility we had in Queens, some 6 miles away. Once all the employees were evacuated and the doors locked, we proceeded to walk to the United Nations Building, the predetermined meet place. We took a head count, everyone had made it.
  15. 15. Spring into actionManagement request “PICK 5 MEN AND GO IN AND RETRIEVE FINANCIAL SERVERS”Response I WILL GO IN IF PROVIDED WITH OXYGEN, PROTECTIVE SUITS AND CASH TO GET PAST THE SECURITY ROADBLOCKS. THESE MEN HAVE FAMILIES AND HAVE NOT EXPERIENCED WAR .Management CASH ONLY
  16. 16. Aftermath set up a hotline using voice mail. The city was completely shutdown. I experienced difficulty updating the voice mail Receive call from management. They attempted to perform a rescue mission of the servers and the financial data. Men that went in, are still receiving mental health therapy. The following day, along with a NYC police officer, two of my people and myself, a cart, dressed in fire retardant suits, with oxygen masks, axes, crowbars, we go get the servers We had to strap them on our backs and take them, one by one, down 5 flights of stairs.
  17. 17. POST MORTEM Proposed DR plan -$100K Cost to get basic functionality back - $500k+ Replacement of lost equipment - $1.2M Insurance picked up around 25% It took around 21 days to get back on line at a loss of approximately $160M Risk to employees, having to recover the servers
  18. 18. PLANNING FOR A DISASTER Types Of Disasters Fire / Explosions Natural Disasters - Tropical Storms and Hurricanes - Tornado and Thunderstorms - Earthquakes - Floods Utilities Disruption - Electric - Gas - Water - Phone - Heat / Air Conditioning - Loss Of Elevators - Computer System Failure
  19. 19. PLANNING FOR A DISASTER Types Of Disasters Security Emergency - Bomb Threats - Criminal Acts on Individuals - Civil Disturbance Hazardous Materials Spills/Releases Food Borne Illness Visitors Incidents Employee Incidents - Employee Serious Injury/Death - Workplace Violence Media Inquiries Evacuation - Site Evacuation Planning Factors - Community Wide Evacuation Shelter-In-Place
  20. 20. PLANNING FOR A DISASTERComponents of a Disaster Recovery Plan Human Continuity Plan  Create Survival Management Teams  Define a scope of what the plan will cover Business Continuity Plan  Define each departments role in the enterprise and determine how critical their functions are to the business  Develop contingencies to critical business processes Facility Response Plan  Create Facility Response Teams  Describe Facilities & Locations  Define a scope of what the plan will cover
  21. 21. PLANNING FOR A DISASTERComponents of a Disaster Recovery Plan Event Management  Create Crisis Management Teams  Detection of events  Incident Detection  Incident Assessment and  Classification  Automatic Detection of Events  Fire Alarms  Intrusion Alarms  Event Response  Notification of proper authorities
  22. 22. PLANNING FOR A DISASTERComponents of a Disaster Recovery Plan Crisis Communication Plan  Determine how the enterprise communicates with it’s employees Post Disaster Review Disaster Testing Plan  Frequency
  23. 23. PLANNING FOR A DISASTERExamples of Components in a Disaster Recovery PlanHuman Continuity Plan Survival Management Team Member  Company will send volunteers for EMT training  Volunteers will be trained in light extrication
  24. 24. PLANNING FOR A DISASTERExamples of Components in a Disaster Recovery PlanBusiness Continuity Plan

×