Disaster Recovery White Feather Solutions Disaster Recovery Planning
Summary First Six Months As CIO of organization in close proximity to twin towers What occurred during and after 9/11 Post Mortem Planning for a Disaster BCP recommendations RIO proposed roadmap of DR facilities and services
First Six Months As CIOwith Boutique Financial Firm left three ominous letters from the previous CIO 1. Evaluate departmental requirements and assess overhead 2. Cut Unnecessary Overhead 3. Cut Extraneous Servers and Systems 4. Cut Personnel
First Six Months As CIO At Financial Firm Discovered report from the previous CIO who was in place during the 1993 bombing of the World Trade Center Enterprise was shutdown for 10 days Losses totaled over 100M No insurance No Business Continuity Plan
New Mission Developed disaster recovery plan Had expertise with this type of firm prior to becoming CIO and wrote several BCP’s for similar firms engaged in Y2K projects Utilized a BCP I developed which was accepted by Deloitte auditors for Y2K and updated it to reflect the present IT procedures
New Mission Interviewed all the departments in order to add their segments of the enterprise to the BCP with the latest business practices Queried department heads as to what were their most critical functions and what needed to be protected and running during a disaster Ascertained what functional interdependencies currently existed between the departments and if they needed to be protected and running
Presentation of the DR Plan to Management Reviewed the outcome of the 1993 bombing and the significant financial impact it had on the organization which, like many in NYC, had been caught unprepared on many levels. Outlined the Business Continuity Plan Reviewed what was needed in terms of resources for BCP Demonstrated graphically how each process would be addressed during an actual disaster Reviewed SLAs negotiated with the vendors Proposed new processes to be incorporated in the Enterprise Reviewed the costs involved in the implementation of the plan, approximated $100,000.
Results Management balked at the business processes and their associated costs. A common roadblock to BCP success: management did not buy into the ongoing cost for backup servers that, as they saw it, were not being used. BCP presentation took place on February 9th, 2001 With zero budget, over the next 5 months I created, implemented and tested a DR plan in my own department. I shared the plan with all department heads. A few of the back office departments implemented the plan. THEN CAME 9/11...
Workday visuals This is what many NYC residents who walked to work everyday routinely saw
September 11, 2001 reports came into the office that a small plane was off course and had hit one of the twin towers. Office talk was buzzing as to whether business should close 15 minutes later the second plane hit . A call came in from management asking for direction. As CIO, every facet of the business touched the data network whose operation was my responsibility. My response was clear. The business had to shut down. Background as Marine fighter pilot post Viet Nam served myself and my company well.
September 11 2001 About an hour or so after that the collapse:
Spring in to action Our communications plan was immediately implemented. Emailed a standard evacuation list to department heads and instructed them to vacate. For those departments who had the plan invoke it and vacate. SMT (Survival Management Teams – 5 two person teams) went through 5 floors of offices making sure everyone was leaving and giving instruction on where to meet, well away from the disaster, which had been predetermined. Ordered a shutdown of all systems and the evacuation of all personnel, and collection of all software backups. Attempted to call vendors, however land and wireless lines were all tied up. I sent emails ordering lines, systems, software to be brought to a second facility we had in Queens, some 6 miles away. Once all the employees were evacuated and the doors locked, we proceeded to walk to the United Nations Building, the predetermined meet place. We took a head count, everyone had made it.
Spring into actionManagement request “PICK 5 MEN AND GO IN AND RETRIEVE FINANCIAL SERVERS”Response I WILL GO IN IF PROVIDED WITH OXYGEN, PROTECTIVE SUITS AND CASH TO GET PAST THE SECURITY ROADBLOCKS. THESE MEN HAVE FAMILIES AND HAVE NOT EXPERIENCED WAR .Management CASH ONLY
Aftermath set up a hotline using voice mail. The city was completely shutdown. I experienced difficulty updating the voice mail Receive call from management. They attempted to perform a rescue mission of the servers and the financial data. Men that went in, are still receiving mental health therapy. The following day, along with a NYC police officer, two of my people and myself, a cart, dressed in fire retardant suits, with oxygen masks, axes, crowbars, we go get the servers We had to strap them on our backs and take them, one by one, down 5 flights of stairs.
POST MORTEM Proposed DR plan -$100K Cost to get basic functionality back - $500k+ Replacement of lost equipment - $1.2M Insurance picked up around 25% It took around 21 days to get back on line at a loss of approximately $160M Risk to employees, having to recover the servers
PLANNING FOR A DISASTER Types Of Disasters Fire / Explosions Natural Disasters - Tropical Storms and Hurricanes - Tornado and Thunderstorms - Earthquakes - Floods Utilities Disruption - Electric - Gas - Water - Phone - Heat / Air Conditioning - Loss Of Elevators - Computer System Failure
PLANNING FOR A DISASTER Types Of Disasters Security Emergency - Bomb Threats - Criminal Acts on Individuals - Civil Disturbance Hazardous Materials Spills/Releases Food Borne Illness Visitors Incidents Employee Incidents - Employee Serious Injury/Death - Workplace Violence Media Inquiries Evacuation - Site Evacuation Planning Factors - Community Wide Evacuation Shelter-In-Place
PLANNING FOR A DISASTERComponents of a Disaster Recovery Plan Human Continuity Plan Create Survival Management Teams Define a scope of what the plan will cover Business Continuity Plan Define each departments role in the enterprise and determine how critical their functions are to the business Develop contingencies to critical business processes Facility Response Plan Create Facility Response Teams Describe Facilities & Locations Define a scope of what the plan will cover
PLANNING FOR A DISASTERComponents of a Disaster Recovery Plan Event Management Create Crisis Management Teams Detection of events Incident Detection Incident Assessment and Classification Automatic Detection of Events Fire Alarms Intrusion Alarms Event Response Notification of proper authorities
PLANNING FOR A DISASTERComponents of a Disaster Recovery Plan Crisis Communication Plan Determine how the enterprise communicates with it’s employees Post Disaster Review Disaster Testing Plan Frequency
PLANNING FOR A DISASTERExamples of Components in a Disaster Recovery PlanHuman Continuity Plan Survival Management Team Member Company will send volunteers for EMT training Volunteers will be trained in light extrication
PLANNING FOR A DISASTERExamples of Components in a Disaster Recovery PlanBusiness Continuity Plan