Millions of dollars are spent to build, furnish and staff a data center or IT-enabled facility, but how much is spent to protect it? Disaster Recovery usually only receives the funding it needs after an incident. However, there are many systems and services you can employ to inexpensively prepare for the inevitable which will also help you begin discussions for improving preparedness. Your organization can drastically improve physical security and disaster readiness. Daniel Hanttula, President-elect of InfraGard Oklahoma and the MS-ISAC Business Continuity, Disaster Recovery and Cyber Exercise Workgroup member tasked with developing a Data Center Recovery Guide, will discuss how to put the spotlight back on your BCP-DR program to help gain funding and awareness for your program – no matter what condition it is currently in.
2. I’M NOT “THAT” DANIEL HANTTULA
State employees cannot:
• Recommend products
• Create presentations without a million reviews
• Give you materials I’ve developed without permission, incurring taxpayer
cost, etc., etc., etc.
• Accept bribes
2
So please:
• Acknowledge that I’m taking lunch, starting riiiiiiiight now.
• Don’t ask me questions about the OMES IS data center
• Take my personal copy of goodies
• Regardless, no matter how much you enjoy this presentation, keep your wallet in
your pocket/purse!
3. AGENDA
Take-Aways
• DR maturity models
• The “$6 sexy”
• Free tools
• The science of disaster response
• A white paper (ooooooh, aaaaaah) to be provided later
3
How to talk to senior management about:
• Funding disaster recovery
• Their expectations during a disaster
• The importance of exercises
• Learn how to “woo” logical & emotional management
4. ASSUMPTIONS
Your company:
Has a disaster recovery plan
The disaster recovery plan:
Is not in a Trapper Keeper™ from 1987
Is not printed from a dot matrix printer
Does not include a roll of dimes to run the emergency call list on a payphone
Is not covered in prehistoric dust/dinosaur fossil remnants
Is known, by management to exist
4
“Recovery on a dime” refers to
the 10¢ cost to make a phone
call in 1981.
5. 5
IF NOT… IDRP IMMEDIATELY
Source:
http://www.dummies.com/programming/networking/10-elements-of-an-interim-it-disaster-recovery-plan/
10 Elements of an Interim IT Disaster Recovery Plan
By Peter H. Gregory, Philip Jan Rothstein
Part of IT Disaster Recovery Planning For Dummies Cheat Sheet
If you don’t have a fully detailed IT disaster recovery plan (DRP) right now, then
implement an interim disaster recovery plan (IDRP) as you develop your long-term
safety net. Sequester two or three experts for one day to develop an interim disaster-
recovery plan that contains:
• A list of people on the emergency response team
• Procedures for declaring a disaster
• Procedures for invoking the DR plan
• Emergency communications
• How to carry out basic recovery plans
• Viable processing center alternatives
• How to enact preventive measures
• A documented interim DR plan
• Wallet-sized emergency contact lists
• Training methods for emergency response team members
6. BUT DOING DR PLANNING IS HARD…
Expecting to “shoot fish in a barrel?”
6
7. WHAT MAKES IT SO DIFFICULT?
• “I need to calculate my downtime cost”
• “I need to perform a business impact analysis for every business unit”
• “I don’t have the proper tools”
• “We don’t have a budget”
8. SO WITH ALL THOSE CHALLENGES, “WHY?”
8
Optimist " We'll never
have a problem... Our
building is safe..."
Pessimist "We're just
one good event away
from getting funded."
9. “BUT WHY?” – PRAGMATIST
9
• Small Business (<10 employees)
$8,220/hr
• Small Business (10-99 employees)
$10,790/hr
• Medium Business (100-499 employees)
$25,600/hr
• Large Business (500+ employees)
$100,000+/hr
One-third of large businesses report $1M+ in damages for each hour lost!
Sources: IDC: The Growth Opportunity for SBC Cloud and Hybrid Business Continuity
ITIC 2017 Reliability and Hourly Cost of Downtime Trends Survey
12. DISASTER RECOVERY REGULATORY “REQUIREMENTS”
FFIEC HIPAA IRS-1075 PCI
Test Annual “Periodic”† Annual Annual
Update “Regular” “Periodic”† Annual Annual
Train “Continuous” Annual “Periodic”
Sources: FFIEC BCP IT Examination Handbook (February 2015), HIPAA Security Rule
164.308(a)(7)(i) (March 2013), IRS 1075 (October 2016), and PCI DSS v3.2
† While HIPAA has a number of strict requirements for continuity, the testing and revision requirements are “addressable.” Often confused with
“optional,” the US Dept. Health & Human Services states “a covered entity must implement an addressable implementation specification if it is
reasonable and appropriate to do so, and must implement an equivalent alternative if the addressable implementation specification is unreasonable
and inappropriate, and there is a reasonable and appropriate alternative.”
16. WHO HATES EXERCISES?
Fire Drills
• 60 flights of stairs
• Held quarterly
• Impacts productivity
• Requires senior management involvement
• Makes notable improvement
• Asked senior management to move a well-
known financial institution out of a major
metropolitan area
16
17. RICK RESCORLA
Bachelor of Arts – English, University of Oklahoma
Master of Arts – English, University of Oklahoma
Law Degree – Oklahoma City University School of Law
Director of Security, Morgan Stanley
18. Event Evacuation Time Causalities
1993 Car Bombing 4 Hours 0
2001 9/11 Attack 90 Minutes 12
MORGAN STANLEY
One of the World Trade Center’s (WTC) largest tenants
Inhabiting space on twenty-five floors of Building Two
Testimonies portray Rescorla, in the hallways, with a stopwatch and
bullhorn, chastising employees who do not move quickly enough. Many
also say that Rescorla was disliked by senior management because of the
impact that his regular evacuation drills (in which the executives were
required to participate) had on the productivity of the firm, and they
rejected his request to move the firm to New Jersey into a four-story
building after the 1993 bombing.
19. ”
“The loss of just a single life is
too many, but when you
consider the incredible
destruction that occurred, the
loss of fewer than 40 of our
people out of the 3,700 who
worked there is a near
miracle.
Philip J. Purcell, chairman and CEO of Morgan Stanley
Firms counting, coping, CNN Money, September 13, 2001
By Robert on Flickr - This file has been extracted from
another file: UA Flight 175 hits WTC south tower 9-11
edit.jpeg, CC BY-SA 2.0
8:46AM
AA FLT 11 impacts WTC North
Tower. Port Authority issued a
“shelter-in-place” advisory to
the WTC South occupants but
Rick Rescorla begins
evacuating Morgan Stanley
employees.
9:02AM
Port Authority reverses order;
instructs tenants in WTC South
to evacuate.
9:03AM
UA FLT 175 strikes WTC South.
9:31AM
Morgan Stanley employees are
clear of the building.
9:59AM
The south tower, where (2,687
Morgan Stanley employees
had been working that
morning) collapses.
20. THE SCIENCE OF EXERCISE
20
https://www.youtube.com/watch?v=4_NW1uX10zc
21. THE SCIENCE OF EXERCISE
21
https://www.youtube.com/watch?v=4_NW1uX10zc
22. 22
ACTIVE SHOOTER SCENARIO
DOES YOUR FACILITY HAVE…
• … No security guard?
• … An unarmed security guard?
• … An armed security guard?
• … An Oklahoma State Trooper?
DOES YOUR PLAN…
• … Have pre-written messages that do not reduce threat severity,
while instilling confidence?
• “There is an active shooter reported in Building 1A, all teachers and
students should lockdown all classrooms and dorm rooms
immediately. Additional information will be sent every 15 minutes.”
23. 23
FEMA TO THE RESCUE!
• Preliminary
• Title
• Version
• Foreword
• Confidentiality Statement
• Introduction
• Intro
• Scope
• Purpose
• Disaster Definition
• Assumptions
• Area-Wide Disasters
• Contractual Arrangement for Recovery Services
• Points of Contact
• System Resources
• Critical Contacts and Resources
• Disruption Impact
• Resource Recovery Priority
• Disaster Recovery Strategy
• System Information
• Backup and Office Storage Procedures
• Offsite Storage Services
• Alternate Site Hardware and Software Configurations
• Testing the Recovery Plan
• Training
• Maintaining the Plan
26. 26
CONVERSATION STARTERS
Soft question: How often should this be updated?
Medium question: Would there be value in automating this list?
- Employee-managed PI updates/HR updates
Hard question: Is there interest in having a system that can ring down the entire organization?
Ironman question: Do our customers expect to be included (e.g. Schools/Universities)
Available for $6 at https://www.statearchivists.org
*** Every time you update, have the conversation again ***
27. MAP YOUR CRISIS TEAM
27
Soft question: What are
our expectations on
employee response times?
Medium question: Do we
need automation to check
in on employees?
Hard question: What is our
succession plan for critical
staff?
Ironman question: Do we
need to monitor a few VIPs
during travel? Give them
executive protection tools?
28. WHO TO “WOO?”
Logical Manager
• Hourly cost of an outage
• Regulatory requirements
• The super model maturity metrics
• The science of disaster exercises
• The PReP phone list
Emotional Manager
• Hourly cost of an outage
• The crawl/walk/run maturity metrics
• The Rick Rescorla story
• “Paul Blart Mall Cop” discussion about your facility
• Crisis team map
29. PROGRAM RESOURCES
Virtual Corp BCM Planning Tool http://bit.ly/2wHt6tW
FEMA BCP Suite http://bit.ly/2f3KilP
For Dummies: 10 Elements of an IDRP http://bit.ly/2gGDudS
PReP envelopes http://bit.ly/2gLeaHl
PReP templates http://bit.ly/2w6ArBj
Steven M. Crimando Video http://bit.ly/2gKPQFG
30