1. IT Continuity of
IT C ti it f
Operations
Minimize the Gaps Between Your
Mi i i th G B t Y
Recovery Capabilities and Functional
Requirements of the Enterprise
Requirements of the Enterprise
Presented By
y
Jeff Roseman
Director, IT Infrastructure ‐ Patterson Medical
March 10, 2009
2. Semper Paratus: Are You Ready?
Semper Paratus: Are You Ready?
• Annual Budget for Disaster Recovery
Annual Budget for Disaster Recovery
& Business Continuity?
• Experienced a Disaster?
Experienced a Disaster?
• Declared a Disaster in Last Year?
• Disaster Recovery Plan?
• Updated DR Plan in Last Year? Let Me
• Tested DR Plan in Last Year? See Your
S Y
y
• Business Continuity Plan? Hands
Semper Paratus (Latin: Always Prepared; U.S. Coast Guard motto)
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 2
3. Disaster Recovery vs.
Business Continuity
• Disaster Recovery (DR)
Disaster Recovery (DR)
– Evolved from Data Center operations
– Strictly a “technical” solution
Strictly a technical solution
– Over time, it was realized that recovery of the platforms
did not mean recovery of the business
• Business Continuity (BC)
– Addresses those “non‐technical” functions
that are required to restore business
th t i dt t b i
– Not just actions taken during a disaster
– An enterprise‐wide project not just IT
An enterprise‐wide project, not just IT
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 3
4. 2008 AT&T Business
Continuity Study
C i i S d
• One in five businesses does not have a business
One in five businesses does not have a business
continuity plan in place
• For the third year in a row, the survey finds that
nearly 30 % of U.S. businesses don t consider
nearly 30 % of U S businesses don't consider
business continuity planning a priority
• Six out of 10 companies have made some type of
p yp
business change in the past year, but only 28 %
updated their plans
• One‐fourth (28 %) have insufficient
One fourth (28 %) have insufficient
storage space
• The vast majority (79 %) have special
arrangements for communicating with
t f i ti ith
key executives during a natural disaster
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 4
5. Every Business Needs a Plan
Every Business Needs a Plan
• A generic DR plan is better than nothing, but it may stress elements that
A generic DR plan is better than nothing, but it may stress elements that
are less important to your business, or worse, leave out critical aspects
• Every organization, regardless of size or industry should have a Business
Continuity Plan (BCP).
o u y a ( )
• Needs vary from business to business and a good availability plan should
be designed for the individual business's needs
– Service Delivery / Call Center / eCommerce
Service Delivery / Call Center / eCommerce
– Manufacturing / Distribution
– Multi‐Site & International Operations
• A Business Continuity Plan is the least expensive insurance
A Business Continuity Plan is the least expensive insurance
any company can have (especially for small companies, as
it costs next to nothing to produce)
– Treat it as an investment not an expense
Treat it as an investment not an expense
– Many businesses NEVER recover from a serious incident
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 5
6. Taking Your Business Continuity
to the Next Level
• It is a huge mistake to
g
develop a business
continuity plan and Enterprise
not make it integral Availability
with your daily
with your daily
business operations
Business
• Availability planning is Continuity
an investment in the
continuing operations Disaster
of the business Recovery
• Transform your
Days Hours Minutes
y
Business Continuity
Business Continuity Functionality/Cost/Recovery Time Objective — RTO
Plan into an
Enterprise Availability Data‐Centric Business Function‐Centric
Plan
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 6
7. Enterprise Availability Plans
Enterprise Availability Plans
• The Process
The Process
– Understand Your Enterprise Requirements
– Prioritize and Map Enterprise Requirements
– Minimize the Gaps between Requirements and
Capabilities
– Test and Modify the Plan to Prevent Future Gaps
Test and Modify the Plan to Prevent Future Gaps
• The Results
– Incident Management Plan – Focused on Crisis Management
– Business Availability Plan – Focused on Work Area Recovery
– Technology Availability Plan – Focused on Technology Recovery
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 7
9. Document Past Downtime Events
Document Past Downtime Events
• A list of known downtime events Common Downtime Events
and their associated costs will
help you identify common (My Personal Stream of Misfortune)
problems and develop solutions
that will improve availability 24/7
that will improve availability 24/7
– Power Loss
– Communication Outage
– Hardware Failures
Hardware Failures
– Scheduled Maintenance
• Your physical location can have a Hardware Failure
lot to do with it Power Outage g
– Multi‐tenant Spaces Weather / Flood
– Construction Malicious Acts
– Weather Patterns Fire / Building
Software Failure
Other
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 9
10. Identify Systems And Recovery
Procedures
• Disaster Recovery Plan (You already have one right?)
Disaster Recovery Plan (You already have one, right?)
• How‐To Guides & Instructions
• Technology Profile
Technology Profile See Appendix for
– Team Members & Skill Sets Technology
– Systems Diagram
y g Profiles Examples
– Hardware Inventory
– DataComm Inventory
– Critical Applications
– PBX Configurations
– Vendors/Partners
– Vital Records
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 10
11. Define Business Functions
Define Business Functions
• Scope of Business Operations
Scope of Business Operations
– Locations (Single, Multi‐Site, International operations)
– Departments / Teams (How is the company organized?)
Departments / Teams (How is the company organized?)
– Processes / Tasks (What does the department do all day?)
– Schedules (Period Close, Peak Seasons, etc.)
– Dependencies (Order processing affected by credit dept.)
• Organization charts and process flow diagrams
can really help IT understand the business.
• Are there manual work arounds?
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 11
12. Identify Critical Business Requirements
Identify Critical Business Requirements
• Document internal key personnel and
Document internal key personnel and
functions (who is their backup?)
• Id if h
Identify who can telecommute
l
• Document external contacts
• Document critical equipment
• Identify critical documents
Identify critical documents
• Identify contingency equipment options
• Id tif
Identify your contingency location
ti l ti
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 12
13. Document Key Internal Personnel and
Functions
• Consider which job functions are critically necessary,
Consider which job functions are critically necessary,
every day, not just in an emergency
• Think about who fills those positions when the
primary job‐holder is on vacation
primary job holder is on vacation
• These are people who fill positions without which
y
your business absolutely cannot function – make the
y
list as large as necessary, but as small as possible
• Decide what non‐critical employees
should do in the event of a disaster.
should do in the event of a disaster
If there is no place for them to work,
will they be in the way of more critical
business functions?
business functions?
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 13
14. Identify Who Can Telecommute
Identify Who Can Telecommute
• Some people in your company might be perfectly
Some people in your company might be perfectly
capable of conducting business from a home office
• Find out who can and who cannot work remotely
• You might consider assuring that your critical staff
h d h l ff
(identified in Step 1) can all telecommute if
necessary y
• This is an easy piece that you can build
into your daily operations
• Key personnel who cannot telecommute
Key personnel who cannot telecommute
will likely need a workstation at your
contingency site
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 14
15. Document Critical External Contacts
Document Critical External Contacts
• Your business partners and vendors can really make or break
p y
your recovery
– Build a contact list that includes a contact information and a
description services they provide
– Include in your list people like the insurance company, attorneys,
l d i li l lik h i
bankers, IT consultants, electricians...anyone that you might need to
call to assist with various operational issues
– Don’t forget utility companies, municipal and community offices, the
g y p , p y ,
post office and FedEx/UPS.
• Keep a list of key customers who you
will want to notify in an emergency
• Create a “Yellow Pages” of external
contacts by function and a “White Pages”
by name
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 15
16. Document Critical Equipment
Document Critical Equipment
• Personal computers often contain critical information (You do
Personal computers often contain critical information (You do
have off‐site backups, don’t you?)
• Some businesses cannot function even for a few hours
without a Fax machine (i.e. 25% of orders come by fax)
• Do you have special printers you absolutely must have?
• What about security and encryption keys?
What about security and encryption keys?
• Do you have hardware license dongles?
• Don’t forget software – that would often
Don t forget software that would often
be considered critical equipment especially
if it is specialized software or if it cannot
be replaced.
b l d
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 16
17. Identify Critical Documents
Identify Critical Documents
• You need to have everything available that would be
You need to have everything available that would be
necessary to start your business over again
– Articles of incorporation and other legal papers
– Insurance policies, banking information, building lease papers
Insurance policies banking information building lease papers
– HR documents, government mandated records, tax returns
– Software Licenses, technical documents and source code
• Remember you might be dealing with a total facility loss
Remember, you might be dealing with a total facility loss
• You keep copies of your DR Plan off‐site, why wouldn’t you do
the same for your critical business documents?
• Store PDFs of critical documents on a
secure, off‐site server that you can access
g y
via the Internet in an emergency
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 17
18. Identify Contingency Equipment
Options
• IT Equipment
q p
– Where would you rent computers?
– Who can provide equipment such as servers on very short notice?
(i.e. CDW has same day service in Chicago)
– Are there components with a particularly long lead time? What are
the alternatives?
• Telecom
– Does your call center require special equipment?
Does your call center require special equipment?
– Can your telecom partner provide you with a loaner?
– What is the turn‐around time to set‐up a new phone
y
system?
• Other Equipment
– Can you use a business service outlet like Kinko’s for copies, fax,
printing, and other critical functions?
– Where would you rent trucks, air conditioners, generators, etc.?
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 18
19. Identify your Contingency Location
Identify your Contingency Location
• This is the place you will conduct business while your primary
p y y p y
offices are unavailable
– It could be a hotel, an adjacent vacant space, or even someone’s
home for a small business
– It could be another company office location
ld b h ffi l i
– Or a 3rd party site or mobile service like IBM or SunGard
– Perhaps telecommuting for everyone is a viable option.
• D idi WHERE t
Deciding WHERE to go depends on the needs of the business
d d th d f th b i
– How much space do you need?
– What facilities and services do you need?
– Will the facility be available to you in a regional disaster?
Will the facility be available to you in a regional disaster?
– What solution will get you back up and running fastest?
• Wherever it is, make sure you have all the
appropriate contact information and include
appropriate contact information and include
a map in your BCP
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 19
21. Conduct a Business Impact Analysis
Conduct a Business Impact Analysis
• Reveal vulnerabilities and potential risks of
Reveal vulnerabilities and potential risks of
worst case scenarios
• M
Measure impact on safety, finances,
i f fi
marketing, legal compliance, and quality
assurance
• Identify the organization’s business
unit processes and the estimated
recovery time frame for each business unit
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 21
22. Identify Risks And Exposures
Identify Risks And Exposures
• We confuse the concept of risk—the probability of success or
p p y
failure—with the concept of exposure— what is at stake
• From a business continuity standpoint, your risks are what is
likely to fail:
y
– Hardware failure (minimized with redundant hardware such as dual
power supplies, RAID arrays, clustered servers)
– Power failure (UPS and/or backup generator)
– C iti l d
Critical documents not stored in fire‐proof safe
t t t d i fi f f
• Your exposure is what is at stake:
– Lost data and information
– L
Loss of business, sales and revenue
fb i l d
– Government penalties (IRS, SOX, HIPPA)
• Understanding the risks and exposures of the business are
fundamental in setting priorities
fundamental in setting priorities
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 22
23. Priority Metrics
Priority Metrics
• Recovery Point Objective – RPO (data)
Recovery Point Objective RPO (data)
– The acceptable level of data loss exposure following an unplanned
event
– This is the point in time (prior to the disaster) to hich lost data can
This is the point in time (prior to the disaster) to which lost data can
be restored, typically the last backup taken offsite
• Recovery Time Objective – RTO (business process)
– The maximum acceptable length of time that can
elapse before the lack of critical business functions
severely impacts the viability of the business
– This is the total time required to recover critical services
– Measured form the time of disaster to resumption of
critical operations (a.k.a – Maximum Allowable Downtime)
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 23
24. Prioritize Your Requirements
Prioritize Your Requirements
• Are there existing Service Level Agreements (SLAs) in place?
• Each business unit should rank their business functions based on most
critical to the organization
– Financial Impact
– Operational Impact
O ti lI t
– Reputation Impact
– Regulatory Impact
• What are interdependencies between business units?
What are interdependencies between business units?
• Set Recovery Time Objectives (RTOs) for business functions and the
applications they depend on
– < 4 Hrs
– < 24 Hrs
– < 72 Hrs
– < 7 days
– 7‐14 days
7 14 days
– > 15 days
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 24
25. Prioritization Process
Prioritization Process
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 25
26. Associate Business Functions With
Applications & Data Sets
• Let the business set the recovery requirements, not the
y q ,
technical capabilities of the organization
– RTO for business function drives RTO for systems
– These gaps are natural
– Gaps will force the technology to improve to meet
the business needs
• Mapping is complicated process
– What are business process interdependencies?
Wh b i i d d i ?
– What are hardware/software dependencies?
– Organize applications in tiers based on business priority
– (10 departments X 10 task) X 5 applications X 10 locations = a very
(10 departments X 10 task) X 5 applications X 10 locations = a very
complex relationship
• You will quickly learn to
– Isolate what are the key resources to recover and in what order
Isolate what are the key resources to recover and in what order
– Build recovery strategies around those priorities
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 26
27. Build a Relationship Diagram
Build a Relationship Diagram
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 27
28. Evaluate Scenarios
Evaluate Scenarios
• What are the most likely scenarios you will face?
– Power Loss
– Telecom Interruption
– Hardware Failure
– Severe Weather
Severe Weather
• What are the most catastrophic scenarios?
– Regional Incidents
– Short Term Loss of Facility Availability
– Complete Facility Destruction
– Global Pandemic
• As your business changes, so will the
scenarios you face
scenarios you face
• It’s hard to prepare for every conceivable disaster, so start of with the
most common outages and work your way up
• These scenarios will be key in putting together your Enterprise Availability
Plan
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 28
29. Develop Strategies for Minimizing
Risk
• Do nothing; assume the risk
Do nothing; assume the risk
• Revert to manual processing
• Be self recoverable via multiple sites
• Contract with a hot‐site/cold‐site
vendor
• Contract a mobile recovery facility
Contract a mobile recovery facility
• Establish a quick‐ship agreement
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 29
30. There Is No "One Size Fits All"
Solution
• It is normal for there to be a mixture of
It is normal for there to be a mixture of
solution types within an organization
• B ild
Build a solution and plan specific to each
l i d l ifi h
business function
• Assume business and technology
requirements will evolve over time
• Think scalability
• Think flexibility
Think flexibility
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 30
31. Building Your Enterprise Availability
Plan
• So you have you a concept of what you will do in a disaster
So you have you a concept of what you will do in a disaster
and tons of supporting documentation, NOW WHAT?
• It’s not enough to just throw it all in a big binder and say
DONE!
• Start off with the Incident Management Plan
– In a crisis, it the first step to recovery
– Most “good practice” standards specify Incident Response
planning now (Sarbanes‐Oxley, ISO, IEEE, ITIL, Payment
Card Industry, etc.)
– Developing your response as an incident is occurring
Developing your response as an incident is occurring
probably will create more stress, cost more, take
more time and not be as well executed
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 31
32. Incident Management Plan
Incident Management Plan
• The Incident Management Plan (IMP) is your Management Playbook
– An enterprise‐wide action plan to help your senior management effectively
and efficiently respond to an incident.
– Your plan includes checklists of required activities, an explanation of roles and
a definition of your resources
y
• Incident Management Coordinator is the Quarterback
– Management Action Team
– Damage Assessment Team
– R
Recovery Team
T
• Incidents usually require a time‐sensitive response –
if staff don’t know what to do, critical information and
options may be lost
• Under stress it is good to know who is capable and
permitted to decide time‐critical issues
• Have an Incident Operations Hub (the “War Room”)
with specific outgoing channels and messages
with specific outgoing channels and messages
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 32
33. Incident Management Overview
Incident Management Overview
Command Control Communicate
Impact Select Post
Pending Incident Escalate Mobilize Plan Recovery
Assess- Recovery Incident
Crisis Occurs to Mgmt Response Execution Mgmt.
ment Plan Analysis
• Establish command and responsibility for managing the
incident then mobilize the response
• Determine the scope of the issue, set priorities, appropriate
Determine the scope of the issue, set priorities, appropriate
response, and take control the overall recovery process
• Coordinate internal and external communication
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 33
34. Business Availability Plan
Business Availability Plan
• Business Availability Plan (BAP) is an action plan
focused on maintaining the availability of critical
business processes when situations—ranging from
minor outages to major disasters—threaten to
disrupt them.
• A detailed series of responses, checklists and
action steps to deal with situations that might
otherwise affect routine work activities
• Each business unit or department should have
Each business unit or department should have
their own plan that meets their particular needs
and rolls up into the Enterprise Plan
• Individual plan also allow you to spread the work around and make it
relevant to the business process owners
relevant to the business process owners
• Added Bonus : You’re better prepared to meet regulatory, legal and
internal audit compliance requirements, with thorough documentation
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 34
35. Getting Your Business Restarted
Getting Your Business Restarted
• Business Function Priorities
– What are the functions most critical to the operation (Consistent with
your technology recovery priories?)
– What processes can be done manually?
• Facilities
– Where can the employees work?
– How do they get there?
• Workstations
– What office equipment do we need?
– What supplies to we need to function?
• Vital Records
– What documents do we need to function?
– How do we write and deposit checks?
– Where is our insurance policy?
h i i li ?
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 35
36. Technology Availability Plan
Technology Availability Plan
• Your existing disaster recovery plan is a good starting point for building a
Technology Availability Plan (TAP)
• It’s a defensive measure that prepares your IT management and team
members to respond to—even help prevent—interruptions
• All‐inclusive, it covers your entire infrastructure as well as
All‐inclusive it covers your entire infrastructure as well as
telecommunications, systems, applications and data within the data
center.
• A detailed series of action steps, activity checklists, personnel role
definitions, resource identification
d fi iti id tifi ti
• Technology recovery priorities
• Benefits of a comprehensive TAP
– Better preparedness for IT disruptions
Better preparedness for IT disruptions
– More agile, more effective response
– Reduced severity and duration of incidents
– Greater ability to mitigate risk—and the
associated increased confidence
associated increased confidence
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 36
37. Lessen the Gaps
L th G
Between Capabilities
Between Capabilities
and Requirements
38. For Most Businesses, 100%
Availability Is a Myth
• In a perfect world, you would have 100% availability, but who
In a perfect world, you would have 100% availability, but who
can afford complete redundancy?
– Smaller business have tighter budgets, but tend to be less complicated
– Large corporations have higher requirements and budgets
Large corporations have higher requirements and budgets
– The mid‐market tends to be in the most challenging position
• The most we can hope for is to lessen the gaps between the
needs and capabilities of the business
needs and capabilities of the business
• How do you make it a reality?
– Management Buy‐in and Support
– Allocation of Resources
– Build Availability into Systems
– Hard Work and Persistence
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 38
39. Selling Availability In Your
Organization
• Management Education
Management Education
– Downtime impact on the business
– Informed managers make better decisions
• Risks and Exposures
Risks and Exposures
• Goal: RTO/RPO acceptance
– What managements needs to approve
– Communicate in business terms ($$$)
• Cost of Ownership
– Initial costs
– Ongoing costs
• Return on investment
– R
Recoverability & More Uptime
bilit & M U ti
– Customer Service / Satisfaction
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 39
40. Cost Of Downtime Analysis
Cost Of Downtime Analysis
• The more complex
The more complex
your environment the
more resource
intensive and
i i d
expensive it is to keep
available
• High availability is not
cheap, but that is
nothing compared to
a business Cost of Prevention
interruption
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 40
41. Example: Downtime Cost to a
$500M Organization
Cost of Outage = $250K/Hr
Length of Outage w/o Preparation (5 days) = $10M
Length of Outage w/ Preparation (1 day) = $2M
SAVINGS = $8M
Cost of Preparation = $75K/year
Odds of Outage 1 in 25 = 4.0%
4.0% x $8M = $320K
Prevention is actually quite cost effective!
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 41
42. Determining ROI Of Availability
Determining ROI Of Availability
• 'Disaster Driven' ROI Solutions…
Disaster‐Driven ROI Solutions
– If Your Business Continuity Solution only
Addresses UNPLANNED, UNPREDICTABLE
Addresses UNPLANNED UNPREDICTABLE
DOWNTIME (Less than 5% of Downtime) it will
take a disaster to find ROI
• ROI from Everyday Solutions
– If Your Business Continuity solution also
If Your Business Continuity solution also
addresses PLANNED, PREDICTABLE DOWNTIME.
(
(95+% of All Downtime), you'll find everyday ROI
), y y y
without the disaster!
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 42
43. Achieving Management Buy In
Achieving Management Buy‐In
• Management support of availability solutions requires
g pp y q
understanding the business requirements
– What are the drivers of the business? Speak the language of business
not just IT
– What is the cost of downtime?
h i h fd i ?
– What are the other non‐technical effects of business interruption?
• Availability is an investment, not an expense
– B ild b i
Build a business case to invest in availability solutions
i i il bili l i
– What is the ROI from implementing availability
solutions?
• Strike when the iron is hot there is no
Strike when the iron is hot, there is no
better time to pitch availability than
after an outage (even a small one)
• Build consensus form the bottom up and the top down
Build consensus form the bottom up and the top down
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 43
44. Keep Current: Update
K C t U d t
Your Plan to Prevent
Your Plan to Prevent
Gaps from Developing
45. Putting it All Together
Putting it All Together
• Your AEP is useless if all the information is scattered
Your AEP is useless if all the information is scattered
about in different places
• Make it easy to update
• Make plenty of copies and give one to each of your
key personnel
– Make hard‐copy emergency “grab binders”
– Keep copies on USB flash drives
• Keep several extra copies off‐site
Keep several extra copies off‐site
– Keep copies at home, in your car, and/or in a safety‐
deposit box.
– Upload a copy to a web‐accessible server hosted off‐site
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 45
46. Communicate, Communicate,
Communicate
• Share your plan don’t just lock it in a
Share your plan, don t just lock it in a
desk drawer!
• M k
Make sure everyone in your company
i
is familiar with the Availability Plan
• Hold mandatory training classes for
every employee whether they are on
the critical list or not
p y y
• Keep availability on everyone's radar
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 46
47. Test Your Plan
Test Your Plan
• You’ve put really good ideas down, accumulated all your
You ve put really good ideas down, accumulated all your
information, identified contingency locations, put your
contact lists in place, but can you pull it off?
• One thing you will definitely learn in the test is that you
One thing you will definitely learn in the test is that you
haven’t gotten it all just exactly right
• Don’t wait until disaster strikes to figure out what you
should do differently next time
should do differently next time
• If you make any major changes, run it again
• Even after you have a solid plan, you should
test it annually
• Run desktop simulations: call your team into a
conference room and run through a mock disaster
g
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 47
48. Plan to Change the Plan
Plan to Change the Plan
• “No battle plan survives contact with the
No battle plan survives contact with the
enemy.” ‐‐Helmuth von Moltke the Elder
• No matter how good your plan is, and no
h d l i d
matter how smoothly your test runs, it is likely
there will be events outside your plan
there will be events outside your plan
– The hotel that was to be your DR site is booked up
– A key member of the recovery team is on vacation
y y
– Your backup tape was defective
– The one weekend, you leave your laptop
at the office, the building burns down
at the office the building burns down
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 48
49. Review, Revise and Redistribute
Review Revise and Redistribute
• Every time something changes update all copies of your EAP
Every time something changes, update all copies of your EAP
– New hardware / new software
– More importantly…new business processes
• Constant updating can be time consuming, consider using a
software tool to manage and update your plans
• Schedule regular reviews of your plan and stick to the
Schedule regular reviews of your plan and stick to the
schedule
• Never let it get out of date…It is a living document
• An out‐of‐date plan can be worse than useless: it makes you
feel safe when you are anything but!
feel safe when you are anything but!
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 49
50. IT Continuity of Operations: Lessons
Learned
• Get out of your comfort zone and focus on the business, not
y ,
just technology
• Embrace availability as a discipline
or methodology gy
• Build higher availability into every project
• Business needs will change over time
• Think flexibility scalability
Think flexibility, scalability
• Strive for continuous improvement
• Test frequently
• Y d ’ l
You don’t always need a million dollar solution, but you need
d illi d ll l i b d
an annual budget
• No matter how prepared you think you are, the unexpected
will always happen…Murphy was an Optimist!
ill l h M h O ti i t!
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 50
51. Questions & Answers
Questions & Answers
“I always tried to turn every disaster into an
I always tried to turn every disaster into an
opportunity.” ‐‐ John D. Rockefeller
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 51
53. Where Else Can I Get Information?
Where Else Can I Get Information?
• Web Sites
Web Sites
– www.drj.com
– www.contingencyplanning.com
ti l i
– www.globalcontinuity.com
– www.recovery.sungard.com
– www.disaster‐resource.com
– www.businesscontinuitytoday.com
• Professional Organizations
• Consultants
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 53
54. More of My Favorite DR Pages
More of My Favorite DR Pages
• Downtime Calculator
Downtime Calculator
– www.visionsolutions.com/Solutions/Disaster‐
Recovery‐toolkit‐downtime‐calc.aspx
y p
• Glossary of Terms
– www.continuitycentral.com/DRGlossaryofTerms.p
df
• Business Continuity and Resiliency Self‐
y y
Assessment Tool
– www.ibm.com/services/us/bcrs/self‐assessment
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 54
55. Document Collection Worksheets
Document Collection Worksheets
• Applications • Vital Records
• Computer
p • Employee Contact
p y
Equipment Info
• Offi E i
Office Equipmentt • E l
Employee Call Trees
C ll T
• Telecom/Voice • 3rd Party Info
• Office Supplies • Alternate Site Space
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 55
56. Applications
– Business Function
Business Function
– Recovery Priority
– Application RTO
– Manual Procedures in Place
– Inter‐dependant Applications
– Vendor
– Version
– # Licenses
– Install Key
– Serial Number
– Media Off Site
Media Off Site
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 56
57. Computer Equipment
Computer Equipment
• Function
• IP Address
• Description
• Service Tag / Code
• Warranty expires
• OS / Service Pack
OS / Service Pack
• Memory
• Hard Drive ‐ number & capacity
• Specialty cards
• Applications supported
• Business function
Business function
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 57
58. Telecom/Voice
• Site Name
Site Name
• Circuit Size
• Equipment
• Circuit ID
• Vendor
• Contact Number
Contact Number
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 58
59. Vital Records
Vital Records
• Description
• Location
• Required By
• Responsible Party
p y
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 59
60. Employee Contact & Call Trees
Employee Contact & Call Trees
• Name
• Role / Title
• Address
• Phone
– Office Phone
– Cell Phone
– Alternate Phone
• E‐mail
– Office E‐mail
Office E mail
– Personal E‐mail
– Alternate E‐mail
• Expertise / Notes
Expertise / Notes
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 60
61. 3rd Party Info
3rd Party Info
• Name
• Customer #
• Telephone
• Contact
• Comments
• Service / Product Provided
Service / Product Provided
• Used in this Recovery Activity
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 61
62. Alternate Site Space
• Workstation Type
Workstation Type
– Hardware/Software
– Phone
• Shared Resources
– Phone System
–Pi
Printers
– Faxes / Copiers
• Seats required by department
Seats required by department
– Match to RTOs (24 hrs, 72 Hrs, etc)
– Not everyone needs to be there Day 1
March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 62