2. DEFINITION OF TERMS
Business Continuity Planning is a framework that addresses all of
the following:
Disaster Recovery Planning (IT systems focused)
Emergency Response Planning – specific action plans to a variety of
foreseeable emergency situations in order to protect employees, tenants,
the public and the organization’s assets.
Business Resumption Planning (again, IT focused, based on workaround
during DRP)
COOP – restore critical functions of the business for a short duration of
time, typically 30-60 days
3. WHY CARE?
Gartner estimates that 2 out of 5 enterprises that
experience a disaster go out of business within 5 years.
Some significant disasters are rather likely.
On a more positive note, it’s a great mechanism to get to
know your businesses key processes and the people
involved.
A well developed and tested BCP becomes an asset to
the company, worthy of mention in the annual report.
Companies can use this expertise to add value to their
customer base.
5. SOME PHILOSOPHICAL POINTS
The best is the enemy of the good
Avoid credenzaware
Get double duty: overlap with SOX audit requirements
Craft your IT Architecture around requirements identified during
Business Continuity Planning
You can outsource systems, but not responsibility. It is still your duty to
perform a BCP on these key systems as well. Outsourcing does help
distribute the risk, but look for SAS70 II audits, failover sites, at least an
annual test of failover, and contractual obligations with financial
penalties against SLAs.
6. THE 6 STAGES OF BCP
1) Understand Risks and “quick win” actions to mitigate
2) Perform a business impact analysis per risk
3) Develop a BC strategy
Recovery Requirements Identification
Recovery options identification
Availability Time Assessment
Cost Capability Assessment
4) Develop a BC plan
Initial Notification and Response
Problem assessment and escalation
Disaster declaration
Plan implementation logistics
Recovery and resumption
Return to Normal
5) Test the BC plan
6) Maintain the BC plan
17. OUR APPROACH
Get Board approval and project review
Follow a formal PM methodology
Use SharePoint or other collaboration tool
Get representatives from all parts of company
Negotiate a realistic timeframe
Have a deliverables focus and concentrate on drillable and
verifiable response plans
18. STAGE 1: UNDERSTAND RISKS
A Limited Initial Set of Scenarios
IT
Loss of 1 to3 servers
Loss of 4 to 5 servers
Loss of entire HQ Data Center
Loss of communications services
Business
Loss of access to HQ
Loss of access to Regional Office
Death of a key executive
19. For each scenario and by department, key personnel and key
processes determine the current impact of each scenario.
AP04 - A/P Voucher Processing
Accounts
Payable
Supervisor
Accounts
Payable
Accounting
Assistant
#2
Accounts
Payable
Accounting
Assistant
Accounting
(Property,
Corporate
or
Construction)
Send invoices to
A/P for Processing
(1)
Review for
completeness –
coding &
approvals
(2)
Incomplete
Key vouchers into
Jenark
(4)
Vendor in
System
(4)
Voucher entry
complete
(4)
Print Batch Edit
Report
(5)
AP03 –
Invoice
Prep
Yes
AP01 – New
Vendor
Process
No
Post Vouchers in
Jenark
(9)
AP05 – Check
Processing
Sort invoices by
Vendor Name
(3)
Peer Review
Batch
(7)
Forward Batch
Edit, invoices for
review
(6)
Batch Edit, and
invoices held for
check printing
(10)
Review Batch
(8)
20. STAGE 3: DEVELOP A BC
STRATEGY
For each scenario and by department, key personnel and key
processes, determine RTO and RPO – how much downtime and
how much data loss we view as an acceptable risk.
Layout the recovery options for work area, IT infrastructure,
personnel and key records and data.
Select the most appropriate recovery options to fit RPO and RTO
objectives most cost effectively
21. STAGE 4: DEVELOP A BC PLAN
Technology Recovery Plan. Recommendation includes a warm
failover site, a second SAN, VMWare ESX virtual servers with
SAN replication of
Communication plan: features Workspeed Notify, an emergency
broadcast system with identification of users, groups, disaster
templates. Starting to use this in conducting drills
Alternate work location plan. Citrix environment means our users
can access from anywhere.
Document management initiative addressing where some of our
critical documents only exist in paper.
22.
23. Business Continuity Policy
Inventory of existing BC/DR/CM Documents
Inventory of Data and Services with Impact Analysis
Inventory of Systems (access currently limited to IT)
Company Firewalls
CompanyLAN
…
Inventory of Processes
Backoffice Process Map
Personnel Plan (KCE, alternate work locations)
Evacuation and Shelter-in-Place Plans
HQ
Regional Offices
Atlanta
Chicago
…
Communication Plan
Tenant Contact List
Workspeed Notify Email To Executive Sponsors
Templates
Death of an Executive Communication
Template
Loss of Access to HQ Communication Template
Loss of Access to Regional Office
Crisis Response Plan
Standard Building Supply List
Technology Response Plan
Checklists
Team Training
Employee and Manager Training
Company Drill Plans and Results
On-Going Support Model
Deliverables Table of
Contents from our
SharePoint Wiki
24. STAGE 5: TEST THE BC PLAN
Test alternate work locations. Actually spend a day with people
working in their alternate locations. You will find things you
missed.
Test emergency communications systems. Review test results
and correct. Wherever possible, make the maintenance of contact
information integrated with your corporate directory.
Test technology failover. When you are ready, physically power
down your data center and bring the failover data center on-line.
Time the exercise. Document results. Again, you will find things
you missed.
25. STAGE 6: MAINTAIN THE BC PLAN
Communications plan refresh – people and roles change
External contact refresh
Addition of new scenarios to handle
Routine testing with evaluation of performance feeding
changes
26. International Organization for Standards ISO 17799
National Fire Protection Association NFPA 1600
FURTHER READING AND RESEARCH
SUGGESTIONS
Editor's Notes
So let’s get past that name, and instead talk about Prior Planning Preventing Poor Performance.
Before we jump right in, let’s make sure we’re talking about the same thing. I think of BCP as a framework that embraces these other areas. I’m a CIO, and so I tend to think first about our IT infrastructure. And these are important components to BCP, be we also have to discuss the processes that make up our businesses, the people and their roles within these processes, and how we can ensure delivery in the face of disruptive events.
For example, one of the things that sometimes surprises me is how some retail outlets are incapable of taking my money for a purchase if they are experiencing a power outage or other problem with their POS systems. They haven’t thought through the key process of “get customers money” and how to execute that process in the face of loss of power.
Sometimes, like with some SOX audit requirements, I’m cynical and think a lot of what I’m asked to do is non-value-add CYA.
But this is not the case with BCP.
Not only can prior planning prevent poor performance, it can prevent pain. Deep, enduring pain.
What I will ask you to do in a moment is imagine an event in your life. A bad event, potentially a catastrophic event.
And what I hope to convince you of is that prior planning for this event or an event like it can make the difference between being mildly annoyed or being faced with catastrophic consequence where your life is never the same.
Before I jump right in, let me get a few points of philosophy out of the way.
There are different ways to slice up the problem of BCP into stages, and here’s the approach we’re using at Company
So let me ask you this….
Tonight, say 2 in the morning, what would happen if your house caught on fire?
It’s an awful thought and I apologize for putting it in your head. It’s relatively unlikely. But not as unlikely as you might think. I have a nephew who is starting out as an actuary at an insurance company and he told me that about 1 in 40 us will experience a severe house fire in our lifetime.
And if you do nothing about it before it happens, then you will end up on the awful, life is never the same end of the pain continuum.
So, what can and should you do in order to make this awful experience the most bearable it can be?
Well, first, do everything you can to avoid this disaster. I’m sure you all have smoke detectors, but let me ask you to think about the last time you tested them? Are they loud enough to wake you up? Are they connected to an alarm system that will contact the fire company?
You should really have fire extinguishers on hand too. They should be distributed around your house. You should practice with one to make sure you know how to operate it. You should replace them when the indicator says so.
And you should really have your irreplaceable documents – and your insurance policy and contacts – in a safety deposit box.
So maybe this 2 am house fire is a hassle – it wakes you up, you get everyone to safety, and you put it out. Maybe all you’ve lost is a little sleep and made a mess that you’ll have to clean up.
But let’s imagine that, despite your best effort to avoid a bad fire, it all happens very quickly and you awaken to a screaming alarm and billowing smoke.
You need to respond and you need to respond now!
But if you haven’t planned, and more importantly practiced your response in advance, when things are calm and you’re clearheaded, then your chances for an effective response during the real emergency grow frighteningly small.
So for me, I know that the first priority is to get all of the people out. I know that there are 4 windows we can climb out and a porch railing that it’s possible to climb down. But as I was putting my thoughts together for this presentation, I realized that I really needed to get one of these fire escape ladders. My step-dad is 81 years old – if he’s staying over there’s no way he’s climbing down my porch railing.
After the people are out, the next priority is my dogs. After that is stuff. My wife has a small, noisy, dirty Amazonian parrot, and in my plan anyway that little bastard is on his own.
Most stuff is replaceable. I make sure to backup my computer files and all of my pictures to DVDs and put those in our safety deposit box. I can’t lie – there are some family heirlooms and other things that would be awful to lose. But our family would carry on – we would continue
So if you get nothing else out of my presentation today, please go home and make your plan. Make sure everyone in the family knows what that plan is. Risk being dorky and have a fire drill at home. For 9 of you listening to me today, your prior planning will help you and your family survive the disaster of a house fire
Well things improve over time. Today, there is a national standard for fire hoses and hydrants. They are all interchangeable.
But the thing that drives change are the lessons learned from living and working through a disaster. So again, I will say that prior planning and practice prevents poor performance.
And still, we have huge wildfires that seem to rage through California every year. We saw plans that hadn’t been properly tested utterly fail New Orleans and the Gulf Coast during Katrina.
To paraphrase a bumper sticker, “Stuff Happens”
But let’s imagine that, despite your best effort to avoid a bad fire, it all happens very quickly and you awaken to a screaming alarm and billowing smoke.
You need to respond and you need to respond now!
But if you haven’t planned, and more importantly practiced your response in advance, when things are calm and you’re clearheaded, then your chances for an effective response during the real emergency grow frighteningly small.
So for me, I know that the first priority is to get all of the people out. I know that there are 4 windows we can climb out and a porch railing that it’s possible to climb down. But as I was putting my thoughts together for this presentation, I realized that I really needed to get one of these fire escape ladders. My step-dad is 81 years old – if he’s staying over there’s no way he’s climbing down my porch railing.
After the people are out, the next priority is my dogs. After that is stuff. My wife has a small, noisy, dirty Amazonian parrot, and in my plan anyway that little bastard is on his own.
Most stuff is replaceable. I make sure to backup my computer files and all of my pictures to DVDs and put those in our safety deposit box. I can’t lie – there are some family heirlooms and other things that would be awful to lose. But our family would carry on – we would continue
So if you get nothing else out of my presentation today, please go home and make your plan. Make sure everyone in the family knows what that plan is. Risk being dorky and have a fire drill at home. For 9 of you listening to me today, your prior planning will help you and your family survive the disaster of a house fire
And when the fire is out and the firefighters go home, you still have a lot of work ahead of you. Someone may need medical assistance; you’re going to need to contact your insurance agent. You may need an alternative place to live for days or weeks or months.
So prior planning prevents poor performance.
And if this presentation was brought to you by the letter P, I could summarize that you should:
Prevent – avoid disasters as much as you possibly can
Prioritize - what’s most important to you – people, pets, paperwork
Have a plan of action in case your prevention measures didn’t work. Have that plan be detailed and up to date. If your plan calls for you to have things on hand like fire extinguishers or escape ladders, then make sure you have those things on hand
And perhaps most important but least done – practice. I can almost guarantee you that there are things missing from your plan. You can either find that out in a real, live crisis or you can find it out in a practice drill
Another local disaster, although it happened in 1904, is instructive about planning and practicing.
There’s a picture of the aftermath of the Baltimore fire of 1904.
At 10:48 in the morning firefighters responded to an alarm at a warehouse. As they pulled up to the scene. The roof blew off and showered burning embers onto the neighboring roofs.
By noon things were out of hand and the entire Baltimore fire brigade was busy. They called Washington and surrounding counties for assistance, and they got on the scene by about 1:30 in the afternoon.
But here was the awful discovery – Washington’s hoses did not fit onto Baltimore’s hydrants.
And so the aftermath was that in just 30 hours, 140 acres, 1526 buildings and 2500 businesses burned to the ground. Think about that - 1,526 buildings – that’s more than 6 times COPT’s entire portfolio, all wiped out in 30 hours.
So that personal example is really explanatory of the steps you take in protecting your business. Let me run you through what we’re doing at Company
I routinely have to remind people that this is the set of disasters that we agreed to address initially. You get a bunch of people in a room discussing this stuff, it’s so easy to start imagining so many different threats.
But I remind them, let’s address this set now, test and practice our response plans, and then we have a solid base that we can add to. It’s important to recognize that this is an on-going process, it is not our one and only chance to get it right. But if we keep on shifting the finish line, we’ll never end up with something actionable.
We’ve been fortunate in being able to leverage our ERP systems replacement work. We’ve conducted in depth interviews and documented our processes and workflows in a comprehensive and consistent way.
Approach testing with a mindset that you will find things wrong. If you don’t find many areas to improve, you didn’t test hard enough.
