"93% of businesses that suffer a significant loss of data are out of business within 4 years." The Bureau of Labor Man-made and natural disasters happen all the time. How resilient is your business? Even if you have a disaster recovery plan, are you really prepared for the unexpected?

1. You Have a Disaster Recovery Plan:
So, Don’t Let a Disaster Defeat Your
Business
MWLUG
August 22, 2012

2. No one is safe from
Mother Nature!
2 August 29, 2012 Velocity Proprietary and Confidential

3. Sobering Statistic
93% of businesses that suffer a significant
loss of data are out of business within 4
years.
The Bureau of Labor

4. Disaster Preparedness Checkpoint #1:
Are Disasters Viewed as a Real Threat?
Disasters never really happen.
It’s just a vendor sales gimmick…
4 August 29, 2012 Velocity Proprietary and Confidential

5. Why Is This Area “ Vital ” ?
Expectations of the IT Services are demanding
Technology is an enabler of business
Penalties are becoming more severe
Business is becoming more competitive
Businesses rely more heavily on their enterprise applications
than any other time in history
5

6. What is a Disaster ?
ANYTHING !!!!
That stops your business from functioning & that cannot be
corrected within an acceptable amount of time….

7. What is Disaster Recovery?
Reaction to a sudden, unplanned event that enables an
organization to continue critical business functions until
normal business operations resume.
“…It is not enough to arrange for hardware replacement;…
planning must address continuation of business operations, or
business continuation.”

8. Common Issues with Disaster Recovery Planning
We tend to let our guard down when times improve.
As planners, we must always be ready and prepared.
• Are the expectations of your DR plan realistic?
• Has your plan kept up to date with your IT integrations?
• Do you have a qualified staff to execute the plan?
• Is DR integrated into your Change Control process?
8 8
August 29, 2012 Velocity Proprietary and Confidential

9. Protect Important Assets
Primary assets needed to operate Information Systems:
• Hardware and networks can be replaced
• Facilities can be rebuilt or relocated
• Your Data is Priceless !
• People are assets too!
9 9
August 29, 2012 Velocity Proprietary and Confidential

10. Checkpoint # 2:
Of course we have a DR plan… Somewhere
Plan it yourself or get someone to do it,
but whatever you do.. Plan!
10 August 29, 2012 Velocity Proprietary and Confidential

11. DR Plans Come in Many Flavors
To be on holidays when the
big one hits
Cross your Fingers
RESUME
Resume stored offsite
11

12. No / Incomplete Disaster Recovery Plan
Guarantees:
• Confusion
• Lack of direction
• Conflict
• Lost Customers or credibility
12

13. The Products of the Plan
Who will execute recovery actions
What is needed to continue, resume, recover or restore business
functions
When business functions and operations must resume
Where to go to resume corporate, business & operational functions
How detailed procedures for continuity, resumption, recovery or
restoration
CLASSIC: WHO-WHAT-WHERE-WHEN-HOW
13

14. Checkpoint # 3:
Ensuring Your Plan Meets Your Business Needs
Does your business know what
your IT department can deliver?
14 August 29, 2012 Velocity Proprietary and Confidential

15. What Are Your Recovery Objectives?
Wks Days Hrs Mins Secs Secs Mins Hrs Days Wks
Recovery Point Recovery Time
RECOVERY POINT RECOVERY TIME
OBJECTIVE OBJECTIVE
Recover data to last known good The amount of time it takes to
point before outage recover production systems
and resume operations
15 August 29, 2012 Velocity Proprietary and Confidential

16. Recovery Solution Examples
RPO - Recovery Point Objective
24 Hours RPO - Tape Recovery
RTO - Recovery Time Objective
48 hours RTO – Systems Availability
16 August 29, 2012 Velocity Proprietary and Confidential

17. Categorize Your Technology Inventory
as Critical vs. Necessary vs. Not at All
Typically look for a 60, 20, 20 breakdown.
• 60% of all your servers are critical for recovery
• 20% are necessary but not required immediately
• Remaining 20% are optional or not required at all.
This means you can focus on 60% not the 100%.
17 August 29, 2012 Velocity Proprietary and Confidential

18. Email
It has become Mission Critical.
It represents a way you communicate with
customers,
partners, employees, and the world.
Imagine no Email !!
Should EMAIL be an H/A application?
18 August 29, 2012 Velocity Proprietary and Confidential

19. Traditional Backup Process
Traditional Backup & Recovery Process
Offsite Storage Backup Offsite Storage
Software ( Bonded Provider )
Facility
19 August 29, 2012 Velocity Proprietary and Confidential

20. How Much Data Can You Afford to Lose?
• Lost data is lost money!
• Most IT shops depend on tape backups to protect their data
• Data considerations for tape
• Minimum of 24 hrs. of lost data
• Application access may be more critical
20 August 29, 2012 Velocity Proprietary and Confidential

21. Recovery Timeline Sample
True recovery process comparison
24 + hrs
Configure Install Configure Load each and Every Install
hardware OS OS application from CD / backup
Tape, FTP etc. agent
Restore Power
VM on VM < 2+ hrs
• Successfully deliver RTO objectives
21 August 29, 2012 Velocity Proprietary and Confidential

22. Does your Business Know What IT Can Deliver?
22 August 29, 2012 Velocity Proprietary and Confidential

23. Management Awareness
Management shocked that the IT folks lost all orders
from previous day.
48 Hour RTO - Who agreed to this ?
Do you have a new Senior Management Team ?
23 August 29, 2012 Velocity Proprietary and Confidential 23

24. Checkpoint #4: Backups
Build it and they will come.
Put into practice what you
say you will do.
24 August 29, 2012 Velocity Proprietary and Confidential

25. Your System Just Lost All of Its Data!
Are you worried ?

26. Tape Backups - Pros
Easy to run
Easy to manage – Tape Management Products
Easy to Automate – Automation Software Solutions
High Speed & High Capacity reduced backup windows
Capacity of tapes reduces number of Tapes
It’s the way we have always done things
Virtual tape libraries
Local Electronic Backups

27. Tape Backups - Cons
• Media, interface, or drive errors
• Seldom validated – How do you know your backups are valid?
• Time to ship, travel and restore tapes at Hotsite in a disaster
• Loss of data because backup only runs once per day
• Backups omitted in lieu of system access
• Lost tapes – Misplaced!
27 27 ugust 29, 2012
A Velocity Proprietary and Confidential

28. Backup = Recovery
How many people backup their system ?
38% are in-complete
19 % iSeries ( AS400) are un-recoverable
32 % Windows are un-recoverable
30 % Oracle DB are un-recoverable
15 % Virtual Infrastructure un-recoverable
28 August 29, 2012 Velocity Proprietary and Confidential 28

29. “Backups ran fine. I never heard of any issues.”
• With a sign off, the process implies correctness
• Many backup solutions are partially broken
• Backup Success Rate = 97%
99
98
Failure rate = 3 %
97
96
95 This equals 11 days per year
94
93
with NO Backups
1 2 3 4 5 6 7 8 9 10 11 12
August 29, 2012
29
29 Velocity Proprietary and Confidential

30. Sample Log
• Starting SAVDLO of folder *ANY to devices TAP01.
• 2574 document library objects saved.
• Starting save of list *LINK to devices TAP01.
• 43917 objects saved. 342 not saved.
• Save of list *LINK completed with errors.
• Starting save of media information at level *OBJ to device
• 18 objects saved from library QUSRBRM.
• Save of BRM media information at level *OBJ complete.
• DAILY *BKU 0070 *EXIT CALL PGM(BBSYSTEM/ENDDAYBU).
• Control group DAILY type *BKU completed with errors.
30

31. Backup/Recovery Best Practices
• Examine your backup strategy and policies for data
• Review backup strategy and policies for server infrastructure
Document server related information with signoff
• Are your network configurations backed up?
• Review off-site tape and vital records storage strategies
• Have you audited your backups?
• Can you demonstrate recovery procedures for daily, weekly, and
monthly backups?
31 August 29, 2012 Velocity Proprietary and Confidential

32. Checkpoint #5:
Building a Recovery Team
Getting the right people:
Do they have the Right Stuff?
32 August 29, 2012 Velocity Proprietary and Confidential

33. Perspective
Too often, companies populate their DR teams with raw or
inexperienced staffers
Volunteers to satisfy an auditor or the sacrificial lamb

34. Ideal Team Member Characteristics
• Considered an expert by his or her peers
• A go-to person for anything and/or everything
• Works well under pressure
• Sense of urgency, willing to fix problems created by others
• Confident
• Trusted by peers
• Dedicated – A company person
34 August 29, 2012 Velocity Proprietary and Confidential

35. The Effects of an Unprepared Recovery
Staff Are Real
Recovery is only possible if your staff is available
to put IT back together again.
Disasters affect people in unpredictable ways:
• It can devastate people both physically and mentally
• Can effect others around them
• Make the individual unable to function
• Emotional breakdown
Respect the situation!
35 August 29, 2012 Velocity Proprietary and Confidential

36. The Role of Executives
Executives are not typically involved and should NOT be.
Why:
• Executive Pressure is hindering
• Intimidating
• Often Nasty
• Glorified Experts
The team must provide regular status reports & the
Executives should be accessible when required.

37. Best Practices During a Disaster
• Feed your staff - Have snacks handy
• Stay away from High Energy Drinks
• Enforce no shift duration to exceed 12 hours
• Encourage people to take breaks
• Provide distractions during breaks

38. Irene was no lady!
Family Comes First
People Evacuated
2, 465, 000
Until the basic personal needs are met
• Family comes first. Always !!
• Staff members will not focus on the Enterprise recovery
• Staff members may or will not be available

39. Checkpoint #6:
Is Your Data Offsite?
It’s eleven o’clock.
Do you know where your data is?
39 August 29, 2012 Velocity Proprietary and Confidential

40. Recipe for Disaster
• One average filing cabinet
• Your recovery tapes/CD’s
• One average business fire
40 August 29, 2012 Velocity Proprietary and Confidential

41. Recipe for Disaster
Directions:
• Place media in filing cabinet.
• Bake in a fire for 20 minutes
at 800°
• Let cool.
• Open filing cabinet.
41 August 29, 2012 Velocity Proprietary and Confidential

42. Storing Backup Media on Site …
Your Business Is Toast!
42 August 29, 2012 Velocity Proprietary and Confidential

43. Offsite Strategy
DAILY EXPOSURE DISASTER
STRIKES
10 AM
W T F S S M T W T F S S M T W T F S S
Full Incremental backup PLANNED
Backup OFFSITE
BACKUP
POSSIBLE
UNPROCESSED
DATA
OPERATIONS RECOVERY
43 August 29, 2012 Velocity Proprietary and Confidential

44. Checkpoint # 7: Virtualization
Practice smart virtualization
44 August 29, 2012 Velocity Proprietary and Confidential

45. Compute
Storage

46. Physical Wintel Recovery Challenges
Today
Production Server “Boot & Pray”
Application cd, tape or Application
ghost image
OS WAN OS
OS files OS files
local local
storage storage
Storage Storage
– OS & applications have 1:1 dependencies on hardware configuration
– Complex toSlow and Unreliable Process,
physically recover OS, applications & data
Expensive Infrastructure
– Separate processes for system and application data
46

47. DR Scenario – Without Virtualization
Hardware Gap - Your Production & DR must always be equal
Gap grows as new severs are deployed
47

48. Can you afford NOT
to modernize your
recovery solution?

49. With Virtualization
Break dependencies between OS
and hardware
Manage OS and application as
single unit by encapsulating
them into VMs
VM’s are hardware-independent:
they can be provisioned anywhere
49

50. Physical Servers Converted into a Virtual
Solution
Production Site DR Site
Image Repository
Import virtual machines
to ESX Servers
Convert images to virtual machines
50

51. Physical to Virtual (P-V) then Recovery
imaging conversion
P2V
JAS
imaging
P2V conversion
Deploy
imaging
conversion Replication
P2V
Load-Balance
How: If Recovery is required:
• Converter creates virtual machines • Boot virtual machines on any hardware
matching physical machines • Start data recovery of application data
• Copy virtual machines to Denver
51

52. Checkpoint # 8: Testing your DR Plan
It worked last year.
Nothing has really changed…
52 August 29, 2012 Velocity Proprietary and Confidential

53. It’s not all bad news if your plan fails
during a test…
Frequent testing identifies gaps in your recovery process
53 August 29, 2012 Velocity Proprietary and Confidential

54. Passive Testing
• Hands on plan review
• Paper walkthrough
– Assessment of current Recovery Plan
– Assessment of risk areas
– Involve every member of your recovery team.
This includes the business and management!
54 August 29, 2012 Velocity Proprietary and Confidential

55. Active Testing
Validates the Recovery Plan in terms of:
1. Recovery capabilities and abilities
2. Alternate site configuration
3. Network recovery
4. Offsite records
5. Identify weaknesses in the plan
6. Provides training
This all equals success during a disaster
55 August 29, 2012 Velocity Proprietary and Confidential

56. Testing / Hotsite Best Practices
• Examine your hardware schedule in your DR contract
— Check for back level compatibility
— Do you need additional capacity?
— Performance Profile – Have database requirements changed?
— Travel and expenses
• Avoid testing from “special” backups
• Test your entire technology infrastructure
• Train your alternates
• Evaluate vendor performance
• Make sure your “hot box” is current for vital records
56 August 29, 2012 Velocity Proprietary and Confidential

57. Disaster Recovery Planning Is an Ongoing Process
…To be prepared, you must
Business
Plan
regularly go through the cycle
Requirements
Test Implement
57 August 29, 2012 Velocity Proprietary and Confidential

58. Remember to Consider All the Key
Checkpoints
1. Recognize that disasters do happen. Know how your business units
may be impacted and understand their expectations
2. Ensure you have an actionable and repeatable DR plan
3. Confirm your DR plan supports your business
4. Backup: Put into practice what you say you will do
5. Build a recovery team
6. Keep your data stored safely offsite and know how to access it.
7. Practice smart virtualization
8. Test your plan, and then update the plan accordingly
58 August 29, 2012 Velocity Proprietary and Confidential

59. What’s on the Mind of CIOs . .
Aging Infrastructure
Testing Risk
Disaster Recovery Standardization
Governance
Career
Staffing

60. DR Preparedness for 21st Century
Understand and are Ready
– Top Gun
Understand and are not Ready
– Sleepless in Seattle
Understand but don’t want DR
- Dumb and Dumber
Don’t Understand. Why Bother !
– Clueless or Armageddon

61. www.mcpressonline.com

62. Richard Dolewski
VP Business Continuity
Richard.Dolewski@velocity.cc
www.velocity.cc
62