The Data Disappearing Act: Mitigating Virtual Data Loss

The Data Disappearing Act: Mitigating Virtual Data Loss April 6-9, 2009 Rosen Shingle Creek Resort Orlando, Florida Jeff Kubacki Chief Information Officer Kroll, a Marsh & McLennan Company jkubacki@kroll.com

The Data Disappearing Act: Mitigating Virtual Data Loss April 6-9, 2009 Rosen Shingle Creek Resort Orlando, Florida

Learning Objectives  Common data loss scenarios in virtual environments April 6-9, 2009  Challenges with recovering virtual Rosen Shingle Creek Resort data Orlando, Florida  Recommendations when virtual data loss occurs  Design recommendations for a virtual environment with data loss prevention in mind

BSOD or PSOD Do They Give You Chills? April 6-9, 2009 Rosen Shingle Creek Resort Orlando, Florida

Common Data Loss Scenarios April 6-9, 2009 Rosen Shingle Creek Resort Orlando, Florida Source: Over 100 Kroll Ontrack Virtual Data Recovery Jobs Over the Past 12 Months

Case Study #1 | Virtual Data Loss  Number of virtual disks disappeared or were somehow deleted  Customer tried a few things to get the data back which April 6-9, 2009 made things worse Rosen Shingle Creek Resort Orlando, Florida  Noticed virtual disks in the wrong subdirectory to truncated snapshot files  Raid was also running in degraded mode  Virtual disks had snapshots and the \"parent pointers\" were no longer correct

Case Study #2 | Virtual Data Loss  While trying to replace a failed disk in another array, an administrator accidentally removed a disk from a healthy array storing all virtual servers which hosted the business critical Microsoft SQL servers April 6-9, 2009 Rosen Shingle Creek Resort  The array sustained critical damage through multiple Orlando, Florida repair attempts

Case Study #3 | Virtual Data Loss  VMware server had two SAN disks it used to store it’s virtual servers  A non-virtual MS Windows server was inadvertently given April 6-9, 2009 Rosen Shingle simultaneous access to the same disks Creek Resort Orlando, Florida  An administrator on the Windows server created and formatted a dynamic mirror at the same time the VMware server was still using the disks to run the company’s Microsoft SQL servers  The creation & formatting of the mirror on the SAN disks caused massive damage to the VMware SQL servers

Common Data Loss Scenarios  At times, the root cause may be related to hardware or software failures April 6-9, 2009 Rosen Shingle Creek Resort Orlando, Florida

Common Data Loss Scenarios  At times, the root cause may be related to hardware or software failures  However, human error appears to play an even larger role in April 6-9, 2009 Rosen Shingle virtual data loss Creek Resort Orlando, Florida

Learning Objectives  Common data loss scenarios in virtual environments  Challenges with recovering virtual data April 6-9, 2009 Rosen Shingle  Recommendations when virtual data loss occurs Creek Resort Orlando, Florida  Design recommendations for a virtual environment with data loss prevention in mind

Challenges with Recovering Virtual Data  Recovery of multiple guests on a single volume  Snapshots, logs and swap files add complexity to recovery April 6-9, 2009  Fragmentation of the virtual file system volume Rosen Shingle Creek Resort  Overall size of the recovery Orlando, Florida  Lack of a good backup that has been tested  Using traditional methods of recovery, such as restore, may make the problem worse

Recommendations When Data Loss Occurs  Don’t panic and don’t update your resume  When troubleshooting, ensure you can rollback to undo April 6-9, 2009 Rosen Shingle changes Creek Resort Orlando, Florida  Don’t write new data to the volume that needs recovery  Restore your backup to a different volume  Don’t reformat the volume that has missing data  Use the support system offered by the software provider

Recommendations When Data Loss Occurs  Restore data to an alternate location and contact a data recovery company with extensive virtual data recovery experience including the ability to perform remote April 6-9, 2009 Rosen Shingle recoveries Creek Resort Orlando, Florida  Definition of data recovery (DR)  DR gets back files from corrupted or inaccessible storage (directly from the failed system, not from a backup)  DR gets back most recent files vs most recent backup (delivers a RPO of zero)  In some cases, DR is faster than restoring from the last backup  DR fits well as part of an overall disaster recovery plan

Design Recommendations  Implement naming conventions for hosts, guests, physical servers and virtual file system volume  Control who has access to the environment April 6-9, 2009 Rosen Shingle Creek Resort  No local storage or disk stores on the host machines Orlando, Florida  Boot the host off a USB key  Use a SAN in your design

Design Recommendations  Document the backup and recovery plan and include the contact information of your preferred data recovery vendor in the plan (i.e. Kroll Ontrack) April 6-9, 2009 Rosen Shingle  Test your backups on a regular basis Creek Resort Orlando, Florida  Use the tools to manage your virtual environment, don’t take shortcuts  Be careful how you use snapshots and do your housekeeping  Monitor the data stores, logs and swaps

Learning Objectives | Summary  Common data loss scenarios in virtual environments  Challenges with recovering virtual data April 6-9, 2009 Rosen Shingle  Recommendations when virtual data loss occurs Creek Resort Orlando, Florida  Design recommendations for a virtual environment with data loss prevention in mind

Thank You! April 6-9, 2009 Rosen Shingle Creek Resort Orlando, Florida Jeff Kubacki Chief Information Officer Kroll, a Marsh & McLennan Company jkubacki@kroll.com

The Data Disappearing Act: Mitigating Virtual Data Loss

0 comments

Notes on slide 1

Favorites, Groups & Events