Disaster Recovery Status Report 22 Feb2012

DISASTER RECOVERY STATUS REPORT
Title Revision
Disaster Recovery Status Report 22 Feb 2012 001
Document Number Issue Date Page
IT-REPORT-1201 29 February 2012 1 of 13

Aviat Disaster Recovery Solution Test

REVISION HISTORY

Revision
Rev Change Order Description of Change Date
001 N/A Initial release of document. 22-Feb-2012
002 N/A Updates to document. 29-FEB-2012
Approvals
Function Approver Title Name Date
Latest rev updated by: Director, Global IT Rees Morgan [dd-mmm-yyyy]
Latest rev approved by: VP, Global IT Gerry Kinder [dd-mmm-yyyy]

PROPRIETARY & CONFIDENTIAL © AVIAT NETWORKS INC. IT DEPARTMENT: This document is the property of the Aviat Networks, Inc. IT
department and may be neither used, copied, reproduced, or communicated outside the Aviat Networks, Inc. IT Department, except with
the written permission of IT department senior staff members.
This document is uncontrolled when printed. Refer to electronic archive for current controlled revision.

Title Revision

CONTENTS

1.0 INTRODUCTION.................................................................................................................................................. 3
1.1 Purpose................................................................................................................................................... 3
1.2 Scope ...................................................................................................................................................... 3
1.3 Abbreviations, Definitions and Terms ..................................................................................................... 3
2.0 SCENARIO .......................................................................................................................................................... 4
2.1 22 Feb DR List of Applications ................................................................................................................ 4
Ordered by Priority................................................................................................................................................... 4
3.0 ENVIRONMENT & SOLUTION PREPARATION ...................................................................................................... 4
3.1 Environment Components and Scope of Work ....................................................................................... 4
3.2 Disaster Recovery Team & Partners ....................................................................................................... 5
4.0 DISASTER RECOVERY TOPOLOGY ....................................................................................................................... 7
4.1 Environment Overview ............................................................................................................................ 7
4.2 Network Overview ................................................................................................................................... 8
5.0 TIMELINE ........................................................................................................................................................... 8
5.1 Timeline Outlined .................................................................................................................................... 8
5.2 Timeline Result ....................................................................................................................................... 9
6.0 DISCOVERIES ................................................................................................................................................... 10
7.0 CONCLUSION .................................................................................................................................................... 11
APPENDIX A: DOCUMENT REFERENCES .................................................................................................................. 11
A.1 I/O Mini Test Errors ............................................................................................................................... 11
A.2 Oracle Database LUN Tier Strategy ..................................................................................................... 12

© AVIAT NETWORKS INC.PROPRIETARY & CONFIDENTIAL: Unauthorized distribution is prohibited.

Title Revision

1.0 INTRODUCTION
1.1 Purpose
The purpose of this document is to provide a status report of the Disaster Recovery test conducted on
22 February 2012 and the action items identified.

1.2 Scope
This document applies to steps, actions and results pertaining to the Disaster Recovery test held on 22
February 2012.

CONFIDENTIAL AND PROPRIETARY © AVIAT NETWORKS INC. IT DEPARTMENT

This document is the property of the Aviat Networks, Inc. IT department and may be neither used,
copied, reproduced, or communicated outside the Aviat Networks IT Department, except with the
written permission of appropriate IT department senior staff members.

1.3 Abbreviations, Definitions and Terms
Term– Disaster Recovery
DR– is the process, policies and procedures related to preparing for recovery or continuation of
technology infrastructure critical to an organization after a natural or human-induced disaster.
Disaster recovery is a subset of business continuity.
Term – Recovery Point Objective
RPO – is the maximum tolerable period in which data might be lost from an IT Service due to a Major
Incident.
Term – Site Recovery Manager
SRM – is the service that coordinates and manages the replication, failover and recovery of VMware
Virtual Machines.


Title Revision

2.0 SCENARIO
The scenario outlined for the initial Disaster Recovery test was that Aviat Santa Clara (host to Aviat’s
core Data Center) was suddenly without network connectivity and thus no longer able to provide
services to Aviat’s Global offices. A list of applications was identified and approved by management to
set the scope of what services require recovery validation. See 2.1 for list of Applications.

2.1 22 Feb DR List of Applications

Ordered by Priority
E-Business
Hyperion FM
Siebel
Agile
Clarify
DAS
B2B /
Business & Financial Shared
Files/Folders (i.e. Durapp1 & SC G: & I: Drive or SC-FS02)
FTP
Data Mart (Data Warehouse)
GLDi-ADI
iDiscoverer
Intranet (SharePoint)
Licensing
SCMNet
Vitria

3.0 ENVIRONMENT & SOLUTION PREPARATION
3.1 Environment Components and Scope of Work
The technology components that were purchased to support the Disaster Recovery environment are EMC’s
RecoverPoint data replication appliance and VMware SRM.
The EMC Recover Point appliances were deployed in pairs to provide redundancy and load balancing at each
site (Santa Clara & San Antonio). VMware Site Recovery Manager (SRM), deployed at both sites, is the
management piece from VMware to manage Recover Point for the replication and recovery of defined Virtual
Machines. The defined Virtual Machines are represented by creating a ‘Recovery Plan of Steps’ in VMware
SRM. The majority of Virtual Machines designated for the DR test are application tier VMs that provide client
access to Oracle applications. In order to test the duplicate (point-in-time copy) of Production while Production
is running, an isolated network was created whereby Administrators could connect to servers via a NAT
(Network Address Translation) IP address that allowed inbound communication but no outbound


Title Revision

communication. This isolation network was labeled as the “Island” or “Bubble” network. The network
separation allowed for point in time duplicate Production environment to be created for the Disaster Recovery
Proof of Concept test. This demonstrated that in the event of a “true” Disaster Recovery Failover, that Santa
Clara would failover to San Antonio with the Applications and Data intact. The Network and Applicatoin
failover has been tested and implemented in a previous project, the Data Center move from Raleigh (RTP) to
Santa Clara (SC).

Our partner and VAR, Nexus IS, assisted us with the preparation of the February test byinstalling the necessary
storage configuration, Recover Point Appliances and VMware SRM configuration. In conjunction with their
work, Aviat Engineers were working and receiving knowledge transfer from Nexus Engineers.
Documentation of the configuration, steps necessary to create Recovery Groups in Recover Point, and VMware
SRM have been created and verified.

3.2 Disaster Recovery Team & Partners

Disaster Recovery Team
Aviat Engineer Aviat Consultants Nexus - Partner
Albert Etienne Alagappan Vairavan Jasen Paris
Chris Snow Ramasubramani Namboor Kevin Byron
Jerome Keller Norman del Prado
Matthew Lass Steven Mitchell
Robert Blankman
Tan Nguyen
Yves Plante


Title Revision

Disaster Recovery Team (Yves Plante, Matt Lass, Steven Mitchell, Jasen Paris, Jerome Keller)


Title Revision

4.0 DISASTER RECOVERY TOPOLOGY
4.1 Environment Overview

Santa Clara San Antonio

VM VM
Business Critical Shared Files Business Critical Shared Files
SQL VM VM SQL

VM VM
DAS Intranet Replication Management DAS Intranet
Equity Edge Magic VM VM Equity Edge Magic

VM Replication VM Replication
Agile Management Aviat Management Agile
E-Business E-Business
Approva GCAS VM Corp Net VM Approva GCAS
B2B GLDI B2B GLDI
Blackline HFM Blackline HFM
Clarify Siebel Clarify Siebel
Discoverer Synergy MLA VM VM Discoverer Synergy MLA
EMDA VASP EMDA VASP
ELIA ELIA

Storage Replication

Protect Protect
Storage Storage

SAN / NAS SAN / NAS


Title Revision

4.2 Network Overview

Production – Santa Clara Disaster Recovery – San Antonio

Internet Internet

DMZ DMZ

3rd Party Shared Data 3rd Party Shared Data

CORP LAN CORP LAN

Client Client
Access Access

Aviat
Corp Net
Restricted Firewall Restricted Firewall
DR DMZ Access DR DMZ Access

Replication

5.0 TIMELINE
5.1 Timeline Outlined
The timeline for the test was scoped and posted to the SharePoint. The overall time projected for the test
exercise was 8 hours or less. A full day window was scoped to allow for variations and process corrections
based on discoveries. The best estimate or intention for the test exercise was 6 hours.
High Level Timeline of Testing
08:00 – Team onsite and logged in to systems and conference bridge
08:30 – Management to declare an emergency and team to begin documented procedures for failover
of Santa Clara to San Antonio
08:35 – San Antonio and Offshore team to initiate technology failover
09:15 – Replicated data mounts to be made available to all teams for recovery of services
10:15 – Services made online to hand over to Application team for verification and beginning of
testing.
13:45 – Application team to provide results or ongoing testing (if needed)


Title Revision

14:00 – Testing complete, verification acknowledged and test completed.

5.2 Timeline Result
While the test started as scheduled, the defined steps and timelines did not follow according to plan. As stated
previously, the timeline was a best estimate or intention for the test exercise.
08:00 – Team onsite and logged in to systems and conference bridge
08:30 – Management to declared an emergency and the team began documented procedures for
failover of Santa Clara to San Antonio
08:35 – On the conference bridge, the DBA informed the server team that the test may be not be able
to proceed due to failures discovered during a mini-test the day before
08:45 – After determining the cause of the errors and resolving them, the failover test began. See A.1
08:50 – Robert Blankman followed the SRM instructions andinitiated the SRM failover by pressing the
“Failover” button. The pressing of this button kicked off the DR test and instructed VMware SRM to
follow the recovery plan steps already configured by Matt Lass and Steven Mitchell.
10:30 – Due to the test and an oversight in the Recovery Plan steps, an additional configuration caused
a hang up in the failover process. Steven and Matt cancelled the process, fixed the configuration issue
and informed Robert to initiate the failover once again.
o Failover through SRM completed in 37 min, 21 sec. 26 Virtual Machines
o This included
 Failover, informing Recover Point to pause any replication
 Make storage available to mount, scan and register Virtual Machines
 Modify Virtual Machine with any networking or other configuration changes to come
online in San Antonio
o All steps were successful
10:38 – Storage was available and provided to Alagappan to be mounted in both Solaris Servers which
host Oracle 8i, 9 & 10g.
10:40 – SharePoint (Intranet) test results were successful
10:40 – FTP test results were successful
10:42 – Oracle E-Business Application and Database became available to testers
12:11 – E-Business test results were successful
12:28 – B2B test results were successful
1:27 – DAS test results were successful
1:27 – Clarify test results were successful
2:23 – Siebel test results partially successful (Application Tier functioning but not able to connect to
Database, testing to continue)

Title Revision

4:38 – Siebel Database test results were successful
Hyperion test results received next day but were successful
Not all test results were received as of 3:30PM PST, the DR Test Exercise was declared completed and the DR
Solution had been validated.

6.0 DISCOVERIES
Various items were identified as a result of the DR test. Majority of the items were unique to the type of test
and preparation needed for the test.
Item Identified Resolution
Need GNET Domain Controller for GNET Create new GNET Domain Controller as VM and set
Authentication in DR Environment to be replicated to DR Site

Oracle Environment used for DR Test was not a full
The Disaster Recovery environment will replicate
copy. Was configured with new data mounts and
Production systems, and captures ALL necessary files
created from Backup Restoration. The manual tasks
and configurations to insure the DR environment is a
performed enabled human error as key files were
true replica.
missed in order to successfully test all applications

The original thinking was to group the Application
Tier and Database Tier for an application together.
Given the two technologies used to bring an
Application Tier VM LUNs were grouped with
Application online at the DR site, it was determined
Database Tier LUNs
it would be better to separate the layers to provide
greater flexibility forrecovery. Application and/or
Database tiers can be restored separately.

During the testing, key files were deemed missing
from human error resulting in a failover and failback
Based on Application Size and Recovery Order, multiple times. Because of the grouping
Applications were grouped together in Recovery configuration, this affects multiple applications, not
Groups just the application that was focused on. Creating
granularity will provide flexibility for such a scenario
in the future and individual application recovery.

Member to Domain Separation: Some Virtual
Machines were not able to authenticate to the GNET
Having a Production Domain Controller replicated
domain controller placed in the DR network.
with the Applications at same point in time, this will
Because the copies of the Applications and the
resolve any separation issues.
Domain Controller were copied at different point in
times, this caused a separation.


Title Revision

7.0 CONCLUSION
Aviat Networks first 2012 Disaster Recovery test was not without issues, but the test itself proved the solution
implemented is valid and functional. The solution validation was the key reason for the February DR test which
is considered a success. With items identified and lessons learned, those changes will be applied to the
Production implementation of DR. The Server and DBA teams feel confident that the solution will provide Aviat
a recovery method in the event the Santa Clara facility is compromised.

The infrastructure deployed for the DR environment validated the need for changes to Production. The
changes require Production databases to reside on independent Network Storage mounts/LUNs. The
separation of mount points provides scale, flexibility and individual Application recovery. See A.2

The 22 Feb 2012 was a constructive learning experience for all teams to validate and work with the technology
implemented for DR. As the changes that were identified are implemented in Production, individual DR tests
will be conducted with each Application per the February Application list. Both the 22 Feb test results and the
individual Application test results will be provide solid evidence Aviat has a viable and compliant DR solution.

Appendix A: Document References
A.1 I/O Mini Test Errors
The I/O or Disk Errors the DBA team received when performing a Mini or component test with the
server team the day prior to the full DR test was a result of the Recover Point Journal filling to
capacity. The Recover Point Journal is used to track any changes made to the Destination or DR
replicated data when made available in a DR scenario. Because we had not initiated a full failover only
a temporary “image access” or bubble test whereby the data is made available while any changes
from the Source (Santa Clara) is queued. The test resulted in too many changes to log and resulted in
offlining the DR data mount points. The DR test, the next day, a full failover was initiated negating the
need for the Recover Point Journal as no changes were needed from San Antonio was not the Primary
site.


Title Revision

A.2 Oracle Database LUN Tier Strategy
Many to one change Migration to One-to-One

Agile
Approva
B2B
Blackline Agile
Clarify Approva
Discoverer B2B
EMDA Blackline
ELIA Clarify
E-Business Discoverer
GCAS EMDA
GLDI ELIA
HFM E-Business
Siebel GCAS
Synergy MLA GLDI
VASP HFM
Siebel
Synergy MLA
VASP


Title Revision


Disaster Recovery Status Report 22 Feb2012

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Disaster Recovery Status Report 22 Feb2012

Similar to Disaster Recovery Status Report 22 Feb2012 (19)

Disaster Recovery Status Report 22 Feb2012