SlideShare a Scribd company logo
1
SAP
Security Categorization: High
Omega Research
Information System Contingency Plan (ISCP)
Version 1
[12/14/14]
Prepared by Tim Haney
[Omega Research]
[3456 Winchester Blvd]
[San Diego, Ca, 94511]
2
Table of Contents
Plan Approval…………………………………………………………………………..…….3
1. Introduction………………………………………………………………………......3
Background…………………………………………………………………………..3
Scope………………………………………………………………………….……….4
Assumptions………………………………………………………………………….4
2. Concept of Operations……………………………………………………………..5
System Description…………………………………………………………………5
Overview of Three Phases…………………………………………………………6
Roles and Responsibilities………………………………………………………..8
3. Activation and Notification………………………………………………………..8
Activation Criteria and Procedure……………………………………………….8
Notification…………………………………………………………………………..8
Outage Assessment………………………………………………………………..10
4. Recovery……………………………………………………………………………..11
Sequence of Recovery Activities………………………………………………..11
Recovery Procedures……………………………………………………………..11
Escalation Notices/Awareness…………………………………………………..12
5. Reconstitution……………………………………………………………………….13
Concurrent Processing…………………………………………………………….13
Validation Data Testing…………………………………………………………….14
Validation Functionality Testing………………………………………………….14
Recovery Declaration………………………………………………………………14
Notifications (Users)………………………………………………………………..14
Cleanup……………………………………………………………………………….14
Offsite Data Storage………………………………………………………………..14
Data Backup…………………………………………………………………………15
Even Documentation………………………………………………………………15
Deactivation…………………………………………………………………………15
Appendices………………………………………………………………………….16
References…………………………………………………………………………..23
3
Plan Approval
As the designated authority for SAP System, I hereby certify that the information system contingency
plan (ISCP) is complete and that the information contained in this ISCP provides an accurate
representation of the application, its hardware,software,and telecommunication components. I further
certify that this document identifies the criticality of the system as it relates to the mission of the Omega
Research,and that the recovery strategies identified will provide the ability to recover the system
functionality in the most expedient and cost-beneficial method in keeping with its level of criticality.
I further attest that this ISCP for SAP System will be tested at least annually. This plan will be tested on
1/15/15; the test,training, and exercise (TT&E)material associatedwith this testcanbe found TT&E results
appendix. This document will be modified as changes occur and will remain under version control, in
accordance with Omega Corporation contingency planning policy.
Tiffany Sabers
ChiefInformation Officer Date 11/24/14
1. Introduction
Information systems are vitalto Omega Research business processes; therefore,it is critical that services
provided by SAP System are able to operate effectively without excessive interruption. This
Information System Contingency Plan (ISCP) establishes comprehensive procedures to recover SAP
System quickly and effectively following a service disruption.
1.1 Background
This SAP System Information System (IS) Contingency Plan (CP) establishes procedures to recover
SAP System following a disruption. The following recovery plan objectives have been established:
 Maximize the effectiveness of contingency operations through an established plan that consists of
the following phases:
 Activation and Notification phase to activate the plan and determine the extent of
damage;
 Recovery phase to restore SAP System operations; and
 Reconstitution phase to ensure that SAP is validated through testing and that normal
operations are resumed.
 Identify the activities, resources, and procedures to carry out SAP System processing
requirements during prolonged interruptions to normal operations.
 Assign responsibilities to designated Omega Research personnel and provide
guidance for
recovering SAP System during prolonged periods of interruption to normal operations.
 Ensure coordination with other personnel responsible for Omega Research contingency
planning strategies. Ensure coordination with externalpoints of contact and vendors associated
with SAP System and execution of this plan.
4
1.2 Scope
This ISCP has been developed for SAP System which is classified as a High-Impact system, in
accordance with Federal Information Processing Standards (FIPS) 199 – Standards for Security
Categorization of Federal Information and Information Systems. Procedures in this ISCP are for High-
Impact systems and designed to recover SAP System within 24 hours. This plan does not address
replacement or purchase of new equipment, short-term disruptions lasting less than 24 hours or loss of
data at the onsite facility or at the user-desktop levels.
1.3 Assumptions
The following assumptions were used when developing this ISCP:
 SAP System has been established as a High-Impact System, in accordance with FIPS 199.
 Alternate processing sites and offsite storage are required and have been established for this
system.
 Current backups of the system software and data are intact and available at the offsite storage
facility in Reston, VA.
 Alternate facilities have been established at Philadelphia, PA and are available if needed for
relocation of SAP System.
 The SAP is inoperable at the Omega Research computer center and cannot be recovered within
24 hours.
 Key SAP System personnel have been identified and trained in their emergency response and
recovery roles; they are available to activate the SAP System Contingency Plan.
 The SAP Contingency Plan does not apply to the following situations:
o Overall recovery and continuity of business operations. The Business
Continuity Plan (BCP) and Continuity of Operations Plan (COOP) address
continuity of business operations.
o Emergency evacuation of personnel. The Occupant Emergency Plan (OEP)
addresses employee evacuation.
5
CONTINGENCY PLANNING GUIDE FOR FEDERAL INFORMATION SYSTEMS (DRAFT)
2. Concept of Operations
The Concept of Operations section provides details about SAP System, an overview of the three phases of
the ISCP (Activation and Notification, Recovery,and Reconstitution), and a description of roles and
responsibilities of Omega Research’s personnel during a contingency activation.
2.1 System Description
San Diego
Salem
Reston Kansas City
TDMTDMTDMTDM TDMTDMTDMTDM
TDMTDMTDMTDM
TDMTDMTDMTDM
6
Reston office: SMTP Mail Gateway, File/ Print Server, Exchange 2000 server, Web Server, DNS,
RAS, PBX, workstations and printers.
San Diego office: SMTP Mail Gateway, File/Print Server, Exchange 2000 mail server, DNS, RAS,
PBX, workstations, and Printers.
Salem office: File/Print server, VPN Gateway, workstations, and Printers.
Kansas City office: File/print Server, Exchange Server, VPN Gateway, workstations, and Printers.
Reston office description: Perimeter protection provided by screening router (ACLs). Remote access is
provided to employees while at home or on travel through PPTP VPN, and dial-up RAS offered by a
Microsoft Windows N.T. Server. All servers in the Reston office have been centrally located to a data
center. The data center supports a 5-keypunch combination lock that is required to have access to the
room and is controlled with HVAC for temperature with isolated services and is not on a raised floor to
control static electricity. Each server and network equipment supports their own mini-UPS.
Email uses Microsoft Exchange Server 2000 mail server running on a Windows 2000 Server with
SMTP mail gateway. Omegaresearch.com is the domain name with a DNS server used for name
resolution for public internet access. Web hosting services are provided by Windows 2000 Server
running Internet Information Services (IIS). X.500 directory services are available through active
directory. Server and client OS (operating system) environments have not been patched. All printers
are networked. The IT department manages more than 170 workstations and 6 servers. Client
machines consist of Microsoft Windows 95, 98, NT workstation 4.0, 2000, and XP. Mac operating
systems (OS) including OS8 and OS-X, and Panther. Productivity applications have not been
standardized. Some users enjoy Corel OfficeSuite while others appreciate Microsoft Office.
San Diego office description: It is essentially a mirror of Reston architecture. The only differences are
that it does not have a web server or support VPN or RAS connections. There are fewer employees
here, and only one IT engineer for the entire office. There are less than 50 client machines with similar
configurations as the main office. All servers have been located in an extra office and do not have
HVAC for a controlled environment for humidity, temperature, or static. There is no access control in
place and there are no redundant power supplies.
Salem office description: Contains only 30 workstations, configured in much the same way as the rest
of the company. It supports a single combined shared file and print server hosted on a Microsoft
Windows NT 4.0 Server. Mail services are obtained through the San Diego office, using mailboxes
setup on the San Diego Exchange Server. No publicly available networked resources. Remote access
is allowed through VPN client to gateway connectivity. It has one IT engineer to manage its IT
resources. All servers have been located in a spare room in San Diego. The servers have no access
restriction, with no HVAC to control humidity, temperature, and static. There are no redundant power
supplies.
Kansas City office description: It is a similar size to the Salem office with the exception that Kansas
City runs a Microsoft Exchange 2000 server for mail services. It has a local system administrator for
support. All servers have been located in a spare office onsite. There is no HVAC or access
restrictions. The office is not controlled for temperature, humidity, or static. There are no redundant
power supplies.
2.2 Overview of Three Phases
This ISCP has been developed to recover the SAP System using a three-phased approach. This
approach ensures that system recovery efforts are performed in a methodical sequence to maximize the
effectiveness of the recovery effort and minimize system outage time due to errors and omissions.
7
The three system recovery phases are:
Activation and Notification Phase – Activation of the ISCP occurs after a disruption or outage that may
reasonably extend beyond the RTO established for a system. The outage event may result in severe
damage to the facility that houses the system, severe damage or loss of equipment, or other damage that
typically results in long-term loss.
Once the ISCP is activated,system owners and users are notified of a possible long-term outage,and a
thorough outage assessment is performed for the system. Information from the outage assessment is
presented to system owners and may be used to modify recovery procedures specific to the cause of the
outage.
Recovery Phase – The Recovery phase details the activities and procedures for recovery of the affected
system. Activities and procedures are written at a level that an appropriately skilled technician can
recover the system without intimate system knowledge. This phase includes notification and awareness
escalation procedures for communication of recovery status to system owners and users.
Reconstitution Phase – The Reconstitution phase defines the actions taken to test and validate system
capability and functionality. This phase consists of two major activities: validating successful recovery
and deactivation of the plan.
During validation, the system is tested and validated as operational prior to returning operation to its
normal state. Validation procedures may include functionality or regression testing, concurrent
processing, and/or data validation. The system is declared recovered and operational by system owners
upon successful completion of validation testing.
Deactivation includes activities to notify users of system operational status. This phase also addresses
recovery effort documentation, activity log finalization, incorporation of lessons learned into plan
updates, and readying resources for any future recovery events.
8
2.3 Roles and Responsibilities
The ISCP establishes severalroles for SAP System recovery and/or recovery support. Persons or teams
assigned ISCP roles have been trained to respond to a contingency event affecting SAP System.
The following are roles and responsibility of each ISCP Support Team:
1. ISCP Director (Bill Hermann, Executive CEO) and alternate (John Sampolous, CFO) have overall
management responsibility for the plan. They are included in a Management Recovery Team.
2. ISCP Coordinator (Tim Haney, IT Manager) and alternate (Linda Okonieski, COO) initiate any
needed escalations or awareness communications, oversee recovery effort,and coordinate with other
recovery teams. They are included in the Recovery Team.
3. Server Recovery Team (Tyler Amdahi, Director of I.T. Operations) and (Rachid Chad, Dir. Of IT)
are responsible for restoration of Web Email, desktop support, email system, and other technical
support.
4. Database Recovery Team (Tiffany Sabers, CIO, Information Technology) and (Sandy Ales,
Director of Sales and Marketing) are responsible for restoration of the SAP system Database.
5. Network Recovery Team (Jackson Davis, Director of Accounting) and (Fionna O’Connor,
Director of Audit and Compliance) are responsible for restoration of the WAN/LAN
telecommunications.
3. Activation and Notification
The Activation and Notification Phase defines initial actions taken once a SAP System disruption has
been detected or appears to be imminent. This phase includes activities to notify recovery personnel,
conduct an outage assessment, and activate the ISCP. At the completion of the Activation and
Notification Phase,SAP System ISCP staff willbe prepared to perform recovery measures to restore
system functions.
3.1 Activation Criteria and Procedure
The SAP System ISCP may be activated if one or more of the following criteria are met:
1. The type of outage indicates SAP System will be down for more than 24 hours
2. The facility housing SAP System is damaged and may not be available within 24 hours
The following persons or roles may activate the ISCP if one or more of these criteria are
met:
1. CIO (Chief Information Officer)
2. Alternate CIO
Tiffany Sabers, 858-934-1245
Tim Haney, 858-234-0943
3.2 Notification
The first step upon activation of the SAP System ISCP is notification of appropriate business and
system support personnel. Contact information for appropriate POCs is included in the contact list
Appendix A and B.
For SAP System, the following method and procedure for notifications are used:
9
1. The Recovery Team Leads,Vendors and the ISCP Director will be contacted by the ISCP Coordinator
or alternate for initial notification. The Team Leads will contact their members as shown in the
notification call tree diagram.
2. When an outage occurs, Business Owners and Stakeholders should also be notified by the ISCP
Coordinator.
3. Use primary email and cell number for outage notification. If these do not work, use the satellite
phone.
4. See the Notification Checklist.
Notification Call Tree
ISCP
Coordinator
ISCP Coordinator
Alternate
SAP/Tape Backup
Vendor
Server Recovery
Team
Lead
Database
Recovery Team
Lead
Network
Recovery Team
Lead
ISCP Director
Alternate
ISCP Director
Database Recovery
Team Member
Server Team
Recovery Member
Network Recovery
Team Member
NotificationChecklist
Number Task Responsibility Date/Time Status
(che ck if
com pleted)
10
1. Notify ISCP Team
with call lists
Refer to
Notification Call
Tree
2. Recovery team meets
to discuss strategy
ISCP
Coordinator
3. Alert Vendor and
Recovery Site
ISCP
Coordinator
4. ISCP Coordinator will
be advised by
Recovery Team to
notify Senior
Management
Recovery Team
Lead
5. Activate Contingency
Plan
CIO
6. Declare a disaster
and activate plan
CIO
7. Notify Hotsite and
Business
Owners/Stakeholders
ISCP
Coordinator
3.3 Outage Assessment
Following notification, a thorough outage assessment is necessary to determine the extent of the
disruption, any damage, and expected recovery time. This outage assessment is conducted by Disaster
11
Recovery Team Lead. Assessment results are provided to the ISCP Coordinator to assist in the
coordination of the recovery of SAP System.
Once the outages occurs, the following procedures should be followed:
1. Determine cause of the outage or disruption;
2. Assess the potential for additional disruptions or damage;
3. Determine the status of physical infrastructure (e.g.,structural integrity of computer room, condition of
electric power, telecommunications, and heating, ventilation and air-conditioning [HVAC]);
4. Inventory and functional status of SAP system and other related applications like email and Web
services.
5. Determine the type of damage to system equipment or data (e.g., water,fire and heat, physical impact,
electrical surge) or other items which may be critical.
6. Determine items to be replaced (e.g.,hardware,software,firmware, supporting materials); and
7. Determine estimated time to restore normal services.
4. Recovery
The Recovery Phase provides formal recovery operations that begin after the ISCP has been activated,
outage assessments have been completed (if possible), personnel have been notified, and appropriate
teams have been mobilized. Recovery Phase activities focus on implementing recovery strategies to
restore system capabilities, repair damage, and resume operational capabilities at the original or new
permanent location. At the completion of the Recovery Phase, SAP System will be functional and
capable of performing the functions identified in the plan.
1. Temporary manual processing like using Excel spreadsheets.
2. Recovery and operation at an alternative site at Philadelphia, PA.
3. Systems with high criticality will be recovered like: Email Server,Web Server, VPN Gateway,DNS,
RAS, and client machines.
4.1 Sequence of RecoveryActivities
The following activities occur during recovery of SAP System:
1. The recovery location is in Philadelphia, PA at SunGard.
2. Recovery services for the SAP system will be for a server environment. The critical system
resources to be recovered first are Email Server, then Web Server, then DNS, then RAS, then
client machines, and last File/Print Server and PBX.
3. Retrieve backup and SAP system installation media;
4. Recover Servers and Windows N.T. Server.
5. Recover SAP system from backup and system installation media at Reston, Va. Or from
Philadelphia, where are backup tapes are sent.
4.2 RecoveryProcedures
12
The following procedures are provided for recovery of SAP system at the established location in
Philadelphia, PA . Recovery procedures are outlined per team and should be executed in the sequence
presented to maintain an efficient recovery effort.
To facilitate Recovery Phase operations, the ISCP should provide detailed procedures to restore the
information system or components to a known state.
Procedures should be assigned to the ISCP recovery teams and typically address the following actions:
1. Obtaining authorization to access damaged facilities and/or geographic area;
2. Notifying internal and external business partners associated with the SAP system;
3. Obtaining necessary office supplies and work space at Philadelphia.
4. Obtaining and installing Email Server, Web Server, DNS, RAS, and client machines.
5. Obtaining and loading backup media from Reston, VA.
6. Restoring critical SAP and Windows Server applications.
7. Restoring SAP system data to a known state;
8. Testing SAP system functionality including security controls;
9. Connecting SAP system to network or other external systems; and
10. Operating alternate equipment at SunGard in Philadelphia, PA successfully.
4.3 Escalation Notices/Awareness
13
Disaster recovery handling, as described in this document, starts with the escalation of an incident,
which is seriously disrupting operations, to a disaster.
It is important to identify the type of error causing the disruption as early aspossible, since the required
recovery phases and applicable activities mainly depend on the error type.
When a disaster is declared, the following main phases of recovery may be applied in this order:
A. Activate the alternate site with SunGard in Philadelphia to stay in business. This can be a technical
switchover to a standby system, or the activation of alternate business processing using workarounds
or emergency plans.
Which options are possible or applicable depends on the solutions in place and the actualtype of error.
Activating a workaround will be easier and faster, if the workaround is already documented in a
recovery plan.
B. Prepare SAP systems for the recovery.
C. If the SAP system or component is down, SAP system recovery or technical recovery, as a first
step, has to reestablish technical availability of the SAP system by fixing any technical error causing
the disruption. This can be done, for example, by exchanging some defect hardware component, by
activating a standby system or by restoring a database from a backup.
D. If all components are up (or were recovered in the previous step), logical errors inside each system
have to be removed to restore integrity of each system in itself. This requires in-depth application
knowledge and is a prerequisite for the next step.
E. If data consistency between systems of the environment was affected,this again requires in-depth
application knowledge and time to fix it.
F. If data was lost and could not be recovered so far,the next effort should aim at reentering such data
into the systems.
G. Having finished all recovery phases, the systems and business functionality should be checked as
a prerequisite for resuming regular operations.
5. Reconstitution
Reconstitution is the process by which a recovered system is tested to validate system capability and
functionality. During Reconstitution, recovery activities are completed and normalsystem operations are
resumed. If the original facility is unrecoverable, the activities in this phase can also be applied to
preparing a new permanent location to support system processing requirements. This phase consists of
two major activities – validating successfulrecovery and deactivation of the plan.
5.1 Concurrent Processing
14
SAP Systems should be fully tested which are interlinked before allowing user to have access to the
system to ensure reliability.
5.2 Validation Data Testing
Validation data testing is the process of testing and validating recovered data to ensure that data files or
databases have been recovered completely. The following procedures will be used to determine that the
recovered data is complete and current to the last available backup:
See Appendix E and J.
5.3 Validation Functionality Testing
Validation functionality testing is the process of verifying that recovered SAP functionality has been
tested, and the system is ready to return to normal operations.
See Appendix E and J.
At the successfulcompletion of the validation testing, ISCP personnelwill be prepared to declare that
reconstitution efforts are complete and that the system is operating normally. This declaration may be made
in a recovery/reconstitution log or other documentation of reconstitution activities. The ISCP Coordinator,
in coordination with the CIO, ISSO,SAISO and with the concurrence of the Authorizing Official, must
determine if the system has undergone significant change and will require reassessment and
reauthorization. The utilization of a continuous monitoring strategy/program can guide the scope of the
reauthorization to focus on those environment/facility controls and any other controls which would be
impacted by the reconstitution efforts.
5.4 RecoveryDeclaration
Upon successfully completing testing and validation, the CIO will formally declare recovery efforts
complete,and that SAP System is in normal operations. SAP System business and technicalPOCs will be
notified of the declaration by the ISCP Coordinator.
5.5 Notifications (users)
Upon return to normal system operations, SAP System users will be notified by ISCP Coordinator
using pre-planned emails, cell phones, or satellite phones.
5.6 Cleanup
Cleanup is the process of cleaning up or dismantling any temporary recovery locations, restocking
supplies used,returning manuals or other documentation to their original locations, and readying the
system for a possible future contingency event.
Return all manuals and Backup Media back to Reston, VA.
5.7 Offsite Data Storage
15
It is important that all backup and installation media used during recovery be returned to the offsite data
storage location. The following procedures should be followed to return backup and installation media to
its offsite data storage location.
See Appendix F.
5.8 Data Backup
As soon as reasonable following recovery,the system should be fully backed up and a new copy of the
current operational system stored for future recovery efforts. This full backup is then kept with other
system backups. The procedures for conducting a full system backup are:
See Appendix F.
Backups in Reston, VA. Backup tapes sent to Philadelphia, PA.
5.9 Event Documentation
It is important that all recovery events be well-documented, including actions taken and problems
encountered during the recovery effort, and lessons learned for inclusion and update to this ISCP. It is the
responsibility of each recovery team or person to document their actions during the recovery effort,and to
provide that documentation to the ISCP Coordinator.
An after-action report with lessons learned should be documented and included for updating ISCP.
Each Recovery Team (Network, Database, and Server) keep activity logs detailing recovery steps and
issues. The results from Functionality and data testing:
1. Development
2. Collection
3. Approval
4. Maintenance
-Testing as an audit tool.
-Testing as a benchmarking.
-Testing as rehearsal.
5.10 Deactivation
Once all activities have been completed and documentation has been updated, the CIO will formally
deactivate the ISCP recovery effort. Notification of this declaration will be provided to all business and
technical POCs.
16
SUGGESTED APPENDICES
APPENDIX A PERSONNEL CONTACT LIST
SAP ISCP Key Personnel
Key Personnel Contact Information
ISCP Director Work 703-123-4567
Bill Hermann, CEO Home 703-567-4321
1234 Smith Street Cellular 703-954-5432
Reston, Virginia, 20190 Email Bhermann@Omega.com
ISCP Director – Alternate Work 703-765-5430
John Sampolous, CFO Home 703-098-9321
1234 Moorpark Ave. Cellular 703-654-3456
Reston, Virginia, 20191 Email Jsampolous@Omega.com
ISCP Coordinator Work 619-123-4567
Tim Haney IT Manager Home 858-234-4321
1234 Fenley Ave. Cellular 619-453-4321
San Diego, Ca , Email Thaney@Omega.com
ISCP Coordinator – Alternate Work 503-876-6543
Linda Okonieski, COO Home 503-986-5421
3456 Winchester Blvd Cellular 503-346-8751
Salem, Oregon, 97303 Email Lokonieski@Omega.com
Recovery Team – Team Lead Work 816-613-8157
Jackson Davis, Dir of Accounting Home 816-025-9641
4567 Stevens Creek Blvd Cellular 816-753-0921
Kansas City, Missouri, 64101 Email Jdavis@Omega.com
Recovery Team – Team Member Work 816-036-6426
Fionna O’Connor, Dir. Of Audit Home 816-841-7631
9843 Andrea Drive Cellular 816-654-8732
Kansas City, Missouri, 64101 Email FOconnor@Omega.com
APPENDIX B VENDOR CONTACT LIST
Vendor Services Address Phone Contacts
IBM Tape Library
TSM Server
522 South Rd.
Poughkeepsie,
NY, 12601
214-451-7747 Steve Barretta
SunGard Recovery
services for
server
environment
401 N Broad St.
Philadelphia, PA
877-456-3966
215-351-1300
-Don Meltin (test
Coordinator)
-Jack Fabrinni
(Acct. Rep)
-Lincoln
Balducci
(Resource
Coord.)
AT&T Leased Line 3495
Winchester
Blvd, Reston,
VA
877-345-5432 Andy Cates
17
APPENDIX C DETAILED RECOVERY PROCEDURES
-Restore the operating system disk with Automated System Recovery (ASR).
-Boot Windows Server 2003 from the installation DVD and press the F2 key.
-For more information, see the Windows ASR documentation.
-Restore non-operating system disks with NTBackup.
-For more information, see the F1 help in NTBackup.
-Restore the database.
APPENDIX D ALTERNATE PROCESSING PROCEDURES
An alternate processing procedure would be to use Excel spreadsheets manually until systems are back
online.
APPENDIX E SYSTEM VALIDATION TEST PLAN
Once the system has been recovered, the following steps will be performed to validate system data and
functionality:
Procedure Expected Results Actual Results Successful?
Performed
by
At the Command Prompt,
type in sysname
System Log-in
Screen appears
Log in as user
testuser, using
password testpass
Initial Screen with
Main Menu shows
From Menu - select 5-
Generate Report
Report Generation
Screen shows
- Select Current Date
Report
- Select Weekly
- Select To Screen
Report is generated
on screen with last
successful transaction
included
- Select Close Report Generation
Screen Shows
- Select Return to
Main Menu
Initial Screen with
Main Menu shows
- Select Log-Off Log-in Screen
appears
18
APPENDIX F ALTERNATE STORAGE, SITE, AND TELECOMMUNICATIONS
Alternate Storage:
The hot-site will be located in Philadelphia, PA. It is located 154 miles from the primary facility. The
alternate storage facility is a third party storage provider SunGard. Points of contact is Jack Fabrianni at
877-456-3966. Those authorized to retrieve media are Don Meltin and Lincoln Balducci at 215-351-
1300. The facility will be protected with a badge and password protected lock inside and outside the
building. In the case of an emergency like an earthquake, the facility may be unavailable to people
outside of the country. The facility has a backup generator in case it loses power. The facility has some
satellite phones available in case phone lines are down. There are servers,EDW (enterprise data
warehouse),Hyperion Express, UFS (Unix file system), Brio SQR (reporting), Portal (tools), and SAP
Dev DB/CI (development instance with database/centralinstance) located at the alternate site.
Alternate processing site:
The hot-site will be located in Philadelphia, PA. It is located 154 miles from the primary facility.
The alternate storage facility is a third party storage provider SunGard. This processing site
provides redundancy. Points of contact is Jack Fabrianni at 877-456-3966. Those authorized to
retrieve media are Don Meltin and Lincoln Balducci at 215-351-1300. The facility will be
protected with a badge and password protected lock inside and outside the building. In the case
of an emergency like an earthquake,the facility may be unavailable to people outside of the
country. The facility has a backup generator in case it loses power. The facility has some
satellite phones available in case phone lines are down. There are servers,EDW (enterprise data
warehouse),Hyperion Express, UFS (Unix file system), Brio SQR (reporting), Portal (tools), and
SAP Dev DB/CI (development instance with database/centralinstance) located at the alternate
processing site. It has SMTP Mail Gateway, File/Print Server,Exchange Mail 2000 Server, Web
Server, DNS (Domain Name Services), RAS (Remote Access Services),and PBX(telephone).
The site has an 180,000 square foot building with 20,000 square feet of office. Power is provided
by PG&E. The telecommunications system is through AT&T. An SLA agreement is in place for
telecommunications after 24 hours.
Alternate telecommunications
The vendor for the alternate telecommunications site is AT&T in Reston, VA. Andy Cates is the
contact at 877-345-5432. There is another location in Reston, VA. An agreement is in place for
a T1 line. The SLA says they only pay when Omega uses it and they will be reimbursed if
service is down for longer than 24 hours. The people authorized to implement or use alternate
telecommunications capacity are Tim Haney at 619-453-4321 and Linda Okonieski at 503-346-
8751.
19
APPENDIX G DIAGRAMS (SYSTEM AND
INPUT/OUTPUT)
20
APPENDIX H SYSTEM INVENTORY
Internal Omega E-mail is supported by a Microsoft Exchange ® 2000 mail server running on a
Microsoft Windows ® 2000 Server. Omega has installed an SMTP mail gateway to support Internet
mail exchange.
Client machines consist of Microsoft Windows ® 95, 98, NT Workstation 4.0, 2000, and XP. Mac
operating systems include OS/8 and OS-X, Panther.
21
List of resources:
-SMTP Mail Gateway
-File/Print Server
-Exchange 2000 Mail Server
-Web Server
-DNS
-RAS
-PBX
-VPN Gateway
APPENDIX I INTERCONNECTIONS TABLE
SAP consists of applications, a database, and other important components which depend on each other to
function. Certain applications will not work if the database is down. The HR application will not work if
the database is down.
22
APPENDIX J TEST AND MAINTENANCE SCHEDULE
Full functional tests will be performed in January and February each year.
Some functional tests will be performed prior to full functional tests. Notification will be email
at Reston site. From backup media, will be a recovery of database or a server.
Step Date Due by Responsible Party Date Scheduled Date Held
Identify failover test facilitator. November 1 ISCP Coordinator October 1
Determine scope of failover
test (include other systems?).
November 15 ISCP Coordinator,
Test Facilitator
October 15
Develop failover test plan. December 1 Test Facilitator November 1
Invite participants. January 1 Test Facilitator December 1
Conduct functional test. February 1 Test Facilitator,
ISCP Coordinator,
POCs
January 1
Finalize after action report and
lessons learned.
March 1 ISCP Coordinator February 1
Update ISCP based on lessons
learned.
April 1 ISCP Coordinator March 1
Approve and distribute
updated version of ISCP.
May 1 ISCP Director,
ISCP Coordinator
April 1
APPENDIX K ASSOCIATED PLANS AND PROCEDURES
Version 1.0. The ICSP in a database in Reston. The primary point of contact is ISCP
Coordinator Tim Haney.
APPENDIX L BUSINESS IMPACT ANALYSIS
Refer to BIA at Share point site.
APPENDIX M DOCUMENT CHANGE PAGE
Modifications made to this plan since the last printing are as follows:
Record of Changes
Page No. Change Comment Date of Change Signature
Thisis the firstversionTim1.0.
23
References
BackingUp andRestoringYour SAPSystemonWindows - SAPNetWeaverbyKeyCapability - SAP
Library.(n.d.).Retrievedfrom
http://help.sap.com/saphelp_nw70ehp2/helpdata/en/85/cd5e76d4b54a0d9463049e0f56f11e/content.
htm
Log in toSlideShare.(n.d.).Retrievedfrom
http://www.slideshare.net/haney888/savedfiles?s_title=emergency-handling-for-recovery-of-sap-
system-landscapes&user_login=BalakrishnaVegi

More Related Content

Similar to Phase II 12.14.14

Phase II Executive Summary 12.14.14
Phase II Executive Summary 12.14.14Phase II Executive Summary 12.14.14
Phase II Executive Summary 12.14.14
haney888
 
52845
5284552845
Increased IT infrastructure effectiveness by 80% with Microsoft system center...
Increased IT infrastructure effectiveness by 80% with Microsoft system center...Increased IT infrastructure effectiveness by 80% with Microsoft system center...
Increased IT infrastructure effectiveness by 80% with Microsoft system center...
Aspire Systems
 
FOISDBA-Ver1.1.pptx
FOISDBA-Ver1.1.pptxFOISDBA-Ver1.1.pptx
FOISDBA-Ver1.1.pptx
ssuser20fcbe
 
Patrick_Rebrook_Resume
Patrick_Rebrook_ResumePatrick_Rebrook_Resume
Patrick_Rebrook_Resume
Patrick Rebrook
 
Muthu_Oracle DBA
Muthu_Oracle DBAMuthu_Oracle DBA
Muthu_Oracle DBA
traffic89
 
Enterprise resource planning_system
Enterprise resource planning_systemEnterprise resource planning_system
Enterprise resource planning_system
Jithin Zcs
 
Sap implementation
Sap implementationSap implementation
Sap implementation
sydraza786
 
Value Proposition for IBM PureFlex System
Value Proposition for IBM PureFlex SystemValue Proposition for IBM PureFlex System
Value Proposition for IBM PureFlex System
IBM India Smarter Computing
 
ShubhashisshubhankarJena
ShubhashisshubhankarJenaShubhashisshubhankarJena
ShubhashisshubhankarJena
Shubhashis Jena
 
Resume (2)
Resume (2)Resume (2)
Resume (2)
Ashok Valeti
 
Senior Systems Engineering ( Microsoft) .
Senior Systems Engineering ( Microsoft) .Senior Systems Engineering ( Microsoft) .
Senior Systems Engineering ( Microsoft) .
chalikars
 
This is my test slideshare
This is my test slideshareThis is my test slideshare
This is my test slideshare
papdev
 
NOTE This sample template is provided to address NIST SP 800-53 s.docx
NOTE This sample template is provided to address NIST SP 800-53 s.docxNOTE This sample template is provided to address NIST SP 800-53 s.docx
NOTE This sample template is provided to address NIST SP 800-53 s.docx
gibbonshay
 
CV_VishalSarode
CV_VishalSarodeCV_VishalSarode
CV_VishalSarode
Vishal Sarode
 
MEDICAL FACILITY ANALYSIS2MEDICAL FACILITY ANALYSIS16.docx
MEDICAL FACILITY ANALYSIS2MEDICAL FACILITY ANALYSIS16.docxMEDICAL FACILITY ANALYSIS2MEDICAL FACILITY ANALYSIS16.docx
MEDICAL FACILITY ANALYSIS2MEDICAL FACILITY ANALYSIS16.docx
ARIV4
 
Failure analysis buisness impact-backup-archive
Failure analysis buisness impact-backup-archiveFailure analysis buisness impact-backup-archive
Failure analysis buisness impact-backup-archive
Davin Abraham
 
System Insight Manager on HP Servers
System Insight Manager on HP ServersSystem Insight Manager on HP Servers
System Insight Manager on HP Servers
koesteruk22
 
Final Anintharan Cisco Ppt
Final Anintharan Cisco PptFinal Anintharan Cisco Ppt
Final Anintharan Cisco Ppt
anintharan
 
Pat Pezzi Resume
Pat Pezzi ResumePat Pezzi Resume
Pat Pezzi Resume
Patricia Pezzi
 

Similar to Phase II 12.14.14 (20)

Phase II Executive Summary 12.14.14
Phase II Executive Summary 12.14.14Phase II Executive Summary 12.14.14
Phase II Executive Summary 12.14.14
 
52845
5284552845
52845
 
Increased IT infrastructure effectiveness by 80% with Microsoft system center...
Increased IT infrastructure effectiveness by 80% with Microsoft system center...Increased IT infrastructure effectiveness by 80% with Microsoft system center...
Increased IT infrastructure effectiveness by 80% with Microsoft system center...
 
FOISDBA-Ver1.1.pptx
FOISDBA-Ver1.1.pptxFOISDBA-Ver1.1.pptx
FOISDBA-Ver1.1.pptx
 
Patrick_Rebrook_Resume
Patrick_Rebrook_ResumePatrick_Rebrook_Resume
Patrick_Rebrook_Resume
 
Muthu_Oracle DBA
Muthu_Oracle DBAMuthu_Oracle DBA
Muthu_Oracle DBA
 
Enterprise resource planning_system
Enterprise resource planning_systemEnterprise resource planning_system
Enterprise resource planning_system
 
Sap implementation
Sap implementationSap implementation
Sap implementation
 
Value Proposition for IBM PureFlex System
Value Proposition for IBM PureFlex SystemValue Proposition for IBM PureFlex System
Value Proposition for IBM PureFlex System
 
ShubhashisshubhankarJena
ShubhashisshubhankarJenaShubhashisshubhankarJena
ShubhashisshubhankarJena
 
Resume (2)
Resume (2)Resume (2)
Resume (2)
 
Senior Systems Engineering ( Microsoft) .
Senior Systems Engineering ( Microsoft) .Senior Systems Engineering ( Microsoft) .
Senior Systems Engineering ( Microsoft) .
 
This is my test slideshare
This is my test slideshareThis is my test slideshare
This is my test slideshare
 
NOTE This sample template is provided to address NIST SP 800-53 s.docx
NOTE This sample template is provided to address NIST SP 800-53 s.docxNOTE This sample template is provided to address NIST SP 800-53 s.docx
NOTE This sample template is provided to address NIST SP 800-53 s.docx
 
CV_VishalSarode
CV_VishalSarodeCV_VishalSarode
CV_VishalSarode
 
MEDICAL FACILITY ANALYSIS2MEDICAL FACILITY ANALYSIS16.docx
MEDICAL FACILITY ANALYSIS2MEDICAL FACILITY ANALYSIS16.docxMEDICAL FACILITY ANALYSIS2MEDICAL FACILITY ANALYSIS16.docx
MEDICAL FACILITY ANALYSIS2MEDICAL FACILITY ANALYSIS16.docx
 
Failure analysis buisness impact-backup-archive
Failure analysis buisness impact-backup-archiveFailure analysis buisness impact-backup-archive
Failure analysis buisness impact-backup-archive
 
System Insight Manager on HP Servers
System Insight Manager on HP ServersSystem Insight Manager on HP Servers
System Insight Manager on HP Servers
 
Final Anintharan Cisco Ppt
Final Anintharan Cisco PptFinal Anintharan Cisco Ppt
Final Anintharan Cisco Ppt
 
Pat Pezzi Resume
Pat Pezzi ResumePat Pezzi Resume
Pat Pezzi Resume
 

Phase II 12.14.14

  • 1. 1 SAP Security Categorization: High Omega Research Information System Contingency Plan (ISCP) Version 1 [12/14/14] Prepared by Tim Haney [Omega Research] [3456 Winchester Blvd] [San Diego, Ca, 94511]
  • 2. 2 Table of Contents Plan Approval…………………………………………………………………………..…….3 1. Introduction………………………………………………………………………......3 Background…………………………………………………………………………..3 Scope………………………………………………………………………….……….4 Assumptions………………………………………………………………………….4 2. Concept of Operations……………………………………………………………..5 System Description…………………………………………………………………5 Overview of Three Phases…………………………………………………………6 Roles and Responsibilities………………………………………………………..8 3. Activation and Notification………………………………………………………..8 Activation Criteria and Procedure……………………………………………….8 Notification…………………………………………………………………………..8 Outage Assessment………………………………………………………………..10 4. Recovery……………………………………………………………………………..11 Sequence of Recovery Activities………………………………………………..11 Recovery Procedures……………………………………………………………..11 Escalation Notices/Awareness…………………………………………………..12 5. Reconstitution……………………………………………………………………….13 Concurrent Processing…………………………………………………………….13 Validation Data Testing…………………………………………………………….14 Validation Functionality Testing………………………………………………….14 Recovery Declaration………………………………………………………………14 Notifications (Users)………………………………………………………………..14 Cleanup……………………………………………………………………………….14 Offsite Data Storage………………………………………………………………..14 Data Backup…………………………………………………………………………15 Even Documentation………………………………………………………………15 Deactivation…………………………………………………………………………15 Appendices………………………………………………………………………….16 References…………………………………………………………………………..23
  • 3. 3 Plan Approval As the designated authority for SAP System, I hereby certify that the information system contingency plan (ISCP) is complete and that the information contained in this ISCP provides an accurate representation of the application, its hardware,software,and telecommunication components. I further certify that this document identifies the criticality of the system as it relates to the mission of the Omega Research,and that the recovery strategies identified will provide the ability to recover the system functionality in the most expedient and cost-beneficial method in keeping with its level of criticality. I further attest that this ISCP for SAP System will be tested at least annually. This plan will be tested on 1/15/15; the test,training, and exercise (TT&E)material associatedwith this testcanbe found TT&E results appendix. This document will be modified as changes occur and will remain under version control, in accordance with Omega Corporation contingency planning policy. Tiffany Sabers ChiefInformation Officer Date 11/24/14 1. Introduction Information systems are vitalto Omega Research business processes; therefore,it is critical that services provided by SAP System are able to operate effectively without excessive interruption. This Information System Contingency Plan (ISCP) establishes comprehensive procedures to recover SAP System quickly and effectively following a service disruption. 1.1 Background This SAP System Information System (IS) Contingency Plan (CP) establishes procedures to recover SAP System following a disruption. The following recovery plan objectives have been established:  Maximize the effectiveness of contingency operations through an established plan that consists of the following phases:  Activation and Notification phase to activate the plan and determine the extent of damage;  Recovery phase to restore SAP System operations; and  Reconstitution phase to ensure that SAP is validated through testing and that normal operations are resumed.  Identify the activities, resources, and procedures to carry out SAP System processing requirements during prolonged interruptions to normal operations.  Assign responsibilities to designated Omega Research personnel and provide guidance for recovering SAP System during prolonged periods of interruption to normal operations.  Ensure coordination with other personnel responsible for Omega Research contingency planning strategies. Ensure coordination with externalpoints of contact and vendors associated with SAP System and execution of this plan.
  • 4. 4 1.2 Scope This ISCP has been developed for SAP System which is classified as a High-Impact system, in accordance with Federal Information Processing Standards (FIPS) 199 – Standards for Security Categorization of Federal Information and Information Systems. Procedures in this ISCP are for High- Impact systems and designed to recover SAP System within 24 hours. This plan does not address replacement or purchase of new equipment, short-term disruptions lasting less than 24 hours or loss of data at the onsite facility or at the user-desktop levels. 1.3 Assumptions The following assumptions were used when developing this ISCP:  SAP System has been established as a High-Impact System, in accordance with FIPS 199.  Alternate processing sites and offsite storage are required and have been established for this system.  Current backups of the system software and data are intact and available at the offsite storage facility in Reston, VA.  Alternate facilities have been established at Philadelphia, PA and are available if needed for relocation of SAP System.  The SAP is inoperable at the Omega Research computer center and cannot be recovered within 24 hours.  Key SAP System personnel have been identified and trained in their emergency response and recovery roles; they are available to activate the SAP System Contingency Plan.  The SAP Contingency Plan does not apply to the following situations: o Overall recovery and continuity of business operations. The Business Continuity Plan (BCP) and Continuity of Operations Plan (COOP) address continuity of business operations. o Emergency evacuation of personnel. The Occupant Emergency Plan (OEP) addresses employee evacuation.
  • 5. 5 CONTINGENCY PLANNING GUIDE FOR FEDERAL INFORMATION SYSTEMS (DRAFT) 2. Concept of Operations The Concept of Operations section provides details about SAP System, an overview of the three phases of the ISCP (Activation and Notification, Recovery,and Reconstitution), and a description of roles and responsibilities of Omega Research’s personnel during a contingency activation. 2.1 System Description San Diego Salem Reston Kansas City TDMTDMTDMTDM TDMTDMTDMTDM TDMTDMTDMTDM TDMTDMTDMTDM
  • 6. 6 Reston office: SMTP Mail Gateway, File/ Print Server, Exchange 2000 server, Web Server, DNS, RAS, PBX, workstations and printers. San Diego office: SMTP Mail Gateway, File/Print Server, Exchange 2000 mail server, DNS, RAS, PBX, workstations, and Printers. Salem office: File/Print server, VPN Gateway, workstations, and Printers. Kansas City office: File/print Server, Exchange Server, VPN Gateway, workstations, and Printers. Reston office description: Perimeter protection provided by screening router (ACLs). Remote access is provided to employees while at home or on travel through PPTP VPN, and dial-up RAS offered by a Microsoft Windows N.T. Server. All servers in the Reston office have been centrally located to a data center. The data center supports a 5-keypunch combination lock that is required to have access to the room and is controlled with HVAC for temperature with isolated services and is not on a raised floor to control static electricity. Each server and network equipment supports their own mini-UPS. Email uses Microsoft Exchange Server 2000 mail server running on a Windows 2000 Server with SMTP mail gateway. Omegaresearch.com is the domain name with a DNS server used for name resolution for public internet access. Web hosting services are provided by Windows 2000 Server running Internet Information Services (IIS). X.500 directory services are available through active directory. Server and client OS (operating system) environments have not been patched. All printers are networked. The IT department manages more than 170 workstations and 6 servers. Client machines consist of Microsoft Windows 95, 98, NT workstation 4.0, 2000, and XP. Mac operating systems (OS) including OS8 and OS-X, and Panther. Productivity applications have not been standardized. Some users enjoy Corel OfficeSuite while others appreciate Microsoft Office. San Diego office description: It is essentially a mirror of Reston architecture. The only differences are that it does not have a web server or support VPN or RAS connections. There are fewer employees here, and only one IT engineer for the entire office. There are less than 50 client machines with similar configurations as the main office. All servers have been located in an extra office and do not have HVAC for a controlled environment for humidity, temperature, or static. There is no access control in place and there are no redundant power supplies. Salem office description: Contains only 30 workstations, configured in much the same way as the rest of the company. It supports a single combined shared file and print server hosted on a Microsoft Windows NT 4.0 Server. Mail services are obtained through the San Diego office, using mailboxes setup on the San Diego Exchange Server. No publicly available networked resources. Remote access is allowed through VPN client to gateway connectivity. It has one IT engineer to manage its IT resources. All servers have been located in a spare room in San Diego. The servers have no access restriction, with no HVAC to control humidity, temperature, and static. There are no redundant power supplies. Kansas City office description: It is a similar size to the Salem office with the exception that Kansas City runs a Microsoft Exchange 2000 server for mail services. It has a local system administrator for support. All servers have been located in a spare office onsite. There is no HVAC or access restrictions. The office is not controlled for temperature, humidity, or static. There are no redundant power supplies. 2.2 Overview of Three Phases This ISCP has been developed to recover the SAP System using a three-phased approach. This approach ensures that system recovery efforts are performed in a methodical sequence to maximize the effectiveness of the recovery effort and minimize system outage time due to errors and omissions.
  • 7. 7 The three system recovery phases are: Activation and Notification Phase – Activation of the ISCP occurs after a disruption or outage that may reasonably extend beyond the RTO established for a system. The outage event may result in severe damage to the facility that houses the system, severe damage or loss of equipment, or other damage that typically results in long-term loss. Once the ISCP is activated,system owners and users are notified of a possible long-term outage,and a thorough outage assessment is performed for the system. Information from the outage assessment is presented to system owners and may be used to modify recovery procedures specific to the cause of the outage. Recovery Phase – The Recovery phase details the activities and procedures for recovery of the affected system. Activities and procedures are written at a level that an appropriately skilled technician can recover the system without intimate system knowledge. This phase includes notification and awareness escalation procedures for communication of recovery status to system owners and users. Reconstitution Phase – The Reconstitution phase defines the actions taken to test and validate system capability and functionality. This phase consists of two major activities: validating successful recovery and deactivation of the plan. During validation, the system is tested and validated as operational prior to returning operation to its normal state. Validation procedures may include functionality or regression testing, concurrent processing, and/or data validation. The system is declared recovered and operational by system owners upon successful completion of validation testing. Deactivation includes activities to notify users of system operational status. This phase also addresses recovery effort documentation, activity log finalization, incorporation of lessons learned into plan updates, and readying resources for any future recovery events.
  • 8. 8 2.3 Roles and Responsibilities The ISCP establishes severalroles for SAP System recovery and/or recovery support. Persons or teams assigned ISCP roles have been trained to respond to a contingency event affecting SAP System. The following are roles and responsibility of each ISCP Support Team: 1. ISCP Director (Bill Hermann, Executive CEO) and alternate (John Sampolous, CFO) have overall management responsibility for the plan. They are included in a Management Recovery Team. 2. ISCP Coordinator (Tim Haney, IT Manager) and alternate (Linda Okonieski, COO) initiate any needed escalations or awareness communications, oversee recovery effort,and coordinate with other recovery teams. They are included in the Recovery Team. 3. Server Recovery Team (Tyler Amdahi, Director of I.T. Operations) and (Rachid Chad, Dir. Of IT) are responsible for restoration of Web Email, desktop support, email system, and other technical support. 4. Database Recovery Team (Tiffany Sabers, CIO, Information Technology) and (Sandy Ales, Director of Sales and Marketing) are responsible for restoration of the SAP system Database. 5. Network Recovery Team (Jackson Davis, Director of Accounting) and (Fionna O’Connor, Director of Audit and Compliance) are responsible for restoration of the WAN/LAN telecommunications. 3. Activation and Notification The Activation and Notification Phase defines initial actions taken once a SAP System disruption has been detected or appears to be imminent. This phase includes activities to notify recovery personnel, conduct an outage assessment, and activate the ISCP. At the completion of the Activation and Notification Phase,SAP System ISCP staff willbe prepared to perform recovery measures to restore system functions. 3.1 Activation Criteria and Procedure The SAP System ISCP may be activated if one or more of the following criteria are met: 1. The type of outage indicates SAP System will be down for more than 24 hours 2. The facility housing SAP System is damaged and may not be available within 24 hours The following persons or roles may activate the ISCP if one or more of these criteria are met: 1. CIO (Chief Information Officer) 2. Alternate CIO Tiffany Sabers, 858-934-1245 Tim Haney, 858-234-0943 3.2 Notification The first step upon activation of the SAP System ISCP is notification of appropriate business and system support personnel. Contact information for appropriate POCs is included in the contact list Appendix A and B. For SAP System, the following method and procedure for notifications are used:
  • 9. 9 1. The Recovery Team Leads,Vendors and the ISCP Director will be contacted by the ISCP Coordinator or alternate for initial notification. The Team Leads will contact their members as shown in the notification call tree diagram. 2. When an outage occurs, Business Owners and Stakeholders should also be notified by the ISCP Coordinator. 3. Use primary email and cell number for outage notification. If these do not work, use the satellite phone. 4. See the Notification Checklist. Notification Call Tree ISCP Coordinator ISCP Coordinator Alternate SAP/Tape Backup Vendor Server Recovery Team Lead Database Recovery Team Lead Network Recovery Team Lead ISCP Director Alternate ISCP Director Database Recovery Team Member Server Team Recovery Member Network Recovery Team Member NotificationChecklist Number Task Responsibility Date/Time Status (che ck if com pleted)
  • 10. 10 1. Notify ISCP Team with call lists Refer to Notification Call Tree 2. Recovery team meets to discuss strategy ISCP Coordinator 3. Alert Vendor and Recovery Site ISCP Coordinator 4. ISCP Coordinator will be advised by Recovery Team to notify Senior Management Recovery Team Lead 5. Activate Contingency Plan CIO 6. Declare a disaster and activate plan CIO 7. Notify Hotsite and Business Owners/Stakeholders ISCP Coordinator 3.3 Outage Assessment Following notification, a thorough outage assessment is necessary to determine the extent of the disruption, any damage, and expected recovery time. This outage assessment is conducted by Disaster
  • 11. 11 Recovery Team Lead. Assessment results are provided to the ISCP Coordinator to assist in the coordination of the recovery of SAP System. Once the outages occurs, the following procedures should be followed: 1. Determine cause of the outage or disruption; 2. Assess the potential for additional disruptions or damage; 3. Determine the status of physical infrastructure (e.g.,structural integrity of computer room, condition of electric power, telecommunications, and heating, ventilation and air-conditioning [HVAC]); 4. Inventory and functional status of SAP system and other related applications like email and Web services. 5. Determine the type of damage to system equipment or data (e.g., water,fire and heat, physical impact, electrical surge) or other items which may be critical. 6. Determine items to be replaced (e.g.,hardware,software,firmware, supporting materials); and 7. Determine estimated time to restore normal services. 4. Recovery The Recovery Phase provides formal recovery operations that begin after the ISCP has been activated, outage assessments have been completed (if possible), personnel have been notified, and appropriate teams have been mobilized. Recovery Phase activities focus on implementing recovery strategies to restore system capabilities, repair damage, and resume operational capabilities at the original or new permanent location. At the completion of the Recovery Phase, SAP System will be functional and capable of performing the functions identified in the plan. 1. Temporary manual processing like using Excel spreadsheets. 2. Recovery and operation at an alternative site at Philadelphia, PA. 3. Systems with high criticality will be recovered like: Email Server,Web Server, VPN Gateway,DNS, RAS, and client machines. 4.1 Sequence of RecoveryActivities The following activities occur during recovery of SAP System: 1. The recovery location is in Philadelphia, PA at SunGard. 2. Recovery services for the SAP system will be for a server environment. The critical system resources to be recovered first are Email Server, then Web Server, then DNS, then RAS, then client machines, and last File/Print Server and PBX. 3. Retrieve backup and SAP system installation media; 4. Recover Servers and Windows N.T. Server. 5. Recover SAP system from backup and system installation media at Reston, Va. Or from Philadelphia, where are backup tapes are sent. 4.2 RecoveryProcedures
  • 12. 12 The following procedures are provided for recovery of SAP system at the established location in Philadelphia, PA . Recovery procedures are outlined per team and should be executed in the sequence presented to maintain an efficient recovery effort. To facilitate Recovery Phase operations, the ISCP should provide detailed procedures to restore the information system or components to a known state. Procedures should be assigned to the ISCP recovery teams and typically address the following actions: 1. Obtaining authorization to access damaged facilities and/or geographic area; 2. Notifying internal and external business partners associated with the SAP system; 3. Obtaining necessary office supplies and work space at Philadelphia. 4. Obtaining and installing Email Server, Web Server, DNS, RAS, and client machines. 5. Obtaining and loading backup media from Reston, VA. 6. Restoring critical SAP and Windows Server applications. 7. Restoring SAP system data to a known state; 8. Testing SAP system functionality including security controls; 9. Connecting SAP system to network or other external systems; and 10. Operating alternate equipment at SunGard in Philadelphia, PA successfully. 4.3 Escalation Notices/Awareness
  • 13. 13 Disaster recovery handling, as described in this document, starts with the escalation of an incident, which is seriously disrupting operations, to a disaster. It is important to identify the type of error causing the disruption as early aspossible, since the required recovery phases and applicable activities mainly depend on the error type. When a disaster is declared, the following main phases of recovery may be applied in this order: A. Activate the alternate site with SunGard in Philadelphia to stay in business. This can be a technical switchover to a standby system, or the activation of alternate business processing using workarounds or emergency plans. Which options are possible or applicable depends on the solutions in place and the actualtype of error. Activating a workaround will be easier and faster, if the workaround is already documented in a recovery plan. B. Prepare SAP systems for the recovery. C. If the SAP system or component is down, SAP system recovery or technical recovery, as a first step, has to reestablish technical availability of the SAP system by fixing any technical error causing the disruption. This can be done, for example, by exchanging some defect hardware component, by activating a standby system or by restoring a database from a backup. D. If all components are up (or were recovered in the previous step), logical errors inside each system have to be removed to restore integrity of each system in itself. This requires in-depth application knowledge and is a prerequisite for the next step. E. If data consistency between systems of the environment was affected,this again requires in-depth application knowledge and time to fix it. F. If data was lost and could not be recovered so far,the next effort should aim at reentering such data into the systems. G. Having finished all recovery phases, the systems and business functionality should be checked as a prerequisite for resuming regular operations. 5. Reconstitution Reconstitution is the process by which a recovered system is tested to validate system capability and functionality. During Reconstitution, recovery activities are completed and normalsystem operations are resumed. If the original facility is unrecoverable, the activities in this phase can also be applied to preparing a new permanent location to support system processing requirements. This phase consists of two major activities – validating successfulrecovery and deactivation of the plan. 5.1 Concurrent Processing
  • 14. 14 SAP Systems should be fully tested which are interlinked before allowing user to have access to the system to ensure reliability. 5.2 Validation Data Testing Validation data testing is the process of testing and validating recovered data to ensure that data files or databases have been recovered completely. The following procedures will be used to determine that the recovered data is complete and current to the last available backup: See Appendix E and J. 5.3 Validation Functionality Testing Validation functionality testing is the process of verifying that recovered SAP functionality has been tested, and the system is ready to return to normal operations. See Appendix E and J. At the successfulcompletion of the validation testing, ISCP personnelwill be prepared to declare that reconstitution efforts are complete and that the system is operating normally. This declaration may be made in a recovery/reconstitution log or other documentation of reconstitution activities. The ISCP Coordinator, in coordination with the CIO, ISSO,SAISO and with the concurrence of the Authorizing Official, must determine if the system has undergone significant change and will require reassessment and reauthorization. The utilization of a continuous monitoring strategy/program can guide the scope of the reauthorization to focus on those environment/facility controls and any other controls which would be impacted by the reconstitution efforts. 5.4 RecoveryDeclaration Upon successfully completing testing and validation, the CIO will formally declare recovery efforts complete,and that SAP System is in normal operations. SAP System business and technicalPOCs will be notified of the declaration by the ISCP Coordinator. 5.5 Notifications (users) Upon return to normal system operations, SAP System users will be notified by ISCP Coordinator using pre-planned emails, cell phones, or satellite phones. 5.6 Cleanup Cleanup is the process of cleaning up or dismantling any temporary recovery locations, restocking supplies used,returning manuals or other documentation to their original locations, and readying the system for a possible future contingency event. Return all manuals and Backup Media back to Reston, VA. 5.7 Offsite Data Storage
  • 15. 15 It is important that all backup and installation media used during recovery be returned to the offsite data storage location. The following procedures should be followed to return backup and installation media to its offsite data storage location. See Appendix F. 5.8 Data Backup As soon as reasonable following recovery,the system should be fully backed up and a new copy of the current operational system stored for future recovery efforts. This full backup is then kept with other system backups. The procedures for conducting a full system backup are: See Appendix F. Backups in Reston, VA. Backup tapes sent to Philadelphia, PA. 5.9 Event Documentation It is important that all recovery events be well-documented, including actions taken and problems encountered during the recovery effort, and lessons learned for inclusion and update to this ISCP. It is the responsibility of each recovery team or person to document their actions during the recovery effort,and to provide that documentation to the ISCP Coordinator. An after-action report with lessons learned should be documented and included for updating ISCP. Each Recovery Team (Network, Database, and Server) keep activity logs detailing recovery steps and issues. The results from Functionality and data testing: 1. Development 2. Collection 3. Approval 4. Maintenance -Testing as an audit tool. -Testing as a benchmarking. -Testing as rehearsal. 5.10 Deactivation Once all activities have been completed and documentation has been updated, the CIO will formally deactivate the ISCP recovery effort. Notification of this declaration will be provided to all business and technical POCs.
  • 16. 16 SUGGESTED APPENDICES APPENDIX A PERSONNEL CONTACT LIST SAP ISCP Key Personnel Key Personnel Contact Information ISCP Director Work 703-123-4567 Bill Hermann, CEO Home 703-567-4321 1234 Smith Street Cellular 703-954-5432 Reston, Virginia, 20190 Email Bhermann@Omega.com ISCP Director – Alternate Work 703-765-5430 John Sampolous, CFO Home 703-098-9321 1234 Moorpark Ave. Cellular 703-654-3456 Reston, Virginia, 20191 Email Jsampolous@Omega.com ISCP Coordinator Work 619-123-4567 Tim Haney IT Manager Home 858-234-4321 1234 Fenley Ave. Cellular 619-453-4321 San Diego, Ca , Email Thaney@Omega.com ISCP Coordinator – Alternate Work 503-876-6543 Linda Okonieski, COO Home 503-986-5421 3456 Winchester Blvd Cellular 503-346-8751 Salem, Oregon, 97303 Email Lokonieski@Omega.com Recovery Team – Team Lead Work 816-613-8157 Jackson Davis, Dir of Accounting Home 816-025-9641 4567 Stevens Creek Blvd Cellular 816-753-0921 Kansas City, Missouri, 64101 Email Jdavis@Omega.com Recovery Team – Team Member Work 816-036-6426 Fionna O’Connor, Dir. Of Audit Home 816-841-7631 9843 Andrea Drive Cellular 816-654-8732 Kansas City, Missouri, 64101 Email FOconnor@Omega.com APPENDIX B VENDOR CONTACT LIST Vendor Services Address Phone Contacts IBM Tape Library TSM Server 522 South Rd. Poughkeepsie, NY, 12601 214-451-7747 Steve Barretta SunGard Recovery services for server environment 401 N Broad St. Philadelphia, PA 877-456-3966 215-351-1300 -Don Meltin (test Coordinator) -Jack Fabrinni (Acct. Rep) -Lincoln Balducci (Resource Coord.) AT&T Leased Line 3495 Winchester Blvd, Reston, VA 877-345-5432 Andy Cates
  • 17. 17 APPENDIX C DETAILED RECOVERY PROCEDURES -Restore the operating system disk with Automated System Recovery (ASR). -Boot Windows Server 2003 from the installation DVD and press the F2 key. -For more information, see the Windows ASR documentation. -Restore non-operating system disks with NTBackup. -For more information, see the F1 help in NTBackup. -Restore the database. APPENDIX D ALTERNATE PROCESSING PROCEDURES An alternate processing procedure would be to use Excel spreadsheets manually until systems are back online. APPENDIX E SYSTEM VALIDATION TEST PLAN Once the system has been recovered, the following steps will be performed to validate system data and functionality: Procedure Expected Results Actual Results Successful? Performed by At the Command Prompt, type in sysname System Log-in Screen appears Log in as user testuser, using password testpass Initial Screen with Main Menu shows From Menu - select 5- Generate Report Report Generation Screen shows - Select Current Date Report - Select Weekly - Select To Screen Report is generated on screen with last successful transaction included - Select Close Report Generation Screen Shows - Select Return to Main Menu Initial Screen with Main Menu shows - Select Log-Off Log-in Screen appears
  • 18. 18 APPENDIX F ALTERNATE STORAGE, SITE, AND TELECOMMUNICATIONS Alternate Storage: The hot-site will be located in Philadelphia, PA. It is located 154 miles from the primary facility. The alternate storage facility is a third party storage provider SunGard. Points of contact is Jack Fabrianni at 877-456-3966. Those authorized to retrieve media are Don Meltin and Lincoln Balducci at 215-351- 1300. The facility will be protected with a badge and password protected lock inside and outside the building. In the case of an emergency like an earthquake, the facility may be unavailable to people outside of the country. The facility has a backup generator in case it loses power. The facility has some satellite phones available in case phone lines are down. There are servers,EDW (enterprise data warehouse),Hyperion Express, UFS (Unix file system), Brio SQR (reporting), Portal (tools), and SAP Dev DB/CI (development instance with database/centralinstance) located at the alternate site. Alternate processing site: The hot-site will be located in Philadelphia, PA. It is located 154 miles from the primary facility. The alternate storage facility is a third party storage provider SunGard. This processing site provides redundancy. Points of contact is Jack Fabrianni at 877-456-3966. Those authorized to retrieve media are Don Meltin and Lincoln Balducci at 215-351-1300. The facility will be protected with a badge and password protected lock inside and outside the building. In the case of an emergency like an earthquake,the facility may be unavailable to people outside of the country. The facility has a backup generator in case it loses power. The facility has some satellite phones available in case phone lines are down. There are servers,EDW (enterprise data warehouse),Hyperion Express, UFS (Unix file system), Brio SQR (reporting), Portal (tools), and SAP Dev DB/CI (development instance with database/centralinstance) located at the alternate processing site. It has SMTP Mail Gateway, File/Print Server,Exchange Mail 2000 Server, Web Server, DNS (Domain Name Services), RAS (Remote Access Services),and PBX(telephone). The site has an 180,000 square foot building with 20,000 square feet of office. Power is provided by PG&E. The telecommunications system is through AT&T. An SLA agreement is in place for telecommunications after 24 hours. Alternate telecommunications The vendor for the alternate telecommunications site is AT&T in Reston, VA. Andy Cates is the contact at 877-345-5432. There is another location in Reston, VA. An agreement is in place for a T1 line. The SLA says they only pay when Omega uses it and they will be reimbursed if service is down for longer than 24 hours. The people authorized to implement or use alternate telecommunications capacity are Tim Haney at 619-453-4321 and Linda Okonieski at 503-346- 8751.
  • 19. 19 APPENDIX G DIAGRAMS (SYSTEM AND INPUT/OUTPUT)
  • 20. 20 APPENDIX H SYSTEM INVENTORY Internal Omega E-mail is supported by a Microsoft Exchange ® 2000 mail server running on a Microsoft Windows ® 2000 Server. Omega has installed an SMTP mail gateway to support Internet mail exchange. Client machines consist of Microsoft Windows ® 95, 98, NT Workstation 4.0, 2000, and XP. Mac operating systems include OS/8 and OS-X, Panther.
  • 21. 21 List of resources: -SMTP Mail Gateway -File/Print Server -Exchange 2000 Mail Server -Web Server -DNS -RAS -PBX -VPN Gateway APPENDIX I INTERCONNECTIONS TABLE SAP consists of applications, a database, and other important components which depend on each other to function. Certain applications will not work if the database is down. The HR application will not work if the database is down.
  • 22. 22 APPENDIX J TEST AND MAINTENANCE SCHEDULE Full functional tests will be performed in January and February each year. Some functional tests will be performed prior to full functional tests. Notification will be email at Reston site. From backup media, will be a recovery of database or a server. Step Date Due by Responsible Party Date Scheduled Date Held Identify failover test facilitator. November 1 ISCP Coordinator October 1 Determine scope of failover test (include other systems?). November 15 ISCP Coordinator, Test Facilitator October 15 Develop failover test plan. December 1 Test Facilitator November 1 Invite participants. January 1 Test Facilitator December 1 Conduct functional test. February 1 Test Facilitator, ISCP Coordinator, POCs January 1 Finalize after action report and lessons learned. March 1 ISCP Coordinator February 1 Update ISCP based on lessons learned. April 1 ISCP Coordinator March 1 Approve and distribute updated version of ISCP. May 1 ISCP Director, ISCP Coordinator April 1 APPENDIX K ASSOCIATED PLANS AND PROCEDURES Version 1.0. The ICSP in a database in Reston. The primary point of contact is ISCP Coordinator Tim Haney. APPENDIX L BUSINESS IMPACT ANALYSIS Refer to BIA at Share point site. APPENDIX M DOCUMENT CHANGE PAGE Modifications made to this plan since the last printing are as follows: Record of Changes Page No. Change Comment Date of Change Signature Thisis the firstversionTim1.0.
  • 23. 23 References BackingUp andRestoringYour SAPSystemonWindows - SAPNetWeaverbyKeyCapability - SAP Library.(n.d.).Retrievedfrom http://help.sap.com/saphelp_nw70ehp2/helpdata/en/85/cd5e76d4b54a0d9463049e0f56f11e/content. htm Log in toSlideShare.(n.d.).Retrievedfrom http://www.slideshare.net/haney888/savedfiles?s_title=emergency-handling-for-recovery-of-sap- system-landscapes&user_login=BalakrishnaVegi