• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Business Continuity Plan
 

Business Continuity Plan

on

  • 3,800 views

 

Statistics

Views

Total Views
3,800
Views on SlideShare
3,800
Embed Views
0

Actions

Likes
1
Downloads
88
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Business Continuity Plan Business Continuity Plan Document Transcript

    • NC Department of Environment and Natural Resources Business Continuity Plan – 2004 North Carolina Department of Environment And Natural Resources Business Continuity Plan This plan meets the requirements of NCGS § 147 – 33.89. For information about Business Continuity Planning in the NC Department of Environment and Natural Resources, Contact any of the following members of the DENR Business Continuity Management Steering Team: Jimmy Carter, Secretary’s Office, DENR: (919) 733-4908 David Johnson, Information Technology Services, DENR: (919) 715-4819 Sam Kornegay, Information Technology Services, DENR: (919) 715-0376 Doug Lewis, Budget Planning and Analysis, DENR: (919) 733-1710 Jill Pafford, Secretary’s Office, DENR: (919) 715-4193 Chris Russo, Secretary’s Office, DENR: (919) 715-4169
    • NCDENR – Business Continuity Plan – 2004 Table of Contents Policy .................................................................................................................................. 4 PART 1 – Read This First! ................................................................................................ 6 Who Should Use This Plan .......................................................................................... 7 What This Plan Does .................................................................................................. 7 When This Plan Is Activated ...................................................................................... 7 Where This Plan Applies ............................................................................................. 7 How This Plan Works .................................................................................................. 7 PART 2 – General Response Procedures ........................................................................ 9 Process Description ................................................................................................... 9 Scope .......................................................................................................................... 9 Process Priority ........................................................................................................... 10 Risk/Impact/Cost ........................................................................................................ 10 Impact Reduction ........................................................................................................ 10 Recovery Organization .............................................................................................. 12 Contingency Plan Steps ............................................................................................. 12 Contingency Plan Maintenance ................................................................................. 13 PART 3 – Division Business Continuity Plans ................................................................ 15 Aquarium At Fort Fisher .............................................................................................. 15 Center For Geographic Information Analysis .............................................................. 20 Introduction ........................................................................................................ 20 Process # 1 – Accounts Receivable, Expenditure Tracking ............................... 20 Process # 2 – GIS Services and Product Delivery .............................................. 22 Process # 3 – Geographic Information Coordinating Council Support ............... 24 Division of Environmental Health ................................................................................ 27 Process Description ........................................................................................... 27 Scope .................................................................................................................. 27 Public Water Supply – Compliance Inspections ................................................. 27 Vector Control – Pest Management ................................................................... 29 Environmental Health Manpower Distribution ................................................... 33 Shellfish Sanitation and Recreational Water Quality ......................................... 36 On-Site Wastewater ........................................................................................... 39 Radiation Protection ........................................................................................... 41 Office of Education and Training ........................................................................ 45 Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 2
    • NCDENR – Business Continuity Plan – 2004 Division of Land Resources ........................................................................................ 49 Environmental Impact management – Dam Safety, Emergency Response ....... 49 Information Technology – GPS Base Station Network ...................................... 52 Division of Marine Fisheries ................................................................................ 56 Restoring Systems ............................................................................................. 61 Division of Waste Management ................................................................................ 67 Office of Public Affairs ................................................................................ 73 Zoological Park ........................................................................................................... 75 General Emergency Guidelines ......................................................................... 75 Severe Weather Emergency .............................................................................. 76 Animal Division Preparedness ........................................................................... 77 Horticulture Division Preparedness .................................................................... 79 Recovery Operations .......................................................................................... 80 Radio Communications ...................................................................................... 82 Response To Bomb Threats .............................................................................. 82 Other Emergencies ............................................................................................. 83 PART 4 – Information Technology, DENR Information Framework .............................. 85 IBEAM Disaster Recovery Plan ................................................................................. 85 Introduction ................................................................................................................ 85 Purpose ............................................................................................................... 85 Scope .................................................................................................................. 85 How To Use This Document ...................................................................... 85 System Architecture ................................................................................................... 85 Contingency Planning Measures ................................................................................ 90 Disaster Recovery Team .................................................................................... 90 Emergency Contact Information ......................................................................... 91 Tape Backup and Support Media Management ......................................................... 92 System Recovery Procedures .................................................................................... 93 APPENDIX A – Risk Assessments for Critical Business Processes .................................. 95 APPENDIX B – Business Continuity Management Team Contacts .................................. 117 APPENDIX C – Crisis Communication Plan, Alternate Work Sites .................................... 118 APPENDIX D – Computer Application Inventory ................................................................ 121 Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 3
    • NCDENR – Business Continuity Plan – 2004 POLICY The Department of Environment and Natural Resources (DENR) recognizes that some services are vital to public well-being and that disruption of those services could cause harmful, costly or irrecoverable impact. As such, DENR resolves to anticipate situations that create risk of business failure and plan maintenance / recovery operations to avoid or lessen the impact of disruption. At all levels of organization, DENR is committed to ensure operational continuity of mission critical business services and supporting information systems. Business continuity management (BCM) is necessary to maintain and/or restore critical business functions according to a well-exercised plan within a predetermined period of time following a disruption that halts business functions. COPYRIGHT © 2004 This document has been created by and for the use of the North Carolina Department of Environment and Natural Resources. Permission for reuse is granted to anyone or organization who would find it useful. Please include appropriate reference to this original document. DENR thanks the Massachusetts Institute of Technology for access to the external release of their business continuity plan found at http://web.mit.edu/security/www/pubplan.htm. MIT’s Business Continuity Plan was helpful in thinking through logical components and layout of a successful plan. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 4
    • NCDENR – Business Continuity Plan – 2004 PART 1 OVERVIEW AND ORIENTATION Read this first! Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 5
    • NCDENR – Business Continuity Plan – 2004 1 READ THIS FIRST! This Business Continuity Plan (BCP) is an active workbook that can be used to implement appropriate contingencies and recovery operations in the event of a critical business disruption. If in doubt, call the most appropriate Business Continuity Contact in your program area from this list and describe the situation. NOTE: Public versions of this plan do not disclose personal contact information. CONTACT ORGANIZATION Work Phone Home Phone George Halsey Air Quality 919-733-0247 Steven Vozzo Air Quality (FaRO) 910-486-1541 Chris Mills Aquariums 919-733-2290 Doug Lewis BPA 919-733-1710 Josh Shepherd Coastal Management 919-733-2293 x236 Bryan Brannon Controller’s Office 919-733-5204 Edythe McKinney Customer Service Center 919-733-0823 Barb Satler Disaster Response Center 919-715-0793 David Robinson Ecosystem Enhancement Program 919-715-2228 Janine Nicholson Environmental Education 919-733-0711 Mike Kelly Environmental Health 919-715-0929 Victor Harris Forest Resources 733-2162 x216 Tim Johnson Geographic Information & Analysis 919-733-2090 Joe Stroup Human Resources 919-715-4808 Mark Hughes Information Technology Services 919-715-0330 David Johnson ITS 919-715-4819 Tracy Davis Land Resources 919-733-4574 Mike Buhl Marine Fisheries 252-726-7021 x104 Steve Busack Museum of Natural Sciences 919-733-7450 x701 Carol Tingley Parks and Recreation 919-715-8691 Gary Hunt Pollution Prevention/Env. Assist. 919-715-6508 Don Reuter Public Affairs Office 919-715-4112 Tracy Rains Purchase and Services 919-715-3889 Lloyd Inman Regional Office Administration 919-715-4149 Jill Pafford Secretary’s Office 919-715-4193 Jimmy Carter Secretary’s Office 919-733-4908 Chris Russo Secretary’s Office 919-7154169 David Williams Soil and Water Conservation 919-715-6103 Linda Culpepper Waste Management 919-733-4996 x203 Tom Fransen Water Resources 919-715-0381 Jeff Poupart Water Quality 919-733-5083 x527 Mary Joan Pugh Zoo 336-879-7110 Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 6
    • NCDENR – Business Continuity Plan – 2004 ________________________________________________________________________ 1.1 Who Should Use This Plan This plan is for DENR personnel involved in maintenance and recovery of critical business processes in the event of a disruption. Each division has a business continuity management team (BCMT) whose members are the primary users of this plan. Members usually include division leadership and administrative staff. All DENR personnel should be familiar with the plan and know how to respond within the most basic recovery operations. ________________________________________________________________________ 1.2 What This Plan Does This plan provides procedural guidance for responding to disruption in critical business functions. The plan provides general instructions for basic departmental response and specific instructions for individual business units. ________________________________________________________________________ 1.3 When This Plan is Activated This plan is considered active anytime an event causes disruption of critical business functions for a period of time that will compromise the delivery of service or product (usually 8 continuous hours). Formal activation of the plan occurs when one key member of the BCMT within an affected business area is notified. ________________________________________________________________________ 1.4 Where This Plan Applies All DENR business functions located across the state and in the Governor’s Washington office fall under the implementation authority of this plan including: Central recovery operations in the Secretary’s Office All field operations in DENR’s seven regions including offices and home based employees not located directly in the regional office. Individual business units in DENR central offices DENR’s Washington Liaison Some DENR business units have emergency response roles and follow an Emergency Operation Plan (EOP) under specific conditions. All DENR EOPs are incorporated into the DENR Business Continuity Plan by reference. To the extent there is conflict in the procedures of this general departmental plan and individual EOPs used by DENR emergency response units, the individual EOPs take precedence. ________________________________________________________________________ 1.5 How This Plan Works The BCP works best when reviewed and tested before an actual disruption occurs. Responsible personnel in DENR become familiar with the procedures and improve them over time as the business need dictates. Upon activation, the user can consult the BCP for contact information and response procedures at all levels of the organization. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 7
    • NCDENR – Business Continuity Plan – 2004 PART 2 GENERAL RESPONSE PROCEDURES Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 8
    • NCDENR – Business Continuity Plan – 2004 2 General Response Procedures This section of the BCP outlines basic recovery procedures for anticipated contingencies applicable to every business unit in DENR. ________________________________________________________________________ 2.1 Process Descriptions DENR is a government agency with mandates to protect the environment and conserve natural resources. The agency operates regulatory, conservation and education programs that: Control (prohibit or minimize) human activities causing impact to human health and the environment Encourage conservation of natural resources ________________________________________________________________________ 2.2 Scope The BCP applies to all areas of DENR service delivery including internal support processes and external citizen interactive processes. This business continuity plan addresses critical business functions. Some functions are routine and others become active only in response to an emergency. 2.2.1 Degree of minimum operations Business functions included in this plan are considered critical and must be maintained or restored to 80% operational within 48 hours of a disruption. 80% operational means: Information is available at least by hard copy Communicate systems function internally and externally Key personnel can be mobilized to points of need Business transactions can be accomplished manually 2.2.2 Capability of the Plan Contingencies for critical functions can be maintained for the duration of a disruption not exceeding 240 hours (10 days). If systems are not restored after 240 hours, resources are deemed stressed and outside assistance is needed. 2.2.3 Event Scenarios DENR activates response and recovery operations for contingencies under the following scenarios: Financial commitments cannot be met (payroll, contract debt, service level agreements) Administrative services are unavailable to support critical emergency response (expedited purchase of products and services for emergency response personnel) Regulatory controls cannot be maintained (field inspections, environmental monitoring, record reviews) Law enforcement personnel cannot access patrol areas (e.g. Marine Fisheries Patrol) Health threat notifications cannot be communicated Life support systems for animals and plants entrusted to the care of DENR are disabled 2.2.4 Triggers for Activation Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 9
    • NCDENR – Business Continuity Plan – 2004 A general activation trigger is any event that disrupts: Electrical power Communication technology (print, web, voice) Mobility Information storage and retrieval systems (technology based or manual) Automated systems that control environmental conditions at DENR facilities Specific activation triggers that DENR can anticipate and are likely: Force majeure – natural uncontrollable events such as hurricanes, tornados, severe thunderstorms Short-term localized power outage Equipment malfunction Specific activation triggers that DENR can anticipate and are not likely: Intentional sabotage – individuals or groups with harmful intent (e.g. international terrorists, disgruntled employees) Resource depletion – loss of personnel ________________________________________________________________________ 2.3 Process Priority Division priority processes are identified in Part 3 of the BCP. DENR business functions and processes are categorized as follows: Critical Business Function – Must be maintained or quickly restored within 48 hours or serious consequences will threaten the future of the mission. For critical functions, maximum level of effort will be applied to maintain even minimum level of service. Essential Business Function – Must be restored within 1 week or replaced by alternate processes within 1 month to ensure a minimal level of service to citizens. Essential functions receive a moderate level of effort to maintain minimum level of service. Necessary Business Function – Can be delayed until support systems (e.g. computer, phone service, transportation) are restored. Desirable Business Function – Can be delayed for the duration of the disruption without serious consequence to the mission. ________________________________________________________________________ 2.4 Risk/Impact/Cost [Level of Risk, Degree of Impact] DENR has used the North Carolina Office of Information Technology Services on line risk assessment tool for an initial analysis of critical business processes. The department plans to routinely analyze the nature of risk and implement measures to close identified gaps. Cost of temporary and long-term remedies has been determined for each critical process DENR developed a simple Impact Analysis Matrix for each process. These documents are attached as Appendix A. High Risk / High Impact Critical Processes: Aquariums Animal Health Animal Life Support Systems Visitor Traffic Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 10
    • NCDENR – Business Continuity Plan – 2004 CGIA Account Management GIS Product Delivery Division of Air Quality Toxic Air Pollutant Monitoring Division of Coastal Management Permitting and licensing Division of Environmental Health Vector Control – Pest Management Public Water Suppy – Compliance Inspections Radiation Protection – Application Review Radiation Protection – Emergency Coordination Shellfish Sanitation – Inspections Shellfish Sanitation – Sampling and Analysis Shellfish Sanitation – Shoreline Discharge Evaluations Shellfish Sanitation – Monitoring Bacteria in Recreational Waters Shellfish Sanitation – Closing Shellfish Waters, Issuing notices at Recreational Waters Division of Land Resources Dam Safety Emergency Response Division of Marine Fisheries Marine Patrol Enforcement Division of Soil and Water Conservation Animal Waste Management Division of Waste Management Emergency Coordination and Response Information Technology Services IBEAM Systems Development RACF Administration Office of Public Affairs Crisis Communication Zoological Park Site Access (employees) Visitor Access Care of Animals Care of Plants Life Support Systems ________________________________________________________________________ 2.5 Impact Reduction Divisions have assessed processes with high risk of business failure and high consequential impact. In cases where service delivery is dependent on technology for communication, mobilization and access to vital information, DENR business units maintain routine backup, duplication capability, or manual alternatives. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 11
    • NCDENR – Business Continuity Plan – 2004 2.5.1 Communication: DENR business units ensure appropriate back up communication systems by providing mobile phones and offsite alternate office space with land line telephone access numbers. 2.5.2 Mobilization: Scenarios that affect mobilization are anticipated such as unexpected equipment failure or natural disaster conditions (e.g. flood). When mobility is compromised, yet needed for critical business functions (e.g. field assessments), vehicles are brought in from other area of the state not affected by the disruption. 2.5.3 Information Access: All computer systems are maintained with periodic backup, secure storage of information and/or remote failover capabilities. Manual printouts of vital information are updated with appropriate frequency. Administrative processes rely on mainframe systems maintained at the state level. ________________________________________________________________________ 2.6 Recovery Organization Each division in DENR maintains a roster of responsible personnel to call in the event of a business disruption. For purposes of business continuity planning, these personnel are identified as business continuity management teams (BCMT). At the department level, a disaster response center (DRC) is activated upon determination by the Secretary that high-level coordination is required. Factors that determine the need for DENR DRC activation include: Activation of the State Emergency Operations Center (EOC) Disruptions likely to extend beyond 48 hours Scenarios that involve multiple DENR programmatic areas Scenarios extending beyond the scope of a single DENR regional office ________________________________________________________________________ 2.7 Contingency Plan Steps 2.7.1 Emergency Determination and Notification [See division plans in section 3 for details.] The Secretary’s Office and each division maintain crisis communication plans including a roster of responsible personnel to be notified in the event of a business disruption. A portion of the Crisis Communication Plan includes alternate work sites and numbers for personnel. 2.7.2 Activation of Response Structure/Teams [See division plans in section 3 for details.] For major events, DENR activates in conjunction with the State Emergency Operations Center (EOC). The DENR Disaster Response Center (DRC) is opened and the contact response structure is enabled. DENR personnel from primary response divisions set up shop in the EOC. A daily situation report is generated by the DRC on the first day of activation and continues until the response is complete. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 12
    • NCDENR – Business Continuity Plan – 2004 For minor events (e.g. local power outage, temporary phone service disruption, vehicle breakdown) the response is proportional. Key personnel ensure that their contacts outside of the affected area know about the event and advise management what is being done to correct the event. The same personnel provide notice when the business process is restored. 2.7.3 Stabilization Steps [See division plans in section 3 for details.] For major events that can be anticipated (hurricane), preparatory work is done to minimize impact and maintain stability. Communication channels are opened between those close to the event and the DRC. From the DRC, resources are mobilized as needed and requested by frontline responders. DENR has the capability to locate the DRC at alternate locations in the state. DENR business units maintain emergency backup generators at key locations. 2.7.4 Recovery Steps [See division plans in section 3 for details.] The DRC and EOC continue operation as needed beyond the event to coordinate movement of resources to affected areas. Throughout response and recovery periods, proper procedure is followed to ensure Federal Emergency Management Agency (FEMA) funds are accessible. ________________________________________________________________________ 2.8 Contingency Plan Maintenance 2.8.1 Management The DENR BCP is maintained in the office of Assistant Secretary for Operations and Development. 2.8.2 Testing Division BCPs are tested at the division level. Revisions are documented periodically as testing results are determined. The DENR Disaster Response Plan is reviewed after each event and modified as needed based on the findings of an After Action Report. 2.8.3 Training BCP training occurs at the division level. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 13
    • NCDENR – Business Continuity Plan – 2004 PART 3 Division BCPs Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 14
    • NCDENR – Business Continuity Plan – 2004 3 Division BCPs This section contains BCPs for site-specific critical business functions and processes. ________________________________________________________________________ 3.1 Aquarium at Fort Fisher Business Continuity Plan & Procedures Business Continuity Plan North Carolina Aquarium at Fort Fisher Post Disaster (hurricane, severe weather, fire, computer failure) Departmental plans are outlined in the following document and are to be included and implemented as required steps to assure a seamless transition from pre and post potential disasters to effectively function. In the event that the Aquarium is closed for any of the described scenarios, the following Staff will be required To report to their work station as soon as possible to co-ordinate all necessary activities to resume operations as soon as possible. Essential Staff Director Director of Husbandry and Operations Department Curators Bldg. Supt. Security Chief Public Relations Coordinator Business Manager Director of Retail Operations, North Carolina Aquarium Society The term “non-essential staff” is not applicable to any effective recovery plan. It will be a requirement of the previously listed essential staff to communicate precisely what the objectives are pre and post disaster and to coordinate safe and expeditious return to their work station to assist in the recovery plan. The following are the departmental action plans for post disaster situations to facilitate Business Continuity. Pre-disaster protocols are outlined in the North Carolina Aquarium at Fort Fisher Emergency Operations Plan Administration Post-Disaster Business Continuity Plan 1. Required computer backups of database and programs would enable staff to work from home to insure timely paychecks and paid invoices. 2. Main Aquarium phone line only would be transferred to Admin staff member’s home using remote codes. 3. Aquarium mail would be picked up at a designated post office branch and distributed the best way possible. 4. Major purchasing would be done on line with the help of Purchase & Services in Raleigh. 5. Supervisor would notify his/her staff when it is safe to relocate to the Aquarium. 6. For a long-term situation, arrangements would be made to lease space/computers for as many staff as possible. Aquariology Department Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 15
    • NCDENR – Business Continuity Plan – 2004 Post-Disaster Business Continuity Procedures 1. Establish communication with staff. 2. Assess the status of the facility and the health of the animal collection. 3. Assess immediate care potential. 4. Determine long-term care necessary for the collection. a) If the facility is safe for the maintenance of the collection, aquariology staff will resume fully staffed and offer assistance in the recovery and cleanup of the facility. b) If the facility is not safe, or is insufficient for the care of the collection, the animals will be moved to our sister facilities and staff will assist with their daily care. If these facilities do not have room, the animals will be released. 5. Assess the condition of our food storage freezer. If problems exist, local service centers will be called to make repairs. If we experience a long-term loss of this freezer, we will contract with a local fish house to hold our supply. Education Post-Disaster Business Continuity Plan 1. Establish damage to building, grounds and collection and communicate to staff. 2. Determine what scheduled programs can be conducted off-site at Carolina Beach Recreation Center, or outdoors as appropriate. 3. Special Activities Coordinator and Registrar call registered program participants to communicate changes. 4. If programs need to be operated off-site for an extended period, the Registrar will have an off-site number established for programs. 5. Education staff will assist in preparing grounds and building for reopening. Exhibits/Design Business Continuity Procedures • Evaluate condition of exhibits: • Safety • Functionality • Access • Determine and produce all necessary emergency signage, and install as soon as possible. • Develop strategies for reconstruction and repair. • Communicate plans to all employees. OPERATIONS Business Continuity & Disaster Recover Procedures Assess damage to building, including structure and electrical, mechanical, and life safety systems. Assess damage to grounds. Inform division as to extent of the damage. Make recommendation as to course of action Assess resources Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 16
    • NCDENR – Business Continuity Plan – 2004 o Supplies o Equipment o Tools o Fuel o Parts Determine course for repairs -- in-house or contracted out o In-house - Call in appropriate staff members to make required repairs or clean-up Identify sources of repair parts, tools, supplies, etc. o Out-source – Identify contractor(s) determine scope of work Clean up grounds to ensure safety of visitors and staff Complete repairs necessary to re-occupy the building Get approval of authorities having jurisdiction for re-occupancy Public Relations Business Continuity Procedures 1. Contingency Planning/Preparations a. Maintain up-to-date media list, stored offsite, with email addresses, phone numbers, and contact names, as necessary. b. Maintain ready media kit and photo file (data file), off-site. c. Maintain portable AND off-site Emergency contact list (DENR, NCA, Off-site phone numbers) d. Identify a backup PR coordinator; provide backup lists for emergencies 2. Emergency Event a. Communicate to media • Use releases or phone calls to inform that the facility is open or closed (and expectation for reopening) • Provide updated information, as appropriate b. If unable to work on-site, advise media of: • PR Coordinator’s off-site location, phone number, email • PR Back-up’s name, location, phone number, email c. If working on-site: • Acquire photos to document damage, repairs, etc. • Maintain contact with key managers, curators, etc. Sales Services Business Continuity Procedures • No admission sales will be conducted until the aquarium is cleared as safe for reuse as a public building. • Admissions staff will assist husbandry, operations, administration, etc., as needed. • Equipment related to admissions sales, cash registers and credit card terminals will be evaluated and replaced if damaged. • If the building is without a security system and at the recommendation of the aquarium director or director of operations, the cash in the safe will be deposited to the NC State Treasurer's account. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 17
    • NCDENR – Business Continuity Plan – 2004 • When a date is determined for the resumption of admission sales, the business manager will arrange for the return of the reserve cash for the aquarium safe. • Before admission sales can begin on site, the aquarium director and the visitor services coordinator or admissions manager must approve the area for the handling of money as secure. • For the duration of cancelled prepaid events, the admissions manager will coordinate with the business manager for the appropriate refunding of earlier payments to the special activity and special event accounts. In a similar manner, if the building is forced to close during regular public hours, any patrons returning that day's admission receipt should have a refund issued. Refunds must be issued with the same method of payment as the original payment. If a credit card was used, refunds will be credited to the account. Cash and check payments will receive a check refund. Security Department Business Continuity and Disaster Sustainability Procedures • Secure facility to prevent vandalism and theft. (Assist in alarm testing or provide 24 hr security.) • Provide daily security to prevent unwanted visitor access. • Assist in money movement and securing new admissions area if needed. • Assist in inspection of Aquarium for visitor and staff safety. Gift Shop Business Continuity Plan/Procedures Pre storm: 1. All floor level merchandise raised 2 feet above floor level. 2. All glass & breakables are removed from windows and doorways. 3. Glass shelving is removed from all windows and stored in a safe place. 4. All donation boxes, pennymen machines, parabolic, duck feeders and spadefish monies are collected. 5. All monies are secured in the safe and all sales and donations are deposited. 6. Computers are brought down, turned off, unplugged and covered with plastic covers. 7. All electrical appliances, credit card terminals, phones, video cameras and recorders, etc are unplugged and covered. 8. Gift shop is locked and secured. 9. Gift shop storage is locked and secured. 10. Gift shop staff then exits the grounds. Re-entry: 1. Director of Retail Operations notifies gift shop Manager of building status and possible re- entry. 2. Gift Shop manager notifies staff to return to the building. 3. All available staff returns to the grounds for clean up and to prepare for visitors. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 18
    • NCDENR – Business Continuity Plan – 2004 4. All glass shelving is replaced. 5. Merchandise is returned to the sales floor and displayed appropriately. 6. Prepare backup drawers for cash registers. 7. Turn on all cash registers and computer terminals. 8. Plug in all electrical appliances. 9. Open the doors for business when given the okay by the Aquarium Director or Director of Operations and Husbandry. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 19
    • NCDENR – Business Continuity Plan – 2004 ________________________________________________________________________ 3.2 Center for Geographic Information Analysis Center for Geographic Information and Analysis Business Continuity Contingency Plan June 17, 2004 Introduction The Center for Geographic Information and Analysis (CGIA) is a cost-recovery agency within the Department of Environment and Natural Resources (DENR). CGIA uses geographic information system (GIS) technology to perform services and deliver products to public sector organizations and other clients. The agency also provides staff support to the Geographic Information Coordinating Council, referenced in General Statute §143-725. CGIA has two offices. The main office is located at 301 N. Wilmington Street, Suite 700, Raleigh, North Carolina. The Asheville Field Office is collocated with the DENR Asheville Regional Office at 2090 US Highway 70, Swannanoa, North Carolina. The Business Continuity Contingency Plan describes processes that need to be sustained in order for CGIA to continue operations. Since the agency relies 100% on receipts for its operations, any disruption results in lost revenue for the agency. Three primary business processes are described below along with the contingencies for handling them. The three processes are: (1) Accounts Receivable, Expenditure Tracking, Property Management, and Procurement; (2) GIS Services and Product Delivery; and (3) Geographic Information Coordinating Council Support. Process #1: Accounts Receivable, Expenditure Tracking, Property Management, and Procurement Process Description This process defines the critical financial side of CGIA operations and is performed in the main office. CGIA generates invoices on a monthly basis based on services provided under contract to a number of clients. The agency deposits those receipts and pays staff salaries, purchases equipment, and pays other debts such as supplies, telephone and network charges. Without the ability to generate invoices, deposit funds, and pay financial obligations, the agency cannot sustain itself and risks its reputation with vendors. Therefore, this business process is essential. CGIA depends on access to its own financial data which is created and processed using Quickbooks software. The agency also relies on access to the State of North Carolina financial/accounting systems including NCAS, XTND, and E-Procurement. Without access, the agency does not have complete knowledge of the financial picture that allows the agency to operate smoothly nor does it have the ability to purchase items to support those operations. Scope CGIA has identified several actions that will be taken to reduce the risk and allow the business process to continue albeit in a less than optimal mode. The agency will make a digital copy of its financial data (and duplicate paper copies of any key paper-only records) and store that data in its Asheville Field Office which is collocated with the DENR Asheville Regional Office in Swannanoa, NC. This backup will be refreshed on a weekly basis. This precaution will provide ready access to information in the QuickBooks financial software package in case of a disaster event in Raleigh. CGIA will put a plan in place in the case where Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 20
    • NCDENR – Business Continuity Plan – 2004 financial systems cannot be restored within 48 hours (two business days). This plan will involve securing a computer, printer, software, and Internet access and placing that configuration in a secondary office space located somewhere other than the vicinity of the CGIA main office (301 N. Wilmington Street, Raleigh). The location may be the CGIA Business Officer’s residence or another suitable facility designated in cooperation with DENR senior management. Access to QuickBooks and associated financial data will allow CGIA to generate new invoices. This is a critical piece of the business process. Depositing receipts and paying salaries and bills could be delayed for a few more days unless the disaster event occurred near a payroll deadline. The disaster events that are most likely to occur based on previous years are hurricanes and ice storms. Power outages have occurred for one or more days in downtown Raleigh following these types of weather events. The loss of electrical power would trigger activation of the Business Continuity Contingency Plan for this business process. Process Priority: High Risk/Impact/Cost This business process has a High Risk/High Impact associated with it because it involves the financial well being of the agency. Since CGIA earns its own receipts and pays its financial obligations with those receipts, it is impacted more by a disruption than some other agencies. A decrease in revenue resulting from a disruption could be significant to the agency, particularly if it extended beyond a few business days. The cost to put systems in place to alleviate most of the risk is less than $10,000 for temporary lease of computer equipment. The cost of alternative office space, particularly in a commercial building, could increase this cost substantially. CGIA considers this to be a larger, departmental issue that could be more cost effectively addressed at that level for all DENR agencies. Impact Reduction CGIA can reduce the impact of the disruption to its essential business functions by preparing and sustaining backup information for all of its key financial matters in the Asheville Field Office as described above. Since the agency has no control over the availability of statewide systems such as NCAS and XTND, it will assume that the responsible agency or agencies with respect to those systems are developing appropriate contingencies. Recovery Organization The following structure has been put in place to prepare for and handle disruptions of this business process. It is led by the Administrative Officer I whose working title is the CGIA Business Officer. This position is responsible for handling the day to day financial matters of the agency. The position reports to the Division Director. Business Continuity Management Responsibilities Representatives Division Director Notify DENR Management Declare normal operations Monitor state of critical process and overall recovery Administrative Officer I Head the Business Recovery Team Implement contingencies Administrative Officer I Coordinate the return of critical functions to the normal functional area Administrative Officer I, Division Maintain recovery plan; schedule training and testing Director Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 21
    • NCDENR – Business Continuity Plan – 2004 Functional Recovery Team Responsibilities Representatives Administrative Officer I Assess problems or functional pieces of the business process that are not available or executable and determine expected recovery time Office Assistant III, Computing Work with Administrative Officer I to determine the outstanding Consultant IV [Asheville Field Office financial transactions to address and list those that are upcoming in Supervisor] the near term; make backup copy of Quickbooks financial data available if necessary Administrative Officer I Meet established recovery times and priorities Administrative Officer I Report progress to the Business Continuity Management Team Contingency Plan Steps The Administrative Officer I as a result of a phone conversation with the Division Director will activate the contingency plan for this business process. The Administrative Officer I will carry out the contingency plan with support from the Office Assistant III and the Computing Consultant IV [Asheville Field Office Supervisor] and in consultation with the Division Director. Problems in executing the plan during or following a disaster event will be brought to the Division Director’s attention for resolution or for escalation through DENR senior management if necessary. The Administrative Officer I will be responsible for bringing CGIA business functions described under this process back to operational levels within 48 hours (two business days) and report to the Division Director that it has occurred. Contingency Plan Maintenance The Administrative Officer I is responsible for maintaining the contingency plan for this business process. This person will: (1) create appropriate exercises that test the plan; (2) involve appropriate staff in the tests; (3) execute those tests on an annual basis; and (4) make adjustments to the plan if necessary. Process #2: GIS Services and Product Delivery Process Description This process outlines the second major component of CGIA’s business – the delivery of GIS services to clients who are paying the agency for those services under a signed agreement or contract. CGIA recruits projects with public sector organizations and signs agreements with a scope of work, delivery schedule, deliverable products, and budget. CGIA agreements/contracts range in duration from one week to several years and range in value from less than $100 to more than $1,000,000. The agency conducts 20-30 projects concurrently with a current services staff of 15 professionals – 11 in the main office and four in the Asheville Field Office. If these staff members are not able to perform technical and management tasks on their computers, CGIA does not generate billable time. Those billable hours are required to generate invoices to clients. Revenues from those invoices keep the agency in operation. Therefore, this business process is essential. Professional staff members use GIS software from ESRI, ERDAS, and other vendors to prepare and deliver products such as software applications, databases, maps, and reports. They also use Microsoft Office tools to support their work in creation of spreadsheets, databases, reports, and other documents. Those staff access GIS data stored in a database previously referred to as the Corporate Geographic Database. This resource is used by a number of federal, state, and local government organizations. It will migrate to a new resource called NC OneMap later in 2004. Both Raleigh and Asheville staff members access the same data over the state government wide area network. Large server computers house the data that is required for projects and large-format plotting devices are used to produce maps for clients. Without the ability to access the network, other equipment, and the GIS data, staff members cease to be productive. In addition, access by external users will be impacted. Scope Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 22
    • NCDENR – Business Continuity Plan – 2004 CGIA will take a number of key actions to minimize the disruption caused by a disaster event such as a hurricane or an ice storm. The agency will relocate a copy of the Corporate Geographic Database (i.e., NC OneMap database later in 2004) to a suite of external, portable hard disk drives and store them offsite. CGIA will establish a communication plan for all staff that will be activated when necessary to ensure a quick return to productivity. To expedite access to replacement computer hardware and software, CGIA will establish agreements with critical vendors that will be activated in an emergency. Finally, the agency will institute a process to scan all original signed agreements and contracts and store them in the Asheville Field Office. The disaster events that are most likely to occur based on previous years are hurricanes and ice storms. Power outages have occurred for one or more days in downtown Raleigh following these types of weather events. The loss of electrical power would trigger activation of the Business Continuity Contingency Plan for this business process. Full execution of the plan for this business process would depend on the extent to which individual CGIA staff members were impacted by the disaster event at their homes. Their inability to leave their home could prevent them from being productive regardless of whether replacement hardware and/or software are available for them to use. Process Priority: High Risk/Impact/Cost This business process has a High Risk/High Impact associated with it because the ability of professional staff to generate billable hours and meet contract obligations is critical to CGIA’s ability to sustain itself financially. A decrease in the agency’s ability to generate revenue resulting from a disruption could be significant. One week of billable staff time is approximately $30,000 in revenue at current CGIA rates. The cost to put systems in place to alleviate most of the risk is between $30,000 and $50,000 for replacement equipment (including temporary desktops/laptops and a temporary server). The cost of alternative office space, particularly in a commercial building, could increase this cost substantially. CGIA considers this and the reality of replacing 15 computers and software to be a larger, departmental issue that could be more cost effectively addressed at that level for all DENR agencies. Impact Reduction CGIA can reduce the impact of a disruption by relocating a copy of its GIS database to a suite of external, portable hard disk drives and store them offsite. The agency will work to establish agreements with critical vendors for computers and software to help CGIA staff return to normal productivity quickly. (Note: These agreements will be put in place if DENR does not pursue this preventive action as a department.) CGIA can also reduce the impact caused by a catastrophic event by scanning a signed original of all contracts with clients for services (i.e., typically handled through a memorandum of agreement or MOA) and storing that information offsite. Finally, CGIA will establish a more specific communication plan for staff which would be activated in a catastrophic event. This would help each staff member understand the process for becoming productive again after an event. Recovery Organization The following structure has been defined to prepare for and handle disruptions of this business process. It is led by the Applications Development Project Supervisor whose working title is CGIA Services Program Manager. This position is responsible for managing the GIS services and product delivery function of the agency. The position reports to the Division Director. Business Continuity Management Responsibilities Representatives Division Director Notify DENR Management Declare normal operations Monitor state of critical process and overall recovery Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 23
    • NCDENR – Business Continuity Plan – 2004 Applications Development Project Head the Business Recovery Team Supervisor Implement contingencies Systems Programmer/Administrator I Coordinate the return of hardware and software to the normal functional area Database Analyst Coordinate the return of the database to the normal functional area Applications Programmer II (position Coordinate the return of CGIA-hosted web mapping applications to #60005) the normal functional area Applications Development Project Maintain recovery plan; schedule training and testing Supervisor, Division Director Functional Recovery Team Responsibilities Representatives Applications Development Project Assess problems or functional pieces of the business process that are Supervisor not available or executable and determine expected recovery time Systems Programmer/ Administrator I, Work with Applications Development Project Supervisor to Database Analyst, Applications determine near term approach to minimize disruption Programmer II (position #60005) Applications Development Project Meet established recovery times and priorities Supervisor Applications Development Project Report progress to the Business Continuity Management Team Supervisor Contingency Plan Steps The contingency plan for this business process will be activated by the Applications Development Project Supervisor as a result of a phone conversation with the Division Director. The Applications Development Project Supervisor will carry out the contingency plan with support from the Systems Programmer/Administrator I, Database Analyst and Applications Programmer II (position #60005), and in consultation with the Division Director. Problems in executing the plan during or following a disaster event will be brought to the Division Director’s attention for resolution or for escalation through DENR senior management if necessary. The Applications Development Project Supervisor will be responsible for bringing CGIA business functions described under this process back to operational levels within one week (five business days) and report to the Division Director that it has occurred. Contingency Plan Maintenance The Applications Development Project Supervisor is responsible for maintaining the contingency plan for this business process. This person will: (1) create appropriate exercises that test the plan; (2) involve appropriate staff in the tests; (3) execute those tests on an annual basis; and (4) make adjustments to the plan if necessary. Process #3: Geographic Information Coordinating Council Support Process Description This process defines the final portion of CGIA’s key business functions. The agency provides staff support to the Geographic Information Coordinating Council (GICC) which is a legislated body through General Statute §143-725. The Council consists of 33 members either named in the statute or appointed by the Governor or the General Assembly. In addition, the GICC has six standing committees who work on issues between quarterly Council meetings to advance technical and policy initiatives. The agency has five staff members who support the GICC; four of those individuals are located in the CGIA main office and the fifth is located in the Asheville Field Office. In the event of a disruption, CGIA’s support of the GICC and the six subcommittees would be delayed if access to digital and paper files, computers (and email), and the Internet were affected. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 24
    • NCDENR – Business Continuity Plan – 2004 Staff members primarily use Microsoft Office tools to support their work in the creation of spreadsheets, databases, reports, and other documents. A subset of those staff members access GIS data stored in a database previously referred to as the Corporate Geographic Database that will migrate to a new version called NC OneMap later in 2004. Both Raleigh and Asheville staff members access the data over the state government wide area network. This function of the agency does not directly generate revenue. Staff members are not billable to client projects. This business process is considered to be necessary since a disruption of one week or less would not significantly impact CGIA or the overall activities of the GICC. Scope CGIA will take specific actions to reduce the risk associated with a disruption in support of the GICC. The agency will perform an examination of existing paper and digital files and determine those that are critical to GICC operations either due to their current value or historical value. Copies will be made of those files and they will be stored in the Asheville Field Office. CGIA will establish a communication plan for the five staff members who work with the GICC or the subcommittees. This plan will be activated when necessary to ensure a quick return to productivity. To expedite access to replacement computer hardware and software, CGIA will work on establishing agreements with vendors that will be activated in an emergency. The disaster events that are most likely to occur based on previous years are hurricanes and ice storms. Power outages have occurred for one or more days in downtown Raleigh following these types of weather events. The loss of electrical power would trigger activation of the Business Continuity Contingency Plan for this business process. Full execution of the plan for this business process would depend on the extent to which individual CGIA staff members were impacted by the disaster event at their homes. Their inability to leave their home could prevent them from being productive regardless of whether replacement hardware and/or software are available for them to use. Process Priority: Medium Risk/Impact/Cost This business process has a High Risk/Medium Impact associated with it because backup systems and contingencies are not currently in place to handle a disruption in the work supporting the GICC. It is not a process that directly generates or handles financial matters that keep the agency operating and is therefore considered to be medium impact. The cost to obtain temporary desktop/laptop computers through a lease agreement to alleviate the risk is approximately $10,000. The cost of alternative office space, particularly in a commercial building, could increase this cost substantially. CGIA considers this and the reality of replacing five computers and software to be a larger, departmental issue that could be more cost effectively addressed at that level for all DENR agencies. Impact Reduction CGIA can reduce the impact of a disruption by evaluating which files related to the GICC are critical to access. Those files will be copied in paper form or scanned into digital documents. Copies will be made of files that are already digital. CGIA will also establish a communication plan for staff that supports the GICC. This plan will be activated in a catastrophic event and enable staff to become productive after the event. Recovery Organization The following structure has been put in place to prepare for and handle disruptions of this business process. It is led by the Computing Consultant V (position # 60015) whose working title is CGIA Coordination Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 25
    • NCDENR – Business Continuity Plan – 2004 Program Manager. This position is responsible for planning and executing the effort to support the GICC and its six standing committees. The position reports to the Division Director. Business Continuity Management Responsibilities Representatives Division Director Notify DENR Management Declare normal operations Monitor state of critical process and overall recovery Computing Consultant V (position Head the Business Recovery Team #60015) Implement contingencies Computing Consultant V (position Coordinate the return of critical functions to the normal functional #60015) area Computing Consultant V (position Maintain recovery plan; schedule training and testing #60015) Functional Recovery Team Responsibilities Representatives Computing Consultant V (#60015) Assess problems or functional pieces of the business process that are not available or executable and determine expected recovery time Computing Consultant III (#6002), Work with Computing Consultant V to determine the near term Computing Consultant III (#60017) approach to minimize disruption Computing Consultant V (#60015) Meet established recovery times and priorities Computing Consultant V (#60015), Report progress to the Business Continuity Management Team Division Director Contingency Plan Steps The contingency plan for this business process will be activated by the Computing Consultant V (position #60015) as a result of a phone conversation with the Division Director. The Computing Consultant V will carry out the contingency plan with support from the Computing Consultant III (position #60017), and in consultation with the Division Director. Problems in executing the plan during or following a disaster event will be brought to the Division Director’s attention for resolution or for escalation through DENR senior management if necessary. The Computing Consultant V will be responsible for bringing CGIA business functions described under this process back to operational levels within one week (five business days) and report to the Division Director that it has occurred. Contingency Plan Maintenance The Computing Consultant V is responsible for maintaining the contingency plan for this business process. This person will: (1) create appropriate exercises that test the plan; (2) involve appropriate staff in the tests; (3) execute those tests on an annual basis; and (4) make adjustments to the plan if necessary. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 26
    • NCDENR – Business Continuity Plan – 2004 ________________________________________________________________________ 3.3 Division of Environmental Health DIVISION OF ENVIRONMENTAL HEALTH BUSINESS CONTINUITY PLAN June 2004 I. Process Description The Mission of the Division of Environmental Health is “To safeguard life, promote human health, and protect the environment through the practice of modern environmental health science, the use of technology, rules, public education, and above all, dedication to the public trust.” II. Scope a. Degree of minimum operations The division will redirect personnel as needed and should be able to maintain a minimum of 80 percent efficiency. b. Capability of the plan This plan incorporates the primary task for each of the six sections in the division and describes the business activities to continue work in the event of an emergency. c. Event Scenarios Emergency situations may occur through any number of activities, such as hurricanes, tornadoes, ice and snowstorms, or terrorist activities. Using the Disaster Notification Roster, DENR would notify the division directors or deputy of a declared emergency . The DEH would utilize its BCP alert roster to notify each of the key division staff to in turn activate their respective section/staff plans. d. Triggers for Activation The division will activate its BCP based on direction from the Governor or DENR secretary in response to the occurrence or impending occurrence of a disaster, regardless of the nature of that disaster. In the event of a loss of power or similar disaster localized to the primary or secondary offices of the division, the division director and/or deputy will activate the BCP as needed. Public Water Supply Section: Compliance/Inspections I. Process Description The compliance inspection business processes is physical, onsite contact to ensure facilities are being maintained in a compliant manner. The sub processes here are the actual sanitary survey required on a recurring frequency determined by EPA for different system types, other onsite compliance visits, and on-site technical assistance. The most important aspect of this process from a business continuity planning perspective is the ability to provide technical assistance for compliance to systems in the event of an emergency. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 27
    • NCDENR – Business Continuity Plan – 2004 II. Scope A. Degree of minimum operations We estimate that we should be able to function at 80 percent efficiency. B. Capability of the plan Business functions must be able to be restored within one to two days after an event. The responsibility for dealing with the emergency or disaster belongs to the water system since they are the ones that provide the water supply. State staff can provide critical technical support to speed recovery in a way that does not compromise public health. C. Event scenarios Physical damage from a natural disaster, such as hurricane, or terrorist event. D. Triggers for activation Loss of power to the water system. Water system contamination. Flooding of a well. III. Process Priority: High. IV. Risk/Impact/Cost Natural disasters such as hurricanes happen in North Carolina. Systems are currently completing emergency response plans as a federal requirement, that will help speed their response and recovery. Some may take mitigative measures to try to reduce risk, such as having back up power supplies. The degree of impact is disaster specific and can be mild to destruction of the water system. State action is reactive. Spending money at the state level will not mitigate risks to the water system. For state response, we have decreased the risk of not being able to provide assistance by field staff bringing paper copies of names and numbers of contact home with them in the event of an approaching storm, so that is they cannot access their office, they may still be able to contact the water systems. If the phone lines are down or busy, their ability to contact the systems is down. In a disaster area, their ability to reach water systems may not exist. The cost of providing cellular phones to field staff for contacting systems if the landlines are down is more than we can sustain. V. Impact Reduction See above. Bringing the lists home has helped. We have also provided contact information to other regional office personnel in the state to assist with telephone contact after an event. VI. Recovery Organization The business continuity management team consists of the Section Chief, Deputy Section Chief, and Branch Heads for the Compliance Services, Technical Services, and Protection and Enforcement Branches. These managers will call on other staff as needed to assist. Primary other staff will include regional office staff in the affected area. The Technical Services Branch Head is the lead contact person for the Emergency Operations Center. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 28
    • NCDENR – Business Continuity Plan – 2004 The functional recovery team is event specific, but includes the regional office staff in the impact areas, each lead by their respective regional PWS supervisor, as well as other regional office staff in non-impact regions that are paired with the impact regions for rapid assessment. VII. Contingency Plan Steps A. Emergency Determination and Notification Individual system contamination incidences are handled as part of routine operations. In an emergency situation, the emergency is determined by the Governor or the State Emergency Operations Center. B. Activation of Response Structure/teams steps We are activated in accordance with EOC policies and procedures. C. Stabilization steps Conducted in accordance with EOC policies and procedures. D. Recovery Steps Conducted in accordance with EOC policies and procedures. VIII. Contingency Plan Maintenance A. Management The Technical Services Branch Head is the Emergency Operations Center Lead Contact. He/she keeps up with any changes to the states Disaster Response Plan that might affect our operations. Management related to area assignments and pairing of regions for assistance is handled by the Section Chief and Protection and Enforcement Branch Head. B. Testing The EOC conducts periodic drills, and unfortunately, hurricanes happen with sufficient frequency to keep things current. After each event, consideration is given to PWS procedures that would benefit from updating. C. Training Limited formal training is available for staff. New staff who will man the EOC go with more experienced personnel for the initial shifts to gain familiarity. In the regions, the Regional supervisor and assistant regional supervisor are responsible for conducting training for new staff. Vector Control—Pest Management Emergency mosquito and mosquito-borne disease control Public Health Pest Management Section, DEH I. Process description We provide emergency mosquito and mosquito-borne disease control services to affected areas in NC after flooding-related disasters and assist local, state and federal agencies in the provision of Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 29
    • NCDENR – Business Continuity Plan – 2004 related services to protect public health per G.S. 130A-346 to 349. Other Section functions are subject to suspension until these emergency responses are completed. Since most such emergency responses have followed hurricane-induced flooding in the past (Hurricane Fran in 1996, Hurricane Floyd in 1999 and Hurricane Isabel in 2003, as well as several minor hurricanes) and are likely to follow hurricanes in the future, we have concentrated on the hurricane season in NC as the most likely time for these business continuity disruptions. Approximately one week before a major hurricane until 4 or more weeks post-hurricane is the usual period of normal business disruption as we implement emergency plans and follow through until the threats are controlled. Business during emergency response periods operates at levels significantly elevated from normal levels and only returns to normal levels as response and post-response phases are completed. From this perspective, the Section’s activities are substantially different from those of non-response agencies, in that we can’t just plan to return to pre-disaster business operations until the indeterminate disaster response period is over for us. A. Primary Contacts Contacts: Dr. Nolan Newton, Chief, PHPM Section 919-733-6407 Dr. Barry Engber, Medical Entomologist 919-733-6407 Dr. Bruce Harrison, Medical Entomologist 336-771-4608 x 367 Carletta Hensley, Office Manager 919-733-6407 II. Scope A. Degree of minimum operations Minimum operations needed during and after disasters will depend on the nature of the disaster and whether or not our program is involved with disaster response. If we are not involved with response (e.g., after ice storms or tornadoes), then our needs are much less than if we are involved with response. The degree of our disaster response involvement will also vary depending on the severity and areal extent of the disaster we are responding to. Operation 1. Technical assistance to local programs: Medium priority Technical assistance is normally preformed by the Raleigh office and four satellite offices: WSRO, Edenton rental lab and office, Wilmington home based office, and a home-based office in Williston. The WSRO office will be assume main office functions if the Raleigh main office is disrupted. It’s staffed by an experienced administrator and a field entomologist who have procedural guidance on-site. The three coastal offices can also serve as operations bases, but these offices are more likely to be disrupted by hurricanes. Operation 2. Administration: Medium priority The WSRO can also serve as the main administrative site if the Raleigh main offices are compromised. All needed operational materials are available there already. Operation 3. Emergency response: Highest priority The scenario in which the main office in Raleigh is disrupted and rendered incapable of managing an emergency response for some period of time has already been played out in the aftermath of Hurricane Fran. Offices were without power for several days and Raleigh-based personnel were occupied with recovery in their own homes. The WSRO will take over management of emergency mosquito-borne disease control if needed. B. Capability of the Plan Having had an emergency plan in place for almost 20 years and having exercised it under disaster conditions several times, the plan has worked well even when plans were disrupted by unforeseen factors. Our plan has been adopted, with revisions, by several states and most coastal counties in Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 30
    • NCDENR – Business Continuity Plan – 2004 NC. We find that a flexible plan that sets goals but doesn’t attempt to dictate every action works well for us. C. Event Scenarios We have continued to do business after hurricanes, power loss, leaking roofs, loss of employee homes, snow and ice storms and flooding. We have not tried the plan in response to tornado damage, but could well experience such damage. With our dispersed personnel in western, central, and coastal NC, all capable of some degree of program management, we feel that we can function after most natural disasters. Man-made disasters, either unintentional or intentional, have not been tested. D. Triggers for activation Several activation triggers can be seen coming several days in advance, hurricanes being the most obvious example. In such cases we will activate immediately and check that the plans are working properly. Other events that could disrupt our business functions are natural and man-made disasters, such as floods and terrorist attacks. Whenever we find that business functions are not operational, for whatever cause, the plans will be activated. III. Process Priority The processes outlined in this business continuity plan represent the top priority for the PHPM Section. The Sleep Products registration and licensing operations of the Section will be deferred as long as necessary for recovery, or operated at reduced levels until recovery is completed. Within this function the priorities are: 1. Find a useable base of operations if the main offices are not useable. 2. Activate the plan to bring operations back to normal. 3. Perform disaster assistance work if needed, which will elevate the level of program activity significantly above normal operating levels. IV. Risk/Impact/Cost The risk from an inability to perform emergency mosquito-borne disease control is very high. West Nile virus infection and eastern equine encephalitis can be severe diseases, and populations of people exposed to the enormous numbers of mosquitoes produced after hurricanes are at increased risks. Local mosquito control programs are likely to be disrupted or rendered incapable of performing their duties after a major storm also. Health departments and media may not be able to adequately warn citizens of the dangers if they are out of commission due to the effects of a disaster. Costs of emergency work are often in the millions of dollars, and the costs of recovery from disruptions caused by disasters can be significant. Estimated recovery costs for PHPM Section if disaster destroys office: Computers: $15,000 Peripherals: $5,000 Software: $20,000 Scientific equipment: $150,000 (mostly microscopes) Supplies: $10,000 Books: $20,000 Furniture: $25,000 Salaries: $25,000 (lost time during recovery) Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 31
    • NCDENR – Business Continuity Plan – 2004 V. Impact Reduction Reducing the impact of disruptions that do not destroy the Raleigh facility is done by dispersing the information, forms, contact info, and other needed materiel to the WSRO and the three field offices and insuring that the personnel in those locations can assume the necessary functions in the event that the central office is disrupted. Field offices are reciprocally covered by the Raleigh office also. VI. Recovery Organization Recovery for the PHPM Section is organized along the same lines as normal organization, with the Raleigh office leading recovery unless it is rendered incapable of doing so. In that case, organization shifts one level down the organizational chart to the WSRO office for implementation. VII. Contingency Plan Steps: A. Emergency determination and notification Emergency determination is a step-wise process. Disaster occurs Personnel are contacted to determine their status and ability to assume duties As soon as safe, survey of damage is made B. Activation of Response structure/teams steps Damage reported to proper agencies, e.g., DENR management, FEMA and CCPS/DEM Assessment of recovery needs Recovery needs that can’t be handled internally are conveyed to agency management Recovery plan implemented C. Stabilization steps Stop damage to equipment and supplies Check on personnel availability Assess physical damage D. Recovery steps Recover computer use Recover phone use Reconstitute files and documents as needed Reschedule personnel as needed VIII. Contingency plan maintenance A. Management Plan maintenance requires reviews on a regular basis and periodic revision as new problems are found or arise. Since our emergency mosquito control plans have been in effect since 1985, we’ve reviewed and revised them many times. Each new disaster requires some revision of plans, and FEMA procedural changes frequently cause us to revise our response plans. B. Testing Testing is done all too frequently due to our need to provide emergency response after hurricanes or other warm weather flooding. The original plan underwent changes after Hurricane Fran in 1996. Hurricanes Dennis and Floyd in 1999 caused us to restructure parts of the plan dealing with mapping. In 2003, Hurricane Isabel again caused us to restructure the plan to accommodate those concerned about aquaculture and organic farms. We’ve spent about 5 years working on spray maps to try to insure that adequate coverage of all 100 counties in NC is done properly. C. Training Training is also ongoing in the PHPM Section and can be broken down into staff training and customer training. Staff are required to review and comment, if appropriate, on each iteration of the emergency plan. Plans are discussed at staff meetings also. Due to the Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 32
    • NCDENR – Business Continuity Plan – 2004 regularity of disasters providing actual response, tabletop or other exercises have not been needed. Environmental Health Manpower Distribution Function (Mutual Aid to County Health Departments) Environmental Health Services Section I. Process Description: Coordinate mutual aid to Environmental Health Programs at local health departments in areas most affected by disasters. II. Scope: Following a disaster, the key public health issues to be addressed by the Division of Environmental Health (DEH) are to assure safe food, water, shelter and sanitation are provided and maintained for all of North Carolina’s citizens. Water distribution can be affected by power failures, flood damage to wells and water treatment plants, damage to or loss of pressure in water distribution systems and chemical contamination of ground water or surface water reservoirs. Availability of safe food can be disrupted by power outages, damage to transportation infrastructure, or flood damage to stored food stocks. Power outages can also cause food spoilage from loss of refrigeration for stored perishable food supplies. Shelter can be affected by wind and water damage to homes making temporary shelters necessary. Sanitation can be a problem from sewage backups and overflows, contaminated floodwaters, flooded out septic tank systems, loss of power to sewage treatment facilities, damage to sewerage and dead animals. DEH is responsible for assisting local health departments with restoring environmental health services. The needs of areas hardest hit by disasters often exceed the resources generally available at the county level and additional manpower must be moved into those areas to quickly restore food and water distribution, shelter and sanitation. DEH Personnel are often among the first people dispatched to help local health departments, but the bulk of assistance is soon provided by volunteers from other counties providing mutual aid. Coordination of mutual aid to restore environmental health services is a key function of the Environmental Health Services Section (EHSS) with a high impact on public health. A. Degree of Minimum Operations: When local health departments need assistance protecting the health of their citizens from environmental health threats, the minimum degree of operations is to dispatch whatever resources the local health director needs to prevent adverse health outcomes. DEH must redirect state employees and recruit volunteers from other agencies and counties to help meet those needs. B. Capability of the plan: This plan is capable of dispatching several environmental health regional specialists from the affected region within 24 hours and sometimes earlier when the disaster is predicted ahead of time. Additional regional specialists from other regions can be moved to the affected areas, usually within another day, and teams of volunteers from other agencies and counties can be functional within one week or less. C. Event Scenarios: 1. Time is of the essence for restoring safe food distribution following a disaster. Commercial restaurants and grocery stores must be visited to assess the condition of perishable foods and determine how to dispose of foods which must not be consumed. Flood damage assessments must be made to determine which foods and establishments are affected and to assure proper cleanup and replacement of affected food stocks. Recovery of commercial food distribution channels is often the first and one of the more manpower intensive environmental health activities following a disaster. EHSS generally dispatches our Environmental Health Regional Specialists to provide assistance with this process. 2. Loss of water pressure, boil water notices and advisories, and flooding of wells can affect the ability of restaurants and emergency shelters to prepare food, provide Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 33
    • NCDENR – Business Continuity Plan – 2004 drinking water or safe ice, and wash dishes or utensils. Medical facilities and group residences may have to curtail operations and evacuate if safe water is not quickly restored. Local health 3. departments must disseminate information about water use precautions, assist businesses and homeowners with disinfections of wells and distribution systems and do a tremendous amount of clearance testing for bacterial contamination in water supplies. These water supply activities are well suited for mutual aid workers from the state and other counties. 4. If conditions destroy or severely damage housing, or the threat of high water makes it necessary to evacuate low-lying areas, emergency shelters must be set up. Local health departments generally handle assessing adequacy of water supplies, sewage disposal, food safety and sanitation in shelters. Depending on other environmental health needs in the community, additional help can be provided to monitor shelter sanitation. 5. Following significant flooding, mold growth in wet buildings can become a major indoor air quality problem and require extensive cleanup and reconstruction efforts. Local health departments help by disseminating information, cleaning supplies, respirators and other assistance to homeowners and businesses. Volunteers are often needed for several weeks to months to assist with hazard mitigation during a flood cleanup. D. Triggers for Activation: Additional manpower is dispatched at the request of the local health director in the affected county. Requests should be initiated by a request for mutual aid on the EM 2000 network, but state personnel are often dispatched through less formal requests when asked for by local health directors. III. Process Priority Assuring that local health departments have the manpower to protect the public from environmental health hazards during an emergency is a high priority for DEH. IV. Risk/Impact/Cost The risk to our citizens from consuming hazardous food or water, lack of safe shelter, or exposure to unsanitary conditions is very high during some types of disasters. The potential for disease outbreaks is greatly increased by a breakdown in our normal distribution channels for food and water. Widespread sewage contamination and loss of shelter can further increase the risks to public health. The cost of providing additional environmental health professionals to serve hard- hit areas during an emergency, though considerable, has been borne by the state agencies and local health departments and their volunteers because the necessity to maintain public health is a shared responsibility which seems to be easily afforded when necessary. V. Impact Reduction Impact reduction is achieved by quickly recognizing what needs to be done and making sure local health departments receive any aid needed to meet those needs. A key component of that is communication. EHSS maintains a directory with contact information for all environmental health professionals in North Carolina. Hard copies are provided to the State EOC to assist emergency communications and the directory is accessible on the Internet when the server at the Parker Lincoln Building is operating. Environmental Health Regional Specialists generally carry cell phones which are valuable tools for communicating needs assessments. The EM 2000 network is a valuable tool for communicating and filling mutual aid requests. Currently we are able to monitor EM-2000 mutual aid requests only from downtown Raleigh and need to be able to Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 34
    • NCDENR – Business Continuity Plan – 2004 monitor those requests from the Parker Lincoln Building. Conditions at the State EOC can be somewhat chaotic and the telephones at the DEH desk are often tied up dealing with public water supply disruptions and clearing boil water advisories. Manpower is often coordinated from the Parker Lincoln Building and the ability to monitor mutual aid requests from that location would speed up the process. VI. Recovery Organization Environmental health is a shared responsibility between DEH and our authorized agents employed by local health departments statewide. Although DEH is capable of providing approximately 20 volunteers to assist local health departments, the bulk of the assistance provided comes from the 900+ local environmental health professionals who volunteer to help one another in times of great need. DEH also has working relations with other agencies including the Department of Health and Human Services, Division of Epidemiology, Division of Public Health, State Laboratory of Public Health, and countless others who help to meet the needs of local public health departments. VII. Contingency Plan Steps A. Emergency Determination and Notification: When weather conditions threaten disaster or a disaster has occurred, the State EOC is activated and DEH dispatches people to SERT to monitor requests by telephone and EM 2000 Tracker. Initial emergency notification is usually provided to DEH from the Department of Crime Control and Public Safety, Division of Emergency Management. B. Activation of Response Structure/teams Steps: Some pre-disaster positioning of state personnel and response planning is coordinated through the State Health Director in anticipation of predicted disasters. Once disaster strikes our manpower distribution functions are driven primarily by requests from local Health Directors in affected counties. Mutual aid requests are ideally sent over the EM 2000 Network, but in some cases are handled informally between the health directors and the agencies or departments providing assistance. Need assessments also are reported by public health teams in the field and we coordinate with the local health director to address those needs. C. Stabilization Steps: EHSS personnel monitor aid requests and needs assessments, solicit volunteers, match the capabilities of volunteers to the tasks needed and assist communication between the volunteers and receiving county. D. Recovery Steps: During the recovery, EHSS attempts to keep track of ongoing manpower deployments and assure that new volunteers are rotated in where ongoing needs exist. As needs change, we attempt to make appropriate substitutions of skills. Volunteers often work as teams from a particular county or agency and agree to be deployed for up to a week at a time. Inevitably we must prepare reports of ongoing efforts for briefings and status reports generated by the agencies involved in disaster relief. VIII. Contingency Plan Maintenance A. Management: DEH prepares public information press releases and handout material on a variety of environmental health threats that may be encountered during a disaster. We train personnel to respond to emergencies and assure we have people available and able to man the State EOC during an emergency. When advance warning is given, we send out information to local health departments and environmental health professionals over the e-mail lists we maintain to prepare them to assess their needs and communicate those needs to us. This also serves as a heads-up to potential volunteers that we may need them to mobilize. B. Testing: Testing is somewhat limited. We do participate in exercises coordinated by the Division of Emergency Management and, unfortunately, have been tested by several actual disasters in recent years. C. Training: North Carolina has a very well trained workforce of environmental health professionals. Through cooperation between DEH, the Environmental Health State of Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 35
    • NCDENR – Business Continuity Plan – 2004 Practice Committee, the UNC School of Public Health Office of Continuing Education and others, we conduct continuing education for environmental health specialists. That training includes training on disaster preparedness and response. The NC Public Health Association, Environmental Health Section has five educational districts which hold quarterly educational meetings and two annual district conferences for local environmental health personnel to network and share information about their ongoing efforts. This continuing education and professional network is invaluable to our teamwork during a disaster. Shellfish Sanitation and Recreational Water Quality Section (Protecting public health from consuming bad shellfish and crustacea) Office Description: The Shellfish Sanitation and Recreational Water Quality Section protect the consuming public from shellfish and crustaceans which could cause illness. Rules and regulations following national guidelines have been implemented to ensure the safety of harvesting waters and the proper sanitation of establishments that process shellfish and crustacean for sale to the general public. The recreational water quality program protects the public health by monitoring the coastal waters where people swim and play and notifying the public of swimming advisories when bacterial levels have been exceeded. I. Process description The Shellfish Sanitation and Recreational Water Quality Section is the primary entity for the inspection of shellfish and crustacean operations in the state. One of the section’s critical functions involves both inspecting and certifying operations for both inter- and intra-state shipping and sale. Further, when other states perform recalls of their product, the section is responsible for taking possession of and destroying the tainted product, to prevent a public health problem. The multimillion-dollar NC shellfish and crustacean industry depends on its reputation of safety and health as well as the capacity of the state to protect the population of residents and visitors from tainted or spoiled product from elsewhere. If section staff were unable to inspect and possibly embargo tainted product and an outbreak were to occur, people’s lives would be at risk, and the shellfish industry as well as the commercial fishing industry could be crippled. Even if the state was protecting the public health, but public perception deteriorated, the same consequences could occur. This is likely to spread beyond the shellfish industry to the wider commercial fishing industry, as was seen during the red tide several years ago. Anything that would compromise this reputation and cause adverse publicity or the erosion of public confidence in the NC shellfish and crustacean industry could cripple this segment of the economy. Another, less likely scenario could occur if the disaster struck just at the time in the fall when the inspectors were recertifying about 1/3 of the operators. Failure to certify them could cause a lapse and prevent the interstate shipping of their product, with significant economic consequences. The key business processes involve communication with other shellfish sanitation states, the federal Food and Drug Administration (FDA) and the operators, to learn of and report recalls, and to inform FDA and other states of the status of NC operators and their stock after inspection and possible embargo if temperature limits were exceeded. Notification of recalls is generally performed by e-mail from states or FDA, while certifications are generally input into a web-based system and saved on the computer network (backed up in Raleigh) and paper copies. The section also has the cooperation of the Division of Marine Fisheries’ 24-hour communication center; personnel there have access to all cell phone and home phone numbers of staff involved in inspections. All the section employees involved in inspections are issued state cell phones and also make their home phones available to their FDA and state contacts, as well as the Marine Fisheries communication center. II. Scope: Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 36
    • NCDENR – Business Continuity Plan – 2004 The section’s business continuity plan is based on achieving full operation as soon as possible, particularly with regard to communications and travel capability. There may be instances where an immediate response is needed in a specific area of the state, such as where a hurricane has affected only a portion of the state, or a recalled shipment of shellfish and/or crustaceans was delivered to a limited number of wholesale/retail outlets. Other times, problems might engulf the entire state. Returning to normal functioning requires the functioning of telephones to contact operators to assess situations and needs, the FDA for status reports and other states to receive recall notices. As noted, the Internet is needed, as recalls are generally distributed by e-mail, because of the detail but in the event of an emergency, the section could be informed by cell phone, provided prompt notification of FDA and partner states was made in an emergency. Accessible roads and four-wheel drive vehicles (to inspect and provide technical assistance to operators, and to embargo product if temperature limits are exceeded when they lost power, or if they received recalled product from interstate shipping. A. Degree of minimum operations: The section will be able to perform its essential functions, albeit on a scaled-down level, provided telephone access and trucks/roads are available. After those two things, Internet and e-mail capacity would greatly increase their abilities and decrease the likelihood of a recall, or perception that the state is not taking precautions to protect public health B. Capability of the plan The plan is designed to help the section be able to perform this critical functions as smoothly and quickly as possible in the event of a disruption. Emergency telephone numbers and cell phone numbers, the marine fisheries communications center and private vehicles and computers are all part of the plan and help insure that employees could operate in some function even if they were unable to get to the section offices. In the event that communications do not go down at the office but staff are unable to get to the office, Marine Fisheries Communications Center number is left on voicemail for use in emergencies; the center is able to contact section staff at home or on cell phones. C. Event scenarios: Loss of operators’ facilities due to hurricane or flood. Loss of office facilities for same reasons or fire. Evacuation of building due to homeland security issues, natural disasters or fire. D. Triggers for Activation: If a state of emergency or disaster area were declared for all or part of the state. Also, if communication tools and accessible roads were unavailable. If we received notice by fax, e-mail, telephone of a recall, or the imminent landfall of a hurricane or other natural disaster. III. Process Priority Priority is given to establish communication with operators in the affected areas, first by phone to assess needs and then in person, to provide technical assistance and embargo product as necessary. Contact with other states and FDA to learn of recalls and report status of NC product. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 37
    • NCDENR – Business Continuity Plan – 2004 IV. Risk/Impact/Cost Communications with operators, states and FDA are critical in the event of a disruption, to prevent the sale of tainted product and seize any that might enter NC. Also to provide status reports and assurance to FDA and other states that the program is running. Providing technical assistance and visits to operators will also be crucial in the event of loss of power or flood damage. . The multimillion-dollar NC shellfish and crustacean industry depends on its reputation of safety and health as well as the capacity of the state to protect the population of residents and visitors from tainted or spoiled product from elsewhere. If section staff were unable to inspect and possibly embargo tainted product and an outbreak were to occur, people’s lives would be at risk, and the shellfish industry as well as the commercial fishing industry could be crippled. Even if the state was protecting the public health, but public perception deteriorated, the same consequences could occur. This is likely to spread beyond the shellfish industry to the wider commercial fishing industry, as was seen during the red tide several years ago. Anything that would compromise this reputation and cause adverse publicity or the erosion of public confidence in the NC shellfish and crustacean industry could cripple this segment of the economy. It would also erode public confidence in the department and ‘state’ and its ability to protect citizens and visitors. Potential implications for commercial fisheries and agriculture as well. V. Impact Reduction The section will update this plan annually, and the cell phone lists are available to all employees and the Marine Fisheries Communications Center. Employees are willing to use personal vehicles in the event state trucks are unavailable. Lists of key state and FDA numbers are kept by the relevant employees. VI. Recovery Organization The section chief and the inspection employees are the initial responders to a disruption. In the chief’s absence, the assistance chief and program manager will be the designee. Other section staff can be called upon to make calls, collect information for the inspectors, as well as reporting to FDA and/or other states and DENR disaster response center. VII. Contingency Plan Steps A. Emergency Determination and Notification—Contact is made from the Division Director, Secretary or Governor’s Office or their representatives. B. Activation of Response Structure/teams Steps—The section is small, with 30 employees. Four employees are involved with the inspection process—one in each coastal region and one for the territory west of Hwy 95. others in the section have served in this capacity and can lend assistance as necessary. C. Stabilization steps—Include contacting key people at FDA, other states, Marine Fisheries Communications, and the operators through whatever means are available. This is delegated by the section chief or his designee. D. Recovery Steps—The return of phone operations and internet/e-mail access, if lost. Access to four-wheel-drive vehicles and accessible roads to visit the operators who lost power. Priority is placed on contacting operators to assess needs and prioritize visits, contacting FDA and other states to assure and report, and communicating with department. Section chief will initiate plan and contact inspection staff, from home if necessary. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 38
    • NCDENR – Business Continuity Plan – 2004 VIII. Contingency Plan Maintenance A. Management—The section will review and update plan (as necessary) on an annual basis- section chief will designate someone to perform this function. B. Testing—The key procedures are in place continually, as recalls could occur at any time. C. Training—New employees are oriented by other staff who’ve served in this capacity. Prior history with hurricanes and recalls has honed current staff’s skills in dealing with disruptions and emergencies. Extensive contact with operators has familiarized them with the need for section contact in an emergency.. On-Site Wastewater Section Repair or replacement of storm damaged on-site wastewater systems I. Process Description Hurricanes and lesser storms may damage or destroy on-site systems, particularly along the coast. It is important to repair or replace these systems in compliance with not only the rules and statutes governing these systems, but also in compliance with the building codes and the CAMA rules and statutes. In order to facilitate this process the section has arranged with local health departments (LHDs), building code officials, and CAMA staff to survey damaged facilities to determine whether they may be repaired or replaced. Part of this process includes the survey of the wastewater treatment and dispersal systems and possible relocation of it. II. Scope The timeliness category for this process is Category III, i.e. for the staff to function in this capacity they must have support systems such as cell phones, computers and vehicles in place. The section must have 80% of its staff field ready in order to perform this function. The section in conjunction with the other agencies can continue to carry out this function for a month or more. A major hurricane or severe northeaster is capable of causing the type of damage or destruction that will trigger the section’s response plan. III. Process Priority The priority of this process is medium, that is transportation to the beach areas must be reestablished before the process may begin. IV. Risk/Impact/Cost Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 39
    • NCDENR – Business Continuity Plan – 2004 Severe storms have struck the North Carolina coast on the average every three to four years causing hundreds of millions of dollars in damage to structures served by on-site wastewater systems. Each storm has required the mobilization of two to eight staff members to assist in the survey of systems. While there is nothing that we can do about the weather; we are better prepared to assist the public by the use of cell phones for communications and laptop computers for record keeping. In addition the division has produced a manual for dealing with various aspects of the aftermath of the storm. The costs of implementing this process are born by the appropriated budget of the section. V. Impact Reduction In preparation for the next severe coastal storm the section has ensured that each field person is fully equipped to conduct such surveys with field tools, communication gear, and the response manual to advise the public. Each of our field people has an assigned vehicle for transportation. The section can do nothing about the weather, but by training in their professional each of our field people are prepared to assist the public in case of a severe coastal storm. VI. Recovery Organization Business Continuity Management Responsibilities Representative Division Director - Notify DENR Management - Implement contingencies - Declare normal operations - Monitor state of critical processes and overall recovery Deputy Division Director - Coordinate and communicate with various recovery team members Section Chief - Head the Business Recovery Team Team Leader for Enforcement - Coordinate the return of critical and Special Projects functions to the normal functional state Team Leader for Field Services - Maintain Recovery Plan/ Schedule Training, etc. Functional Recovery Team Responsibilities Representatives Process Team Leader - Develop specific lists of damages - Track activities of field staff and allocate as emergency demands - Report progress to Business Team Regional Soil Specialist - Cooperate with local officials to Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 40
    • NCDENR – Business Continuity Plan – 2004 Or Enviro. Engineer survey, permit repairs of systems, or bring systems back onto line - Report activities to the Process Team Leader VII. Contingency Plans Steps Section Chief contacts the coastal counties effected by the storm immediately following its cessation to determine the needs of those counties for assistance from the section in repairing or replacing systems on the beaches. Should that assistance be required the Section Chief will advise the Process Team Leader to activate the plan in the specific areas requiring assistance. The Process Team Leader will notify the Regional Soil Specialist (RSS) responsible for that geographical area of the state and the Environmental Engineer (EE) responsible and any other RSSs and/or EEs needed to form local survey teams. The Local Survey Teams will establish survey areas based on the most accessible beaches first. The survey is a door- to-door determination of the condition of the structure, the condition of the system and its potential for repair/replacement, and the location of the “CAMA Line.” Once the situation in the county allows assistance to end in the opinion of the Process Team Leader and, based on data received from the field, with the concurrence of the Section Chief; staff will be withdrawn to be reallocated or stood down. VIII. Contingency Plan Management The Process Team Leader is the party responsible for managing the plan once the Section Chief activates it. Following each implementation of the plan the Process Team Leader will hold a debriefing of the personnel involved and any modifications necessary will be made. Each of the members of the field teams is a professional who performs the duties that will be required for the contingency plan and will not require periodic training to carry out the plan. The Section Chief and the Process Team Leader should review the plan annually to remain familiar with their responsibilities. Radiation Protection Section (Licensing /registering radioactive devices): I. Process Description A. Business System: Licensing and registration databases B. Business Owner: Beverly O. Hall, Section Chief C. Primary Technical Contact: Danny Zweigart D. Function: The licensing and registration databases are used to maintain facility specific information regarding all facilities that are not under exclusive federal jurisdiction that receive, possess, use, transfer, own or acquire any source of radiation within the State of North Carolina. These databases are the sole source of information utilized by Radiation Protection’s billing process as well as the information regarding the status of payment of the bills issued by the agency. II. Scope A. Degree of minimum operations Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 41
    • NCDENR – Business Continuity Plan – 2004 1. 80% efficiency required for overall IT support; however, the 20% inefficiency must be limited to Internet connectivity. 100% efficiency is needed for database performance regardless of connectivity to outside agencies. Licensing and registration databases are routinely updated on a daily basis. An interruption of 8-48 hours would have a moderate impact on operations during a normal business day. A backlog would be created that would require immediate attention upon the resumption of operations. The annual billing normally takes 3-5 days to complete, which includes the generation, printing, folding, envelope stuffing and mailing of the invoices. An interruption of 8-48 hours during the annual billing would potentially have a high negative impact on public perception, as the invoices would be received at a date later than that mandated by regulations. 2. 80% efficiency required for overall IT support; however, the 20% inefficiency must be limited to Internet connectivity. 100% efficiency is needed for database performance regardless of connectivity to outside agencies. The accounting database is utilized for the posting of payment information for all accounts receivable. The auditor’s office has mandated that checks be processed and deposited into the bank within 24 hours of receipt. An interruption of 8-48 hours would create a backlog of un-posted and/or un-deposited checks that would require immediate attention. The potential for a high negative impact on public perception could result from the agency not complying with the requirements of the Auditor’s office. B. Capability of the plan 1. Use of live data a. Assuming that external power is available to the building at 3825 Barrett Drive, all database functions would remain available since our data resides inside our router and is protected by a firewall. The internal network would continue to function as normal. Operations could continue for an indefinite period of time. b. Assuming that external power is not available to the building at 3825 Barrett Drive, all database functions would remain available for the approximate 4-5 hours that the UPS back-up would supply power to the network. Reducing the number of computers connected to the network to the minimum necessary to continue essential functions could extend that time by as much as 2 hours. 2. Use of back-up data a. Assuming that the network is down but external power is available to 3825 Barrett Drive, databases can be configured to run on individual PCs in order to complete essential functions. Operations could continue for an indefinite period of time. Once normal network operation has been restored, data would have to be merged back into the live data system. b. Assuming that the network is down and external power is not available to 3825 Barrett Drive, databases can be configured to run on laptop computers to complete essential functions. Operations could continue until the battery supply is exhausted. Once normal network operation and power have been restored, data would have to be merged back into the live data system. C. Event Scenarios 1. Inability to generate, print, and mail invoices. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 42
    • NCDENR – Business Continuity Plan – 2004 2. Inability to post payments and deposit checks in required timeframe. 3. Inability to update license and registration information. 4. Inability to generate reports regarding licenses or registrations. D. Event Triggers 1. Loss of power 2. Loss of electronic data 3. Loss of access to or use of the offices at 3825 Barrett Drive III. Process Priority A. Assessment of the Disruption Factors indicates that restoration of normal business is a HIGH priority. B. Assessment of the Business Impact Factors indicates that a HIGH potential exists for a threat to Citizen Health/Environment by delaying the issuance of licenses and registrations to facilities, particularly those providing health care support. Associated with delays in the provision of these licenses and registrations is a threat to the Public Trust as the agency could be viewed as an impediment to the provision of health services. IV. Risk/Impact/Cost A. Risk: The Risk Level is HIGH due to the statewide nature of the business process. Support for database development and maintenance is an internal process for Radiation Protection. In the event of a catastrophic interruption of normal business, the resources normally assigned to database maintenance and support could be reassigned to other critical duties. IT support is 100% dedicated to hardware, connectivity and software compatibility issues and would likely not be available for database maintenance support in the event of a catastrophic interruption. No additional IT positions are available or funded to assume the database maintenance and support duties during such a business process interruption. B. Impact: A high level of confidence in this agency exists with the regulated community and the citizens of North Carolina. The primary impact would likely be damage to the Public Trust. We currently have no direct dependencies on other agencies in this area. A dependency on DEH at Parker-Lincoln could exist if all three-event triggers occur at the same time; however, the likelihood of this is extremely low. C. Cost: The overall cost to the agency would be less than $10,000. V. Impact Reduction A. The live data is maintained on a separate server from other business functions. Access is limited to those personnel directly responsible for maintenance and support of the databases. The data is copied daily to tape and an internal server. In addition, a copy is made to the network for server so that the staff can use data that is no more than one-day- old for other work purposes. A copy of the data is also sent to DEH at Parker-Lincoln. Each Branch has individuals assigned the responsibility of daily back up of not only databases but also MS Word documents vital to business continuity. B. Radiation Protection maintains a sufficient number of laptop computers such that business could be conducted at remote locations once data was copied to them. Events such as billing, posting, license updates and registration updates could occur with the use of one laptop computer and one dedicated printer. In the absence of external power, billing would be delayed but the data could be updated for as long as power was available via battery. The more laptop computers that are available with properly trained staff members to operate them, the higher the impact reduction factor. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 43
    • NCDENR – Business Continuity Plan – 2004 VI. Recovery Operations A. The priority would be the restoration of access and external power to Radiation Protection; however, this would likely be outside of our immediate control. B. Once access and external power were restored, the next important order of business would be the restoration of the network. C. Once the network was restored, resources would be dedicated to merging any data that had been generated on desktop computers, laptop computers or at workstations outside of 3825 Barrett Drive. D. The final order of business would be to immediately back-up the merged data. VII. Contingency plan steps A. Emergency determination and notification: The Section Chief retains primary responsibility for the determination of the degree of interruption and any notifications to be made; however, the management team (Branch Managers) and the IT Technical Contact are an integral part of the decision making. The Section Chief would be responsible for transmitting this information to the Division Staff, if available. Updates would be provided if/when substantial changes occur with a minimum frequency of daily. B. Activation of response structure/team steps: Once the initial assessment of the event is complete, the Section Chief, IT Technical Contact and the Branch Managers would be responsible for assessing available resources, developing a plan for how to use technical and administrative staff in the notification and recovery process, and for ensuring maximum efficiency in the return to normal business. Assigning teams in generated and transmitted to the Division Staff, if possible. As a minimum, updates would be provided if/when substantial changes occur; however, it is likely that updates could occur hourly, with a minimum of daily for events with long recovery times. C. Stabilization steps: The Section Chief, IT Technical Contact and the Branch Managers would have to assign priorities to the critical work functions. Based on the circumstances of the incident, workstations would have to be assigned to the available staff such that critical work functions could be addressed as efficiently as possible. As the incident mitigates, the Section Chief, IT Technical Contact and the Branch Managers would be responsible for returning available staff to their regular workstations and priorities. The Section Chief would be responsible for transmitting this information to the Division Staff, if available. Updates would be provided if/when substantial changes occur with a minimum frequency of daily. D. Recovery steps: The Section Chief, IT Technical Contact and the Branch Managers would be responsible for identifying a team of technical staff members to utilize in the merging of data from any remote locations. Once the data is properly merged into the main system, all aspects of the databases should be checked for operability. Back-ups of the remote data will be maintained until data integrity is verified. In addition, the Section Chief, IT Technical Contact and the Branch Managers would be responsible for ensuring that each individual that is assigned to a work station is working with the correct version of the database and that all copies are removed to ensure integrity. Assuming that printers have been removed from the network in order to facilitate business continuity, the oversight team is responsible for ensuring that all equipment required for normal business is deemed fully functional prior to its attempted use. The Section Chief would be responsible for transmitting this information to the Division Staff, if available. Updates would be provided if/when substantial changes occur with a minimum frequency of daily. Contingency Plan Maintenance Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 44
    • NCDENR – Business Continuity Plan – 2004 A. Management: The Section Chief and the IT Technical Contact have primary responsibility for managing the plan. In the event that the Section Chief is unavailable, a designee can be appointed from the Branch Managers. B. Testing: At present, the contingency plan has not been fully tested. There have been occasions where electronic data has been lost and recovery from back up has been implemented. Additionally, there have been rare occasions where access to 3825 Barrett Drive has been limited or when external power has been unavailable for short periods of time, but none of these occurrences have necessitated the implementation of the contingency plan as normal work resumed in less than 4 hours. Routine testing of this process on at least an annual basis could be useful. The testing could be achieved via full or partial scale exercises that mock likely event scenarios. The tests could be performed by available in- house staff. C. Training: The training required for this business process would be minimal. Once the contingency plan is implemented, the user interface will appear exactly the same to the users so no additional training is needed for them. There are several staff members that are proficient in data manipulation; however, the training of additional staff in the details of current data structure and how to merge remote data into the live data system would be useful. This would require some advanced MS Access skills as well as practical exposure to current databases. Ideally, it would take less than one year for an astute technical staff member to gain sufficient skill to manage the required activities. If database support were to continue to be solely an in-house function, it would be wise for each individual to have a specific assigned work function that keeps his or her database skills honed to the highest degree possible. D. Improvement Opportunity: Databases could also routinely be backed up to a dedicated laptop computer which could be maintained in a secure location. In the event of a far- reaching catastrophic event, such as the extended loss of power in Raleigh, this computer could be removed to a remote location to ensure minimum business continuity. The downside of this alternative is that it would be impossible for one individual to complete all of the work functions contained in all of the databases; however, it would provide a starting place in the event of a prolonged incident. Office of Education and Training (Certification of Water Treatment Plant Operators) I. Process description The Office of Education and Training herein referred to as OET is comprised of three sections: North Carolina Water Treatment Operators Certification Board (NCWTOCB), Centralized Intern Training (CIT), and Delegation of Authority to environmental health specialist. • NCWTOCB is responsible for managing the water treatment operator certification program for North Carolina. This includes curriculum development, training, and certifying by examination operators within the state who oversee the operation and maintenance of public water facilities and systems. • CIT provides the initial training for environmental health specialists who work within the local health departments that provide inspections of food, lodging, institutions, residential septic systems, public swimming pools, tattoo parlors, child care centers, child lead prevention, pest management, water sampling, and other environmental health related activities. • Authorization is the delegation of authority granted by the Department of Environment and Natural Resources (DENR) Division of Environmental Health (DEH) to individual Environmental Health Specialist (EHS) employed by Local Health Departments to administer and enforce the NC General Statutes and the sanitation rules of the Commission of Health Services as required in GS 130A-4(b). Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 45
    • NCDENR – Business Continuity Plan – 2004 In cases of emergencies, such as natural disasters, OET provides administrative support to the other sections within the Division of Environmental Health (DEH). A. Primary Contacts Contacts: Stefanie deOlloqui, Training Manager 919-715-9571 Tony Arnold, Education and Training Specialist 919-715-9572 Kristopher Joyce, Education and Training Specialist 919-715-3205 II. Scope A. Degree of minimum operations Minimum operations needed during and after disasters will depend on the nature of the disaster and whether or not our program is involved with disaster response. We are not typically involved with initial emergency response. Depending upon the emergency situation, OET can assist in the following manners: 1. Administrative assistance for the Division: Prepare reports based on information received from the sections and submit a summarize report to the Department. Through communication with all sections, information is documented concerning their current status and any specific needs of the respective section. 2. Critical Record Maintenance: Prior to emergencies, such as a natural disaster, OET has the ability to provide specific information on emergency personnel within the local health departments. If a request is made for information regarding a certified water system operators, the NCWTOCB staff can retrieve necessary data. 3. Emergency response: If the need should arise during an emergency, OET personnel are available to assist all other sections within the Division. B. Capability of the Plan Due to the fact that request for services from OET vary, our emergency operation plans remain flexible but are centered on administrative assistance to the Division. Other than assisting other sections within the Division, and maintaining critical records, no other emergency plan is necessary. C. Event Scenarios OET has continued to do business as usual after past natural disasters, such as hurricanes, power loss, leaking roofs, snow and ice storms, and flooding. D. Triggers for activation In the event of an emergency situation, OET verifies that appropriate documents are available, and the office functions as normal unless request to provide assistance to the various sections. III. Process Priority The processes outlined in this business continuity plan represent the top priority for the OET Section. Within this function the priorities are: 1. Daily services to certified operators and permitted facilities. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 46
    • NCDENR – Business Continuity Plan – 2004 2. Daily services to environmental health specialists and local health departments. 3. Provide administrative assistance to the Division, if needed. IV. Risk/Impact/Cost The following impacts could occur if OET was incapacitated during an emergency situation: • Cancellation of operator certification examinations. • Delay of correspondence and reports to regulatory agencies, general public, local health departments, and certified water facility operators. • Delay of authorization approvals. • Cancellation of CIT. • Break down of communication between OET and the general public. • Estimated recovery costs for OET if disaster destroys office: Computers: $25,000 Peripherals: $50,000 Software: $20,000 Supplies: $15,000 Books: $20,000 Furniture: $25,000 V. Impact Reduction Reducing the impact of disruptions that does not destroy the office in which OET is located, would involve critical record maintenance and personnel availability. VI. Recovery Organization Recovery and stabilization for OET would occur as the infrastructure within the office is brought back online, and normal communication is available. VII. Contingency Plan Steps: A. Emergency determination and notification After the disaster has occurred, personnel within OET are contacted by the supervisor to determine their capacity to resume their assigned duties. Employees reporting to work can survey any damages, and communicate with other sections within the Division to determine necessary assistance. B. Activation of Response structure/teams steps Damage is reported to supervisor upon arrival to OET office. Assessment of recovery needs. Supervisor implements the recovery plan. C. Stabilization steps Prevent additional damages to critical equipment and supplies, first. Check on personnel availability. Finally, begin to minimize additional damages to non-critical equipment and supplies, and offer assistance to other sections, as needed. D. Recovery steps Recover computer use; recover phone use; reconstitute files and documents as needed; reschedule personnel as needed. Contingency plan maintenance A. Management Annually review the plan to verify applicability and efficiency. OET may adjust specific sections of the plan where unforeseen problems arise and revisions are necessary. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 47
    • NCDENR – Business Continuity Plan – 2004 B. Testing After an emergency has past, any deficiencies can be corrected and addressed in the plan. C. Training OET personnel have the responsibility of being familiar with the specific plan protocol, outlined in this document. Through staff discussions, any comments, concerns, questions, or recommendations concerning the plan should be addressed to the supervisor. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 48
    • NCDENR – Business Continuity Plan – 2004 ________________________________________________________________________ 3.4 Division of Land Resources BUSINESS CONTINUITY PLAN DIVISION OF LAND RESOURCES LAND QUALITY SECTION DATE: June 11, 2004 Environmental Impact Management: Dam Safety Emergency Response Business Process Owner: Maxwell R. Fowler, PE Requestor: James D. Simons, PG, PE Facilitator: Tracy E. Davis, PE Process Description: This critical business process is the core of the State’s Dam Safety Program. In the event of a dam failure, staff of the Division of Land Resources must mobilize, coordinate, communicate and respond immediately or loss of life and severe property damage may occur. This process relies on highly trained staff and the availability and use of the dam safety inventory database, project files, equipment, communication devices and coordination with state and local emergency management contacts. See Worksheet 1. Scope: This critical business process must be 100 % operational within 8 hours of a disruption. A maximum level of effort will be applied to maintain even a minimal level of service as citizens’ health and safety are at risk. In cases where the Dam Safety Program staff cannot be fully mobilized immediately upon notification of a dam emergency, partial mobilization of no less than 80 % of program staff within one hour should be sufficient, in addition to regular Emergency Management personnel. If the State Dam Safety Engineer and his immediate staff are unable to mobilize from their base station, other trained staff throughout the State will be contacted and mobilized, in conjunction with local and state Emergency Management personnel. In the event of a power outage at the database server host location, the dam safety inventory database will not be accessible online but may be retrieved from back up copies stored across the State. Hard copy files of the dams in imminent failure are available at the central office and the regional office responsible for the dam. Equipment is also distributed across the State in each regional office, which is readily available as needed. These interim contingencies should be sufficient for the business process to continue until full capacity is achieved. Event triggers that may cause disruption of this business process include, but may not be limited to, loss of power, loss of database, loss of equipment, loss of use of facilities, flooding, inaccessible roadways, loss of landlines and cellular phone coverage. The scenarios that would result would include, but may not be limited to, inability of staff to mobilize from home, inability of staff to access office, inability of staff to access dam emergency location, inability to access database, inability to access hard copy files, and inability to contact and coordinate with other agencies to evacuate public downstream of dam. Priority: The overall importance of this business process is HIGH, as loss of life and severe property damage could occur if this critical process is minimally disrupted. See Worksheet 2. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 49
    • NCDENR – Business Continuity Plan – 2004 Risk/Impact/Cost: The risk level for this critical business process is HIGH. The failure to provide this business process could result in significant health and safety concerns for a large number of citizens of NC. The consequences could be catastrophic and failure to respond could result in damage to the organization’s reputation and public trust. The cost to lower the risk of a business process disruption would be low, as the interim and permanent contingencies could be implemented with existing staff and budget resources. See results of online risk assessment. Impact Reduction: The following actions are being taken to accelerate emergency response and limit the impact on returning to minimum operations: 1. Entire IBEAM dam safety database will be downloaded quarterly by at least 3 regional offices outside the Raleigh area (where database is currently located) onto a local drive; 2. State Dam Safety Engineer, as a minimum, will be trained in the inner workings and administration of the IBEAM dam safety database as a backup to the current technical contact for the system; 3. Regional Offices will maintain, in perpetuity, dam safety (hard copy) files for its region’s dams as a back up to the central office files and electronic database; 4. Central Office and Regional Offices will maintain, in hard copy format, the latest IBEAM “Single Page Report” for each regulated dam in the front of each corresponding dam file as a backup to the IBEAM database system. 5. Emergency protocol will be updated with current staff’s contact information and detailed descriptions of each person’s responsibilities for dam failure response; 6. Emergency response equipment, including vehicles and cell phones, will be made available to each regional office and the central office at all times in case of an emergency. Recovery Organization: The recovery management infrastructure (and responsibilities) to execute disruption response, stabilization, and recovery requirements of the Plan is provided in this section. This infrastructure focuses on both recovery of functions to normal conditions (Business Continuity Management Team) and recovery of processes (Functional Team). Business Continuity Management (BCM) Team: The plan describes the central authority managing and coordinating the implementation and execution of actions to stabilize and eliminate disruptions. BCM Representative: Responsibilities: Division Director - Notify DENR Management - Implement contingencies - Declare normal operations - Monitor state of critical processes and overall recovery Section Chief - Coordinate and communicate with various recovery team members Dam Safety Engineer - Head the Business Recovery Team - Maintain Recovery Plan/Schedule training Asst. Dam Safety Engineer - Coordinate and assist in returning critical functions to normal conditions Regional Office Staff - Assist in returning critical functions to normal conditions Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 50
    • NCDENR – Business Continuity Plan – 2004 Functional Recovery (FR) Team: The plan describes the established teams that respond to the critical processes needing to be reestablished as required by the Contingency Plan. FR Representative: Responsibilities: State Dam Safety Engineer - Assess damages and estimate recovery time - Develop specific list of damages for the Business Continuity Management Team - Report progress to Business Continuity Management Team State Dam Safety Engineer/ - Meet established recovery times and priorities of critical Asst. Dam Safety Engineer/ processes Regional Office Staff Contingency Plan Steps: The following steps will be taken to activate and resolve the Contingency Plan: 1. Emergency Determination and Notification: The State Dam Safety Engineer will be the first contact after a business process disruption has occurred. He will notify the appropriate Dam Safety Program personnel in the central office and regional offices immediately and maintain contact with all parties, including the Division of Emergency Management, on an hourly basis by cell phone and land lines, if available. 2. Activation of Contingencies: The State Dam Safety Engineer will determine which parts of the business process have been disrupted and, through communication with Dam Safety Program regional and central office staff, what resources are available and where they are located. He will coordinate the redistribution of these available resources in order to ensure that the critical business process is restored to minimum levels and eventually to full recovery to normal operations. Once assigned their tasks, staff will activate the contingencies as efficiently and effectively as possible. Staff will communicate on an hourly basis until such contingencies are in place. 3. Stabilization: All Dam Safety Program staff will utilize the resources available to properly and timely respond to dam safety emergencies as directed by the State Dam Safety Engineer. Coordination and communication will continue on an hourly basis until the dam emergency is resolved. 4. Recovery: Once the dam safety emergency is resolved, the State Dam Safety Engineer, with the assistance and input from central office and regional office program staff, will assess the success of the Contingency Plan, inventory the items that need to be recovered/replaced, and work together to restore the business process back to its normal operation. Coordination and communication will continue on a daily basis until normal operations are achieved, and the Contingency Plan is updated and put in place. Contingency Plan Maintenance: The State Dam Safety Engineer is the owner of this critical business process. Therefore, he is the person responsible for managing, implementing and updating the Plan. Any updates to the Plan will be discussed with and approved by the Section Chief of the Land Quality Section and the Director of the Division of Land Resources. The State Dam Safety Engineer will review/test the Plan on a quarterly basis by ensuring that all components of the Plan are in place in the central and regional offices. This would include periodic Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 51
    • NCDENR – Business Continuity Plan – 2004 updates in the contact information and responsibilities outlined in the emergency protocol. New employee training will be conducted within six months of employment, routine training will be provided for existing staff through training sessions in the office and in the field, periodic meetings with Emergency Management personnel will be held, and audits of the regional offices will be conducted to ensure that backup electronic and hard (single page reports) copies of the IBEAM dam safety inventory database are in place. BUSINESS CONTINUITY PLAN DIVISION OF LAND RESOURCES GEODETIC SURVEY SECTION DATE: June 11, 2004 Information Technology: GPS Base Station Network Business Process Owner: Gary Thompson, PLS Requestor: James D. Simons, PG, PE Facilitator: Tracy E. Davis, PE Process Description: This essential business process is one of the core missions of the NC Geodetic Survey Section. Citizens, surveyors, local and state government agencies and others use this business process to complete important survey, engineering and GIS projects. Without this process, the State’s infrastructure would be adversely affected, as projects would become more costly, delays in project completion would occur, and accuracy of projects would be compromised. This process relies on a main DENR ITS FTP server that provides access to the data from the network of GPS base stations. Clientele access this server to download the GPS data they need to complete their survey and measurement projects. See Worksheet 1. NOTE: This Plan only covers the GPS Base Station Network’s remote base station sites overseen by the NC Geodetic Survey, not the DENR ITS FTP server. DENR ITS maintains the DENR ITS FTP server and, therefore, it’s maintenance and contingency plan is not addressed in this document). Scope: This essential business process must be 80 % operational within 8 to 48 hours of a disruption (this is with respect to the base station network, not the DENR ITS FTP server). It is important to note that each remote base station is located at a facility that provides electrical power and Internet access for the base station. Therefore, if there is a problem with the electrical power or Internet access that keeps the base station from working correctly, the Business Process Owner must rely on that facility to resolve the problem. The Business Process Owner is responsible only for the GPS equipment and computer at each site. In cases where one or more of the base stations is out of service, the rest of the base station network will continue to function. Therefore, the client can access the data from other nearby base stations to obtain the Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 52
    • NCDENR – Business Continuity Plan – 2004 information that is needed until the disrupted base station is back online. This may result in less accurate data and some time delays to the client, but he can continue to function and complete his project using his GPS equipment. Only when the DENR ITS FTP server is down is the entire base station network out of service. The Business Process Owner and his staff monitor the base station network on a daily basis and notify the local base station contact when the base station in question is down. The staff offers technical assistance to assist the contact in bringing the base station back on line as soon as possible. If replacement equipment is needed, the staff will travel to the base station location to install that equipment. The staff also redirects users to other base stations in the interim until the base station in question is online. This can be a matter of days but usually less than one month at a time. Event triggers that may cause disruption of this business process include, but may not be limited to, loss of power, loss of connection to database, loss of equipment, loss of use of facilities, flooding, lightning, and lack of backup power supplies. The scenarios that would result would include, but may not be limited to, inability of base station contacts or NCGS staff to access facility, inability to access faulty equipment, inability to access FTP server, and inability to contact and coordinate with other agencies to determine solution to get the base station back online. Priority: The overall importance of this business process is HIGH, as various individuals, businesses, local and state governments rely on this base station network to complete their work in a timely and cost effective way, with a high degree of accuracy. This essential business process could function at a minimal level if only a few base stations remained online while others were disrupted – if the main DENR ITS FTP server was still functioning. See Worksheet 2. Risk/Impact/Cost: The risk level for this essential business process is HIGH. The failure to provide this business process could result in time delays and cost increases in infrastructure projects across the State. While the users of this base station network are relatively small in number compared to the overall number of citizens in NC, but increasing daily, the impact to the State’s infrastructure could be significant. The consequences could result in damage to the organization’s reputation and public trust and increased costs to federal, state and local governments and private citizens. The cost to lower the risk of a business process disruption would be low, as the interim and permanent contingencies could be implemented with existing staff and budget resources (with the exception of the backup FTP server, mentioned in the next section, which will cost the Business Owner approximately $3,500). See results of online risk assessment. Impact Reduction: The following actions are being taken to accelerate emergency response and limit the impact on returning to minimum operations: 1. FTP automated monitoring program will be implemented through DENR ITS that will automatically check each base station every hour and email the Business Process Owner (main Technical Contact and a backup) and DENR ITS if there is a disruption; 2. Business Process Owner and DENR ITS are implementing backup FTP capability for the GPS base station network; 3. Business Process Owner will maintain a backup technical contact that has decision making authority in absence of primary technical contact – to assist local contact in restoring base station to full capacity; 4. Backup equipment and replacement parts, such as antennas, GPS receivers, computers, device servers, lightning protectors, etc., will be inventoried and maintained for immediate distribution to local owners if base station equipment is damaged and needs immediate replacement; Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 53
    • NCDENR – Business Continuity Plan – 2004 5. Business Process Owner adding more base stations across the State that will provide more backup coverage/redundancy in case one or more base stations in area have a disruption. Recovery Organization: The recovery management infrastructure (and responsibilities) to execute disruption response, stabilization, and recovery requirements of the Plan is provided in this section. This infrastructure focuses on both recovery of functions to normal conditions (Business Continuity Management Team) and recovery of processes (Functional Team). Business Continuity Management (BCM) Team: The plan describes the central authority managing and coordinating the implementation and execution of actions to stabilize and eliminate disruptions. BCM Representative: Responsibilities: Division Director - Notify DENR Management - Implement contingencies - Declare normal operations - Monitor state of critical processes and overall recovery Section Chief - Coordinate and communicate with various recovery team members Technical Contact - Head the Business Recovery Team - Maintain Recovery Plan/Schedule training with local base station contacts - Coordinate and assist in returning critical functions to normal conditions Local Base Station Contacts - Assist in returning critical functions to normal conditions by implementing corrective actions DENR ITS - Coordinate and assist in returning critical functions to normal conditions Functional Recovery (FR) Team: The plan describes the established teams that respond to the essential processes needing to be reestablished as required by the Contingency Plan. FR Representative: Responsibilities: Technical Contact/ - Assess damages and estimate recovery time Local Base Station Contacts - Develop specific list of damages for the Business Continuity Management Team - Report progress to Business Continuity Management Team Local Base Station Contacts - Meet established recovery times and priorities of essential processes DENR ITS - Establish recovery time for FTP server Contingency Plan Steps: The following steps will be taken to activate and resolve the Contingency Plan: 1. Emergency Determination and Notification: The Technical Contact will be the first contact to determine that a business process disruption has occurred. He will notify the appropriate Geodetic Survey Section personnel and the local base station contact of the disruption and work with the contact to determine the cause of the disruption Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 54
    • NCDENR – Business Continuity Plan – 2004 and what can be done to correct it. This is accomplished by email and telephone with as many calls as necessary to develop an agreeable and timely solution to the disruption. 2. Activation of Contingencies: The Technical Contact, through communication and coordination with the local base station contact, will offer appropriate assistance (through technical advice and replacement equipment). He will coordinate with the local contact and Geodetic Survey Section staff in the distribution of available resources in order to ensure that the essential business process is restored to minimum levels and eventually to full recovery to normal operations at the base station in question. Once assigned their tasks, both the local contact and Section staff will activate the contingencies as efficiently and effectively as possible. Staff will communicate on an hourly basis until such contingencies are in place at the base station in question. In the meantime, the Technical Contact will advise clientele, through bulletins posted on the Section’s web page, electronic messages, and by telephone, that the base station is down and redirect them to fully functional base stations in their area. If technical assistance is needed to correct the data generated from a secondary base station to meet the client’s needs, the Technical Contact will assist the client. 3. Stabilization: The Technical Contact, local base station contact and Geodetic Survey Section staff involved in the recovery will utilize the resources available to properly and timely respond to the business disruption to minimize the inconvenience to its clientele. Coordination and communication will continue on an hourly basis until the disruption is resolved. 4. Recovery: Once the necessary actions have been taken to get the base station back online, the Technical Contact, with the assistance and input from the Geodetic Survey Section staff and local base station contact involved in the recovery, will assess the success of the Contingency Plan, determine if the fixes are permanent (if not, actions will be taken to put the proper equipment in place), and work together to restore the business process back to its normal operation on a permanent basis. Coordination and communication will continue on a daily basis until normal operations are achieved, and the Contingency Plan is updated and put back in place. Contingency Plan Maintenance: The Section Chief is the owner of this essential business process. However, he has delegated the responsibility for managing, implementing and updating the Plan to the Technical Contact as he is more familiar with the technical and day-to-day operation of the base station network and its components. Any updates to the Plan will be discussed with and approved by the Section Chief of the Geodetic Survey Section and the Director of the Division of Land Resources. The Technical Contact will review/test the Plan on a quarterly basis by ensuring that all components of the Plan are in place in the Geodetic Survey Section offices and at the various base station locations. This would include periodic updates in the contact information and responsibilities of each base station contact and its staff, including back up equipment, parts and power supplies. Periodic guidance and training of base station contacts will be held to increase the likelihood that proper maintenance and contingencies are in place at each base station. Maintenance at these remote base station sites is difficult for the Geodetic Survey Section to manage as it does not have direct authority over the facilities IT network, the quality of their technical/maintenance staff, and the types of contingencies that they put in place to prevent a business process disruption (such as back up generators in the event of a power outage). Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 55
    • NCDENR – Business Continuity Plan – 2004 ________________________________________________________________________ 3.5 Division of Marine Fisheries June 10, 2004 Business Continuity Contingency Plan Process Description: Enforcement The North Carolina Division of Marine Fisheries (NCDMF) Patrol Section enforces N.C. General Statutes relating to the protection of marine resources and to the Marine Fisheries Commission rules and regulations. The Marine Patrol Section monitors compliance regarding these rules and regulations utilizing various platforms. These include vehicles, aircraft and vessels of various types. The Section is made up of three (3) Districts with each District having approximately 18 Officers (including supervisors). Support Sections include an Aviation Section with two (2) fixed wing aircraft and two (2) helicopters. A large patrol vessel supports the Districts around large bodies of water (i.e. Pamlico Sound, and the Atlantic Ocean area). A communications center provides violations, license and other crucial information to the Field Officers as needed. The Marine Patrol Section also assists with other functions when requested by other agencies such as disaster recovery, “Homeland Security”, manhunts and other searches. Scope: Degree of Minimum Operations 80% efficiency Capability of the Plan 8 Hours Event Scenarios Inability to communicate and coordinate with officers through the Communications Center Inability to verify licenses and other crucial information Disruption of normal Marine Patrol activities Triggers for Activation Loss of power Loss of use of facilities NCIC / FIN system not operational Natural or man – made disaster alert (i.e. hurricane prep, etc.) Process Priority: High Risk/Impact/Cost: Level of Risk High Impact Citizen health / environment is threatened Cost $0 – procedures and training already in place to reduce risk. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 56
    • NCDENR – Business Continuity Plan – 2004 Impact Reduction: Impact Reduction Develop and maintain a written contingency plan that outlines how to maintain communication and coordination throughout the Marine Patrol in the event of a natural or man – made disaster. Develop a method to ensure the ability to validate license information. Train all staff on the process and details of the contingency plan, annually. Keep all assigned equipment in ready mode Recovery Organization: Business Continuity Management Representative Responsibilities Division Director Notify DENR Management Implement contingencies Declare normal operations Monitor state of critical processes and overall recovery Deputy Division Director Coordinate and communicate with various recovery team members Maintain Recovery Plan/Schedule training, etc. Administrative Services Office Head the Business Recovery Team Section Chief Marine Patrol Section Chief Head the Functional Recovery Team Representatives Information Technology Section Head the Information Technology Recovery Team Representative Chief Functional Recovery Team Representatives Responsibilities Marine Patrol Section Chief, or Develop contingency plan. designee Train staff to implement contingency plan. Assess alerts or damage and estimate recovery time. If damage is IT related, work with IT Section Chief to assess damage and estimated recovery time. Contact L&S section to provide hardcopies of license information to Communications Section. Communicate alerts or damage and estimated recovery time to staff and NCDMF management. Information Technology Section Assess IT related damage and recovery time. Chief Communicate damage and estimated recovery time to L&S section, Marine Patrol Section, NCDMF management, and NCDENR ITS. Communications Staff Provide assistance to Marine Patrol Section Chief, or designee, in contacting Marine Patrol supervisors. Communicate any event or damage that prevents normal processes. Take necessary steps to secure communications equipment. Communicate any additional damage to Marine Patrol Section Chief, or designee, and, if needed, IT Section Chief through the DMF Help Desk. Maintain contact with Marine Patrol Staff throughout plan Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 57
    • NCDENR – Business Continuity Plan – 2004 via any available resource. These include, but not limited to, telephone, computer, cell phone, radio dispatch, or Mobile Data Terminal (MDT) in Patrol vehicles. Marine Patrol Major and District Inform Marine Patrol staff to implement contingency plan Captains within 2 hours of alert or damage. Communicate damage and estimated recovery time to Patrol Supervisors. Communicate any event or damage that prevents normal processes. Communicate any additional damage to Marine Patrol Section Chief, or designee, and, if needed, IT Section Chief through the DMF Help Desk. District Lieutenants and Inform Field Officers to implement contingency plan within Sergeants 2 hours of alert or damage. Communicate alerts or damage and estimated recovery time to Field Officers. Communicate any additional damage to Marine Patrol Captain, Marine Patrol Section Chief, or designee. All Marine Patrol Officers and Be very familiar with contingency plan and keep all staff equipment to be used in ready mode. This includes having vehicles, aircraft, and vessels fueled or secured. Communicate any event or damage that prevents normal processes. Implement contingency plan when informed by Marine Patrol Supervisor, Marine Patrol Section Chief, or other designee. Communicate any additional damage to Marine Patrol Supervisor, Marine Patrol Section Chief, or designee. Contingency Plan Steps: 1. Emergency Determination and Notification: Responsibility Steps Marine Patrol Advise Marine Patrol supervisors and the Communications Center of current alerts Section Chief, or or conditions and actions to be taken via any available resource. These include, but designee not limited to, telephone, computer, cell phone, radio dispatch or Mobile Data Terminal (MDT) in Patrol vehicles. Communications Provide assistance to Marine Patrol Section Chief, or designee, in contacting Staff Marine Patrol supervisors via any available resource. These include, but not limited to, telephone, computer, cell phone, radio dispatch or MDT in Patrol vehicles. Marine Patrol Advise Officers and staff of these alerts or conditions and actions via any available Supervisors resource. These include, but not limited to, telephone, computer, cell phone, radio dispatch or MDT in Patrol vehicles. 2. Activation of Response structure/team Steps Responsibility Steps Marine Patrol If Communications is without computer capability (FIN, NCIC), work Section Chief, or with IT Section Chief to assess damage and estimated recovery time. designee Oversee operations of Marine Patrol Section. Contact L&S section to provide hardcopies of license information for Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 58
    • NCDENR – Business Continuity Plan – 2004 Communications staff. Communicate any special needs (i.e. request for other agency assistance) to Marine Patrol Supervisors and to NCDMF management. IT Section Provide support as needed. L&S Section Provide support as needed. Communications Provide assistance to Marine Patrol Section Chief, or designee, in Staff contacting Marine Patrol supervisors. If hurricane, or other imminent threat, secure communications equipment, as needed. If main communications radio is inoperative, attempt to make contact with officers utilizing base unit in District II office. If this is inoperative, procure handheld unit from District II office. Utilize any available resource to communicate with Marine Patrol Officers. These include, but not limited to, telephone, computer, cell phone, radio dispatch or Mobile Data Terminal (MDT) in Patrol vehicles. For any equipment that is inoperative, take appropriate steps to contact responsible party for repairs. Utilize hardcopy of license information provided by L&S section for any request of license checks. Have supplies to manually record request available for use. Marine Patrol Provide assistance to Marine Patrol Section Chief, or designee as required. Supervisors If hurricane or other imminent threat, secure area of responsibility (i.e. facilities, aircraft, vehicles, vessels, and other equipment) as needed. Make any needed equipment (i.e. food rations, water, generator, vehicles, and vessels) to perform desired task available to Marine Patrol Officers and staff. Provide assistance to Marine Patrol staff as required. Marine Patrol Provide assistance to Marine Patrol Section Chief, or designee as required. Officers and staff If hurricane or other imminent threat, secure area of responsibility (i.e. facilities, aircraft, vehicles, vessels, and other equipment) as needed. Procure any needed equipment (i.e. food rations, water, generator, vehicles, and vessels) to perform desired service. Provide required service (i.e. security, evacuation, fisheries enforcement). Communicate any event or damage that prevents normal processes. 3. Stabilization Steps Responsibility Steps License Clerks Steps for issuing a temporary license/permit: 1. The Clerk completes the appropriate applications for the customer and has the customer sign the application and all needed Certification Statement Form for Licenses/Permits (signed and notarized). 2. To verify qualifying license information, ask to view the fisherman’s previous licenses/permits to show proof. If the fisherman presents the license/permit information that shows him eligible to hold the desired license and the identity of the Participant has been verified as the license/permit holder, then the fisherman has qualified. 3. The Clerk is to stamp the license/permit application with the ‘Temporary License’ stamp. On the provided lines, the Clerk is to write in: a. Beginning date = date of issuance. b. Expiration date = date the temporary license/ permit expires (at least 10 days, check with supervisor for number of days; this will vary depending on the reason temporary licenses/permits are being issued). Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 59
    • NCDENR – Business Continuity Plan – 2004 c. Clerk ID number and a control number (see below for explanation of issuing and tracking the control number). 4. The Clerk is to instruct the customer that the control number is the license number to be recorded on trip tickets. This number is to be utilized until the fisherman receives the permanent license with the permanent number (will be mailed to the customer). Then the customer is to use the permanent number on trip tickets and discontinue the use of the temporary license number. a. CONTROL NUMBER: License Clerks are responsible for tracking their own control numbers. One unique control number is recorded on each temporary license/permit issued. The format of the control number is 7 digits (first two digits are the Clerk’s identification number and the other 5 are sequential starting with 00001). Example: 0100001 (01 = Clerk id number, 00001 = unique sequential number) 0100002 (01 = Clerk id number, 00002 = next unique sequential number) 0100003 (01 = Clerk id number, 00003 = next unique sequential number) b. Clerks are not to issue duplicate control numbers. Clerks are to record the control number on a form along with the name and date of the license. The next available number will be used for the next temporary license and so forth. c. Control Number Tracking Form is located in the License/Permit SOPs. 5. The Clerk photocopies the paper license/permit and gives the photocopy to the customer and the NCDMF retains the original with the signed and notarized Certification Statement Form for Licenses. 6. The Clerk is to charge the individual the correct fee by consulting the correct license year License Fee Reference Sheet. The amount charged is to be itemized and totaled on a receipt. Once monies are exchanged, the customer is to be given one part of the receipt and the NCDMF retains the other part. The Clerk is to ensure the receipt also has the Clerk id, date and other necessary information recorded for auditing purposes. At the end of the day, the Clerk is to close out and balance cash drawers according to the fiscal management procedures. 7. The Clerk is to batch all appropriate applications, documents and signatures for each issued temporary license/permit. This information will be entered into FIN at a later date and the permanent license/permit will be produced and mailed to the customer. 4. Recovery Steps Responsibility Steps License Staff and License staff and any other available NCDMF staff will enter the information into any other Available FIN once the system is operational. Data entry can occur on license staff NCDMF Staff computers and the NCDMF training lab computers. All appropriate licenses (vinyl stock cards, decals, one copy of proof of purchase, etc.) will be printed and mailed to the customers. The NCDMF will file a copy of the Proof of Purchase Form along with the copies of the temporary licenses, documents and signed and notarized Certification Statement Form for Licenses/Permits. License Staff and Responsible for correcting the trip ticket information with the correct permanent Trip Ticket Staff number once the permanent license is issued. It is very important the get the correct, renewable fisherman’s, vessel, and dealer number in the system. This will require dedicated editing and re-entering data to correct, but it must be done. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 60
    • NCDENR – Business Continuity Plan – 2004 Contingency Plan Maintenance: Management License and Statistics Section Chief Testing Reviewed annually. Every time the contingency plan is invoked, an evaluation of the process will be conducted. Training License and Permit Standard Operating Procedures (SOP) are reviewed and, if necessary, updated, annually. These include the contingency plan for licenses and permits SOP training and mock sales training, annually Restoring Systems Disaster Committee A disaster recovery planning committee will be formed within the division. The chair of this Committee will be the Deputy Director. This committee will encompass all sections of the division. The purpose of this committee is to address the possibility of a disaster and how we will handle the business of doing business in an event of a disaster. Facility loss and recovery Plans to move or relocate staff and or equipment will be addressed by the Director or Deputy Director’s office at the time of the disaster. The alternative work location in the event of a catastrophic loss of the Morehead City office is the DENR Wilmington Regional Office in Wilmington. Manual procedures Each section will establish and maintain a manual set of operating procedures (manual meaning without a computer as it may be sometime before our computer system will be fully operational) for key business functions. These SOPs will be available for implementation in the case of any emergency. Employees will be aware of those operating procedures and be trained to manually implement them event of an emergency. Process priorities These functions will be restored in priority order. • Marine Patrol – Law Enforcement • License and Statistics – Access to FIN system • Information Technology – Network Admin and Support • Administration – covered under the Department Level plan for Business Continuity • Fisheries Management – Creation of Proclamations • Public Awareness – Access to Email and Internet for distribution of information Current data loss potential Note all server backups are full backups. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 61
    • NCDENR – Business Continuity Plan – 2004 Based upon a total disaster this is what we can expect to lose: • Servers: Maximum 1 week data if loss occurs on a subsequent Thursday or Friday Morning. Minimum 24hrs if loss occurs on a weekend or a Monday We are currently working on plans to have a 48-hr maximum data loss from the servers. PCs depend upon user backups. Problem determination LAN Administrator needs to determine nature and extent of problem and Report to OITS the extent of damages. If Hardware determine of replacement parts are available and replace the parts that are needed. If software locate software and restore as needed Contact Office of Information Technology Services Services or Information Technology Services (ITS) immediately 1-800-722-3946. Server priorities NetWare Dmf06 * Wins And PDC Dmf03 * Production DMF08 * CAD Dmf04 * Crystal and reporting Dmf02 * Backup Wins / IIS server or Dev01 * Test All Others Our Main server is the file server, which is needed for all logins. The DMF06 and DMF02 servers are needed for NT login. Manual copies of the server configurations, user lists, and master copies of software will be located off site in a secondary vault box at Wachovia Bank in Mitchell Village. These lists will be updated on a monthly basis. OITS will maintain an image of all the server configuration components. Testing A disaster recovery test on both servers will be done annually in April or August. This is offsite testing which doesn’t interfere with regular operations. Hardware protection All servers at DMF are currently connected to a UPS. All main Network hubs and switches are connected to a UPS. All desktop CPUs are connected to a Surge protection device. Backups schedule Backups are scheduled to be run nightly Monday- Friday at 2:AM All backups are full backups. Recovery of Seagate Crystal Reports See Seagate recovery docs for this procedure. A copy of the aforementioned document will be included with this document. Recovery of SAS Basically reloading of the SAS application on the servers. Recovery of FIN database Backups are done on a nightly basis on the databases. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 62
    • NCDENR – Business Continuity Plan – 2004 Our DBA has available documentation as to changes made in the database structure. A copy of his instructions will become a part of this document. Equipment Recovery DMF IT will determine the extent of the problem and will take action accordingly. • If equipment is under warranty a warranty replacement or repair will be made. • If parts and repair are available not to exceed the equipment value then a repair will be done. • Otherwise replacement of the equipment by the section will be the order of the day. Time spans for equipment repair/replacement will not exceed 72 hours. Backup equipment DMF IT maintains numerous PC’s in offsite storage to be used as temporary replacements for lost equipment. This equipment will be issued to sections based upon priority needs. This equipment will be maintained under IT inventory. Data recovery Data on personal desktop computers are currently the clients’ responsibility DMF IT will recover software and OS to the best of their ability for these PC’s. Software recovered will be the following • OS Win95 NT or 98 • Windows 95 patches including TCP/IP upgrades • Novell Intranetware • Mqseries • Test for y2k compliance • Asset insight • Netscape 4.7 • Adobe Acrobat 4.0 • Office 97/2000 • WordPerfect 6.1/11 • Harvard Graphics 3.0 • WS FTP lite • QWS3270 • Crystal Reports, Arcview and SAS depending upon license. A list of SAS, Crystal Reports and Arcview users are available for Helpdesk staff. • OITS is not responsible for special programs added by the user. We will assist the users as much as possible according to prior agreements with the user. Basic copies of the programs are on the LAN. However some data may not be on the LAN. Users will need to be aware that this data needs to be uploaded to the LAN for backup purposes. Time spans for data recovery on the PC currently average about 3 hrs per machine. Backup Tapes and Date Restore The DMF backup tapes will contain data and programs, although some programs may have to be restored separately The OITS MAPS, HP Recover Personnel and DMF will perform the physical restore of data from tape to servers Below is a list of data and separate software that may have to be manually restored Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 63
    • NCDENR – Business Continuity Plan – 2004 • DMF Crystal reports –04 server • DMF SAS –file server • DMF Fin Data –03 server • DMF Biological Database –file server • DMF CAD Users will be responsible for the restore Users Workstation Data Recovery Plan for NT servers Responsible party Action Completion time DMF Obtain current backup from Will depend upon access to external depository etc tapes in vault and whether we will transport them to Raleigh for a restore or not. 24 hrs If Servers will be brought to DMF with OS Software and Backup Exec already operating from Raleigh 24- 48 hrs DMF/OITS Obtain Windows NT server Within 24 hrs 4 CD DMF/OITS Install windows NT to a See above Tempnt directory Remember you are only needing it to restore the software then the whole system DMF/OITS Install current backup exec See Above software DMF Catalog backup tape 30 minutes after servers are replaced DMF Do a complete restore from Depends upon the amount the tape of data etc 3 hrs DMF Restore will overwrite files etc DMF Delete your Tempnt Directory DMF/OITS Restore applications and DBA/Programmer will be databases as necessary responsible for database restores and Crystal Reports. OITS will be responsible for restore of Mqseries etc 3-6 hrs Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 64
    • NCDENR – Business Continuity Plan – 2004 Recovery plans for Novell Responsible Party Action Completion Time DMF/OITS Install OS/NetWare, Ensure Will depend upon access to that NDS was set up as tapes in vault and whether before same server name we will transport them to and Admin password If the Raleigh for a restore or not. setup had volumes other 24 hrs than SYS that were not If Servers will be brought to affected you must purge DMF with OS Software and those files. Otherwise Backup Exec already trustee assignments may operating from Raleigh 24- be invalid. Use (load) 48 hrs DSRepair and choose the “Unattended full repair option” OITS Reinstall Directory Services 4-8hrs OITS Reinstall backup product 1hr Install Backup Exec DMF/OITS Catalog backup tape or 30 ins tapes OITS Restore directory services 4-8 Hrs Restore schema Restore root deselecting admin, Backup_exec and BKUPEXEC_servername objects DMF Restore NetWare file 8Hrs system DMF/OITS If Vol1 files were ok then 8Hrs restore only the trustee assignments under the restore option in Backup Exec for NetWare client options “exclude data streams “ Otherwise restore the data. DMF/OITS Verify that the server is on 1Hr the tree with time synchronization, etc… Contacts OITS contacts John Whelan Jim Fox 919-981-2666 247-1174 Charles Otstot Al Schmidt Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 65
    • NCDENR – Business Continuity Plan – 2004 919-875-3631 Melodie Parnell OITS Help Desk Richard Jewett 800-722-3946 Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 66
    • NCDENR – Business Continuity Plan – 2004 ________________________________________________________________________ 3.6 Division of Waste Management Division of Waste Management 6/22/2004 Business Continuity Plan – Disaster Recovery Communication Process Description The Division of Waste Management’s (DWM) Disaster Recovery Communication team (DRC) gathers, collates and delivers appropriate emergency data and reports for the division, agency and/or governor during emergencies. Team members’ roles in DWM emergency preparedness and response complement their DWM job responsibilities. The DRC Chairperson is the primary liaison between the team members and DWM management. The Division Director sponsors the DRC team and ensures that they have the necessary resources. During emergencies, the team members play a critical role. They communicate their Section’s emergency response activities (scope of incident, initial response measures, coordination/resource needs, and short/long term action items), advise teammates of incident situations and the subsequent actions that need to be taken. The DRC’s DWM State Emergency Response Team serves as liaison to the N.C. Division of Emergency Management. The Superfund DRC member is the liaison with EPA’s Region 4 Emergency Response program. The Division’s LAN Administrator supports the DRC as needed, based on situational computer requirements. DRC members: disseminate information about emergency preparedness and response activities within their Sections and among DRC members; coordinate the DWM emergency response plan development; help implement the DWM plan; contact appropriate resources, as needed, to assist in emergency response. All DRC members are considered essential personnel. Scope Acceptable level of function is 100 percent within eight hours after the onset of a disaster. Hurricane / Severe storm: Hazardous Waste: Previous Section activities include pre-planning for these types of natural occurrences. Pre-planning includes calling hazardous waste Treatment, Storage and Disposal Facilities and Large Generators with outside storage areas. These entities are instructed to take appropriate measures. Staff also recommend they implement their facility’s contingency plan(s) to respond to the emergency. Post incident response includes tracking orphaned containers, identifying waste and providing technical assistance to assess and clean up affected areas. Solid Waste: This section’s preparedness planning includes training with local governments. They evaluate damage or threats to facilities (e.g., leachate storage, power supply, access, etc.), help identify and approve storage/staging /processing /disposal sites for vegetative debris and demolished structure debris (homes, mobile homes, other buildings, etc.). Staff also evaluate disposal capacity issues. The section communicates and collaborates with FEMA, DOT, DAQ, etc. The Solid Waste Section is DWM’s lead Section for this type of disaster response. Superfund: Identify sites in affected areas and survey sites to determine damage. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 67
    • NCDENR – Business Continuity Plan – 2004 UST: Evaluate the potential for the release of petroleum from damaged pumps served by USTs. Winter Storm Hazardous Waste: At present, this Section has no emergency response experience in this scenario. Predicted items could include the loss of power affecting alarms systems for HW tanks and other systems. The section could provide technical assistance and guidance. Solid Waste: Experience with preparedness planning and training with local governments. Staff evaluate damage or threats to facilities (e.g., leachate storage, power supply, access, etc.) They also help identify and approve storage/staging /processing /disposal sites for vegetative debris and demolished structures debris (homes, mobile homes, other buildings, etc.) and evaluate disposal capacity issues. Staff actively communicate and collaborate with FEMA, DOT, DAQ, etc. This section serves as the Division’s lead Section in this scenario. Superfund: No response anticipated. UST: No response anticipated. Flood Hazardous Waste: Pre and Post responses are identical to "Hurricane/Severe Storm" response. Solid Waste: Experience with preparedness planning and training with local governments. Staff evaluate damage or threats to facilities (e.g., leachate storage, power supply, access, etc.) They also help identify and approve storage/staging /processing /disposal sites for vegetative debris and demolished structure debris (homes, mobile homes, other buildings, etc.) and evaluate disposal capacity issues. Staff actively communicate and collaborate with FEMA, DOT, DAQ, etc. This Section serves as the Division’s lead Section in this scenario. Superfund: Identify sites in affected area and survey to determine any damage. UST: Evaluate the potential for flooded USTs to release product. Evaluate the potential for product releases from flooded USTs that have floated out of the ground. Nuclear Power Plant Disaster Hazardous Waste: Nuclear plants usually have minor HW drum storage. We would provide any technical assistance that would be necessary. This section serves as the Division’s lead Section in this scenario. Solid Waste: No response anticipated. Superfund: No response anticipated. UST: No response anticipated. Terrorist Attack on Infrastructure Hazardous Waste: In the case of a hijacked truck with hazardous waste/material, the section would respond in all capacities with the exception of “first responder mode.” Acts as Division’s lead Section in this scenario. Solid Waste: Help identify and approve storage/staging /processing/disposal sites for demolished structure debris. Superfund: Identify sites in affected area and survey to determine any damage. UST: Identify sites in affected areas and survey to determine any damage. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 68
    • NCDENR – Business Continuity Plan – 2004 Biological Warfare / Terrorism / Spill / Release Hazardous Waste: Post incident response could include waste identification and technical assistance to assess and clean affected sites. Solid Waste: No response anticipated. Superfund: No direct effect anticipated. Would take lead in overseeing removal action contract. UST: No response anticipated. Chemical Warfare / Terrorism / Spill / Release Hazardous Waste: Post incident response could include waste identification and technical assistance on assessment and cleanup Solid Waste: No response anticipated. Superfund: No direct effect anticipated. Would take lead in overseeing removal action contract(s). UST: No response anticipated. Additional Information Hazardous Waste: This Section can contribute additional resources that include issuing emergency permits to dispose/destroy chemicals and munitions. Section can actively collaborate and communicate with EPA's Emergency Response and Removal Branch to assist in any cleanups. Solid Waste: No response anticipated Superfund: No response anticipated UST: No response anticipated Priority Maintaining and activating the DRC is a critical high priority. Other DWM processes such as compliance assistance, inspection/evaluations, permitting, and remediation/restoration are categorized as essential or necessary. Environmental education assistance is considered desirable. Worksheets 1 & 2 are attached. Risk/Impact/Cost The overall risk is high, given the need for integrated communication and decision-making capabilities. Impact Reduction Electronic information is backed up periodically at remote locations. DRC members maintain hard copies of critical information including, but not limited to: available emergency response vendors, regulated facility lists and DRC contact information. The DWM central office and Parker Lincoln office also maintain periodically updated contact information. Recovery Organization Business Continuity Management Responsibilities Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 69
    • NCDENR – Business Continuity Plan – 2004 Representative Division Director Notify DENR Management. Implement contingencies. Communicate DENR policies. Declare normal operations. Monitor state of critical processes and overall recovery. Deputy Division Director Coordinate and communicate with various recovery team members. Serve as FEMA contact. DRC Chairperson Head DRC. Maintain recovery plan. Schedule/conduct training. DWM SERT contact Primary liaison with N.C. Emergency Management. DRC Members Coordinate the return of critical functions to the normal functional area. DWM HR/Budget Provide Division Director and DRC Chairperson with HR/Budget policies. Serve as FEMA contact. DWM Public Information Officer Communicate assessment and critical public information. Functional Recovery Teams: The primary business functions within DWM work through field offices to provide response as demanded across the state. Staff may be reassigned to assist other Sections or Divisions to meet response needs. Functional Recovery Team Representatives Responsibilities SW/HW/UST Field Operations Assess damage and estimate recovery time. Assist in providing Section business functions across the state. DRC Section Contacts Develop specific list of damages for the DRC. Report progress to DRC Chairperson. Sections, Division staff Assist in supporting Field Operations Branches to assess and establish recovery times. Section Chiefs Prioritize critical processes and identify resources to meet needs. DWM LAN Administrator Provide remote access to electronic information as needed. Contingency Plan Steps Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 70
    • NCDENR – Business Continuity Plan – 2004 Emergency Determination/Notification The DWM Division Director alerts the DRC Chairperson verbally or electronically at the onset of a disaster event. If electronic means are available, a wider broadcast to Section Chiefs, all staff or other identified groups will be made. The DRC Chairperson will communicate specific needs and the status of available resources to DRC members. DRC members will communicate assessment needs and available resources to their Section’s staff. Activation of Contingencies DRC members will invoke contingency plans within the Section as needed to assess the scope of the incident. The status will be reported to the DRC Chairperson. As remote access needs are coordinated, the situations will be periodically communicated to the Director. The DWM LAN Administrator will work directly with DRC members to retrieve critical information. Stabilization and Actions Taken DRC members will periodically report their assessment and response status to the DRC Chairperson. The DRC Chairperson will set report intervals. The DRC Chairperson compiles information and submits reports to the Division Director as requested. The Deputy Director and Division HR/Budget contacts will provide resource information to the Director as information becomes available. The PIO will maintain communication with DENR’s PIO Director and broadcast public announcements as needed to help stabilize Division functions and as directed by the Department. The DWM LAN Administrator will re-establish electronic data management tools and provide the Director with status reports as needed. Recovery Section Chiefs and Division staff will document the resources extended and the recovery phase status. Section Chiefs will notify the Director on the status of essential and necessary business functions and when full recovery is achieved. The PIO and LAN Administrator will inform the Director of outstanding assistance requests. Contingency Plan Maintenance The DRC Chairperson maintains the Business Continuity Plan. Annual plan reviews are conducted with the Division Director in December. Annual testing with the N.C. Division of Emergency Management and EPA’s Region 4 will be conducted in March. Improvement recommendations will be forwarded to the Division Director and the Quality Management Steering Team as appropriate. Training for emergency response measures is conducted annually via eight-hour OSHA updates. The DWM SERT contact participates in emergency response training as advised by the N.C. Division of Emergency Management. The Deputy Director and HR/Budget contacts participate in annual DENR FEMA training by July of each year. Definitions Business Continuity Process Contingency Plan – a collection of procedures and information developed, compiled and maintained in readiness for use in the event of an emergency or disaster. Emergency Determination/Notification – determining degree of disruption and need for contingency operations, then initiating emergency notifications. Activation – initial organizational actions taken when disruptions occurs or before an expected event. Stabilization – bringing critical processes up to minimum operating levels. Recovery – the work of rebuilding processes, operations or infrastructure after a disruption or emergency event. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 71
    • NCDENR – Business Continuity Plan – 2004 Functional Recovery – restoring critical processes to minimal levels during the disruption and eventually restoring essential processes. Business Recovery – restoring all processes for all functions to attain normal operations. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 72
    • NCDENR – Business Continuity Plan – 2004 ________________________________________________________________________ 3.7 Office of Public Affairs N.C. Department of Environment and Natural Resources Office of Public Affairs Business Continuity Plan May 25, 2004 Office Description: The Office of Public Affairs coordinates DENR’s communications, public information and outreach efforts through media relations, publication production, Web site development and maintenance, special events and a variety of other programs and initiatives. 1. Process description – The Office of Public Affairs is the primary entity for communicating about department issues with the media and general public. The office’s critical function in continuing business operations is to maintain or re-establish communications (public education and outreach) in the event of potential or real business disruption. The key process business process in this function is crisis communication to the public, through the media and any other appropriate mechanisms. While news releases, the department’s Web site and electronic messaging are common communications tools for sharing information during normal business operations, the key tool utilized by the office is the telephone (cell or land line). The Office of Public Affairs already operates under essentially an on- going business continuity plan. The office provides home phone numbers for staff to be reached during non-business hours. The office staff members are reachable 24 hours, seven days a week. This operational practice was necessitated by the unpredictable nature of environmental disasters, such as spills, explosions and fish kills. The office also maintains a close operational relationship with the state’s Emergency Operations Center and provides staff to support the center during declared emergencies. 2. Scope – The OPA business continuity plan is based on achieving full operational (100 percent) as soon as possible, but in many cases, the return to normalcy will be dependent upon other business functions, such as those covering computer systems, servers, the Web site, etc. a. Degree of minimum operations - The office will be able to perform its essential functions, albeit with more challenges, with the use of working telephone operations. After the cell phone, a working computer and fax machine would assist agency performance. b. Capability of the Plan – This plan is designed to help OPA perform its critical functions to support the agency as quickly as possible. The plan also includes the department’s crisis communications plan and alternative site locations for key departmental personnel. Office staff can work from home if roads are impassible or travel into state offices is not feasible. OPA maintains home phone and cell number lists and carries them at all times. c. Event scenarios – Loss of use of primary office facilities due to power outages, building evacuations due to bomb scares or other homeland security issues, natural disasters, such as hurricanes, snow storms and tornadoes. d. Triggers for activation – Customary communications tools such as fax machines and electronic communications are unavailable. The alternative work location for the office director is the state EOC, where the director will coordinate communication efforts with department leadership and utilize communication tools available through the EOC. The EOC is the logical place for relocation since a disaster or disruption forcing such a change could likely result in activation of the center. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 73
    • NCDENR – Business Continuity Plan – 2004 3. Process Priority – The process priority is to re-establish phone communications as soon as possible. That allows OPA to re-establish contact with department and administration leadership and the media/public about events and to share other critical public health and safety information. 4. Risk/Impact/Cost – Communications are critical during a crisis or business disruption. Over the past decade, business has been disrupted about once every three years due to snowstorms, hurricanes or building evacuations. During each of these events, communications was maintained because phone operations were not lost. The potential impact of the loss of external communication capabilities is significant because DENR is responsible for protecting the health and well-being of the public and North Carolina’s natural resources. The agency’s integrity and credibility are integrally linked with its ability to communicate with the public in the event of a business disruption. Costs are not a functional issue in the process. 5. Impact Reduction – The office updates the crisis management plan and the home and cell phone number lists for key employees every six months and the alternative work site locations are updated annually. 6. Recovery Organization – The office director is the initial responder to disruption events. In his absence, the department PIO in the office is the designee. As business recovers, OPA staff are pulled in for relief. Agency-level PIOs are also called upon for assistance, depending upon their division involvement, as needed and can be used in recovery if needed. 7. Contingency Plan Steps a. Emergency Determination and Notification – Contact is made from the director, or his/her representative with the department secretary and/or deputy secretary. b. Activation of Response structure/teams Steps – The Office of Public Affairs is a small agency with six full-time staff. The director and the department PIO are the key people involved, and they will communicate and plan actions for business continuity. c. Stabilization Steps – This includes contacting key people in the organization and establishing communication with the media and public through means available. This is coordinated by the director. d. Recovery Steps – The return of phone operation, if lost, is the first recovery step. The use of generators and priority placed on restoring operations at the EOC make this the logical location for the director to set up shop when transportation becomes feasible. The director will initiate communications and outreach from home until that is feasible. The other recovery steps that follow involve re-establishing communication with key people in the department, the EOC and the governor’s office. 8. Contingency Plan Maintenance a. Management – The OPA continually updates the crisis plans, phone and alternative work location lists. b. Testing – The key pieces of materials are kept with staff at all times and are sent out for review on a regular basis. c. Training – The office and department PIOs continually support the efforts of the state EOC and also participate in training drills it coordinates, including those required by the Nuclear Regulatory Commission for the state’s nuclear power plants. Through these efforts, staff members become well-equipped for business disruptions and emergencies. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 74
    • NCDENR – Business Continuity Plan – 2004 ________________________________________________________________________ 3.8 Zoological Park NORTH CAROLINA ZOOLOGICAL PARK EMERGENCY OPERATIONS POLICY _________________________________ Effective Date: December 13, 2000 Director’s Signature EOP- 1 GENERAL EMERGENCY GUIDELINES DENR requires that contingency plans establish sufficient criteria to minimize disruption of operations during the following emergencies and to maximize public safety, employee safety, security of property and protection of the environment. 1. Acts of Nature: A direct, sudden, and irresistible action of natural forces, which may or may not have been foreseen or prevented. 2. Mechanical failure: Extended disruption of services from failure of mechanical device or system that is deemed necessary to continue business and protect the plant and animal collection. 3. Utility disruption: Non-availability of gas, electricity, heat, water, sewer, or communications for an extended period of any cause. 4. Disease: An illness or threat of illness that has the potential to infect or affect many individuals in a particular facility. 5. Fire: the destruction or damage to a workplace that renders it unsuitable or unsafe for use due to fire. 6. Vandalism: deliberately mischievous or malicious destruction or damage to property. 7. Terrorism: the use of methods by individuals or groups for the purpose of causing fear, harm, or submission upon an individual or a group. 8. Bomb threats: communication of intent to use an explosive device within or upon a structure or other area. 9. Protests/picketing: an expression or declaration of objection, disapproval or dissent by one or more persons. Such activity shall be deemed inappropriate when any statements or actions are intended to cause fear, harm or intimidation to others, disrupt the work place, or unlawfully invade State property. 10. Uninvited intrusion: an act of unwelcome or inappropriate entry into any part of an exhibit, workplace or the Park in general. NCZP has the added requirement to be prepared for site-specific emergencies to include an Animal Escape Plan; Emergency Access to Restricted Areas and specific details for recovery from weather related damage. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 75
    • NCDENR – Business Continuity Plan – 2004 DENR Policy requires each agency to designate a Local Response Team to initiate action in response to plan provisions. DENR specifies that members of the Local Response Team or employee performing duties at the direction of the Team, shall not be impeded in executing duties nor shall they be reprimanded or disciplined so long as their actions are consistent with approved site response plans. The Local Response Team for NCZP consists of the Director; General Curator; Facilities Officer; Visitor Services Officer; Curator of Horticulture; and Public Affairs Officer. Other members of staff may be added to the Team or consulted depending on the situation. Due to the twenty-four hour coverage provided by the Visitor Services Ranger Branch, ZooCom shall maintain current plans and emergency communications call lists for designated staff, local support functions and State agency representatives having responsibility for the various types of emergencies. The NCZP Local Response Team may require specified members of staff to remain or be recalled to the Park to make appropriate preparations for a possible emergency and or to initiate recovery operations. The Section Managers for Animal, Horticulture, Facilities and Visitor Services may unilaterally retain or recall their staff without Local Response Team approval. When an employee, who is not in a mandatory operation, chooses not to report to work or chooses to leave early or when it has been determined that services will be suspended, work time lost will be accounted for in accordance with the Adverse Weather Policy. EOP – 2 SEVERE WEATHER EMERGENCY Severe weather emergencies encompass any situations and conditions associated with hurricanes, tornadoes, thunderstorms, snow-ice storms or other damaging weather not necessarily associated with a National Weather Service alert. All NCZP supervisors should monitor impending conditions and take the initiative to reasonably and prudently initiate preparatory measures within their areas of responsibility. ZooCom has the primary responsibility to monitor all available sources necessary to provide general information and Park-wide notifications using National Weather Service alert levels to include applicable definitions of each alert. CONSIDERATIONS i Top off all fuel tanks; portable fuel tanks, generators and powered tools; provide for a temporary water supply; evaluate animal food supply level. i Prepare commercial plaza, workspaces and exhibit areas to minimize potential damage due to unsecured items. i Priorities for protection are human safety, animal safety, plants and facilities in that order. i “High danger” buildings are those with extensive perimeter glass or other structural conditions that make them most vulnerable to damage and are least protective of human safety. Aviary, Sonora Desert, upper Africa Pavilion, Greenhouses, Upper Education Center, Junction and North America Plaza Commercial Buildings and Admissions Booths. i ‘Medium Danger” buildings are those with structural characteristics that make them less susceptible to peripheral damage. Veterinary Center, Streamside, Touch and Learn Center, Service Support Office, Metal Buildings, Propagation Buildings and Wildlife Rehabilitation Center. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 76
    • NCDENR – Business Continuity Plan – 2004 i “Low Danger” buildings are those most suitable for human safety. Animal Holding/staff areas, Public Restrooms, Lower Rocky Coast, Africa Plaza Commercial Buildings, Africa Ranger Station (ZooCom). i Most “High” and “Medium Danger” buildings have some areas within them that are relatively safe for shelter. Supervisors should identify these areas in assigned workspaces and exhibit areas. i During severe wind and or rainstorms, visitors will be reluctant to leave whatever structure they are in even if it is a high-risk building. The staff will need to manage the situation as best as possible until alternatives can be coordinated to include the feasibility of using bus transportation to evacuate the high-risk area. Visitors that are determined to leave a structure or the Park should not be detained. i Visitor Services Division has primary responsibility for communications, public safety, transportation of visitors, evacuation, medical response, coordination of fire and rescue operations, and maintaining order/security. i Animal Division assumes temporary responsibility for the public at indoor exhibits and primary responsibility for the animal collection. Comment [jsm1]: i Horticulture Division has primary responsibility for clearing debris or snow removal for emergency access to animal areas, service roads for emergency vehicle access, Veterinary Hospital, rear access to Ranger Station, access to Medical Room at North America and access to the radio tower at Bison Barn. i Facilities Division has primary responsibility for protection and restoration of utilities, maintaining generator operations and emergency repairs. EOP – 3 ANIMAL DIVISION PREPAREDNESS The Animal Division has the following minimum considerations applicable to preparedness when a severe weather emergency is predicted. The stated or implied meaning of these steps also applies to situations having no warning. 1. Curators and Keepers III will monitor storm track and predicted landfall locations and times for all named tropical storms and hurricanes and insure all keeper staff are kept informed. Curators and Keepers III will utilize information and advice from State/Federal authorities. 2. Curators and Keepers III will meet and discuss specific response plans minimally 48 hours in advance of predicted local impact time for any storm that appears to be tracking toward Central North Carolina . 3. Keepers III will meet with all work unit staff 24 to 48 hours in advance of predicted local impact to discuss potential dangers, roles and responsibilities during preparations and staffing availability for next several days. This will include specific discussion of when and how to secure all collection animals. Appropriate Curator or Keeper III will perform this duty for any work units whose assigned Keeper III is off-duty. 4. Keepers III will review list of high, medium and low danger buildings and all other elements of this plan. Discuss vulnerability of facilities in each work unit and closest safe areas for possible use as personal, public and animal shelter. 5. All Keeper staff will store or secure any equipment, objects, and visitor trash cans from outdoor areas in and around their work units that could become dangerous projectiles in high Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 77
    • NCDENR – Business Continuity Plan – 2004 wind conditions. In consultation with Maintenance staff, evaluate need to remove shade-tarps or similar materials that could be damaged or contribute to structural damage in high wind conditions. Areas for special attention are those where animal containment includes glass barriers. 6. Gather and make ready emergency equipment, emergency lights; nets, crates, traps and other animal capture equipment. Locate a store of the most useful equipment (flashlights, boots, gloves) in a highly accessible place when leaving the “night before”. Charge portable radios. 7. Gather 3 days of animal care supplies that would be needed if electrical power or water were interrupted (animal drinking water, bedding, bulk food supplies). 8. Stage all vehicles in open areas of the North America parking lot and at Service Support on the “night before” to prevent vehicles becoming trapped inside the Zoo by downed trees. Leave vehicle keys in the vehicles. 9. Staff assigned to report the “morning after” will take portable radios and chargers home to monitor ZooCom. 10. If Keeper staff can not receive ZooCom radio reports at the pre-arranged times they are to contact a supervisor or Curator by phone. If phone service is interrupted, Keepers are to use their best judgment as to when it is safe and reasonable to proceed to the Zoo. SECURING ANIMALS Species to be secured will be somewhat dependent on the predicted severity of the storm. Always secure all dangerous animals in the most storm resistant/secure facility available (see lists of dangerous animals in NCZP Animal Escape Plan). This should normally include elephants, rhinos, cats, bears, bison, elk, chimpanzees, gorillas, and venomous reptiles. Secure any birds from outdoor exhibits into barn stalls as they are at significant risk in high wind conditions. Secure animals in housing that would best facilitate meeting husbandry needs in the event animals must remain in these quarters for several days with limited or no shifting due to power outage. Do not secure any animal in a facility in which electrified fencing is a primary containment element. STAFFING GUIDELINES Keepers III will assess scheduled staffing and staff availability for the days leading up to and immediately following the storm. Two “critical staff” keepers will be identified for each work unit who will attempt to report to work on the “morning after” and on succeeding days. These staff should live close enough to the Zoo to insure a reasonable chance of reaching the Zoo to provide animal care. Keepers III will contact off- duty staff due to return to work on the “morning after” to advise them of staffing plans and arrangements. Keepers III will develop plans for getting sufficient animal care staff to the Zoo in the event that storm debris or damage makes it difficult for staff to get to the Zoo in personal vehicles. All keeper staff scheduled for duty on the “morning after” will meet at pre-determined rendezvous points outside the Animal perimeter fence (typically in the Employee Lounge at Service Support and in the lobby of the Education Center). All Zoo staff other than the Search and Secure Teams may not enter past this fence line. Curators will review weapons and radio staging areas to insure maximum safe availability the for search and security inspections. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 78
    • NCDENR – Business Continuity Plan – 2004 EOP – 4 HORTICULTURE DIVISION PREPAREDNESS The Horticulture Division has the following minimum considerations applicable to preparedness when a severe weather emergency is predicted. The stated or implied meaning of these steps also applies to situations having no warning. 1. Curator of Horticulture will assign staff to report to the Zoo after the initial storm, who they report to, and when to report. 2. Horticulture Curatorial and Supervisory staff will monitor storm track and make necessary adjustments to preparations. 3. Supervisors will meet with their assigned staff within 24 hours of predicted local impact to designate individual roles and responsibilities, expected staffing availability, and make final preparations as needed. 4. Supervisors will review list of Zoo structures that are deemed safe and unsafe under such conditions. This would include plastic covered and glass greenhouses, and other structures that would be easily damaged by flying debris such as the Aviary and Desert exhibit. 5. Staff should close and lock all interior and exterior doors when leaving the Zoo prior to the storm. 6. Supervisors will assign staff to secure and store tools, and equipment as needed, paying attention to areas around windows, glass greenhouses, and polyhouses minimize possible projectiles during the storm. 7. Curator of Horticulture, Supervisors, and staff within the closest proximity of the Zoo will take radios and radio chargers home to monitor the status reports issued by ZooCom during the course of the storm. Condition updates will be broadcast by ZooCom at pre-determined times. 8. Supervisors will insure that all vehicles are fueled and parked away from any trees that could fall during the storm. Keys will be located in a designated place that would be easily accessible for all staff. Assigned staff will take a Horticulture 4-wheel drive vehicle home for the purpose of being able to pick up crews the next day. 9. Supervisors will ensure that pertinent tools are readied, blowers are fueled, rakes, shovels, brooms, and flashlights are gathered for easy access. Also ensure that a sufficient quantity of fuel and fuel-mix is on hand. Drinking water will be made ready for employees and stored with the tools and supplies. 10. Arboreal staff will fuel all chainsaws and have them readily available in the seed barn, which will be left unlocked. STAFFING GUIDELINES The Curator of Horticulture will designate those staff members needed to function as the Horticulture First Response Team for recovery operations. It is critical that this initial team makes every reasonable effort to report to the Service Support office as soon as possible. Do not enter the Park past the animal perimeter fence line until ZooCom broadcasts an “All Clear”. Staff not assigned to the First Response Team will report to work as soon as possible. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 79
    • NCDENR – Business Continuity Plan – 2004 EOP – 5 RECOVERY OPERATIONS PHASE 1 PARK CLOSURE The NCZP Local Response Team will keep the Director informed of the situation and offer rational for recommending that the Park be closed to the public. If the Park is closed to the public it does not mean that it is closed to the staff. See Adverse Weather Policy. Should the Park be closed to the Public, ZooCom will immediately change the message on the Visitor Information Telephone System. Marketing staff will return the system to normal mode when notified that the Park will re-open. The Director or a designee will inform DENR and the Public Affairs Officer will notify media representatives. PHASE 2 SEARCH AND SECURE TEAM The Animal Division Search and Secure Team shall inspect the Park for escaped or otherwise unsecured dangerous animals prior to anyone being allowed in the Park. Employees reporting to the Education Center or Service Support Office for information, assignments and to confirm if the Search and Secure Team has authorized an “All Clear” status through ZooCom may use gate 1, 11, 12 or 15. Two Horticulture arbor staff may be part of this team. In the event of a severe storm, damage to animal containment will be assumed. Oversight for search and security operations will be the responsibility of the General Curator, in consultation with Section Heads or designated representatives from Veterinary, Facilities, Horticulture and Visitor Services. Team members will determine timing and methods for entering the Park. Decisions by these teams will guide the search and security procedures. Armed two-person teams of animal care staff (Weapons Team members) will proceed in secure vehicles to carry out the search and security inspections. Inspections will insure all animal holding facilities are secure before other staff enters the Park. Effort should be made to check highest security risk/most dangerous animal areas first. Horticulture Arbor staff will stay with the Weapons Team at all times until the “All Clear” condition is announced. A Curator or Keeper III will be assigned as a Staging Area Leader to direct staff at each staging area. A Keeper will be assigned to guard gates 4, 5, 6 and 16 to prevent any staff other than the Search and Secure Teams from entering inside the perimeter fence until the “All Clear” condition is broadcast. Animal Husbandry and Veterinary staff using guidelines from the NCZP Animal Escape Plan will handle escaped or unsecured dangerous animal situations. SEARCH AND SECURE TEAM MEMBERS SHOULD CONSIDER THE FOLLOWING FACTORS AND GUIDANCE: Contact ZooCom with final decision for operations and reaffirm that no one is to enter the Animal perimeter fence until the Search and Secure Teams completes their work. If pre-arranged staging should be changed, instruct ZooCom to announce new plan details. All staff should listen for new pre-entry instructions via ZooCom radio announcements at one-hour intervals. Proceed to the Zoo and obtain recommended equipment, then rendezvous at designated Staging Area. RECOMMENDED EQUIPMENT LIST FOR SEARCH AND SECURE TEAMS: (PER TWO- PERSON TEAM) • hard-hats (2) • high-powered flashlight (2) • work gloves (2 pair) • weapons (2 shotguns & 1 or 2 rifles) • binoculars Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 80
    • NCDENR – Business Continuity Plan – 2004 • chainsaws, chainsaw operating safety equipment for Arbor staff • pepper spray PROTOCOL FOR SEARCH AND SECURE TEAMS 1. Check all animal-holding facilities; prioritize checking dangerous/high risk animals first. 2. Each team is to proceed in a logical order around their assigned “continental section” when doing inspection. 3. A “cruiser ready” shotgun should be removed from its case and carried by one team member during all inspections. 4. Check the entire perimeter of the exterior of each animal holding facility for any signs of structural damage or potentially compromised animal containment. 5. Look through any windows or safe openings available to assess interior security, but do not enter holding areas. 6. Notify staging area leader by radio of security status as you complete the inspection for each facility. 7. Observe and assess security conditions for all exhibits and perimeter fencing along the inspection route but, do not perform full inspections of these areas if it significantly delays completing checks of animal holding facilities. 8. Be vigilant at all times, especially when out of vehicles, for the possibility of escaped collection animals. GUIDELINES FOR KEEPERS ENTERING ANIMAL HOLDING FACILITIES 1. Remember the Search and Secure Teams do not enter holding buildings to check security; only the outside of these facilities is checked for signs of major damage. 2. Proceed slowly and cautiously and take nothing for granted. 3. Perform a thorough inspection of all animal holding caging and containment. 4. Count all animals as soon as possible. 5. Perform a thorough inspection of building integrity and utilities. 6. Turn all electrical devices on carefully and in stages, unless there is standing water. PHASE 3 EMERGENCY DAMAGE MITIGATION When the Search and Secure Team have cleared the Park, the Horticulture First Response Team and other staff members will start clean-up operations and emergency repairs to facilities and utilities. The Horticulture staff may obtain initial damage assessment from ZooCom and the Search and Secure Teams. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 81
    • NCDENR – Business Continuity Plan – 2004 EOP – 6 RADIO COMMUNICATIONS During any preparedness stage of operations each NCZP Division Manager will designate critical members of staff to take individual radios and chargers home so as to monitor conditions on Channel 1. ZooCom will blind broadcast periodic status reports during the night at 2300 hours; 0500 hours; 0600 hours, 0700 hours and 0800 hours. The broadcast will include current NOAA forecast; State Highway Patrol warnings (SHP does not recommend driving in sustained winds of 30+ MPH) and any on-site observations of conditions conducted by Ranger vehicle patrol (foot patrol suspended.) During Search and Secure Team operations radio transmissions are strictly limited to the Team and Rangers. After the “All Clear” has been announced staff will revert to regularly assigned channels. EOP – 7 RESPONSE TO BOMB THREATS The communication of a bomb threat or the discovery of suspicious related conditions, packages or vehicles must be taken seriously. Visitors and employees at potential risk must be informed and it is critical that all actions following a threat be taken with calm deliberation and discipline. If the threat is communicated by telephone or e-mail, the recipient should make every attempt to maintain communications and ascertain as much specific information as possible. THREAT ASSESSMENT If the threat recipient is in communication with the person making or reporting a bomb threat, responses to the following questions could be critical. i Why was it placed at the Zoo? i Where is the bomb located? i What time will it be detonated? i What does it look like? While the threat is being communicated or reported it is critical to remain calm, do not interrupt or argue with the caller because the recipient should also try to gather the following impressions of the caller: Background noise; speech characteristics; knowledge of Zoo; sex; age estimate; general attitude and level of conviction. THREAT RESPONSE The following actions should be taken in whatever order the situation and available information dictates: i Assume the threat is legitimate and try to obtain specifics. i If occupied with caller, have another staff member notify ZooCom preferably by telephone. i ZooCom will dispatch two (2) Park Rangers to quickly collect and assess all available information from the threat recipient and ZooCom will concurrently notify County Emergency 911 Communications. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 82
    • NCDENR – Business Continuity Plan – 2004 i ZooCom will notify the following personnel in the order listed, temporarily by-passing notification of anyone not on Zoo grounds: Visitor Services Officer; Duty Officer, Chief Ranger; Director; Public Affairs Officer. i If the threat area and or building(s) were communicated, Park Rangers shall orderly evacuate the area or structure. Employees familiar with the threat zone shall assist the Park Rangers as directed. i When evacuating a known or purported threat zone the distance from the zone is important. Disbursement of evacuees is also critical because the primary explosive devise may not be placed as reported. A secondary explosive device placed in a “logical” assembly area is also plausible. EOP – 8 OTHER EMERGENCIES EOP – 1 lists other threats or potential emergencies not specifically addressed in this policy. However, they are covered in the Visitor Operations Policy VOP – 11 Ranger Operational Guidelines and VOP – 7 Impermissible Activities and Items in Park. These policies and others are located on the Visitor Services Homepage. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 83
    • NCDENR – Business Continuity Plan – 2004 PART 4 Information Technology DENR Departmental Information Framework Internet Based Enterprise Application Management (IBEAM) Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 84
    • NCDENR – Business Continuity Plan – 2004 DENR ITS IBEAM Disaster Recovery Plan Effective September 31, 2003 Information Technology Services Division INTRODUCTION Purpose A disaster, within the context of this document, relates to a complete computer system failure that requires server replacement and file system recovery. These procedures may be performed at the existing computing facility or at a new facility if the disaster forced Information Technology (IT) operations to relocate. The primary purpose of this plan is to avoid or minimize disruptions of IT services in the event of a disaster, facilitate co-ordination of recovery tasks and document recovery procedures. Scope This document covers IBEAM disaster recovery procedures for IT resources and services within the DENR ITS LANs. This document does not cover media recovery procedures for such things as an application failure, corrupted database, or hard disk failure. Media recovery procedures should be documented in standard operating procedures for applications, database and systems administration. How To Use This Document The disaster recovery procedures included in this plan are dependent upon both DENR ITS and support from external organizations. Examples of outside support include recovering telecom services and routers. Although these external support procedures are mentioned to present a comprehensive strategy, their inclusion does not suggest that DENR ITS can control these situations. This document also references policies and procedures, system documentation, and resources that do not exist. For instance, DENR ITS does have documented policies and procedures for offsite storage of tape backups at the Raleigh Regional Office. DENR ITS has also not identified a replacement strategy for hardware should they have to relocate because of a disaster and they have not officially selected a remote site that they could relocate to. References to this missing information have been underscored in this text and recommendations are provided to assist DENR ITS is amending this plan. SYSTEM ARCHITECTURE Firewalls DENR ITS has a PIX 515urfo firewall with six NICs to provide security based upon server security requirements. This is a locally managed firewall. Servers DENR ITS has an IBEAM System, which includes 13 servers. The configuration of these servers is listed in the table below. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 85
    • NCDENR – Business Continuity Plan – 2004 NRMSITS019 Make/Model Dell Power Workstation Operating System WINDOWS 2000 Server Processor One Pentium 450 processors Disk Drives One 8 GB drives (Raid Level 1) RAM 256 MB CD ROM One bay Drive Configuration C – 8 GB Use IBEAM Domain Controller IP Address, Mask 10.251.128.19 255.255.255.192 & Gateway 10.251.128.1 Remove History only NRMSITS020 Make/Model Dell Power Workstation Operating System WINDOWS 2000 Server Processor One Pentium 450 processors Disk Drives One 8 GB drives (Raid Level 1) RAM 256 MB CD ROM One bay Drive Configuration C – 8 GB Use IBEAM Domain Controller IP Address, Mask 10.251.128.20 255.255.255.192 & Gateway 10.251.128.1 NRMSITS021 Make/Model Dell Power Edge 2650 Operating System WINDOWS 2000 Advance Server Processor Two Pentium 2400 gig processors Disk Drives Two 36GB drives (Raid Level 1) RAM 3000 MB CD ROM One bay Tape Drive PV132 Tape Library SAN 200 Gigs of storage Drive Configuration C - 10 GB, D - 23 GB, F - 131 GB, G - 65.9 GB Use Production Oracle Data Base IP Address, Mask, & 10.251.128.21 255.255.255.192 Gateway 10.251.128.1 NRMSITS022 Make/Model Dell Power Edge 2650 Operating System WINDOWS 2000 Advance Server Processor Two Pentium 2400 gig processors Disk Drives Two 36GB drives (Raid Level 1) RAM 3000 MB CD ROM One bay Tape Drive PV132 Tape Library SAN 200 Gigs of storage Drive Configuration C - 10 GB, D - 23 GB, F - 131 GB, G - 65.9 GB Use Future logical standby Oracle Data Base Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 86
    • NCDENR – Business Continuity Plan – 2004 IP Address, Mask, & 10.251.128.22 255.255.255.192 Gateway 10.251.128.1 NRMSITS023 Make/Model Dell Power Edge 2650 Operating System WINDOWS 2000 Advance Server Processor One Pentium 2400 gig processors Disk Drives Two 36GB drives (Raid Level 1) RAM 3000 MB CD ROM One bay Tape Drive None SAN None Drive Configuration C - 10 GB, D - 23 GB Use Web Logic IP Address & Mask 10.251.128.23 255.255.255.192 Gateway 10.251.128.1 NRMSITS024 Make/Model Dell Power Edge 2650 Operating System WINDOWS 2000 Advance Server Processor One Pentium 2400 gig processors Disk Drives Two 36GB drives (Raid Level 1) RAM 2000 MB CD ROM One bay Tape Drive None SAN None Drive Configuration C - 10 GB, D - 23 GB Use Crystal Reports IP Address, Mask, 10.251.128.24 255.255.255.192 & Gateway 10.251.128.1 NRMSITS025 Make/Model Dell Power Edge 1650 Operating System WINDOWS 2000 Server Processor One Pentium 1400 gig processors Disk Drives Two 36GB drives (Raid Level 1) One 134GB Raid 0 RAM 1000 MB CD ROM One bay Tape Drive None SAN None Drive Configuration C - 10 GB, D - 23 GB E- 130GB Use Development/Test IP Address, Mask, 10.251.128.25 255.255.255.192 & Gateway 10.251.128.1 NRMSITS026 Make/Model Dell Power Edge 2650 Operating System WINDOWS 2000 Server Processor One Pentium12400 gig processors Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 87
    • NCDENR – Business Continuity Plan – 2004 Disk Drives Two 36GB drives (Raid Level 1) RAM 1000 MB CD ROM One bay Tape Drive None SAN None Drive Configuration C - 10 GB, D - 23 GB Use Production/Test IP Address, Mask, 10.251.128.26 255.255.255.192 & Gateway 10.251.128.1 NRMSITS029 Make/Model Compaq Proliant 5500 Operating System WINDOWS 2000 Server Processor Two Pentium 200 processors Disk Drives Three 9 GB drives (Raid Level 5) RAM 512 MB CD ROM One bay Drive Configuration C – 8 GB, D – 9GB Use PVCS and MOM IP Address, Mask 149.168.72.199 255.255.255.0 & Gateway 149.168.72.1 NRMSITS080 Make/Model Dell Power Edge 650 Operating System WINDOWS 2000 Server Processor Two Pentium 550 meg processors Disk Drives Two 8 GB drives (Raid Level 1), One 36 GB RAM 2000 MB CD ROM One bay Tape Drive None SAN None Drive Configuration C - 8 GB, D - 33 GB Use Jboss DEV Test and Production Builds IP Address, Mask, & 10.251.128.80 255.255.255.192 Gateway 10.251.128.65 NRMSITS085 Make/Model Dell Power Edge 2650 Operating System WINDOWS 2000 Advance Server Processor One Pentium 2400 gig processors Disk Drives Two 36GB drives (Raid Level 1) One 146GB drives RAM 3000 MB CD ROM One bay Tape Drive None SAN None Drive Configuration C - 10 GB, D - 23 GB, E – 134GB Use Web Logic IP Address & Mask 10.251.128.85 255.255.255.192 Gateway 10.251.128.65 Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 88
    • NCDENR – Business Continuity Plan – 2004 NRMSITS0140 Make/Model Dell Power Workstation Operating System WINDOWS 2000 Server Processor One Pentium 450 processors Disk Drives One 8 GB drives (Raid Level 1) RAM 256 MB CD ROM One bay Drive Configuration C – 8 GB Use IBEAM Web logic Proxy, Apache IP Address, Mask 10.251.128.140 255.255.255.192 & Gateway 10.251.128.129 NRMSITS0145 Make/Model Dell Power Workstation Operating System WINDOWS 2000 Server Processor One Pentium 450 processors Disk Drives One 8 GB drives (Raid Level 1) RAM 256 MB CD ROM One bay Drive Configuration C – 8 GB Use IBEAM Crystal Reports, Proxy, IIS IP Address, Mask 10.251.128.145 255.255.255.192 & Gateway 10.251.128.129 NRMSITS0xx Make/Model Dell Power Edge 1750 Operating System Linux Processor Two Pentium 2400 gig processors Disk Drives Two 73GB drives (Raid Level 1) RAM 1000 MB CD ROM One bay Tape Drive None SAN None Drive Configuration To be determined Use Future Web Search Engine IP Address, Mask, & To be determined Gateway IBEAM also interacts with GIS servers whose disaster recovery info is covered in the DENR Legacy Server Tape Backup Drives and Backup Software DENR ITS uses PV-132T Tape Library with two 100/200 LTO tape drives in a tape library to back up the IBEAM servers. The backup software is Backup Exec 8.6. Server Applications Applications that reside on the DENR IBEAM servers include: • Oracle • Web Logic • Crystal Reports Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 89
    • NCDENR – Business Continuity Plan – 2004 • PVCS • MOM • IIS • Apache • Backup Exec Server Application Databases DENR IBEAM servers support Oracle and SQL databases. Telecommunication Equipment DENR ITS acquires their routers from the NC Office of Information Technology Services (OITS) and requisitions telecom switches, network hubs and Local Area Network (LAN) wiring from State Purchasing based on recommendations from OITS. DENR ITS currently has one 5 Meg fiber line and is using a Cisco 3550 router for Wide Area Network (WAN) and Internet connections. CONTINGENCY PLANNING MEASURES Business Impact Analysis IBEAM Services DENR ITS is responsible for coordinating the replacement and full recovery of the IBEAM servers. Currently, there is an emergency equipment replacement strategy using the Air Quality equipment. Similarly configured implies an Intel system with equal or greater processors, disk space and memory. Desktop Services Not applicable. Telecommunication Services DENR ITS is responsible for contacting OITS and requesting the replacement and configuration of a new network router. DENR ITS is responsible for coordinating the replacement and recovery of network connection equipment such as network hubs, wiring. Currently, there is no emergency equipment replacement strategy to replace network equipment in the event of a disaster rendered the existing system inoperable. OITS is responsible for administering Internet Protocol (IP) addresses. A static IP address is assigned to the server and a range of IP addresses is allocated to the router from which the Dynamic Host Configuration Protocol (DHCP) service assigns dynamic client IP addresses. Disaster Recovery Team Disaster Recovery Coordinator - This person is responsible for the overall management and coordination of IT disaster recovery planning and the recovery process. These responsibilities include • Coordinating new facility arrangements for IT assets • Coordinating the replacement of computer hardware and support equipment • Restoring computer operations, Work Pager Home Email Phone No. No. Phone No. Address Bill Knight 919-715-6544 919-507-0100 919-233-8124 Bill.Knight@ncmail.net Systems Administrator - This person is responsible for restoring the operating system and systems software support including establishing and maintaining appropriate security capabilities. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 90
    • NCDENR – Business Continuity Plan – 2004 Work Pager Home Email Phone No. No. Phone No. Address Arlyn Kinsey 919-715-0378 919-857-5981 919-481-4708 Arlyn.Kinsey@ncmail.net Operations Team - This team is responsible for computer operations, facility preparation and replacement of computer hardware and support equipment such as LAN and WAN connections. This team will consist of DENR ITS staff. NOTE: Contact information for each team member should be listed in this document. Emergency Contact Information This section lists contact information for DENR management and various services that may need to be informed or contacted in an emergency situation. CONTACT PHONE NO. Office of the Secretary 919-733-4984 Administration 919-733-4984 Budget, Planning and Analysis 919-733-8800 Controller 919-733-8800 Human Resources 919-715-4500 Information Technology Services 919-715-0327 Public Affairs 919-715-4112 Purchase and Services 919-715-3880 Parks and Recreation Unit 919-733-4181 Help Desk 919-715-0333 Air Quality 919-733-3340 Customer Service Center 877-623-6748 Forest Resources 919-733-2162 Land Resources 919-733-3833 Environmental Education 919-733-0711 Server Support Dell Vendor Technical Support 800-234-1490 Novell Support NC OITS Help Desk 919-981-5555 Novell Technical Support 800-255-2707 Telecommunications Support 919-981-5555 NC OITS Help Desk 919-981-5555 Facility Maintenance Building Superintendent 919-715-3880 Telephone Co. Electric Co. Insurance and Warrantee Information Management DENR ITS administrative support has the yearly insurance information that covers the IT resources needed to support critical business systems or warrantee information for IT hardware and software. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 91
    • NCDENR – Business Continuity Plan – 2004 Remote Site Operations The LAN Administrator believes that his operations would most likely move to the Air Quality facility in the Parker/Lincoln building. Communication Procedures DENR ITS does not a formal communication plan that covers internal and public notification procedures that should be followed in the event of an emergency. TAPE BACKUP AND SUPPORT MEDIA MANAGEMENT Tape Backup Process DENR ITS performs a daily full backup of its IBEAM servers. The Friday and monthly tapes are transported to the Raleigh Regional Office for storage. A schema of daily, weekly, and monthly backup tapes is listed below. MON A TUE A WED A THU A FRI #1 MON A TUE A WED A THU A FRI #2 MON A TUE A WED A THU A FRI #3 MON A TUE A WED A THU A FRI #4 MON A TUE A WED A THU A FRI #5 Monthly Tape Backup and Recovery Procedures Each daily and full backup fits on an individual tape, which is capable of storing 100GB uncompressed and 200GB compressed DENR ITS uses Backup Exec 8.6 as their backup software and follows standard operating procedures for tape backup and recovery. Data and Vital Records Storage DENR ITS stores all backup tape media and critical system documentation onsite at the Raleigh Regional Office. The following person(s) are contacts to gain access to the locked room: Sarah Beall Work Phone No. 919-471-4700 Home Phone No Pager No. E-Mail Address Sarah.Beall@ncmail.net The following resources are stored in the white cabinet at the Raleigh Regional Office: • Windows 2000 server and advance server installation media and documentation • Tape backups Sets Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 92
    • NCDENR – Business Continuity Plan – 2004 SYSTEM RECOVERY PROCEDURES Dependencies A successful system recovery will be affected by the following dependencies: 1. DENR ITS will be able to restore their server to a similarly configured machine. Similar configuration implies equal or greater processor performance, memory and disk space and a tape drive that can read the backup tapes. 2. The following media should be available to facilitate a system recovery: • Windows 2000 server and Advance server installation software and documentation • Tape backups, A remote site must have sufficient telecommunications bandwidth to support their operations and the additional business applications that will tax their system when a recovered system is installed at this remote site. File Server Recovery Existing Site Recovery Procedures Telecommunications Recovery operations for a router failure within the DENR ITS facility are to contact OITS for their assistance. IBEAM server In the event of a server failure, the following recovery steps should be performed on Server or a similarly configured server. 1. Install the Windows operating system. 2. Install the applicable service pack from backup media (CD or downloaded copy). 3. Install Backup Exec onto the server. 4. Restore all volumes. 5. Restart the new server and check out logins and file rights for user accounts. Desktop Systems Each division is responsible for replacing its desktop systems. System Accessories Each division is responsible for replacing its network printers. Remote Site Recovery Procedures Telecommunications After selecting the remote site, determine if the existing router and bandwidth configuration can support both the resident needs and the additional utilization requirements that the relocated system will impose. If additional telecommunication equipment is required: Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 93
    • NCDENR – Business Continuity Plan – 2004 1. If a router is not available or if the existing router needs to be scaled up, contact NC OITS and request a router that will handle the projected capacity. This request should specify the internal IP address of the existing router and the location where the new router will be installed. 2. If necessary, contact NC OITS and request expanded bandwidth (i.e., multiple T-1 lines) for the WAN backbone. 3. Develop a wiring diagram based on the facility's floor plan and submit it to NC OITS for contractor services to install the required LAN cable. 4. If necessary, identify an appropriate part number for a hub or switch component from the NC State Hardware Convenience Contract and submit a requisition request to State Purchasing. File/Print server Desktop Systems Same as existing site conditions. System Accessories Same as existing site conditions. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 94
    • NCDENR – Business Continuity Plan – 2004 Appendix A: Risk Assessments for Critical Business Processes Division of Coastal Management – Permitting and Licensing Risk Level High Owner/Manager -- Ted Tyndall Department -- DENR Owner Detail Organization -- DCM Location -- Morehead City, NC Detailed description of gap analysis: Identify HIGH level risks and begin development strategies for mitigation. Look for methods to reduce vulnerability to risks. Detail the impact to the Line of Business: This line of business (Permitting and Licensing) is critical to the recovery of coastal communities and property owners after natural disasters such as Hurricanes and other coastal storms. Regaining the Division's ability to review and respond to permit request is of vital importance to private and public interests. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) While there is no permanent solution, to reduce all risk from HIGH, there can be methods and approaches to reduce the impacts of these risks. One of which is to continuing update/rework of the Divisions Disaster Response Manual. Length of time needed to correct or reduce risk to an acceptable level: 6 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $1 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Immediate Update of Disaster Response Manual Are the interim controls currently in place? Yes Division of Environmental Health, Radiation Protection – Application Review and Decisions Risk Level High Owner/Manager -- Beverly Hall Department -- DENR Owner Detail Organization -- DEH/Radiation Protection Location -- Raleigh, NC Detailed description of gap analysis: (Draft) Detail the impact to the Line of Business: (Draft) Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) (Draft) Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 95
    • NCDENR – Business Continuity Plan – 2004 Length of time needed to correct or reduce risk to an acceptable level: 6 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $10,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) (Draft) Are the interim controls currently in place? Division of Environmental Health, Radiation Protection – Emergency Coordination and Response Risk Level High Owner/Manager -- Beverly Hall Department -- DENR Owner Detail Organization -- DEH/Radiation Protection Location -- Raleigh, NC Detailed description of gap analysis: (Draft) Detail the impact to the Line of Business: (Draft) Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) (Draft) Length of time needed to correct or reduce risk to an acceptable level: 6 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $14,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) (Draft) Are the interim controls currently in place? (Draft) Division of Environmental Health, Public Health Pest Management – Vector Control Risk Level High Owner/Manager -- Nolan Newton Department -- DENR Owner Detail Organization -- PHPM Section/DEH Location -- Parker-Lincoln Detailed description of gap analysis: I'm guessing that a "gap analysis" is a description of the needs vs. present state of the response plans. Our plans are up-to-date but departmental fiscal management problems always plague emergency vector control activities. Detail the impact to the Line of Business: Emergency vector-borne disease control is a critical public health activity post hurricanes and other flooding Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 96
    • NCDENR – Business Continuity Plan – 2004 events. Need is immediate and planning is constantly being refined to manage the risks. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Ongoing refinements of procedural and budgetary requirements for effective actions. Length of time needed to correct or reduce risk to an acceptable level: 1 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $100,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Ongoing refinements of procedural and budgetary requirements for effective actions. Are the interim controls currently in place? Controls have been in place, with updated revisions, since ca. 1985. They're continually updated and refined based on new information and new procedures. Comments: NC is the national leader in responding to emergency vector control needs. Division of Waste Management – Emergency Response Risk Level High Owner/Manager -- DWM ERC Team - Mark Poindexter Department -- DENR Owner Detail Organization -- DWM Location -- 401 Oberlin Rd, Suite 150 Raleigh, NC Detailed description of gap analysis: Need to cross-train Team members on accessing facility identification information from Section databases, and remote location access procedures to access systems. Need to document levels of decision-making - who triggers what. Need for routine hard copy reports archived for response staff. Detail the impact to the Line of Business: Ability to substitute Team members among program areas to respond to emergency needs if one or more offices are not functioning. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Training on IT systems and remote access capabilities. Documentation secured. Hard copies periodically maintained in known locations. Actions will reduce risk to an acceptable level within current available projected resources. Length of time needed to correct or reduce risk to an acceptable level: 6 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $1,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Intranet implemented - date Training on system access and use - on-going and complete 30 days after Intranet implemented Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 97
    • NCDENR – Business Continuity Plan – 2004 Documentation completed on decision-making - 7/31/04 Hard copies secured - 6/30/04 Are the interim controls currently in place? IT system has security measures in place. Individual team members have knowledge of accessing data from their own systems and members generally know whom to contact for data. Members have worked together through several natural disaster events and know "who" decides "what" though further documentation is being developed. Comments: Concurrently working on FEMA documentation that will support the Emergency Coordination/Response component of the BCP. Division of Environmental Health – Operator Authorization and Certification / Intern Training Risk Level High Owner/Manager -- Stefanie deOlloqui Department -- DENR Owner Detail Organization -- DEH Location -- Parker Lincoln Building, Raleigh Detailed description of gap analysis: Our business services have paper back up for all certification information. Loss of ability to use IT equipment will slow down business processes. Detail the impact to the Line of Business: Failure to certify drinking water operators or authorize local environmental health specialists result in the statutory inability for these professional to operate water systems and conduct sanitation inspections/permits. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Regularly scheduled printing of reports will be established so that manual program administration can be accomplished better. Length of time needed to correct or reduce risk to an acceptable level: 1 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $0 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Print reports monthly Are the interim controls currently in place? No Center for Geographic Information Analysis – Admin Processes Risk Level High Owner Detail Owner/Manager -- Mindy Callaway Department -- DENR Organization -- CGIA Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 98
    • NCDENR – Business Continuity Plan – 2004 Location -- 301 N. Wilmington St., Raleigh Detailed description of gap analysis: CGIA measured what is needed to maintain critical business office functions against current methods/plans. Since CGIA generates its own revenues on a monthly basis and pays salaries, vendors, and others with those revenues, The loss of financial data related to invoices generated and payments made is the most significant problem. Currently, duplicate copies of financial records are not in place. This is a shortcoming that will be remedied. Detail the impact to the Line of Business: There are several impacts if a disruption in operations occurs: (1) CGIA bills are not paid, (2) invoices are not produced for services rendered by CGIA staff and, therefore, revenues are not generated, (3) equipment and supplies are not able to be purchased, (4) historical financial data could be lost, and (5) CGIA is unable to collect debts as a result of services rendered. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) CGIA will take the following actions: (1) duplicate financial data and store in its Asheville Office, updating on a weekly basis, (2) establish a plan for secondary office space with computer, software, and Internet access, and (3) document a procedure describing a contingency plan in the event of a catastrophic event. Length of time needed to correct or reduce risk to an acceptable level: 3 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $3,500 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) CGIA will begin copying financial data from QuickBooks on a regular basis until the more permanent solution is implemented. Are the interim controls currently in place? No DENR Information Technology Services – RACF Administration Risk Level High Owner/Manager -- David Johnson Department -- Department of Environment and Natural Resources Owner Detail Organization -- Information Technology Services Location -- Archdale Building Detailed description of gap analysis: The review of the moderates and the high-risk results from Phase I and II exposed the following: 1. The policies and procedures for RACF Administration require modifications to clearly identify guidelines and penalties for misuse and sharing of user Ids. 2. State guidelines and the recent security audit identify the need for an agency Security Administrator with staff. Funding and resources have not been established for this function in DENR 3. RACF Administrators that are not under the supervision of the Information Technology Services Division (ITS) Client Services Manager must be made aware of the legal and non-compliance policy requirements and the responsibilities associated with the role. 4. Adequate funding, for RACF Administrator staff, and training are issues to be addressed by Senior Management of the Department of Environment and Natural Resources (DENR). The ITS budget request should include a request to fund these items. 5. Client Services staff is currently revising and creating documentation required for RACF Administration. All staff is to perform the administrative functions consistent with the documented procedures 6. Coordination of recovery procedures for the DENR ITS Network with recovery procedures of the Office of Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 99
    • NCDENR – Business Continuity Plan – 2004 Information Technology Services (OITS) for the Mainframe computer is critical to reducing down time. 7. DENR ITS has a firewall in place; however the acquisition of a true intrusion detection system is in the planning stage with a dependency on funding. 8. DENR ITS is not permitted to perform scanning on the network. This service is performed by the OITS when requested. Procedures need to be put in place for regular scanning of the network at a minimum of two times a year. 9. RACF is an application that resides on the State's computer and is maintained by the Office of Information Technology Services. There is a major dependency on OITS to have the RACF application functioning at all times. DENR Client Services has the responsibility of administering RACF for the DENR staff. The two agencies use two different approaches to manipulate users using the application. Often this produces communication failures between the two parties when trying to resolve issues. Additional training of the Client Services staff on RACF is needed to close this gap. 10. Business Continuity Planning for RACF Administration will follow as the next step to this Risk Assessment. Detail the impact to the Line of Business: RACF Administration is a key process to allow the Agency staff access to the State's computer. Without this service access to the state Accounting System, the state Personnel System, and certain billing and reporting facilities will be closed to DENR staff. A number of inter-agency communications will be limited without proper functioning of the RACF Administration Process. Authority given to OITS to support RACF in the event of a disaster for DENR would increase the risk of unauthorized access, since DENR is a large agency with staff all over the state. Legal and statutory limits to data could be compromised if RACF Administration is not appropriately applied. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Develop and document identification procedures for DENR staff that would allow RACF Administrators to accurately give access privileges to staff as required within defined limits. Establish and document procedures relating to the RACF Administration of network facilities from alternate locations. Develop and document a Business Continuity plan that effectively provides the agency with the ability to recover and maintain a high level of security for state computer access in the event of a disruption or disaster. Review and document resource requirements for RACF Administration as a part of the DENR ITS Security Administration direction. These requirements will also include staff training. Length of time needed to correct or reduce risk to an acceptable level: 9 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $15,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) A DENR RACF Administration policy is in places that requires a written request signed by the supervisor of the person to whom access privileges are to be granted. A verbal request is acceptable to reset an expired or revoked password as long as the affected person is employed with DENR. By July 2004, a written policy will be available that specifies that a division RACF Coordinator or Network Administrator submits a request to reset a user's password. Validation will be completed by end of July 2004 that all RACF Administrators in DENR have access to DENR RACF policies. Notification will be given that adherence to stated policies and procedures are required. Until funding is made available, knowledge sharing among RACF staff in DENR and OITS will continue to Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 100
    • NCDENR – Business Continuity Plan – 2004 be the mode toward process improvement. This informal process started March 11, 2004. A Business Continuity Plan for RACF Administration will be documented June 2004. Are the interim controls currently in place? To ensure that staff is available in the event of a disruption, all members of the Client Services Team have been given special RACF privileges to perform the required sixty percent of the RACF Administration process. CGIA – GIS Product and Service Delivery Risk Level High Owner/Manager -- Jeffrey Brown Department -- DENR Owner Detail Organization -- CGIA Location -- 301 N. Wilmington St., Raleigh Detailed description of gap analysis: CGIA measured current procedures against what is necessary to sustain the delivery of GIS products and services. The NC MapNet database is currently backed up and stored offsite but staff would not have ready access to the data to continue to perform their tasks (and generate revenue). An improved plan is needed to ensure that staff could access the database and continue working. Detail the impact to the Line of Business: The following impacts would be felt if a disruption occurred: (1) billable staff would not be able to generate revenue without computers, GIS software, and data, (2) CGIA would not be able to meet contract commitments with its clients if #1 did not occur, and (3) the database would not be available to staff who rely upon it to perform work tasks. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) CGIA will take the following actions: (1) relocate a copy of the database to a suite of external, portable hard disk drives and store them offsite, (2) establish a communication plan for staff which would be activated when necessary to ensure a quick return to productivity, (3) establish agreements with vendors for computers and software to help CGIA staff return to normal productivity quickly, and (4) scan contract documents and store them offsite. Length of time needed to correct or reduce risk to an acceptable level: 3 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $52,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) CGIA will purchase a scanner and begin the process of scanning contract documents. CGIA will acquire the necessary external, portable hard disk drives and will begin discussions with vendors about providing necessary computers and software. Finally, CGIA will begin preparation of a staff communication plan. Are the interim controls currently in place? No Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 101
    • NCDENR – Business Continuity Plan – 2004 Division of Environmental Health, Environmental Health Section – Information Management Risk Level High Owner/Manager -- Bart Campbell Department -- DENR Owner Detail Organization -- DEH Location -- Environmental Health Services Detailed description of gap analysis: The Environmental Health Services Section staff the Emergency Operation Center during an emergency and will be able to communicate with local health departments through the EOC system. EHSS staff will assist with public information and policy development and will be able to communicate and receive information from that location. Detail the impact to the Line of Business: Local emergency situations sometimes call for interim guidance and policy to be communicated to local health departments. This information is needed to protect public health as the regulated facilities return to operation. It is very important that emergency guidance and policy be distributed in a consistent manner to all affected counties. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) State cell phones, press release templates available at alt. work locations, lists of emergency contact numbers for staff and dept and the Emergency Operations Center. An employee of EHSS provides this guidance from the Emergency Operations Center through the Public Health Desk. Length of time needed to correct or reduce risk to an acceptable level: 1 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $0 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) All staff have cell phones, which can be used to convey info to the department and/or the EOC on environmental health emergency advisories. Lists of emergency, off-hours contacts are available and are updated regularly. Templates of press releases are kept in the Raleigh office and the home computers of Regional staff. Are the interim controls currently in place? yes Division of Environmental Health, Environmental Health Section – Environmental Resource Coordination Risk Level High Owner/Manager -- Bart Campbell Department -- DENR Owner Detail Organization -- DEH Location -- Environmental Health Services Detailed description of gap analysis: The Environmental Health Services Section staff the Emergency Operation Center during an emergency and will be able to communicate with local health departments through the EOC system. EHSS staff will assist with coordination of State and local environmental health manpower when requested by the EOC from a county health department. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 102
    • NCDENR – Business Continuity Plan – 2004 Detail the impact to the Line of Business: During an emergency some local health departments do not have the manpower to review the environmental health conditions of establishments in a timely manner. The Environmental Health Services Section has worked a clearinghouse for allocating state and local environmental staff. Inability to coordinate this effort could result in delays in the opening of businesses. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) State cell phones, press release templates available at alt. work locations, lists of emergency contact numbers for staff and dept and EOC. Length of time needed to correct or reduce risk to an acceptable level: 1 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $0 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) The Environmental Health Services Section coordinates the county emergency manpower needs and manpower resources available through the Emergency Operations Center. Hard copies are maintained of all environmental health manpower for North Carolina and are available at the Public Health desk of the EOC Are the interim controls currently in place? yes Division of Environmental Health, Shellfish Sanitation – Shellfish Water Closure and Recreational Swimming Advisories Risk Level High Owner/Manager -- Wayne Mobley, J.D. Potts Department -- DENR Owner Detail Organization -- DEH Location -- Morehead City Detailed description of gap analysis: Ideally, shellfish closures and swimming advisories could be conveyed via phone, fax and internet. Shellfish Sanitation does not close the shellfish waters; the section gives the recommendation for opening or closure to the Division of Marine Fisheries, who issues the proclamations closing the waters. In practice, however, shell fishermen call the Shellfish Sanitation office to get their information. Swimming advisories are issued directly from our section. If we were to lose internet access and fax/landline capability, we would still need to get the information to the department and/or the EOC. Cell phones will be sufficient, and all staff with communication responsibilities has cell phones. Detail the impact to the Line of Business: If shellfish were harvested from closed waters, the consumption of this potentially contaminated stock could pose serious public health risk to some populations, as well as significant adverse publicity if people were to become ill from eating contaminated NC shellfish. The industry could suffer a loss of local and interstate sales, which currently generates several million dollars annually. Also, if people were to become ill from swimming in coastal waters, the tourism industry could be negatively impacted from the ensuing publicity. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) State cell phones, press release templates available at alt. work locations, lists of emergency contact numbers for staff and dept and EOC. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 103
    • NCDENR – Business Continuity Plan – 2004 Length of time needed to correct or reduce risk to an acceptable level: 1 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $1 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) All staff have cell phones which can be used to convey info to the department and/or the EOC on closures and swimming advisories. Lists of emergency, off-hours contacts are available and are updated regularly. Templates of press releases are kept in the Raleigh office as well as on the home computers of Morehead staff. Are the interim controls currently in place? YES Division of Environmental Health, Shellfish Sanitation – Shellfish / Crustacea Inspections Risk Level High Owner/Manager -- Wayne Mobley Department -- DENR Owner Detail Organization -- DEH Location -- Morehead City Detailed description of gap analysis: If a disruption were to occur when the operators are getting recertified for the coming year, and this inspection/certification process were held up, the FDA could stop interstate shipment of the multi-million- dollar NC shellfish and crustacea industry. In addition, inspectors must be able to visit operations to determine if temp limits have been exceeded when/if the plants lose electricity. If so, product must be discarded and the permit suspended under imminent hazard until the facility is operational again. Consuming shellfish that has not met temperature restrictions could be hazardous to people's health. The inspectors need to be able to convey that as well as provide technical assistance. That requires vehicles, telephones, passable roads, and a means of communicating the recommendations. Detail the impact to the Line of Business: If the inspectors cannot access the operations, they cannot inspect/certify them. If they cannot convey their recommendations, product could get out into the market that could be dangerous to the public health. If multiple operations were closed to interstate shipping, due to inspectors’ inability to visit them, there could be considerable adverse economic impacts, due to the size of the industry. If contaminated product were not removed, it could pose a public health hazard. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Much depends on accessible roads, but in the event of impassable roads, inspectors could make telephone contact with operators to tell them to embargo product if they lose power. Recertification requires an inspection, however. Length of time needed to correct or reduce risk to an acceptable level: 1 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $100 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Inspectors have state cell phones and access to four-wheel drive vehicles and boats to facilitate their visits. Are the interim controls currently in place? Yes Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 104
    • NCDENR – Business Continuity Plan – 2004 Division of Land Resources – Dam Safety Emergency Response Risk Level High Owner/Manager -- Maxwell R. Fowler, PE Department -- NC Dept. of Environment and Natural Resources Owner Detail Organization -- Land Resources Location -- Land Quality Section - Archdale Detailed description of gap analysis: Database information located in central location without backup by remote program locations "off-line"; one technical contact currently trained to maintain and administer database system (no backup); emergency protocol has not been updated with current staff contacts and their respective contact information for use by various emergency management agencies; program does not have "in-house" IT position to support and enhance the technical aspects of the database and related technology - PDAs and GIS needs (presently done with existing engineering/geology staff); contingencies have been discussed but not documented by staff Detail the impact to the Line of Business: Delays in database access and program staff deployment could result in loss of life and significant impact on property and infrastructure in the event of a dam failure; local, state-wide and national press coverage would be likely and could tarnish program's integrity if business process disrupted/delayed; program could have professional and financial liability for dam failure damages if it failed to respond in timely manner Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Entire IBEAM dam safety database will be downloaded quarterly by at least 3 regional offices outside the Raleigh area (where database currently located) onto a local drive; The State Dam Safety Engineer, as a minimum, will be trained in the interworkings and administration of the IBEAM dam safety database as a backup to current technical contact; Regional Offices will maintain, in perpetuity, dam safety (hard copy) files for its region's dams as a back up to the central office files and database; The emergency protocol will be updated with current staff's contact information and detailed descriptions of each person's responsibilities for dam failure response; emergency response equipment, including vehicles and cell phones, will be made available to each regional office and the central office at all times in case of an emergency Length of time needed to correct or reduce risk to an acceptable level: 6 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $0 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) The permanent controls mentioned above are in progress and should all be completed/in place within 6 months. Are the interim controls currently in place? Some backup of the database and duplication of dam files (hard copies) is currently in place; an emergency protocol is partially in place (updated contact information is needed); most equipment, vehicles and cell phones are in place or on order. NC Zoo – Site Security Risk Level High Owner Detail Owner/Manager -- Yancy King Department -- DENR Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 105
    • NCDENR – Business Continuity Plan – 2004 Organization -- NCZP Location -- Asheboro Detailed description of gap analysis: Natural disasters causing downed trees and damage to security of grounds cannot be prevented. To reduce risk need to have 24 hour 7 days/week security coverage, emergency contact info available, vehicle fueled and ready, and radio and/or telephone (land line or cell) working. Detail the impact to the Line of Business: Grounds must be secure to prevent animal escape and damage to facilities. Service needs to be recovered as soon as possible. Communications must be operational. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Cannot be reduced any further. Continue maintain 24 hour 7 days/week staffing. Continue to keep emergency contacts available. Continue to have vehicle fueled and ready. Upgrade radio system - replace one repeater and 20 portable units. Continue to have cell phone available. Length of time needed to correct or reduce risk to an acceptable level: 12 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $12,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Repaired one repeater. Are the interim controls currently in place? Yes. Zoo – Site Access Risk Level High Owner/Manager -- Gin Wall Department -- DENR Owner Detail Organization -- NCZP Location -- Asheboro Detailed description of gap analysis: Zoo staff needs to be able to get into the Zoo to take care of Animals, Plants, and life support systems. During storm events, trees may be down blocking access and compromising security fencing. Horticulture needs to be able to drive around the Park to assess the problems, communicate with Zoo communications via radios, have equipment such as chainsaws, bobcats, 4WD vehicles, and plows and supplies such as sand, shovels, gloves to clear the way. Detail the impact to the Line of Business: Access is key to getting to holding areas and exhibit areas containing animal and plant collections and life support system spread over 500 acres and 6 miles of service roads and to getting to service support where supplies, animal food and equipment are located. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Replace a repeater and 20 radio units. Replace six aging 4WD vehicles and the dump truck. Purchase two blades for scraping roads. Length of time needed to correct or reduce risk to an acceptable level: Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 106
    • NCDENR – Business Continuity Plan – 2004 12 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $180,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Continue to have staff, vehicles and equipment ready ahead of a storm event as specified in the Emergency Procedures. Are the interim controls currently in place? Yes. Zoo – Care of Animals Risk Level High Owner/Manager -- Ron Morris Department -- DENR Owner Detail Organization -- NCZP Location -- Asheboro Detailed description of gap analysis: Within a 24 hour period animals have to be fed and cared for so access to animal areas which are spread over 500 acres and six miles of service roads is important as is access to veterinarian care and supplies such as animal food, vet and cleaning. The life support systems for heating and cooling, water filtration, and pumps must be operational. Some animals are more critical than other such as puffins, which need water and air filtration and temperature between 50 and 55 degrees F. Communications via radio is important given the distances between animal areas. Detail the impact to the Line of Business: Without care the animals will be sick and may die. The Zoo could lose its USDA license to display animals and have to close. Also the Zoo could lose its AZA accreditation and have difficulty in acquiring animals. It would become a PR nightmare with the public. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Upgrade backup generator at Aviary and add backup generator to Rocky Coast, which has no backup. - $100,000. Replace six 4WD vehicles used to transport keepers to Zoo and animal areas and blades for scraping to clear access ways. - $160,000. Replace repeater and 20 radio units. - $12,000. Replace vet incinerator, which is used to dispose of dead animals. - $70,000. Add 20,000-gallon propane tank to increase propane supply from 3-4 days to 7-8 days. - $100,000 Length of time needed to correct or reduce risk to an acceptable level: 12 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $442,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Continue to have staff, vehicles and supplies ready in advance of a storm in accordance with Emergency Procedures. Continue to use backup generators and repaired repeater and radios. Continue to have quick response from REMC on electricity. Stay current on propane deliveries in winter. Continue to bury dead animals in accordance with State law. Are the interim controls currently in place? Yes. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 107
    • NCDENR – Business Continuity Plan – 2004 Zoo – Care of Plants Risk Level High Owner/Manager -- Gin Wall Department -- DENR Owner Detail Organization -- NCZP Location -- Asheboro Detailed description of gap analysis: The Zoo's plant collection requires care within 24 to 48 hours of a storm event or power outage so access to indoor exhibits and the greenhouse, which are spread over 500 acres and six miles of service roads, is important as is access to supplies such. The life support systems for heating and cooling must be operational. Some plants are more critical than other such as the Desert plants which require low humidity and temperatures 80 degrees F year round. Communication vis radio important given the distance between greenhouses and indoor exhibits. Detail the impact to the Line of Business: The plant collection is what makes the exhibit look natural. Without the plants, the Zoo's reputation would suffer and attendance would fall. The indoor plant collections are worth over $1 million and some plants cannot be replaced. Likely have PR problems regarding loss of plant collection. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Upgrade backup generator at Aviary - $30,000. Replace six 4WD vehicles used to transport horticulturists as well as keepers and staff needed to clear access ways. - $150,000. Replace repeater and 20 radio units. - $12,000. Replace vet incinerator, which is used to dispose of dead animals, so that Horticulture staff does not have to bury dead animals. - $70,000. Add 20,000-gallon propane tank to increase propane supply used to heat three indoor exhibits from 3-4 days to 7-8 days. - $100,000 Length of time needed to correct or reduce risk to an acceptable level: 12 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $332,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Continue to have staff, vehicles and supplies ready in advance of a storm in accordance with Emergency Procedures. Continue to use backup generators and repaired repeater and radios. Continue to have quick response from REMC on electricity. Stay current on propane deliveries in winter. Continue to bury dead animals in accordance with State law. Are the interim controls currently in place? Yes. Zoo – Life Support Systems Risk Level High Owner/Manager -- John McDonald, Brian Purnell Department -- DENR Owner Detail Organization -- NCZP Location -- Asheboro Detailed description of gap analysis: The Zoo has a variety of life support systems that are critical to the animal and plant collections. The electricity for the entire Zoo is backed up by a megagenerator operated by REMC and the Zoo. However, if the outage is inside the Zoo the megagenerator does not operate. The indoor exhibits are backed up by diesel Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 108
    • NCDENR – Business Continuity Plan – 2004 powered generators. However, the generator for the Aviary was not upgraded when the building was renovated and more HVAC installed. Also Rocky Coast containing chillers, pumps, and water filtration does not have it own backup generator. Rocky Coast as well as Streamside, has large pools that require hundreds of thousands of gallons of water with no backup. This is a problem if the water to these two exhibits is interrupted. The Zoo uses propane stored in two large tanks (30,000 gallons) to heat three indoor exhibits. The storage is enough for 3-4 days in cold weather. A third tank would extend the storage to 7-8 days in cold weather. The Zoo is the largest user of propane of all State facilities. The tanks are difficult to access if significant snow or ice is on the ground. Piedmont Natural Gas says the Zoo is too large to serve with their current line, which is less than a mile from the Zoo. Detail the impact to the Line of Business: The Aviary has 80-90 exotic birds and a large collection of tropical plants, which could sustain major damage and loss if electricity were out for two or more hours in the summer or winter. Rocky Coast has polar bears, seals, sea lions and sea birds, which require electricity for cooling, and water and air filtration. Particularly the sea birds could sustain major damage and/or loss if the power were out for two or more hours in the spring, summer or fall. An interruption in water flow to Rocky Coast and Streamside for more than 48 hours especially in hot weather could cause major damage and loss especially to Streamside fish collection and to Rocky Coast sea birds. An interruption in propane for 24 hours in cold weather would cause major damage and loss to the animal and plant collections in the Aviary, Sonora Desert and African Pavilion. The animal and plant collections are worth more than $2 million and in many cases could not be replaced. In all three cases - electricity, water and propane the Zoo could lose its USDA license and have to close and/or lose its AZA accreditation and have difficulty acquiring animals. The financial impact would be severe since the Zoo depends on admission, gift and food receipts for 40% ($5 million) of its current year operating budget. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Upgrade backup generator at Aviary and add backup generator to Rocky Coast. - $100,000. Add 20,000- gallon propane tank to increase propane supply from 3-4 days to 7-8 days. - $100,000 Use two potential groundwater wells to provide water to be stored at Streamside for backup for Streamside and Rocky Coast. - $50,000 Length of time needed to correct or reduce risk to an acceptable level: 12 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $250,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Continue to use backup generators. Continue to have quick response from REMC on electricity. Stay current on propane deliveries in winter. Are the interim controls currently in place? Yes Zoo – Visitor Access Risk Level High Owner Detail Owner/Manager -- Karen Auman Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 109
    • NCDENR – Business Continuity Plan – 2004 Department -- DENR Organization -- NCZP Location -- Asheboro Detailed description of gap analysis: The Zoo's budget is based on 40% ($5 million) from admission, gift and food receipts. Thus if the Zoo has to close due a storm event or power outage, the financial loss can be significant. To open to visitors, roads, parking lots and trails have to be cleared, admissions, food and gift need to be open, cash needs to be available and POS including credit card system if possible, exhibits have to be open, animals on exhibit, restrooms clean, trams and buses operational and rangers (EMTs) on Park. During a typical winter storm or hurricane event, the critical factor is how soon the service roads, parking lots can be cleared to bring in staff to care for animals and plants and life support system. DOT does not clear the two large parking lots or the one-mile internal road between the parking lots. The one mile NC 159 Spur access to the Entrance and Panther Creek Road which is access to Service Support are on DOT's secondary list which can take two or more days before they are cleared. Next is to open exhibits, clear trails, and bring in Visitor Services staff to clean restrooms and operate admissions, gift, cash management and transport. The biggest gap is the staffing and equipment to clear the Zoo of snow, ice and/or storm debris. Aging 4WD vehicles, lack of blade attachments for scraping roads and lack of a bucket truck for hanging limbs are problems. In the 2002 ice storm, the Zoo could not open certain trails to exhibits due to hanging limbs and no bucket truck for up to six months. Another major risk gap is the Point of Sale system and credit card process. These systems do not function in a power outage and sometimes fail on high attendance days due to aging software. The Zoo can function without POS and credit card systems, however every task including sales, cash management and inventory has to be done manually which is time and labor intensive. Given that it is responsible for $5+ million in receipts each year, the POS and credit card software need to be replaced. Detail the impact to the Line of Business: The longer the Zoo is closed due to storm cleanup, the more revenue it loses. Since the Zoo is required to generate $5 million out of a $13 million budget, the financial loss can be significant. The same is true of the POS and credit card systems failing whether it is due to a storm or high attendance. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Have DOT clear the one-mile NC 159 Spur, Panther Creek Road, the one-mile internal road and road to the Education Center as priorities. - Written agreement Replace six 4WD vehicles used to bring in staff and clear the roads and parking lots. Purchase a bucket truck to use to clear hanging limbs as well as to do routine arbor maintenance and blades to scrape roads. - $250,000 Replace POS and credit card software - $110,000 Length of time needed to correct or reduce risk to an acceptable level: 12 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $360,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Continue to have staff, equipment and vehicles ready in advance of storm events. Continue to clear tram roads and trails as quickly as possible with current staff, equipment and vehicles. Continue to use manual system when POS and credit card systems fail. Are the interim controls currently in place? Yes Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 110
    • NCDENR – Business Continuity Plan – 2004 Division of Air Quality – Toxic Air Pollutant Monitoring Risk Level High Owner/Manager -- Lee Daniel/ Lori Cherry Department -- DENR Owner Detail Organization -- Air Quality Location -- Parker Lincoln, Reedy Creek Road Laboratory Detailed description of gap analysis: A full gap analysis needs to be determined however deficiencies have been noted in the areas of IT support and in business continuity planning. Detail the impact to the Line of Business: Greater IT support could greatly increase the analytical capabilities of the mobile emergency laboratory's assessment abilities; especially in the areas of GIS and the ability to access the Division's own data remotely. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) To be identified once resources available. For data. However, part of the permanent solution would be a mobile dish connection back to the main server so all pertinent information could be obtained in real time. For the short term (and also as a backup measure should the network be down) an automated routine needs to be established to make copies of key information about facilities, GIS, and other support documents that would be manually stored on the MAML computer. For the Mobile Lab. The only way to eliminate the risk would be to have a second duplicate Mobile Amb Monitoring Lab. However this is highly unlikely due to the extreme cost ($855 K) Length of time needed to correct or reduce risk to an acceptable level: 6 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $855,000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) To be determined once resources are identified. However for the data needs as stated above, here is a proposed stopgap schedule to get key data stored to the MAML: (1) July 15th 2004. a) Determine which data elements need to be automatically copied to a storage system. b) Determine frequency of the need (likely monthly). c) Develop the process for getting the information to the MAML d) Provide IT with the specifications to create the automated data dump. e) Concurrently IT and ITSC will begin to evaluate the mobile satellite dish conncection. Discuss with DENR, OITS, etc. (2) August 15, 2004. IT will have a prototype of the automated data dump. TCB will review and return comments within one week. IT would then create the final system. (3) Sept 15, 2004. TCB test of using the system and placing in MAML. (4) Sept 30, 2004. TCB will begin to employ the system on a monthly basis (or whatever frequency was established). Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 111
    • NCDENR – Business Continuity Plan – 2004 Longterm. (*) Concurrent with the above items, the mobile dish connection will be evaluated for the long term solution. At this time, DAQ is not certain that this is approvable. Are the interim controls currently in place? There are some measures currently active Aquariums – Exhibit Animal Health Risk Level High Owner/Manager -- Directors of Operations Department -- DENR Owner Detail Organization -- Aquarium Division Location -- Roanoke Island, Fort Fisher Detailed description of gap analysis: None Detail the impact to the Line of Business: The loss of the exhibit wildlife would effectively shut down the effected Aquarium for as long as four weeks. Revenue loss, coupled with replacement cost could run into the hundreds of thousands of dollars. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) All viable risk mitigation triggers are currently in place. Length of time needed to correct or reduce risk to an acceptable level: 1 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $0 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) All controls are in place. Are the interim controls currently in place? Yes Aquariums – Visitor Traffic Risk Level High Owner/Manager -- Facility Directors Department -- DENR Owner Detail Organization -- Aquarium Division Location -- Roanoke Island, Fort Fisher Detailed description of gap analysis: None Detail the impact to the Line of Business: Must of all Aquarium operational areas on line to re-open for visitors ASAP. The longer the facilities take to recover from business disruption and initiate operational readiness the more severe the economic impact to the division and department. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 112
    • NCDENR – Business Continuity Plan – 2004 to be taken within next 12 months) Risk levels have been addressed and all possible mitigating blocks are in place. Length of time needed to correct or reduce risk to an acceptable level: 1 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $0 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Emergency generators, fuel storage, employee scheduling matrix. Are the interim controls currently in place? Yes Aquariums – Conservatory Plant Health Risk Level High Owner/Manager -- Aquarium Division Department -- DENR Owner Detail Organization -- North Carolina Aquariums Location -- Fort Fisher & Roanoke Island Detailed description of gap analysis: None Detail the impact to the Line of Business: None Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) None Length of time needed to correct or reduce risk to an acceptable level: 1 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $1 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) None Are the interim controls currently in place? Not Applicable Aquariums – Animal Health Risk Level High Owner/Manager -- Animal Husbandry Curator Department -- DENR Owner Detail Organization -- Aquarium Division Location -- Roanoke Island, Fort Fisher, Pine Knoll Shores Detailed description of gap analysis: None Detail the impact to the Line of Business: Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 113
    • NCDENR – Business Continuity Plan – 2004 The loss of the exhibit Life Support Systems for animals in the exhibit tanks would result in the death and closure of any effected tank within 24 hours of losing power. The Aquarium facility would have to be closed until new exhibit specimens could be brought in. This could mean a revenue loss to the facility on a peak period of more than $100,000. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Actions are already in place with the installation of emergency generators powerful enough to operate the facility along with 10,000 gallon fuel tanks which can keep the generators running for 5 days. A second fuel tank is planned for the future as a redundant fuel source. Length of time needed to correct or reduce risk to an acceptable level: 1 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $0 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) None Are the interim controls currently in place? All mitigating procedures are currently in place. DENR Information Technology Services – Systems Development Risk Level High Owner/Manager -- DENR ITS / Arvind Sinha, Ed King Department -- DENR Owner Detail Organization -- ITS Location -- Archdale Building Detailed description of gap analysis: Analysis from Phases I and II indicated that: 1. There are negative consequences for non-compliance with legal, regulatory, or policy requirements, 2. A man-made threat is the most likely threat and that it will result in >4 & <24 hours downtime, 3. That the IT planning process does not include operational resources to support any automated business process, 4. That operational resources are not adequate to deliver this essential service, 5. That a delay in the delivery of this line of business would result in missed deliverables, deadlines, or reporting requirements, 6. That there are gaps in IT staff training, 7. That staff turnover and/or unexpected loss are not adequately planned for, 8. That system coverage is not adequately maintained, 9. That there is not adequate funding for IT positions, 10.That agency interdependencies do not always operate seamlessly, 11.That there are gaps in system and end-user documentation, and that the system does not qualify for ongoing vendor support, 12.That a BCP does not currently exist, 13.That all key roles do not have a designated backup, 14.That all system administrators do not have adequate security skills. Detail the impact to the Line of Business: 1. The development of new systems is done in accordance with DENR, state, and in particular, IRMC policies. All systems are developed in accordance with those policies and result in the negative consequences related to those policies if that are not followed, as with all systems developed in the state. 2. The amount of downtime identified in the Phase 1 question >4 to <24 hours, while inconvenient, would be acceptable for this line of business. Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 114
    • NCDENR – Business Continuity Plan – 2004 3. DENR ITS staffing levels are not adequate to support any automated business process. The question should probably have been interpreted differently, but it will probably never be adequate to support ANY business process. There may have been a misunderstanding of the intent of the question. 4. Operational resources are not currently adequate. N.C. has just gone through (and may not be finished with) a budget crisis. DENR ITS lost positions as a result of budget issues at the beginning of implementing a major IT initiative (IBEAM). The impact on the lob is that there is great need for automation across the department that may not be addressed. 5. In the event of a disruption in the lob, system development goals and milestones may not be met. Reporting requirements would probably not be at issue as that would be more likely associated with the running of existing production applications as opposed to the development of new ones. Given the most likely type of disruption and its frequency of occurrence, the delay would probably be acceptable. 6.-9. The IBEAM initiative required the implementation of a comprehensive development environment, staff retraining, process development, and changes in corporate culture. This occurred at the beginning of massive state budget problems. As a result, positions were lost which affected all aspects of the initiative, but primarily in the resources available to work on implementation tasks. 10. Agency inter-dependencies generally operate without incident. However, the question was stated as a absolute (seamlessly). Perhaps the question should have been answered differently. 11. Given staffing issues, some procedures related to system development should be better documented. The effect on development is that additional time may be required to understand how development should proceed. 12. End-user documentation is developed by client staff. Because of this, it is better for some systems than for others. This may result in additional time required for developers working with client staff to discuss and test system functions. 13. The (IBEAM) framework used for system development was developed in-house by DENR ITS staff and contract staff as well. DENR ITS staff supports it. 14. The current exercise will result in a BCP. While this line of business is desirable, necessary, or even essential, it is not critical in the same sense as the need to maintain the ability to run the production applications. 15. The impact is that some key functions do not occur if key individuals are not present. 16. Security as it relates to system development is controlled by the infrastructure. Security skills are not relevant as related to this lob. 6. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) The identified risks that have an unacceptable impact on the lob include: 1. Staff issues as related to system coverage, ability to provide the lob, and the ability to document system infrastructure processes, and coverage (backup) of key staff funtions. Length of time needed to correct or reduce risk to an acceptable level: 12 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $ Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) Management is currently involved in reorientation of staff roles to allow for better coverage of key functions related to supporting system development. In the past this has involved movement of some staff from systems to development to other key areas like, client services. In the past and currently, staff are being repositioned to provide more infrastructure support. As buget allows, more staff will be allocated to the system development function itself. Are the interim controls currently in place? Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 115
    • NCDENR – Business Continuity Plan – 2004 DENR ITS – IBEAM Risk Level High Owner/Manager -- Bill Knight Department -- ITS Owner Detail Organization -- Department of Environment and Natural Resources Location -- Archdale Building, Raleigh Detailed description of gap analysis: The review of the moderates and the high-risk results from Phase I and II exposed the following: 1. Operational resources are currently not adequate to deliver this essential service due to the lack of staffing, inadequate training, and inadequate funding. 2. There would be a delay in delivery of services in the event of a business disruption. 3. Recovery strategies have not been reviewed and tested within the last 12 months. Detail the impact to the Line of Business: I-BEAM is the key software infrastructure framework upon which the centralized DENR information system processes are created and managed. Without this framework, I-BEAM based applications would become unavailable to the DENR staff who may be dependent on the data to make informed decisions in their job. Detail the permanent solution to eliminate or effectively reduce the risk to an acceptable level: (Action to be taken within next 12 months) Seek funding for additional staff positions and appropriate training expenditures. Implement a hot-site failover and recovery system in an offsite location. This is currently being developed and will be complete by Fall 2004. Review and test recovery strategies that utilize the hot-site recovery system and integrate the procedures into the DENR business continuity plan. Length of time needed to correct or reduce risk to an acceptable level: 12 Months What will it cost to eliminate or effectively reduce the risk to an acceptable level? (In US Dollars) $100000 Detail interim controls to reduce or mitigate the risk: (Include dates, tasks) 1. Document procedures for manually rebuilding system and recovering data. 2. Perform regular tape backups of all systems and data. 3. Store backup tapes in an offsite location. 4. Share duties among staff to cross-train and share skills. Are the interim controls currently in place? Yes Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 116
    • NCDENR – Business Continuity Plan – 2004 Appendix B: Workgroup Contact List CONTACT ORGANIZATION VOICE NO. EMAIL ADDRESS Chris Mills Aquariums 919-733-2290 Chris.Mills@ncmail.net Doug Lewis BPA 919-733-1710 Doug.Lewis@ncmail.net Tim Johnson Geographic Information & Analysis 919-733-2090 Tim.R.Johnson@ncmail.net Bryan Brannon Controller’s Office 919-733-5204 bryan.brannon@ncmail.net Edythe McKinney Customer Service Center 919-733-0823 edythe.mckinney@ncmail.net George Halsey Air Quality 919-733-0247 George.Halsey@ncmail.net Steven Vozzo Air Quality (FaRO) 910-486-1541 steven.vozzo@ncmail.net Josh Shepherd Coastal Management 919-733-2293 x236 Josh.Shepherd@ncmail.net Mike Kelly Environmental Health 919-715-0929 Mike.A.Kelly@ncmail.net Victor Harris Forest Resources 733-2162 x216 Victor.Harris@ncmail.net Joe Stroup Human Resources 919-715-4808 Joe.Stroup@ncmail.net Mark Hughes Information Technology Services 919-715-0330 Mark.Hughes@ncmail.net Tracy Davis Land Resources 919-733-4574 Tracy.Davis@ncmail.net Mike Buhl Marine Fisheries 252-726-7021 x104 Michael.Buhl@ncmail.net Gary Hunt Pollution Prevention/Env. Assist. 919-715-6508 Gary.Hunt@ncmail.net Carol Tingley Parks and Recreation Tracy Rains Purchase and Services 919-715-3889 traci.rains@ncmail.net David Williams Soil and Water Conservation 919-715-6103 David.B.Williams@ncmail.net Linda Culpepper Waste Management 919-733-4996 x203 Linda.Culpepper@ncmail.net Tom Fransen Water Resources 919-715-0381 Tom.Fransen@ncmail.net Jeff Poupart Water Quality Steve Busack Museum of Natural Sciences 919-733-7450 x701 steve.busack@ncmail.net Janine Nicholson Environmental Education 919-733-0711 Janine.Nicholson@ncmail.net Jim Stanfill OEEP [backup: Barb Satler] Don Reuter Office of Public Affairs 919-715-4112 Don.Reuter@ncmail.net Mary Joan Pugh Zoo 336-879-7110 MaryJoan.Pugh@ncmail.net CONTACT ORGANIZATION VOICE NO. EMAIL ADDRESS Jimmy Carter Secretary’s Office 919-733-4908 Jimmy.Carter@ncmail.net Jill Pafford Secretary’s Office 919-715-4193 Jill.Pafford@ncmail.net David Johnson ITS 919-715-4819 David.Johnson@ncmail.net Sam Kornegay ITS 919-715-0376 Sam.Kornegay@ncmail.net Chris Russo Secretary’s Office 919-7154169 Chris.Russo@ncmail.net Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 117
    • NCDENR – Business Continuity Plan – 2004 Appendix C: Crisis Communication Plan – Alternate Work Sites NOTE: Public versions of this plan do not disclose personal contact information. Name/Title  Division  Alternative Work  A.W.S.  Cell No.  Pager E‐Mail  Home Phone  Site  Phone No.  (if site is ʹhomeʹ) Bill Ross Administration Governor's Office 733-5811 Secretary DOA Dempsey Benton Administration Governor's Office 733-5811 815-3589 Deputy Secretary DOA Robin Smith, Administration 218-3833 Asst. Sec./Policy & Planning Dan Oakley Administration AG's Office, 716-6600 614-2859 General Counsel Environmental Protection, 3rd fl. Jimmy Carter Administration DAQ, Parker 733-3340 218-7426 Asst. Sec. Lincoln Bldg. Operations & Dev. Don Reuter Public Affairs Emergency 733-3825 218-8593 505-9304 Director Operations, DOA Bldg. Vacant PIO Public Affairs Emergency 733-3825 Operations, DOA Bldg. Lewis Ledford Parks & Rec Yorkshire Bldg. 846-9991 810-5950 Director Richard Rogers Conservation & Legislative (reach 604-7557 604-7557 N/A Director Community via cell) Affairs Lloyd Inman Administrative Raleigh Regional 571-4700 414-8548 983-8311 Director Services Office Terry Pierce Environmental Emergency 733-3825 810-5837 N/A Director Health Operations, DOA Bldg. Harlan Frye Human Raleigh Regional 571-4700 696-4091 Director Resources Office Charles Jones Coastal Mgmt. Raleigh Regional 571-4700 252-241-8185 Director Office Michael Bryant Purchasing Raleigh Regional 571-4700 630-6484 Director Office Rex Whaley Budget Planning Raleigh Regional 571-4700 Director & Analysis Office Rod Davis Controller's Office Raleigh Regional 571-4700 Director Office Stan Adams Forestry Griffith's Forestry 553-6178 218-3087 Director Center /Clayton Betsy Bennett Museum of Reedy Creek Lab 733-7450 225-2639 N/A Director Natural Sciences x750 Edythe McKinney Customer Service Raleigh Regional 571-4700 608-3114 Director Center Office Johanna Reese Legislative & Aquariums Office, 733-2290 606-1327 Director Intergovernmental Blount St. Affairs Alan Klimek Water Quality DWQ/Parker 733-6900 801-0451 1-800-614-2058 Director Lincoln Bldg. Steve Wall Policy Administration 672-2234 Analyst Mary Penny Administration AG's Office, Env. 716-6600 539-0707 Thompson Asst. Div., 3rd Floor General Counsel Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 118
    • NCDENR – Business Continuity Plan – 2004 Name/Title  Division  Alternative Work  A.W.S.  Cell No.  Pager E‐Mail  Home Phone  Site  Phone No.  (if site is ʹhomeʹ) Bill Gilmore Ecosystem Home 218-1774 Program Manager Enhancement Program Jill Pafford Administration Home Planning and Policy Analyst Chris Russo Administration State Emergency 733-3825 656-8319 1-888-475-7523 Director, Operation Center Organizational Excellence Craig Deal DOT Administration State Emergency 733-3300 218-4221 748-1049 Liaison Operation Center Jim Simons Land Resources Raleigh Regional 571-4700 614-2585 1-877-292-1093 Director Office Preston Pate Marine Fisheries Wilmington 910-395-3900 919-808-7315 1-800-283-1212 Director Regional Off. Dexter Matthews Waste Mgmt. Parker Lincoln 733-8486 218-4336 Director Bldg. David Jones NC Zoo Home 336-460-4603 Director Gary Hunt Pollution Oberlin Rd.-Waste 733-4996 Director Prevention Mgmt. 571-4700 or Raleigh Regional Office John Morris Water Resources Parker Lincoln 733-3340 Director Bldg. Tim Johnson CGIA Home 919-818-0526 Director Keith Overcash Air Quality Archdale Bldg. 733-4984 218-0138 Director David Vogel Soil & Water Home Director Bill Flournoy Sr. Consv. & Com. Home Consv. Spec. Aff. Tom Mather PIO Air Quality Bob Roush PIO Aquarium/Ft. Home 910-512-3901 Fisher Charles S. Powell Aquarium/Roanok Home 923-6303 PIO e Michele Walker Coastal Mgmt. PIO Tad Boggs PIO Ecosystem 218-1723 Enhancement Program Charlie Peek PIO Parks & 218-4622 Recreation Pat Kelly PIO Customer Service 877-623-6748 Ctr. Diana Kees PIO Pollution 368-2461 Prevention & Environmental Assist. Cathy Akroyd PIO Soil & Water 218-3456 Conser. Chrystal Bartlett Waste Parker Lincoln 733-8486 218-4239 PIO Management Susan Water Quality Home 810-5996 Massengale PIO Rod Hackney PIO Zoological Park Home 336-953-2983 vacant PIO Environmental Health Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 119
    • NCDENR – Business Continuity Plan – 2004 Name/Title  Division  Alternative Work  A.W.S.  Cell No.  Pager E‐Mail  Home Phone  Site  Phone No.  (if site is ʹhomeʹ) Jamie Kritzer PIO Forest Resources Home 1-800-283-1212, Pin #: 888-0328 Nancy Fish PIO Marine Fisheries Home 252-342-2151 vacant PIO Museum Of Natural Sciences Lisa Schell PIO CWMTF Home 244-4084 Lisa Tolley Off. Of Env. Home Director Education Mark Hughes Info. Technology Home 215-5869 505-5016 Director Svcs. NOTE: All area codes are 919 unless otherwise indicated.  Revised May 13, 2004  Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 120
    • NCDENR – Business Continuity Plan – 2004 Appendix D: Computer Application Inventory This table is a compilation of all registered computer applications in DENR that have been identified previously as “mission critical”. DENR is in the process of assessing these applications for backup and failover capabilities. DIVISION App Name Description/function Software Code SERVER SERVER O/S BackupExec Data Backup and Restore NROAR07 NetWare 5.Yes ADMIN Software ADMIN Office View Electronic Sign-in/Out Board NROAR07 NetWare 5.Yes ADMIN Word Perfect Word processing NROAR07 NetWare 5.Yes Budget Allows for revision and data entry COBOL IBM Mainframe MVS All revision pgm of budget data CICSNR Provides database and tools for COBOL IBM Mainframe MVS Natural Resource management statewide - Boat licenses, wildlife licenses, internal (DMF) All convictions and warning Contract Tracks contracts VB Dell Win 2000 All Tracking All Eprocurement Purchasing of Commodities unk unk unk Fixed Assets Participating agency in fixed COBOL IBM 0S/370 All asset management tool All Lease Tracking Track leases unk unk unk Mainframe Provides Statistical analysis of COBOL IBM Mainframe MVS All SAS data sets on the mainframe NC MapNet Provides GIS data from the ArcIMS Dell Win 2000 Corporate Geographic Database to other agencies and the public via the Internet. Data includes base map information such as roads, rivers and streams, and municipal boundaries, as well as thematic data such such as water and sewer systems and conservation lands. NC mapnet enables CGIA to make data available for decision making and other purposes in an easy to use All form. All Office View Electronic Sign-in/Out Board DellPowerEdge 2300 Netware 5.Yes Parking Tracks Assignment of Parking unk unk unk All Spaces TSO Provides statewide mainframe Unknown IBM Mainframe MVS All access All White Pages Phone directory unk unk unk NCAS Participating agency in financial COBOL Dell Netware 5.Yes All - (OSC) management tool All - OfficeViewPro Electronic Sign-in/Out Board and unknown NROAR06 Novell NetWare Asheville internal messaging system 5.Yes Regional Office Xtnd Provides centralized financial COBOL IBM Mainframe MVS all - OSC reports toagencies and dvisions Personnel Provide database and tools to COBOL Sun Solaris Management manage personnel records and Information actions for state agencies and All - OSP System (PMIS) the university system Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 121
    • NCDENR – Business Continuity Plan – 2004 DIVISION App Name Description/function Software Code SERVER SERVER O/S Decision DSS Facility Information WEB NRMSITSYes40/Yes45 MS WIN2K Support Template for States IiNTERFACE & /02Yes/022/023/024/02 Server & Adv System WEB LOGIC & 6/PVCS Server ALL CRYSTAL Regulatory REPORTS & divisions ORACLE Decision Dss state environmental WEB NRMSITSYes40/Yes45 MS WIN2K Support protection act (sepa) tracking IiNTERFACE & /02Yes/022/023/024/02 Server & Adv System system WEB LOGIC & 6/PVCS Server All CRYSTAL Regulatory REPORTS & divisions ORACLE DAQ Access Complaints database Access NROAR05 NW5 DAQ Access Emissions inventory Access NROAR05 NW5 Access Reports/annual & quarterly Access NROAR05 NW5 DAQ facility DAQ Actions Compiance Tracking application java Dell Microsoft Windows 2000 Server DAQ AERO Emission Inventory Online java Dell Microsoft application Windows NT Server 4.0 DAQ Air awareness Airawareness/Ozone/PM2.5/Cont java Dell Microsoft acts application Windows NT Server 4.0 DAQ Air Emissions Emission Inventory application java Dell Microsoft Inventory Windows NT Server 4.0 DAQ Air Quality Permit Writer application java Dell Microsoft Permit Writer Windows NT Server 4.0 DAQ Ambient Ambient Monitoring Yes hour java Dell Microsoft Monitoring application Windows 2000 Server DAQ Ambient Sites Ambient Sites application java Dell Microsoft Windows NT Server 4.0 DAQ AQI Air Quality Index application java Dell Microsoft Windows NT Server 4.0 DAQ ArcInfo/View ESRI GIS software unknown Dell Linux DAQ Budget Module Budget application java Dell Microsoft Windows NT Server 4.0 DAQ Calendar Enterprise wide Calendar unknown unknown unknown system, supervisors only DAQ CMAQ EPA Air Quality Modeling unknown Dell Linux software DAQ Comment/Mes Message Board application java Dell Microsoft sage Board Windows NT Server 4.0 DAQ Complaint Complaints application java Dell Microsoft Tracker Windows NT Server 4.0 Compliance Archived Complaints Prior to May NROAR07 NetWare 5.Yes DAQ Database 2003 DAQ DAQ External Division public external web site html unknown unknown Web Site DAQ DAQ Internal Division wide Internal Web Site java Dell Microsoft Web Site Windows NT Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 122
    • NCDENR – Business Continuity Plan – 2004 DIVISION App Name Description/function Software Code SERVER SERVER O/S Server 4.0 DAQ Document Facility Documents application java Dell Microsoft Management Windows NT system Server 4.0 emissions Tracks submittals of annual Excel nroar04 netware 5.Yes DAQ inventory emissions inventories DAQ ESM Emission Sources application java Dell Microsoft Windows NT Server 4.0 DAQ Exceed Hummingbird terminal emulation unknown Dell Linux DAQ Facilities Facilities application java Dell Microsoft Windows NT Server 4.0 Facility Tracks permits, inspections and Excel Dell PowerEdge 2300 Netware 5.Yes DAQ Compliance general facility information DAQ Fees Fees application java Dell Microsoft Windows NT Server 4.0 DAQ FELIX Management software unknown Dell Linux DAQ File/Print Division Central Office and Lab n/a Dell Novell Netware, Services File/Print, ftp, DHCP, LDAP, Microsoft PDC, BDC, wins, Backup Exec Windows NT Server 4.0 DAQ Forecasters Infrastructure for Forecasters unknown Dell Linux Modeling Modeling software; Red Hat Infrastructure Linux Hog Odor Tracks hog odor complaints Access Dell PowerEdge 2300 Netware 5.Yes DAQ Complaints HogOdor - Compliance report tracking NROAR07 NetWare 5.Yes DAQ Access DAQ IEPA Electronic Applications java Dell Microsoft application Windows 2000 Server DAQ IMPAQ vYes Framework for IMPAQ java Dell Microsoft Framework applications, Authentication, Windows NT Authorization, Navigation, Server 4.0 Scheduler, Messenger, … I-Steps Tracks permits, inspections and Dell PowerEdge 2300 Netware 5.Yes DAQ general facility information iTools Tracks DAQ permit holder FoxPro nroar04 netware 5.Yes DAQ deadlines DAQ MM5 NCAR Meteorological Modeling unknown Dell Linux software DAQ NC Mail Enterprise wide Email system unknown unknown unknown DAQ Norton Division Central Office and Lab n/a Dell Microsoft AntiVirus Norton antivirus Windows NT Server 4.0 NowDataLAN- Compliance report tracking NROAR07 NetWare 5.Yes DAQ Access DAQ PAVE CEP graphical software unknown Dell Linux DAQ Permit Permit applications application java Dell Microsoft Application Windows NT Management Server 4.0 System DAQ PGI Portland Group compiler unknown Dell Linux technology DAQ Pollen Pollen application java Dell Microsoft Windows NT Server 4.0 DAQ SMOKE MCNC Emission Modeling unknown Dell Linux Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 123
    • NCDENR – Business Continuity Plan – 2004 DIVISION App Name Description/function Software Code SERVER SERVER O/S software DAQ Source Test Source Test application java Dell Microsoft Module Windows 2000 Server DAQ Toxics Lab Toxics Lab Automation java Dell Microsoft Automation application Windows 2000 Server DAQ Training Training application java Dell Microsoft Tracker Windows 2000 Server DAQ Violations Violations application java Dell Microsoft Windows NT Server 4.0 DAQ Vis5D Sourceforge graphical software unknown Dell Linux CAMA Permit Database used to track permits Cobol SIPS Mainframe Tracker issued. Only tracks DCM general/majors. CDAITS Database used to track permits VB.NET/ASP.NE DCM2kServer Windows 2000 DCM issued. T SP2 Consistency Database used to track Dbase NROBA02 Netware 5 6a Tracking consistency determinations and Database generate letters for DCM mailout/comments. DEH List serv Web based email distribution list Python Dell Power App Yes00 Windows 2000 app web server/Dell PowerEdge6300 DEH Database Server DEH.Dnet System to consolidate many SQL/ASP Dell Power App Yes00 Windows 2000 sectional databases into one web server/Dell PowerEdge6300 DEH Database Server DSWC Deh web calandar SQL/ASP Dell Power App Yes00 Windows 2000 web server/Dell PowerEdge6300 DEH Database Server EHS Tracking system for Run from the NA DEH Authorizations Environmental Specialist workstation FAS Computer hardware inventory SQL/ASP Dell Power App Yes00 Windows 2000 tracking web server/Dell PowerEdge6300 DEH Database Server HR Employee personnel and training SQL/ASP Dell Power App Yes00 Windows 2000 data web server/Dell PowerEdge6300 DEH Database Server Intern Training Tracking system for MS Access/VB Run from the NA Environmental Specialist Intern workstation DEH Training DEH NAV Nortons antivirus Dell PowerEdge Server Windows 2000 EHS Plans EHS Plan Review tracking SQL/ASP Dell Power App Yes00 Windows 2000 Review web server/Dell PowerEdge6300 DEH - EHS Database Server LeadSiteActivit Tracking system of Lead Poison SQL/ASP Dell Power App Yes00 Windows 2000 y cases web server/Dell PowerEdge6300 DEH - EHS Database Server Wade Online Wade Tracking system for SQL/ASP Dell Power App Yes00 Windows 2000 DEH - OSWW web server /Dell OSWW PowerEdge6300 Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 124
    • NCDENR – Business Continuity Plan – 2004 DIVISION App Name Description/function Software Code SERVER SERVER O/S Database Server West Nile dead Database for dead birds found in SQL/ASP Dell Power App Yes00 Windows 2000 bird tracking NC web server/Dell DEH - PowerEdge6300 PHPM Database Server On-line Data On-line Data Review Perl Dell Power Debian Linux DEH - PWS Review EdgeYes300 PWS Inventory Pws inventory Itext/Java/JSP Dell Power App Yes00 Windows 2000 web server/Dell PowerEdge6300 DEH - PWS Database Server DLR Access Sediment database Access NROAR05 NW5 DAMS & IBEAM dams & mine tracking WEB NRMSITSYes40/Yes45 MS WIN2K MINES system IiNTERFACE & /02Yes/022/023/024/02 Server & Adv SYSTEM WEB LOGIC & 6/PVCS Server CRYSTAL REPORTS & DLR ORACLE FRO Dams - Dam statistics -old paradox NROAD07 NetWare 5.Yes DLR Access FRO Mines - Mine stats - old paradox NROAD07 NetWare 5.Yes DLR Access FRO Sediment Sediment statistics - old paradox NROAD07 NetWare 5.Yes DLR - Access DLR Sediment Tracks active SPCA projects Access nroar04 netware 5.Yes Sediment Track Field Inspections on Paradox Dell PowerEdge 2300 Netware 5.Yes DLR Database Projects Access Dam and Mining Inventory Access NRDLR0Yes NetWare 5 DLR / DSW Database DLR / DSW Asset Insight Inventory tracking application NRDLR0Yes NetWare 5 Backup Exec 7 Network operating and file NRDLR0Yes NetWare 5 for Windows system back up DLR / DSW NT Backup Exec Network operating and file NRDLR0Yes NetWare 5 8.5 for system back up DLR / DSW NetWare DLR / DSW NetWare 5 Network operating system NRDLR0Yes NetWare 5 NetWare 5 Host for DLR divisional web site NRDLRWEB NetWare 5 DLR / DSW Web Server Symantec Antivirus software NRDLR0Yes NetWare 5 DLR / DSW Antivirus Arcview/SQL Provides a link between Arcview Arcview, SQL Compaq Proliant Windows 2000 link maps and SQL datasets Server, DMF ArcAvenue Biological Data Marine Fisheries collection of SAS, FTP, IBM Mainframe MVS DMF Base data COBOL Biological Data Provides DMF access to PERL, xitami, PC Dell PC Windows 98 Base/ Web Biological Database via intranet SAS DMF Server server FIN Provides support for DMF SQL Server, Compaq Proliant Windows 2000 Commercial/Recreational fishing Visual Basic, C++ DMF licensees and Permits Fisheries Program that tracks proclamation dbase Compaq Proliant Netware 5 DMF Proclamations release recipients Fisheries Marine Fisheries collection of MS Access Compaq Proliant Netware 5 Quota quota monitoring of catch data DMF Monitoring License Inquiry Provides access to fisheies dbase Compaq Proliant Netware 5 license information prior to DMF Yes999 Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 125
    • NCDENR – Business Continuity Plan – 2004 DIVISION App Name Description/function Software Code SERVER SERVER O/S NCFisheries Provides external access and MS Frontpage Compaq Proliant Windows 2000 DMF web page news to fisheires information Trip Ticket Provides mainframe access to SAS,FTP IBM Mainframe MVS DMF catch information Vessel Old vessel licensing program COBOL IBM Mainframe MVS DMF Licenses / IMS ENVIRONMEN Ibeam, environmental WEB NRMSITSYes40/Yes45 MS WIN2K TAL management system IiNTERFACE & /02Yes/022/023/024/02 Server & Adv MANAGEMEN WEB LOGIC & 6/PVCS Server T SYSTEM CRYSTAL REPORTS & DPPEA ORACLE DPR Asset Insight Hardware/software inventory Off Shelf NRDPR03 Netware 5.Yes Butterflies of Butterfly inventory FileMaker Pro Web Server Mac DPR North Carolina OS8/Webstar Case Incident Visitor emergency/infraction FileMaker Pro Database Server Mac DPR Reporting record OS8/Webstar Comman Virus protection Off Shelf NRDPR03 Netware 5.Yes DPR Antivirus DPR Document tracking FileMaker Pro Database Server Mac correspondenc OS8/Webstar DPR e DPR Internet Online discussion forum MySQL, PHP Database Server Mac Discussion OSYes0/Apach DPR Forum e Yes.3.27 DPR mailing Data for Large Mailing FileMaker Pro Database Server Mac DPR labels OS8/Webstar DPR Training Employee training calendar MySQL, PHP Database Server Mac Calendar OSYes0/Apach DPR e Yes.3.27 Environmental Search Tool for EELE Workshop FileMaker Pro Web Server Mac Educ. Learning OS8/Webstar Experience DPR Workshops Environmental Ongoing Statistical Record of FileMaker Pro Database Server Mac Education & Programs Provided OS8/Webstar Interpretation Programs and DPR Statistics Identification Employee ID Info and log FileMaker Pro Database Server Mac Cards for OS8/Webstar commissioned DPR DPR personnel Identification Employee ID Info and log FileMaker Pro Database Server Mac Cards for non- OS8/Webstar commissioned DPR DPR personnel Natural Statistical log of rare species or FileMaker Pro Web Server Mac Heritage habitat OS8/Webstar Program Element Occurrence by DPR County Natural Statistical log of rare species or FileMaker Pro Web Server Mac Heritage habitat OS8/Webstar Program Element Occurrence by DPR Topo Quad DPR Natural Inventory of park natural MySQL, PHP Database Server Mac Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 126
    • NCDENR – Business Continuity Plan – 2004 DIVISION App Name Description/function Software Code SERVER SERVER O/S Resource resources OSYes0/Apach Inventory e Yes.3.27 Database Parks & In-house Training Calendar MySQL, PHP Database Server Mac Recreation OSYes0/Apach Training - e Yes.3.27 Interpretation & DPR Education State Park Park activity search tool FileMaker Pro Web Server Mac DPR Activities OS8/Webstar State Park Monthly events information FileMaker Pro Web Server Mac Calendar of search tool OS8/Webstar DPR Events State Park Park facility open/closed status FileMaker Pro Web Server Mac DPR Closings OS8/Webstar State Park Project Info, Log and Forecast MySQL, PHP Database Server Mac Strategic OSYes0/Apach Planning for e Yes.3.27 DPR I&E State Park Data Used to Create Web Based MySQL, PHP Database Server Mac Virtual Tours Views of Facilities _ pictoral tour OSYes0/Apach DPR e Yes.3.27 IIS through IIS through Oracle passthrough DSW_ACSP Windows NT 4.0 Oracle to access Oracle database in DSW DENR ITSD DSWC FileMaker Pro Tracks swine operation reviews FileMaker Pro DellPowerEdge 2300 Netware 5.Yes Able EPA program used to determine MS Access NRUST0Yes Netware 5.Yes financial resonsibility for Compaq 2500* corporations. Used for Eligibility DWM process for Federal Trust Funds Application Provides access to the scanned MS SQL Compaq Proliant 370 Win 2000 Xtender files from the Hazardous Waste Server File room. This consists of information from Yes980 through DWM the present. Biennial This application tracks the Visual Foxpro Compaq 5500 Netware 5.Yes Reporting hazardous waste generation, System treatment and transportation, biennially, of the large quantity generators and hazardous waste treatment, storage and disposal DWM facilities in NC. Consultants Database used to generate MS Access NRUST0Yes Netware 5.0 labels and mailing lists to Compaq 2500* DWM external consultnats, lawyers, etc Disaster Debris Used in establishment and MS Access Compaq 5500 Netware 5.Yes Sites organization of disaster debris DWM storm sites DSCA_Master Data storage file for the Access MSAccess Compaq 5500 Netware 5.Yes database application serving the Drycleaning Solvent Cleanup Act DWM Program. Eligibility Supports all Regional Offices, MS Access NRUST0Yes Netware 5.Yes Database provides data entry, data Compaq 2500* management and data extraction DWM functions for STF eligibility. Enforcement Provides data entry, data MS Access NRUST0Yes Netware 5.Yes management and data extraction Compaq 2500* functions for incidents that have DWM gone to legal Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 127
    • NCDENR – Business Continuity Plan – 2004 DIVISION App Name Description/function Software Code SERVER SERVER O/S FacilityYes Tracks contacts and addresses MS Access Compaq 5500 Netware 5.Yes for all permitted solid waste facilities, tracks waste received at these facilities by fiscal year. (this data is used to maintain a list of solid waste facilities and contacts used by the public as well as provides data to support the solid waste annual report on a yearly DWM basis.) Federal Trust Supports Central & Regional MS Access NRUST0Yes Netware 5.Yes Fund Database Offices, provides data entry, data Compaq 2500* management and data extraction functions for Federal fund DWM payment/reimbursement FITS Provides data entry, data MS Access NRUST0Yes Netware 5.Yes management and data Compaq 2500* manipulation functions for Facility Information Tracking for States DWM uploads. Hazardous This application tracks the fee Access Compaq 5500 Netware 5.Yes Waste collection of facilities handling Facilities Fee hazardous waste. Used by Collection personnel of Waste DWM Management. Inactive Cataloging of hazardous MSAccess Compaq 5500 Netware 5.Yes Hazardous substance disposal sites Sites addressed by program. Mandatory for management of work and reporting to the DWM legislature. IndiPay EPA program used to determine MS Access NRUST0Yes Netware 5.Yes financial resonsibility for Compaq 2500* individuals. Used for Eligibility DWM process for Federal Trust Funds Notifications Maintains Permit Information for MS Access Compaq 5500 Netware 5.Yes Yard Waste and T&P DWM Notifications Old Landfills Cataloging of old unlined lined MSAccess Compaq 5500 Netware 5.Yes Tracking landfills requiring investigation. Data is very preliminary and subject to inaccuracies. Mandatory for management of work and reporting to the DWM legislature. Permitting Local copy of data from MS Access NRUST0Yes Netware 5.Yes Underground Storage Tank Compaq 2500* DWM Permitting/Fee database Regional Supports all Regional Offices, MS Access NRUST0Yes Netware 5.Yes Underground provides data entry, data Compaq 2500* Storage Tank management and data extraction DWM Database functions Rolodex Mailing information for county MS Access Compaq 5500 Netware 5.Yes DWM officials and interested parties Septage Fee Maintains Permit, Fee and MS Access Compaq 5500 Netware 5.Yes Collection Compliance information for the DWM Septage Management Program SF Fedsite Tracking database for CERCLA Fox Pro Compaq 5500 Netware 5.Yes DWM documents DWM Soil Reports Septage site soil report data MS Access Compaq 5500 Netware 5.Yes Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 128
    • NCDENR – Business Continuity Plan – 2004 DIVISION App Name Description/function Software Code SERVER SERVER O/S State Trust Supports Central & Regional MS Access NRUST0Yes Netware 5.Yes Fund Database Offices, provides data entry, data Compaq 2500* management and data extraction functions for payments/reimbursements for DWM incidents Tax Used in the development of tax MS Access Compaq 5500 Netware 5.Yes Certification certifications, compliance reviews, inspection evaluations, violations, compliance orders, financial assurance, monthly DWM reports, status of sw sites MANAGEMEN Management of hazardous waste WEB NRMSITSYes40/Yes45 MS WIN2K T SYSTEM rcra - epa system IiNTERFACE & /02Yes/022/023/024/02 Server & Adv WEB LOGIC & 6/PVCS Server CRYSTAL DWM REPORTS & RCRA ORACLE 4D Database used in tracking and DELL NT Server storing data related to DWQ certifications of operators 4D Server Used in tracking and storing data DELL Win NT Server related to certifications of DWQ operators Above Ground Stores data regarding above Access97 DELL PowerEdge 2400 Novell Storage Tank ground storage tanks DWQ Database Arc Info Application used to develp GIS Commercial Windows 2000 DWQ coverages Software BEA Weblogic Used in the development of BIMS DELL W2K server Client/Develop DWQ er BEA Used in the development of DELL W2K server WeblogicServe Basinwide Information r Management System (BIMS - Manages a centralized source of data about WQ Section's activities and activities affecting DWQ water quality). BIMS Application that provides and java DELL W2K server DWQ stores DWQ critical data Complaint Stores environmental complaints Visual Basic DELL PowerEdge 2400 Novell System submitted by the public to the DWQ Regional Offices Compliance Stores test data submitted by Access97/Parado DELL PowerEdge 2400 Novell Monitoring permitted facilities regarding x DWQ Database nondischarge sites. DWQ DB2 Used in the development of BIMS SUN UNIX Enforcement Stores data regarding Paradox DELL PowerEdge 2400 Novell Database enforcement action taken by the DWQ Groundwater Section FileMaker Pro Units within section use this DELL Netware 5.0 application to create databases Application to store vital information related object to their work. These are shared DWQ across section/division FileMaker Pro Hosting numerous databases DELL Win NT Server Server valuable to specific units within DWQ the section DWQ GWYes's Stores data regarding well Access97 DELL PowerEdge 2400 Novell Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 129
    • NCDENR – Business Continuity Plan – 2004 DIVISION App Name Description/function Software Code SERVER SERVER O/S Database construction IBEAM IBEAM version of the above Java DELL PowerEdge 2400 Novell Incident DWQ Management Incident Allows the public to determine Visual Basic DELL PowerEdge 2400 Windows 2000 Contamination the distance between Server Area Search contamination incidents and a DWQ defined Latitude/Longitude Incident Permits Regional Offices to Visual Basic DELL PowerEdge 2400 Novell Management submit Groundwater DWQ Contamination Incident Data Internet Main web server html Dell PowerEdge 2300 Windows NT Information DWQ Services Inventory Stores hardware inventory Paradox DELL PowerEdge 2400 Novell Database information for the Groundwater DWQ Section Java Webstart Manages the deployment of java DELL W2K server BIMS and BIMS Reports DWQ Applications Jreports Used in the development of BIMS DELL W2Ks erver DWQ Designer Jreports Used in the development of BIMS DELL W2K server Enterprise DWQ Server NDPerm Stores data regarding non- Paradox DELL PowerEdge 2400 Novell DWQ Database Discharge permits Novell Netware Provides File/Print sharing of WQ Dell Netware 5.0 DWQ staff data Novell Desktop mgmt application used DELL Netware 5.0 DWQ Zenworks in association with Netware OS Public Incident Allows the public to retrieve Visual Basic DELL PowerEdge 2400 Windows 2000 Management incident data through the web Server DWQ Query System Pump Registration information Paradox DELL PowerEdge 2400 Novell Installers regarding Pump Installers DWQ Database DWQ Rational Rose Used in the development of BIMS ALR Win NT Symantec Antivirus application DELL Win NT DWQ Antivirus UIC Database Stores data regarding the Paradox DELL PowerEdge 2400 Novell permitting of Underground DWQ Injection Control systems Veritas Backup Backup Exec - - remote agents DELL W2K DWQ Exec Well Stores data regarding Well Access97 DELL PowerEdge 2400 Novell Contractors Contractor certification, testing, Database continuing education and DWQ payments. Wells Stores permit data regarding Paradox DELL PowerEdge 2400 Novell DWQ Database groundwater withdrawal DWQ Envir 4D Database for Self Monitoring proprietary Dell PowerEdge 2300 Windows NT Sciences stations DWQ Envir 4D Database for Biological proprietary Dell PowerEdge 2300 Windows NT Sciences monitoring DWQ Envir Oracle Database unknown Dell PowerEdge 2300 Windows NT Sciences DWQ Envir STORET Stores water sampling data for unknown Dell PowerEdge 2300 Windows NT Sciences water quality testing DWQ, Sample Combination of Excel '97 Microsoft Excel NRDWQLB03 None Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 130
    • NCDENR – Business Continuity Plan – 2004 DIVISION App Name Description/function Software Code SERVER SERVER O/S DSWM Tracking and Spreadsheets used to track Receiving sample data flow and archive sample results within Laboratory Section. Financial Provide the ability to relate Sybase SQL Compaq Netware Information project level moneys to the state 4.YesYes Control Sytem NCAS system and provide a valid DWQ/CGL (FICS) audit trail of all transactions. DWQ/SWC FileMaker Pro Dwq database application NROAD07 NetWare 5.Yes Environmen HSIS Inspection, Tracking and Billing COBOL OITS tal Health System (Mainframe System) MAINFRAME Groundwat All Sample Track sample well data NROAD07 NetWare 5.Yes er Data-Excel Groundwat Soil Permits- Track soil permits NROAD07 NetWare 5.Yes er Excel Groundwat Well Permits- Track well permits NROAD07 NetWare 5.Yes er Excel HR SYSTEM Job Posting and Application WEB MS WIN2K Entry IiNTERFACE & Server & Adv WEB LOGIC & Server HUMAN CRYSTAL RESOURC REPORTS & ES ORACLE CISCO Pix firewall management CISCO WORKS NRMSITS250 MS WIN2K ITS WORKS Server GIS IBEAM GIS functions ARCIMS NRGIS/NRGISDEV/NR MS WIN2K ITS MSITS006 Server GIS/FTP Gis ftp systems FTP NRGISDEV MS WIN2K ITS SYSTEMS Server NortonAntivirus Server antivirus NortonAntivirus All Servers MS WIN2K ITS Server ORACLE Manage oracle database servers OEM NRMSITS005/0YesYes MS WIN2K ENTERPRISE Server ITS MANAGER PVCS Maintain source code PVCS NRMSITS029 MS WIN2K ITS Server SOURCESAF Maintain source code SOURCESAFE NRCUPSDEVWB WIN2K, WIN2K ITS E Adv Server NetWare 5.0 LAN operating system, allows Compaq Win 95 shared components to be utilized for communication and cost Museum effectiveness Norton Antivirus software Museum Enterprise Museum QWS3270 Mainframe access software Museum XTND Mainframe access software Natural Biological and Provides database and tool to unknown Redhat Linux 9 Heritage Conservation manage information on rare Program Database species and natural communities. BizTracker Tracks visitor and gift sales and unknown Dell PowerEdge 4600 Netware 5.Yes NC ZOO Point-of-Sale gift inventory BizTracker Tracks visitor and gift sales and unknown HP Netserver Netware 5.Yes Point-of-Sale gift & general warehouse NC ZOO inventory TM Vista Schedules Aquarium Programs, SQL Dell Workstation Windows 2000 allows visitors to register for programs, provides calendar of NCA at FF events NCA at TM Vista Schedules Aquarium Programs, SQL Dell Workstation Windows 2000 PKS allows visitors to register for Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 131
    • NCDENR – Business Continuity Plan – 2004 DIVISION App Name Description/function Software Code SERVER SERVER O/S programs, provides calendar of events TM Vista Schedules Aquarium Programs, SQL Dell Workstation Windows 2000 allows visitors to register for programs, provides calendar of NCA at RI events Office of Ipswitch FTP Allow Public Affair to transfer web ? ? Public WS_FTP LE files to SIPS server Affairs Cost Cost distribution system COBOL OITS Distribution (mainframe system) MAINFRAME System (Mainframe OOC System) Health Plan Health plan reconciliation system COBOL OITS Reconsolation (mainframe system) MAINFRAME System (Mainframe OOC System) TLAS Denr timesheet for personnel Oracle, IIS, NRCUPSPRODWB & MS WIN2K Adv OOC Comm+ NRMSITS00Yes Server Pur & Serv Contracts Tracks contracts unk unk unk Pur & Serv Dbase PO #'s before PO Tracking unk unk unk Pur & Serv Lotus Organizational charts unk unk unk Pur & Serv PO Tracking Tracks PO's unk unk unk Compliance Track regional ust compliance NROAD07 NetWare 5.Yes UST DB-Access UST RUST -Access Regional ust database NROAD07 NetWare 5.Yes Soil Permitting Track soil acceptance NROAD07 NetWare 5.Yes UST DB State Trust Track stf pre approvals NROAD07 NetWare 5.Yes UST Fund Pre App License sales Provides access via telephone COBOL IBM Mainframe MVS WRC lines for WRC license sales Rev: JUL 06 2004 • NCDENR Business Continuity Plan (Internal Version) 132