SlideShare a Scribd company logo
Trillium Software Disaster Recovery /
Business Continuity Response Plan
Revision History:
When Who Changes
01/03/16 Percuoco Updated Cloud DR / BC Strategy external document creation
01/05/16 Percuoco Updated Cloud DR / BC Strategy with various small edits
01/05/16 Percuoco Updated RTO and RPO with Will’s feedback
01/06/16 Percuoco Updated annual exercise BC language
Trillium Software Disaster Recovery / Business Continuity
Response Plan
Confidential Page 1 2/10/2016
1. Disaster Recovery / Business Continuity Management................................................ 2
1.1. Defining the Disaster Recovery / Business Continuity Mission....................... 2
1.1.1. Manufacturing / Fulfillment........................................................................ 2
1.1.2. Customer Support ....................................................................................... 3
1.1.3. Development Engineering Support............................................................. 3
1.2. Recovery Time Objective and Recovery Point Objective ................................ 3
1.3. Cloud Disaster Recovery / Business Continuity Practices ............................... 4
2. Trillium Disaster Recovery / Business Continuity Architecture .................................. 7
3. Management, Communication, Coordination............................................................... 8
4. Exercising, Maintenance and Review........................................................................... 8
5. Contact Information...................................................................................................... 9
Trillium Software Disaster Recovery / Business Continuity
Response Plan
Confidential Page 2 2/10/2016
1 Disaster Recovery / Business Continuity
Management
Disaster Recovery / Business Continuity Management (BCM) is a process that identifies
potential critical areas that could impact an organization’s ability to deliver under both
normal daily or extreme disaster conditions. BCM provides a framework for location
resilience and the capability for an effective response in the face of unknown events.
BCM stresses identifying, creating, testing and improving contingency strategies with
respect to contractual deliverables and services, critical resources and processes to allow
the creation and support of the Harte-Hanks Trillium Software products and customer
base both in normal and/or extreme conditions.
1.1 Defining the Disaster Recovery / Business Continuity
Mission
It is important to understand that a successful Disaster Recovery / Business Continuity
plan is not an exercise that is done once and is over. As business functions change over
time, reacting to both internal and external conditions, the defined critical areas and the
contingency strategies for those critical areas need to be updated with regularity. Also,
there needs to be an exercise regimen to test and ensure that the contingency strategies
and resources are solid and will work over time in adverse conditions. Lastly, Disaster
Recovery / Business Continuity is a mission which will require an annual exercise,
analysis of results and investment to continue to deliver a quality response in adverse
conditions.
Critical to the Disaster Recovery / Business Continuity mission is to understand each
critical service and deliverable in the Trillium organization. Each critical area’s
deliverables and processes are documented and the urgency with which these activities
needed to be resumed if they are disrupted. The following contractual critical response
areas are:
1.1.1 Manufacturing / Fulfillment of Postal Directory updates
and critical Trillium Software System software patches
Manufacturing and fulfillment is a critical process in the Trillium organization as a
result of the data and product file creation and fulfillment related to the Postal
Directory subscription business. As there are financial penalties and Service Level
Agreement (SLA) support implications if contractual delivery obligations are not
met, continuity of the Postal Directory build and release process is critical. The
majority of Postal Directory client subscriptions are monthly although some are
created quarterly and even annually. Manufacturing also is responsible for
maintaining, posting and distributing digital product release and patch media to the
global client audience. All of these releases are transferred through our secure
Trillium Software Disaster Recovery / Business Continuity
Response Plan
Confidential Page 3 2/10/2016
Cloud MoveIt FTP (File Transfer Protocol) website hosted in Microsoft Azure that
is globally Cloud enabled and automatically digitally redundant to a different
geography’s Azure virtual data center in the event of disaster.
1.1.2 Customer Support of Trillium Software System software
maintenance contracts
Customer Support is defined to be a critical process because of the nature and
obligations of the departments’ mission. At Trillium software client issues and
support is everyone’s primary focus and responsibility. Trillium Software has
always fostered a service and support culture as defined by the Service Level
Agreements (SLA) with our customers. We are morally and contractually bound to
provide critical maintenance support functions and critical bug resolution to our
clients even in adverse scenarios. All customer support personnel work from
virtual workstation environments in a secure private cloud in the Harte Hanks
NDC Data Center which has global connectivity to satellite support offices in
Australia and Germany. Dedicated support resources in all global locations are
available to login remotely and resolve critical client issues. The Harte Hanks
NDC data center is digitally synchronized and fully redundant with automated
failover to the Harte Hanks SDC data center.
1.1.3 Development Engineering of Postal build process and
critical Trillium Software System software patches
Development Engineering is a critical Trillium process to enable the creation and
fulfillment of Trillium Software System, critical patches and releases as defined by
contractual maintenance and support obligations. All program development and
production resource file builds reside in a secure private cloud in the Harte Hanks
NDC Data Center which has global connectivity to engineering offices in the US
and UK. Dedicated engineering resources in both locations are available to create
critical patches or create contractual periodic Postal file updates. The Harte Hanks
NDC data center is digitally synchronized and fully redundant with automated
failover to the Harte Hanks SDC data center.
1.2 Recovery Time Objective and Recovery Point Objective
Trillium Software uses two common industry terms for Disaster Recovery / Business
Continuity planning:
 Recovery time objective (RTO) — RTO is the time it takes after a disruption to
restore a business process to its service level, as defined by the Service level
agreement (SLA). For example, if a disaster occurs at 12:00 PM (noon) and the
Trillium Software Disaster Recovery / Business Continuity
Response Plan
Confidential Page 4 2/10/2016
RTO is one hour, the DR process should restore the business process to the
acceptable service level by 1:00 PM. The client production environments that are
not on premise exist in an AWS private Cloud instance one per client with fully
redundant failover to a different AWS global availability zone. Once an official
disaster is declared by Trillium Cloud Operations, under standard operating
procedures during normal business hours, a new production instance can be spun
up and the last virtual snapshot loaded (taken every 5 secs in production) in a
second global facility with an objective RTO of a 10-12 minute window.
 Recovery point objective (RPO) — RPO is the acceptable amount of data loss
measured in time. For example, if a disaster occurs at 12:00 PM (noon) and the
RPO is one hour, the system should recover all data that was in the system before
11:00 AM. Data loss will span only one hour, between 11:00 AM and 12:00 PM
(noon). As already stated the client production environments that are not on
premise exist in an AWS private Cloud instance one per client with fully
redundant failover to a different AWS global availability zone. Once an official
disaster is declared by Trillium Cloud Operations, under standard operating
procedures during normal business hours, a new production instance can be spun
up and the last virtual snapshot loaded (taken every 5 secs in production) in a
second global facility with an objective RPO of the same 10-12 minute window.
1.3 Cloud Disaster Recovery / Business Continuity Practices
The Trillium Software approach to Disaster Recovery and associated Disaster Recovery /
Business Continuity involves different levels of off-site duplication of data and
infrastructure. Critical business services are set up and maintained on these digitally
redundant infrastructures and tested at regular intervals. The disaster recovery
environment’s location and the source infrastructure should be a significant physical
distance apart to ensure that the disaster recovery environment is isolated from faults that
could impact the source site.
Trillium Software uses Amazon Web Services Cloud availability in multiple regions
around the globe, so there is a choice for the most appropriate location for the Disaster
Recovery / Business Continuity site, in addition to the site where the client production
system is fully deployed. AWS has multiple general purpose regions in North America,
South America, Europe, the Middle East, Africa and Asia Pacific. In this way the location
of the client servers are configurable and particular international data regulations like the
Safe Harbor act can be architected and adhered to.
Trillium Software in the Cloud uses Amazon’s simple storage service (Amazon S3)
which provides a highly durable storage infrastructure designed for mission-critical
applications and primary data storage. Storage objects are redundantly stored on multiple
devices (3) across multiple facilities within a region, designed to provide a durability of
99.999999999% (11 9s). Trillium provides further protection for data retention and
Trillium Software Disaster Recovery / Business Continuity
Response Plan
Confidential Page 5 2/10/2016
archiving through AWS multi-factor authentication (AWS MFA), storage bucket
policies, and AWS Identity and Access Management (IAM) roles.
The Trillium Software System Cloud enablement uses Amazon Elastic Block Store
(Amazon EBS) to provide the ability to create point-in-time snapshots of data volumes.
These snapshots are taken every 5 seconds, with minimal effect on performance, in the
client production instance and include the client data and the running state of the system
at that moment. These snapshots are then used as the starting point for the disaster
recovery virtual images that are instantiated when needed. Client data is protected for
long-term durability because snapshots are stored within Amazon S3. After a snapshot is
created it can be attached to a failover replication Disaster Recovery / Business
Continuity Amazon EC2 instance in another predefined global location.
In relation to Trillium’s Cloud Production client instances of the Trillium Software
System, Amazon Machine Images (AMIs) are virtual machines that are preconfigured
with the host operating system, hardened with virus checking and malware software, then
loaded with the current licensed and production released version of the Trillium Software
System. In the context of Disaster Recovery / Business Continuity objectives golden
image Trillium AMIs are preconfigured and launched and spun up in a different global
availability zone as part of the production recovery procedure.
AWS Availability Zones are distinct locations that are engineered to be insulated from
failures in other Availability Zones. By launching instances in separate Availability
Zones Trillium Software can protect its client’s applications from the failure of a single
location. Trillium client Regions consist of two or more Availability Zones and each
AWS private cloud Trillium client production instance always has two fully synchronized
virtual machine instances concurrently running in two distinct global Availability Zones.
Updating and executing between the two client virtual images in different Availability
Zones is how the Trillium Postal Directories and software is maintained and upgraded
with minimal down time.
Trillium Software uses a Disaster Recovery / Business Continuity scenario in which
recovery versions are provisioned and launched in a different Availability Zone to restore
business-critical services using pre-configured virtual servers bundled as Amazon
Machine Images (AMIs), which are ready to be started up automatically at a moment’s
notice. When starting redundant recovery servers instances from these AMIs boot up and
digital state snapshots are loaded very quickly.
Trillium Software automates the provisioning and configuration of the infrastructure
Disaster Recovery / Business Continuity resources which is a significant benefit to save
time and protect against human error.
Key steps for Disaster Recovery / Business Continuity preparation:
 Set up Amazon EC2 instances to replicate or mirror data.
Trillium Software Disaster Recovery / Business Continuity
Response Plan
Confidential Page 6 2/10/2016
 Ensure that the Trillium Software System and all pertinent components are
available in AWS.
 Create and maintain Trillium golden image AMIs of key servers where fast
recovery is required.
 Regularly run and test these servers as part of Trillium Quality Assurance
applying any software updates and configuration changes.
After recovery, redundancy is restored as quickly as possible. A failure of the Disaster
Recovery / Business Continuity environment shortly after the client production
environment fails is unlikely but possible so please be aware of this risk.
Trillium Software Disaster Recovery / Business Continuity
Response Plan
Confidential Page 7 2/10/2016
2 Trillium Disaster Recovery / Business Continuity
Architecture
TSS
Development
VM1
TSS Batch VM3
TSS Batch VM4
TSS
Development
VM2
Local
F:
Local
F:
Local
F:
Trillium
Control
Center
Trillium
Deployed
Batch project
Trillium
Deployed
Batch project
Client
Input/
output
Trillium
Project
Postal
Update
Client Input file
Output file, Stats, logs
AMI Golden Image File
AWS Snapshot File
Project / Postals
Local
F:
Trillium
Control
Center
Instance
Reboot
Instance
Reboot
Trillium Software Disaster Recovery / Business Continuity
Response Plan
Confidential Page 8 2/10/2016
3 Management, Communication, Coordination
Communication of the plan and its execution is the responsibility of the VP of Cloud
Operations and the VP of Software Development acting in an operational capacity. Their
responsibility is to:
 Communicate to affected areas that they should execute on previously
documented Disaster Recovery / Business Continuity strategies. Part of this
communication is a published Emergency Contact Hierarchy that defines the BCR
response teams including contact information, area and responsibilities.
 Define the Disaster Recovery / Business Continuity areas, processes and
objectives into a prioritized and coordinated Disaster Recovery / Business
Continuity project plan designed specifically to get the most throughput and time
efficiencies in the event of an incident.
 Coordinate with state and local agencies in the event of a communicated state
emergency.
4 Exercising, Maintenance and Review
As part of the Trillium Disaster Recovery / Business Continuity Response capability a
reliable plan must be exercised, maintained and updated. The Disaster Recovery /
Business Continuity response program is a continual process that matures over time as
business needs, deliverables and department processes change. The central management
resource is responsible for:
 Periodic Disaster Recovery / Business Continuity Response drills per critical
function to make sure critical area processes are updated, documented and
working.
 Periodic audits and updates of critical function Disaster Recovery / Business
Continuity Response plans to reflect business need changes in the area’s critical
objectives and processes.
Trillium Software Disaster Recovery / Business Continuity
Response Plan
Confidential Page 9 2/10/2016
5 Contact Information
Company Contact Phone Email
Trillium Rick Percuoco
– VP Cloud
Operations
978-670-
4980
richard_percuoco@trilliumsoftware.com
Trillium. Aaron Frenger
– VP of
Sofware
Development
978-439-
3816
aaron_frenger@trilliumsoftware.com
Trillium Fiona
MacDonald –
Primary
Engineer Cloud
Operations
978-436-
2903
fiona_macdonald@trilliumsoftware.com
Trillium Jason Renaud-
Customer
Support
Manager
978-436-
2854
jason_renaud@trilliumsoftware.com

More Related Content

Similar to Business Continuity Plan V5

Bluelock's Recovery Suite
Bluelock's Recovery SuiteBluelock's Recovery Suite
Bluelock's Recovery Suite
Bluelock
 
Disaster Recovery: Develop Efficient Critique for an Emergency
Disaster Recovery: Develop Efficient Critique for an EmergencyDisaster Recovery: Develop Efficient Critique for an Emergency
Disaster Recovery: Develop Efficient Critique for an Emergency
sco813f8ko
 
ProfitBricks-white-paper-Disaster-Recovery-US
ProfitBricks-white-paper-Disaster-Recovery-USProfitBricks-white-paper-Disaster-Recovery-US
ProfitBricks-white-paper-Disaster-Recovery-USMudia Akpobome
 
Disaster Recovery: Understanding Trend, Methodology, Solution, and Standard
Disaster Recovery:  Understanding Trend, Methodology, Solution, and StandardDisaster Recovery:  Understanding Trend, Methodology, Solution, and Standard
Disaster Recovery: Understanding Trend, Methodology, Solution, and Standard
PT Datacomm Diangraha
 
How to Make an Effective Cloud Disaster Recovery Strategy.pdf
How to Make an Effective Cloud Disaster Recovery Strategy.pdfHow to Make an Effective Cloud Disaster Recovery Strategy.pdf
How to Make an Effective Cloud Disaster Recovery Strategy.pdf
Sysvoot Antivirus
 
Boomerang Total Recall
Boomerang Total RecallBoomerang Total Recall
Boomerang Total Recall
bdoyle05
 
cloud Resilience
cloud Resilience cloud Resilience
cloud Resilience
Integral university, India
 
Enterprise-Grade Disaster Recovery Without Breaking the Bank
Enterprise-Grade Disaster Recovery Without Breaking the BankEnterprise-Grade Disaster Recovery Without Breaking the Bank
Enterprise-Grade Disaster Recovery Without Breaking the Bank
Donna Perlstein
 
Enterprise-Grade Disaster Recovery Without Breaking the Bank
Enterprise-Grade Disaster Recovery Without Breaking the BankEnterprise-Grade Disaster Recovery Without Breaking the Bank
Enterprise-Grade Disaster Recovery Without Breaking the Bank
CloudEndure
 
ACIC Rome & Veritas: High-Availability and Disaster Recovery Scenarios
ACIC Rome & Veritas: High-Availability and Disaster Recovery ScenariosACIC Rome & Veritas: High-Availability and Disaster Recovery Scenarios
ACIC Rome & Veritas: High-Availability and Disaster Recovery Scenarios
Accenture Italia
 
Disaster Recovery Deep Dive
Disaster Recovery Deep DiveDisaster Recovery Deep Dive
Disaster Recovery Deep DiveLiberteks
 
Disaster Recovery Plan
Disaster Recovery PlanDisaster Recovery Plan
Disaster Recovery PlanDavid Donovan
 
COMPANY Disaster Recovery Plan (DRP) for [PRODU.docx
COMPANY    Disaster Recovery Plan (DRP) for [PRODU.docxCOMPANY    Disaster Recovery Plan (DRP) for [PRODU.docx
COMPANY Disaster Recovery Plan (DRP) for [PRODU.docx
monicafrancis71118
 
Joe Graziano – Challenge 2 Design Solution (Part 1)
Joe Graziano – Challenge 2 Design Solution (Part 1)Joe Graziano – Challenge 2 Design Solution (Part 1)
Joe Graziano – Challenge 2 Design Solution (Part 1)
tovmug
 
V mware quick start guide to disaster recovery
V mware   quick start guide to disaster recoveryV mware   quick start guide to disaster recovery
V mware quick start guide to disaster recoveryVMware_EMEA
 
Mastering Backup and Disaster Recovery: Ensuring Data Continuity and Resilience
Mastering Backup and Disaster Recovery: Ensuring Data Continuity and ResilienceMastering Backup and Disaster Recovery: Ensuring Data Continuity and Resilience
Mastering Backup and Disaster Recovery: Ensuring Data Continuity and Resilience
MaryJWilliams2
 
Symantec Disaster Recovery Orchestrator: One Click Disaster Recovery to the C...
Symantec Disaster Recovery Orchestrator: One Click Disaster Recovery to the C...Symantec Disaster Recovery Orchestrator: One Click Disaster Recovery to the C...
Symantec Disaster Recovery Orchestrator: One Click Disaster Recovery to the C...
Symantec
 
Data Protection and Disaster Recovery Solutions: Ensuring Business Continuity
Data Protection and Disaster Recovery Solutions: Ensuring Business ContinuityData Protection and Disaster Recovery Solutions: Ensuring Business Continuity
Data Protection and Disaster Recovery Solutions: Ensuring Business Continuity
MaryJWilliams2
 
Shielding Data Assets: Exploring Data Protection and Disaster Recovery Strate...
Shielding Data Assets: Exploring Data Protection and Disaster Recovery Strate...Shielding Data Assets: Exploring Data Protection and Disaster Recovery Strate...
Shielding Data Assets: Exploring Data Protection and Disaster Recovery Strate...
MaryJWilliams2
 

Similar to Business Continuity Plan V5 (20)

Bluelock's Recovery Suite
Bluelock's Recovery SuiteBluelock's Recovery Suite
Bluelock's Recovery Suite
 
Disaster Recovery: Develop Efficient Critique for an Emergency
Disaster Recovery: Develop Efficient Critique for an EmergencyDisaster Recovery: Develop Efficient Critique for an Emergency
Disaster Recovery: Develop Efficient Critique for an Emergency
 
ProfitBricks-white-paper-Disaster-Recovery-US
ProfitBricks-white-paper-Disaster-Recovery-USProfitBricks-white-paper-Disaster-Recovery-US
ProfitBricks-white-paper-Disaster-Recovery-US
 
Disaster Recovery: Understanding Trend, Methodology, Solution, and Standard
Disaster Recovery:  Understanding Trend, Methodology, Solution, and StandardDisaster Recovery:  Understanding Trend, Methodology, Solution, and Standard
Disaster Recovery: Understanding Trend, Methodology, Solution, and Standard
 
How to Make an Effective Cloud Disaster Recovery Strategy.pdf
How to Make an Effective Cloud Disaster Recovery Strategy.pdfHow to Make an Effective Cloud Disaster Recovery Strategy.pdf
How to Make an Effective Cloud Disaster Recovery Strategy.pdf
 
Boomerang Total Recall
Boomerang Total RecallBoomerang Total Recall
Boomerang Total Recall
 
cloud Resilience
cloud Resilience cloud Resilience
cloud Resilience
 
Enterprise-Grade Disaster Recovery Without Breaking the Bank
Enterprise-Grade Disaster Recovery Without Breaking the BankEnterprise-Grade Disaster Recovery Without Breaking the Bank
Enterprise-Grade Disaster Recovery Without Breaking the Bank
 
Enterprise-Grade Disaster Recovery Without Breaking the Bank
Enterprise-Grade Disaster Recovery Without Breaking the BankEnterprise-Grade Disaster Recovery Without Breaking the Bank
Enterprise-Grade Disaster Recovery Without Breaking the Bank
 
ACIC Rome & Veritas: High-Availability and Disaster Recovery Scenarios
ACIC Rome & Veritas: High-Availability and Disaster Recovery ScenariosACIC Rome & Veritas: High-Availability and Disaster Recovery Scenarios
ACIC Rome & Veritas: High-Availability and Disaster Recovery Scenarios
 
Disaster Recovery Deep Dive
Disaster Recovery Deep DiveDisaster Recovery Deep Dive
Disaster Recovery Deep Dive
 
Disaster Recovery Plan
Disaster Recovery PlanDisaster Recovery Plan
Disaster Recovery Plan
 
COMPANY Disaster Recovery Plan (DRP) for [PRODU.docx
COMPANY    Disaster Recovery Plan (DRP) for [PRODU.docxCOMPANY    Disaster Recovery Plan (DRP) for [PRODU.docx
COMPANY Disaster Recovery Plan (DRP) for [PRODU.docx
 
Joe Graziano – Challenge 2 Design Solution (Part 1)
Joe Graziano – Challenge 2 Design Solution (Part 1)Joe Graziano – Challenge 2 Design Solution (Part 1)
Joe Graziano – Challenge 2 Design Solution (Part 1)
 
V mware quick start guide to disaster recovery
V mware   quick start guide to disaster recoveryV mware   quick start guide to disaster recovery
V mware quick start guide to disaster recovery
 
Mastering Backup and Disaster Recovery: Ensuring Data Continuity and Resilience
Mastering Backup and Disaster Recovery: Ensuring Data Continuity and ResilienceMastering Backup and Disaster Recovery: Ensuring Data Continuity and Resilience
Mastering Backup and Disaster Recovery: Ensuring Data Continuity and Resilience
 
Symantec Disaster Recovery Orchestrator: One Click Disaster Recovery to the C...
Symantec Disaster Recovery Orchestrator: One Click Disaster Recovery to the C...Symantec Disaster Recovery Orchestrator: One Click Disaster Recovery to the C...
Symantec Disaster Recovery Orchestrator: One Click Disaster Recovery to the C...
 
Disaster Recovery is Dead
Disaster Recovery is DeadDisaster Recovery is Dead
Disaster Recovery is Dead
 
Data Protection and Disaster Recovery Solutions: Ensuring Business Continuity
Data Protection and Disaster Recovery Solutions: Ensuring Business ContinuityData Protection and Disaster Recovery Solutions: Ensuring Business Continuity
Data Protection and Disaster Recovery Solutions: Ensuring Business Continuity
 
Shielding Data Assets: Exploring Data Protection and Disaster Recovery Strate...
Shielding Data Assets: Exploring Data Protection and Disaster Recovery Strate...Shielding Data Assets: Exploring Data Protection and Disaster Recovery Strate...
Shielding Data Assets: Exploring Data Protection and Disaster Recovery Strate...
 

Business Continuity Plan V5

  • 1. Trillium Software Disaster Recovery / Business Continuity Response Plan Revision History: When Who Changes 01/03/16 Percuoco Updated Cloud DR / BC Strategy external document creation 01/05/16 Percuoco Updated Cloud DR / BC Strategy with various small edits 01/05/16 Percuoco Updated RTO and RPO with Will’s feedback 01/06/16 Percuoco Updated annual exercise BC language
  • 2. Trillium Software Disaster Recovery / Business Continuity Response Plan Confidential Page 1 2/10/2016 1. Disaster Recovery / Business Continuity Management................................................ 2 1.1. Defining the Disaster Recovery / Business Continuity Mission....................... 2 1.1.1. Manufacturing / Fulfillment........................................................................ 2 1.1.2. Customer Support ....................................................................................... 3 1.1.3. Development Engineering Support............................................................. 3 1.2. Recovery Time Objective and Recovery Point Objective ................................ 3 1.3. Cloud Disaster Recovery / Business Continuity Practices ............................... 4 2. Trillium Disaster Recovery / Business Continuity Architecture .................................. 7 3. Management, Communication, Coordination............................................................... 8 4. Exercising, Maintenance and Review........................................................................... 8 5. Contact Information...................................................................................................... 9
  • 3. Trillium Software Disaster Recovery / Business Continuity Response Plan Confidential Page 2 2/10/2016 1 Disaster Recovery / Business Continuity Management Disaster Recovery / Business Continuity Management (BCM) is a process that identifies potential critical areas that could impact an organization’s ability to deliver under both normal daily or extreme disaster conditions. BCM provides a framework for location resilience and the capability for an effective response in the face of unknown events. BCM stresses identifying, creating, testing and improving contingency strategies with respect to contractual deliverables and services, critical resources and processes to allow the creation and support of the Harte-Hanks Trillium Software products and customer base both in normal and/or extreme conditions. 1.1 Defining the Disaster Recovery / Business Continuity Mission It is important to understand that a successful Disaster Recovery / Business Continuity plan is not an exercise that is done once and is over. As business functions change over time, reacting to both internal and external conditions, the defined critical areas and the contingency strategies for those critical areas need to be updated with regularity. Also, there needs to be an exercise regimen to test and ensure that the contingency strategies and resources are solid and will work over time in adverse conditions. Lastly, Disaster Recovery / Business Continuity is a mission which will require an annual exercise, analysis of results and investment to continue to deliver a quality response in adverse conditions. Critical to the Disaster Recovery / Business Continuity mission is to understand each critical service and deliverable in the Trillium organization. Each critical area’s deliverables and processes are documented and the urgency with which these activities needed to be resumed if they are disrupted. The following contractual critical response areas are: 1.1.1 Manufacturing / Fulfillment of Postal Directory updates and critical Trillium Software System software patches Manufacturing and fulfillment is a critical process in the Trillium organization as a result of the data and product file creation and fulfillment related to the Postal Directory subscription business. As there are financial penalties and Service Level Agreement (SLA) support implications if contractual delivery obligations are not met, continuity of the Postal Directory build and release process is critical. The majority of Postal Directory client subscriptions are monthly although some are created quarterly and even annually. Manufacturing also is responsible for maintaining, posting and distributing digital product release and patch media to the global client audience. All of these releases are transferred through our secure
  • 4. Trillium Software Disaster Recovery / Business Continuity Response Plan Confidential Page 3 2/10/2016 Cloud MoveIt FTP (File Transfer Protocol) website hosted in Microsoft Azure that is globally Cloud enabled and automatically digitally redundant to a different geography’s Azure virtual data center in the event of disaster. 1.1.2 Customer Support of Trillium Software System software maintenance contracts Customer Support is defined to be a critical process because of the nature and obligations of the departments’ mission. At Trillium software client issues and support is everyone’s primary focus and responsibility. Trillium Software has always fostered a service and support culture as defined by the Service Level Agreements (SLA) with our customers. We are morally and contractually bound to provide critical maintenance support functions and critical bug resolution to our clients even in adverse scenarios. All customer support personnel work from virtual workstation environments in a secure private cloud in the Harte Hanks NDC Data Center which has global connectivity to satellite support offices in Australia and Germany. Dedicated support resources in all global locations are available to login remotely and resolve critical client issues. The Harte Hanks NDC data center is digitally synchronized and fully redundant with automated failover to the Harte Hanks SDC data center. 1.1.3 Development Engineering of Postal build process and critical Trillium Software System software patches Development Engineering is a critical Trillium process to enable the creation and fulfillment of Trillium Software System, critical patches and releases as defined by contractual maintenance and support obligations. All program development and production resource file builds reside in a secure private cloud in the Harte Hanks NDC Data Center which has global connectivity to engineering offices in the US and UK. Dedicated engineering resources in both locations are available to create critical patches or create contractual periodic Postal file updates. The Harte Hanks NDC data center is digitally synchronized and fully redundant with automated failover to the Harte Hanks SDC data center. 1.2 Recovery Time Objective and Recovery Point Objective Trillium Software uses two common industry terms for Disaster Recovery / Business Continuity planning:  Recovery time objective (RTO) — RTO is the time it takes after a disruption to restore a business process to its service level, as defined by the Service level agreement (SLA). For example, if a disaster occurs at 12:00 PM (noon) and the
  • 5. Trillium Software Disaster Recovery / Business Continuity Response Plan Confidential Page 4 2/10/2016 RTO is one hour, the DR process should restore the business process to the acceptable service level by 1:00 PM. The client production environments that are not on premise exist in an AWS private Cloud instance one per client with fully redundant failover to a different AWS global availability zone. Once an official disaster is declared by Trillium Cloud Operations, under standard operating procedures during normal business hours, a new production instance can be spun up and the last virtual snapshot loaded (taken every 5 secs in production) in a second global facility with an objective RTO of a 10-12 minute window.  Recovery point objective (RPO) — RPO is the acceptable amount of data loss measured in time. For example, if a disaster occurs at 12:00 PM (noon) and the RPO is one hour, the system should recover all data that was in the system before 11:00 AM. Data loss will span only one hour, between 11:00 AM and 12:00 PM (noon). As already stated the client production environments that are not on premise exist in an AWS private Cloud instance one per client with fully redundant failover to a different AWS global availability zone. Once an official disaster is declared by Trillium Cloud Operations, under standard operating procedures during normal business hours, a new production instance can be spun up and the last virtual snapshot loaded (taken every 5 secs in production) in a second global facility with an objective RPO of the same 10-12 minute window. 1.3 Cloud Disaster Recovery / Business Continuity Practices The Trillium Software approach to Disaster Recovery and associated Disaster Recovery / Business Continuity involves different levels of off-site duplication of data and infrastructure. Critical business services are set up and maintained on these digitally redundant infrastructures and tested at regular intervals. The disaster recovery environment’s location and the source infrastructure should be a significant physical distance apart to ensure that the disaster recovery environment is isolated from faults that could impact the source site. Trillium Software uses Amazon Web Services Cloud availability in multiple regions around the globe, so there is a choice for the most appropriate location for the Disaster Recovery / Business Continuity site, in addition to the site where the client production system is fully deployed. AWS has multiple general purpose regions in North America, South America, Europe, the Middle East, Africa and Asia Pacific. In this way the location of the client servers are configurable and particular international data regulations like the Safe Harbor act can be architected and adhered to. Trillium Software in the Cloud uses Amazon’s simple storage service (Amazon S3) which provides a highly durable storage infrastructure designed for mission-critical applications and primary data storage. Storage objects are redundantly stored on multiple devices (3) across multiple facilities within a region, designed to provide a durability of 99.999999999% (11 9s). Trillium provides further protection for data retention and
  • 6. Trillium Software Disaster Recovery / Business Continuity Response Plan Confidential Page 5 2/10/2016 archiving through AWS multi-factor authentication (AWS MFA), storage bucket policies, and AWS Identity and Access Management (IAM) roles. The Trillium Software System Cloud enablement uses Amazon Elastic Block Store (Amazon EBS) to provide the ability to create point-in-time snapshots of data volumes. These snapshots are taken every 5 seconds, with minimal effect on performance, in the client production instance and include the client data and the running state of the system at that moment. These snapshots are then used as the starting point for the disaster recovery virtual images that are instantiated when needed. Client data is protected for long-term durability because snapshots are stored within Amazon S3. After a snapshot is created it can be attached to a failover replication Disaster Recovery / Business Continuity Amazon EC2 instance in another predefined global location. In relation to Trillium’s Cloud Production client instances of the Trillium Software System, Amazon Machine Images (AMIs) are virtual machines that are preconfigured with the host operating system, hardened with virus checking and malware software, then loaded with the current licensed and production released version of the Trillium Software System. In the context of Disaster Recovery / Business Continuity objectives golden image Trillium AMIs are preconfigured and launched and spun up in a different global availability zone as part of the production recovery procedure. AWS Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones. By launching instances in separate Availability Zones Trillium Software can protect its client’s applications from the failure of a single location. Trillium client Regions consist of two or more Availability Zones and each AWS private cloud Trillium client production instance always has two fully synchronized virtual machine instances concurrently running in two distinct global Availability Zones. Updating and executing between the two client virtual images in different Availability Zones is how the Trillium Postal Directories and software is maintained and upgraded with minimal down time. Trillium Software uses a Disaster Recovery / Business Continuity scenario in which recovery versions are provisioned and launched in a different Availability Zone to restore business-critical services using pre-configured virtual servers bundled as Amazon Machine Images (AMIs), which are ready to be started up automatically at a moment’s notice. When starting redundant recovery servers instances from these AMIs boot up and digital state snapshots are loaded very quickly. Trillium Software automates the provisioning and configuration of the infrastructure Disaster Recovery / Business Continuity resources which is a significant benefit to save time and protect against human error. Key steps for Disaster Recovery / Business Continuity preparation:  Set up Amazon EC2 instances to replicate or mirror data.
  • 7. Trillium Software Disaster Recovery / Business Continuity Response Plan Confidential Page 6 2/10/2016  Ensure that the Trillium Software System and all pertinent components are available in AWS.  Create and maintain Trillium golden image AMIs of key servers where fast recovery is required.  Regularly run and test these servers as part of Trillium Quality Assurance applying any software updates and configuration changes. After recovery, redundancy is restored as quickly as possible. A failure of the Disaster Recovery / Business Continuity environment shortly after the client production environment fails is unlikely but possible so please be aware of this risk.
  • 8. Trillium Software Disaster Recovery / Business Continuity Response Plan Confidential Page 7 2/10/2016 2 Trillium Disaster Recovery / Business Continuity Architecture TSS Development VM1 TSS Batch VM3 TSS Batch VM4 TSS Development VM2 Local F: Local F: Local F: Trillium Control Center Trillium Deployed Batch project Trillium Deployed Batch project Client Input/ output Trillium Project Postal Update Client Input file Output file, Stats, logs AMI Golden Image File AWS Snapshot File Project / Postals Local F: Trillium Control Center Instance Reboot Instance Reboot
  • 9. Trillium Software Disaster Recovery / Business Continuity Response Plan Confidential Page 8 2/10/2016 3 Management, Communication, Coordination Communication of the plan and its execution is the responsibility of the VP of Cloud Operations and the VP of Software Development acting in an operational capacity. Their responsibility is to:  Communicate to affected areas that they should execute on previously documented Disaster Recovery / Business Continuity strategies. Part of this communication is a published Emergency Contact Hierarchy that defines the BCR response teams including contact information, area and responsibilities.  Define the Disaster Recovery / Business Continuity areas, processes and objectives into a prioritized and coordinated Disaster Recovery / Business Continuity project plan designed specifically to get the most throughput and time efficiencies in the event of an incident.  Coordinate with state and local agencies in the event of a communicated state emergency. 4 Exercising, Maintenance and Review As part of the Trillium Disaster Recovery / Business Continuity Response capability a reliable plan must be exercised, maintained and updated. The Disaster Recovery / Business Continuity response program is a continual process that matures over time as business needs, deliverables and department processes change. The central management resource is responsible for:  Periodic Disaster Recovery / Business Continuity Response drills per critical function to make sure critical area processes are updated, documented and working.  Periodic audits and updates of critical function Disaster Recovery / Business Continuity Response plans to reflect business need changes in the area’s critical objectives and processes.
  • 10. Trillium Software Disaster Recovery / Business Continuity Response Plan Confidential Page 9 2/10/2016 5 Contact Information Company Contact Phone Email Trillium Rick Percuoco – VP Cloud Operations 978-670- 4980 richard_percuoco@trilliumsoftware.com Trillium. Aaron Frenger – VP of Sofware Development 978-439- 3816 aaron_frenger@trilliumsoftware.com Trillium Fiona MacDonald – Primary Engineer Cloud Operations 978-436- 2903 fiona_macdonald@trilliumsoftware.com Trillium Jason Renaud- Customer Support Manager 978-436- 2854 jason_renaud@trilliumsoftware.com