Software Testing Center of
21351 Ridgetop Circle, Suite 400 ● Dulles, Virginia 20166 ● 703-404-9293 ● firstname.lastname@example.org ● www.cigital.com
Cigital > Software Testing Center of Excellence 2
Table of Contents
1 SOFTWARE TESTING CENTER OF EXCELLENCE (STCE) ......................3
2 STCE STRUCTURE ......................................................................................4
2.1 Core Functions ................................................................................................... 5
2.1.1 Testing Services .............................................................................................. 5
2.1.2 Best Practices and Processes........................................................................... 7
2.1.3 Knowledge Management ................................................................................ 8
2.1.4 Tools and Technologies .................................................................................. 9
2.2 Supporting Functions ....................................................................................... 9
2.2.1 Configuration Management ............................................................................ 9
2.2.2 Release Management .................................................................................... 11
2.2.3 Independent Test and Evaluation Facility (ITEF) ........................................ 12
3 SUMMARIZED SAMPLING OF STCE ACTIVITIES ...................................13
4 STCE BASIC STAFFING STRUCTURE .....................................................16
5 STCE COMPONENT INTERACTIONS .......................................................17
6 RISKS AND MITIGATION ...........................................................................17
Cigital > Software Testing Center of Excellence 3
1 Software Testing Center of Excellence (STCE)
A Software Testing Center of Excellence is a unified and balanced organization that
provides a full range of software quality and testing services to support the effective,
efficient and consistent delivery of quality software.
What is the problem?
Organizations developing significant amounts of software are often challenged with
providing quality assurance and testing of their software in an effective and efficient
manner. This issue is compounded by the increasing movement towards outsourcing
software development services.
What is the solution?
An independently staffed and/or managed Software Testing Center of Excellence
providing a unified and balanced combination of:
• Testing Services
• Best Practices and Processes
• Knowledge Management
• Tools & Technologies
• Configuration Management
• Release Management
• Independent Test and Evaluation Facility
What are the benefits of a Software Testing Center of Excellence?
A Software Testing Center of Excellence provides:
• Scalability – ability to scale software quality and testing services to changing
demand and still maintain adequate quality of service
• Objectivity – holds development accountable for the software they build by
providing independent assurance of software quality.
• Consistency – standardizing on testing best practice process and tools guarantees
that software testing activities will be performed in a consistent and repeatable
• Constant Improvement – on-going training, measurement/metrics, and process
improvement assure continued strides toward a best of breed software testing
• Better, cheaper, faster products – objective, full lifecycle software assurance
following standards and best practices reduces software development rework as
critical defects are found earlier in the process and corrected.
Cigital > Software Testing Center of Excellence 4
2 STCE Structure
An effective and scalable STCE will be organized along four mutually supportive core
functions along with three optional supporting functions shown below. Each of these
functions will be described in greater detail in the following sections.
Core Functions: Supporting Functions:
• Testing Services • Configuration Management
• Best Practices & Processes • Release Management
• Knowledge Management • Independent Test & Evaluation
• Tools & Technologies Facility
Cigital > Software Testing Center of Excellence 5
2.1 Core Functions
2.1.1 Testing Services
The Testing Services core function provides actual management, planning and execution
services for full lifecycle software quality and testing activities for software development
and maintenance. These services include but are not limited to:
1) Requirements analysis
2) Software test strategy and planning
3) Test automation
4) Test design, execution and results validation
5) Artifact review
6) Overall risk management
7) Security testing
An STCE Testing Service core function will provide independent software testing and
analysis to assure the successful delivery of high quality software and leverage defined
best practices and processes through the associated knowledge portal.
Benefits derived from an STCE Testing Services function include:
• Objective assessments of software quality
• Full lifecycle analysis to decrease rework activities during software development
• Cost-effective automation of testing
• Organization scalability to match capability with demand
• A continuously learning and improving testing staff supported by the other STCE
core and supporting functions
Full lifecycle testing services provided through the STCE will include both Systems
Acceptance Testing (SAT) and User Acceptance Testing (UAT) methodologies, as
briefly defined below, to support several types of testing:
1. SAT: The Test and Evaluation (T&E) Team uses the system functional and non-
functional requirements to determine if the product correctly performs to required
specifications and fulfills the business functions as needed by the user. Problems
identified during testing are documented as Test Problem Reports (TPRs). The TPRs
are passed to the Development Team, which assesses them and determines the level
of effort required to implement a fix. At the completion of SAT, project stakeholders
should be confident in their understanding of what level the software product either
satisfies or does not satisfy the specified requirements.
2. UAT: UAT relies on actual system users to perform testing, to ensure that it meets
their operational needs, before the system is released into the production
environment. T&E facilitates the UAT by preparing appropriate test procedures and
scenarios, assisting users during the UAT, and documenting TPRs identified during
the UAT. The TPRs are passed to the Development Team, which assesses them and
determines the level of effort required to implement a fix.
Cigital > Software Testing Center of Excellence 6
Testing types to be performed as part of the independent T&E function include but are
not limited to:
• Functional Testing - Verify that the system or application functions properly,
satisfies the requirements defined in the Functional Requirements Document, and
performs adequately in the host environment to ensure that potential system errors are
identified and addressed prior to deployment.
• Data Handling and Integrity Testing - Ensure that the integrity of the data is
maintained from all points of input for a system, through its handling and
manipulation, to its storage in the persistence layer and eventual presentation. Any
derivations done on the data are checked for correctness to validate that the data
remains reliable. These tests are executed by validating the form and content of data
from system inputs, validating the derivations done to the data, validating its
appropriate storage and validating the presentation of the data to the user. This type
of testing is important for systems dealing with mission-critical data.
• Systems Security Testing - Determine the overall assurance profile and security risk
of the system through testing of system security requirements, conducting
architectural risk analysis, testing of the system for the presence of known common
security weaknesses, penetration and red team testing, as well as evaluating
compliance of the operational system with organizational security and data integrity
guidelines, as well as federal security regulations. Part of system security requirement
testing involves validating how well a system meets predefined technical control
security requirements concerning unauthorized internal or external access or willful
damage. Security testing also establishes an application security baseline and
identifies a level of security risk prior to production implementation. Applications
are tested on standard secure platform configurations to ensure that normal operations
are not impeded by the security configurations themselves. Security testing also
includes vulnerability assessments using automated scanning tools, as well as testing
patches and security alerts and warnings for both applications and images.
Additionally, security testing may include disaster recovery and COOP planning and
• Reliability Testing – Verify if a system is capable of recuperating gracefully from
failure conditions. It tests to guarantee that a system can recover and continue
operating in the event of a major outage, web server, application server, database,
etc. These tests are executed by simulating these outages during normal system
processing. Reliability is important for systems dealing with mission-critical data
• Usability Testing – Verify the aesthetic and efficiency qualities of user interfaces and
examines their ease of use. The user interfaces are evaluated on how logical data
entry into the system is and how intuitive the presentation of data to the user is. This
type of testing is more of an art form than a science. The ultimate goal is to remove
any thing that might be confusing or ambiguous to the user.
Cigital > Software Testing Center of Excellence 7
• Integrated Performance and Stress Testing - Ensure the product delivered to the
field performs with the desired response times, and as expected under projected user
load using the existing infrastructures. Determine the load at which the application
and/or hardware can no longer meet acceptable processing metrics as defined during
the requirements and design process.
Interface and Interoperability Testing - Assess the compatibility and potential
impact of multiple, cooperatively employed systems through the validation of their
operation and conformity to approved standards; validate standard images before they
are released into the field.
• Interoperability testing validates that applications or COTS products
installed in combination on production platform(s) operate correctly or
work with the approved Production baseline. It also validates that the new
version of a platform baseline image permits the functioning of Production
applications and systems.
• Interface testing verifies communication and interaction between systems
by ensuring that the system’s interface design requirements are satisfied.
Interface testing addresses calls made to other modules, communication
interfaces between modules, and the integration of COTS software and
• Regression Testing - Ensure that program changes have not degraded the overall
functionality of the system.
• Infrastructure Testing - Ensure that new and proposed infrastructure components
such as servers, workstations, peripherals, operating systems, and office productivity
software are compatible with the current systems and applications.
• Image Testing - Ensure that developed workstation and server images are compatible
with the current systems and applications
• Installation Testing - Analyze the impact of the installation of new systems or
components on deployed systems and baselines ensuring the installation meets the
project’s requirements and does not negatively impact production.
2.1.2 Best Practices and Processes
The Best Practices and Processes core function will identify, define, deploy, track and
improve standard quality assurance and testing processes for the STCE based on industry
best practices and lessons learned internal to the STCE.
Key processes and practices to be defined by the Best Practice and Processes core
function include areas not typically covered by most QA and testing organizations
including but not limited to:
• Test strategy and planning
• Test automation
• Software metrics
Cigital > Software Testing Center of Excellence 8
• Requirements review and traceability
• Security testing and analysis
• Risk management
These will be integrated with CMMI and ISO 9001 quality assurance and management
practices as appropriate.
Activities will be used to assess current testing process and practices to identify existing
best practices and gaps and to develop comprehensive improvement roadmaps that
incrementally improve software testing processes while not impacting release schedules.
In addition to deployment through the software testing portal, there will be
documentation of these best practices-based software testing processes and
methodologies. Measurements will be measured to determine the impact of process
change and will be incrementally improved.
The STCE will be a driving force for improvement in the organization, but will amend
existing processes incrementally to reduce culture shock.
2.1.3 Knowledge Management
The Knowledge Management core function will provide three primary services: software
quality assurance and testing knowledge capture and transfer, including training; process,
knowledge and collaboration deployment through a software testing portal; and
management of a certified components repository to house previously vetted components
for strategic reuse.
The overall purpose is to provide knowledge to software testing and development teams
that will increase their productivity and capabilities and drive on-going software test
training (both classroom and online) to improve individual skill sets.
Ongoing activities for the STCE Knowledge Management function will include:
1) Development, maintenance and population of a knowledge portal that provides
information to both software developers and testers on testing best practices, plan
and report templates, technology quality guidelines
2) Incorporate component repository of tested components to drive reuse and
3) Maintain and deliver software quality assurance and test training curriculum.
Targeted content for management should include, but not be limited to, the following:
• Knowledge Management online repository/ “Portal” to house STCE measurements,
costs, testing reports, daily summaries, CM status accounting, status of document
assessments - and anything else deemed as important information to share across the
organization. The SCTE Portal is role-based and gives instant insight into STCE
activities and documentation including the SLM process and allows for users to
quickly search for relevant topics of interest. The portal serves as the repository for
the latest changes to STCE methodologies and procedures and houses the latest
Cigital > Software Testing Center of Excellence 9
• Periodic meetings with development teams, IT Project Managers, IT Operations, and
the STCE to discuss project schedules, technology advancements, process
improvement, and anticipated changes to the environment, ensuring all parties are
aware of pending issues that can be resolved before they become problems.
• Yearly self-evaluations that result in documented “Lessons Learned,” ensuring the
STCE is constantly maturing and improving.
• Online and in-person training of all STCE processes, including the systems
development lifecycle process.
• Newsletters developed to foster awareness of systems development methodology and
standards, to highlight areas of frequent questions, and to communicate with system
development projects in easily accessible targeted briefs
• Enterprise Systems Assurance Plan (ESAP) serves as a “how-to” guide for
implementing the STCE. The ESAP provides instructions for carrying out specific
CM, Release Management (RM), and T&E activities, and delineates the
responsibilities of these activities for STCE Teams and project teams.
• Status reports highlight all STCE activities accomplished each week.
• Service Level offerings to enable system owners and project managers to assist in
determining depth and breadth of testing appropriate for a specific release given the
project context, mandates, and willingness to accept risk.
• Daily reporting as required on independent testing and evaluation progress.
2.1.4 Tools and Technologies
The purpose of STCE Tools and Technologies branch is to examine and select
appropriate tools and testing techniques for use within the testing services core function.
STCE Tools and Technologies function activities provide tool evaluation, testing
techniques evaluation, and documentation of tool and technique best practices.
The benefits of the STCE Tools and Technologies function include but are not limited to:
1) Assures that effective software tools are selected for use by both software
development and testing based upon business/mission criteria
2) Documented information provides developers and testers useful knowledge for
effectively using appropriate tools
2.2 Supporting Functions
2.2.1 Configuration Management
The Configuration Management (CM) supporting function includes planning, defining,
and providing the change management environment to ensure the delivery of quality
systems. Change management identifies and tracks changes to system components
through administration tools such as version control software, system change request
software, and a central repository for system documentation. As requirements change,
system change requests (SCRs) will be tracked through the Governance process based on
automated tools, and discussed and reviewed during CCBs and other Governance
reviews. Upon approval by the CCB, the requirement, data, code, or architecture (CIs)
Cigital > Software Testing Center of Excellence 10
are updated in its respective repository to ensure integrity and tracking for all changes to
the established baselines. CM ensures software and systems release integrity as a release
or infrastructure modification moves through the development, test, and production
environments and ensures that software and documentation assets are well protected.
Configuration Management requirements to be supported include the following:
• Configuration Management Planning
• Developing an overall CM plan documenting CM process and procedures to be
implemented at specific phases and milestones of the system development
• Monitoring and controlling the configuration management process by initiating,
controlling, tracking, and auditing changes, deviations, and waivers.
• Communicating with and training those performing or supporting the
configuration management process on as needed basis.
• Supporting the operation of the Enterprise Change Control Board (CCB).
• Configuration Identification
• Providing configuration identification and documentation for software, hardware,
and other configurable items (CIs) within the organizational IT environment.
• Configuration Change Control
• Implementing CM policy and change controls to meet system security
certification and accreditation requirements
• Tracking and controlling changes to software and hardware configurations.
• Establishing and maintaining an automated CM and change management system
for controlling work products.
o Analyze and evaluate CM software tools - researching products
and technical specifications.
o Providing and administering enterprise-wide use of version control
software to ensure organizational investments and software assets
are consistently maintained.
• Maintaining and operating an enterprise centralized repository and enterprise
library of documents (including processes, procedures, workflow, etc.), software,
and infrastructure. This leverages the software test portal managed and deployed
by the Knowledge Management core function.
• Ensuring that version control software and the document repository are
compatible with legacy CM tools such that data can be migrated.
• Working with each software development project to ensure configuration
management activities are properly incorporated into project plans.
• Making baselined application code available to staging teams.
• Managing all baselines including but not limited to applications, workstations,
servers, and images.
o Identifying and documenting baseline contents
o Tracking baseline changes.
Cigital > Software Testing Center of Excellence 11
• Configuration Status Accounting
o Certifying baselines for internal use and for delivery to the field.
o Producing a global configuration status accounting report.
o Reviewing the activities, status, and results of the configuration
management process with organizational management.
• CM Audits
o Participating in the development of audit trails to specify what changes,
i.e., executables, configuration files, documents were deployed and where.
o Conducting configuration audits and reviews to maintain integrity of the
configuration baselines and the enterprise central repository and library of
documents and software.
2.2.2 Release Management
The Release Management supporting function provides services to assure quality in the
deployment of new system releases. Applications and COTS Software Staging and Pre-
Deployment Services ensure that planned single or bundled releases of software and/or
hardware changes, COTS upgrades, configuration changes, patches, and images are
available for deployment to field locations. The Release Management function will stage
releases using various platforms including FTP servers, web servers, CDs, and automated
tools. The organizational IT Operations Division will physically deploy the software to
field sites. The intent is to deploy quality software to production facilities in a seamless
and transparent manner.
Applications and COTS Software Staging and Pre-Deployment Services requirements
include the following:
• Developing staging/ release processes, methodologies, and plans.
• Creating installation packages for system releases using industry best practices and
• Ensuring the release package can be successfully installed by IT field personnel who
do not have an intimate knowledge of either the system or deployment process.
• Notifying the Help Desk that a new system release is imminent so that a message may
• Staging application updates using industry best practices and tools/ methods (e.g.,
automated tools, CD-ROM, FTP Site and Websites).
• Participating in developing a standardized methodology to work with system business
owners, IT project managers, users if necessary, and others, as required to
communicate staging and deployment milestones, business impacts, roles/
responsibilities, problem resolution, other special needs for a specific IT component.
• Creating installation CD-ROMs and send to the designated point of contact on the
distribution list supplied by the Development Team, IT Project Managers, etc.
• Developing fallback procedures and checkpoints (milestones for staging).
• Developing a standardized, post staging review process.
Cigital > Software Testing Center of Excellence 12
2.2.3 Independent Test and Evaluation Facility (ITEF)
The Independent Test and Evaluation Facility supporting function will provide an
independent testing environment to support the testing activities of the Testing Services
core function. The ITEF will have the appropriate size, equipment, security, and
functionality to support the STCE and envisioned enterprise testing support. It will be
able to reproduce organizational operating environments and technologies. It will be
responsible for creating an environment which mirrors the workstation / printer /
peripheral environment of the field locations as closely as possible. It will be scalable.
The ITEF will include all hardware and software for performing testing, CM, release
management, reporting, and other STCE functions that is not GFE.
Where the sponsoring organization chooses to utilize a third-party ITEF to support the
STCE, the STCE will develop a physical and logical configuration plan and management
process to be implemented by the third-party ITEF. This plan will address the following
• Location (e.g., in proximity to the organizational office location)
• Layout/ partitions
• Infrastructure/ tools
• Access Control/ Security
• COOP and Disaster Recovery
Cigital > Software Testing Center of Excellence 13
3 Summarized Sampling of STCE Activities
A summarized sampling of some typical detailed STCE activities include but are not
• Test Planning and Management Oversight
Establishing and maintaining enterprise-wide testing standards and procedures in
accordance with the system development life cycle, and ensuring that development
testers follow established standards. (Best Practices and Processes function)
Serving as a testing point of contact for organization generated test questions or
issues. (Test Services function)
Attending control board meetings, review board meetings, and other system
lifecycle process required reviews and addressing any T&E issues. (STCE
Management and Test Services function)
Using requirements documents in preparing a detailed specification describing the
physical test lab solution for each application and release being tested. (Individual
Test and Evaluation Facility function)
Providing test engineering guidance for T&E personnel. (Best Practices and
Developing/ delivering documentation such as test plans, which clearly define the
goals, requirements, testing needs, entry/exit criteria, test data needs, and detailed
test cases or test scripts for automated testing. (Testing Services function)
Defining and establishing—with CM, QA, and Release Management team
standard processes for the management of automated test scripts. (Best Practices
and Processes function)
Providing all deliverables, as required, in a well documented, timely manner.
(Testing Services function)
Conducting functional testing to assess whether the final software product meets
the approved requirements and design specifications. The Systems T&E Team
uses the functional requirements as described in the current Functional
Requirements Document (FRD) to determine if the product performs the business
functions as needed by the user. Functional testing includes both manual and
automated processes. (Testing Services function)
Conducting end-to-end integrated performance testing that measures response
time, and throughput using load, stress and WAN-emulation tests. (Testing
Executing performance diagnostics tuning to identify and resolve the root cause of
performance bottlenecks. (Testing Services function)
Conducting Interoperability Testing to assess the compatibility and potential
impact of new or updated systems on existing systems. (Testing Services
Developing a methodology for conducting independent Security Testing
Evaluation, and conducting security testing that will aid the Security team in
Cigital > Software Testing Center of Excellence 14
making a decision(s) about an application’s certification to be deployed securely.
(Best Practices and Processes function)
Performing risk-based testing that focuses test efforts on critical areas when there
is the latitude to choose the areas for test focus based on functional criticality,
complexity, and risk. (Testing Services function)
Developing and maintaining a suite of automated regression testing tools. (Tools
and Technologies function and Testing Services function)
Electronically capturing and executing automated test scripts using approved
software products. (Testing Services function)
Facilitating User Acceptance Testing by creating results based testing scenarios.
(Testing Services function)
Recording and tracking TPRs during SAT and UAT. (Testing Services function)
• Support Systems Development Lifecycle Processes
Providing guidance to organization Systems Assurance management on the
benefits and function of test support through the system development lifecycle.
(STCE Management and Best Practices and Processes function)
Participating in control board meetings and review board meetings, and
conducting Test Readiness Reviews (TRR) and Release Readiness Reviews (RRR)
with organization System Owners, IT Project Managers, IT Operations and
Systems Assurance. (STCE Management and Testing Services function)
Providing technical assessments and risk assessments of Test Problem Reports
(TPRs) during the RRR to assist organization in determining if planned changes
are certified for deployment into production. (Testing Services function)
Integrating the test process with the configuration management process such that
when problems are discovered during testing, they can be reproduced, diagnosed,
and fixed against the version of the application code that failed and on the same
platform. (Best Practices and Processes function and Configuration
Evaluating Version Description Documents and Maintenance Release Notes
provided by the development teams, IT Infrastructure or Security to validate the
information and ensure target software can be installed in an independent test
facility. (Release Management function, Independent Test and Evaluation
Facility function, Testing Services function and Configuration Management
Evaluating application software and making recommendations for improvement.
(Testing Services function)
Evaluating office automation and support tools, OS, COTS, hardware, security
template, configuration settings. (Tools and Technologies function and
Knowledge Management function)
Resolving questions and issues that arise between application development teams
and assigned testers. Generating test-related routines and special reports as
required. (Testing Services function and Best Practices and Processes function)
Assessing readiness of software and hardware for delivery to the Government
upon completion of testing. (Testing Services function)
Cigital > Software Testing Center of Excellence 15
Support Disaster Recovery and COOP planning/ testing as required. (Testing
• Systems Testing Administration
Providing systems administration for the independent T&E lab equipment, so as to
duplicate the organizational environment for accurate T&E. (Independent Test
and Evaluation Facility function)
Providing DBA Administration for the Systems Assurance ITEF, which may
include: setup of independent test areas for all applications undergoing T&E,
managing database configuration control, setup of performance test databases,
performing analysis of performance testing on the database, ensuring database
configuration meets security requirements, manipulation of tables for creating
different testing scenarios. (Independent Test and Evaluation Facility function)
Providing automated test tool administration to include the following:
management of procurement and implementation, license management,
maintenance and upgrades, technical support, scheduling of formal training, and
serving as a general point of contact for all automated testing tool related inquiries.
(Tools and Technologies function and Knowledge Management function)
• Tracking and Analysis/ Reporting
Ensuring that individual tests provide testing schedule estimates, test plans, daily
test summary reports upon request, and test analysis reports to the organization
Systems Assurance Manager, and in general, developing a standardized,
comprehensive reporting process. (Best Practices and Processes function and
Testing Services function)
Identifying, recording, and tracking software and hardware defects. (Testing
Developing Test Analysis Reports (TARs) to comprehensively show analysis and
the results from all levels of testing. TARs detail items such as the testing type
and methods used, test case results, test environment utilized, test problems
encountered (open and/or resolved), and test metrics. (Testing Services function)
Cigital > Software Testing Center of Excellence 16
4 STCE Basic Staffing Structure
The following staffing structure would form the baseline model for a balanced and
scalable approach to staffing an STCE. This structure provides the key roles required to
fully execute the STCE services defined above and can be easily scaled to support
anywhere from a very small number of projects to a very large number of projects. As
demand increases, the staffing of the Test Services function under the Test Manager will
scale at a much faster rate than the rest of the core and supporting functions.
Cigital > Software Testing Center of Excellence 17
5 STCE Component Interactions
The STCE’s four core functions (Testing Services, Best Practices and Processes,
Knowledge Management, and Tools and Technologies) are constantly interacting in a
synergistic, self-reinforcing mechanism, as shown in the figure below. Mission-oriented
metrics will be developed and refined to ensure their relevancy to the organization. This
constant interaction and mutual dependence is what lends strength and flexibility to this
solution. It ensures that the STCE remains current and relevant, that it remains flexible
and scalable and that it is continually evolving to achieve higher levels of effectiveness
and efficiency while adapting to the needs of the sponsoring organization.
Best Practices & Pro
Process definition re sult g
Management Portal access Services
Tool knowledge edb g
l fe orin
Too m ent
Tools & T ool
6 Risks and Mitigation
There are some implicit risks in effectively implementing an STCE in an organization. It
is always wise to identify such risks and establish planned actions to mitigate them as
early as possible. The following are some of the more common risks with establishing an
• Risk #1
Description: STCE gravitating to a “body shop” approach to software testing over
Mitigation: All four core components of the STCE must be implemented day one
and a strong measurement program must be put in place.
• Risk # 2
Description: Best practices and processes are documented but seldom followed.
Cigital > Software Testing Center of Excellence 18
1) Appropriate training must be developed and delivered
2) Best practice and processes must include enabling technologies
3) Performance measurement must include productivity measures
• Risk #3
Description: Software development views software testing as an adversary instead
of a partner.
Mitigation: Implement and clearly communicate the value of a full lifecycle
software testing processes that will help identify key defects earlier in the process
and save software development time and effort.