An assessment of a software testing organization can identify strengths, weaknesses, and improvement opportunities by comparing it to industry benchmarks. It provides a roadmap for reducing costs and improving quality. SDT offers assessments to evaluate metrics, methods, tools, practices, and environment used for testing. An assessment identifies high priority areas for enhancing effectiveness and efficiency based on best practices.

1. WHITE PAPER
Organization Assessments
For Software Testing
An assessment provides a vision and roadmap for future
success in testing.
A Test Organization Assessment can be a powerful tool to determine a software test
organization’s real capabilities, strengths, weaknesses, advantages, and disadvantages.
It puts into perspective how the test organization compares to other industry test
groups. It identifies the highest priority areas for improvement and provides guidance
on how to make those improvements.

2. WHITE PAPER: ORGANIZATION ASSESSMENTS FOR SOFTWARE TESTING
2
Software Development Considerations for an
Technologies (SDT) provides
software test services to product Assessment
development and IT partners. SDT Software testing is critically important to
creates new test groups or extends software development organizations and is
becoming increasingly difficult and more
the software testing capability of
expensive as the complexity and pervasiveness
our clients. SDT can design, of systems grow. As important as testing is,
implement and execute Function, companies should focus on testing in the most
System, and Performance tests to efficient, cost effective way. Increasingly, the
augment a partner's test capability. costs of testing are going up. As the complexity
Alternatively, SDT can assess a of software increases, the number of errors
software testing organization or grows and customers’ tolerance for problems
declines. A 2002 study by National Institute of
assess and then transform a Standards and Technology (NIST) 1 concluded:
software testing organization, by
introducing an automation • 80% of the total cost of software
framework, a wide range of development is spent to fix bugs
• total loss in the USA due to inadequate
software test tools, and a complete testing to both software developers and
set of test processes. users: $59.5B
• total cost of errors to software developers:
Tim Townsend, Senior Director at $21.2B
Sun Information Resources, Sun • potential savings with better testing: $10.6 B
Microsystems, commented about • test tools today are primitive and evolving
SDT assessment and • over half of software bugs are caught
transformation consulting. downstream
An assessment is often the first step of a major
Tim said, “SDT came in and quickly
improvement program. We should not take
assessed our existing development organizational change lightly as people are
and test methods and architected a directly affected (on the one end, they worry
new, large-scale test process. They about their jobs and on the other extreme they
also designed and rolled out a new are hoping for change). There is an expectation
technical reviews and inspections that something will be done when an assessment
program, resulting in greatly is completed. The assessment is the first phase
for planning the changes. The change program
improved communications and may be long and painful – but it may also be
earlier defect detection. These very successful, especially with the right
processes greatly fortified SunWay, planning.
the foundation life-cycle
methodology for the construction An assessment provides a baseline measurement
and engineering management of and description of the test organization, which
can be compared to industry benchmark
complex Sun programs. These
measures. This is the best way to identify gaps in
processes orchestrated the performance in comparison with typical and best
program’s alignment and in class operations. Often an assessment is
integration activities across requested because management believes that
production releases giving us a testing costs too much or provides too little
quality foundation from which to value. Software quality may be unknown or
manage successfully.”
1
NIST Report 02-3; costs derived from Table
ES-4, p. ES11

3. WHITE PAPER: ORGANIZATION ASSESSMENTS FOR SOFTWARE TESTING
3
Learn More declining. Although it may be clear to
management that something is wrong, it is
seldom clear what is wrong, or what to do to
Software Testing in the Real World improve. An assessment should provide specific
is an informational guide that is answers to managers’ questions.
essentially a toolkit for continuous,
sustainable improvement of the A software test assessment is an unbiased
software testing process. Written evaluation of the metrics, methods, tools,
practices, organization and environment used to
by Edward Kit, CEO of Software
produce software, with a focus on the testing of
Development Technologies, the software. The objective is to identify the
Software Testing in the Real World highest priority areas for improvement and
addresses the most frequently provide guidance on how to make those
asked questions about improvements based on first hand experience
methodologies, tools, technology, with best practices at similar organizations. The
and organizational issues being scope and formality of an assessment depends on
the organization size and needs.
posed in the testing community
today. The assessment process is not an audit, even
though assessments and audits share many
To request a discussion with SDT similarities. An audit compares an organization
and gain insights into how SDT can to a specific, established, theoretical model. It
assess your software testing assumes that every organization should be
operating exactly as the model describes. An
organization, please call James
assessment describes the organization as it is and
Wright, SDT’s General Manager, compares it with other similar successful
Test Service Operations, at 405- organizations to identify areas and methods for
232-6000 x11. To find out more improvement. It allows for differences within
about Software Development and between organizations as long as the
Technologies, visit operation is effective and efficient. The
www.sdtcorp.com. assessment focuses on supporting strengths and
improving areas of weakness.
Test Assessment Benefits
There are many potential benefits, which can be
derived from an assessment:
• Provide a roadmap for improvements
• Reduce and control testing costs
• Provide the potential for predictability
• Increase organizational productivity
• Improve the quality of work products
• Learn how the organization compares with
other industry test organizations
• Clarify areas of confusion and weakness
• Wake up the troops (to avoid people
becoming complacent)
• Improve the use of test technology / tooling
• Apply different kinds and levels of testing
expertise through a release cycle and
between release cycles
• Improve effectiveness, efficiency, and
reduce stress for testing organizations

4. WHITE PAPER: ORGANIZATION ASSESSMENTS FOR SOFTWARE TESTING
4
Considerations for Selecting test industry benchmark knowledge, and
organization change management skills
an Assessor Outside consultants do not have these same
The test organization being assessed is in issues. They should have current test industry
partnership with the Assessment Team. The knowledge. What they lack, though, is
relationship should be mutually supportive and knowledge about the specific organization
based on mutual respect because the team must (which is both an advantage and disadvantage).
become intimately familiar with the operations to Other benefits available from using an outside
generate meaningful and useful results. Thee assessor:
team’s purpose is to understand the strengths and
• Obtain a neutral opinion and evaluation
weaknesses in order to point the way to
improving the organization. • Leverage experience from industry experts
• Avoid the “Groupthink” issues of insiders
An Assessment Team needs broad experience • Benchmark against industry and
with industry test practices in a variety of marketplace best practices
industries to discern practices and compare with • Gain access to world-class capabilities
industry benchmarks. The team also needs
experience at all levels of management and all The important factors to consider when choosing
aspects of testing to identify a set of an assessment organization are:
development and testing methodologies,
including automation, to be implemented to • Knowledge and experience performing
transition testing from a manual, back-end-only assessments in multiple organizations,
task to a lifecycle-wide approach. This will industries, technologies and with industry
enable the organization to identify defects at best practices
every stage (i.e. have a “defect plan” for each • World-class capabilities
phase) of the life cycle integrated with the • Specific experience with similar companies /
software development lifecycle. The Assessment applications
Team also needs practical experience with • Availability of domain expertise
organizational transformations in order to make • Existing customizable process definitions
reasonable recommendations. Few test and templates
organizations have in-house expertise with the • Courseware corresponding to processes
breadth and depth of experience required to • Ability to perform client selected
assess the state of testing and quality. Companies improvements
that specialize in software testing and assessment
vary greatly in their knowledge, quality of
assessments, and ability to follow up. Test Assessment
Many organizations attempt to do their own Deliverables
assessments with varying levels of success. An assessment should provide actionable
Several hurdles must be overcome for internal information in a form that can be understood and
assessments to work: discussed and should include:
• It is very difficult to observe what we are • An accurate description of the current state
doing wrong ourselves (if we know a better of testing and quality by observing actual
way we use it) behaviors and artifacts and listening to what
• People inside the organization have existing people say
relationships and agendas that may get in the • A list of the current strengths and areas for
way of objective analysis improvement within the software
• People in the test group may not have development and testing processes
experience working in other test • A concrete set of recommendations for
organizations improvement, both immediate and longer
• People inside the organization seldom have term, including recommendations on people
experience with doing assessments, current and process.

5. WHITE PAPER: ORGANIZATION ASSESSMENTS FOR SOFTWARE TESTING
5
• A “roadmap” for implementing the
recommended methodologies and best
Focus Areas for the
practices, along with reference material of Assessment
where to get more information about a given There are seven primary focus areas in a test
methodology and required training for the assessment. These factors represent the major
recommended approach. Where more than organizational components critical in
one methodology exists, recommendations determining the current baseline state of a test
of the “best fit” for the company with organization. These same areas are used for gap
guidance on conducting pilots to evaluate analysis and later to categorize improvements.
one approach relative to another. The areas SDT focuses on are:
This paper describes a method for getting a clear
• Organization
picture of a test organization’s current practices,
• People
strengths, weaknesses, advantages, and
disadvantages in comparison with industry • Process
benchmarks, standards, tools, and techniques. It • Technology
also identifies some best practice and evaluation • Environment
criteria applicable for test organizations. • Outsourcing
• Integration
Testing Best Practices Organization
The organization describes the work team’s roles
• Understand company goals and objectives and relationships. It defines the hierarchy and
• Understand strategic vision and plan relationships required to produce and test the
• Select the right practices software. Important elements of the organization
• Get a baseline measure of the include:
organization and compare to industry • Structure
benchmarks • Roles and responsibilities
• Involve testing early in the life cycle – • Location(s)
starting with requirements reviews and • Culture
continuing throughout the life cycle. • Communication styles es
• Charter an internal professional test
management group to oversee the tools, People
techniques, workflow, and metrics People are the means by which software is
• Document and enforce entry and exit created and tested. Unqualified, untrained, or
criteria unsupported people are inefficient and
• Measure and monitor work regularly ineffective. They must understand what is
expected from them and what they need to be
• Document and maintain requirements
successful. To do their jobs, people within the
traceability (a mapping from tests to
organization must:
requirements)
• Be technically qualified
• Generate senior executive support and
involvement • Be trained to do the job
• Use ROI metrics to justify • Have the proper tools
expenditures/practices • Share an understanding of the inputs they
• Use outside expertise, as needed require and the work products they create
• Plan for continuing changes Process
• Document internal process for specifying The processes of the organization are the means
work and measuring progress by which work is done. The test processes are
• Apply balanced scorecard for testing central to the effective and efficient functioning
metrics of the test organization, but the entire
• Closely manage and control production- development life cycle is involved in creating the
like test environments quality of the deliverable software. The
assessment must cover the test process and
whichever other processes have direct positive or

6. WHITE PAPER: ORGANIZATION ASSESSMENTS FOR SOFTWARE TESTING
6
negative impact on the work done in the test around the clock operations, and more. This is
group or on the software quality itself. The also an area littered with failed projects and
existing processes must be understood disappointments from both outsourcing across
sufficiently to make valid comparisons with the street and overseas.
applicable industry best processes. Process
questions include: There are a host of opportunities and issues
surrounding outsourcing testing. Potential
• What are the process tasks and timetable(s)
problems include:
• Is adequate time given for the tasks
• Cultural misunderstandings
• How uniform and stable are the support
• Lack of internal processes for specifying
processes (e.g., defect management, source
work
code control, requirements management,
etc.) • Lack of internal project management skills
• Losing control of the resource
Technology • Losing critical internal expertise
This reflects the choice of tools and their • Problems communicating with cross-
utilization, efficiency, and effectiveness. The functional groups
types of questions addressed include:
• What tools are used for development and Integration
testing It is critical to know how well the areas play
• Are they the right tools for the jobs together. It is not enough to have the right
• Are they the best tools for the specific people, tools, and processes if they do not work
requirements together effectively. Some questions to answer
are:
• Are they used effectively
• Where are the disconnects?
Environment • Are there disagreements about what needs to
Software development and test is done in a
be delivered -- by/to whom, or when?
technical environment, which includes hardware,
• Are people aware of their roles and
software, networks, and communication
interdependencies?
mechanisms. Controlled and uniform Test
Environments are required for consistent test
results and to replicate and fix the problems SDT Test Assessment
found. High levels of environmental control and
monitoring are helpful for everyone working in Process
the Test Environments since the test results A Test Assessment can take a few days to
depend upon the regulation of outside factors, several months, depending upon the size,
which could corrupt or invalidate the test team’s diversity, and dispersion of the test organization.
work. Factors to investigate in this are include: The overall assessment process remains basically
• Adequacy of test hardware, software, and the same, however.
other systems
• Test systems and components that can be Initially, the parties agree on the purpose and
shared among projects on a scheduled and timing of the assessment. The organization being
predictable basis assessed, desired outcomes, and specific
• Test Environment infrastructure deliverables are identified and the scope of the
• Test Environment procedures assessment is determined. Since many functions
within an organization directly affect the product
Outsourcing quality and the testing of software, the test
The state of software development today is in assessment must include representatives from
flux. Global competition in information these groups, as well. The Assessment team must
technology has caused significant changes in have the latitude, political backing, and expertise
how and where organizations get work done. necessary to follow threads far beyond the
Virtual organizations are possible, with teams borders of the Test Organization to assess
consisting of members who never meet in important factors influencing quality and thus
person. Outsourcing is a method for staff provide recommendations that apply.
augmentation, gaining experts for specific tasks,

7. WHITE PAPER: ORGANIZATION ASSESSMENTS FOR SOFTWARE TESTING
7
The assessment process progresses through three • Test organization, charter, schedule.
major phases, as outlined below: • Results of any prior assessments or
1. Pre-Planning: The client and the organizational reviews.
Assessment Team Lead meet to agree upon • Software development and test
the scope of the assessment, identify the deliverables for the projects, such as
major participants, and plan the schedule. Test Plans, example tests, presentations
The Assessment Team gathers and digests at project reviews, etc.
background information and prepares for the • Provide a detailed schedule of assessment
on-site meetings. activities and milestones.
• Plan site visits
2. Conducting the Assessment: The • Generate agenda and schedule for Kickoff
organization’s activities and artifacts are Meeting
observed and evaluated. An on-site
assessment includes interviewing and The assessment project planning process
observing designated personnel from provides the content for Kickoff Meetings
Software Engineering groups (i.e. presented at each site to familiarize participants
Development and Test), Senior with the purpose and methods employed in the
Management, and other functions that assessment. A typical Kickoff Meeting agenda
interface with Software Engineering at might include the following items:
multiple sites.
• Introduce SDT and the Assessment Team
3. Analysis and Recommendations:
• Describe Assessment Purpose
Observations are compared to industry
• Assessment Phases
norms with relative strengths and areas for
improvement identified and specific • Pre-Planning
recommendations/actions made. A • Conducting the Assessment
management presentation documents the • Analysis and Recommendations
recommendations for organization, people, • Assessment Schedule
process, technology, environment,
outsourcing, and integration. Phase 2: Conduct the
Software Test Assessment
Phase 1: Plan the The Assessment Team customizes its proprietary
Assessment Project process capability assessment for each client.
The Assessment Team works with designated The Assessment Team will interview and
Senior Management to define the project scope observe people within the development, test
in the following areas: planning, and test execution teams to understand
• Internal functions to be included / excluded existing people, organizational structure,
and their locations. processes, and the issues/goals and objectives in
• External entities to be included (partners, improving the existing environment. The
customers) Assessment Team may also interview critical
• Assessment focus (such as which projects to customers or partners to learn their expectations,
examine) integration issues, and processes. From these
meetings, the Assessment Team will develop an
Preliminary information is gathered by having overview of:
selected personnel complete a short Pre- • The big picture testing process at the client
Assessment Questionnaire. The steps in the • How the test process fits into the software
Assessment Planning Process are: development process
• Testing architecture, test implementations,
• Complete the Pre-Planning Questionnaire and test practices
• Gather process/project documentation and • Onsite / offshore outsource model
provide SDT with access. Documentation to • Approach to inspections and reviews
include: • Approach to defect metrics
• Company development/test lifecycle.

8. WHITE PAPER: ORGANIZATION ASSESSMENTS FOR SOFTWARE TESTING
8
• Approach to staffing and organization, skill The Evaluation produces the baseline description
sets, and job roles of the client organization. The Initial
• Different levels of testing employed Presentation gives the executives a preview of
• Use of tools for testing the findings and recommendations.
Documentation of the analysis and
recommendations is generated, and the
Phase 3: Analysis and Presentation/ Follow Up disseminates relevant
information from the assessment and prepares
Recommendations the client organization for implementation of the
The Assessment Team analyzes the existing recommended improvements.
approach to software testing and identifies
missing and sub-optimal components based on Evaluation and Baseline
applicable industry standards and practices. The
Assessment Team analyzes the input received Description
and examines: Based on the information gathered in the
Evaluation and Initial Presentation steps, the
• How the approach could be improved to Assessment Team describes the baseline
better meet the rapid development and organizational behaviors and uses that as the
deployment needs at the client basis for evaluation. The baseline is compared to
• How the existing processes could be made industry practices and gaps identified between
more effective within the constraints of the the organization and the most effective software
client’s rapidly changing business development and testing.
environment
• What the client’s software development/test
needs are Initial Executive Presentation
From the baseline description and evaluation, the
The Assessment Team composes the following: Assessment Team provides preliminary feedback
• A recommended set of critical and recommendations to executive management
improvements for testing on changes and improvements to its
• Actions that can immediately be architecture/strategy in the software development
implemented, as well as long term actions and testing process areas. The 2 - 4 hour
within the client organization to improve management meeting outlines the observations
software/systems development and testing and conclusions while providing for discussion
• A specific recommended roadmap for of opportunities for improvement. This initial
improvement of QA/Test inclusive of test presentation serves as a sanity check for the
environment, test methods, and best findings and recommendations and informs
practices across the defined product executive management of the relative strengths
development cycle and weaknesses identified.
• An overall training plan for the organization
and assistance with training vendor Documentation of Analysis and
identification for courses outside of SDT’s Recommendations
expertise
Incorporating the feedback from the Initial
• The definition of what document templates
Executive Presentation, specific written
should be put in place
documentation is provided on people, process
• The definition of an overall testing and organizational recommendations; the
architecture including an automation QA/Test improvement roadmap; training
framework recommendations; and references to
recommended methodologies.
Phase 3 is further broken down into four steps:
1. Evaluation Presentation and Follow Up
2. Initial Presentation
Based on feedback from the Initial Executive
3. Documentation
Presentation, the Assessment Team provides
4. Presentation/Follow Up

9. WHITE PAPER: ORGANIZATION ASSESSMENTS FOR SOFTWARE TESTING
9
further information and recommendations to Environment, Outsourcing, and the Integration
designated client managers on specific areas of of the factors.
interest. The Assessment Team may discuss
alternatives for achieving the recommendations When done well, the assessment provides a
and refinements of the roadmap elements. The vision and roadmap for future success in testing.
Assessment Team may also discuss refinements It should reflect the reality of the existing
of the schedule for improving employee skill organization and a direction that clearly leads to
levels and for improving the employees’ improvement.
understanding and use of metrics, processes, and
best practices.
Often, a formal presentation to the management
team is made by SDT to communicate, clarify,
and address any questions about the observations
and recommendations. A view of the
requirements to achieve stated goals may be
included, detailing required training and
automation and giving an outline of the overall
requirements, investment estimates, and benefits
of implementing the improvements. An agenda
for a typical assessment presentation might
include:
• Introductions
• SDT Assessment Purpose and Approach
• Keys to Software Testing Success
• Observations and Challenges
• Recommendations
• Roadmap to Success
• Summary and Next Steps
Summary and
Conclusions
A Test Assessment can be a powerful tool to
determine a software test organization’s real
capabilities, strengths, weaknesses, advantages,
and disadvantages. It puts into perspective how
the test organization compares to other industry
test groups. It can identify the highest priority
areas for improvement and provide guidance on
how to make those improvements.
Since many parts of an organization can directly
affect product quality and the testing of software,
the test assessment should include participants
from many organizations besides the test and
development groups.
The assessment process progresses through three
phases: Planning, Conducting the Assessment,
and Analysis and Recommendations. Many
factors are considered during the assessment:
Organization, People, Process, Technology,