Getting Started with Risk-Based Testing

MC
Full Day Tutorial
10/13/2014 8:30:00 AM
"Getting Started with Risk-Based
Testing"
Presented by:
Dale Perry
Software Quality Engineering
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Dale Perry
Software Quality Engineering
Dale Perry has more than thirty-eight years of experience in information technology as a
programmer/analyst, database administrator, project manager, development manager, tester,
and test manager. Dale’s project experience includes large-system development and
conversions, distributed systems, and both client/server and web-based online applications. A
professional instructor for more than twenty-four years, he has presented at numerous industry
conferences on development and testing. With Software Quality Engineering for eighteen years,
Dale has specialized in training and consulting on testing, inspections and reviews, and other
testing and quality-related topics.
Speaker Presentations

1© SQE Training - STAR West 2014

This page left blank

Notice of Rights
Entire contents © 2010-2014 by SQE Training, unless otherwise noted on specific
items. All rights reserved. No material in this publication may be reproduced in any
form without the express written permission of SQE Training.
Home Office
SQE Training
340 Corporate Way, Suite 300
Orange Park FL 32073 U.S.A.
904. 278.0524
904.278.4380 fax
www.sqetraining.com
Notice of Liability
The information provided in this book is distributed on an “as is” basis, without
warranty. Neither the author nor SQE Training shall have any liability to any person or
entity with respect to any loss or damage caused or alleged to have been caused
directly or indirectly by the content provided in this course.

Many of the development models in use today originated with the concepts
developed by Dr. Shewhart in 1938 at Bell Laboratories and extended to
manufacturing and quality control by Dr. Demming in the 1950s.
This concept was called the life cycle of life cycles.

Static testing: Testing of a component or system at specification or
implementation level without execution of that software, e.g., reviews orimplementation level without execution of that software, e.g., reviews or
static code analysis.
Dynamic testing: Testing that involves the execution of the software of a
component or system.

Testing every possible data value, every possible navigation path through
the code, and every possible combination of input values is almost always
an infinite task which never can be completed.
Even if it were possible, it is not necessarily a good idea because many of
the test cases would be redundant, consume resources to create, delay time
to market, and not add anything of value.
For example, a single screen (GUI) or data stream, with thirteen variables,
each with three values
• To test every possible combination is 3(13) possible combinations
or 1,594,323 tests
• Plus testing of interfaces, etc.

The purpose of discussing software risk is to determine the primary focus of
testing. Generally speaking, most organizations find that their resources are
inadequate to test everything in a given release.
Outlining software risks helps the testers prioritize what to test and allows
them to concentrate on those areas that are likely to fail or those areas that
will critically impact the customer if they do fail.
Risks are used to decide where to start testing and where to test more.
Testing is used to reduce the risk of an adverse effect occurring or to reduce
the impact of an adverse effect.

Organizations that work on safety-critical software usually can use the
information from their safety and hazard analysis to identify areas of risk.
However, many companies make no attempt to verbalize software risks in
any fashion. If your company does not currently do any type of risk analysis,
try a brainstorming session among a small group of users, developers, and
testers to identify concerns.

Risk Factor
1 Ambiguous Improvement Targets
2 Artificial Maturity Levels
3 Canceled Projects
4 Corporate Politics
5 Cost Overruns
6 Creeping User Requirements
7 Crowded Office Conditions
8 Error Prone Modules
9 Excessive Paperwork
10 Excessive schedule Pressure
11 Excessive Time to Market
1212 False Productivity Claims
13 Friction Between Clients and Software
Contractors
14 Friction Between Software Management and
Senior Executives
15 High Maintenance Costs
16 Inaccurate Cost Estimating
17 Inaccurate Metrics
18 Inaccurate Quality Estimating
19 Inaccurate Sizing of Deliverables
20 Inadequate Assessments
21 Inadequate Compensation Plans

Risk Factor
22 Inadequate Configuration Control and
Project Repositories
23 Inadequate Curricula (Software Engineering)
24 Inadequate Curricula (Software
Management)
25 Inadequate Measurement
26 Inadequate Package Acquisition Methods
27 Inadequate Research and Reference
Facilities
28 Inadequate Software Policies and Standards
29 Inadequate Project Risk Analysis
30 Inadequate Project value Analysis
31 Inadequate Tools and Methods (Project31 Inadequate Tools and Methods (Project
Management)
32 Inadequate Tools and Methods (Quality
Assurance)
33 Inadequate Tools and Methods (Software
Engineering)
34 Inadequate Tools and Methods (Technical
Documentation)
35 Lack of Reusable Architecture
36 Lack of Reusable Code
37 Lack of Reusable Data
38 Lack of Reusable Designs (Blueprints)
39 Lack of Reusable Documentation
40 Lack of Reusable Estimates (Templates)

Risk Factor
41 Lack of Reusable Human Interfaces
42 Lack of Reusable Project Plans
43 Lack of Reusable Requirements
44 Lack of Reusable Test Plans. Test Cases
and Test Data
45 Lack of Specialization
46 Long Service Life of Obsolete Systems
47 Low Productivity
48 Low Quality
49 Low Status of Software Personnel and
Management
50 Low User Satisfaction
51 Malpractice (Project Management)
52 Malpractice (Technical Staff)
53 Missed Schedules
54 Partial Life-Cycle Definitions
55 Poor Organization Structures
56 Poor Technology Investments
57 Severe Layoffs and Cutbacks of Staff
58 Short-Range Improvement Planning
59 Silver Bullet Syndrome
60 Slow Technology Transfer

Managers tend to focus on two key elements
• Controlling and managing costs
• Return on investment (ROI)
Marketing and sales people tend to be driven by a singular goal, “competitive
advantage”
• The concerns of marketing and sales are not necessarily related to
functionality, they need an edge
Engineers (developers, analysts, etc.) tend to be focused on the technology
itself
• Driven by the use of “new” technology and techniques
• Not interested in functionality except as it relates to the use of
technology
• Some technically valid decisions may cause functional
problems

Customers are seeking the answer to a very simple question, “can I do my
job?”
Their view of a product is a tool to do their job
If they can’t use your product to that end, it’s a bad product

Quality assurance is part of an organizations’ overall set of processes.
Testing primarily focuses on product risk. Testing is an evaluation activity, it
uses the processes defined within and organization to assess the quality of a
product as defined within the scope of a project. Testing also assesses the
process used within the organization to ensure they are usable, effective and
appropriate. A more accurate acronym for testing is Q/C (Quality Control).
Simply stated: We can think of testing (Q/C) as the application of process to
a product within the context of a project focusing on risks.

The concept of risk driven testing applies to all software development models
and processes. It is critical to developing quality software that meets
user/customer expectations and is the focus of both the STEP™ methodology
and many of the new agile development processes.
If you analyze the newer “agile” development methods, this is one of the key
concepts. It’s interesting that this is not really a new concept at all; it has been
around for a couple of decades.

There are many different software lifecycle approaches: waterfall, spiral,
incremental delivery, prototyping (evolutionary and throwaway), RAD,
extreme programming (XP), SCRUM, DSDM, etc. The key is to know which
process the project is following and to integrate into that process as soon as
possible (reasonable). The later you get involved, the less chance you have
to prevent problems.

Planning is essential for all projects, regardless of type of development
model. There are certain essential decisions that need to be made early in
the project to ensure the project will be a success and that the testing can be
realistically accomplished within the constraints of the project.

In testing it is critical to identify those elements within the scope of the
project that require testing. Once identified those elements need to be
assessed as to what is absolutely critical to the operation of the software and
the user’s needs and expectations and what can possibly be deferred.
Not all elements within the scope of a project are of equal importance.

This model is based on the risk analysis model within the STEP™ testing
methodology developed by Software Quality Engineering. The general
process can be applied to any type of product and within any type of
development model.

All project start with something. The only reason to develop or change
software is because one of the stakeholders in an organization has a
problem with the existing processes or systems.
A problem does not necessarily mean that the software does not function,
although there may be problems with the software there cane be other
reasons for someone to want to change the system. The business model
may have shifted or a government entity may have changed some rule or
process. It may be that an internal marketing team is looking to add some
advantages to the system to improve the competitive nature of the system.
Regardless of the reason, software is developed or changed because
someone has a problem, understanding the nature of the problem is
essential to both develop a solution and to ensure it works (test the solution).

Reference materials include any information available that can assist in
determining the testing objects/conditions.
Some lifecycles do not have formal sources of documentation. No formal
requirements are written. However, there is usually some information about
what type of system is being created, the platform on which it will run, the
goals of the client, etc.
Any information you can gather will help you better understand the test
requirements for this project.

Different groups have different ideas about software. The more of theseDifferent groups have different ideas about software. The more of these
disparate groups you can combine, the more accurate the picture you will
have of the risks, priorities, and goals for development, and the more
accurate the testing goals and objects/conditions for this project become.
In addition to the stakeholders noted earlier you may also want to consider
others who have knowledge of current and past issues associated with the
system under discussion. Groups such as the help desk and user support
tend to have in-depth knowledge of current as well as past issues.
The key is to include testers in the discussions to help focus the team on the
testing issues and to help determine the priority of the features to be
developed. Testers need to know the risks and issues in order to properly
analyze and design reasonable tests to mitigate/control those risks or to at
least know what level of risk the stakeholders are willing to accept.

As soon as the project is established and the general focus of the software is
known, either new development or maintenance, the process of risk
assessment begins.

When designing an inventory or elements/objects to test we need to asses
three different aspects of the software/system
Functional – Identification of “what” the user/customer expects the
system/.application to provide. These are often referred to as “black box” or
specification based elements (as they are typically derived from some form
of specification (document) or model.
Non-Functional – Identifying aspects of the system that need to be tested
that are not necessarily part of the functions required by the user/customer.
i.e. performance, usability, response time, etc. These are characteristics or
attributes of the system and reflect “how” the system works not “what” it
does or how it’s built.
Structural – Identification of the physical elements of the infrastructure,
architecture, design or code that also require testing. There are often
elements within the system that are not part of user/customer’s requirements
but they still need testing. These represent “how” the system is constructed.

The inventory can be a separate artifact/document or in can be incorporated
into and existing artifact/document such as a risk prioritized requirements
specification, use case model or story backlog. The inventory can represent
an entire system, part of a system or a single requirement, use case or story.
The key to an inventory is to identify what needs to be tested from as many
viewpoints as possible.

The inventory process is an iterative process. You begin the process at
requirements and continue the process at each stage of the development
process. How far you take the process is determined by the scope and risks
associated with the software being tested. In addition to the process being
iterative, it is also cumulative.
The information from requirements is used to improve the requirements
(static testing/reviews), to focus the design, and possibly to improve the
design.
At the design stage of a project the information from the requirements
inventory process is used to evaluate the design and to ensure problems are
corrected and additional items are gathered from the design. This process
can be continued as far as the risks to the project warrant.

These can be specific elements to be tested or they can be aspects
(characteristics, attributes) of elements identified in other work products such
as an expansion of a user story or use case from a testing and risk
perspective.
A single story, use case or requirement may have multiple aspects that
requiring testing. E.g. a function may have inputs, and/or outputs that are
retrieved from and sent to a data store via an interface or transaction and
those elements may have specific relationships (rules or constraints), all of
which need to be tested.
The identified (requirement, story or use case) may also participate in a
larger sequence of events (scenario) that also must eventually be tested to
ensure the identified element works correctly at all levels within an
application or system (end-to-end).
Some aspects of the identified element may also be sensitive to the
environment and so will require additional testing in multiple enrivonments.

While creating the inventory we may also identify aspects of the identified
elements that are not completely defined or may be vague, incomplete,
ambiguous, etc. These issues and questions are logged to ensure that they
are not lost. S
ome of the identified issues may not be resolvable at the requirements stage
of a project and may be further defined later in the development process.
This is perfectly acceptable, provided the information is logged and tracked
somehow. The information can be part of an existing issues/defect tracking
system, part of a backlog, or may be a separate artifact/document. How it is
managed will depend to a large degree on the development process
employed. The key is not to lose tract of the identified issues and questions.
For incomplete information we would come back to our list of identified
issues and questions at the appropriate time within the development process
and ensure that they have, in fact, been addressed to a reasonable degree
and are now testable.

Once the requirements have been assessed, the initial inventory created and
any issues and questions logged, the team may have to wait for the system
design (architecture and detail aspects) to be defined.
When the design is ready we can continue the inventory process by
assessing the design in the same manner as we did the requirements. The
existing inventory and the list of known issues and questions is the input into
the next step in the process.

During the assessment of the design, issues and questions that were
deferred until the design was defined can now be revisited. At this point we
should have additional information that will help us clarify open issues from
the requirements assessment.
At this stage three things are occurring:
1. Identification of additional elements to add to our inventory
2. Resolution of issues and questions from the requirements process
3. Identification of new issues and questions related to the design
How far we take the inventory process depends on the development model,
the organizational processes in place and the level of potential risk. There is
no specific rule as to how far the inventory process if taken within a project.

Reviewing designs may require additional technical skills that not all testers
have. At least one person on the test team needs to have some technical
skills that relate to the system/product being tested in order to ask critical
questions.
Early tests, especially scenario based tests, developed during the
requirements part of the process can be used as walk through models for
the design.
Walking through a design with a set of functional and non-functional tests
can often reveal problems that would be missed if only looked at from an
engineering perspective. After all, development (software engineering) is the
process of solving a problem with a computer. People solving problems have
a different perspective than those trying to ensure the solution works. This
process may even reveal places where changes can be made to aid in
testing, such as exposing the information in an API or memory table.

There are some common aspects of applications that can be drawn from the
design specifications.
Updating the inventory will be about providing clarification of earlier issues
and questions raised during the requirements inventory review as well as
adding new objects/elements to the inventory. Many of the elements added
at this point will be technical in nature and late will be tested primarily at the
component or integration stages of development. It is critical to know where
these technical elements exist so that the get proper testing.
May failures discovered in system testing are often symptoms of incomplete
or ineffective testing of technical elements within a product/system.

Analyzing an individual requirement, use case, story, interface etc. allows the
tester to more accurately understand the scope of testing related to a
particular element/object. This can also help in estimating the scope of effort
required to properly test a particular element/object.

A single requirement, use case or story may have multiple aspects that need
to be tested. Just as we can be a general list of overall elements/objects to
test we can also construct an inventory specific to a single element.

Just like a functional aspect can be analyzed for elements to test so can a
structural element. The single element is the interface but there are many
aspects of the interface that need to be tested to ensure it works correctly.

Impact is sometimes referred to as severity.
Once the inventory has been built, the next step is to determine the Impact
and likelihood of something going wrong with each of the elements identified
in the inventory.
• Determine the impact (loss or damage) and likelihood (frequency or
probability) of the feature or attribute failing.
While some organizations like to use percentages, number of days/years
between occurrences, or even probability “half lives,” using a set of simple
categories such as the ones listed in the slide above typically provide
sufficient accuracy.
If the likelihood or impact of something going wrong is none or zero, then
this item could be removed from the analysis. However, the removal should
be documented.
• This is not recommended. Just leave it in the inventory, it will
naturally drop to the bottom.

Many of the factors that affect likelihood are technical in nature. In
development much of the code developed is about handling the errors and
exceptions that occur within the system. There is typically more code
covering the errors and exceptions than there is code implementing the
general rules and processes requested b y the stakeholders.

Impact is typically viewed as the impact on the user of the product./system.
Impact is normally assessed from a business perspective, what is the
damage to the business should an identified risk occur.

Most organization use a combination of both methods.
• Qualitative helps express risk in terms a person can understand.
• Quantitative is then used in conjunction with qualitative categories
to create a matrix with each risk given a weighted priority.
Each qualitative category needs to be defined in the organization’s overall
process directives. Ideally, there needs to be several examples of each risk
assessment category to aid people is determining which category is
appropriate to the object they are assessing.
Risk is in the eye of the beholder as noted earlier.
• Any two people may look at the same event and see an entirely
different set of issues. What is critical to one may be trivial to the
other.

Likelihood = The probability or chance of an event occurring (e.g., the
likelihood that a user will make a mistake and, if a mistake is made, the
likelihood that it will go undetected by the software).
Impact = The damage that results from a failure (e.g., the system crashing or
corrupting data might be considered high impact).
Something could have a low likelihood (technical) risk and yet have a very
high impact (business) risk and vice-versa. It is the combination of the two
factors that helps to determine the overall risk level for a particular element.

Within an organization it may be necessary to set up different risk models for
different products/systems. This is not a problem as not all products/systems
should necessarily follow the exact same model.
The “quantitative” factors can be changed and adjusted as needed, however,
the “qualitative” elements should remain consistent across different risk
models. This will ensure that risks are assessed the same across the
organization and still allow for adjustments for differences in relative risk
levels.
By adjusting the “quantitative” factors we can allow for applications that have
more business than technical risk and vice-versa but still have a consistent
model that can be used across the organization.

Under likelihood and impact, there may be differences of opinion as to the
risk. It can be high business risk but low technical risk, etc. So you may have
to compromise on an acceptable level of risk.
The numbers are calculated using the values from our original matrix (page
70 and multiplying them.
H = High which has a value of 3
M = Medium which has a value of 2
L = Low which has a value of 1

Make adjustments and sort by the agreed priority. We now have a risk-based
assessment of what needs to be tested.

Of course, successful prioritization of risks does not help unless test cases
are defined for each risk—with the highest priority risks being assigned the
most comprehensive tests and priority scheduling. The object of each test
case is to mitigate at least one of these risks.
If time or resources are an issue, then the priority associated with each
feature or attribute can be used to determine which test cases should be
created and/or run. If testing must be cut, then the risk priority can be used
to determine how and what to drop.
• Cut low risk completely (indicated by the horizontal line).
If you plan to ship the low risk features, you may want to consider an across
the board approach, where the high risk features are fully tested and the
lower risk are tested a little less based on the level of risk associated with
the feature. So medium risk elements get less that high but mode than low
risk elements. At least that way, the features do not ship untested (risk
unknown). This will entail some additional risk as higher risk features get
less testing.

Appendix has an expanded version of the FMEA (Failure Modes and Effects
Analsys) model as well as the ISO 9126 quality model.

Risk mitigation/Risk control: The process through which decisions are
reached and protective measures are implemented for reducing risks to, or
maintaining risks within, specified levels.
Risk type: A specific category of risk related to the type of testing that can
mitigate (control) that category. For example the risk of user-interactions
being misunderstood can be mitigated by usability testing.

Murphy’s Law: “If anything can go wrong, it will, and will do so at the worst
possible time”.

Although exploratory testing primarily relies on the skills and knowledge of
the tester and tends to be more dynamic than traditional technique-driven
design, it too can be more formalized. Using the inventory process as part
of an exploratory test process can add structure to the definition of the areas
to be investigated rather than relying only on the skills of the individual
tester.

The goal is to avoid gaps in the testing as well as to avoid overlapping
testing too much.
Depending on how you define your inventories, based on generic groupings
or application specific groupings, the idea to decide who will test which
objects at what stage/level.
Some objects cannot be tested until later stages of the process (i.e.,
scenarios and usage based objects). Conversely some elements, such as
field edits, valid ranges, error messages etc., are best tested in the earlier
stages. These code logic elements, created by the programmers, are best
tested at that stage of the process. Finding such errors late in the process
can be very costly.

These three aspects of test execution have to be evaluated together.
Looking at any single aspect provides no real useful information.

Requirements Coverage:
• Which requirements have been tested?
• What are the key risk indicators?
• Degree of coverage for each requirement
• Relative to defined tests
• Failure rates
• Defect information (density, patterns, trends, root cause,
etc.
Design Coverage
• Have we addressed key design issues?
• Do we require coverage measures?
• Integration complexity
• Interface (API) coverage
• What are the key risk indicators?
• Failure rates within the design
• Defect information (density, patterns, trends, root
cause, etc.

Risk Coverage
• Are we covering the high risk areas first?
• Are the executed tests balanced across the risk areas or narrowly
focused on specific risk areas?
• What risks areas are still unaddressed?
• What is the level of risk in untested elements?
Code Coverage
• What percentage of the code has been tested?
• Which coverage measures are important?
• Statement coverage
• Decision/branch coverage
• Condition coverage
• Path coverage
• Key measures such as cyclomatic complexity
• Other code coverage measures

Web site references
General testing sites for research etc.
www.Stickyminds.com
www.techwell.com
www.sqaforums.com
www.softwareqatest.com
www.testingeduction.org
Sites for open source elements (tools etc.)
www.sourceforge.net
www.opensourcetesting.org
www.pairwise.org
www.pairwise.org/tools.asp
Sites related to performance, performance services and statistics
Performance-testing.org
www.sandvine.com
TPC.org
www.internetworldstats.com
www.keynote.comwww.keynote.com
www.webtrends.com
www.shunra.com
www.alexa.com
www.mentora.com
www.plunkettresearch.com
Sites related to usability etc.
www.useit.com
www.nngroup.com
www.webpagesthatsuck.com (yes this is a serious site)
sumi.ucc.ie
www.wammi.com
General security related sites
www.sans.org
www.nist.gov
csrc.nist.gov

This page left blank.

Related terms:
FMECA
SFMEA
SFMECA
Every product has modes of failure. The effects represent the impact of
failures.
— adapted from the Quality Training Portal

FMEA can be applied at any level and should be done iteratively.

ISO 9126 extract
6.1 Functionality
The capability of the software product to provide functions, which meet stated and implied
needs when the software is used under specified conditions. This characteristic is concerned
with what the software does to fulfill needs, whereas the other characteristics are mainly
concerned with when and how it fulfils needs. For a system, which is operated by a user, the
combination of functionality, reliability, usability and efficiency can be measured externally by
quality in use.
Subcategories
6.1.1 Suitability - The capability of the software product to provide an appropriate set of
functions for specified tasks and user objectives.
6.1.2 Accuracy - The capability of the software product to provide the right, or agreed
results, or effects with the needed degree of precision.
6.1.3 Interoperability - The capability of the software product to interact with one or
more specified systems.
6.1.4 Security - The capability of the software product to protect information and data6.1.4 Security - The capability of the software product to protect information and data
so that unauthorized persons or systems cannot read or modify them and authorized
persons or systems are not denied access to them.
6.1.5 Functionality compliance - The capability of the software product to adhere to
standards, conventions or regulations in laws and similar prescriptions relating to
functionality.
6.2 Reliability
The capability of the software product to maintain a specified level of performance when used
under specified conditions. Wear or ageing does not occur in software. Limitations in reliability
are due to faults in requirements, design, and implementation. Failures due to these faults
depend on the way the software product is used and the program options selected rather than
on elapsed time. The definition of reliability in ISO/IEC 2382-14:1997 is "The ability of functional
unit to perform a required function...". In this document, functionality is only one of the
characteristics of software quality. Therefore, the definition of reliability has been broadened to
"maintain a specified level of performance..." instead of "...perform a required function".
Subcategories
6.2.1 Maturity
The capability of the software product to avoid failure as a result of faults in the
software.

6.2.2 Fault tolerance - The capability of the software product to maintain a specified
level of performance in cases of software faults or of infringement of its specified
interface.
NOTE The specified level of performance may include fail safe capability.
6.2.3 Recoverability - The capability of the software product to re-establish a specified
level of performance and recover the data directly affected in the case of a failure.
NOTE 1 Following a failure, a software product will sometimes be down for a certain
period of time, the length of which is assessed by its recoverability.
NOTE 2 Availability is the capability of the software product to be in a state to perform
a required function at a given point in time, under stated conditions of use. Externally,
availability can be assessed by the proportion of total time during which the software
product is in an up state. Availability is therefore a combination of maturity (which
governs the frequency of failure), fault tolerance and recoverability (which governs the
length of down time following each failure). For this reason it has not been included as a
separate subcharacteristic.
6.2.4 Reliability compliance - The capability of the software product to adhere to
standards, conventions or regulations relating to reliability.
6.3 Usability
The capability of the software product to be understood, learned, used and attractive to the
user, when used under specified conditions. Some aspects of functionality, reliability and
efficiency will also affect usability, but for the purposes of ISO/IEC 9126 they are not classified
as usability. Users may include operators, end users and indirect users who are under the
influence of or dependent on the use of the software. Usability should address all of the
different user environments that the software may affect, which may include preparation for
usage and evaluation of results.
Subcategories
Understandability - The capability of the software product to enable the user to
understand whether the software is suitable, and how it can be used for particular tasks
and conditions of use. This will depend on the documentation and initial impressions
given by the software.
Learnability - The capability of the software product to enable the user to learn its
application. The internal attributes correspond to suitability for learning as defined in
ISO 9241-10.

Operability - The capability of the software product to enable the user to operate and
control it.
NOTE 1 Aspects of suitability, changeability, adaptability and installability may affect
operability.
NOTE 2 Operability corresponds to controllability, error tolerance and conformity with
user expectations as defined in ISO 9241-10.
NOTE 3 For a system which is operated by a user, the combination of functionality,
reliability, usability andefficiency can be measured externally by quality in use.
Attractiveness - The capability of the software product to be attractive to the user. This
refers to attributes of the software intended to make the software more attractive to
the user, such as the use of color and the nature of the graphical design.
Usability compliance - The capability of the software product to adhere to standards,
conventions, style guides or regulations relating to usability.
6.4 Efficiency - The capability of the software product to provide appropriate performance,
relative to the amount of resources used, under stated conditions. Resources may include other
software products, the software and hardware configuration of the system, and materials (e.g.software products, the software and hardware configuration of the system, and materials (e.g.
print paper, diskettes). For a system which is operated by a user, the combination of
functionality, reliability, usability and efficiency can be measured externally by quality in use.
Subcategories
6.4.1 Time behavior - The capability of the software product to provide appropriate
response and processing times and throughput rates when performing its function,
under stated conditions.
6.4.2 Resource utilization - The capability of the software product to use appropriate
amounts and types of resources when the software performs its function under stated
conditions. Human resources are included as part of productivity.
6.4.3 Efficiency compliance - The capability of the software product to adhere to
standards or conventions relating to efficiency.
6.5 Maintainability
The capability of the software product to be modified. Modifications may include corrections,
improvements or adaptation of the software to changes in environment, and in requirements
and functional specifications.

Subcategories
6.5.1 Analysability - The capability of the software product to be diagnosed for
deficiencies or causes of failures in the software, or for the parts to be modified to be
identified.
6.5.2 Changeability - The capability of the software product to enable a specified
modification to be implemented. Implementation includes coding, designing and
documenting changes. If the software is to be modified by the end user, changeability
may affect operability.
6.5.3 Stability - The capability of the software product to avoid unexpected effects from
modifications of the software.
6.5.4 Testability - The capability of the software product to enable modified software to
be validated.
6.5.5 Maintainability compliance - The capability of the software product to adhere to
standards or conventions relating to maintainability.
6.6 Portability - The capability of the software product to be transferred from one environment
to another. The environment may include organizational, hardware or software environment.
Subcategories
6.6.1 Adaptability - The capability of the software product to be adapted for different
specified environments without applying actions or means other than those provided
for this purpose for the software considered. Adaptability includes the scalability of
internal capacity (e.g. screen fields, tables, transaction volumes, report formats, etc.). If
the software is to be adapted by the end user, adaptability corresponds to suitability for
individualization as defined in ISO 9241-10, and may affect operability.
6.6.2 Installability - The capability of the software product to be installed in a specified
environment.
NOTE If the software is to be installed by an end user, installability can affect the
resulting suitability and operability.
6.6.3 Co-existence - The capability of the software product to co-exist with other
independent software in a common environment sharing common resources.

6.6.4 Replaceability - The capability of the software product to be used in place of
another specified software product for the same purpose in the same environment. For
example, the replaceability of a new version of a software product is important to the
user when upgrading. Replaceability is used in place of compatibility in order to avoid
possible ambiguity with interoperability. Replaceability may include attributes of both
installability and adaptability. The concept has been introduced as a sub characteristic of
its own because of its importance.
6.6.5 Portability compliance - The capability of the software product to adhere to
standards or conventions relating to portability

Getting Started with Risk-Based Testing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to Getting Started with Risk-Based Testing

Similar to Getting Started with Risk-Based Testing (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Getting Started with Risk-Based Testing