Seng421 08 handout

Contents
Software quality
Software quality models: Boehm’s model,
SENG 421: McCall’s model, ISO 9126 model, etc.
Software Metrics Software quality metrics
Quality management models
Measuring External Product Attributes: Measuring customer satisfaction
Software Quality
Software Quality Assurance (SQA)
(Chapter 8)
Department of Electrical & Computer Engineering, University of Calgary

B.H. Far （far@ucalgary.ca）
http://www.enel.ucalgary.ca/People/far/Lectures/SENG421/08/

far@ucalgary.ca 2

What is Quality? What is Software Quality? /1
Quality popular views: Conventional definitions:
Something “good” but not Conformance to requirement
quantifiable The requirements are clearly stated and the product must
Something luxury and classy conform to it.
Any deviation from the requirements is regarded as a
defect.
Quality professional views: A good quality product contains fewer “bugs.”
Conformance to requirement
Fitness for use
(Crosby, 1979)
Fit to “user” expectations: meet user’s needs.
Fitness for use (Juran, 1970)
A good quality product provides better user satisfaction.

Both Dependable computing system
far@ucalgary.ca 3 far@ucalgary.ca 4

What is Software Quality? /2 Definition: Software Quality
Various viewpoints/perspectives: ISO 8402 definition of quality:
Conformance to customers’ requirements The totality of features and characteristics of a
Requirement, design and development quality product or a service that bear on its ability to
Process quality vs. end-product quality satisfy stated or implied needs
Process quality: higher usability and dependability Conformance to customers’ requirements
End-product quality: less failure Containing fewer bugs
Internal vs. external characteristics Providing better user satisfaction
Relativity: advantage over similar products
Conformance: conformance to standards


Software Quality: Difficulties Software Quality: Classification
Need to account for By observation:
“creativity” in the “design” of Leonardo
da Vinci Internal quality (quality while the product is
the product and the being produced, including process and checks)
Mona Lisa
“requirements” rather than the (1479) External quality (final product quality)
product itself. By process:
Design quality
Kind of art … Implementation quality
Which one has a higher Pablo Picasso
Test quality
quality? Dorra Maar
(1937)
Maintenance quality


Demming’s 14 Quality Points Demming’s 14 Quality Points
1. Create constancy of purpose for improvement of 8. Drive out fear.
product or service. 9. Break down barriers between staff areas.
2. Adopt the new philosophy. 10. Eliminate slogans, exhortations, and targets for the
3. Cease dependence on inspection to achieve quality. work force.
11. Eliminate numerical quotas for the work force and
4. End the practice of awarding business on the basis W. E. Deming numerical goals for management. W. E. Deming
of price tag alone. Instead, minimize total cost by (1900-1993)
12. Remove barriers that rob people of pride of
(1900-1993)
The Father of The Father of
working with a single supplier. Modern Quality workmanship. Eliminate the annual rating or merit Modern Quality
5. Improve constantly and forever every process for system.
planning, production and service. 13. Institute a vigorous program of education and self-
6. Institute training on the job. improvement for everyone.
7. Adopt and institute leadership. 14. Put everybody in the company to work to
accomplish the transformation.


TQM: Total Quality Management The Practice of TQM: Points
TQM is a style of Focus on quality
management aiming at Cooperate with customers
achieving “long-term” Continuously improve development process
success by linking
quality with customer Encourage constructive critics and empower
satisfaction employees
Other variations: Use the problem solving/problem prevention cycle
Studying customer needs, gather
Total Quality Control customer requirements, measuring Use measurements to back decisions
customer satisfaction
(HP)
Achieve business and product
Market Driven Quality development process improvement
(IBM)
Creation of company-wide quality
culture Figure from Kan’s Book


Modeling Software Quality Example: CUPRIMDA Model
Selecting some attributes (or factors) Quality parameters
Plotting relationship among attributes (many-to- (parameters for
fitness):
many relationships)
Capability
or Usability
Selecting some criteria (or intermediate and Performance
primitive constructs) to represent the attributes Reliability
Installability
Mapping criteria (or primitive constructs) to metrics
Maintainability
Documentation Reference: S.H. Kan (1995)
Availability


Example: Boehm’s Model Example: McCall’s Model

Figure from Fenton’s Book Figure from Fenton’s Book


Bayesian Causal Model Quality Model: ISO 9126 /1
Quality is affected Specifying software quality for a product yet to be developed
Customer is difficult for the customer and/or supplier.
by many other Satisfaction The customer needs to understand and communicate
attributes, each requirements for the product to be developed.
Process
driven by some set The supplier needs to understand the requirement and to
Complexity
of metrics assess with confidence whether it is possible to provide the
product with the right level of quality.
A Bayesian Network Reliability
ISO 9126 will serve to eliminate any misunderstanding
to represent such a between customer and supplier.
model can be ISO 9126 is the software product evaluation standard. This
Quality
devised international standard defines six characteristics that
describe, with minimal overlap, software quality.


Quality Model: ISO 9126 Quality Model: ISO 9126 /2
1. Functionality is the set of attributes that bear on the existence of a set of
functions and their specified properties. The functions are those that
satisfy stated or implied needs.
2. Reliability is the set of attributes that bear on the capability of software
to maintain its level of performance under stated conditions for a stated
period of time.
3. Usability is the set of attributes that bear on the effort needed for use,
and on the individual assessment of such use, by a stated or implied set
of users.
4. Efficiency is the set of attributes that bear on the relationship between
the level of performance of the software and the amount of resources
used, under stated conditions.
5. Maintainability is the set of attributes that bear on the effort needed to
make specified modifications.
6. Portability is the set of attributes that bear on the ability of software to
be transferred from one environment.


Quality Model: ISO 9126 /3 ISO 9126: Users’ View
Original ISO 9126 does not provide sub-characteristics and metrics, nor Users are mainly interested in using the software, its
the method for measurement, rating and assessment. performance and the effects of using the software.
Characteristics Attributes Users evaluate the software without knowing the internal
Functionality Suitability Interoperability Accuracy aspects of the software, or how the software is developed.
Compliance Security
Users’ questions may include:
Reliability Maturity Recoverability Fault tolerance
Are the required functions available in the software?
Crash frequency
How reliable is the software?
Usability Understandability Learnability Operability
How efficient is the software?
Efficiency Time behaviour Resource behaviour
Is the software easy to use?
Maintainability Analyzability Stability Changeability
How easy is it to transfer the software to another environment?
Testability
How easy is to change this software?
Portability Adaptability Installability Conformance
Replacability


ISO 9126: Developers’ View ISO 9126: Managers’ View
The process of development requires the user and the A manager is typically more interested in the overall quality
developer to use the same software quality characteristics, rather than in a specific quality characteristic, and for this
since they apply to requirement and acceptance. When reason will need to assign weights, reflecting business
developing off-the-shelf software, the implied needs must be requirements, to the individual characteristics.
reflected in the quality requirement.
Since developers are responsible for producing software The manager may also need to balance the quality
which will satisfy quality requirements they are interested in improvement with management criteria such as schedule
the intermediate product quality as well as in the final delay or cost overrun, because he/she wishes to optimise
product quality. In order to evaluate the intermediate product quality within limited cost, human resources and time-frame.
quality at each phase of the development cycle, the
developers have to use different metrics for the same
characteristics because the same metrics are not applicable to
all phases of the cycle.


ISO 9126: Extensions /1 ISO 9126: Extensions /2
The new edition of ISO/IEC 9126 is divided into three parts: ISO/IEC 9126-2:
ISO/IEC 9126-1: (Released in 2001) Information technology - Software quality characteristics
Information technology - Software quality characteristics & metrics & metrics - Part 2: External metrics.
- Part 1: Quality characteristics and subcharacteristics.
This part provides the concepts introduced in the original standard is a This part provides external metrics for measuring software
recommended quality model which categorizes software quality in six quality characteristics.
characteristics, which are further sub-divided into subcharacteristics. The An external metric is a quantitative scale and measurement
subcharacteristics have been moved from the annex to become part of the method, which can be used for measuring an attribute or
standard. They have been reworded and several new ones added.
There is also a definition of quality in use which defines the user’s view
characteristic of a software product, derived from the
as a result of using the software. behaviour of the system of which it is a part.
External metrics are applicable to an executable software
product during testing or operating in later stage of
development and after entering to operation process.


ISO 9126: Extensions /3 Flexible Quality Model
ISO/IEC 9126-3: No need to stick to a fixed (published) model.
Information technology - Software quality characteristics
& metrics - Part 3: Internal metrics. You can define your own quality model
This part provides internal metrics for measuring software based on users (customers and company)
quality characteristics.
An internal metric is a quantitative scale and measurement consensus. The model will be composed of
method, which can be used for measuring an attribute or quality attributes which are important for the
characteristic of a software product, derived from the given product.
product itself, either direct or indirect.
Internal metrics are applicable to a non executable software The model should be verified by actual
product during designing and coding in early stage of
development process. measurement.


Example: Attribute Expansion
Design by measurable Quality objective
objectives:
Incremental design is
Availability
Software Quality:
User friendliness
ISO 9126 Standard
evaluated to check whether
the goal for each increment
was achieved.
% of planned Days on job to
System uptime learn task supplied
By new system

Worst: 95% Worst: 7 days
Best: 99% Best: 1 day

far@ucalgary.ca 29

Review: Software Quality Quality Model: ISO 9126
ISO 8402 definition of quality:
The totality of features and characteristics of a
product or a service that bear on its ability to
satisfy stated or implied needs
Conformance to customers’ requirements
Containing fewer bugs
Providing better user satisfaction


Quality Model: ISO 9126 ISO 9126: 1. Functionality
Suitability: Attributes of software that bear on the presence
Characteristics Attributes and appropriateness of a set of functions for specified tasks.
Functionality Suitability Interoperability Accuracy
Accuracy: Attributes of software that bear on the provision
Compliance Security of right or agreed results or effects.
Reliability Maturity Recoverability Fault tolerance Interoperability: Attributes of software that bear on its
Crash frequency ability to interact with specified systems.
Usability Understandability Learnability Operability Compliance: Attributes of software that make the software
Efficiency Time behaviour Resource behaviour adhere to application related standards or conventions or
Maintainability Analyzability Stability Changeability regulations in laws and similar prescriptions.
Testability Security: Attributes of software that bear on its ability to
Portability Adaptability Installability Conformance prevent unauthorized access, whether accidental or
Replacability deliberate, to programs and data.


ISO 9126: 2. Reliability ISO 9126: 3. Usability
Maturity: Attributes of software that bear on the frequency Understandability: Attributes of software that bear
of failure by faults in the software. on the users’ effort for recognizing the logical
Fault tolerance: Attributes of software that bear on its concept and its applicability.
ability to maintain a specified level of performance in cases
of software faults or of infringement of its specified Learnability: Attributes of software that bear on
interface.
the users’ effort for learning its application (for
Crash frequency: The number of the system crashes per example, operation control, input, output).
unit of time.
Recoverability: Attributes of software that bear on the
capability to re-establish its level of performance and recover Operability: Attributes of software that bear on the
the data directly affected in case of a failure and on the time users’ effort for operation and operation control.
and effort needed for it.


ISO 9126: 4. Efficiency ISO 9126: 5. Maintainability
Efficiency: The extent to which a product or Analysability: Attributes of software that bear on
the effort needed for diagnosis of deficiencies or
process can operate using the fewest possible causes of failures, or for identification of parts to be
resources. modified.
Time behaviour: Attributes of software that bear on Changeability: Attributes of software that bear on
response and processing times and on throughput rates in the effort needed for modification, fault removal or
performing its function.
for environmental change.
Resource behaviour: Attributes of software that bear on Stability: Attributes of software that bear on the
the amount of resources used and the duration of such use risk of unexpected effect of modifications.
in performing its function. Testability: Attributes of software that bear on the
effort needed for validating the modified software.


ISO 9126: 6. Portability 1. Functionality: Accuracy /1
Adaptability: Attributes of software that bear on the Definition:
opportunity for its adaptation to different specified
environments without applying other actions or means than Attributes of software that bear on the provision
those provided for this purpose for the software considered. of right or agreed results or effects.
Installability: Attributes of software that bear on the effort The degree to which a component is free from faults in its
needed to install the software in a specified environment. specification, design, and implementation.
Conformance: Attributes of software that make the software
adhere to standards or conventions relating to portability. The degree to which a component meets specified
requirements or user needs and expectations.
Replaceability: Attributes of software that bear on the
opportunity and effort of using it in the place of specified The ability of a component to produce specified outputs
other software in the environment of that software. when given specified inputs, and the extent to which they
match or satisfy the requirements.


1. Functionality: Accuracy /2 1. Functionality: Security /1
Measurement: Definition: Attributes of software that bear on its
Number of problem reports per phase, priority, category, ability to prevent unauthorized access, whether
or cause accidental or deliberate, to programs and data.
Number of reported problems per time period Software system security is defined at the following
Number of open real problems per time period levels:
Level 0: no security at all
Number of closed real problems per time period
Level 1: firewalls
Number of unevaluated problem reports
Level 2: encryption
Age of open real problem reports Level 3: authentication (digital ID verification)
Age of unevaluated problem reports Level 4: intrusion protection
Age of real closed problem reports Level 5: combination of level 1-4
Rate of error discovery


1. Functionality: Security /2 2. Reliability: Maturity /1
Measurement: Security level (Lsc) Definition: Attributes of software that bear on
nt the frequency of failure by faults in the software.
Lsc =
nint IEEE 982.2 - 1988 defines the Software Maturity
Index (SMI). This index is useful for assessing
nt : number of successful intrusions release readiness when changes, additions, or
nint : total number of intrusion attempts deletions are made to existing software systems.

Lsc takes values of 0.1 - 0.001 for level 1 and
for level 4 is 10−7 −10−9


2. Reliability: Maturity /2 3. Usability (UA)
Measurement: Definition: The set of attributes that bear on the
SMI = Mt - ( Fc + Fa + Fd) / Mt effort needed for use, and on the individual
where assessment of such use, by a stated or implied set
of users.
Mt is the number of software functions/modules in the
current release Usability can be defined in terms of the number of
Fc is the number of functions/modules that contain available functions (naf) divided by the number of
changes from the previous release system required functions (nrf)
Fa is the number of functions/modules that contain
additions to the previous release
Fd is the number of functions/modules that are deleted
UA = (naf / nrf) × 100%
from the previous release


5. Maintainability System Recovery Time (Trc)
(T
Definition: The set of attributes that bear on the Definition: System Recovery Time
effort needed to make specified modifications.
It can be defined in terms of the probability that the Trc = τ r − τ f
system can be restored within a given time after a
failure. τ r : time to system recovery
Maintainability can be measured by: τ f : time to system failure
System recovery time (Trc)
System service degradation rate (Rsd) Unit: minute or hour or day
Time to switch (Tsw)


Service Degradation Rate (Rsd)
(R Time to Switch (Tsw)
(T
Definition: Service Degradation Time Definition: Time to Switch
⎛n ⎞ Tsw = τ sw − τ f
Rsd = ⎜ fu ⎟ × 100%
⎝ N⎠
τ sw : time to stand-by system activated
n fu : number of unrecoverable functions
τ f : time to system failure
(after maintenance)
N : total number of functions Unit: minute or hour or day
Unit: normalized 0-100%


5. Maintainability: Analyzability 5. Maintainability: Stability
Definition: Attributes of software that Definition: Attributes of software that
bear on the effort needed for diagnosis of bear on the risk of unexpected effect of
deficiencies or causes of failures, or for modifications.
identification of parts to be modified. Metrics:
Metrics: Number of parameters referenced
Cyclomatic number Number of global variables
Number of statements Number of parameters changed
Comments rate Number of called relationships


5. Maintainability: Changeability 5. Maintainability: Testability
Definition: Attributes of software that Definition: Attributes of software that
bear on the effort needed for modification, bear on the effort needed for validating the
fault removal or for environmental modified software.
change.
Metrics:
Metrics:
Number of non-cyclic path
Number of jumps
Number of nested levels Number of nested levels
Average size of statement Cyclomatic number
Number of variables Number of call-paths


6. Portability /1 6. Portability /2
Definition: The set of attributes that bear Total development cost in an environment e1 without
portability
on the ability of software to be transferred
from one environment to another. Cdev (req, e1 ) =
Cdesign (req ) + Ccode (req, e1 ) + Ctest (req, e1 ) + Cdoc (req, e1 )
The principal role of portability metrics is to
help characterize the costs and benefits of The total cost to develop an original portable design plus an
incorporating portability in a software design, implementation for environment e1, is:

or of porting an existing software unit. Cdevp (req, e1 ) = Cdev (req, e1 ) + C pa (req )
Portability cost


6. Portability /3 6. Portability /4
The cost of redevelopment of a software unit (su) with the In general we expect that Cptest < Crtest and
same specifications, targeted for a new environment e2 is: Cpdoc < Crdoc
Furthermore, if a portable design has not been developed, it
Crdev (req, e2 ) = is not unlikely that
Crdesign (req ) + Crcode (req, e2 ) + Crtest (req, e2 ) + Crdoc (req, e2 ) Cmod > Crdesign + Crcode
since the modifications begin with code designed for a
Where, usually Crx ≤ Cx different target, whereas the redevelopment begins with
The cost to port a software unit (su) to a new environment is more generic specifications.
composed primarily of components for manual modification, If there is an effective portable design, however, the
test, and documentation inequality becomes
C port ( su , e2 ) = Cmod ( su , e2 ) + C ptest (req, e2 ) + C pdoc (req, e2 )
Cmod << Crdesign + Crcode

Modification cost

6. Portability /5 Conclusion
Definition: degree of portability (DP) of a software
Characteristics Attributes
unit (su). Functionality Suitability Interoperability Accuracy
⎛ C ( su , e2 ) ⎞ Compliance Security
DP( su ) = 1 − ⎜ port
⎜ C ( req, e ) ⎟⎟ Reliability Maturity Recoverability Fault tolerance
⎝ rdev 2 ⎠ Crash frequency
Usability Understandability Learnability Operability
DP has a maximum value of 1, representing perfect Efficiency Time behaviour Resource behaviour
portability (Cport = 0). Maintainability Analyzability Stability Changeability
Portability is cost-effective if and only if DP > 0. Testability
To appreciate its significance in a particular case it Portability Adaptability Installability Conformance
would be necessary to state the values of Cport and Replacability
Crdev as well.


TQM: Customer Satisfaction
Software Quality: Studies show that
Measuring Customer It is five times more costly to recruit a new
customer than it is to keep an old customer
Satisfaction and
Dissatisfied customers tell 7 to 20 people
about their experiences, while satisfied
customers tell only 3 to 5

far@ucalgary.ca 62

TQM: Customer Satisfaction Methods to Collect Data /1
Total quality management (TQM) is aimed at How to collect data from customers?
long-term business success by linking quality Three common methods to gather data:
with customer satisfaction. Personal face-to-face interviews
With ever-increasing market competition, Telephone interviews
customer focus is the only way to retain the Mail questionnaires (self-administered)
existing customers and to expand market All require a kind of sampling to select a
share. subset of the total customers.


Methods to Collect Data /2 Sampling Methods
Comparison of the data gathering techniques For any survey, the sampling design is of
Type of survey
In Person
Phone Mail
utmost importance in obtaining unbiased,
Interview - = Disadvantage
Cost -- + ++ -- = Worst representative data.
Sampling +- + -
Response Rate +- ++ -- + = Advantage 4 basic types of probability sampling:
Speed +- + - ++ = Best
Reliability ++ + -
1. Simple random sampling
++ - - +- = Could be either an
Observations
Advantage or
2. Systematic sampling
Length of Interview + - +
Exhibits + - +- Disadvantage 3. Stratified sampling
Validity ++ + +
4. Cluster sampling

Figure from Kan’s Book


1. Simple Random Sampling 2. Systematic Sampling
A sample of size n is drawn from a population in such a way that every In systematic sampling instead of using a table of random
possible sample of size n has the same chance of being selected.
To take a simple random sample, each individual in the population must
numbers, one simply goes down a list taking every kth
be listed once and only once. individual, starting with a randomly selected case among the
A mechanical procedure (i.e., using a random number table, or random first k individuals.
number generating program) is used to draw the sample. k is the ratio between the size of the population and the size
To avoid repeated drawing of the same individual, the sampling is done
without replacement. of the sample to be drawn. (1/k : sampling fraction)
On each successive draw the probability of an individual being selected Example: draw a sample of 500 customers from a
increases slightly because there are fewer and fewer individuals left population of 20,000.
unselected from the population.
k = 20000/500 = 40
If, on any given draw, the probabilities are equal of all
remaining items being selected, then we have a simple Starting with a random number between 1 and 40 (say, 18), then we
would draw every fortieth on the list (58, 98, 138, …).
random sample.


3. Stratified Sampling 4. Cluster Sampling
In a stratified sample items are classified into non- Sometimes it is much more cost effective to divide the
overlapping groups, called strata, and then simple random population into a large number of groups, called clusters,
samples are selected from each stratum. and to sample among the clusters.
The strata are usually formed based on important variables A cluster sample is a simple random sample in which each
pertaining to the parameter of interest. sampling unit is a cluster of elements.
Example: customers with high-speed internet systems may Usually geographical units such as cities, districts, schools,
have a set of satisfaction criteria for software products that is or work plants are used as units for cluster sampling.
different from those who have low-speed systems. A Example: if a company has many branch offices throughout
stratified sample should include customer type as one of the the world and an in-depth face-to-face interview with a
stratification variables. sample of its customers is desired, then a cluster sample
Stratified samples can be designed to yield greater accuracy using branch offices as clusters (of customers) may be the
for the same cost, or for the same accuracy with less cost. best sampling approach.


Sample Size /1 Sample Size /2
How large a sample is sufficient? Sample size for simple random sampling:
The answer depends on the confidence level we N x Z 2 × p (1 − p )
want and the margin of error we can tolerate. n=
NB 2 + ⎡ Z 2 × p (1 − p ) ⎤
⎣ ⎦
How to bind sample
size with customer N : population size
satisfaction? Z : Z statistic from normal distribution:
for 80% confidence level, Z = 1.28
Larger for 85% confidence level, Z = 1.45
for 90% confidence level, Z = 1.65
sample for 95% confidence level, Z = 1.96
Smaller size Higher
error level of p : estimated satisfaction level
margin confidence B : margin of error.


Sample Size /3 Example
Sample sizes for 10,000 customers for Suppose that you are going to design a questionnaire to be
sent to customers who bought your software product. The
80% - 95% levels of confidence (Z) and population size is 10,000 customers and margins of error is
5% and 3% margin of error (B) supposed to be 5%.
a) Calculate the sample size for the expected customer
satisfaction of 90% and confidence factor of 80%.
Larger sample size

Expected 80% 85% 90% 95% N = 10,000
Z = 1.28 for 80% confidence level
Customer confidence confidence confidence confidence p = 0.9
Satisfaction (p) ±5% ±3% ±5% ±3% ±5% ±3% ±5% ±3% B = 0.05
80% 104 283 133 360 171 462 240 639
N Z 2 × p (1 − p ) 10, 000 × (1.28 ) × 0.9 × 0.1
2
85% 83 227 106 289 137 271 192 516
n= =
N B 2 + ⎡ Z 2 × p (1 − p ) ⎤ 10, 000 × ( 0.05 ) + (1.28 ) × 0.9 × 0.1
90% 50 161 75 206 97 265 135 370 2 2

95% 31 86 40 110 51 142 72 199 ⎣ ⎦
Smaller n = 59
Sample size


Example (cont’d) Analyze Satisfaction Data
b) Suppose that you change the sample size to n=136 The five-point satisfaction scale (very satisfied, satisfied,
and the other parameters except the confidence level neutral, dissatisfied, and very dissatisfied) is often used in
remain unchanged. Calculate the new confidence customer satisfaction surveys.
level. Percent satisfied and percent dissatisfied are common
metrics.
N = 10,000 p = 0.9
n = 136 B = 0.05 Satisfaction data is collected for the whole software as well
Z=? as its various attributes, such as CUPRIMDA
N Z 2 × p (1 − p ) 10, 000 × Z 2 × 0.09 U: usability P: performance
136 = =
N B 2 + ⎡ Z 2 × p (1 − p ) ⎤ 10, 000 × ( 0.05 ) + Z 2 × 0.09
2
⎣ ⎦ R: reliability I: installability
3400 M: maintainability D: documentation
Z2 = = 3.829 Z = 1.96
887.76 A: availability C: capability
Confidence level is 95% for this value of Z


Example Example: Collected Data
Analyzing the relationship between satisfaction Correlation Matrix, Means, and Standard Deviations
level with specific attributes and overall satisfaction Overall U P R I M D A
for a software product. U (usability) 0.61
P (performance) 0.43 0.46
Overall customer satisfaction is the dependent R (reliability) 0.63 0.56 0.42
variable, and satisfaction levels with CUPRIMDA I (installability) 0.51 0.57 0.39 0.47
M (maintainability) 0.40 0.39 0.31 0.40 0.38
are the independent variables. D (documentation) 0.45 0.51 0.34 0.44 0.45 0.35
A (availability) 0.39 0.39 0.52 0.46 0.32 0.28 0.31
The purpose of the analysis is to determine the
priority for improvement by assessing the extent to Mean 4.20 4.18 4.35 4.41 3.98 4.15 3.97 4.57
which each of the UPRIMDA parameters affects Standard deviation 0.75 0.78 0.75 0.66 0.90 0.82 0.89 0.64
% Satisfaction 85.5 84.1 91.1 93.8 75.3 82.9 73.3 94.5
overall customer satisfaction.
Lowest satisfaction

Example from Kan’s Book


Example: Statistical Analysis Example: Visualize Results
Using regression analysis Variable Regression Odds
one can determine the
R (reliability)
Coefficient
1.216
Ratio
11.4
Priority
degree of contribution of U (usability) 0.701 4.1
each attribute (i.e., A (availability) 0.481 2.6
regression ratio) to the I (installability) 0.410 2.3
M (maintainability) 0.376 2.1
overall customer P (performance) 0.321 1.9
satisfaction. D (documentation) 0.164 1.4

Odds ratio indicates
relative importance of the Candidates for
attribute in the model. further
improvement

Example from Kan’s Book Example from Kan’s Book


Example: Improvement Priority Question: How Good is Good Enough?
How to determine the priority of improvement How much customer satisfaction is good
among the specific quality attributes?
enough?
Determine the order of significance of each quality
attribute on overall satisfaction by statistical modeling Should my company invest $2,000,000 to
(such as the regression model in the example).
improve satisfaction from 85% to 90%?
Plot the coefficient of each attribute from the model (Y-
axis) against its satisfaction level (X-axis). Given that my company’s customer
Use the plot to determine priority by: satisfaction is at 95%, should I invest another
Going from top to bottom, and
Going from left to right, if the coefficients of importance have million dollars to improve it further or should
the same values.
I do this later?


Babich’s Study /1 Babich’s Study /2
Babich (1992) Model based on a simplified model of A : number of A customers
customer satisfaction and market share that contains only
three companies: A, B, and C. When customers are B : number of B customers
dissatisfied with company A, they choose company B or C, C : number of C customers
etc.
⎛ At ⎞ ⎛ At ⎞ ⎛ At ⎞
G : number of new customers to market
At +1 = At (1 − x ) + Bt y ⎜ ⎟ + Ct z ⎜ ⎟+G⎜ ⎟
⎝ At + Ct ⎠ ⎝ At + Bt ⎠ ⎝ At + Bt + Ct ⎠ x : dissatisfaction level with A products
⎛ Bt ⎞ ⎛ Bt ⎞ ⎛ Bt ⎞ y : dissatisfaction level with B products
Bt +1 = Bt (1 − y ) + At x ⎜ ⎟ + Ct z ⎜ ⎟+G⎜ ⎟
⎝ Bt + Ct ⎠ ⎝ At + Bt ⎠ ⎝ At + Bt + Ct ⎠ z : dissatisfaction level with C products
⎛ Ct ⎞ ⎛ Ct ⎞ ⎛ Ct ⎞
Ct +1 = Ct (1 − z ) + At x ⎜ ⎟ + Bt y ⎜ ⎟+G⎜ ⎟ t : time
⎝ Bt + Ct ⎠ ⎝ At + Ct ⎠ ⎝ At + Bt + Ct ⎠


Babich’s Study /3 Babich’s Study /4
Market shares of the three If the satisfaction levels
companies assuming of companies B and C
satisfaction levels of 95%, were 98% and 99%,
91%, and 90% for A, B, respectively, and
and C, respectively, over a
number of time periods and company A’s
equal initial market share. satisfaction level
After 12 time periods the remained at 95%,
95% satisfaction product company A’s product
(company A) would would have less than
basically own the market. 10% market share in 24
time periods.

Figure from Kan’s Book Example from Kan’s Book


Answer: How Good is Good Enough? References
From Babich’s simple model and examples the Stephen H. Kan, “Metrics and Models in
answer is:
Software Quality Engineering,” 2nd Edition,
You have to be better than your competitors!
Addison-Wesley, 2002.
Therefore, it is important to measure not only one’s P. Babich, “Customer Satisfaction: How
customer satisfaction level, but also the competitors Good Is Good Enough?” Quality Progress,
satisfaction level. December 1992, pp. 65-67.
Indeed, many companies have been doing exactly
that.


Reliable Software
Reliable software products are those that run
Software Quality: correctly and consistently, have few
Software Quality Assurance remaining defects, handle abnormal situation
(SQA) properly, and need few installation effort.
Developing reliable software requires:
Devising Software Quality Assurance
(SQA) program
Establishing Software Reliability
Engineering (SRE) process

far@ucalgary.ca 90

Software Quality Assurance SQA: Definition
Software quality Assurance (SQA) is a planned and SQA is the planned and systematic set of
systematic approach to ensure that both software process and
software product conform to the established standards, activities that ensure that both software
processes, and procedures. process and product conform to
The goals of SQA are to improve software quality by requirements, standards, and procedures.
monitoring both software and the development process to
ensure full compliance with the established standards and Process includes all activities involved in
procedures. designing, coding, testing and maintaining.
Steps to establish an SQA program Product includes software, associated data,
Get the top management’s agreement on its goal and support.
Identify SQA issues, write SQA plan, establish standards and SQA documentation, and all supporting and
functions, implement the SQA plan and evaluate SQA program. reporting paperwork.


SQA: Role SQA: Goals
The role of SQA is to give management the assurance that The goals of SQA is to reduce the risks by:
the officially established process is actually being
implemented. Appropriately monitoring the software and the
development process.
SQA ensures that:
An appropriate development methodology is in place. Ensuring full compliance with standards and procedures
The projects use standards and procedures in their work. for software and process.
Reviews and audits are conducted. Ensuring that inadequacies in product, process, or
Documents are produced to support maintenance and enhancement. standards are brought to management’s attention so that
Software configuration management is set up to control change.
they can be fixed.
Testing is performed and passed.
Deficiencies and deviations are identified, documented and brought SQA is not responsible for producing quality products.
to management’s attention. It is responsible for auditing the quality actions and for
alerting management to any deviations.


SQA: Responsibilities Establishing SQA Program
To achieve its goals, SQA is responsible for: How to establish SQA program?
Review all development and quality plans for completeness.
Participate as inspection moderators in design and code
inspections. 1. Creating SQA Plan
Review all test plans for adherence to standards. Documentation
Review samples of all test results to determine adherence to Standards and Procedures
plans.
Periodically audit the performance to determine adherence to 2. Establishing SQA Activities
standards.
Participate in all project phase reviews and write down any
3. Project specific concerns
nonconformance.


Creating SQA Plan /1 Creating SQA Plan /2
SQA plan specifies its goals, tasks to be performed, IEEE standard for SQA plan preparation contains
and the standards and procedures against which the the following (contd.):
development work is to be measured. Software Configuration Management
IEEE standard for SQA plan preparation contains Problem Reporting and Corrective Action
the following: Tools, Techniques, and Methodologies
Purpose Code Control
Reference Documents Media Control
Management Supplier Control
Documentation Records Collection, Maintenance, and Retention
Standards, Practices, and Conventions
Reviews and Audits


SQA: Documentation /1 SQA: Documentation /2
Software verification and validation plan: describes the
Documentation step describes the documents methods used to verify that the requirements are
to be produced and how it is to be previewed. implemented in the design, that the design is
implemented in the code, and that the code meets the
Documentation includes: requirements.
Software verification and validation report: reports on
Software requirement specification: specifies the SQA verification and validation activities.
each software function, performance parameter, User Documentation: describes the required installation,
interface, or other attribute with sufficient operation, and maintenance of the software.
Other documents: include software development plan,
precision to permit its verification. software configuration management plan, standards and
Software design description: describes the major procedures manual.
Review plan: describes the planned review methods.
components, databases, and internal interfaces.


SQA: Standards & Procedures SQA: Standards & Procedures
Standards are the criteria to which software Minimum requirement for standards include:
Documentation Standards: specify form and content for
products are compared (i.e., standards define planning, control, and product documentation and
what should be done). provide consistency throughout a project.
Design Standards: specify the form and content of the
Procedures are the criteria to which design product. They provide rules for translating the
development and control processes are software requirements into the software design and for
representing it in the design documentation.
compared (i.e., procedures define how the
Code Standards: specify the language in which the code
work is actually to be done, by whom, when is to be written and define any restrictions on use of
and what is done with the results). language features. They define legal language structure,
style conventions, rules for data structures, and internal
code documentation.


SQA: Activities SQA: Project Specific Concerns
SQA activities include: Each project has its specific attributes and
Production evaluation is to ensure that standards are SQA program should be tailored to
followed. It assures that clear and achievable standards
exist and evaluate compliance of software product with
accommodate to the project needs.
the standards. Common project specific characteristics are:
Process monitoring is to ensure that appropriate steps to Mission critical of project
carry out process are being followed. SQA monitors
processes by comparing actual steps performed with Schedule and budget
established procedures. Size and complexity of project
Audit looks at a process or product in depth, comparing Size and complexity of project staff organization
them with established standards and procedures.


SQA: Project Specific Concerns SQA: Other Considerations
The relationship between SQA program and mission critical To make SQA program successful, the first step is
level is straight forward. The more critical the project, the to get senior management commit to it. Without
more important and formal the SQA. management support, SQA can not be effective.
The relationship between SQA and budget and schedule is
not so intuitive; the tighter the budget and schedule, the more Even with senior managers support experienced
critical it is to have a well planned and effective SQA SQA staff is a necessity.
program. The hardest problems the software managers face is
The project organization structure influences the SQA to get experiences staff into SQA. Good engineers
program. are usually preferred to stay with the development
For large or dispersed staff, more formal SQA program is required.
A small, centralized development staff can easily inform each other
group. Usually, software development transfers its
of the nonconformance and helping each other in meeting standards. poorer performers to SQA and not taking them
back. Rotation scheme may be effective.


Seng421 08 handout

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Viewers also liked

Viewers also liked (6)

Similar to Seng421 08 handout

Similar to Seng421 08 handout (20)

Recently uploaded

Recently uploaded (20)

Seng421 08 handout