Chapter 8 software quality assurance and configuration audit
CHAPTER 8: SOFTWARE QUALITY ASSURANCE AND CONFIGURATION
8.1 Quality Assurance Over-view
What is quality assurance
- is the forecasting and prevention of quality related problems.
o Every effort is taken to meet the customer quality requirements. Examples of quality
assurance tools include:
Policies and procedures
Periodic checking and rechecking
Examples of quality control tools include:
Software Quality Assurance
♦ Software quality can be defined as:
♦ Conformance to explicitly-stated functional and performance requirements, explicitly-
documented development standards, and implicit characteristics that are expected of all
professionally developed software.
♦ This definition thus emphasizes on three important points:
• Quality will be measured with Software Requirements, i.e. lack of conformance to
these requirements will mean lack of quality.
• Specified standards define a set of development criteria that guide the manner in
which software is engineered. If the criteria is not followed, lack of quality will result.
• The set of implicit requirements, e.g. good maintainability, must still be followed. If
these are not met, software quality is suspect.
• Software quality assurance(SQA)
- is a planned systematic pattern for all actions necessary to provide adequate
confidence that the product, or process by which the product is developed, conforms to
- The primary goal of software quality assurance is to enhance the quality of
software. To this end, a new concept, Clean Room engineering, is evolving where
by a software is developed to assure quality.
♦ Software quality is thus a mix of factors that will vary across different applications and
different customers. The sections below will thus be concerned about:
• Identification of Software Quality Factors
• Human activities required to achieve them
Quality management activities
1 Quality assurance
Establish organisational procedures and standards for quality.
is concerned with the forecasting and prevention of quality problems. It is pro-active.
2 Quality planning
Select applicable procedures and standards for a particular project and modify these as
3 Quality control
Ensure that procedures and standards are followed by the software development team.
is concerned with detecting and reporting quality-related problems. It is reactive.
Quality management should be separate from project management to ensure independence.
8.2 Software Quality Factors
♦ In order to help us categorize software quality factors, McCall proposes a
categorisation which focuses on three important aspects of a software product:
• Product Operations: A software product's operational characteristics;
• Product Reunion: Its ability to undergo change,
• Produced Transition: Its adaptability to new environments.
8.2.1 Product Operations
• Does it do what I want?
- The extent to which a program satisfies its specification and fulfils the mission
• Does it do it accurately all the time?
- The extent to which a program can be expected to perform its intended
function with required precision.
• Will it run on my hardware as well as it can?
- The amount of computing resources and code required by a program to
perform a function.
• Is it secure?
- The extent to which access to software or data by unauthorised persons can be
• Is it designed for the user?
- The effort required to learn, operate, prepare input, and interpret output of a
8.2.2 Product Revision
• Can I fix it?
- The effort required to locate and fix an error in a program.
• Can I change it?
- The effort required to modify an operational program.
• Can I test it?
- The effort required to test a program to ensure that it performs its intended
8.2.3 Product Transition
• Will I be able to use it on another machine?
- The effort to transfer the program from one hardware and/or software system
environment to another.
• Will I be able to reuse some of the software?
- The extent to which a program (or parts of a program) can be reused in other
applications-related to the packaging and scope of the functions that the
• Will I be able to interface it with another system?
- The effort required to couple one system to another.
8.3 Software Quality Assurance Major Activities
♦ Software Quality Assurance (SQA) is a planned and systematic pattern of actionsª
that are required to ensure quality in software.
• The SQA group in any company serves as the customer's in-house representative. That is,
the people who perform SQA must look at the software from the customer's point of
Does the software adequately meet the quality factors from the customer's point of
Does the software adequately meet the quality factors described earlier?
Has software development been conducted according to pre-established standards?
Have technical disciplines properly performed their roles as part of the SQA activity?
8.3.1 SQA Activities
♦ SQA is comprised of a variety of tasks associated with seven major activities:
• Application of Technical Methods
- SQA begins with a set of technical methods and tools that help the analyst to achieve
a high-quality specification and the designer to develop a high-quality design.
• Conduct of Formal Technical Review
- Activity that accomplishes quality assessment for the specification and the design.
The FTR is a stylised meeting conducted by technical staff with the sole purpose of
uncovering quality problems.
• Testing of Software
- Combines a multistep strategy with a series of test case design methods that help
ensure effective error detection.
• Enforcement of standards
- Degree in which this is applied varies from company to company. In many cases,
standards are dictated by customers or regulatory mandates. In other situations,
standards are self-imposed.
• Control of Change
- Every change to software has the potential of introducing errors or creating side
effects that propagate errors. The change control process thus contributes directly to
software quality by formalising requests for change, evaluating the nature of change,
and controlling the impact of change.
- Track software quality and assess the impact of methodological and procedural
changes on improved software quality.
• Record keeping and recording
- Collection and dissemination of SQA information. The results of reviews, audits,
change control, testing, and other SQA activities must become part of a historical
record for a project and should be disseminated to the development staff on a need-to-
8.4 Formal Technical Reviews
♦ Formal technical review is
• A class of reviews that include walkthroughs, inspections, round-robin reviews, and other
small group technical assessments of software
• A planned and controlled meeting attended by a group of diversified people
8.4.1 Objectives of FTR
♦ To uncover errors in function, logic, or implementation for any representation of the
♦ To verify that the software under review meets its requirements
♦ To ensure that the software has been represented according to predefined standards
♦ To achieve software that is developed in a uniform manner
♦ To make projects more manageable
8.4.2 Effects of FTR
♦ Early discovery of software defects so the development and maintenance phase is
♦ Serves as a training ground, enabling junior engineers to observe different approaches
to software analysis, design, and implementation
♦ Serves to promote backup and continuity because a number of people become
familiar with parts of the software that they may not have otherwise seen
8.4.3 Guidelines for the Organisation and Preparation of FTR
♦ Should involve between three and five people
♦ Advance preparation should occur but should not require no more than 2 hours of
work for each person
♦ The duration of the review meeting should be less than 2 hours
♦ Focus on a components of the software (eg. a portion of requirements specification, a
detailed module design, a source code listing for a module)
♦ Producer should report his/her progress
♦ A recorder should actively record all issues as
• What was reviewed?
• Who reviewed it?
• What were the findings and conclusions?
♦ The attenders must make a decision at the end of the session as to
• Accept the product
• Reject the product
• Accept the product provisionally, subject to further modification
8.4.4 Review Guidelines during the FTR
♦ Review the product, not the producer
• Tone of the meeting should be loose and constructive, and intent should not be to
embarrass or belittle.
♦ Set an agenda and maintain it
• One main problem in all types of meetings is the tendency to drift. The FTR must
be kept on track and on schedule.
♦ Limit debate and rebuttal
• There may not be universal agreement on some issues. Rather than spending time
debating the issue, the issue should be recorded for further discussion off-line.
♦ Enunciate problem areas, but don't attempt to solve every problem noted
• A review is not a problem-solving session. The solution of a problem can often be
accomplished by the producer alone. Problem solving should be postponed until
after the review meeting.
♦ Take written notes
• Makes notes on a wall board so that wording and prioritisation can be assessed by
the other reviewers as information is recorded.
♦ Limit the number of participants and insist upon advance preparation
• Keep the number of people involved to the necessary minimum.
♦ Develop a checklist for each product that is likely to be reviewed
• A checklist helps the review leader to structure the FTR meeting and helps each
reviewer to focus on important issues. Checklists should be developed for
analysis, design, code and even test documents.
♦ Allocate resources and time schedule for FTRs
• For reviews to be effective, they should be scheduled as tasks during the software
♦ Conduct meaningful training for all reviewers
• To be effective, all review participants should receive some formal training. The
training should stress both process-related issues and the human psychological
side of reviews.
♦ Review your early reviews
• Debriefing can be beneficial in uncovering problems with the review process
8.5 Software Reliability
♦ Most hardware-related reliability models are predicated on failure due to wear rather
than failure due to design defects. In hardware, failures due to physical wear (e.g. the effects
of temperature, corrosion, shock) are more likely than a design-related failure. The opposite
is true for software: in fact, all software failures can be traced to design or implementation
problems; wear does not enter into the picture.
♦ Software Reliability: The probability of failure free operation of a computer program
in a specified environment for a specified time.
♦ A simple measure of reliability is mean time between failure (MTBF), where
MTBF = MTTF + MTTR
♦ In addition to this reliability measure, a measure of availability can be calculated.
♦ Software availability is the probability that a program is operating according to
requirements at a given point in time and is defined by:
8.6 Software Quality Assurance Approach
♦ At the low end of the scale, quality is the sole responsibility of the individual who
may engineer, review, and test at any comfort level. At the high end of the scale, an SQA
group is charged with the responsibility standards and procedures for achieving software
quality and ensuring that each is followed.
♦ Before formal quality assurance procedures are instituted, a software development
organisation should adopt software engineering procedures, methods, and tools. This
methodology, when combined with an effective paradigm for software development, can do
much to improve the quality of all the software produced by the organisation.
♦ Manager/managers and practitioners are not interested in establishing formal SQA
• Managers are reluctant to incur the extra up-front cost
• Practitioners feel they are already doing everything that needs to be done
• No one knows where to put such a function organisationally
• Everyone wants to avoid the "red tape" that SQA is preceived to introduce
MTTF + MTTR
8.6.1 Benefits of SQA
♦ Software will have fewer latent defects, resulting in reduced effort and time spent
during testing and maintenance
♦ Higher reliability will result in greater customer satisfaction
♦ Maintenance costs (a substantial percentage of all software costs can be reduced)
♦ Overall life cycle cost of software is reduced
8.6.2 Constraints of SQA
♦ Diffcult to institute in small organisations, where available resources to perform the
neccessary activities are not available
♦ SQA represents cultural change - and change is never easy
♦ SQA required the expenditure of dollars that would not otherwise be explicitly
budgeted to software engineering or quality assurance
Configuration management (CM) is concerned with managing evolving software systems: system
change is a team activity; CM aims to control the costs and effort involved in making changes to a
Somehow, all modifications to the software system (including all documentation, data, and
training material) must be identified, organized, and controlled.
Involves the development and application of procedures and standards to manage an evolving software
May be seen as part of a more general quality management process.
Principles of Software Configuration Management
Software configuration management has a number of component parts:
1. Configuration Identification – Knowing what items are under configuration control and how to
2. Configuration Control – How to handle Change Request, emergencies, procedures describing
how to build releases and control variants;
3. Status Accounting – Tracking the items and managing transitions between one status and
4. Configuration Auditing – Proving the above are being done properly.
When released to CM, software systems are sometimes called baselines as they are a starting point for
further development (i.e. further change).
A "baseline" is A specification or product that has been formally reviewed and agreed upon, that
thereafter serves as the basis for further development, and that can be changed only through formal
change control procedures.
The sequence of events for a change might look something like:
ι. Someone identifies a change they feel is needed (e.g. adaptive/corrective/enhancement
maintenance) and generates a change request
ιι. The request is evaluated by a member of the development group and generates a change report
ιιι. The change report is evaluated by the change control authority (e.g. project admin, or task leader),
and either the requested change is denied (and the initiator informed) or is approved
The rest of this sequence assumes the change request has been approved
a. Individuals are assigned to those objects under configuration control which must be
b. The individuals "check out" the objects (so no one else can alter the objects at the
c. The changes are applied
d. The changes are reviewed/audited
e. The objects are "checked in"
f. The changes are tested
g. The software is rebuilt if appropriate
h. The cumulative set of changes to all the configuration objects is reviewed/audited
i. The changes are included in the new version
j. The new version is distributed
Some of the items that may fall under formal change control include
The project plan
The requirements and specifications document(s)
The user manual(s)
The design specification document(s)
The source code listings
The test plan, test cases, and recorded results
The operation and installation manual(s)
The executable program
The database description document(s)
The bug reports, maintenance requests, and change orders
The standards and procedures documents
Reviews/audits are geared towards the following
Ensuring the requested change has been made, and that only the requested change has been
Ensuring a formal technical review has assessed the change
Ensuring that all the defined standards have been followed
Ensuring that the change has been clearly identified in the appropriate documentation and all
appropriate configuration items updated
Ensuring that the change information (author, date, etc) has been properly recorded
CM should always be based on a set of standards which are applied within an organisation.
Applied standards will cover both the process followed and the product resulting (i.e.
standards regarding methods, and standards regarding things like document/code formats)
CM standards should define how items are identified, how changes are controlled and how
new versions are managed.
These may be based on external CM standards (e.g., IEEE standard).
The most common existing cm standards are based on a waterfall model, effective new
standards are needed for other techniques such as evolutionary development.
Document standards might specify the sections which must be included in the document, content items
that must be addressed in each section, the order of sections, forms which must be included, the
document layout (fonts, page formats, footnote and bibliography styles), the support tools used (e.g.
specific word processing packages, spell checkers, etc)
Configuration management planning
All products of the software process may have to be managed:
o test data,
o user manuals.
Thousands of separate documents are generated for a large software system.
Starts during the early phases of the project.
Must define the documents or document classes which are to be managed.
Documents which might be required for future system maintenance should be identified and
specified as managed documents.
Configuration item identification
Large projects typically produce thousands of documents which must be uniquely identified.
Some of these documents must be maintained for the lifetime of the software.
Document naming scheme should be defined so that related documents have related names.
A hierarchical scheme with multi-level names is probably the most flexible approach.
All CM information should be maintained in a CM database.
Should allow queries about configurations to be answered. Who has a particular system
version? What platform is required for a particular version? What versions are affected by a
change to component X? How many reported faults in version T?
Record of changes applied to a document or code component.
Should record, in outline, the changes made, the rationale for the change, who made the
change and when it was implemented.
May be included as a comment in code. If a standard prologue style is used for the derivation
history, tools can process this automatically.
Version and release management
CM should allow a user to specify a variety of possible configurations of the software by
selecting from different available versions - each supported by all the appropriate tools and
Invent identification scheme for system versions.
Plan when new system version is to be produced.
Ensure that version management procedures and tools are properly applied.
Plan and distribute new system releases.
Version: an instance of a system which is functionally distinct in some way from other
Variant: an instance of a system which is functionally identical but non-functionally distinct
from other instances of a system.
Release: an instance of a system which is distributed to users outside of the development
Attributed version identification
Attributes can be associated with a version with the combination of attributes identifying that
Examples of attributes: Date, Creator, Programming Language, Customer, Status, etc.
More flexible than an explicit naming scheme for version retrieval, can cause problems with
Needs an associated name for easy reference.
Risk and risk management
Because software projects represent a substantial business investment, and because there is tremendous
potential for cost and schedule overruns or failed delivery of requirements, risk management is an
extremely important aspect of software engineering projects
We can consider five steps in risk management:
1. Establish a framework for categorizing types of risk and severity of risks
2. Evaluate the risks associated with the project
3. Establish means to avoid the assessed risks, with emphasis on the risks which are most
likely to occur and/or have the most severe impact
4. Establish metrics which can be used to monitor the increasing/decreasing potential for
different risks as the project progresses
5. Establish contingency plans for mitigating risks should they actualize during the project
Not all risks can be planned for, but sound risk management strategies can salvage many projects that
would be doomed otherwise
1. Frameworks for categorizing risks
We can categorize risks by their severity, e.g.: catastrophic, critical, marginal, or negligible
We can also categorize risks by the aspect of the project they impact, e.g.: system performance, product
support, cost/budget, and schedule
A simple table helps ensure everyone has similar ideas about the severity classification of different types
of risk, e.g.:
Risk type: Performance Support Cost Schedule
Unable to meet core
Product will be
Delivery date in
Potential for early
2. Evaluation of specific risks
We now try to categorize all forseeable risks. For each risk identified, establish a rough probability, and
classify the severity of its impact.
E.g. We estimate there is a 40% chance that upper management will re-assign two of our key
personnel because of problems on another project.
This would have a critical impact on both the product and the schedule, due to loss of their
Some of the common risks, and factors affecting their severity and probability, are outlined below.
1. Organizational risks
a) The strength of organizational committment to the project in terms of budget,
staffing/resources, decision-making, has a substantial impact on risk.
What is the likelihood of staff/resource/funding cutbacks or delays, and what is the
impact of each?
Factors to consider here include:
the potential impact of the product on company revenue,
the visibility of the product to senior management,
the regulatory constraints on the project,
the costs of late or incomplete delivery,
management awareness of these costs and constraints, etc.
b) What is the likelihood of delays/technical deficiencies due to staff instability - i.e.
are the best people available?
will they remain available?
do they have appropriate training?
2. Customer-related risks
a. Lack of support, or actual obstruction, from the customer can also represent a
substantial risk to software projects.
In assessing the customer-related risks for a project, the factors to consider include:
i. Does the customer understand the technical aspects of their requests and
the software development process?
ii. Is the customer willing to participate in the review and evaluation process
and give timely feedback to the software team?
iii. Does the customer have a clear idea of their requirements?
Your confidence in assessing these factors will also differ depending on how close and
how frequent your previous contact with the customer has been.
A lack of familiarity means the risk factor is higher (since there is less confidence in
original assessments of the nature of the customer)
3. Risks associated with the development process
b. The effectiveness and appropriateness of the development process obviously has a
significant impact on the successful and timely completion of the project.
Factors to consider when evaluating process-associated risks include:
i. Is the process clearly specified, in writing, for the entire team,
management, and the clients?
ii. Are the team members, management, and the client genuinely
willing/able to follow the process
iii. Does the process account for regular reviews, standards, and CM?
iv. Is there adequate software tool support for the process?
c. Delays or technical deficiencies can be a result of using new or unusual
development processes. The risk assessment should thus consider
i. Do the requirements call for unconventional development methods?
ii. Do the customer requirements call for new (to the team and/or
organization) analysis, design, or test methods?
4. Risks associated with the product
a. Product size can be estimated in many ways:
ii. number of files,
iii. number of transactions,
iv. size of database,
v. number of users, etc
What is the likelihood that the end product will be catastrophically larger than the
What would be the impact on performance/support/cost/schedule?
Influencing factors to consider include
i) the stability of requirements,
ii) the amount of re-use expected,
iii)the level of technical experience on the part of the original estimators, etc.
b. How stable are the product requirements?
I.e. what is the risk of substantial requirements changes forcing substantial product
changes and redevelopment?
In part this depends on the nature of the product and its dependence on the external
In part this depends on the clarity, focus, and consistency of the customer's view of the
c. What are the risks with respect to new/unfamiliar techniques/technology?
Factors to consider include:
i. is the technology new to your organization and/or team members?
ii. Does the product interact with new/unproven hardware or software?
iii. Is a specialized/unconventional user interface required?
iv. How restrictive are the performance constraints?
3. Risk avoidance
Once risks have been identified, the team should evaluate which risks most need attention - based on
probability of occurence and severity of impact
The team then develops strategies to help avoid the specific risks targetted - this will be very risk-
• If staff turnover leading to delayed production is identified as a key risk, possible avoidance
o Reducing turnover through efforts to improve staff morale
improved working conditions,
listening to staff and acting on feedback
o Taking measures to reduce lost time when/if turnover does occur:
developing training plans to bring new personnel up to speed quickly,
ensuring that for each key area at least two people have a working understanding
of the material
ensuring that documentation is clear, thorough, and up-to-date
4. Risk monitoring
Once a risk has been identified, we can begin monitoring relevant information to give us ample warning
if the risk shows signs of materializing
• Considering the staff turnover risk again, relevant data to monitor might include:
o Monitoring the availability of jobs in other areas of the company or outside the company
(i.e. what is the potential for turnover)
o Monitor the typical conditions/salaries of competing jobs
o Monitor the general attitude of team members, particularly in response to project
pressures and personal interactions
5. Contingency planning
Assuming efforts to avoid the risk have failed and the risk has become a reality, contingency planning is
needed to counteract the effects of the risk
• Continuing the staff turnover example, assume key people have left the project (and that we've
followed our "avoidance" policies above)
o since at least two people have been associated with each critical area, there is a back-up
person to move into the vacated area
o since there is adequate training material and documentation, a new person can be brought
into the project and brought up to speed in reasonable time
o assuming the departing personnel have given some advance notice, they spend their final
time documenting key information their replacement must learn, and/or training that