In practice Quality of a system can be a vague, undefined, attribute.
We therefore need to define precisely what qualities we require of a system.
For any software system, there should be three specifications:
• A functional specification describing what the system is to do
• A quality (or attribute) specification concerned with how well the functions are to operate;
• A resource specification concerned with how much is to be spent on the system.
To identify specific product qualities that are appropriate to software, James A McCall grouped
software qualities into three sets of quality factors
• Product operation quality factors
• Correctness: Traceability, consistency, completeness
• Reliability: Error tolerance, consistency, accuracy
• Efficiency: Execution and storage efficiency
• Integrity: Access control, access audit
• Usability: Operability, training, communicativeness, Input / Output volume,
Input / Output rate
• Product revision quality factors
• Maintainability: Consistency, simplicity, modularity, self-descriptiveness
• Testability: Simplicity, modularity, instrumentation , self-descriptiveness
• Flexibility: Modularity, generality, expandability , self-descriptiveness
• Product transition quality factors
• Portability: Modularity, self-descriptiveness, machine independence,
software system independence
• Reusability: Generality, modularity, software system independence,
machine independence, self-descriptiveness
• Interoperability: Modularity, communications commonality, data
This standard was published to tackle the question of the definition of software quality.
ISO 9126 identifies six software quality characteristics:
• Functionality: suitability, accuracy, interoperability (interaction with other systems),
• Reliability: maturity (in terms of failure in system), fault tolerance, recoverability
• Usability: understandability, learnability, operability
• Efficiency: time behavior, resource behavior
• Maintainability: analyzability, changeability, stability, testability
• Portability: adaptability, installability, conformance, replaceability
Techniques to help enhance Software Quality
• Increasing visibility
• Procedural structure
• Checking immediate stages through inspections
Process and Product Quality
The quality of a developed product is influenced by the quality of the production process.
This is important in software development as some product quality attributes are hard to
assess. However, there is a very complex and poorly understood relationship between
software processes and product quality.
• There is a straightforward link between process and product in manufactured goods.
• More complex for software because:
– The application of individual skills and experience is particularly important in
– External factors such as the novelty of an application or the need for an accelerated
development schedule may impair product quality.
• Care must be taken not to impose inappropriate process standards - these could reduce
rather than improve the product quality.
Quality management activities
• Quality assurance
– Establish organisational procedures and standards for quality.
• Quality planning
– Select applicable procedures and standards for a particular project and modify these
• Quality control
– Ensure that procedures and standards are followed by the software development
• Quality management should be separate from project management to ensure
Software quality assurance (SQA) consists of a means of monitoring the software engineering
processes and methods used to ensure quality. It does this by means of audits of the quality
management system under which the software system is created. These audits are backed by one or
more standards, usually ISO 9000 or CMMI.
Typical quality assurance activities include:
Establishing a quality management system
Establishing mechanisms for quality assurance
Providing guidance to project managers
Effecting continuous improvement
Software Quality Assurance encompasses the entire software development process, which includes
processes such as software design, coding, source code control, code reviews, change management,
configuration management, and release management. Whereas software quality control is a control
of products, software quality assurance is a control of processes.
It is distinct from software quality control which includes reviewing requirements documents, and
Cost of quality can be divided into the following groups:
• Preventive cost
• Detection cost
• Internal failure cost
• External failure cost
• Start-up costs
• Metrics identification
• Project related costs
• Quality plan
• Quality assurance support
• Supervision cost
• Quality control costs e.g. reviews, testing
• Continuing costs
• Staff training
• Standards maintenance
• Metrics maintenance
Quality assurance benefits
• Reduced costs
• Greater efficiency
• Better performance
• Less unplanned work
• Fewer disputes
• Improved visibility and predictability
• Reduced risk
• Problems show up earlier
• Better quality
• Improved customer confidence
• Portable and reusable product
• Better control over contracted products
As per IEEE, a quality plan might have entries for:
• Purpose: scope of plan;
• List of references to other documents;
• Management , including organization, tasks and responsibilities;
• Documentation to be produced;
• Standard practices and conventions;
• Reviews and audits;
• Testing and configuration management;
• Problem reporting and corrective action;
• Tools techniques and methodologies;
• Code, media and supplier control;
• Records collection, maintenance and retention;
• Risk management
Quality Review involves checking the software development process to ensure that procedures and
standards are being followed. There are two approaches to quality control
– Quality reviews;
A group of people carefully examine part or all of a software system and its
associated documentation. Code, designs, specifications, test plans,
standards, etc. can all be reviewed. Software or documents may be 'signed
off' at a review which signifies that progress to the next development stage
has been approved by management.
• Review functions
Quality function - they are part of the general quality management process.
Project management function - they provide information for project
managers. Training and communication function - product knowledge is
passed between development team members.
• Review results
Comments made during the review should be classified
No action. No change to the software or documentation is required;
Refer for repair. Designer or programmer should correct an identified fault;
Reconsider overall design. The problem identified in the review impacts
other parts of the design. Some overall judgement must be made about the
most cost-effective way of solving the problem;
Requirements and specification errors may have to be referred to the client.
– Automated software assessment and software measurement.
Software Quality Management System
A quality management system (quality system) is the principal methodology used by organizations to
ensure that the products they develop have the desired quality. It consists of the following:
• Managerial structure and individual responsibilities. It is the responsibility of the
organization as a whole. It should have the support of the top management.
• Quality system activities include
• Auditing of the projects
• Review of the quality system
• Development of standards
• Production of reports for top management
Quality Assurance and Standards
• Standards are the key to effective quality management.
• They may be international, national and organizational or project standards.
• Product standards define characteristics that all components should exhibit e.g. a common
• Process standards define how the software process should be enacted.
Importance of Standards
• Encapsulation of best practice- avoids repetition of past mistakes.
• They are a framework for quality assurance processes - they involve checking compliance to
• They provide continuity - new staff can understand the organisation by understanding the
standards that are used.
International standards such as ISO 9000 provide guidance on how to organize and maintain a
It specifies a set of guidelines for repeatable and high quality product development.
ISO 9000 and ISO 9003 apply to software organizations.
Benefits of ISO 9000 Certification
• Confidence of customers
• Well-documented software
• Makes the process focused, efficient and cost-effective
• Weak points of the organization are identified and remedial action implemented
• Sets the basic framework for total quality management
How to get ISO 9000 certification
• Documents review and adequacy of audits
• Compliance audit
• Continued surveillance
Summary of ISO 9001 Requirements
• Management responsibility
• Quality System
• Contract Reviews
• Design Control
• Document control
• Purchaser Supplied product
• Product Identification
• Process control
• Inspection and testing
• Inspection, Measuring and Test Equipment
• Inspection and Test Status
• Control of Nonconforming Product
• Corrective action
• Quality records
• Quality Audits
• Statistical Techniques
Contract Review: ISO 9001 requires that contracts be reviewed to determine whether the
requirements are adequately defined, agreed with the bid, and can be implemented.
Design Control: It requires that procedures to control and verify the design be established. This
includes planning design activities, identifying inputs and outputs, verifying the design, and
controlling design changes.
Document Control: The distribution and modification of documents be controlled
Purchasing: The purchased products should confirm to their specified requirements. This
includes assessment of potential sub-contractors and verification of purchased products.
Purchaser-Supplied product: It requires that any purchaser-supplied material be verified and
Product Identification and Traceability: It requires that the product be identified and traceable
during all stages of production, delivery, and installation.
Process Control: It requires that production processes be defined and planned. This includes
carrying out production under controlled conditions, according to documented instructions.
Special processes that cannot be fully verified are continuously monitored and controlled.
Inspection and Testing: It requires that incoming materials be inspected or certified before use
and that in-process inspection and testing be performed. Final inspection and testing are
performed prior to release of finished product. Records of inspection and testing are kept.
Inspection, Measuring, and Test Equipment: It requires that equipment used to demonstrate
conformance be controlled, calibrated, and maintained. When test hardware or software be
used, it is checked before use and rechecked at prescribed intervals.
Inspection and Test Status: It requires that the status of inspections and tests be maintained for
items as they progress through various processing steps.
Control of nonconforming product: It requires that nonconforming product be controlled to
prevent inadvertent use or installation.
Corrective action: It requires that the causes of nonconforming product be identified. Potential
causes of nonconforming product are eliminated; procedures are changed resulting from
Handling, Storage, Packaging, and Delivery: It requires that procedures for handling, storage,
packaging, and delivery be established and maintained.
Quality records: It requires that quality records be collected, maintained, and dispositioned.
Internal Quality Audits: It requires that audits be planned and performed. The results of audits
are communicated to management, and any deficiencies found are corrected.
Training: It requires that training needs be identified and that training be provided, since
selected tasks may require qualified personnel. Records of training are maintained.
Servicing: It requires that servicing activities be performed as specified.
Statistical Techniques: It states that appropriate, adequate statistical techniques are identified
and should be used to verify the acceptability of process capability and product characteristics.
Shortcomings of ISO 9000 Certification
• It requires a software production process to be adhered to, but does not guarantee the
process to be of high quality
• No international accreditation agency exists
• Organizations getting ISO 9000 certification often tend to downplay domain expertise.
• ISO 9000 does not automatically lead to continuous process improvement.
Capability Maturity Model
It classifies software development industry into the following five maturity models.
1. Initial: characterized by ad hoc activities
2. Repeatable: Management practices like tracking costs and schedule are established
3. Defined: The processes for both management and development activities are defined and
documented. The processes though defined, product qualities are not measured.
4. Managed: Product and process metrics are measured and used for evaluating project
performance rather than improving the process
5. Optimizing: Continuous process improvement is achieved both by carefully analyzing the
quantitative feedback from process measurements and from application of innovative ideas
Comparison between ISO 9000 and CMM
• ISO 9000 can be quoted in official • CMM is for internal purposes
documents for communication
• Applies specially to Software
with external parties
• ISO 9000 applies to a range of
• CMM aims to achieve Total Quality
• ISO aims at level 3 of CMM
• Provides a list of Key process areas
• States important standards which (KPAs), on which the organization
all organizations should maintain at any maturity level needs to
• Product based
• Process based
Six sigma at many organizations simply means striving for near perfection. To achieve six sigma, a
process must not produce more than the acceptable defects
The fundamental object of six sigma methodology is the implementation of a measurement-based
strategy that focuses on process improvement and variation reduction through the application of six
sigma improvement projects.
What is Sigma?
Sigma Level Defects Per Million Rate of
2σ 308,000 2 times
3σ 66,800 5 times
4σ 6,210 11 times
5σ 230 27 times
6σ 3.4 68 times
It has three core steps:
• Define customer requirements, deliverables, and project goals via well-defined methods of
• Measure the existing process and its output to determine current quality performance
(collect defect metrics)
• Analyze defect metrics and determine the vital few causes
It also suggests additional steps for improvement:
• Improve the process by eliminating the root cause of defects
• Control the process to ensure that future work does not reintroduce the causes of defects
Six Sigma Project Methodology
Applying Six Sigma to software development makes product development and other projects
transparent to both management and customers. Transparency requires an important cultural
change. As a result, after transparency is achieved, completing accurate project estimations while
meeting both deadlines and customer requirements becomes a lot easier.
There is no SW metrics that serves all. It depends what kind of software we are developing
or installing. Finding good metrics for software development or deployment is a major task in itself.
Software development must adapt to the ever changing human ideas, beliefs, fears, and the
corresponding environment. We cannot restrict ourselves to software engineering, we need focus
on project management to improve both software development and deployment.
And we need measurements in order to know that we implemented the targets we had
Principle No 1
P Measure customer related metrics only
o Use Combinatory Metrics to cover all topics
Principle No 2
P Adjust to moving targets
o Your goals may need change; accept change and manage it accordingly
Principle No 3
P Enforce measurement
o Do not enforce meeting targets
What Makes Six Sigma Different?
• Breakthrough improvements
• Financial results focus
• Process focus
• Structured & disciplined problem solving methodology using scientific tools and techniques
• Customer centered
• Involvement of leadership is mandatory.
• Training is mandatory;
• Action learning (25% class room, 75 % application)
• Creating a dedicated organisation for problem solving (85/50 Rule).
Benefits of Six Sigma
• Generates sustained success
• Sets performance goal for everyone
• Enhances value for customers;
• Accelerates rate of improvement;
• Promotes learning across boundaries;
• Executes strategic change
Software audit is a systematic and independent examination to determine the following:
Are quality related arrangements implemented as planned?
Are results consistent with expectations
Adequacy Audit determines the extent to which documented QMS conforms to applicable
Compliance Audit is aimed at determining the extent to which the implemented system
conforms to documented systems
Surveillance Audit ensures continued adherence to standards and continuous improvement
First party audits:
Ensures effective implementation
Evaluate any particular quality problem
Second party audits
Performed by clients
Third party audits
Particularly important - documents are the tangible manifestation of the software.
Documentation process standards
Concerned with how documents should be developed, validated and maintained.
Concerned with document contents, structure, and appearance.
Document interchange standards
Concerned with the compatibility of electronic documents.
• Document identification standards
– How documents are uniquely identified.
• Document structure standards
– Standard structure for project documents.
• Document presentation standards
– Define fonts and styles, use of logos, etc.
• Document update standards
– Define how changes from previous versions are reflected in a document.
• Interchange standards allow electronic documents to be exchanged, mailed, etc.
• Documents are produced using different systems and on different computers. Even when
standard tools are used, standards are needed to define conventions for their use e.g. use of
style sheets and macros.
• Need for archiving. The lifetime of word processing systems may be much less than the
lifetime of the software being documented. An archiving standard may be defined to ensure
that the document can be accessed in future.