hshskskasjkjjajjsjsdjsdjsjjdsjsjsjkdsjkskskjk

1. Software Testing & Quality Assurance
Prof. Dr. Tariq Javid
Graduate School of Engineering Sciences & Information Technology
Faculty of Engineering Sciences & Technology
Hamdard University
Spring 2024

2. Software Quality Models and
Metrics
2

3. Outline
• Introduction
• Software Quality (SQ)
• SQ Models
• SQ Metrics
3

4. Introduction, 1/2
• How to measure software quality (SQ)
• How to use measurements to improve software development
process
• Categories of SQ Models and SQ Metrics: Quality management,
software reliability and projection, complexity, and customer view
4

5. Introduction, 2/2
• Discuss fundamentals of measurement theory, specific quality
metrics and tools, and methods for applying metrics to software
development process
5

6. Quality, 1/2
• Perception and interpretation of quality
• Quality Engineering discipline: Quality can, and should, be
operationally defined, measured, monitored, managed, and
improved
6

7. Quality, 2/2
• Quality: Luxury, class, taste
• Quality: Expensive
• Quality: Conformance to requirements … Verification
• Quality: Fitness for use … Validation
7

8. Quality: Conformance to Requirements
• Conformance to requirements implies that requirements must be
clearly stated
• In development and production process, measurements are taken
regularly to determine conformance to requirements
• Non-conformances are defects leads to absence of quality
• If a product meets requirements, then it is a quality product
8

9. Quality: Fitness for Use, 1/2
• Fitness for use: Customers' requirements and expectations
• Different customers may use products in different ways
• Products must possess multiple elements
9

10. Quality: Fitness for Use, 2/2
• Each element is a quality characteristic
• Elements are classified into categories or parameters for fitness
for use
• Important parameters: Quality of design and quality of
conformance
10

11. Quality of Design
• Quality of design: Grades or models related to purchasing power
• Determination of requirements and specifications
• Automobiles provide transportation service to users
• Automobiles models differ in size, comfort, performance, style,
economy, status, etc.
11

12. Quality of Conformance
• Quality of conformance: Product conforms to intent of design
• Conformance to requirements
12

13. Quality Definitions Revisited
• Quality ( Conformance to Requirements, Fitness for Use)
• Both definitions of quality are essentially similar
• Difference: Fitness for use … customers' requirements and
expectations
13

14. Customer’s View of Quality
• Quality: Customer's perceived value of product s/he purchased,
based on a variety of variables such as price, performance,
reliability, and satisfaction
14

15. Customers Buy Quality
• Your customers are in a perfect position to tell you about quality,
because that's all they're really buying. They're not buying a
product. They're buying your assurances that their expectations
for that product will be met.
• And you haven't really got anything else to sell them but those
assurances. You haven't really got anything else to sell but quality.
15

16. Conformance to Requirements Process
• Customers' requirements must be first gathered and analyzed
• Specifications to meet those requirements must be produced
• Product must be developed and manufactured accordingly
16

17. Errors and Defects in Process Phases
• In each phase of process, errors can occur affecting quality of
product
• Requirements may have errors
• Development and manufacturing process may have variables that
induce defects
17

18. Product Quality (Producer, Product,
Customer)
• Producer's perspective: Once requirements are specified, developing
and producing product in accordance with specifications is path to
achieving quality
• Product quality: Lack of defects and good reliability are most basic
measures
• Customer's perspective: Satisfaction after purchase of product is
ultimate validation that product conforms to requirements and is fit to
use
18

19. Two-Level Approach to Quality
• Small q Approach: intrinsic product quality, often operationally
limited to product's defect rate and reliability
• Big Q Approach: Product quality, process quality, and customer
satisfaction
19

20. Implementation Cycle for Two-Level
Approach
• Customer's wants and needs
• Requirements and specifications
• Products designed, developed, and manufactured in accordance with
requirements, and with continuous focus on process improvement
• Excellent product quality, plus good distribution and service processes
• Total customer satisfaction
20

21. Software Quality
• Software product quality is commonly recognized as lack of "bugs" in product
• It is most basic meaning of conformance to requirements
• If software contains too many functional defects, basic requirement of
providing desired function is not met
• Defect rate (e.g., number of defects per million lines of source code, per
function point, or other unit)
• Reliability (e.g., number of failures per n hours of operation, mean time to
failure, or the probability of failure-free operation in a specified time)
21

22. Measurement of Customer’s Satisfaction
• Customer satisfaction is usually measured by percent satisfied or non-
satisfied (neutral and dissatisfied) from customer satisfaction surveys
• To reduce bias, techniques such as blind surveys (interviewer does not
know customer and customer does not know company interviewer
represents) are usually used
• In addition to overall customer satisfaction with software product,
satisfaction toward specific attributes is also gauged
22

23. Examples
• IBM monitors satisfaction with its software products in levels of
CUPRIMDSO (capability [functionality], usability, performance,
reliability, installability, maintainability, documentation/information,
service, and overall)
• Hewlett-Packard focuses on FURPS (functionality, usability, reliability,
performance, and serviceability)
• Other companies use similar dimensions of software customer
satisfaction
23

24. Quality
• To increase overall customer satisfaction as well as satisfaction with
various quality attributes, the quality attributes must be taken into
account in the planning and design of the software
• Large customers with sophisticated networks and real-time
processing, performance and reliability may be most important
attributes
• Customers with standalone systems and simple operations, ease of
use, installability, and documentation may be more important
24

25. 25

26. Requirements Errors and Quality
• Quality (conformance to customers' requirements) is especially
relevant to software industry
• Requirements errors constitute major problem categories in
software development
• 15% or more of all software defects are requirements errors
• Development process that does not address requirements quality
is bound to produce poor-quality software
26

27. Process Quality vs. Product Quality
• From customer requirements to delivery of software products, development
process is complex and often involves a series of stages, each with feedback
paths
• In each stage, an intermediate deliverable is produced for an intermediate user
—the next stage
• Each stage also receives an intermediate deliverable from the preceding stage
• Each intermediate deliverable has certain quality attributes that affect quality
of end product.
27

28. 28

29. Process Quality
• Concept of external and internal customers leads to process quality
• Each stage of development process meets requirements of intermediate user
(next stage)
• End product developed and produced meet specified requirements
• Each stage numerous factors exist affecting stage's ability to fulfill its
requirements
29

30. Development Process Quality
Improvement
• To improve quality during development, we need to:
• Select and deploy specific methods and approaches, and proper
tools and technologies
• Measure characteristics and quality parameters as well as metrics
and models
30

31. Total Quality Management
• TQM: Management approach, style to improve quality
• To ensure long-term success through (quality, customer
satisfaction)
• ISO 9000 Standard
• Recall: Process, product, service
31

32. Example Implementation of TQM, 1/3
• Hewlett-Packard's Total Quality Control
• Key areas: Management commitment, leadership, customer focus,
total participation, and systematic analysis … to drive quality
improvement
• Objective: To achieve success through customer satisfaction
• In software development, Software Quality and Productivity
Analysis program is one of the approaches to improve quality
32

33. Example Implementation of TQM , 2/3
• Motorola's Six Sigma strategy: To achieve stringent quality levels in order
to obtain total customer satisfaction
• Cycle time reduction and participative management are among key
initiatives
• Six Sigma is a measure of quality level for product design improvements
and reductions in process variations
• Six Sigma is applied to product quality as well as everything that can be
supported by data and measurement
33

34. Example Implementation of TQM , 3/3
• "Customer is the final arbiter" them of IBM's Market Driven
Quality strategy
• Strategy comprises four initiatives: Defect elimination, cycle time
reduction, customer and business partner satisfaction, and
adherence to Baldrige assessment discipline
34

35. Software Quality and Productivity
Analysis
• SQPA at HP
• Software Engineering review process
• Management decision support system
• Assesses state of each HP lab and draws comparisons with other
HP labs and industry
35

36. Baldrige Assessment
• Baldrige self-assessment helps organizations assess whether they
are developing and deploying a sound, balanced and systematic
approach for running their organization
• Baldrige Excellence Framework and Criteria
• Conduct a self-assessment of organization
• To evaluate how well your organization is meeting its goals and
objectives
36

37. Key Elements of TQM, 1/5
• Customer focus: The objective is to achieve total customer satisfaction
• Process: The objective is to reduce process variations and to achieve
continuous process improvement
• Human side of quality: The objective is to create a companywide quality
culture
• Measurement and analysis: The objective is to drive continuous
improvement in all quality parameters by the goal-oriented measurement
system
37

38. Key Elements of TQM, 2/5
• Customer focus: The objective is to achieve total customer
satisfaction
• It focus includes studying customers' wants and needs, gathering
customers' requirements, and measuring and managing
customers' satisfaction
38

39. Key Elements of TQM, 3/5
• Process: The objective is to reduce process variations and to
achieve continuous process improvement
• This element includes both the business process and the product
development process
• Through process improvement, product quality will be enhanced
39

40. Key Elements of TQM, 4/5
• Human side of quality: The objective is to create a companywide
quality culture
• Focus areas include leadership, management commitment, total
participation, employee empowerment, and other social,
psychological, and human factors
40

41. Key Elements of TQM, 5/5
• Measurement and analysis: The objective is to drive continuous
improvement in all quality parameters by the goal-oriented
measurement system
• Furthermore, an organization that practices TQM must have
executive leadership, must focus on infrastructure, training, and
education, and must do strategic quality planning
41

42. 42

43. Quality Frameworks
• Various organizational frameworks have been proposed to improve
quality that can be used to substantiate the TQM philosophy
• Plan-Do-Check-Act
• Quality Improvement Paradigm/Experience Factory Organization
• Software Engineering Institute (SEI) Capability Maturity Model
• Lean Enterprise Management
43

44. Software Quality Metrics, 1/4
• Software quality metrics focus on quality aspects (product, process,
project)
• Grouped into categories in accordance with software development
life cycle
• End-product quality metrics
• In-process quality metrics
• Maintenance quality metrics
44

45. Software Quality Metrics, 2/4
• End-product quality metrics:
• Mean time to failure
• Defect density
• Customer-reported problems
• Customer satisfaction
45

46. Software Quality Metrics, 3/4
• In-process quality metrics:
• Phase-based defect removal pattern
• Defect removal effectiveness
• Defect density during formal machine testing
• Defect arrival pattern during formal machine testing
46

47. Software Quality Metrics, 4/4
• Maintenance quality metrics:
• Fix backlog
• Backlog management index
• Fix response time and fix responsiveness
• Percent delinquent fixes
• Defective fixes
47

48. Software Quality Models, 1/6
• Software Quality Models:
• Reliability and projection models
• Quality management models
• Complexity metrics and models
• Customer-oriented metrics, measurements, and models
48

49. Software Quality Models, 2/6
• Reliability and projection models:
• Rayleigh model
• Exponential model
• Weibull distribution with different shape parameters
• Reliability growth models that were developed specifically for software
• The polynomial regression models for defect arrivals and backlog
projections
49

50. Software Quality Models, 3/6
• Quality management models (1/2):
• Rayleigh model framework for quality management The phase-
based defect removal model and defect removal effectiveness
• Code integration pattern metric or heuristic model
• Defect arrival (PTR) nonparametric (heuristic) model
• Polynomial regression models for defect arrivals and backlog
projections
50

51. Software Quality Models, 4/6
• Quality management models (2/2):
• Use of reliability models to set targets for the current project
• Models for spreading the total projected field defects over time for
maintenance planning (heuristic or using the reliability models)
• Statistical models on customer satisfaction analysis
• Effort/outcome model for establishing in-process metrics and for
assessing in-process quality status
51

52. Software Quality Models, 5/6
• Complexity metrics and models:
• Analysis and statistical models linking complexity metrics,
syntactic constructs, structure metrics, and other variables to
quality
• Analysis and statistical models linking the object-oriented design
and complexity metrics to quality and managerial variables
52

53. Software Quality Models, 6/6
• Customer-oriented metrics, measurements, and models:
• System availability and outage metrics
• Methods and models to measure and analyze customer
satisfaction
53

54. Tasks
• Read the Standard: ISO/IEC/IEEE 29119-2
• https://www.youtube.com/watch?v=BBmA5Qp6Ghk
54