3. Software Quality Assurance
• What is “quality”?
• IEEE Glossary: Degree to which a system,
component, or process meets (1) specified
requirements, and (2) customer or user needs
or expectations
4. Software Quality Assurance
• What is “quality”?
• IEEE Glossary: Degree to which a system,
component, or process meets (1) specified
requirements, and (2) customer or user needs
or expectations
• ISO: the totality of features and
characteristics of a product or service that
bear on its ability to satisfy specified or
implied needs
5. Software Quality Assurance
• An alternate view of Quality:
– is not absolute
– is multidimensional, can be difficult to quantify
– has aspects that are not easy to measure
– assessment is subject to constraints (e.g., cost)
– is about acceptable compromises
– criteria are not independent, can conflict
7. What is Software Quality Assurance
(SQA)?
• “Set of systematic activities providing
evidence of the ability of the software
process to produce a software product that
is fit to use”
– G. Schulmeyer and J. McManus, Software
Quality Handbook, Prentice Hall, 1998.
8. What is SQA?
• Monitoring processes and products throughout
the software development lifecycle to ensure
the quality of the delivered product(s)
• Monitoring the processes
– Provides management with objective feedback
regarding process compliance to approved plans,
procedures, standards, and analyses
• Monitoring the products
– Focus on the quality of product within each phase
of the SDLC
• e.g., requirements, test plan, architecture, etc.
– Objective: identify and remove defects throughout
the lifecycle, as early as possible
9. Quality of Software developed in-house
& COTS components
• SQA processes apply when integrating
purchased or customer-supplied software
products into the developed product
• Question. How do you determine the
“quality” of COTS components?
– Current research problem
10. Process Assessment
• Use of standards and process models has a positive
impact on the quality of the software product
– Disciplined, controlled development process
• Examples include:
– ISO 9001
– CMM
• CMU SEI, 5 levels
– SPICE
• Developing a standard for software process assessment
• ISO joint committee, Europe, Australia
– IEEE 1074, IEEE 12207, …
12. Product Assessment
• Reviews, inspections, walkthroughs of
Plans, reports, models, standards
– Project management, quality assurance,
training, test plan(s)
– Requirements, analysis, architecture, detailed
design model, test cases
– Issue or problem reports
– Metric reports
– Traceability reports
– Documentation, coding standards
– …
13. Software Reviews
• They may include managerial reviews, acquirer-supplier reviews,
technical reviews, inspections, walkthroughs, and audits.
• Inspection:
– A formal evaluation technique in which an artifact (e.g., software
requirements, design, or code) is examined in detail by a person or group
other than the originator
– detect faults, violations of development standards, and other problems.
– review members are peers (equals) of the designer or programmer.
– data is collected during inspections for later analysis and to assist in future
inspections.
Note
– Introduced by Fagan, 1976.
– Fagan, M., “Design and Code Inspections to Reduce Errors in Program
Development”, IBM Systems Journal, 15, 3 (1976), pp. 182-211
– Fagan, M., “Advances in Software Inspections”, IEEE Transactions on Software
Engineering, 12, 7(July 1986), pp. 744-751
14. Picture from “Inspections” presentation
http://www.math.uaa.alaska.edu/~afkjm/cs470/handouts/inspections.pdf
15. Defect Checklists
• Useful to support reviews, inspections, walkthroughs
• Expertise is captured in a list format
– Less experienced people can use
• Straightforward to use (each check should be clear, simple to assess/apply)
– Improve consistency of assessments
• Example architecture checklist used in undergrad./grad. courses
for OO
– spreadsheet in in the course materials subdirectory
• One or more architectural styles are selected.
• Capabilities and interfaces are defined for subsystems.
• Capabilities of and interfaces among subsystems support all of the use cases.
• Concurrency defined.
• Distribution defined.
• Error handling defined.
• Start up and shut down defined.
• Data persistency defined.
• Rationale for the model is provided.
• Other
16. Verifying Formal Specifications
• Formal specifications may be verified in a number of
different ways:
–Syntax, typechecking
• If the notation is typed
–Simulated
–Model checked (e.g., SPIN)
–Proven correct (e.g., HOL, PVS)
• More straightforward? Less assurance of correctness; fully
automated
• Less straightforward? Higher assurance of correctness; not
fully automated
More
straightforward
Less
straightforward
17. Problem Reporting, Tracking, and Resolving
• Describe the practices and procedures to be followed
for reporting, tracking, and resolving problems
– Who can report a problem?
– How is it reported?
– How is is tracked?
– Who determines if it is a problem that going to be resolved?
– How is it assigned for resolution?
– How does the person indicate it has been corrected?
– Who reviews it to determine if it can be closed?
• Problems can be product or process related
– e.g. incorrect requirement, incomplete class definition, code
defect, ambiguous description in user documentation,
process to review detailed design is not clearly defined, etc.
18. Metrics
• Metrics for each artifact
• e.g., Requirements
– Number of requirements
– Number of changes per requirement
• Called “churn” rate
– Characterization of defects
• Not testable, ambiguous, inconsistent, incorrect,
incomplete redundant, infeasible, …
• Major or minor defect
• Phase defect detected
• Cost to fix
19. Tools, techniques, training
• What tools?
– e.g., CVS for CM, excel spreadsheet for
problem reporting/tracking, ...
• What techniques?
– e.g., formal peer review for deliverables,
checklists for defect detection, ...
• What training is needed on tools,
techniques?
20. Media Control
• Identify the media for each intermediate and
deliverable artifact
• Documentation required to store the media, including
the backup and restore process
• Protect computer program physical media from:
– unauthorized access
– inadvertent damage
– degradation
21. Architecture Analysis Methods
• Why evaluate an architecture?
http://www.slideshare.net/kevinjew/evaluating-software-
architectures-presentation
• Specialized techniques available:
http://www.slideshare.net/timmenzies/architecture-
tradeoff-analysis-method-presentation
SEI presentation and technical report on ATAM are in the
course subdirectory
