2. Introduction
• Proposed by Mills Dyer and Linger during 1980s
• The philosophy focuses on defect avoidance rather
than defect removal
• Avoids costly defect removal processes by writing
code increments right the first time and verify that
correctness before testing
3. Cleanroom software engineering involves the integrated
use of
- software engineering modeling
- program verification
- statistical software quality assurance.
Verifies design specification using mathematically-based
proof of correctness
Relies heavily on statistical use testing to uncover high
impact errors
Generally follows an incremental development process
What is it?
4. Cleanroom is Shift in Pracrice
• From
– Individual
craftsmanship
– Sequential
development
– Individual unit
testing
– Informal coverage
testing
– Unknown reliability
– Informal design
• To
– Peer reviewed
engineering
– Incremental
development
– Team correctness
verification
– Statistical usage testing
– Measured reliability
– Disciplined engineering
specification and design
5. Benefits
• Zero failures in the field
– that’s the goal any way
– a realistic expectation is < 5 failures per KLOC on first program
execution in the first team project
• Short development cycles
– results from use incremental strategy and avoidance of rework
– new teams should experience a two-fold productivity increase on
the first project and continue the increase
• Longer product life
– investments detailed specifications and usage models help keep a
product viable longer
6. Why are Cleanroom Techniques
Not Widely Used
Some people believe cleanroom techniques are too
theoretical, too mathematical, and too radical for use in real
software development
Relies on correctness verification and statistical quality
control rather than unit testing (a major departure from
traditional software development)
Organizations operating at the ad hoc level of the
Capability Maturity Model, do not make rigorous use of the
defined processes needed in all phases of the software life
cycle
8. A pipeline of software increments is developed by small independent
software teams. As each increment is certified it is integrated into the
whole
Increment Planning: adopts the incremental strategy. The functionality
of each increment, its projected size and a cleanroom development
schedule is created
Requirements Gathering: defines a description of customer level
requirements (for each increment)
Box Structure Specification: describes the functional specification. Box
structures isolate and separate the creative definition of behavior, data and
procedures at each level of refinement
Formal Design: specifications (called “black boxes”) are iteratively
refined (with an increment) to become analogous to architectural and
procedural designs (called “state boxes” and “clear boxes,” respectively)
The Cleanroom Process Model
9. The Cleanroom Process Model
Correctness Verification: verification begins with the highest level box
structure (specification) and moves toward design detail and code using
a set of “correctness questions.” If these do not demonstrate that the
specification is correct, more formal (mathematical) methods for
verification are used
Code Generation, Inspection and Verification: the box structure
specifications, represented in a specialized language, are transmitted into
the appropriate programming language.
Statistical Test Planning: a suite of test cases that exercise of
“probability distribution” of usage are planned and designed
Statistical Usage Testing: execute a series of tests derived from a
statistical sample (the probability distribution noted above) of all
possible program executions by all users from a targeted population
Certification: once verification, inspection and usage testing have been
completed (and all errors are corrected) the increment is certified as
ready for integration.
10. What makes Cleanroom Different?
It makes explicit use of statistical quality control.
It verifies design specifications using a mathematically
based proof of correctness.
It implements testing techniques that have a high
likelihood of uncovering high impact errors.
11. Functional Specification
Cleanroom SE complies with the operational analysis principles by
using a method called Box Structure Specification.
A “Box” encapsulates the system (or some aspect of the system) at
some level in detail.
Through a process of elaboration or stepwise refinement, boxes are
refined into a hierarchy where each box has referential
transparency.
The information content of each box specification is sufficient to
define its refinement, without depending on implementation of any
other box.
Three types of boxes are used.
Black Box
State Box
Clear Box
13. Black Box
• Specifies a set of transition rules that describe the behavior of
system components as responses to specific stimuli, makes use of
inheritance in a manner similar to classes
• Specifies system function by mapping all possible stimulus
histories to all possible responses.
• The function f, is applied to the sequence, S* of inputs, S and
transforms them into an output(Response) R .
S* R
stimulus history responses
14. • Generalization of a state machine, encapsulates the data
and operations similar to an object, the inputs (stimuli) and
outputs (responses) are represented, data that must be
retained between transitions is encapsulated
• The state is the encapsulation of the stimulus history
• State variables are invented to save any stimuli that need to
retained
S x T R x T
stimuli X state data responses X state data
State Box
15. • Contains the procedural design of the state box, in a manner
similar to structured programming
• Specifies both data flow and control flow
S x T R x T
stimuli X state data responses X state data
• State update and response production is allowed
Clear Box
16. Box Principles
• Transaction closure of stimuli and responses
– users and uses are considered including security and
error recovery
• State migration within box hierarchy
– downward migration of state data is possible whenever
new black boxes are created inside a clear box
– upward migration of state date is desirable when
duplicate data is updated in several places in the tree
• Common services
– reusable boxes from library
17. Design Refinement & Verification
If a function f is expanded into a sequence g and h, the
correctness condition for all input to f is:
• Does g followed by h do f?
When a function f is refined into a conditional (if-then-
else), the correctness condition for all input to f is:
• Whenever condition <c> is true does g do f and
whenever <c> is false, does h do f?
When function f is refined as a loop, the correctness
conditions for all input to f is:
• Is termination guaranteed?
• Whenever <c> is true does g followed by f do f, and
whenever <c> is false, does skipping the loop still do
f?
18. Design Verification Advantages
• Reduces verification to a finite process
• Improves quality
• Lets cleanroom teams verify every line of code
• Results in near zero levels of defects
• Scales up to larger systems and higher levels
• Produces better code than unit testing
19. Certification Steps
• Usage scenarios must be created
• Usage profile is specified
• Test cases generated from the usage profile
• Tests are executed and failure data are recorded
and analyzed
• Reliability is computed and recorded
20. Statistical Testing
• Generation of test cases
– each test case begins in a start state and represents a random walk
through the usage model ending at a designated end state
• Control of statistical testing
– a well-defined procedure is performed under specified conditions
– each performance is a trial and can be used as part of an empirical
probability computation
• Stopping criteria for testing
– when testing goals or quality standards are achieved
– when the difference between the predicted usage chain and the
actual testing chain becomes very small
21. Reliability Estimation
ConfidenceLevel
Reliability 90% 95% 99% 99.9%
.9 22 29 44 66
.95 45 59 90 135
.99 230 299 459 688
.999 2302 2995 4603 6905
The binomial distribution can be used to estimate the
number of error-free test cases are needed to assume a
given level of reliability at a specified confidence level.
22. Cleanroom Certification Models
• Sampling model
– determines the number if random cases that need to be
executed to achieve a particular reliability level
• Component model
– allows analyst to determine the probability that a given
component in a multi-component system fails prior to
completion
• Certification model
– projected overall reliability of system
