This document summarizes research conducted at West Virginia University and the Jet Propulsion Laboratory under NASA grants. It discusses an approach called STAR (Simulated Annealing with collars and Rules) for handling uncertainty in software project models. STAR uses simulated annealing to stochastically sample different input settings, scores the outputs, and favors higher scoring settings to generate rules. The document compares STAR to other modeling methods and finds that even without tuning, STAR produces surprisingly accurate results, often within 25% of methods using historical data to learn input relationships. This performance is attributed to key inputs being tightly constrained by a few influential "collar" variables.

1. This work was conducted at West Virginia University and the Jet Propulsion
Laboratory under grants with NASA's Software Assurance Research
Program. Reference herein to any specific commercial product, process, or
service by trademark, manufacturer, or otherwise, does not constitute or
imply its endorsement by the United States Government.
If you fix everything you
lose fixes for everything else
Tim Menzies (WVU)
Jairus Hihn (JPL)
Oussama Elrawas (WVU)
Dan Baker (WVU)
Karen Lum (JPL)
tim@menzies.us
International Workshop on Living with Uncertainty,
IEEE ASE 2007, Atlanta, Georgia, oelrawas@mix.wvu.edu
Nov 5, 2007

2. What does this mean?
A supposedly np-hard task
abduction over first-
order theories
nogood/2
Q: for what models does (a few peeks) = (many hard stares)?
2

3. Grow

A: models with Monte Carlo a model
–
Picking input settings at

“collars” random
For each run
–
Score each output

Add score to each input

“Collar” variables set the other settings

variables Harvest

Rule generation experiments,
Narrows –
–
favoring settings with better

Amarel in the 60s

scores
Minimal environments
–
If “collars”, then

DeKleer ’85

… small rules …
–
Master variables
–
– … learned quickly …
Crawford & Baker ‘94

– … will suffice
Feature subset selection
–
Kohavi & John ‘97

Back doors
–
Williams et al ‘03

Etc
–
Implications for uncertainty?

Feather & Menzies RE’02
3

4. STAR: collars + simulated annealing on
For
example
Boehm’s USC’s software process models
USC software process models for effort, defects, threats

controllable
y[i] = impact[i] * project[i] + b[i] for i ∈ {1,2,3,…}
–
α ≤ project[i] ≤ β : uncertainty in project description
–
χ ≤ impact[i] ≤ δ : uncertainty in model calibration
–
uncontrollable
Random solution

pick project[i] and impact[i] from any α .. β , χ .. δ
–
– α .. β set via domain knowledge;
e.g. process maturity in 3 to 5
– range of χ .. δ known from history;
Score solution by effort (Ef),

defects (De) and Threat (Th)
4

5. Two studies
y[i] = impact[i] * project[i] + b[i]
one two
Certain methods

Methods with more uncertainty

Using much historical data
–
Using no historical data
–
Learn the magnitude of the
–
– Monte Carlo at random across
impact[i] relationship
the project[i] settings and
– With fixed impact[I] Tame
impact[i] settings
Monte Carlo at uncontroll-

andom across the ables via
project[i] settings historical
E.g.

E.g. records

STAR
–
Regression-based tools that
–
– Monte Carlo a model
learn impact[I] from historical
records – Score each output
– 93 records of JPL systems – Sort settings by their “C”,
– SCAT: “C”= cumulative score

JPL’s current methods

Rule generation experiments,
–
2CEE:
–
favoring settings with better “C”.

WVU’s improvement over

SCAT (currently under test)
5

6. Bad
Inside STAR
1. sampling
- simulated annealing Good
2. summarizing
- post-processor
38 not-so- good ideas
for setting ∈ Sx { value[setting] += E }

Sort all settings by their value

Ignore uncontrollables impact[I]
–
Assume the top
–
(1 ≤ i ≤ max) project[I] settings
Randomly select the rest
–
“Policy point” :

smallest I with lowest E
–
Median = 50% percentile

Spread = (75-50)% percentile
–
22 good ideas
6

7. SCAT vs
2CEE vs
STAR project[i]
7

8. Control impact[I] via
historical data
SCAT vs
2CEE vs
STAR project[i]
8

9. Control impact[I] via
historical data
SCAT vs
2CEE vs
STAR project[i]
Stagger around
superset of possible
impact[I]
9

10. Control impact[I] via
historical data
SCAT vs
2CEE vs
STAR project[i]
Stagger around
superset of possible
impact[I]
Median:
50% point
Spread :
(75 - 50)%
10

11. Control impact[I] via
historical data
SCAT vs
2CEE vs
STAR project[i]
Stagger around
superset of possible
impact[I]
Median:
50% point
Spread :
(75 - 50)%
STAR/2cee= 50/ 800= 6%
STAR/scat= 50/1300= 4%
11

12. Control impact[I] via
historical data
SCAT vs
2CEE vs
STAR project[i]
Stagger around
superset of possible
impact[I]
Median:
50% point
Spread :
(75 - 50)%
STAR/2cee= 400/1600= 25% STAR/2cee= 180/ 400= 45%
STAR/2cee= 30/620= 5%
STAR/2cee= 50/ 800= 6%
STAR/scat= 400/1900= 21% STAR/scat= 180/1900= 60%
STAR/scat= 30/730= 4%
STAR/scat= 50/1300= 4%
12

13. Control impact[I] via
historical data
SCAT vs
2CEE vs
STAR project[i]
Stagger around
superset of possible
impact[I]
Median:
50% point
Spread :
(75 - 50)%
STAR/2cee= 400/1600= 25% STAR/2cee= 180/ 400= 45%
STAR/2cee= 30/620= 5%
STAR/2cee= 50/ 800= 6%
STAR/scat= 400/1900= 21% STAR/scat= 180/1900= 60%
STAR/scat= 30/730= 4%
STAR/scat= 50/1300= 4%
13

14. Control impact[I] via
historical data
SCAT vs
2CEE vs
STAR project[i]
Stagger around
superset of possible
impact[I]
Median:
50% point
Spread :
(75 - 50)%
STAR/2cee= 400/1600= 25% STAR/2cee= 180/ 400= 45%
STAR/2cee= 30/620= 5%
STAR/2cee= 50/ 800= 6%
STAR/scat= 400/1900= 21% STAR/scat= 180/1900= 60%
STAR/scat= 30/730= 4%
STAR/scat= 50/1300= 4%
Ignoring historical data is useful (!!!?) 14

15. Control impact[I] via
historical data
SCAT vs
2CEE vs
STAR project[i]
Stagger around
superset of possible
impact[I]
Median:
50% point
Spread :
(75 - 50)%
STAR/2cee= 400/1600= 25% STAR/2cee= 180/ 400= 45%
STAR/2cee= 30/620= 5%
STAR/2cee= 50/ 800= 6%
STAR/scat= 400/1900= 21% STAR/scat= 180/1900= 60%
STAR/scat= 30/730= 4%
STAR/scat= 50/1300= 4%
Ignoring historical data is useful (!!!?) 15

16. Control impact[I] via
historical data
SCAT vs
2CEE vs
STAR project[i]
Stagger around
superset of possible
impact[I]
Median:
50% point
Spread :
(75 - 50)%
STAR/2cee= 400/1600= 25% STAR/2cee= 180/ 400= 45%
STAR/2cee= 30/620= 5%
STAR/2cee= 50/ 800= 6%
STAR/scat= 400/1900= 21% STAR/scat= 180/1900= 60%
STAR/scat= 30/730= 4%
STAR/scat= 50/1300= 4%
Ignoring historical data is useful (!!!?) If you fix everything, you lose fixes for everything else16

17. Luke, trust the force,
I mean, collars
IEEE Computer, Jan 2007
“The strangest thing about software”

18. Extra Material

19. Related work
Abduction :

World W = minimal set of
–
assumptions (w.r.t. size) such that
Feather, DDP, treatment learning

T ∪ A => G

Optimization of
Not(T U A => error) –

requirement models
Framework for
–
validation,

XEROC PARC, 1980s, qualitative

diagnosis,

representations (QR)
planning,

monitoring,
 not overly-specific,
–
explanation,
 Quickly collected in a new
–
tutoring,
 domain.
test case generation,

– Used for model diagnosis
prediction,…
 and repair
Theoretically slow (NP-hard) but
– – Can found creative solutions in
this should be practical: larger space of possible
Abduction + stochastic sampling
 qualitative behaviors,
Find collars
 than in the tighter space of precise

Learn constraints on collars quantitative behaviors

19

20. Possible optimizations
(not used here)
STAR, an example of a general BORE (best or rest)
 
process: n runs
–
Stochastic sampling Best= top 10% scores
– –
– Sort settings by “value” Rest = remaining 90%
–
– Rule generation experiments {a,b} = frequency of
–
discretized range in {best, rest
favoring highly “value”-ed settings

See also, elite sampling in the Sort settings by
 – Ask
cross-entropy method -1 * (a/n)2 / (a/n + b/n) me why,
off-line
If SA convergence too slow

Other valuable tricks:

Try moving back select into the SA;
–
Incremental discretization:
–
– Constrain solution mutation to Gama&Pinto’s PID +
prefer highly “value”-ed settings Fayyad&Irani
– Limited discrepancy search:
Harvey&Ginsberg
– Treatment learning: Menzies&Yu
20

21. “Uncertainty
helps
planning”
(questions? comments?)

22. At the “policy point”, diff
diff
STAR’s random solutions
are surprisingly accurate
diff
diff
LC : learn impact[i] via regression (JPL data)
STAR: no tuning, randomly pick impact[i]
Diff = ∑ mre(lc)/ ∑ mre(star)
Mre = abs(predicted - actual) /actual same
diff
∑ mre(lc) / ∑ mre(star) strategic tactical
ground 66% 63%
all 91% 75%
OSP2 99% 125% ●❍ same
same
OSP 112% ●❍ 111% ●❍
flight 101% ●❍ 121% ●❍
same at {95, 99}% confidence (MWU)
{ “●” “❍”}
same
same
Why so little Diff (median= 75%)?
Most influential inputs tightly constrained
–
22

23. (Model uncertainty = collars) << inputs
In many models, a few “collar” variables set the other variables

Narrows (Amarel in the 60s)
–
Minimal environments (DeKleer ’85)
–
Master variables (Crawford & Baker ‘94)
–
– Feature subset selection (Kohavi & John ‘97)
– Back doors (Williams et al ‘03)
– See “The Strangest Thing About Software (IEEE Computer, Jan’07)”
Collars appear in all execution traces (by definition)

You don’t have to find the collars, they’ll find you
–
So, to handle uncertainty

Write a simulator
–
Stagger over uncertainties
–
This talk: a very simple example of this process
From stagger, find collars
–
Constrain collars
–
23

24. Comparisons
Standard software process modeling

Models written more than run (PROSIM community)
–
Limited sensitivity analysis

Limited trade space

Or, expensive, error-prone, incomplete data collection
–
programs
Point solutions

Here:

No data collection
–
Found stable conclusions
–
within a space of possibilities
– Search : very simple
– Solution, not brittle
With trade-off space

22 good ideas, sorted
24

25. Bad
Summary
Living with uncertainty

Sometimes, simpler than you Good
–
may think
more useful than you might
–
think
Simple:

Here, the smallest change
–
to simulating annealing
Useful:

Sometimes uncertainty can
–
teach you more than certainty
If you fix everything, you lose
–
fixes to everything else
An example you
Collars control certainty
 can explain to
Uncertainty plus constrained
–
any business user
collars → more certainty
Also, can drive model to
–
better performance
An example you
can explain to
any business user 22 good ideas, sorted
25