Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Uncertainty, Risk, and Information Value in
Software Requirements and Architecture
Emmanuel Letier, David Stefan, Earl T. ...
Embrace uncertainty!
2
Software Design Decisions
Uncertainty is inevitable
We must decide without knowing everything
3
What software to build? Wh...
The Surfer’s Approach to Uncertainty
Instead of learning to
surf, conventional
organizations try to
control the waves. Thi...
The Surfer’s Approach to Uncertainty
Instead of learning to
surf, conventional
organizations try to
control the waves. Thi...
The Surfer’s Approach to Uncertainty
Instead of learning to
surf, conventional
organizations try to
control the waves. Thi...
The Scientific Approach to Uncertainty
Decision Analysis, a discipline
for understanding, formalising,
analysing, and comm...
The Pseudo-Scientific Approach
Use formulae that resembles a
scientific approach, except that
•  the decision criteria are...
What do we mean by uncertainty ?
9
Uncertainty
Uncertainty is the lack of complete knowledge about a state
or quantity. There is more than one possible value...
Key observation
11
For any uncertain quantity, we always know something even
if our uncertainty is large
How many auto ric...
12
Things Software Engineers Say ...
Clients don’t know what they want
Requirements change is inevitable
It’s not possible...
13
Things Academics Say ...
Requirements are inherently
unknowable!
Linda Northrop “Does Scale Really Matter? – Ultra-Larg...
What they mean ...
Requirements are uncertain
14
We always know something about the
requirements for a system even if our
...
Software Design Decisions
Uncertainty is inevitable
We must decide without knowing everything
15
What software to build? W...
Our approach in 6 steps
16
1. Model the design
decision problem
Architecture decisions
and impact on software qualities
Re...
Our approach in 6 steps
17
1. Model the design
decision problem
2. Define the decision risks
What do we mean by risk?
Risk is a state of uncertainty where some of the possibilities
involve a loss, catastrophe, or ot...
Our approach in 6 steps
19
1. Model the design
decision problem
2. Define the decision risks
3. Elicit what decision maker...
Our approach in 6 steps
20
1. Model the design
decision problem
2. Define the decision risks
3. Elicit what decision maker...
0.0 0.2 0.4 0.6 0.8 1.0
1214161820
Project Failure risk
ExpectedNetBenefit
Shortlisting candidate architectures
•  Monte-C...
Our approach in 6 steps
22
1. Model the design
decision problem
2. Define the decision risks
3. Elicit what decision maker...
The Expected Value of Perfect Information (EVPI)
•  Useful against measurement inversion bias: measuring
what we can be pr...
Our approach in 6 steps
24
1. Model the design
decision problem
2. Define the decision risks
3. Elicit what decision maker...
Application to an example from ICSE’13
A mobile system for coordinating
emergency rescue teams
•  Design space: 10 design
...
Conclusion
26
Research Roadmap
27
Scientific Approach
to Software
Decisions
Parameter uncertainty
Model uncertainty: quantifying “good e...
Please, help us!
28
We need more research on Software Decision Analysis
‒  Our ‘moda’ R package is available to facilitate...
A Call to Action
Who do you want to inform critical
IT decisions?
29
Uncertainty will be at the heart of
many important de...
Upcoming SlideShare
Loading in …5
×

Uncertainty, Risk, and Information Value in Software Requirements and Architecture

1,147 views

Published on

Slides from the ICSE 2014 presentation

E. Letier, D. Stefan, E. T. Barr, Uncertainty, Risk, and Information Value in Software Requirements and Architecture, Proc. 36th International Conference on Software Engineering 2014 (ICSE 2014).

Uncertainty complicates early requirements and architecture decisions and may expose a software project to significant risk. Yet software architects lack support for evaluating uncertainty, its impact on risk, and the value of reducing uncertainty before making critical decisions. We propose to apply decision analysis and multi-objective optimisation techniques to provide such support. We present a systematic method allowing software architects to describe uncertainty about the impact of alternatives on stakeholders' goals; to calculate the consequences of uncertainty through Monte-Carlo simulation; to shortlist candidate architectures based on expected costs, benefits and risks; and to assess the value of obtaining additional information before deciding. We demonstrate our method on the design of a system for coordinating emergency response teams. Our approach highlights the need for requirements engineering and software cost estimation methods to disclose uncertainty instead of hiding it.

Published in: Science, Technology, Business
  • Be the first to comment

Uncertainty, Risk, and Information Value in Software Requirements and Architecture

  1. 1. Uncertainty, Risk, and Information Value in Software Requirements and Architecture Emmanuel Letier, David Stefan, Earl T. Barr University College London, UK 1 Hyderabad, 6 June 2014, ICSE 2014
  2. 2. Embrace uncertainty! 2
  3. 3. Software Design Decisions Uncertainty is inevitable We must decide without knowing everything 3 What software to build? What functions? What quality level? What architectural style? What components and interfaces? How to deploy them?
  4. 4. The Surfer’s Approach to Uncertainty Instead of learning to surf, conventional organizations try to control the waves. This almost never works. — Allen Ward 4 Mary Poppendieck, “Learning to Surf”, industry keynote @ ICSE2013
  5. 5. The Surfer’s Approach to Uncertainty Instead of learning to surf, conventional organizations try to control the waves. This almost never works. — Allen Ward 5 Mary Poppendieck, “Learning to Surf”, industry keynote @ ICSE2013
  6. 6. The Surfer’s Approach to Uncertainty Instead of learning to surf, conventional organizations try to control the waves. This almost never works. -- Allen Ward 6
  7. 7. The Scientific Approach to Uncertainty Decision Analysis, a discipline for understanding, formalising, analysing, and communicating insights about situations in which important decisions must be made 7 Ron Howard, Stanford
  8. 8. The Pseudo-Scientific Approach Use formulae that resembles a scientific approach, except that •  the decision criteria are numbers without verifiable meaning •  the decision models are not falsifiable •  no retrospective evaluation of decisions and outcomes Most widely used example, the Analytical Hierarchy Process (AHP) 8
  9. 9. What do we mean by uncertainty ? 9
  10. 10. Uncertainty Uncertainty is the lack of complete knowledge about a state or quantity. There is more than one possible value and the “true” value is not known. Measurement of uncertainty. A set of possible values with a probability assigned to each. 10 Will I submit a paper to ICSE 2015? yes no 0.8 0.2 How many papers will be submitted to ICSE 2015? 500 600350 distribution
  11. 11. Key observation 11 For any uncertain quantity, we always know something even if our uncertainty is large How many auto rickshaw journeys in Hyderabad today? 10m100k
  12. 12. 12 Things Software Engineers Say ... Clients don’t know what they want Requirements change is inevitable It’s not possible to discover the true requirements before building the system
  13. 13. 13 Things Academics Say ... Requirements are inherently unknowable! Linda Northrop “Does Scale Really Matter? – Ultra-Large-Scale Systems Seven Years after the Study” plenary keynote @ ICSE2013
  14. 14. What they mean ... Requirements are uncertain 14 We always know something about the requirements for a system even if our uncertainty is large
  15. 15. Software Design Decisions Uncertainty is inevitable We must decide without knowing everything 15 What software to build? What functions? What quality level? What architectural style? What components and interfaces? How to deploy them?
  16. 16. Our approach in 6 steps 16 1. Model the design decision problem Architecture decisions and impact on software qualities Requirements decisions and impact on business value
  17. 17. Our approach in 6 steps 17 1. Model the design decision problem 2. Define the decision risks
  18. 18. What do we mean by risk? Risk is a state of uncertainty where some of the possibilities involve a loss, catastrophe, or other undesirable outcome. Measurement of Risk. A set of possibilities each with quantified probabilities and quantified losses. (D. Hubbard) 18 Goal failure risk: software not meeting minimally acceptable level of goal satisfaction (e.g. unacceptable performance, reliability, availability, cost, etc.) Project failure risk: software not meeting minimally acceptable level for at least one of its goals In our approach Paper novelty: first method to quantify goal failure risks and project failure risk to inform software design decisions
  19. 19. Our approach in 6 steps 19 1. Model the design decision problem 2. Define the decision risks 3. Elicit what decision makers already know (their prior probability distributions) We rely on simple, effective methods to counter cognitive biases (overconfidence, anchoring, etc.)
  20. 20. Our approach in 6 steps 20 1. Model the design decision problem 2. Define the decision risks 3. Elicit what decision makers already know (their prior probability distributions) 4. Shortlist candidate architectures based on expected value, expected cost, and risks
  21. 21. 0.0 0.2 0.4 0.6 0.8 1.0 1214161820 Project Failure risk ExpectedNetBenefit Shortlisting candidate architectures •  Monte-Carlo simulation to evaluate candidate architectures under uncertainty •  Multi-objective optimisation to find shortlist (shortlist = set of Pareto-optimal candidates) 21 •  Paper novelties (for SBSE experts): - Pareto strip = Pareto front with uncertainty margins - Identification of closed and open decisions in shortlist Design space: ~ 7,000 candidates Shortlist (in red): 10 candidates
  22. 22. Our approach in 6 steps 22 1. Model the design decision problem 2. Define the decision risks 3. Elicit what decision makers already know (their prior probability distributions) 4. Shortlist candidate architectures based on expected value, expected cost, and risks 5. Compute the expected value of information Should we seek additional info about shortlisted candidates?
  23. 23. The Expected Value of Perfect Information (EVPI) •  Useful against measurement inversion bias: measuring what we can be precise about rather than what is most valuable to decision •  Paper novelty: computing impact of perfect information on risk 23 EVPI(X) = the expected gain in business value ($) from obtaining perfect information about X to inform decision (Ronald Howard, 1966) How much would you be willing to pay for perfect information about X?
  24. 24. Our approach in 6 steps 24 1. Model the design decision problem 2. Define the decision risks 3. Elicit what decision makers already know (their prior probability distributions) 4. Shortlist candidate architectures based on expected value, expected cost, and risks 5. Compute the expected value of information 6. Seek additional info where valuable (creates posterior distributions) Should we seek additional info about shortlisted candidates?
  25. 25. Application to an example from ICSE’13 A mobile system for coordinating emergency rescue teams •  Design space: 10 design decisions; around 7,000 candidate architectures •  Objectives: Cost, Response Time, Reliability, Battery Life, ... •  Models given by design team: Utility score defined as weighted sum of objectives satisfaction •  Lessons Learnt –  Measuring risks leads to a different shortlist –  Need to reason about model uncertainty in addition to parameter uncertainty –  Decision models must be falsifiable 25 (Esfahani, Malek, & Razavi)
  26. 26. Conclusion 26
  27. 27. Research Roadmap 27 Scientific Approach to Software Decisions Parameter uncertainty Model uncertainty: quantifying “good enough” Incremental value delivery Showing applicability Showing cost-effectiveness Overcoming cultural barriers ??? Incremental evidence-based model tuning
  28. 28. Please, help us! 28 We need more research on Software Decision Analysis ‒  Our ‘moda’ R package is available to facilitate uptake ‒  Related work: CBAM @ ICSE’01, ICSE’03; GuideArch @ ICSE’13; Fenton et al @ ICSE’04 We need industry partners willing to try scientific approach to decision making –  portfolio decisions –  release planning –  architecture decisions –  risk-based testing –  process decisions
  29. 29. A Call to Action Who do you want to inform critical IT decisions? 29 Uncertainty will be at the heart of many important decisions for the 21st Century The Surfers The ScientistsThe Pseudo-Scientists

×