Global Sensitivity Analysis of Predictor Models in Software Engineering

Global Sensitivity Analysis of Predictor Models in Software Engineering Stefan Wagner Software & Systems Engineering Technische Universit¨t M¨nchen a u Germany wagnerst@in.tum.de May 20, 2007 Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 1 / 27

Introduction Predictor Models in Software Engineering Problem Example: Fischer-Wagner Model Questions About the Model Overview Introduction Global Sensitivity Analysis Approach for SE Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 2 / 27

Predictor Models in Software Engineering Predictor models describe situations in software engineering Introduction s Predictor Models in Various types, aims, . . . Software s Engineering Examples s Problem Example: Fischer-Wagner COCOMO (costs) x Model Questions About the Musa-Okumoto (reliability) x Model Overview Fault-proneness x Global Sensitivity Analysis Approach for SE Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 3 / 27

Problem The models themselves can be complex Introduction s Predictor Models in Their use needs a lot of eﬀort Software s Engineering How can I analyse those models themeselves? s Problem Example: How can I simplify them? s Fischer-Wagner Model How do I improve their predictive power? s Questions About the Model Overview Global Sensitivity Analysis Approach for SE Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 4 / 27

Example: Fischer-Wagner Model Reliability model used at Siemens COM Introduction s Predictor Models in Two parameters estimated by failure data Software s Engineering Failure probability of a fault: pa = p1 · d(a−1) s Problem Example: Geometrical distribution : Fa (t) = 1 − (1 − pa )t s Fischer-Wagner Model Cummulated failures up to t: s Questions About the Model µ(t) = ∞ 1 − (1 − p1 · d(a−1) ) Overview a=1 Input parameters s Global Sensitivity Analysis p1 : Highest failure probability of a fault x Approach for SE d: Complexity of the system ]0; 1[ x Summary t: Execution time (incidents) x inf : approximation of ∞ x Output: µ(t) s More details: S. Wagner, H. Fischer. Ada-Europe, 2006 Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 5 / 27

Questions About the Model How do the input parameters inﬂuence the output? Introduction s Predictor Models in Which parameter(s) do I have to estimate best to get a good Software s Engineering prediction? Problem Example: Are there insigniﬁcant parameters that I can remove? s Fischer-Wagner Model Questions About the Model Overview Global Sensitivity Analysis Approach for SE Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 6 / 27

Overview Introduction Introduction Predictor Models in Software Global Sensitivity Analysis Engineering Problem Example: Approach for SE Fischer-Wagner Model Summary Questions About the Model Overview Global Sensitivity Analysis Approach for SE Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 7 / 27

Introduction Global Sensitivity Analysis Definition Method Variance-based Methods Main Effect Higher-Order Effects Total Effect Global Sensitivity Analysis Approach for SE Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 8 / 27

Definition Introduction Saltelli (2000): “Sensitivity analysis studies the relationships s Global Sensitivity between information flowing in and out of the model.” Analysis Definition How do the input parameters influence the output? s Method How can this influence by quantified? Variance-based s Methods Global properties s Main Effect Higher-Order Effects Inclusion of influence of shape and scale x Total Effect Approach for SE Multidimensional averaging x Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 9 / 27

Method Introduction Input Model Output Global Sensitivity Analysis Definition x1 Method Variance-based Methods Main Effect Higher-Order Effects x2 Total Effect Approach for SE Summary Y f (x) x3 xn (Based on slide of S. Tarantola) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 10 / 27

Variance-based Methods Global analysis usually variance-based Introduction s Global Sensitivity “model-free” s Analysis Definition Analysis on the basis of sensitivity indices s Method Variance-based Main or first-order effect x Methods Main Effect x Higher-order effects Higher-Order Effects x Total effect Total Effect Approach for SE Decomposition in main effects and interactions s Summary k y = f (x) = f0 + fi (xi ) + fij (xi , xj ) i=1 i j>i + . . . + f1,2,...,k (x1 , x2 , . . . , xk ) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 11 / 27

Variance-based Methods Global analysis usually variance-based Introduction s Global Sensitivity “model-free” s Analysis Definition Analysis on the basis of sensitivity indices s Method Variance-based Main or first-order effect x Methods Main Effect x Higher-order effects Higher-Order Effects x Total effect Total Effect Approach for SE Decomposition in main effects and interactions s Summary For example k = 3: f (x) = f0 + f1 (x1 ) + f2 (x2 ) + f3 (x3 ) + f12 (x1 , x2 ) +f13 (x1 , x3 ) + f23 (x2 , x3 ) + f123 (x1 , x2 , x3 ) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 11 / 27

Main Effect Introduction Decomposition of the variance V of f (x) Global Sensitivity Analysis k Definition Var(y) = V = Vi + Vij Vijk + . . . + V1,2,...k Method Variance-based Methods i=1 i j i j k Main Effect Higher-Order Effects First-order sensitivity coefficient Total Effect Approach for SE Vi Si = Summary V (Factors Priorisation) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 12 / 27

Higher-Order Effects Higher-order indices analog Introduction Global Sensitivity Analysis Si1 ...io = Vi1 ...io /V Definition Method Variance-based Methods Main Effect Higher-Order Effects Total Effect Approach for SE With k = 4: Summary 2. order: S12 , S13 , S14 , S23 , S24 , S34 3. order: S123 , S124 , S134 , S234 4. order: S1234 Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 13 / 27

Total Effect Sum of the first- and higher-order effects of xi Introduction s Global Sensitivity Removal of the non-relevant parts s Analysis Definition Method Variance-based S1 + S2 + S3 + S4 + S12 + S13 + S14 + S23 + S24 + S34 + Methods Main Effect S123 + S124 + S134 + S234 + S1234 = 1 Higher-Order Effects Total Effect Approach for SE Summary (Factors Fixing) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 14 / 27

Total Effect Sum of the first- and higher-order effects of xi Introduction s Global Sensitivity Removal of the non-relevant parts s Analysis Definition Method Variance-based S1 + //2 + S3 + S4 + S12 + S13 + S14 + S23 + S24 + S34 + S/ Methods Main Effect Higher-Order Effects S123 + S124 + S134 + S234 + S1234 = 1 Total Effect Approach for SE Summary (Factors Fixing) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 14 / 27

Total Effect Sum of the first- and higher-order effects of xi Introduction s Global Sensitivity Removal of the non-relevant parts s Analysis Definition Method Variance-based S1 + //2 + / 3 + S4 + S12 + S13 + S14 + S23 + S24 + S34 + S/ S // Methods Main Effect Higher-Order Effects S123 + S124 + S134 + S234 + S1234 = 1 Total Effect Approach for SE Summary (Factors Fixing) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 14 / 27

Total Effect Sum of the first- and higher-order effects of xi Introduction s Global Sensitivity Removal of the non-relevant parts s Analysis Definition Method Variance-based S1 + //2 + / 3 + /// + S12 + S13 + S14 + S23 + S24 + S34 + S/ S // S4 Methods Main Effect Higher-Order Effects S123 + S124 + S134 + S234 + S1234 = 1 Total Effect Approach for SE Summary (Factors Fixing) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 14 / 27

Total Effect Sum of the first- and higher-order effects of xi Introduction s Global Sensitivity Removal of the non-relevant parts s Analysis Definition Method Variance-based S1 + //2 + / 3 + /// + S12 + S13 + S14 + S23 + S24 + S34 + S/ S // S4 // // // // // // Methods Main Effect Higher-Order Effects S123 + S124 + S134 + S234 + S1234 = 1 Total Effect Approach for SE Summary (Factors Fixing) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 14 / 27

Total Effect Sum of the first- and higher-order effects of xi Introduction s Global Sensitivity Removal of the non-relevant parts s Analysis Definition Method Variance-based S1 + //2 + / 3 + /// + S12 + S13 + S14 + S23 + S24 + S34 + S/ S // S4 // // // // // // Methods Main Effect Higher-Order Effects S123 + S124 + S134 + / 234 + S1234 = 1 S // // / Total Effect Approach for SE Summary (Factors Fixing) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 14 / 27

Total Effect Sum of the first- and higher-order effects of xi Introduction s Global Sensitivity Removal of the non-relevant parts s Analysis Definition Method Variance-based S1 + //2 + / 3 + /// + S12 + S13 + S14 + S23 + S24 + S34 + S/ S // S4 // // // // // // Methods Main Effect Higher-Order Effects S123 + S124 + S134 + / 234 + S1234 = 1 S // // / Total Effect Approach for SE ST 1 = S1 + S12 + S13 + S14 + S123 + S124 + S134 + S1234 Summary (Factors Fixing) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 14 / 27

Total Effect Sum of the first- and higher-order effects of xi Introduction s Global Sensitivity Removal of the non-relevant parts s Analysis Definition Method Variance-based S1 + //2 + / 3 + /// + S12 + S13 + S14 + S23 + S24 + S34 + S/ S // S4 // // // // // // Methods Main Effect Higher-Order Effects S123 + S124 + S134 + / 234 + S1234 = 1 S // // / Total Effect Approach for SE ST 1 = S1 + S12 + S13 + S14 + S123 + S124 + S134 + S1234 Summary ST 2 = S2 + S12 + S23 + S24 + S123 + S124 + S234 + S1234 (Factors Fixing) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 14 / 27

Introduction Global Sensitivity Analysis Approach for SE Overview of the Approach Determining Distributions Example: Distributions Approach for SE Visualising Using Scatterplots p1 and µ d and µ t and µ inf and µ Applying Global SA Indizes Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 15 / 27

Overview of the Approach Introduction Global Global Sensitivity Visualising Determining Analysis sensitivity using distributions Approach for SE analyses scatterplots Overview of the Approach Determining Distributions Example: Distributions Visualising Using Scatterplots .375 p1 and µ .045 d and µ .003 t and µ .002 inf and µ Input Sensitivity Applying Global SA Scatterplots distributions indices Indizes Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 16 / 27

Determining Distributions Distribution of the values of the input parameteters Introduction s Global Sensitivity Derived from s Analysis Approach for SE scientiﬁc literature x Overview of the Approach physical boundaries x Determining expert opinion x Distributions Example: surveys x Distributions Visualising Using experiments x Scatterplots p1 and µ d and µ t and µ inf and µ Applying Global SA Indizes Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 17 / 27

Example: Distributions Based on expert opinion Introduction s Global Sensitivity General parameter s Analysis Approach for SE p1 uniformly distributed between 0 and 1 x Overview of the Approach d uniformly distributed between 0.9 and 1 x Determining Distributions Dependent on application s Example: Distributions Visualising Using inf e.g.. uniformly distributed between 50 and 500 x Scatterplots p1 and µ For t e.g. t ∼ N (1000, 200) x d and µ t and µ inf and µ Applying Global SA Indizes Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 18 / 27

Visualising Using Scatterplots Sampling using the distributions Introduction s Global Sensitivity Pairwise relationship between factors s Analysis Detection of errors in the model s Approach for SE Overview of the Indications of inﬂuences s Approach Determining Distributions Example: Distributions Visualising Using Scatterplots p1 and µ d and µ t and µ inf and µ Applying Global SA Indizes Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 19 / 27

p1 and µ 900 Introduction Global Sensitivity Analysis 800 Approach for SE 700 Overview of the Approach Determining 600 Distributions Example: µ 500 Distributions Visualising Using Scatterplots 400 p1 and µ d and µ 300 t and µ inf and µ 200 Applying Global SA Indizes 100 Summary 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 p1 Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 20 / 27

d and µ 900 Introduction Global Sensitivity Analysis 800 Approach for SE 700 Overview of the Approach Determining 600 Distributions Example: 500 Distributions µ Visualising Using Scatterplots 400 p1 and µ d and µ 300 t and µ inf and µ 200 Applying Global SA Indizes 100 Summary 0 0.9 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1 d Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 21 / 27

t and µ 900 Introduction Global Sensitivity 800 Analysis Approach for SE 700 Overview of the Approach 600 Determining Distributions Example: 500 Distributions µ Visualising Using 400 Scatterplots p1 and µ 300 d and µ t and µ 200 inf and µ Applying Global SA 100 Indizes Summary 0 99200 99400 99600 99800 100000 100200 100400 100600 100800 t Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 22 / 27

inf and µ 900 Introduction Global Sensitivity Analysis 800 Approach for SE 700 Overview of the Approach Determining 600 Distributions Example: 500 Distributions µ Visualising Using Scatterplots 400 p1 and µ d and µ 300 t and µ inf and µ 200 Applying Global SA Indizes 100 Summary 0 400 450 500 550 600 650 700 750 800 850 900 inf Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 23 / 27

Applying Global SA Various methods for calculating indices Introduction s Global Sensitivity FAST (Fourier Amplitude Sensitivity Test) gives quantitative s Analysis results Approach for SE Overview of the Uses sampled data s Approach Determining Tool-support: Simlab s Distributions Example: Distributions Visualising Using Scatterplots p1 and µ d and µ t and µ inf and µ Applying Global SA Indizes Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 24 / 27

Indizes Main Eﬀect Introduction Global Sensitivity Analysis t 6.88e-5 Approach for SE p1 0.0090 Overview of the Approach d 0.9026 Determining Distributions 0.0158 inf Example: Distributions Visualising Using Scatterplots p1 and µ d and µ t and µ inf and µ Applying Global SA Indizes Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 25 / 27

Indizes Main Eﬀect Introduction Global Sensitivity Analysis t 6.88e-5 Approach for SE p1 0.0090 Overview of the Approach d 0.9026 Determining Distributions 0.0158 inf Example: Distributions Visualising Using Total Eﬀect Scatterplots p1 and µ d and µ t 0.005788 t and µ p1 0.022365 inf and µ Applying Global SA d 0.975155 Indizes 0.086735 inf Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 25 / 27

Introduction Global Sensitivity Analysis Approach for SE Summary Conclusions Summary Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 26 / 27

Conclusions Sensitivity analysis useful tool for predictor models Introduction s Global Sensitivity SA can help to s Analysis Approach for SE ﬁnd errors in the models x Summary simplify the models x Conclusions identify interactions between input parameters x identify parameters that should be investigated more x get more robust predictions x Experience with s reliability model x QA economics model x process model x expert system for IT tools x Good tool-support available (Simlab: s http://simlab.jrc.cec.eu.int/) Dr. Stefan Wagner, TU M¨nchen u PROMISE – May 20, 2007 – 27 / 27

Global Sensitivity Analysis of Predictor Models in Software Engineering

by promise07 on May 20, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 2

Statistics

Global Sensitivity Analysis of Predictor Models in Software Engineering — Presentation Transcript