Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Successfully reported this slideshow.

Like this presentation? Why not share!

- Whither Software Engineering Resear... by David Rosenblum 28862 views
- Jogging While Driving, and Other So... by David Rosenblum 22052 views
- Career Management (invited talk at ... by David Rosenblum 24010 views
- Known Unknowns: Testing in the Pres... by David Rosenblum 21349 views
- Applications and Abstractions: A Ca... by David Rosenblum 29890 views
- Felicitous Computing (invited Talk ... by David Rosenblum 26442 views

24,398 views

Published on

No Downloads

Total views

24,398

On SlideShare

0

From Embeds

0

Number of Embeds

20,337

Shares

0

Downloads

1

Comments

0

Likes

3

No embeds

No notes for slide

- 1. NASAC 2013,Tianjin, 9 November 2013 Probability and Uncertainty in Software Engineering David S. Rosenblum! Dean, School of Computing! National University of Singapore
- 2. NASAC 2013,Tianjin, 9 November 2013 Software Engineering at NUS Hugh Anderson Chin Wei Ngan Dong Jin Song Aquinas Hobor Joxan! Jaffar Stan Jarzabek Khoo Siau Cheng Damith Rajapakse David! Rosenblum Abhik Roychoudhury Bimlesh Wadhwa Yap Hock Chuan, Roland
- 3. NASAC 2013,Tianjin, 9 November 2013 Certainty in Software Engineering Engineering of software is centered around simplistic,“yes/no” characterizations of artifacts
- 4. NASAC 2013,Tianjin, 9 November 2013 Certainty in Software Engineering Engineering of software is centered around simplistic,“yes/no” characterizations of artifacts Program is correct/incorrect Program execution ﬁnished/crashed Compilation completed/aborted Test suite succeeded/failed Speciﬁcation is satisﬁed/violated
- 5. NASAC 2013,Tianjin, 9 November 2013 Example! Model Checking ! ¬p → ◊q( )∧"( ) Model Checker ✓ ✕ State Machine! Model Temporal Property Results Counterexample! Trace System Requirements
- 6. NASAC 2013,Tianjin, 9 November 2013 Example! Model Checking ! ¬p → ◊q( )∧"( ) Model Checker ✕ State Machine! Model Temporal Property Results Counterexample! Trace System Requirements
- 7. NASAC 2013,Tianjin, 9 November 2013 Uncertainty in Software Engineering ✓Nondeterminism ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics
- 8. NASAC 2013,Tianjin, 9 November 2013 Uncertainty in Software Engineering ✓Nondeterminism ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics Probabilistic Modeling and Analysis
- 9. NASAC 2013,Tianjin, 9 November 2013 Probabilistic Model Checking ! ¬p → ◊q( )∧"( ) Model Checker ✓ ✕ State Machine! Model Temporal Property Results Counterexample! Trace System Requirements P≥0.95 [ ] 0.4 0.6 Probabilistic Probabilistic
- 10. NASAC 2013,Tianjin, 9 November 2013 Probabilistic Model Checking ! ¬p → ◊q( )∧"( ) Model Checker ✓ ✕ State Machine! Model Temporal Property Results Counterexample! Trace System Requirements P=? [ ] 0.4 0.6 Quantitative Results 0.9732Probabilistic Probabilistic
- 11. NASAC 2013,Tianjin, 9 November 2013 Example Die Tossing Simulated by Coin Flipping Knuth-Yao algorithm, from the PRISM group (Kwiatkowska et al.) 0 3 2 1 6 4 5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
- 12. NASAC 2013,Tianjin, 9 November 2013 Example Die Tossing Simulated by Coin Flipping Knuth-Yao algorithm, from the PRISM group (Kwiatkowska et al.) The behavior is governed by a! theoretical probability distribution 0 3 2 1 6 4 5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
- 13. NASAC 2013,Tianjin, 9 November 2013 Probabilistic Model Checking ! ¬p → ◊q( )∧"( ) Model Checker ✓ State Machine! Model Temporal Property Results Counterexample! Trace System Requirements P≥0.95 [ ] 0.4 0.6 Quantitative Results 0.9732Probabilistic Probabilistic
- 14. NASAC 2013,Tianjin, 9 November 2013 Probabilistic Model Checking ! ¬p → ◊q( )∧"( ) Model Checker ✕ State Machine! Model Temporal Property Results Counterexample! Trace System Requirements P≥0.95 [ ] Quantitative Results Probabilistic Probabilistic 0.41 0.59 0.6211
- 15. NASAC 2013,Tianjin, 9 November 2013 Example! Zeroconf Protocol s1s0 s2 s3 q 1 1 {ok} {error} {start} s4 s5 s6 s7 s8 1 1-q 1-p 1-p 1-p 1-p p p p p 1 from the PRISM group (Kwiatkowska et al.)
- 16. NASAC 2013,Tianjin, 9 November 2013 Example! Zeroconf Protocol s1s0 s2 s3 q 1 1 {ok} {error} {start} s4 s5 s6 s7 s8 1 1-q 1-p 1-p 1-p 1-p p p p p 1 The behavior is governed by an! empirically estimated probability distribution from the PRISM group (Kwiatkowska et al.) packet-loss rate
- 17. NASAC 2013,Tianjin, 9 November 2013 Perturbed Probabilistic Systems! (Current Research) • Starting Points! ✓Discrete-Time Markov Chains (DTMCs)! ✓… with one or more probability parameters! ✓… veriﬁed against reachability properties: S? ∪ S! Guoxin Su and David S. Rosenblum, “Asymptotic Bounds for QuantitativeVeriﬁcation of Perturbed Probabilistic Systems”, Proc. ICFEM 2013
- 18. NASAC 2013,Tianjin, 9 November 2013 Parametric Markov Chains • A distribution parameter in a DTMC is represented as a vector x of parameters xi! • The norm of total variance represents the amount of perturbation:! ! • The parameter is allowed a “sufﬁciently small” perturbation with respect to ideal reference values r:! ! • Can generalize to multiple parameters v = vi∑ x − r ≤ Δ
- 19. NASAC 2013,Tianjin, 9 November 2013 Perturbation Bounds • Perturbation Function! ! where A is the transition probability sub-matrix for S? and b is the vector of one-step probabilities from S? to S! ! • Condition Numbers! ! ρ x( )= ι? i A x i i b x( )− Ai i b( )( )i=0 ∞ ∑ κ = lim δ→0 sup ρ(x − r) δ : x − r ≤ δ,δ > 0 ⎧ ⎨ ⎩ ⎫ ⎬ ⎭
- 20. NASAC 2013,Tianjin, 9 November 2013 Results! Noisy Zeroconf (35000 Hosts, PRISM) p Actual Collision Probability Predicted Collision Probability 0.095 -19.8% -21.5% 0.096 -16.9% -17.2% 0.097 -12.3% -12.9% 0.098 -8.33% -8.61% 0.099 -4.23% -4.30% 0.100 1.8567 — 0.101 +4.38% +4.30% 0.102 +8.91% +8.61% 0.103 +13.6% +12.9% 0.104 +18.4% +17.2% 0.105 +23.4% +21.5%
- 21. NASAC 2013,Tianjin, 9 November 2013 Additional Aspects • Models ✓Markov Decision Processes (MDPs)! ✓Continuous-Time Markov Chains (CMTCs) • Veriﬁcation ✓LTL Model Checking! using Deterministic Rabin Automata! ✓PCTL Model Checking! with singular perturbations due to nested P[ ] operators! ✓Reward Properties! ✓Alternative Norms and Bounds! Kullback-Leibler Divergence, Quadratic Bounds
- 22. NASAC 2013,Tianjin, 9 November 2013 Other Forms of Uncertainty “There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say, we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.”! ! — Donald Rumsfeld
- 23. NASAC 2013,Tianjin, 9 November 2013 Uncertainty in Testing! (New Research) 1982: Weyuker: Non-Testable Programs! - Impossible/too costly to efﬁciently check results! - Example: mathematical software! 2010: Garlan: Intrinsic Uncertainty! - Systems embody intrinsic uncertainty/imprecision! - Cannot easily distinguish bugs from “features”! - Example: ubiquitous computing
- 24. NASAC 2013,Tianjin, 9 November 2013 Example! Google Latitude ~ 500m ~ 50m ~ 2m
- 25. NASAC 2013,Tianjin, 9 November 2013 Example! Google Latitude When is an incorrect location! a bug, and when is it a “feature”? ~ 500m ~ 50m ~ 2m
- 26. NASAC 2013,Tianjin, 9 November 2013 Example! Google Latitude When is an incorrect location! a bug, and when is it a “feature”? And how do! you know? ~ 500m ~ 50m ~ 2m
- 27. NASAC 2013,Tianjin, 9 November 2013 Example! Affective Computing
- 28. NASAC 2013,Tianjin, 9 November 2013 Example! Affective Computing When is an! incorrect! classiﬁcation a bug,! and when is it a! “feature”?
- 29. NASAC 2013,Tianjin, 9 November 2013 Example! Affective Computing When is an! incorrect! classiﬁcation a bug,! and when is it a! “feature”? And how do! you know?
- 30. NASAC 2013,Tianjin, 9 November 2013 Sources of Uncertainty ✓Output: results, characteristics of results! ✓Sensors: redundancy, reliability, resolution! ✓Context: sensing, inferring, fusing! ✓Machine learning: imprecision, user training
- 31. NASAC 2013,Tianjin, 9 November 2013 Sources of Uncertainty ✓Output: results, characteristics of results! ✓Sensors: redundancy, reliability, resolution! ✓Context: sensing, inferring, fusing! ✓Machine learning: imprecision, user training These create signiﬁcant challenges for software engineering research and practice!
- 32. NASAC 2013,Tianjin, 9 November 2013 Conclusion ✓Software engineering (certainly) suffers from excessive certainty! ✓A probabilistic mindset offers greater insight! ✓But signiﬁcant challenges remain for probabilistic veriﬁcation! ✓And other forms of uncertainty are equally challenging to address
- 33. NASAC 2013,Tianjin, 9 November 2013 Probability and Uncertainty in Software Engineering David S. Rosenblum! Dean, School of Computing! National University of Singapore ThankYou!

No public clipboards found for this slide

Be the first to comment