Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Successfully reported this slideshow.

Like this presentation? Why not share!

- Known Unknowns: Testing in the Pres... by David Rosenblum 20226 views
- Applications and Abstractions: A Ca... by David Rosenblum 29869 views
- Whither Software Engineering Resear... by David Rosenblum 28825 views
- Career Management (invited talk at ... by David Rosenblum 23966 views
- Felicitous Computing (invited Talk ... by David Rosenblum 26420 views
- Probability and Uncertainty in Soft... by David Rosenblum 24300 views

22,025 views

Published on

Published in:
Technology

No Downloads

Total views

22,025

On SlideShare

0

From Embeds

0

Number of Embeds

18,878

Shares

0

Downloads

1

Comments

0

Likes

3

No embeds

No notes for slide

- 1. Jogging While Driving! and Other Software Engineering Research Problems David S. Rosenblum! Dean, School of Computing! National University of Singapore
- 2. Singapore
- 3. Singapore
- 4. Singapore
- 5. Singapore Universities
- 6. Singapore Universities
- 7. Singapore Universities
- 8. NUS School of Computing ✓Ranked #1 in Asia, #9 in the world [QS World University Rankings by Subject]! ✓2 Departments: Computer Science and Information Systems! ✓111 Academic Staff (tenure-track & teaching track)! ✓115 Research Staff! ✓1800 Undergraduate Students! ✓180 Masters Students! ✓350 PhD Students! ✓S$25 million operating budget! ✓S$10 million+ in research income per annum
- 9. Certainty in Software Engineering Engineering of software is centered around simplistic,“yes/no” characterizations of artifacts
- 10. Certainty in Software Engineering Engineering of software is centered around simplistic,“yes/no” characterizations of artifacts Program is correct/incorrect Program execution ﬁnished/crashed Compilation completed/aborted Test suite succeeded/failed Speciﬁcation is satisﬁed/violated
- 11. Example! Model Checking Model Checker ✓ ✕ State Machine! Model Temporal Properties Results System Requirements ! ¬p → ◊q( )∧"( )
- 12. Example! Model Checking Model Checker ✕ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements ! ¬p → ◊q( )∧"( )
- 13. Uncertainty in Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics
- 14. Uncertainty in Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics Custom Model Checking Algorithms
- 15. CAAAs Context-Aware Adaptive Applications
- 16. CAAAs Context-Aware Adaptive Applications
- 17. CAAAs Context-Aware Adaptive Applications
- 18. CAAAs Context-Aware Adaptive Applications
- 19. CAAAs Context-Aware Adaptive Applications
- 20. Adaptation in CAAAs Physical Context Sensed Context Inferred Context Presumed Context Environment Context! Manager Application Adaptation! Manager Middleware M. Sama, D.S. Rosenblum, Z.Wang and S. Elbaum,“Multi-Layer Faults in the Architectures of Mobile, Context-Aware Adaptive Applications”, Journal of Systems and Software,Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
- 21. Adaptation in CAAAs Physical Context Sensed Context Inferred Context Presumed Context Environment Context! Manager Application Adaptation! Manager Middleware Rule Engine M. Sama, D.S. Rosenblum, Z.Wang and S. Elbaum,“Multi-Layer Faults in the Architectures of Mobile, Context-Aware Adaptive Applications”, Journal of Systems and Software,Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
- 22. Adaptation in CAAAs Physical Context Sensed Context Inferred Context Presumed Context Environment Context! Manager Application Adaptation! Manager Middleware 3rd-Party Libraries Rule Engine M. Sama, D.S. Rosenblum, Z.Wang and S. Elbaum,“Multi-Layer Faults in the Architectures of Mobile, Context-Aware Adaptive Applications”, Journal of Systems and Software,Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
- 23. Approach 1.Derive Adaptation Finite-State Machine (A-FSM) from rule logic! 2.Explore state space of A-FSM to discover all potential faults! ✓Enumerative algorithms! ✓Symbolic algorithms! 3.(Conﬁrm existence of discovered faults) M. Sama, S. Elbaum, F. Raimondi and D.S. Rosenblum,“Context-Aware Adaptive Applications: Fault Patterns and Their Automated Identiﬁcation”, IEEETransactions on Software Engineering,Vol. 36, No. 5, Sep./Oct. 2010, pp. 644-661.
- 24. PhoneAdapter
- 25. PhoneAdapter normal,! vibrate silent, vibrate loud, vibratesilent, divert to voicemail loud,! divert to! hands-free
- 26. PhoneAdapter normal,! vibrate silent, vibrate loud, vibratesilent, divert to voicemail loud,! divert to! hands-free
- 27. PhoneAdapter A-FSM Ofﬁce Driving! Fast Meeting Driving Sync General Home Outdoor Jogging
- 28. PhoneAdapter A-FSM ActivateMeeting DeactivateMeeting Ofﬁce Driving! Fast Meeting Driving Sync General Home Outdoor Jogging
- 29. PhoneAdapter A-FSM checking location implies GPS is on! locations are mutually exclusive! speeds monotonically increase! a meeting’s end time is later than its start time Global constraints: ActivateMeeting DeactivateMeeting Ofﬁce Driving! Fast Meeting Driving Sync General Home Outdoor Jogging
- 30. Example Faults in PhoneAdapter OfﬁceGeneral Home
- 31. Example Faults in PhoneAdapter User’s phone discovers ofﬁce PC at home OfﬁceGeneral Home
- 32. Example Faults in PhoneAdapter Nondeterminism! OfﬁceGeneral Home
- 33. Example Faults in PhoneAdapter General
- 34. Example Faults in PhoneAdapter User decides to go somewhere else GeneralOutdoor
- 35. Example Faults in PhoneAdapter User starts driving before Bluetooth detects hands-free system Driving GeneralOutdoor
- 36. Example Faults in PhoneAdapter Activation hazard! Driving GeneralOutdoor Jogging
- 37. Example Faults in PhoneAdapter Activation hazard! Driving GeneralOutdoor Jogging
- 38. Faults in CAAAs • Behavioral Faults! Nondeterminism! Dead rule! Dead state! ! ! ! ! ! Unreachable state! Activation race! Activation cycle • Hazards! Hold hazard! Activation hazard! ! Priority inversion hazard
- 39. PhoneAdapter Results Behavioral Faults: Enumerative, Symbolic TABLE 2 Faulty Input Conﬁgurations Reported for PhoneAdapter State Nondeterministic Dead Adaptation Unreachable Adaptations Predicates Races Cycles States General 37 1 45 13 0 Outdoor 3 0 135 23 0 Jogging 0 0 97 19 0 Driving 0 0 36 13 0 DrivingFast 0 0 58 19 0 Home 0 0 76 19 0 Ofﬁce 0 0 29 1 0 Meeting 0 0 32 1 0 Sync 0 0 27 5 1
- 40. PhoneAdapter Results Hazards: Enumerative n PhoneAdapter aptation Races and Cycles Context Hazards signments Race Cycle Paths Hold Activ. Prior. 3968 45 13 14085 0 11 3182 3968 135 23 161 0 0 52 3072 97 19 2 0 0 0 2560 36 13 16 2 2 4 3072 58 19 2 0 0 0 2816 76 19 104 8 0 13 2848 29 1 82634 1828 368 2164 2048 32 1 0 0 0 0 1024 27 5 2 2 0 0 ned a formal model of a key complex behavioral char- eristic, namely adaptation, of an increasingly large and Table 2: Faults State Vars. Nondet. Adaptation Dead Pred Assignments Faults Assignments General 7 128 37 128 Outdoor 5 32 3 17 Jogging 2 4 0 1 Driving 3 8 0 7 DrivingFast 2 4 0 2 Home 4 16 0 9 O ce 7 128 1 65 Meeting 1 2 0 2 Sync 2 4 0 1 6.4 Detecting Context Hazards This class of faults corresponds to sequences of asynchr
- 41. CAAAs Summary ✓Rule-based CAAAs can be extremely fault- prone, even with a small set of rules! ✓The model checking algorithms ﬁnd many actual faults, with different tradeoffs! ✓Some alternative to rule-based adaptation may be preferable
- 42. Uncertainty in Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics
- 43. Uncertainty in Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics Probabilistic Model Checking
- 44. Probabilistic Model Checking Model Checker ✓ ✕ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements ! ¬p → ◊q( )∧"( )
- 45. Probabilistic Model Checking Model Checker ✓ ✕ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements 0.4 0.6 Probabilistic ! ¬p → ◊q( )∧"( )
- 46. P≥0.95 [ ] Probabilistic Model Checking Model Checker ✓ ✕ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements 0.4 0.6 Probabilistic Probabilistic ! ¬p → ◊q( )∧"( )
- 47. P=? [ ] Probabilistic Model Checking Model Checker ✓ ✕ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements 0.4 0.6 Quantitative Results 0.9732Probabilistic Probabilistic ! ¬p → ◊q( )∧"( )
- 48. Example Die Tossing Simulated by Coin Flipping Knuth-Yao algorithm, from the PRISM group (Kwiatkowska et al.) 0 3 2 1 6 4 5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
- 49. Example Die Tossing Simulated by Coin Flipping Knuth-Yao algorithm, from the PRISM group (Kwiatkowska et al.) The behavior is governed by a! theoretical probability distribution 0 3 2 1 6 4 5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
- 50. P≥0.95 [ ] Probabilistic Model Checking Model Checker ✓ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements 0.4 0.6 Quantitative Results 0.9732Probabilistic Probabilistic ! ¬p → ◊q( )∧"( )
- 51. P≥0.95 [ ] Probabilistic Model Checking Model Checker ✓ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements Quantitative Results 0.9732Probabilistic Probabilistic 0.41 0.59 ! ¬p → ◊q( )∧"( )
- 52. P≥0.95 [ ] Probabilistic Model Checking Model Checker ✕ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements Quantitative Results Probabilistic Probabilistic 0.41 0.59 0.6211 ! ¬p → ◊q( )∧"( )
- 53. Example! Zeroconf Protocol s1s0 s2 s3 q 1 1 {ok} {error} {start} s4 s5 s6 s7 s8 1 1-q 1-p 1-p 1-p 1-p p p p p 1 from the PRISM group (Kwiatkowska et al.)
- 54. Example! Zeroconf Protocol s1s0 s2 s3 q 1 1 {ok} {error} {start} s4 s5 s6 s7 s8 1 1-q 1-p 1-p 1-p 1-p p p p p 1 The behavior is governed by an! empirically estimated probability distribution from the PRISM group (Kwiatkowska et al.) packet-loss rate
- 55. Perturbed Probabilistic Systems • Starting Points! ✓Discrete-Time Markov Chains (DTMCs)! ✓… with one or more probability parameters! ✓… veriﬁed against reachability properties:! ! ✓… and (more recently) LTL properties S? ∪ S! Guoxin Su and David S. Rosenblum,“Asymptotic Bounds for QuantitativeVeriﬁcation of Perturbed Probabilistic Systems”, Proc. ICFEM 2013! ! Guoxin Su and David S. Rosenblum,“Perturbation Analysis of Stochastic Systems with Empirical Distribution Parameters”, Proc. ICSE 2014
- 56. Parametric Markov Chains • A distribution parameter in a DTMC is represented as a vector x of parameters xi! • The norm of total variance represents the amount of perturbation:! ! • The parameter is allowed a “sufﬁciently small” perturbation with respect to ideal reference values r:! ! • Can generalize to multiple parameters v = vi∑ x − r ≤ Δ
- 57. Perturbation Bounds • Perturbation Function! ! where A is the transition probability sub-matrix for S? and b is the vector of one-step probabilities from S? to S! ! • Condition Numbers: [ICFEM 2013]! ! • Quadratic Bounds: [ICSE 2014]! ρ x( )= ι? i A x i i b x( )− Ai i b( )( )i=0 ∞ ∑ κ = lim δ→0 sup ρ(x − r) δ : x − r ≤ δ,δ > 0 ⎧ ⎨ ⎩ ⎫ ⎬ ⎭ f − (δ )− inf ρ(x − r) + f + (δ )− supρ(x − r) = o(δ 2 )
- 58. Results! Noisy Zeroconf (35000 Hosts, PRISM) p Actual Collision Probability Predicted Collision Probability (κ) 0.095 -19.8% -21.5% 0.096 -16.9% -17.2% 0.097 -12.3% -12.9% 0.098 -8.33% -8.61% 0.099 -4.23% -4.30% 0.100 1.8567 — 0.101 +4.38% +4.30% 0.102 +8.91% +8.61% 0.103 +13.6% +12.9% 0.104 +18.4% +17.2% 0.105 +23.4% +21.5%
- 59. Additional Aspects • Models ✓Markov Decision Processes (MDPs)! ✓Continuous-Time Markov Chains (CMTCs)! • Veriﬁcation ✓PCTL Model Checking! with singularities due to nested P[ ] operators! ✓Reward Properties! ✓Alternative Norms and Bounds! Kullback-Leibler Divergence! ✓Parameters as random variables
- 60. Other Forms of Uncertainty “There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say, we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.”! ! — Donald Rumsfeld
- 61. Uncertainty in Testing 1982: Elaine Weyuker: Non-Testable Programs! - Impossible/too costly to efﬁciently check results! - Example: mathematical software! 2010: David Garlan: Intrinsic Uncertainty! - Systems embody intrinsic uncertainty/imprecision! - Cannot easily distinguish bugs from “features”! - Example: ubiquitous computing
- 62. Example! Google Latitude ~ 500m ~ 2m ~ 50m
- 63. Example! Google Latitude When is an incorrect location! a bug, and when is it a “feature”? ~ 500m ~ 2m ~ 50m
- 64. Example! Google Latitude When is an incorrect location! a bug, and when is it a “feature”? And how do! you know? ~ 500m ~ 2m ~ 50m
- 65. Example! Affective Computing
- 66. Example! Affective Computing When is an! incorrect emotion! classiﬁcation a bug,! and when is it a! “feature”?
- 67. Example! Affective Computing When is an! incorrect emotion! classiﬁcation a bug,! and when is it a! “feature”? And how do! you know?
- 68. Sources of Uncertainty ✓Output: results, characteristics of results! ✓Sensors: redundancy, reliability, resolution! ✓Context: sensing, inferring, fusing! ✓Machine learning: imprecision, user-speciﬁcity
- 69. Sources of Uncertainty ✓Output: results, characteristics of results! ✓Sensors: redundancy, reliability, resolution! ✓Context: sensing, inferring, fusing! ✓Machine learning: imprecision, user-speciﬁcity These create signiﬁcant challenges for software engineering research and practice!
- 70. Conclusion ✓Software engineering (certainly) suffers from excessive certainty! ✓A probabilistic mindset offers some insight! ✓But signiﬁcant challenges remain for probabilistic veriﬁcation! ✓And other forms of uncertainty remain a challenge to address
- 71. Jogging While Driving! and Other Software Engineering Research Problems David S. Rosenblum! Dean, School of Computing! National University of Singapore

No public clipboards found for this slide

Be the first to comment