Like this presentation? Why not share!

- Felicitous Computing (invited Talk ... by David Rosenblum 14644 views
- Applications and Abstractions: A Ca... by David Rosenblum 18280 views
- Whither Software Engineering Resear... by David Rosenblum 17026 views
- Career Management (invited talk at ... by David Rosenblum 11188 views
- Probability and Uncertainty in Soft... by David Rosenblum 11684 views
- Known Unknowns: Testing in the Pres... by David Rosenblum 7184 views

10,037

-1

-1

Published on

Published in:
Technology

No Downloads

Total Views

10,037

On Slideshare

0

From Embeds

0

Number of Embeds

9

Shares

0

Downloads

1

Comments

0

Likes

3

No embeds

No notes for slide

- 1. Jogging While Driving! and Other Software Engineering Research Problems David S. Rosenblum! Dean, School of Computing! National University of Singapore
- 2. Singapore
- 3. Singapore
- 4. Singapore
- 5. Singapore Universities
- 6. Singapore Universities
- 7. Singapore Universities
- 8. NUS School of Computing ✓Ranked #1 in Asia, #9 in the world [QS World University Rankings by Subject]! ✓2 Departments: Computer Science and Information Systems! ✓111 Academic Staff (tenure-track & teaching track)! ✓115 Research Staff! ✓1800 Undergraduate Students! ✓180 Masters Students! ✓350 PhD Students! ✓S$25 million operating budget! ✓S$10 million+ in research income per annum
- 9. Certainty in Software Engineering Engineering of software is centered around simplistic,“yes/no” characterizations of artifacts
- 10. Certainty in Software Engineering Engineering of software is centered around simplistic,“yes/no” characterizations of artifacts Program is correct/incorrect Program execution ﬁnished/crashed Compilation completed/aborted Test suite succeeded/failed Speciﬁcation is satisﬁed/violated
- 11. Example! Model Checking Model Checker ✓ ✕ State Machine! Model Temporal Properties Results System Requirements ! ¬p → ◊q( )∧"( )
- 12. Example! Model Checking Model Checker ✕ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements ! ¬p → ◊q( )∧"( )
- 13. Uncertainty in Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics
- 14. Uncertainty in Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics Custom Model Checking Algorithms
- 15. CAAAs Context-Aware Adaptive Applications
- 16. CAAAs Context-Aware Adaptive Applications
- 17. CAAAs Context-Aware Adaptive Applications
- 18. CAAAs Context-Aware Adaptive Applications
- 19. CAAAs Context-Aware Adaptive Applications
- 20. Adaptation in CAAAs Physical Context Sensed Context Inferred Context Presumed Context Environment Context! Manager Application Adaptation! Manager Middleware M. Sama, D.S. Rosenblum, Z.Wang and S. Elbaum,“Multi-Layer Faults in the Architectures of Mobile, Context-Aware Adaptive Applications”, Journal of Systems and Software,Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
- 21. Adaptation in CAAAs Physical Context Sensed Context Inferred Context Presumed Context Environment Context! Manager Application Adaptation! Manager Middleware Rule Engine M. Sama, D.S. Rosenblum, Z.Wang and S. Elbaum,“Multi-Layer Faults in the Architectures of Mobile, Context-Aware Adaptive Applications”, Journal of Systems and Software,Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
- 22. Adaptation in CAAAs Physical Context Sensed Context Inferred Context Presumed Context Environment Context! Manager Application Adaptation! Manager Middleware 3rd-Party Libraries Rule Engine M. Sama, D.S. Rosenblum, Z.Wang and S. Elbaum,“Multi-Layer Faults in the Architectures of Mobile, Context-Aware Adaptive Applications”, Journal of Systems and Software,Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
- 23. Approach 1.Derive Adaptation Finite-State Machine (A-FSM) from rule logic! 2.Explore state space of A-FSM to discover all potential faults! ✓Enumerative algorithms! ✓Symbolic algorithms! 3.(Conﬁrm existence of discovered faults) M. Sama, S. Elbaum, F. Raimondi and D.S. Rosenblum,“Context-Aware Adaptive Applications: Fault Patterns and Their Automated Identiﬁcation”, IEEETransactions on Software Engineering,Vol. 36, No. 5, Sep./Oct. 2010, pp. 644-661.
- 24. PhoneAdapter
- 25. PhoneAdapter normal,! vibrate silent, vibrate loud, vibratesilent, divert to voicemail loud,! divert to! hands-free
- 26. PhoneAdapter normal,! vibrate silent, vibrate loud, vibratesilent, divert to voicemail loud,! divert to! hands-free
- 27. PhoneAdapter A-FSM Ofﬁce Driving! Fast Meeting Driving Sync General Home Outdoor Jogging
- 28. PhoneAdapter A-FSM ActivateMeeting DeactivateMeeting Ofﬁce Driving! Fast Meeting Driving Sync General Home Outdoor Jogging
- 29. PhoneAdapter A-FSM checking location implies GPS is on! locations are mutually exclusive! speeds monotonically increase! a meeting’s end time is later than its start time Global constraints: ActivateMeeting DeactivateMeeting Ofﬁce Driving! Fast Meeting Driving Sync General Home Outdoor Jogging
- 30. Example Faults in PhoneAdapter OfﬁceGeneral Home
- 31. Example Faults in PhoneAdapter User’s phone discovers ofﬁce PC at home OfﬁceGeneral Home
- 32. Example Faults in PhoneAdapter Nondeterminism! OfﬁceGeneral Home
- 33. Example Faults in PhoneAdapter General
- 34. Example Faults in PhoneAdapter User decides to go somewhere else GeneralOutdoor
- 35. Example Faults in PhoneAdapter User starts driving before Bluetooth detects hands-free system Driving GeneralOutdoor
- 36. Example Faults in PhoneAdapter Activation hazard! Driving GeneralOutdoor Jogging
- 37. Example Faults in PhoneAdapter Activation hazard! Driving GeneralOutdoor Jogging
- 38. Faults in CAAAs • Behavioral Faults! Nondeterminism! Dead rule! Dead state! ! ! ! ! ! Unreachable state! Activation race! Activation cycle • Hazards! Hold hazard! Activation hazard! ! Priority inversion hazard
- 39. PhoneAdapter Results Behavioral Faults: Enumerative, Symbolic TABLE 2 Faulty Input Conﬁgurations Reported for PhoneAdapter State Nondeterministic Dead Adaptation Unreachable Adaptations Predicates Races Cycles States General 37 1 45 13 0 Outdoor 3 0 135 23 0 Jogging 0 0 97 19 0 Driving 0 0 36 13 0 DrivingFast 0 0 58 19 0 Home 0 0 76 19 0 Ofﬁce 0 0 29 1 0 Meeting 0 0 32 1 0 Sync 0 0 27 5 1
- 40. PhoneAdapter Results Hazards: Enumerative n PhoneAdapter aptation Races and Cycles Context Hazards signments Race Cycle Paths Hold Activ. Prior. 3968 45 13 14085 0 11 3182 3968 135 23 161 0 0 52 3072 97 19 2 0 0 0 2560 36 13 16 2 2 4 3072 58 19 2 0 0 0 2816 76 19 104 8 0 13 2848 29 1 82634 1828 368 2164 2048 32 1 0 0 0 0 1024 27 5 2 2 0 0 ned a formal model of a key complex behavioral char- eristic, namely adaptation, of an increasingly large and Table 2: Faults State Vars. Nondet. Adaptation Dead Pred Assignments Faults Assignments General 7 128 37 128 Outdoor 5 32 3 17 Jogging 2 4 0 1 Driving 3 8 0 7 DrivingFast 2 4 0 2 Home 4 16 0 9 O ce 7 128 1 65 Meeting 1 2 0 2 Sync 2 4 0 1 6.4 Detecting Context Hazards This class of faults corresponds to sequences of asynchr
- 41. CAAAs Summary ✓Rule-based CAAAs can be extremely fault- prone, even with a small set of rules! ✓The model checking algorithms ﬁnd many actual faults, with different tradeoffs! ✓Some alternative to rule-based adaptation may be preferable
- 42. Uncertainty in Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics
- 43. Uncertainty in Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics Probabilistic Model Checking
- 44. Probabilistic Model Checking Model Checker ✓ ✕ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements ! ¬p → ◊q( )∧"( )
- 45. Probabilistic Model Checking Model Checker ✓ ✕ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements 0.4 0.6 Probabilistic ! ¬p → ◊q( )∧"( )
- 46. P≥0.95 [ ] Probabilistic Model Checking Model Checker ✓ ✕ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements 0.4 0.6 Probabilistic Probabilistic ! ¬p → ◊q( )∧"( )
- 47. P=? [ ] Probabilistic Model Checking Model Checker ✓ ✕ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements 0.4 0.6 Quantitative Results 0.9732Probabilistic Probabilistic ! ¬p → ◊q( )∧"( )
- 48. Example Die Tossing Simulated by Coin Flipping Knuth-Yao algorithm, from the PRISM group (Kwiatkowska et al.) 0 3 2 1 6 4 5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
- 49. Example Die Tossing Simulated by Coin Flipping Knuth-Yao algorithm, from the PRISM group (Kwiatkowska et al.) The behavior is governed by a! theoretical probability distribution 0 3 2 1 6 4 5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
- 50. P≥0.95 [ ] Probabilistic Model Checking Model Checker ✓ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements 0.4 0.6 Quantitative Results 0.9732Probabilistic Probabilistic ! ¬p → ◊q( )∧"( )
- 51. P≥0.95 [ ] Probabilistic Model Checking Model Checker ✓ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements Quantitative Results 0.9732Probabilistic Probabilistic 0.41 0.59 ! ¬p → ◊q( )∧"( )
- 52. P≥0.95 [ ] Probabilistic Model Checking Model Checker ✕ State Machine! Model Temporal Properties Results Counterexample! Trace System Requirements Quantitative Results Probabilistic Probabilistic 0.41 0.59 0.6211 ! ¬p → ◊q( )∧"( )
- 53. Example! Zeroconf Protocol s1s0 s2 s3 q 1 1 {ok} {error} {start} s4 s5 s6 s7 s8 1 1-q 1-p 1-p 1-p 1-p p p p p 1 from the PRISM group (Kwiatkowska et al.)
- 54. Example! Zeroconf Protocol s1s0 s2 s3 q 1 1 {ok} {error} {start} s4 s5 s6 s7 s8 1 1-q 1-p 1-p 1-p 1-p p p p p 1 The behavior is governed by an! empirically estimated probability distribution from the PRISM group (Kwiatkowska et al.) packet-loss rate
- 55. Perturbed Probabilistic Systems • Starting Points! ✓Discrete-Time Markov Chains (DTMCs)! ✓… with one or more probability parameters! ✓… veriﬁed against reachability properties:! ! ✓… and (more recently) LTL properties S? ∪ S! Guoxin Su and David S. Rosenblum,“Asymptotic Bounds for QuantitativeVeriﬁcation of Perturbed Probabilistic Systems”, Proc. ICFEM 2013! ! Guoxin Su and David S. Rosenblum,“Perturbation Analysis of Stochastic Systems with Empirical Distribution Parameters”, Proc. ICSE 2014
- 56. Parametric Markov Chains • A distribution parameter in a DTMC is represented as a vector x of parameters xi! • The norm of total variance represents the amount of perturbation:! ! • The parameter is allowed a “sufﬁciently small” perturbation with respect to ideal reference values r:! ! • Can generalize to multiple parameters v = vi∑ x − r ≤ Δ
- 57. Perturbation Bounds • Perturbation Function! ! where A is the transition probability sub-matrix for S? and b is the vector of one-step probabilities from S? to S! ! • Condition Numbers: [ICFEM 2013]! ! • Quadratic Bounds: [ICSE 2014]! ρ x( )= ι? i A x i i b x( )− Ai i b( )( )i=0 ∞ ∑ κ = lim δ→0 sup ρ(x − r) δ : x − r ≤ δ,δ > 0 ⎧ ⎨ ⎩ ⎫ ⎬ ⎭ f − (δ )− inf ρ(x − r) + f + (δ )− supρ(x − r) = o(δ 2 )
- 58. Results! Noisy Zeroconf (35000 Hosts, PRISM) p Actual Collision Probability Predicted Collision Probability (κ) 0.095 -19.8% -21.5% 0.096 -16.9% -17.2% 0.097 -12.3% -12.9% 0.098 -8.33% -8.61% 0.099 -4.23% -4.30% 0.100 1.8567 — 0.101 +4.38% +4.30% 0.102 +8.91% +8.61% 0.103 +13.6% +12.9% 0.104 +18.4% +17.2% 0.105 +23.4% +21.5%
- 59. Additional Aspects • Models ✓Markov Decision Processes (MDPs)! ✓Continuous-Time Markov Chains (CMTCs)! • Veriﬁcation ✓PCTL Model Checking! with singularities due to nested P[ ] operators! ✓Reward Properties! ✓Alternative Norms and Bounds! Kullback-Leibler Divergence! ✓Parameters as random variables
- 60. Other Forms of Uncertainty “There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say, we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.”! ! — Donald Rumsfeld
- 61. Uncertainty in Testing 1982: Elaine Weyuker: Non-Testable Programs! - Impossible/too costly to efﬁciently check results! - Example: mathematical software! 2010: David Garlan: Intrinsic Uncertainty! - Systems embody intrinsic uncertainty/imprecision! - Cannot easily distinguish bugs from “features”! - Example: ubiquitous computing
- 62. Example! Google Latitude ~ 500m ~ 2m ~ 50m
- 63. Example! Google Latitude When is an incorrect location! a bug, and when is it a “feature”? ~ 500m ~ 2m ~ 50m
- 64. Example! Google Latitude When is an incorrect location! a bug, and when is it a “feature”? And how do! you know? ~ 500m ~ 2m ~ 50m
- 65. Example! Affective Computing
- 66. Example! Affective Computing When is an! incorrect emotion! classiﬁcation a bug,! and when is it a! “feature”?
- 67. Example! Affective Computing When is an! incorrect emotion! classiﬁcation a bug,! and when is it a! “feature”? And how do! you know?
- 68. Sources of Uncertainty ✓Output: results, characteristics of results! ✓Sensors: redundancy, reliability, resolution! ✓Context: sensing, inferring, fusing! ✓Machine learning: imprecision, user-speciﬁcity
- 69. Sources of Uncertainty ✓Output: results, characteristics of results! ✓Sensors: redundancy, reliability, resolution! ✓Context: sensing, inferring, fusing! ✓Machine learning: imprecision, user-speciﬁcity These create signiﬁcant challenges for software engineering research and practice!
- 70. Conclusion ✓Software engineering (certainly) suffers from excessive certainty! ✓A probabilistic mindset offers some insight! ✓But signiﬁcant challenges remain for probabilistic veriﬁcation! ✓And other forms of uncertainty remain a challenge to address
- 71. Jogging While Driving! and Other Software Engineering Research Problems David S. Rosenblum! Dean, School of Computing! National University of Singapore

No public clipboards found for this slide

×
### Save the most important slides with Clipping

Clipping is a handy way to collect and organize the most important slides from a presentation. You can keep your great finds in clipboards organized around topics.

Be the first to comment