• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Jogging While Driving, and Other Software Engineering Research Problems (invited talk for UIC Computer Science Distinguished Lecturer Series)
 

Jogging While Driving, and Other Software Engineering Research Problems (invited talk for UIC Computer Science Distinguished Lecturer Series)

on

  • 1,396 views

invited talk presented for the Distinguished Lecturer Series of the Department of Computer Science at the University of Illinois at Chicago, 10 April 2014

invited talk presented for the Distinguished Lecturer Series of the Department of Computer Science at the University of Illinois at Chicago, 10 April 2014

Statistics

Views

Total Views
1,396
Views on SlideShare
422
Embed Views
974

Actions

Likes
2
Downloads
0
Comments
0

7 Embeds 974

http://www.comp.nus.edu.sg 963
https://www.linkedin.com 3
http://www.linkedin.com 2
https://www.google.com.sg 2
http://www.google.com.sg 2
http://www0.comp.nus.edu 1
http://www.google.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Jogging While Driving, and Other Software Engineering Research Problems (invited talk for UIC Computer Science Distinguished Lecturer Series) Jogging While Driving, and Other Software Engineering Research Problems (invited talk for UIC Computer Science Distinguished Lecturer Series) Presentation Transcript

    • Jogging While Driving! and Other Software Engineering Research Problems David S. Rosenblum! Dean, School of Computing! National University of Singapore
    • Singapore
    • Singapore
    • Singapore Universities
    • Singapore Universities
    • NUS
 School of Computing ✓Ranked #1 in Asia, #9 in the world [QS World University Rankings by Subject]! ✓2 Departments: Computer Science and Information Systems! ✓111 Academic Staff (tenure-track & teaching track)! ✓115 Research Staff! ✓1800 Undergraduate Students! ✓180 Masters Students! ✓350 PhD Students! ✓S$25 million operating budget! ✓S$10 million+ in research income per annum
    • Certainty in
 Software Engineering Engineering of software is centered around simplistic,“yes/no” characterizations of artifacts Program is correct/incorrect Program execution finished/crashed Compilation completed/aborted Test suite succeeded/failed Specification is satisfied/violated
    • Example! Model Checking Model Checker ✓ ✕ State Machine! Model Temporal
 Properties Results System Requirements ! ¬p → ◊q( )∧"( )
    • Example! Model Checking Model Checker ✕ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements ! ¬p → ◊q( )∧"( )
    • Uncertainty in
 Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics
    • Uncertainty in
 Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics Custom Model Checking Algorithms
    • CAAAs Context-Aware Adaptive Applications
    • CAAAs Context-Aware Adaptive Applications
    • CAAAs Context-Aware Adaptive Applications
    • CAAAs Context-Aware Adaptive Applications
    • Adaptation in CAAAs Physical Context Sensed Context Inferred Context Presumed Context Environment Context! Manager Application Adaptation! Manager Middleware 3rd-Party Libraries Rule Engine M. Sama, D.S. Rosenblum, Z.Wang and S. Elbaum,“Multi-Layer Faults in the Architectures of Mobile,
 Context-Aware Adaptive Applications”, Journal of Systems and Software,Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
    • Approach 1.Derive Adaptation Finite-State Machine
 (A-FSM) from rule logic! 2.Explore state space of A-FSM to discover all potential faults! ✓Enumerative algorithms! ✓Symbolic algorithms! 3.(Confirm existence of discovered faults) M. Sama, S. Elbaum, F. Raimondi and D.S. Rosenblum,“Context-Aware Adaptive Applications: Fault Patterns and Their Automated Identification”, IEEETransactions on Software Engineering,Vol. 36, No. 5, Sep./Oct. 2010, pp. 644-661.
    • PhoneAdapter
    • PhoneAdapter normal,! vibrate silent, vibrate loud, vibratesilent, divert to voicemail loud,! divert to! hands-free
    • PhoneAdapter A-FSM checking location implies GPS is on! locations are mutually exclusive! speeds monotonically increase! a meeting’s end time is later than its start time Global constraints: ActivateMeeting DeactivateMeeting Office Driving! Fast Meeting Driving Sync General Home Outdoor Jogging
    • Example Faults in PhoneAdapter OfficeGeneral Home
    • Example Faults in PhoneAdapter User’s phone discovers office PC at home OfficeGeneral Home
    • Example Faults in PhoneAdapter Nondeterminism! OfficeGeneral Home
    • Example Faults in PhoneAdapter General
    • Example Faults in PhoneAdapter User decides to go somewhere else GeneralOutdoor
    • Example Faults in PhoneAdapter User starts driving before Bluetooth detects hands-free system Driving GeneralOutdoor
    • Example Faults in PhoneAdapter Activation hazard! Driving GeneralOutdoor Jogging
    • Example Faults in PhoneAdapter Activation hazard! Driving GeneralOutdoor Jogging
    • Faults in CAAAs • Behavioral Faults! Nondeterminism! Dead rule! Dead state! ! ! ! ! ! Unreachable state! Activation race! Activation cycle • Hazards! Hold hazard! Activation hazard! ! Priority inversion
 hazard
    • PhoneAdapter Results Behavioral Faults: Enumerative, Symbolic TABLE 2 Faulty Input Configurations Reported for PhoneAdapter State Nondeterministic Dead Adaptation Unreachable Adaptations Predicates Races Cycles States General 37 1 45 13 0 Outdoor 3 0 135 23 0 Jogging 0 0 97 19 0 Driving 0 0 36 13 0 DrivingFast 0 0 58 19 0 Home 0 0 76 19 0 Office 0 0 29 1 0 Meeting 0 0 32 1 0 Sync 0 0 27 5 1
    • PhoneAdapter Results Hazards: Enumerative in PhoneAdapter Adaptation Races and Cycles Context Hazards Assignments Race Cycle Paths Hold Activ. Prior. 3968 45 13 14085 0 11 3182 3968 135 23 161 0 0 52 3072 97 19 2 0 0 0 2560 36 13 16 2 2 4 3072 58 19 2 0 0 0 2816 76 19 104 8 0 13 2848 29 1 82634 1828 368 2164 2048 32 1 0 0 0 0 1024 27 5 2 2 0 0 efined a formal model of a key complex behavioral char- cteristic, namely adaptation, of an increasingly large and Table 2: Faults D State Vars. Nondet. Adaptation Dead Predica Assignments Faults Assignments General 7 128 37 128 Outdoor 5 32 3 17 Jogging 2 4 0 1 Driving 3 8 0 7 DrivingFast 2 4 0 2 Home 4 16 0 9 O ce 7 128 1 65 Meeting 1 2 0 2 Sync 2 4 0 1 6.4 Detecting Context Hazards This class of faults corresponds to sequences of asynchron
    • CAAAs Summary ✓Rule-based CAAAs can be extremely fault- prone, even with a small set of rules! ✓The model checking algorithms find many actual faults, with different tradeoffs! ✓Some alternative to rule-based adaptation may be preferable
    • Uncertainty in
 Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics Probabilistic Model Checking
    • Probabilistic
 Model Checking Model Checker ✓ ✕ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements ! ¬p → ◊q( )∧"( )
    • P≥0.95 [ ] Probabilistic
 Model Checking Model Checker ✓ ✕ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements 0.4 0.6 Probabilistic Probabilistic ! ¬p → ◊q( )∧"( )
    • P=? [ ] Probabilistic
 Model Checking Model Checker ✓ ✕ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements 0.4 0.6 Quantitative Results 0.9732Probabilistic Probabilistic ! ¬p → ◊q( )∧"( )
    • Example
 Die Tossing Simulated by Coin Flipping Knuth-Yao algorithm,
 from the PRISM group
 (Kwiatkowska et al.) 0 3 2 1 6 4 5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
    • Example
 Die Tossing Simulated by Coin Flipping Knuth-Yao algorithm,
 from the PRISM group
 (Kwiatkowska et al.) The behavior is governed by a! theoretical probability distribution 0 3 2 1 6 4 5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
    • P≥0.95 [ ] Probabilistic
 Model Checking Model Checker ✓ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements 0.4 0.6 Quantitative Results 0.9732Probabilistic Probabilistic ! ¬p → ◊q( )∧"( )
    • P≥0.95 [ ] Probabilistic
 Model Checking Model Checker ✕ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements Quantitative Results Probabilistic Probabilistic 0.41 0.59 0.6211 ! ¬p → ◊q( )∧"( )
    • Example! Zeroconf Protocol s1s0 s2 s3 q 1 1 {ok} {error} {start} s4 s5 s6 s7 s8 1 1-q 1-p 1-p 1-p 1-p p p p p 1 from the PRISM group
 (Kwiatkowska et al.)
    • Example! Zeroconf Protocol s1s0 s2 s3 q 1 1 {ok} {error} {start} s4 s5 s6 s7 s8 1 1-q 1-p 1-p 1-p 1-p p p p p 1 The behavior is governed by an! empirically estimated probability distribution from the PRISM group
 (Kwiatkowska et al.) packet-loss rate
    • Perturbed Probabilistic Systems • Starting Points! ✓Discrete-Time Markov Chains (DTMCs)! ✓… with one or more probability parameters! ✓… verified against reachability properties:! ! ✓… and (more recently) LTL properties S? ∪ S! Guoxin Su and David S. Rosenblum,“Asymptotic Bounds for QuantitativeVerification of
 Perturbed Probabilistic Systems”, Proc. ICFEM 2013! ! Guoxin Su and David S. Rosenblum,“Perturbation Analysis of Stochastic Systems with
 Empirical Distribution Parameters”, Proc. ICSE 2014
    • Parametric
 Markov Chains • A distribution parameter in a DTMC is represented as a vector x of parameters xi! • The norm of total variance represents the amount of perturbation:! ! • The parameter is allowed a “sufficiently small” perturbation with respect to ideal reference values r:! ! • Can generalize to multiple parameters v = vi∑ x − r ≤ Δ
    • Perturbation Bounds • Perturbation Function! ! where A is the transition probability sub-matrix for S? and b is the vector of one-step probabilities from S? to S! ! • Condition Numbers: [ICFEM 2013]! ! • Quadratic Bounds: [ICSE 2014]! ρ x( )= ι? i A x i i b x( )− Ai i b( )( )i=0 ∞ ∑ κ = lim δ→0 sup ρ(x − r) δ : x − r ≤ δ,δ > 0 ⎧ ⎨ ⎩ ⎫ ⎬ ⎭ f − (δ )− inf ρ(x − r) + f + (δ )− supρ(x − r) = o(δ 2 )
    • Results! Noisy Zeroconf (35000 Hosts, PRISM) p Actual Collision Probability Predicted Collision Probability (κ) 0.095 -19.8% -21.5% 0.096 -16.9% -17.2% 0.097 -12.3% -12.9% 0.098 -8.33% -8.61% 0.099 -4.23% -4.30% 0.100 1.8567 — 0.101 +4.38% +4.30% 0.102 +8.91% +8.61% 0.103 +13.6% +12.9% 0.104 +18.4% +17.2% 0.105 +23.4% +21.5%
    • Additional Aspects • Models ✓Markov Decision Processes (MDPs)! ✓Continuous-Time Markov Chains (CMTCs)! • Verification ✓PCTL Model Checking! with singularities due to nested P[ ] operators! ✓Reward Properties! ✓Alternative Norms and Bounds! Kullback-Leibler Divergence! ✓Parameters as random variables
    • Other Forms of Uncertainty “There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say, we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.”! ! — Donald Rumsfeld
    • Uncertainty in Testing 1982: Elaine Weyuker: Non-Testable Programs! - Impossible/too costly to efficiently check results! - Example: mathematical software! 2010: David Garlan: Intrinsic Uncertainty! - Systems embody intrinsic uncertainty/imprecision! - Cannot easily distinguish bugs from “features”! - Example: ubiquitous computing
    • Example! Google Latitude ~ 500m ~ 2m ~ 50m
    • Example! Google Latitude When is an
 incorrect location! a bug, and when
 is it a “feature”? And how do! you know? ~ 500m ~ 2m ~ 50m
    • Example! Affective Computing
    • Example! Affective Computing When is an! incorrect emotion! classification a bug,! and when is it a! “feature”? And how do! you know?
    • Sources of
 Uncertainty ✓Output: results, characteristics of results! ✓Sensors: redundancy, reliability, resolution! ✓Context: sensing, inferring, fusing! ✓Machine learning: imprecision, user-specificity These create significant challenges for
 software engineering research and practice!
    • Conclusion ✓Software engineering (certainly) suffers from excessive certainty! ✓A probabilistic mindset offers some insight! ✓But significant challenges remain for probabilistic verification! ✓And other forms of uncertainty remain a challenge to address
    • Jogging While Driving! and Other Software Engineering Research Problems David S. Rosenblum! Dean, School of Computing! National University of Singapore