Jogging While Driving!
and Other Software Engineering
Research Problems
David S. Rosenblum!
Dean, School of Computing!
Nat...
Singapore
Singapore
Singapore
Singapore Universities
Singapore Universities
Singapore Universities
NUS

School of Computing
✓Ranked #1 in Asia, #9 in the world [QS World University Rankings by Subject]!
✓2 Departments: Co...
Certainty in

Software Engineering
Engineering of software is centered around
simplistic,“yes/no” characterizations of art...
Certainty in

Software Engineering
Engineering of software is centered around
simplistic,“yes/no” characterizations of art...
Example!
Model Checking
Model
Checker
✓
✕
State Machine!
Model
Temporal

Properties
Results
System
Requirements
! ¬p → ◊q(...
Example!
Model Checking
Model
Checker
✕
State Machine!
Model
Temporal

Properties
Results
Counterexample!
Trace
System
Req...
Uncertainty in

Software Engineering
✓Nondeterminism and Asynchrony
✓Randomized Algorithms
✓“Good Enough Software”
✓Test C...
Uncertainty in

Software Engineering
✓Nondeterminism and Asynchrony
✓Randomized Algorithms
✓“Good Enough Software”
✓Test C...
CAAAs
Context-Aware Adaptive Applications
CAAAs
Context-Aware Adaptive Applications
CAAAs
Context-Aware Adaptive Applications
CAAAs
Context-Aware Adaptive Applications
CAAAs
Context-Aware Adaptive Applications
Adaptation in CAAAs
Physical Context
Sensed Context
Inferred Context
Presumed Context
Environment
Context!
Manager
Applica...
Adaptation in CAAAs
Physical Context
Sensed Context
Inferred Context
Presumed Context
Environment
Context!
Manager
Applica...
Adaptation in CAAAs
Physical Context
Sensed Context
Inferred Context
Presumed Context
Environment
Context!
Manager
Applica...
Approach
1.Derive Adaptation Finite-State Machine

(A-FSM) from rule logic!
2.Explore state space of A-FSM to discover
all...
PhoneAdapter
PhoneAdapter
normal,!
vibrate
silent, vibrate
loud, vibratesilent, divert to voicemail
loud,!
divert to!
hands-free
PhoneAdapter
normal,!
vibrate
silent, vibrate
loud, vibratesilent, divert to voicemail
loud,!
divert to!
hands-free
PhoneAdapter A-FSM
Office
Driving!
Fast
Meeting
Driving
Sync
General
Home
Outdoor
Jogging
PhoneAdapter A-FSM
ActivateMeeting DeactivateMeeting
Office
Driving!
Fast
Meeting
Driving
Sync
General
Home
Outdoor
Jogging
PhoneAdapter A-FSM
checking location implies GPS is on!
locations are mutually exclusive!
speeds monotonically increase!
a...
Example Faults in
PhoneAdapter
OfficeGeneral
Home
Example Faults in
PhoneAdapter
User’s phone discovers office PC at home
OfficeGeneral
Home
Example Faults in
PhoneAdapter
Nondeterminism!
OfficeGeneral
Home
Example Faults in
PhoneAdapter
General
Example Faults in
PhoneAdapter
User decides to go somewhere else
GeneralOutdoor
Example Faults in
PhoneAdapter
User starts driving before Bluetooth detects hands-free system
Driving
GeneralOutdoor
Example Faults in
PhoneAdapter
Activation hazard!
Driving
GeneralOutdoor
Jogging
Example Faults in
PhoneAdapter
Activation hazard!
Driving
GeneralOutdoor
Jogging
Faults in CAAAs
• Behavioral Faults!
Nondeterminism!
Dead rule!
Dead state!
!
!
!
!
!
Unreachable state!
Activation race!
...
PhoneAdapter Results
Behavioral Faults: Enumerative, Symbolic
TABLE 2
Faulty Input Configurations Reported for PhoneAdapter...
PhoneAdapter Results
Hazards: Enumerative
n PhoneAdapter
aptation Races and Cycles Context Hazards
signments Race Cycle Pa...
CAAAs
Summary
✓Rule-based CAAAs can be extremely fault-
prone, even with a small set of rules!
✓The model checking algorit...
Uncertainty in

Software Engineering
✓Nondeterminism and Asynchrony
✓Randomized Algorithms
✓“Good Enough Software”
✓Test C...
Uncertainty in

Software Engineering
✓Nondeterminism and Asynchrony
✓Randomized Algorithms
✓“Good Enough Software”
✓Test C...
Probabilistic

Model Checking
Model
Checker
✓
✕
State Machine!
Model
Temporal

Properties
Results
Counterexample!
Trace
Sy...
Probabilistic

Model Checking
Model
Checker
✓
✕
State Machine!
Model
Temporal

Properties
Results
Counterexample!
Trace
Sy...
P≥0.95 [ ]
Probabilistic

Model Checking
Model
Checker
✓
✕
State Machine!
Model
Temporal

Properties
Results
Counterexampl...
P=? [ ]
Probabilistic

Model Checking
Model
Checker
✓
✕
State Machine!
Model
Temporal

Properties
Results
Counterexample!
...
Example

Die Tossing Simulated by Coin Flipping
Knuth-Yao algorithm,

from the PRISM group

(Kwiatkowska et al.)
0
3
2
1
6...
Example

Die Tossing Simulated by Coin Flipping
Knuth-Yao algorithm,

from the PRISM group

(Kwiatkowska et al.)
The behav...
P≥0.95 [ ]
Probabilistic

Model Checking
Model
Checker
✓
State Machine!
Model
Temporal

Properties
Results
Counterexample!...
P≥0.95 [ ]
Probabilistic

Model Checking
Model
Checker
✓
State Machine!
Model
Temporal

Properties
Results
Counterexample!...
P≥0.95 [ ]
Probabilistic

Model Checking
Model
Checker
✕
State Machine!
Model
Temporal

Properties
Results
Counterexample!...
Example!
Zeroconf Protocol
s1s0 s2 s3
q
1
1
{ok} {error}
{start} s4
s5
s6
s7
s8
1
1-q
1-p
1-p
1-p
1-p
p p p
p
1
from the P...
Example!
Zeroconf Protocol
s1s0 s2 s3
q
1
1
{ok} {error}
{start} s4
s5
s6
s7
s8
1
1-q
1-p
1-p
1-p
1-p
p p p
p
1
The behavi...
Perturbed Probabilistic
Systems
• Starting Points!
✓Discrete-Time Markov Chains (DTMCs)!
✓… with one or more probability p...
Parametric

Markov Chains
• A distribution parameter in a DTMC is represented as a
vector x of parameters xi!
• The norm o...
Perturbation Bounds
• Perturbation Function!
!
where A is the transition probability sub-matrix for S?
and b is the vector...
Results!
Noisy Zeroconf (35000 Hosts, PRISM)
p
Actual
Collision Probability
Predicted
Collision Probability (κ)
0.095 -19....
Additional Aspects
• Models
✓Markov Decision Processes (MDPs)!
✓Continuous-Time Markov Chains (CMTCs)!
• Verification
✓PCTL...
Other Forms of
Uncertainty
“There are known knowns; there are things we know
we know. We also know there are known unknown...
Uncertainty in Testing
1982: Elaine Weyuker: Non-Testable Programs!
- Impossible/too costly to efficiently check results!
-...
Example!
Google Latitude
~ 500m
~ 2m
~ 50m
Example!
Google Latitude
When is an

incorrect location!
a bug, and when

is it a “feature”?
~ 500m
~ 2m
~ 50m
Example!
Google Latitude
When is an

incorrect location!
a bug, and when

is it a “feature”?
And how do!
you know?
~ 500m
...
Example!
Affective Computing
Example!
Affective Computing
When is an!
incorrect emotion!
classification a bug,!
and when is it a!
“feature”?
Example!
Affective Computing
When is an!
incorrect emotion!
classification a bug,!
and when is it a!
“feature”?
And how do!...
Sources of

Uncertainty
✓Output: results, characteristics of results!
✓Sensors: redundancy, reliability, resolution!
✓Cont...
Sources of

Uncertainty
✓Output: results, characteristics of results!
✓Sensors: redundancy, reliability, resolution!
✓Cont...
Conclusion
✓Software engineering (certainly) suffers
from excessive certainty!
✓A probabilistic mindset offers some insigh...
Jogging While Driving!
and Other Software Engineering
Research Problems
David S. Rosenblum!
Dean, School of Computing!
Nat...
Upcoming SlideShare
Loading in …5
×

Jogging While Driving, and Other Software Engineering Research Problems (invited talk for UIC Computer Science Distinguished Lecturer Series)

10,037
-1

Published on

invited talk presented for the Distinguished Lecturer Series of the Department of Computer Science at the University of Illinois at Chicago, 10 April 2014

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,037
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
1
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Jogging While Driving, and Other Software Engineering Research Problems (invited talk for UIC Computer Science Distinguished Lecturer Series)

  1. 1. Jogging While Driving! and Other Software Engineering Research Problems David S. Rosenblum! Dean, School of Computing! National University of Singapore
  2. 2. Singapore
  3. 3. Singapore
  4. 4. Singapore
  5. 5. Singapore Universities
  6. 6. Singapore Universities
  7. 7. Singapore Universities
  8. 8. NUS
 School of Computing ✓Ranked #1 in Asia, #9 in the world [QS World University Rankings by Subject]! ✓2 Departments: Computer Science and Information Systems! ✓111 Academic Staff (tenure-track & teaching track)! ✓115 Research Staff! ✓1800 Undergraduate Students! ✓180 Masters Students! ✓350 PhD Students! ✓S$25 million operating budget! ✓S$10 million+ in research income per annum
  9. 9. Certainty in
 Software Engineering Engineering of software is centered around simplistic,“yes/no” characterizations of artifacts
  10. 10. Certainty in
 Software Engineering Engineering of software is centered around simplistic,“yes/no” characterizations of artifacts Program is correct/incorrect Program execution finished/crashed Compilation completed/aborted Test suite succeeded/failed Specification is satisfied/violated
  11. 11. Example! Model Checking Model Checker ✓ ✕ State Machine! Model Temporal
 Properties Results System Requirements ! ¬p → ◊q( )∧"( )
  12. 12. Example! Model Checking Model Checker ✕ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements ! ¬p → ◊q( )∧"( )
  13. 13. Uncertainty in
 Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics
  14. 14. Uncertainty in
 Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics Custom Model Checking Algorithms
  15. 15. CAAAs Context-Aware Adaptive Applications
  16. 16. CAAAs Context-Aware Adaptive Applications
  17. 17. CAAAs Context-Aware Adaptive Applications
  18. 18. CAAAs Context-Aware Adaptive Applications
  19. 19. CAAAs Context-Aware Adaptive Applications
  20. 20. Adaptation in CAAAs Physical Context Sensed Context Inferred Context Presumed Context Environment Context! Manager Application Adaptation! Manager Middleware M. Sama, D.S. Rosenblum, Z.Wang and S. Elbaum,“Multi-Layer Faults in the Architectures of Mobile,
 Context-Aware Adaptive Applications”, Journal of Systems and Software,Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
  21. 21. Adaptation in CAAAs Physical Context Sensed Context Inferred Context Presumed Context Environment Context! Manager Application Adaptation! Manager Middleware Rule Engine M. Sama, D.S. Rosenblum, Z.Wang and S. Elbaum,“Multi-Layer Faults in the Architectures of Mobile,
 Context-Aware Adaptive Applications”, Journal of Systems and Software,Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
  22. 22. Adaptation in CAAAs Physical Context Sensed Context Inferred Context Presumed Context Environment Context! Manager Application Adaptation! Manager Middleware 3rd-Party Libraries Rule Engine M. Sama, D.S. Rosenblum, Z.Wang and S. Elbaum,“Multi-Layer Faults in the Architectures of Mobile,
 Context-Aware Adaptive Applications”, Journal of Systems and Software,Vol. 83, Issue 6, Jun. 2010, pp. 906–914.
  23. 23. Approach 1.Derive Adaptation Finite-State Machine
 (A-FSM) from rule logic! 2.Explore state space of A-FSM to discover all potential faults! ✓Enumerative algorithms! ✓Symbolic algorithms! 3.(Confirm existence of discovered faults) M. Sama, S. Elbaum, F. Raimondi and D.S. Rosenblum,“Context-Aware Adaptive Applications: Fault Patterns and Their Automated Identification”, IEEETransactions on Software Engineering,Vol. 36, No. 5, Sep./Oct. 2010, pp. 644-661.
  24. 24. PhoneAdapter
  25. 25. PhoneAdapter normal,! vibrate silent, vibrate loud, vibratesilent, divert to voicemail loud,! divert to! hands-free
  26. 26. PhoneAdapter normal,! vibrate silent, vibrate loud, vibratesilent, divert to voicemail loud,! divert to! hands-free
  27. 27. PhoneAdapter A-FSM Office Driving! Fast Meeting Driving Sync General Home Outdoor Jogging
  28. 28. PhoneAdapter A-FSM ActivateMeeting DeactivateMeeting Office Driving! Fast Meeting Driving Sync General Home Outdoor Jogging
  29. 29. PhoneAdapter A-FSM checking location implies GPS is on! locations are mutually exclusive! speeds monotonically increase! a meeting’s end time is later than its start time Global constraints: ActivateMeeting DeactivateMeeting Office Driving! Fast Meeting Driving Sync General Home Outdoor Jogging
  30. 30. Example Faults in PhoneAdapter OfficeGeneral Home
  31. 31. Example Faults in PhoneAdapter User’s phone discovers office PC at home OfficeGeneral Home
  32. 32. Example Faults in PhoneAdapter Nondeterminism! OfficeGeneral Home
  33. 33. Example Faults in PhoneAdapter General
  34. 34. Example Faults in PhoneAdapter User decides to go somewhere else GeneralOutdoor
  35. 35. Example Faults in PhoneAdapter User starts driving before Bluetooth detects hands-free system Driving GeneralOutdoor
  36. 36. Example Faults in PhoneAdapter Activation hazard! Driving GeneralOutdoor Jogging
  37. 37. Example Faults in PhoneAdapter Activation hazard! Driving GeneralOutdoor Jogging
  38. 38. Faults in CAAAs • Behavioral Faults! Nondeterminism! Dead rule! Dead state! ! ! ! ! ! Unreachable state! Activation race! Activation cycle • Hazards! Hold hazard! Activation hazard! ! Priority inversion
 hazard
  39. 39. PhoneAdapter Results Behavioral Faults: Enumerative, Symbolic TABLE 2 Faulty Input Configurations Reported for PhoneAdapter State Nondeterministic Dead Adaptation Unreachable Adaptations Predicates Races Cycles States General 37 1 45 13 0 Outdoor 3 0 135 23 0 Jogging 0 0 97 19 0 Driving 0 0 36 13 0 DrivingFast 0 0 58 19 0 Home 0 0 76 19 0 Office 0 0 29 1 0 Meeting 0 0 32 1 0 Sync 0 0 27 5 1
  40. 40. PhoneAdapter Results Hazards: Enumerative n PhoneAdapter aptation Races and Cycles Context Hazards signments Race Cycle Paths Hold Activ. Prior. 3968 45 13 14085 0 11 3182 3968 135 23 161 0 0 52 3072 97 19 2 0 0 0 2560 36 13 16 2 2 4 3072 58 19 2 0 0 0 2816 76 19 104 8 0 13 2848 29 1 82634 1828 368 2164 2048 32 1 0 0 0 0 1024 27 5 2 2 0 0 ned a formal model of a key complex behavioral char- eristic, namely adaptation, of an increasingly large and Table 2: Faults State Vars. Nondet. Adaptation Dead Pred Assignments Faults Assignments General 7 128 37 128 Outdoor 5 32 3 17 Jogging 2 4 0 1 Driving 3 8 0 7 DrivingFast 2 4 0 2 Home 4 16 0 9 O ce 7 128 1 65 Meeting 1 2 0 2 Sync 2 4 0 1 6.4 Detecting Context Hazards This class of faults corresponds to sequences of asynchr
  41. 41. CAAAs Summary ✓Rule-based CAAAs can be extremely fault- prone, even with a small set of rules! ✓The model checking algorithms find many actual faults, with different tradeoffs! ✓Some alternative to rule-based adaptation may be preferable
  42. 42. Uncertainty in
 Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics
  43. 43. Uncertainty in
 Software Engineering ✓Nondeterminism and Asynchrony ✓Randomized Algorithms ✓“Good Enough Software” ✓Test Coverage Metrics Probabilistic Model Checking
  44. 44. Probabilistic
 Model Checking Model Checker ✓ ✕ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements ! ¬p → ◊q( )∧"( )
  45. 45. Probabilistic
 Model Checking Model Checker ✓ ✕ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements 0.4 0.6 Probabilistic ! ¬p → ◊q( )∧"( )
  46. 46. P≥0.95 [ ] Probabilistic
 Model Checking Model Checker ✓ ✕ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements 0.4 0.6 Probabilistic Probabilistic ! ¬p → ◊q( )∧"( )
  47. 47. P=? [ ] Probabilistic
 Model Checking Model Checker ✓ ✕ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements 0.4 0.6 Quantitative Results 0.9732Probabilistic Probabilistic ! ¬p → ◊q( )∧"( )
  48. 48. Example
 Die Tossing Simulated by Coin Flipping Knuth-Yao algorithm,
 from the PRISM group
 (Kwiatkowska et al.) 0 3 2 1 6 4 5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  49. 49. Example
 Die Tossing Simulated by Coin Flipping Knuth-Yao algorithm,
 from the PRISM group
 (Kwiatkowska et al.) The behavior is governed by a! theoretical probability distribution 0 3 2 1 6 4 5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
  50. 50. P≥0.95 [ ] Probabilistic
 Model Checking Model Checker ✓ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements 0.4 0.6 Quantitative Results 0.9732Probabilistic Probabilistic ! ¬p → ◊q( )∧"( )
  51. 51. P≥0.95 [ ] Probabilistic
 Model Checking Model Checker ✓ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements Quantitative Results 0.9732Probabilistic Probabilistic 0.41 0.59 ! ¬p → ◊q( )∧"( )
  52. 52. P≥0.95 [ ] Probabilistic
 Model Checking Model Checker ✕ State Machine! Model Temporal
 Properties Results Counterexample! Trace System Requirements Quantitative Results Probabilistic Probabilistic 0.41 0.59 0.6211 ! ¬p → ◊q( )∧"( )
  53. 53. Example! Zeroconf Protocol s1s0 s2 s3 q 1 1 {ok} {error} {start} s4 s5 s6 s7 s8 1 1-q 1-p 1-p 1-p 1-p p p p p 1 from the PRISM group
 (Kwiatkowska et al.)
  54. 54. Example! Zeroconf Protocol s1s0 s2 s3 q 1 1 {ok} {error} {start} s4 s5 s6 s7 s8 1 1-q 1-p 1-p 1-p 1-p p p p p 1 The behavior is governed by an! empirically estimated probability distribution from the PRISM group
 (Kwiatkowska et al.) packet-loss rate
  55. 55. Perturbed Probabilistic Systems • Starting Points! ✓Discrete-Time Markov Chains (DTMCs)! ✓… with one or more probability parameters! ✓… verified against reachability properties:! ! ✓… and (more recently) LTL properties S? ∪ S! Guoxin Su and David S. Rosenblum,“Asymptotic Bounds for QuantitativeVerification of
 Perturbed Probabilistic Systems”, Proc. ICFEM 2013! ! Guoxin Su and David S. Rosenblum,“Perturbation Analysis of Stochastic Systems with
 Empirical Distribution Parameters”, Proc. ICSE 2014
  56. 56. Parametric
 Markov Chains • A distribution parameter in a DTMC is represented as a vector x of parameters xi! • The norm of total variance represents the amount of perturbation:! ! • The parameter is allowed a “sufficiently small” perturbation with respect to ideal reference values r:! ! • Can generalize to multiple parameters v = vi∑ x − r ≤ Δ
  57. 57. Perturbation Bounds • Perturbation Function! ! where A is the transition probability sub-matrix for S? and b is the vector of one-step probabilities from S? to S! ! • Condition Numbers: [ICFEM 2013]! ! • Quadratic Bounds: [ICSE 2014]! ρ x( )= ι? i A x i i b x( )− Ai i b( )( )i=0 ∞ ∑ κ = lim δ→0 sup ρ(x − r) δ : x − r ≤ δ,δ > 0 ⎧ ⎨ ⎩ ⎫ ⎬ ⎭ f − (δ )− inf ρ(x − r) + f + (δ )− supρ(x − r) = o(δ 2 )
  58. 58. Results! Noisy Zeroconf (35000 Hosts, PRISM) p Actual Collision Probability Predicted Collision Probability (κ) 0.095 -19.8% -21.5% 0.096 -16.9% -17.2% 0.097 -12.3% -12.9% 0.098 -8.33% -8.61% 0.099 -4.23% -4.30% 0.100 1.8567 — 0.101 +4.38% +4.30% 0.102 +8.91% +8.61% 0.103 +13.6% +12.9% 0.104 +18.4% +17.2% 0.105 +23.4% +21.5%
  59. 59. Additional Aspects • Models ✓Markov Decision Processes (MDPs)! ✓Continuous-Time Markov Chains (CMTCs)! • Verification ✓PCTL Model Checking! with singularities due to nested P[ ] operators! ✓Reward Properties! ✓Alternative Norms and Bounds! Kullback-Leibler Divergence! ✓Parameters as random variables
  60. 60. Other Forms of Uncertainty “There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say, we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.”! ! — Donald Rumsfeld
  61. 61. Uncertainty in Testing 1982: Elaine Weyuker: Non-Testable Programs! - Impossible/too costly to efficiently check results! - Example: mathematical software! 2010: David Garlan: Intrinsic Uncertainty! - Systems embody intrinsic uncertainty/imprecision! - Cannot easily distinguish bugs from “features”! - Example: ubiquitous computing
  62. 62. Example! Google Latitude ~ 500m ~ 2m ~ 50m
  63. 63. Example! Google Latitude When is an
 incorrect location! a bug, and when
 is it a “feature”? ~ 500m ~ 2m ~ 50m
  64. 64. Example! Google Latitude When is an
 incorrect location! a bug, and when
 is it a “feature”? And how do! you know? ~ 500m ~ 2m ~ 50m
  65. 65. Example! Affective Computing
  66. 66. Example! Affective Computing When is an! incorrect emotion! classification a bug,! and when is it a! “feature”?
  67. 67. Example! Affective Computing When is an! incorrect emotion! classification a bug,! and when is it a! “feature”? And how do! you know?
  68. 68. Sources of
 Uncertainty ✓Output: results, characteristics of results! ✓Sensors: redundancy, reliability, resolution! ✓Context: sensing, inferring, fusing! ✓Machine learning: imprecision, user-specificity
  69. 69. Sources of
 Uncertainty ✓Output: results, characteristics of results! ✓Sensors: redundancy, reliability, resolution! ✓Context: sensing, inferring, fusing! ✓Machine learning: imprecision, user-specificity These create significant challenges for
 software engineering research and practice!
  70. 70. Conclusion ✓Software engineering (certainly) suffers from excessive certainty! ✓A probabilistic mindset offers some insight! ✓But significant challenges remain for probabilistic verification! ✓And other forms of uncertainty remain a challenge to address
  71. 71. Jogging While Driving! and Other Software Engineering Research Problems David S. Rosenblum! Dean, School of Computing! National University of Singapore

×