Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Known Unknowns: Testing in the Presence of Uncertainty (talk at ACM SIGSOFT FSE 2014 Visions & Challenges Track)

21,961 views

Published on

talk presented in the Visions & Challenges Track of the ACM SIGSOFT 22nd International Symposium on the Foundations of Software Engineering (FSE 2014), Hong Kong, 20 November 2014; the paper won 2nd Prize in the track

Published in: Software

Known Unknowns: Testing in the Presence of Uncertainty (talk at ACM SIGSOFT FSE 2014 Visions & Challenges Track)

  1. 1. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 Known Unknowns:! Testing in the Presence of Uncertainty Sebastian Elbaum! University of Nebraska–Lincoln! elbaum@cse.unl.edu! David S. Rosenblum National University of Singapore! david@comp.nus.edu.sg
  2. 2. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 Known Unknowns “There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say, we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.”! ! — Donald Rumsfeld
  3. 3. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 Known Unknowns in Software Testing Test
 Execution System Under Test Result Interpretation
  4. 4. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 Known Unknowns in Software Testing Test
 Execution System Under Test Result Interpretation Acceptable✓
  5. 5. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 Known Unknowns in Software Testing Test
 Execution System Under Test Result Interpretation Unacceptable Acceptable✓ ✕
  6. 6. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 Known Unknowns in Software Testing Test
 Execution System Under Test Result Interpretation Unacceptable Acceptable Acceptable ✓ ✕ ✕
  7. 7. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example! Amazon Recommendations Amazon
 Music Result Interpretation
  8. 8. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example! Amazon Recommendations Amazon
 Music Result Interpretation
  9. 9. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example! Amazon Recommendations Amazon
 Music Result Interpretation Acceptable✓
  10. 10. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example! Amazon Recommendations Amazon
 Music Result Interpretation Acceptable✓
  11. 11. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example! Amazon Recommendations Amazon
 Music Result Interpretation Unacceptable Acceptable✓ ✕
  12. 12. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example! Amazon Recommendations Amazon
 Music Result Interpretation Unacceptable Acceptable✓ ✕
  13. 13. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example! Amazon Recommendations Amazon
 Music Result Interpretation Unacceptable Acceptable Acceptable ✓ ✕ ✕
  14. 14. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 The Problem Such acceptable misbehavior
 may be due to the! intrinsic uncertainty of some component of the system under test or it may be masking real faults!
  15. 15. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 The Problem Such acceptable misbehavior
 may be due to the! intrinsic uncertainty of some component of the system under test How can we reduce the uncertainty of these
 known unknowns? or it may be masking real faults!
  16. 16. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 Handling Uncertainty Today! Amazon Recommendations Amazon
 Music Result Interpretation Unacceptable Acceptable Acceptable ✓ ✕ ✕
  17. 17. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 Handling Uncertainty Today! Amazon Recommendations Amazon
 Music Result Interpretation Unacceptable Acceptable Acceptable ✓ ✕ ✕
  18. 18. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 Handling Uncertainty Today! Amazon Recommendations Amazon
 Music Result Interpretation Unacceptable Acceptable✓ ✕ ✕ Unacceptable
  19. 19. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 Handling Uncertainty Today! Amazon Recommendations Amazon
 Music Result Interpretation Unacceptable Acceptable✓ ✕
  20. 20. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example Approach! HMMs and Distribution Fitting System Under Test Training
 Data WEKA
  21. 21. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example Approach! HMMs and Distribution Fitting System Under Test WEKA
  22. 22. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example Approach! HMMs and Distribution Fitting System Under Test WEKA
  23. 23. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example Approach! HMMs and Distribution Fitting System Under Test WEKA
  24. 24. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example Approach! HMMs and Distribution Fitting System Under Test Result Interpretation Acceptablep < 0.99 Test
 Execution WEKA
  25. 25. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example Approach! HMMs and Distribution Fitting System Under Test Result Interpretation Unacceptable Acceptablep < 0.99 Test
 Execution WEKA p < 0.0027
  26. 26. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example Approach! HMMs and Distribution Fitting System Under Test Result Interpretation Unacceptable Acceptable Inconclusive p < 0.99 Test
 Execution WEKA p < 0.37 p < 0.0027
  27. 27. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 An Example Approach! HMMs and Distribution Fitting System Under Test Result Interpretation Unacceptable Acceptable Inconclusive p < 0.99 Test Execution W p < 0.37 p < 0.0027 see the paper for more details!
  28. 28. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 Conclusion • Software systems increasingly exhibit many kinds of uncertainty • This uncertainty greatly complicates testing • New testing approaches are needed for revealing faults masked by uncertainty
  29. 29. SIGSOFT FSE 2014 Visions & Challenges! ! ! ! 20 November 2014 Known Unknowns:! Testing in the Presence of Uncertainty Sebastian Elbaum! University of Nebraska–Lincoln! elbaum@cse.unl.edu! David S. Rosenblum! National University of Singapore! david@comp.nus.edu.sg Thank you!

×