Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Test Case as Executable Example

2,376 views

Published on

CWI Lectures on Understanding Software, in honor of Paul Klint. June 14, 2012.

Published in: Technology, Education
  • Be the first to comment

The Test Case as Executable Example

  1. 1. The Test Case as Executable ExampleCWI Lectures – Understanding Software June 14, 2012 Arie van Deursen @avandeursen 1
  2. 2. 20th International Conference on Program Comprehension Study ofthe human activity of comprehending software;the processes and technologies for supporting it 2
  3. 3. Programming by ExampleProgramming by Example, Spreadsheet Data Manipulation UsingDaniel Halbert, 1984, Xerox Examples, CACM, 2012 3
  4. 4. The TestingPerspective• A test case is an executable example of system behavior• Stakeholder communication needs examples• Throughout the full development cycle 4
  5. 5. webrat_steps.rb:Cucumber Then /^I should see "([^"]*)"$/ do |text| response.should contain(text) endFeature: Log in and out As an administrator I want to restrict access to certain portions of my site In order to prevent users from changing the content Scenario: Logging in Given I am not logged in as an administrator When I go to the administrative page And I fill in the fields | Username | admin | | Password | secret | And I press "Log in" Then I should be on the administrative page And I should see "Log out" Scenario: Logging out ... 5
  6. 6. Plugin / Junit Testing in Eclipse. 1000s of tests 6
  7. 7. Test Suite Information Needs Test Confessions: A Study of Testing Practices for Plug-In SystemsMichaela Greiler, Arie van Deursen, Margaret-Anne Storey, ICSE 2012 7
  8. 8. Challenges• Set of examples is incomplete• Set of examples is too large – Some grouping needed – Redundancy Test Similarity• Set of examples is disconnected – Requirements traceability – Backward coverage 8
  9. 9. Requirements End-to-End Analysis Test ... ... High Level Integration Design Test Code Unit TestMeasuring Test Case Similarity to Support Test Suite Understanding Michaela Greiler, Arie van Deursen, Andy Zaidman, TOOLS 2012 9
  10. 10. Measuring Test Case Similarity Shared word count Test-aware Test Tests rankedinstrumentation Traces by similarity 10
  11. 11. Preliminary Evaluation• 75 unit tests • 37 unit• 14 acceptance tests • 6 end to end• Ground truth by hand • Ground truth from book• Most similar match: #1 or #2 in ground truth• Top 10: all (but one) explainable & useful• Surprises: mostly point to issue in test suite.• Work in progress: Analyzing Eclipse test suites. 11
  12. 12. Rethinking “Test Cases as Examples” • Languages for writing examples – Different levels of abstraction / stakeholders – Executable – Analyzable – Connected to execution traces• Modularization and grouping of examples – Different levels of abstraction / stakeholders – Requirements / design / code coverage – Multiple views – Connected via execution traces 12
  13. 13. In Conclusion• A test case is an executable example• Never enough, yet also too many• Test case connection via trace similarities• Language engineering opportunities ahead 13
  14. 14. Photo credits• Passau: Andreas Brandl http://icpc12.sosy-lab.org/• Programming by Example, picture JanHeering (CWI). Photo Eelco Visser. http://eelcovisser.org/post/91/example-driven-research• The Testing Perspective: Photo “Reaching for Understanding”, Flickr http://www.flickr.com/photos/elizabethsalib/2741807789/• Andromeda Galaxy http://www.nasa.gov/mission_pages/spitzer/multimedia/spitzer- 20070604.html• Colliding galaxies http://hubblesite.org/newscenter/archive/releases/2007/08/• Paul Klint, lecturing. http://www.frankwatching.com/archive/2009/05/18/het-nationaal-ict- event/ 14

×