The Test Case as Executable Example


Published on

CWI Lectures on Understanding Software, in honor of Paul Klint. June 14, 2012.

Published in: Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

The Test Case as Executable Example

  1. 1. The Test Case as Executable ExampleCWI Lectures – Understanding Software June 14, 2012 Arie van Deursen @avandeursen 1
  2. 2. 20th International Conference on Program Comprehension Study ofthe human activity of comprehending software;the processes and technologies for supporting it 2
  3. 3. Programming by ExampleProgramming by Example, Spreadsheet Data Manipulation UsingDaniel Halbert, 1984, Xerox Examples, CACM, 2012 3
  4. 4. The TestingPerspective• A test case is an executable example of system behavior• Stakeholder communication needs examples• Throughout the full development cycle 4
  5. 5. webrat_steps.rb:Cucumber Then /^I should see "([^"]*)"$/ do |text| response.should contain(text) endFeature: Log in and out As an administrator I want to restrict access to certain portions of my site In order to prevent users from changing the content Scenario: Logging in Given I am not logged in as an administrator When I go to the administrative page And I fill in the fields | Username | admin | | Password | secret | And I press "Log in" Then I should be on the administrative page And I should see "Log out" Scenario: Logging out ... 5
  6. 6. Plugin / Junit Testing in Eclipse. 1000s of tests 6
  7. 7. Test Suite Information Needs Test Confessions: A Study of Testing Practices for Plug-In SystemsMichaela Greiler, Arie van Deursen, Margaret-Anne Storey, ICSE 2012 7
  8. 8. Challenges• Set of examples is incomplete• Set of examples is too large – Some grouping needed – Redundancy Test Similarity• Set of examples is disconnected – Requirements traceability – Backward coverage 8
  9. 9. Requirements End-to-End Analysis Test ... ... High Level Integration Design Test Code Unit TestMeasuring Test Case Similarity to Support Test Suite Understanding Michaela Greiler, Arie van Deursen, Andy Zaidman, TOOLS 2012 9
  10. 10. Measuring Test Case Similarity Shared word count Test-aware Test Tests rankedinstrumentation Traces by similarity 10
  11. 11. Preliminary Evaluation• 75 unit tests • 37 unit• 14 acceptance tests • 6 end to end• Ground truth by hand • Ground truth from book• Most similar match: #1 or #2 in ground truth• Top 10: all (but one) explainable & useful• Surprises: mostly point to issue in test suite.• Work in progress: Analyzing Eclipse test suites. 11
  12. 12. Rethinking “Test Cases as Examples” • Languages for writing examples – Different levels of abstraction / stakeholders – Executable – Analyzable – Connected to execution traces• Modularization and grouping of examples – Different levels of abstraction / stakeholders – Requirements / design / code coverage – Multiple views – Connected via execution traces 12
  13. 13. In Conclusion• A test case is an executable example• Never enough, yet also too many• Test case connection via trace similarities• Language engineering opportunities ahead 13
  14. 14. Photo credits• Passau: Andreas Brandl• Programming by Example, picture JanHeering (CWI). Photo Eelco Visser.• The Testing Perspective: Photo “Reaching for Understanding”, Flickr• Andromeda Galaxy 20070604.html• Colliding galaxies• Paul Klint, lecturing. event/ 14