This document presents a technique for generating unit test blueprints from execution traces of existing code. The technique instruments a program to collect execution traces and object flow data. An analysis of the traces then extracts test scenarios to generate blueprints showing how to set up test fixtures, execute units under test, and verify expected behavior. An initial case study found the tool worked well at producing tests but had difficulties selecting execution units and initializing objects. The blueprints provide guidance for writing tests for legacy code even when it is not well understood.
Streamlining Python Development: A Guide to a Modern Project Setup
Test Blueprints
1. Test Blueprints
Exposing Side Effects in Execution
Traces to Suport Writing Unit Tests
Adrian Lienhard, Tudor Gîrba,
Orla Greevy and Oscar Nierstrasz
Software Composition Group
University of Bern, Switzerland
Thanks for the support:
2. Unit Testing
Everybody knows
...unit tests are important!
...write tests first!
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
3. Unit Testing
Everybody knows
...unit tests are important!
...write tests first!
Fair enough, but what if
...the code exists already?
...and you don’t understand it well?
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
4. Unit Testing
Yet, writing unit tests for
legacy systems is important.
“Tests are your life insurance”
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
5. Challenges of Testing
Legacy Code
1. Selecting a unit to test
choose unit of appropriate complexity
2. Creating a fixture
instantiate and set up graph of objects
3. Executing the unit under test
stimulate the fixture
4. Verifying the expected behavior
assert side effects and return value
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
6. Idea
Use example runs of the program to extract
test scenarios.
Perform a dynamic analysis to extract a test
blueprint that shows:
- how to create a fixture for a selected unit
- how to execute the unit
- how to verify the expected behavior
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
7. Program Instrumentation
Example program execution Execution of existing tests
object flow +
coverage data
execution trace data
Analysis
Tool
Execution Trace Test Blueprint
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
8. Test Blueprint
Execution Trace Test Blueprint
ASTChecker>>declareVariableNode:
...
FunctionScope>>addTemp:
Execution
TempVar class>>new
...initialization...
Unit
TempVar>>name:
TempVar>>scope:
KeyedSet>>add:
...library code...
...
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
9. Test Blueprint
Legend :Class existing instance
existing reference
:Class new instance
new reference (side effect)
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
10. Creating the Fixture
Unit Test for addTemp: method
1 fscope := FunctionScope new.
2 varname := ’x’.
Legend :Class existing instance
existing reference
:Class new instance
new reference (side effect)
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
11. Executing the Unit under Test
Unit Test for addTemp: method
1 fscope := FunctionScope new.
2 varname := ’x’.
3 var := fscope addTemp: name.
Legend :Class existing instance
existing reference
:Class new instance
new reference (side effect)
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
12. Verifying Expected Behavior
Unit Test for addTemp: method
1 fscope := FunctionScope new.
2 varname := ’x’.
3 var := fscope addTemp: name.
4 self assert: var class = TempVar.
5 self assert: var name = varname.
6 self assert: var scope = fscope.
7 self assert: ( Legend :Class existing instance
fscope tempVars includes: var). existing reference
:Class new instance
new reference (side effect)
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
13. Blueprint Examples
IRBuilder>>add: FunctionScope>>lookupVar:
NonClosureScopeFixer>>acceptVarNode:
InstanceScope>>newMethodScope
ace Test Blueprint
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
14. Implementation
dynamic model static model Aliases are created
Instance * 1
Class
when an object is:
1 ‣ instantiated
children
* 1 activations
0..1 *
1
* ‣ stored into or read
Alias Activation * Method from a field
receiver *
0..1 * creator 1 root
parent ‣ stored into or read
createdAliases from an array
Execution-
Unit ‣ passed as argument/
return value
Object Flow meta-model
The flow of an object in the system is tracked by
the parent-child relationship of its Aliases
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
15. Implementation
Execution Unit
w r r
object_1
w
object_2
w r w
object_3
Imported Exported
references references
t
Imported references = used state (->fixture)
Exported references = side effects (->assertions)
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
16. Initial Case Study I
How straightforward is it to use the
proposed tool?
Target system: Insurance broker
application (520 classes, 6 years old)
Observations
- One developer produced 12 tests in 2h
- Average fixture size: 5
- Average side effects: 4
- Test Blueprint worked surprisingly well
...yet, sporadically resorted to consulting source code
- Insufficient support for selecting execution units
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
17. Initial Case Study II
How do tests written using our approach differ compared
to conventional tests written by an expert?
Content Management System (377 classes)
Study: selected 14 unit tests, removed all 84 assertions
and then systematically rewrote them using our tool
Resulting assertions
- 72 identical
- 5 additional
- 12 missed ==> verification that special objects are left unmodified
our approach does not detect unmodified objects
that are worthwhile to verify!
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
18. Conclusions
Yes, we can write unit test for not well known code
...by exploiting runtime information
Test Blueprints tell us
...how the fixture and assertions have to look like
Difficulties are how to
...select good execution units in the trace
...initialize objects to bring them into the desired state
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch
19. Conclusions
Yes, we can write unit test for not well known code
...by exploiting runtime information
Test Blueprints tell us
...how the fixture and assertions have to look like
Difficulties are how to
...select good execution units in the trace
...initialize objects to bring them into the desired state
Questions?
CSMR’08 : : Test Blueprints : : Adrian Lienhard : : www.adrian-lienhard.ch : : lienhard@iam.unibe.ch