1. Jonathan Misurda (jmisurda@cs.pitt.edu)
James A. Clause, Juliya L. Reed, Bruce R. Childers
Department of Computer Science
University of Pittsburgh
Pittsburgh, Pennsylvania 15260 USA
Mary Lou Soffa
Department of Computer Science
University of Virginia
Charlottesville, Virginia 22904 USA
Demand-Driven Structural Testing
with Dynamic Instrumentation

2. Software Testing
n What is Software Testing?
q Gather information about the behavior of the software being
developed or modified.
n Why test software?
q Gain confidence and insights into software correctness
q Create quality, robust programs
n Why is testing hard?
q Testing is expensive
n 50-60% of the total cost of software development
q Adds complexity to development process
n One testing technique is not enough

3. Structural Software Testing
n Structural Testing
q The process of discovering run-time properties of a
program specifically pertaining to control-flow
q Demonstrate adequacy of the test cases
q Different granularities of structures:
n Node coverage records statements (basic blocks)
n Branch coverage records control-flow edges
n Def-Use coverage records pairs of variable definitions
and uses
q Over multiple test cases until coverage criteria

4. Current Testing Tools
n E.g.: JCover, Clover, IBM Rational PurifyPlus
n Static instrumentation with test probes
q Probes are injected instrumentation to gather coverage
information
n Limitations:
q Expensive in both time and space
q Only one type of test
q Limited code regions and scopes
q Cannot define their own ways to test

5. Our Approach
n Demand-driven structural testing
q Specification driven: User written tests
q Test plans: Recipe of how & where to test
q Path specific: Instrument only what is needed
q Dynamic: Insert & remove instrumentation
n Jazz: A scalable & flexible framework for testing
q Automatically apply multiple test strategies
q Handle large programs
q Allows user customization of tests

6. Jazz Overview
Application
IDE & Test GUI
Test Specification
Test Planner
Test Plan
Test Analyzer and Result Reporter
Test Dynamic Instrumenter
Test Results
Select the regions and test types
Allows user customization
Automatically generate where and
how to instrument
Run the program with demand-driven
instrumentation
Collect the results

7. Test Specifier
1a
1b
3
2
Specify the regions to test
1a. Highlight line regions
1b. Select classes and methods
List of desired tests
Click to create a test

8. Test Planner
n Test plan: Where and how to test program
n Test storage, probe location table,
instrumentation payload
n Can combine payloads to create custom strategies
Application
Test
Planner
Test
Specification
Test Instrumentation
Payload
Test Plan
Global Storage & Probe Location Table

9. Test Planner
n Challenges:
q When to insert test probes
q Where to test/instrument
q What to do at each probe
n A planner determines three things:
q Seed Probe Set – probes which are initially inserted in a
test region.
q Coverage Probe Set – probes that can be inserted and
removed on demand along a path.
q Probe Lifetime – whether a probe must be re-inserted
after being hit by its “control flow successor” basic blocks.

10. Branch Coverage Example Plan
public class simpleDemo {
public static void main(String[] args) {
int evenSum = 0;
int oddSum = 0;
for(int i = 0; i < 100; i++) {
if(i%2==0) {
evenSum += i;
}
else {
oddSum += i;
}
}
System.out.println("The sum of the even numbers from 0 to 100 is: " + evenSum);
System.out.println("The sum of the odd numbers from 0 to 100 is: " + oddSum);
}
}
DEFINITIONS {
NAME: d_method, REGION_D,
LOCATION: FILE simpleDemo.java {
CLASS simpleDemo, METHOD main
}
}
BODY {
DO BRANCH_TEST ON REGION d_method UNTIL: 85%
}
simpleDemo.java
simpleDemo.testspec

11. Branch Coverage Planner
n Record which edges are executed
q Determine (source, sink) edge pairs hit at run-time
q Source is a branch
q Sink can be taken & not-taken target of branch
Probe Set Members
Seed Probe First block in testing region
Coverage Probe Sinks of currently instrumented block
Probe Lifetime Until all incoming edges have been
covered

12. Branch Coverage Example
Probe Loc Tab Storage
Block Next Hit
1 2,3
2 4
3 4
4 1
Test Payload
Mark edge hit
Insert at next point
Remove instrumentation
1
2 3
4
Coverage probe
Hit
N, N
N
N
N
Y, Y
Y
Y
Y
Seed probe

13. Demo
n Branch coverage of music player
q Test region is whole program (JOrbis)
q Test input is a song
n Compared
q Traditional approach with static instrumentation
q Demand-driven approach with dynamic instrumentation
With Dynamic
Instrumentation
With Static
Instrumentation

14. Demo
n Branch coverage of music player
q Test region is whole program (JOrbis)
q Test input is a song
n Compared
q Traditional approach with static instrumentation
q Demand-driven approach with dynamic instrumentation
With Dynamic
Instrumentation
With Static
Instrumentation

15. Demo
n Branch coverage of music player
q Test region is whole program (JOrbis)
q Test input is a song
n Compared
q Traditional approach with static instrumentation
q Demand-driven approach with dynamic instrumentation
With Dynamic
Instrumentation
With Static
Instrumentation

16. Other Planners
n Node Coverage Planner
q Record which statements are executed
Probe Set Members
Seed Probe First statement in the testing region
Coverage Probe Next reachable statement
Probe Lifetime Removed as soon as statement is executed
n Def-Use Coverage Planner
q Record which variable definition reaches an executed use
Probe Set Members
Seed Probe All variable definitions
Coverage Probe Uses associated with all definitions
Probe Lifetime Defs removed after all reachable uses are
covered. Uses are removed as executed

17. Dynamic Instrumentation
n Test plan targets an instrumentation API
n FIST instrumentation engine [WOSS’04]
q Retargetable & reconfigurable
q Dynamic insertion & removal of instrumentation
q Binary level instrumentation (post JIT)
n Uses fast breakpoints [Kessler]: Replace existing
instruction with a jump to instrumentation

18. Experimental Methodology
n Test Specification
q Implemented as an Eclipse 3.0 plugin
n Test Planner & Dynamic Instrumenter
q Implemented using the Jikes RVM version 2.1.1
q On the x86 platform
n SPECjvm98 benchmarks test input
q unloaded 2.4 Ghz Pentium IV, 1GB of memory
q RedHat Linux 7.3
n Traditional vs. Jazz (in same implementation)
q Compared coverage, run-time, memory

19. Coverage Results
n Coverage – same reported by both tools
Branch Node Def-Use
compress
58% 90.6% 89.8%
jess
46.8% 80.3% 71.8%
db
44.4% 76.9% 75.0%
javac
38.9% 75.0% 66.9%
mpegaudio
60.9% 88.7% 90.5%
mtrt
50.6% 90.3% 87.3%
jack
55.6% 82.2% 73.4%

20. Run-time Performance
Static Branch
Dynamic Branch
Static Node Static Def-Use
Dynamic Node Dynamic Def-Use

21. Memory Requirements
Branch Node Def-Use
Demand Static Demand Static Demand Static
compress
7.9
7.5
4.4 3.7 8.8 5.6
jess
50.2
60.3
34.1 29.7 70.3 44.8
db
9.7
8.9
5.0 4.1 9.8 5.1
javac
178.9
186
107.3 85.3 202.0 125.3
mpegaudio
24.7
29.5
15.0 14.7 34.8 22.3
mtrt
22.4
23
12.5 11.0 26.1 16.1
jack
73.4
78
46.5 37.6 89.0 46.8
• In Kilobytes

22. Summary
n New demand-driven approach
q A tool (Jazz) for structural testing
q Dynamic instrumentation guided by a program’s
execution
q Minimally intrusive
q User configurable and flexible testing
n Very low overhead
q E.g., branch coverage tool is 3-4x faster than
traditional approaches

23. For Further Information…
jmisurda@cs.pitt.edu
http://www.cs.pitt.edu/coco

24. Test Results
1 Click to run test suite
2 Results

25. Static Instrumentation
n Shortcomings of static instrumentation:
q Not scalable: Instrumentation left in place causes
unnecessary increase in overhead
q Inflexible: Only certain tests, languages, and
platforms
q Cumbersome: Requires recompilations for
dedicated testing
q Intrusive: Addition of testing code may alter or
mask defects

26. Run-time Performance
-10%
30%
70%
110%
150%
190%
230%
270%
compress jess db javac mpeg mtrt jack avg
%slowdownoverbaseline
Static Branch Dynamic Branch Static Node
Dynamic Node Static Def-Use Dynamic Def-Use

27. Branch Testing
0
10
20
30
40
50
60
70
80
90
100
Time(seconds)
compress jess db javac mpeg mtrt jack avg
Dynamic Static None

28. Node Testing
0
10
20
30
40
50
60
70
80
90
100
Time(seconds)
compress jess db javac mpeg mtrt jack avg
Dynamic Static None

29. Def-Use Testing
0
10
20
30
40
50
60
70
80
90
100
Time(seconds)
compress jess db javac mpeg mtrt jack avg
Dynamic Static None