A Survey on Automatic Test Generation and Crash Reproduction

1

A SURVEY ON
AUTOMATIC TEST GENERATION
AND CRASH REPRODUCTION
PhD Qualifying Examination

Ning Chen

2

Outline
• Automatic Test Generation
• Feedback-directed Random Generation
• Captured-Object based Generation
• Genetic Evolution based Generation
• Symbolic Execution
• Dynamic Symbolic Execution

• Automatic Crash Reproduction
• Record-and-replay Approaches
• Static analysis based Approaches

• Summary

3

Automatic Test Generation

4

Automatic Test
Generation

Symbolic
Search
Execution
Based
Based

Feedback- Object- Genetic Symbolic Dynamic Symbolic
directed capturing Evolution Execution Execution

Common Challenge:
My Approach
Object Creation Challenge

5

Categories Representatives
Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11
Captured-Object Based MSeqGen’09, OCAT’10
Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09
Symbolic Execution King’76, Clarke’76, JPF’04
Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08

6

Feedback-directed Random Test Generation

Randomly
select a new
method call
Classes to
test
Error-
Generate Execute Examine revealing test
Method Method Execution cases
Sequences Sequences Result
Time Limit
Regression
test cases

Properties
Method
to check Sequences
Method Arguments If Sequence Legal
Generated

7


Randomly
select a new
Classes to method call
test
Error-
Generate Execute revealing test
Method Method cases
Sequences Sequences
Time Limit
Regression
test cases

Properties
Method
to check Sequences
Feedback to guide If Sequence Legal
Generated

8

• Advantages:
• Scalable to large subject programs
• Known to be easy to implement
• Generates no or little redundant method call sequences

• Challenges:
• Generates too many test cases (thousands of test cases)

• Cannot generate complex method sequences, thus may not
achieve good coverage for programs with complex structures

9

01 void execute(tinySQLStatement stmt, String sqlStr) {
02 if (stmt.execute(sqlStr)) {
03 // target branch
04 }
05 }

• cannot cover the target branch because it is difficult to
generate the proper sql statement randomly

• In short, Random approach fails to create test inputs that
can reach deeper part of the subject program.

10

Follow-up works
Combining Dynamic Exec. with Random Generation:
S. Artzi, M. D. Ernst, A. Kie˙zun, C. Pacheco, and J. H. Perkins. Finding the
needles in the haystack: Generating legal test inputs for object-oriented
programs. In 1st Workshop on Model-Based Testing and Object-Oriented,
2006. (Palulu)

Combining Static Exec. with Random Generation:
W. Zheng, Q. Zhang, M. Lyu, and T. Xie. Random unit-test generation with mut-
aware sequence recommendation. In ASE, 2010. (RecGen)

Combining Dynamic + Static Exec. with Random Generation:
S. Zhang, D. Saff, Y. Bu, and M. D. Ernst, Combined static and dynamic
automated test generation, in ISSTA, 2011 (Palus)

11

Follow-up works
• The common goal: more diverse and complex test inputs
that can reach deeper part of the subject program.

• However, there is an alternative…

12

Captured-Object Based MSeqGen’09, OCAT 10

13

Object Captured Based

Randomly
select a new
method call
Classes to
test
Generate Execute Error-
Method Method revealing test
Sequences Sequences cases

Time Limit
Arguments

Regression
Method test cases
Sequences
Properties If Sequence Legal
Generated
to check
Runtime Existing Test
Captured Cases / Normal
Objects Executions

14

• Advantages:
• Can generate test inputs that are complex for random approaches
• Can be integrated with other test generation approaches.
• E.g. Seed objects for Random generation.

• Challenges:
• Fail to demonstrate an object generating method sequence.
• Making a test case less useful.

• Rely heavily on the quality and quantity of existing test cases.

• Captured objects may contain sensitive private information.
• James Clause et. al - ICSE2011

15

01 void test_execute() {
02 SQLStatement stmt = load(“…/SQLStatement/001”);
03 String sqlStr = load(“…/String/123”);
02 if (stmt.execute(sqlStr)) {
03 // target branch
04 }
05 }

• Apparently, such a test case would be less useful as it
does not provide much information.

16


17

Genetic Evolution Based
• Adopts evolutionary computation techniques to synthesize
method sequences for reaching higher code coverage.

• Method sequences (chromosome) are crossed-over and
mutated to produce different varieties of offspring.

• The fitness score is computed according to the code
coverage.
• Method sequences that can achieve higher code coverage will be
selected for future evolution.

18

• Advantages:
• Have stronger capabilities of generating complex test inputs over
random approaches.

• Challenges:
• Might not scale to large subject programs.
• Factors such as fitness functions, mutation operators, evolution
algorithms can greatly affect the quality of the generations.

19


Search Based Test Generation Approaches

20


Symbolic Execution Based Test Generation Approaches

21

Symbolic Execution
• Proposed as early as the 70s but become realistic only in
recent years

• Generate tests by symbolically executing all paths in the
program

• Rely on constraint solvers to compute test inputs (i.e.
models)

22

Symbolic Execution
01 void foo(int x, int y) {
02 if (x > y + 1) {
03 // branch 1
04 }
05 else {
06 // branch 2
07 }
08 }

23

Symbolic Execution
01 void foo(int x, int y) { Symbolic Input: X, Y

02 if (x > y + 1) {
Path Condition: X > Y + 1
03 // branch 1
04 }
05 else { Constraint Solver -> X = 2, Y = 0

06 // branch 2
07 } Generate actual test case

08 }

24

Symbolic Execution
01 void foo(int x) { Symbolic Input: X, Y

02 if (x > y + 1) {
Path Condition: X <= Y + 1
03 // branch 1
04 }
05 else { Constraint Solver -> X = 0, Y = 0

06 // branch 2
07 } Generate actual test case

08 }

25

Symbolic Execution
• Advantages:
• Feasible in recent years with the increase of computation power.

• Capable of achieving higher structural coverage than search based
techniques.

26

Symbolic Execution
• Challenges:
• Constraint Solving Limitations:
• Non-linear
• Floating point,
• Bit-vector, etc.

• External Method Calls:
• External Libraries, System Level Calls, etc.
• Cannot Symbolically execution these external method calls.

• The Object Creation Problem:
• In OO, test inputs (objects) needs to be constructed in sequence of
method calls.

27

Follow-up work
Complex Constraint Solving Challenge:
C. S. P˘as˘areanu, N. Rungta, and W. Visser. Symbolic execution with mixed
concrete-symbolic solving. In ISSTA ‟11

I. Erete and A. Orso. Optimizing constraint solving to better support symbolic
execution. In CSTVA‟11

The Object Creation Problem:
S. Thummalapenta, T. Xie, N. Tillmann, P. de Halleux, and W. Schulte.
MSeqGen: Object-oriented unit-test generation via mining source code. In
ESEC/FSE‟09

28


29

Dynamic Symbolic Execution
• Symbolic execution Ideally can achieve complete
coverage, but usually not the case in reality

• Recent years, Dynamic Symbolic Execution is proposed.
• EXE/KLEE „05
• DART „05
• CUTE „05, JCUTE „06
• PEX „08

• DSE is designed to handle two major challenges in
Symbolic Execution:
• 1) Constraint Solving, and
• 2) External Method Calls

30

01 void foo(int x, int y) {
02 if (external (x) == y) { No source access for external()
03 // branch 1
04 }
05 else if (hash(x) > y) { hash() is complex arithmetic
06 // branch 2
07 }
08 }

Key Idea:
Combines Dynamic execution + Symbolic execution

31

01 void foo(int x, int y) { Dynamic Exec.

02 if (external(x) == y) { external(0) == 1
03 // branch 1
04 }
05 else if (hash(x) < y) { hash(0) == 10
06 // branch 2
07 }
08 }

First Iteration (Random Input):
Dynamic Exec: x = 0 and y = 0

32

01 void foo(int x, int y) { Symbolic Exec.

02 if (external(x) == y) { external(X) == Y
03 // branch 1
04 } external(0) == 1
05 else if (hash(x) < y) {
06 // branch 2 Constraint Solver:
07 } X == 0, Y == 1

08 }

Second Iteration:
Path Condition: external(X) == Y

33

01 void foo(int x, int y) { Symbolic Exec.

02 if (external(x) == y) { external(X) != Y
03 // branch 1
04 } hash(X) < Y

05 else if (hash(x) < y) {
06 // branch 2 external(0) == 1
hash(0) == 10
07 }
08 } Constraint Solver:
X == 0, Y == 11

Third Iteration:
Path Condition: external(x) == y && hash(x) < y

34

• Advantage:
• Can handle situations where traditional symbolic executions cannot
handle. i.e. Constraint Solving, External Method Calls

• Achieve higher coverage compared to only symbolic execution.

35

• Challenges:
• A recent paper by Xiao ‟11. investigates the major challenges faced
by a state-of-the-art DSE framework for C#, Pex.
Project LOC Cov % Object Creation Problem
SvnBridge 17.1K 56.26 11 (42.31%)
xUnit 11.4K 15.54 8 (72.73%)
Math.Net 3.5K 62.84 17 (70.83%)
QuickGraph 8.3K 53.21 10 (100%)
Total 40.3K 49.87 46 (64.79%)

• The object creation challenge accounts for 65% of all cases that
cause code blocks to be not covered by Pex.

36

The Object Creation Problem
void foo(Container container) {
if (container.size >= 10) { Model:
// not covered branch container.size = 10

…

}

The Object Creation Problem :
How do we create and mutate a Container object into size = 10?

37

• In object-oriented programs, modifying a non-public field
requires a set of legal method sequence
• e.g. Container.size == 10
• Possible method sequence:
Container.add(Object);
…
Container.add(Object);

• But as the structure of the classes become more complex,
so will the object creation problem.

• Therefore, the object creation problem is an important, yet
unsolved problem in automatic test generation.

38

Demand-driven Object Generation
• To address this OCP, I am working on a project for
Demand-Driven Object Generation.

• Goal: Given a target state -> generate a legal method
sequence that can create and mutate object that satisfy
this target state
• Target state: container.size == 10
• Possible Output: container.add(new Object()); ten times

39

Demand-driven Object Generation
• The approach should be demand-driven, so only legal
sequence satisfying the target state can be generated.

• It is helpful to solve the object creation problem
mentioned.

• Thus, it can improve the coverage for dynamic symbolic
execution approaches for object-oriented programs.

40

Automatic Crash Reproduction

41


Automatic Crash
Reproduction

Static
Record-
Analysis
and-replay
Based

Common Challenges My Approach

42


Record-and-replay Jrapture’00, BugNet’05, ReCrash’08, LEAP’10
Static Analysis Approaches PSE’04, XYLem’09, SnuggleBug’09

43

Record-and-Replay
• Approach:
• Monitoring Phase: Captures/Stores runtime heap & stack objects.
• Test Generation Phase: Generates tests that loads the correct
objects with the crashed methods.
Original Program Execution

Store from heap & stack

Stored
Objects

Load as crashed method params

Recreated Test Case

44

Record-and-Replay
• Advantage:
• Can usually reproduce target crashes reliability

• Challenges:
• Require instrumentation in advance
• Extra runtime overhead, i.e. ReCrash:
• 13-64% performance overhead, and
• 3-90% memory overhead
• Extensive amount of data from users: objects in heap, running
context, etc
• Difficult to obtain ( > 100M )
• Privacy issues: James Clause et. al @ ICSE2011

45

Record-and-Replay
• Existing Frameworks:
• Jrapture ‟00,
• BugNet ’05,
• ReCrash/ReCrashJ ‟08
• LEAP/LEAN ’10

• Existing approaches have been trying various techniques
to reduce the runtime overhead yet maintain a good
reproduction result.

46


Record-and-replay Jrapture’00, BugNet’05, ReCrash’08, LEAP’10
Static Analysis Approaches PSE’04, XYLem’09, SnuggleBug’09

47

Static Analysis Approaches
• Microsoft PSE ‘04
• Postmortem static analysis framework.
• Explains the potential cause of a crash through symbolic execution.

• XyLem ’09
• Null-dereference analysis tool for Java language.
• Identifies the precise paths along which null values may ﬂow to de-
reference.

• IBM SnuggleBug ‘09
• Demand-driven backward symbolic execution framework.
• Computes the weakest preconditions of crashes.
• But does not generate actual crash triggering objects from the
computed preconditions.

48

STAR: automatically reproduces failures
• All of the above approaches either only finds error path, or
computes the error triggering preconditions.

• They do not support the full reproduction of a crash from the
crash information.

• We propose STAR, an automatic crash reproduction framework
• Requires only minimal information (crash stack trace, crash type)
• No client site deployment and recording necessary
• No performance overhead
• No need to collect any private data from user.
• Supports both finding out the failure preconditions as well as
recreating the actual failures.

49


crash precondition
stack trace processing crash condition preparation
computation
1 2 3
bug report x() at nnn {var == null}
y() at yyy {true}
z() at zzz …
crash
4 test case generation reproduction
program
test
TestGen: A novel test generation technique
cases

50

Backward Symbolic Execution
• Goal: Given a crash location, compute at a public method
entry, the precondition that can trigger the crash

• There have been several previous work in this direction
• XYLem ICSE ‟09
• SnuggleBug PLDI ‟09

• STAR has a more efficient backward symbolic execution
engine that improves existing approaches with heuristics:
• Pruning irrelevant branches
• Smarter backtracking
• Early detection of inner contradictions

51


crash precondition
stack trace processing crash condition preparation
computation
1 2 3
bug report x() at nnn {var == null}
y() at yyy {true}
z() at zzz …
crash
4 test case generation reproduction
program
test
TestGen: A novel test generation technique
cases

52

Crash Test Case Generation
• Essentially, this is the same problem as the demand-
driven method sequence generation we‟ve discussed.

• Thus, we combine our efficient backward symbolic
execution approach with the demand-driven method
sequence generation to achieve automatic crash
reproduction.

53

Summary – Automatic Test Generation
Categories Efficiency Coverage Complex Input
Generation
Feedback-directed Random Based

Captured-Object Based


Symbolic Execution


54

Summary – Automatic Test Generation
Generation



Symbolic Execution


Demand Driven Object Generation

55

Summary – Crash Reproduction
Generation



Symbolic Execution


Categories Reproducibility Minimal No Performance
Information Overhead
Record-and-replay


56

Summary – Crash Reproduction
Generation



Symbolic Execution


Categories Reproducibility Minimal No Performance
Information Overhead
Record-and-replay


STAR

A Survey on Automatic Test Generation and Crash Reproduction

More Related Content

What's hot

Viewers also liked

Similar to A Survey on Automatic Test Generation and Crash Reproduction

More from Sung Kim

Recently uploaded

A Survey on Automatic Test Generation and Crash Reproduction

Editor's Notes