Hyunmin's FSE presentation.

1. How We Get There:
A Context-Guided Search
Strategy in Concolic Testing
Hyunmin Seo and Sunghun Kim
The Hong Kong University of Science and Technology
Nov. 19 2014
FSE, Hong Kong

2. Concolic Testing
2
푝푐1
푝푐2
푝푐3
푝푐4
휋1
푰ퟏ
b9
푃퐶 = 푝푐1⋀푝푐2⋀푝푐3⋀푝푐4⋀…
푃퐶′ = 푝푐1⋀푝푐2⋀¬푝푐3
푰ퟑ
휋2 휋1 휋3
b1
b2
b3
b4
푰ퟐ
⟹
Execution Tree
푰ퟐ
b1
b2
b9
b10
휋2 휋1

3. Path Explosion Challenge
• Exponentially many execution paths
3
4 conditional nodes 16 (24) execution paths

4. Path Explosion Challenge
• grep – text search utility
• 19K LOC
• Statically - 4K branches in CFG
• Dynamically - 8K branches in
an execution path
4
*CFG of re_match_2_internal in grep

5. Search Strategy
• Given a limited testing budget, select high-priority
branches first to improve coverage fast
5
• Run 푷 with a concrete input 푰
• Execution path 흅 = 풃ퟏ풃ퟐ풃ퟑ …
• Select a branch 풃풊
• Generate 푰′ for 흅′ = 풃ퟏ풃ퟐ풃ퟑ…풃 풊
• By symbolic execution and constraint solving
Repeat

6. Existing Search Strategies
6
DFS BFS Random

7. Coverage-Optimized Strategies
7
• CFG – How far is any
uncovered branch from this?
• CarFast – How many can be
covered by this?
• Generational – How many
are actually covered by this?

8. Context-Guided Search
Considers context of a branch
and selects a branch
in the new context
8

9. Intuition behind CGS
Explore diverse state of P
by avoiding exploring the same state
- “same” in the limited context
9

10. CGS Example
10
b7 b8
b7 b7 b7 b7
휋1 휋2 휋3 휋4
b5 b6

11. CGS Example
b6 b6 b6 b5
11
b5 b6
b7 b8
b7 b7 b7 b7
휋1 휋2 휋3 휋4
SELECT SKIP SKIP SELECT

12. Context
•k-Context of branch b
• A sequence of k preceding branches
of b in an execution path
• Example
• 3-Context of b6 : (b6, b4, b1)
12
b2
b4
b6
π1
b1
b3
b5

13. Optimal k
• 1-Context
• Select each branch only once
• ∞-Context
• Select all branches
• Optimal k for the best coverage depends on
• Testing budget
• Characteristic of the subjects
13

14. Incremental Search
14
BFS traversal
1-context 2-context 3-context

15. Dominators
if every path from the entry node
to node n must go through node d
15
Node d dominates node n,
Dom(b11) {b9, b7, b5, b3}
2-Context of b11 along 흅ퟏ= (b11, b9)
2-Context of b11 along 흅ퟐ= (b11, b9)
2-Context of b11 along 흅ퟏ= (b11, b1)
2-Context of b11 along 흅ퟐ= (b11, b2)
b1 b2
b3
b5
b7
b9
b11
b4
b6
b8
b10
b12
π1 π2

16. Research Questions
• RQ1 – Given the same testing budget, how many
branches can be covered?
• RQ2 – Given a target coverage goal, how many
iterations are required to achieve the goal?
• RQ3 – What is the effect of dominators and
incremental k?
• RQ4 – How different are the covered branch sets by
different strategies?
16

17. Evaluation Subjects
17
Subject Testing Tool Language LOC
grep CREST C 19K
replace CREST C 0.5K
expat CREST C 18K
cdaudio CREST C 2K
floppy CREST C 1.5K
kbfiltr CREST C 1K
tp300 CarFastTool Java 0.3K
tp600 CarFastTool Java 0.6K
tp1k CarFastTool Java 1.5K
tp2k CarFastTool Java 2.4K
tp5k CarFastTool Java 5.8K
tp10k CarFastTool Java 28K

18. 18
RQ1-Coverage – C Subjects

19. 19
RQ1-Coverage – C Subjects

20. RQ1-Coverage - Java Subjects
20
350
300
250
200
150
100
tp600
0 500 1000 1500
Number of branches covered
Iterations
CGS
CarFast

21. RQ2 – Reaching the Target
(C Subjects)
21

22. RQ2 – Reaching the Target
(Java Subjects)
22

23. RQ3 - Effect of Dominators
23

24. RQ3 - Effect of Incremental
Search
24

25. RQ4 - Comparison of
Covered Branch Sets
CovCGS - A set of branches covered by CGS
CovOthers - A set of branches covered by other strategies
CovOthers ≤ |CovCGS|
Cov CovCGS Others
25
CovCGS
CovOthers

26. RQ4 - Comparison of
Covered Branch Sets
26
CovCGS
CovOthers
replace, cdaudio, floppy, kbfiltr
Tp300, tp600, tp1k, tp2k
CovOthers ⊆ |CovCGS|
grep
CovOthers
−
CovCGS
|CovCGS|
= 0.1 ~ 3%
Cov CovCGS CFG
61 1606 383

27. Threats to Validity
• Precision in Symbolic Execution
• Non-linear expression, Floating-Point operations, Symbolic
pointer dereferencing
• Input vector
• Size of input, Optional arguments
• External Validity
• Test subjects and strategies might not be representative
27

28. Summary
• Path explosion challenge in Concolic testing
• Search strategies prioritizes branches according to some criteria
• CGS
• Selects branches in the new context
• Use dominators to exclude irrelevant branches
• BFS search + incrementally increase of the size of context
• Evaluation on six C and six Java subjects
• Achieved the highest coverage on all twelve subjects
• Reached the target much faster on most subjects
28

29. Backup Slides
29

30. Coverage
for
C Subjects
30

31. Coverage for Java Subjects
31
Mann-Whitney U test P-value < 0.01

32. Related Work
• Pruning Redundant Path
• RWset [Cristian ‘08]
• Interpolation [Jaffar ’13]
• Function Summary
• Compositional [Godefroid ‘07, ‘10]
• Demand-driven compositional [Anand ‘08]
• Others
• Fitness-guided approach [Xie ’09], Sub-path guided [Li ‘13]
• Hybrid [Majumdar ‘07]
32

33. Branch Selection in CGS
33

34. Search Strategies in KLEE
34