Behavioral Model Seeding for Search-based Crash Reproduction, Lorentz workshop: In-Vivo Analytics for Big Software Quality

1. Behavioral Model Seeding for
Search-based Crash Reproduction
Pouria Derakhshanfar, Xavier Devroey,
Gilles Perrouin, Andy Zaidman and Arie van Deursen

2. !2

3. !3
java.lang.ClassCastException: […]
at org…..SolrEntityReferenceResolver.getWikiReference(....java:93)
at org…..SolrEntityReferenceResolver.getEntityReference(….java:70)
at org…..SolrEntityReferenceResolver.resolve(….java:63)
at org…..SolrDocumentReferenceResolver.resolve(….java:48)
at …
Crash!?!

4. Java Stack Trace (Issue XWIKI-13031)
!4
java.lang.ClassCastException: […]
at org…..SolrEntityReferenceResolver.getWikiReference(....java:93)
at org…..SolrEntityReferenceResolver.getEntityReference(….java:70)
at org…..SolrEntityReferenceResolver.resolve(….java:63)
at org…..SolrDocumentReferenceResolver.resolve(….java:48)
at …
Exception
Frames
{Target→

5. !5
Issue XWIKI-13031

6. !6
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
Random initial
test suite
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a()
c() e()d()
e()
c() b() e()
a()
c() e()d()
e()
c() b() e()
Evolutionary
search
a() e()d()
Exception:
at x(…)
at y(…)
at e(…)
Exception:
at x(…)
at y(…)
at e(…)
Crash
reproducing
test case
Stack trace
Soltani, M., Panichella, A. and van Deursen, A. 2018. Search-Based Crash Reproduction and Its Impact on
Debugging. Software Engineering, IEEE Transactions on. (2018).

7. !7
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
Random initial
test suite
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a()
c() e()d()
e()
c() b() e()
a()
c() e()d()
e()
c() b() e()
Evolutionary
search
a() e()d()
Exception:
at x(…)
at y(…)
at e(…)
Exception:
at x(…)
at y(…)
at e(…)
Crash
reproducing
test case
Stack trace
Isn’t an initial test suite close to actual usage of the classes
more likely to lead to a crash reproduction?

8. Test Seeding
• Use existing tests to generate
the initial test suite
• J. M. Rojas, et al. , “Seeding
strategies in search- based unit
test generation,” STVR, 2016.
• Applied to crash replication
!8
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() c()
c() e()c()
e()
c() a()
Exception:
at x(…)
at y(…)
at e(…)
Stack trace
a() c()
c() c()
b() a() e()
Existing
tests
e()
Random initial
test suite
Existing tests
subset

9. Behavioral Model Seeding
• Use a model of method usage
to generate objects used in the
initial test suite
• Relies on Model-based Testing
• Select abstract test cases
• Concretized into objects
!9
Random initial
test suite
Exception:
at x(…)
at y(…)
at e(…)
Stack trace
a()b()
c() e()
d()
Model
Model-driven
initial test
suite
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
a() b()
c() e()d()
e()
c() e()
b() a()
c() e()d()
e()
c() e()
a()

10. size()
add(Object)iterator()
S0
get(int)
remove(int)
S1
S2S3
remove(int)
S4
S5
SX
size()
add(Object)
size()
add(Object)
!10
Abstract test case selection
<add(Object), add(Object)>Abstract test case
int[] t = new int[7];
t[3] = (-2147483647);
EuclideanIntegerPoint ep = new […](t);
LinkedList<[...]> lst = new LinkedList<>();
lst.add(ep);
lst.add(ep);
Concrete object

11. Test MATH-79b, Frame 2
!11
java.lang.NullPointerException
at ...KMeansPlusPlusClusterer.assignPointsToClusters()
at ...KMeansPlusPlusClusterer.cluster()
…
Input stack trace
public void testCluster() throws Exception{
int[] t = new int[7];
t[3] = (-2147483647);
EuclideanIntegerPoint ep = new EuclideanIntegerPoint(t);
LinkedList<[...]> lst = new LinkedList<>();
lst.add(ep);
lst.add(ep);
KMeansPlusPlusClusterer<[...]> kmean = new KMeansPlusPlusClusterer<>(12);
lst.offerFirst(ep);
kmean.cluster(lst, 1, (-1357));
}
Crash reproducing test

12. Behavioral Model Inference
• Inference from sequences of
method calls
• Coming from
• Source code
• Static analysis
• Test cases
• Dynamic analysis
• Operations logs (future work)
• Online analysis
!12
a()b()
c() e()
d()
N-gram inference
[b(), a(), e()]
[c(), d(), a(), e()]
[b(), a(), d(), a(), d(), a(), e()]
…

13. Behavioral Model Inference
• Inference from sequences of
method calls
• Coming from
• Source code
• Static analysis
• Test cases
• Dynamic analysis
• Operations logs (future work)
• Online analysis
!12
a()b()
c() e()
d()
N-gram inference
[b(), a(), e()]
[c(), d(), a(), e()]
[b(), a(), d(), a(), d(), a(), e()]
…

14. Behavioral Model Inference
• Inference from sequences of
method calls
• Coming from
• Source code
• Static analysis
• Test cases
• Dynamic analysis
• Operations logs (future work)
• Online analysis
!12
a()b()
c() e()
d()
N-gram inference
[b(), a(), e()]
[c(), d(), a(), e()]
[b(), a(), d(), a(), d(), a(), e()]
…

15. Evaluation on 45 crashes
!13
0
25
50
75
not started failed line reached ex. thrown reproduced
Numberofframes
Configurations
no s.
test s. 0.2
test s. 0.5
test s. 0.8
test s. 1.0
model s. 0.2
model s. 0.5
model s. 0.8
model s. 1.0

16. Evaluation on 45 crashes
!13
0
25
50
75
not started failed line reached ex. thrown reproduced
Numberofframes
Configurations
no s.
test s. 0.2
test s. 0.5
test s. 0.8
test s. 1.0
model s. 0.2
model s. 0.5
model s. 0.8
model s. 1.0

17. Evaluation on 45 crashes
!13
0
25
50
75
not started failed line reached ex. thrown reproduced
Numberofframes
Configurations
no s.
test s. 0.2
test s. 0.5
test s. 0.8
test s. 1.0
model s. 0.2
model s. 0.5
model s. 0.8
model s. 1.0

18. Execution Time
!14
●
●
653 653
641
638
629 699 690 697 704
1e+01
1e+03
1e+05
nos.
tests.0.2
tests.0.5
tests.0.8
tests.1.0
models.0.2
models.0.5
models.0.8
models.1.0
Numberoffitnessevaluations

19. Execution Time
!14
●
●
653 653
641
638
629 699 690 697 704
1e+01
1e+03
1e+05
nos.
tests.0.2
tests.0.5
tests.0.8
tests.1.0
models.0.2
models.0.5
models.0.8
models.1.0
Numberoffitnessevaluations

20. Influencing Factors
• Seeding abstract test cases
• <add(Object), add(Object)>
• Having dissimilar abstract test cases
• Multiple information sources
• Static analysis
• Dynamic analysis
• Prioritising abstract test cases selection
• Select abstract test cases for classes in the stack trace
!15

21. https://www.stamp-project.eu
https://github.com/STAMP-project