This document discusses automated unit test generation and the costs of software bugs. It provides statistics on the annual costs of software bugs in the US ($59.9B) and worldwide ($312B). It then covers different techniques for automated unit test generation including random data generation, search-based approaches, and symbolic/concolic-based approaches. Examples of specific tools that use each technique like EvoSuite, PEX, and jCUTE are also provided. The document concludes with best practices for using automatically generated unit tests and a question about automatically generating GUI/API tests.

1. Automated Unit Test Generation
Vassil Popovski, VMware

4. The Cost of Software Bugs?
$59.9 B per year in USA (NIST, 2002)*
(*) http://www.nist.gov/director/planning/upload/report02-3.pdf
$312 B per year worldwide (Cambridge University, 2012)**
(**) http://www.roguewave.com/DesktopModules/Bring2mind/DMX/Download.aspx?entryid=1606&command=core_download&PortalId=0&TabId=607

5. $312,000,000,000
The Cost of Software Bugs?

6. The Cost of Software Bugs?

7. Total market capitalization for all 3 companies(*):
!
$177.36 B
(*) Google finance - as of Nov 2nd, 2013

8. The Cost of World Hunger?
$40B per year(*)
(*) According to United Nations

10. The Cost of Fixing Bugs?
http://www.amazon.com/Assessments-Benchmarks-Addison-Wesley-Information-Technology/dp/0201485427?
ie=UTF8&s=books&qid=1209056706&sr=1-1

11. Software Testing Pyramid
http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid/
Unit Testing: 80-90%
Integration testing: 4-15%
E2E/GUI testing: 1-5%

12. Techniques/Tools
1. Random data generation
2. Search based
3. Symbolic/Concolic based
The goal: High (close to 100% code coverage*)

13. Anatomy of a Unit Test
Arrange
Act
Assert

14. Anatomy of a Unit Test
Arrange Data
Arrange Mocks/Stubs
Act
Assert

15. Anatomy of a Unit Test

16. Evaluation of Tools
Arrange/Generate
Data
Arrange/Generate
Mocks/Stubs
Act/Invoke
methods
Assert/Verify
the results

17. Techniques/Tools
x != 100 x == 100
x>=z x<z
if
if

18. Techniques/Tools
x>=z x<z
if
ifRandom data generation:
!
Feed x,y with random data
x != 100 x == 100

19. Demo

20. Evaluation of Tools
Arrange/Generate
Data
Arrange/Generate
Mocks/Stubs
Act/Invoke
methods
Assert/Verify
the results
junit-quickcheck
feed4JUnit
yes - some
no
no
no

21. Techniques/Tools
1. Random Data Generation:
• junit-quickcheck (Java)
https://github.com/pholser/junit-quickcheck/
• Feed4JUnit & Feed4TestNG + Benerator (java)
http://databene.org/feed4-tools.html
http://databene.org/databene-benerator.html
• Randoop (Java + .NET)
https://code.google.com/p/randoop/
• T2 (Java)
https://code.google.com/p/t2framework/

22. Techniques/Tools
x != 100 x == 100
x>=z x<z
if
ifSearch based approach:
!
1. Start with random/initial data
(x=1, y=1)

23. Techniques/Tools
x != 100 x == 100
x>=z x<z
if
ifSearch based approach:
!
1. Start with random/initial data
(x=1, y=1)
x=1, y=1

24. Techniques/Tools
x != 100
x>=z x<z
if
ifSearch based approach:
!
1. Start with random/initial data
(x=1, y=1)
x=1, y=1
x == 100

25. Techniques/Tools
x == 100
x>=z x<z
if
ifSearch based approach:
!
1. Start with random/initial data
(x=1, y=1)
2. Based on coverage information
- select new value(s) (x==100)
!
x != 100
x=1, y=1

26. Techniques/Tools
x == 100
x>=z x<z
if
ifSearch based approach:
!
1. Start with random/initial data
(x=1, y=1)
2. Based on coverage information
- select new value(s) (x==100)
x=100, y=1
x != 100

27. Demo

30. Evaluation of Tools
Arrange/Generate
Data
Arrange/Generate
Mocks/Stubs
Act/Invoke
methods
Assert/Verify
the results
yes - some
no
no
no
EvoSuite
yes
no
yes
yes
junit-quickcheck
feed4JUnit

31. Techniques/Tools
2. Search Based:
• EvoSuite (Java)
http://www.evosuite.org/
• Testful (Java)
https://code.google.com/p/testful/

32. Techniques/Tools
x != 100 x == 100
x>=z x<z
if
ifSymbolic based approach:
!
1. Evaluate path conditions (PC)
on every branch PC: X!=100, Y PC: X == 100, Y
x=X, y=Y, z=2*Y
PC: X == 100,
X>=2*Y
PC: X == 100,
X<2*Y
1. X!=100, Y => (x=1, y=1)
2. X=100, X>=2*Y =>(x=100, y=1)
3. X=100, X<2*Y =>(x=100, y=51)
2. For each leaf - calculate X & Y
using constraints solver:

33. Techniques/Tools
x != 100 x == 100
x>=z x<z
if
ifConcolic based approach:
!
Optimization of symbolic
execution - but use real values
for some of the branches

34. Demo

35. Evaluation of Tools
Arrange/Generate
Data
Arrange/Generate
Mocks/Stubs
Act/Invoke
methods
Assert/Verify
the results
yes - some
no
no
no
EvoSuite
yes
no
yes
yes
PEX (+ Moles)
yes
yes - some
yes
yes
junit-quickcheck
feed4JUnit

36. Techniques/Tools
3. Symbolic/Concolic based:
• Microsoft Pex (.NET)
http://research.microsoft.com/en-us/projects/pex/
• jCUTE (Java)
http://osl.cs.uiuc.edu/software/jcute/
• jalangi (Java Script)
https://github.com/SRA-SiliconValley/jalangi
• AgitarOne (Java, commercial)
http://www.agitar.com/solutions/products/agitarone.html
• Jtest (Java, commercial)
http://www.parasoft.com/jsp/products/jtest.jsp

37. Best usage of Auto Generated Unit Tests
1.Regression testing of legacy code
!
2.To support your unit testing / TDD

38. Automatic Generation of GUI/API tests?

39. Q&A

40. Thank you
!
Vassil Popovski
!
Director, R&D, VMware
vpopovski@vmware.com