Keynote Talk in the Workshop on Testing: Academia-Industry Collaboration, Practice and Research Techniques (TAIC PART 2016) http://www2016.taicpart.org/
Transferring Software Testing and Analytics Tools to Practice
1. Tao Xie
University of Illinois at Urbana-Champaign
Part of the research work described in this talk was done in collaboration with the Pex team and SoftwareAnalytics
Group @Microsoft Research, students @IllinoisASE, and other collaborators
2. Research impact: inspiring/impactful
ideas/directions/subareas… for researchers
Example: model checking
Practice impact: Practice adoption of
tools/systems/technologies… for practitioners
Some examples discussed in this talk
Societal impact: inspiring/impactful
ideas/thinking/awareness… for general public
Example: computational thinking, privacy, medical-
device security, MOOCs, …
3. Publishing research results technologies
there adopted by companies, e.g.,
ICSE 00 Daikon paper by Ernst et al. Agitar Agitator
https://homes.cs.washington.edu/~mernst/pubs/invariants-relevance-icse2000.pdf
ASE 04 Rostra paper by Xie et al. Parasoft Jtest improvement
http://taoxie.cs.illinois.edu/publications/ase04.pdf
PLDI/FSE 05 DART/CUTE papers by Sen et al. MSR SAGE, Pex
http://srl.cs.berkeley.edu/~ksen/papers/dart.pdf
http://srl.cs.berkeley.edu/~ksen/papers/C159-sen.pdf
…
5. Transferring research results to product groups
tools/products used inside company or
outside, e.g.,
SAGE
Flash Fill
…
CloudBuild
Tools for Software Engineers
Fakes
6. Release open source infrastructures or libraries
to engage academic/industrial communities to
use and contribute, e.g.,
▪ MPI/PETSc by Bill Gropp et al.
▪ Charm++ by Laxmikant (Sanjay) Kale et al.
▪ LLVM byVikram Adve, Chris Lattner, et al.
“The openness of the LLVM technology and the quality of its architecture
and engineering design are key factors in understanding the success it has
had both in academia and industry.”
KLEE? JPF?FindBugs?Shipshape? Soot? WALA? …
7. 50 years of automated debugging research
N papers only 5 evaluated with actual programmers
“
” [Parnin&Orso ISSTA’11]
http://dl.acm.org/citation.cfm?id=2001445
8. Human
Expensive, incomplete, …
Brute Force
Pairwise, predefined data, etc…
Tool Automation!!
10. Pex (released on May 2008)
Shipped with Visual Studio 15 as IntelliTest
30,388 download# (20 months, Feb 08-Oct 09)
22,466 download# (10 months, Apr 13-Jan 14): Code Digger
Active user community: 1,436 forum posts during ~3 years
(Oct 08- Nov 11)
Moles (released on Sept 2009)
Shipped with Visual Studio 12 as Fakes
“Provide Microsoft Fakes w/ allVisual Studio editions” got
1,457 community votes
14. Code Hunt can identify
top coders
http://programming2015.cstnet.cn/
15. Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
classTest {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
15
16. “It really got me *excited*.The part that got me most is
about spreading interest in teaching CS: I do think that it’s
REALLY great for teaching | learning!”
“I used to love the first person shooters and the
satisfaction of blowing away a whole team of
Noobies playing Rainbow Six, but this is far more
fun.”
“I’m afraid I’ll have to constrain myself to spend just an hour
or so a day on this really exciting stuff, as I’m really stuffed
with work.”
X
17. NOT Random:
Cheap, Fast
“It passed a thousand tests” feeling
…
But Dynamic Symbolic Execution:
e.g., Pex, CUTE,EXE
White box
Constraint Solving
18. Code to generate inputs for:
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==1234567890
void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a!=null &&
a.Length>0 &&
a[0]!=1234567890
a!=null &&
a.Length>0 &&
a[0]==1234567890
Data
null
{}
{0}
{123…}
a==null
a.Length>0
a[0]==123…
T
TF
T
F
F
Execute&MonitorSolve
Choose next path
Done: There is no path left.
Negated condition
19. There are decision procedures for individual path
conditions, but…
Number of potential paths grows exponentially with
number of branches
Reachable code not known initially
Without guidance, same loop might be unfolded
forever
Fitnex search strategy
[Xie et al. DSN 09]
http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf
20. public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
TestLoop(0, {0})
Path condition:
!(x == 90)
↓
New path condition:
(x == 90)
↓
New test input:
TestLoop(90, {0})
21. public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
TestLoop(90, {0})
Path condition:
(x == 90) && !(y[0] ==15) && !(x == 110)
↓
New path condition:
(x == 90) && (y[0] ==15)
↓
New test input:
TestLoop(90, {15})
22. public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
TestLoop(90, {15})
Path condition:
(x == 90) && (y[0] ==15)
&& !(x+1 == 110)
↓
New path condition:
(x == 90) && (y[0] ==15)
&& (x+1 == 110)
↓
New test input:
No solution!?
24. public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
TestLoop(90, {15})
We can have infinite paths!
Manual analysis need at
least 20 loop iterations to
cover the target branch
Exploring all paths up to 20
loop iterations is infeasible:
220 paths
25. public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
Key observations: with respect to the
coverage target
not all paths are equally promising for
branch-node flipping
not all branch nodes are equally
promising to flip
• Our solution:
– Prefer to flip branch nodes on the most promising paths
– Prefer to flip the most promising branch nodes on paths
– Fitness function to measure “promising” extents
TestLoop(90, {15, 0})
TestLoop(90, {15, 15})
[Xie et al. DSN 2009]
http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf
26. FF computes fitness value (distance between the
current state and the goal state)
Search tries to minimize fitness value
[Tracey et al. 98, Liu at al. 05, …]
27. public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
Fitness function: |110 – x |
28. public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
(90, {0}) 20
(90, {15}) 19
(90, {15, 0}) 19
(90, {15, 15}) 18
(90, {15, 15, 0}) 18
(90, {15, 15, 15}) 17
(90, {15, 15, 15, 0}) 17
(90, {15, 15, 15, 15}) 16
(90, {15, 15, 15, 15, 0}) 16
(90, {15, 15, 15, 15, 15}) 15
…
FitnessValue(x, y)
Fitness function: |110 – x |
Give preference to flip paths with better fitness values
We still need to address which branch node to flip on paths …
30. For a flipped node leading to Fnew, find out
the old fitness value Fold before flipping
• Assign Fitness Gain (Fold – Fnew) for the branch of the
flipped node
• Assign Fitness Gain (Fnew – Fold ) for the other branch of
the branch of the flipped node
Compute the average fitness gain for each
branch over time
31. Each branch node candidate for being flipped is
prioritized based on its composite fitness value:
• (Fitness value of node – Fitness gain of its branch)
Select first the one with the best composite
fitness value
32. Pex (released on May 2008)
Shipped with Visual Studio 15 as IntelliTest
30,388 download# (20 months, Feb 08-Oct 09)
22,466 download# (10 months, Apr 13-Jan 14): Code Digger
Active user community: 1,436 forum posts during ~3
years (Oct 08- Nov 11)
Moles (released on Sept 2009)
Shipped with Visual Studio 12 as Fakes
“Provide Microsoft Fakes w/ allVisual Studio editions” got
1,457 community votes
How to make such
successful case????
33. voidTestAdd(ArrayList a, object o) {
Assume.IsTrue(a!=null);
int i = a.Count;
a.Add(o);
Assert.IsTrue(a[i] == o);
}
Parameterized UnitTests Supported by Pex
Moles/Fakes
Code Digger
Pex4Fun/Code Hunt
Surrounding (Moles/Fakes)
Simplifying (Code Digger)
Retargeting (Pex4Fun/Code Hunt)
34. Developer/manager: “Who is using your tool?”
Pex team: “Do you want to be the first?”
Developer/manager: “I love your tool but no.”
Tool Adoption by (Mass)Target Users
Tool Shipping withVisual Studio
Macro Perspective
Micro Perspective
35. Developer: “Code digger generates a lot of “0” strings
as input. I can’t find a way to create such a string via my
own C# code. Could any one show me a C# snippet? I
meant zero terminated string.”
Pex team: “In C#, a 0 in a string does not mean zero-
termination. It’s just yet another character in the string
(a very simple character where all bits are zero), and
you can create as Pex shows the value: “0”.”
Developer: “Your tool generated “0””
Pex team: “What did you expect?”
Developer: “Marc.”
36. Developer: “Your tool generated a test called Foo001. I
don’t like it.”
Pex team: “What did you expect?”
Developer:“Foo_Should_Fail_When_Bar_Is_Negative.”
37. Object Creation messages suppressed
(related to Covana by Xiao et al. [ICSE’11])
ExceptionTreeView
Exploration TreeView
Exploration ResultsView
http://taoxie.cs.illinois.edu/publications/icse11-covana.pdf
38. public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
Key observations: with respect to the
coverage target
not all paths are equally promising for
branch-node flipping
not all branch nodes are equally
promising to flip
• Our solution:
– Prefer to flip branch nodes on the most promising paths
– Prefer to flip the most promising branch nodes on paths
– Fitness function to measure “promising” extents
Fitnex by Xie et al. [DSN’09]
To avoid local optimal or biases, the
fitness-guided strategy is integrated
with Pex’s fairness search strategies
http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf
39. “Simply one mouse click and then everything would work
just perfectly”
Often need environment isolation w/ Moles/Fakes or factory
methods, …
“One mouse click, a test generation tool would detect all
or most kinds of faults in the code under test”
Developer: “Your tool only finds null references.”
Pex team: “Did you write any assertions?”
Developer: “Assertion???”
“I do not need test generation; I already practice unit
testing (and/orTDD).Test generation does not fit into the
TDD process”
40. Gathered feedback from target tool users
Directly, e.g., via
MSDN Pex forum, tech support, outreach to MS engineers and
.NET user groups
Indirectly, e.g., via
interactions with MSVisual Studio team (a tool vendor to its huge
user base)
Motivations of Moles
Refactoring testability issue faced resistance in practice
Observation at Agile 2008: high attention on mock
objects and tool supports
41. Win-win collaboration model
Win (Ind Lab): longer-term research innovation, man power,
research impacts, …
Win (Univ): powerful infrastructure, relevant/important
problems in practice, both research and industry impacts, …
Industry-located Collaborations
Faculty visits, e.g., Fitnex, Pex4Fun
Student internships, e.g., FloPSy, DyGen, state cov
Academia-locatedCollaborations
43. Pex practice impacts
Moles/Fakes, Code Digger, Pex4Fun/Code Hunt
Lessons in transferring tools
Started as (Evolved) Dream
Chicken and Egg
Human Factors
Best vs.Worst Cases
Tool Users’ Stereotypical Mindset or Habits
Practitioners’Voice
Collaboration w/ Academia
44. Start a startup
but desirable to have right people (e.g., former students) to
start
Release free tools/libraries to aim for adoption
but a lot of efforts to be invested on “non-researchy” stuffs
Collaborate with industrial research labs
but many research lab projects may look like univ. projects
Collaborate with industrial product groups
but many probs faced by product groups may not be
“researchy”
45. NikolaiTillmann, Jonathan de Halleux, and Tao Xie. Transferring an Automated Test Generation
Tool to Practice: From Pex to Fakes and Code Digger. In Proceedings of ASE 2014, Experience
Papers. http://taoxie.cs.illinois.edu/publications/ase14-pexexperiences.pdf
Jian-Guang Lou, Qingwei Lin, Rui Ding, Qiang Fu, Dongmei Zhang, and Tao Xie. Software Analytics
for Incident Management of Online Services: An Experience Report. In Proceedings ASE 2013,
Experience Paper. http://taoxie.cs.illinois.edu/publications/ase13-sas.pdf
Dongmei Zhang, Shi Han,Yingnong Dang, Jian-Guang Lou, Haidong Zhang, andTao Xie. Software
Analytics in Practice. IEEE Software, Special Issue on the Many Faces of Software Analytics, 2013.
http://taoxie.cs.illinois.edu/publications/ieeesoft13-softanalytics.pdf
Yingnong Dang, Dongmei Zhang, Song Ge, Chengyun Chu,Yingjun Qiu, and Tao Xie. XIAO:Tuning
Code Clones at Hands of Engineers in Practice. In Proceedings of ACSAC 2012.
http://taoxie.cs.illinois.edu/publications/acsac12-xiao.pdf
47. Start a startup
but desirable to have right people (e.g., former students) to
start
Release free tools/libraries to aim for adoption
but a lot of efforts to be invested on “non-researchy” stuffs
Collaborate with industrial research labs
but many research lab projects may look like univ. projects
Collaborate with industrial product groups
but many probs faced by product groups may not be
“researchy”