Transferring Software Testing and Analytics Tools to Practice

Tao Xie
Tao XieSoftware Engineering Researcher & Educator
Tao Xie
University of Illinois at Urbana-Champaign
Part of the research work described in this talk was done in collaboration with the Pex team and SoftwareAnalytics
Group @Microsoft Research, students @IllinoisASE, and other collaborators
 Research impact: inspiring/impactful
ideas/directions/subareas… for researchers
 Example: model checking
 Practice impact: Practice adoption of
tools/systems/technologies… for practitioners
 Some examples discussed in this talk
 Societal impact: inspiring/impactful
ideas/thinking/awareness… for general public
 Example: computational thinking, privacy, medical-
device security, MOOCs, …
 Publishing research results  technologies
there adopted by companies, e.g.,
ICSE 00 Daikon paper by Ernst et al.  Agitar Agitator
https://homes.cs.washington.edu/~mernst/pubs/invariants-relevance-icse2000.pdf
ASE 04 Rostra paper by Xie et al.  Parasoft Jtest improvement
http://taoxie.cs.illinois.edu/publications/ase04.pdf
PLDI/FSE 05 DART/CUTE papers by Sen et al.  MSR SAGE, Pex
http://srl.cs.berkeley.edu/~ksen/papers/dart.pdf
http://srl.cs.berkeley.edu/~ksen/papers/C159-sen.pdf
…
 Commercializing research results in startup 
tools/products used by companies, e.g.,
Reactis®
…
 Transferring research results to product groups
 tools/products used inside company or
outside, e.g.,
SAGE
Flash Fill
…
CloudBuild
Tools for Software Engineers
Fakes
 Release open source infrastructures or libraries
to engage academic/industrial communities to
use and contribute, e.g.,
▪ MPI/PETSc by Bill Gropp et al.
▪ Charm++ by Laxmikant (Sanjay) Kale et al.
▪ LLVM byVikram Adve, Chris Lattner, et al.
“The openness of the LLVM technology and the quality of its architecture
and engineering design are key factors in understanding the success it has
had both in academia and industry.”
KLEE? JPF?FindBugs?Shipshape? Soot? WALA? …
 50 years of automated debugging research
 N papers  only 5 evaluated with actual programmers
“
” [Parnin&Orso ISSTA’11]
http://dl.acm.org/citation.cfm?id=2001445
 Human
 Expensive, incomplete, …
 Brute Force
 Pairwise, predefined data, etc…
 Tool Automation!!
Running Symbolic PathFinder ...
…
============================================
========== results
no errors detected
============================================
========== statistics
elapsed time: 0:00:02
states: new=4, visited=0,
backtracked=4, end=2
search: maxDepth=3,
constraints=0
choice generators: thread=1, data=2
heap: gc=3, new=271, free=22
instructions: 2875
max memory: 81MB
loaded code: classes=71, methods=884
…
9
 Pex (released on May 2008)
 Shipped with Visual Studio 15 as IntelliTest
 30,388 download# (20 months, Feb 08-Oct 09)
 22,466 download# (10 months, Apr 13-Jan 14): Code Digger
 Active user community: 1,436 forum posts during ~3 years
(Oct 08- Nov 11)
 Moles (released on Sept 2009)
 Shipped with Visual Studio 12 as Fakes
 “Provide Microsoft Fakes w/ allVisual Studio editions” got
1,457 community votes
https://visualstudio.uservoice.com/forums/121579-visual-studio-2015/suggestions/6773265-make-intellitest-available-to-visual-studio-profes
https://visualstudio.uservoice.com/forums/121579-visual-studio-2015/suggestions/2216195-include-pex-and-moles-with-all-visual-studio-editi
1,753,594 clicked 'Ask Pex!'
http://pex4fun.com/
https://www.codehunt.com/
Code Hunt can identify
top coders
http://programming2015.cstnet.cn/
Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
classTest {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
15
“It really got me *excited*.The part that got me most is
about spreading interest in teaching CS: I do think that it’s
REALLY great for teaching | learning!”
“I used to love the first person shooters and the
satisfaction of blowing away a whole team of
Noobies playing Rainbow Six, but this is far more
fun.”
“I’m afraid I’ll have to constrain myself to spend just an hour
or so a day on this really exciting stuff, as I’m really stuffed
with work.”
X
 NOT Random:
 Cheap, Fast
 “It passed a thousand tests” feeling
 …
 But Dynamic Symbolic Execution:
e.g., Pex, CUTE,EXE
 White box
 Constraint Solving
Code to generate inputs for:
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==1234567890
void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a!=null &&
a.Length>0 &&
a[0]!=1234567890
a!=null &&
a.Length>0 &&
a[0]==1234567890
Data
null
{}
{0}
{123…}
a==null
a.Length>0
a[0]==123…
T
TF
T
F
F
Execute&MonitorSolve
Choose next path
Done: There is no path left.
Negated condition
There are decision procedures for individual path
conditions, but…
 Number of potential paths grows exponentially with
number of branches
 Reachable code not known initially
 Without guidance, same loop might be unfolded
forever
Fitnex search strategy
[Xie et al. DSN 09]
http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
TestLoop(0, {0})
Path condition:
!(x == 90)
↓
New path condition:
(x == 90)
↓
New test input:
TestLoop(90, {0})
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
TestLoop(90, {0})
Path condition:
(x == 90) && !(y[0] ==15) && !(x == 110)
↓
New path condition:
(x == 90) && (y[0] ==15)
↓
New test input:
TestLoop(90, {15})
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
TestLoop(90, {15})
Path condition:
(x == 90) && (y[0] ==15)
&& !(x+1 == 110)
↓
New path condition:
(x == 90) && (y[0] ==15)
&& (x+1 == 110)
↓
New test input:
No solution!?
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
TestLoop(90, {15})
Path condition:
(x == 90) && (y[0] ==15)
&& (0 < y.Length)
&& !(1 < y.Length)
&& !(x+1 == 110)
↓
New path condition:
(x == 90) && (y[0] ==15)
&& (0 < y.Length)
&& (1 < y.Length)
 Expand array size
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
TestLoop(90, {15})
We can have infinite paths!
Manual analysis  need at
least 20 loop iterations to
cover the target branch
Exploring all paths up to 20
loop iterations is infeasible:
220 paths
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
Key observations: with respect to the
coverage target
 not all paths are equally promising for
branch-node flipping
 not all branch nodes are equally
promising to flip
• Our solution:
– Prefer to flip branch nodes on the most promising paths
– Prefer to flip the most promising branch nodes on paths
– Fitness function to measure “promising” extents
TestLoop(90, {15, 0})
TestLoop(90, {15, 15})
[Xie et al. DSN 2009]
http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf
 FF computes fitness value (distance between the
current state and the goal state)
 Search tries to minimize fitness value
[Tracey et al. 98, Liu at al. 05, …]
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
Fitness function: |110 – x |
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
(90, {0}) 20
(90, {15}) 19
(90, {15, 0}) 19
(90, {15, 15}) 18
(90, {15, 15, 0}) 18
(90, {15, 15, 15}) 17
(90, {15, 15, 15, 0}) 17
(90, {15, 15, 15, 15}) 16
(90, {15, 15, 15, 15, 0}) 16
(90, {15, 15, 15, 15, 15}) 15
…
FitnessValue(x, y)
Fitness function: |110 – x |
Give preference to flip paths with better fitness values
We still need to address which branch node to flip on paths …
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
(90, {0}) 20
(90, {15})  flip b4 19
(90, {15, 0})  flip b2 19
(90, {15, 15})  flip b4 18
(90, {15, 15, 0})  flip b2 18
(90, {15, 15, 15})  flip b4 17
(90, {15, 15, 15, 0})  flip b2 17
(90, {15, 15, 15, 15})  flip b4 16
(90, {15, 15, 15, 15, 0})  flip b2 16
(90, {15, 15, 15, 15, 15})  flip b4 15
…
FitnessValue(x, y)
Fitness function: |110 – x |
Branch b1: i < y.Length
Branch b2: i >= y.Length
Branch b3: y[i] == 15
Branch b4: y[i] != 15
•Flipping Branch b4 (b3) gives us average 1
(-1) fitness gain (loss)
•Flipping branch b2 (b1) gives us average 0
fitness gain (loss)
 For a flipped node leading to Fnew, find out
the old fitness value Fold before flipping
• Assign Fitness Gain (Fold – Fnew) for the branch of the
flipped node
• Assign Fitness Gain (Fnew – Fold ) for the other branch of
the branch of the flipped node
 Compute the average fitness gain for each
branch over time
 Each branch node candidate for being flipped is
prioritized based on its composite fitness value:
• (Fitness value of node – Fitness gain of its branch)
 Select first the one with the best composite
fitness value
 Pex (released on May 2008)
 Shipped with Visual Studio 15 as IntelliTest
 30,388 download# (20 months, Feb 08-Oct 09)
 22,466 download# (10 months, Apr 13-Jan 14): Code Digger
 Active user community: 1,436 forum posts during ~3
years (Oct 08- Nov 11)
 Moles (released on Sept 2009)
 Shipped with Visual Studio 12 as Fakes
 “Provide Microsoft Fakes w/ allVisual Studio editions” got
1,457 community votes
How to make such
successful case????
voidTestAdd(ArrayList a, object o) {
Assume.IsTrue(a!=null);
int i = a.Count;
a.Add(o);
Assert.IsTrue(a[i] == o);
}
Parameterized UnitTests Supported by Pex
Moles/Fakes
Code Digger
Pex4Fun/Code Hunt
 Surrounding (Moles/Fakes)
 Simplifying (Code Digger)
 Retargeting (Pex4Fun/Code Hunt)
 Developer/manager: “Who is using your tool?”
 Pex team: “Do you want to be the first?”
 Developer/manager: “I love your tool but no.”
Tool Adoption by (Mass)Target Users
Tool Shipping withVisual Studio
Macro Perspective
Micro Perspective
 Developer: “Code digger generates a lot of “0” strings
as input. I can’t find a way to create such a string via my
own C# code. Could any one show me a C# snippet? I
meant zero terminated string.”
 Pex team: “In C#, a 0 in a string does not mean zero-
termination. It’s just yet another character in the string
(a very simple character where all bits are zero), and
you can create as Pex shows the value: “0”.”
 Developer: “Your tool generated “0””
 Pex team: “What did you expect?”
 Developer: “Marc.”
 Developer: “Your tool generated a test called Foo001. I
don’t like it.”
 Pex team: “What did you expect?”
 Developer:“Foo_Should_Fail_When_Bar_Is_Negative.”
Object Creation messages suppressed
(related to Covana by Xiao et al. [ICSE’11])
ExceptionTreeView
Exploration TreeView
Exploration ResultsView
http://taoxie.cs.illinois.edu/publications/icse11-covana.pdf
public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
Key observations: with respect to the
coverage target
 not all paths are equally promising for
branch-node flipping
 not all branch nodes are equally
promising to flip
• Our solution:
– Prefer to flip branch nodes on the most promising paths
– Prefer to flip the most promising branch nodes on paths
– Fitness function to measure “promising” extents
Fitnex by Xie et al. [DSN’09]
To avoid local optimal or biases, the
fitness-guided strategy is integrated
with Pex’s fairness search strategies
http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf
 “Simply one mouse click and then everything would work
just perfectly”
 Often need environment isolation w/ Moles/Fakes or factory
methods, …
 “One mouse click, a test generation tool would detect all
or most kinds of faults in the code under test”
 Developer: “Your tool only finds null references.”
 Pex team: “Did you write any assertions?”
 Developer: “Assertion???”
 “I do not need test generation; I already practice unit
testing (and/orTDD).Test generation does not fit into the
TDD process”
Gathered feedback from target tool users
 Directly, e.g., via
 MSDN Pex forum, tech support, outreach to MS engineers and
.NET user groups
 Indirectly, e.g., via
 interactions with MSVisual Studio team (a tool vendor to its huge
user base)
 Motivations of Moles
 Refactoring testability issue faced resistance in practice
 Observation at Agile 2008: high attention on mock
objects and tool supports
 Win-win collaboration model
 Win (Ind Lab): longer-term research innovation, man power,
research impacts, …
 Win (Univ): powerful infrastructure, relevant/important
problems in practice, both research and industry impacts, …
 Industry-located Collaborations
 Faculty visits, e.g., Fitnex, Pex4Fun
 Student internships, e.g., FloPSy, DyGen, state cov
 Academia-locatedCollaborations
Academia-located Collaborations
 Immediate indirect impacts, e.g.,
 Reggae [ASE’09s]  Rex
 MSeqGen [FSE’09]  DyGen
 Guided Cov [ICSM’10]  state coverage
 Long-term indirect impacts, e.g.,
 DySy by Csallner et al. [ICSE’08]
 Seeker [OOPSLA’11]
 Covana [ICSE’11]
 Pex  practice impacts
 Moles/Fakes, Code Digger, Pex4Fun/Code Hunt
 Lessons in transferring tools
 Started as (Evolved) Dream
 Chicken and Egg
 Human Factors
 Best vs.Worst Cases
 Tool Users’ Stereotypical Mindset or Habits
 Practitioners’Voice
 Collaboration w/ Academia
 Start a startup
 but desirable to have right people (e.g., former students) to
start
 Release free tools/libraries to aim for adoption
 but a lot of efforts to be invested on “non-researchy” stuffs
 Collaborate with industrial research labs
 but many research lab projects may look like univ. projects
 Collaborate with industrial product groups
 but many probs faced by product groups may not be
“researchy”
 NikolaiTillmann, Jonathan de Halleux, and Tao Xie. Transferring an Automated Test Generation
Tool to Practice: From Pex to Fakes and Code Digger. In Proceedings of ASE 2014, Experience
Papers. http://taoxie.cs.illinois.edu/publications/ase14-pexexperiences.pdf
 Jian-Guang Lou, Qingwei Lin, Rui Ding, Qiang Fu, Dongmei Zhang, and Tao Xie. Software Analytics
for Incident Management of Online Services: An Experience Report. In Proceedings ASE 2013,
Experience Paper. http://taoxie.cs.illinois.edu/publications/ase13-sas.pdf
 Dongmei Zhang, Shi Han,Yingnong Dang, Jian-Guang Lou, Haidong Zhang, andTao Xie. Software
Analytics in Practice. IEEE Software, Special Issue on the Many Faces of Software Analytics, 2013.
http://taoxie.cs.illinois.edu/publications/ieeesoft13-softanalytics.pdf
 Yingnong Dang, Dongmei Zhang, Song Ge, Chengyun Chu,Yingjun Qiu, and Tao Xie. XIAO:Tuning
Code Clones at Hands of Engineers in Practice. In Proceedings of ACSAC 2012.
http://taoxie.cs.illinois.edu/publications/acsac12-xiao.pdf
Questions ?
https://sites.google.com/site/asergrp/
http://research.microsoft.com/pex
http://research.microsoft.com/sa/
taoxie@illinois.edu
 Start a startup
 but desirable to have right people (e.g., former students) to
start
 Release free tools/libraries to aim for adoption
 but a lot of efforts to be invested on “non-researchy” stuffs
 Collaborate with industrial research labs
 but many research lab projects may look like univ. projects
 Collaborate with industrial product groups
 but many probs faced by product groups may not be
“researchy”
1 of 47

Recommended

Next Generation Developer Testing: Parameterized Testing by
Next Generation Developer Testing: Parameterized TestingNext Generation Developer Testing: Parameterized Testing
Next Generation Developer Testing: Parameterized TestingTao Xie
1.6K views52 slides
Csise15 codehunt by
Csise15 codehuntCsise15 codehunt
Csise15 codehuntTao Xie
658 views24 slides
Advances in Unit Testing: Theory and Practice by
Advances in Unit Testing: Theory and PracticeAdvances in Unit Testing: Theory and Practice
Advances in Unit Testing: Theory and PracticeTao Xie
1.3K views90 slides
Testing for Educational Gaming and Educational Gaming for Testing by
Testing for Educational Gaming and Educational Gaming for TestingTesting for Educational Gaming and Educational Gaming for Testing
Testing for Educational Gaming and Educational Gaming for TestingTao Xie
595 views54 slides
Mutation @ Spotify by
Mutation @ Spotify Mutation @ Spotify
Mutation @ Spotify STAMP Project
100 views47 slides
Java Puzzlers by
Java PuzzlersJava Puzzlers
Java PuzzlersMike Donikian
1.6K views14 slides

More Related Content

What's hot

Java Language fundamental by
Java Language fundamentalJava Language fundamental
Java Language fundamentalInfoviaan Technologies
115 views39 slides
54240326 copy by
54240326   copy54240326   copy
54240326 copyVinayak Shedgeri
165 views28 slides
Final JAVA Practical of BCA SEM-5. by
Final JAVA Practical of BCA SEM-5.Final JAVA Practical of BCA SEM-5.
Final JAVA Practical of BCA SEM-5.Nishan Barot
8.5K views23 slides
Java practical(baca sem v) by
Java practical(baca sem v)Java practical(baca sem v)
Java practical(baca sem v)mehul patel
1.1K views103 slides
Java programs by
Java programsJava programs
Java programsMukund Gandrakota
987 views53 slides
DCN Practical by
DCN PracticalDCN Practical
DCN PracticalNiraj Bharambe
522 views18 slides

What's hot(20)

Final JAVA Practical of BCA SEM-5. by Nishan Barot
Final JAVA Practical of BCA SEM-5.Final JAVA Practical of BCA SEM-5.
Final JAVA Practical of BCA SEM-5.
Nishan Barot8.5K views
Java practical(baca sem v) by mehul patel
Java practical(baca sem v)Java practical(baca sem v)
Java practical(baca sem v)
mehul patel1.1K views
Object oriented programming la bmanual jntu by Khurshid Asghar
Object oriented programming la bmanual jntuObject oriented programming la bmanual jntu
Object oriented programming la bmanual jntu
Khurshid Asghar464 views
Important java programs(collection+file) by Alok Kumar
Important java programs(collection+file)Important java programs(collection+file)
Important java programs(collection+file)
Alok Kumar85.1K views
Google mock for dummies by Harry Potter
Google mock for dummiesGoogle mock for dummies
Google mock for dummies
Harry Potter1.9K views
OCJP Samples Questions: Exceptions and assertions by Hari kiran G
OCJP Samples Questions: Exceptions and assertionsOCJP Samples Questions: Exceptions and assertions
OCJP Samples Questions: Exceptions and assertions
Hari kiran G3.3K views
Introduction to Software Testing by Sergio Arroyo
Introduction to Software TestingIntroduction to Software Testing
Introduction to Software Testing
Sergio Arroyo2.8K views

Viewers also liked

HotSoS16 Tutorial "Text Analytics for Security" by Tao Xie and William Enck by
HotSoS16 Tutorial "Text Analytics for Security" by Tao Xie and William EnckHotSoS16 Tutorial "Text Analytics for Security" by Tao Xie and William Enck
HotSoS16 Tutorial "Text Analytics for Security" by Tao Xie and William EnckTao Xie
1.2K views78 slides
Common Technical Writing Issues by
Common Technical Writing IssuesCommon Technical Writing Issues
Common Technical Writing IssuesTao Xie
3.3K views55 slides
Impact-Driven Research on Software Engineering Tooling by
Impact-Driven Research on Software Engineering ToolingImpact-Driven Research on Software Engineering Tooling
Impact-Driven Research on Software Engineering ToolingTao Xie
754 views26 slides
Software Analytics - Achievements and Challenges by
Software Analytics - Achievements and ChallengesSoftware Analytics - Achievements and Challenges
Software Analytics - Achievements and ChallengesTao Xie
1.8K views65 slides
User Expectations in Mobile App Security by
User Expectations in Mobile App SecurityUser Expectations in Mobile App Security
User Expectations in Mobile App SecurityTao Xie
942 views47 slides
Transferring Software Testing Tools to Practice by
Transferring Software Testing Tools to PracticeTransferring Software Testing Tools to Practice
Transferring Software Testing Tools to PracticeTao Xie
539 views34 slides

Viewers also liked(7)

HotSoS16 Tutorial "Text Analytics for Security" by Tao Xie and William Enck by Tao Xie
HotSoS16 Tutorial "Text Analytics for Security" by Tao Xie and William EnckHotSoS16 Tutorial "Text Analytics for Security" by Tao Xie and William Enck
HotSoS16 Tutorial "Text Analytics for Security" by Tao Xie and William Enck
Tao Xie1.2K views
Common Technical Writing Issues by Tao Xie
Common Technical Writing IssuesCommon Technical Writing Issues
Common Technical Writing Issues
Tao Xie3.3K views
Impact-Driven Research on Software Engineering Tooling by Tao Xie
Impact-Driven Research on Software Engineering ToolingImpact-Driven Research on Software Engineering Tooling
Impact-Driven Research on Software Engineering Tooling
Tao Xie754 views
Software Analytics - Achievements and Challenges by Tao Xie
Software Analytics - Achievements and ChallengesSoftware Analytics - Achievements and Challenges
Software Analytics - Achievements and Challenges
Tao Xie1.8K views
User Expectations in Mobile App Security by Tao Xie
User Expectations in Mobile App SecurityUser Expectations in Mobile App Security
User Expectations in Mobile App Security
Tao Xie942 views
Transferring Software Testing Tools to Practice by Tao Xie
Transferring Software Testing Tools to PracticeTransferring Software Testing Tools to Practice
Transferring Software Testing Tools to Practice
Tao Xie539 views
Software Mining and Software Datasets by Tao Xie
Software Mining and Software DatasetsSoftware Mining and Software Datasets
Software Mining and Software Datasets
Tao Xie2.5K views

Similar to Transferring Software Testing and Analytics Tools to Practice

Transferring Software Testing Tools to Practice (AST 2017 Keynote) by
Transferring Software Testing Tools to Practice (AST 2017 Keynote)Transferring Software Testing Tools to Practice (AST 2017 Keynote)
Transferring Software Testing Tools to Practice (AST 2017 Keynote)Tao Xie
390 views66 slides
Software Testing:
 A Research Travelogue 
(2000–2014) by
Software Testing:
 A Research Travelogue 
(2000–2014)Software Testing:
 A Research Travelogue 
(2000–2014)
Software Testing:
 A Research Travelogue 
(2000–2014)Alex Orso
2.5K views36 slides
Mutation Testing: Start Hunting The Bugs by
Mutation Testing: Start Hunting The BugsMutation Testing: Start Hunting The Bugs
Mutation Testing: Start Hunting The BugsAri Waller
107 views62 slides
COM1407: Program Control Structures – Repetition and Loops by
COM1407: Program Control Structures – Repetition and Loops COM1407: Program Control Structures – Repetition and Loops
COM1407: Program Control Structures – Repetition and Loops Hemantha Kulathilake
409 views51 slides
ParallelProgrammingBasics_v2.pdf by
ParallelProgrammingBasics_v2.pdfParallelProgrammingBasics_v2.pdf
ParallelProgrammingBasics_v2.pdfChen-Hung Hu
2 views40 slides
.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech by
.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech
.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTechNETFest
570 views73 slides

Similar to Transferring Software Testing and Analytics Tools to Practice(20)

Transferring Software Testing Tools to Practice (AST 2017 Keynote) by Tao Xie
Transferring Software Testing Tools to Practice (AST 2017 Keynote)Transferring Software Testing Tools to Practice (AST 2017 Keynote)
Transferring Software Testing Tools to Practice (AST 2017 Keynote)
Tao Xie390 views
Software Testing:
 A Research Travelogue 
(2000–2014) by Alex Orso
Software Testing:
 A Research Travelogue 
(2000–2014)Software Testing:
 A Research Travelogue 
(2000–2014)
Software Testing:
 A Research Travelogue 
(2000–2014)
Alex Orso2.5K views
Mutation Testing: Start Hunting The Bugs by Ari Waller
Mutation Testing: Start Hunting The BugsMutation Testing: Start Hunting The Bugs
Mutation Testing: Start Hunting The Bugs
Ari Waller107 views
COM1407: Program Control Structures – Repetition and Loops by Hemantha Kulathilake
COM1407: Program Control Structures – Repetition and Loops COM1407: Program Control Structures – Repetition and Loops
COM1407: Program Control Structures – Repetition and Loops
ParallelProgrammingBasics_v2.pdf by Chen-Hung Hu
ParallelProgrammingBasics_v2.pdfParallelProgrammingBasics_v2.pdf
ParallelProgrammingBasics_v2.pdf
Chen-Hung Hu2 views
.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech by NETFest
.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech
.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech
NETFest570 views
Recommending job ads to people by Fabian Abel
Recommending job ads to peopleRecommending job ads to people
Recommending job ads to people
Fabian Abel1.5K views
JavaOne 2017 - The hitchhiker’s guide to Java class reloading by Anton Arhipov
JavaOne 2017 - The hitchhiker’s guide to Java class reloadingJavaOne 2017 - The hitchhiker’s guide to Java class reloading
JavaOne 2017 - The hitchhiker’s guide to Java class reloading
Anton Arhipov902 views
JavaOne 2017 - The hitchhiker’s guide to Java class reloading by Anton Arhipov
JavaOne 2017 - The hitchhiker’s guide to Java class reloadingJavaOne 2017 - The hitchhiker’s guide to Java class reloading
JavaOne 2017 - The hitchhiker’s guide to Java class reloading
Anton Arhipov269 views
Types Working for You, Not Against You by C4Media
Types Working for You, Not Against YouTypes Working for You, Not Against You
Types Working for You, Not Against You
C4Media277 views
DSR Testing (Part 2) by Steve Upton
DSR Testing (Part 2)DSR Testing (Part 2)
DSR Testing (Part 2)
Steve Upton705 views
Building Single-Page Web Appplications in dart - Devoxx France 2013 by yohanbeschi
Building Single-Page Web Appplications in dart - Devoxx France 2013Building Single-Page Web Appplications in dart - Devoxx France 2013
Building Single-Page Web Appplications in dart - Devoxx France 2013
yohanbeschi9K views
STAMP Descartes Presentation by STAMP Project
STAMP Descartes PresentationSTAMP Descartes Presentation
STAMP Descartes Presentation
STAMP Project105 views
Boost delivery stream with code discipline engineering by Miro Wengner
Boost delivery stream with code discipline engineeringBoost delivery stream with code discipline engineering
Boost delivery stream with code discipline engineering
Miro Wengner22 views
DSR Testing (Part 1) by Steve Upton
DSR Testing (Part 1)DSR Testing (Part 1)
DSR Testing (Part 1)
Steve Upton1.1K views

More from Tao Xie

MSR 2022 Foundational Contribution Award Talk: Software Analytics: Reflection... by
MSR 2022 Foundational Contribution Award Talk: Software Analytics: Reflection...MSR 2022 Foundational Contribution Award Talk: Software Analytics: Reflection...
MSR 2022 Foundational Contribution Award Talk: Software Analytics: Reflection...Tao Xie
309 views62 slides
DSML 2021 Keynote: Intelligent Software Engineering: Working at the Intersect... by
DSML 2021 Keynote: Intelligent Software Engineering: Working at the Intersect...DSML 2021 Keynote: Intelligent Software Engineering: Working at the Intersect...
DSML 2021 Keynote: Intelligent Software Engineering: Working at the Intersect...Tao Xie
694 views42 slides
Intelligent Software Engineering: Synergy between AI and Software Engineering by
Intelligent Software Engineering: Synergy between AI and Software EngineeringIntelligent Software Engineering: Synergy between AI and Software Engineering
Intelligent Software Engineering: Synergy between AI and Software EngineeringTao Xie
713 views39 slides
Diversity and Computing/Engineering: Perspectives from Allies by
Diversity and Computing/Engineering: Perspectives from AlliesDiversity and Computing/Engineering: Perspectives from Allies
Diversity and Computing/Engineering: Perspectives from AlliesTao Xie
1.3K views44 slides
Intelligent Software Engineering: Synergy between AI and Software Engineering... by
Intelligent Software Engineering: Synergy between AI and Software Engineering...Intelligent Software Engineering: Synergy between AI and Software Engineering...
Intelligent Software Engineering: Synergy between AI and Software Engineering...Tao Xie
917 views46 slides
MSRA 2018: Intelligent Software Engineering: Synergy between AI and Software ... by
MSRA 2018: Intelligent Software Engineering: Synergy between AI and Software ...MSRA 2018: Intelligent Software Engineering: Synergy between AI and Software ...
MSRA 2018: Intelligent Software Engineering: Synergy between AI and Software ...Tao Xie
828 views33 slides

More from Tao Xie(19)

MSR 2022 Foundational Contribution Award Talk: Software Analytics: Reflection... by Tao Xie
MSR 2022 Foundational Contribution Award Talk: Software Analytics: Reflection...MSR 2022 Foundational Contribution Award Talk: Software Analytics: Reflection...
MSR 2022 Foundational Contribution Award Talk: Software Analytics: Reflection...
Tao Xie309 views
DSML 2021 Keynote: Intelligent Software Engineering: Working at the Intersect... by Tao Xie
DSML 2021 Keynote: Intelligent Software Engineering: Working at the Intersect...DSML 2021 Keynote: Intelligent Software Engineering: Working at the Intersect...
DSML 2021 Keynote: Intelligent Software Engineering: Working at the Intersect...
Tao Xie694 views
Intelligent Software Engineering: Synergy between AI and Software Engineering by Tao Xie
Intelligent Software Engineering: Synergy between AI and Software EngineeringIntelligent Software Engineering: Synergy between AI and Software Engineering
Intelligent Software Engineering: Synergy between AI and Software Engineering
Tao Xie713 views
Diversity and Computing/Engineering: Perspectives from Allies by Tao Xie
Diversity and Computing/Engineering: Perspectives from AlliesDiversity and Computing/Engineering: Perspectives from Allies
Diversity and Computing/Engineering: Perspectives from Allies
Tao Xie1.3K views
Intelligent Software Engineering: Synergy between AI and Software Engineering... by Tao Xie
Intelligent Software Engineering: Synergy between AI and Software Engineering...Intelligent Software Engineering: Synergy between AI and Software Engineering...
Intelligent Software Engineering: Synergy between AI and Software Engineering...
Tao Xie917 views
MSRA 2018: Intelligent Software Engineering: Synergy between AI and Software ... by Tao Xie
MSRA 2018: Intelligent Software Engineering: Synergy between AI and Software ...MSRA 2018: Intelligent Software Engineering: Synergy between AI and Software ...
MSRA 2018: Intelligent Software Engineering: Synergy between AI and Software ...
Tao Xie828 views
SETTA'18 Keynote: Intelligent Software Engineering: Synergy between AI and So... by Tao Xie
SETTA'18 Keynote: Intelligent Software Engineering: Synergy between AI and So...SETTA'18 Keynote: Intelligent Software Engineering: Synergy between AI and So...
SETTA'18 Keynote: Intelligent Software Engineering: Synergy between AI and So...
Tao Xie956 views
ISEC'18 Tutorial: Research Methodology on Pursuing Impact-Driven Research by Tao Xie
ISEC'18 Tutorial: Research Methodology on Pursuing Impact-Driven ResearchISEC'18 Tutorial: Research Methodology on Pursuing Impact-Driven Research
ISEC'18 Tutorial: Research Methodology on Pursuing Impact-Driven Research
Tao Xie1.8K views
ISEC'18 Keynote: Intelligent Software Engineering: Synergy between AI and Sof... by Tao Xie
ISEC'18 Keynote: Intelligent Software Engineering: Synergy between AI and Sof...ISEC'18 Keynote: Intelligent Software Engineering: Synergy between AI and Sof...
ISEC'18 Keynote: Intelligent Software Engineering: Synergy between AI and Sof...
Tao Xie4.2K views
Intelligent Software Engineering: Synergy between AI and Software Engineering by Tao Xie
Intelligent Software Engineering: Synergy between AI and Software EngineeringIntelligent Software Engineering: Synergy between AI and Software Engineering
Intelligent Software Engineering: Synergy between AI and Software Engineering
Tao Xie2.9K views
Software Analytics: Data Analytics for Software Engineering and Security by Tao Xie
Software Analytics: Data Analytics for Software Engineering and SecuritySoftware Analytics: Data Analytics for Software Engineering and Security
Software Analytics: Data Analytics for Software Engineering and Security
Tao Xie606 views
Planning and Executing Practice-Impactful Research by Tao Xie
Planning and Executing Practice-Impactful ResearchPlanning and Executing Practice-Impactful Research
Planning and Executing Practice-Impactful Research
Tao Xie1.5K views
Software Analytics: Data Analytics for Software Engineering by Tao Xie
Software Analytics: Data Analytics for Software EngineeringSoftware Analytics: Data Analytics for Software Engineering
Software Analytics: Data Analytics for Software Engineering
Tao Xie2K views
Text Analytics for Security by Tao Xie
Text Analytics for SecurityText Analytics for Security
Text Analytics for Security
Tao Xie2.5K views
Gamifying Teaching and Learning of Software Engineering and Programming by Tao Xie
Gamifying Teaching and Learning of Software Engineering and ProgrammingGamifying Teaching and Learning of Software Engineering and Programming
Gamifying Teaching and Learning of Software Engineering and Programming
Tao Xie959 views
Towards Mining Software Repositories Research that Matters by Tao Xie
Towards Mining Software Repositories Research that MattersTowards Mining Software Repositories Research that Matters
Towards Mining Software Repositories Research that Matters
Tao Xie1.9K views
Tutorial: Text Analytics for Security by Tao Xie
Tutorial: Text Analytics for SecurityTutorial: Text Analytics for Security
Tutorial: Text Analytics for Security
Tao Xie1.6K views
Software Analytics: Towards Software Mining that Matters (2014) by Tao Xie
Software Analytics:Towards Software Mining that Matters (2014)Software Analytics:Towards Software Mining that Matters (2014)
Software Analytics: Towards Software Mining that Matters (2014)
Tao Xie1.7K views
Teaching and Learning Programming and Software Engineering via Interactive Ga... by Tao Xie
Teaching and Learning Programming and Software Engineering via Interactive Ga...Teaching and Learning Programming and Software Engineering via Interactive Ga...
Teaching and Learning Programming and Software Engineering via Interactive Ga...
Tao Xie1.3K views

Recently uploaded

Navigating container technology for enhanced security by Niklas Saari by
Navigating container technology for enhanced security by Niklas SaariNavigating container technology for enhanced security by Niklas Saari
Navigating container technology for enhanced security by Niklas SaariMetosin Oy
14 views34 slides
Keep by
KeepKeep
KeepGeniusee
77 views10 slides
Quality Engineer: A Day in the Life by
Quality Engineer: A Day in the LifeQuality Engineer: A Day in the Life
Quality Engineer: A Day in the LifeJohn Valentino
6 views18 slides
Copilot Prompting Toolkit_All Resources.pdf by
Copilot Prompting Toolkit_All Resources.pdfCopilot Prompting Toolkit_All Resources.pdf
Copilot Prompting Toolkit_All Resources.pdfRiccardo Zamana
10 views4 slides
WebAssembly by
WebAssemblyWebAssembly
WebAssemblyJens Siebert
51 views18 slides
Fleet Management Software in India by
Fleet Management Software in India Fleet Management Software in India
Fleet Management Software in India Fleetable
11 views1 slide

Recently uploaded(20)

Navigating container technology for enhanced security by Niklas Saari by Metosin Oy
Navigating container technology for enhanced security by Niklas SaariNavigating container technology for enhanced security by Niklas Saari
Navigating container technology for enhanced security by Niklas Saari
Metosin Oy14 views
Quality Engineer: A Day in the Life by John Valentino
Quality Engineer: A Day in the LifeQuality Engineer: A Day in the Life
Quality Engineer: A Day in the Life
John Valentino6 views
Copilot Prompting Toolkit_All Resources.pdf by Riccardo Zamana
Copilot Prompting Toolkit_All Resources.pdfCopilot Prompting Toolkit_All Resources.pdf
Copilot Prompting Toolkit_All Resources.pdf
Riccardo Zamana10 views
Fleet Management Software in India by Fleetable
Fleet Management Software in India Fleet Management Software in India
Fleet Management Software in India
Fleetable11 views
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra... by Marc Müller
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra....NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
Marc Müller40 views
Sprint 226 by ManageIQ
Sprint 226Sprint 226
Sprint 226
ManageIQ5 views
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium... by Lisi Hocke
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...
Lisi Hocke35 views
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P... by NimaTorabi2
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
Unlocking the Power of AI in Product Management - A Comprehensive Guide for P...
NimaTorabi212 views
Dapr Unleashed: Accelerating Microservice Development by Miroslav Janeski
Dapr Unleashed: Accelerating Microservice DevelopmentDapr Unleashed: Accelerating Microservice Development
Dapr Unleashed: Accelerating Microservice Development
Miroslav Janeski10 views
Headless JS UG Presentation.pptx by Jack Spektor
Headless JS UG Presentation.pptxHeadless JS UG Presentation.pptx
Headless JS UG Presentation.pptx
Jack Spektor8 views
FIMA 2023 Neo4j & FS - Entity Resolution.pptx by Neo4j
FIMA 2023 Neo4j & FS - Entity Resolution.pptxFIMA 2023 Neo4j & FS - Entity Resolution.pptx
FIMA 2023 Neo4j & FS - Entity Resolution.pptx
Neo4j8 views
DSD-INT 2023 Process-based modelling of salt marsh development coupling Delft... by Deltares
DSD-INT 2023 Process-based modelling of salt marsh development coupling Delft...DSD-INT 2023 Process-based modelling of salt marsh development coupling Delft...
DSD-INT 2023 Process-based modelling of salt marsh development coupling Delft...
Deltares7 views
20231129 - Platform @ localhost 2023 - Application-driven infrastructure with... by sparkfabrik
20231129 - Platform @ localhost 2023 - Application-driven infrastructure with...20231129 - Platform @ localhost 2023 - Application-driven infrastructure with...
20231129 - Platform @ localhost 2023 - Application-driven infrastructure with...
sparkfabrik7 views
Airline Booking Software by SharmiMehta
Airline Booking SoftwareAirline Booking Software
Airline Booking Software
SharmiMehta6 views
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI... by Marc Müller
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...
Marc Müller41 views

Transferring Software Testing and Analytics Tools to Practice

  • 1. Tao Xie University of Illinois at Urbana-Champaign Part of the research work described in this talk was done in collaboration with the Pex team and SoftwareAnalytics Group @Microsoft Research, students @IllinoisASE, and other collaborators
  • 2.  Research impact: inspiring/impactful ideas/directions/subareas… for researchers  Example: model checking  Practice impact: Practice adoption of tools/systems/technologies… for practitioners  Some examples discussed in this talk  Societal impact: inspiring/impactful ideas/thinking/awareness… for general public  Example: computational thinking, privacy, medical- device security, MOOCs, …
  • 3.  Publishing research results  technologies there adopted by companies, e.g., ICSE 00 Daikon paper by Ernst et al.  Agitar Agitator https://homes.cs.washington.edu/~mernst/pubs/invariants-relevance-icse2000.pdf ASE 04 Rostra paper by Xie et al.  Parasoft Jtest improvement http://taoxie.cs.illinois.edu/publications/ase04.pdf PLDI/FSE 05 DART/CUTE papers by Sen et al.  MSR SAGE, Pex http://srl.cs.berkeley.edu/~ksen/papers/dart.pdf http://srl.cs.berkeley.edu/~ksen/papers/C159-sen.pdf …
  • 4.  Commercializing research results in startup  tools/products used by companies, e.g., Reactis® …
  • 5.  Transferring research results to product groups  tools/products used inside company or outside, e.g., SAGE Flash Fill … CloudBuild Tools for Software Engineers Fakes
  • 6.  Release open source infrastructures or libraries to engage academic/industrial communities to use and contribute, e.g., ▪ MPI/PETSc by Bill Gropp et al. ▪ Charm++ by Laxmikant (Sanjay) Kale et al. ▪ LLVM byVikram Adve, Chris Lattner, et al. “The openness of the LLVM technology and the quality of its architecture and engineering design are key factors in understanding the success it has had both in academia and industry.” KLEE? JPF?FindBugs?Shipshape? Soot? WALA? …
  • 7.  50 years of automated debugging research  N papers  only 5 evaluated with actual programmers “ ” [Parnin&Orso ISSTA’11] http://dl.acm.org/citation.cfm?id=2001445
  • 8.  Human  Expensive, incomplete, …  Brute Force  Pairwise, predefined data, etc…  Tool Automation!!
  • 9. Running Symbolic PathFinder ... … ============================================ ========== results no errors detected ============================================ ========== statistics elapsed time: 0:00:02 states: new=4, visited=0, backtracked=4, end=2 search: maxDepth=3, constraints=0 choice generators: thread=1, data=2 heap: gc=3, new=271, free=22 instructions: 2875 max memory: 81MB loaded code: classes=71, methods=884 … 9
  • 10.  Pex (released on May 2008)  Shipped with Visual Studio 15 as IntelliTest  30,388 download# (20 months, Feb 08-Oct 09)  22,466 download# (10 months, Apr 13-Jan 14): Code Digger  Active user community: 1,436 forum posts during ~3 years (Oct 08- Nov 11)  Moles (released on Sept 2009)  Shipped with Visual Studio 12 as Fakes  “Provide Microsoft Fakes w/ allVisual Studio editions” got 1,457 community votes
  • 12. 1,753,594 clicked 'Ask Pex!' http://pex4fun.com/
  • 14. Code Hunt can identify top coders http://programming2015.cstnet.cn/
  • 15. Secret Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } Player Implementation class Player { public static int Puzzle(int x) { return x; } } classTest { public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); } } behavior Secret Impl == Player Impl 15
  • 16. “It really got me *excited*.The part that got me most is about spreading interest in teaching CS: I do think that it’s REALLY great for teaching | learning!” “I used to love the first person shooters and the satisfaction of blowing away a whole team of Noobies playing Rainbow Six, but this is far more fun.” “I’m afraid I’ll have to constrain myself to spend just an hour or so a day on this really exciting stuff, as I’m really stuffed with work.” X
  • 17.  NOT Random:  Cheap, Fast  “It passed a thousand tests” feeling  …  But Dynamic Symbolic Execution: e.g., Pex, CUTE,EXE  White box  Constraint Solving
  • 18. Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890 void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123…} a==null a.Length>0 a[0]==123… T TF T F F Execute&MonitorSolve Choose next path Done: There is no path left. Negated condition
  • 19. There are decision procedures for individual path conditions, but…  Number of potential paths grows exponentially with number of branches  Reachable code not known initially  Without guidance, same loop might be unfolded forever Fitnex search strategy [Xie et al. DSN 09] http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf
  • 20. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } TestLoop(0, {0}) Path condition: !(x == 90) ↓ New path condition: (x == 90) ↓ New test input: TestLoop(90, {0})
  • 21. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } TestLoop(90, {0}) Path condition: (x == 90) && !(y[0] ==15) && !(x == 110) ↓ New path condition: (x == 90) && (y[0] ==15) ↓ New test input: TestLoop(90, {15})
  • 22. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } TestLoop(90, {15}) Path condition: (x == 90) && (y[0] ==15) && !(x+1 == 110) ↓ New path condition: (x == 90) && (y[0] ==15) && (x+1 == 110) ↓ New test input: No solution!?
  • 23. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } TestLoop(90, {15}) Path condition: (x == 90) && (y[0] ==15) && (0 < y.Length) && !(1 < y.Length) && !(x+1 == 110) ↓ New path condition: (x == 90) && (y[0] ==15) && (0 < y.Length) && (1 < y.Length)  Expand array size
  • 24. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } TestLoop(90, {15}) We can have infinite paths! Manual analysis  need at least 20 loop iterations to cover the target branch Exploring all paths up to 20 loop iterations is infeasible: 220 paths
  • 25. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Key observations: with respect to the coverage target  not all paths are equally promising for branch-node flipping  not all branch nodes are equally promising to flip • Our solution: – Prefer to flip branch nodes on the most promising paths – Prefer to flip the most promising branch nodes on paths – Fitness function to measure “promising” extents TestLoop(90, {15, 0}) TestLoop(90, {15, 15}) [Xie et al. DSN 2009] http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf
  • 26.  FF computes fitness value (distance between the current state and the goal state)  Search tries to minimize fitness value [Tracey et al. 98, Liu at al. 05, …]
  • 27. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Fitness function: |110 – x |
  • 28. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } (90, {0}) 20 (90, {15}) 19 (90, {15, 0}) 19 (90, {15, 15}) 18 (90, {15, 15, 0}) 18 (90, {15, 15, 15}) 17 (90, {15, 15, 15, 0}) 17 (90, {15, 15, 15, 15}) 16 (90, {15, 15, 15, 15, 0}) 16 (90, {15, 15, 15, 15, 15}) 15 … FitnessValue(x, y) Fitness function: |110 – x | Give preference to flip paths with better fitness values We still need to address which branch node to flip on paths …
  • 29. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } (90, {0}) 20 (90, {15})  flip b4 19 (90, {15, 0})  flip b2 19 (90, {15, 15})  flip b4 18 (90, {15, 15, 0})  flip b2 18 (90, {15, 15, 15})  flip b4 17 (90, {15, 15, 15, 0})  flip b2 17 (90, {15, 15, 15, 15})  flip b4 16 (90, {15, 15, 15, 15, 0})  flip b2 16 (90, {15, 15, 15, 15, 15})  flip b4 15 … FitnessValue(x, y) Fitness function: |110 – x | Branch b1: i < y.Length Branch b2: i >= y.Length Branch b3: y[i] == 15 Branch b4: y[i] != 15 •Flipping Branch b4 (b3) gives us average 1 (-1) fitness gain (loss) •Flipping branch b2 (b1) gives us average 0 fitness gain (loss)
  • 30.  For a flipped node leading to Fnew, find out the old fitness value Fold before flipping • Assign Fitness Gain (Fold – Fnew) for the branch of the flipped node • Assign Fitness Gain (Fnew – Fold ) for the other branch of the branch of the flipped node  Compute the average fitness gain for each branch over time
  • 31.  Each branch node candidate for being flipped is prioritized based on its composite fitness value: • (Fitness value of node – Fitness gain of its branch)  Select first the one with the best composite fitness value
  • 32.  Pex (released on May 2008)  Shipped with Visual Studio 15 as IntelliTest  30,388 download# (20 months, Feb 08-Oct 09)  22,466 download# (10 months, Apr 13-Jan 14): Code Digger  Active user community: 1,436 forum posts during ~3 years (Oct 08- Nov 11)  Moles (released on Sept 2009)  Shipped with Visual Studio 12 as Fakes  “Provide Microsoft Fakes w/ allVisual Studio editions” got 1,457 community votes How to make such successful case????
  • 33. voidTestAdd(ArrayList a, object o) { Assume.IsTrue(a!=null); int i = a.Count; a.Add(o); Assert.IsTrue(a[i] == o); } Parameterized UnitTests Supported by Pex Moles/Fakes Code Digger Pex4Fun/Code Hunt  Surrounding (Moles/Fakes)  Simplifying (Code Digger)  Retargeting (Pex4Fun/Code Hunt)
  • 34.  Developer/manager: “Who is using your tool?”  Pex team: “Do you want to be the first?”  Developer/manager: “I love your tool but no.” Tool Adoption by (Mass)Target Users Tool Shipping withVisual Studio Macro Perspective Micro Perspective
  • 35.  Developer: “Code digger generates a lot of “0” strings as input. I can’t find a way to create such a string via my own C# code. Could any one show me a C# snippet? I meant zero terminated string.”  Pex team: “In C#, a 0 in a string does not mean zero- termination. It’s just yet another character in the string (a very simple character where all bits are zero), and you can create as Pex shows the value: “0”.”  Developer: “Your tool generated “0””  Pex team: “What did you expect?”  Developer: “Marc.”
  • 36.  Developer: “Your tool generated a test called Foo001. I don’t like it.”  Pex team: “What did you expect?”  Developer:“Foo_Should_Fail_When_Bar_Is_Negative.”
  • 37. Object Creation messages suppressed (related to Covana by Xiao et al. [ICSE’11]) ExceptionTreeView Exploration TreeView Exploration ResultsView http://taoxie.cs.illinois.edu/publications/icse11-covana.pdf
  • 38. public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false; } Key observations: with respect to the coverage target  not all paths are equally promising for branch-node flipping  not all branch nodes are equally promising to flip • Our solution: – Prefer to flip branch nodes on the most promising paths – Prefer to flip the most promising branch nodes on paths – Fitness function to measure “promising” extents Fitnex by Xie et al. [DSN’09] To avoid local optimal or biases, the fitness-guided strategy is integrated with Pex’s fairness search strategies http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf
  • 39.  “Simply one mouse click and then everything would work just perfectly”  Often need environment isolation w/ Moles/Fakes or factory methods, …  “One mouse click, a test generation tool would detect all or most kinds of faults in the code under test”  Developer: “Your tool only finds null references.”  Pex team: “Did you write any assertions?”  Developer: “Assertion???”  “I do not need test generation; I already practice unit testing (and/orTDD).Test generation does not fit into the TDD process”
  • 40. Gathered feedback from target tool users  Directly, e.g., via  MSDN Pex forum, tech support, outreach to MS engineers and .NET user groups  Indirectly, e.g., via  interactions with MSVisual Studio team (a tool vendor to its huge user base)  Motivations of Moles  Refactoring testability issue faced resistance in practice  Observation at Agile 2008: high attention on mock objects and tool supports
  • 41.  Win-win collaboration model  Win (Ind Lab): longer-term research innovation, man power, research impacts, …  Win (Univ): powerful infrastructure, relevant/important problems in practice, both research and industry impacts, …  Industry-located Collaborations  Faculty visits, e.g., Fitnex, Pex4Fun  Student internships, e.g., FloPSy, DyGen, state cov  Academia-locatedCollaborations
  • 42. Academia-located Collaborations  Immediate indirect impacts, e.g.,  Reggae [ASE’09s]  Rex  MSeqGen [FSE’09]  DyGen  Guided Cov [ICSM’10]  state coverage  Long-term indirect impacts, e.g.,  DySy by Csallner et al. [ICSE’08]  Seeker [OOPSLA’11]  Covana [ICSE’11]
  • 43.  Pex  practice impacts  Moles/Fakes, Code Digger, Pex4Fun/Code Hunt  Lessons in transferring tools  Started as (Evolved) Dream  Chicken and Egg  Human Factors  Best vs.Worst Cases  Tool Users’ Stereotypical Mindset or Habits  Practitioners’Voice  Collaboration w/ Academia
  • 44.  Start a startup  but desirable to have right people (e.g., former students) to start  Release free tools/libraries to aim for adoption  but a lot of efforts to be invested on “non-researchy” stuffs  Collaborate with industrial research labs  but many research lab projects may look like univ. projects  Collaborate with industrial product groups  but many probs faced by product groups may not be “researchy”
  • 45.  NikolaiTillmann, Jonathan de Halleux, and Tao Xie. Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger. In Proceedings of ASE 2014, Experience Papers. http://taoxie.cs.illinois.edu/publications/ase14-pexexperiences.pdf  Jian-Guang Lou, Qingwei Lin, Rui Ding, Qiang Fu, Dongmei Zhang, and Tao Xie. Software Analytics for Incident Management of Online Services: An Experience Report. In Proceedings ASE 2013, Experience Paper. http://taoxie.cs.illinois.edu/publications/ase13-sas.pdf  Dongmei Zhang, Shi Han,Yingnong Dang, Jian-Guang Lou, Haidong Zhang, andTao Xie. Software Analytics in Practice. IEEE Software, Special Issue on the Many Faces of Software Analytics, 2013. http://taoxie.cs.illinois.edu/publications/ieeesoft13-softanalytics.pdf  Yingnong Dang, Dongmei Zhang, Song Ge, Chengyun Chu,Yingjun Qiu, and Tao Xie. XIAO:Tuning Code Clones at Hands of Engineers in Practice. In Proceedings of ACSAC 2012. http://taoxie.cs.illinois.edu/publications/acsac12-xiao.pdf
  • 47.  Start a startup  but desirable to have right people (e.g., former students) to start  Release free tools/libraries to aim for adoption  but a lot of efforts to be invested on “non-researchy” stuffs  Collaborate with industrial research labs  but many research lab projects may look like univ. projects  Collaborate with industrial product groups  but many probs faced by product groups may not be “researchy”