Advances in Unit Testing: Theory and Practice

Tao Xie1 NikolaiTillmann2 Pratap Lakshman2
1 University of Illinois at Urbana-Champaign
2 MicrosoftTools for Software Engineers/Developer Division
Materials: http://taoxie.cs.illinois.edu/courses/testing/

2
A unit test is a small program with assertions.
[TestMethod] public void Add(HashSet s,int x, int y)
{
Assume.IsTrue(!s.IsMember(x));
int oldCount = s.GetSize();
s.Add(x);
Assert.AreEqual(set.Count, oldCount + 1);
}
Many developers write such unit tests by hand.This involves
 determining a meaningful sequence of method calls,
 selecting exemplary argument values (the test inputs),
 stating assertions.

 Design and specification
 by example
 Code coverage and regression testing
 confidence in correctness
 preserving behavior
 Short feedback loop
 unit tests exercise little code
 failures are easy to debug
 Documentation

 Coverage: Are all parts of the program exercised?
 statements
 basic blocks
 explicit/implicit branches
 …
 Assertions: Does the program do the right thing?
 test oracle
Experience:
 Just high coverage or large number of assertions is
no good quality indicator.
 Only both together are!

 Concrete, easy to understand
 Don’t need new language
 Easy to see if program meets the spec
 Making tests forces you to talk to customer
and learn the problem
 Making tests forces you to think about design
of system (classes, methods, etc.)
5

 Too specific
 Hard to test that something can’t happen
 Can’t withdraw more money than you have in the
system
 Can’t break into the system
 Can’t cause a very long transaction that hangs the
system
 Tends to be verbose
6

Parameterized UnitTest
 A parameterized unit test is a small program that
takes some inputs and states assumptions and
assertions.
7

Parameterized UnitTests
 serve as specifications
 can be leveraged by (automatic) test input generators
 fit in development environment, evolve with the code

Hand-written
9
// FooTest.cs
[TestClass, PexClass]
partial class FooTest
{
[PexMethod]
voidTest(Foo foo) {…}
// FooTest.Test.cs
partial class FooTest
{
[TestMethod]
void Test_1()
{ this.Test(new Foo(1)); }
[TestMethod]
void Test_1()
{ this.Test(new Foo(2)); }
…
}
Pex
• User writes parameterized tests
• Lives inside a test class • Generated unit tests
• Pex not required for re-execution
• xUnit unit tests
xUnit AttributesPex Attributes
Partial Class
Generated
http://msdn.microsoft.com/en-us/library/wa80x488(VS.80).aspx

PUTs separate two concerns:
(1) The specification of external behavior
(i.e., assertions)
(2) The selection of internal test inputs
(i.e., coverage)
In many cases, a test generation tool (e.g., Pex)
can construct a small test suite with high
coverage !

A PUT can be read as a universally quantified,
conditional axiom.
 int name, int data.
name ≠ null ⋀ data ≠ null ⇒
equals(
ReadResource(name,
WriteResource(name, data)),
data)

12
import org.junit.*;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;
import static org.junit.Assert.*;
import java.util.*;
@RunWith(Parameterized.class)
public class ParamTest {
public int sum, a, b;
public ParamTest (int sum, int a, int b) { this.sum = sum; this.a = a; this.b = b; }
@Parameters public static Collection<Object[]> parameters() {
return Arrays.asList (new Object [][] {{0, 0, 0}, {2, 1, 1}});
}
@Test public void additionTest() { assertEquals(sum, a+b); }
}

13
 These Are UnitTests With Actual Parameters
 So Far,We’ve Only Seen ParameterlessTest Methods
 Contract Model: Assume, Act, Assert
 Assumptions (Preconditions) LimitValues Appropriately
 Action Performs Activity Under Scrutiny
 Assertions (Postconditions)Check Result
@Theory public void removeThenAddDoesNotChangeSet(
Set<String> set, String string) { // Parameters!
assumeTrue(set.contains(string)) ; // Assume
Set<String> copy = new HashSet<String>(set); // Act
copy.remove(string);
copy.add(string);
assertTrue (set.equals(copy)); // Assert //
// System.out.println(“Instantiated test: “ + set + “, “ + string);
}

14
Answer:
 All Combinations ofValues from @DataPoint Annotations
Where Assume Clause isTrue
 Four (of Nine) Combinations inThis Particular Case
 Note: @DataPoint Format is an Array.
@DataPoints
public static String[] string = {"ant", "bat", "cat"};
@DataPoints
public static Set[] sets = {
new HashSet(Arrays.asList("ant", "bat")),
new HashSet(Arrays.asList(“bat", “cat", “dog“, “elk”)),
new HashSet(Arrays.asList(“Snap”, “Crackle”, “Pop"))
};

15
import org.junit.*;
import org.junit.runner.RunWith;
import static org.junit.Assert.*;
import static org.junit.Assume.*;
import org.junit.experimental.theories.DataPoint;
import org.junit.experimental.theories.DataPoints;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.Theory;
import java.util.*;
@RunWith(Theories.class)
public class SetTheoryTest {
… // See Earlier Slides
}

 Human
 Expensive, incomplete, …
 Brute Force
 Pairwise, predefined data, etc…
 Tool Automation!!

Running Symbolic PathFinder ...
…
============================================
========== results
no errors detected
============================================
========== statistics
elapsed time: 0:00:02
states: new=4, visited=0,
backtracked=4, end=2
search: maxDepth=3,
constraints=0
choice generators: thread=1, data=2
heap: gc=3, new=271, free=22
instructions: 2875
max memory: 81MB
loaded code: classes=71, methods=884
…
17

http://research.microsoft.com/pex/

 As new feature of “IntelliTest”
https://www.visualstudio.com/news/vs2015-vs#Testing

1,767,012 clicked 'Ask Pex!'
http://pex4fun.com/default.aspx?language=CSharp&sample=_Template

130,000 users
as of March 2015
after 1 year launch
https://www.codehunt.com/

 NOT Random:
 Cheap, Fast
 “It passed a thousand tests” feeling
 …
 But Dynamic Symbolic Execution:
e.g., Pex, CUTE,EXE
 White box
 Constraint Solving

http://research.microsoft.com/pex/
Dynamic Symbolic Execution (DSE)
aka. ConcolicTesting
[Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08]
Instrument code to explore feasible paths

Code to generate inputs for:
Constraints to solve
a!=null
a!=null &&
a.Length>0
a!=null &&
a.Length>0 &&
a[0]==1234567890
void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
Observed constraints
a==null
a!=null &&
!(a.Length>0)
a!=null &&
a.Length>0 &&
a[0]!=1234567890
a!=null &&
a.Length>0 &&
a[0]==1234567890
Data
null
{}
{0}
{123…}
a==null
a.Length>0
a[0]==123…
T
TF
T
F
F
Execute&MonitorSolve
Choose next path
Done: There is no path left.
Negated condition

There are decision procedures for individual path
conditions, but…
 Number of potential paths grows exponentially with
number of branches
 Reachable code not known initially
 Without guidance, same loop might be unfolded
forever
Fitnex search strategy
[Xie et al. DSN 09]
http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf

 Pex (released on May 2008)
 Shipped with Visual Studio 15 as IntelliTest
 30,388 download# (20 months, Feb 08-Oct 09)
 22,466 download# (10 months, Apr 13-Jan 14): Code Digger
 Active user community: 1,436 forum posts during ~3
years (Oct 08- Nov 11)
 Moles (released on Sept 2009)
 Shipped with Visual Studio 12 as Fakes
 “Provide Microsoft Fakes w/ allVisual Studio editions” got
1,457 community votes
A Journey of Bringing
Automated UnitTest
Generation to Practice

voidTestAdd(ArrayList a, object o) {
Assume.IsTrue(a!=null);
int i = a.Count;
a.Add(o);
Assert.IsTrue(a[i] == o);
}
Parameterized UnitTests Supported by Pex
Moles/Fakes
Code Digger
Pex4Fun/Code Hunt
 Surrounding (Moles/Fakes)
 Simplifying (Code Digger)
 Retargeting (Pex4Fun/Code Hunt)

 Developer/manager: “Who is using your tool?”
 Pex team: “Do you want to be the first?”
 Developer/manager: “I love your tool but no.”
Tool Adoption by (Mass)Target Users
Tool Shipping withVisual Studio
Macro Perspective
Micro Perspective

 Developer: “Code digger generates a lot of “0” strings
as input. I can’t find a way to create such a string via my
own C# code. Could any one show me a C# snippet? I
meant zero terminated string.”
 Pex team: “In C#, a 0 in a string does not mean zero-
termination. It’s just yet another character in the string
(a very simple character where all bits are zero), and
you can create as Pex shows the value: “0”.”
 Developer: “Your tool generated “0””
 Pex team: “What did you expect?”
 Developer: “Marc.”

 Developer: “Your tool generated a test called Foo001. I
don’t like it.”
 Pex team: “What did you expect?”
 Developer:“Foo_Should_Fail_When_Bar_Is_Negative.”

Object Creation messages suppressed
(related to Covana by Xiao et al. [ICSE’11])
ExceptionTreeView
Exploration TreeView
Exploration ResultsView
http://taoxie.cs.illinois.edu/publications/icse11-covana.pdf

public bool TestLoop(int x, int[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
Key observations: with respect to the
coverage target
 not all paths are equally promising for
branch-node flipping
 not all branch nodes are equally
promising to flip
• Our solution:
– Prefer to flip branch nodes on the most promising paths
– Prefer to flip the most promising branch nodes on paths
– Fitness function to measure “promising” extents
Fitnex by Xie et al. [DSN’09]
To avoid local optimal or biases, the
fitness-guided strategy is integrated
with Pex’s fairness search strategies
http://taoxie.cs.illinois.edu/publications/dsn09-fitnex.pdf

 “Simply one mouse click and then everything would work
just perfectly”
 Often need environment isolation w/ Moles/Fakes or factory
methods, …
 “One mouse click, a test generation tool would detect all
or most kinds of faults in the code under test”
 Developer: “Your tool only finds null references.”
 Pex team: “Did you write any assertions?”
 Developer: “Assertion???”
 “I do not need test generation; I already practice unit
testing (and/orTDD).Test generation does not fit into the
TDD process”

Gathered feedback from target tool users
 Directly, e.g., via
 MSDN Pex forum, tech support, outreach to MS engineers and
.NET user groups
 Indirectly, e.g., via
 interactions with MSVisual Studio team (a tool vendor to its huge
user base)
 Motivations of Moles
 Refactoring testability issue faced resistance in practice
 Observation at Agile 2008: high attention on mock
objects and tool supports

 Win-win collaboration model
 Win (Ind Lab): longer-term research innovation, man power,
research impacts, …
 Win (Univ): powerful infrastructure, relevant/important
problems in practice, both research and industry impacts, …
 Industry-located Collaborations
 Faculty visits, e.g., Fitnex, Pex4Fun
 Student internships, e.g., FloPSy, DyGen, state cov
 Academia-locatedCollaborations

Academia-located Collaborations
 Immediate indirect impacts, e.g.,
 Reggae [ASE’09s]  Rex
 MSeqGen [FSE’09]  DyGen
 Guided Cov [ICSM’10]  state coverage
 Long-term indirect impacts, e.g.,
 DySy by Csallner et al. [ICSE’08]
 Seeker [OOPSLA’11]
 Covana [ICSE’11]

 Pex  practice impacts
 Moles/Fakes, Code Digger, Pex4Fun/Code Hunt
 Lessons in transferring tools
 Started as (Evolved) Dream
 Chicken and Egg
 Human Factors
 Best vs.Worst Cases
 Tool Users’ Stereotypical Mindset or Habits
 Practitioners’Voice
 Collaboration w/ Academia

 NikolaiTillmann, Jonathan de Halleux, and Tao Xie. Transferring an Automated Test Generation
Tool to Practice: From Pex to Fakes and Code Digger. In Proceedings of ASE 2014, Experience
Papers. http://taoxie.cs.illinois.edu/publications/ase14-pexexperiences.pdf
 Jian-Guang Lou, Qingwei Lin, Rui Ding, Qiang Fu, Dongmei Zhang, and Tao Xie. Software Analytics
for Incident Management of Online Services: An Experience Report. In Proceedings ASE 2013,
Experience Paper. http://taoxie.cs.illinois.edu/publications/ase13-sas.pdf
 Dongmei Zhang, Shi Han,Yingnong Dang, Jian-Guang Lou, Haidong Zhang, andTao Xie. Software
Analytics in Practice. IEEE Software, Special Issue on the Many Faces of Software Analytics, 2013.
http://taoxie.cs.illinois.edu/publications/ieeesoft13-softanalytics.pdf
 Yingnong Dang, Dongmei Zhang, Song Ge, Chengyun Chu,Yingjun Qiu, and Tao Xie. XIAO:Tuning
Code Clones at Hands of Engineers in Practice. In Proceedings of ACSAC 2012.
http://taoxie.cs.illinois.edu/publications/acsac12-xiao.pdf

using System;
using Microsoft.Pex.Framework;
using Microsoft.Pex.Framework.Settings;
[PexClass]
public class Set {
[PexMethod]
public static void testMemberAfterInsertNotEqual(Set s, int i, int j) {
PexAssume.IsTrue(s != null);
PexAssume.IsTrue(i != j);
bool exist = s.member(i);
s.insert(j);
PexAssert.IsTrue(exist);
}
….
}
39

Class IntSet {
public IntSet() {…};
public void insert(int e) { … }
public Bool member(int e) { … }
public void remove(int e) { … }
}
sort IntSet imports Int, Bool
signatures
new : -> IntSet
insert : IntSet × Int -> IntSet
member : IntSet × Int -> Bool
remove : IntSet × Int -> IntSet
40http://www.cs.unc.edu/~stotts/723/adt.html

Class IntSet {
public IntSet() {…};
public void insert(int e) { … }
public Bool member(int e) { … }
public void remove(int e) { … }
}
See the Set.cs that can be downloaded from
http://taoxie.cs.illinois.edu/courses/testing/Set.cs
Let’s copy it to
http://pex4fun.com/default.aspx?language=CSharp&sample=_Template
And Click “Ask Pex”
41

using System;
[PexClass]
public class Set {
[PexMethod]
bool existOld = s.member(i);
s.insert(j);
PexAssert.IsTrue(existOld == exist);
}
….
}
42
Pex4Fun supports only one PexMethod at a time;
you can write multiple PexMethods but comment out
other lines of “[PexMethod]” except one

variables i, j : Int; s : IntSet
Axioms:
member(new(), i) = false
member(insert(s, j), i) =
if i = j then true
else member(s, i)
43
http://www.cs.unc.edu/~stotts/723/adt.html
Is this complete?
How do we know?

 Classify methods:
 constructors: return IntSet
 inspectors: take IntSet as argument, returning some
other value.
 Identify key constructors, capable of constructing
all possible object states
 e.g., insert, new.
 Identify others as auxiliary,
 e.g., remove is a destructive constructor
 Completeness requires (at least):
 every inspector/auxiliary constructor is defined by one
equation for each key constructor.
44

 remove(new(), i) = new()
 remove(insert(s, j), i) =
if i = j then remove(s, i)
else insert(remove(s, i), j)
45
Are we done yet?
The completeness criterion (an equation defining member
and remove for each of the new and insert constructors)
is satisfied.

 But does this really specify sets? Do the
following properties hold?
 Order of insertion is irrelevant.
 insert(insert(s, i), j) = insert(insert(s, j), i)
 Multiple insertion is irrelevant.
 insert(insert(s, i), i) = insert(s, i)
46

Class UIntStack {
public UIntStack() {…};
public void Push(int k) { … }
public void Pop() { … }
public intTop() { … }
public bool IsEmpty() { … }
public int MaxSize() { … }
public bool IsMember(int k) { … }
public bool Equals(UIntStack s) { … }
public int GetNumberOfElements() { … }
public bool IsFull() { … }
}
47
See the UIntStack.cs that can be downloaded from
http://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs

Class UIntStack {
public UIntStack() {…};
public void Push(int k) { … }
public void Pop() { … }
public intTop() { … }
public bool IsEmpty() { … }
public int MaxSize() { … }
public bool IsMember(int k) { … }
public bool Equals(UIntStack s) { … }
public int GetNumberOfElements() { … }
public bool IsFull() { … }
}
48
Let’s copy it to
http://pex4fun.com/default.aspx?language=C
Sharp&sample=_Template
And Click “Ask Pex”
Reminder: you have to
comment earlier written
“[PexMethod]” before you
try Pex on your current
PUT (Pex4Fun can handle
only one PUT at a time)
See the UIntStack.cs that can be downloaded from
http://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs

using System;
[PexClass]
public class Set {
[PexMethod]
bool existOld = s.member(i);
s.insert(j);
PexAssert.IsTrue(existOld == exist);
}
….
}
49

using System;
[PexClass]
public class Set {
[PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)]
s.insert(j);
PexAssert.IsTrue(exist);
}
….
}
50

In class, we show the factory method as below automatically synthesized
by Pex after a user clicks “1 Object Creation” issue and then click
“Accept/Edit Factory Method”. But it is not good enough to generate
various types of object states.
[PexFactoryMethod(typeof(UIntStack))]
public static UIntStack Create(int k_i)
{
UIntStack uIntStack = new UIntStack();
uIntStack.Push(k_i);
return uIntStack;
//TODO: Edit factory method of UIntStack
//This method should be able to configure the object in all possible
ways.
// Add as many parameters as needed,
// and assign their values to each field by using the API.
}
51

Below is a manually edited/created good factory method to guide
Pex to generate various types of object states. Note that Pex also
generates argument values for the factory method.
[PexFactoryMethod(typeof(UIntStack))]
public static UIntStackCreateVariedSizeAnyElemsStack(int[] elems)
{
PexAssume.IsNotNull(elems);
UIntStack s = new UIntStack();
PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1));
for (int i = 0; i < elems.Length; i++)
s.Push(elems[i]);
return s;
}
52

Below is a manually edited/created good factory method to guide
Pex to generate various types of object states. Note that Pex also
generates argument values for the factory method.
[PexMethod]
public voidTestPush([PexAssumeUnderTest]UIntStack s, int i)
{
//UIntStack s = new UIntStack();
PexAssume.IsTrue(!s.IsMember(i));
int oldCount = s.GetNumberOfElements();
s.Push(i);
PexAssert.IsTrue(s.Top() == i);
PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1);
PexAssert.IsFalse(s.IsEmpty());
}
53

If you try PUTs on Pex4Fun, which doesn’t support factory method, you
can “embed” the factory method like the highlighted code portion below
[PexMethod]
public voidTestPush(int[] elems, int i)
{
PexAssume.IsNotNull(elems);
UIntStack s = new UIntStack();
PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1));
for (int i = 0; i < elems.Length; i++)
s.Push(elems[i]);
//UIntStack s = new UIntStack();
PexAssume.IsTrue(!s.IsMember(i));
int oldCount = s.GetNumberOfElements();
s.Push(i);
PexAssert.IsTrue(s.Top() == i);
PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1);
PexAssert.IsFalse(s.IsEmpty());
}
54

• Setup: basic set up for invoking the method
under test
• Checkpoint: Run Pex to make sure that you
don't miss any Pex assumptions (preconditions)
for the PUT
• Assert: add assertions for asserting
behavior of the method under test, involving
• Adding Pex assertions
• Adding Pex assumptions for helping assert
• Adding method sequences for helping assert

• Select your method under test m
• Put its method call in your PUT
• Create a parameter for your PUT as the
class under test c (annotated it with
[PexAssumeUnderTest])
• Create other parameters for your PUT for
parameters of m if any
• Add Pex assumptions for preconditions for
all these parameters of PUT if any

[PexMethod]
public void TestPush([PexAssumeUnderTest]UIntStack s, int i) {
s.Push(i);
}
You may write your factory method to help Pex in test
generation
If you get exceptions thrown
• if indicating program faults, fix them
• If indicating lack of PUT assumptions, add PUT assumptions
• If indicating insufficient factory method assumptions or
inappropriate scenarios, add PUT assumptions or improve
factory method.

• Think about how you can assert the
behavior
• Do you need to invoke other (observer) helper
methods in your assertions (besides asserting
return values)?
• Do you need to add assumptions so that your
assertions can be valid?
• Do you need to add some method sequence
before the method under test to set up
desirable state and cache values to be used in
the assertions?

• Return value of the method under test (MUT)
• Argument object of MUT
• Receiver object properties being modified by
MUT (if public fields, directly assertable)
• How to assert them?
• Think about the intended behavior!
• If you couldn't do so easily, follow the guidelines
discussed next

• A property value before invoking MUT
may need to be cached and later used.
Pattern 2.1/2.2: Assume, Arrange, Act, Assert
[PexMethod]
void AssumeActAssert(ArrayList list, object item) {
PexAssume.IsNotNull(list); // assume
var count = list.Count; // arrange
list.Add(item); // act
Assert.IsTrue(list.Count == count + 1); // assert
}

• Argument value of MUT may be used
Pattern 2.3:Constructor Test
[PexMethod]
void Constructor(int capacity) {
var list = new ArrayList(capacity); // create
AssertInvariant(list); // assert invariant
Assert.AreEqual(capacity, list.Capacity); // assert
}

• Receiver or argument value of a
method before invoking MUT
Pattern 2.4/5:Roundtrip
[PexMethod]
void ToStringParseRoundtrip(int value) {
// two-way roundtrip
string s = value.ToString();
int parsed = int.Parse(s);
// assert
Assert.AreEqual(value, parsed);
}
value  s  parsed

• Invoking observer methods on the
modified object state
Pattern 2.6: State Relation
[PexMethod]
void InsertContains(string value) {
var list = new List<string>();
list.Add(value);
Assert.IsTrue(list.Contains(value));
}
Each modified object property should be
read by at least one observer method.

• Forcing observer methods to return
specific values (e.g., true or false) can
force you to add specific assumptions
or scenarios
[PexMethod]
void PushIsFull([PexAssumeUnderTest]UIntStack s, int value) {
PexAssume.IsTrue(s.GetSize() == (s.GetMaxSize()-1));
s.Push (value);
Assert.IsTrue(s.IsFull ());
}

• Invoking another method/method
sequence to produce a value to be
used
Pattern 2.7: Commutative Diagram
[PexMethod]
void CommutativeDiagram1(int x, int y) {
// compute result in one way
string z1 = Multiply(x, y).ToString();
// compute result in another way
string z2 = Multiply(x.ToString(), y.ToString());
// assert equality if we get here
PexAssert.AreEqual(z1, z2);
}

• Split possible outcomes into cases (each
with pre and post condition)
Pattern 2.8: Cases
[PexMethod]
void BusinessRules(int age, Job job) {
var salary = SalaryManager.ComputeSalary(age, job);
PexAssert
.Case(age < 30)
.Implies(() => salary < 10000)
.Case(job == Job.Manager && age > 35)
.Implies(() => salary > 10000)
.Case(job == Job.Manager && age < 20)
.Implies(() => false); }

• If class invariant checker (repOk) exists
or you would be willing to write one,
use it to assert
Pattern 2.3:Constructor Test
[PexMethod]
void Constructor(int capacity) {
var list = new ArrayList(capacity); // create
AssertInvariant(list); // assert invariant
Assert.AreEqual(capacity, list.Capacity); // assert
}

• Pattern 2.9: Allowed exceptions
• [PexAllowedException(typeof(ArgumentNullException))]
• [ExpectedException(typeof(ArgumentNullException))]
• Pattern 2.10: Reachability
• [PexExpectedGoals] + throw new PexGoalException();
• Pattern 2.11: Parameterized Stub
• No scenarios or assertions
• Pattern 2.12: Input OutputTest
• void Add(int a, int b, out int result) { … }
• int Substract(int a, int b) { … }
• Pattern 2.13/14: RegressionTests
• bool Parse(string input) { … }
• PexStore.ValueForValidation("result", result);
http://research.microsoft.com/en-us/projects/pex/patterns.pdf

 Basic Idea:
 Write tests before code
 Refine code with new tests
 In more detail,TDD is a cycle of steps:
 Add a test,
 Run it and watch it fail,
 Change the code as little as possible such that the
test should pass,
 Run the test again and see it succeed,
 Refactor the code if needed.

 TDD encourages writing specifications before
code
 Exemplary specification
 Later, we will generalizeTDD to
ParameterizedTDD
 Axiomatic specifications

Write/refine Contract
as PUT
Write/refine Code
of Implementation
Fix-it (with Pex),
Debug with generated tests
Use GeneratedTests
for Regression
Run Pex
Bug in PUT
Bug in Code
failures
no failures

Pex computes “semantic diff” in cloud
secret reference implementation vs.
code written in browser
You win when Pex finds no differences
For more info, see our ICSE 2013 SEE paper:
http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf

Secret Implementation
class Secret {
public static int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player Implementation
class Player {
public static int Puzzle(int x) {
return x;
}
}
classTest {
public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x))
throw new Exception(“Mismatch”);
}
}
behavior
Secret Impl == Player Impl
75
1,594,0921,594,092

Code Hunt Programming Game
https://www.codehunt.com/

iterative gameplay
adaptive
personalized
no cheating
clear winning criterion
code
test cases

Questions ?
http://research.microsoft.com/pex
taoxie@illinois.edu
Materials: http://taoxie.cs.illinois.edu/courses/testing/

Advances in Unit Testing: Theory and Practice

More Related Content

What's hot

Viewers also liked

Similar to Advances in Unit Testing: Theory and Practice

More from Tao Xie

Recently uploaded

Advances in Unit Testing: Theory and Practice