Automated Developer Testing: Achievements and Challenges
Upcoming SlideShare
Loading in...5
×
 

Automated Developer Testing: Achievements and Challenges

on

  • 505 views

Automated Developer Testing: Achievements and Challenges

Automated Developer Testing: Achievements and Challenges

Statistics

Views

Total Views
505
Views on SlideShare
505
Embed Views
0

Actions

Likes
0
Downloads
16
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Automated Developer Testing: Achievements and Challenges Automated Developer Testing: Achievements and Challenges Presentation Transcript

  • Automated Developer Testing:Achievements and ChallengesTao XieNorth Carolina State Universitycontact: taoxie@gmail.com
  • Automation in Developer Testing• Background on developer testing– http://www.developertesting.com/– Kent Beck’s 2004 talk on “Future of Developer Testing”http://www.itconversations.com/shows/detail301.html• This talk focuses on developer testing– Not system testing etc. conducted by testers• Unit Test Automation commonly referred to writing unittest cases manually, executed automatically• Automation here is broad, including automatic testgeneration 2
  • Software Testing Setup=?Outputs ExpectedOutputsProgram+TestinputsTest Oracles3
  • Software Testing Problems=?Outputs ExpectedOutputsProgram+TestinputsTest Oracles4• Faster: How can tools help developers create and run tests faster?
  • Software Testing Problems=?Outputs ExpectedOutputsProgram+TestinputsTest Oracles5• Faster: How can tools help developers create and run tests faster?• Better Test Inputs: How can tools help generate new better test inputs?
  • Software Testing Problems=?Outputs ExpectedOutputsProgram+TestinputsTest Oracles6• Faster: How can tools help developers create and run tests faster?• Better Test Inputs: How can tools help generate new better test inputs?• Better Test Oracles: How can tools help generate better test oracles?
  • Example Unit Test Case=?Outputs ExpectedOutputsProgram+TestinputsTest Oracles7void addTest() {ArrayList a = new ArrayList(1);Object o = new Object();a.add(o);AssertTrue(a.get(0) == o);}• Appropriate method sequence• Appropriate primitive argument values• Appropriate assertionsTest Case = Test Input + Test Oracle
  • Levels of Test Oracles• Expected output for an individual test input– In the form of assertions in test code• Properties applicable for multiple test inputs– Crash (uncaught exceptions) or not, related torobustness issues, supported by most tools– Properties in production code: Design by Contract(precondition, postcondition, class invariants)supported by Parasoft Jtest, Google CodePro AnalytiX– Properties in test code: Parameterized unit testssupported by MSR Pex, AgitarOneX. Xiao, S. Thummalapenta, and T. Xie. Advances on Improving Automation in Developer Testing.In Advances in Computers, 2012 http://people.engr.ncsu.edu/txie/publications.htm#ac12-devtest
  • Economics of Test Oracles9• Expected output for an individual test input– Easy to manually verify for one test input– Expensive/infeasible to verify for many test inputs– Limited benefits: only for one test input• Properties applicable for multiple test inputs– Not easy to write (need abstraction skills)– But once written, broad benefits for multiple testinputs
  • Assert behavior of multiple test inputsDesign by Contract• Example tools: Parasoft Jtest, Google CodePro AnalytiX,MSR Code Contracts, MSR Pex• Class invariant: properties being satisfied by an object (ina consistent state) [AgitarOne allows a class invarianthelper method used as test oracles]• Precondition: conditions to be satisfied (on receiverobject and arguments) before a method can be invoked• Postcondition: properties being satisfied (on receiverobject and return) after the method has returned• Other types of specs also existhttp://research.microsoft.com/en-us/projects/contracts/
  • Microsoft Research Code ContractsFeaturespublic virtual int Add(object value){Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 );Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if (count == items.Length) EnsureCapacity(count + 1);items[count] = value;return count++;}- Slide adapted from MSR RiSEhttp://research.microsoft.com/en-us/projects/contracts/
  • Microsoft Research Code ContractsFeatures Language expression syntaxpublic virtual int Add(object value){Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 );Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if (count == items.Length) EnsureCapacity(count + 1);items[count] = value;return count++;}- Slide adapted from MSR RiSEhttp://research.microsoft.com/en-us/projects/contracts/
  • Microsoft Research Code ContractsFeatures Language expression syntax Type checking / IDEpublic virtual int Add(object value){Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 );Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if (count == items.Length) EnsureCapacity(count + 1);items[count] = value;return count++;}- Slide adapted from MSR RiSEhttp://research.microsoft.com/en-us/projects/contracts/
  • Microsoft Research Code ContractsFeatures Language expression syntax Type checking / IDE Declarativepublic virtual int Add(object value){Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 );Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if (count == items.Length) EnsureCapacity(count + 1);items[count] = value;return count++;}- Slide adapted from MSR RiSEhttp://research.microsoft.com/en-us/projects/contracts/
  • Microsoft Research Code ContractsFeatures Language expression syntax Type checking / IDE Declarative Special Encodings Result and Oldpublic virtual int Add(object value){Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 );Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if (count == items.Length) EnsureCapacity(count + 1);items[count] = value;return count++;}- Slide adapted from MSR RiSEhttp://research.microsoft.com/en-us/projects/contracts/
  • Microsoft Research Code Contracts[ContractInvariantMethod]void ObjectInvariant() {Contract.Invariant( items != null );}Features Language expression syntax Type checking / IDE Declarative Special Encodings Result and Oldpublic virtual int Add(object value){Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 );Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if (count == items.Length) EnsureCapacity(count + 1);items[count] = value;return count++;}- Slide adapted from MSR RiSEhttp://research.microsoft.com/en-us/projects/contracts/
  • Parameterized Unit Testingvoid TestAdd(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}• Parameterized Unit Test =Unit Test with Parameters• Separation of concerns– Data is generated by a tool– Developer can focus on functional specification[Tillmann&Schulte ESEC/FSE 05]http://research.microsoft.com/apps/pubs/default.aspx?id=77419
  • Parameterized Unit Tests areFormal SpecificationsAlgebraic Specifications• A Parameterized Unit Test can be read as auniversally quantified, conditional axiom.void TestReadWrite(Res r, string name, string data) {Assume.IsTrue(r!=null & name!=null && data!=null);r.WriteResource(name, data);Assert.AreEqual(r.ReadResource(name), data);}∀ string name, string data, Res r:r ≠ null ⋀ name ≠ null ⋀ data ≠ null ⇒equals(ReadResource(WriteResource(r, name, data).state, name),data)
  • http://research.microsoft.com/pex/Parameterized Unit Tests in Pex
  • Parameterized Unit TestingGetting PopularParameterized Unit Tests (PUTs) commonly supported byvarious test frameworks• .NET: Supported by .NET test frameworks– http://www.mbunit.com/– http://www.nunit.org/– …• Java: Supported by JUnit 4.X– http://www.junit.org/Generating test inputs for PUTs supported by tools• .NET: Supported by Microsoft Research Pex– http://research.microsoft.com/Pex/• Java: Supported by Agitar AgitarOne– http://www.agitar.com/
  • ParameterizedTest-Driven DevelopmentWrite/refine Contractas PUTWrite/refine Codeof ImplementationFix-it (with Pex),Debug with generated testsUse Generated Testsfor RegressionRun PexBug in PUTBug in Codefailuresno failures
  • Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCode- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  • Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCodeSoftwareAgitation- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  • Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCodeSoftwareAgitationObservationson code behavior,plusTest Coverage data- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  • Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCodeSoftwareAgitationObservationson code behavior,plusTest Coverage data- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  • Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCodeSoftwareAgitationObservationson code behavior,plusTest Coverage dataIf an Observationreveals a bug, fix it- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  • Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCodeSoftwareAgitationObservationson code behavior,plusTest Coverage dataIf an Observationreveals a bug, fix itIf it describes desired behavior,click to create a Test Assertion- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  • Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCodeSoftwareAgitationObservationson code behavior,plusTest Coverage dataIf an Observationreveals a bug, fix itIf it describes desired behavior,click to create a Test AssertionCodeCompileReviewAgitate- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  • Software Agitation in AgitarOne18Image from http://www.agitar.com/
  • Automated Test Generation19 Recent advanced technique: DynamicSymbolic Execution/Concolic Testing Instrument code to explore feasible paths Example tool: Pex from Microsoft Research(for .NET programs)P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random testing. In Proc. PLDI2005K. Sen, D. Marinov, and G. Agha. CUTE: a concolic unit testing engine for C. In Proc. ESEC/FSE2005N. Tillmann and J. de Halleux. Pex - White Box Test Generation for .NET. In Proc. TAP 2008
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}InputnullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullInputnullExecute&MonitorObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullConstraints to solvea!=nullInputnullExecute&MonitorChoose next pathObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullConstraints to solvea!=nullInputnull{}Execute&MonitorSolveChoose next pathObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0FTFa==nullConstraints to solvea!=nullInputnull{}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0FTFa==nullConstraints to solvea!=nulla!=null &&a.Length>0Inputnull{}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0Inputnull{}{0}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0Inputnull{}{0}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}{123…}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullTConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}{123…}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullTConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}{123…}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890Done: There is no path left.Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  • Automating Test Generation• Method sequences– MSeqGen/Seeker [Thummalapenta et al. OOSPLA 11, ESEC/FSE 09],Covana [Xiao et al. ICSE 2011], OCAT [Jaygarl et al. ISSTA 10],Evacon [Inkumsah et al. ASE 08], Symclat [dAmorim et al. ASE 06]• Environments e.g., db, file systems, network, …– DBApp Testing [Taneja et al. ESEC/FSE 11], [Pan et al. ASE 11]– CloudApp Testing [Zhang et al. IEEE Soft 12]• Loops– Fitnex [Xie et al. DSN 09]@NCSU ASEhttp://people.engr.ncsu.edu/txie/publications.htm
  • Pex on MSDN DevLabsIncubation Project for Visual StudioDownload counts (20 months)(Feb. 2008 - Oct. 2009 )Academic: 17,366Devlabs: 13,022Total: 30,388http://research.microsoft.com/projects/pex/
  • Open Source Pex extensionshttp://pexase.codeplex.com/Publications: http://research.microsoft.com/en-us/projects/pex/community.aspx#publications
  • Writing Test Oracles Learning Formal Methods!?• Parameterized Unit Test =Unit Test with Parameters• Separation of concerns– Data is generated by a tool– Developer can focus on functional specificationvoid TestAdd(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}
  • Automatic Test Generation Human Assistance to Test Generation?!Running Symbolic PathFinder ...…====================================================== resultsno errors detected====================================================== statisticselapsed time: 0:00:02states: new=4, visited=0,backtracked=4, end=2search: maxDepth=3,constraints=0choice generators: thread=1, data=2heap: gc=3, new=271, free=22instructions: 2875max memory: 81MBloaded code: classes=71, methods=884…25
  • Automatic Test Generation Human Assistance to Test Generation?!Running Symbolic PathFinder ...…====================================================== resultsno errors detected====================================================== statisticselapsed time: 0:00:02states: new=4, visited=0,backtracked=4, end=2search: maxDepth=3,constraints=0choice generators: thread=1, data=2heap: gc=3, new=271, free=22instructions: 2875max memory: 81MBloaded code: classes=71, methods=884…25
  • ChallengesFaced by Test Generation Tools object-creation problems (OCP) - 65% external-method call problems (EMCP) – 27%Total block coverage achieved is 50%, lowest coverage 16%.26 Example: Dynamic Symbolic Execution/Concolic Testing Instrument code to explore feasible paths Challenge: path explosion
  • ChallengesFaced by Test Generation Tools object-creation problems (OCP) - 65% external-method call problems (EMCP) – 27%Total block coverage achieved is 50%, lowest coverage 16%.26 Example: Dynamic Symbolic Execution/Concolic Testing Instrument code to explore feasible paths Challenge: path explosion
  •  A graph example fromQuickGraph library00: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  •  A graph example fromQuickGraph library Includes two classesGraphDFSAlgorithm00: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  •  A graph example fromQuickGraph library Includes two classesGraphDFSAlgorithm GraphAddVertex00: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  •  A graph example fromQuickGraph library Includes two classesGraphDFSAlgorithm GraphAddVertexAddEdge: requiresboth vertices to bein graph00: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  • 5600: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  • 57 Test target: Cover truebranch (B4) of Line 2400: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  • 58 Test target: Cover truebranch (B4) of Line 24 Desired objectstate: graph shouldinclude at least oneedge00: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  • 59 Test target: Cover truebranch (B4) of Line 24 Desired objectstate: graph shouldinclude at least oneedge Target sequence:Graph ag = new Graph();Vertex v1 = new Vertex(0);Vertex v2 = new Vertex(1);ag.AddVertex(v1);ag.AddVertex(v2);ag.AddEdge(v1, v2);DFSAlgorithm algo = newDFSAlgorithm(ag);algo.Compute(v1);00: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  • ChallengesFaced by Test Generation Tools object-creation problems (OCP) - 65% external-method call problems (EMCP) – 27%Total block coverage achieved is 50%, lowest coverage 16%.29 Example: Dynamic Symbolic Execution/Concolic (Pex) Instrument code to explore feasible paths Challenge: path explosion
  • ChallengesFaced by Test Generation Tools object-creation problems (OCP) - 65% external-method call problems (EMCP) – 27%Total block coverage achieved is 50%, lowest coverage 16%.29 Example: Dynamic Symbolic Execution/Concolic (Pex) Instrument code to explore feasible paths Challenge: path explosion
  • Example External-Method CallProblems (EMCP)30123
  • Example External-Method CallProblems (EMCP) Example 1: File.Exists has data dependencieson program input Subsequent branch at Line 1 usingthe return value of File.Exists.30123
  • Example External-Method CallProblems (EMCP) Example 1: File.Exists has data dependencieson program input Subsequent branch at Line 1 usingthe return value of File.Exists. Example 2: Path.GetFullPath has datadependencies on program input Path.GetFullPath throwsexceptions.30123
  • Example External-Method CallProblems (EMCP) Example 1: File.Exists has data dependencieson program input Subsequent branch at Line 1 usingthe return value of File.Exists. Example 2: Path.GetFullPath has datadependencies on program input Path.GetFullPath throwsexceptions. Example 3: String.Format donot cause any problem30123
  • Human Can Help!Object Creation Problems (OCP)Tackle object-creation problems with Factory Methods31
  • Human Can Help!External-Method Call Problems (EMCP)Tackle external-method call problems with Mock Methods orMethod InstrumentationMocking System.IO.File.ReadAllText32
  • State-of-the-Art/PracticeTesting ToolsRunning Symbolic PathFinder ...…====================================================== resultsno errors detected====================================================== statisticselapsed time: 0:00:02states: new=4, visited=0,backtracked=4, end=2search: maxDepth=3,constraints=0choice generators: thread=1, data=2heap: gc=3, new=271, free=22instructions: 2875max memory: 81MBloaded code: classes=71, methods=884…Tools typically don’tcommunicate challenges facedby them to enable cooperationbetween tools and users.We typically don’t teach peoplehow to cooperate with tools.33X. Xiao, T. Xie, N. Tillmann, and J. de Halleux. Precise Identification of Problems for StructuralTest Generation. In Proc. ICSE 2011http://people.engr.ncsu.edu/txie/publications/icse11-covana.pdf
  • Coding Duels1,206,095 clicked Ask Pex!
  • Coding DuelsPex computes “semantic diff” in cloudcode written in browser vs.secret reference implementationYou win when Pex finds no differences
  • Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}class Test {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl36
  • Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}class Test {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl36
  • Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}class Test {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl36
  • Coding DuelsFun and EngagingIterative gameplayAdaptivePersonalizedNo cheatingClear winning criterion
  • Example User Feedback“It really got me *excited*. The part that got me most isabout spreading interest in teaching CS: I do think that it’sREALLY great for teaching | learning!”“I used to love the first person shooters and thesatisfaction of blowing away a whole team ofNoobies playing Rainbow Six, but this is far morefun.”“I’m afraid I’ll have to constrain myself to spend just an houror so a day on this really exciting stuff, as I’m really stuffedwith work.”Released since 2010X
  • Coding Duel Competition@ICSE 2011http://pexforfun.com/icse2011
  • Teaching and Learning
  • Coding Duels for Automatic Grading@Grad Software Engineering Coursehttp://pexforfun.com/gradsofteng
  • Coding Duels for Training Testingpublic static string Puzzle(int[] elems, int capacity, int elem) {if ((maxsize <= 0) || (elems == null) || (elems.Length > (capacity + 1)))return "Assumption Violation!";Stack s= new Stack(capacity);for (int i = 0; i < elems.Length; i++)s.Push(elems[i]);int origSize = s.GetNumOfElements();//Please fill in below test scenario on the s stack//The lines below include assertions to assert the program behaviorPexAssert.IsTrue(s.GetNumOfElements() == origSize + 1);PexAssert.IsTrue(s.Top() == elem); PexAssert.IsTrue(!s.IsEmpty());PexAssert.IsTrue(s.IsMember(elem));return s.GetNumOfElements().ToString() + "; “ + s.Top().ToString() + "; “+ s.IsMember(elem).ToString() + "; " + s.IsEmpty();}Set up a stack with some elementsCache values used in assertions
  • Usage Scenarios of Pex4Fun• Massive Open Online Courses (MOOC): Challenges– Grading, addressed by Pex4Fun– Cheating [Open Challenge]• Course assignments (students/professionals)– E.g., intro programming, software engineering• Student/professional competitions– E.g., coding-duel competition at ICSE 2011• Assessment of testing/programming/problemsolving skills for job applicants– Not just final results of problem solving but also process!
  • More ReadingNikolai Tillmann, Jonathan De Halleux, Tao Xie,Sumit Gulwani and Judith BishopTeaching and Learning Programming andSoftware Engineering via Interactive GamingIn Proceedings of the 35th InternationalConference on Software Engineering (ICSE 2013),Software Engineering Education (SEE), SanFrancisco, CA, May 2013.http://people.engr.ncsu.edu/txie/publications/icse13see-pex4fun.pdf
  • Conclusion• Software testing is important and yet costly; needsautomation• Better Test Inputs: help generate new better testinputs– Generate method arguments– Generate method sequences• Better Test Oracles: help generate better test oracles– Assert behavior of individual test inputs– Assert behavior of multiple test inputs• Software Testing  Educational Gaming– http://www.pexforfun.com/45
  • Example IndustrialDeveloper Testing Tools• Agitar AgitatorOne http://www.agitar.com/• Parasoft Jtest http://www.parasoft.com/• Google CodePro AnalytiX https://developers.google.com/java-dev-tools/codepro/doc/• SilverMark Test Mentor http://www.silvermark.com/• Microsoft Research Pex (for .NET)http://research.microsoft.com/Pex/• Microsoft Research Spec Explorer (for .NET)http://research.microsoft.com/specexplorer/46
  • Trends in Practice• Regression Test Selection/Prioritization• Cloud Computing for Test Execution, e.g.,http://www.skytap.com/• Crowdsourcing for Testing, e.g.,http://www.utest.com/• Mocking Environments– Google: EasyMock– Microsoft VS: Fake/Moleshttp://research.microsoft.com/en-us/projects/pex/• Automatic Test Generation– Microsoft: Pex, SAGE http://research.microsoft.com/en-us/um/people/pg/
  • Q & AThank you!contact: taoxie@gmail.comAcknowledgments: NSF grants CCF-0845272, CCF-0915400, CNS-0958235, CNS-1160603,a Microsoft Research SEIF Award, and a Microsoft Research Award.
  • Automated Combinatorial TestingGoals – reduce testing cost, improve cost-benefit ratioAccomplishments – huge increase in performance,scalability, 200+ users, most major IT firms and othersAlso non-testing applications – modelling andsimulation, genomehttp://csrc.nist.gov/groups/SNS/acts/index.html
  • Failure-triggering Interactions• Additionalstudiesconsistent• > 4,000failure reportsanalyzed• Conclusion:failurestriggered byfew variables
  • NIST ACTS Tool• Covering array generator• Coverage analysis - what is the combinatorial coverage ofexisting test set?• .NET configuration file generator• Fault characterization -ongoingCurrentusershttp://csrc.nist.gov/groups/SNS/acts/documents/comparison-report.htmlapproximately 200 users as ofJuly 2009, in IT, defense, finance,telecom, and many otherindustries
  • Defining a New System
  • Variable Interaction Strength
  • Constraints
  • Covering Array Output