Automated Developer Testing: Achievements and Challenges

770 views

Published on

Automated Developer Testing: Achievements and Challenges

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
770
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
27
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Automated Developer Testing: Achievements and Challenges

  1. 1. Automated Developer Testing:Achievements and ChallengesTao XieNorth Carolina State Universitycontact: taoxie@gmail.com
  2. 2. Automation in Developer Testing• Background on developer testing– http://www.developertesting.com/– Kent Beck’s 2004 talk on “Future of Developer Testing”http://www.itconversations.com/shows/detail301.html• This talk focuses on developer testing– Not system testing etc. conducted by testers• Unit Test Automation commonly referred to writing unittest cases manually, executed automatically• Automation here is broad, including automatic testgeneration 2
  3. 3. Software Testing Setup=?Outputs ExpectedOutputsProgram+TestinputsTest Oracles3
  4. 4. Software Testing Problems=?Outputs ExpectedOutputsProgram+TestinputsTest Oracles4• Faster: How can tools help developers create and run tests faster?
  5. 5. Software Testing Problems=?Outputs ExpectedOutputsProgram+TestinputsTest Oracles5• Faster: How can tools help developers create and run tests faster?• Better Test Inputs: How can tools help generate new better test inputs?
  6. 6. Software Testing Problems=?Outputs ExpectedOutputsProgram+TestinputsTest Oracles6• Faster: How can tools help developers create and run tests faster?• Better Test Inputs: How can tools help generate new better test inputs?• Better Test Oracles: How can tools help generate better test oracles?
  7. 7. Example Unit Test Case=?Outputs ExpectedOutputsProgram+TestinputsTest Oracles7void addTest() {ArrayList a = new ArrayList(1);Object o = new Object();a.add(o);AssertTrue(a.get(0) == o);}• Appropriate method sequence• Appropriate primitive argument values• Appropriate assertionsTest Case = Test Input + Test Oracle
  8. 8. Levels of Test Oracles• Expected output for an individual test input– In the form of assertions in test code• Properties applicable for multiple test inputs– Crash (uncaught exceptions) or not, related torobustness issues, supported by most tools– Properties in production code: Design by Contract(precondition, postcondition, class invariants)supported by Parasoft Jtest, Google CodePro AnalytiX– Properties in test code: Parameterized unit testssupported by MSR Pex, AgitarOneX. Xiao, S. Thummalapenta, and T. Xie. Advances on Improving Automation in Developer Testing.In Advances in Computers, 2012 http://people.engr.ncsu.edu/txie/publications.htm#ac12-devtest
  9. 9. Economics of Test Oracles9• Expected output for an individual test input– Easy to manually verify for one test input– Expensive/infeasible to verify for many test inputs– Limited benefits: only for one test input• Properties applicable for multiple test inputs– Not easy to write (need abstraction skills)– But once written, broad benefits for multiple testinputs
  10. 10. Assert behavior of multiple test inputsDesign by Contract• Example tools: Parasoft Jtest, Google CodePro AnalytiX,MSR Code Contracts, MSR Pex• Class invariant: properties being satisfied by an object (ina consistent state) [AgitarOne allows a class invarianthelper method used as test oracles]• Precondition: conditions to be satisfied (on receiverobject and arguments) before a method can be invoked• Postcondition: properties being satisfied (on receiverobject and return) after the method has returned• Other types of specs also existhttp://research.microsoft.com/en-us/projects/contracts/
  11. 11. Microsoft Research Code ContractsFeaturespublic virtual int Add(object value){Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 );Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if (count == items.Length) EnsureCapacity(count + 1);items[count] = value;return count++;}- Slide adapted from MSR RiSEhttp://research.microsoft.com/en-us/projects/contracts/
  12. 12. Microsoft Research Code ContractsFeatures Language expression syntaxpublic virtual int Add(object value){Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 );Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if (count == items.Length) EnsureCapacity(count + 1);items[count] = value;return count++;}- Slide adapted from MSR RiSEhttp://research.microsoft.com/en-us/projects/contracts/
  13. 13. Microsoft Research Code ContractsFeatures Language expression syntax Type checking / IDEpublic virtual int Add(object value){Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 );Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if (count == items.Length) EnsureCapacity(count + 1);items[count] = value;return count++;}- Slide adapted from MSR RiSEhttp://research.microsoft.com/en-us/projects/contracts/
  14. 14. Microsoft Research Code ContractsFeatures Language expression syntax Type checking / IDE Declarativepublic virtual int Add(object value){Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 );Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if (count == items.Length) EnsureCapacity(count + 1);items[count] = value;return count++;}- Slide adapted from MSR RiSEhttp://research.microsoft.com/en-us/projects/contracts/
  15. 15. Microsoft Research Code ContractsFeatures Language expression syntax Type checking / IDE Declarative Special Encodings Result and Oldpublic virtual int Add(object value){Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 );Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if (count == items.Length) EnsureCapacity(count + 1);items[count] = value;return count++;}- Slide adapted from MSR RiSEhttp://research.microsoft.com/en-us/projects/contracts/
  16. 16. Microsoft Research Code Contracts[ContractInvariantMethod]void ObjectInvariant() {Contract.Invariant( items != null );}Features Language expression syntax Type checking / IDE Declarative Special Encodings Result and Oldpublic virtual int Add(object value){Contract.Requires( value != null );Contract.Ensures( Count == Contract.OldValue(Count) + 1 );Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );if (count == items.Length) EnsureCapacity(count + 1);items[count] = value;return count++;}- Slide adapted from MSR RiSEhttp://research.microsoft.com/en-us/projects/contracts/
  17. 17. Parameterized Unit Testingvoid TestAdd(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}• Parameterized Unit Test =Unit Test with Parameters• Separation of concerns– Data is generated by a tool– Developer can focus on functional specification[Tillmann&Schulte ESEC/FSE 05]http://research.microsoft.com/apps/pubs/default.aspx?id=77419
  18. 18. Parameterized Unit Tests areFormal SpecificationsAlgebraic Specifications• A Parameterized Unit Test can be read as auniversally quantified, conditional axiom.void TestReadWrite(Res r, string name, string data) {Assume.IsTrue(r!=null & name!=null && data!=null);r.WriteResource(name, data);Assert.AreEqual(r.ReadResource(name), data);}∀ string name, string data, Res r:r ≠ null ⋀ name ≠ null ⋀ data ≠ null ⇒equals(ReadResource(WriteResource(r, name, data).state, name),data)
  19. 19. http://research.microsoft.com/pex/Parameterized Unit Tests in Pex
  20. 20. Parameterized Unit TestingGetting PopularParameterized Unit Tests (PUTs) commonly supported byvarious test frameworks• .NET: Supported by .NET test frameworks– http://www.mbunit.com/– http://www.nunit.org/– …• Java: Supported by JUnit 4.X– http://www.junit.org/Generating test inputs for PUTs supported by tools• .NET: Supported by Microsoft Research Pex– http://research.microsoft.com/Pex/• Java: Supported by Agitar AgitarOne– http://www.agitar.com/
  21. 21. ParameterizedTest-Driven DevelopmentWrite/refine Contractas PUTWrite/refine Codeof ImplementationFix-it (with Pex),Debug with generated testsUse Generated Testsfor RegressionRun PexBug in PUTBug in Codefailuresno failures
  22. 22. Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCode- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  23. 23. Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCodeSoftwareAgitation- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  24. 24. Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCodeSoftwareAgitationObservationson code behavior,plusTest Coverage data- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  25. 25. Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCodeSoftwareAgitationObservationson code behavior,plusTest Coverage data- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  26. 26. Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCodeSoftwareAgitationObservationson code behavior,plusTest Coverage dataIf an Observationreveals a bug, fix it- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  27. 27. Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCodeSoftwareAgitationObservationson code behavior,plusTest Coverage dataIf an Observationreveals a bug, fix itIf it describes desired behavior,click to create a Test Assertion- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  28. 28. Assert behavior of multiple test inputsSoftware Agitation in AgitarOneCodeSoftwareAgitationObservationson code behavior,plusTest Coverage dataIf an Observationreveals a bug, fix itIf it describes desired behavior,click to create a Test AssertionCodeCompileReviewAgitate- Slide adapted from Agitar Software Inc.http://www.agitar.com/
  29. 29. Software Agitation in AgitarOne18Image from http://www.agitar.com/
  30. 30. Automated Test Generation19 Recent advanced technique: DynamicSymbolic Execution/Concolic Testing Instrument code to explore feasible paths Example tool: Pex from Microsoft Research(for .NET programs)P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random testing. In Proc. PLDI2005K. Sen, D. Marinov, and G. Agha. CUTE: a concolic unit testing engine for C. In Proc. ESEC/FSE2005N. Tillmann and J. de Halleux. Pex - White Box Test Generation for .NET. In Proc. TAP 2008
  31. 31. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  32. 32. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}InputnullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  33. 33. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullInputnullExecute&MonitorObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  34. 34. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullConstraints to solvea!=nullInputnullExecute&MonitorChoose next pathObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  35. 35. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullConstraints to solvea!=nullInputnull{}Execute&MonitorSolveChoose next pathObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  36. 36. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0FTFa==nullConstraints to solvea!=nullInputnull{}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  37. 37. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0FTFa==nullConstraints to solvea!=nulla!=null &&a.Length>0Inputnull{}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  38. 38. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0Inputnull{}{0}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  39. 39. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0Inputnull{}{0}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  40. 40. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  41. 41. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}{123…}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  42. 42. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullTConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}{123…}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  43. 43. void CoverMe(int[] a){if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length>0a[0]==123…TFTFFa==nullTConstraints to solvea!=nulla!=null &&a.Length>0a!=null &&a.Length>0 &&a[0]==123456890Inputnull{}{0}{123…}Execute&MonitorSolveChoose next pathObserved constraintsa==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890Done: There is no path left.Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork
  44. 44. Automating Test Generation• Method sequences– MSeqGen/Seeker [Thummalapenta et al. OOSPLA 11, ESEC/FSE 09],Covana [Xiao et al. ICSE 2011], OCAT [Jaygarl et al. ISSTA 10],Evacon [Inkumsah et al. ASE 08], Symclat [dAmorim et al. ASE 06]• Environments e.g., db, file systems, network, …– DBApp Testing [Taneja et al. ESEC/FSE 11], [Pan et al. ASE 11]– CloudApp Testing [Zhang et al. IEEE Soft 12]• Loops– Fitnex [Xie et al. DSN 09]@NCSU ASEhttp://people.engr.ncsu.edu/txie/publications.htm
  45. 45. Pex on MSDN DevLabsIncubation Project for Visual StudioDownload counts (20 months)(Feb. 2008 - Oct. 2009 )Academic: 17,366Devlabs: 13,022Total: 30,388http://research.microsoft.com/projects/pex/
  46. 46. Open Source Pex extensionshttp://pexase.codeplex.com/Publications: http://research.microsoft.com/en-us/projects/pex/community.aspx#publications
  47. 47. Writing Test Oracles Learning Formal Methods!?• Parameterized Unit Test =Unit Test with Parameters• Separation of concerns– Data is generated by a tool– Developer can focus on functional specificationvoid TestAdd(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}
  48. 48. Automatic Test Generation Human Assistance to Test Generation?!Running Symbolic PathFinder ...…====================================================== resultsno errors detected====================================================== statisticselapsed time: 0:00:02states: new=4, visited=0,backtracked=4, end=2search: maxDepth=3,constraints=0choice generators: thread=1, data=2heap: gc=3, new=271, free=22instructions: 2875max memory: 81MBloaded code: classes=71, methods=884…25
  49. 49. Automatic Test Generation Human Assistance to Test Generation?!Running Symbolic PathFinder ...…====================================================== resultsno errors detected====================================================== statisticselapsed time: 0:00:02states: new=4, visited=0,backtracked=4, end=2search: maxDepth=3,constraints=0choice generators: thread=1, data=2heap: gc=3, new=271, free=22instructions: 2875max memory: 81MBloaded code: classes=71, methods=884…25
  50. 50. ChallengesFaced by Test Generation Tools object-creation problems (OCP) - 65% external-method call problems (EMCP) – 27%Total block coverage achieved is 50%, lowest coverage 16%.26 Example: Dynamic Symbolic Execution/Concolic Testing Instrument code to explore feasible paths Challenge: path explosion
  51. 51. ChallengesFaced by Test Generation Tools object-creation problems (OCP) - 65% external-method call problems (EMCP) – 27%Total block coverage achieved is 50%, lowest coverage 16%.26 Example: Dynamic Symbolic Execution/Concolic Testing Instrument code to explore feasible paths Challenge: path explosion
  52. 52.  A graph example fromQuickGraph library00: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  53. 53.  A graph example fromQuickGraph library Includes two classesGraphDFSAlgorithm00: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  54. 54.  A graph example fromQuickGraph library Includes two classesGraphDFSAlgorithm GraphAddVertex00: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  55. 55.  A graph example fromQuickGraph library Includes two classesGraphDFSAlgorithm GraphAddVertexAddEdge: requiresboth vertices to bein graph00: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  56. 56. 5600: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  57. 57. 57 Test target: Cover truebranch (B4) of Line 2400: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  58. 58. 58 Test target: Cover truebranch (B4) of Line 24 Desired objectstate: graph shouldinclude at least oneedge00: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  59. 59. 59 Test target: Cover truebranch (B4) of Line 24 Desired objectstate: graph shouldinclude at least oneedge Target sequence:Graph ag = new Graph();Vertex v1 = new Vertex(0);Vertex v2 = new Vertex(1);ag.AddVertex(v1);ag.AddVertex(v2);ag.AddEdge(v1, v2);DFSAlgorithm algo = newDFSAlgorithm(ag);algo.Compute(v1);00: class Graph : IVEListGraph { …03: public void AddVertex (IVertex v) {04: vertices.Add(v); // B1 }06: public Edge AddEdge (IVertex v1, IVertex v2) {07: if (!vertices.Contains(v1))08: throw new VNotFoundException("");09: // B210: if (!vertices.Contains(v2))11: throw new VNotFoundException("");12: // B314: Edge e = new Edge(v1, v2);15: edges.Add(e); } }//DFS:DepthFirstSearch18: class DFSAlgorithm { …23: public void Compute (IVertex s) { ...24: if (graph.GetEdges().Size() > 0) { // B425: isComputed = true;26: foreach (Edge e in graph.GetEdges()) {27: ... // B528: }29: } } } [Thummalapenta et al. OOPSLA 11]
  60. 60. ChallengesFaced by Test Generation Tools object-creation problems (OCP) - 65% external-method call problems (EMCP) – 27%Total block coverage achieved is 50%, lowest coverage 16%.29 Example: Dynamic Symbolic Execution/Concolic (Pex) Instrument code to explore feasible paths Challenge: path explosion
  61. 61. ChallengesFaced by Test Generation Tools object-creation problems (OCP) - 65% external-method call problems (EMCP) – 27%Total block coverage achieved is 50%, lowest coverage 16%.29 Example: Dynamic Symbolic Execution/Concolic (Pex) Instrument code to explore feasible paths Challenge: path explosion
  62. 62. Example External-Method CallProblems (EMCP)30123
  63. 63. Example External-Method CallProblems (EMCP) Example 1: File.Exists has data dependencieson program input Subsequent branch at Line 1 usingthe return value of File.Exists.30123
  64. 64. Example External-Method CallProblems (EMCP) Example 1: File.Exists has data dependencieson program input Subsequent branch at Line 1 usingthe return value of File.Exists. Example 2: Path.GetFullPath has datadependencies on program input Path.GetFullPath throwsexceptions.30123
  65. 65. Example External-Method CallProblems (EMCP) Example 1: File.Exists has data dependencieson program input Subsequent branch at Line 1 usingthe return value of File.Exists. Example 2: Path.GetFullPath has datadependencies on program input Path.GetFullPath throwsexceptions. Example 3: String.Format donot cause any problem30123
  66. 66. Human Can Help!Object Creation Problems (OCP)Tackle object-creation problems with Factory Methods31
  67. 67. Human Can Help!External-Method Call Problems (EMCP)Tackle external-method call problems with Mock Methods orMethod InstrumentationMocking System.IO.File.ReadAllText32
  68. 68. State-of-the-Art/PracticeTesting ToolsRunning Symbolic PathFinder ...…====================================================== resultsno errors detected====================================================== statisticselapsed time: 0:00:02states: new=4, visited=0,backtracked=4, end=2search: maxDepth=3,constraints=0choice generators: thread=1, data=2heap: gc=3, new=271, free=22instructions: 2875max memory: 81MBloaded code: classes=71, methods=884…Tools typically don’tcommunicate challenges facedby them to enable cooperationbetween tools and users.We typically don’t teach peoplehow to cooperate with tools.33X. Xiao, T. Xie, N. Tillmann, and J. de Halleux. Precise Identification of Problems for StructuralTest Generation. In Proc. ICSE 2011http://people.engr.ncsu.edu/txie/publications/icse11-covana.pdf
  69. 69. Coding Duels1,206,095 clicked Ask Pex!
  70. 70. Coding DuelsPex computes “semantic diff” in cloudcode written in browser vs.secret reference implementationYou win when Pex finds no differences
  71. 71. Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}class Test {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl36
  72. 72. Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}class Test {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl36
  73. 73. Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x <= 0) return 1;return x * Puzzle(x-1);}}Player Implementationclass Player {public static int Puzzle(int x) {return x;}}class Test {public static void Driver(int x) {if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);}}behaviorSecret Impl == Player Impl36
  74. 74. Coding DuelsFun and EngagingIterative gameplayAdaptivePersonalizedNo cheatingClear winning criterion
  75. 75. Example User Feedback“It really got me *excited*. The part that got me most isabout spreading interest in teaching CS: I do think that it’sREALLY great for teaching | learning!”“I used to love the first person shooters and thesatisfaction of blowing away a whole team ofNoobies playing Rainbow Six, but this is far morefun.”“I’m afraid I’ll have to constrain myself to spend just an houror so a day on this really exciting stuff, as I’m really stuffedwith work.”Released since 2010X
  76. 76. Coding Duel Competition@ICSE 2011http://pexforfun.com/icse2011
  77. 77. Teaching and Learning
  78. 78. Coding Duels for Automatic Grading@Grad Software Engineering Coursehttp://pexforfun.com/gradsofteng
  79. 79. Coding Duels for Training Testingpublic static string Puzzle(int[] elems, int capacity, int elem) {if ((maxsize <= 0) || (elems == null) || (elems.Length > (capacity + 1)))return "Assumption Violation!";Stack s= new Stack(capacity);for (int i = 0; i < elems.Length; i++)s.Push(elems[i]);int origSize = s.GetNumOfElements();//Please fill in below test scenario on the s stack//The lines below include assertions to assert the program behaviorPexAssert.IsTrue(s.GetNumOfElements() == origSize + 1);PexAssert.IsTrue(s.Top() == elem); PexAssert.IsTrue(!s.IsEmpty());PexAssert.IsTrue(s.IsMember(elem));return s.GetNumOfElements().ToString() + "; “ + s.Top().ToString() + "; “+ s.IsMember(elem).ToString() + "; " + s.IsEmpty();}Set up a stack with some elementsCache values used in assertions
  80. 80. Usage Scenarios of Pex4Fun• Massive Open Online Courses (MOOC): Challenges– Grading, addressed by Pex4Fun– Cheating [Open Challenge]• Course assignments (students/professionals)– E.g., intro programming, software engineering• Student/professional competitions– E.g., coding-duel competition at ICSE 2011• Assessment of testing/programming/problemsolving skills for job applicants– Not just final results of problem solving but also process!
  81. 81. More ReadingNikolai Tillmann, Jonathan De Halleux, Tao Xie,Sumit Gulwani and Judith BishopTeaching and Learning Programming andSoftware Engineering via Interactive GamingIn Proceedings of the 35th InternationalConference on Software Engineering (ICSE 2013),Software Engineering Education (SEE), SanFrancisco, CA, May 2013.http://people.engr.ncsu.edu/txie/publications/icse13see-pex4fun.pdf
  82. 82. Conclusion• Software testing is important and yet costly; needsautomation• Better Test Inputs: help generate new better testinputs– Generate method arguments– Generate method sequences• Better Test Oracles: help generate better test oracles– Assert behavior of individual test inputs– Assert behavior of multiple test inputs• Software Testing  Educational Gaming– http://www.pexforfun.com/45
  83. 83. Example IndustrialDeveloper Testing Tools• Agitar AgitatorOne http://www.agitar.com/• Parasoft Jtest http://www.parasoft.com/• Google CodePro AnalytiX https://developers.google.com/java-dev-tools/codepro/doc/• SilverMark Test Mentor http://www.silvermark.com/• Microsoft Research Pex (for .NET)http://research.microsoft.com/Pex/• Microsoft Research Spec Explorer (for .NET)http://research.microsoft.com/specexplorer/46
  84. 84. Trends in Practice• Regression Test Selection/Prioritization• Cloud Computing for Test Execution, e.g.,http://www.skytap.com/• Crowdsourcing for Testing, e.g.,http://www.utest.com/• Mocking Environments– Google: EasyMock– Microsoft VS: Fake/Moleshttp://research.microsoft.com/en-us/projects/pex/• Automatic Test Generation– Microsoft: Pex, SAGE http://research.microsoft.com/en-us/um/people/pg/
  85. 85. Q & AThank you!contact: taoxie@gmail.comAcknowledgments: NSF grants CCF-0845272, CCF-0915400, CNS-0958235, CNS-1160603,a Microsoft Research SEIF Award, and a Microsoft Research Award.
  86. 86. Automated Combinatorial TestingGoals – reduce testing cost, improve cost-benefit ratioAccomplishments – huge increase in performance,scalability, 200+ users, most major IT firms and othersAlso non-testing applications – modelling andsimulation, genomehttp://csrc.nist.gov/groups/SNS/acts/index.html
  87. 87. Failure-triggering Interactions• Additionalstudiesconsistent• > 4,000failure reportsanalyzed• Conclusion:failurestriggered byfew variables
  88. 88. NIST ACTS Tool• Covering array generator• Coverage analysis - what is the combinatorial coverage ofexisting test set?• .NET configuration file generator• Fault characterization -ongoingCurrentusershttp://csrc.nist.gov/groups/SNS/acts/documents/comparison-report.htmlapproximately 200 users as ofJuly 2009, in IT, defense, finance,telecom, and many otherindustries
  89. 89. Defining a New System
  90. 90. Variable Interaction Strength
  91. 91. Constraints
  92. 92. Covering Array Output

×