Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Next Generation Developer Testing: Parameterized Testing

1,151 views

Published on

A talk given at the Next Generation Testing Conference at Chicago, September 17, 2015

Published in: Spiritual
  • Be the first to comment

Next Generation Developer Testing: Parameterized Testing

  1. 1. Next Generation Developer Testing: Parameterized Testing Tao Xie Department of Computer Science University of Illinois at Urbana-Champaign Email: taoxie@illinois.edu http://taoxie.cs.illinois.edu/ In Collaboration with Microsoft Research
  2. 2. 2 Introduction  Developer (aka unit) testing – a widely adopted practice for ensuring high quality software  A conventional unit test (CUT): small program with test inputs and test assertions void AddTest() { HashSet set = new HashSet(); set.Add(7); set.Add(3); Assert.IsTrue(set.Count == 2); } Test Scenario Test Assertions Test Data
  3. 3. 3 Parameterized Unit Tests (PUTs)  Recent advances in unit testing introduced PUTs void AddSpec(int x, int y) { HashSet set = new HashSet(); set.Add(x); set.Add(y); Assert.AreEqual(x == y, set.Count == 1); Assert.AreEqual(x != y, set.Count == 2); }  PUTs separate two concerns: • Specification of externally visible behavior (assertions) • Selection of internally relevant test inputs (coverage)
  4. 4. 4 Parameterized Unit Tests (PUTs)  More beneficial than CUTs • Help describe behaviors for all test arguments  Address two main issues with CUTs • Missing test data required for exercising important behaviors  Low fault-detection capability • Including test data that exercises the same behaviour  Redundant unit tests
  5. 5. 5 An Example using IntStack public void CUT1() { int elem = 1; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } Three CUTs public void CUT2() { int elem = 30; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } public void CUT3() { int elem1 = 1, elem2 = 30; IntStack stk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count()); }  CUT1 and CUT2 exercise push with different test data  CUT3 exercises push when stack is not empty Two main issues with CUTs:  Fault-detection capability issue: undetected defect where things go wrong when passing a negative value to push  Redundant test issue: CUT2 is redundant with respect to CUT1
  6. 6. 6 An Example using IntStack public void CUT1() { int elem = 1; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } Three CUTs public void CUT2() { int elem = 30; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } public void CUT3() { int elem1 = 1, elem2 = 30; IntStack stk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count()); }  No need to describe test data • Generated automatically  Single PUT replaces multiple CUTs • With reduced size of test code public void PUT(int[] elem) { Assume.IsTrue(elem != null); IntStack stk = new IntStack(); for(int i in elem) stk.push(elem); Assert.AreEqual(elem.Length, stk.count()); } An equivalent PUT
  7. 7. Parameterized Unit Tests are Algebraic Specifications • A Parameterized Unit Test can be read as a universally quantified, conditional axiom. void ReadWrite(Storage s, string name, string data) { Assume.IsTrue(s!=null && name!=null && data!=null); s.WriteResource(name, data); var readData = s.ReadResource(name); Assert.AreEqual(data, readData); }  Storage s, string name, string data: s ≠ null ⋀ name ≠ null ⋀ data ≠ null ⇒ equals( ReadResource(WriteResource(s,name,data).state, name).retval, data)
  8. 8. Parameterized Unit Testing is going mainstream Parameterized Unit Tests (PUTs) commonly supported by various test frameworks • .NET: Supported by .NET test frameworks – http://www.mbunit.com/ – http://www.nunit.org/ … • Java: Supported by JUnit 4.X – http://www.junit.org/ Generating test inputs for PUTs supported by tools • .NET: Supported by Microsoft Visual Studio 2015 IntelliTest – Formerly Microsoft Research Pex: http://research.microsoft.com/pex/ • Java: Supported by Agitar AgitarOne – http://www.agitar.com/
  9. 9. Parameterized Tests in JUnit 9 https://github.com/junit-team/junit/wiki/Parameterized-tests
  10. 10. JUnit Theories 10 https://github.com/junit-team/junit/wiki/Theories
  11. 11. Assumptions and Assertions void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException(); } void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException(); } • Assumptions and assertions induce branches • Executions which cause assumption violations are ignored, not reported as errors or test cases
  12. 12. Test Data Generation • Human – Expensive, incomplete, … • Brute Force – Pairwise, predefined data, etc… • Semi - Random – Cheap, Fast – “It passed a thousand tests” feeling • Dynamic Symbolic Execution: IntelliTest/Pex, SAGE, CUTE, … – Automated white-box – Not random – Constraint Solving
  13. 13. 13 void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } a.Length>0 a[0]==123… TF T F F a==null T Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==123456890 Input null {} {0} {123…} Execute&MonitorSolve Choose next path Observed constraints a==null a!=null && !(a.Length>0) a==null && a.Length>0 && a[0]!=1234567890 a==null && a.Length>0 && a[0]==1234567890  Generates test data systematically Done: There is no path left. Background: DSE
  14. 14. Pex4Fun http://pex4fun.com/ Nikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani and Judith Bishop. Teaching and Learning Programming and Software Engineering via Interactive Gaming. In ICSE 2013, SEE. http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf 1,703,247 clicked 'Ask Pex!'
  15. 15. Pex4Fun • Click http://pex4fun.com/default.aspx?language=CSharp&sa mple=_Template • Copy and modify the following code snippet in the code editing box (or simply click here) using System; using Microsoft.Pex.Framework; using Microsoft.Pex.Framework.Settings; [PexClass] public class Program { [PexMethod]//[PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)] public static string testMethod(int x, int y) { PexAssume.IsTrue(y >= 0);//replace here with your assumption //... enter your code under test here //if (x == 10000) throw new Exception(); PexAssert.IsTrue(y >= 0);//replace here with your assertion return PexSymbolicValue.GetPathConditionString(); } } https://sites.google.com/site/teachpex/Home/pex-usage-tips
  16. 16. Microsoft Visual Studio 2015 IntelliTest 16 https://msdn.microsoft.com/en-us/library/dn823749.aspx
  17. 17. Microsoft Visual Studio 2015 IntelliTest 17https://msdn.microsoft.com/en-us/library/dn823749.aspx
  18. 18. 18 Recall: An Example using IntStack public void CUT1() { int elem = 1; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } Three CUTs public void CUT2() { int elem = 30; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count()); } public void CUT3() { int elem1 = 1, elem2 = 30; IntStack stk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count()); }  No need to describe test data • Generated automatically  Single PUT replaces multiple CUTs • With reduced size of test code public void PUT(int[] elem) { Assume.IsTrue(elem != null); IntStack stk = new IntStack(); for(int i in elem) stk.push(elem); Assert.AreEqual(elem.Length, stk.count()); } An equivalent PUT
  19. 19. 19 Test Generalization: CUTs  PUT Major Steps • S1: Parameterize • S2: Generalize Test Oracle • S3: Add Assumptions Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011 http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
  20. 20. Example 00: public class SettingsGroup{ 01: MSS storage; ... 02: public SettingsGroup(MSS storage) { 03: this.storage = storage; 04: } 05: public void SaveSetting(string sn, object sv) { 06: object ov = storage.GetSetting( sn ); 07: //Avoid change if there is no real change 08: if (ov != null ) { 09: if(ov is string && sv is string && (string)ov==(string)sv || 10: ov is int && sv is int && (int)ov==(int)sv || 11: ov is bool&& sv is bool&& (bool)ov==(bool)sv || 12: ov is Enum&& sv is Enum&& ov.Equals(sv)) 13: return; 14: } 15: storage.SaveSetting(sn, sv); 16: if (Changed != null) 17: Changed(this, new SettingsEventArgs(sn)); 18: }} 20 SettingsGroup class of NUnit with the SaveSetting method under test
  21. 21. An Existing CUT 00: public class SettingsGroup{ 01: MSS storage; ... 02: public SettingsGroup(MSS storage) { 03: this.storage = storage; 04: } 05: public void SaveSetting(string sn, object sv) { 06: object ov = storage.GetSetting( sn ); 07: //Avoid change if there is no real change 08: if (ov != null ) { 09: if(ov is string && sv is string && (string)ov==(string)sv || 10: ov is int && sv is int && (int)ov==(int)sv || 11: ov is bool&& sv is bool&& (bool)ov==(bool)sv || 12: ov is Enum&& sv is Enum&& ov.Equals(sv)) 13: return; 14: } 15: storage.SaveSetting(sn, sv); 16: if (Changed != null) 17: Changed(this, new SettingsEventArgs(sn)); 18: }} 21 00: //tg is of type SettingsGroup 01: [Test] 02: public void TestSettingsGroup() { 03: tg.SaveSetting("X",5); 04: tg.SaveSetting("NAME","Tom"); 05: Assert.AreEqual(5,tg.GetSetting("X")); 06: Assert.AreEqual("Tom",tg.GetSetting("NAME")); 07: } Existing CUT SettingsGroup class of NUnit with the SaveSetting method under test
  22. 22. ? Issues with Existing CUT 22 Only CUT for verifying SaveSetting method • Does not verify the behavior for the types bool and enum • Does not cover the true branch in Line 8 Does test generalization addresses these two issues … 06: object ov = storage.GetSetting( sn ); 07: //Avoid change if there is no real change 08: if (ov != null ) { …
  23. 23. 23 S1 - Parameterize  Promote all primitive values as arguments • name of setting as a parameter of type string • string “TOM” and int 5 as a parameter of type object helping IntelliTest/Pex generate concrete values based on the constraints encountered in different paths  Promote non-primitive objects such as receiver objects as arguments helping IntelliTest/Pex generate object states for the receiver objects that can cover additional paths //Original CUT 02: public void TestSettingsGroup() { 03: tg.SaveSetting("X",5); 04: tg.SaveSetting("NAME","Tom");…. //New PUT 02: public void TestSave( SettingsGroup st, string sn, object sv){ ….
  24. 24. 24 S2 – Generalize Test Oracle  Replace the constant value, “TOM” and 5, with the relevant parameter of the PUT //Original CUT 02: public void TestSettingsGroup() { 03: tg.SaveSetting("X",5); 04: tg.SaveSetting("NAME","Tom"); 05: Assert.AreEqual(5,tg.GetSetting("X")); 06: Assert.AreEqual("Tom",tg.GetSetting("NAME")); …. //New PUT 02: public void TestSave(SettingsGroup st, string sn, object sv){ 03: st.SaveSetting(sn, sv); 04: PexAssert.AreEqual(sv,st.GetSetting(sn));
  25. 25. //New PUT 02: public void TestSave(SettingsGroup st, string sn, object sv){ 03: st.SaveSetting(sn, sv); 04: PexAssert.AreEqual(sv,st.GetSetting(sn)); 25 S3 – Add Assumptions IntelliTest/Pex requires guidance in generating legal values for the parameters E.g., Add the tag PexAssumeUnderTest (PAUT) with the parameter, i.e., generated value should not be null //New PUT 02: public void TestSave([PAUT]SettingsGroup st, [PAUT]string sn, [PAUT] object sv){ 03: st.SaveSetting(sn, sv); 04: PexAssert.AreEqual(sv,st.GetSetting(sn));
  26. 26. 26 Example Summary  Resulting PUT Result of S1  Parameters “sn” and “sv”  Result of S2  Line 4: “st.GetSetting(sn)”  Result of S3  Parameter “st” 00: //PAUT: PexAssumeUnderTest 01: [PexMethod] 02: public void TestSave([PAUT]SettingsGroup st, [PAUT] string sn, [PAUT] object sv) { 03: st.SaveSetting(sn, sv); 04: PexAssert.AreEqual(sv,st.GetSetting(sn)); 05: }
  27. 27. 27 Example - Test Generalization Results  Achieved 10% branch cov  Required 2 method calls with values of type string and int 2: public void TestSave([PAUT]SettingsGroup st, [PAUT]string sn, [PAUT] object sv){ 3: st.SaveSetting(sn, sv); 4: PexAssert.AreEqual (sv,st.GetSetting(sn)); 2: public void TestSettingsGroup() { 3: tg.SaveSetting("X",5); 4: tg.SaveSetting("NAME","Tom");….  Achieved 90% branch cov  Required only 1 method call, sufficient to test for all types (parameter of type object) Original CUT New PUT
  28. 28. EXAMPLE PATTERNS FOR PARAMETERIZED UNIT TESTS http://research.microsoft.com/pex/patterns.pdf
  29. 29. Pattern 4A • Assume, Arrange, Act, Assert [PexMethod] void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act Assert(target.Count == count + 1)//assert }
  30. 30. Pattern Roundtrip • For an API f(x), f-1(f(x)) = x for all x void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); Assert.AreEqual(value, parsed); }
  31. 31. Pattern State Relation • Observe a state change void ContainedAfterAdd(string value) { var list = new List<string>(); list.Add(value); Assert.IsTrue(list.Contains(value)); }
  32. 32. Pattern Commutative Diagram • Given two implementations f and g of the same function, each possible requiring a different number of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(… (gm(x)…)), then it should hold that f1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x. string Multiply(string x, string y); int Multiply(int x, int y); void CommutativeDiagram1(int x, int y) { string z1 = Multiply(x, y).ToString(); string z2 = Multiply(x.ToString(), y.ToString()); PexAssert.AreEqual(z1, z2); }
  33. 33. Code Hunt: Turning Pex/PUT into a Coding Game https://www.codehunt.com/
  34. 34. Behind the Scene of Code Hunt Secret Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); } } Player Implementation class Player { public static int Puzzle(int x) { return x; } } class Test { public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); } } behavior Secret Impl == Player Impl 34
  35. 35. It’s a game! • iterative gameplay • adaptive • personalized • no cheating • clear winning criterion code test cases Bishop et al. Code Hunt: Experience with Coding Contests at Scale. ICSE 2015, JSEET http://taoxie.cs.illinois.edu/publications/icse15jseet-codehunt.pdf
  36. 36. Next Generation Developer Testing: Parameterized Testing Microsoft Visual Studio 2015 - IntelliTest JUnit TheoriesPexMethod Test Generalization CUTs  PUTs PUT Patterns  PUTs …
  37. 37. Thank You 37 https://sites.google.com/site/teachpex/ http://research.microsoft.com/pex http://taoxie.cs.illinois.edu/ Collaboration is Welcome! Email: taoxie@illinois.edu
  38. 38. Next Generation Developer Testing: Parameterized Testing Microsoft Visual Studio 2015 - IntelliTest JUnit TheoriesPexMethod Test Generalization CUTs  PUTs PUT Patterns  PUTs … http://taoxie.cs.illinois.edu/taoxie@illinois.edu https://sites.google.com/site/teachpex/
  39. 39. Pattern Roundtrip • For an API f(x), f-1(f(x)) = x for all x void PropertyRoundtrip(string value) { var target = new Foo(); target.Name = value; var roundtripped = target.Name; Assert.AreEqual(value, roundtripped); } http://research.microsoft.com/pex/patterns.pdf
  40. 40. Pattern Sanitized Roundtrip • For an API f(x), f-1(f(f-1(x)) = f-1(x) for all x void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate); Assert(normalized == roundtripped); } http://research.microsoft.com/pex/patterns.pdf
  41. 41. Pattern Reachability • Indicate which portions of a PUT should be reachable. [PexExpectedGoals] public void InvariantAfterParsing(string input) { ComplexDataStructure x; bool success = ComplexDataStructure.TryParse( input, out x); PexAssume.IsTrue(success); PexGoal.Reached(); x.AssertInvariant(); } http://research.microsoft.com/pex/patterns.pdf
  42. 42. Pattern Regression Tests • Generated test asserts any observed value – Return value, out parameters, PexGoal • When code evolves, breaking changes in observable will be discovered int AddTest(int a, int b) { return a + b; } void AddTest01() { var result = AddTest(0, 0); Assert.AreEqual(0, result); } http://research.microsoft.com/pex/patterns.pdf
  43. 43. Pattern Same Observable Behavior • Given two methods f(x) and g(x), and a method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x. public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right)); } http://research.microsoft.com/pex/patterns.pdf
  44. 44. Pattern Allowed Exception • Allowed exception -> negative test case [PexAllowedException(typeof(ArgumentException))] void Test(object item) { var foo = new Foo(item) // validates item // generated test (C#) [ExpectedException(typeof(ArgumentException))] void Test01() { Test(null); // argument check } http://research.microsoft.com/pex/patterns.pdf
  45. 45. 45 S4 – Add Factory Method  IntelliTest/Pex requires guidance handling non- primitive objects e.g., Add factory methods that generate instances of non-primitive objects 00: //PAUT: PexAssumeUnderTest 01: //MSS: MemorySettingsStorage (class) 01: [PexFactoryMethod(typeof(MSS))] 02: public static MSS Create([PAUT] string[] sn, [PAUT]object[] sv) { 03: PexAssume.IsTrue(sn.Length == sv.Length); 04: PexAssume.IsTrue(sn.Length > 0); 05: MSS mss = new MSS(); 06: for(int count = 0; count < sn.Length; count++) { 07: mss.SaveSetting(sn[count], sv[count]); 08: } 09: return mss; 10: } • Help create different object states for MSS Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011 http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
  46. 46. 46 S5 – Add Mock Object  Pex faces challenges in handling code that interacts with external environment, e.g., file system • Write mock objects to assist Pex and to test features in isolation Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011 http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
  47. 47. 47 Empirical Study  RQ1: Branch coverage • How much higher percentage of branch coverage is achieved by retrofitted PUTs compared to existing CUTs?  RQ2: Defect detection • How many new defects (that are not detected by existing CUTs) are detected by PUTs and vice-versa?  RQ3: Generalization effort • How much effort is required for generalizing CUTs to PUTs? Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011 http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
  48. 48. Study Setup  Three Subject Applications 48 Subjects Downloads Code Under Test # Classes # Methods KLOC Test Code # Classes # CUTs KLOC NUnit 193,563 9 87 1.4 9 49 0.9 DSA 3,239 27 259 2.4 20 337 2.5 QuickGraph 7,969 56 463 6.2 9 21 1.2 TOTAL 92 809 10 38 407 4.6  High downloads count  Lines of code under test: 10 KLOC  Number of CUTs: 407 Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011 http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
  49. 49. Study Setup – cont. 49  Conducted by the first and second authors • No prior knowledge of subjects • Two years of experience with PUTs and Pex • Retrofitted 407 CUTs (4.6 KLOC) as 224 PUTs (4.0 KLOC) Generated three categories of CUTs • C1: Existing CUTs • C2: CUTs generated from PUTs • C3: Existing CUTs + RTs (tests generated using Randoop)  Help show that benefits of generalization cannot be achieved by generating tests randomly Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011 http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
  50. 50. RQ1: Branch Coverage 50 Subjects # RTs Branch Coverage CUTs (%) CUTs + RTs (%) PUTs (%) Overall Increase (%) Maximum Increase (%) NUnit 144 78 78 88 10 52 DSA 615 91 91 92 1 1 QuickGrap h 3628 87 88 89 2 11 CUTs + RTs:  Added 144, 615, and 3628 tests using Randoop  Branch coverage increased by 0%, 0%, and 1% CUTs generated from PUTs:  Branch coverage increased by 10%, 1%, and 2% Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011 http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
  51. 51. RQ2: Defect Detection 51  RTs: CUTs generated using Randoop  Pex without generalized PUTs: CUTs generated by applying Pex on public methods  CUTs generated from PUTs: CUTs generated by applying Pex on generalized PUTs • Detected all defects detected by the first two categories CUTs category #Failing Tests DSA NUnit QuickGraph # Real Defects Basic CUTs 0 0 0 0 RTs 90 25 738 4 Pex without generalized PUTs 23 170 17 2 CUTs generated from PUTs 15 4 0 19 Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011 http://taoxie.cs.illinois.edu/publications/fase11-put.pdf
  52. 52. RQ3: Generalization Effort 52  Both authors conducted comparable amount of generalization  Effort spent in hours • NUnit: 2.8 hrs • DSA: 13.8 hrs • QuickGraph: 1.5 hrs  Effort is worthwhile compared to benefits of test generalization Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011 http://taoxie.cs.illinois.edu/publications/fase11-put.pdf

×