ICSE 2014: Unit Test Virtualization with VMVM

604 views

Published on

Presentation of the ACM SigSoft Distinguished Paper, "Unit Test Virtualization with VMVM" given at ICSE 2014. Unit Test Virtualization significantly decreases the amount of time necessary to run unit tests.

Published in: Science, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
604
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

ICSE 2014: Unit Test Virtualization with VMVM

  1. 1. @_jon_bell_ICSE 2014 June 5, 2014 Unit Test Virtualization with VMVM Jonathan Bell and Gail Kaiser Columbia University Forkm e on Github
  2. 2. @_jon_bell_ICSE 2014 June 5, 2014 Good news: We have tests! No judgement on whether they are complete or not, but we sure have a lot of them Number of tests Apache Tomcat 1,734 Closure Compiler 7,949 Commons I/O 1,022
  3. 3. @_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests!
  4. 4. @_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests! • Much work has focused on improving the situation: • Test Suite Prioritization • E.g. Wong [ISSRE ’97], Rothermel [ICSM ’99]; Elbaum [ICSE ’01]; Srivastava [ISSTA ’02] and more
  5. 5. @_jon_bell_ICSE 2014 June 5, 2014 Bad news: We have to run a lot of tests! • Much work has focused on improving the situation: • Test Suite Prioritization • E.g. Wong [ISSRE ’97], Rothermel [ICSM ’99]; Elbaum [ICSE ’01]; Srivastava [ISSTA ’02] and more • Test Suite Minimization • E.g. Harrold [TOSEM ’93]; Wong [ICSE ’95]; Chen [IST ’98]; Jones [TOSEM ’03]; Tallam [PASTE ’05]; Jeffrey [TSE ’07]; Orso [ICSE ’09] Hao [ICSE ’12] and more
  6. 6. @_jon_bell_ICSE 2014 June 5, 2014 Testing still takes too long.
  7. 7. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization
  8. 8. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62%
  9. 9. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Apache Tomcat: From 26 minutes to 18 minutes
  10. 10. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Integrates with JUnit, ant, and mvn on unmodified JVMs. Apache Tomcat: From 26 minutes to 18 minutes
  11. 11. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach: Unit Test Virtualization Reduces test execution time by up to 97%, on average 62% Integrates with JUnit, ant, and mvn on unmodified JVMs. Apache Tomcat: From 26 minutes to 18 minutes Available on GitHub
  12. 12. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite
  13. 13. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite
  14. 14. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test!
  15. 15. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM?
  16. 16. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM? Unit tests as fast as 3-5 ms
  17. 17. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test! Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec)
  18. 18. @_jon_bell_ICSE 2014 June 5, 2014 JUnit Test Execution Start JVM Execute Test Terminate App Begin Test Start Test Suite 1.4 sec (combined)! For EVERY test!Up to 4,153%, avg 618% Overhead of restarting the JVM? Unit tests as fast as 3-5 ms JVM startup time is fairly constant (1.4 sec) *From our study of 20 popular FOSS apps
  19. 19. @_jon_bell_ICSE 2014 June 5, 2014 Do applications really use a new JVM for each test? • Checked out the 1,000 largest Java projects from Ohloh • 81% of projects with more than 1,000 tests do it • 71% of projects with more than 1 million LOC do it • Overall: 41% of all of the projects do
  20. 20. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent
  21. 21. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests?
  22. 22. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence • We typically assume that tests are order- independent • Might rely on developers to completely reset the system under test between tests • Who tests the tests? • Dangerous: If wrong, can have false positives or false negatives (Muşlu [FSE ’11], Zhang [ISSTA ’14])
  23. 23. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue();
  24. 24. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue(); This field is set once, when the class that owns it is initialized
  25. 25. @_jon_bell_ICSE 2014 June 5, 2014 Test Independence /**#If#true,#cookie#values#are#allowed#to#contain#an#equals# character#without#being#quoted.#*/# public#static#boolean#ALLOW_EQUALS_IN_VALUE#=# ####Boolean.valueOf(System.getProperty("org.apache.tomcat.# ####util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE","false"))# ########.booleanValue(); This field is set once, when the class that owns it is initialized This field’s value is dependent on an external property
  26. 26. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue();
  27. 27. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue();
  28. 28. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test
  29. 29. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test But our static field is stuck! TestDontAllowEqualsInValue
  30. 30. @_jon_bell_ICSE 2014 June 5, 2014 A Tale of Two Tests TestAllowEqualsInValue TestDontAllowEqualsInValue Sets environmental variable to true Start Tomcat, run test public#static#boolean#ALLOW_EQUALS_IN_VALUE#=#Boolean.valueOf(# # System.getProperty(“org.apache.tomcat.util.http.ServerCookie.# # ALLOW_EQUALS_IN_VALUE","false")).booleanValue(); Sets environmental variable to false Start Tomcat, run test TestAllowEqualsInValue
  31. 31. @_jon_bell_ICSE 2014 June 5, 2014 Our Approach Unit Test Virtualization: Allow tests to leave side-effects. But efficiently contain them.
  32. 32. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs
  33. 33. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance We think in terms of object graphs
  34. 34. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Accessible! Objects references We think in terms of object graphs
  35. 35. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references We think in terms of object graphs
  36. 36. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references Accessible! Objects references Test Case n references We think in terms of object graphs
  37. 37. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented Test Runner Instance Test Case 1 references Test Case 2 references Accessible! Objects references Accessible! Objects references Accessible! Objects references Test Case n references We think in terms of object graphs No cross-talk No cross-talk
  38. 38. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class#A Static# Fields Class#B Static# Fields Static fields: owned by a class, NOT by an instance
  39. 39. @_jon_bell_ICSE 2014 June 5, 2014 How do Tests Leak Data? Java is memory-managed, and object oriented We think in terms of object graphs Class#A Static# Fields Class#B Static# Fields Static fields: owned by a class, NOT by an instance These are leakage points references references
  40. 40. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2
  41. 41. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes
  42. 42. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads
  43. 43. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads Writes
  44. 44. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static# Fields Class%B Static# Fields Class%C Static# Fields Test 1 Test 2 Writes Reads Reads Static% Fields Writes
  45. 45. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* Static% Fields
  46. 46. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception*
  47. 47. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception*
  48. 48. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* So, don’t touch them! These classes had no possible conflicts
  49. 49. @_jon_bell_ICSE 2014 June 5, 2014 Isolating Side Effects Class%A Static& Fields Class%B Static& Fields Class%C Static& Fields Test 1 Test 2 Writes Reads Reads Writes *Interception* So, don’t touch them! These classes had no possible conflicts Key Insight:! No need to re-initialize the entire application in order to isolate tests
  50. 50. @_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM
  51. 51. @_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit
  52. 52. @_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation
  53. 53. @_jon_bell_ICSE 2014 June 5, 2014 VMVM: Unit Test Virtualization • Isolates in-memory side effects, just like restarting JVM • Integrates easily with ant, maven, junit • Implemented completely with application byte code instrumentation • No changes to JVM, no access to source code required
  54. 54. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time
  55. 55. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T
  56. 56. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T
  57. 57. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status
  58. 58. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Release lock on T Notinitialized
  59. 59. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Release lock on T Notinitialized
  60. 60. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Release lock on T Notinitialized
  61. 61. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Release lock on T Acquire lock on T Notinitialized
  62. 62. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Acquire lock on T Notinitialized
  63. 63. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Notinitialized
  64. 64. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Notinitialized First new instance or static reference of T per test
  65. 65. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Re-initialize T’s super classes Notinitialized First new instance or static reference of T per test
  66. 66. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization Emulate exactly what happens when a class is initialized the first time First new instance or static reference of T Acquire lock on T Check initialization status Initialize T’s super classes Run initializer for T Mark init done Release lock on T Release lock on T Acquire lock on T Re-initialize T’s super classes Notinitialized First new instance or static reference of T per test Re-initialize T
  67. 67. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs
  68. 68. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class
  69. 69. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java
  70. 70. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process
  71. 71. @_jon_bell_ICSE 2014 June 5, 2014 Efficient Reinitialization • Does not require any modifications to the JVM and runs on commodity JVMs • The JVM calls a special method, <clinit> to initialize a class • We do the same, entirely in Java • Add guards to trigger this process • Register a hook with test runner to tell us when a new test starts
  72. 72. @_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization?
  73. 73. @_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM?
  74. 74. @_jon_bell_ICSE 2014 June 5, 2014 Experiments • RQ1: How does VMVM compare to Test Suite Minimization? • RQ2: What are the performance gains of VMVM? • RQ3: Does VMVM impact fault finding ability?
  75. 75. @_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches
  76. 76. @_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault finding ability, Harrold [TOSEM ‘93]'s technique
  77. 77. @_jon_bell_ICSE 2014 June 5, 2014 RQ1: VMVM vs Test Minimization • Study design follows Zhang [ISSRE ‘11]’s evaluation of four minimization approaches • Compare to the minimization technique with least impact on fault finding ability, Harrold [TOSEM ‘93]'s technique • Study performed on the popular Software Infrastructure Repository dataset
  78. 78. @_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! RQ1: VMVM vs Test Minimization Larger is better
  79. 79. @_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% RQ1: VMVM vs Test Minimization Larger is better
  80. 80. @_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% RQ1: VMVM vs Test Minimization Larger is better
  81. 81. @_jon_bell_ICSE 2014 June 5, 2014 0%! 10%! 20%! 30%! 40%! 50%! 60%! 70%! 80%! 90%! Antv1!Antv2!Antv3!Antv4!Antv5!Antv6!Antv7!Antv8! JM eterv1! JM eterv2! JM eterv3! JM eterv4! JM eterv5! jtopas v1! jtopas v2! jtopas v3! xm l-sec v1! xm l-sec v2! xm l-sec v3! ReductioninTestingTime! Application! Test Suite Minimization! VMVM! Combined! 13% 46% 49% RQ1: VMVM vs Test Minimization Larger is better
  82. 82. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes
  83. 83. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes
  84. 84. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation • Previous study: well-studied suite of 4 projects, which average 37,000 LoC and 51 test classes • This study: manually collected repository of 20 projects, average 475,000 LoC and 56 test classes • Range from 5,000 LoC - 5,692,450 LoC; 3 - 292 test classes; 3.5-15 years in age
  85. 85. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openfire! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Larger is better
  86. 86. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openfire! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Average: 62% Larger is better
  87. 87. @_jon_bell_ICSE 2014 June 5, 2014 RQ2: Broader Evaluation 0%! 20%! 40%! 60%! 80%! 100%! upm! JTor! Openfire! Trove for Java! FreeRapid Downloader! JAXX! Commons Validator! Commons Codec! Closure Compiler! betterFORM! Apache Ivy! mkgmap! gedcom4j! btrace! Apache River! Commons IO! Jetty! Apache Tomcat! Apache Nutch! Bristlecone! Relative Speedup! Max: 97% Average: 62% Larger is better
  88. 88. @_jon_bell_ICSE 2014 June 5, 2014 Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup
  89. 89. @_jon_bell_ICSE 2014 June 5, 2014 Factors that impact reduction • Looked for relationships between number of tests, lines of code, age of project, total testing time, time per test, and VMVM’s speedup • Result: Only average time per test is correlated with VMVM’s speedup (in fact, quite strongly; p < 0.0001)
  90. 90. @_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR)
  91. 91. @_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though?
  92. 92. @_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications
  93. 93. @_jon_bell_ICSE 2014 June 5, 2014 RQ3: Impact on Fault Finding • No impact on fault finding from seeded faults (SIR) • Does VMVM correctly isolate tests though? • Compared false positives and negatives between un- isolated execution, traditionally isolated execution, and VMVM-isolated execution for these 20 complex applications • Result: False positives occur when not isolated. VMVM shows no false positives or false negatives.
  94. 94. @_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases
  95. 95. @_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases • VMVM provides up to a 97% reduction in testing time through more efficient isolation (average 62%)
  96. 96. @_jon_bell_ICSE 2014 June 5, 2014 Conclusions • Most large applications isolate their test cases • VMVM provides up to a 97% reduction in testing time through more efficient isolation (average 62%) • VMVM does not risk a reduction in fault finding
  97. 97. @_jon_bell_ICSE 2014 June 5, 2014 Unit Test Virtualization with VMVM Jonathan Bell and Gail Kaiser Columbia University https://github.com/Programming-Systems-Lab/vmvm See a demo of VMVM at 2:30 today! Room MR G1-3 Forkm e on Github

×