Advanced Java Testing @ POSS 2019

1. New types of tests for Java projects Vincent Massol, CTO XWiki SAS, December 2019

2. Agenda •Context & Current status quo •Coverage testing •Mutation testing •Environment testing •Crash reproduction

3. Context: XWiki • Open source wiki • 15 years • 10-15 active committers • Very extensible, scripting in wiki pages • Platform for developing ad-hoc web applications • Strong build practices using Maven and lots of “Quality” plugins • Using Jenkins & custom pipeline library for the CI https://xwiki.org

4. Context: STAMP • Automatic Test Ampliﬁcation • XWiki SAS participating, 3 years • Experiments on XWiki project Mutation testing Environment Testing Production Testing

5. Current Testing Status • 10986 automated tests (in 2.5 hours): • Unit tests (using Mockito) • Integration tests (using Mockito) • Functional (UI) tests (using Selenium/Webdriver)

6. New questions • Are my tests testing enough? Coverage • How good are my tests? Mutation testing • Do my software work in various setups? Environment testing • How can I reproduce bugs found in production? Production testing = in place w/ strategy = in progress

7. Test Coverage - Local • Using Jacoco and Clover • Strategy - “Ratchet effect”: • Each Maven module has a threshold • Jacoco Maven plugin fails if new code has less coverage than before in % • Dev is allowed to increase threshold Of course TPC is not panacea. You could have 100% and app not working. Also need functional tests. Aim for 80%.

8. Test Coverage - Global • Issue: Local coverage can increase and global decrease • Removed code with high TPC • Code tested indirectly by functional tests and code refactoring led to different paths used • New module with lower TPC than average Global TPC evolution

9. Test Coverage - Global • Strategy: • Global Clover TPC computed automatically every night on Jenkins for all repos combined, using a pipeline • Email sent to developers with report in email (see next slide) • Developers ﬁx module they have been working on • Release Manager (RM) ensures that report passes before release & we add one step in our Release Plan check list. Source: http://massol.myxwiki.org/xwiki/bin/view/Blog/ComparingCloverReports

10. Test Coverage - Global

11. Mutation Testing • Using PIT/Gregor, PIT/Descartes • Concepts of PIT • Modify code under test (mutants) and run tests • Good tests kill mutants • Generates a mutation score similar to the coverage % • Descartes = extreme mutations that execute fast and have high values https://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescartes

12. Mutation - Descartes Image courtesy of Oscar LuisVera Perez / INRIA / STAMP project

13. Mutation - Example

14. Mutation - Example result = (getId() == macroId.getId() || (getId() != null && getId().equals(macroId.getId()))) && (getSyntax() == macroId.getSyntax() || (getSyntax() != null && getSyntax().equals( macroId.getSyntax())));

15. Mutation - Example @Test public void testEquality() { MacroId id1 = new MacroId("id", Syntax.XWIKI_2_0); MacroId id2 = new MacroId("id", Syntax.XWIKI_2_0); MacroId id3 = new MacroId("otherid", Syntax.XWIKI_2_0); MacroId id4 = new MacroId("id", Syntax.XHTML_1_0); MacroId id5 = new MacroId("otherid", Syntax.XHTML_1_0); MacroId id6 = new MacroId("id"); MacroId id7 = new MacroId("id"); Assert.assertEquals(id2, id1); // Equal objects must have equal hashcode Assert.assertTrue(id1.hashCode() == id2.hashCode()); Assert.assertFalse(id3 == id1); Assert.assertFalse(id4 == id1); Assert.assertFalse(id5 == id3); Assert.assertFalse(id6 == id1); Assert.assertEquals(id7, id6); // Equal objects must have equal hashcode Assert.assertTrue(id6.hashCode() == id7.hashCode()); } Not testing for inequality! Improved thanks to Descartes!

16. Mutation - Limitations • Takes time to ﬁnd interesting things to look at and decide if that’s an issue to handle or not. Need better categorisation in report (now reported by Descartes): • Strong pseudo-tested methods:The worst! No matter what the return values are the tests always fail • Pseudo-tested methods: Grey area.The tests pass with at least one modiﬁed value. • Multi module support - PITmp • But slow on large projects (e.g. 7+ hours just for xwiki-rendering)

17. Mutation - Strategy • Seems to be working ok so far (6+ months of feedback now) • But still young and not enough data about evolution • Fail the build when the mutation score of a given module is below a deﬁned threshold in the pom.xml • The idea is that new tests should, in average, be of quality equal or better than past tests. • Other idea: hook on CI to run it only on modiﬁed code/tests. General goal with coverage + mutation: maintain quality

18. Mutation: Going further • Using DSpot • Uses PIT/Descartes but injects results to generate new tests • Adds assertions to existing tests • Generate new test methods • Selector can be PIT/Gregor, PIT/ Descartes, Jacoco (instruction coverage), Clover (Branch coverage) https://massol.myxwiki.org/xwiki/bin/view/Blog/TestGenerationDspot

19. Mutation: Dspot Example 1 public void escapeAttributeValue2() { String escapedText = XMLUtils.escapeAttributeValue("a < a' && a' < a" => a < a" {"); // AssertGenerator add assertion Assert.assertEquals("a < a' && a' < a" => a < a" {", escapedText); // AssertGenerator create local variable with return value of invocation boolean o_escapeAttributeValue__3 = escapedText.contains("<"); // AssertGenerator add assertion Assert.assertFalse(o_escapeAttributeValue__3); // AssertGenerator create local variable with return value of invocation boolean o_escapeAttributeValue__4 = escapedText.contains(">"); // AssertGenerator add assertion Assert.assertFalse(o_escapeAttributeValue__4); // AssertGenerator create local variable with return value of invocation boolean o_escapeAttributeValue__5 = escapedText.contains("'"); // AssertGenerator add assertion Assert.assertFalse(o_escapeAttributeValue__5); // AssertGenerator create local variable with return value of invocation boolean o_escapeAttributeValue__6 = escapedText.contains("""); // AssertGenerator create local variable with return value of invocation boolean o_escapeAttributeValue__7 = escapedText.contains("&&"); // AssertGenerator add assertion Assert.assertFalse(o_escapeAttributeValue__7); // AssertGenerator create local variable with return value of invocation boolean o_escapeAttributeValue__8 = escapedText.contains("{"); // AssertGenerator add assertion Assert.assertFalse(o_escapeAttributeValue__8); } Generated test New test @Test public void escapeAttributeValue() { String escapedText = XMLUtils.escapeAttributeValue("a < a' && a' < a" => a < a" {"); assertFalse("Failed to escape <", escapedText.contains("<")); assertFalse("Failed to escape >", escapedText.contains(">")); assertFalse("Failed to escape '", escapedText.contains("'")); assertFalse("Failed to escape "", escapedText.contains(""")); assertFalse("Failed to escape &", escapedText.contains("&&")); assertFalse("Failed to escape {", escapedText.contains("{")); } Original test

20. Mutation: Dspot Example 2 Generated test Original test Also increase coverage Before: 70.5% After: 71.2%

21. Mutation: Dspot Strategy • DSpot is very slow to execute (between 3 to 20mn on small modules) • One strategy is to run it on CI and in the pipeline commit generated tests in a different source root. • And run it only on Tests affected by commit changeset • Conﬁgure Maven to add a new test directory source using the Maven Build Helper plugin. • Work in progress: small coverage and mutation score improvements on XWiki so far.

22. Environment Testing • Environment = combination of Servlet container & version, DB & version, OS, Browser & version • Future: cluster mode, LibreOfﬁce integration, external SOLR, etc • Need: Be able to run/debug functional tests on local dev machines as well as on CI • Using Docker / TestContainers

23. Environment Testing https://massol.myxwiki.org/xwiki/bin/view/Blog/EnvironmentTestingExperimentations

24. Environment Testing

25. Environment Testing • Feedback: takes about 3 minutes to deploy all (and 1 minute for the test) • Strategy • Run on CI (Jenkins) • 3 jobs • “latest”: latest versions of all elements (DB, Servlet Container, Browser, etc). Once per day • “all”: all supported versions. Once per week • “unsupported”: what we want to support in the future. Once per month. • Future: IE/Edge + Docker in Docker • Some instability with Docker and DinD/DooD.

26. Crash Reproduction • Tool: Botsing • Concept:Take a stack trace and generates a test that, when executed, leads to this stack trace • i.e. ﬁnd the conditions that leads to the problem

27. Botsing Example 10 frames reproduced!

28. Botsing - Feedback • Can take a long time to reproduce, doesn’t always succeed • Generates a test that reproduces the problem, not the ﬁx! • Often you’d write a test at a different level (usually up in the call chain, to be more meaningful to the use case) • Is useful for newcomers who don’t know the codebase well as it helps pinpoint the problem. Acts as a timesaver.

29. Parting words • Experiment, push the limit! • Some other types of tests not covered and that also need automation • Backward compatibility testing • Performance/Stress testing • Usability testing • others?

30. Q&A Me

31. Vincent Massol vincent@xwiki.com skype: vmassol http://about.me/vmassol http://xwiki.org http://xwiki.com

Advanced Java Testing @ POSS 2019

You are reading a preview.

Check these out next

Advanced Java Testing @ POSS 2019

More Related Content

Slideshows for you (20)

Similar to Advanced Java Testing @ POSS 2019 (20)

More from Vincent Massol (20)

Recently uploaded (20)