GeeCON 2011 Who Watches The Watchmen? - On Quality Of Tests


Published on

Slides of my GeeCON 2011 talk on quality of tests. See my blog for additional information

Published in: Technology, Lifestyle
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

GeeCON 2011 Who Watches The Watchmen? - On Quality Of Tests

  1. 1.
  2. 2. www.coachingkidssports.com
  3. 3. CODE www.coachingkidssports.comCODER TESTS
  4. 4. Who watches the watchmen? - on quality of tests Tomek Kaczanowski
  5. 5. Should developers test?
  6. 6. Should developers test? Oh yes!
  7. 7. Traditional deploymentQA finished? → we can deploy to production.
  8. 8. Continuous deploymentAll tests green? → automatic deploy to production
  9. 9. Continuous deploymentAll tests green? → automatic deploy to production
  10. 10. Power of automatic tests• A mix of test – 15 000 of test cases – 1h of Selenium tests – Unit & integration tests• 4.5h of testing – Run in parallel on 30-40 machines – Reduces execution time to 9 minutes• All tests green? → deploy!
  11. 11. Tests written by developers can beEXTREMELYPOWERFUL ...if done right
  12. 12. Why should we care?• Because we – write more tests than production code – use TDD – are agile – deploy continously
  13. 13. Why should we care?• Because we – write more tests than production code – use TDD – are agile – deploy continously We rely on our tests!
  14. 14. Software QualityInternal vs. External
  15. 15. Software Quality - Tests• External quality: – Client perspective – Really testing my application? – Quality measure: bugs found• Internal quality: – Developer perspective – Is it easy to maintain them? – Quality measure: development problems
  16. 16. A high quality test• Ends green or red• Fully automated• Covers important functionality• Is understandable• Is simple (in terms of logic)• Respects DRY principle• Respects SRP principles• Runs fast
  17. 17. Code reviews
  18. 18. Why code review tests?• Learning – Is our API convenient to use? – Is our design testable?
  19. 19. Why code review tests?• Learning • Verification of tests – Is our API – Do they add value or convenient to use? hinder – Is our design modifications? testable? – Are they easy to understand?
  20. 20. ExternalQuality of Test
  21. 21. External Quality of Tests• Complete• Understandable
  22. 22. External Quality of Tests• Complete – Code coverage – Test parameters – Mutational testing• Understandable – DSLs
  23. 23. External quality of tests Code Coverage
  24. 24. Code Coverage• Powerful tools – IDE, build tools, CI, Sonar• Various types – line, branch, … – source/byte-code, offline/runtime• Cool tricks – Coverage reports merge
  25. 25. Code Coverage• Issues – Conditional statements – When loop is covered? – Threads ignored• It is all about quantity, – not quality!• How much is good enough? – 100% – green/red is bad!
  26. 26. Human nature... we do what we are paid for
  27. 27. Human nature... we do what we are paid for
  28. 28. Human nature... we do what we are paid for Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam id nulla elit. Phasellus turpis diam, dictum sed venenatis luctus, dictum ac odio. Aenean at nunc non libero posuere aliquet.
  29. 29. Human nature... we do what we are paid for
  30. 30. Human nature... we do what we are paid for Lorem ipsum? - dolor sit amet consectetur - adipiscing elit. - Aliquam id nulla elit? - Phasellus turpis diam - dictum sed venenatis luctus? - dictum ac odio! Aenean at nunc non libero: - posuere aliquet.
  31. 31. Human nature... we do what we are paid for
  32. 32. Human nature... we do what we are paid for Lo-lo-lo-rem ipsu-su-su-su- sum dolor sit amet consectetur adipiscing eli- li-li-li-lit. Aliquam id nulla eli-li-li-li-lit. Pha-pha-pha- pha-phasellus turpis diam, dictum sed venenatis Colin Firth luctus, dictum ac o-o-o-o- o-o-dio. Aenean at nu-nu- nu-nu-nu-nu-nunc non libe-e-e-e-e-e-ro posuere aliquet.
  33. 33. Human nature... we do what we are paid for If you cant measure what you want to get, you will only get what you can measure.
  34. 34. Code Coverage will help, if:• People are mature• Code reviews are performed• Coverage reports from all types of tests are considered ...still, code coverage says little about the quality of tests
  35. 35. External quality of tests Test parameters
  36. 36. Test parameters• Even the simplest functions are impossible to test exhaustively: – public boolean whatever(int x)• Tests for all three groups: – Normal – Boundary – Strange
  37. 37. Many test parametrs – naive approach public class StackTest {   @Test  public void pushTest() {    stack.push(1);    assertEquals(stack.peek(), 1);    stack.push(2);    assertEquals(stack.peek(), 2);    stack.push(3);    assertEquals(stack.peek(), 3);    stack.push(4);    assertEquals(stack.peek(), 4);  }
  38. 38. Many test parameters - JUnit@RunWith(value = Parameterized.class)public class StackTest { Stack<Integer> stack; private int number; public StackTest(int number) {    this.number = number;  } @Parameters public static Collection data() {   return Arrays.asList(1, 2, 3, 4); }@Test public void pushTest() {   stack.push(number);   assertEquals(stack.peek(), number); }
  39. 39. Many test parameters - TestNG@Testpublic class StackTest { Stack<Integer> stack; @DataProviderpublic Object[][] getNumbers() { return new Object[][] {   { 1 }, {2 }, {3 }, {4 }  };}@Test(dataProvider = „getNumbers”) public void pushTest(int number) {   stack.push(number);   assertEquals(stack.peek(), number); }
  40. 40. Many test parameters – outsource it :)• Generate them randomly - JCheck@RunWith(org.jcheck.runners.JCheckRunner.class)class SimpleTest {    @Test public void simpleAdd(int i, int j) {        Money miCHF= new Money(i, "CHF");        Money mjCHF= new Money(j, "CHF");        Money expected= new Money(i+j, "CHF");        Money result= miCHF.add(mjCHF);        assertEquals(expected, result);    }}
  41. 41. Many test parameters – outsource it :)• Generate them randomly - QuickCheck@Testpublic void sortedListCreation() {    for (List<Integer> any : someLists(integers())) {        SortedList sortedList = new SortedList(any);        List<Integer> expected = ...        assertEquals(expected, sortedList.toList());    }}
  42. 42. Combinations of test parameterspublic void whatever(boolean a, boolean b, enum c)• Number of test cases: 2 x 2 x enum.size()• Rather cumbersome to test all combinations
  43. 43. Combinations of test parameters• Pairwise testing (all pairs testing) to the rescue• Observation: – “most faults are caused by interactions of at most two factors” – Generation of test suite which covers all input combinations of two and is therefore much smaller than exhaustive testing.
  44. 44. External quality of tests Mutational Testing
  45. 45. Mutational Testing TESTSORIGINAL CODE
  47. 47. Mutational Testing• Mutations - examples – Conditionals, a > b to !(a > b) – Arithmetic operators + to - – Switch statements – Some language-specific e.g. public modifiers• Issues – Time! - creation of mutants, execution of tests
  48. 48. External quality of tests Readability
  49. 49. Make it readable. For everyone."/"); sel.type("Bugzilla_login", "admin"); sel.type("Bugzilla_password", "admin");"log_in"); sel.waitForPageToLoad("30000");"link=Reports"); sel.waitForPageToLoad("30000");"link=My Requests"); sel.waitForPageToLoad("30000");
  50. 50. Make it readable. For everyone. gotoHomePage(); loginAsAdmin(); gotoReportsPage(); gotoRequestsPage();
  51. 51. Make it readable. For everyone. Scenario: Going to page Services   Given that I am on spriteCloud Home   When I click on link Services   Then the page title should be "spriteCloud  ­ Services"
  52. 52. Make it readable. For everyone.• Tools are mature & robust – Fitnesse, Concordion, Cucumber, Twist• People seems to be the problem!
  53. 53. External Quality of Tests• Code coverage & mutational testing can help• Test parameters are important• DSLs can help• No tools will make external quality happen• Code review the tests!• Reasonable people are crucial
  54. 54. InternalQuality of Tests
  55. 55. Internal Quality of Tests• Managable• Run fast or even faster• Helpful in bug fixing
  56. 56. Internal quality of tests Readability
  57. 57. Naming – two options• Follow the structure of classes – Class: • MyClassTest – Method: • testAddClient• Follow the behaviour – Class: • WhenAddingClients – Methods: • shouldRaiseExceptionIfClientIsNull • shouldReturnUniqueIdOfCreatedClient
  58. 58. Naming – two options• Follow the structure of classes – Class: • MyClassTest – Method: • testAddClient• Follow the behaviour – Class: • WhenAddingClients – Methods: • shouldRaiseExceptionIfClientIsNull • shouldReturnUniqueIdOfCreatedClient
  59. 59. Naming – variables and resources @DataProvider    public static Object[][] userPermissions() {        return new Object[][]{                {"user_1", READ, true},  {"user_1", WRITE, true},                {"user_2", READ, false}, {"user_2", WRITE, false},                {"user_3", READ, false}, {"user_3", WRITE, false}, ...        };    } User_1 does not@Test(dataprovider = „userPermissions”)hasPermission(String user, Perm perm, boolean  have READresult) { permission, but he assertEquals(permService.hasPerm(user, perm),  should (...shouldresult); he?).}
  60. 60. Naming – variables and resources @DataProvider    public static Object[][] userPermissions() {        return new Object[][]{                {"admin", READ, true},  {"admin", WRITE, true},                {"logged", READ, false}, {"logged", WRITE, false},                {"guest", READ, false}, {"guest", WRITE, false}, ...        };    } Admin does not@Test(dataprovider = „userPermissions”)hasPermission(String user, Perm perm, boolean  have READresult) { permission, but he assertEquals(permService.hasPerm(user, perm),  should.result);}
  61. 61. Matchers – more assertionsList<Character> fellowshipOfTheRing = ...assertThat(fellowshipOfTheRing).hasSize(9)                                   .contains(frodo, sam)                                   .excludes(sauron);assertThat(fellowshipOfTheRing).onProperty("")                                   .contains("Hobbit", "Man", "Elf")                                   .excludes("Orc");
  62. 62. Matchers – custom assertionsServerSocketAssertion socket =          new ServerSocketAssertion(server.getSocket());assertThat(socket).isConnectedTo(2000);public class ServerSocketAssertion implements AssertExtension {  private final ServerSocket socket;  public ServerSocketAssertion(ServerSocket socket) {    this.socket = socket;  }  public ServerSocketAssert isConnectedTo(int port) {    assertThat(socket.isBound()).isTrue();    assertThat(socket.getLocalPort()).isEqualTo(port);    assertThat(socket.isClosed()).isFalse();    return this;  }}
  63. 63. KISS• No logic please• Do nothing more than – Create object – Execute methods – Verify results• Avoid complex test dependencies – Explicit and implicit
  64. 64. Helpful inbug fixing
  65. 65. Assertion Messages Before fixing the code, make sure the assertion message clearly indicates the problem.• Assertion message should contain: – information on what should have happened (but did not) – arguments
  66. 66. TestNG diagram example• Red – failed• Yellow – skipped
  67. 67. TestNG logs output• Dont miss the important information! log4j.rootCategory=ALL, stdout, testLogFile #only FATAL stuff (i.e. nothing) goes to standard output log4j.appender.stdout.threshold=FATAL ... #everything is logged to a file log4j.appender.testLogFile=org.apache.log4j.FileAppender log4j.appender.testLogFile.threshold=DEBUG log4j.appender.testLogFile.File=target/test­logs.log ...
  68. 68. Internal quality of tests Static Code Analysis
  69. 69. Static Code Analysis• TestCase has no tests• Proper spelling of setUp/tearDown• Assertions always with message• Use proper assertions (e.g. assertEquals instead of assertTrue)• Every test with assertion• Test class must have some test cases• Unnecessary boolean assertions – e.g. assertFalse(true)• The suite() method in a JUnit test needs to be both public and static
  70. 70. Static Code Analysis• Mostly Junit (3.8!)• Many checks rendered obsolete by: – Test-first coding – Use of IDE• Test are dead simple (in terms of language constructs) Cant tell you anything interestingabout real value/quality of your test
  71. 71. Internal quality of tests Make it fast
  72. 72. Make it fast• Boost it: – More CPU/RAM – Create (costly) context once – Write real unit tests – Run tests in parallel = 3, invocationCount = 10, timeOut = 10000)public void testServer() {
  73. 73. Make it fast• Make it fail fast – Test-first – First fast tests, then slow ones – Pre-tested commit – Use test dependencies wisely – Use timeouts
  74. 74. Make it fast• No Thread.sleep();• No System.currentTimeMillis();• Dependency injection – Interface TimeSource – Production code and tests uses different implementations
  75. 75. Internal quality of tests Portability
  76. 76. Portability• Problems: – new File(„C:My Documentswhatever.txt”) – DriverManager.  getConnection("jdbc:mysql://localhost:8080, ...);• Not only Java code, but also test scripts
  77. 77. Portability• Heuristic: – Checkout on a clean system – Run them – ...and see what went wrong• But start with – other developers machines – CI server
  78. 78. Internal quality of tests Modification Leads to Failing Tests
  79. 79. My main problem at the moment with unit tests iswhen I change a design I get a stack of failing tests.This means Im either going to write less tests or makefewer big design changes. Both of which are badthings. Unicode Andy (cited from’s overwhelmingly easy to write bad unit tests thatadd very little value to a project while inflating thecost of code changes astronomically. Steve Sanderson
  80. 80. Why does it happen?• Acceptance/Functional testing – Testing via user interface – Move to a higher abstraction level• Unit tests – With test doubles – Price we pay for meddling in objects private affairs
  81. 81. Red?! Yes!Yes, some tests shouldbe failing if you change your production code. This is desirable.
  82. 82. Solutions (well, sort of...)• Use „nice” mocks (or Mockito)• Respect DRY principleThink about what is really important. Is it reallyimportant that this getter is called only once bythe tested method? No? Then record it as a stub. DoI care about this passed parameter? No? Just usethe anyObject matcher. That seems silly but that’show you get resilient tests. Henri Tremblay, EasyMock lead developer
  83. 83. Testingantipatterns
  84. 84. Testing antipatterns•• The sloppy worker – A test that creates persistent resources but doesn’t clean after itself.• The Mockery - [...] a unit test contains so many mocks, stubs, and/or fakes that the system under test isn’t even being tested at all, instead data returned from mocks is what is being tested.• The Dodo – [...] tests behavior no longer required in the application.
  85. 85. Testing antipatterns• Many reasons to fail (SRP violation)• Testing the same thing (DRY violation)• Happy path testing• Side effects (database change)• Fixed values – e.g. after saving user:users = userDao.getAllUsers();assertEquals(users.size(), 2);
  86. 86. Tautological testsClientDAO clientDAO;void saveClient(Client client) {;}ClientService clientService = ...;ClientDAO clientDAO = ...; // create mockClient client = ...;@Testvoid makeSureThatClientIsSaved() {    expect(clientDAO.saveClient(client));;    verify(clientDAO);}
  87. 87. Tautological tests[...] instead of using the tests to drive the domain model, the developer ends up writing the implementation code twice Bill Six
  88. 88. Now it is worth testingClientDAO clientDAO;void saveClient(Client client) { if (client.something()) { .... } else {; }}
  89. 89. So what should I do?
  90. 90. Quality - Same techniques Production Code TestsMake it readable Extract Method Extract Method DLS DSL MatchersVerify it Tests Code Review Code Review Code Coverage Static Analysis Mutational TestingDRY Code it once Test it onceSRP One reason to One reason to fail changeKISS Limited responsibility, No logic, stick to no awkward stuff common patterns
  91. 91. What should I do?• Respect tests• Use the right tools • Stick to principles: DRY,• Code review tests SRP, KISS• Test on the right level • TDD rules!• Choose test parameters • Examine coverage well reports• Make them portable • Be consistent• Make them readable • Encourage people to• Make them fast participate.• Helpful for diagnostic
  92. 92.
  93. 93. Thank you Tomek Kaczanowski be great if you could provide some feedback at