• Like
  • Save
GeeCON 2011 Who Watches The Watchmen? - On Quality Of Tests
Upcoming SlideShare
Loading in...5
×
 

GeeCON 2011 Who Watches The Watchmen? - On Quality Of Tests

on

  • 1,100 views

Slides of my GeeCON 2011 talk on quality of tests. See my blog for additional information http://kaczanowscy.pl/tomek

Slides of my GeeCON 2011 talk on quality of tests. See my blog for additional information http://kaczanowscy.pl/tomek

Statistics

Views

Total Views
1,100
Views on SlideShare
981
Embed Views
119

Actions

Likes
0
Downloads
15
Comments
0

3 Embeds 119

http://kaczanowscy.pl 111
http://www.kaczanowscy.pl 7
http://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    GeeCON 2011 Who Watches The Watchmen? - On Quality Of Tests GeeCON 2011 Who Watches The Watchmen? - On Quality Of Tests Presentation Transcript

    • http://alexsfunplace.com www.coachingkidssports.com
    • www.coachingkidssports.comhttp://alexsfunplace.com http://cdn.plussizecostumesupercenter.com
    • CODE www.coachingkidssports.comCODER TESTShttp://alexsfunplace.com http://cdn.plussizecostumesupercenter.com
    • Who watches the watchmen? - on quality of tests Tomek Kaczanowski http://kaczanowscy.pl/tomek
    • Should developers test? http://www.codeodor.com
    • Should developers test? Oh yes! http://nickjosevski.files.wordpress.com
    • Traditional deploymentQA finished? → we can deploy to production.
    • Continuous deploymentAll tests green? → automatic deploy to production
    • Continuous deploymentAll tests green? → automatic deploy to production
    • Power of automatic tests• A mix of test – 15 000 of test cases – 1h of Selenium tests – Unit & integration tests• 4.5h of testing – Run in parallel on 30-40 machines http://www.mmostation.com/ – Reduces execution time to 9 minutes• All tests green? → deploy!
    • Tests written by developers can beEXTREMELYPOWERFUL ...if done right
    • Why should we care?• Because we – write more tests than production code – use TDD – are agile – deploy continously
    • Why should we care?• Because we – write more tests than production code – use TDD – are agile – deploy continously We rely on our tests!
    • Software QualityInternal vs. External http://blog.castsoftware.com/
    • Software Quality - Tests• External quality: – Client perspective – Really testing my application? – Quality measure: bugs found• Internal quality: http://www.cypressmfg.com – Developer perspective – Is it easy to maintain them? – Quality measure: development problems
    • A high quality test• Ends green or red• Fully automated• Covers important functionality• Is understandable• Is simple (in terms of logic)• Respects DRY principle• Respects SRP principles• Runs fast
    • Code reviews
    • Why code review tests?• Learning http://wapedia.mobi/ – Is our API convenient to use? – Is our design testable?
    • Why code review tests? http://wapedia.mobi/ http://zuskin.com• Learning • Verification of tests – Is our API – Do they add value or convenient to use? hinder – Is our design modifications? testable? – Are they easy to understand?
    • ExternalQuality of Test
    • External Quality of Tests• Complete http://www.planetdan.net• Understandable dearyongjoon.blogspot.com
    • External Quality of Tests• Complete http://www.planetdan.net – Code coverage – Test parameters – Mutational testing• Understandable dearyongjoon.blogspot.com – DSLs
    • External quality of tests Code Coverage
    • Code Coverage• Powerful tools http://www.ibm.com/developerworks – IDE, build tools, CI, Sonar• Various types – line, branch, … – source/byte-code, offline/runtime• Cool tricks http://www.phpunit.de – Coverage reports merge
    • Code Coverage• Issues http://www.ibm.com/developerworks – Conditional statements – When loop is covered? – Threads ignored• It is all about quantity, – not quality!• How much is good enough? http://www.phpunit.de – 100% – green/red is bad!
    • Human nature... we do what we are paid for
    • Human nature... we do what we are paid forhttp://peureport.blogspot.com/
    • Human nature... we do what we are paid for Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam id nulla elit. Phasellus turpis diam, dictum sed venenatis luctus, dictum ac odio.http://peureport.blogspot.com/ Aenean at nunc non libero posuere aliquet.
    • Human nature... we do what we are paid forhttp://peureport.blogspot.com/
    • Human nature... we do what we are paid for Lorem ipsum? - dolor sit amet consectetur - adipiscing elit.http://peureport.blogspot.com/ - Aliquam id nulla elit? - Phasellus turpis diam - dictum sed venenatis luctus? - dictum ac odio! Aenean at nunc non libero: - posuere aliquet.
    • Human nature... we do what we are paid forhttp://peureport.blogspot.com/
    • Human nature... we do what we are paid for Lo-lo-lo-rem ipsu-su-su-su- sum dolor sit amet consectetur adipiscing eli- li-li-li-lit. Aliquam id nulla eli-li-li-li-lit. Pha-pha-pha-http://peureport.blogspot.com/ pha-phasellus turpis diam, dictum sed venenatis Colin Firth luctus, dictum ac o-o-o-o- o-o-dio. Aenean at nu-nu- nu-nu-nu-nu-nunc non libe-e-e-e-e-e-ro posuere aliquet.
    • Human nature... we do what we are paid for If you cant measure what you want to get, you will only get what you can measure.
    • Code Coverage will help, if:• People are mature• Code reviews are performed• Coverage reports from all types of tests are considered ...still, code coverage says little about the quality of tests
    • External quality of tests Test parameters
    • Test parameters• Even the simplest functions are impossible to test exhaustively: – public boolean whatever(int x)• Tests for all three groups: – Normal – Boundary – Strange
    • Many test parametrs – naive approach public class StackTest {   @Test  public void pushTest() {    stack.push(1);    assertEquals(stack.peek(), 1);    stack.push(2);    assertEquals(stack.peek(), 2);    stack.push(3);    assertEquals(stack.peek(), 3);    stack.push(4);    assertEquals(stack.peek(), 4);  }
    • Many test parameters - JUnit@RunWith(value = Parameterized.class)public class StackTest { Stack<Integer> stack; private int number; public StackTest(int number) {    this.number = number;  } @Parameters public static Collection data() {   return Arrays.asList(1, 2, 3, 4); }@Test public void pushTest() {   stack.push(number);   assertEquals(stack.peek(), number); }
    • Many test parameters - TestNG@Testpublic class StackTest { Stack<Integer> stack; @DataProviderpublic Object[][] getNumbers() { return new Object[][] {   { 1 }, {2 }, {3 }, {4 }  };}@Test(dataProvider = „getNumbers”) public void pushTest(int number) {   stack.push(number);   assertEquals(stack.peek(), number); }
    • Many test parameters – outsource it :)• Generate them randomly - JCheck@RunWith(org.jcheck.runners.JCheckRunner.class)class SimpleTest {    @Test public void simpleAdd(int i, int j) {        Money miCHF= new Money(i, "CHF");        Money mjCHF= new Money(j, "CHF");        Money expected= new Money(i+j, "CHF");        Money result= miCHF.add(mjCHF);        assertEquals(expected, result);    }}
    • Many test parameters – outsource it :)• Generate them randomly - QuickCheck@Testpublic void sortedListCreation() {    for (List<Integer> any : someLists(integers())) {        SortedList sortedList = new SortedList(any);        List<Integer> expected = ...        assertEquals(expected, sortedList.toList());    }}
    • Combinations of test parameterspublic void whatever(boolean a, boolean b, enum c)• Number of test cases: 2 x 2 x enum.size()• Rather cumbersome to test all combinations
    • Combinations of test parameters• Pairwise testing (all pairs testing) to the rescue• Observation: – “most faults are caused by interactions of at most two factors” – Generation of test suite which covers all input combinations of two and is therefore much smaller than exhaustive testing.
    • External quality of tests Mutational Testing
    • Mutational Testing TESTSORIGINAL CODE
    • Mutational Testing TESTSORIGINAL CODE TESTSMUTATED OR CODE
    • Mutational Testing• Mutations - examples – Conditionals, a > b to !(a > b) – Arithmetic operators + to - – Switch statements – Some language-specific e.g. public modifiers• Issues – Time! - creation of mutants, execution of tests
    • External quality of tests Readability
    • Make it readable. For everyone. sel.open("/"); sel.type("Bugzilla_login", "admin"); sel.type("Bugzilla_password", "admin"); sel.click("log_in"); sel.waitForPageToLoad("30000");http://www.eviltester.com/ sel.click("link=Reports"); sel.waitForPageToLoad("30000"); sel.click("link=My Requests"); sel.waitForPageToLoad("30000");
    • Make it readable. For everyone. gotoHomePage(); loginAsAdmin(); gotoReportsPage();http://www.eviltester.com/ gotoRequestsPage();
    • Make it readable. For everyone.http://blog.spritecloud.com/ Scenario: Going to page Services   Given that I am on spriteCloud Home   When I click on link Services   Then the page title should be "spriteCloud  ­ Services"
    • Make it readable. For everyone.• Tools are mature & robust – Fitnesse, Concordion, Cucumber, Twist• People seems to be the problem!
    • External Quality of Tests• Code coverage & mutational testing can help• Test parameters are important• DSLs can help• No tools will make external quality happen• Code review the tests!• Reasonable people are crucial http://www.ourdailyjourney.org/
    • InternalQuality of Tests
    • Internal Quality of Tests http://www.rightattitudes.com• Managable http://www.q8ieng.com• Run fast or even faster http://www.raysofhealinglight.com• Helpful in bug fixing
    • Internal quality of tests Readability
    • Naming – two options• Follow the structure of classes – Class: • MyClassTest – Method: • testAddClient• Follow the behaviour http://www.dailymail.co.uk – Class: • WhenAddingClients – Methods: • shouldRaiseExceptionIfClientIsNull • shouldReturnUniqueIdOfCreatedClient
    • Naming – two options• Follow the structure of classes – Class: • MyClassTest – Method: • testAddClient• Follow the behaviour http://www.dailymail.co.uk – Class: • WhenAddingClients – Methods: • shouldRaiseExceptionIfClientIsNull • shouldReturnUniqueIdOfCreatedClient
    • Naming – variables and resources @DataProvider    public static Object[][] userPermissions() {        return new Object[][]{                {"user_1", READ, true},  {"user_1", WRITE, true},                {"user_2", READ, false}, {"user_2", WRITE, false},                {"user_3", READ, false}, {"user_3", WRITE, false}, ...        };    } User_1 does not@Test(dataprovider = „userPermissions”)hasPermission(String user, Perm perm, boolean  have READresult) { permission, but he assertEquals(permService.hasPerm(user, perm),  should (...shouldresult); he?).}
    • Naming – variables and resources @DataProvider    public static Object[][] userPermissions() {        return new Object[][]{                {"admin", READ, true},  {"admin", WRITE, true},                {"logged", READ, false}, {"logged", WRITE, false},                {"guest", READ, false}, {"guest", WRITE, false}, ...        };    } Admin does not@Test(dataprovider = „userPermissions”)hasPermission(String user, Perm perm, boolean  have READresult) { permission, but he assertEquals(permService.hasPerm(user, perm),  should.result);}
    • Matchers – more assertionsList<Character> fellowshipOfTheRing = ...assertThat(fellowshipOfTheRing).hasSize(9)                                   .contains(frodo, sam)                                   .excludes(sauron);assertThat(fellowshipOfTheRing).onProperty("race.name")                                   .contains("Hobbit", "Man", "Elf")                                   .excludes("Orc");
    • Matchers – custom assertionsServerSocketAssertion socket =          new ServerSocketAssertion(server.getSocket());assertThat(socket).isConnectedTo(2000);public class ServerSocketAssertion implements AssertExtension {  private final ServerSocket socket;  public ServerSocketAssertion(ServerSocket socket) {    this.socket = socket;  }  public ServerSocketAssert isConnectedTo(int port) {    assertThat(socket.isBound()).isTrue();    assertThat(socket.getLocalPort()).isEqualTo(port);    assertThat(socket.isClosed()).isFalse();    return this;  }}
    • KISS• No logic please• Do nothing more than – Create object – Execute methods – Verify results http://robertstrongmarketing.org• Avoid complex test dependencies – Explicit and implicit
    • Helpful inbug fixing
    • Assertion Messages Before fixing the code, make sure the assertion message clearly indicates the problem.• Assertion message should contain: – information on what should have happened (but did not) – arguments
    • TestNG diagram example• Red – failed• Yellow – skipped
    • TestNG logs output• Dont miss the important information! log4j.rootCategory=ALL, stdout, testLogFile #only FATAL stuff (i.e. nothing) goes to standard output log4j.appender.stdout.threshold=FATAL ... #everything is logged to a file log4j.appender.testLogFile=org.apache.log4j.FileAppender log4j.appender.testLogFile.threshold=DEBUG log4j.appender.testLogFile.File=target/test­logs.log ...
    • Internal quality of tests Static Code Analysis
    • Static Code Analysis• TestCase has no tests• Proper spelling of setUp/tearDown• Assertions always with message• Use proper assertions (e.g. assertEquals instead of assertTrue)• Every test with assertion• Test class must have some test cases• Unnecessary boolean assertions – e.g. assertFalse(true)• The suite() method in a JUnit test needs to be both public and static
    • Static Code Analysis• Mostly Junit (3.8!)• Many checks rendered obsolete by: – Test-first coding – Use of IDE http://zazzle.com• Test are dead simple (in terms of language constructs) Cant tell you anything interestingabout real value/quality of your test
    • Internal quality of tests Make it fast
    • Make it fast• Boost it: – More CPU/RAM – Create (costly) context once – Write real unit tests – Run tests in parallel http://memegenerator.net@Test(threadPoolSize = 3, invocationCount = 10, timeOut = 10000)public void testServer() {
    • Make it fast• Make it fail fast – Test-first – First fast tests, then slow ones – Pre-tested commit – Use test dependencies wisely http://thumannresources.files.wordpress.com – Use timeouts
    • Make it fast• No Thread.sleep();• No System.currentTimeMillis();• Dependency injection – Interface TimeSource http://images.icanhascheezburger.com – Production code and tests uses different implementations
    • Internal quality of tests Portability
    • Portability• Problems: – new File(„C:My Documentswhatever.txt”) – DriverManager.  getConnection("jdbc:mysql://localhost:8080, ...);• Not only Java code, but also test scripts
    • Portability• Heuristic: – Checkout on a clean system – Run them – ...and see what went wrong• But start with – other developers machines – CI server
    • Internal quality of tests Modification Leads to Failing Tests
    • My main problem at the moment with unit tests iswhen I change a design I get a stack of failing tests.This means Im either going to write less tests or makefewer big design changes. Both of which are badthings. Unicode Andy (cited from http://www.codinghorror.com/)It’s overwhelmingly easy to write bad unit tests thatadd very little value to a project while inflating thecost of code changes astronomically. Steve Sanderson
    • Why does it happen?• Acceptance/Functional testing – Testing via user interface – Move to a higher abstraction level• Unit tests – With test doubles – Price we pay for meddling in objects private affairs
    • Red?! Yes!Yes, some tests shouldbe failing if you change your production code. This is desirable.
    • Solutions (well, sort of...)• Use „nice” mocks (or Mockito)• Respect DRY principleThink about what is really important. Is it reallyimportant that this getter is called only once bythe tested method? No? Then record it as a stub. DoI care about this passed parameter? No? Just usethe anyObject matcher. That seems silly but that’show you get resilient tests. Henri Tremblay, EasyMock lead developer
    • Testingantipatterns
    • Testing antipatterns• http://blog.james-carr.org/2006/11/03/tdd-anti-patterns/• The sloppy worker – A test that creates persistent resources but doesn’t clean after itself.• The Mockery - [...] a unit test contains so many mocks, stubs, and/or fakes that the system under test isn’t even being tested at all, instead data returned from mocks is what is being tested.• The Dodo – [...] tests behavior no longer required in the application.
    • Testing antipatterns• Many reasons to fail (SRP violation)• Testing the same thing (DRY violation)• Happy path testing• Side effects (database change)• Fixed values – e.g. after saving user:users = userDao.getAllUsers();assertEquals(users.size(), 2);
    • Tautological testsClientDAO clientDAO;void saveClient(Client client) {    clientDAO.save(client);}ClientService clientService = ...;ClientDAO clientDAO = ...; // create mockClient client = ...;@Testvoid makeSureThatClientIsSaved() {    expect(clientDAO.saveClient(client));    clientService.save(client);    verify(clientDAO);}
    • Tautological tests[...] instead of using the tests to drive the domain model, the developer ends up writing the implementation code twice Bill Six
    • Now it is worth testingClientDAO clientDAO;void saveClient(Client client) { if (client.something()) { .... } else { clientDAO.save(client); }}
    • So what should Ihttp://www.desicomments.com do?
    • Quality - Same techniques Production Code TestsMake it readable Extract Method Extract Method DLS DSL MatchersVerify it Tests Code Review Code Review Code Coverage Static Analysis Mutational TestingDRY Code it once Test it onceSRP One reason to One reason to fail changeKISS Limited responsibility, No logic, stick to no awkward stuff common patterns
    • What should I do?• Respect tests http://www.tefldaddy.com• Use the right tools • Stick to principles: DRY,• Code review tests SRP, KISS• Test on the right level • TDD rules!• Choose test parameters • Examine coverage well reports• Make them portable • Be consistent• Make them readable • Encourage people to• Make them fast participate.• Helpful for diagnostic
    • http://studyzone.org
    • Thank you Tomek Kaczanowskihttp://kaczanowscy.pl/tomekWould be great if you could provide some feedback at tkaczano@poczta.onet.pl