GeeCON 2011 Who Watches The Watchmen? - On Quality Of Tests

http://alexsfunplace.com

www.coachingkidssports.com


http://cdn.plussizecostumesupercenter.com

CODE

CODER

TESTS

http://cdn.plussizecostumesupercenter.com

Who watches the watchmen?
- on quality of tests
Tomek Kaczanowski
http://kaczanowscy.pl/tomek

Should developers test?

http://www.codeodor.com

Should developers test?

Oh yes!

http://nickjosevski.files.wordpress.com

Traditional deployment

QA finished? → we can deploy to production.

Continuous deployment

All tests green? → automatic deploy to production

Power of automatic tests

• A mix of test – 15 000 of test cases
– 1h of Selenium tests
– Unit & integration tests
• 4.5h of testing
– Run in parallel on 30-40 machines
http://www.mmostation.com/

– Reduces execution time to 9 minutes

• All tests green? → deploy!

Tests written by developers
can be

EXTREMELY
POWERFUL
...if done right

Why should we care?

• Because we
– write more tests than production code
– use TDD
– are agile
– deploy continously

Why should we care?

• Because we
– write more tests than production code
– use TDD
– are agile
– deploy continously

We rely on our tests!

Software Quality

Internal vs. External

http://blog.castsoftware.com/

Software Quality - Tests
• External quality:
– Client perspective
– Really testing my application?
– Quality measure: bugs found

• Internal quality:
http://www.cypressmfg.com

– Developer perspective
– Is it easy to maintain them?
– Quality measure: development problems

A high quality test

• Ends green or red
• Fully automated
• Covers important functionality
• Is understandable
• Is simple (in terms of logic)
• Respects DRY principle
• Respects SRP principles
• Runs fast

Why code review tests?

• Learning http://wapedia.mobi/

– Is our API
convenient to use?
– Is our design
testable?

Why code review tests?

http://wapedia.mobi/

http://zuskin.com
• Learning • Verification of tests
– Is our API – Do they add value or
convenient to use? hinder
– Is our design modifications?
testable? – Are they easy to
understand?

External Quality of Tests

• Complete

http://www.planetdan.net
• Understandable

dearyongjoon.blogspot.com


• Complete

http://www.planetdan.net
– Code coverage
– Test parameters
– Mutational testing

• Understandable

dearyongjoon.blogspot.com
– DSLs

External quality of tests

Code
Coverage

Code Coverage

• Powerful tools

http://www.ibm.com/developerworks
– IDE, build tools, CI, Sonar
• Various types
– line, branch, …
– source/byte-code, offline/runtime
• Cool tricks

http://www.phpunit.de
– Coverage reports merge

Code Coverage

• Issues

http://www.ibm.com/developerworks
– Conditional statements
– When loop is covered?
– Threads ignored
• It is all about quantity,
– not quality!
• How much is good enough?

http://www.phpunit.de
– 100%
– green/red is bad!

Human nature... we do what we are
paid for

paid for
http://peureport.blogspot.com/

paid for

Lorem ipsum dolor sit amet, consectetur
adipiscing elit. Aliquam id nulla elit.
Phasellus turpis diam, dictum sed
venenatis luctus, dictum ac odio.

Aenean at nunc non libero posuere
aliquet.

paid for

Lorem ipsum?
- dolor sit amet consectetur
- adipiscing elit.

- Aliquam id nulla elit?
- Phasellus turpis diam
- dictum sed venenatis luctus?
- dictum ac odio!
Aenean at nunc non libero:
- posuere aliquet.

paid for

Lo-lo-lo-rem ipsu-su-su-su-
sum dolor sit amet
consectetur adipiscing eli-
li-li-li-lit. Aliquam id nulla
eli-li-li-li-lit. Pha-pha-pha-

pha-phasellus turpis diam,
dictum sed venenatis Colin Firth

luctus, dictum ac o-o-o-o-
o-o-dio. Aenean at nu-nu-
nu-nu-nu-nu-nunc non
libe-e-e-e-e-e-ro posuere
aliquet.

paid for

If you can't measure
what you want to get,
you will only get
what you can measure.

Code Coverage will help, if:

• People are mature
• Code reviews are performed
• Coverage reports from all types of tests are
considered

...still, code coverage says little about the
quality of tests


Test
parameters

Test parameters

• Even the simplest functions are impossible to
test exhaustively:
– public boolean whatever(int x)

• Tests for all three groups:
– Normal
– Boundary
– Strange

Many test parametrs – naive approach

public class StackTest {

  @Test
public void pushTest() {
   stack.push(1);
   assertEquals(stack.peek(), 1);
   stack.push(2);
   stack.push(3);
   stack.push(4);
}

Many test parameters - JUnit
@RunWith(value = Parameterized.class)
Stack<Integer> stack;
private int number;

public StackTest(int number) {    this.number = number;  }

@Parameters
public static Collection data() {
   return Arrays.asList(1, 2, 3, 4);
}

@Test
public void pushTest() {
   stack.push(number);
   assertEquals(stack.peek(), number);
}

Many test parameters - TestNG
@Test
Stack<Integer> stack;

@DataProvider
public Object[][] getNumbers() {
return new Object[][] {
   { 1 }, {2 }, {3 }, {4 }
};
}

@Test(dataProvider = „getNumbers”)
public void pushTest(int number) {
   stack.push(number);
   assertEquals(stack.peek(), number);
}

Many test parameters – outsource it :)

• Generate them randomly - JCheck

@RunWith(org.jcheck.runners.JCheckRunner.class)
class SimpleTest {
    @Test public void simpleAdd(int i, int j) {
        Money miCHF= new Money(i, "CHF");
        Money mjCHF= new Money(j, "CHF");
        Money expected= new Money(i+j, "CHF");
        Money result= miCHF.add(mjCHF);
        assertEquals(expected, result);
    }
}

Many test parameters – outsource it :)

• Generate them randomly - QuickCheck

@Test
public void sortedListCreation() {
    for (List<Integer> any : someLists(integers())) {
        SortedList sortedList = new SortedList(any);
        List<Integer> expected = ...
        assertEquals(expected, sortedList.toList());
    }
}

Combinations of test parameters
public void whatever(boolean a, boolean b, enum c)

• Number of test cases: 2 x 2 x enum.size()
• Rather cumbersome to test all combinations

Combinations of test parameters

• Pairwise testing (all pairs testing) to the rescue
• Observation:
– “most faults are caused by interactions of at
most two factors”
– Generation of test suite which covers all input
combinations of two and is therefore much
smaller than exhaustive testing.


Mutational
Testing

Mutational Testing
TESTS
ORIGINAL
CODE

Mutational Testing
TESTS
ORIGINAL
CODE

TESTS
MUTATED OR
CODE

Mutational Testing

• Mutations - examples
– Conditionals, a > b to !(a > b)
– Arithmetic operators + to -
– Switch statements
– Some language-specific e.g. public modifiers
• Issues
– Time! - creation of mutants, execution of tests


Readability

Make it readable. For everyone.

sel.open("/");
sel.type("Bugzilla_login", "admin");
sel.type("Bugzilla_password", "admin");
sel.click("log_in");
sel.waitForPageToLoad("30000");
http://www.eviltester.com/

sel.click("link=Reports");
sel.click("link=My Requests");


gotoHomePage();

loginAsAdmin();

gotoReportsPage();
http://www.eviltester.com/

gotoRequestsPage();

http://blog.spritecloud.com/

Scenario: Going to page Services
  Given that I am on spriteCloud Home
  When I click on link Services
  Then the page title should be "spriteCloud
Services"


• Tools are mature & robust
– Fitnesse, Concordion, Cucumber, Twist
• People seems to be the problem!


• Code coverage & mutational testing can help
• Test parameters are important
• DSLs can help

• No tools will make external quality happen
• Code review the tests!
• Reasonable people are crucial

http://www.ourdailyjourney.org/

Internal Quality of Tests

http://www.rightattitudes.com
• Managable

http://www.q8ieng.com
• Run fast or even faster

http://www.raysofhealinglight.com
• Helpful in bug fixing

Internal quality of tests

Readability

Naming – two options
• Follow the structure of classes
– Class:
• MyClassTest
– Method:
• testAddClient
• Follow the behaviour http://www.dailymail.co.uk

– Class:
• WhenAddingClients
– Methods:
• shouldRaiseExceptionIfClientIsNull
• shouldReturnUniqueIdOfCreatedClient

Naming – variables and resources
@DataProvider
   public static Object[][] userPermissions() {
       return new Object[][]{
               {"user_1", READ, true},  {"user_1", WRITE, true},
               {"user_2", READ, false}, {"user_2", WRITE, false},
               {"user_3", READ, false}, {"user_3", WRITE, false},
...
       };
   }

User_1 does not
@Test(dataprovider = „userPermissions”)
hasPermission(String user, Perm perm, boolean
have READ
result) { permission, but he
assertEquals(permService.hasPerm(user, perm), should (...should
result); he?).
}

Naming – variables and resources
@DataProvider
   public static Object[][] userPermissions() {
       return new Object[][]{
               {"admin", READ, true},  {"admin", WRITE, true},
               {"logged", READ, false}, {"logged", WRITE, false},
               {"guest", READ, false}, {"guest", WRITE, false},
...
       };
   }

Admin does not
@Test(dataprovider = „userPermissions”)
hasPermission(String user, Perm perm, boolean
have READ
result) { permission, but he
assertEquals(permService.hasPerm(user, perm), should.
result);
}

Matchers – more assertions
List<Character> fellowshipOfTheRing = ...

assertThat(fellowshipOfTheRing).hasSize(9)
                                   .contains(frodo, sam)
                                   .excludes(sauron);

assertThat(fellowshipOfTheRing).onProperty("race.name")
                                   .contains("Hobbit",
"Man", "Elf")
                                   .excludes("Orc");

Matchers – custom assertions
ServerSocketAssertion socket =
      new ServerSocketAssertion(server.getSocket());
assertThat(socket).isConnectedTo(2000);

public class ServerSocketAssertion implements AssertExtension {
  private final ServerSocket socket;
  public ServerSocketAssertion(ServerSocket socket) {
    this.socket = socket;
  }
  public ServerSocketAssert isConnectedTo(int port) {
    assertThat(socket.isBound()).isTrue();
    assertThat(socket.getLocalPort()).isEqualTo(port);
    assertThat(socket.isClosed()).isFalse();
    return this;
  }
}

KISS

• No logic please

• Do nothing more than
– Create object
– Execute methods
– Verify results
http://robertstrongmarketing.org

• Avoid complex test dependencies
– Explicit and implicit

Assertion Messages

Before fixing the code,
make sure the
assertion message
clearly indicates the
problem.

• Assertion message should contain:
– information on what should have happened (but did not)
– arguments

TestNG diagram example

• Red – failed
• Yellow – skipped

TestNG logs output

• Don't miss the important information!

log4j.rootCategory=ALL, stdout, testLogFile

#only FATAL stuff (i.e. nothing) goes to standard output
log4j.appender.stdout.threshold=FATAL
...

#everything is logged to a file
log4j.appender.testLogFile=org.apache.log4j.FileAppender
log4j.appender.testLogFile.threshold=DEBUG
log4j.appender.testLogFile.File=target/testlogs.log
...


Static
Code
Analysis

Static Code Analysis
• TestCase has no tests
• Proper spelling of setUp/tearDown
• Assertions always with message
• Use proper assertions (e.g. assertEquals instead of
assertTrue)
• Every test with assertion
• Test class must have some test cases
• Unnecessary boolean assertions
– e.g. assertFalse(true)
• The suite() method in a JUnit test needs to be both
public and static

Static Code Analysis
• Mostly Junit (3.8!)
• Many checks rendered obsolete by:
– Test-first coding
– Use of IDE http://zazzle.com

• Test are dead simple (in terms of language
constructs)

Can't tell you anything interesting
about real value/quality of your test


Make it
fast

Make it fast

• Boost it:
– More CPU/RAM
– Create (costly) context once
– Write real unit tests
– Run tests in parallel
http://memegenerator.net

@Test(threadPoolSize = 3, invocationCount = 10,
timeOut = 10000)
public void testServer() {

Make it fast

• Make it fail fast
– Test-first
– First fast tests, then slow ones
– Pre-tested commit
– Use test dependencies wisely http://thumannresources.files.wordpress.com

– Use timeouts

Make it fast

• No Thread.sleep();
• No System.currentTimeMillis();

• Dependency injection
– Interface TimeSource http://images.icanhascheezburger.com

– Production code and tests uses different
implementations


Portability

Portability

• Problems:
– new File(„C:My Documentswhatever.txt”)
– DriverManager.
getConnection("jdbc:mysql://localhost:8080, ...);

• Not only Java code, but also test scripts

Portability

• Heuristic:
– Checkout on a clean system
– Run them
– ...and see what went wrong

• But start with
– other developers machines
– CI server


Modification
Leads to
Failing Tests

My main problem at the moment with unit tests is
when I change a design I get a stack of failing tests.
This means I'm either going to write less tests or make
fewer big design changes. Both of which are bad
things. Unicode Andy (cited from http://www.codinghorror.com/)

It’s overwhelmingly easy to write bad unit tests that
add very little value to a project while inflating the
cost of code changes astronomically.
Steve Sanderson

Why does it happen?

• Acceptance/Functional testing
– Testing via user interface
– Move to a higher abstraction level
• Unit tests
– With test doubles
– Price we pay for meddling in object's private
affairs

Red?! Yes!

Yes, some tests should
be failing if you change
your production code.
This is desirable.

Solutions (well, sort of...)

• Use „nice” mocks (or Mockito)
• Respect DRY principle

Think about what is really important. Is it really
important that this getter is called only once by
the tested method? No? Then record it as a stub. Do
I care about this passed parameter? No? Just use
the anyObject matcher. That seems silly but that’s
how you get resilient tests.
Henri Tremblay, EasyMock lead developer

Testing antipatterns
• http://blog.james-carr.org/2006/11/03/tdd-anti-patterns/
• The sloppy worker – A test that creates
persistent resources but doesn’t clean after
itself.
• The Mockery - [...] a unit test contains so many
mocks, stubs, and/or fakes that the system
under test isn’t even being tested at all,
instead data returned from mocks is what is
being tested.
• The Dodo – [...] tests behavior no longer
required in the application.

Testing antipatterns
• Many reasons to fail (SRP violation)
• Testing the same thing (DRY violation)
• Happy path testing
• Side effects (database change)
• Fixed values
– e.g. after saving user:
users = userDao.getAllUsers();
assertEquals(users.size(), 2);

Tautological tests
ClientDAO clientDAO;
void saveClient(Client client) {
    clientDAO.save(client);
}

ClientService clientService = ...;
ClientDAO clientDAO = ...; // create mock
Client client = ...;

@Test
void makeSureThatClientIsSaved() {
    expect(clientDAO.saveClient(client));
    clientService.save(client);
    verify(clientDAO);
}

Tautological tests

[...] instead of using the tests to
drive the domain model, the
developer ends up writing the
implementation code twice

Bill Six

Now it is worth testing
ClientDAO clientDAO;

void saveClient(Client client) {
if (client.something()) {
....
}
else {
clientDAO.save(client);
}
}

So what
should I
http://www.desicomments.com
do?

Quality - Same techniques
Production Code Tests
Make it readable Extract Method Extract Method
DLS DSL
Matchers

Verify it Tests Code Review
Code Review Code Coverage
Static Analysis Mutational Testing

DRY Code it once Test it once
SRP One reason to One reason to fail
change
KISS Limited responsibility, No logic, stick to
no awkward stuff common patterns

What should I do?
• Respect tests http://www.tefldaddy.com

• Use the right tools
• Stick to principles: DRY,
• Code review tests SRP, KISS
• Test on the right level • TDD rules!
• Choose test parameters • Examine coverage
well reports
• Make them portable • Be consistent
• Make them readable • Encourage people to
• Make them fast participate.
• Helpful for diagnostic

Thank you

Tomek Kaczanowski
http://kaczanowscy.pl/tomek
Would be great if you could provide some feedback at
tkaczano@poczta.onet.pl

GeeCON 2011 Who Watches The Watchmen? - On Quality Of Tests

More Related Content

Similar to GeeCON 2011 Who Watches The Watchmen? - On Quality Of Tests

More from Tomek Kaczanowski

Recently uploaded

GeeCON 2011 Who Watches The Watchmen? - On Quality Of Tests