Slides from my Confitura 2012 presentation. The issues discussed during the talk will be described in my new (free!) ebook - see https://github.com/tomekkaczanowski/bad-tests-good-tests
3. Tomek Kaczanowski
• Developer
• Team lead
• Blogger
• http://kaczanowscy.pl/tomek
• Book author
• http://practicalunittesting.com
• Working at CodeWise (Krakow)
...we are hiring, wanna join us?
4. Before we begin
• Most of the examples are real but:
Obfuscated
− to protect the innocents
Truncated
− imagine much more complex domain objects
• Asking questions is allowed
...but being smarter than me is not ;)
7. Please...
no more...
http://thedogatemycareplan.wordpress.com
8. Before we begin
• The tests were written in 2004-2006.
• No automation, no CI.
• Some tests do not compile.
• In some tests you can read a comment that "WARNING:
test requires the divide param to be set to 20" but the
code is so ugly, that there is no way to inject this value.
• Some test data are available in form of serialized objects
(*.ser) that can not be currently deserialized, because
the classes have changed.
• The project is now in maintenance.
Courtesy of Bartosz http://twitter.com/#!/bocytko
9. We don't need no stinkin' asserts!
public void testAddChunks() {
System.out.println("*************************************");
System.out.println("testAddChunks() ... ");
ChunkMap cm = new ChunkMap(3);
cm.addChunk(new Chunk("chunk"));
List testList = cm.getChunks("chunk",null);
if (testList.isEmpty())
fail("there should be at least one list!");
Chunk chunk = cm.getActualChunk("chunk",null);
if (chunk.getElements().isEmpty())
fail("there should be at least one element!");
if (cm.getFinalChunkNr() != 1)
fail("there should be at least one chunk!");
// iterate actual chunk
for (Iterator it = chunk.getElements().iterator();
it.hasNext();) {
Element element = (Element) it.next();
System.out.println("Element: " + element);
}
showChunks(cm);
System.out.println("testAddChunks() OK ");
}
10. Success is not an option...
/**
* Method testFailure.
*/
public void testFailure() {
try {
Message message = new Message(null,true);
fail();
} catch(Exception ex) {
ExceptionHandler.log(ExceptionLevel.ANY,ex);
fail();
}
}
11. What has happened? Well, it failed...
public void testSimple() {
IData data = null;
IFormat format = null;
LinkedList<String> attr = new LinkedList<String>();
attr.add("A");
attr.add("B");
try {
format = new SimpleFormat("A");
data.setAmount(Amount.TEN);
data.setAttributes(attr);
IResult result = format.execute();
System.out.println(result.size());
Iterator iter = result.iterator();
while (iter.hasNext()) {
IResult r = (IResult) iter.next();
System.out.println(r.getMessage());
...
}
catch (Exception e) {
fail();
}
}
12. What has happened? Well, it failed...
public void testSimple() {
IData data = null;
IFormat format = null;
LinkedList<String> attr = new LinkedList<String>();
attr.add("A");
attr.add("B"); data is still null here.
Ready or not, NPE is coming.
try {
format = new SimpleFormat("A");
data.setAmount(Amount.TEN);
data.setAttributes(attr);
IResult result = format.execute();
System.out.println(result.size());
Iterator iter = result.iterator();
while (iter.hasNext()) {
IResult r = (IResult) iter.next();
System.out.println(r.getMessage());
...
}
catch (Exception e) {
fail();
}
}
13. Talk to me
//wait for messages
do {
input = "";
try {
System.out.print(">");
read = System.in.read(buf);
//convert characters to string
input = new String(buf, 0, read - newline.length());
System.out.println(input);
if (input.equals("end") || input.equals("exit")
|| input.equals("stop") || input.equals("quit")) {
System.out.println("Terminating Test please wait...");
System.out.println("******* Test terminated *******");
toStop = true;
}
else {
System.out.println("Commands:" + newline + "'end',
'exit', 'stop' or 'quit' terminates this test ");
}
} catch (Exception e) {
e.printStackTrace();
}
} while (!toStop);
14. Tests are boring – let us autogenerate them!
/** protected void tearDown() throws Exception {
* Generated by JUnitDoclet, a tool provided by // JUnitDoclet begin method testcase.tearDown
* ObjectFab GmbH under LGPL. adapter = null;
* Please see www.junitdoclet.org, www.gnu.org super.tearDown();
* and www.objectfab.de for informations about // JUnitDoclet end method testcase.tearDown
* the tool, the licence and the authors.
*/ public void testMain() throws Exception {
public class AdapterTest // JUnitDoclet begin method testMain
// JUnitDoclet begin extends_implements Adapter.main(new String [] {"ADAPTER"});
extends TestCase // JUnitDoclet end method testMain
// JUnitDoclet end extends_implements }
{
// JUnitDoclet begin class
Adapter adapter = null; /**
// JUnitDoclet end class * JUnitDoclet moves marker to this method, if there is not match
* for them in the regenerated code and if the marker is not empty.
public AdapterTest(String name) { * This way, no test gets lost when regenerating after renaming.
// JUnitDoclet begin method AdapterTest * Method testVault is supposed to be empty.
super(name); */
// JUnitDoclet end method AdapterTest public void testVault() throws Exception {
} // JUnitDoclet begin method testcase.testVault
// JUnitDoclet end method testcase.testVault
public Adapter createInstance() throws Exception { }
// JUnitDoclet begin method testcase.createInstance
return new Adapter(); public static void main(String[] args) {
// JUnitDoclet end method testcase.createInstance // JUnitDoclet begin method testcase.main
} junit.textui.TestRunner.run(AdapterTest.class);
// JUnitDoclet end method testcase.main
protected void setUp() throws Exception { }
// JUnitDoclet begin method testcase.setUp }
super.setUp();
adapter = createInstance();
// JUnitDoclet end method testcase.setUp
}
15. Tests are boring – let us autogenerate them!
public void testSetGetTimestamp() throws Exception {
// JUnitDoclet begin method setTimestamp getTimestamp
java.util.Calendar[] tests = {new GregorianCalendar(), null};
for (int i = 0; i < tests.length; i++) {
adapter.setTimestamp(tests[i]);
assertEquals(tests[i], adapter.getTimestamp());
}
// JUnitDoclet end method setTimestamp getTimestamp
}
public void testSetGetParam() throws Exception {
// JUnitDoclet begin method setParam getParam
String[] tests = {"a", "aaa", "---", "23121313", "", null};
for (int i = 0; i < tests.length; i++) {
adapter.setParam(tests[i]);
assertEquals(tests[i], adapter.getParam());
}
// JUnitDoclet end method setParam getParam
}
17. Conclusions
• Automation!
• Running
• Verification
• Tests are to be written not generated
• You should be informed why your test failed
• Master your tools
…at least learn the basics!
19. Why bother with tests?
• System works as expected
• Changes do not hurt
• Documentation
http://twitter.com/#!/devops_borat
20. Tests help to achieve quality
Not sure when I saw this picture –
probably in GOOS?
21. What happens if we do it wrong?
• Angry clients
• Depressed developers
http://www.joshcanhelp.com
22. When I started out with unit tests, I was
enthralled with the promise of ease and
security that they would bring to my
projects. In practice, however, the
theory of sustainable software through
unit tests started to break down. This
difficulty continued to build up, until I
finally threw my head back in anger and
declared that "Unit Tests have become
more trouble than they are worth."
Llewellyn Falco and Michael Kennedy, Develop Mentor August 09
30. Asking for troubles...
LoggingPropertyConfigurator configurator
= mock(LoggingPropertyConfigurator.class);
BaseServletContextListener baseServletContextListener
= new BaseServletContextListener(configurator);
Should load some
@Test public void shouldLoadConfigProperties() { default config
baseServletContextListener.contextInitialized();
verify(configurator).configure(any(Properties.class));
}
@Test(expected = LoggingInitialisationException.class)
Should load this
public void shouldThrowLoggingException() { specific file
System.setProperty("logConfig", "nonExistingFile");
baseServletContextListener.contextInitialized();
}
32. Mock'em All!
public String getUrl(User user, String timestamp) {
String name=user.getFullName();
String url=baseUrl
+"name="+URLEncoder.encode(name, "UTF-8")
+"×tamp="+timestamp;
return url;
}
public String getUrl(User user) {
Date date=new Date();
Long time=(date.getTime()/1000); //convert ms to seconds
String timestamp=time.toString();
return getUrl(user, timestamp);
}
33. Mock'em All!
public String getUrl(User user, String timestamp) {
String name=user.getFullName();
String url=baseUrl
+"name="+URLEncoder.encode(name, "UTF-8")
+"×tamp="+timestamp;
return url;
}
public String getUrl(User user) {
Date date=new Date();
Long time=(date.getTime()/1000); //convert ms to seconds
String timestamp=time.toString();
return getUrl(user, timestamp);
}
@Test
public void shouldUseTimestampMethod() {
//given
Util spyUtil = spy(util);
//when
spyUtil.getUrl(user);
//then
verify(spyUtil).getUrl(eq(user), anyString());
}
34. Use the front door
@Test
public void shouldAddTimestampToGeneratedUrl() {
//given
util = new ....
TimeProvider timeProvider = mock(TimeProvider.class);
when(timeProvider.getTime()).thenReturn("12345");
util.set(timeProvider);
//when
String url = util.getUrl(user);
//then
assertThat(url).contains("timestamp=12345");
}
36. Use the front door
@Test
public void shouldAddTimeZoneToModelAndView() {
//given
UserFacade userFacade = mock(UserFacade.class);
ModelAndView modelAndView = new ModelAndView();
given(userFacade.getTimezone()).willReturn("timezone X");
//when
new UserDataInterceptor(userFacade)
.postHandle(null, null, null, modelAndView);
//then
assertThat(modelAndView).constains("timezone", "timezone X");
}
38. SRP for tests
A test should have one and only one reason to fail.
P.S. This is definitely a good advice for unit tests, but rather not valid
for integration and end-to-end tests.
39. Testing two things at once
@DataProvider
public Object[][] data() {
return new Object[][] { {"48", true}, {"+48", true},
{"++48", true}, {"+48503", true}, {"+4", false},
{"++4", false}, {"", false},
{null, false}, {" ", false}, };
}
@Test(dataProvider = "data")
public void testQueryVerification(String query, boolean expected) {
assertEquals(expected, FieldVerifier.isValidQuery(query));
}
40. Testing two things at once
@DataProvider
public Object[][] data() {
return new Object[][] { {"48", true}, {"+48", true},
Data
{"++48", true}, {"+48503", true}, {"+4", false},
{"++4", false}, {"", false},
{null, false}, {" ", false}, };
}
@Test(dataProvider = "data") Algorithm / Logic
public void testQueryVerification(String query, boolean expected) {
assertEquals(expected, FieldVerifier.isValidQuery(query));
}
41. Testing two things at once
@DataProvider
public Object[][] data() {
return new Object[][] { {"48", true}, {"+48", true},
{"++48", true}, {"+48503", true}, {"+4", false},
{"++4", false}, {"", false},
{null, false}, {" ", false}, };
}
@Test(dataProvider = "data")
public void testQueryVerification(String query, boolean expected) {
assertEquals(expected, FieldVerifier.isValidQuery(query));
}
testQueryVerification1() {
assertEquals(true, FieldVerifier.isValidQuery(„48”));
}
testQueryVerification2() {
assertEquals(true, FieldVerifier.isValidQuery(„+48”));
}
testQueryVerification3() {
assertEquals(true, FieldVerifier.isValidQuery(„++48”));
}
testQueryVerification4() {
assertEquals(true, FieldVerifier.isValidQuery(„+48503”));
}
...
42. Testing two things at once
@DataProvider
public Object[][] data() {
return new Object[][] { {"48", true}, {"+48", true},
{"++48", true}, {"+48503", true}, {"+4", false},
{"++4", false}, {"", false},
{null, false}, {" ", false}, };
}
@Test(dataProvider = "data")
public void testQueryVerification(String query, boolean expected) {
assertEquals(expected, FieldVerifier.isValidQuery(query));
}
43. Concentrate on one feature
@DataProvider
public Object[][] validQueries() {
return new Object[][] { {"48"}, {"48123"},
{"+48"}, {"++48"}, {"+48503"}};
}
@Test(dataProvider = "validQueries")
public void shouldRecognizeValidQueries(
String validQuery) {
assertTrue(FieldVerifier.isValidQuery(validQuery));
}
@DataProvider
public Object[][] invalidQueries() {
return new Object[][] {
{"+4"}, {"++4"},
{""}, {null}, {" "} };
}
@Test(dataProvider = "invalidQueries")
public void shouldRejectInvalidQueries(
String invalidQuery) {
assertFalse(FieldVerifier.isValidQuery(invalidQuery));
}
46. Happy paths are for wimps
2 + 2
2 + -2
2 + -5
0 + 2
2 + 0
Integer.MAX_VALUE + something
etc.
http://kidskidskids.tumblr.com/post/1145294997
47. Avoiding happy paths
Start with one:
testSum() {
assertEquals(Math.sum(2,2), 4);
}
Do the simplest thing that works:
sum(int x, int y) {
return 4;
}
And then listen to your code.
Because it tells you something. http://kidskidskids.tumblr.com/post/1145294997
48. Avoiding happy paths
sum(int x, int y) {
return 4;
}
You moron!
Your test is so pathetic,
that I can make it pass
by doing such a silly thing.
Try harder!
http://looneytunes09.files.wordpress.com/2010/07/lisa-yell.gif
55. Do not make me learn the API!
server = new MockServer(responseMap, true,
new URL(SERVER_ROOT).getPort(), false);
56. Do not make me learn the API!
server = new MockServer(responseMap, true,
new URL(SERVER_ROOT).getPort(), false);
private static final boolean RESPONSE_IS_A_FILE = true;
private static final boolean NO_SSL = false;
server = new MockServer(responseMap, RESPONSE_IS_A_FILE,
new URL(SERVER_ROOT).getPort(), NO_SSL);
57. Do not make me learn the API!
server = new MockServer(responseMap, true,
new URL(SERVER_ROOT).getPort(), false);
server = new MockServerBuilder()
.withResponse(responseMap)
.withResponseType(FILE)
.withUrl(SERVER_ROOT)
.withoutSsl().create();
66. “should” is better than “test”
• Starting test method names
with “should” steers you in
the right direction.
http://jochopra.blogspot.com/
• “test” prefix makes your test
method a limitless bag
where you throw everything
worth testing
http://www.greenerideal.com/
67. Test methods names are important
@Test
public void testQuery(){
when(q.getResultList()).thenReturn(null);
assertNull(dao.findByQuery(Transaction.class, q, false));
assertNull(dao.findByQuery(Operator.class, q, false));
assertNull(dao.findByQuery(null, null, false));
List result = new LinkedList();
when(q.getResultList()).thenReturn(result);
assertEquals(dao.findByQuery(Transaction.class, q, false), result);
assertEquals(dao.findByQuery(Operator.class, q, false), result);
assertEquals(dao.findByQuery(null, null, false), null);
when(q.getSingleResult()).thenReturn(null);
assertEquals(dao.findByQuery(Transaction.class, q, true).size(), 0);
assertEquals(dao.findByQuery(Operator.class, q, true).size(), 0);
assertEquals(dao.findByQuery(null, null, true), null);
when(q.getSingleResult()).thenReturn(t);
assertSame(dao.findByQuery(Transaction.class, q, true).get(0), t);
when(q.getSingleResult()).thenReturn(o);
assertSame(dao.findByQuery(Operator.class, q, true).get(0), o);
when(q.getSingleResult()).thenReturn(null);
assertSame(dao.findByQuery(null, null, true), null);
}
69. Assertion part is freaking huge
public void shouldPreDeployApplication() {
// given
Artifact artifact = mock(Artifact.class);
when(artifact.getFileName()).thenReturn("war-artifact-2.0.war");
ServerConfiguration config
= new ServerConfiguration(ADDRESS, USER, KEY_FILE, TOMCAT_PATH, TEMP_PATH);
Tomcat tomcat = new Tomcat(HTTP_TOMCAT_URL, config);
String destDir = new File(".").getCanonicalPath() + SLASH + "target" + SLASH;
new File(destDir).mkdirs();
// when
tomcat.preDeploy(artifact, new FakeWar(WAR_FILE_LENGTH));
//then
JSch jsch = new JSch();
jsch.addIdentity(KEY_FILE);
Session session = jsch.getSession(USER, ADDRESS, 22);
session.setConfig("StrictHostKeyChecking", "no");
session.connect();
Channel channel = session.openChannel("sftp");
session.setServerAliveInterval(92000);
channel.connect();
ChannelSftp sftpChannel = (ChannelSftp) channel;
sftpChannel.get(TEMP_PATH + SLASH + artifact.getFileName(), destDir);
sftpChannel.exit();
session.disconnect();
File downloadedFile = new File(destDir, artifact.getFileName());
assertThat(downloadedFile).exists().hasSize(WAR_FILE_LENGTH);
}
70. Just say it
public void shouldPreDeployApplication() {
Artifact artifact = mock(Artifact.class);
when(artifact.getFileName())
.thenReturn(ARTIFACT_FILE_NAME);
ServerConfiguration config
= new ServerConfiguration(ADDRESS, USER,
KEY_FILE, TOMCAT_PATH, TEMP_PATH);
Tomcat tomcat = new Tomcat(HTTP_TOMCAT_URL, config);
tomcat.preDeploy(artifact, new FakeWar(WAR_FILE_LENGTH));
SSHServerAssert.assertThat(ARTIFACT_FILE_NAME)
.existsOnServer(config).hasSize(WAR_FILE_LENGTH);
}
71. Asserting using private methods
@Test
public void testChargeInRetryingState() throws Exception {
// given
TxDTO request = createTxDTO(RequestType.CHARGE);
AndroidTransaction androidTransaction = ...
// when
final TxDTO txDTO = processor.processRequest(request);
// then
assertState(request, androidTransaction,
CHARGED, CHARGE_PENDING, AS_ANDROID_TX_STATE,
ClientMessage.SUCCESS, ResultCode.SUCCESS);
}
73. Matchers vs. private methods
@Test
public void testChargeInRetryingState() throws Exception {
// given
TxDTO request = createTxDTO(CHARGE);
AndroidTransaction androidTransaction = ...
// when
final TxDTO txDTO = processor.processRequest(request);
// then
assertThat(androidTransaction).hasState(CHARGED)
.hasMessage(ClientMessage.SUCCESS)
.hasPreviousState(CHARGE_PENDING)
.hasExtendedState(null);
assertEquals(txDTO.getResultCode(),
ResultCode.SUCCESS);
}
74. What is asserted?
@Test
public void testCompile_32Bit_FakeSourceFile() {
CompilerSupport _32BitCompilerSupport
= CompilerSupportFactory.getDefault32BitCompilerSupport();
testCompile_FakeSourceFile(_32BitCompilerSupport);
}
75. What is asserted?
@Test
public void testCompile_32Bit_FakeSourceFile() {
CompilerSupport _32BitCompilerSupport
= CompilerSupportFactory.getDefault32BitCompilerSupport();
testCompile_FakeSourceFile(_32BitCompilerSupport);
}
private void testCompile_FakeSourceFile(
CompilerSupport compilerSupport) {
compiledFiles
= compilerSupport.compile(new File[] { new File("fake") });
assertThat(compiledFiles, is(emptyArray()));
}
83. Running SUT's code concurrently
@Test(threadPoolSize = 3, invocationCount = 10)
public void testServer() {
// this method will be run in parallel by 3 thread
// 10 invocations (in total)
}
84. Dependent test methods
@Test
public void shouldConnectToDB() {
// verifying that you can
// estabilish a connection with DB
}
@Test(dependsOnMethods = „shouldConnectToDB”)
public void should…() {
// some operations on DB
}
85. Know your tools
• Unit testing framework Additional libraries
Use of temporary file rule
Hamcrest, FEST, Mockito,
Listeners catch-exception, awaitility, …
Concurrency • Build tool
@Before/@After Parallel execution
Parametrized tests CI
Test dependencies • IDE
Templates
Shortcuts
87. What do you really want to test?
@Test
public void shouldAddAUser() {
User user = new User();
userService.save(user);
assertEquals(dao.getNbOfUsers(), 1);
}
88. You wanted to see that the number increased
@Test
public void shouldAddAUser() {
int nb = dao.getNbOfUsers();
User user = new User();
userService.save(user);
assertEquals(dao.getNbOfUsers(), nb + 1);
}
90. Doing it wrong
public void myTest() {
SomeObject obj = new SomeObject(
a, b, c, productCode());
// testing of obj here
}
private String productCode(){
String[] codes =
{"Code A", "Code B",
"Code C", "Code D"};
int index = rand.nextInt(codes.length);
return codes[index];
}
91. The dream of stronger, random-powered tests
public void myTest() {
SomeObject obj = new SomeObject(
randomName(), randomValue(), ....);
// testing of obj here
}
Does it make your test stronger?
92. The dream of stronger, random-powered tests
public void myTest() {
SomeObject obj = new SomeObject(
randomName(), randomValue(), ....);
// testing of obj here
}
Does it make your test stronger?
...or does it only bring confusion?
Test failed
Expected
SomeObject(„a”, „b”, ....)
but got
SomeObject(„*&O*$NdlF”, „#idSLNF”, ....)
94. There is more to it
• Integration / end-to-end tests which are not parametrized
(so they all try to set up jetty on port 8080),
• Tests which should be really unit, but use Spring context
to create objects,
• Tests with a lot of dependencies between them (a
nightmare to maintain!),
• Tests which run slow
• Tests which try to cover the deficiencies of production
code and end up being a total mess,
• Tests which verify methods instead of verifying
responsibilities of a class
• etc., etc.
95. Test-last? No!
• makes people not write tests at all
• makes people do only happy-testing
• tests reflect the implementation
96. Always TDD?
For six or eight hours spread over the next few weeks I
struggled to get the first test written and running. Writing
tests for Eclipse plug-ins is not trivial, so it’s not
surprising I had some trouble. [...] In six or eight hours
of solid programming time, I can still make significant
progress. If I’d just written some stuff and verified it by
hand, I would probably have the final answer to whether
my idea is actually worth money by now. Instead, all I
have is a complicated test that doesn’t work, a pile
of frustration, eight fewer hours in my life, and the
motivation to write another essay.
Kent Beck, Just Ship, Baby
97. Treat tests as the first class citizens
• do it everyday or forget about it • make tests readable using matchers,
• use the right tool for the job builders and good names
• and learn to use it! • test behaviour not methods
• do not live with broken windows • be pragmatic about the tests you write
• respect KISS, SRP, DRY (?) • TDD always?
• write good code, and you will also write • what is the best way to test it?
good tests unit/integration/end-to-end ?
• or rather write good tests and you • automate!
will get good code for free • always concentrate on what is worth
• code review your tests testing
• do more than happy path testing • ask yourself questions like: 'is it
really important that X should send
• do not make the reader learn the API,
message Y to Z?'
make it obvious
• use the front door – state testing before
• bad names lead to bad tests
interaction testing (mockc)
99. Thank you!
Thank you for watching these
slides! You can learn more about
wirting high quality tests by
reading my book – „Practical Unit
Testing with TestNG and
Mockito”.
You can also participate in
writing of my
new (free!) e-book devoted to
bad and good tests.