Software testing
COMP9321
Julien Ponge (julienp@cse.unsw.edu.au)
Invited lecture: september 7th 2006
Computer School of Engineering
The University of New South Wales, Sydney, Australia

• We will see:
• why tests are important
• what habits you should get rid of
• how to create tests
• tips to make your life easier with tests

Introduction
3

Testing... but what?
• Unit testing
• Functional testing
• Conformance testing
• Performance testing
4

Unit testing
• Typically classes
• Isolate as much as possible
• Self-contained and automated
• “Developers love writing tests”
• Metrics: passed tests and code coverage
5

What to expect?
• Less stupid bugs + test more often
• Prevent regression bugs
• Reveal bad design early
• Easier deep refactorings
6

The cost of tests
• Requires more 4,8
efforts in the early
stages 4
3,2
• A code base
without tests can 2,4
become too difficult 1,6
to manage as it
grows 0,8
0 0,4 0,8 1,2 1,6 2 2,4 2,8 3,2
• Test suites help
when scaling! 7

Old-school testing
8

Typical approach
• Write a main method (console or GUI)
• Perform a few checkings and/or output
values (System.out.println(...))
• You may feed data manually and/or
interactively
• Manually check if it works or not
9

Example 1
import my.package.NiceClass;
public class OldSchoolTesting {
public static void main(String[] args) {
NiceClass nc = new NiceClass();
System.out.println(\"Hey!\");
nc.doSomething();
System.out.println(\"It didn't crash!!! (yet)\");
System.out.println(nc.getSomeValue());
}
}
10

Example 2
public class MyGreatClass {
// (...)
public void doSomethingGreat() {
String str = grabItSomewhereElse();
System.out.println(\"=> str is \" + str);
if (!str.equals(ONE_CONSTANT)) {
cry();
goToBed();
} else {
enjoy();
haveOneBeerOrTwo();
}
}
// (...)
} 11

Example 3
12

Problems
• Not automated:
• you check
• you feed the data
• you need to run it
• System.out.println + messy outputs
• Most of the time... you throw it away
when it works!
13

To address this...
• We will see an automated unit testing
framework: JUnit
• We will see how to properly output
data when need be: Logging
14

JUnit
15

JUnit
• Most famous for Java, many extensions
• Very stupid set of classes simple
• Composite pattern for test cases & test
suites
• Primarily for unit testing, but functional
testing can be conducted as well
• Derivatives: NUnit, PyUnit, CppUnit, ...
16

Running JUnit
• It reports:
• passed & failed tests
• errors (exceptions)
• It provides a text-mode and a Swing-
based runners
• Most IDEs embed JUnit and display a
progress bar (green / red)
17

How it works
• Base class is TestCase
• Start testing methods names with test
• Override setUp() & tearDown() if
initialization and cleanups are needed
• Check values with assertEquals,
assertNotSame, assertNull,
assertNotNull, ...
18

Basic test case canvas
• Create a class that extends TestCase, and
name it as MyClassTest for MyClass
• Create a test method testFoo for each
non-trivial method foo
• Instantiate data, invoke methods
• Use JUnit assertions
19

Example 1
public class DifferenceOperatorTest extends TestCase {
private DifferenceOperator operator = new DifferenceOperator(new
BusinessProtocolFactoryImpl());
public void testApply() throws DocumentException {
BusinessProtocol p1 = TestUtils.loadProtocol(\"difference/p1.wsprotocol\");
BusinessProtocol p2 = TestUtils.loadProtocol(\"difference/p2.wsprotocol\");
BusinessProtocol result = operator.apply(p2, p1);
BusinessProtocol expected = TestUtils.loadProtocol(\"difference/p2-diff-
p1.wsprotocol\");
TestCase.assertEquals(expected, result);
}
public void testComputeComplement() throws DocumentException {
BusinessProtocol p1 = TestUtils.loadProtocol(\"difference/p1.wsprotocol\");
BusinessProtocol expected = TestUtils.loadProtocol(\"difference/compl-p1.wsprotocol\");
BusinessProtocol result = operator.computeComplement(p1);
TestCase.assertEquals(expected, result);
}
}
20

Example 2
public class OperationImplTest extends TestCase {
public void testEqualsObject() {
State s1 = new StateImpl(\"s1\", false);
State s2 = new StateImpl(\"s2\", false);
State s3 = new StateImpl(\"s3\", false);
State s4 = new StateImpl(\"s4\", false);
Message m1 = new MessageImpl(\"a\", Polarity.POSITIVE);
Message m2 = new MessageImpl(\"a\", Polarity.NEGATIVE);
OperationImpl o1 = new OperationImpl(\"T1\", s1, s2, m1);
OperationImpl o2 = new OperationImpl(\"T2\", s3, s4, m2);
TestCase.assertEquals(o1, o1);
TestCase.assertNotSame(o1, o2);
}
public void testToString() {
State s1 = new StateImpl(\"s1\", false);
State s2 = new StateImpl(\"s2\", false);
Message m1 = new MessageImpl(\"a\", Polarity.POSITIVE);
OperationImpl o1 = new OperationImpl(\"T1\", s1, s2, m1);
TestCase.assertEquals(\"T1: ((s1),[a](+),(s2),explicit)\", o1.toString());
}
}
21

What your tests should do
• Test both good and bad cases
• Challenge your code: push it with
misuses and stupid values
• Make sure you get a good coverage:
• use dedicated tools
• Eclipse since version 3.2
22

Separate tests
from the rest
of the code!
23

Build integration
• Apache Ant is similar to Make, but for
Java
• There is a Ant task
• Apache Maven is project-oriented, and
expects JUnit tests
• Tests must pass for many Maven goals
24

Ant + JUnit
<target name=\"tests\" depends=\"build,build.tests\">
<junit printsummary=\"yes\" fork=\"yes\"
haltonfailure=\"yes\">
<formatter type=\"plain\"/>
<batchtest fork=\"yes\" todir=\"${reports.tests}\">
<fileset dir=\"${src.tests}\">
<include name=\"**/*Test*.java\"/>
<exclude name=\"**/AllTests.java\"/>
</fileset>
</batchtest>
</junit>
</target>
25

IDE integration
keep the
bar green!

Logging
27

Why?
• System.out and System.err are limited
• Not all events are of the same
importance: informations, debugging
messages, errors, ...
• Activate some messages on-demand,
output to console, network, dbs, files, ...
• Useful for server-side applications, but
also standalone applications or libraries
28

Logging in Java
• Log4J is a popular library
• Java 1.4+ has java.util.logging
• Apache commons-logging is a popular
choice to abstract and dynamically
discover the logging API
• Very minor impact on performances
• Never throw messages, keep them!
29

Logging levels
• Fatal : critical errors
• Error : errors that can be recovered
• Warn : warnings
• Info : general informations (start,
stop, ...)
• Debug : only-useful for debugging
• Trace : only for critical debugging
30

Log4J
outputs
root logger
formatters
logger.A
logger.B
logger.B.a
filters
31

Sample Log4J config
log4.properties
# Root logger
log4j.rootLogger=INFO, consoleAppender
# A console appender
log4j.appender.consoleAppender=org.apache.log4j.ConsoleAppender
log4j.appender.consoleAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.consoleAppender.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
32

Example
public class OperationImpl implements Operation {
(...)
/** Logger. */
private static Log log = LogFactory.getLog(OperationImpl.class);
(...)
public void setMessage(Message message) {
Message oldMessage = this.message;
this.message = message;
listeners.firePropertyChange(MESSAGE_PROPERTY_CHANGE, oldMessage, message);
if (log.isDebugEnabled()) {
log.debug(\"Changing the message from \" + oldMessage + \" to \" + message);
}
}
(...)
}
33

Beyond just tests
34

Life of a tests suite
• A test suite is like good wine: it gets better
over time!
• A good example is to reproduce bugs as
they get identified: you will prevent
regressions
35

Handling bugs
(ah ah I’m back) (I am a bug)
Test case
Tests suite
36

Tests are also...
• Useful to enforce API contracts
• An excellent API usage documentation
• A solid test suite allows deep
refactorings: it acts as a safeguard
against regressions
• Tests reveal bad design early: it is
difficult to test, then you need to
refactor!
37

Self-containment
38

Self-containment
• Your test suites should be as self-
contained as possible
• Avoid environment setup requirements
(databases servers, application
servers, ...)
• Eases automation in any context
39

Examples
data access relational
objects database
HTTP HTTP
client server
40

Self-contained DB
public abstract class DAOTestBase extends TestCase {
protected Map<String, String> queries;
protected QueryRunner runner;
base class
protected Connection connection;
public DAOTestBase() throws IOException {
super();
DbUtils.loadDriver(\"org.hsqldb.jdbcDriver\");
runner = new QueryRunner();
queries = QueryLoader.instance().load(DBConstants.INIT_DB_SQL_RESOURCE);
}
protected void setUp() throws Exception {
connection = DriverManager.getConnection(\"jdbc:hsqldb:mem:CustomerDAOTest\", \"sa\", \"\");
runner.update(connection, queries.get(\"create.customers\"));
runner.update(connection, queries.get(\"create.transactions\"));
runner.update(connection, queries.get(\"data.1\"));
runner.update(connection, queries.get(\"data.2\"));
runner.update(connection, queries.get(\"data.3\"));
runner.update(connection, queries.get(\"data.4\"));
runner.update(connection, queries.get(\"data.5\"));
}
protected void tearDown() throws Exception {
runner.update(connection, queries.get(\"drop.transactions\"));
runner.update(connection, queries.get(\"drop.customers\"));
DbUtils.closeQuietly(connection);
}
41
}

Self-contained DB
public class CustomerDAOTest extends DAOTestBase {
public void testSaveAndUpdate() throws IOException, SQLException {
CustomerDAO dao = new CustomerDAO(connection);
Customer customer = new Customer();
test class
customer.setName(\"Bernard Minet\");
customer.setEmail(\"bernard@les-muscles.com\");
customer.setAddress(\"France\");
dao.save(customer);
Customer readCustomer = dao.loadCustomerByName(\"Bernard Minet\");
assertEquals(customer, readCustomer);
customer.setName(\"Framboisier\");
customer.setEmail(\"framboisier@les-muscles.com\");
customer.setAddress(\"Paris, France\");
customer.setUrl(\"http://blog.les-muscles.com/framboisier/\");
customer.creditBalance(50.0);
dao.update(customer);
readCustomer = dao.loadCustomerByName(\"Bernard Minet\");
assertNull(readCustomer);
readCustomer = dao.loadCustomerByName(\"Framboisier\");
assertNotNull(readCustomer);
assertEquals(customer, readCustomer);
} 42

Self-contained HTTP
public abstract class AbstractHttpSoapTransportTest extends TestCase {
private Server server;
base class
private Servlet servlet;
private ServletHandler servletHandler;
protected AbstractHttpSoapTransportTest(Servlet servlet) {
this.servlet = servlet;
server = new Server();
SelectChannelConnector connector = new SelectChannelConnector();
connector.setPort(8085);
server.setConnectors(new Connector[]{connector});
servletHandler = new ServletHandler();
servletHandler.addServletWithMapping(new ServletHolder(servlet), \"/dtc\");
server.setHandler(servletHandler);
}
protected void setUp() throws Exception {
server.start();
}
protected void tearDown() throws Exception {
server.stop();
}
} 43

Self-contained HTTP
public class DummyServlet extends HttpServlet {
protected void doPost(HttpServletRequest request, HttpServletResponse
httpServletResponse) throws ServletException, IOException {
InputStream in = request.getInputStream();
OutputStream out = httpServletResponse.getOutputStream();
byte[] buffer = new byte[1024];
servlet for
int nbytes = 0;
while ((nbytes = in.read(buffer, 0, 1024)) != -1) {
out.write(buffer, 0, nbytes);
testing
}
}
protected void doGet(HttpServletRequest request, HttpServletResponse
httpServletResponse) throws ServletException, IOException {
InputStream in = SoapMessageTest.class.getResourceAsStream(\"simple-soap-
message.xml\");
OutputStream out = httpServletResponse.getOutputStream();
byte[] buffer = new byte[1024];
int nbytes = 0;
while ((nbytes = in.read(buffer, 0, 1024)) != -1) {
out.write(buffer, 0, nbytes);
}
}
} 44

Self-contained HTTP
public class HttpSoapTransportWithDummyServletTest extends
AbstractHttpSoapTransportTest {
private NodeComparator nodeComparator = new NodeComparator();
public HttpSoapTransportWithDummyServletTest() {
test class
super(new DummyServlet());
}
public void testSend() throws DocumentException, IOException,
XmlPullParserException, SoapTransportException {
HttpSoapTransport soapTransport = new HttpSoapTransport();
SoapMessage message = SoapMessageUtils.readSoapMessageFromStream
(SoapMessageTest.class
.getResourceAsStream(\"simple-soap-message.xml\"));
SoapMessage response = soapTransport.send(\"http://localhost:8085/dtc\",
message, null);
assertEquals(0, nodeComparator.compare(message, response));
}
}
45

HttpUnit: testing webapps
46

HttpUnit
• Functional testing
• Works like a web browser / HTTP
client (cookies, form submissions, ...)
• Embeds a JavaScript engine (Rhino)
• JUnit integration
• Provides ServletUnit to unit-test servlets
in isolation of servlet containers
47

Main classes
• WebConversation: stateful client session
• WebRequest: a single GET or POST
HTTP request
• WebResponse: HTTP response
• WebForm, WebLink, (...): many helper
classes to access the pages content
48

Example
public void testGoodLogin() throws Exception {
WebConversation conversation = new WebConversation();
WebRequest request = new GetMethodWebRequest(
\"http://www.meterware.com/servlet/TopSecret\");
WebResponse response = conversation.getResponse(request);
WebForm loginForm = response.getForms()[0];
request = loginForm.getRequest();
request.setParameter( \"name\", \"master\" );
response = conversation.getResponse( request );
assertTrue( \"Login not accepted\",
response.getText().indexOf( \"You made it!\" ) != -1 );
assertEquals( \"Page title\", \"Top Secret\", response.getTitle() );
}
49 http://fb2.hu/x10/Articles/HttpUnit.html

(from the cookbook)
WebConversation wc = new WebConversation();
WebResponse resp = wc.getResponse( \"http://www.httpunit.org/doc/cookbook.html\" );
WebLink link = resp.getLinkWith( \"response\" );
link.click();
WebResponse jdoc = wc.getCurrentPage();
WebTable table = resp.getTables()[0];
assertEquals( \"rows\", 4, table.getRowCount() );
assertEquals( \"columns\", 3, table.getColumnCount() );
assertEquals( \"links\", 1, table.getTableCell( 0, 2 ).getLinks().length );
WebForm form = resp.getForms()[0];
assertEquals( \"La Cerentolla\", form.getParameterValue( \"Name\" ) );
assertEquals( \"Chinese\", form.getParameterValue( \"Food\" ) );
assertEquals( \"Manayunk\", form.getParameterValue( \"Location\" ) );
assertEquals( \"on\", form.getParameterValue( \"CreditCard\" ) );
form.setParameter( \"Food\", \"Italian\" );
form.removeParameter( \"CreditCard\" );
form.submit();
50

ServletUnit example
ServletRunner sr = new ServletRunner();
sr.registerServlet( \"myServlet\", StatefulServlet.class.getName() );
ServletUnitClient sc = sr.newClient();
WebRequest request = new PostMethodWebRequest(\"http://test.meterware.com/myServlet\");
request.setParameter( \"color\", \"red\" );
WebResponse response = sc.getResponse( request );
assertNotNull( \"No response received\", response );
assertEquals( \"content type\", \"text/plain\", response.getContentType() );
assertEquals( \"requested resource\", \"You selected red\", response.getText() );
public class StatefulServlet extends HttpServlet {
protected void doPost(HttpServletRequest req, HttpServletResponse resp)
throws ServletException,IOException {
resp.setContentType( \"text/plain\" );
writeSelectMessage( req.getParameter( \"color\" ), resp.getWriter() );
setColor( req, req.getParameter( \"color\" ) );
}
protected void writeSelectMessage(String color, PrintWriter pw)
throws IOException {
pw.print( \"You selected \" + color );
pw.close();
}
void setColor( HttpServletRequest req, String color )
throws ServletException {
req.getSession().setAttribute( \"color\", color );
}
51
}

Tests-friendly patterns
52

Problem: dependencies
AdressBook PersonDAO
+ loadPerson(...)
+ savePerson(...)
53

Hard-dependencies
• Isolating the class to test is difficult
and/or impossible
• Failures may be induced by a
dependency
• More generally, it reveals potential code
evolution problems: strong coupling
54

Depending on interfaces
<<interface>>
AdressBook IPersonDAO
+ loadPerson(...)
+ savePerson(...)
DBPersonDAO FilePersonDAO MockPersonDAO
55

Interfaces dependencies
• Interfaces introduce loose coupling
• At execution time: use the real class
• At test time: use a mock class
• Future evolution is easier
• BTW, don’t over-use interfaces!
56

Mock objects
• Give a simple behavior: fixed values,
random values, values passed by
methods of the mock class, ...
• They allow you to test even if the real
class hasn’t been created yet
• EasyMock, JMockIt allow to generate
mocks on-the-fly
57

Object.equals()
public boolean equals(Object obj)
{
if (obj instanceof ComparisonNode)
{
ComparisonNode other = (ComparisonNode) obj;
return symbol.equals(other.symbol)
&& leftChild.equals(other.leftChild)
&& rightChild.equals(other.rightChild);
}
return false;
important for
}
TestCase.assertXXX(...)
58

Dependency injection
• 2 types of injection: setter-based &
constructor-based
• Constructor injection is preferable, and
make your objects ready right after
their instanciation
• Clarifies classes dependencies
• Principle used in inversion of control
frameworks: PicoContainer, Spring, ...
59

Example
private IConstraintNode leftChild;
private IConstraintNode rightChild;
private String symbol;
public ComparisonNode(String symbol, VariableNode var, ConstantNode cst)
{
this(var, cst, symbol);
}
public ComparisonNode(String symbol, ConstantNode cst, VariableNode var)
{
this(cst, var, symbol);
}
protected ComparisonNode(IConstraintNode leftChild, IConstraintNode rightChild,
String symbol)
{
super();
this.symbol = symbol;
this.leftChild = leftChild;
this.rightChild = rightChild;
}
60

A few tips
61

Do
• Try to really write your tests first
• Test as soon as you create a new class
• Test often
• Start with sensible simple scenarios
• Load data from resources
(Class.getResourceAsStream(“/path/to”))
• Check for code coverage
62

Don’t
• Write overly exhaustive test cases
• Write tests for trivial methods (getters,
setters, ...) even if it lowers coverage
• Catch exceptions, unless you actually
test that
• Fix bugs without first reproducing them
in a test case
63

Traversing object graphs
• Sometimes you will need to walk
through deep object graphs to test
values
• This is really boring
• JXPath uses XPath on objects, and
replaces iterations/loops by requests
64

JXPath example
Iterator it;
String str;
JXPathContext ctx = JXPathContext.newContext(bp2);
TestCase.assertEquals(\"s0\", bp2.getInitialState().getName());
it = ctx.iterate(\"finalStates/name\");
TestCase.assertTrue(it.hasNext());
TestCase.assertEquals(\"s1\", (String) it.next());
TestCase.assertFalse(it.hasNext());
it = ctx.iterate(\"states[name='s0']/successors/name\");
TestCase.assertTrue(it.hasNext());
TestCase.assertEquals(\"s1\", (String) it.next());
TestCase.assertEquals(\"s0\", (String) it.next());
TestCase.assertFalse(it.hasNext());
(...)
Operation toRemove = (Operation) ctx.getValue(\"operations[message/name='a']\");
bp2.removeOperation(toRemove);
List remainingOps = (List) ctx.getValue(\"states[name='s0']/incomingOperations\");
TestCase.assertTrue(remainingOps.size() == 1);
65

The case of ‘old’ software
• Test cases can be added to existing
applications
• In many situations the design can make
it difficult to test / isolate...
• You cannot break existing APIs
• Don’t expect to introduce tests cases for
100% of the code base...
• It will still allow you to spot new bugs!
66

Continuous integration
67

Problem
• Developers work on different portions
of the code base
• Traditional integration may not be
frequent enough (~ weekly)
• It can be difficult to find who, when and
how the build was broken
• Continuous integration aims at
detecting problems in nearly real-time
68

C.I. environment
4 email
developers
instant messaging
1 continuous integration
changeset
(Continuum, ...)
3
execute
notification via hooks
2
source code tests suite
repository
69
(SVN, CVS, ...)

• Demo:
• project using Maven2
• Apache Continuum configuration
• launch the tests from Continuum
70

Thanks!
71