Software Testing - Invited Lecture at UNSW Sydney

2,275 views

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,275
On SlideShare
0
From Embeds
0
Number of Embeds
72
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Software Testing - Invited Lecture at UNSW Sydney

  1. 1. Software testing COMP9321 Julien Ponge (julienp@cse.unsw.edu.au) Invited lecture: september 7th 2006 Computer School of Engineering The University of New South Wales, Sydney, Australia
  2. 2. • We will see: • why tests are important • what habits you should get rid of • how to create tests • tips to make your life easier with tests
  3. 3. Introduction 3
  4. 4. Testing... but what? • Unit testing • Functional testing • Conformance testing • Performance testing 4
  5. 5. Unit testing • Typically classes • Isolate as much as possible • Self-contained and automated • “Developers love writing tests” • Metrics: passed tests and code coverage 5
  6. 6. What to expect? • Less stupid bugs + test more often • Prevent regression bugs • Reveal bad design early • Easier deep refactorings 6
  7. 7. The cost of tests • Requires more 4,8 efforts in the early stages 4 3,2 • A code base without tests can 2,4 become too difficult 1,6 to manage as it grows 0,8 0 0,4 0,8 1,2 1,6 2 2,4 2,8 3,2 • Test suites help when scaling! 7
  8. 8. Old-school testing 8
  9. 9. Typical approach • Write a main method (console or GUI) • Perform a few checkings and/or output values (System.out.println(...)) • You may feed data manually and/or interactively • Manually check if it works or not 9
  10. 10. Example 1 import my.package.NiceClass; public class OldSchoolTesting { public static void main(String[] args) { NiceClass nc = new NiceClass(); System.out.println(quot;Hey!quot;); nc.doSomething(); System.out.println(quot;It didn't crash!!! (yet)quot;); System.out.println(nc.getSomeValue()); } } 10
  11. 11. Example 2 public class MyGreatClass { // (...) public void doSomethingGreat() { String str = grabItSomewhereElse(); System.out.println(quot;=> str is quot; + str); if (!str.equals(ONE_CONSTANT)) { cry(); goToBed(); } else { enjoy(); haveOneBeerOrTwo(); } } // (...) } 11
  12. 12. Example 3 12
  13. 13. Problems • Not automated: • you check • you feed the data • you need to run it • System.out.println + messy outputs • Most of the time... you throw it away when it works! 13
  14. 14. To address this... • We will see an automated unit testing framework: JUnit • We will see how to properly output data when need be: Logging 14
  15. 15. JUnit 15
  16. 16. JUnit • Most famous for Java, many extensions • Very stupid set of classes simple • Composite pattern for test cases & test suites • Primarily for unit testing, but functional testing can be conducted as well • Derivatives: NUnit, PyUnit, CppUnit, ... 16
  17. 17. Running JUnit • It reports: • passed & failed tests • errors (exceptions) • It provides a text-mode and a Swing- based runners • Most IDEs embed JUnit and display a progress bar (green / red) 17
  18. 18. How it works • Base class is TestCase • Start testing methods names with test • Override setUp() & tearDown() if initialization and cleanups are needed • Check values with assertEquals, assertNotSame, assertNull, assertNotNull, ... 18
  19. 19. Basic test case canvas • Create a class that extends TestCase, and name it as MyClassTest for MyClass • Create a test method testFoo for each non-trivial method foo • Instantiate data, invoke methods • Use JUnit assertions 19
  20. 20. Example 1 public class DifferenceOperatorTest extends TestCase { private DifferenceOperator operator = new DifferenceOperator(new BusinessProtocolFactoryImpl()); public void testApply() throws DocumentException { BusinessProtocol p1 = TestUtils.loadProtocol(quot;difference/p1.wsprotocolquot;); BusinessProtocol p2 = TestUtils.loadProtocol(quot;difference/p2.wsprotocolquot;); BusinessProtocol result = operator.apply(p2, p1); BusinessProtocol expected = TestUtils.loadProtocol(quot;difference/p2-diff- p1.wsprotocolquot;); TestCase.assertEquals(expected, result); } public void testComputeComplement() throws DocumentException { BusinessProtocol p1 = TestUtils.loadProtocol(quot;difference/p1.wsprotocolquot;); BusinessProtocol expected = TestUtils.loadProtocol(quot;difference/compl-p1.wsprotocolquot;); BusinessProtocol result = operator.computeComplement(p1); TestCase.assertEquals(expected, result); } } 20
  21. 21. Example 2 public class OperationImplTest extends TestCase { public void testEqualsObject() { State s1 = new StateImpl(quot;s1quot;, false); State s2 = new StateImpl(quot;s2quot;, false); State s3 = new StateImpl(quot;s3quot;, false); State s4 = new StateImpl(quot;s4quot;, false); Message m1 = new MessageImpl(quot;aquot;, Polarity.POSITIVE); Message m2 = new MessageImpl(quot;aquot;, Polarity.NEGATIVE); OperationImpl o1 = new OperationImpl(quot;T1quot;, s1, s2, m1); OperationImpl o2 = new OperationImpl(quot;T2quot;, s3, s4, m2); TestCase.assertEquals(o1, o1); TestCase.assertNotSame(o1, o2); } public void testToString() { State s1 = new StateImpl(quot;s1quot;, false); State s2 = new StateImpl(quot;s2quot;, false); Message m1 = new MessageImpl(quot;aquot;, Polarity.POSITIVE); OperationImpl o1 = new OperationImpl(quot;T1quot;, s1, s2, m1); TestCase.assertEquals(quot;T1: ((s1),[a](+),(s2),explicit)quot;, o1.toString()); } } 21
  22. 22. What your tests should do • Test both good and bad cases • Challenge your code: push it with misuses and stupid values • Make sure you get a good coverage: • use dedicated tools • Eclipse since version 3.2 22
  23. 23. Separate tests from the rest of the code! 23
  24. 24. Build integration • Apache Ant is similar to Make, but for Java • There is a Ant task • Apache Maven is project-oriented, and expects JUnit tests • Tests must pass for many Maven goals 24
  25. 25. Ant + JUnit <target name=quot;testsquot; depends=quot;build,build.testsquot;> <junit printsummary=quot;yesquot; fork=quot;yesquot; haltonfailure=quot;yesquot;> <formatter type=quot;plainquot;/> <batchtest fork=quot;yesquot; todir=quot;${reports.tests}quot;> <fileset dir=quot;${src.tests}quot;> <include name=quot;**/*Test*.javaquot;/> <exclude name=quot;**/AllTests.javaquot;/> </fileset> </batchtest> </junit> </target> 25
  26. 26. IDE integration keep the bar green!
  27. 27. Logging 27
  28. 28. Why? • System.out and System.err are limited • Not all events are of the same importance: informations, debugging messages, errors, ... • Activate some messages on-demand, output to console, network, dbs, files, ... • Useful for server-side applications, but also standalone applications or libraries 28
  29. 29. Logging in Java • Log4J is a popular library • Java 1.4+ has java.util.logging • Apache commons-logging is a popular choice to abstract and dynamically discover the logging API • Very minor impact on performances • Never throw messages, keep them! 29
  30. 30. Logging levels • Fatal : critical errors • Error : errors that can be recovered • Warn : warnings • Info : general informations (start, stop, ...) • Debug : only-useful for debugging • Trace : only for critical debugging 30
  31. 31. Log4J outputs root logger formatters logger.A logger.B logger.B.a filters 31
  32. 32. Sample Log4J config log4.properties # Root logger log4j.rootLogger=INFO, consoleAppender # A console appender log4j.appender.consoleAppender=org.apache.log4j.ConsoleAppender log4j.appender.consoleAppender.layout=org.apache.log4j.PatternLayout log4j.appender.consoleAppender.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n 32
  33. 33. Example public class OperationImpl implements Operation { (...) /** Logger. */ private static Log log = LogFactory.getLog(OperationImpl.class); (...) public void setMessage(Message message) { Message oldMessage = this.message; this.message = message; listeners.firePropertyChange(MESSAGE_PROPERTY_CHANGE, oldMessage, message); if (log.isDebugEnabled()) { log.debug(quot;Changing the message from quot; + oldMessage + quot; to quot; + message); } } (...) } 33
  34. 34. Beyond just tests 34
  35. 35. Life of a tests suite • A test suite is like good wine: it gets better over time! • A good example is to reproduce bugs as they get identified: you will prevent regressions 35
  36. 36. Handling bugs (ah ah I’m back) (I am a bug) Test case Tests suite 36
  37. 37. Tests are also... • Useful to enforce API contracts • An excellent API usage documentation • A solid test suite allows deep refactorings: it acts as a safeguard against regressions • Tests reveal bad design early: it is difficult to test, then you need to refactor! 37
  38. 38. Self-containment 38
  39. 39. Self-containment • Your test suites should be as self- contained as possible • Avoid environment setup requirements (databases servers, application servers, ...) • Eases automation in any context 39
  40. 40. Examples data access relational objects database HTTP HTTP client server 40
  41. 41. Self-contained DB public abstract class DAOTestBase extends TestCase { protected Map<String, String> queries; protected QueryRunner runner; base class protected Connection connection; public DAOTestBase() throws IOException { super(); DbUtils.loadDriver(quot;org.hsqldb.jdbcDriverquot;); runner = new QueryRunner(); queries = QueryLoader.instance().load(DBConstants.INIT_DB_SQL_RESOURCE); } protected void setUp() throws Exception { connection = DriverManager.getConnection(quot;jdbc:hsqldb:mem:CustomerDAOTestquot;, quot;saquot;, quot;quot;); runner.update(connection, queries.get(quot;create.customersquot;)); runner.update(connection, queries.get(quot;create.transactionsquot;)); runner.update(connection, queries.get(quot;data.1quot;)); runner.update(connection, queries.get(quot;data.2quot;)); runner.update(connection, queries.get(quot;data.3quot;)); runner.update(connection, queries.get(quot;data.4quot;)); runner.update(connection, queries.get(quot;data.5quot;)); } protected void tearDown() throws Exception { runner.update(connection, queries.get(quot;drop.transactionsquot;)); runner.update(connection, queries.get(quot;drop.customersquot;)); DbUtils.closeQuietly(connection); } 41 }
  42. 42. Self-contained DB public class CustomerDAOTest extends DAOTestBase { public void testSaveAndUpdate() throws IOException, SQLException { CustomerDAO dao = new CustomerDAO(connection); Customer customer = new Customer(); test class customer.setName(quot;Bernard Minetquot;); customer.setEmail(quot;bernard@les-muscles.comquot;); customer.setAddress(quot;Francequot;); dao.save(customer); Customer readCustomer = dao.loadCustomerByName(quot;Bernard Minetquot;); assertEquals(customer, readCustomer); customer.setName(quot;Framboisierquot;); customer.setEmail(quot;framboisier@les-muscles.comquot;); customer.setAddress(quot;Paris, Francequot;); customer.setUrl(quot;http://blog.les-muscles.com/framboisier/quot;); customer.creditBalance(50.0); dao.update(customer); readCustomer = dao.loadCustomerByName(quot;Bernard Minetquot;); assertNull(readCustomer); readCustomer = dao.loadCustomerByName(quot;Framboisierquot;); assertNotNull(readCustomer); assertEquals(customer, readCustomer); } 42
  43. 43. Self-contained HTTP public abstract class AbstractHttpSoapTransportTest extends TestCase { private Server server; base class private Servlet servlet; private ServletHandler servletHandler; protected AbstractHttpSoapTransportTest(Servlet servlet) { this.servlet = servlet; server = new Server(); SelectChannelConnector connector = new SelectChannelConnector(); connector.setPort(8085); server.setConnectors(new Connector[]{connector}); servletHandler = new ServletHandler(); servletHandler.addServletWithMapping(new ServletHolder(servlet), quot;/dtcquot;); server.setHandler(servletHandler); } protected void setUp() throws Exception { server.start(); } protected void tearDown() throws Exception { server.stop(); } } 43
  44. 44. Self-contained HTTP public class DummyServlet extends HttpServlet { protected void doPost(HttpServletRequest request, HttpServletResponse httpServletResponse) throws ServletException, IOException { InputStream in = request.getInputStream(); OutputStream out = httpServletResponse.getOutputStream(); byte[] buffer = new byte[1024]; servlet for int nbytes = 0; while ((nbytes = in.read(buffer, 0, 1024)) != -1) { out.write(buffer, 0, nbytes); testing } } protected void doGet(HttpServletRequest request, HttpServletResponse httpServletResponse) throws ServletException, IOException { InputStream in = SoapMessageTest.class.getResourceAsStream(quot;simple-soap- message.xmlquot;); OutputStream out = httpServletResponse.getOutputStream(); byte[] buffer = new byte[1024]; int nbytes = 0; while ((nbytes = in.read(buffer, 0, 1024)) != -1) { out.write(buffer, 0, nbytes); } } } 44
  45. 45. Self-contained HTTP public class HttpSoapTransportWithDummyServletTest extends AbstractHttpSoapTransportTest { private NodeComparator nodeComparator = new NodeComparator(); public HttpSoapTransportWithDummyServletTest() { test class super(new DummyServlet()); } public void testSend() throws DocumentException, IOException, XmlPullParserException, SoapTransportException { HttpSoapTransport soapTransport = new HttpSoapTransport(); SoapMessage message = SoapMessageUtils.readSoapMessageFromStream (SoapMessageTest.class .getResourceAsStream(quot;simple-soap-message.xmlquot;)); SoapMessage response = soapTransport.send(quot;http://localhost:8085/dtcquot;, message, null); assertEquals(0, nodeComparator.compare(message, response)); } } 45
  46. 46. HttpUnit: testing webapps 46
  47. 47. HttpUnit • Functional testing • Works like a web browser / HTTP client (cookies, form submissions, ...) • Embeds a JavaScript engine (Rhino) • JUnit integration • Provides ServletUnit to unit-test servlets in isolation of servlet containers 47
  48. 48. Main classes • WebConversation: stateful client session • WebRequest: a single GET or POST HTTP request • WebResponse: HTTP response • WebForm, WebLink, (...): many helper classes to access the pages content 48
  49. 49. Example public void testGoodLogin() throws Exception { WebConversation conversation = new WebConversation(); WebRequest request = new GetMethodWebRequest( quot;http://www.meterware.com/servlet/TopSecretquot;); WebResponse response = conversation.getResponse(request); WebForm loginForm = response.getForms()[0]; request = loginForm.getRequest(); request.setParameter( quot;namequot;, quot;masterquot; ); response = conversation.getResponse( request ); assertTrue( quot;Login not acceptedquot;, response.getText().indexOf( quot;You made it!quot; ) != -1 ); assertEquals( quot;Page titlequot;, quot;Top Secretquot;, response.getTitle() ); } 49 http://fb2.hu/x10/Articles/HttpUnit.html
  50. 50. (from the cookbook) WebConversation wc = new WebConversation(); WebResponse resp = wc.getResponse( quot;http://www.httpunit.org/doc/cookbook.htmlquot; ); WebLink link = resp.getLinkWith( quot;responsequot; ); link.click(); WebResponse jdoc = wc.getCurrentPage(); WebTable table = resp.getTables()[0]; assertEquals( quot;rowsquot;, 4, table.getRowCount() ); assertEquals( quot;columnsquot;, 3, table.getColumnCount() ); assertEquals( quot;linksquot;, 1, table.getTableCell( 0, 2 ).getLinks().length ); WebForm form = resp.getForms()[0]; assertEquals( quot;La Cerentollaquot;, form.getParameterValue( quot;Namequot; ) ); assertEquals( quot;Chinesequot;, form.getParameterValue( quot;Foodquot; ) ); assertEquals( quot;Manayunkquot;, form.getParameterValue( quot;Locationquot; ) ); assertEquals( quot;onquot;, form.getParameterValue( quot;CreditCardquot; ) ); form.setParameter( quot;Foodquot;, quot;Italianquot; ); form.removeParameter( quot;CreditCardquot; ); form.submit(); 50
  51. 51. ServletUnit example ServletRunner sr = new ServletRunner(); sr.registerServlet( quot;myServletquot;, StatefulServlet.class.getName() ); ServletUnitClient sc = sr.newClient(); WebRequest request = new PostMethodWebRequest(quot;http://test.meterware.com/myServletquot;); request.setParameter( quot;colorquot;, quot;redquot; ); WebResponse response = sc.getResponse( request ); assertNotNull( quot;No response receivedquot;, response ); assertEquals( quot;content typequot;, quot;text/plainquot;, response.getContentType() ); assertEquals( quot;requested resourcequot;, quot;You selected redquot;, response.getText() ); public class StatefulServlet extends HttpServlet { protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException,IOException { resp.setContentType( quot;text/plainquot; ); writeSelectMessage( req.getParameter( quot;colorquot; ), resp.getWriter() ); setColor( req, req.getParameter( quot;colorquot; ) ); } protected void writeSelectMessage(String color, PrintWriter pw) throws IOException { pw.print( quot;You selected quot; + color ); pw.close(); } void setColor( HttpServletRequest req, String color ) throws ServletException { req.getSession().setAttribute( quot;colorquot;, color ); } 51 }
  52. 52. Tests-friendly patterns 52
  53. 53. Problem: dependencies AdressBook PersonDAO + loadPerson(...) + savePerson(...) 53
  54. 54. Hard-dependencies • Isolating the class to test is difficult and/or impossible • Failures may be induced by a dependency • More generally, it reveals potential code evolution problems: strong coupling 54
  55. 55. Depending on interfaces <<interface>> AdressBook IPersonDAO + loadPerson(...) + savePerson(...) DBPersonDAO FilePersonDAO MockPersonDAO 55
  56. 56. Interfaces dependencies • Interfaces introduce loose coupling • At execution time: use the real class • At test time: use a mock class • Future evolution is easier • BTW, don’t over-use interfaces! 56
  57. 57. Mock objects • Give a simple behavior: fixed values, random values, values passed by methods of the mock class, ... • They allow you to test even if the real class hasn’t been created yet • EasyMock, JMockIt allow to generate mocks on-the-fly 57
  58. 58. Object.equals() public boolean equals(Object obj) { if (obj instanceof ComparisonNode) { ComparisonNode other = (ComparisonNode) obj; return symbol.equals(other.symbol) && leftChild.equals(other.leftChild) && rightChild.equals(other.rightChild); } return false; important for } TestCase.assertXXX(...) 58
  59. 59. Dependency injection • 2 types of injection: setter-based & constructor-based • Constructor injection is preferable, and make your objects ready right after their instanciation • Clarifies classes dependencies • Principle used in inversion of control frameworks: PicoContainer, Spring, ... 59
  60. 60. Example private IConstraintNode leftChild; private IConstraintNode rightChild; private String symbol; public ComparisonNode(String symbol, VariableNode var, ConstantNode cst) { this(var, cst, symbol); } public ComparisonNode(String symbol, ConstantNode cst, VariableNode var) { this(cst, var, symbol); } protected ComparisonNode(IConstraintNode leftChild, IConstraintNode rightChild, String symbol) { super(); this.symbol = symbol; this.leftChild = leftChild; this.rightChild = rightChild; } 60
  61. 61. A few tips 61
  62. 62. Do • Try to really write your tests first • Test as soon as you create a new class • Test often • Start with sensible simple scenarios • Load data from resources (Class.getResourceAsStream(“/path/to”)) • Check for code coverage 62
  63. 63. Don’t • Write overly exhaustive test cases • Write tests for trivial methods (getters, setters, ...) even if it lowers coverage • Catch exceptions, unless you actually test that • Fix bugs without first reproducing them in a test case 63
  64. 64. Traversing object graphs • Sometimes you will need to walk through deep object graphs to test values • This is really boring • JXPath uses XPath on objects, and replaces iterations/loops by requests 64
  65. 65. JXPath example Iterator it; String str; JXPathContext ctx = JXPathContext.newContext(bp2); TestCase.assertEquals(quot;s0quot;, bp2.getInitialState().getName()); it = ctx.iterate(quot;finalStates/namequot;); TestCase.assertTrue(it.hasNext()); TestCase.assertEquals(quot;s1quot;, (String) it.next()); TestCase.assertFalse(it.hasNext()); it = ctx.iterate(quot;states[name='s0']/successors/namequot;); TestCase.assertTrue(it.hasNext()); TestCase.assertEquals(quot;s1quot;, (String) it.next()); TestCase.assertEquals(quot;s0quot;, (String) it.next()); TestCase.assertFalse(it.hasNext()); (...) Operation toRemove = (Operation) ctx.getValue(quot;operations[message/name='a']quot;); bp2.removeOperation(toRemove); List remainingOps = (List) ctx.getValue(quot;states[name='s0']/incomingOperationsquot;); TestCase.assertTrue(remainingOps.size() == 1); 65
  66. 66. The case of ‘old’ software • Test cases can be added to existing applications • In many situations the design can make it difficult to test / isolate... • You cannot break existing APIs • Don’t expect to introduce tests cases for 100% of the code base... • It will still allow you to spot new bugs! 66
  67. 67. Continuous integration 67
  68. 68. Problem • Developers work on different portions of the code base • Traditional integration may not be frequent enough (~ weekly) • It can be difficult to find who, when and how the build was broken • Continuous integration aims at detecting problems in nearly real-time 68
  69. 69. C.I. environment 4 email developers instant messaging 1 continuous integration changeset (Continuum, ...) 3 execute notification via hooks 2 source code tests suite repository 69 (SVN, CVS, ...)
  70. 70. • Demo: • project using Maven2 • Apache Continuum configuration • launch the tests from Continuum 70
  71. 71. Thanks! 71

×