Lecture 18


Published on

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Lecture 18

  1. 1. The Testing Process CS320 – Fundamentals of Software Engineering
  2. 2. What is Testing? <ul><li>IEEE: </li></ul><ul><ul><li>“The process of operating a system or component under specified condition, observing or recording the results, and making an evaluation of some aspect of the system or component.” </li></ul></ul><ul><li>Specified Condition = Test Cases </li></ul>
  3. 3. What is testing? <ul><li>Rick Craig and Stefan Jaskiel: </li></ul><ul><ul><li>“ Testing is a concurrent lifecycle process of engineering, using and maintaining testware in order to measure and improve the quality of the software being tested” </li></ul></ul><ul><li>Includes planning, analysis, and design that lead to test cases </li></ul>
  4. 4. What is testing? <ul><li>Testing is the process of comparing “what is” with “what ought to be” </li></ul>
  5. 5. Boris Beizer’s Five Levels <ul><li>Level 0 – No difference between testing and debugging </li></ul><ul><li>Level 1 – Purpose is to show software works </li></ul><ul><li>Level 2 – Purpose is to show that software doesn’t work (tough cases, boundary values) </li></ul><ul><li>Level 3 – Purpose is not to prove, but to reduce the perceived risk of not working at an acceptable value </li></ul><ul><li>Level 4 – Mental discipline that result in low risk software (Make software more testable from inception) </li></ul>
  6. 6. Challenges <ul><li>Not enough time </li></ul><ul><li>Too many combos </li></ul><ul><li>Difficult to determine the expected result of each test </li></ul><ul><li>Non-existent or rapidly changing req’s </li></ul><ul><li>No training </li></ul><ul><li>No tool support </li></ul><ul><li>Management doesn't understand testing </li></ul>
  7. 7. Test Cases <ul><li>Need to be designed: </li></ul><ul><ul><li>Systematic approach </li></ul></ul><ul><ul><li>Documentation & Planning </li></ul></ul><ul><ul><li>Goal and purpose </li></ul></ul>
  8. 8. Test Case <ul><li>Inputs </li></ul><ul><li>Expected Outputs </li></ul><ul><li>Order of Execution </li></ul><ul><li>(and Testing Criteria) how do I know when I’m done testing??? </li></ul>
  9. 9. Inputs <ul><li>Data </li></ul><ul><ul><li>Files </li></ul></ul><ul><ul><li>Interfacing Systems (keyboards, mice …) </li></ul></ul><ul><ul><li>databases </li></ul></ul>
  10. 10. Outputs <ul><li>Excepted Results (oracle) </li></ul><ul><ul><li>Program, process or data </li></ul></ul><ul><ul><li>Regression test suites </li></ul></ul><ul><ul><li>Validation data </li></ul></ul><ul><ul><li>Purchase test suites </li></ul></ul><ul><ul><li>Existing program </li></ul></ul>
  11. 11. Order of Execution <ul><li>Cascading test cases (build on each other) </li></ul><ul><ul><li>Ex. Create record in database </li></ul></ul><ul><ul><ul><li>Use record </li></ul></ul></ul><ul><ul><ul><li>Delete record etc. </li></ul></ul></ul><ul><li>Indecent test case (order does not matter) </li></ul>
  12. 12. Testing Criteria <ul><li>When do I stop testing: </li></ul><ul><ul><li>Ex1: When I cover all def use pairs </li></ul></ul><ul><ul><li>Ex2: All paths </li></ul></ul>
  13. 13. Types of Testing <ul><li>Black Box </li></ul><ul><ul><li>Reqs and spec, no knowledge of internal structure </li></ul></ul><ul><li>White Box </li></ul><ul><ul><li>Based on internal paths, structure </li></ul></ul><ul><ul><ul><li>Ex1 - definition use pairs </li></ul></ul></ul>
  14. 14. Testing Levels <ul><li>Unit Testing : smallest piece </li></ul><ul><li>Integration Testing : assemble pieces into larger units and test </li></ul><ul><li>System Testing : all of the software hardware, user manuals etc. </li></ul><ul><li>Acceptance Testing : testing, that when completed will result in delivering the software to the customer </li></ul>
  15. 15. Impossible to Test Everything <ul><li>int scale(int j){ </li></ul><ul><li>j = j-1; // should be j = j+1 </li></ul><ul><li>j = j/30000; </li></ul><ul><li>return j; </li></ul><ul><li>} </li></ul><ul><li>Input range -32768 to 32767 = 65,536 test cases (6 test cases reveal problem) </li></ul><ul><li>-30,001, -30000, -1, 0, 29999, 30000 </li></ul>
  16. 16. Unit testing with JUnit
  17. 17. One minute summary <ul><li>‘Code that isn’t tested doesn’t work’ </li></ul><ul><ul><li>Why test? </li></ul></ul><ul><li>Code that isn’t regression tested suffers from code rot (breaks eventually) </li></ul><ul><ul><li>Why regression test? </li></ul></ul><ul><li>A unit testing framework help make unit and regression testing more efficient </li></ul>
  18. 18. What is unit testing? <ul><li>Testing is often divided into categories such as: </li></ul><ul><ul><li>Unit testing </li></ul></ul><ul><ul><ul><li>Testing a ‘unit’ of code, usually a class </li></ul></ul></ul><ul><ul><li>Integration testing </li></ul></ul><ul><ul><ul><li>Testing a module of code (e.g. a package) </li></ul></ul></ul><ul><ul><li>Application testing </li></ul></ul><ul><ul><ul><li>Testing the code as the user would see it (black box) </li></ul></ul></ul>
  19. 19. What is a testing framework? <ul><li>A test framework provides reusable test functionality which: </li></ul><ul><ul><li>Is easier to use (e.g. don’t have to write the same code for each class) </li></ul></ul><ul><ul><li>Is standardized and reusable </li></ul></ul><ul><ul><li>Provides a base for regression tests </li></ul></ul>
  20. 20. Why formalize unit testing? <ul><li>Unit testing has often been done, but in an ad hoc manner </li></ul><ul><ul><li>E.g. adding a main method to a class, which runs tests on the class </li></ul></ul><ul><li>Axiom: </li></ul><ul><ul><li>Code that isn’t tested doesn’t work </li></ul></ul><ul><ul><li>‘ If code has no automated test case written for it to prove that it works, it must be assumed not to work.’ (Hightower and Lesiecki) </li></ul></ul>
  21. 21. Why use a testing framework? <ul><li>Each class must be tested when it is developed </li></ul><ul><li>Each class needs a regression test </li></ul><ul><li>Regression tests need to have standard interfaces </li></ul><ul><li>Thus, we can build the regression test when building the class and have a better, more stable product for less work </li></ul>
  22. 22. Regression testing <ul><li>New code and changes to old code can affect the rest of the code base </li></ul><ul><ul><li>‘Affect’ sometimes means ‘break’ </li></ul></ul><ul><li>I need to run tests on the old code, to verify it works – these are regression tests </li></ul><ul><li>Regression testing is required for a stable, maintainable code base </li></ul>
  23. 23. Regression Testing <ul><li>Regression testing - retesting code to check for errors due to changes </li></ul><ul><li>Rothermel and Harrold (1994) provide a simple definition for regression testing ( T0 = original tests): </li></ul><ul><ul><li>Identify changes to the software </li></ul></ul><ul><ul><li>Evaluate which tests from set T0 are no longer valid and create a subset T1 </li></ul></ul><ul><ul><li>Use T1 to test software </li></ul></ul><ul><ul><li>Identify if any parts of the system have not been tested adequately (based upon test adequacy criteria) and generate a new set of tests T2 </li></ul></ul><ul><ul><li>Use T2 to test software </li></ul></ul>
  24. 24. Regression Testing <ul><li>Rothermel and Harrold built framework of several retest selection techniques (1996) </li></ul><ul><ul><li>Minimization </li></ul></ul><ul><ul><ul><li>Linear equations </li></ul></ul></ul><ul><ul><li>Coverage </li></ul></ul><ul><ul><ul><li>ID changed definition-use pairs in control flow graphs </li></ul></ul></ul><ul><ul><li>Safe techniques </li></ul></ul><ul><ul><ul><li>Control dependency graphs. </li></ul></ul></ul>
  25. 25. Background: Regression Testing <ul><li>Leung and White classify test cases according to reuse (1989). </li></ul><ul><ul><li>Reusable test cases cover unmodified code (kept but not used) </li></ul></ul><ul><ul><li>Retestable test cases cover modified code (used) </li></ul></ul><ul><ul><li>Obsolete test-cases cover removed code (deleted) </li></ul></ul>
  26. 26. Regression Testing <ul><li>Briand et al. (TR 2004) uses Use Cases, Sequence Diagrams, and Class Diagrams to regression test code. </li></ul><ul><li>Creates mapping between design changes and code changes to classify code test cases. </li></ul><ul><li>Uses Leung and White approach to classify test cases </li></ul><ul><li>Makes assumptions regarding how the UML is used: </li></ul><ul><ul><li>(1) assumes all possible use-case scenarios can be realized by Sequence Diagrams </li></ul></ul><ul><ul><li>(2) assumes that each test case maps to a Sequence Diagram Scenario </li></ul></ul>
  27. 27. Regression testing and refactoring <ul><li>‘Refactoring is a technique to restructure code in a disciplined way.’ (Martin Fowler) </li></ul><ul><li>Refactoring is an excellent way to break code. </li></ul><ul><li>Regression testing allows developers to refactor safely – if the refactored code passes the test suite, it works </li></ul>
  28. 28. Running automated tests <ul><li>The real power of regression tests happens when they are automated </li></ul><ul><ul><li>This requires they report pass/fail results in a standardized way </li></ul></ul><ul><li>Can set up jobs to </li></ul><ul><ul><li>Clean & check out latest build tree </li></ul></ul><ul><ul><li>Run tests </li></ul></ul><ul><ul><li>Put results on a web page & send mail (if tests fail) </li></ul></ul><ul><li>JUnit & ant have code to do all of this </li></ul>
  29. 29. Some background <ul><li>eXtreme programming (XP) is a ‘programming methodology that stresses (among other things) testing and refactoring </li></ul><ul><li>The Apache community provides open-source software, including the Jakarta project (server side java tools, e.g. tomcat (servlet engine)) </li></ul><ul><li>Ant is a build tool (like make) </li></ul><ul><ul><li>Part of the Jakarta project </li></ul></ul><ul><ul><li>Is becoming the de facto standard for java projects </li></ul></ul><ul><ul><li>Is written in java, uses xml to specify build targets </li></ul></ul>
  30. 30. What is Junit? <ul><li>JUnit is a regression testing framework written by Erich Gamma and Kent Beck </li></ul><ul><li>It is found at www.junit.org </li></ul><ul><li>It consists of classes that the developer can extend to write a test </li></ul><ul><li>Notably: junit.framework.TestCase and related structure to run and report the tests </li></ul>
  31. 31. JUnit.. <ul><li>JUnit helps the programmer: </li></ul><ul><ul><li>Define and execute tests and test suites </li></ul></ul><ul><ul><li>Formalize requirements and clarify architecture </li></ul></ul><ul><ul><li>Write and debug code </li></ul></ul><ul><ul><li>Integrate code and always be ready to release a working version </li></ul></ul>
  32. 32. Installing Junit and CppUnit <ul><li>For JUnit </li></ul><ul><ul><li>Download (and unpack) the distribution </li></ul></ul><ul><ul><li>Put the jar file on your classpath </li></ul></ul><ul><li>For CppUnit </li></ul><ul><ul><li>Download (and unpack) the distribution </li></ul></ul><ul><ul><li>On linux, run make (configure; make; make install) </li></ul></ul><ul><ul><li>Bob Goddard figured out the required switches on Solaris, contact Bob, Pete Brodsky or I for details </li></ul></ul>
  33. 33. Tool integration <ul><li>Java IDEs </li></ul><ul><ul><li>Netbeans - http://www.netbeans.org/index.html </li></ul></ul><ul><ul><ul><li>Claims to include Junit integrationIt’s free </li></ul></ul></ul><ul><ul><li>Idea - http://www.intellij.com/idea/ </li></ul></ul><ul><ul><ul><li>Includes the Junit jar & is Junit savvy </li></ul></ul></ul><ul><ul><ul><li>Also supports ant </li></ul></ul></ul><ul><ul><ul><li>Not free, but not expensive ($400-$600) </li></ul></ul></ul>
  34. 34. What JUnit does <ul><li>JUnit runs a suite of tests and reports results </li></ul><ul><li>For each test in the test suite: </li></ul><ul><ul><li>JUnit calls setUp() </li></ul></ul><ul><ul><ul><li>This method should create any objects you may need for testing </li></ul></ul></ul>
  35. 35. What JUnit does… <ul><ul><li>JUnit calls one test method </li></ul></ul><ul><ul><ul><li>The test method may comprise multiple test cases; that is, it may make multiple calls to the method you are testing </li></ul></ul></ul><ul><ul><ul><li>In fact, since it’s your code, the test method can do anything you want </li></ul></ul></ul><ul><ul><ul><li>The setUp() method ensures you entered the test method with a virgin set of objects; what you do with them is up to you </li></ul></ul></ul><ul><ul><li>JUnit calls tearDown() </li></ul></ul><ul><ul><ul><li>This method should remove any objects you created </li></ul></ul></ul>
  36. 36. Creating a test class in JUnit <ul><li>Define a subclass of TestCase </li></ul><ul><li>Override the setUp() method to initialize object(s) under test. </li></ul><ul><li>Override the tearDown() method to release object(s) under test. </li></ul><ul><li>Define one or more public testXXX() methods that exercise the object(s) under test and assert expected results. </li></ul><ul><li>Define a static suite() factory method that creates a TestSuite containing all the testXXX() methods of the TestCase. </li></ul><ul><li>Optionally define a main() method that runs the TestCase in batch mode. </li></ul>
  37. 37. Fixtures <ul><li>A fixture is just a some code you want run before every test </li></ul><ul><li>You get a fixture by overriding the method </li></ul><ul><ul><li>protected void setUp() { …} </li></ul></ul><ul><li>The general rule for running a test is: </li></ul><ul><ul><li>protected void runTest() { setUp(); <run the test> tearDown(); } </li></ul></ul><ul><ul><li>so we can override setUp and/or tearDown, and that code will be run prior to or after every test case </li></ul></ul>
  38. 38. Implementing setUp() method <ul><li>Override setUp () to initialize the variables, and objects </li></ul><ul><li>Since setUp() is your code, you can modify it any way you like (such as creating new objects in it) </li></ul><ul><li>Reduces the duplication of code </li></ul>
  39. 39. Implementing the tearDown() method <ul><li>In most cases, the tearDown() method doesn’t need to do anything </li></ul><ul><ul><li>The next time you run setUp() , your objects will be replaced, and the old objects will be available for garbage collection </li></ul></ul><ul><ul><li>Like the finally clause in a try-catch-finally statement, tearDown() is where you would release system resources (such as streams) </li></ul></ul>
  40. 40. The structure of a test method <ul><li>A test method doesn’t return a result </li></ul><ul><li>If the tests run correctly, a test method does nothing </li></ul><ul><li>If a test fails, it throws an AssertionFailedError </li></ul><ul><li>The JUnit framework catches the error and deals with it; you don’t have to do anything </li></ul>
  41. 41. Test suites <ul><li>In practice, you want to run a group of related tests (e.g. all the tests for a class) </li></ul><ul><li>To do so, group your test methods in a class which extends TestCase </li></ul><ul><li>Running suites we will see in examples </li></ul>
  42. 42. assert X methods <ul><li>static void assertTrue(boolean test ) </li></ul><ul><li>static void assertFalse(boolean test ) </li></ul><ul><li>assertEquals( expected , actual ) </li></ul><ul><li>assertSame(Object  expected , Object  actual ) </li></ul><ul><li>assertNotSame(Object  expected , Object  actual ) </li></ul><ul><li>assertNull(Object  object ) </li></ul>
  43. 43. assert X methods <ul><li>assertNotNull(Object  object ) </li></ul><ul><li>fail() </li></ul><ul><li>All the above may take an optional String message as the first argument, for example, static void assertTrue(String message , boolean test ) </li></ul>
  44. 44. Organize The Tests <ul><li>Create test cases in the same package as the code under test </li></ul><ul><li>For each Java package in your application, define a TestSuite class that contains all the tests for validating the code in the package </li></ul><ul><li>Define similar TestSuite classes that create higher-level and lower-level test suites in the other packages (and sub-packages) of the application </li></ul><ul><li>Make sure your build process includes the compilation of all tests </li></ul>
  45. 45. JUnit framework
  46. 46. Writing a test <ul><li>Directions can be found in the Junit cookbook - http://junit.sourceforge.net/doc/cookbook/cookbook.htm </li></ul><ul><li>Summary </li></ul><ul><ul><li>Create an instance of TestCase : </li></ul></ul><ul><ul><li>Write methods which run your tests, calling them test<Foo> </li></ul></ul><ul><ul><li>Call a test runner with your test </li></ul></ul><ul><ul><li>When you want to check a value, call an assert method (e.g. assertTrue()) and pass a condition that is true if the test succeeds </li></ul></ul>
  47. 47. Example: Counter class <ul><li>For the sake of example, we will create and test a trivial “counter” class </li></ul><ul><ul><li>The constructor will create a counter and set it to zero </li></ul></ul><ul><ul><li>The increment method will add one to the counter and return the new value </li></ul></ul><ul><ul><li>The decrement method will subtract one from the counter and return the new value </li></ul></ul>
  48. 48. Example: Counter class <ul><li>We write the test methods before we write the code </li></ul><ul><ul><li>This has the advantages described earlier </li></ul></ul><ul><ul><li>Depending on the JUnit tool we use, we may have to create the class first, and we may have to populate it with stubs (methods with empty bodies) </li></ul></ul><ul><li>Don’t be alarmed if, in this simple example, the JUnit tests are more code than the class itself </li></ul>
  49. 49. JUnit tests for Counter <ul><li>public class CounterTest extends junit.framework.TestCase { Counter counter1; </li></ul><ul><li> public CounterTest() { } // default constructor </li></ul><ul><li> protected void setUp() { // creates a (simple) test fixture counter1 = new Counter(); } </li></ul><ul><li> protected void tearDown() { } // no resources to release </li></ul>
  50. 50. JUnit tests for Counter… <ul><li>public void testIncrement() { assertTrue(counter1.increment() == 1); assertTrue(counter1.increment() == 2); } </li></ul><ul><li>public void testDecrement() { assertTrue(counter1.decrement() == -1); } } // End from last slide </li></ul>
  51. 51. The Counter class itself <ul><li>public class Counter { int count = 0; public int increment() { return ++count; } public int decrement() { return --count; } </li></ul><ul><li>public int getCount() { return count; } } </li></ul>
  52. 52. Result <ul><li>We will see with the help of tool </li></ul><ul><li>When ?? </li></ul>!! Now !!
  53. 53. Why JUnit <ul><li>Allow you to write code faster while increasing quality </li></ul><ul><li>Elegantly simple </li></ul><ul><li>Check their own results and provide immediate feedback </li></ul><ul><li>Tests is inexpensive </li></ul><ul><li>Increase the stability of software </li></ul><ul><li>Developer tests </li></ul><ul><li>Written in Java </li></ul><ul><li>Free </li></ul><ul><li>Gives proper uniderstanding of unit testing </li></ul>
  54. 54. Problems with unit testing <ul><li>JUnit is designed to call methods and compare the results they return against expected results </li></ul><ul><ul><li>This ignores: </li></ul></ul><ul><ul><ul><li>Programs that do work in response to GUI commands </li></ul></ul></ul><ul><ul><ul><li>Methods that are used primary to produce output </li></ul></ul></ul>
  55. 55. Problems with unit testing… <ul><li>I think heavy use of JUnit encourages a “functional” style, where most methods are called to compute a value, rather than to have side effects </li></ul><ul><ul><li>This can actually be a good thing </li></ul></ul><ul><ul><li>Methods that just return results, without side effects (such as printing), are simpler, more general, and easier to reuse </li></ul></ul>
  56. 56. Junit addons <ul><li>Junit has a bunch of add on stuff that has been written for it (your mileage may vary) </li></ul><ul><li>Examples </li></ul><ul><ul><li>Code generators – make guesses at what tests are needed </li></ul></ul><ul><ul><li>Load testers </li></ul></ul><ul><ul><li>Code coverage </li></ul></ul><ul><ul><li>Testers for thread safety </li></ul></ul><ul><ul><li>‘ Helpful classes’ (e.g. RecursiveTestSuite is a TestSuite that will recursively walk through the directory structure looking for subclasses of TestCase and will add them.) </li></ul></ul>
  57. 57. Cactus, HttpUnit, Mock Objects <ul><li>Cactus (from jakarta) is a simple test framework for unit testing server-side java code (Servlets, EJBs, Tag Libs, Filters, ...). </li></ul><ul><li>HttpUnit emulates the relevant portions of browser behavior to allow automated testing (on sourceforge) </li></ul><ul><li>Mock objects emulate objects from other parts of the application, but have controlled behaviour </li></ul>
  58. 58. Resources <ul><li>Web sites </li></ul><ul><ul><li>Junit - http://www.junit.org </li></ul></ul><ul><ul><li>CppUnit - http://cppunit.sourceforge.net/ </li></ul></ul><ul><ul><li>Ant - http://jakarta.apache.org/ant/index.html </li></ul></ul>