Your SlideShare is downloading. ×
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. The Testing Process CS320 – Fundamentals of Software Engineering
  • 2. What is Testing?
    • IEEE:
      • “The process of operating a system or component under specified condition, observing or recording the results, and making an evaluation of some aspect of the system or component.”
    • Specified Condition = Test Cases
  • 3. What is testing?
    • Rick Craig and Stefan Jaskiel:
      • “ Testing is a concurrent lifecycle process of engineering, using and maintaining testware in order to measure and improve the quality of the software being tested”
    • Includes planning, analysis, and design that lead to test cases
  • 4. What is testing?
    • Testing is the process of comparing “what is” with “what ought to be”
  • 5. Boris Beizer’s Five Levels
    • Level 0 – No difference between testing and debugging
    • Level 1 – Purpose is to show software works
    • Level 2 – Purpose is to show that software doesn’t work (tough cases, boundary values)
    • Level 3 – Purpose is not to prove, but to reduce the perceived risk of not working at an acceptable value
    • Level 4 – Mental discipline that result in low risk software (Make software more testable from inception)
  • 6. Challenges
    • Not enough time
    • Too many combos
    • Difficult to determine the expected result of each test
    • Non-existent or rapidly changing req’s
    • No training
    • No tool support
    • Management doesn't understand testing
  • 7. Test Cases
    • Need to be designed:
      • Systematic approach
      • Documentation & Planning
      • Goal and purpose
  • 8. Test Case
    • Inputs
    • Expected Outputs
    • Order of Execution
    • (and Testing Criteria) how do I know when I’m done testing???
  • 9. Inputs
    • Data
      • Files
      • Interfacing Systems (keyboards, mice …)
      • databases
  • 10. Outputs
    • Excepted Results (oracle)
      • Program, process or data
      • Regression test suites
      • Validation data
      • Purchase test suites
      • Existing program
  • 11. Order of Execution
    • Cascading test cases (build on each other)
      • Ex. Create record in database
        • Use record
        • Delete record etc.
    • Indecent test case (order does not matter)
  • 12. Testing Criteria
    • When do I stop testing:
      • Ex1: When I cover all def use pairs
      • Ex2: All paths
  • 13. Types of Testing
    • Black Box
      • Reqs and spec, no knowledge of internal structure
    • White Box
      • Based on internal paths, structure
        • Ex1 - definition use pairs
  • 14. Testing Levels
    • Unit Testing : smallest piece
    • Integration Testing : assemble pieces into larger units and test
    • System Testing : all of the software hardware, user manuals etc.
    • Acceptance Testing : testing, that when completed will result in delivering the software to the customer
  • 15. Impossible to Test Everything
    • int scale(int j){
    • j = j-1; // should be j = j+1
    • j = j/30000;
    • return j;
    • }
    • Input range -32768 to 32767 = 65,536 test cases (6 test cases reveal problem)
    • -30,001, -30000, -1, 0, 29999, 30000
  • 16. Unit testing with JUnit
  • 17. One minute summary
    • ‘Code that isn’t tested doesn’t work’
      • Why test?
    • Code that isn’t regression tested suffers from code rot (breaks eventually)
      • Why regression test?
    • A unit testing framework help make unit and regression testing more efficient
  • 18. What is unit testing?
    • Testing is often divided into categories such as:
      • Unit testing
        • Testing a ‘unit’ of code, usually a class
      • Integration testing
        • Testing a module of code (e.g. a package)
      • Application testing
        • Testing the code as the user would see it (black box)
  • 19. What is a testing framework?
    • A test framework provides reusable test functionality which:
      • Is easier to use (e.g. don’t have to write the same code for each class)
      • Is standardized and reusable
      • Provides a base for regression tests
  • 20. Why formalize unit testing?
    • Unit testing has often been done, but in an ad hoc manner
      • E.g. adding a main method to a class, which runs tests on the class
    • Axiom:
      • Code that isn’t tested doesn’t work
      • ‘ If code has no automated test case written for it to prove that it works, it must be assumed not to work.’ (Hightower and Lesiecki)
  • 21. Why use a testing framework?
    • Each class must be tested when it is developed
    • Each class needs a regression test
    • Regression tests need to have standard interfaces
    • Thus, we can build the regression test when building the class and have a better, more stable product for less work
  • 22. Regression testing
    • New code and changes to old code can affect the rest of the code base
      • ‘Affect’ sometimes means ‘break’
    • I need to run tests on the old code, to verify it works – these are regression tests
    • Regression testing is required for a stable, maintainable code base
  • 23. Regression Testing
    • Regression testing - retesting code to check for errors due to changes
    • Rothermel and Harrold (1994) provide a simple definition for regression testing ( T0 = original tests):
      • Identify changes to the software
      • Evaluate which tests from set T0 are no longer valid and create a subset T1
      • Use T1 to test software
      • Identify if any parts of the system have not been tested adequately (based upon test adequacy criteria) and generate a new set of tests T2
      • Use T2 to test software
  • 24. Regression Testing
    • Rothermel and Harrold built framework of several retest selection techniques (1996)
      • Minimization
        • Linear equations
      • Coverage
        • ID changed definition-use pairs in control flow graphs
      • Safe techniques
        • Control dependency graphs.
  • 25. Background: Regression Testing
    • Leung and White classify test cases according to reuse (1989).
      • Reusable test cases cover unmodified code (kept but not used)
      • Retestable test cases cover modified code (used)
      • Obsolete test-cases cover removed code (deleted)
  • 26. Regression Testing
    • Briand et al. (TR 2004) uses Use Cases, Sequence Diagrams, and Class Diagrams to regression test code.
    • Creates mapping between design changes and code changes to classify code test cases.
    • Uses Leung and White approach to classify test cases
    • Makes assumptions regarding how the UML is used:
      • (1) assumes all possible use-case scenarios can be realized by Sequence Diagrams
      • (2) assumes that each test case maps to a Sequence Diagram Scenario
  • 27. Regression testing and refactoring
    • ‘Refactoring is a technique to restructure code in a disciplined way.’ (Martin Fowler)
    • Refactoring is an excellent way to break code.
    • Regression testing allows developers to refactor safely – if the refactored code passes the test suite, it works
  • 28. Running automated tests
    • The real power of regression tests happens when they are automated
      • This requires they report pass/fail results in a standardized way
    • Can set up jobs to
      • Clean & check out latest build tree
      • Run tests
      • Put results on a web page & send mail (if tests fail)
    • JUnit & ant have code to do all of this
  • 29. Some background
    • eXtreme programming (XP) is a ‘programming methodology that stresses (among other things) testing and refactoring
    • The Apache community provides open-source software, including the Jakarta project (server side java tools, e.g. tomcat (servlet engine))
    • Ant is a build tool (like make)
      • Part of the Jakarta project
      • Is becoming the de facto standard for java projects
      • Is written in java, uses xml to specify build targets
  • 30. What is Junit?
    • JUnit is a regression testing framework written by Erich Gamma and Kent Beck
    • It is found at
    • It consists of classes that the developer can extend to write a test
    • Notably: junit.framework.TestCase and related structure to run and report the tests
  • 31. JUnit..
    • JUnit helps the programmer:
      • Define and execute tests and test suites
      • Formalize requirements and clarify architecture
      • Write and debug code
      • Integrate code and always be ready to release a working version
  • 32. Installing Junit and CppUnit
    • For JUnit
      • Download (and unpack) the distribution
      • Put the jar file on your classpath
    • For CppUnit
      • Download (and unpack) the distribution
      • On linux, run make (configure; make; make install)
      • Bob Goddard figured out the required switches on Solaris, contact Bob, Pete Brodsky or I for details
  • 33. Tool integration
    • Java IDEs
      • Netbeans -
        • Claims to include Junit integrationIt’s free
      • Idea -
        • Includes the Junit jar & is Junit savvy
        • Also supports ant
        • Not free, but not expensive ($400-$600)
  • 34. What JUnit does
    • JUnit runs a suite of tests and reports results
    • For each test in the test suite:
      • JUnit calls setUp()
        • This method should create any objects you may need for testing
  • 35. What JUnit does…
      • JUnit calls one test method
        • The test method may comprise multiple test cases; that is, it may make multiple calls to the method you are testing
        • In fact, since it’s your code, the test method can do anything you want
        • The setUp() method ensures you entered the test method with a virgin set of objects; what you do with them is up to you
      • JUnit calls tearDown()
        • This method should remove any objects you created
  • 36. Creating a test class in JUnit
    • Define a subclass of TestCase
    • Override the setUp() method to initialize object(s) under test.
    • Override the tearDown() method to release object(s) under test.
    • Define one or more public testXXX() methods that exercise the object(s) under test and assert expected results.
    • Define a static suite() factory method that creates a TestSuite containing all the testXXX() methods of the TestCase.
    • Optionally define a main() method that runs the TestCase in batch mode.
  • 37. Fixtures
    • A fixture is just a some code you want run before every test
    • You get a fixture by overriding the method
      • protected void setUp() { …}
    • The general rule for running a test is:
      • protected void runTest() { setUp(); <run the test> tearDown(); }
      • so we can override setUp and/or tearDown, and that code will be run prior to or after every test case
  • 38. Implementing setUp() method
    • Override setUp () to initialize the variables, and objects
    • Since setUp() is your code, you can modify it any way you like (such as creating new objects in it)
    • Reduces the duplication of code
  • 39. Implementing the tearDown() method
    • In most cases, the tearDown() method doesn’t need to do anything
      • The next time you run setUp() , your objects will be replaced, and the old objects will be available for garbage collection
      • Like the finally clause in a try-catch-finally statement, tearDown() is where you would release system resources (such as streams)
  • 40. The structure of a test method
    • A test method doesn’t return a result
    • If the tests run correctly, a test method does nothing
    • If a test fails, it throws an AssertionFailedError
    • The JUnit framework catches the error and deals with it; you don’t have to do anything
  • 41. Test suites
    • In practice, you want to run a group of related tests (e.g. all the tests for a class)
    • To do so, group your test methods in a class which extends TestCase
    • Running suites we will see in examples
  • 42. assert X methods
    • static void assertTrue(boolean test )
    • static void assertFalse(boolean test )
    • assertEquals( expected , actual )
    • assertSame(Object  expected , Object  actual )
    • assertNotSame(Object  expected , Object  actual )
    • assertNull(Object  object )
  • 43. assert X methods
    • assertNotNull(Object  object )
    • fail()
    • All the above may take an optional String message as the first argument, for example, static void assertTrue(String message , boolean test )
  • 44. Organize The Tests
    • Create test cases in the same package as the code under test
    • For each Java package in your application, define a TestSuite class that contains all the tests for validating the code in the package
    • Define similar TestSuite classes that create higher-level and lower-level test suites in the other packages (and sub-packages) of the application
    • Make sure your build process includes the compilation of all tests
  • 45. JUnit framework
  • 46. Writing a test
    • Directions can be found in the Junit cookbook -
    • Summary
      • Create an instance of TestCase :
      • Write methods which run your tests, calling them test<Foo>
      • Call a test runner with your test
      • When you want to check a value, call an assert method (e.g. assertTrue()) and pass a condition that is true if the test succeeds
  • 47. Example: Counter class
    • For the sake of example, we will create and test a trivial “counter” class
      • The constructor will create a counter and set it to zero
      • The increment method will add one to the counter and return the new value
      • The decrement method will subtract one from the counter and return the new value
  • 48. Example: Counter class
    • We write the test methods before we write the code
      • This has the advantages described earlier
      • Depending on the JUnit tool we use, we may have to create the class first, and we may have to populate it with stubs (methods with empty bodies)
    • Don’t be alarmed if, in this simple example, the JUnit tests are more code than the class itself
  • 49. JUnit tests for Counter
    • public class CounterTest extends junit.framework.TestCase { Counter counter1;
    • public CounterTest() { } // default constructor
    • protected void setUp() { // creates a (simple) test fixture counter1 = new Counter(); }
    • protected void tearDown() { } // no resources to release
  • 50. JUnit tests for Counter…
    • public void testIncrement() { assertTrue(counter1.increment() == 1); assertTrue(counter1.increment() == 2); }
    • public void testDecrement() { assertTrue(counter1.decrement() == -1); } } // End from last slide
  • 51. The Counter class itself
    • public class Counter { int count = 0; public int increment() { return ++count; } public int decrement() { return --count; }
    • public int getCount() { return count; } }
  • 52. Result
    • We will see with the help of tool
    • When ??
    !! Now !!
  • 53. Why JUnit
    • Allow you to write code faster while increasing quality
    • Elegantly simple
    • Check their own results and provide immediate feedback
    • Tests is inexpensive
    • Increase the stability of software
    • Developer tests
    • Written in Java
    • Free
    • Gives proper uniderstanding of unit testing
  • 54. Problems with unit testing
    • JUnit is designed to call methods and compare the results they return against expected results
      • This ignores:
        • Programs that do work in response to GUI commands
        • Methods that are used primary to produce output
  • 55. Problems with unit testing…
    • I think heavy use of JUnit encourages a “functional” style, where most methods are called to compute a value, rather than to have side effects
      • This can actually be a good thing
      • Methods that just return results, without side effects (such as printing), are simpler, more general, and easier to reuse
  • 56. Junit addons
    • Junit has a bunch of add on stuff that has been written for it (your mileage may vary)
    • Examples
      • Code generators – make guesses at what tests are needed
      • Load testers
      • Code coverage
      • Testers for thread safety
      • ‘ Helpful classes’ (e.g. RecursiveTestSuite is a TestSuite that will recursively walk through the directory structure looking for subclasses of TestCase and will add them.)
  • 57. Cactus, HttpUnit, Mock Objects
    • Cactus (from jakarta) is a simple test framework for unit testing server-side java code (Servlets, EJBs, Tag Libs, Filters, ...).
    • HttpUnit emulates the relevant portions of browser behavior to allow automated testing (on sourceforge)
    • Mock objects emulate objects from other parts of the application, but have controlled behaviour
  • 58. Resources
    • Web sites
      • Junit -
      • CppUnit -
      • Ant -