Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Do unit testing as much as possible  minimize faults to lead to the system testing level, where the cost of finding errors multiply.
  • Testing each methods in isolation will not reveal the error. Only if you do cancel() ; pricePerMonth() you get error.
  • If some fields of Subscription, e.g. "price" , are open to package-level access, but constrained by a class invariant, you must in principle also check that other classes in the same package respect the classinv.
  • This is probably seldom done in practice. Furthermore, this assumes you do write class invariant, which is also seldom done in practice. Of course if there is simply no constraint on values of those fields, then there is no problem either.
  • Explain "oracle"
  • Also mention that in a way this is also a form of system test.
  • Note that the above code is that of ApplicationLogic_SPEC.
  • C14

    1. 1. Testing MSO 08/09, WP
    2. 2. Why testing ? <ul><li>Obvious. </li></ul><ul><ul><li>Still, how many of you actually test your assignment before submitting them? </li></ul></ul><ul><ul><li>How do you know if you have tested enough?? </li></ul></ul><ul><ul><li>How much testing really cost? Seemingly a lot, since there are plenty of software testing companies in NL; they seem to make good business! </li></ul></ul>
    3. 3. Some numbers <ul><li>US financial sector suffers 1.5 billion USD (2002) due to inadequate testing (US Nat. Inst. of Standards and Tech.) </li></ul><ul><li>Testing cost 30 – 70% of total development cost! (various studies) </li></ul><ul><li>Effort to find a bug (Ellims et all 2004, JTest) at system testing 12 – 20 hrs (600 Eur per bug!) at unit testing 0 – 3 hrs (90 Eur per bug) </li></ul>
    4. 4. Why is it so hard ? <ul><li>Size At the system level you can be dealing with millions LOCs! </li></ul><ul><li>OO adds a multitude of other complications. </li></ul><ul><li>Concurrency </li></ul>
    5. 5. Plan <ul><li>Unit test, integration test, system test </li></ul><ul><li>OO issues, concurrency issues </li></ul><ul><li>Testing with JUnit </li></ul><ul><li>Adequacy of your tests </li></ul><ul><li>Assignment-7 </li></ul><ul><li>Specification-based testing </li></ul><ul><li>Regression testing </li></ul>Dennis et al only discuss testing very globally 
    6. 6. Taxonomy of testing <ul><li>According to the &quot;scope&quot; : </li></ul><ul><ul><li>Unit testing </li></ul></ul><ul><ul><li>Integration testing </li></ul></ul><ul><ul><li>System testing </li></ul></ul><ul><li>According to ... &quot;other things&quot;  : </li></ul><ul><ul><li>Functional testing </li></ul></ul><ul><ul><li>GUI testing </li></ul></ul><ul><ul><li>Acceptance testing </li></ul></ul><ul><ul><li>Performance testing </li></ul></ul><ul><ul><li>Regression testing </li></ul></ul>
    7. 7. Unit Testing <ul><li>Testing each &quot; program unit &quot; in isolation. </li></ul><ul><li>Manageable size; finding bugs are easier at this level. </li></ul><ul><li>Widely supported by xUnit tools. </li></ul><ul><li>Automated tools e.g. T2. </li></ul><ul><li>We'll take a look at Junit. </li></ul>
    8. 8. Unit Testing <ul><li>What is &quot;unit&quot; ? Options : </li></ul><ul><ul><li>method </li></ul></ul><ul><ul><li>class </li></ul></ul><ul><ul><li>package </li></ul></ul><ul><li>Typically people take method as &quot;unit&quot; But you should be aware that in OO methods can influence each other through their side effects on objects' states. This implies that there are methods that should not be tested in isolation. </li></ul>
    9. 9. OO issues complicates testing <ul><li>Intra-class side effect makes it difficult to test each method in isolation: </li></ul>class Subscription { int price ; int n = 12 ; public Subscription(p) { price = p } public int pricePerMonth() { return price/n } public cancel() { price=0; n=0 } }
    10. 10. OO issues <ul><li>We fix Subscription and find no more error. So, now we're sure that every client can safely call &quot;pricePerMonth&quot;. </li></ul><ul><li>Watch out!! Inheritance adds another complication  </li></ul>class client { m(Subscription s) { s.cancel() ; return s.pricePerMonth() } class EvilSubscription extends Subscription { public int pricePerMonth() { return 0/0 } ... This implies that whenever you override a method, you need to test if you still respect the spec of the original method (Lipkov substitution).
    11. 11. Integration test <ul><li>Test if a set of &quot;units&quot; (that logically belong together) functions expectedly. </li></ul><ul><li>But what is a &quot;set of units&quot; ?  Class or package </li></ul><ul><li>Class level testing. Given a class C, check that </li></ul><ul><ul><li>Lots of work to do by hand; use automated tool (T2). </li></ul></ul>Various sequences of calls to C's methods and updates to C's exposed fields do not violate C's class invariant and the specs of each method.
    12. 12. Integration test <ul><li>Package level testing. </li></ul>Subscription price ... Customer SubsctiptionPackage classinv price  0 Service Potential problem: if there are fields in Subscription which are open to package-level access, but their values are constrained by a class invariant. Implying you have to test that every method in the package repects the class invariant of every class in the package.
    13. 13. Concurrency <ul><li>Concurrent threads multiply the number of possible scenarios to consider: </li></ul><ul><li>This implies you need lots of tests to cover all scenarios  expensive!! </li></ul><ul><li>But even that is not possible without special tools: </li></ul><ul><ul><li>We can't control JVM to do a specific scheduling. </li></ul></ul><ul><ul><li>This also implies that you can't replay an error in a concurrent setup! </li></ul></ul>thread 1 : { price++ ; price++ } thread 2 : { price = 0 } Will this satisfy : 0  price  2 ? And multi-cores are coming! (shudder)
    14. 14. The famous V model of testing Image is taken from Wikipedia.
    15. 15. V model of testing This approach assumes a water-fall SDLC model. System Requirement System Specifications Package/class level specifications Implementation unit testing integration testing system testing acceptance testing But more generally phases can run parallel.
    16. 16. System test <ul><li>System testing tests if a software as a whole functions as expected. </li></ul><ul><li>Problem: at this level a software often has to be interacted through its (complex) GUI. </li></ul><ul><li>Testing through GUI is HARD to write and hard to automate!! </li></ul><ul><li>Alternative: split system testing to </li></ul><ul><ul><li>Functional system testing </li></ul></ul><ul><ul><li>Separate GUI testing </li></ul></ul>
    17. 17. Functional System Testing <ul><li>You need a well defined interface (set of methods) through which we can directly access a software functionality (without going through the GUI). We will test through this interface. </li></ul><ul><li>Ideally this interface is prepared all the way from your design phase. </li></ul>Test Interface GUI Business Logic
    18. 18. Test Plan <ul><li>For a large application you (obviously) need a test plan. Because it involves lots of activities (and huge investment). </li></ul><ul><li>IEEE 829, test plan is a document: </li></ul><ul><ul><li>describing scope, approach, resources, and schedules of testing activities. </li></ul></ul><ul><ul><li>identifying test items, features to test, testing tasks, who will do the tasks, risks, and contingency plans. </li></ul></ul>
    19. 19. Test Plan <ul><li>IEEE 829  </li></ul><ul><ul><li>c) Approach </li></ul></ul><ul><ul><li>d) Test items </li></ul></ul><ul><ul><li>e) Features to be tested; features not to be tested </li></ul></ul><ul><ul><li>g) Pass/fail criteria </li></ul></ul><ul><ul><li>h) Suspension and resumption criteria </li></ul></ul><ul><ul><li>i) Test deliverables </li></ul></ul><ul><li>Oh well, a test plan essentially: </li></ul><ul><ul><li>Defines the scope of the testing </li></ul></ul><ul><ul><li>Explains how we do the testing </li></ul></ul><ul><ul><li>Estimates the resources needed </li></ul></ul>
    20. 20. (Unit) Testing with Junit C Ctest Junit report Java public class Subscription { private int price ; private int length ; ... public double pricePerMonth() { double r = (double) price / (double) length ; return r ; } }
    21. 21. Testing with Junit import org.junit.* ; import static org.junit.Assert.* ; public class SubscriptionTest { @Test public void test_if_pricePerMonth_returns_Euro() { Subscription S = new Subscription(200,2) ; assertTrue(S.pricePerMonth() == 1.0) ; } @Test public void test_if_pricePerMonth_rounds_up() { Subscription S3 = new Subscription(200,3) ; assertTrue(S3.pricePerMonth() == (double) 0.67) ; } } Read the Tutorial, linked from Opdracht-7. C Ctest Junit report Java
    22. 22. JUnit with Eclipse
    23. 23. JUnit with Eclipse
    24. 24. Have we tested enough ?? <ul><li>In principle, no way to know that. </li></ul><ul><li>Still, we can use &quot;coverage&quot; as indication: The precentage of &quot;parts&quot; of your code that are executed during your tests. </li></ul><ul><li>Keep in mind: 100% coverage does not imply absence of bugs!! But low coverage is obviously not good. </li></ul>
    25. 25. Various concepts of &quot;coverage&quot; <ul><li>Very abstract: </li></ul><ul><ul><li>Class coverage (have we tested all classes?) </li></ul></ul><ul><ul><li>Method coverage (have we tested all methods?) </li></ul></ul><ul><li>Code level: </li></ul><ul><ul><li>Line coverage </li></ul></ul><ul><ul><li>Branch coverage </li></ul></ul><ul><ul><li>Path coverage </li></ul></ul>
    26. 26. Line, branch, and path coverage <ul><li>Line coverage : percentage of lines executed by your tests. </li></ul><ul><li>E.g. a test (1x) with s.period = 0 s.price = 0 will give you 100% line coverage. </li></ul><ul><li>Very weak. But in practice is used most  </li></ul>discount(Subscription s) { if (s.period > 12) d = 0.05 else d=0 if (s.price > 1000) d += 0.05 return d }
    27. 27. Line, branch, and path coverage <ul><li>Branch coverage : percentage of branches (of &quot;if&quot;, &quot;while&quot; etc) executed by your tests. </li></ul><ul><li>E.g. the tests (2x) s1.price=0 s1.period=0 s2.price=2000 s2.period=2000 gives 100% branch coverage. </li></ul>discount(Subscription s) { if (s.period > 12) d = 0.05 else d=0 if (s.price > 1000) d += 0.05 return d } s.period > 12 s.price > 1000 But you still miss the &quot;red&quot; case!!
    28. 28. Line, branch, and path coverage <ul><li>Path coverage : percentage of paths in the &quot;control flow graph&quot; that your tests pass through. </li></ul><ul><li>There are 4 paths. You will need 4 tests to cover them all. </li></ul><ul><li>Much stronger. </li></ul><ul><li>Complication with loop & recursion  take e.g. &quot;prime paths&quot;. </li></ul>discount(Subscription s) { if (s.period > 12) d = 0.05 else d=0 if (s.price > 1000) d += 0.05 return d } s.period > 12 s.price > 1000
    29. 29. Measuring Coverge in Eclipse Read the Tutorial, linked from Opdracht-7.
    30. 30. Assignment 7 : Foo Subscription System
    31. 31. Foo architecture in more detail
    32. 32. Foo architecture in more detail
    33. 33. Discount <ul><li>Abstract class DiscountStrategy </li></ul><ul><ul><li>description () </li></ul></ul><ul><ul><li>calcDiscoun t(customer, activeDiscountStrategies) </li></ul></ul><ul><li>Two concrete implementations: </li></ul><ul><ul><li>Discount_5top </li></ul></ul><ul><ul><li>Discount_10pack </li></ul></ul><ul><li>TASK 1 : test these two classes (unit testing) </li></ul><ul><ul><li>Verify against the specifications (of the above two methods) </li></ul></ul><ul><ul><li>100% (emma) coverage! </li></ul></ul>
    34. 34. Problem with traditional tests <ul><li>We need concrete data as inputs and oracles  need manual effort. </li></ul><ul><li>This is very fragile; if we change the functionality a bit, you'll have to re-calculate them. </li></ul>@Test public void test_if_pricePerMonth_returns_Euro() { Subscription S = new Subscription(200,2) ; assertTrue(S.pricePerMonth() == 1.0) ; } @Test public void test_if_pricePerMonth_rounds_up() { Subscription S3 = new Subscription(200,3) ; assertTrue(S3.pricePerMonth() == (double) 0.67) ; }
    35. 35. Alternative: specification-based testing <ul><li>We'll write specifications in-code (in Java). </li></ul><ul><li>Alternative: use JML or OCL. </li></ul><ul><li>You'll need additional tools. </li></ul><ul><li>Your need to train your programmers in them. </li></ul><ul><li>TASK 2 : (optional, 1.5 pt) Write specifications for ApplicationLogic , and do specification-based testing. We'll limit to the methods addCustomer and removeCustomer of ApplicationLogic. </li></ul>
    36. 36. Architecture ApplicationLogic ApplicationLogic_SPEC addCustomer removeCustomer classinv ApplicationLogic_SPEC_test app int addCustomer(String name, String email) { ... // pre-condition int result = app.addCustomer (name,email) ; .... // post-condition return result }
    37. 37. Example of in-code specification : spec-1 Suppose this is the spec of addCustomer: &quot; Name and email should not be null nor empty (pre-cond). If name does not occur in the system, then add the customer, return +1. Else don't add, return -1.&quot; int addCustomer(String name, String email) { assertTrue(name != null && email != null && ! name.equals(&quot;&quot;) && ... ) boolean occur = false ; for (String n : getCustNames()) if (n.equals(name)) { occur = true ; break } int result = app.addCustomer (name,email) ; if (occur) assertTrue(result == 1) else assertTrue(result == -1) ; return result }
    38. 38. Example of in-code specification : spec-2 ApplicationLogic ApplicationLogic_SPEC addCustomer removeCustomer classinv app Suppose we want to impose this class invariant on the application logic: &quot; The value of the variable NextCustId should be non-negative. &quot; boolean classinv() { assertTrue(app.nextCustID  0) ; return true ; }
    39. 39. Examples of the tests import org.junit.*; import static org.junit.Assert.*; public class ApplicationLogic_test { @Test public void test1() { ApplicationLogic.reset() ApplicationLogic_SPEC applogic = new ApplicationLogic_SPEC() assertTrue (applogic.classinv()) applogic.addCustomer (&quot;Pinky&quot;,&quot;Pinky@world.com&quot;) assertTrue (applogic.classinv()) applogic.addCustomer (&quot;Pinky&quot;,&quot;Pinky@world.com&quot;) assertTrue (applogic.classinv()) } }
    40. 40. Advatanges of spec-based testing <ul><li>No (fragile) oracle! </li></ul><ul><li>But we still have to manually write the test sequences and inputs. Can be automated!  e.g. with T2 </li></ul><ul><li>You must combine this with pair-programming. </li></ul><ul><li>Disadvantage: </li></ul><ul><ul><li>writing specs takes time </li></ul></ul><ul><ul><li>Incure maintenace cost. </li></ul></ul>public void test1() { ... assertTrue (applogic.classinv()) applogic.addCustomer (&quot;Pinky&quot;,&quot;Pinky@world.com&quot;) assertTrue (applogic.classinv()) applogic.addCustomer (&quot;Pinky&quot;,&quot;Pinky@world.com&quot;) assertTrue (applogic.classinv()) }
    41. 41. Regression test <ul><li>The re-testing of a software that has been modified. </li></ul><ul><li>Is a serious problem for large software  may take days! </li></ul><ul><li>Between versions, the changes are usually very limited (to some modules). </li></ul><ul><li>However, the impact may be global due to interactions between modules. </li></ul><ul><li>This forces you to test the entire software. That is, you need a test set that will deliver full coverage over your entire software. </li></ul><ul><li>But if we just re-execute the whole set of tests collected so far, this will take too long!! </li></ul><ul><li>Challenge: how do we identify a good minimum set? </li></ul>