Software Testing


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Software Testing

  1. 1. Motivation People are not perfect We make errors in design and code Goal of testing: given some code, uncover Software Testing as many errors are possible Important and expensive activity Not unusual to spend 30-40% of total project effort on testing For critical systems (e.g. flight control): cost can be several times the cost of all other activities combined CSE 757 - Software Testing 1 CSE 757 - Software Testing 2 A Way of Thinking Testing Objectives Design and coding are creative Testing is a process of executing software with the intent of finding Testing is destructive errors The primary goal is to “break” the software Good testing has a high probability of Very often the same person does both finding as-yet-undiscovered errors coding and testing Need “split personality”: when you start Successful testing discovers unknown testing, become paranoid and malicious errors Surprisingly hard to do: people don’t like If did not find any errors, need to ask finding out that they made mistakes whether our testing approach is good CSE 757 - Software Testing 3 CSE 757 - Software Testing 4 Basic Definitions Testing Approaches Test case: specifies In this course: small sample of Inputs + pre-test state of the software approaches for testing Expected results (outputs and state) White-box testing Black-box testing: ignores the internal Control-flow-based testing logic of the software Data-flow-based testing Given this input, was the output correct? Black-box testing White-box testing: uses knowledge of Equivalence partitioning the internal structure of the software e.g. write tests to “cover” internal paths CSE 757 - Software Testing 5 CSE 757 - Software Testing 6 1
  2. 2. Control-flow-based Testing Example of a Control Flow Graph Traditional form of white-box testing s:=0; 1 d:=0; Step 1: From the source code, create a while (x<y) { graph describing the flow of control 2 x:=x+3; Called the control flow graph y:=y+2; 3 The graph is created (extracted from the if (x+y < 100) source code) manually or automatically s:=s+x+y; 4 else Step 2: Design test cases to cover d:=d+x-y; 5 6 certain elements of this graph } 7 Nodes, edges, paths 8 CSE 757 - Software Testing 7 CSE 757 - Software Testing 8 Elements of a Control Flow Graph IF-THEN, IF-THEN-ELSE, SWITCH Three kinds of nodes: if (c) if (c) switch (c) Statement nodes: single-entry-single-exit then … then … case 1: … sequences of statements join point else … case 2: … Predicate (decision) nodes: conditions for join point . . . branching join point Auxiliary nodes: (optional) for easier understanding (e.g. “merge points” for IF) Edges: possible flow of control ... ... ... ........... CSE 757 - Software Testing 9 CSE 757 - Software Testing 10 Example Mapping for Loops switch (position) case CASHIER: if (empl_yrs > 5) while (c) { bonus := 1; … ... else } ..... bonus := 0.7; break; case MANAGER: Note: other loops (e.g. FOR, bonus := 1.5; if (retiring_soon) DO-WHILE, …) are mapped similarly bonus := 1.2 * bonus break; Mini-assignment: figure out how this is done case ... endswitch CSE 757 - Software Testing 11 CSE 757 - Software Testing 12 2
  3. 3. Statement Coverage Example Given the control flow graph, define a Suppose that we write and execute two test cases 1 “coverage target” and write test cases to achieve it Test case #1: follows path 2 1-2-exit Traditional target: statement coverage 3 Test case #2: 1-2-3-4-5- Test cases that cover all nodes 7-8-2-3-4-5-7-8-2-exit (loop twice, and both times T 4 F Code that has not been executed during take the true branch) testing is more likely to contain errors 5 6 Problem: node 6 is never 7 Often this is the “low-probability” code executed, so we don’t have 100% statement coverage 8 CSE 757 - Software Testing 13 CSE 757 - Software Testing 14 Branch Coverage Branch Coverage Target: write test cases that cover all Statement coverage does not imply branches of predicate nodes branch coverage True and false branches of each IF Example: if (c) then s; The two branches corresponding to the By executing only with c=true, we will achieve condition of a loop statement coverage, but not branch All alternatives in a SWITCH coverage In modern languages, branch coverage Motivation: experience shows that many implies statement coverage errors occur in “decision making” Plus, it subsumes statement coverage CSE 757 - Software Testing 15 CSE 757 - Software Testing 16 Example Achieving Branch Coverage Same example as Branch coverage: a necessary minimum 1 before: two test cases Pick a set of start-to-end paths that cover Path 1-2-exit 2 all branches, and write tests cases to Path 1-2-3-4-5-7-8-2-3- execute these paths 4-5-7-8-2-exit 3 Basic strategy Problem: the “false” T 4 F Add a new path that covers at least one edge branch of 4 is never 5 6 that is not covered by the current paths taken - don’t have Sometimes the set of paths chosen with this 100% branch coverage 7 strategy is called the “basis set” 8 Good example in Pressman, Sect. 17.4.3 CSE 757 - Software Testing 17 CSE 757 - Software Testing 18 3
  4. 4. Data-flow-based Testing Example Test connections between variable assume y is already initialized 1 2 3 definitions (“write”) and variable uses 1 s:=0; (“read”) 2 x:=0; DEF(1)={s} USE(1)=∅ 4 3 while (x<y) { DEF(2)={x} USE(2)=∅ Variation of the control flow graph 4 x:=x+3; DEF(3)=∅ USE(3)={x,y} 5 A node represents a single statement, not a 5 y:=y+2; DEF(4)={x} USE(4)={x} single-entry-single-exit chain of statements 6 if (x+y<10) DEF(5)={y} USE(5)={y} 6 7 s:=s+x+y; DEF(6)=∅ USE(6)={x,y} Set DEF(n) contains variables that are else DEF(7)={s} USE(7)={s,x,y} 7 8 defined at node n (i.e., they are written) 8 s:=s+x-y; DEF(8)={s} USE(8)={s,x,y} 9 } DEF(9)=∅ USE(9)=∅ Set USE(n): variables that are read DEF(10)=∅ USE(10)=∅ 10 CSE 757 - Software Testing 19 CSE 757 - Software Testing 20 Reaching Definitions Def-Use Pairs A definition of x at n1 A def-use (DU) pair for variable x is a reaches n2 if and only if 1 2 3 pair of nodes (n1,n2) such that there is a path between n1 x is in DEF(n1) and n2 that does not 4 the definition of x at n1 reaches n2 contain a definition of x 5 x is in USE(n2) DEF(1)={s} USE(1)=∅ Reaches DEF(2)={x} USE(2)=∅ 6 i.e., the value that is assigned to x at n1 DEF(3)=∅ USE(3)={x,y} nodes 2, 3, DEF(4)={x} USE(4)={x} is used at n2 7 8 4, 5, 6, 7, DEF(5)={y} USE(5)={y} Since the definition reaches n2, the value is 9 not “killed” along some path n1…n2 8, but not DEF(6)=∅ USE(6)={x,y} DEF(7)={s} USE(7)={s,x,y} 9, 10 DEF(8)={s} USE(8)={s,x,y} 10 CSE 757 - Software Testing 21 CSE 757 - Software Testing 22 Example of Def-Use Pairs Data-flow-based Testing Reaches nodes 2, 3, 4, 5, Identify all DU pairs and construct test 6, 7, 8, but not 9, 10 1 2 3 cases that cover these pairs For this 4 Variations with different “strength” definition: All-DU-paths: for each DU pair (n1,n2) two DU 5 DEF(1)={s} USE(1)=∅ for x, exercise all possible paths n1…n2 pairs DEF(2)={x} USE(2)=∅ that are clear of definitions of x 1-7, 1-8 DEF(3)=∅ USE(3)={x,y} 6 DEF(4)={x} USE(4)={x} 7 8 All-uses: for each DU pair (n1,n2) for x, DEF(5)={y} USE(5)={y} exercise at least one path n1…n2 that is DEF(6)=∅ USE(6)={x,y} 9 DEF(7)={s} USE(7)={s,x,y} clear of definitions of x DEF(8)={s} USE(8)={s,x,y} 10 CSE 757 - Software Testing 23 CSE 757 - Software Testing 24 4
  5. 5. Data-flow-based Testing Black-box Testing All-definitions: for each definition, cover Unlike white-box testing: don’t use any at least one DU pair for that definition knowledge about the internals of the code i.e., if x is defined at n1, execute at least Test cases are designed based on one path n1…n2 such that x is in USE(n2) and the path is clear of definitions of x specifications Motivation: see the effects of using the Example: search for a value in an array values produced by computations Postcondition: return value is the index of some occurrence of the value, or -1 if the Focuses on the data, while control-flow- value does not occur in the array based testing focuses on the control CSE 757 - Software Testing 25 CSE 757 - Software Testing 26 Equivalence Partitioning Equivalence Classes Consider input/output domains and Examples partition them into equivalence classes Input x in range [a..b]: three classes “x<a”, For different values from the same class, “a<=x<=b”, “b<x” the software should behave equivalently boolean: classes “true” and “false” Some classes may represent invalid input Test values from each class Example for input range 2..5: “less than 2”, Choosing test values “between 2 and 5”, and “greater than 5” Choose a typical value in the middle of the class(es) that represent valid input Testing with values from different classes is more likely to find errors than Choose values at the boundaries of classes testing with values from the same class e.g. for [a..b], use a-1, a, a+1, b-1, b, b+1 CSE 757 - Software Testing 27 CSE 757 - Software Testing 28 Example Another Example Spec says that the code accepts between Similarly for the output: exercise 4 and 24 inputs; each is a 3-digit integer boundary values One partition: number of inputs Spec: the output is between 3 and 6 Classes “x<4”, “4<=x<=24”, “24<x” integers, each in the range 1000-2500 Chosen values: 3,4,5,14,23,24,25 Try to design inputs that produce Another partition: integer values 3 outputs with value 1000 Classes: “x<100”, “100<=x<=999”, “999<x” 3 outputs with value 2500 Chosen values: 99,100,101,500,998,999,1000 6 outputs with value 1000 6 outputs with value 2500 CSE 757 - Software Testing 29 CSE 757 - Software Testing 30 5
  6. 6. Example: Searching Example: Searching Search for a value in an array Array Value Output Return: index of some occurrence of the empty 5 -1 value, or -1 if the value does not occur [7] 7 0 One partition: size of the array Programmer errors are often made for size [7] 2 -1 1: a separate equivalence class [1,6,4,7,2] 1 0 Classes: “empty array”, “array with one element”, “array with many elements” [1,6,4,7,2] 4 2 [1,6,4,7,2] 2 4 Another partition: location of the value “first element”, “last element”, “middle [1,6,4,7,2] 3 -1 element”, “not found” CSE 757 - Software Testing 31 CSE 757 - Software Testing 32 Testing Strategies Many issues beyond what we discussed Who does the testing? Which techniques should we use and when? Testing Strategies No universal strategies Principles that have been useful in practice e.g. the notions of unit testing and integration testing CSE 757 - Software Testing 33 CSE 757 - Software Testing 34 Scope and Focus Unit Testing Unit testing: scope = individual component Scope: one component from the design Focus: component correctness Often corresponds to the notion of White-box and black-box techniques “compilation unit” from the prog. language Integration testing: scope = set of Responsibility of the developer interacting components Focus: correctness of component interactions Both white-box and black-box techniques Mostly black-box, some white-box May be necessary to create stubs: “fake” System testing: scope = entire system code that replaces called modules Focus: overall system correctness If not yet implemented, or not yet tested Only black-box techniques CSE 757 - Software Testing 35 CSE 757 - Software Testing 36 6
  7. 7. Basic Strategy for Unit Testing Test-First Programming Create black-box tests Modern practices: the importance of unit Based on the specification of the unit testing during development Evaluate the tests using white-box Example: test-first programming techniques (test adequacy criteria) Before you start writing any code, first How well did the tests cover statements, write the tests for this code branches, paths, DU-pairs, etc.? Write a little test code, write the Many possible criteria; at the very least corresponding unit code, make sure it passes need 100% branch coverage the tests, and then repeat Create more tests for the inadequacies: E.g., used in Extreme Programming: any e.g. to increase coverage of DU-pairs program feature without a corresponding test simply doesn’t exist CSE 757 - Software Testing 37 CSE 757 - Software Testing 38 Advantages of Test-First Programming System Testing Developers don’t “skip” unit testing Goal: find whether the program does Satisfying for the programmer: feeling of what the customer expects to see accomplishment when the tests pass Black-box Helps clarify interface and behavior From requirements analysis, there should before programming be validation criteria To write tests for something, first you need How are the developers and the customers to understand it well going to agree that the software is OK? Software evolution Many issues: functionality, performance, After changing existing code, rerun the tests usability, portability, etc. to gain confidence (regression testing) CSE 757 - Software Testing 39 CSE 757 - Software Testing 40 System Testing (cont) System Testing (cont) Initial system testing is done by the For multiple customers: two phases software producer Alpha testing: at the vendor’s site by a Eventually need testing by the customers few customers Customers are great testers Beta testing: distributed to many end- If the software is built for a single users customer: series of acceptance tests Users run it in their own environment Deploy in the customer environment and have Sometimes done by thousands of users end-users run it CSE 757 - Software Testing 41 CSE 757 - Software Testing 42 7
  8. 8. Stress Testing Stress Testing (cont) Form of system testing: behavior under Find how the system deals with overload very heavy load Reason 1: determine failure behavior e.g. normally there are 1-2 interrupts per If the load goes above the intended, how second; what if there are 10 interrupts? “gracefully” does the system fail? e.g. what if data sets that are an order of magnitude larger than normal? Reason 2: expose bugs that only occur e.g. what if our server gets 10 times more under heavy loads client requests than normally expected? Especially for OS, middleware, servers, etc. e.g. memory leaks, incorrect resource allocation and scheduling, race conditions CSE 757 - Software Testing 43 CSE 757 - Software Testing 44 Regression Testing Rerun old tests to see if anything was “broken” by a change Changes: bug fixes, module integration, maintenance enhancements, etc. Testing of Object-Oriented Software Need test automation tools Load tests, execute them, check correctness Everything has to be completely automatic Could happen at any time: during initial development or after deployment CSE 757 - Software Testing 45 CSE 757 - Software Testing 46 Object-Oriented Software One Difference: Unit Testing Widespread popularity in the last decade Traditional view of “unit”: a procedure Initial hopes: it would be easier to test In OO: a method is similar to a procedure OO software than procedural software But a method is part of a class, and is Soon became clear that this is not true tightly coupled with other methods and fields in the class Some of the older testing techniques are still useful The smallest testable unit is a class It doesn’t make sense to test a method as a New testing techniques are designed separate entity specifically for OO software Unit testing in OO = class testing CSE 757 - Software Testing 47 CSE 757 - Software Testing 48 8
  9. 9. Class Testing Example: Inter-method DU Pairs class A { Traditional black-box and white-box private int index; techniques still apply public void m1() { E.g. testing with boundary values index = …; Inside each method: at least 100% branch … coverage; also, DU-pairs inside a method m2(); } Extension: DU pairs that cross method private void m2() { … x = index; … } boundaries public void m3() { … z = index; … } Example: inside method m1, field f is } assigned a value; inside method m2, this test 1: call m1, which calls m2 and value is read reads the value of index test 2: call m1, and then call m3 CSE 757 - Software Testing 49 CSE 757 - Software Testing 50 Possible Test Suite Polymorphism public class MainDriver { A variable may refer to objects of public static void main(String[] args) { A a = new A(); different classes at different points of … time a.m1(); e.g., in Java, if D is a subclass of C, a a.m3(); variable of type C may refer to instances of … C or to instances of D } An operation may be defined in several Note: need to ensure that the actual execution classes and may have different exercises definition-free paths for the two DU implementations in each class pairs Method overriding CSE 757 - Software Testing 51 CSE 757 - Software Testing 52 Example of Polymorphism Testing of Polymorphism Example: class A with subclasses B and C During class testing of X: “drive” call site class A { … void m() {…} …} a.m() through all possible bindings class B extends A { … } All-receiver-classes: execute with at class C extends A { … void m() {…} … } least one receiver of class A, at least one Suppose inside class X there is call a.m(), receiver of class B, and at least one where variable a is of type A receiver of class C Could potentially send message m() to an All-invoked-methods: need to execute instance of A, instance of B, or instance of C with receivers that cover A.m and C.m The invoked method could be A.m or C.m i.e. (A or B receiver) and (C receiver) CSE 757 - Software Testing 53 CSE 757 - Software Testing 54 9
  10. 10. State-based Testing Example: Stack Natural representation with finite-state States machines Initial: before creation States correspond to certain values of the Empty: number of elements = 0 attributes Holding: number of elements >0, but less Transitions correspond to methods than the max capacity Full: number elements = max FSM can be used as basis for testing Final: after destruction e.g. “drive” the class through all transitions, and verify the response and the resulting Transitions: starting state, ending state, state action that triggers the transition, and possibly some response to the action CSE 757 - Software Testing 55 CSE 757 - Software Testing 56 Examples of Transitions Finite State Machine for a Stack Initial -> Empty: action = “create” Initial e.g. “s = new Stack()” in Java create add,delete Empty -> Holding: action = “add” add Empty -> Full: action = “add” Empty Holding delete if max_capacity = 1 add delete Empty -> Final: action = “destroy” add delete destroy e.g. destructor call in C++, garbage collection destroy in Java Final destroy Full Holding -> Empty: action = “delete” CSE 757 - Software Testing 57 CSE 757 - Software Testing 58 FSM-based Testing Inheritance Each valid transition should be tested People thought that inheritance will Verify the resulting state using a state reduce the need for testing inspector that has access to the internals of Claim 1: “If we have a well-tested superclass, the class we can reuse its code (in subclasses, through inheritance) without retesting inherited Each invalid transition should be tested code” to ensure that it is rejected and the Claim 2: “A good-quality test suite used for a state does not change superclass will also be good for a subclass” e.g. Full -> Full is not allowed: we should call add on a full stack Both claims are wrong CSE 757 - Software Testing 59 CSE 757 - Software Testing 60 10
  11. 11. Problems with Inheritance Testing of Inheritance Incorrect initialization of superclass Principle: inherited methods should be attributes by the subclass retested in the context of a subclass Missing overriding methods Example 1: if we change some method m Typical example: equals and clone is a superclass, we need to retest m Direct access to superclass fields from inside all subclasses that inherit it the subclass code Example 2: if we add or change a Can create subtle side effects that break unsuspecting superclass methods subclass, we need to retest all methods inherited from a superclass in the A subclass violates an invariant from the context of the new/changed subclass superclass, or creates an invalid state CSE 757 - Software Testing 61 CSE 757 - Software Testing 62 Example Another Example class A { class A { int x; // invariant: x > 100 void m() { … m2(); … } void m() { // correctness depends on void m2 { … } … } // the invariant … } … } class B extends A { class B extends A { void m2() { … } … } void m2() { x = 1; … } … } If inside B we override a method from A, If m2 has a bug and breaks the invariant, this indirectly affects other methods m is incorrect in the context of B, even inherited from A though it is correct in A e.g. m now calls B.m2, not A.m2: so, we cannot be sure that m is correct anymore Therefore m should be retested on B objects and we need to retest it inside B CSE 757 - Software Testing 63 CSE 757 - Software Testing 64 Testing of Inheritance (cont) Testing of Interacting Classes Test cases for a method m defined in Until now we only talked about testing of class X are not necessarily good for individual classes retesting m in subclasses of X Class testing is not sufficient e.g. if m calls m2 in A, and then some OO design: several classes collaborate to subclass overrides m2, we have a completely implement the desired functionality new interaction Still, it is essential to run all superclass A variety of methods for interaction tests on a subclass testing We will only consider testing based on UML Goal: check behavioral conformance of the interaction diagrams subclass w.r.t. the superclass (LSP) CSE 757 - Software Testing 65 CSE 757 - Software Testing 66 11
  12. 12. UML Interaction Diagrams for Testing Collaboration Diagram UML interaction diagrams: sequences of prepare() messages among a set of objects :Order 1.2.1:needsReorder:= There may be several diagrams showing needToReorder() 1*[all order lines]: different variations of the interaction prepare() * Basic idea: run tests that cover all :OrderLine 1.1:hasStock:=check() :StockItem diagrams, and all messages and conditions 1.2[hasStock]:remove() inside each diagram 1.3[hasStock]: If a diagram does not have conditions and create() 1.2.2[needsReorder]:create() iteration, it contains only one path :DeliveryItem :ReorderItem CSE 757 - Software Testing 67 CSE 757 - Software Testing 68 Coverage Requirements Run enough tests to cover all messages and conditions test with 0 loop iterations and >=1 iterations test with hasStock=true and hasStock=false test with needsReorder=true and needsReorder=false To cover each one: pick a particular path in the diagram and “drive” the objects through that path CSE 757 - Software Testing 69 12