software-testing.ppt

12,779 views
12,648 views

Published on

1 Comment
3 Likes
Statistics
Notes
No Downloads
Views
Total views
12,779
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
457
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide

software-testing.ppt

  1. 1. Πληρουοριακά Συστήματα Επιχειρήσεων Διδάσκων Κώστας Κοντογιάννης Αναπλ. Καθηγητής, Ε.Μ.Π 1
  2. 2. Course Outline • Introduction to software engineering • Requirements Engineering • Design Basics • Traditional Design • OO Design • Design Patterns • Software Architecture • Design Documentation Verification & Validation • Software Process Management and Economics 2
  3. 3. • These slides are based on: – Lecture slides by Ian Summerville, see http://www.comp.lancs.ac.uk/computing/resources/ser/ 3
  4. 4. Overview Basics of Testing • Testing & Debugging Activities • Testing Strategies – Black-Box Testing – White-Box Testing • Testing in the Development Process – Unit Test – Integration Test – System Test – Acceptance Test – Regression Test • Practical Considerations 4
  5. 5. Static and dynamic V&V Static verification Requirements Detailed Implementation Implementation Architecture specification design V1 V1 Prototype Dynamic V &V Special cases • Executable specifications • 5 Animation of formal specs
  6. 6. Program testing • Can reveal the presence of errors NOT their absence – Only exhaustive testing can show a program is free from defects. However, exhaustive testing for any but trivial programs is impossible • A successful test is a test which discovers one or more errors • Should be used in conjunction with static verification • Run all tests after modifying a system 6 ©Ian Sommerville 1995
  7. 7. Testing in the V-Model Requirements Acceptance Customer test Developer Architectural System test Design Detailed Integration test Design Functional (BB) Module Structural Unit test (WB) implementation 7
  8. 8. Testing stages • Unit testing – Testing of individual components • Integration testing – Testing to expose problems arising from the combination of components • System testing – Testing the complete system prior to delivery • Acceptance testing – Testing by users to check that the system satisfies requirements. Sometimes called alpha testing 8
  9. 9. Types of testing • Statistical testing – Tests designed to reflect the frequency of user inputs. Used for reliability estimation. – Covered later in section on Software reliability. • Defect testing – Tests designed to discover system defects. – A successful defect test is one which reveals the presence of defects in a system. 9 ©Ian Sommerville 1995
  10. 10. Some Terminology – Failure • A failure is said to occur whenever the external behavior does not conform to system spec. – Error • An error is a state of the system which, in the absence of any corrective action, could lead to a failure. – Fault • An adjudged cause of an error. 10
  11. 11. Some Terminology fault, bug, error, It is there in the program… defect Fault Program state Error Observed Failure 11
  12. 12. Overview • Basics of Testing Testing & Debugging Activities • Testing Strategies – Black-Box Testing – White-Box Testing • Testing in the Development Process – Unit Test – Integration Test – System Test – Acceptance Test – Regression Test • Practical Considerations 12
  13. 13. Testing and debugging • Defect testing and debugging are distinct processes • Defect testing is concerned with confirming the presence of errors • Debugging is concerned with locating and repairing these errors • Debugging involves formulating a hypothesis about program behaviour then testing these hypotheses to find the system error 13 ©Ian Sommerville 1995
  14. 14. Debugging Activities Locate error & fault Design fault repair Repair fault Re-test program 14 ©Ian Sommerville 1995
  15. 15. Testing Activities Identify Test conditions (―What‖): an item or event to be verified. How the ―what‖ can be tested: realization Design Build test cases (imp. scripts, data) Build Run the system Execute Test case outcome with Compare expected outcome Test result 15
  16. 16. Testing Activities • Test condition – What: Descriptions of circumstances that could be examined (event or item). – Categories: functionality, performance, stress, robustness… – Derive • Using testing techniques (to be discussed) • (Refer to the V-Model) 16
  17. 17. Testing Activities • Design test cases: the details – Input values – Expected outcomes • Things created (output) • Things changed/updated  database? • Things deleted • Timing • … – Expected outcomes • Known • Unknown (examine the first actual outcome) – Environment prerequisites: file, net connection … 17
  18. 18. Testing Activities • Build test cases (implement) – Implement the preconditions (set up the environment) – Prepare test scripts (may use test automation tools) Structure of a test case Simple linear Tree (I, EO) {(I1, EO1), (I2, EO2), …} I EO1 EO2 18
  19. 19. Testing Activities • Scripts contain data and instructions for testing – Comparison information – What screen data to capture – When/where to read input – Control information • Repeat a set of inputs • Make a decision based on output – Testing concurrent activities 19
  20. 20. Testing Activities • Compare (test outcomes, expected outcomes) – Simple/complex (known differences) – Different types of outcomes • Variable values (in memory) • Disk-based (textual, non-textual, database, binary) • Screen-based (char., GUI, images) • Others (multimedia, communicating apps.) 20
  21. 21. Testing Activities • Compare: actual output == expected output?? – Yes • Pass (Assumption: Test case was ―instrumented.‖) – No • Fail (Assumption: No error in test case, preconditions) 21
  22. 22. Overview • Basics of Testing • Testing & Debugging Activities Testing Strategies – Black-Box Testing – White-Box Testing • Testing in the Development Process – Unit Test – Integration Test – System Test – Acceptance Test – Regression Test • Practical Considerations 22
  23. 23. Goodness of test cases • Exec. of a test case against a program P – Covers certain requirements of P; – Covers certain parts of P‘s functionality; – Covers certain parts of P‘s internal logic.  Idea of coverage guides test case selection. 23
  24. 24. Overview • Basics of Testing • Testing & Debugging Activities • Testing Strategies Black-Box Testing – White-Box Testing • Testing in the Development Process – Unit Test – Integration Test – System Test – Acceptance Test – Regression Test • Practical Considerations 24
  25. 25. Black-box testing • Approach to testing where the program is considered as a ‗black-box‘ • The program test cases are based on the system specification • Test planning can begin early in the software process • Possible test design strategy – Equivalence partitioning 25 ©Ian Sommerville 1995
  26. 26. Equivalence partitioning Invali d in pu ts Vali d in pu ts S y stem Ou tput s 26 ©Ian Sommerville 1995
  27. 27. Equivalence partitioning • Partition system inputs and outputs into ‗equivalence sets‘ – If input is a 5-digit integer between 10000 and 99999, equivalence partitions are <10000, 10000-99999 and >99999 – If you can predict that certain set of inputs will be treated differently in processing from another one, put them into separate partitions, e.g., Canadian and US addresses (will validate state/province), addresses in other countries (no state validation), etc. • Two test case selection strategies – Choose one random test case from each partition – Choose test cases at the boundary of the partitions • 09999, 10000, 99999, 100000 27 ©Ian Sommerville 1995
  28. 28. Equivalence partitions 09999 10000 50000 99999 100000 Less than 10000 Between 10000 and 99999 More than 99999 28 ©Ian Sommerville 1995
  29. 29. Search routine specification procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ; Pre-condition -- the array has at least one element T’FIRST <= T’LAST Post-condition -- the element is found and is referenced by L ( Found and T (L) = Key) or -- the element is not in the array ( not Found and not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key )) 29 ©Ian Sommerville 1995
  30. 30. Strategy for input partitions (I) (General) • Inputs which conform to the pre-conditions • Inputs where a pre-condition does not hold • Inputs derived from the post-condition – Inputs where the key element is a member of the array – Inputs where the key element is not a member of the array 30 ©Ian Sommerville 1995 [modified]
  31. 31. Strategy for input partitions (II) (Arrays) • Test software with arrays which have only a single value • Test with arrays of zero length (if allowed by programming language) • Use arrays of different sizes in different tests • Derive tests so that the first, middle and last elements of the array are accessed 31 ©Ian Sommerville 1995
  32. 32. Search routine - input partitions A rray Elem ent Single value In array Single value N ot in array More than 1 value First element in array More than 1 value L ast element in array More than 1 value Middle element in array More than 1 value N ot in array No value Not in array 32 ©Ian Sommerville 1995 [modified]
  33. 33. Search routine - test cases Input array (T ) Key (Key ) Output (Found, L ) 17 17 true, 1 17 0 false, ? ? 17, 29, 21, 23 17 true, 1 41, 18, 9, 31, 30, 16, 45 45 true, 6 17, 18, 21, 23, 29, 41, 38 23 true, 4 21, 23, 29, 33, 38 25 false, ? ? () 1 ??,?? 33 ©Ian Sommerville 1995 [modified]
  34. 34. Overview • Basics of Testing • Testing & Debugging Activities • Testing Strategies – Black-Box Testing White-Box Testing • Testing in the Development Process – Unit Test – Integration Test – System Test – Acceptance Test – Regression Test • Practical Considerations 34
  35. 35. White-box testing • Sometime called structural testing • Derivation of test cases according to program structure. Knowledge of the program is used to identify additional test cases • Possible test design strategy – Equivalence partitioning – Code coverage 35 ©Ian Sommerville 1995
  36. 36. Binary search (Ada) procedure Binary_search (Key ELEM ; T: ELEM_ARRAY ; : Found: in out BOOLEAN ; L: in out ELEM_INDEX ) is - Preconditions -- T’FIRST < =T’LAST and -- forall i: T’FIRST..T’LAST-1, T (i) <= T(i+1) Bott : ELEM_INDEX := T’FIRST ; Top : ELEM_INDEX := T’LAST ; Mid : ELEM_INDEX; begin L := (T’FIRST + T’LAST ) / 2; Found := T( L ) = Key; while Bott <= Top and not Found loop Mid := (Top + Bott) mod 2; if T( Mid ) = Key then Found := true; L := Mid; elsif T( Mid ) < Key then Bott := Mid + 1; else Top := Mid - 1; end if ; end loop ; end Binary_search; 36 ©Ian Sommerville 1995
  37. 37. Binary search (C++) void Binary_search (elem key, elem* T, int size, boolean &found, int &L) { int bott, top, mid ; bott = 0 ; top = size -1 ; L = ( top + bott ) / 2 ; if (T[L] == key) found = true ; else found = false ; while (bott <=top && !found) { mid = top + bott / 2 ; if ( T [mid] == key ) { found = true; L = mid ; } else if (T [mid] < key ) bott = mid + 1 ; else top = mid-1 ; } // while 37 } //binary_search ©Ian Sommerville 1995
  38. 38. Binary search - equiv. partitions • Pre-conditions satisfied, key element in array • Pre-conditions satisfied, key element not in array • Pre-conditions unsatisfied, key element in array • Pre-conditions unsatisfied, key element not in array • Input array has a single value • Input array has an even number of values • Input array has an odd number of values 38 ©Ian Sommerville 1995
  39. 39. Binary search equiv. partitions Eq ui valen ce cl ass bo un dari es El em ent s < M i d El em ent s > M i d M i d-p oi nt 39 ©Ian Sommerville 1995
  40. 40. Binary search - test cases Input arra y (T ) Key ( Key ) Output (Found, L ) 17 17 true, 1 17 0 false, ? ? 1 7, 2 1 , 2 3 , 29 17 true, 1 9 , 16 , 18 , 3 0, 3 1 , 4 1 , 45 45 true, 7 1 7, 1 8 , 2 1 , 23 , 2 9, 3 8 , 4 1 23 true, 4 1 7, 1 8 , 2 1 , 23 , 2 9, 3 3 , 3 8 21 true, 3 1 2, 1 8 , 2 1 , 23 , 3 2 23 true, 4 2 1, 2 3 , 2 9 , 33 , 3 8 25 false, ? ? 40 ©Ian Sommerville 1995
  41. 41. Code Coverage • Statement coverage – Elementary statements: assignment, I/O, call – Select a test set T such that by executing P for each case in T, each statement of P is executed at least once. – Read(x); read(y); if x > 0 then write(“1”); else write(“2”); if y > 0 then write(“3”); else write(“4”); – T: {<x = -13, y = 51>, <x = 2, y = -3>} 41
  42. 42. Code Coverage - Construct a control-flow graph of a program (module) c ~c c c ~c S1 S2 S1 S1 Assignment, I/O, call If c then S1 If c then S1 while c S1 else S2 -Edge coverage - Select a set T such that, by executing P for each member in T, each edge of P’s control-flow graph is traversed at least once. 42
  43. 43. Code Coverage • Condition coverage – Edge coverage plus – All possible values of the constituents of compound conditions are exercised at least once. (constituents: atomic formula  rel./Boolean var) – Example: • while (~found) and counter <= NumOfItems 43
  44. 44. Code Coverage • Path coverage – Path coverage focuses on executing distinct paths rather than just edges or statements – Paths through a program can be feasible or infeasible (e.g., due to contradicting conditions) – Ideally, one would like to cover all feasible paths, but the number of feasible paths usually explodes very quickly with the size of the program – Therefore, there are various path coverage strategies targeting specific subsets of the feasible paths 44
  45. 45. Code Coverage -Path coverage - Select a test set T such that, by executing P for each member of T, all paths leading from the initial node to the final node of P‘s control-flow graph are traversed. - Control-flow graph S1 S c Seq. do S until c If c While c then S S2 If c then S1 else S2 -Note: Separate node for each member of a compound condition -How many paths? 45
  46. 46. Code Coverage • Path testing based on cyclomatic complexity – (McCabe‘s) Cyclomatic complexity • V(G) = E – N + 2  E: # of edges, N: # of nodes • V(G) = p + 1  p is the # of predicate nodes • V(G) = # of regions (area surrounded by nodes/edges) – V(G): Upper bound on the # of independent paths • Independent path: A path with at least one new node/edge – Example: – V(G) = E – N + 2 = 17 - 13 + 2 = 6 – V(G) = p + 1 = 5 + 1 – V(G) = 6 – Advantage: The number of test cases is proportional to the size of the program 46
  47. 47. Code Coverage -Path coverage: example 1 : I = 1; 2 1 TI = TV = 0; F sum = 0; DO WHILE (value[I] <> -999 and TI < 100) 3 4 TI++; 2 3 R1 if (value[I] >= min and value[I] <= max){ 5 6 F 4 then {TV++; sum = sum + value[I];} } 7 5 R4 8 I++; F 10 R2 ENDDO T 9 6 T If TV > 0 10 12 R5 11 R3 av = sum/TV; 11 8 7 Else av = -999; 13 12 R6 9 13  final node 47 Outer region
  48. 48. V(G) as An Upper Bound 1 2 3 1 2 V(G) = 3 48
  49. 49. V(G)-Preserving Transformation SWITCH Nested IFs V(G) = 3 V(G) = 3 49
  50. 50. Code Coverage • More complete path testing strategies target loops, e.g., – 0 and 1 passes through every loop is tested – Multiple passes exercising different paths through the body of a loop – n passes (max = n) • Trying to be more complete on loop testing quickly becomes infeasible 50
  51. 51. Code Coverage -Data flow testing ( D = Definition, U = Use) -DU chain: -Assign a unique number to each statement -DEF(S): set of all variables defined in statement number S -USE(S): set of all variables used in statement number S -Live variable: Def.of var X at S is live at S‘ if -There is a path from S to S‘ that contains no other def of X -DU chain: [X, S, S‘] -X is in DEF(S) and in USE(S‘), and -Def of X at S is live at S‘ -DU coverage -Every DU chain be covered at least once. 51
  52. 52. Overview • Basics of Testing • Testing & Debugging Activities • Testing Strategies – Black-Box Testing – White-Box Testing • Testing in the Development Process Unit Test – Integration Test – System Test – Acceptance Test – Regression Test • Practical Considerations 52
  53. 53. Unit Testing Objective: Find differences between specified units and their imps. Unit: component ( module, function, class, objects, …) Unit test environment: Driver Test cases Test result Unit under Effectiveness? • Partitioning test • Code coverage Stub Stub Dummy modules 53
  54. 54. Overview • Basics of Testing • Testing & Debugging Activities • Testing Strategies – Black-Box Testing – White-Box Testing • Testing in the Development Process – Unit Test Integration Test – System Test – Acceptance Test – Regression Test • Practical Considerations 54
  55. 55. Integration Testing • Objectives: • To expose problems arising from the combination • To quickly obtain a working solution from comps. • Problem areas – Internal: between components • Invocation: call/message passing/… • Parameters: type, number, order, value • Invocation return: identity (who?), type, sequence – External: • Interrupts (wrong handler?) • I/O timing – Interaction 55
  56. 56. Integration Testing • Types of integration – Structural • “Big bang”  no error localization • Bottom-up: terminal, driver/module, (driver  module) • Top-down: top, stubs, (stub  module), early demo – Behavioral • (next slide) 56
  57. 57. Integration Testing (Behavioral: Path-Based) A B C MM-path: Interleaved sequence of module exec path and messages Module exec path: entry-exit path in the same module Atomic System Function: port input, … {MM-paths}, … port output Test cases: exercise ASFs 57
  58. 58. Overview • Basics of Testing • Testing & Debugging Activities • Testing Strategies – Black-Box Testing – White-Box Testing • Testing in the Development Process – Unit Test – Integration Test System Test – Acceptance Test – Regression Test • Practical Considerations 58
  59. 59. System Testing • Concerns with the app‘s externals • Much more than functional – Load/stress testing – Usability testing – Performance testing – Resource testing 59
  60. 60. System Testing • Functional testing – Objective: Assess whether the app does what it is supposed to do – Basis: Behavioral/functional specification – Test case: A sequence of ASFs (thread) • (Refer to pages 22-24 of ECE 355 PBX Project Desc.) 60
  61. 61. System Testing • Functional testing: coverage • Event-based coverage – PI1: each port input event occurs – PI2: common sequences of port input event occurs – PI3: each port input in every relevant data context – PI4: for a given context, all possible input events – PO1: each port output event – PO2: each port output event occurs for each cause • Data-based – DM1: Exercise cardinality of every relationship – DM2: Exercise (functional) dependencies among relationships 61
  62. 62. System Testing • Stress testing: push it to its limit + beyond Volume Users Application response : (System) rate Resources: phy. + logical 62
  63. 63. System Testing • Performance testing – Performance seen by • users: delay, throughput • System owner: memory, CPU, comm – Performance • Explicitly specified or expected to do well • Unspecified  find the limit • Usability testing – Human element in system operation • GUI, messages, reports, … 63
  64. 64. Test Stopping Criteria • Meet deadline, exhaust budget, …  management • Achieved desired coverage • Achieved desired level failure intensity 64
  65. 65. Overview • Basics of Testing • Testing & Debugging Activities • Testing Strategies – Black-Box Testing – White-Box Testing • Testing in the Development Process – Unit Test – Integration Test – System Test Acceptance Test – Regression Test • Practical Considerations 65
  66. 66. Acceptance Testing • Purpose: ensure that end users are satisfied • Basis: user expectations (documented or not) • Environment: real • Performed: for and by end users (commissioned projects) • Test cases: – May reuse from system test – Designed by end users 66
  67. 67. Overview • Basics of Testing • Testing & Debugging Activities • Testing Strategies – Black-Box Testing – White-Box Testing • Testing in the Development Process – Unit Test – Integration Test – System Test – Acceptance Test Regression Test • Practical Considerations 67
  68. 68. Regression Testing • Whenever a system is modified (fixing a bug, adding functionality, etc.), the entire test suite needs to be rerun – Make sure that features that already worked are not affected by the change • Automatic re-testing before checking in changes into a code repository • Incremental testing strategies for big systems 68
  69. 69. Overview • Basics of Testing • Testing & Debugging Activities • Testing Strategies – Black-Box Testing – White-Box Testing • Testing in the Development Process – Unit Test – Integration Test – System Test – Acceptance Test – Regression Test Practical Considerations 69
  70. 70. How to Design Practical Test Cases (Experience of Hitachi Software Engineering) Bug detection during a development phase 100% 50% 0% Desk Unit Integrate System Field failure 10 engineers x 12 months  100,000 LOC 0.02% 1000 bugs  < 1 bug appears at the customer‘s site 70
  71. 71. Testing effectiveness • In an experiment, black-box testing was found to be more effective than structural testing in discovering defects • Static code reviewing was less expensive and more effective in discovering program faults 71
  72. 72. Practical test cases • Document test cases – New viewpoint to functional spec – Validation of test cases • Well-balanced: normal, abnormal, boundary • Correctness test cases – Estimating quality • 100/1000 test cases show 4 bugs  40 bugs 72
  73. 73. Practical test cases • Schedule (rough idea) – 10 engineers, 12 months, 100,000 LOC in C – Apportioning 12 months • SRS: 2; Design: 3; Coding: 2 • Debugging (U + I + S tests): 3 • QA testing: 2 (shippable??, redesign tests—no reuse) • How many test cases (30-year empirical study) – 1 test case per 10-15 LOC – 100,000 LOC  1000 test cases – 2 weeks to design 1000 test cases – 2 months to execute 1000 test cases (25/day) 73
  74. 74. Practical test cases (steps for designing test cases) • Designing test cases – Only one rule: Cover all the features. • Visit all functions: normal, abnormal, boundary val Ex.: Admission fee varies by age 0 <= age <= 6 years: no fee 6 < age <= 12: $5 12 < age <= 18: $8 18 < age: $10 Test case: {2, 10, 14, 30}, {-1,0,6,7,12,13,18,19} • White-box: IF is a better measure than LOC. • Loops: 0, 1, 2, average, max-1, max, max+1 74
  75. 75. Practical test cases • Designing test cases (contd.) – Distribution of test cases • Normal: 60% • Boundary: 10% • Error: 15% • Environmental (platform + performance): 15% – Finishing touch • 48-hour continuous operation test (basic function)  memory leakage, deadlock, connection time-out 75
  76. 76. Practical test cases • Checking test case quality – Check correctness – Balance between normal, abnormal, boundary, environmental – Balance between black-box and white-box • The level of testing driven by target reliability of the system (see Lecture on Reliability) 76
  77. 77. Overview • Basics of Testing • Testing & Debugging Activities • Testing Strategies – Black-Box Testing – White-Box Testing • Testing in the Development Process – Unit Test – Integration Test – System Test – Acceptance Test – Regression Test • Practical Considerations 77

×