Software Testing Software Testing

2,151 views
2,090 views

Published on

Software Testing Software Testing

  1. 1. 2.03.2006 IAF0030 IAF0030 Arvutitehnika erikursus I Programmers are in a race with the Universe to create bigger and better idiot-proof programs. Software Testing While the Universe is trying to create bigger and better idiots. So far the Universe is winning Gert Jervan Arvutitehnika instituut (ATI) Tallinn University Tallinna Tehnikaülikool of Technology Some slides © Hyoung Hong Concordia University, CA Lecture Outline Software Life Cycle Introduction Requirements Design Test Economics Implementation Types of Testing Testing Testing coverage Maintenance IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 3 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 4 The Product Development Cycle Software Development Costs Cost For life-critical software (e.g. flight control, reactor monitoring), testing can cost 3 to 5 Customer & market Release Testing times as much as all Driven inputs to Software other activities ar e SW manufacture Development Unit combined. f tw c Tes So Spe t New System HW-SW Product Design and Product Implementation Stop testing is a Spec H Integration Verification Idea ar business decision d Sp war est it T - There is always ec e Hardware Un HW Development something more to Requirements test Product Line Product - Risk based decision Engineering Development Management & Verification functions Engineering functions inputs IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 5 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 6 Gert Jervan, TTÜ/ATI 1
  2. 2. 2.03.2006 Software Life Cycle Costs Software Qualities Correctness Reliability (dependability) Cost Maintenance Robustness Safety Security (survivability) Performance Development Productivity Maintainability, portability, interoperability, … IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 7 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 8 Software Verification and Validation Techniques for V&V Verification Static 0 Are we building the product right? 0 Collects information about a software without 0 Process-oriented executing it • Does the product of a given phase fulfill the • Reviews, walkthroughs, and inspections requirements established during the previous phase? • Static analysis • Formal verification Validation 0 Are we building the right product? Dynamic 0 Product-oriented 0 Collects information about a software with • Does the product of a given phase fulfill the user’s executing it requirements? • Testing: finding errors • Debugging: removing errors IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 9 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 10 Static Analysis Formal Verification Control flow analysis and data flow analysis Given a model of a program and a property, 0 Extensively used for compiler optimization and software determine whether the model satisfies the engineering property based on mathematics Examples 0 Unreachable statements Examples 0 Variables used before initialization 0 Safety 0 Variables declared but never used • If the light for east-west is green, then the light for 0 Variables assigned twice but never used between south-north should be red assignments 0 Liveness 0 Variables used twice with no intervening assignment • If a request occurs, there should be a response 0 Possible array bound violations eventually in the future IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 11 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 12 Gert Jervan, TTÜ/ATI 2
  3. 3. 2.03.2006 Introduction to Testing Testing Debugging and testing are not the same thing! Apply input Observe output Software Testing is a systematic attempt to break a program. 0 Correct, bug-free programs by construction are the goal but until that is possible (if ever!) we Validate the observed output have testing. 0 Since testing is basically destructive in nature, Is the observed output the same as the expected output? it requires that the tester discard preconceived notions of the correctness of the software to be tested IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 13 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 14 Software Testing Fundamentals Limitations of Testing (I) Testing objectives include To test all possible inputs is impractical or 0 Testing is a process of executing a program impossible with the intent of finding an error. int foo(int x) { 0 A good test case is one that has a high y = very-complex-computation(x); probability of finding an as yet undiscovered write(y); error. } 0 A successful test is one that uncovers an as To test all possible paths is impractical or yet undiscovered error. impossible int foo(int x) { for (index = 1; index < 10000; index++) write(x); } IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 15 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 16 Limitations of Testing (II) Economics of Testing (I) Dijkstra, 1972 The characteristic S-curve for error removal 0 Testing can be used to show the presence of bugs, but never their absence Goodenough and Gerhart, 1975 We need 0 Testing is successful if the program fails other techniques Number of Testing is The (modest) goal of testing defects effective Cutoff point found 0 Testing cannot guarantee the correctness of software but can be effectively used to find errors (of certain types) Time spent testing IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 17 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 18 Gert Jervan, TTÜ/ATI 3
  4. 4. 2.03.2006 Economics of Testing (II) Economics of Testing (III) Testing tends to intercept errors in order of Verification is insensitive to the probability their probability of occurrence of occurrence of errors Progress of testing Number of Number of defects defects Progress of Not yet found verification Found Found Not yet found Less likely = Less likely = More critical More critical IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 19 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 20 Fundamental Questions in Testing Types of Testing When can we stop testing? Level 0 Test coverage regression What should we test? acceptance system 0 Test generation integration Is the observed output correct? unit Accessibility 0 Test oracle functional white grey black reliability How well did we do? robustness box box box 0 Test efficiency performance Who should test your program? usability 0 Independent V&V Aspect IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 21 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 22 Levels of Testing Component/Unit Testing (I) A unit of testing What users really need 0 Functions in procedural programming Acceptance testing languages such as C, Fortran, … Test driver F1(int x1, y1) { …… Requirements System testing F2(x1+1, y1-1); } Test unit F2(int x2, y2) { …… Design Integration testing F3(x2+2, y2-1); } Unit testing Test stub F3(int x3, y3) { Code …… } IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 23 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 24 Gert Jervan, TTÜ/ATI 4
  5. 5. 2.03.2006 Component/Unit Testing (II) Component/Unit Testing (III) Require knowledge of code Test case 0 High level of detail 0 Input, expected outcome, purpose 0 Deliver thoroughly tested components to 0 Selected according to a strategy, e.g., branch integration coverage Stopping criteria Outcome 0 Code Coverage 0 Pass/fail result 0 Quality 0 Log, i.e., chronological list of events from execution IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 25 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 26 Integration Testing (I) Integration Testing (II) Interactions among units (assembled Strategies for integration testing components that must be tested and 0 Top-down accepted previously) • Stubs are needed 0 Import/export type compatibility Main 0 Bottom-up 0 Import/export range errors • Drivers are needed F2 • F1 calls F2 with a parameter of array F1 0 Big-bang • F1 assumes array of size 8, while F2 assumes an array of size 10 0 Functional 0 Import/export representation • F1 calls F2 with a parameter Elapsed_time 0 Drivers & Fm Fn • F1 thinks in seconds, while F2 thinks in miliseconds stubs have to tested as well! IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 27 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 28 System Testing (I) System Testing (II) Tests the overall system (the integrated hardware Non-functional testing and software) to determine whether the system meets its requirements Quality attributes Focuses on the use and interaction of system 0 Performance, can the system handle required functionalities rather than details of throughput? implementations 0 Reliability, obtain confidence that system is Test cases derived from specification reliable Should be carried out by a group independent of 0 Timeliness, testing whether the individual tasks the code developers meet their specified deadlines Should be planned with the same rigor as other 0 etc. phases of the software development Use-case focus IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 29 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 30 Gert Jervan, TTÜ/ATI 5
  6. 6. 2.03.2006 Acceptance Testing Re-Test and Regression Testing (I) Re- User (or customer) involved Conducted after a change Environment as close to field use as Re-test aims to verify whether a fault is possible removed Focus on: 0 Re-run the test that revealed the fault 0 Building confidence Regression test aims to verify whether new 0 Compliance with defined acceptance criteria in faults are introduced the contract 0 Re-run all tests 0 Should preferably be automated IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 31 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 32 Re-test & Regression Testing (II) Re- Accessibility of Testing Development versus maintenance White box testing (structural testing, program- based testing) 0 Development costs: 1/3 White box testing is a test case design method that 0 Maintenance costs: 2/3 uses the control structure of the procedural design Testing in maintenance phase to derive test cases. Test cases can be derived that 0 How can we test modified or newly inserted 0 guarantee that all independent paths within a module have been exercised at least once, programs? 0 exercise all logical decisions on their true and false sides, • Ignore old test suites and make new ones from the scratch or 0 execute all loops at their boundaries and within their operational bounds, and • Reuse old test suites and reduce the number of new test suites as many as possible 0 exercise internal data structures to ensure their validity. IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 33 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 34 Accessibility of Testing (II) Program-Based Testing (I) Black box testing (functional testing, Main steps specification-based testing) 0 Examine the internal structure of a program 0 Design a set of inputs satisfying a coverage criterion 0 Assumes that the program is unavailable or 0 Apply the inputs to the program and collect the actual testers do not want to look at the details of the outputs program 0 Compare the actual outputs with the expected outputs • Derives test cases from the requirements of the program Limitations • Controls and observes the program only through 0 Cannot catch omission errors external interfaces • What requirements are missing in the program? • Ideally done by independent test group (not original 0 Cannot provide test oracles programmer) • What is the expected output for an input? Grey box testing IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 35 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 36 Gert Jervan, TTÜ/ATI 6
  7. 7. 2.03.2006 Program-Based Testing (II) Statement Coverage Statement coverage of a set of test cases is defined to be the proportion of statements in a unit covered by those test cases. Apply input Observe output Program 100% statement coverage for a set of tests means that all statements are covered by the tests. That is, all statements will be Validate the observed output against the expected output executed at least once by running the tests. Who will take care of test oracles? IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 37 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 38 Branch Coverage Example – Branch coverage Branch coverage is determined by the proportion of decision branches that are A What branch coverage is achieved exercised by a set of proposed test cases. C by ABG, ACDFG, ACEFG? B 100% branch coverage is where every D E decision branch in a unit is visited by at least one test in the set of proposed test F cases. G IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 39 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 40 Example – Branch coverage Example – Branch coverage A What branch coverage is achieved A What branch coverage is achieved by ABG, ACDFG, ACEFG? by ABG, ACDFG, ACEFG? B C B C D E D E F F G G IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 41 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 42 Gert Jervan, TTÜ/ATI 7
  8. 8. 2.03.2006 Example – Branch coverage Example – Branch coverage A What branch coverage is achieved A What branch coverage is achieved by ABG, ACDFG, ACEFG? by ABG, ACDFG, ACEFG? B C B C D E D E 4 in total. F F 4 covered G G So 4/4 = 100% branch coverage IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 43 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 44 Path Coverage Example – Path coverage Path coverage is determined by assessing the proportion of execution paths through a unit A What path coverage is achieved by exercised by the set of proposed test cases. ABG, ACDFG, ACEFG? 100% path coverage is where every path in the B C unit is executed at least once by the set of proposed test cases. D E 100% path coverage is achieved by an ideal test set. As we saw the other week, it is all but F impossible or infeasible in most programs of any size. G IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 45 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 46 Example – Path coverage Example – Path coverage A What path coverage is achieved by A What path coverage is achieved by ABG, ACDFG, ACEFG? ABG, ACDFG, ACEFG? B C B C D E D E 3/3=100% F F G G IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 47 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 48 Gert Jervan, TTÜ/ATI 8
  9. 9. 2.03.2006 Coverage An example It is possible to have 100% statement coverage without 100% branch coverage Test cases covering ABDEG and ACDFG It is possible to have 100% branch cover 4/4 branches coverage without 100% path coverage (100%) and 7/7 statements (100%) They, however, only 100% path coverage implies 100% branch cover 2/4 paths (50%). coverage and 100% branch coverage implies 100% statement coverage IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 49 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 50 An example An example Test cases covering ABDEG Test cases covering ABDEG and ACDFG cover 4/4 and ACDFG cover 4/4 branches (100%) and 7/7 branches (100%) and 7/7 statements (100%) statements (100%) They, however, only cover They, however, only cover 2/4 paths (50%). 2/4 paths (50%). 2 more tests are required to 2 more tests are required to achieve 100% path coverage achieve 100% path coverage 0 ABDFG 0 ABDFG, ACDEG IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 51 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 52 Loop Testing Mutation testing It is usually impossible or infeasible to test all Create a number of mutants, i.e., faulty versions of paths in a program involving loops program 0 Each mutant contains one fault Basis Path Testing 0 Fault created by using mutant operators 0 Zero path: Test zero iterations of the loop body (Guard is negated by loop initialisation) Run test on the mutants (random or selected) 0 One path: Test a single iteration of the loop body (Good 0 When a test case reveals a fault, save test case and remove mutant from the set, i.e., it is killed idea to try for 100% path coverage of loop body if loop body is not iterative) 0 Continue until all mutants are killed Results in a set of test cases with high quality 0 Does not consider maximum iteration termination in Need for automation many cases 0 Does not consider combinations of loop body paths in successive iterations IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 53 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 54 Gert Jervan, TTÜ/ATI 9
  10. 10. 2.03.2006 Specification-Based Testing (I) Specification-Based Testing (II) Main steps 0 Examine the structure of the program’s specification 0 Design a set of inputs from the specification satisfying a Expected output coverage criterion Specification 0 Apply the inputs to the specification and collect the Apply input expected outputs 0 Apply the inputs to the program and collect the actual outputs Actual output 0 Compare the actual outputs with the expected outputs Program Limitations 0 Specifications are not usually available • Many companies still have only code, there is no other Validate the observed output against the expected output document. IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 55 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 56 Object-Oriented Program Testing Object- Steps to Testing Nirvana Kernighan, Pike, “The Practice of Programming” Unit testing for OO Programs 0 A class is a set of variables and member functions Think about potential problems as you design and 0 50% of member functions are just 10 lines of code implement. Make a note of them and develop tests 0 A class is often a unit of testing in C++ or Java that will exercise these problem areas. Integration testing for OO Programs 0 Document all loops and their boundary conditions, all arrays and their boundary conditions, all variables and 0 Rule of thumb in OO development their range of permissible values. • Make a large number of small classes in a bottom-up fashion 0 Pay special attention to parameters from the command line and into functions and what are their valid and 0 There are several relationships between classes invalid values. • Association, aggregation, inheritance, concurrency 0 Enumerate the possible combinations and situations for a piece of code and design tests for all of them. 0 GIGO - what happens when garbage goes in? IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 57 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 58 Steps to Testing Nirvana Steps to Testing Nirvana Test systematically, starting with easy tests Within a module, test incrementally as you and working up to more elaborate ones. code 0 Often leads to “bottom up” testing, starting 0 Write, test, add more code, test again, repeat with simplest modules at the lowest level of 0 The earlier that errors are detected, the easier calling they are to locate and fix. 0 When those are working, test their callers 0 Testing is not only concerning code 0 Document (and/or automate) this testing so • Documents and models should also be subject to that it can be repeated (regression testing) testing constantly as the code grows and changes. IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 59 IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 60 Gert Jervan, TTÜ/ATI 10
  11. 11. 2.03.2006 Tricks of the Trade Test boundary conditions. 0 loops and conditional statements should be Questions? checked to ensure that loops are executed the correct number of times and that branching is correct 0 if code is going to fail, it usually fails at a boundary 0 check for off-by-one errors, empty input, empty output Gert Jervan Department Of Computer Engineering Tallinn University of Technology Tallinn University of Technology Estonia IAF0030 – Lecture 5 Gert Jervan, TTÜ/ATI 61 Gert Jervan, TTÜ/ATI 11

×