Software Testing and Validation


Published on

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Software Testing and Validation

  1. 1. 1. Introduction • The software engineering approach, as described in literature and software Software Testing and Validation engineering standards, works toward a single goal: to produce high-quality Juha Taina software. – The users of current software products are University of Helsinki surprisingly tolerant to software problems, Department of Computer Science low-quality software always becomes expensive in the long run. Software Testing and Validation, page -1- Software Testing and Validation, page -2- • While this looks clean enough, we still have • For our purposes, software quality is to answer the fundamental questions: defined as: – What kind of software fulfils the high-quality – Conformance to explicitly stated functional and software requirements? performance requirements, explicitly – How can we ensure that we create high-quality documented development standards, and software? implicit characteristics that are expected of all professionally developed software (Pressman, • High quality software views: Software Engineering, 1997). – fulfils customers’ needs, – follows requirements specifications, – gives plenty of functionality, – best what one can buy with this price. Software Testing and Validation, page -3- Software Testing and Validation, page -4- • The definition gives us the following points: • One of the implicit software requirement is – Requirements specifications define the basis for that is it well tested. high-quality software. A software product that • We cannot test a program completely, but does not follow its specifications is not a high- we can create enough test cases to create a quality product. good test coverage for the product. – The theory of software engineering defines the framework where high-quality software must • Why can’t we test a program completely? be developed. If used software engineering – The number of possible test inputs and methods are not within the framework, the execution paths through the program can easily result is not a high-quality program. exceed our computing capacity. – A high-quality software follows both implicit – Even if this was possible, each bug fix would and explicit software requirements. require a rerun of the test cases. Software Testing and Validation, page -5- Software Testing and Validation, page -6- 1
  2. 2. • Let’s take an example. Myers uses the • Although the problem looks simple at first following program as a self-assessment for sight, a significant number of test cases his readers’ ability to specify adequate must be developed: testing (Myers, the Art of Software Testing, – All input-related problems: Wiley, 1979): • too few or too many parameters, – A program reads three integer values. The three values • parameters are not integers, are interpreted as representing the lengths of the sides • negative or zero value parameters, of a triangle. The program prints a message that states • parameters with +-sign and/or spaces, whether the triangle is scalene, isosceles, or equilateral. Develop a set of test cases that you feel will adequately • integer value overflow problems, test this program. • etc. • Try to think of the possible test cases before peeking the next slides. Software Testing and Validation, page -7- Software Testing and Validation, page -8- – Considering that the input values are correct – Output-related problems: integers: • test that the output message is correct (no spelling • test that the three parameters actually form a errors in any of the messages etc.), triangle, • test indirect output (unix pipes for instance), • test that the program results a scalene triangle if and • test that output is correct in localised environments only if none of the parameters are equal, (Chinese characters etc). • test that the program results a isosceles triangle if • etc. and only if exactly two of the parameters are equal, • Conclusion: Even in such a small program • test that the program results an equilateral triangle if and only if all three parameters are equal. as this, the number of test cases grows very large. • Corollary: Testing is not as straightforward as it might first sound like. Software Testing and Validation, page -9- Software Testing and Validation, page -10- 1.1. Testing as a verification and • Testing is a tool for assuring that the tested validation tool product fulfils high-quality software requirements. It is used in Quality • We define software testing as follows: assurance: – Testing is a managed activity that – Quality assurance: (1) A planned and systematically by using theoretically accepted systematic pattern of all actions necessary to testing methods tries to reduce the number of provide adequate confidence that an item or defects in a software product. product conforms to established technical • Thus, we consider testing to be requirements. (2) A set of activities designed to – theoretically sound, evaluate the process by which products are – managed, and developed or manufactured. – systematic. Software Testing and Validation, page -11- Software Testing and Validation, page -12- 2
  3. 3. • The quality assurance activities can be • Testing is not the only tool for verification categorised into verification and validation. and validation. – Verification refers to the set of activities that – Inspections are used to verify a small piece of ensure that the software correctly implements a documentation or software code. An inspection specific function. group gathers into an inspection meeting to – Validation encompasses the set of activities that review the inspected component. All found ensure that the software is traceable to problems are written down but their possible customers’ requirements and users’ needs. solutions are not discussed in the meeting. • Usually this is stated as follows: – Simulation is used especially in validation. A finished software is put into a simulated – Verification: Are we building the product right? environment to check for possible defects. This – Validation: Are we building the right product? often saves both time and money. Software Testing and Validation, page -13- Software Testing and Validation, page -14- • More verification and validation tools: – Modeling belongs to all high-quality software 1.2. Principles of software testing creation. Several models of various abstraction • Why testing? Why systematic testing? levels can be used in validation. A most common example of this is the use of Universal – According to Beizer in “Software Testing Modeling Language UML. Techniques”, a professionally produced commercial software system contains 1-3 – Formal methods are used whenever we have a defects per 100 executable statements when the mathematical model of the product. With software engineering group has declared it mathematical formalism it is possible both to error-free. automatically create a frame for the software product and to prove its correctness. • Unfortunately this approach is suitable for only a small subset of software products. Software Testing and Validation, page -15- Software Testing and Validation, page -16- – Also Beizer in “Software System Testing and – Common estimates of the cost of finding and Quality Assurance” reported his private bug fixing errors in programs range from 40% to rate - how many mistakes he made in designing 80% of the total development cost. and coding a program - as 1.5 errors per – The average debugging effort is about 12 hours executable statement. This includes all of working time for a single defect, testing is mistakes, including typing errors. one of the most costly phases in a software – While most programmers catch and fix more engineering process. than 99% of their mistakes before releasing a • Corollary: Companies don’t spend fortunes program for testing, there is still the missing to testing and maintenance to “verify that a 1% left. program works.” They spend the money – Human work is needed in test case design, implementation, management, and analysis. because the program does not work - it has bugs which need to be found. Software Testing and Validation, page -17- Software Testing and Validation, page -18- 3
  4. 4. • Testing is a process of executing or • A successful test is one that uncovers an as- analysing a software program, its structure yet undiscovered error. Thus, testing is or documentation with the intent of finding destructive behaviour since its intent is to an error. Thus, testing is not restricted to demolish an existing product. the software code. • Testing cannot show the absence of defects. • A good test case is one that has a high It can only show that software errors are probability of finding an as-yet present. Thus, testing cannot prove that a undiscovered error. Thus, test cases must program is correct. cover the complete product. • Testing is by itself a process that must be systematically managed. Thus, testing is part of software process management. Software Testing and Validation, page -19- Software Testing and Validation, page -20- • Before applying methods to design effective • Basic principles continue. test cases, a software engineer must – Testing should begin “in the small” and understand the basic principles that guide progress toward testing “in the large”. software testing. – Exhaustive testing is not possible. – All tests should be traceable to implicit and – To be most effective, testing should be explicit customer requirements. conducted by an independent third party. – Tests should be planned before testing begins. • A software engineer should design a – 80% of all errors uncovered during testing will computer program that is easy to test. likely be traceable to 20% of all program – Fortunately a program that is designed using modules. generally accepted software design and analysis methods is also easy to test. Software Testing and Validation, page -21- Software Testing and Validation, page -22- The “V” model of software testing (Paakki, 1.3. The V-model of testing Software Testing, 2000): • Like in other steps in software management Requirements Acceptance elicitation processes, also testing needs to be planned testing and conducted in steps. Requirements System analysis testing • Usually the overall organisation of testing Architecture design Integration testing follows the V-model. Module Module design testing • The model integrates testing and Coding construction by showing how the testing levels verify the constructive phases. Specification Planning Testing level (phase) Software Testing and Validation, page -23- Software Testing and Validation, page -24- 4
  5. 5. • The testing process is organised in levels • The presented V-model testing levels are according to the V-model. module testing, integration testing, system • The levels reflect the different viewpoints testing, and acceptance testing. of testing on different levels of detail. – In module testing, both module structure and functionality are tested. The structure testing is • Using the V-model integrates testing better often called white-box testing since source code to the complete software engineering is visible in it. The functionality testing, on the process. As such it helps to manage testing. other hand, is called black-box testing, since in • The presented V-model is not the only one. it the internal structure of the module is not In fact most companies have their own visible - only its definition is known. • Module testing is mostly white-box testing. variations of the model. Software Testing and Validation, page -25- Software Testing and Validation, page -26- – In integration testing the objective is to verify – Acceptance testing is the final testing stage that modules co-operate correctly. This is before taking the system into full operation. In achieved by testing both interfaces and global acceptance testing customers and users check relationships between integrated modules. that the system fulfils their actual needs. • Integration testing is mostly black-box testing, Acceptance testing is first conducted in although also white-box testing can be used to verify laboratory conditions (so called alpha-testing) module relationships (for instance global variables). and finally in a real environment (so called – At the system testing level, the whole system beta-testing). including hardware, software, and external • Acceptance testing is completely black-box testing programs, is tested under the assumption that as was system testing. Since these testing phases are individual elements are already tested. very similar, they are often combined. • System testing is completely black-box testing. No source code should be visible anywhere. Software Testing and Validation, page -27- Software Testing and Validation, page -28- • Since this material is about testing, we will 2. Software quality and errors concentrate on the flaw part of quality: How • The primary task of testing is to find and will we know a flaw when we find it? report errors. – Thus, what constitutes an error? • The result of testing is hopefully • A software error is a mismatch between the improvement in product quality. program and its specification? • A program’s quality depends on: – Not really. Bad specification leads to bad programs, and implicit requirements exist – the features that make the customer want to use outside the specification. the program, and – the flaws that make the customer wish he’d bought something else. Software Testing and Validation, page -29- Software Testing and Validation, page -30- 5
  6. 6. • A software error is present when the • Kaner et al. describes 13 major software program does not do what its end user error categories in “Testing Computer reasonably expects it to do? Software”: – Better. However, since different users have – User interface errors: different expectations, what constitutes a • missing functionality, software error also varies. One man’s cheese is • missing (on-line) documentation, other man’s rotten milk. • bad command structure, • Thus, there can never be an absolute • missing user commands, definition for software errors. The extent to • bad performance, • badly designed output. which a program has faults is measured by the extent to which it fails to be useful. Software Testing and Validation, page -31- Software Testing and Validation, page -32- – Error handling: – Calculation errors: • failure to anticipate the possibility of errors and • formula errors, protect against them, • precision errors, • failure to notice error conditions, • rounding and truncation errors, • failure to deal with a detected error in a reasonable • calculation algorithm errors. way. – Initial and later states: – Boundary-related errors: • program fails in the first execution round, • numeric boundaries: very small numbers, very large • program fails at the second execution round, numbers, number near zero; • back-up and undo processes are flawed. • program use boundaries: first time, second time, a very large time, after a new document etc; – Control flow errors: • memory configuration errors: too little memory, too • wrong execution order, much memory (yes, it’s possible). • wrong condition branch. Software Testing and Validation, page -33- Software Testing and Validation, page -34- – Errors in handling or interpreting data: – Hardware: • misinterpreted data, • program sends bad data to devices, • missed latest changes during data exchange. • program ignores device status information, – Race conditions: • program doesn’t recognise hardware failures. • program expects two events A and B to occur in – Source and version control: order AB while also order BA is possible (while • old incompatible versions of modules or libraries very rare). used in a new program, – Load conditions: • incorrect copyright messages, sign-on screens, or • program fails under a high volume (much work over version numbers, a long period), • incorrect platform information. • program fails under a high stress (maximum load at – Documentation: one time). • poor documentation, • incomplete documentation. Software Testing and Validation, page -35- Software Testing and Validation, page -36- 6
  7. 7. – Testing errors: • errors in test code, 3. Testing management • errors in test data, • misunderstood program functionality in tests, • Testing can be seen as a process by itself. It • incomplete test coverage. needs to be planned, executed according to • Recognising these error classes gives us the the plan, documented, and reported. first basis for error detection. A more • A well planned and managed testing process detailed analyse is of course needed for simplifies test procedures, helps to create each tested product. high-quality software, and also helps to • Errors do vary. Some are inconvenient. repeat tests when necessary. Others cause loss of human lives. Software Testing and Validation, page -37- Software Testing and Validation, page -38- • The following documents can be used for – Test-case specification defines a test case managing the testing process (Source: identified by a test-design specification. IEE829 Standard for Software Test – Test-procedure specification specifies the steps Documentation). The most important for executing a set of test cases. documents are in bold. – Test-item transmittal report identifies the test items being transmitted for testing, including – Test plan prescribes the scope, approach, the person responsible for each item, its resources, and schedule of the testing activities. physical location, and its status. – Test design specification specifies the – Test log provides a chronological record of refinements of the test approach and identifies relevant details about the execution of tests. the features to be tested by the design and its associated tests. Software Testing and Validation, page -39- Software Testing and Validation, page -40- Project Item – Test-incident report (or a Bug report) Doc Doc Test Item documents any event that occurs during the testing process which requires investigation. Test Plan – Test-summary report summarises the results of the designated activities and provides Test Design Test Design Test Design evaluations based on these results. Spec Spec Spec • The list must be seen as a guideline from Test Case which each organisation can specify and Spec tailor its testing documentation practices. Test Proc Test Item Transmittal Spec • The document relationships can be seen in Report the next two slides. Test Execution Software Testing and Validation, page -41- Software Testing and Validation, page -42- 7
  8. 8. Test Item Test Proc Transmittal • The test process model may vary between Spec Report Test Execution testing groups. • The V-model defines the general steps in Test Test testing. Each V-model test phase has private Log Log process structure that may be linear or Test Incident Test Incident iterative. Report Report • One possible test process model as presented by Paakki is in the next slide. It Test Summary is a minimal model in the sense that it includes only the most necessary steps. Software Testing and Validation, page -43- Software Testing and Validation, page -44- • Next to the test process management, also testing staff must be managed. There are Requirements Require- Test summary engineering ments report Test Test Evaluation Res ults four major ways to share testing responsibilities (Paakki, 2000): plan planning of results Inspection Management Bug reports – 1. Testing is each person’s responsibility: the of testing product developers are also responsible for testing their own code. Drawback: testing must Test case specific ation Test cas es Scripting Test scripts Test exec ution be done independently; programmers are too biased by their own solutions and blind to their errors. Inspection Software Testing and Validation, page -45- Software Testing and Validation, page -46- – 2. Testing is each group’s responsibility: the – 4. Testing is performed by a test organization: product developers within the project group test testing is a component of quality assurance with each other’s code. Microsoft: “joint ownership a dedicated team. Drawback: additional and responsibility”. Drawback: usually organizations and processes. development efforts finally dominate over • Some testing tasks are among the most testing efforts. mechanical tasks in a software engineering – 3. Testing is performed by a dedicated process (test execution from someone else’s resource: there is a person in the project group who only concentrates on testing. Drawback: a scripts, for instance). It is important to person might be nominated who is less ensure that the testing staff does not wear effective in development (and probably in out in testing. testing as well). Software Testing and Validation, page -47- Software Testing and Validation, page -48- 8
  9. 9. • In white-box testing we test program 4. White-box testing structure. We traverse through possible paths, test conditions, loops, and module • White-box or Glass-box testing is a testing parameters. phase where source code is visible. • In the other testing type, black-box testing, • In white-box testing, the testing group uses we test program functionality. Program their understanding and access to the source functionality is tested from outside, which is code to develop test cases. how the customer will work with it. • White-box testing is the kind of testing that • Both white-box and black-box testing are is best suited to do once a module or module needed. White-box testing finds low-level integration is complete. bugs, black-box finds high-level bugs. Software Testing and Validation, page -49- Software Testing and Validation, page -50- • White-box testing provides the following – Control flow: The programmer can add test- benefits: specific code so that the program constantly reports of what it is doing. The tester can use – Focused testing: White-box testing is especially this code, or run a debugger to track the order suitable for module-level testing. It’s much in which lines of code are executed. The easier to test an individual module with white- trouble points can be examined and reported box methods than with black-box methods. immediately. – Testing coverage: The tester can find out – Data integrity: The tester can see which parts of which parts of the program are exercised by the program modify any item or data. By any test. White-box testing can tell which lines tracking a data item through the system, she can of code, which branches, or which paths spot data manipulation by inappropriate haven’t yet been tested, and she can add tests modules. that she knows will cover areas not yet touched. Software Testing and Validation, page -51- Software Testing and Validation, page -52- – Internal boundaries: The tester can see internal • White-box testing methods should always boundaries in the code that are completely be present in the programming process. The invisible to black-box testers. For instance, earlier white-box testing is exercised, the some programs store data into a temporary buffer if too much comes in too quickly. This earlier low-level bugs are found. is visible only at white-box testing. • In an optimal case in the testing process the – Algorithm-specific testing: Some test cases can testers could concentrate on black-box be derived directly from the used algorithms. testing. In practice this is not possible, and For example, there are many ways to invert a testers should at least be familiar with matrix, and well understood ways to white-box testing methods. miscalculate the result. The programmer can apply standard numerical analysis techniques to • White-box testing can be at least partially predict the results. automated with appropriate testing tools. Software Testing and Validation, page -53- Software Testing and Validation, page -54- 9
  10. 10. • The principle in white-box testing is to • The criterion for selecting the execution select test cases such that they force the paths depends on the chosen white-box program to move along different kinds of method. However, all methods are based on paths during its execution. the analysis of the control flow of the • A path is one flow of control logic from the program. beginning of the tested entity to the end of • The control flow is usually presented as a the entity during execution. flow graph. The nodes in a flow graph • When the tested entity is a program module, present program statements, and edges a path starts from the module start and ends present control transfer between the to one of the exit points in the module. statements. Software Testing and Validation, page -55- Software Testing and Validation, page -56- Flow-graph structures: yes s s p yes no p s1 yes no no p (c) Loop (while-do) statement: while (p) do s (d) Loop (do-while) statement: s2 s1; do s while (p) s2; OR …; s1 s2 s1 sn e1 s1 e2 e s2 yes sn p s … en no sn (b) Conditional (if) statement: s2 (a) Statement sequence: if (p) then s1 else s2 (f) Switch (case) statement: s1; s2; …; sn (e) Iterative (for) statement: switch (e) {case e1: s1; for (s1; p; s2) do s case e2: s2; …; case en: sn} Software Testing and Validation, page -57- Software Testing and Validation, page -58- • Coverage of a program is quantitative. We 4.1. Coverage can calculate the percentage of coverage in • A white-box testing strategy is designed and a module. This depends on the chosen measured in terms of coverage. It defines coverage criterion. how well a module is tested with respect of • Before we cover various coverage criteria the chosen test policy. we need an informal definition for an • For the sake of simplicity we consider a execution path: tested unit to be a module. – An execution path in a flow graph is a sequence of nodes starting from the beginning of the • Coverage is defined with a coverage module and ending to a module exit point. criterion. It indicates how extensively the module has been tested with given test cases. Software Testing and Validation, page -59- Software Testing and Validation, page -60- 10
  11. 11. • The definition of the execution path is • At least the following coverage criteria has important for all white-box testing. been defined: • Thus, in white-box testing we assume that a – Statement coverage: A set P of execution paths flow graph of the tested module is present. satisfied the statement coverage criterion if and only if for all nodes n in the flow graph, there is • Although we assume that the tested unit is a at least one path p in P such that p contains the module, the methods can be expanded to node n. any program unit where source code is • In other words, statement coverage criterion is present. fulfilled if test cases force each statement of the tested module to be executed at least once. • If a set of execution paths satisfies the statement coverage criterion, we are said to have achieved a complete statement coverage. Software Testing and Validation, page -61- Software Testing and Validation, page -62- – Branch coverage: A set P of execution paths – Control coverage: A set P of execution paths satisfies the branch coverage criterion if and satisfies the condition coverage criterion if and only if for all edges e in the flow graph, there is only if for every control node in the flow graph at least one path p in P such that p contains the consisting of atomic predicates (, ci edge e. yields true when evaluated within a path p1 in • In other words, the test cases must cover each P, and ci yields false when evaluated within a branch for both the true and the false values. path p2 in P, i = 1,..n • The branch coverage criterion is a stronger criterion • In other words, each atomic test (e.g. a==1) must be than the statement coverage criterion. For instance, evaluated at least once to true, and at least once to a test “if (a==1) do_something;” needs a one test false in the test cases. case for statement coverage (a=1) but two test cases • If a branch node has n atomic predicates (e.g. a<1 || for branch coverage (a=1, a=0). a >2 || b==0), as many test cases are needed to fulfil condition coverage criterion (tff,ftf,fft). Software Testing and Validation, page -63- Software Testing and Validation, page -64- – Multicondition coverage: A set P of execution • The essential differences between statement paths satisfies the multicondition coverage coverage, branch coverage, condition criterion if and only if for every control node in coverage, and multicondition coverage can the flow graph consisting of atomic predicates (c1,…cn), each possible combination of their be summarized by the following conditional truth values (true,false) is evaluated within at statement and its flow graph: least one path p in P. if ((x != 0) && (z >=0)) y = z/x; • In other words, when a branch node has n atomic predicates (e.g. a<1 || a>2 || b==0), n*n test cases are needed to fulfil the multicondition coverage (x!=0) yes criterion (fff,fft,ftf,ftt,tff,tft,ttf,ttt). && y=z/x (z>=0) • This is the most complete coverage criterion of no practical relevance when testing is driven through exploration of control predicates. Software Testing and Validation, page -65- Software Testing and Validation, page -66- 11
  12. 12. – Complete statement coverage over the module can be reached with a single test: (x=1,z=0). • Next to the branch-based coverage criterion, – Complete branch coverage calls for at least two we can define path-based coverage sets: (x=1,z=0) for the yes-branch, and (x=1,z=- criterion. Here we define two criteria: path 1) for the no-branch. coverage and independent path coverage. – Complete condition coverage needs at least two – Path coverage: A set P of execution paths tests: (x=0,z=0) for the atomic combination satisfies the path coverage criterion if and only (false,true) over the control predicate, and if P contains all execution paths from the begin- (x=1,z=-1) for the atomic combination node to the end-node in the flow graph. (true,false). • This is an ultimate white-box coverage criterion. – Complete multicondition coverage needs four • Unfortunately loops make it practically impossible to reach full path coverage for even for relatively tests: (x=1,z=0), (x=1,z=-1),(x=0,z=0), and simple programs. (x=0,z=-1). (tt,tf,ft,ff). Software Testing and Validation, page -67- Software Testing and Validation, page -68- – Independent path coverage: A set P of • The independent path coverage is a good execution paths satisfies the independent path criterion for path-based testing. It can be coverage criterion if and only if P contains all made more complete by breaking tests to the p+1 independent paths over the flow graph, where p is the number of predicate nodes in the atomic predicates with the following simple graph. rules: if (A && B) s1; else s2; ==> if (A) if (B) s1; else s2; else • A set of paths is called independent if none of them s2; is a linear combination of the others. In other if (A || B) s1; else s2; ==> if (A) s1; else if (B) s1; else s2; words, a linear path cannot be combined from other linear paths. – This way independent path coverage criterion • Independent paths collapse loops into single comes closer to the multicondition coverage iterations and therefore dramatically reduce the criterion. number of test cases needed to explore the module. – Similarly multipredicate branches are broken Software Testing and Validation, page -69- into atomicSoftware Testing and Validation, page -70- predicates. true false true false A&&B A • The set of independent paths can be false constructed by starting from the shortest s1; s2; s2; B execution path, and incrementally true introducing a new path by adding at least one edge that is not yet contained in any s1; path in the set. true false false true A||B A – Sometimes this leads to impossible execution paths. Some fine tuning is then necessary. s1; s2; B true s1; • If the tested module has several exit points, false a new artificial exit node can be introduced s2; to simplify the test path generation. Software Testing and Validation, page -71- Software Testing and Validation, page -72- 12
  13. 13. • When a particular coverage criterion has • Example: Create test cases for the following been chosen, the test cases that satisfy the module using independent path coverage. criterion must be selected. In principle this PROCEDURE average is straightforward: * This procedure computes the average of 100 or fewer numbers that lie between bounding values. It – 1. Construct the flow graph of the tested also computes the sum and the total number valid. module. INTERFACE RETURNS average, total.input, – 2. Choose the minimum number of execution total.valid paths that satisfy the criterion, and INTERFACE ACCEPTS value, minimum, maximum – 3. For each selected path, prepare a test case TYPE value [1:100] IS SCALAR ARRAY that activates the execution of the path. TYPE average, total.input, total.valid, minimum, maximum, sum IS SCALAR • These tasks can be partially automated. TYPE i IS INTEGER Software Testing and Validation, page -73- Software Testing and Validation, page -74- * Example continues * Example continues i=1 increment i by 1 total.input = total.valid = 0 ENDDO sum = 0 IF total.valid > 0 THEN DO WHILE value[1] <> -999 AND total.input < 100 average = sum / total.valid increment total.input by 1 ELSE IF value[i] >= minimum AND value[i] <= average = -999 maximum THEN ENDIF increment total.valid by 1 END-PROCEDURE average sum = sum + value[i] ENDIF Software Testing and Validation, page -75- Software Testing and Validation, page -76- B Flow-graph without atomic predicates: • Flow-graph nodes are as follows: 1 4 paths. 1: i=1, total.input = total.valid = 0, sum = 0 2: value[i] <> -999 AND total.input < 100 false 3: increment total.input by 1 2 true 4: value[i] >= minimum AND value[i] <= maximum false true 5: increment total.valid by 1, sum=sum+value[i] 7 3 6: increment i by 1 false true 7: total.valid > 0 8 9 4 5 6 8: average = -999 9: average = sum / total.valid E Software Testing and Validation, page -77- Software Testing and Validation, page -78- 13
  14. 14. • The four independent paths are: • The original independent path coverage 1) B,1,2,7,8,E gives rather poor white-box test cases. 2) B,1,2,7,9,E 3) B,1,2,3,4,6,2,...,7,8,E – The maximum number of data items is never 4) B,1,2,3,4,5,6,2,...,7,9,E tested. (This is a better alternative to path B,1,2,3,4,5,6,2,...,7,8,E which is – The test against a maximum input value is also independent of paths 1-3) never tested. • The test cases could be: 1) value[0] = -999 • To get better test cases we must break the 2) Path cannot be tested standalone. non-atomic predicates in branches to atomic 3) minimum = 1, value[0] = 0, value[1] = -999 predicates. This is done in the next slide. 4) minimum = 1, value[0] = 1, value[1] = -999 Software Testing and Validation, page -79- Software Testing and Validation, page -80- B Flow-graph with atomic predicates: • Flow-graph nodes are as follows: 1 6 paths. 1: Same as earlier 2.1: value[i] <> -999 false 2.1 2.2: total.input < 100 true 3: Same as earlier false 2.2 4.1: value[i] >= minimum true false 7 true 4.2: value[i] <= maximum 3 5-9: Same as earlier 8 9 false 4.1 true true 4.2 5 6 E false Software Testing and Validation, page -81- Software Testing and Validation, page -82- • Now the six independent paths are: • It is important to note that some 1) B,1,2.1,7,8,E independent paths cannot be tested in 2) B,1,2.1,7,9,E 3) B,1,2.1,2.2,...,7,9,E standalone fashion. 4) B,1,2.1,2.2,3,4.1,6,2,...,7,9,E – The combination of data required to traverse 5) B,1,2.1,2.2,3,4.1,4.2,6,2,...,7,9,E the path cannot be achieved in the normal flow 6) B,1,2.1,2.2,3,4.1,4.2,5,6,2,...,7,9,E of the program. • The test cases could be: • In such cases, these paths are tested as part 1) value[0] = -999 2) Cannot be tested standalone of another path test. 3) minimum=1,maximum=2,value[0]..value[99]=1 • Sometimes an independent path must be 4) minimum=1,maximum=2,value[0]=1,value[1]=0,value[2]=-999 5) minimum=1,maximum=2,value[0]=1,value[1]=3,value[2]=-999 expanded to achieve the end condition. The 6) minimum=1,maximum=2,value[0]=1, value[1]=2, value[2]=-999 path 3 earlier is a typical example of this. Software Testing and Validation, page -83- Software Testing and Validation, page -84- 14
  15. 15. • Loop testing is a white-box technique that 4.2. Loop testing focuses on the validity of looping constructs with respect to control flow of the program. • One problem of the independent path coverage testing is that it collapses loops • Four different classes of loops can be into a single iteration when possible. Thus, defined, each with a testing strategy of its many loop-related bugs are not discovered own: by the conventional coverage methods. – Simple loops, – nested loops , • Yet loops are a cornerstone of a vast majority of algorithms. They must be tested – concatenated loops, and with more than the smallest number of test – unstructured loops. cases that span a particular coverage. Software Testing and Validation, page -85- Software Testing and Validation, page -86- 7.2. Loop testing • Typically loop bugs lurk in corners and attack at boundaries. Thus interesting loop iterations are: – 0 rounds (skip the loop entirely) – 1 round – 2 rounds – m rounds (a typical value) – max-1 rounds (c) – max rounds (a) Simple (b) Nested (d) loops loops Concatenated loops Unstructured loops – max+1 rounds Software Testing and Validation, page -87- Software Testing and Validation, page -88- • If it is known that the loop has internal • Simple loops: boundaries, the boundaries are treated the – Each simple loop is analysed with the same way than maximum values: previously explained passes (0,1,2,typical,etc.) – A loop reads a physical file that is made of – If the number of iterations is fixed or restricted, blocks of 1024 bytes then we do: we may be able to test just a few of the iteration • 0,1,2,typical size,1023,1024,1025,max- patterns given earlier. 1,max,max+1 iterations. FOR i=0 to 10 DO .... – A dynamic array grows physically after 256 • Nested loops: added data items: – If we extend the simple loop strategy to nested • 0,1,2,typical size,255,256,257,max-1,max,max+1 it. loops, the number of necessary tests will grow – Sometimes the max value is hardware- geometrically with the level of nesting. dependent, sometimes it’s difficult to calculate. Software Testing and Validation, page -89- Software Testing and Validation, page -90- 15
  16. 16. – The geometrical growth soon creates too many – Since the number of test cases soon grows to test cases to be practical. A two-level or at impractical numbers, the following optimised most a three-level nesting is the maximum for strategy may be applied: this kind of complete testing. For instance, 1) Start at the innermost loop. Set all other loops to minimum values. when we have a nested loop with a known 2) Conduct simple loop tests for the innermost loop while inside boundary (like a file block size) we get holding the outer loops at their minimum iteration the following cases: parameter (e.g. loop counter) values. Add other tests for out-of-range or excluded values. • Two-level nesting needs 10*9+1 = 91 tests. 3) Work outward, conducting tests for the next loop, but • Three-level nesting needs 10*9*9+1=811 tests etc. keeping all other outer loops at minimum values and other • The extra one case comes from the zero case when nested loops to “typical” values. all loops are skipped. 4) Continue until all loops have been tested. • Combinations (0,0,any) and (0,any,any) are not possible. Software Testing and Validation, page -91- Software Testing and Validation, page -92- • Concatenated loops: • Unstructured loops: – Concatenated loops can be tested using the – Whenever possible, this class of loops should approach defined in the simple loops if each of be redesigned to reflect the use of well- the loops is independent of the other. behaving loops (simple, nested, and – If two loops are concatenated and the loop concatenated loops). counter for loop 1 is used as the initial value for – Such restructuring is always possible since loop 2, then the loops are not independent. nested loops along with condition branches – When the loops are not independent, the allow any combination of loop iterations. approach applied to nested loops is recommended. Software Testing and Validation, page -93- Software Testing and Validation, page -94- • Even if every path is executed once, the 4.3.Data-flow testing program can still have bugs. Here is an example: • The white-box testing methods presented a = b/c; earlier are based on studying the control – This works if c is nonzero, but what if c is 0? flow over the program under test. The difference between the two cases is a • Although various path testing coverage difference in data, not in paths. criteria are highly effective, they are not • A path is called error-sensitive if an error in sufficient alone. it can be detected when it is exercised. A • The path testing coverage criteria are based path is error-revealing if the error always on paths. Yet some bugs lurk in data. appears when the path is taken. Software Testing and Validation, page -95- Software Testing and Validation, page -96- 16
  17. 17. • Any path that includes the division b/c is • The value use may be in computation or in a error-sensitive since the error depends on control predicate. These cases are also the value of c. evaluated in data-flow testing. • Due to this, a still more detailed category of • Actually the value assignment may be in a methods, data-flow testing, takes control predicate as well, for instance in C- additionally into account the things that language and in other similar languages. might happen to data when the module is • A special flow graph called data flow graph executed. is generated for each analysed variable. It is • In data-flow testing, the central events then used to define variable-specific test related to variables are value assignments cases. (value writes) and value uses (value reads). Software Testing and Validation, page -97- Software Testing and Validation, page -98- r e ad ( i) • A data flow graph is similar to a traditional flow graph, except that it has added syntax y es (i < 0) | | (i > 10 0 ) for variable presentation. The presentations no vary between models, but mostly they show er r or ( ) su m =0 ; x= 0 d variable definitions and use. • An example of a data flow graph is in the c no x < i yes x =x + 1 next slide. p ri n t( s um ) y es no ( i == 1 ) cd || ( i == 1 0) d s um = 1 s u m= s u m+ x Software Testing and Validation, page -99- Software Testing and Validation, page -100- • The data-flow methods are usually complex, relatively difficult to use, and not 5. Black-box testing very good at detecting data errors. • Black-box testing focuses on the functional • The data-flow analysis is usually based on a requirements of the software. The program single variable. Thus, each variable needs is considered a black box without structure. its own analysis and test cases. This can • Program logic may be present in black-box lead to a very complex testing procedure. analysis, but implementation details are not. • While lately some progress is being made in The internal details of modules and data-flow testing, value-dependent error subsystems are hidden and cannot be studied detection is still mostly left for black-box from outside. testing. Software Testing and Validation, page -101- Software Testing and Validation, page -102- 17
  18. 18. • Black-box testing concentrates on the • Unlike white-box testing, which is interfaces of modules and subsystems. It is performed early in the testing process, used to test externally observable black-box testing tends to be applied during functionality and input-output behaviour. It later stages of testing. is often called functional testing, while • The theory of black-box testing is not as white-box testing is procedural testing. mature as the theory of white-box testing • Black-box testing is not an alternative to due to lack of mathematical background for white-box techniques. Rather, it is a functionality. complementary approach that is likely to • Yet black-box testing is the kind of testing uncover different class of errors than white- that most people consider “real testing”. box methods. • Still black-box testing need not to be magic. Software Testing and Validation, page -103- Software Testing and Validation, page -104- • The black-box testing is based on input • Even with a small input value set the values. If we could test the program with number of possible test cases would grow all possible input values and evaluate the too large. Due to this we need to prune the results of the tests then we could say that we input value set. have a complete black-box testing. • The solution for the pruning lies in • Unfortunately this would imply equivalence classes. – all possible valid inputs, – An equivalence class is a set of input values – all possible invalid inputs, that cause an equal execution path in the – all possible edited inputs, program. – all possible combinations of inputs, and – Thus, any value in the class represents the – all possible input timings. behaviour of all values in the class. Software Testing and Validation, page -105- Software Testing and Validation, page -106- • Here are a few recommendations for • Hence, when we divide the input set into a looking for equivalence classes: set of equivalence classes and pick one – Look for invalid inputs. input value from each class we get a • If all invalid inputs are processed similarly they complete black-box testing. create an equivalence class. If some cause different • Unfortunately (again) the chosen program execution (not just a different error message) they create several equivalence classes. equivalence classes depend on the analyser. – Look for ranges of numbers. Two people analysing a program will come • A range creates several equivalence classes. Let’s up with a different list of equivalence say that we accept values 1-99. Then we have classes. – a class of values 1-99, – a class of values < 1 – a class of values >99, and – a class of non-numbers. Software Testing and Validation, page -107- Software Testing and Validation, page -108- 18
  19. 19. – Look for membership in a group. – Create time-determined equivalence classes. • If an input must belong to a group, one equivalence • When timing matters to input values, we get several class includes all members of the group. Another time-related equivalence classes. Let’s say that time includes everything else. t is a critical input time. Then we get the following • Sometimes it is reasonable to subdivide the classes equivalence classes: into smaller subclasses. – input values long before time t, – input values shortly before time t, – Analyse responses to lists and menus. – input values at time t, • When the program responds differently to each – input values shortly after time t, and alternative of a list or a menu, the alternatives are – input values long after time t. equivalence classes. • Naturally this kind of partitioning is relevant only • If several alternatives create a same response (a when time plays a critical role in the system. menu and a hot key for instance) they form a single equivalence class. Software Testing and Validation, page -109- Software Testing and Validation, page -110- – Look for variable groups that must calculate to – Look for equivalent output events. certain value of range. • The equivalence class may be determined by an • Sometimes a function determines the equivalence output event. For instance, if a program drives a classes. Let’s assume that we have a program that plotter that can drive lines up to four inches long, a accepts three angles of a triangle. Thus we have a line might be within the valid range, there might be function a1+a2+a3 = 180, where a1, a2, and a3 are no line, the program might try to plot a line longer given angles. Then we have four equivalence than four inches, or it might try to plot something classes: else altogether. – The sum of the input values is 180. • Some difficulty lies in determining what inputs to – The sum of the input values is <180. feed the program to generate these different outputs. – The sum of the input values is >180. • Sometimes many classes of inputs should have the – One or more of the input values is not a valid number. same effect, and unless we are know for certain that they are treated similarly in the program they should be separated to different equivalence classes. Software Testing and Validation, page -111- Software Testing and Validation, page -112- – Look for equivalent operating environments. • Usually we are not dealing with scalar • If a program has expected or known hardware inputs. Instead, we need an input vector limitations, they create equivalence classes. For instance, if a program is stated to work with memory that defines several input parameters to the sizes of 64M - 256M, the three equivalence classes program. include a memory < 64M, a memory in 64M - 256M, and a memory > 256M. • In that case the input domain of the system • Sometimes hardware expectations cause severe in a mathematical sense is a cartesian failures. For instance, if the program has a race product: condition it is possible that a slower (or a faster) D = D1 x D2x...x Dn, where D1, D2, ... , Dn are CPU triggers it and the program fails. the input domains of the vector elements. • All hardware-specific limitations create equivalence classes (where the input value is hardware instead of data). Software Testing and Validation, page -113- Software Testing and Validation, page -114- 19
  20. 20. • Then the equivalence classes for the whole • Altogether we get 16 equivalence classes: input can be designed as combinations of – (x valid, y valid), the classes for the elementary domains. – (x < 100, y valid), • For instance, let’s assume that a program – (x > 200, y valid), accepts two integers between 100 and 200. – (x non-integer, y valid), • For each of the input integers we get the – (x valid, y < 100), following equivalence classes: – (x valid, y > 200), – valid numbers 100...200 – (x valid, y non-integer), – integers < 100 – ... – integers > 200 – (x non-integer, y non-integer). – non-integers Software Testing and Validation, page -115- Software Testing and Validation, page -116- • As can be noticed, a complete cartesian product even over relatively simple domains 5.1. Class representative selection soon leads to an explosion in the number of • In an optimal case the equivalence class equivalence classes. partition is sound and complete. No matter • Also important is to notice that almost all which representative we elect the result of the cases are of illegal combinations. the corresponding test remains the same. • One way to prune the equivalence class set is to take a few cases for closer inspection. • This sort of leads to equivalence classes of equivalence classes. Software Testing and Validation, page -117- Software Testing and Validation, page -118- • In the formal sense, “identical” behaviour is • If the chosen input values result different not a solvable problem. So, without test results then we know that the executing a program it is usually not equivalence class is too large: each member possible to know whether two different in an equivalence class should generate the input values are actually processed in the same test response. same way. • In such a case the equivalence class is • Since we cannot ensure that our partition is partitioned into two smaller cases such that perfect, it is bad testing practice to pick just the differing inputs are no longer in the a single value from each equivalence class. same class. • Instead we take two or three values from each class and run the tests with them. Software Testing and Validation, page -119- Software Testing and Validation, page -120- 20