Software Testing and Reliability Testing Graphical User ...


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Software Testing and Reliability Testing Graphical User ...

  1. 1. Software Testing and Reliability Testing Graphical User Interfaces Aditya P. Mathur Purdue University May 19-23, 2003 @ Guidant Corporation Minneapolis/St Paul, MN Last update: April 17, 2003 Graduate Assistants : Ramkumar Natarajan Baskar Sridharan
  2. 2. References <ul><ul><li>2. Generating test cases for GUI responsibilities using complete interaction sequences , Lee White and Husain Almezen, International Symposium on Software Reliability Engineering, San Jose, CA, pp. 110-121, October 2000. </li></ul></ul><ul><ul><li>3. User-based testing of GUI sequences and their interactions , Lee White,Husain Almezen, and Nasser Alzeidi </li></ul></ul><ul><ul><li>1. Regression Testing of GUI Event-Interactions , Lee White, Proc. of the International Conference on Software Maintenance, Washington DC, pp. 350-358, November 1996. </li></ul></ul>Material from 1. and 2. is used extensively in this presentation.
  3. 3. References [contd.] <ul><ul><li>6. Orthogonal Latin Squares , Alexander Bogomolny, </li></ul></ul><ul><ul><li>4 . Hierarchical GUI test case generation using automated planning . Atif Memon, Martha E. Pollack, Mary Lou Soffa, IEEE Transactions on Software Engineering, V27, N 2, February 2001, pp144-155. </li></ul></ul><ul><ul><li>5. Coverage criteria for GUI testing, Atif Memon, Mary Lou Soffa, and Martha E. Pollack , , 8th European Conference and 9th ACM SIGSOFT Foundation of Software Engineering (FSE-9), Austria, September 10-14, 2001. </li></ul></ul>
  4. 4. References [contd.] <ul><ul><li>7 . Model Checking Graphical User Interfaces Using Abstractions , Matthew B. Dwyer, Vicki Carr, Laura Hines , Proceedings of the Sixth European Software Engineering Conference (ESEC/FSE 97), September 1997. </li></ul></ul><ul><ul><li>8. The Black Art of GUI Testing, Lawrence Kepple, Dr. Dobb’s Journal, pp. 40-46, Feb. 1994. </li></ul></ul>
  5. 5. GUI Testing <ul><li>Learning objectives- </li></ul><ul><ul><li>1. GUI test tools: strengths and weaknesses </li></ul></ul><ul><ul><li>2. Modeling and test generation for GUI testing </li></ul></ul><ul><ul><li>3. Automated test oracles for GUI testing </li></ul></ul><ul><ul><li>4. Coverage Criteria for GUI testing </li></ul></ul><ul><ul><li>2. Why is GUI testing difficult? </li></ul></ul>
  6. 6. GUI Testing: Questions <ul><li>What types of GUI test tools are available? </li></ul><ul><li>Are there formal techniques for testing GUIs? If so, then what are they and how well do they function? </li></ul><ul><li>What are the advantages and drawbacks of capture-replay techniques and ho can one do better? </li></ul><ul><li>What makes GUI testing difficult? </li></ul>
  7. 7. Features of GUI Test Tools <ul><li>Record and playback of physical events </li></ul><ul><li>Screen image capture and comparison </li></ul><ul><li>Shell scripts to control and execute test runs of the GUI </li></ul>We focus on how to test interactions amongst GUI objects .
  8. 8. GUI Testing: Techniques <ul><li>Pair-wise testing of GUI interactions. [ Due to Lee White.] </li></ul><ul><li>Testing GUIs using Complete Interaction Sequences corresponding to responsibilities . [ Due to Lee White, and Husain Almezen.] </li></ul>
  9. 9. GUI Interaction Testing <ul><li>Testing for interactions amongst all GUI objects. </li></ul>Problem : How to automatically and efficiently devise tests for testing pair wise interactions amongst GUI objects? <ul><li>GUI interactions: Static, Dynamic, and a mix of the two. </li></ul>
  10. 10. Sample GUI Options Total number of possible combination of selections: 5x6x5x7=1050 5 Modes 6 options 5 options 7 options X Y Z
  11. 11. Testing Static Interactions <ul><li>In the previous example, there is a total of 1050 pair wise interactions. </li></ul>Can we reduce the number of pair wise tests and if so then how ? <ul><li>Testing is considered static in this example because only a single screen is involved. </li></ul><ul><li>There are too many higher order interactions. At present we focus only on testing second order , or pair wise, interactions. </li></ul>
  12. 12. Brute Force versus Efficient Test Generation <ul><li>Suppose that a GUI has three factors denoted by X, Y, and Z. </li></ul><ul><li>Let X, Y, and Z, have, respectively, 2, 3, and 2 possible selections. </li></ul><ul><li>Using a brute force scheme, a total of 12 tests are needed to cover all pair wise selections. </li></ul><ul><li>Sample tests: <X1, Y1, Z1>, <X2, Y1, Z2>, <X1, Y3, Z2> </li></ul><ul><li>Reduced test set consisting of only 7 tests:: </li></ul><X1, Y1, Z1>, <X1, Y2, Z2>, <X2, Y3, Z2> <X2, Y1, Z1>, <X1, Y3, Z1>, <X1, Y1, Z2> <X1, Y2, Z1>
  13. 13. Controlling the number of possible interactions <ul><li>In both static and dynamic cases, the number of possible interactions could be very large. </li></ul><ul><li>In the dynamic case a selection might bring up a new screen where additional selections are made. </li></ul><ul><li>One method to tackle the combinatorial growth was proposed by Dalal [1994]. </li></ul>
  14. 14. Problem abstraction <ul><li>Consider each GUI object from which selections are made as a factor and denote it by F’ i , for the i t h factor. </li></ul><ul><li>In our GUI example, we have four factors F’1, F’2, F’3, and F’4, corresponding to Mode, X, Y, and Z. </li></ul><ul><li>Order the factors by their cardinality such that </li></ul><ul><li>In the GUI shown in an earlier example we get: |F1|=7, |F2|=6, |F3|=5, |F4|=5. </li></ul>|F1|>=|F2|….>=|Fk|
  15. 15. Problem Statement <ul><li>Question: What is the lower bound on the number of tests? </li></ul>The problem now is to select a set of tests each set consisting of one selection from each factor .
  16. 16. Testing Dynamic Interactions <ul><li>This involves making a selection from one or more menus, buttons, or any other GUI object on one screen. </li></ul><ul><li>Such a selection brings up a new screen where additional selections are made. </li></ul><ul><li>Assumption: Regardless of the selection made, the new screen is always dependent on the current screen and not on the entire past history. </li></ul>
  17. 17. Hierarchy of Screens Prune Prune Prune S1 S1 S2 S5 S1 S3 S5
  18. 18. Pruning Paths <ul><li>Starting with the initial screen S1, generate a hierarchy where each path of screen transitions ends at a pruned screen . </li></ul><ul><li>Select all paths from S1 to pruned screens but do not include the pruned screens. </li></ul><ul><li>With each selected path, associate only those GUI events which lead to the paths being selected . </li></ul><ul><li>Map these factors and selections onto the GUI Interaction test problem. </li></ul>
  19. 19. Review: Selection of Test cases <ul><li>The problem is to select a set of test cases, each test case consisting of one selection from each factor. </li></ul><ul><li>The lower bound on the number of tests is 7x6=42. </li></ul><ul><li>One can often get close to the lower bound. </li></ul>
  20. 20. Three Methods for GUI Test Generation <ul><li>GUI Test : Enumerate each factor and duplicate the elements when necessary. Cover all possible interaction pairs. </li></ul><ul><li>Random Test : Generate the elements of each factor randomly, duplicating an element when necessary. Cover all possible interaction pairs. </li></ul><ul><li>Mutually Orthogonal Latin Squares (MOLS Test): Generate the elements of each factor by using MOLS. </li></ul>
  21. 21. Latin Squares <ul><li>A Latin square of order n is an n x n matrix of n symbols in which every symbol occurs exactly once in each row and column. </li></ul><ul><li>Introduced by Leonhard Euler in 1783. </li></ul>a b b a Latin Square of order 2 x y z z x y y z x Latin Square of order 3
  22. 22. Mutually Orthogonal Latin Squares <ul><li>A pair of latin squares A=(a ij ) and B=(b ij ) are orthogonal iff the ordered pairs (a ij ,b ij ) are distinct for all i and j. </li></ul>1 2 3 2 3 1 3 1 2 A 1 2 3 3 1 2 2 3 1 B 1 1 2 2 3 3 2 3 3 1 1 2 3 2 1 3 2 1 A and B superimposed
  23. 23. Procedure to Generate Tests Using MOLS <ul><li>2. Arrange factors in descending order of cardinality. </li></ul>|X|=3, |Y|=4, |Z|=2 1. Identify factors. Determine their cardinalities . |Y|>=|X|>=|Z|, rewritten as: |F1|>=|F2>=|F3|, where F1, F2, and F3 denote, respectively, Y, X, and Z. 3. Let k=number of factors and n=|F2| k=3, n=3
  24. 24. Procedure to Generate Tests Using MOLS [Contd.] 4. Prepare a table containing k columns and k x n rows divided into |F1| blocks. Label columns as F1, F2, ..Fk. F1 F2 F3 Block 1 Block 2 Block 3 Block 4
  25. 25. Procedure to Generate Tests Using MOLS [Contd.] 5. Fill column 1 with 1, 2, …|F1| such that all rows in block 1 contain a 1, all rows in block 2 contain a 2, and so on. F1 F2 F3 4 4 4 1 1 1 2 2 2 3 3 3 Block 1 Block 2 Block 3 Block 4
  26. 26. Procedure to Generate Tests Using MOLS [Contd.] 6. Fill each block in column 2 with 1,2,…|F2|. F1 F2 F3 4 4 4 1 1 1 2 2 2 3 3 3 Block 1 Block 2 Block 3 Block 4 3 1 2 1 2 3 1 2 3 1 2 3
  27. 27. Procedure to Generate Tests Using MOLS [Contd.] 7. Determine, if possible, (k-2) mutually orthogonal latin squares of order n . We will denote these by M 1 , M 2 , ….M (k-2) 8. Fill entries in block 1 of column F3 with entries from column 1 of M1. 1 2 3 2 3 1 3 1 2 M 1 =
  28. 28. Procedure to Generate Tests Using MOLS [Contd.] F1 F2 F3 4 4 4 1 1 1 2 2 2 3 3 3 Block 1 Block 2 Block 3 Block 4 3 1 2 1 2 3 1 2 3 1 2 3 1 2 3 2 3 1 3 1 2 M 1 = 1 2 3
  29. 29. Procedure to Generate Tests Using MOLS [Contd.] 9. Fill entries in blocks 2 and 3 of column F3 with entries from columns 2 and 3 of M1.
  30. 30. Procedure to Generate Tests Using MOLS [Contd.] F1 F2 F3 4 4 4 1 1 1 2 2 2 3 3 3 Block 1 Block 2 Block 3 Block 4 3 1 2 1 2 3 1 2 3 1 2 3 M 1 = 1 2 3 2 3 1 3 1 2 1 2 3 2 3 1 3 1 2 ?
  31. 31. Procedure to Generate Tests Using MOLS [Contd.] 10. We have now exhausted all columns of M1. How do we fill block 4 of column F3? Reuse columns of M 1 starting with column 1.
  32. 32. Procedure to Generate Tests Using MOLS [Contd.] F1 F2 F3 4 4 4 1 1 1 2 2 2 3 3 3 Block 1 Block 2 Block 3 Block 4 3 1 2 1 2 3 1 2 3 1 2 3 M 1 = 1 2 3 2 3 1 3 1 2 1 2 3 2 3 1 3 1 2 1 2 3
  33. 33. Procedure to Generate Tests Using MOLS [Contd.] Therefore we remove all instances of 3 from column F3 of our table. 11. Note that F3 corresponds to factor Z which can assume only one of two values.
  34. 34. Procedure to Generate Tests Using MOLS [Contd.] Y X Z F1 F2 F3 4 4 4 1 1 1 2 2 2 3 3 3 Block 1 Block 2 Block 3 Block 4 3 1 2 1 2 3 1 2 3 1 2 3 M 1 = 1 2 3 2 3 1 3 1 2 1 2 2 1 1 2 1 2
  35. 35. Procedure to Generate Tests Using MOLS [Contd.] Not yet! We need to (a) fill in the blank entries and (b) check if all interaction pairs are covered. Let us begin with (b). Are we done generating the tests? 12. We have a total of 3x4x2=24 interaction pairs amongst factors X, Y, and Z. It is easy to see from the table that all (X,Y) pairs are covered. Also, despite the blank entries under column F3 (this corresponds to Z), we note that all (Y,Z) and (X,Z) pairs are covered. Voila! We are done and have generated 12, the minimum, number of tests required to cover all pairs in this example..
  36. 36. Procedure to Generate Tests Using MOLS [Contd.] 13. Fill in the blank entries in column F3. F1 F2 F3 4 4 4 1 1 1 2 2 2 3 3 3 Block 1 Block 2 Block 3 Block 4 3 1 2 1 2 3 1 2 3 1 2 3 1 2 2 1 1 2 1 2 Y X Z 1 2 1 1
  37. 37. Procedure to Generate Tests Using MOLS [Contd.] 14. The table we just completed provides test requirements. It is easy to derive test specifications from this table. Sample tests for X={New, Open, Close), Y= {Select All, Cut, Paste, Undo}, Z={Symbol, Equation} From Row 1 : X= New Y = Select All Z= Symbol From Row 12 : X= Close Y = Undo Z= Symbol
  38. 38. Special Cases: k-2>n <ul><li>k-2>n: This implies that while filling in the columns for factors, you will run out of matrices. </li></ul><ul><li>For example, suppose that k=10 , and n=7 , i.e., that there are 10 factors and we will be using MOLS of order 7. </li></ul><ul><li>Now, it turns out that there are 6 MOLS of order 7. We can use matrices M 1 , M 2 ,…M 6 for factors F3, F4,…F8. What do we do for F9 and F10? </li></ul><ul><li>The solution to the problem is to randomize the generation of entries in columns F9 and F10. </li></ul><ul><li>Of course, this might mean that some pairs may remain uncovered and hence additional tests may need to be generated. </li></ul>
  39. 39. Special Cases: |Fi|<n <ul><li>|Fi|<n, i>2: This implies that factor Fi has a cardinality less than the order of the MOLS we are using. F3 in our earlier example is one such factor. </li></ul><ul><li>As illustrated earlier, this will lead to blank entries. These are known as “ don’t care ” entries and can be filled appropriately. </li></ul>
  40. 40. Special Cases: MOLS of order n do not exist <ul><li>This is true for n=2 and 6. </li></ul>Think of the advantages and disadvantages of this approach. <ul><li>In this case we are out of luck! However, one could always use MOLS of the next higher degree . This will lead to many don’t care entries that can be handled as described earlier. </li></ul>
  41. 41. Special Cases: n<|F1| <ul><li>When the order of MOLS is less than the cardinality of |F1|, we run out of matrix columns to use. </li></ul><ul><li>In this case we reuse the matrix columns. </li></ul>
  42. 42. Experiments with three algorithms <ul><li>Lee White conducted experiments to answer the above question. </li></ul><ul><li>Recall the three algorithms for generating GUI tests: GUITest, Random, and MOLS test. Which of these is the best? </li></ul><ul><li>Results follow.. </li></ul>
  43. 43. (Sample) Number of tests generated by three algorithms GUITest Random MOLS Min. Factors k 10 8 6 6 3/2/2/2 4 30 20 20 20 5/4/3/3/2 5 63 38 38 36 6/6/5/4/3/3/2 7 315 161 127 100 10/10/10/9/9/8/7/6/5/4 10 485 217 163 121 11/11/11/11/10/10/9/9/9/8/8/8/7 13
  44. 44. Conclusions from the Experiments <ul><li>GUITest and Random perform comparably for small k but become worse as k increases. </li></ul><ul><li>MOLS achieves lower bound in most cases. Not in n=6 and 10 and when the randomization is to be done. </li></ul><ul><li>In all cases, MOLS performed better than Random and GUITest, and Random performed better than GUITest. </li></ul>
  45. 45. Implications for GUI Maintenance: Addition of a Screen <ul><li>The incremental changes to a GUI must be tested. </li></ul><ul><li>Development of GUIs is often incremental; screens are often developed one or a few at a time and full functionality of all GUI objects may not be provided.. </li></ul>How does the addition of a screen, or any change in the GUI, affect the set of tests already developed? <ul><li>If the screen transitions depend only on the current screen, and not on the entire history, then the addition of a screen will mean the addition of GUI events to corresponding test sequences and not to all tests sequences. </li></ul>
  46. 46. GUI Object Modification or Screen Deletion <ul><li>In general, any screen transitions could be affected by a change and must be checked. </li></ul><ul><li>Similar effect on test sequences could occur when a GUI object is modified or a screen is deleted . </li></ul>What is the effect on a given test sequence of adding a new GUI object? <ul><li>Note that a new object corresponds to a new factor . </li></ul>
  47. 47. Addition of a GUI Object <ul><li>The remaining columns would not be affected unless the cardinality of the new factor is greater than that of F1 or F2. </li></ul><ul><li>If there exists an unused MOLS matrix then the test table can be easily modified by adding a column in the proper location corresponding to the new factor. </li></ul><ul><li>In this latter case the number of tests would increase and the entire table regenerated. </li></ul>What is the effect on a given test sequence of deleting a GUI object, ie., deleting a factor?
  48. 48. Deletion of a GUI Object <ul><li>There will be no change in the number of tests if the factor deleted is not F1 or F2. </li></ul>What is the effect on a given test sequence of adding a GUI screen that has new GUI objects? What is the effect on a given test sequence of deleting a GUI screen and thereby deleting some GUI objects??
  49. 49. Stability of Tests <ul><li>If the GUI is modified slightly then MOLS is the most stable algorithm as long as there are more MOLS matrices available. </li></ul>What is the impact on the test sequence of small changes in the GUI? <ul><li>The Random algorithm is less stable but allows the reuse of previous tests. </li></ul><ul><li>The GUITest algorithm is the least stable and requires many more tests to be added. </li></ul>Which of the three approaches would you recommend when the GUI is unpredictable both during design and maintenance?
  50. 50. Using Don’t Care Entries: Prioritized List <ul><li>Use these entries to test for additional interactions between modified GUI objects. </li></ul>How best to use the don’t care entries generated when using the MOLS approach? <ul><li>Use these entries to test for modified GUI elements and closely related GUI objects or screens; for example other unmodified GUI objects within the same screen, or within screens that immediately precede or follow a modified screen. </li></ul><ul><li>Use these entries to test for interactions between modified GUI objects and other arbitrary GUI objects in the test sequence. </li></ul>
  51. 51. Using Don’t Care Entries: More Ideas! <ul><li>Identify screens and GUI objects where failures contribute to a higher overall risk to the successful operation of the system. </li></ul><ul><li>Use these entries to test for interactions between higher and lower risk elements of the GUI. </li></ul><ul><li>Use these elements to utilize criteria such as boundary value testing. </li></ul>
  52. 52. Questions <ul><li>Are the tests generated using MOLS feasible? </li></ul><ul><li>How would you treat sub-menus? </li></ul><ul><li>How would you detect and handle infeasible tests? </li></ul><ul><li>Is pair wise testing sufficient? </li></ul>
  53. 53. GUI Testing Using Responsibilities <ul><li>A responsibility is an end-effect that the user of a GUI wants to achieve. We assume that the end-effect is observable in the surrounding environment that includes memory, peripheral device, application software, etc. </li></ul>Open a file. <ul><li>Examples: </li></ul>Copy a section of a file and paste it at another place within the same file. Select a sequence of vertically placed objects, align them at their left edges, and evenly distribute them vertically. Establish communication with a pacemaker, and download data recorded in its internal memory during the past 24 hours.
  54. 54. Complete Interaction Sequences (CIS) <ul><li>A CIS is sequence of objects and selections made by a user to produce a desired response (or to fulfill a desired responsibility). </li></ul>[Show examples from Powerpoint.] <ul><li>Examples: </li></ul><ul><li>A CIS might overlap with another, or be fully contained, or simply share GUI objects.. </li></ul>
  55. 55. Identification of Responsibilities and CISs <ul><li>Responsibilities are identified using all available sources that include design documents. </li></ul><ul><li>A CIS is defined for each responsibility. </li></ul><ul><li>A finite state machine is constructed for each CIS. </li></ul><ul><li>Several transformations are applied to each FSM to reduce its complexity and thus tackle the state explosion problem. These transformations abstract two types of components: </li></ul><ul><li>Each reduced FSM is used to generate tests for the corresponding CIS. </li></ul><ul><li>Strongly connected component </li></ul><ul><li>Structural symmetry component </li></ul>
  56. 56. Strongly Connected Components <ul><li>A subFSM is a strongly connected component if for any pair of states (s1,s2), there exists a directed path from s1 to s2. </li></ul>I1 I2 P Q R S O1
  57. 57. Design Tests <ul><li>In the example below we need two design tests that match up the two inputs with the output and include all states. </li></ul>Design tests: {<I1 Q, R, S, P, Q, R, O1>, <I2, P, Q, R, S, P, Q, R, O1>} P Q R S I1 I2 O1
  58. 58. Implementation Tests <ul><li>Implementation tests include all transitions that do not occur in the design but do occur in the implementation. </li></ul><ul><li>To obtain implementation tests determine all unused selections of each GUI object in the component under consideration. </li></ul><ul><li>If these selections correspond to transitions outside of this component to other states of the CIS, then they produce new outputs of the component. </li></ul><ul><li>Similarly, new inputs to a component are obtained by considering possible transitions to the states of a component. </li></ul>
  59. 59. Implementation Tests: Additional Transitions P Q R S I1 I2 O I3 O2 A different GUI selection, not in the original design, produces this transition. *
  60. 60. Implementation Tests <ul><li>The revised state diagram leads to implementation tests . </li></ul>Implementation tests:: {<I1, Q, R, S, Q, R, S, P, Q, R, S, P*, Q, R, O1>, <I1, Q, R, S, Q, R, S, P, Q, R, S, P*, Q, R, S, O2>, There are four more tests. Can you derive them?    }
  61. 61. Another example: Edit-Cut-Copy CIS Strongly connected component A Select Text E Complete D Copy C Cut B Edit
  62. 62. Tests for the Select Text-Edit-Cut-Copy CIS <ul><li>Two tests for this CIS are given below. </li></ul><A, B, C, B, D, B, C, E>, There are two more tests. Can you derive them? <A, B, D, B, C, B, D, E> <ul><li>Notice that there are two different ways to get to E for Cut an Copy, respectively.. </li></ul><ul><li>Further, for each of these two ways, there are different ways in which the sequence could be started, one with C and the other with D. </li></ul>
  63. 63. Another example: Account Withdrawal How many and what test(s) are required for this CIS? Strongly connected component A Start E Select other transactions C Withdraw D Print result B Select Account
  64. 64. Structural Symmetry Components: Definition <ul><li>A component has structural symmetry if it has one input to state s1, one output from state s2, and one or more directed paths from s1 to s2. </li></ul><ul><li>Here we assume that each directed path from s1 to s2 contains only one state other than s1 and s2. </li></ul><ul><li>Structural symmetry components must also satisfy the following constraints: </li></ul>
  65. 65. Structural Symmetry Components: Constraints <ul><li>The path followed to get to the input of the component has no effect on the internal paths and states of the component. </li></ul><ul><li>Any path traversed following the output of the component is independent of the path that was traversed within the component. </li></ul>
  66. 66. Structural Symmetry Components: Example Are the symmetry conditions met? I1 A Open File Dialog C Select File by Selection B Select File by Name D O1 File Selected UNDO Are they also met if UNDO transitions are added from B and C to A ? UNDO
  67. 67. Structural Symmetry Component: Design Tests <I1,A, B, D, O1> <I1,A, C, D, O1> <ul><li>For the component in the previous example, without the UNDO transitions, we need two design tests: </li></ul>I1 A Open File Dialog C Select File by Selection B Select File by Name D O1 File Selected
  68. 68. Destruction of Structural Symmetry Component <ul><li>For implementation testing we need to discover additional transitions. </li></ul><ul><li>The additional transitions will likely destroy the structural symmetry This will lead to an increase in the number of implementation tests. </li></ul><ul><li>However, the discovery and abstraction of components reduces the number of tests required to test a CIS FSM. Why? </li></ul><ul><li>This is because when testing the CIS FSM, we select only one of the several paths in each test of the FSM. Of course, we make sure that each path of the abstracted component is tested at least once. </li></ul>
  69. 69. Testing the Reduced FSM: Design Tests <ul><li>Each abstracted component in the CIS FSM is replaced by a superstate . </li></ul><ul><li>To conduct a design test , construct a sufficient test set as follows: </li></ul><ul><li>Each test case must correspond to a sequence of GUI actions that begin at the start node and ends at the terminating node. </li></ul><ul><li>The test set must cover all paths in the reduced FSM. each time a superstate is entered, an appropriate path of the corresponding component is selected. </li></ul><ul><li>All design tests for each abstracted component must be exercised in the complete test of the FSM. </li></ul>
  70. 70. Testing the Reduced FSM: Implementation Tests <ul><li>The reduced FSM for the implementation tests is likely to be larger in the its number of states than the one used in design test.. </li></ul><ul><li>Some abstracted components might be invalidated and therefore replaced by their “complete” versions. </li></ul><ul><li>A sufficient set of implementation test cases must: </li></ul>(b) include all the implementation tests for each abstracted component at least once. (a) cause all paths in the modified FSM to be executed and
  71. 71. Other Considerations <ul><li>The FSM does not model the effects of the GUI and therefore the result of traversing each path must be examined carefully. </li></ul><ul><li>A cycle may need to be traversed different number of times at different points along a path. </li></ul><ul><li>All distinct paths must be generated from the start to the terminating state. However, we can exclude any path where a cycle is traversed more than once. </li></ul><ul><li>Note that this method is more exhaustive than simply covering each state and each transition once. </li></ul>Why?
  72. 72. Example <ul><li>This example shows how a CIS FSM can be reduced and tests derived from the reduced FSM. </li></ul><ul><li>The CIS considered is an Edit-Cut-paste-Copy-Paste sequence that involves opening two files. </li></ul><ul><li>We begin by developing the FSM for the given CIS. </li></ul><ul><li>Next, we identify the strongly connected and structural symmetry components. </li></ul><ul><li>All such components are replaced by a superstate . </li></ul><ul><li>We can now develop tests for the reduced FSM. </li></ul>
  73. 73. FSM for Edit-Cut-Paste-Copy-Paste CIS Total tests required: 50 C2 C1 C3 Initial Paste Finish Move Cursor File Name HLT Select Open Cut Ready/ Edit Copy Move Cursor Open2 Select Name HLT File Edit
  74. 74. Reduced FSM 2 tests 4 tests 2 tests Total tests required: 8 Initial Move Cursor File Paste Finish Move Cursor File C1 Ready/ Edit C2 C3
  75. 75. Evaluation of the CIS-based Technique <ul><li>The GUIs of two applications , A and B, were subjected to CIS based testing technique. </li></ul><ul><li>The primary objective s of the experiment were to determine (a) how good is the technique in detecting defects in GUI implementation and (b) the reduction in the number tests achieved through the reduction of the FSM. </li></ul><ul><li>Defect : Serious departure from the specified behavior. </li></ul><ul><li>Surprise : User recognized departure from expected behavior. </li></ul>
  76. 76. Experiment Details <ul><li>Application A: MS Windows 98. </li></ul><ul><li>Application A 1 : Arabic enabled </li></ul><ul><li>Application A 2 : Arabic version </li></ul><ul><li>Application A 3 : English version </li></ul><ul><li>Application B: GVISUAL, a component of an object-oriented multimedia data base system. </li></ul><ul><li>Application A: Sample CIS: Install an application, e.g. MS Word, from a CD. </li></ul>
  77. 77. Experiment Details [contd.] Results follow… <ul><li>Sample responsibilities: </li></ul><ul><li>Application A: Install an application, e.g MS Word, from a CD. </li></ul><ul><li>Application B: A condition box that forces certain constraints on the relations between objects. </li></ul><ul><li>Sample defect seeded: Application B: </li></ul><ul><li>Creating a method without identifying its base class. </li></ul>Total of six defects seeded.
  78. 78. Results of the Evaluation of the CIS-based Technique *Defects/surprises found by implementation tests include those found by Design tests . # of Tests *Defects/Surprises Found 28 32 28 32 28 32 58 112 6/3 12/4 4/2 4/2 4/1 6/1 1/2 2/2 Application A 1 Design Implementation #CIS: 13 Application A 2 Design Implementation #CIS: 13 Application A 3 Design Implementation #CIS: 13 Application B Design Implementation #CIS: 52 5/0 6/0 Seeded defects found
  79. 79. Reduced versus all directed paths: # of Design Tests Reduced FSM All directed paths 28 58 40 86 Application A Application B *Defects/surprises found by implementation tests include those found by Design tests .
  80. 80. Conclusions <ul><li>Though the design tests are useful, implementations tests are highly recommended. </li></ul><ul><li>The savings in # of tests due to reduction is not impressive. However, this is due to the low complexity of the CIS used. More complex CIS will likely to to more savings. </li></ul><ul><li>Consistency in the ratio [Tests/CIS] suggests that the number of tests could be estimated from the number of CIS. </li></ul>