Automated Testing of Graphical User Interfaces


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Automated Testing of Graphical User Interfaces

  1. 1. Proceedings of National Conference on Challenges & Opportunities in Information Technology (COIT-2007) RIMT-IET, Mandi Gobindgarh. March 23, 2007. Automated Testing of Graphical User Interfaces Er. Monika Passi Department Of MCA, PunjabTechnicalUniversity Lala Lajpat Rai Institute of Engineering and Technology Moga, E-mail: object. All CR-Tools are object-oriented, i.e. they recognize Abstract each selected GUI element which is selected (such as button, radio-box, toolbar, etc.) and capture all object characteristics The present paper discusses strengths and weaknesses of (name, color, label, value), and not just the X/Y-coordinates of mouse-click. commercially available Capture-and-Replay GUI testing tools (CR-Tools) and presents a pragmatic and economic approach B. Programming: The captured test steps are stored by the CR- for testing Graphical User Interfaces using such tools. Today's software systems usually featured with Graphical User tool in a test script equivalent to C or Basic. With all the Interfaces (GUI's). Because of the varied possibilities for user functions of a programming language in the test script (case differentiation, loops, subroutines), even complex test processes interaction and the number of control elements (buttons, pull- down menus, toolbars, etc.) available with GUI's, their testing can be implemented. is extremely time-consuming and costly. Manual testing of C. Checkpoints: In order to determine if the program being GUI's is labor-intensive, frequently monotonous, and not well liked by software engineers or software testers. A promising tested (SuT: Software under Test) is functioning correctly or if remedy is offered by automation[3], and several tools[3] for there have been any errors, the tester (during the test capture or in the script editing) can insert additional checkpoints in the test computer-based testing of GUI's are already commercially available. script. By this means the layout-relevant characteristics of windows-objects (color, position, size, etc.) can be verified along with functional characteristics of the SuT (a mask value, contents of a message box, etc.). 1. INTRODUCTION D. Replay mode: Once captured, tests can be replayed and thus Testing is the process of analyzing a software item to detect the in principal are repeatable at any time. The aforementioned difference between existing and required condition (i.e. bugs) object-oriented test capture permits GUI elements to be re- and to evaluate the features of the software items. In other recognized when the test is repeated, even if the GUI has words, it is the process of analyzing a program with the intent meanwhile been modified by a change in the software. If the of finding errors [5].All software needs to be tested. Software test object behaves differently during a repeat test or if testing is major part of software development process. The checkpoints are violated, then the test fails. The CR-Tool main principles for testing are: records this as an error in the test object. Tool vendor promote CR-Tools as a very fast and easy method of automating GUI • Tests should be planned long before testing begins. testing. But, in reality there are lots of traps or pitfalls, which • To be more effective, an independent third party can impair the effectiveness of CR-Tool, based testing. should conduct testing • Exhaustive testing is not possible. 3. TESTING OF THE GUI In a GUI-based software system, all functions are represented in 2. CAPTURE-AND-REPLAY TOOLS the GUI by visible "interaction points" (IP), and the activation of intended functions is achieved by directly pointing to their All CR-Tools currently commercially available are similar in visible representations. As a GUI tester you first must be able to function and operation: recognize all these IP's (what is no problem for the human tester). Then you have to perform a set of test cases to test the functionality of the software underlying the GUI (does the A. Capture mode: During a test session, the Capture-and-Replay system really store the entered data on the database server?). Tool (CR-Tool) captures all manual user-interactions on the test These tests are specific for each product being tested, and this 46
  2. 2. Proceedings of National Conference on Challenges & Opportunities in Information Technology (COIT-2007) RIMT-IET, Mandi Gobindgarh. March 23, 2007. functional testing is an essential component of GUI testing. You also always have to verify that the visible representations of Can you be sure that the data reread was not already stored on functions are consistent and make sense to the end-user (why is the server by your tester colleague? (You have to be sure of this the save-button not labeled "save"?). These "style guide" tests fact. Otherwise your test case makes no sense at all). If you are are usually specific for a product line, for all the software sure, because - during testing -- the system hasn't popped up an products of a specific company, or for software running under a overwrite data yes/no message, then you have a new problem: specific windows-system. Of course, In practice there is no Your test case is not reusable and must be rewritten into strict borderline between these two types of tests (when I something like: enter data, store it, if ´overwrite data yes/no´- change that data field's value and close the mask - why doesn't message appears press ´yes´, reload it, check if values of the system automatically store the altered data?). Now, where input/output are the same. are the traps? Test cases depend on the SuT's history and current state; this problem is well known in test automation. However, in GUI test 4. TRAPS WHEN ACCESSING GUI-OBJECTS automation this problem will occur with nearly every test case: As mentioned above, CR-Tools are supposed to recognize IP's • Buttons are enabled/disabled depending on field- or GUI objects with object-oriented methodology and to also re- values, recognize them if the GUI layout has been changed. However, you must make sure this is also true for your tool. If your test • Data fields change color to gray and become read-only, tool is able to recognize GUI objects created by C++ development systems, it might not recognize objects within • Toolbars and menu entries change during operation, Oracle-applications or, for example, OCX-controls. In this case you would have to look for an additional interface kit offered by • The bitmap you stored as a reference bitmap contains your test tool vendor. system clock output, • Sometimes during testing a word-processor will be Since CR-Tools recognize GUI elements by their ID, running in parallel eating up windows resources, and individualizing the specific element within its context, or via a sometimes not, combination of attribute values that uniquely identify the element, they are sensitive to changes of these values. However, • An email message popping up captures the focus and software development tools (e.g. Microsoft's Visual C++) stops your test-run, sometimes change ID's of GUI-objects without even notifying the developers (the idea is to release the developer from • The application iconifies, responsibility for assigning individual ID's to GUI elements). In this case, whenever a product is re-tested during a new release, • And other surprises. the CR-Tool must be manually re-taught about all changed GUI objects. A similar problem occurs if GUI objects are eliminated or redesigned due to functional changes in the new product 6. TRAPS OF STYLE GUIDE TESTING release. In order to give software a consistent Look & Feel, its designers try to establish appropriate layout-rules in a so-called Style 5. TRAPS WHEN TESTING FUNCTIONALITY Guide. The list below gives some examples: If the CR-Tool captures all your mouse-clicks, it’s likely that no • ´OK´ and ´Cancel´ are always located at the bottom or real test is recorded, such as: press the ´save´-button and wait on the right, but never on top or on the left. until ´saved´ occurs in a message box. If you really want to test if the data has been saved on the server, you have to look for the • All fields must be accessed via tabs in the same order. specific data on the server itself (a checkpoint normally not If fields are grouped together, then tabs must access performed by a GUI-testing tool), or you must reload the data the fields within the group. record to check if it's the same as the one stored before. Capturing the sequence enter data, store it, reload it, check if • All fields must be perfectly aligned. values of input/output are the same is therefore a better approach, but there are even more potential traps: 47
  3. 3. Proceedings of National Conference on Challenges & Opportunities in Information Technology (COIT-2007) RIMT-IET, Mandi Gobindgarh. March 23, 2007. • Texts must be legible and displayed with sufficient contrast. FIG. 1: THE TESTING PROCESS • A context-sensitive help function must be accessible for each button. These on-line help instructions must 8. GUI TEST SPECIFICATION be useful and understandable. GUI tests can be divided into product-specific functional tests The Style Guide must not only be followed for one mask, but and universally valid style guide tests. The functional tests can also for the complete application. Automated testing would thus be subdivided into testing categories such as performance tests, certainly lead to a quality improvement for tested products. Test which must be implemented for each tested product differently automation without formalization of the Style Guide is doomed but can have a similar definition in the test specification. If this to failure from the beginning. is taken into consideration when organizing the test specification, a reusable specification template can be obtained. Figure 2 shows a good basis for a GUI test specification: 7. MAKING GUI TEST-AUTOMATION WORK TABLE 1. TEST SPECIFICATION The capture mode of CR-Tools can only help to achieve an initial prototype implementation of a test case. Therefore, GUI 1. References test automation is a programming task. On the other hand, it is a 1.1 Reference documents software specification task, too. The test cases must be defined 1.2 Reference configurations before capturing, and their specification must be much more 1.3 Reference data objects detailed than for manual testing. Any organization that is 2. Functional Testing planning to automate testing must already have an established 2.1 Functional tests object 1 testing process in place; otherwise test development is doomed to failure. The diagram below shows a Testing Model of the 2.2 Functional tests object 2 imbus company, which specifies a structured process from test 3. Style Guide Testing planning to test specification, test implementation and finally 3.1 Window-System rules, (automated) testing. 3.2 Application specific rules 4. Other Testing Issues 4.1 Concurrency 4.2 Volume & Performance 4.3 Security Aspects 4.4 System administration 4.5 Installation A template like this lists the test cases to be performed in each GUI test. But what is inside a test case? Each test case definition should answer the following questions and therefore consist of the following parts: TABLE 2. TEST CASE TEMPLATE Test Configuration: Which s/w object is tested and which test equipment is needed? Test Assertion: Which feature is tested by the test case? Test Pre-Condition: What is the state of the SuT before this test can be performed? 48
  4. 4. Proceedings of National Conference on Challenges & Opportunities in Information Technology (COIT-2007) RIMT-IET, Mandi Gobindgarh. March 23, 2007. Test Sequence: How do we test this? This library contains prototypes for test scripts (to ensure script implementation according to the rules described above), Test Verification: What’s the checkpoint? How do we verify if extensions of the test script language, and, in particular, ready- the test failed or passed? to-run implementations of Style Guide test cases from our GUI test specification running under Windows 95 and Windows NT. Test Post-Condition: What is the state of the SuT after test is With the increasing growth of this library, we expect once to performed. cover the full set of Style Guide rules. Accordingly, running the library’s test scripts can check Style Guide conformance of a software product, and in the future a style guide written in plain text will no longer be needed. But additional work is still 9. GUI TEST IMPLEMENTATION necessary here; especially tests checking the ergonomic aspects of software’s GUI have to be programmed. The following are After you have defined a method of specifying tests using examples of Style Guide checks already implemented: templates then you are one step closer to automated GUI testing. TABLE 4. EXAMPLES FOR GUI TEST LIBRARY FUNCTIONS But you have to make sure about it’s proper documentation and adequate modularization of the scripts. Header: Describe purpose & usage of script. As a minimum requirement, each test script should consist of Test Initializations: Set up all the test preconditions. the following six sections: TABLE 3. TEST SCRIPT STRUCTURES Test Sequence: Perform Test. Test Verification: Check if test failed or passed. Test Log: Write test results to log file. Clean-Up: Guarantees that the SuT is left in a defined state, even if test has failed. StdCheckHelp (): Checks within a window if the help function can be activated by pressing the related button and the, F1 key, Checks if the contents of the Help window are related to the current dialog (via comparison of keywords), checks if, after closing the Help-window, the same window re-appear as before activating, Help. StdCheckShortcuts (): Validates the uniqueness of all In the long-term such a library will only be used if it is the shortcuts within a window. constantly maintained and updated: new or modified tests in S/W-projects must be periodically checked for reusability. All If all tests are programmed in this manner, then test modules reusable tests must be perfected and documented; company- can be obtained which are relatively easy to combine into test wide access must be guaranteed to any new versions of the test- sequences. This not only supports the reusability of tests when case library. Therefore, in parallel to the implementation of the testing subsequent product releases, but also the reusability of test case library, an appropriate process for library maintenance the test programs by other testers, in different test conditions, should be established. The following figure illustrates the steps and/or for testing other products. needed: 10. BUILDING A TEST CASE LIBRARY The implementation of test scripts according to the rules described above necessitates relatively high expenditures for test-programming as well as corresponding know-how in CR- Tool programming. In order to "conserve" this knowledge and make it available to less qualified test-tool users, we started to develop a Test Case Library [3], which completes the GUI Test Specification Template at the implementation level. 49
  5. 5. Proceedings of National Conference on Challenges & Opportunities in Information Technology (COIT-2007) RIMT-IET, Mandi Gobindgarh. March 23, 2007. V m: = Expenditure for test specification. V a: = Expenditure for test specification + implementation. D m: = Expenditure for single, manual test execution. D a: = Expenditure for test interpretation after automated testing. Time for test process not counted since executed without FIG. 2: TEST LIBRARY MAINTENANCE STEPS supervision via CR-Tool. V and D are given in hours of work. E n: = A a /A m = (V a + n*D a)/ (V m + n*D m). 11. MEASUREMENTS OF EXPENDITURES The question of main interest is: How often does a specific test have to be repeated before automated testing becomes cheaper To determine how much more economical automated GUI- than manual testing? In the above table this "break-even" point testing really is compared to manual testing; we have measured is represented by the N-factor, in accordance with the equation and compared the expenditures for both methods during our PIE. E N = A a /A m = 100%. The measurements undertaken within our experiments show that a break-even can already be attained The baseline project we chose for measuring this data was the by the 2nd regression test cycle (N total =2, 03). This break- development of an integrated PC software tool for radio base even, however, does have two prerequisites: the tests must run stations (for GSM radio communication)[4]. This application completely without human interaction (e.g. overnight test runs), provides a graphical interface for commissioning, and no further test script modifications are necessary to rerun parameterization, hardware diagnostics, firmware downloading, the tests in later releases of the product. As already mentioned equipment data base creation, and in-field and off-line in this lecture, this is not easy to achieve. diagnostics of multiple types of base stations, including a full- graphics editor for equipment definition. The application was If all you do is buy a CR-Tool and begin capturing, then your developed using Microsoft Visual C++/Visual Studio and is testing costs will increase to between 125% and 150% of approximately 100000 lines of C++ Code in size. manual testing costs (see E 1 in fig. 7). And there will be additional costs associated with each test run because of traps The table below shows the measurement results: such as test script maintenance. On the other hand, if you establish a complete framework for GUI test automation (where TABLE 5. MEASUREMENT RESULTS the CR-Tool is a cornerstone and not the complete solution), 50
  6. 6. Proceedings of National Conference on Challenges & Opportunities in Information Technology (COIT-2007) RIMT-IET, Mandi Gobindgarh. March 23, 2007. then a decrease of costing down to about 40% for a typical product test cycle (E 10 ) is realistic. 12. CONCLUSION Putting GUI test automation to work is a software development effort, and the complexity of this effort calls for the professional: experienced software testers with a solid software development background working within the framework of a well established testing process. High investments are necessary, not only for tools, but also for test implementation and test maintenance. On the other hand, after making these start-up investments, automated testing of Graphical User Interfaces can decrease your testing costs down to 40% compared to manual testing. REFERENCES [1] [2] [3] [4] Matthias Rauterberg: ´Quantitative Test Metrics to Measure the Quality of User Interfaces´, in: Proceedings of 4th European Conference SOFTWARE TESTING ANALYSIS & REVIEW--EuroSTAR'96, 2-6 December 1996, Amsterdam. [5] Whitten and Bentley, System analysis and design methods, 7th Edition (Tata McGraw Hills). 51