Automating GUI Testing for Eclipse RCP Applications


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Automating GUI Testing for Eclipse RCP Applications

  1. 1. This article will be appearing in the April 2007 issue of Software Test & Performance. Automating GUI Testing for Eclipse RCP Applications by Dan Rubel, Chief Technology Officer Phil Quitslund, Senior Architect Instantiations Graphical user interfaces (GUIs) today are among ware code to see if it the most complex and interactive software tools delivers the desired Automated GUI Testing Approaches designed and customers expect the same high qual- result. Automated test- Automated GUI Testing ity in them as they get from simpler software tools. ing can be split into Issues When users press a key, click the mouse or select two broad categories: • Finding widgets menus, they expect the same result consistently. API testing and GUI Companies concerned about time-to-market are testing. Both involve • Verifying state (assertions) moving to build their software products in as little driving the applica- as one-sixth the time previously required. Tossing tion and verifying • Staying in-sync a GUI over the wall to a testing or quality assur- results, but Automated • Handling the ance department is no longer an option. To build API testing calls unpredictable software faster and still meet customer expecta- methods and validates • Platform differences tions for high quality, companies must discover return values, while • Maintaining tests any bugs before their customers do. This is both Automated GUI more cost effective and fosters greater customer testing interacts with the application like a user satisfaction. To create high quality user interfaces, through the user interface. Of the two approaches, testing means checking the UI throughout the API testing is more common while GUI testing is development process and it includes exhaustively more difficult and as a result less common. Both exercising a user interface that is today universally approaches are necessary for complete testing of graphic in nature. an application. Manual Testing Building in quality means validating the UI early in the process and then frequently throughout Manual GUI testing is time consuming, labor the development cycle to assure new fixes don’t intensive and expensive. Based on written direc- introduce new bugs. Development and quality tions explaining the steps and expected results, it assurance testing must be more integrated today requires a labor force, either the QA department or and without placing an undue burden on the devel- an outsourced team. It requires writing a “script” opment teams. Because it is “machine powered” that describes user actions and the expected rather than “human powered,” Automated GUI outcomes. This process assumes individuals will testing can be performed more frequently and follow their directions explicitly, uncover any earlier in the development cycle than Manual GUI problems, record those problems and return the testing, but not all tests can be automated. As a results to the development team. Since it’s “human result, the best practice is to automate those tests powered,” the process is tedious, error prone and that can be automated, execute the automated costly. Because of the cost, this process is usually tests during an automated build process, and have done just before a software release and not con- manual tests for UI interactions that are too dif- tinuously throughout the development cycle. ficult or impossible to automate. Either method Automated Testing results in better quality software, but by combining the two approaches, you can achieve the best qual- Automated testing uses a “test” program to drive ity at the least cost. a set of inputs to exercise some other part of soft-
  2. 2. Automated GUI Testing Approaches be used to replay the recorded actions. Depending Writing a comprehensive test procedure is tedious upon the tool used, these scripts can be parameter- and the process of automating and maintaining ized later with different inputs for the same test that procedure is difficult. Fortunately there are so that a single test can cover multiple situations. tools emerging to help with this, and even though These recorded actions can be stored in many dif- they are not perfect, they reduce the time and cost ferent forms such as an XML script or a Java test considerably. (See table 1 for a list of tools.) The case. Click-and-record products provide an easy strengths and weaknesses of each tools depend way to get started where tests can be recorded by upon the design choices made when designing non-programmers. the tools in the first place. Some tools take a pure Other products focus on providing a UI testing recording approach, hiding test source code behind library upon which UI tests can be hand coded. a UI; other tools are source code libraries used in These libraries and tests are typically written in hand-coded automated UI tests; still others take a general purpose programming language such a hybrid approach generating source code as a as Java. The initial hand coded tests are written starting point but facilitating hand-modification of to directly call the API provided by the testing recorded test. library. Over time, as the number of tests grow, Click-and-Record UI testing tools generate test common elements of the tests being written are scripts by watching a user interact with an appli- factored out into utility classes and methods cation-under-test. Typically, the user launches the making the creation of subsequent tests that much tool and the application-under-test and then starts easier. Although UI testing libraries take more the UI test recorder. As the user interacts with the time up front to understand and code UI tests, they application-under-test, the UI test record is gather- provide a much more flexible and robust solution ing all of the UI events and capturing state infor- in the long run. mation about the visible UI components such as Still other products provide a hybrid of these whether a button is enabled, a checkbox is selected two approaches. Hybrid products have recorder or the text in a particular field. Either during the functionality to observe a user interacting with the recording process or after the user has stopped the application-under-test and generate test scripts. UI test recorder, the test tool emits a script that can Rather than storing the generated test script in a Table 1: Some of the Tools for GUI Testing Tool Availability Type Abbot open source library (Java) Costello open source recorder (XML) GUI Dancer commercial recorder JDemo open source library (Java) Mercury WinRunner commercial recorder QF-Test commercial recorder (XML/Jython) Rational Functional Tester commercial recorder TPTP AGR* open source recorder (XML/Java) WindowTester commercial hybrid (Java) * TPTP AGR = Eclipse Test and Performance Tools Platform Automated GUI Recorder Automating GUI Testing –– April 2007
  3. 3. descriptive format such as XML, hybrid products Regardless of which tool developers chose, they generate test suites in a general purpose program- need to understand that they face several issues ming language such as Java and on top of a rich UI regardless of the tools they choose to create and testing library. The goal of a hybrid product is to exercise their GUI tests, including finding widgets, provide the ease of click-and-record functionality verifying states (assertions), staying in-sync, han- to generate the initial tests combined with the flex- dling the unpredictable, platform differences and ibility and robustness of a UI testing library for the maintaining tests. long haul. Finding widgets Automated GUI Testing Issues For each UI operation such as clicking a button Admittedly, automated UI testing is not perfect. or selecting a menu item, you must locate the UI Different tools and different approaches to provid- component to be manipulated. Problems arise ing GUI tests for your application mean many around uniquely identifying widgets, creating choices. Some questions to be answered when searching ambiguities. For example, there can be choosing an automated GUI testing tool or library identical items in menu or tree, or multiple labels include: or buttons with the same text. One best practice • How quickly can developers and non-develop- involves inserting code to associate a unique inter- ers start producing tests? nal identifier with each widget so that it can be uniquely identified when testing. Another solution • Can recorded tests be easily modified to meet involves locating a widget based upon its container changing requirements? hierarchy. Locating a widget based upon its screen • How easily can tests be parameterized to cover position is fragile and should only be used as a last a wider range of scenarios? resort. • Can individually recorded tests be bundled into Different tools solve this problem in different larger test suites? ways. TPTP AGR (see table 1) stores test scripts • Can test execution be integrated into your in XML and identifies the widget using a weighted automated build environment? widget identifier. The widget identifier is stored as a command element in the test script and might One of the most difficult issues about automated look something like this: software testing is recording because it takes command considerable effort to read the code generated and descriptive=Transfer Shell Menu Action understand what it is doing. Today no system over- type=select comes these recording issues, but a hybrid system contextId=menus widgetId=org.eclipse.ui.internal. approaches the ideal and supports today’s best ActionSetContributionItem practices for GUI testing. Using a hybrid approach #{{Transfer-amp;Transfer}}-{{0.8}}{{5|0}}- is similar to using either Abbot or TPTP AGR (see {{0.6}}{{true}}-{{0.1}}{{1}}-{{0.1}}{{first}}- {{0.2}}{{last}}-{{0.2}} / Table 1) but provides more flexibility because it taps into Java code. Similar to click-and-record, In contrast, Abbot provides a programmatic solu- the developer launches recording sessions, records tion to this problem by providing matchers used to actions, makes assertions and generates the test, identify the desired widget from within a container but then has the full flexibility of a general purpose hierarchy. Locating the First widget of type “Tree” programming language under the covers to modify in a container hierarch might look something like the script to handle unforeseen situations and adapt this: to the changing UI requirements. tree = finder.find(new IndexMatcher(new ClassMatcher(Tree.class), 1)); Automating GUI Testing –– April 2007
  4. 4. Verifying state (assertions) that detects when the transaction is complete and Before and after a UI operation, it is useful to continue executing the test. verify that various aspects of the GUI are exactly Some click-and-record tools such as TPTP AGR as you’d expect them. Implicit verification occurs hide all of the UI test playback complexity behind when a widget is located to be manipulated; if the a playback UI. The price of this simplicity is the widget cannot be located, then the test fails. Not all inability of the framework to deal with unpredict- UI state is interesting, but UI state that changes in able timing. Hybrid solutions take a different one widget as a result of a UI operation in another approach; playback is handled through a rich should be explicitly verified as part of the test. API that hides the complexity but provides more For example, verifying that the content of a tree flexibility to deal with unpredictable timing. Pure has been updated as a result of clicking a button library API solutions such as Abbot provide the is interesting while verifying that a checkbox has necessary flexibility in this regard, but that flex- changed state as a result of clicking it is typically ibility comes at a price; playback issues such not interesting. as threading must be tackled by the user. For Tools that provide a UI testing API in a general example, in Abbot, waiting for a progress dialog purpose program language such as Java provide to close indicating that an operation might look maximum flexibility in this regard. Any widget can something like this. Robot.wait(new Condition() { be accessed and any application state can be veri- boolean inProgress = true; fied. For example, in Abbot, verifying that a tree public boolean test() { has a particular selection might look something Display.getDefault().syncExec(new Runnable() { like this: public void run() { Shell shell = Display.getDefault(). selection = treeTester.getSelection(tree); getActiveShell(); assertEquals(1, selection.length); if (shell == null || !shell.getText(). assertEquals(report.pdf, equals(Progress Information)) treeTester.getText(selection[0])); inProgress = false; } Click-and-record tools have different levels of }); possible verification depending upon the design of return !inProgress; the tool. Some provide a fixed set of verification } }); operations while others provide the ability to call a class or method in a general purpose programming Handling the unpredictable language to perform verification. The outcome of UI operations are hard to predict Staying in-sync and not all unpredictable outcomes are error condi- Making the playback robust and keeping the test tions. For example, a dialog may appear the first synchronized with the UI can be complicated. time a UI operation is performed but not subse- Typically the test executes in a separate thread quent times in a single session. This situation can from the UI; a good test framework hides this be coded into the first test that is executed, but that issue from the developer that they can focus on approach is fragile; better to setup a condition han- what should be tested and now on how it is accom- dler that watches for this dialog to appear, during plished. For example a UI operation that involves whichever test, and respond accordingly. a transaction over the network may take an unpre- Test libraries and hybrid solutions provide the dictable amount of time. One way to account for greatest flexibility when dealing with the unpre- the unpredictable duration is to delay the test some dictable; these tools provide a UI testing API in a fixed amount of time and hope that the transaction general purpose programming language such as has completed before the test completes. A better Java to programmatically react to specific unpre- and less fragile way is to set up a wait condition dictable situations. For example, given Abbot and Automating GUI Testing –– April 2007
  5. 5. the unpredictable dialog situation described above, with click-and-record tools so long as they provide you could fork a thread (see Figure 1) to wait for a call-out hook to a general purpose programming the appearance of a dialog with a particular title, language. and when that dialog appears, click the button to dismiss the dialog and resume the test. Platform differences final Thread handleWarningThread = Different operating platforms have different accel- new Thread(Handle Warning Thread) { erator key labels (Win32 is Alt+Shift+N while final Object lock = new Object(); public void run() { Linux is Shift+Alt+N), default selection events, while (true) { focus differences and tree item selection. Ideally, Display.getDefault().asyncExec tests should not include platform specific refer- (new Runnable() { public void run() { ences so that the same tests can exercise and vali- lock.notifyAll(); date the same application. In others, like Eclipse Shell shell = Display.getDefault(). the UI changes with different versions. getActiveShell(); if (shell != null How platform differences are handled depends shell.getText().equals(Access Error)) { very much on the tool being used. Some click- Button button = (Button) finder.find( new ClassMatcher(Button.class)); and-record tools such as TPTP AGR (see table 1) new ButtonTester().actionClick(button); provide data parameterization which may be used } to smooth out some differences between platforms. } }); Others allow for regular expressions to be used lock.wait(); as parameters in the test script. And of course, Thread.sleep(500); given a tool that provides a UI testing API in a } } general purpose program language, you have the }; full power of that language at your disposal to deal handleWarningThread.setPriority(Thread.MAX_PRIORITY); with the situation. handleWarningThread.setDaemon(true); handleWarningThread.start(); Maintaining tests Given a hybrid tool with a rich API and this same As the GUI changes during the lifetime of the situation, the solution might be a single call to product, adapting the test can be tricky. Something notify the runtime engine and handle the unpre- as simple and changing a label from “disconnect” dictable dialog. This same technique could be used Figure 1. Running an Abbot test uses three threads. The SWT thread accesses widgets on that thread only. The test thread executes the test and asserts results. The condition thread handles unpredictable situations. Automating GUI Testing –– April 2007
  6. 6. to “terminate” means a change in the test details; purpose programming language such as Java, developers must make changes to GUI tests as the test maintenance is very much dependent the test interface evolves. If the test script is readable, it library API and how the tests were created and re- makes test maintenance much easier. factored over time. The readability of the test script depends very Conclusion much upon the chosen tool and the effort expended during test creation to structure the tests. Given An automated GUI testing tool must be easy a click-and-record tool, some changes in data to use to encourage adoption while at the same inputs are very easy to make using that tool’s user time provide the flexibility to react to unforeseen interface; other changes require modifications to situations and the maintainability to adapt to UI the click-and-record tool’s test script which might modifications over time. With the time between an be much more difficult. At worst, the test can be application concept and deployment is shrinking, re-recorded in the face of a dramatic UI change. it is necessary to automate as much of the testing For tools that provide a UI testing API in a general process as possible and automated GUI testing is an important piece of that process. Dan Rubel is Chief Technology Officer and Phil Quitslund is a senior architect at Instantiations, Inc. in Oregon. Rubel is also co-author of Eclipse: Building Commercial-Quality Plug-Ins, published by Addison-Wesley. Automating GUI Testing –– April 2007