Test Driven Development  &  Testing the Graphical User Interface Prepared by Yibo Sun Soha Makady
Agenda <ul><li>What is GUI testing? </li></ul><ul><li>Generation of test cases </li></ul><ul><li>Running test cases </li><...
Definition <ul><li>Th e process of testing a software product with graphical user interface </li></ul><ul><li>Like testing...
Importance <ul><li>GUI accounts for as much as 60% of the total application code </li></ul><ul><li>Use in safety critical ...
Benefits <ul><li>Safety </li></ul><ul><li>Robustness </li></ul><ul><li>Usability </li></ul>
Difficulty <ul><li>Scope:  huge number of GUI operations, states which need to be tested </li></ul><ul><li>Sequence:  GUI ...
Coverage Criteria <ul><li>Conventional software testing uses coverage criteria which define a set of rules telling the typ...
Regression  A.F. Memon and M.L. Soffa, “Regression testing of GUIs”,  Software Engineering Notes, Sep. 2003 <ul><li>Test c...
Generation of Test Cases <ul><li>Manual </li></ul><ul><li>Automated </li></ul><ul><li>Other: Monkey Testing </li></ul>
Automated Test Case Generation <ul><li>Approaches for automated test case generation: </li></ul><ul><ul><li>State machine ...
Automated Test Case Generation  - State Machine Modelling State 1 State 2 State 3 Event/Transition 1 Event/Transition 2 St...
State Machine Modelling <ul><li>Modelling GUI as a state machine with states and events (transitions)  : </li></ul><ul><ul...
Automated Planning System <ul><li>Based on an artificial intelligence algorithm </li></ul><ul><li>Defines  </li></ul><ul><...
Genetic Algorithm <ul><li>Generate tests simulating novice usage, a more random path achieves more system states </li></ul...
What is a “Good” GUI Suite?  Xie and Memon, ISSRE, 2006 <ul><li>Three experiments were conducted to answer these questions...
Used Terminology in Experiments <ul><li>Test case: A sequence of events </li></ul><ul><li>Fault: A mismatch between the ex...
Effect of Test Suite Size Fault Detection Effectiveness vs. Test Suite Size for TerpCalc Median For all suites: Fixed test...
Effect of Test Case Length Fault Detection Effectiveness vs. Test case length Size for TerpCalc For all suites: Fixed suit...
Effect of Event Composition <ul><li>Fixed test suite size </li></ul><ul><li>Fixed test case length </li></ul><ul><li>Resul...
Running GUI Test Cases <ul><li>Generated test cases (manually/automatically) can be run in two ways: </li></ul><ul><ul><li...
Record and Playback <ul><li>Capture events and put states assertions in scripts </li></ul><ul><li>Playback events in scrip...
Record and Playback Tools <ul><li>Java AWT and Swing </li></ul><ul><ul><li>Marathon </li></ul></ul><ul><ul><li>Jacareto </...
Record and playback tools -  Comparison Costello Jacareto Marathon Complexity Complex Complex Simple Playback Use Abbot Ro...
Selenium <ul><li>Runs directly in a browser </li></ul><ul><li>Supports web page testing </li></ul>
Demo
Drawbacks of Record & playback <ul><li>Requires the GUI to be built first </li></ul><ul><li>Assumes the GUI under test can...
GUI Architectures for TDD <ul><li>Following, are two proposed GUI architectures to enable GUI unit testing: </li></ul><ul>...
The Humble Dialog Box Michael Feathers, Object Mentor, Inc. 2002 <ul><li>Creates  smart objects  that contain the function...
The Humble Dialog -Example Paul Hamill, Unit Test Frameworks, O’Reilly , 2004 <ul><li>Consider a Library GUI application t...
The Humble Dialog Box – Example (Cont’d) <ul><li>Unit test for the  smart object : AddBook </li></ul><ul><li>public class ...
The Humble Dialog Box – Example (Cont’d) <ul><li>Unit test for the  humble dialog:  AddBookViewTest.java </li></ul><ul><li...
The Humble Dialog Box – Example (Cont’d) <ul><li>Unit test for the humble dialog:  AddBookViewTest.java </li></ul><ul><li>...
Drawbacks of the Humble Dialog Box Approach <ul><li>Tested GUI components have to be visible as non-private instance varia...
Presenter First (PF) Micah Alles et al., Presenter First, AGILE’06 <ul><li>A design and development technique to produce f...
Model View Presenter Pattern <ul><li>Model:  The data and logic for business aspects </li></ul><ul><li>View:  what the use...
MVP Variant for PF <ul><li>Model and View are isolated from each other </li></ul><ul><li>Model and View are members of Pre...
MVP Variant for PF (Cont’d) Presenter IModel <<interface>> IView <<interface>> Mock Model Mock View Used for testing the p...
PF Example – Presenter Class <ul><li>public class TBPresenter { </li></ul><ul><li>private ITBModel m_model; </li></ul><ul>...
The Burke Porter Dyno Host Project   http://atomicobject.com/pages/Burke+Porter+Dyno+Host <ul><li>Done in Presenter First ...
GUI TDD Unit Testing Problems <ul><li>Difficulty in verifying some behaviors  </li></ul><ul><ul><li>Example:  Check that t...
java.awt.Robot Class <ul><li>Emulates end-user interactions using the keyboard and/or mouse. </li></ul><ul><li>Limitations...
Programmatic GUI Testing Tools (Comparison) Abbot Jemmy JFCUnit Tests Robustness Robust Robust Robust Unit testing framewo...
Abbot Java GUI Test Framework <ul><li>Supports record and playback (Costello) and programmatic handling </li></ul><ul><li>...
Demo
Conclusion <ul><li>Record and playback tools cannot be used when developing a GUI following TDD </li></ul><ul><li>Designin...
References <ul><li>A.F. Memon, “GUI testing pitfalls and process”, Computer, v 35, n 8, Aug., pp. 87-88, 2002 </li></ul><u...
References (Cont’d) <ul><li>D.J. Kasik and H.G. George, “Toward Automatic Generation of Novice User Test Scripts,” Proc. C...
Thank you
Upcoming SlideShare
Loading in …5
×

Test Driven Development

1,311 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,311
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
39
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Test Driven Development

  1. 1. Test Driven Development & Testing the Graphical User Interface Prepared by Yibo Sun Soha Makady
  2. 2. Agenda <ul><li>What is GUI testing? </li></ul><ul><li>Generation of test cases </li></ul><ul><li>Running test cases </li></ul><ul><li>Design approaches for TDD of GUI </li></ul><ul><li>GUI TDD problems </li></ul><ul><li>Programmatic GUI Testing </li></ul><ul><li>Conclusion </li></ul>
  3. 3. Definition <ul><li>Th e process of testing a software product with graphical user interface </li></ul><ul><li>Like testing any other kind of codes, graphical user interface testing requires test cases </li></ul><ul><li>All GUI functionalities of the SUT should be covered by test cases </li></ul>
  4. 4. Importance <ul><li>GUI accounts for as much as 60% of the total application code </li></ul><ul><li>Use in safety critical systems </li></ul>
  5. 5. Benefits <ul><li>Safety </li></ul><ul><li>Robustness </li></ul><ul><li>Usability </li></ul>
  6. 6. Difficulty <ul><li>Scope: huge number of GUI operations, states which need to be tested </li></ul><ul><li>Sequence: GUI operations are dependent and rely on other operations, when the sequence is very long, the test case will be very long and error prone. </li></ul><ul><li>Regression Test: GUI may change significantly across versions, test cases need to be changed repeatedly </li></ul>
  7. 7. Coverage Criteria <ul><li>Conventional software testing uses coverage criteria which define a set of rules telling the type and amount of code to test </li></ul><ul><ul><li>It is hard to map GUI states and events to underlying code </li></ul></ul><ul><ul><li>GUI states </li></ul></ul><ul><li>It is hard to determine the most important subset of all possible test cases </li></ul>
  8. 8. Regression A.F. Memon and M.L. Soffa, “Regression testing of GUIs”, Software Engineering Notes, Sep. 2003 <ul><li>Test cases become obsolete </li></ul><ul><li>Recognize GUI components by names or IDs instead of their placements </li></ul><ul><li>74% of test cases are broken per version (study on Adobe Reader 4.0 and 5.0) </li></ul>
  9. 9. Generation of Test Cases <ul><li>Manual </li></ul><ul><li>Automated </li></ul><ul><li>Other: Monkey Testing </li></ul>
  10. 10. Automated Test Case Generation <ul><li>Approaches for automated test case generation: </li></ul><ul><ul><li>State machine modelling </li></ul></ul><ul><ul><li>Automated Planning System </li></ul></ul><ul><ul><li>Genetic Algorithm </li></ul></ul>
  11. 11. Automated Test Case Generation - State Machine Modelling State 1 State 2 State 3 Event/Transition 1 Event/Transition 2 State 4 Event/Transition 3
  12. 12. State Machine Modelling <ul><li>Modelling GUI as a state machine with states and events (transitions) : </li></ul><ul><ul><li>FSM (Finite State Machine) </li></ul></ul><ul><ul><ul><li>Scaling problem </li></ul></ul></ul><ul><ul><li>VFSM (Variable Finite State Machine) </li></ul></ul><ul><ul><li>Assertion for GUI states to be inserted manually, at points determined by test designers </li></ul></ul>
  13. 13. Automated Planning System <ul><li>Based on an artificial intelligence algorithm </li></ul><ul><li>Defines </li></ul><ul><ul><li>an initial state </li></ul></ul><ul><ul><li>a goal state </li></ul></ul><ul><ul><li>a set of operations </li></ul></ul><ul><li>A test plan is determined as a path from an initial state to a goal state </li></ul><ul><li>Testers focus on what function to test instead of how to test a function which is determined by the test plan path </li></ul><ul><li>Limitation: Manually decide the tasks (initial and goal states) to be tested and poorly chosen tasks can produce a test suite without adequate GUI testing coverage </li></ul>
  14. 14. Genetic Algorithm <ul><li>Generate tests simulating novice usage, a more random path achieves more system states </li></ul><ul><li>A gene is a path through GUI states transitions generated for some task </li></ul><ul><li>Choose an initial gene to mimic a novice path. </li></ul><ul><li>More genes are then generated by modifying and extending the initial gene </li></ul><ul><li>Genes that complete tasks best are kept, scored by some criteria as best novice paths. </li></ul><ul><li>Surviving genes are replicated and the natural selection process is repeated with more genes. </li></ul><ul><li>Eventually, multiple test cases to mimic novice paths (genes) are produced </li></ul><ul><li>Limitation: The initial gene has to be chosen manually, and thus this approach largely relies on the human expert’s ability to choose the initial gene. </li></ul>
  15. 15. What is a “Good” GUI Suite? Xie and Memon, ISSRE, 2006 <ul><li>Three experiments were conducted to answer these questions: </li></ul><ul><ul><li>How many test cases should be generated? </li></ul></ul><ul><ul><li>What should be the length of each test case? </li></ul></ul><ul><ul><li>What events should be put into a test case? </li></ul></ul><ul><li>Tested applications: TerpWord, TerpSpreadSheet, TerpPaint, and TerpCalc </li></ul><ul><li>The primary measured variable is the fault-detection effectiveness of a test suite </li></ul>
  16. 16. Used Terminology in Experiments <ul><li>Test case: A sequence of events </li></ul><ul><li>Fault: A mismatch between the expected and actual GUI states (asserted for every GUI property in a certain GUI state) </li></ul><ul><li>Fault-detection effectiveness: the number of unique faults detected by the test cases in the suite </li></ul>
  17. 17. Effect of Test Suite Size Fault Detection Effectiveness vs. Test Suite Size for TerpCalc Median For all suites: Fixed test case length Fixed event composition Smallest observation Largest observation Lower quartile Upper quartile
  18. 18. Effect of Test Case Length Fault Detection Effectiveness vs. Test case length Size for TerpCalc For all suites: Fixed suite size Fixed event composition
  19. 19. Effect of Event Composition <ul><li>Fixed test suite size </li></ul><ul><li>Fixed test case length </li></ul><ul><li>Results revealed that absence of an event (that interacted with a functional unit ) in a test suite directly effects the detection of a fault </li></ul><ul><li>Example: When running a test suite that does not contain a “click” event on a form full of buttons, not all faults are detected </li></ul>
  20. 20. Running GUI Test Cases <ul><li>Generated test cases (manually/automatically) can be run in two ways: </li></ul><ul><ul><li>Playing scripts: using Record & playback technique </li></ul></ul><ul><ul><li>Programmatically send commands/events: using API functions </li></ul></ul>
  21. 21. Record and Playback <ul><li>Capture events and put states assertions in scripts </li></ul><ul><li>Playback events in scripts </li></ul><ul><li>Compare expected state in scripts to actual UI state to determine the correctness of execution </li></ul><ul><li>Scripts can be edited when UI is changed </li></ul>
  22. 22. Record and Playback Tools <ul><li>Java AWT and Swing </li></ul><ul><ul><li>Marathon </li></ul></ul><ul><ul><li>Jacareto </li></ul></ul><ul><ul><li>Abbot(Costello) </li></ul></ul><ul><li>Web </li></ul><ul><ul><li>Selenium </li></ul></ul>
  23. 23. Record and playback tools - Comparison Costello Jacareto Marathon Complexity Complex Complex Simple Playback Use Abbot Robot Re-generate events Jython Component reference Unique IDs Numbering Text match in Python Other advantageous features Run test cases in JUnit Data analysis of user behaviour, multimedia demo Highly readable Python script, easy to manipulate
  24. 24. Selenium <ul><li>Runs directly in a browser </li></ul><ul><li>Supports web page testing </li></ul>
  25. 25. Demo
  26. 26. Drawbacks of Record & playback <ul><li>Requires the GUI to be built first </li></ul><ul><li>Assumes the GUI under test can already be functional </li></ul><ul><li>The generated test scripts cannot keep up with design changes </li></ul>
  27. 27. GUI Architectures for TDD <ul><li>Following, are two proposed GUI architectures to enable GUI unit testing: </li></ul><ul><ul><li>The Humble Dialog Box </li></ul></ul><ul><ul><li>Presenter First </li></ul></ul>
  28. 28. The Humble Dialog Box Michael Feathers, Object Mentor, Inc. 2002 <ul><li>Creates smart objects that contain the functional behavior of a particular GUI object </li></ul><ul><li>Adds a thin GUI view object ( a humble dialog ) to display the smart object’s information (and has no application logic) </li></ul><ul><li>Tests the smart objects mainly </li></ul><ul><li>As a result, the application behavior will be covered by unit tests, and the display code (GUI) will not require complex GUI tests </li></ul>
  29. 29. The Humble Dialog -Example Paul Hamill, Unit Test Frameworks, O’Reilly , 2004 <ul><li>Consider a Library GUI application that allows the user to add new books. </li></ul><ul><li>GUI design sketch for the AddBook dialog Window </li></ul><ul><li>The smart object AddBook and its humble dialog (view class) AddBookView </li></ul>
  30. 30. The Humble Dialog Box – Example (Cont’d) <ul><li>Unit test for the smart object : AddBook </li></ul><ul><li>public class AddBookTest extends TestCase { public void testAddBook( ) { </li></ul><ul><li>Library library = new Library( ); </li></ul><ul><li>AddBook addBook = new AddBook(library); addBook.add(&quot; Java &quot;, &quot;Carl Sagan&quot;); </li></ul><ul><li>assertNotNull( library.getBooksByTitle(&quot; Java &quot;) ); </li></ul><ul><li>} } </li></ul>
  31. 31. The Humble Dialog Box – Example (Cont’d) <ul><li>Unit test for the humble dialog: AddBookViewTest.java </li></ul><ul><li>public class AddBookViewTest extends TestCase {… </li></ul><ul><li>public void setUp( ) { … } </li></ul><ul><li>public void testControlValues( ) { </li></ul><ul><li>AddBookView view = new AddBookView( ); assertEquals( &quot;Add Book&quot;, view.getTitle( ) ); assertEquals( &quot;&quot;, view.titleField.getText( ) ); assertEquals( &quot;&quot;, view.authorField.getText( ) ); assertEquals( &quot;Add&quot;, view.addButton.getText( ) ); assertEquals( &quot;Cancel&quot;, view.cancelButton.getText( ) ); } </li></ul>Testing application GUI
  32. 32. The Humble Dialog Box – Example (Cont’d) <ul><li>Unit test for the humble dialog: AddBookViewTest.java </li></ul><ul><li>public class AddBookViewTest extends TestCase {… </li></ul><ul><li>… </li></ul><ul><li>public void testAddButton( ) { </li></ul><ul><li>view.titleField.setText(“Java&quot;); view.authorField.setText(&quot;Carl Sagan&quot;); view.addButton.doClick( ); </li></ul><ul><li>assertEquals(1, library.getBooksByTitle(“Java&quot;).size( )); </li></ul><ul><li>} } </li></ul>Testing application functionality
  33. 33. Drawbacks of the Humble Dialog Box Approach <ul><li>Tested GUI components have to be visible as non-private instance variables, or through getter methods, which conflicts with the normal design convention, and increases GUI code. </li></ul><ul><li>GUI components do not operate over the actual event mechanism, thus not all errors will be revealed. </li></ul><ul><ul><li>Example: Invisible and disabled components can be addressed in the unit tests. </li></ul></ul>
  34. 34. Presenter First (PF) Micah Alles et al., Presenter First, AGILE’06 <ul><li>A design and development technique to produce fully tested GUI applications following TDD </li></ul><ul><li>A variant of the Model View Presenter (MVP) design pattern </li></ul>
  35. 35. Model View Presenter Pattern <ul><li>Model: The data and logic for business aspects </li></ul><ul><li>View: what the user sees and interacts with </li></ul><ul><li>Presenter: Behavior that corresponds to customer stories (application writing) </li></ul>Any interaction between View and Model takes place through Presenter
  36. 36. MVP Variant for PF <ul><li>Model and View are isolated from each other </li></ul><ul><li>Model and View are members of Presenter, and do not know of its presence </li></ul><ul><li>Presenter works with Model and View interfaces </li></ul><ul><li>Testing the View can be automated using tools, or done manually </li></ul>Dashed lines are events, solid lines are object messages
  37. 37. MVP Variant for PF (Cont’d) Presenter IModel <<interface>> IView <<interface>> Mock Model Mock View Used for testing the presenter View Model Actual application items
  38. 38. PF Example – Presenter Class <ul><li>public class TBPresenter { </li></ul><ul><li>private ITBModel m_model; </li></ul><ul><li>private ITBView m_view; </li></ul><ul><li>public TBPresenter(ITBModel model, ITBView view) { </li></ul><ul><li>m_model = model; </li></ul><ul><li>m_view = view; </li></ul><ul><li>m_view.SubscribeItemDrag(new UpdateDelegate(ItemDrag)); </li></ul><ul><li>m_model.SubscribeToolboxItemsChanged(new </li></ul><ul><li>UpdateDelegate(ToolboxItemsChanged)); </li></ul><ul><li>}} </li></ul>
  39. 39. The Burke Porter Dyno Host Project http://atomicobject.com/pages/Burke+Porter+Dyno+Host <ul><li>Done in Presenter First Style </li></ul><ul><li>Status: </li></ul><ul><ul><li>6 developers for 9 months </li></ul></ul><ul><ul><li>One week iterations </li></ul></ul><ul><ul><li>99% estimation accuracy over project lifetime </li></ul></ul><ul><ul><li>Full test driven development (5200+ automated unit tests) </li></ul></ul><ul><ul><li>Project: A Visually configured automotive test and measurement application </li></ul></ul><ul><li>But… </li></ul><ul><ul><li>The level of experience of the developers is not mentioned as a factor in the estimates </li></ul></ul>
  40. 40. GUI TDD Unit Testing Problems <ul><li>Difficulty in verifying some behaviors </li></ul><ul><ul><li>Example: Check that the dialog box pops up modally showing the alert icon and the warning message in red.) </li></ul></ul><ul><li>Using ONLY JUnit has its limitations: </li></ul><ul><ul><li>Tests may start before the application starts as they run in a separate thread </li></ul></ul><ul><ul><li>When starting the application from within the unit test, it may take time loading, thus the test may fail due to the time out waiting for the GUI </li></ul></ul><ul><ul><li>There is no easy way to find and interact with GUI components except through Java Robot class </li></ul></ul>
  41. 41. java.awt.Robot Class <ul><li>Emulates end-user interactions using the keyboard and/or mouse. </li></ul><ul><li>Limitations: </li></ul><ul><ul><li>Takes control of the mouse and keyboard, so the user cannot do anything else on the computer while the test is running </li></ul></ul><ul><ul><li>Tests are fragile, where any change in the layout will break them </li></ul></ul><ul><ul><li>The user would have to add getter methods, or loop through the component hierarchy to reference a certain component. </li></ul></ul>
  42. 42. Programmatic GUI Testing Tools (Comparison) Abbot Jemmy JFCUnit Tests Robustness Robust Robust Robust Unit testing frameworks JUnit / TestNG None JUnit Component finding technique One finder class for all components A finder class for every component Through component naming Tests readability High Low Low
  43. 43. Abbot Java GUI Test Framework <ul><li>Supports record and playback (Costello) and programmatic handling </li></ul><ul><li>Supports both AWT and Swing components </li></ul><ul><li>Based on Java AWT Robot </li></ul><ul><li>Main components: </li></ul><ul><ul><li>A single finder class </li></ul></ul><ul><ul><li>A tester class for one/several components </li></ul></ul>
  44. 44. Demo
  45. 45. Conclusion <ul><li>Record and playback tools cannot be used when developing a GUI following TDD </li></ul><ul><li>Designing an application following the PF style, makes it easier to follow a TDD approach. </li></ul><ul><li>Testing the application GUI (view) can be done through a programmatic GUI testing tool that can find components regardless of their layout/position </li></ul><ul><li>There is still no clear criteria for determining the GUI testing coverage (e.g. no tool like Coverlipse for GUI testing coverage) </li></ul><ul><li>Unit testing GUI scenarios is complicated due to the need to simulate many actions to reach the initial state for a certain test case </li></ul>
  46. 46. References <ul><li>A.F. Memon, “GUI testing pitfalls and process”, Computer, v 35, n 8, Aug., pp. 87-88, 2002 </li></ul><ul><li>A.F. Memon and M.L. Soffa, “Regression testing of GUIs”, Software Engineering Notes, v 28, n 5, Sept., pp. 118-27, 2003 </li></ul><ul><li>R.K. Shehady and D.P. Siewiorek, “A Method to Automate User Interface Testing Using Variable Finite State Machines,” Proc. 27 th Ann. Int'l Symp. Fault-Tolerant Computing (FTCS '97), pp. 80-88, June 1997 </li></ul><ul><li>A.F. memon, M.E. Pollock, and M.L. Soffa, “ Hierarchical GUI test case generation using automated planning”, IEEE Transaction on Software Engineering, Vol. 27, Issue 2, Feb., pp. 144-55, 2001. </li></ul>
  47. 47. References (Cont’d) <ul><li>D.J. Kasik and H.G. George, “Toward Automatic Generation of Novice User Test Scripts,” Proc. Conf. Human Factors in Computing Systems: Common Ground, M.J. Tauber, V. Bellotti, R. Jeffries, J.D. Mackinlay, and J. Nielsen, eds., pp. 244-251, Apr. 1996 </li></ul><ul><li>Jacareto, http:// jacareto.sourceforge.net / </li></ul><ul><li>Abbot, http:// abbot.sourceforge.net/doc/overview.shtm </li></ul><ul><li>Selenium, http:// www.openqa.org /selenium/ . </li></ul><ul><li>Marathon, http:// www. marathon .com </li></ul><ul><li>JFCUnit, http:// jfcunit.sourceforge.net / </li></ul><ul><li>MVC and MVP patterns: http:// www.martinfowler.com </li></ul>
  48. 48. Thank you

×