GUI Test Patterns

3,602 views

Published on

Final presentation of my GUI Test Patterns Investigation

0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,602
On SlideShare
0
From Embeds
0
Number of Embeds
54
Actions
Shares
0
Downloads
0
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

GUI Test Patterns

  1. 1. Test Patterns for GUI’s MIEIC - Introdução à Investigação - 2008/2009 Rafael Araújo Pires Supervised by: Prof. Ana Paiva
  2. 2. “ Program testing can be used to show the presence of bugs, but never to show their absence” Dijkstra , E. W.; (1970) Notes On Structured Programming
  3. 3. Contents <ul><li>Objectives </li></ul><ul><li>What is a Pattern? </li></ul><ul><li>GUI test Patterns </li></ul><ul><li>Common problems testing GUI's </li></ul><ul><li>GUI's testing methods </li></ul><ul><li>Future work </li></ul>
  4. 4. Objectives <ul><li>Understand pattern concept </li></ul><ul><li>GUI test patterns </li></ul><ul><li>Identification of common problems </li></ul><ul><li>Methods and tools </li></ul><ul><li>Develop appropriated patterns </li></ul><ul><li>Patterns validation </li></ul>
  5. 5. What is a pattern?
  6. 6. What is a pattern? What is a pattern § GUI test patterns § Common problems § Testing methods § Future work “ a recurring solution to a recurring problem in a context”, Gerard Meszaros <ul><li>Forms… Description document of a pattern </li></ul><ul><li>Describing concrete solutions for known software problems </li></ul><ul><li>Languages... Pattern catalogs, collections or grammar </li></ul><ul><li>Some problems require the use of many patterns </li></ul>
  7. 7. GUI test patterns...
  8. 8. GUI test patterns What is a pattern § GUI test patterns § Common problems § Testing methods § Future work <ul><li>Research literature is not very extensive, when compared to the importance of this area </li></ul><ul><li>There are none developed or implemented test patterns about GUI testing </li></ul>
  9. 9. Common problems...
  10. 10. Common problems What is a pattern § GUI test patterns § Common problems § Testing methods § Future work <ul><li>Three main types of GUI failures/errors: </li></ul><ul><li>Usability errors </li></ul><ul><ul><li>Problems that the user has to pass through while using the system </li></ul></ul><ul><li>Functionality errors </li></ul><ul><ul><li>Appear when the system reacts in an unexpected way </li></ul></ul><ul><li>Performance errors </li></ul><ul><ul><li>Efficiency of the system </li></ul></ul><ul><ul><li>(Pimenta, 2006) </li></ul></ul>
  11. 11. Common problems <ul><li>Very large input spaces </li></ul><ul><li>Large number of states to be tested </li></ul><ul><li>Complex dependencies </li></ul><ul><li>External effects caused by GUI at any point of its execution </li></ul><ul><li>“ the greatest difficulty is detecting an effect when an error occurs, because it may not be observable on the screen” </li></ul><ul><li>(White, 2001) / (White, 2002) </li></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  12. 12. Common problems <ul><li>Test oracles... source of expected results for a test case </li></ul><ul><li>GUI tests demands every event checking </li></ul><ul><li>Oracle can’t execute only at the end of the test case </li></ul><ul><li>GUI test case should end as early as possible... </li></ul><ul><li>...because the final output can be correct but some intermediate events were incorrect </li></ul><ul><li>(Memon, 2002) </li></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  13. 13. Common problems <ul><li>it obliges a verification during the test case execution </li></ul><ul><ul><ul><li>Another solution... </li></ul></ul></ul><ul><li>Specify an expected output for each event, what is a resource intensive task </li></ul><ul><li>(Xie, 2006) </li></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  14. 14. Common problems <ul><li>How to evaluate the adequacy of GUI test cases? </li></ul><ul><li>Usually are used a code base criteria </li></ul><ul><ul><li>Do not imply that problematic interactions between GUI events have been tested. </li></ul></ul><ul><li>New criteria is needed </li></ul><ul><ul><li>Is essential to consider the possible number of paths that can be taken by users. </li></ul></ul><ul><ul><li>(Xie, 2006) / (Ruiz et al., 2008) </li></ul></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  15. 15. Common problems <ul><li>GUIs are often modified and continuously re-tested </li></ul><ul><ul><li>First generated test cases quickly become obsolete </li></ul></ul><ul><ul><li>New ones adapted to the latest version of the GUI are required </li></ul></ul><ul><li>After the writing of GUI tests the difficulties don’t end </li></ul><ul><ul><li>Maintaining them and solve those that fail can be so hard or even harder than writing them </li></ul></ul><ul><ul><li>(Xie, 2006) / (Ruiz et al., 2008) </li></ul></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  16. 16. Common problems <ul><li>Regression testing, guarantee that updates to the code will not introduce errors. </li></ul><ul><li>The dependency of the GUI layout is a challenge </li></ul><ul><ul><li>A change in the layout could modify the input/output mapping making old test cases ineffective. </li></ul></ul><ul><ul><li>(Memon, 2002) </li></ul></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  17. 17. Common problems <ul><li>Relatively to automation... </li></ul><ul><li>Transformation of the obtained information from many GUI components on machine readable data </li></ul><ul><li>Difficult for many GUI components like: </li></ul><ul><ul><li>Displays, screen navigation and program states </li></ul></ul><ul><ul><li>(Alsmadi, 2007) </li></ul></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  18. 18. Common problems <ul><li>… relatively to automation </li></ul><ul><li>Slower response time when compared to API tests </li></ul><ul><li>Test case explosion </li></ul><ul><ul><li>Due to the several ways to get to the same state </li></ul></ul><ul><li>Obstacles controlling the GUI actions </li></ul><ul><li>Complexity of checking outputs </li></ul><ul><li>(Pimenta, 2006) </li></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  19. 19. GUI testing methods...
  20. 20. GUI testing methods <ul><li>Capture/Replay... </li></ul><ul><li>Labor intensive technique which records GUI navigation path </li></ul><ul><li>Reproduce the records to recreate the evens with different inputs </li></ul><ul><li>Facilitate execution of tests </li></ul><ul><ul><li>Creation and maintenance are very resource intensive </li></ul></ul><ul><ul><li>Reuse of this type of test is almost impossible </li></ul></ul><ul><ul><li>(Memon, 2002) / (Xie, 2006) / (Jiang, 2002) </li></ul></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  21. 21. GUI testing methods <ul><li>...Capture/Replay... inconvenients </li></ul><ul><li>It only can be executed at the end of the process of development </li></ul><ul><li>Lack of support to design test cases and coverage criteria evaluation </li></ul><ul><li>Dependent on a physical interface </li></ul><ul><ul><li>invalidation of all tests when changes happen in the layout </li></ul></ul><ul><ul><li>Then is generated a re-capture, and everything is repeated again </li></ul></ul><ul><ul><li>(Pimenta, 2006) . </li></ul></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  22. 22. GUI testing methods <ul><li>CIS ( Complete Interaction Sequences )... </li></ul><ul><li>Focused in user sequences of GUI objects and selection </li></ul><ul><li>Concept called “responsibility” that represents the activities that interact with one or more GUI objects (tasks) </li></ul><ul><li>Responsibilities have an observable effect in the environment </li></ul><ul><ul><li>Changes in memory, behave of devices... </li></ul></ul><ul><li>Each responsibility has an associated CIS </li></ul><ul><li>(White, 2000) </li></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  23. 23. GUI testing methods <ul><li>...CIS ( Complete Interaction Sequences ) </li></ul><ul><li>Reduce the number of test cases required </li></ul><ul><li>Defects associated to each CIS </li></ul><ul><li>Technique especially resource intensive </li></ul><ul><li>With modification in GUI: </li></ul><ul><li>Responsibilities and the associated CIS have to be identified manually and rectified </li></ul><ul><li>(White, 2000) / (Xie, 2006) </li></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  24. 24. GUI testing methods <ul><li>Manual testing... </li></ul><ul><li>Complexity of GUI testing makes the manual approach unfit </li></ul><ul><li>Used to detect usability errors </li></ul><ul><li>Not systematic </li></ul><ul><ul><li>Weak coverage criteria </li></ul></ul><ul><ul><li>Doesn’t assure that all features are covered </li></ul></ul><ul><li>Dependent on the tester experience </li></ul><ul><li>(Pimenta, 2006) / (Memon, 2002) </li></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  25. 25. GUI testing methods <ul><li>...Manual testing </li></ul><ul><li>Difficulty of reproduction </li></ul><ul><li>Construct, execute or analyze the results of test cases is laborious </li></ul><ul><li>Tedious process and are influenced by human errors </li></ul><ul><li>When a change occurs in a GUI: </li></ul><ul><li>All test cases must be executed over again </li></ul><ul><li>A popular alternative is to turn available beta copies of the software and let the users be an active part on testing </li></ul><ul><li>(Memon, 2002) / (Pimenta, 2006) . </li></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  26. 26. GUI testing methods <ul><li>Random input testing </li></ul><ul><li>Also known as stochastic test or monkey test </li></ul><ul><li>Random interaction with the computer </li></ul><ul><li>Two main types of monkeys </li></ul><ul><ul><li>Dumb monkeys </li></ul></ul><ul><ul><li>Smart monkeys </li></ul></ul><ul><ul><li>(Nyman, 2000) / (Pimenta, 2006) </li></ul></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  27. 27. GUI testing methods <ul><li>Random input testing </li></ul><ul><li>Dumb monkeys </li></ul><ul><ul><li>Don't know the current state of the application </li></ul></ul><ul><ul><li>Don't recognizes software bugs </li></ul></ul><ul><ul><li>Objective is to crash the system during the test </li></ul></ul><ul><ul><li>Cost-effective tool on the discovery of faults </li></ul></ul><ul><ul><li>Semi-smart monkey, a variation that is able to recognize a fail on the system </li></ul></ul><ul><ul><li>(Nyman, 2000) / (Pimenta, 2006) </li></ul></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  28. 28. GUI testing methods <ul><li>Random input testing </li></ul><ul><li>Smart monkeys </li></ul><ul><ul><li>Tests is more difficult and expensive to develop </li></ul></ul><ul><ul><li>Knowledge about the application states and how to surpass and recognize the states </li></ul></ul><ul><ul><li>Capable to find more bugs </li></ul></ul><ul><ul><li>Useful for load and stress testing on multi-user systems </li></ul></ul><ul><ul><li>(Nyman, 2000) / (Pimenta, 2006) </li></ul></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  29. 29. GUI testing methods <ul><li>Random input testing </li></ul><ul><li>Difficult to ensure repeatability </li></ul><ul><li>Technique is arbitrary, time consuming and expensive </li></ul><ul><ul><li>Don't provide reliable coverage </li></ul></ul><ul><ul><li>“ James Tierney, former Director of Testing at Microsoft, has reported in internal presentations that some Microsoft applications groups have found 10 to 20 percent of the bugs in their projects using monkey test tools“ </li></ul></ul><ul><ul><li>(Dabóczi et al., 2003) / (Nyman, 2000) </li></ul></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  30. 30. GUI testing methods <ul><li>Unit testing frameworks </li></ul><ul><li>Imply manual generation of test cases </li></ul><ul><li>Interaction between the user and the GUI is simulated by the code that is written by the test </li></ul><ul><li>The test verifies if the output is the one expected </li></ul><ul><li>Centered on classes isolation </li></ul><ul><li>Is not appropriate for GUI testing </li></ul><ul><li>(Pimenta, 2006) / (Functional Test, 2007) </li></ul>What is a pattern § GUI test patterns § Common problems § Testing methods § Future work
  31. 31. GUI testing methods What is a pattern § GUI test patterns § Common problems § Testing methods § Future work <ul><li>Model-based testing </li></ul><ul><li>Testing applications trough their Application Programming Interface (API) </li></ul><ul><ul><li>Visual Test Development Environment </li></ul></ul><ul><ul><li>Integration Design and Automated Test Case Generation (IDATG) </li></ul></ul><ul><ul><li>GUI Testing framework (GUITAR) </li></ul></ul><ul><ul><li>(Beer et al., 1998) / (Ostrand et al., 1998) / (Pimenta, 2006) </li></ul></ul>
  32. 32. Future work...
  33. 33. Future work What is a pattern § GUI test patterns § Common problems § Testing methods § Future work <ul><li>Critical analysis of the state of the art </li></ul><ul><li>Development of patterns appropriate to solve the GUI testing problems </li></ul><ul><li>Validation of the test pattern </li></ul>
  34. 34. References...
  35. 35. References - Alsmadi, I. M.; (2007) Thesis: Building a user interface test automation framework using the data model, Graduate Faculty of the North Dakota State University of Agriculture and Applied Science - Beer, A.; Mohacsi, S.; Stary, C.; (1998) IDATG: an open tool for automated testing of interactive software, Proceedings of the COMPSAC'98 - The Twenty-Second Annual International Conference Computer Software and Applications, 470-475 - Dabóczi, T.; Kollár, I.; Simon, G.; Megyeri, T.; (2003) How to test graphical user interfaces, IEEE Instrumentation & Measurement Magazine, 6(3):27-33 - Dijkstra, E. W.; (1970) Notes On Structured Programming, 70-WSK-03 - Functional Test; (2007) [Online]; Available from, URL: http://www.c2.com/cgi/wiki?FunctionalTest - Memon, A. M.; (2002) GUI testing: pitfalls an process, Software Technologies, 35(8):87-88 - Nyman, N.; (2000) Using monkey test tools, STQE – Software Testing and Quality Engineering Magazine, 2(1):18-21
  36. 36. References - Pimenta, A.; (2006) Thesis: Automated specification-based testing of graphical user interfaces, Faculdade de Engenharia da Universidade do Porto - Ruiz, A.; Price, Y. W.; (2008) GUI testing made easy, Practice and Research Techniques, TAIC PART '08. Testing: Academic & Industrial Conference, 99-103 - White, L.; Almezen, H.; (2002) Generating test cases for GUI responsibilities using complete interaction sequences, 11th International Symposium on Software Reliability Engineering, 110 -121 - White, L.; Almezen, H.; Alzeidi, N.; (2001) User-based testing of GUI sequences and their interactions, 12th International Symposium on Software Reliability Engineering, 54 – 63 - Xie, Q.; (2006) Thesis: Developing cost-effective model-based techniques for GUI testing, University of Maryland, College Park
  37. 37. Thank you for your attention!! MIEIC - Introdução à Investigação - 11/02/2009 Rafael Araújo Pires Supervised by: Prof. Ana Paiva

×