UI Test Automation Effectiveness

1,103 views

Published on

Доклад Ильина Александра на SQA Days 7

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,103
On SlideShare
0
From Embeds
0
Number of Embeds
33
Actions
Shares
0
Downloads
20
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

UI Test Automation Effectiveness

  1. 1. <Insert Picture Here> UI Test Automation Effectiveness Alexandre (Shura) Iline Java SE and JavaFX Quality architect.
  2. 2. The following is intended to outline our general product direction. It is intended for information purposes, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle.
  3. 3. This presentation shares experience gotten from testing such products as JavaFX Authoring Tool
  4. 4. What's effectiveness • Regular functional testing • Saving human time • Special groups of tests (Things that could only be done through automated testing) • Doing it with less human time
  5. 5. UI testing … by Wikipedia «GUI software testing is the process of testing a product that uses a graphical user interface, to ensure it meets its written specifications.»
  6. 6. UI testing … most often ... «Checking whether usage of a product UI leads to results expected by the the person who performs testing»
  7. 7. Sample test scenario ●Start text editor ●Push «File/Open» ●Verify file chooser directory ●Select some file ●Verify editor area content ●Verify application title ●Verify buttons availabilities ● ....
  8. 8. Test automation ... for me «Building a process which exercises and reports certain product characteristics while run unattended.»
  9. 9. Test automation is like development • Putting logic in a code • Same lifecycle: • Requirements • Design • Implementation • Same set of problems • Bugs • Instabilities • Scope • defined through test specification/plan
  10. 10. Test automation is not like development • Big fat dependency – the tested product • vs many libraries and platform • Many small programs • vs one big program • Does one thing good – reports status • vs does many thing ... good • Perfectness is not the goal • other than maintenance cost, ease of use
  11. 11. UI Test automation • Avoidable • Testing could be done with no automation • Less obvious • Many misconceptions • More expensive • Requires specific tools • Such as the test drivers
  12. 12. Misconceptions • Automated testing replaces manual testing • it will find all the bugs • it will find more bugs • it does the same amount of work • Create once use forever • This is easy • This is too hard • No need to create test plan • “Will just buy the XXX tool and it will solve all out ptoblems”
  13. 13. UI changes • “Improvements” • Specification/requirement changes • Bug fixes If UI is not changing ... Why bother testing?
  14. 14. Test base characteristics • Coverage • How well the test base tests the product? • Initial creation cost • How expensive it is to initially implement it? • Sustainability • How expensive it is to support it? • Reliability • Does it have intermittent failures?
  15. 15. Coverage • Implementation coverage • Line, block, condition, sequence, ... • Specification coverage • How well the tests covering the functional specification • Public API coverage • Whether the public API is tested fully • UI coverage • Whether the tests cover UI fully • Combine with bug density • First cover the areas with more bugs • Combine with code complexity • First cover the more complicated code.
  16. 16. Initial creation • Record && replay • Inexpensive in creation • Lacks functionality • (More) expensive to support • Disables effective UI test automation • Coding • Expensive to create • Require experienced engineers • Encourages effective UI testing • (Much) Less expensive to support
  17. 17. Tests code needed to be changed • Test code needs to be updated every time UI changes colorComboBox.selectColor(“Grey”) turns into colorButton.push(); ... colorChooser.selectColor(Color.GREY);
  18. 18. Application UI Product UI Product UI 18 18
  19. 19. Coordinates • click(134,32) //selects some record • click(215,122) //hits “Properties” • sleep(5) //sleeps to let dialog be painted • click(64,182) //expands color combo • click(235,182) //selects Gray • click(235,212) //hit OK 19 19
  20. 20. Widgets • Find “Car records” frame • Find table • Select “1abc234” cell • Push “Properties” button • Wait for “1abc234” dialog • Select “Gray” color in combo box • Push “OK” 20 20
  21. 21. Widgets or coordinates Car record Domain Domain model model Model Model Make VIN VIN Color Year License plate Make Color Year License plate Product UI Product UI Test 21 21
  22. 22. UI Primitives • Find car list frame CarListFrame list = new CarListFrame() • Open properties dialog for car “1abc234” CarDialog propDialog = list.carProperties(“1abc234”); • Set color to gray propDialog.setColor(Color.GRAY); • Apply changes propDialog.ok(); 22 22
  23. 23. Library Domain Car record Domain Model Make VIN Color Year License plate model Model Make VIN Color Year License plate model Product UI Product UI CarListFrame Test library Test library CarDialog Test 23 23
  24. 24. Domain model • Set color to gray for a car “1abc234” new CarRecord(“1abc234”). setColor(Color.GRAY); Underneath the cover, CarRecord class does all described earlier 24 24
  25. 25. Domain library Domain Car record Domain model Model Make VIN Color Year License plate model Model Make VIN Color Year License plate Product Product UI UI CarList UI test library UI test library CarDialog CarRecord Domain test library Domain test library Test 25 25
  26. 26. The formula? TM * NR * NC EA = TD + TS * NR * NC EA – automation effectiveness To be used for every particular product. NR and NC are unique for a product. TM is a characteristic of a test suite. Smaller TD and TS - higher the EA. Coefficient depend on the way you write your t
  27. 27. Td and Ts together 8 7.5 7 Td/Tm 6 Ts/Tm 5 5 4 3 3 2 1.1 1 1 0.5 0.1 0.05 0 Coordinates Widgets UI Library Domain library
  28. 28. TD and TS for NC=3, NR=8, TM=1 30 25.1 Td 25 24 Ts Td+(Ts*Nc*Nr) 20 15 15 12 10 8.7 7.4 7.5 5 5 3 2.4 1.1 1.2 0 Coordinates Widgets UI Library Domain library
  29. 29. 3.5 EA for NC=3, NR=8 3.24 3 2.76 2.5 2 1.6 1.5 0.96 1 0.5 0 Coordinates Widgets UI Library Domain library Ea
  30. 30. Other Ts improvements • Store resources in a property file • Take from the application itself
  31. 31. Testbase reliability • Select right tool • No sleeps – only waitings • Use event queue • Right lookup criteria • By ID • By text • By location • Not by index
  32. 32. Special test types • Continuous build tests • aka sanity aka acceptance aka smoke • Pre-integration • Performance 32 32
  33. 33. Continuous build Build Executed Code automatically Success Commit changes after commit Yes. No Analysis. Rollback!!! Development Promote Test further Code is compilable
  34. 34. Continuous build with testing Code Build Success changes Commit No Rollback!!! Yes. = Compilation Test Analysis. successful changes No Testing Test further Passed Build is good Yes Is it working? Code line is healthy Go on ...
  35. 35. Pre-integration Code changes Yes Testing Passed Commit Test changes No
  36. 36. How to benefit from UI automation? Use it! • Sanity • Pre-integration • Attach to bug reports • Make is a quality criteria • Run it for every build • Show it to the boss :) • Don't forget to show it to the dev. team
  37. 37. <Insert Picture Here> SQE reporting infrastructure improvements Alexandre (Shura) Iline Java and JavaFX Quality architect. shura@sun.com

×