Олег Бунин
Testing JavaFX UI
Tips from JavaFX quality team
Alexandre (Shura) Iline
Java and JavaFX quality architect
Oracle
Agenda
●UI testing overview
●Test automation approaches
●Jemmy library
●Test automation effectiveness
●Test library design...
UI testing … by Wikipedia
«GUI software testing is the process of
testing a product that uses a graphical user
interface, ...
UI testing … most often ...
«Checking whether usage of a product UI
leads to results expected by the the person
who perfor...
●Start text editor
●Push «File/Open»
●Verify file chooser directory
●Select some file
●Verify editor area content
●Verify ...
UI Testing Manual Automated
Initial step Design test specification. Create tests.
Establish regular runs.
Ongoing Click, c...
Development
Continuous build
Commit
Build
Executed
automagically
after commit
Success
Analysis.
Rollback!!!
Code
changes
N...
Continuous build with testing
Commit
Build Success
Testing
Is it working?
Passed
Analysis.
Rollback!!!
Code
changes
Test
c...
Automation
approach
Record && Replay Coding
Test creation Inexpensive*
Usually just repeating manual test
in special envir...
Jemmy
Jemmy v2
Started as a tool to tests TeamWare UI (1999)
Used for NetBeans extensions (2000)
Official test tool for NetBeans...
Jemmy usages
●V2 (Swing/AWT)
● Internal: Swing, NetBeans, Glassfish
● External:
● Used by: Boeing (x2), agfa.com, compuwar...
Demo...
Finally!
Jemmy v3JavaFX
AWT/Swing
SWT
Scenegraph
LCDUI
JemmyCore
Interfaces
AWT robot FX robot
Hierarchy
Input
Scene, node
containe...
UI test
Find Do VerifyPass Pass
Pass
Fail Fail Fail
Failure analysis
Find next control
To perform operation
On in
Verify t...
Lookup
● By ID
● Easiest but may not be possible
● «open_file_btn»
● By type
● Most common
● Button, text, combo box, etc
...
UIcouldbe
verycomplicated
UIcouldbe
verycomplicated
UIcouldbe
verycomplicated
Lookup demo
Interfaces
Interface Control types Description
Mouse,
Keyboard,
Drag
Enything Low level input
Parent Containers, list (for...
Interfaces demo
Verifications
UI feedback Non UI feedback
Dialog displayed
Test changed
Image updated
Progress bar changed position
File c...
Waiting
●UI is a multy-thread environment
Things happen in background
Test code is in another thread
●Not much could be re...
Verification demo
Test automation
effectiveness
TD + *TS
NR
TM * NR
NC*
EA =
NC
*
EA
– automation effectiveness
NR
and NC
are characteristics for a product.
TM
is a chara...
1 2 3 4 5 6 7 8
0
5
10
15
20
25
0
1
2
3
4
Nc = 1 Ratio
Auto Manual Effect
Time
1 2 3 4 5 6 7 8
0
5
10
15
20
25
0
1
2
3
4
N...
Td
or Ts
– what to minimize
TS
- if (NC
* NR
) is big
Multi-platform
Compatibility with external products (servers, browse...
Tests fail every now and then ...
… because the tested UI is changed
Tests fail every now and then ...
Ah! And also because errors are made ...
End2End demo
Ts
continuends
Time spent on What to do %% of time
Allocating failures Use test harness 1% - 5%
Anderstandint the
failure ...
Harness
TestNGTestNG
Understanding failure
Logging
UI operations
Going to push ... button
Selecting ... menu
UI responces
Found list view: …
UI...
Demo
Jemmy logging functionality
Fix the tests
One UI, many-many tests.
Do as few test code changes as possible
Ideally … one change in test code for one
c...
Decrease maintenance cost
●Implement reusable operations in a library
openFile(String filename)
●Decrease implementation d...
Demo
Refining test.
Library design approaches
Remember the formula?
TD
+ *TS
NR
TM
* NR
NC
*
EA
=
NC
*
EA
– automation effectiveness
To be used for every particular pro...
TS
mainly consists of time for test modification.
When product changes, tests need to be changed accordingly.
Many tests! ...
Application domain modelCarrecordCarrecordCarrecord
ColorColor
ModelModel MakeMake
YearYear License plateLicense plate
VIN...
Application UI
Product UIProduct UI
Coordinates
● click(134,32) //selects some record
● click(215,122) //hits “Properties”
● sleep(5) //sleeps to let dialog b...
Widgets
● Find “Car records” frame
● Find table
● Select “1abc234” cell
● Push “Properties” button
● Wait for “1abc234” di...
ProductUIProductUIDomain
model
Domain
model
Car record
ColorColorModelModel
MakeMake YearYear License plateLicense plateVI...
UI Primitives● Find car list frame
CarListFrame list = new CarListFrame()
● Open properties dialog for car “1abc234”
CarDi...
ProductUIProductUIDomain
model
Domain
model
Car record
Test libraryTest library
Color
Color
Model
Model
Make
Make
Year
Yea...
Domain model
● Set col or t o gr ay f or a car
“ 1abc234”
new Car Recor d( “ 1abc234” ) .
set Col or ( Col or . GRAY) ;
Un...
Product
UI
Product
UI
Domain
model
Domain
model
Car record
Domain test libraryDomain test library
Color
Color
Model
Model
...
Coordinates Widgets UI Library Domain library
0
1
2
3
4
5
6
7
8
1.1
3
5
7.5
1
0.5
0.1 0.05
Td/Tm
Ts/Tm
Td
and Ts
together
Coordinates Widgets UI Library Domain library
0
5
10
15
20
25
30
1.1
3
5
7.5
24
12
2.4
1.2
25.1
15
7.4
8.7
Td
Ts
Td+(Ts*Nc...
EA
for NC
=3, NR
=8
Coordinates Widgets UI Library Domain library
0
0.5
1
1.5
2
2.5
3
3.5
0.96
1.6
3.24
2.76
Ea
Олег Бунин
Testing JavaFX UI
Tips from JavaFX quality team
Alexandre (Shura) Iline
Java and JavaFX quality architect
Oracle
Links
●http://jemmy.dev.java.net
Upcoming SlideShare
Loading in...5
×

Alexandre Iline Rit 2010 Java Fxui

708

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
708
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Note that in reality all the verifications should be done throwg waiting
  • Effectiveness is estimated under assumtion that the testing is thoruogh and there are many tests
  • This is a well accepeted practice nowadays.
    However it is not really possible without testing.
    And it is not really possible with manual testing as the turnaround is too long to rely on.
  • This is a well accepeted practice nowadays.
    However it is not really possible without testing.
    And it is not really possible with manual testing as the turnaround is too long to rely on.
  • If we get to it, I will be howing figures which explain why do we think so
  • It is not a toy
  • ditto
  • The values of course depend on how the fast product is changing
    The estimations are for NetBeans – we went throughthis stage
    Now would happen if the combobox is replaced by color chooser
    page up test would fail, 'cause it's looking for combobox page down
    Most importantly ... all tests would fail!
  • Obvious solution
  • Alexandre Iline Rit 2010 Java Fxui

    1. 1. Олег Бунин Testing JavaFX UI Tips from JavaFX quality team Alexandre (Shura) Iline Java and JavaFX quality architect Oracle
    2. 2. Agenda ●UI testing overview ●Test automation approaches ●Jemmy library ●Test automation effectiveness ●Test library design approaches
    3. 3. UI testing … by Wikipedia «GUI software testing is the process of testing a product that uses a graphical user interface, to ensure it meets its written specifications.»
    4. 4. UI testing … most often ... «Checking whether usage of a product UI leads to results expected by the the person who performs testing»
    5. 5. ●Start text editor ●Push «File/Open» ●Verify file chooser directory ●Select some file ●Verify editor area content ●Verify application title ●Verify buttons availabilities ●....
    6. 6. UI Testing Manual Automated Initial step Design test specification. Create tests. Establish regular runs. Ongoing Click, click, click Analize results File bugs Change the specification Change the tests Qualification High, low for test executors High Effectiveness Low High Other Inexpensive to start Continuous quality monitoring , inexpensive to reuse Fun Bo-o-o-ring Much like programming
    7. 7. Development Continuous build Commit Build Executed automagically after commit Success Analysis. Rollback!!! Code changes No Yes. Promote Code is compilableTest further
    8. 8. Continuous build with testing Commit Build Success Testing Is it working? Passed Analysis. Rollback!!! Code changes Test changes No No Test further Build is good Code line is healthy Go on ... Yes Yes. = Compilation successful
    9. 9. Automation approach Record && Replay Coding Test creation Inexpensive* Usually just repeating manual test in special environment Must be accompanied by other means Expensive* Consists of programming Test execution Does not depend on approach Test maintanance Higher (in most cases)* Very much depends on test format Lower* Depends on principles of building test library Test analysis Does not depend on approach (*) Much more information closer to the end of this presentation
    10. 10. Jemmy
    11. 11. Jemmy v2 Started as a tool to tests TeamWare UI (1999) Used for NetBeans extensions (2000) Official test tool for NetBeans (2001) Open-source (2001) Jemmy v3 Started in (2008) as a proof of concept experiemnt Extended to support JavaFX (2009) Opensource with support of JavaFX 1.2 (2009) Developed in close-source for 1.3 since then
    12. 12. Jemmy usages ●V2 (Swing/AWT) ● Internal: Swing, NetBeans, Glassfish ● External: ● Used by: Boeing (x2), agfa.com, compuware.com, attix5.com, exgate.tek.com, globo.com, gmx.de, non.agilent.com ● Evaluated by: syntelinc.com, covast.com, visus-tt.com, covast.com, nokia.com, twt.com, ubs.com ●V3 (FX) ● Internal: JavaFX, Authoring Tool, JavaStore
    13. 13. Demo... Finally!
    14. 14. Jemmy v3JavaFX AWT/Swing SWT Scenegraph LCDUI JemmyCore Interfaces AWT robot FX robot Hierarchy Input Scene, node containers Scroller, slider, list, text, ... Generic interfaces Covered in this presentation
    15. 15. UI test Find Do VerifyPass Pass Pass Fail Fail Fail Failure analysis Find next control To perform operation On in Verify that expected State reached Perform necessary actions
    16. 16. Lookup ● By ID ● Easiest but may not be possible ● «open_file_btn» ● By type ● Most common ● Button, text, combo box, etc ● By index ● Unavoidable ● 2nd button with text «browse» ● By toString() ● Sucks ● projectmanager.Main$Main $Script$1Scene$ObjLit$20 @4d16318b ● By text, tooltip, associated label ● Best, if possible ● Button with text «browse»
    17. 17. UIcouldbe verycomplicated
    18. 18. UIcouldbe verycomplicated
    19. 19. UIcouldbe verycomplicated
    20. 20. Lookup demo
    21. 21. Interfaces Interface Control types Description Mouse, Keyboard, Drag Enything Low level input Parent Containers, list (for its content) Something you could look within Selectable Toggle button, radio button, combobox, check box, lists, etc A control which provides limited number of choices CaretOwner Text box, scroll bar, slider A control which has a number value which changes within some range Text Text box Editable text container
    22. 22. Interfaces demo
    23. 23. Verifications UI feedback Non UI feedback Dialog displayed Test changed Image updated Progress bar changed position File created Database updated Sunset happened :)
    24. 24. Waiting ●UI is a multy-thread environment Things happen in background Test code is in another thread ●Not much could be really verified ●Everything should be waited for
    25. 25. Verification demo
    26. 26. Test automation effectiveness
    27. 27. TD + *TS NR TM * NR NC* EA = NC * EA – automation effectiveness NR and NC are characteristics for a product. TM is a characteristic of a test suite TD and TS depend on test automation approach Smaller TD and TS - higher the EA .
    28. 28. 1 2 3 4 5 6 7 8 0 5 10 15 20 25 0 1 2 3 4 Nc = 1 Ratio Auto Manual Effect Time 1 2 3 4 5 6 7 8 0 5 10 15 20 25 0 1 2 3 4 Nc = 3 Ratio Auto Manual Effect Time Assumptions: TM = 1 engineer*week TS = 0.1 * TM TD = 5 * TM NR = 8
    29. 29. Td or Ts – what to minimize TS - if (NC * NR ) is big Multi-platform Compatibility with external products (servers, browsers, ...) Long-living TD - if (NC * NR ) is small Proof of concept Preview
    30. 30. Tests fail every now and then ... … because the tested UI is changed
    31. 31. Tests fail every now and then ... Ah! And also because errors are made ...
    32. 32. End2End demo
    33. 33. Ts continuends Time spent on What to do %% of time Allocating failures Use test harness 1% - 5% Anderstandint the failure reason Use test logging, save images, save UI state 10% - 80% Fix the tests Move common code to the library (*) 90% - 10% (*) The only way it is different from programming is that there are a lot of tests. More on this later.
    34. 34. Harness TestNGTestNG
    35. 35. Understanding failure Logging UI operations Going to push ... button Selecting ... menu UI responces Found list view: … UI hierarchy Scene «…» Group «…» Button «…» UI states Images
    36. 36. Demo Jemmy logging functionality
    37. 37. Fix the tests One UI, many-many tests. Do as few test code changes as possible Ideally … one change in test code for one change in UI code.
    38. 38. Decrease maintenance cost ●Implement reusable operations in a library openFile(String filename) ●Decrease implementation dependencies lookup(Button.class, new ByText<Button>(«Low»)) ●Increasing stability .wrap(0).mouse().click(); Thread.sleep(1000); ...
    39. 39. Demo Refining test.
    40. 40. Library design approaches
    41. 41. Remember the formula? TD + *TS NR TM * NR NC * EA = NC * EA – automation effectiveness To be used for every particular product. NR and NC are unique for a product. TM is a characteristic of a test suite. Smaller TD and TS - higher the EA . Coefficient depend on the way you write your tests
    42. 42. TS mainly consists of time for test modification. When product changes, tests need to be changed accordingly. Many tests! Hundreds. “Less changes of test code per a change in the product UI.” Ideally ... no more than one. But ... how? You are the coders – you know: Move code to test library.
    43. 43. Application domain modelCarrecordCarrecordCarrecord ColorColor ModelModel MakeMake YearYear License plateLicense plate VINVIN ...... That's ... car catalog of some sort
    44. 44. Application UI Product UIProduct UI
    45. 45. Coordinates ● click(134,32) //selects some record ● click(215,122) //hits “Properties” ● sleep(5) //sleeps to let dialog be painted ● click(64,182) //expands color combo ● click(235,182) //selects Gray ● click(235,212) //hit OK TTdd ~= 1.1 * T~= 1.1 * Tmm ,, TTss ~= 1 * T~= 1 * TmmNever tried, but ...
    46. 46. Widgets ● Find “Car records” frame ● Find table ● Select “1abc234” cell ● Push “Properties” button ● Wait for “1abc234” dialog ● Select “Gray” color in combo box ● Push “OK”
    47. 47. ProductUIProductUIDomain model Domain model Car record ColorColorModelModel MakeMake YearYear License plateLicense plateVINVIN Test Widgets or coordinates TTDD ~= 3 * T~= 3 * TMM ,, TTSS ~= .5 * T~= .5 * TMM
    48. 48. UI Primitives● Find car list frame CarListFrame list = new CarListFrame() ● Open properties dialog for car “1abc234” CarDialog propDialog = list.carProperties(“1abc234”); ● Set color to gray propDialog.setColor(Color.GRAY); ● Apply changes propDialog.ok();
    49. 49. ProductUIProductUIDomain model Domain model Car record Test libraryTest library Color Color Model Model Make Make Year Year License plate License plate VIN VIN CarListFrame CarDialog Test TTDD ~= 5 * T~= 5 * TMM ,, TTSS ~= .2 * T~= .2 * TMM
    50. 50. Domain model ● Set col or t o gr ay f or a car “ 1abc234” new Car Recor d( “ 1abc234” ) . set Col or ( Col or . GRAY) ; Underneath the cover, CarRecord class does all described earlier
    51. 51. Product UI Product UI Domain model Domain model Car record Domain test libraryDomain test library Color Color Model Model Make Make Year Year License plate License plate VIN VIN UI test libraryUI test libraryCarList CarDialog CarRecord Test TTDD ~= 7.5 * T~= 7.5 * TMM ,, TTSS ~= .05 * T~= .05 * TMM
    52. 52. Coordinates Widgets UI Library Domain library 0 1 2 3 4 5 6 7 8 1.1 3 5 7.5 1 0.5 0.1 0.05 Td/Tm Ts/Tm Td and Ts together
    53. 53. Coordinates Widgets UI Library Domain library 0 5 10 15 20 25 30 1.1 3 5 7.5 24 12 2.4 1.2 25.1 15 7.4 8.7 Td Ts Td+(Ts*Nc*Nr) TD and TS for NC =3, NR =8, TM =1
    54. 54. EA for NC =3, NR =8 Coordinates Widgets UI Library Domain library 0 0.5 1 1.5 2 2.5 3 3.5 0.96 1.6 3.24 2.76 Ea
    55. 55. Олег Бунин Testing JavaFX UI Tips from JavaFX quality team Alexandre (Shura) Iline Java and JavaFX quality architect Oracle
    56. 56. Links ●http://jemmy.dev.java.net
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×