В докладе рассказано о нескольких подходах к автоматизации тестирования через пользовательский интерфейс. Вы узнаете как автоматизировать приложения на Java Swing. Также будет рассмотрен инструмент автоматизированного тестирования Jemmy, продемонстрирована работа с ним. Еще вы познакомитесь с новыми возможностями Jemmy 3.

1. <Insert Picture Here>
Automating JFC UI application testing with Jemmy.
Alexandre (Shura) Iline
Java SE and JavaFX Quality architect.

2. The following is intended to outline our general product
direction. It is intended for information purposes, and
may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or
functionality, and should not be relied upon in making
purchasing decisions.
The development, release, and timing of any features
or functionality described for Oracle's products remains
at the sole discretion of Oracle.

3. This presentation shares experience gotten
from testing such products as
JRockit JavaFX AT

4. What's in scope
• All sorts of definitions
“Testing”, “UI testing”, “Test automation”
• Jemmy library
• Automation approaches effectiveness and cost.
• Bunch of source code
• Basic Jemmy operations
• Less basic operations
• Switches and settings
• Tricks and tips
• Misconceptions

5. What's not in scope
• The economical effects
• A.k.a. what's “quality”
• Why to automate
• Automation cost vs. manual testing cost
• When to automate
• What to automate
• Non-economical benefits of automation
A.k.a. let's leave it for later. :)

6. UI testing … by Wikipedia
«GUI software testing is the
process of testing a product
that uses a graphical user
interface, to ensure it meets its
written specifications.»

7. UI testing … most often ...
«Checking whether usage of a
product UI leads to results
expected by the the person
who performs testing»

8. Sample test scenario
●Start text editor
●Push «File/Open»
●Verify file chooser directory
●Select some file
●Verify editor area content
●Verify application title
●Verify buttons availabilities
●
....

9. UI test workflow
Pass
Perform necessary
actions
Find Pass Do Pass Verify
Find next control Verify that expected
To perform operation State reached
Fail Fail Fail
Failure analysis

10. Test automation ... for me
«Building a process which
exercises and reports certain
product characteristics while
run unattended.»

11. Test automation is like development
• Putting logic in a code
• Same lifecycle:
• Requirements
• Design
• Implementation
• Same set of problems
• Bugs
• Instabilities
• Scope
• defined through test specification/plan

12. Test automation is not like
development
• Big fat dependency – the tested product
• vs many libraries and platform
• Many small programs
• vs one big program
• Does one thing good – reports status
• vs does many thing ... good
• Perfectness is not the goal
• other than maintenance cost, ease of use

13. Jemmy

14. Jemmy history
• Started as a tool to tests TeamWare UI (1999)
• Used for NetBeans extensions (2000)
• Official test tool for NetBeans (2001)
• Open-source (2001)
• Is used for NetBeans and extensions since then
• Adopted as a test tool for Swing (~2003)
• Used outside SUN Microsystems (next slide)
• Jemmy v3 created (2008)
• The test tool for JavaFX SDK (2008 – now)
• Extended to support SWT
• Test tool for JRMC

15. Jemmy and extensions
Jemmy v2 LCDUI
NetBeans JemmySG
AWT & Swing
JDK JemmyAWT JemmyCore JemmyRemote
JemmySWT
Jemmy v3 JemmyFX JemmyFXRemote
JRMC
JavaFX Authoring
JavaFX
Tool

16. Jemmy v2
JFC

17. Same VM.
Test code runs in the same VM as the application code
• Benefits
• Full access to the application UI objects
• As well as the application domain object (sometimes).
• Control over event queue
• Drawbacks
• Impacts the tested UI
• Options
• Run application from test
• Run test from application
• Use accessibility hook

18. Demo
Using Jemmy

19. Operators
JFrameOperator
JFrameOperator
JTextFieldOperator
JTextFieldOperator getTitle(),
getText(), getTitle(),
getText(), ...
...
typeText(...), ...
typeText(...), ...
JButtonOperator JComboBoxOperator
JComboBoxOperator
JButtonOperator
getText(), getSelectedItem(),
getSelectedItem(),
getText(),
push() selectItem(...)
selectItem(...)
push()
Test code
Test code

20. Demo
Operators

21. Threading
“noblocking”
operation
periodical
checks
action waiting action waiting action
action

22. Demo
Waiting
Timeouts

23. Event queue
User
UI
Other
threads

24. Event queue
with test
Lookup
Actions
Verifications
Test
UI
Other
threads

25. Demo
Event queue tools

26. Drivers
TextDriver
typeText() WindowDriver
...
JFrameOperator
JTextFieldOperator
ButtonDriver ListDriver
push(...) slectItem(...)
... ...
JButtonOperator JComboBoxOperator
Test code

27. Demo
Drivers

28. Dispatching modes
• Events
• Proper events in a proper order are dispatched to a
component making it thinking some user actions are
performed.
• Shortcut
• Same as events except the events are posted into the event
queue at once.
• Robot
• java.awt.Robot is used
Implemented with driver sets

29. Demo
Dispatching models

30. Image
• Fragile
• Needed for custom controls
• Comparison
• Pixel-to-pixel
• %s of pixels
• Color distance
• Average
• Maximum

31. Demo
Verifying images

32. Automation
approaches

33. Application UI
Product UI
Product UI
33
33

34. Coordinates
• click(134,32) //selects some record
• click(215,122) //hits “Properties”
• sleep(5) //sleeps to let dialog be painted
• click(64,182) //expands color combo
• click(235,182) //selects Gray
• click(235,212) //hit OK
34
34

35. Widgets
• Find “Car records” frame
• Find table
• Select “1abc234” cell
• Push “Properties” button
• Wait for “1abc234” dialog
• Select “Gray” color in combo box
• Push “OK”
35
35

36. Widgets or coordinates
Car record
Domain
Domain
model
model
Model
Model Make VIN
VIN Color Year License plate
Make Color Year License plate
Product UI
Product UI
Test
36
36

37. Demo
Cars test
Widgets

38. UI Primitives
• Find car list frame
CarListFrame list = new CarListFrame()
• Open properties dialog for car “1abc234”
CarDialog propDialog =
list.carProperties(“1abc234”);
• Set color to gray
propDialog.setColor(Color.GRAY);
• Apply changes
propDialog.ok();
38
38

39. Library
Domain
Car record
Domain
Model Make VIN Color Year License plate
model
Model Make VIN Color Year License plate
model
Product UI
Product UI
CarListFrame Test library
Test library CarDialog
Test
39
39

40. Demo
Cars test
UI Library

41. Domain model
• Set color to gray for a car “1abc234”
new CarRecord(“1abc234”).
setColor(Color.GRAY);
Underneath the cover, CarRecord class does
all described earlier
41
41

42. Domain library
Domain
Car record
Domain
model
Model Make VIN Color Year License plate
model
Model Make VIN Color Year License plate
Product
Product
UI
UI
CarList UI test library
UI test library CarDialog
CarRecord Domain test library
Domain test library
Test
42
42

43. Demo
Cars test
Domain Library

44. Automation
effectiveness

45. The formula?
TM * NR * NC
EA =
TD + TS * NR * NC
EA – automation effectiveness
To be used for every particular product.
NR and NC are unique for a product.
TM is a characteristic of a test suite.
Smaller TD and TS - higher the EA.
Coefficient depend on the way you write your tests

46. Td and Ts together
8
7.5
7 Td/Tm
6
Ts/Tm
5
5
4
3
3
2
1.1 1
1
0.5
0.1 0.05
0
Coordinates Widgets UI Library Domain library

47. TD and TS for NC=3, NR=8, TM=1
30
25.1 Td
25 24
Ts
Td+(Ts*Nc*Nr)
20
15
15
12
10 8.7
7.4 7.5
5
5
3 2.4
1.1 1.2
0
Coordinates Widgets UI Library Domain library

48. EA for NC=3, NR=8
3.5
3.24
3
2.76
2.5
2
1.6
1.5
0.96
1
0.5
0
Coordinates Widgets UI Library Domain library
Ea

49. Misconceptions
• Automated testing replaces manual testing
• it will find all the bugs
• it will find more bugs
• it does the same amount of work
• Create once use forever
• This is easy
• This is too hard
• No need to create test plan
• “Will just buy the XXX tool and it will solve all out
problems”

50. Resources
jemmy.dev.java.net

51. <Insert Picture Here>
Automating JFC UI application testing.
With Jemmy.
The technical aspects.
Alexandre (Shura) Iline
Java SE and JavaFX Quality architect.

52. Backup slides

53. Coverage
• Implementation coverage
• Line, block, condition, sequence, ...
• Specification coverage
• How well the tests covering the functional specification
• Public API coverage
• Whether the public API is tested fully
• UI coverage
• Whether the tests cover UI fully
• Combine with bug density
• First cover the areas with more bugs
• Combine with code complexity
• First cover the more complicated code.

54. Continuous build
Build
Executed
Code automatically Success
Commit
changes after commit
Yes.
No
Analysis.
Rollback!!!
Development
Promote
Test further Code is compilable

55. Continuous build with testing
Code Build Success
changes Commit
No
Rollback!!! Yes.
= Compilation
Test Analysis. successful
changes No
Testing
Test further Passed
Build is good Yes Is it working?
Code line is healthy
Go on ...

56. Pre-integration
Code
changes
Yes
Testing Passed Commit
Test
changes No

57. How to benefit from UI automation?
Use it!
• Sanity
• Pre-integration
• Attach to bug reports
• Make is a quality criteria
• Run it for every build
• Show it to the boss :)
• Don't forget to show it to the dev. team