1.
PRACTICAL SIKULI
USING SCREENSHOTS FOR GUI AUTOMATION AND TESTING
Tsung-Hsiang (Sean) Chang
MIT CSAIL
Collaborators: Tom Yeh, Rob Miller
1

2.
WHAT IS SIKULI?
2

3.
WHAT IS SIKULI?
2

4.
2
3
1
3

5.
Low-level API?
Mac’s system design?
4

6.
EMPTY THE TRASH CAN
5

7.
OUTLINE
• What Sikuli is
• How Sikuli works
• Sikuli for GUI automation
• Sikuli for GUI testing
• Conclusions
6

8.
How Sikuli Works
7

9.
TEMPLATE MATCHING
click(
)
8

10.
TEMPLATE MATCHING
click(
)
8

11.
SYSTEM DESIGN
9

12.
DEMO 1:
EMPTY TRASH CAN
2
3
1
10

13.
REGION AND MATCH
Region
match = region.find( or “text pattern”)
11

14.
REGION
• basic attributes: x, y, width, height
• Match is a Region
• Screen is also a Region
• chained methods: Screen(0).find(W).click(X)
12

15.
KEY METHODS OF REGION
• find, findAll, exists, wait, waitVanish
• Mouse Actions
click, doubleClick, rightClick, hover, dragDrop
mouseDown, mouseUp
• Keyboard Actions
type(text), paste(text)
keyDown, keyUp
13

16.
EX2: FACEBOOK STALKER
14

17.
EX2: FACEBOOK STALKER
14

18.
RESTRICT SEARCH REGION
15

19.
But the Region constant is bad
16

20.
APP CLASS
• firefox = App.open(“Firefox”)
• firefox.focus(), firefox.close()
• firefox.window([n])
• returns the Region of Firefox’s n-th window
17

21.
FIND IN CERTAIN APP
18

22.
EX3: MUTE SPEAKERS
19

23.
EX3: MUTE SPEAKERS
19

24.
SPATIAL OPERATORS
20

25.
SPATIAL OPERATORS
20

26.
SPATIAL OPERATORS (2)
21

27.
TEXT MATCHING AND
RECOGNITION
• click(“OK”), find(“Cancel”)
• region.text() - read the text in a region
22

28.
VISUAL-EVENT-DRIVEN
PROGRAMMING
• observes visual events in any Region
• Region.onAppear(Pattern, handler)
• Region.onVanish(Pattern, handler)
• Region.onChange(Pattern, handler)
23

29.
VISUAL-EVENT-DRIVEN
PROGRAMMING
24

30.
VISUAL-EVENT-DRIVEN
PROGRAMMING
24

31.
VISUAL DICTIONARY
25

32.
GUI
TESTING
26

33.
VISUAL ASSERTION
• make sure the appearance of visual feedback
27

34.
VISUAL ASSERTION
• make sure the appearance of visual feedback
27

35.
JUNIT INTEGRATION
28

36.
SIKULI IN COMMAND LINE
usage:
Sikuli-­‐IDE
[-­‐-­‐args
<arguments>]
[-­‐h]
[-­‐r
<sikuli-­‐file>]
[-­‐s]
[-­‐t
<sikuli-­‐test-­‐case>]
-­‐-­‐args
<arguments>
specify
the
arguments
passed
to
Jython's
sys.argv
-­‐h,-­‐-­‐help
print
this
help
message
-­‐r,-­‐-­‐run
<sikuli-­‐file>
run
.sikuli
or
.skl
file
-­‐s,-­‐-­‐stderr
print
runtime
errors
to
stderr
instead
of
popping
up
a
message
box
-­‐t,-­‐-­‐test
<sikuli-­‐test-­‐case>
run
.sikuli
as
a
unit
test
case
with
junit's
text
UI
runner
29

37.
CONCLUSIONS
30

38.
PLATFORM INDEPENDENCE
• Works
on ANY GUI that can be displayed on
Windows/Linux/Mac
• Virtual machines
• Remote desktop
• Mobile simulators: Android, iPhone
• Web: Flash, HTML+Javascript
31

39.
PROGRAM AGAINST UI
Sikuli programs are written against the user interface
instead of an API
UI: visible, familiar, always exists
API: faster, probably more stable
32

40.
FUTURE WORK
• fast and accurate OCR on screen
• Accessibility API integration
33

41.
Thanks!
http://sikuli.org
or, just google “sikuli”
New Sikuli X 1.0-rc2 released.
Follow us on twitter @sikuli
Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.)
34