PRACTICAL SIKULI
USING SCREENSHOTS FOR GUI AUTOMATION AND TESTING
Tsung-Hsiang (Sean) Chang
MIT CSAIL
Collaborators: Tom Yeh, Rob Miller
1

WHAT IS SIKULI?
2

WHAT IS SIKULI?
2

2
3
1
3

Low-level API?
Mac’s system design?
4

EMPTY THE TRASH CAN
5

OUTLINE
• What Sikuli is
• How Sikuli works
• Sikuli for GUI automation
• Sikuli for GUI testing
• Conclusions
6

How Sikuli Works
7

TEMPLATE MATCHING
click(
)
8

TEMPLATE MATCHING
click(
)
8

SYSTEM DESIGN
9

DEMO 1:
EMPTY TRASH CAN
2
3
1
10

REGION AND MATCH
Region
match = region.find( or “text pattern”)
11

REGION
• basic attributes: x, y, width, height
• Match is a Region
• Screen is also a Region
• chained methods: Screen(0).find(W).click(X)
12

KEY METHODS OF REGION
• find, findAll, exists, wait, waitVanish
• Mouse Actions
click, doubleClick, rightClick, hover, dragDrop
mouseDown, mouseUp
• Keyboard Actions
type(text), paste(text)
keyDown, keyUp
13

EX2: FACEBOOK STALKER
14

EX2: FACEBOOK STALKER
14

RESTRICT SEARCH REGION
15

But the Region constant is bad
16

APP CLASS
• firefox = App.open(“Firefox”)
• firefox.focus(), firefox.close()
• firefox.window([n])
• returns the Region of Firefox’s n-th window
17

FIND IN CERTAIN APP
18

EX3: MUTE SPEAKERS
19

EX3: MUTE SPEAKERS
19

SPATIAL OPERATORS
20

SPATIAL OPERATORS
20

SPATIAL OPERATORS (2)
21

TEXT MATCHING AND
RECOGNITION
• click(“OK”), find(“Cancel”)
• region.text() - read the text in a region
22

VISUAL-EVENT-DRIVEN
PROGRAMMING
• observes visual events in any Region
• Region.onAppear(Pattern, handler)
• Region.onVanish(Pattern, handler)
• Region.onChange(Pattern, handler)
23

VISUAL-EVENT-DRIVEN
PROGRAMMING
24

VISUAL-EVENT-DRIVEN
PROGRAMMING
24

VISUAL DICTIONARY
25

GUI
TESTING
26

VISUAL ASSERTION
• make sure the appearance of visual feedback
27

VISUAL ASSERTION
• make sure the appearance of visual feedback
27

JUNIT INTEGRATION
28

SIKULI IN COMMAND LINE
usage:
Sikuli-­‐IDE
[-­‐-­‐args
<arguments>]
[-­‐h]
[-­‐r
<sikuli-­‐file>]
[-­‐s]
[-­‐t
<sikuli-­‐test-­‐case>]
-­‐-­‐args
<arguments>
specify
the
arguments
passed
to
Jython's
sys.argv
-­‐h,-­‐-­‐help
print
this
help
message
-­‐r,-­‐-­‐run
<sikuli-­‐file>
run
.sikuli
or
.skl
file
-­‐s,-­‐-­‐stderr
print
runtime
errors
to
stderr
instead
of
popping
up
a
message
box
-­‐t,-­‐-­‐test
<sikuli-­‐test-­‐case>
run
.sikuli
as
a
unit
test
case
with
junit's
text
UI
runner
29

CONCLUSIONS
30

PLATFORM INDEPENDENCE
• Works
on ANY GUI that can be displayed on
Windows/Linux/Mac
• Virtual machines
• Remote desktop
• Mobile simulators: Android, iPhone
• Web: Flash, HTML+Javascript
31

PROGRAM AGAINST UI
Sikuli programs are written against the user interface
instead of an API
UI: visible, familiar, always exists
API: faster, probably more stable
32

FUTURE WORK
• fast and accurate OCR on screen
• Accessibility API integration
33

Thanks!
http://sikuli.org
or, just google “sikuli”
New Sikuli X 1.0-rc2 released.
Follow us on twitter @sikuli
Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.)
34