Sikuli is a tool that uses image recognition to automate and test graphical user interfaces (GUIs). It works by using screenshots of the screen and image matching to interact with GUI elements. This allows it to be platform independent and work across different operating systems and applications. Sikuli can be used for test automation by adding visual assertions and integrating with testing frameworks like JUnit. It also supports event-driven programming by monitoring screens for visual changes.

1. PRACTICAL SIKULI
USING SCREENSHOTS FOR GUI AUTOMATION AND TESTING
Tsung-Hsiang (Sean) Chang
MIT CSAIL
Collaborators: Tom Yeh, Rob Miller
1

2. WHAT IS SIKULI?
2

3. WHAT IS SIKULI?
2

4. 2
3
1
3

5. Low-level API?
Mac’s system design?
4

6. EMPTY THE TRASH CAN
5

7. OUTLINE
• What Sikuli is
• How Sikuli works
• Sikuli for GUI automation
• Sikuli for GUI testing
• Conclusions
6

8. How Sikuli Works
7

9. TEMPLATE MATCHING
click(
)
8

10. TEMPLATE MATCHING
click(
)
8

11. SYSTEM DESIGN
9

12. DEMO 1:
EMPTY TRASH CAN
2
3
1
10

13. REGION AND MATCH
Region
match = region.find( or “text pattern”)
11

14. REGION
• basic attributes: x, y, width, height
• Match is a Region
• Screen is also a Region
• chained methods: Screen(0).find(W).click(X)
12

15. KEY METHODS OF REGION
• find, findAll, exists, wait, waitVanish
• Mouse Actions
click, doubleClick, rightClick, hover, dragDrop
mouseDown, mouseUp
• Keyboard Actions
type(text), paste(text)
keyDown, keyUp
13

16. EX2: FACEBOOK STALKER
14

17. EX2: FACEBOOK STALKER
14

18. RESTRICT SEARCH REGION
15

19. But the Region constant is bad
16

20. APP CLASS
• firefox = App.open(“Firefox”)
• firefox.focus(), firefox.close()
• firefox.window([n])
• returns the Region of Firefox’s n-th window
17

21. FIND IN CERTAIN APP
18

22. EX3: MUTE SPEAKERS
19

23. EX3: MUTE SPEAKERS
19

24. SPATIAL OPERATORS
20

25. SPATIAL OPERATORS
20

26. SPATIAL OPERATORS (2)
21

27. TEXT MATCHING AND
RECOGNITION
• click(“OK”), find(“Cancel”)
• region.text() - read the text in a region
22

28. VISUAL-EVENT-DRIVEN
PROGRAMMING
• observes visual events in any Region
• Region.onAppear(Pattern, handler)
• Region.onVanish(Pattern, handler)
• Region.onChange(Pattern, handler)
23

29. VISUAL-EVENT-DRIVEN
PROGRAMMING
24

30. VISUAL-EVENT-DRIVEN
PROGRAMMING
24

31. VISUAL DICTIONARY
25

32. GUI
TESTING
26

33. VISUAL ASSERTION
• make sure the appearance of visual feedback
27

34. VISUAL ASSERTION
• make sure the appearance of visual feedback
27

35. JUNIT INTEGRATION
28

36. SIKULI IN COMMAND LINE
usage:
Sikuli-­‐IDE
[-­‐-­‐args
<arguments>]
[-­‐h]
[-­‐r
<sikuli-­‐file>]
[-­‐s]
[-­‐t
<sikuli-­‐test-­‐case>]
-­‐-­‐args
<arguments>
specify
the
arguments
passed
to
Jython's
sys.argv
-­‐h,-­‐-­‐help
print
this
help
message
-­‐r,-­‐-­‐run
<sikuli-­‐file>
run
.sikuli
or
.skl
file
-­‐s,-­‐-­‐stderr
print
runtime
errors
to
stderr
instead
of
popping
up
a
message
box
-­‐t,-­‐-­‐test
<sikuli-­‐test-­‐case>
run
.sikuli
as
a
unit
test
case
with
junit's
text
UI
runner
29

37. CONCLUSIONS
30

38. PLATFORM INDEPENDENCE
• Works
on ANY GUI that can be displayed on
Windows/Linux/Mac
• Virtual machines
• Remote desktop
• Mobile simulators: Android, iPhone
• Web: Flash, HTML+Javascript
31

39. PROGRAM AGAINST UI
Sikuli programs are written against the user interface
instead of an API
UI: visible, familiar, always exists
API: faster, probably more stable
32

40. FUTURE WORK
• fast and accurate OCR on screen
• Accessibility API integration
33

41. Thanks!
http://sikuli.org
or, just google “sikuli”
New Sikuli X 1.0-rc2 released.
Follow us on twitter @sikuli
Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.)
34