1.
PRACTICAL SIKULIUSING SCREENSHOTS FOR GUI AUTOMATION AND TESTING Tsung-Hsiang (Sean) Chang MIT CSAIL Collaborators: Tom Yeh, Rob Miller 1

2.
WHAT IS SIKULI? 2

3.
WHAT IS SIKULI? 2

4.
2 3 13

5.
Low-level API?Mac’s system design? 4

6.
EMPTY THE TRASH CAN 5

7.
OUTLINE• What Sikuli is• How Sikuli works• Sikuli for GUI automation• Sikuli for GUI testing• Conclusions 6

8.
How Sikuli Works 7

9.
TEMPLATE MATCHINGclick( ) 8

10.
TEMPLATE MATCHINGclick( ) 8

11.
SYSTEM DESIGN 9

12.
DEMO 1:EMPTY TRASH CAN 2 3 1 10

13.
REGION AND MATCH Regionmatch = region.find( or “text pattern”) 11

14.
REGION• basic attributes: x, y, width, height• Match is a Region• Screen is also a Region• chained methods: Screen(0).find(W).click(X) 12

15.
KEY METHODS OF REGION• find, findAll, exists, wait, waitVanish• Mouse Actions click, doubleClick, rightClick, hover, dragDrop mouseDown, mouseUp• Keyboard Actions type(text), paste(text) keyDown, keyUp 13

16.
EX2: FACEBOOK STALKER 14

17.
EX2: FACEBOOK STALKER 14

18.
RESTRICT SEARCH REGION 15

19.
But the Region constant is bad 16

20.
APP CLASS• firefox = App.open(“Firefox”)• firefox.focus(), firefox.close()• firefox.window([n]) • returns the Region of Firefox’s n-th window 17

21.
FIND IN CERTAIN APP 18

22.
EX3: MUTE SPEAKERS 19

23.
EX3: MUTE SPEAKERS 19

24.
SPATIAL OPERATORS 20

25.
SPATIAL OPERATORS 20

26.
SPATIAL OPERATORS (2) 21

27.
TEXT MATCHING AND RECOGNITION• click(“OK”), find(“Cancel”)• region.text() - read the text in a region 22

28.
VISUAL-EVENT-DRIVEN PROGRAMMING• observes visual events in any Region • Region.onAppear(Pattern, handler) • Region.onVanish(Pattern, handler) • Region.onChange(Pattern, handler) 23

29.
VISUAL-EVENT-DRIVEN PROGRAMMING 24

30.
VISUAL-EVENT-DRIVEN PROGRAMMING 24

31.
VISUAL DICTIONARY 25

32.
GUITESTING 26

33.
VISUAL ASSERTION• make sure the appearance of visual feedback 27

34.
VISUAL ASSERTION• make sure the appearance of visual feedback 27

35.
JUNIT INTEGRATION 28

36.
SIKULI IN COMMAND LINEusage:Sikuli-­‐IDE [-­‐-­‐args <arguments>] [-­‐h] [-­‐r <sikuli-­‐file>] [-­‐s] [-­‐t <sikuli-­‐test-­‐case>] -­‐-­‐args <arguments> specify the arguments passed to Jythons sys.argv -­‐h,-­‐-­‐help print this help message -­‐r,-­‐-­‐run <sikuli-­‐file> run .sikuli or .skl file -­‐s,-­‐-­‐stderr print runtime errors to stderr instead of popping up a message box -­‐t,-­‐-­‐test <sikuli-­‐test-­‐case> run .sikuli as a unit test case with junits text UI runner 29

37.
CONCLUSIONS 30

38.
PLATFORM INDEPENDENCE• Works on ANY GUI that can be displayed on Windows/Linux/Mac • Virtual machines • Remote desktop • Mobile simulators: Android, iPhone • Web: Flash, HTML+Javascript 31

39.
PROGRAM AGAINST UISikuli programs are written against the user interfaceinstead of an API UI: visible, familiar, always exists API: faster, probably more stable 32

40.
FUTURE WORK• fast and accurate OCR on screen• Accessibility API integration 33

41.
Thanks! http://sikuli.org or, just google “sikuli” New Sikuli X 1.0-rc2 released. Follow us on twitter @sikuliThanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.) 34