PRACTICAL SIKULIUSING SCREENSHOTS FOR GUI AUTOMATION AND TESTING             Tsung-Hsiang (Sean) Chang                    ...
WHAT IS SIKULI?       2
WHAT IS SIKULI?       2
2    3            13
Low-level API?Mac’s system design?         4
EMPTY THE TRASH CAN         5
OUTLINE• What     Sikuli is• How      Sikuli works• Sikuli   for GUI automation• Sikuli   for GUI testing• Conclusions    ...
How Sikuli Works       7
TEMPLATE MATCHINGclick(	  	  	  	  	  	  	  	  	  	  	  	  	  	  )                                                        ...
TEMPLATE MATCHINGclick(	  	  	  	  	  	  	  	  	  	  	  	  	  	  )                                                        ...
SYSTEM DESIGN      9
DEMO 1:EMPTY TRASH CAN                  2            3                      1       10
REGION AND MATCH                                            Regionmatch = region.find(        or “text pattern”)           ...
REGION• basic   attributes: x, y, width, height• Match    is a Region• Screen    is also a Region• chained    methods: Scr...
KEY METHODS OF REGION•   find, findAll, exists, wait, waitVanish•   Mouse Actions    click, doubleClick, rightClick, hover, ...
EX2: FACEBOOK STALKER          14
EX2: FACEBOOK STALKER          14
RESTRICT SEARCH REGION          15
But the Region constant is bad              16
APP CLASS• firefox   = App.open(“Firefox”)• firefox.focus(), firefox.close()• firefox.window([n])  • returns   the Region of F...
FIND IN CERTAIN APP         18
EX3: MUTE SPEAKERS        19
EX3: MUTE SPEAKERS        19
SPATIAL OPERATORS        20
SPATIAL OPERATORS        20
SPATIAL OPERATORS (2)          21
TEXT MATCHING AND           RECOGNITION• click(“OK”), find(“Cancel”)• region.text()   - read the text in a region          ...
VISUAL-EVENT-DRIVEN          PROGRAMMING• observes   visual events in any Region • Region.onAppear(Pattern, handler) • Reg...
VISUAL-EVENT-DRIVEN   PROGRAMMING         24
VISUAL-EVENT-DRIVEN   PROGRAMMING         24
VISUAL DICTIONARY        25
GUITESTING   26
VISUAL ASSERTION• make   sure the appearance of visual feedback                           27
VISUAL ASSERTION• make   sure the appearance of visual feedback                           27
JUNIT INTEGRATION        28
SIKULI IN COMMAND LINEusage:Sikuli-­‐IDE	  [-­‐-­‐args	  <arguments>]	  [-­‐h]	  [-­‐r	  <sikuli-­‐file>]	  [-­‐s]	  [-­‐t...
CONCLUSIONS     30
PLATFORM INDEPENDENCE• Works      on ANY GUI that can be displayed on Windows/Linux/Mac • Virtual   machines • Remote     ...
PROGRAM AGAINST UISikuli programs are written against the user interfaceinstead of an API	   UI: visible, familiar, always...
FUTURE WORK• fast   and accurate OCR on screen• Accessibility API   integration                                    33
Thanks!           http://sikuli.org      or, just google “sikuli”                                            New Sikuli X ...
Upcoming SlideShare
Loading in …5
×

Practical Sikuli: using screenshots for GUI automation and testing

101,985 views

Published on

This's a Sikuli tutorial I gave to the Megason Lab at Harvard Medical School.

Published in: Technology
9 Comments
50 Likes
Statistics
Notes
  • Recently published a test automation framework that is powered by Sikuli. Check it out here:
    https://github.com/smysnk/sikuli-framework
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Interesting, I will try it later
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Interesting tool.. liked it so far
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • @Arye probably you may not set the path for SIKULI &python in environmental variables...
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • why it is not working for me? this is the error:
    [error] Error message: Traceback (most recent call last):
    File 'C:\DOCUME~1\ANUDEL~1\LOCALS~1\Temp\sikuli-tmp6054047037597739699.py', line 2, in
    while not r.exits('1325492826226.png'):
    AttributeError: 'Region' object has no attribute 'exits'
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
101,985
On SlideShare
0
From Embeds
0
Number of Embeds
1,819
Actions
Shares
0
Downloads
1,792
Comments
9
Likes
50
Embeds 0
No embeds

No notes for slide

Practical Sikuli: using screenshots for GUI automation and testing

  1. 1. PRACTICAL SIKULIUSING SCREENSHOTS FOR GUI AUTOMATION AND TESTING Tsung-Hsiang (Sean) Chang MIT CSAIL Collaborators: Tom Yeh, Rob Miller 1
  2. 2. WHAT IS SIKULI? 2
  3. 3. WHAT IS SIKULI? 2
  4. 4. 2 3 13
  5. 5. Low-level API?Mac’s system design? 4
  6. 6. EMPTY THE TRASH CAN 5
  7. 7. OUTLINE• What Sikuli is• How Sikuli works• Sikuli for GUI automation• Sikuli for GUI testing• Conclusions 6
  8. 8. How Sikuli Works 7
  9. 9. TEMPLATE MATCHINGclick(                            ) 8
  10. 10. TEMPLATE MATCHINGclick(                            ) 8
  11. 11. SYSTEM DESIGN 9
  12. 12. DEMO 1:EMPTY TRASH CAN 2 3 1 10
  13. 13. REGION AND MATCH Regionmatch = region.find( or “text pattern”) 11
  14. 14. REGION• basic attributes: x, y, width, height• Match is a Region• Screen is also a Region• chained methods: Screen(0).find(W).click(X) 12
  15. 15. KEY METHODS OF REGION• find, findAll, exists, wait, waitVanish• Mouse Actions click, doubleClick, rightClick, hover, dragDrop mouseDown, mouseUp• Keyboard Actions type(text), paste(text) keyDown, keyUp 13
  16. 16. EX2: FACEBOOK STALKER 14
  17. 17. EX2: FACEBOOK STALKER 14
  18. 18. RESTRICT SEARCH REGION 15
  19. 19. But the Region constant is bad 16
  20. 20. APP CLASS• firefox = App.open(“Firefox”)• firefox.focus(), firefox.close()• firefox.window([n]) • returns the Region of Firefox’s n-th window 17
  21. 21. FIND IN CERTAIN APP 18
  22. 22. EX3: MUTE SPEAKERS 19
  23. 23. EX3: MUTE SPEAKERS 19
  24. 24. SPATIAL OPERATORS 20
  25. 25. SPATIAL OPERATORS 20
  26. 26. SPATIAL OPERATORS (2) 21
  27. 27. TEXT MATCHING AND RECOGNITION• click(“OK”), find(“Cancel”)• region.text() - read the text in a region 22
  28. 28. VISUAL-EVENT-DRIVEN PROGRAMMING• observes visual events in any Region • Region.onAppear(Pattern, handler) • Region.onVanish(Pattern, handler) • Region.onChange(Pattern, handler) 23
  29. 29. VISUAL-EVENT-DRIVEN PROGRAMMING 24
  30. 30. VISUAL-EVENT-DRIVEN PROGRAMMING 24
  31. 31. VISUAL DICTIONARY 25
  32. 32. GUITESTING 26
  33. 33. VISUAL ASSERTION• make sure the appearance of visual feedback 27
  34. 34. VISUAL ASSERTION• make sure the appearance of visual feedback 27
  35. 35. JUNIT INTEGRATION 28
  36. 36. SIKULI IN COMMAND LINEusage:Sikuli-­‐IDE  [-­‐-­‐args  <arguments>]  [-­‐h]  [-­‐r  <sikuli-­‐file>]  [-­‐s]  [-­‐t  <sikuli-­‐test-­‐case>]  -­‐-­‐args  <arguments>                          specify  the  arguments  passed  to  Jythons  sys.argv  -­‐h,-­‐-­‐help                                            print  this  help  message  -­‐r,-­‐-­‐run  <sikuli-­‐file>                  run  .sikuli  or  .skl  file  -­‐s,-­‐-­‐stderr                                        print  runtime  errors  to  stderr  instead  of  popping  up  a  message  box  -­‐t,-­‐-­‐test  <sikuli-­‐test-­‐case>      run  .sikuli  as  a  unit  test  case  with  junits  text  UI  runner 29
  37. 37. CONCLUSIONS 30
  38. 38. PLATFORM INDEPENDENCE• Works on ANY GUI that can be displayed on Windows/Linux/Mac • Virtual machines • Remote desktop • Mobile simulators: Android, iPhone • Web: Flash, HTML+Javascript 31
  39. 39. PROGRAM AGAINST UISikuli programs are written against the user interfaceinstead of an API UI: visible, familiar, always exists API: faster, probably more stable 32
  40. 40. FUTURE WORK• fast and accurate OCR on screen• Accessibility API integration 33
  41. 41. Thanks! http://sikuli.org or, just google “sikuli” New Sikuli X 1.0-rc2 released. Follow us on twitter @sikuliThanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.) 34

×