Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Practical Sikuli: using screenshots for GUI automation and testing

111,176 views

Published on

This's a Sikuli tutorial I gave to the Megason Lab at Harvard Medical School.

Published in: Technology
  • Want to become expert in Sikuli?. Just visit and learn Sikuli courses from Simpliv. https://www.simpliv.com/developmenttool/show-and-tell-sikuli-pattern-matching-and-automation
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Recently published a test automation framework that is powered by Sikuli. Check it out here:
    https://github.com/smysnk/sikuli-framework
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Interesting, I will try it later
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Interesting tool.. liked it so far
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • @Arye probably you may not set the path for SIKULI &python in environmental variables...
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Practical Sikuli: using screenshots for GUI automation and testing

  1. 1. PRACTICAL SIKULIUSING SCREENSHOTS FOR GUI AUTOMATION AND TESTING Tsung-Hsiang (Sean) Chang MIT CSAIL Collaborators: Tom Yeh, Rob Miller 1
  2. 2. WHAT IS SIKULI? 2
  3. 3. WHAT IS SIKULI? 2
  4. 4. 2 3 13
  5. 5. Low-level API?Mac’s system design? 4
  6. 6. EMPTY THE TRASH CAN 5
  7. 7. OUTLINE• What Sikuli is• How Sikuli works• Sikuli for GUI automation• Sikuli for GUI testing• Conclusions 6
  8. 8. How Sikuli Works 7
  9. 9. TEMPLATE MATCHINGclick(                            ) 8
  10. 10. TEMPLATE MATCHINGclick(                            ) 8
  11. 11. SYSTEM DESIGN 9
  12. 12. DEMO 1:EMPTY TRASH CAN 2 3 1 10
  13. 13. REGION AND MATCH Regionmatch = region.find( or “text pattern”) 11
  14. 14. REGION• basic attributes: x, y, width, height• Match is a Region• Screen is also a Region• chained methods: Screen(0).find(W).click(X) 12
  15. 15. KEY METHODS OF REGION• find, findAll, exists, wait, waitVanish• Mouse Actions click, doubleClick, rightClick, hover, dragDrop mouseDown, mouseUp• Keyboard Actions type(text), paste(text) keyDown, keyUp 13
  16. 16. EX2: FACEBOOK STALKER 14
  17. 17. EX2: FACEBOOK STALKER 14
  18. 18. RESTRICT SEARCH REGION 15
  19. 19. But the Region constant is bad 16
  20. 20. APP CLASS• firefox = App.open(“Firefox”)• firefox.focus(), firefox.close()• firefox.window([n]) • returns the Region of Firefox’s n-th window 17
  21. 21. FIND IN CERTAIN APP 18
  22. 22. EX3: MUTE SPEAKERS 19
  23. 23. EX3: MUTE SPEAKERS 19
  24. 24. SPATIAL OPERATORS 20
  25. 25. SPATIAL OPERATORS 20
  26. 26. SPATIAL OPERATORS (2) 21
  27. 27. TEXT MATCHING AND RECOGNITION• click(“OK”), find(“Cancel”)• region.text() - read the text in a region 22
  28. 28. VISUAL-EVENT-DRIVEN PROGRAMMING• observes visual events in any Region • Region.onAppear(Pattern, handler) • Region.onVanish(Pattern, handler) • Region.onChange(Pattern, handler) 23
  29. 29. VISUAL-EVENT-DRIVEN PROGRAMMING 24
  30. 30. VISUAL-EVENT-DRIVEN PROGRAMMING 24
  31. 31. VISUAL DICTIONARY 25
  32. 32. GUITESTING 26
  33. 33. VISUAL ASSERTION• make sure the appearance of visual feedback 27
  34. 34. VISUAL ASSERTION• make sure the appearance of visual feedback 27
  35. 35. JUNIT INTEGRATION 28
  36. 36. SIKULI IN COMMAND LINEusage:Sikuli-­‐IDE  [-­‐-­‐args  <arguments>]  [-­‐h]  [-­‐r  <sikuli-­‐file>]  [-­‐s]  [-­‐t  <sikuli-­‐test-­‐case>]  -­‐-­‐args  <arguments>                          specify  the  arguments  passed  to  Jythons  sys.argv  -­‐h,-­‐-­‐help                                            print  this  help  message  -­‐r,-­‐-­‐run  <sikuli-­‐file>                  run  .sikuli  or  .skl  file  -­‐s,-­‐-­‐stderr                                        print  runtime  errors  to  stderr  instead  of  popping  up  a  message  box  -­‐t,-­‐-­‐test  <sikuli-­‐test-­‐case>      run  .sikuli  as  a  unit  test  case  with  junits  text  UI  runner 29
  37. 37. CONCLUSIONS 30
  38. 38. PLATFORM INDEPENDENCE• Works on ANY GUI that can be displayed on Windows/Linux/Mac • Virtual machines • Remote desktop • Mobile simulators: Android, iPhone • Web: Flash, HTML+Javascript 31
  39. 39. PROGRAM AGAINST UISikuli programs are written against the user interfaceinstead of an API UI: visible, familiar, always exists API: faster, probably more stable 32
  40. 40. FUTURE WORK• fast and accurate OCR on screen• Accessibility API integration 33
  41. 41. Thanks! http://sikuli.org or, just google “sikuli” New Sikuli X 1.0-rc2 released. Follow us on twitter @sikuliThanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.) 34

×