Advertisement
Advertisement

More Related Content

Viewers also liked(20)

Advertisement

Similar to Practical Sikuli: using screenshots for GUI automation and testing(20)

Advertisement

Practical Sikuli: using screenshots for GUI automation and testing

  1. PRACTICAL SIKULI USING SCREENSHOTS FOR GUI AUTOMATION AND TESTING Tsung-Hsiang (Sean) Chang MIT CSAIL Collaborators: Tom Yeh, Rob Miller 1
  2. WHAT IS SIKULI? 2
  3. WHAT IS SIKULI? 2
  4. 2 3 1 3
  5. Low-level API? Mac’s system design? 4
  6. EMPTY THE TRASH CAN 5
  7. OUTLINE • What Sikuli is • How Sikuli works • Sikuli for GUI automation • Sikuli for GUI testing • Conclusions 6
  8. How Sikuli Works 7
  9. TEMPLATE MATCHING click(                            ) 8
  10. TEMPLATE MATCHING click(                            ) 8
  11. SYSTEM DESIGN 9
  12. DEMO 1: EMPTY TRASH CAN 2 3 1 10
  13. REGION AND MATCH Region match = region.find( or “text pattern”) 11
  14. REGION • basic attributes: x, y, width, height • Match is a Region • Screen is also a Region • chained methods: Screen(0).find(W).click(X) 12
  15. KEY METHODS OF REGION • find, findAll, exists, wait, waitVanish • Mouse Actions click, doubleClick, rightClick, hover, dragDrop mouseDown, mouseUp • Keyboard Actions type(text), paste(text) keyDown, keyUp 13
  16. EX2: FACEBOOK STALKER 14
  17. EX2: FACEBOOK STALKER 14
  18. RESTRICT SEARCH REGION 15
  19. But the Region constant is bad 16
  20. APP CLASS • firefox = App.open(“Firefox”) • firefox.focus(), firefox.close() • firefox.window([n]) • returns the Region of Firefox’s n-th window 17
  21. FIND IN CERTAIN APP 18
  22. EX3: MUTE SPEAKERS 19
  23. EX3: MUTE SPEAKERS 19
  24. SPATIAL OPERATORS 20
  25. SPATIAL OPERATORS 20
  26. SPATIAL OPERATORS (2) 21
  27. TEXT MATCHING AND RECOGNITION • click(“OK”), find(“Cancel”) • region.text() - read the text in a region 22
  28. VISUAL-EVENT-DRIVEN PROGRAMMING • observes visual events in any Region • Region.onAppear(Pattern, handler) • Region.onVanish(Pattern, handler) • Region.onChange(Pattern, handler) 23
  29. VISUAL-EVENT-DRIVEN PROGRAMMING 24
  30. VISUAL-EVENT-DRIVEN PROGRAMMING 24
  31. VISUAL DICTIONARY 25
  32. GUI TESTING 26
  33. VISUAL ASSERTION • make sure the appearance of visual feedback 27
  34. VISUAL ASSERTION • make sure the appearance of visual feedback 27
  35. JUNIT INTEGRATION 28
  36. SIKULI IN COMMAND LINE usage: Sikuli-­‐IDE  [-­‐-­‐args  <arguments>]  [-­‐h]  [-­‐r  <sikuli-­‐file>]  [-­‐s]  [-­‐t  <sikuli-­‐test-­‐case>]  -­‐-­‐args  <arguments>                          specify  the  arguments  passed  to  Jython's  sys.argv  -­‐h,-­‐-­‐help                                            print  this  help  message  -­‐r,-­‐-­‐run  <sikuli-­‐file>                  run  .sikuli  or  .skl  file  -­‐s,-­‐-­‐stderr                                        print  runtime  errors  to  stderr  instead  of  popping  up  a  message  box  -­‐t,-­‐-­‐test  <sikuli-­‐test-­‐case>      run  .sikuli  as  a  unit  test  case  with  junit's  text  UI  runner 29
  37. CONCLUSIONS 30
  38. PLATFORM INDEPENDENCE • Works on ANY GUI that can be displayed on Windows/Linux/Mac • Virtual machines • Remote desktop • Mobile simulators: Android, iPhone • Web: Flash, HTML+Javascript 31
  39. PROGRAM AGAINST UI Sikuli programs are written against the user interface instead of an API UI: visible, familiar, always exists API: faster, probably more stable 32
  40. FUTURE WORK • fast and accurate OCR on screen • Accessibility API integration 33
  41. Thanks! http://sikuli.org or, just google “sikuli” New Sikuli X 1.0-rc2 released. Follow us on twitter @sikuli Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.) 34
Advertisement