Successfully reported this slideshow.

Practical Sikuli: using screenshots for GUI automation and testing

59

Share

Upcoming SlideShare
Sikuli Slides
Sikuli Slides
Loading in …3
×
1 of 41
1 of 41

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Practical Sikuli: using screenshots for GUI automation and testing

  1. 1. PRACTICAL SIKULI USING SCREENSHOTS FOR GUI AUTOMATION AND TESTING Tsung-Hsiang (Sean) Chang MIT CSAIL Collaborators: Tom Yeh, Rob Miller 1
  2. 2. WHAT IS SIKULI? 2
  3. 3. WHAT IS SIKULI? 2
  4. 4. 2 3 1 3
  5. 5. Low-level API? Mac’s system design? 4
  6. 6. EMPTY THE TRASH CAN 5
  7. 7. OUTLINE • What Sikuli is • How Sikuli works • Sikuli for GUI automation • Sikuli for GUI testing • Conclusions 6
  8. 8. How Sikuli Works 7
  9. 9. TEMPLATE MATCHING click(                            ) 8
  10. 10. TEMPLATE MATCHING click(                            ) 8
  11. 11. SYSTEM DESIGN 9
  12. 12. DEMO 1: EMPTY TRASH CAN 2 3 1 10
  13. 13. REGION AND MATCH Region match = region.find( or “text pattern”) 11
  14. 14. REGION • basic attributes: x, y, width, height • Match is a Region • Screen is also a Region • chained methods: Screen(0).find(W).click(X) 12
  15. 15. KEY METHODS OF REGION • find, findAll, exists, wait, waitVanish • Mouse Actions click, doubleClick, rightClick, hover, dragDrop mouseDown, mouseUp • Keyboard Actions type(text), paste(text) keyDown, keyUp 13
  16. 16. EX2: FACEBOOK STALKER 14
  17. 17. EX2: FACEBOOK STALKER 14
  18. 18. RESTRICT SEARCH REGION 15
  19. 19. But the Region constant is bad 16
  20. 20. APP CLASS • firefox = App.open(“Firefox”) • firefox.focus(), firefox.close() • firefox.window([n]) • returns the Region of Firefox’s n-th window 17
  21. 21. FIND IN CERTAIN APP 18
  22. 22. EX3: MUTE SPEAKERS 19
  23. 23. EX3: MUTE SPEAKERS 19
  24. 24. SPATIAL OPERATORS 20
  25. 25. SPATIAL OPERATORS 20
  26. 26. SPATIAL OPERATORS (2) 21
  27. 27. TEXT MATCHING AND RECOGNITION • click(“OK”), find(“Cancel”) • region.text() - read the text in a region 22
  28. 28. VISUAL-EVENT-DRIVEN PROGRAMMING • observes visual events in any Region • Region.onAppear(Pattern, handler) • Region.onVanish(Pattern, handler) • Region.onChange(Pattern, handler) 23
  29. 29. VISUAL-EVENT-DRIVEN PROGRAMMING 24
  30. 30. VISUAL-EVENT-DRIVEN PROGRAMMING 24
  31. 31. VISUAL DICTIONARY 25
  32. 32. GUI TESTING 26
  33. 33. VISUAL ASSERTION • make sure the appearance of visual feedback 27
  34. 34. VISUAL ASSERTION • make sure the appearance of visual feedback 27
  35. 35. JUNIT INTEGRATION 28
  36. 36. SIKULI IN COMMAND LINE usage: Sikuli-­‐IDE  [-­‐-­‐args  <arguments>]  [-­‐h]  [-­‐r  <sikuli-­‐file>]  [-­‐s]  [-­‐t  <sikuli-­‐test-­‐case>]  -­‐-­‐args  <arguments>                          specify  the  arguments  passed  to  Jython's  sys.argv  -­‐h,-­‐-­‐help                                            print  this  help  message  -­‐r,-­‐-­‐run  <sikuli-­‐file>                  run  .sikuli  or  .skl  file  -­‐s,-­‐-­‐stderr                                        print  runtime  errors  to  stderr  instead  of  popping  up  a  message  box  -­‐t,-­‐-­‐test  <sikuli-­‐test-­‐case>      run  .sikuli  as  a  unit  test  case  with  junit's  text  UI  runner 29
  37. 37. CONCLUSIONS 30
  38. 38. PLATFORM INDEPENDENCE • Works on ANY GUI that can be displayed on Windows/Linux/Mac • Virtual machines • Remote desktop • Mobile simulators: Android, iPhone • Web: Flash, HTML+Javascript 31
  39. 39. PROGRAM AGAINST UI Sikuli programs are written against the user interface instead of an API UI: visible, familiar, always exists API: faster, probably more stable 32
  40. 40. FUTURE WORK • fast and accurate OCR on screen • Accessibility API integration 33
  41. 41. Thanks! http://sikuli.org or, just google “sikuli” New Sikuli X 1.0-rc2 released. Follow us on twitter @sikuli Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.) 34

×