Sikuli: Using Screenshots for GUI Automation and Testing

  • 11,901 views
Uploaded on

 

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
11,901
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
475
Comments
0
Likes
9

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. SIKULI USING SCREENSHOTS FOR GUI AUTOMATION AND TESTING Tsung-Hsiang (Sean) Chang, Rob Miller MIT CSAIL Collaborator: Tom Yeh 1
  • 2. How to tell computers what to do? 2
  • 3. case $rtval in 1) rm -f /tmp/dirip.$$ ; return ;; 255) rm -f /tmp/dirip.$$ ; return ;; esac mfile=`cat /tmp/dirip.$$` if [ -z $mfile ] then mfile=`pwd`/* SHELL SCRIPT else grep "*" /tmp/dirip.$$ if [ $? -eq 1 ] then ! mfile=$mfile/* fi fi for i in $mfile do if [ -f $i ] then ! echo "$i Delete?" >> /tmp/finallist.$$ fi! done 3
  • 4. 4
  • 5. GUI AUTOMATION 5
  • 6. 2 3 1 6
  • 7. 7
  • 8. click click 7
  • 9. SIKULI SCRIPT 8
  • 10. SIKULI TEST 9
  • 11. OUTLINE • Problem - How to refer to a GUI widget? • Sikuli Script - examples and demos • Sikuli Test - demo • Test By Demonstration • Support of testing practices • Conclusions 10
  • 12. HOW TO REFER TO A GUI WIDGET? 11
  • 13. HOW TO REFER TO A GUI WIDGET? <AWTTestScript> Abbot <launch classpath="src/demo" args="[]" class="CelsiusConverter" method="main" /> <action args="45" method="actionKeyString"/> <action class="javax.swing.AbstractButton" args="Convert..." method="actionClick"/> <assert value="113 Fahrenheit" component="113 Fahrenheit" method="getText"/> by pre-programmed references <terminate/> </AWTTestScript> 11
  • 14. HOW TO REFER TO A GUI WIDGET? <AWTTestScript> Abbot <launch classpath="src/demo" args="[]" Autoit class="CelsiusConverter" method="main" /> <action args="45" method="actionKeyString"/> <action class="javax.swing.AbstractButton" args="Convert..." method="actionClick"/> <assert value="113 Fahrenheit" component="113 Fahrenheit" method="getText"/> by pre-programmed references <terminate/> by absolute locations </AWTTestScript> 11
  • 15. Use Screenshots! 12
  • 16. TEMPLATE MATCHING click(                            ) 13
  • 17. TEMPLATE MATCHING click(                            ) 13
  • 18. EXAMPLE 1: EMPTY TRASH CAN 2 3 1 14
  • 19. EX2: FACEBOOK STALKER 15
  • 20. EX2: FACEBOOK STALKER 15
  • 21. 16
  • 22. RESTRICT SEARCH REGION 17
  • 23. EX3: MUTE SPEAKERS 18
  • 24. EX3: MUTE SPEAKERS 18
  • 25. SPATIAL OPERATORS 19
  • 26. SPATIAL OPERATORS 19
  • 27. SPATIAL OPERATORS (2) 20
  • 28. EX4: SET IP ADDRESS 21
  • 29. VISUAL-EVENT-DRIVEN PROGRAMMING 22
  • 30. VISUAL-EVENT-DRIVEN PROGRAMMING 22
  • 31. SIKULI TEST 23
  • 32. VISUAL ASSERTION • Verify appearance 24
  • 33. VISUAL ASSERTION • Verify replacement 25
  • 34. TEST BY DEMONSTRATION detect changes click 26
  • 35. Demo 27
  • 36. GENERATE ACTION COMMANDS mouse mouse ‘A’ ‘A’ ‘b’ ‘b’ down up down up down up 28
  • 37. GENERATE ACTION COMMANDS low-level events mouse mouse ‘A’ ‘A’ ‘b’ ‘b’ down up down up down up 28
  • 38. GENERATE ACTION COMMANDS low-level events mouse mouse ‘A’ ‘A’ ‘b’ ‘b’ down up down up down up high-level commands click type “Ab” 28
  • 39. GENERATE ACTION COMMANDS low-level events mouse mouse down up ‘A’ ‘A’ ‘b’ ‘b’ click down up down up high-level commands click type “Ab” combine with screenshots 28
  • 40. GENERATE ASSERTIONS click 29
  • 41. GENERATE ASSERTIONS click 1. find differences / manually hint 29
  • 42. GENERATE ASSERTIONS click 1. find differences / manually hint 2. group together 29
  • 43. GENERATE ASSERTIONS click 1. find differences / manually hint 2. group together 3. combine with assertExist 29
  • 44. GENERATE ASSERTIONS click 1. find differences / manually hint 2. group together 3. combine with assertExist 29
  • 45. jUNIT INTEGRATION 30
  • 46. TEST-DRIVEN DEVELOPMENT 31
  • 47. READABILITY OF CODE 32
  • 48. PLATFORM INDEPENDENCE • Works on ANY GUI that can be displayed on Windows/Linux/Mac • Virtual machines • Remote desktop • Mobile simulators: Android, iPhone • Web: Flash, HTML+Javascript 33
  • 49. 34
  • 50. 34
  • 51. LIMITATIONS Theme Variations Visibility Constraints 35
  • 52. DISCUSSION Sikuli programs are written against the user interface instead of an API UI: visible, familiar, always exists API: faster, probably more stable 36
  • 53. FUTURE WORK • fast and accurate OCR on screen • more robust against different skins 37
  • 54. Thanks! http://sikuli.csail.mit.edu or, just google “sikuli” New Sikuli 0.10 released. Follow us on twitter @sikuli Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.) 38