SIKULI
USING SCREENSHOTS FOR GUI
 AUTOMATION AND TESTING
 Tsung-Hsiang (Sean) Chang, Rob Miller
              MIT CSAIL


...
How to
tell computers what to do?



            2
case $rtval in
                         1) rm -f /tmp/dirip.$$ ; return ;;
                         255) rm -f /tmp/dirip....
4
GUI AUTOMATION




      5
2

    3


            1

6
7
click   click




           7
SIKULI SCRIPT
      8
SIKULI TEST
     9
OUTLINE
• Problem     - How to refer to a GUI widget?

• Sikuli   Script - examples and demos

• Sikuli Test   - demo

  •...
HOW TO REFER TO A GUI WIDGET?




              11
HOW TO REFER TO A GUI WIDGET?


<AWTTestScript>                      Abbot
 <launch classpath="src/demo" args="[]"
   clas...
HOW TO REFER TO A GUI WIDGET?


<AWTTestScript>                      Abbot
 <launch classpath="src/demo" args="[]"
       ...
Use Screenshots!



       12
TEMPLATE MATCHING

click(	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  )




                                   ...
TEMPLATE MATCHING

click(	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  )




                                   ...
EXAMPLE 1:
EMPTY TRASH CAN
                  2

            3


                      1
       14
EX2: FACEBOOK STALKER




          15
EX2: FACEBOOK STALKER




          15
16
RESTRICT SEARCH REGION




          17
EX3: MUTE SPEAKERS




        18
EX3: MUTE SPEAKERS




        18
SPATIAL OPERATORS




        19
SPATIAL OPERATORS




        19
SPATIAL OPERATORS (2)




          20
EX4: SET IP ADDRESS




         21
VISUAL-EVENT-DRIVEN
   PROGRAMMING




         22
VISUAL-EVENT-DRIVEN
   PROGRAMMING




         22
SIKULI TEST



     23
VISUAL ASSERTION
• Verify   appearance




                        24
VISUAL ASSERTION
• Verify   replacement




                         25
TEST BY DEMONSTRATION
               detect changes

click




          26
Demo



 27
GENERATE ACTION
                COMMANDS


mouse mouse
               ‘A’ ‘A’ ‘b’ ‘b’
down   up
              down up down...
GENERATE ACTION
                COMMANDS

                                      low-level events

mouse mouse
            ...
GENERATE ACTION
                COMMANDS

                                      low-level events

mouse mouse
            ...
GENERATE ACTION
                COMMANDS

                                              low-level events

mouse mouse
down...
GENERATE ASSERTIONS



              click




         29
GENERATE ASSERTIONS



                                          click

1. find differences / manually hint




           ...
GENERATE ASSERTIONS



                                          click

1. find differences / manually hint
2. group togeth...
GENERATE ASSERTIONS



                                          click

1. find differences / manually hint
2. group togeth...
GENERATE ASSERTIONS



                                          click

1. find differences / manually hint
2. group togeth...
jUNIT INTEGRATION




        30
TEST-DRIVEN DEVELOPMENT




           31
READABILITY OF CODE




         32
PLATFORM INDEPENDENCE

• Works
      on ANY GUI that can be displayed on
 Windows/Linux/Mac

 • Virtual   machines

 • Rem...
34
34
LIMITATIONS
Theme Variations



                       Visibility Constraints




                        35
DISCUSSION
Sikuli programs are written against the user interface
instead of an API




	

   UI: visible, familiar, alway...
FUTURE WORK


• fast   and accurate OCR on screen

• more    robust against different skins




                          ...
Thanks!


     http://sikuli.csail.mit.edu

      or, just google “sikuli”

                                            Ne...
Upcoming SlideShare
Loading in …5
×

Sikuli: Using Screenshots for GUI Automation and Testing

14,057 views

Published on

Published in: Technology, Education
0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
14,057
On SlideShare
0
From Embeds
0
Number of Embeds
31
Actions
Shares
0
Downloads
555
Comments
0
Likes
11
Embeds 0
No embeds

No notes for slide

Sikuli: Using Screenshots for GUI Automation and Testing

  1. 1. SIKULI USING SCREENSHOTS FOR GUI AUTOMATION AND TESTING Tsung-Hsiang (Sean) Chang, Rob Miller MIT CSAIL Collaborator: Tom Yeh 1
  2. 2. How to tell computers what to do? 2
  3. 3. case $rtval in 1) rm -f /tmp/dirip.$$ ; return ;; 255) rm -f /tmp/dirip.$$ ; return ;; esac mfile=`cat /tmp/dirip.$$` if [ -z $mfile ] then mfile=`pwd`/* SHELL SCRIPT else grep "*" /tmp/dirip.$$ if [ $? -eq 1 ] then ! mfile=$mfile/* fi fi for i in $mfile do if [ -f $i ] then ! echo "$i Delete?" >> /tmp/finallist.$$ fi! done 3
  4. 4. 4
  5. 5. GUI AUTOMATION 5
  6. 6. 2 3 1 6
  7. 7. 7
  8. 8. click click 7
  9. 9. SIKULI SCRIPT 8
  10. 10. SIKULI TEST 9
  11. 11. OUTLINE • Problem - How to refer to a GUI widget? • Sikuli Script - examples and demos • Sikuli Test - demo • Test By Demonstration • Support of testing practices • Conclusions 10
  12. 12. HOW TO REFER TO A GUI WIDGET? 11
  13. 13. HOW TO REFER TO A GUI WIDGET? <AWTTestScript> Abbot <launch classpath="src/demo" args="[]" class="CelsiusConverter" method="main" /> <action args="45" method="actionKeyString"/> <action class="javax.swing.AbstractButton" args="Convert..." method="actionClick"/> <assert value="113 Fahrenheit" component="113 Fahrenheit" method="getText"/> by pre-programmed references <terminate/> </AWTTestScript> 11
  14. 14. HOW TO REFER TO A GUI WIDGET? <AWTTestScript> Abbot <launch classpath="src/demo" args="[]" Autoit class="CelsiusConverter" method="main" /> <action args="45" method="actionKeyString"/> <action class="javax.swing.AbstractButton" args="Convert..." method="actionClick"/> <assert value="113 Fahrenheit" component="113 Fahrenheit" method="getText"/> by pre-programmed references <terminate/> by absolute locations </AWTTestScript> 11
  15. 15. Use Screenshots! 12
  16. 16. TEMPLATE MATCHING click(                            ) 13
  17. 17. TEMPLATE MATCHING click(                            ) 13
  18. 18. EXAMPLE 1: EMPTY TRASH CAN 2 3 1 14
  19. 19. EX2: FACEBOOK STALKER 15
  20. 20. EX2: FACEBOOK STALKER 15
  21. 21. 16
  22. 22. RESTRICT SEARCH REGION 17
  23. 23. EX3: MUTE SPEAKERS 18
  24. 24. EX3: MUTE SPEAKERS 18
  25. 25. SPATIAL OPERATORS 19
  26. 26. SPATIAL OPERATORS 19
  27. 27. SPATIAL OPERATORS (2) 20
  28. 28. EX4: SET IP ADDRESS 21
  29. 29. VISUAL-EVENT-DRIVEN PROGRAMMING 22
  30. 30. VISUAL-EVENT-DRIVEN PROGRAMMING 22
  31. 31. SIKULI TEST 23
  32. 32. VISUAL ASSERTION • Verify appearance 24
  33. 33. VISUAL ASSERTION • Verify replacement 25
  34. 34. TEST BY DEMONSTRATION detect changes click 26
  35. 35. Demo 27
  36. 36. GENERATE ACTION COMMANDS mouse mouse ‘A’ ‘A’ ‘b’ ‘b’ down up down up down up 28
  37. 37. GENERATE ACTION COMMANDS low-level events mouse mouse ‘A’ ‘A’ ‘b’ ‘b’ down up down up down up 28
  38. 38. GENERATE ACTION COMMANDS low-level events mouse mouse ‘A’ ‘A’ ‘b’ ‘b’ down up down up down up high-level commands click type “Ab” 28
  39. 39. GENERATE ACTION COMMANDS low-level events mouse mouse down up ‘A’ ‘A’ ‘b’ ‘b’ click down up down up high-level commands click type “Ab” combine with screenshots 28
  40. 40. GENERATE ASSERTIONS click 29
  41. 41. GENERATE ASSERTIONS click 1. find differences / manually hint 29
  42. 42. GENERATE ASSERTIONS click 1. find differences / manually hint 2. group together 29
  43. 43. GENERATE ASSERTIONS click 1. find differences / manually hint 2. group together 3. combine with assertExist 29
  44. 44. GENERATE ASSERTIONS click 1. find differences / manually hint 2. group together 3. combine with assertExist 29
  45. 45. jUNIT INTEGRATION 30
  46. 46. TEST-DRIVEN DEVELOPMENT 31
  47. 47. READABILITY OF CODE 32
  48. 48. PLATFORM INDEPENDENCE • Works on ANY GUI that can be displayed on Windows/Linux/Mac • Virtual machines • Remote desktop • Mobile simulators: Android, iPhone • Web: Flash, HTML+Javascript 33
  49. 49. 34
  50. 50. 34
  51. 51. LIMITATIONS Theme Variations Visibility Constraints 35
  52. 52. DISCUSSION Sikuli programs are written against the user interface instead of an API UI: visible, familiar, always exists API: faster, probably more stable 36
  53. 53. FUTURE WORK • fast and accurate OCR on screen • more robust against different skins 37
  54. 54. Thanks! http://sikuli.csail.mit.edu or, just google “sikuli” New Sikuli 0.10 released. Follow us on twitter @sikuli Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.) 38

×