What you’ll learn today
Visual approach to search andautomation of graphical user interfaces         using screenshots.
Just to see if it can be done     Usually needed to automate GUIs:       support from developers       API access      ...
Demo
Template Matching for small patterns    Invariance      Resized versions      Texturally similar, different color pallets
   Learn     Extract model from training pattern        ▪ Invariant to scale & rotation     Encode in the model the rel...
find(   )
f = Finder(path-to-imagefile)f.find(path-to-imagefile, [similarity])// iterates through Match objectswhile(f.hasNext):  pr...
Pattern(string)Pattern(string).similar(0.9).targetOffset(10,30)     Abstraction for visual patterns     Are used by find...
Region(x, y, w, h)Region(region)Region(Rectangle)    Search in a given region    Observe a region in background for chan...
click(   )
click(PSMLR),doubleClick(PSMLR)dragDrop(PSMLRtarget, PSMLR destination)
type(PSMLR, text)text()   Uses the Tesseract OCR engine
mouseDown(button), mouseUp(button)keyDown(keys), keyUp(keys)   They can be used along with KeyModifiers
wait(PS), waitVanish(PS)
onAppear(PS, handler)onVanish(PS, handler)onChange([minChengedSize], handler)
How performant do you want it to be?    A typical call to find() for a 100x100 target on     a 1600x1200 screen takes les...
Why not give it a try and make your own?    Record-playback    Sikuli Guide    …
Ok, the moment of truth!    Screenshots – unstable interfaces    Visibility constraints A paradigm shift requires a thin...
DemoWhat we’ve made of it
Automation with Sikuli
Automation with Sikuli
Upcoming SlideShare
Loading in...5
×

Automation with Sikuli

3,129

Published on

What is Sikuli? How it works? How to use it?

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,129
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
128
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Automation with Sikuli

  1. 1. What you’ll learn today
  2. 2. Visual approach to search andautomation of graphical user interfaces using screenshots.
  3. 3. Just to see if it can be done  Usually needed to automate GUIs:  support from developers  API access  language/OS dependency  position/naming dependencies
  4. 4. Demo
  5. 5. Template Matching for small patterns  Invariance  Resized versions  Texturally similar, different color pallets
  6. 6.  Learn  Extract model from training pattern ▪ Invariant to scale & rotation  Encode in the model the relative position of the center Search  Extract invariant features  Infer possible model center position  Cluster consistent features  Validate supposition
  7. 7. find( )
  8. 8. f = Finder(path-to-imagefile)f.find(path-to-imagefile, [similarity])// iterates through Match objectswhile(f.hasNext): print “found match: “ + f.next().getScore()
  9. 9. Pattern(string)Pattern(string).similar(0.9).targetOffset(10,30)  Abstraction for visual patterns  Are used by finding operations  Methods can be chained to refine the pattern  Can define click points
  10. 10. Region(x, y, w, h)Region(region)Region(Rectangle)  Search in a given region  Observe a region in background for changes  Retrieve matches  Optimize the search by chaining regions  No need to be aware of the content
  11. 11. click( )
  12. 12. click(PSMLR),doubleClick(PSMLR)dragDrop(PSMLRtarget, PSMLR destination)
  13. 13. type(PSMLR, text)text() Uses the Tesseract OCR engine
  14. 14. mouseDown(button), mouseUp(button)keyDown(keys), keyUp(keys) They can be used along with KeyModifiers
  15. 15. wait(PS), waitVanish(PS)
  16. 16. onAppear(PS, handler)onVanish(PS, handler)onChange([minChengedSize], handler)
  17. 17. How performant do you want it to be?  A typical call to find() for a 100x100 target on a 1600x1200 screen takes less than 200 msec  Improve your searches
  18. 18. Why not give it a try and make your own?  Record-playback  Sikuli Guide  …
  19. 19. Ok, the moment of truth!  Screenshots – unstable interfaces  Visibility constraints A paradigm shift requires a thinking shift
  20. 20. DemoWhat we’ve made of it
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×