Automation with Sikuli

4,439 views

Published on

What is Sikuli? How it works? How to use it?

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,439
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
163
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Automation with Sikuli

  1. 1. What you’ll learn today
  2. 2. Visual approach to search andautomation of graphical user interfaces using screenshots.
  3. 3. Just to see if it can be done  Usually needed to automate GUIs:  support from developers  API access  language/OS dependency  position/naming dependencies
  4. 4. Demo
  5. 5. Template Matching for small patterns  Invariance  Resized versions  Texturally similar, different color pallets
  6. 6.  Learn  Extract model from training pattern ▪ Invariant to scale & rotation  Encode in the model the relative position of the center Search  Extract invariant features  Infer possible model center position  Cluster consistent features  Validate supposition
  7. 7. find( )
  8. 8. f = Finder(path-to-imagefile)f.find(path-to-imagefile, [similarity])// iterates through Match objectswhile(f.hasNext): print “found match: “ + f.next().getScore()
  9. 9. Pattern(string)Pattern(string).similar(0.9).targetOffset(10,30)  Abstraction for visual patterns  Are used by finding operations  Methods can be chained to refine the pattern  Can define click points
  10. 10. Region(x, y, w, h)Region(region)Region(Rectangle)  Search in a given region  Observe a region in background for changes  Retrieve matches  Optimize the search by chaining regions  No need to be aware of the content
  11. 11. click( )
  12. 12. click(PSMLR),doubleClick(PSMLR)dragDrop(PSMLRtarget, PSMLR destination)
  13. 13. type(PSMLR, text)text() Uses the Tesseract OCR engine
  14. 14. mouseDown(button), mouseUp(button)keyDown(keys), keyUp(keys) They can be used along with KeyModifiers
  15. 15. wait(PS), waitVanish(PS)
  16. 16. onAppear(PS, handler)onVanish(PS, handler)onChange([minChengedSize], handler)
  17. 17. How performant do you want it to be?  A typical call to find() for a 100x100 target on a 1600x1200 screen takes less than 200 msec  Improve your searches
  18. 18. Why not give it a try and make your own?  Record-playback  Sikuli Guide  …
  19. 19. Ok, the moment of truth!  Screenshots – unstable interfaces  Visibility constraints A paradigm shift requires a thinking shift
  20. 20. DemoWhat we’ve made of it

×