Sikuli

15,642 views
15,405 views

Published on

This Content is directly copied from http://sikuli.org/ site.

This is not an updated document.

All the credit goes to my friends
1. Sundar Ram Raju
2. Shiva Prasad Raju
for knowing me about this tool

Thanks to my manager Srinivasa Movva for encouraging me to learn this tool.

© Copyright 2009-2010 Sikuli.org • All Rights Reserved with Sikuli org

Published in: Technology
2 Comments
14 Likes
Statistics
Notes
No Downloads
Views
Total views
15,642
On SlideShare
0
From Embeds
0
Number of Embeds
24
Actions
Shares
0
Downloads
889
Comments
2
Likes
14
Embeds 0
No embeds

No notes for slide

Sikuli

  1. 1. Sikuli Satish Gorripotu ome SIKULI SATISH GORRIPOTU Gsatish_leela@yahoo.com 9885121381
  2. 2. Sikuli Satish Gorripotu TOPICS AND STATUS Topic Comments Page NumberSikuli DocumentedSikuli and Other Supporting Softwares Required DocumentedInstalling/Configuring JDK or JRE / Setting Class Path DocumentedInstalling Sikuli DocumentedStarting Sikuli IDE DocumentedStop the script during run time: DocumentedSikuli Working Architecture: DocumentedCheck Box: DocumenteddragDrop: DocumentedSliders DocumentedTesting on Skype Application: DocumentedMeeting Request Arrival Notifier: DocumentedImages Dictionary/Repository: DocumentedOpen any application in run time. DocumentedEmpty Recycle Bin: DocumentedRun Sikuli from Command Line: DocumentedCreate Unit Testing Scripts for GUI: DocumentedSikuli Script in JAVA programs: DocumentedSikuli together with other IDE‘s - Eclipse: DocumentedImport Packages/ Files DocumentedControlling Sikuli Scripts and their Behavior: DocumentedSetting DocumentedControlling Applications and their Windows: DocumentedApp DocumentedInteracting with the User DocumentedListening to Global Hotkeys DocumentedGeneral Settings and Access to Environment Information DocumentedRegion DocumentedExtending a Region DocumentedFinding inside a Region and Waiting for a Visual Event DocumentedObserving Visual Events in a Region DocumentedSikuliEvent DocumentedLow-level Mouse and Keyboard Actions DocumentedException FindFailed DocumentedScreen DocumentedMatch DocumentedPattern DocumentedFinder DocumentedKey DocumentedConstants related to Mouse and Keyboard Actions Documented
  3. 3. Sikuli Satish GorripotuExtensions DocumentedSummary DocumentedJava Docs DocumentedJython Basics DocumentedReferences DocumentedExcel Handling in Sikuli Yet to CompleteObject Repository Yet to CompleteHandling Synchronization Problems Yet to CompleteMethods that are mostly used Yet to CompleteMaintaining the Test Results Yet to CompleteFrameworks Yet to CompleteDatabase Testing using Sikuli Yet to CompleteSending the test results to email by using Sikuli Yet to CompleteChallenges faced Yet to Complete
  4. 4. Sikuli Satish Gorripotu Sikuli Sikuli is an open-source research project. This is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots). It includes 1. Sikuli Script, 2. Visual scripting API for Jython 3. Sikuli IDE. Sikuli Script automates anything you see on the screen without internal APIs support. We can programmatically control a web page, a Windows/Linux/Mac OS X desktop application, or even an iphone or android application running in a simulator or via VNC. Download and unzip the Sikuli Using following link http://sikuli.org/download.shtml Sikuli and Other Supporting Softwares Required 1. Sikuli-X-1.0rc2-win32.exe 2. Java Runtime Environment (JRE) 6+ Make sure to have only one instance of each software is available on machine. You are provided with Admin access to configure the paths in the Environmental variables. Installing/Configuring JDK or JRE / Setting Class Path Install the JDK package and update the JAVA_HOME environment variable (User) in the Advanced tab. Of System Properties. In system variable edit PATH by adding C:Program FilesJavajdk1.6.0_14bin; I . Right click on my computer->Properties->Advanced->Environment variables II . 1. Click on "New" in system variables 2. Create a new variable as shown below Variable name = JAVA_HOME Variable value = C:Program FilesJavajdk1.6.0_14 // [path of jdk is installed] III. Select the existing path variable and edit it as shown below PATH = % JAVA_HOME%bin; % JAVA_HOME%lib; IV. Click on OK
  5. 5. Sikuli Satish Gorripotu V . START-> RUN ->type CMD VI. Now enter "javac" to check whether the java correctly installed or not.
  6. 6. Sikuli Satish Gorripotu Installing Sikuli Sikuli-X-1.0rc2-win32.exe is self extracting file. Just double click and click next … next … until finish whilst follow the instructions. 1. It will automatically install the software. 2. It will configure the environment variables. 3. In desk top or program files we can see the short cut for Sikuli IDE as Starting Sikuli IDE Double click on the Sikuli IDE will give the below IDE There are two ways to capture the screen images 1. Click on in the IDE 2. Press a hot-key (Ctrl + Shift + 2)
  7. 7. Sikuli Satish Gorripotu Screen will look darker and freeze momentarily. The entire desktop becomes like a canvas where we can draw a rectangle around the target to capture an image. Now that image will be captured and inserted in IDE.We can use Auto List command capture to enable the basic commands in the IDEinstead of typing the commands manually.1. Find a. find b. find all c. wait d. waitvanish e. exists2. Mouse Actions a. Click b. doubleClick c. rightClick d. hover e. dragDrop3. Keyboard Actions a. type b. paste4. Event Observation a. onAppear b. OnVanish c. onChange d. observeBelow script will open the Keane web site ―www.keane.com‖ and search with the keyword ―Testing‖.
  8. 8. Sikuli Satish Gorripotu Type: type ("http://sikuli.org" + Key.ENTER) miscellanous keys ENTER - TAB - ESC - BACKSPACE - DELETE - INSERT function keys F1 - F2 - F3 - F4 - F5 - F6 - F7 - F8 - F9 - F10 - F11 - F12 - F13 - F14 - F15 navigation keys HOME - END - LEFT - RIGHT - DOWN - UP - PAGE_DOWN - PAGE_UP key modifiers ALT - CMD - CTRL - META - SHIFT - WIN
  9. 9. Sikuli Satish Gorripotu Stop the script during run time: Press alt-shift-c
  10. 10. Sikuli Satish GorripotuSikuli Working Architecture:Sikuli ScriptSikuli Script is a Jython and Java library that automates GUI interaction using image patterns to directkeyboard/mouse events. The core of Sikuli Script is a Java library that consists of two parts:java.awt.Robot, which delivers keyboard and mouse events to appropriate locations, and a C++ enginebased on OpenCV, which searches given image patterns on the screen. The C++ engine is connected toJava via JNI and needs to be compiled for each platform.On top of the Java library, a thin Jython layer is provided for end-users as a set of simple and clearcommands. Therefore, it should be easy to add more thin layers for other languages running on JVM, e.g.JRuby, Scala, Javascript, etc.The Structure of a Sikuli source/executable script (.sikuli, .skl)A Sikuli script (.sikuli) is a directory that consists of a Python source file (.py), and all the image files(.png) used by the source file.All images used in a Sikuli script are simply a path to the .png file in the .sikuli bundle. Therefore, thePython source file can also be edited by any text editor.While saving a script using Sikuli IDE, an extra HTML file is also created in the .sikuli directory so thatusers can share the scripts on the web easily.A Sikuli executable script (.skl) is simply a zipped file of all files in the .sikuli directory. When a script ispassed to Sikuli IDE as a command line argument, Sikuli IDE recognizes its type by check its filenameextension. If a .skl is seen, Sikuli IDE runs it without showing the IDE window. If a .Sikuli is seen, SikuliIDE opens it in a source code editor.
  11. 11. Sikuli Satish GorripotuSikuli IDESikuli IDE edits and runs Sikuli source scripts. Sikuli IDE integrates screen capturing and a custom texteditor (SikuliPane) to optimize the usability of writing a Sikuli script.To show embedded images in the SikuliPane, all string literals that ends with ".png" are replaced by acustom JButton object, ImageButton. If a user adjusts the image patterns similarity, a Pattern() isautomatically constructed on top of the image.To execute a Sikuli script, Sikuli IDE creates a org.python.util.PythonInterpreter and automatically passesa few lines of headers (e.g. to import Sikulis Jython modules, and to set the path to .sikuli directory) tothe interpreter.Once these headers are set, the .py script is simply executed byPythonInterpreter.execfile().Working with Check Box:1. Check the all check boxes: findAll() : searches the screen for all the matching visual patterns and returns a list of locations. Then we can use click() function to check the all check box.2. Check the particular check box:
  12. 12. Sikuli Satish Gorripotu Step 1: we have to select the region Step 2: Then identify check box Below is the code for that.
  13. 13. Sikuli Satish GorripotudragDrop:The above script will drag and drop all the word files into specified folder.
  14. 14. Sikuli Satish GorripotuWorking with Sliders Same procedure as check boxes. But we can use dragDrop() function to move slide.Way 1:Way 2: Need to work with Coordinates. Moving to Null. Need to resolve by Ram Raju.Find Below Screen shot. There x, Y are the coordinates.Main Scenario is: Zoom in and Zoom Out of the the word document.
  15. 15. Sikuli Satish Gorripotuhover(find( ).nearby(200).right().find( ).below().find( ).getCenter().offset(3,18))
  16. 16. Sikuli Satish GorripotuTesting on Skype Application:Now status of the ―Family – Bujji‖ is . We have to wait up to status as .We can use any of the below way toWay 1: while not exists( ):
  17. 17. Sikuli Satish Gorripotu sleep(5)Way 2: wait( ,FOREVER)Way 3: wait( ,3600)#wait for an hourWay 4: while exists( ): sleep(5)Way 5: waitVanish( , FOREVER) popup(―Oyee Satti !!!!!! Your Wife Online‖)
  18. 18. Sikuli Satish GorripotuMeeting Request Arrival Notifier:Define a region and focus only on that region to search for a particular visual pattern. So tool don‘t have toscan the whole screen and we can resize the browser to show just that portion of the map, while leaving alot of screen space to do something else.To do so, click on the ―Create region‖ button.The entire screen will freeze and turn darker, similar to what happen in the screen capture mode. Simplydraw a rectangle to cover the entire neighborhood map. The region covered by this rectangle is visuallyrepresented as a thumbnail image of the entire desktop where the region is shaded in red.
  19. 19. Sikuli Satish GorripotuNext, we capture the screenshot image of the meeting symbol so that we can tell Sikuli Script to watch forits appearance.We can write the following script to check the mails and watch for the appearance of a meeting Requestsymbol:Way 1:Way 2:Images Dictionary/Repository: 1. Store Images as Predefined.
  20. 20. Sikuli Satish Gorripotu Filename = VDict() Filename [ ]= Recycle Filename [ ] = Skype 2. Store Images in runtime img = capture() Filename [img] = input("Please enter Name to Captured image") 3. Retrieve name of the Image on popup. Way 1: query = capture() gotit = Filename[query][0] popup(gotit) Way 2: popup("select a icon in the ") query = capture() Result = Filename [query] if result: popup(Result[0]) else: popup("unknown Image")Open any application in run time. setThrowException(True) setAutoWaitTimeout(10000) switchApp("C:WINDOWSsystem32calc.exe")
  21. 21. Sikuli Satish GorripotuEmpty Recycle Bin:Run Sikuli from Command Line: PATH-TO-SIKULI/sikuli-ide.exe [options] or PATH-TO-SIKULI/sikuli-ide.bat [options] Command Line Options Sikuli-IDE --args <arguments> specify the arguments passed to Jython‘s sys.argv -h,--help print this help message
  22. 22. Sikuli Satish Gorripotu usage: Sikuli-IDE [--args <arguments>] [-h] [-r <sikuli-file>] [-s] [-t <sikuli-file>] --args <arguments> specify the arguments passed to Jythons sys.argv -h,--help print this help message -r,--run <sikuli-file> run .sikuli or .skl file -s,--stderr print runtime errors to stderr instead of popping up a message box -t,--test <sikuli-file> run .sikuli as a unit test case with junits text UI runner -r,--run <sikuli-file> run .sikuli or .skl file -s,--stderr print runtime errors to stderr instead of popping up a message box -t,--test <sikuli-file> run a .sikuli (.skl not possible) as a unit test with junit‘s text UI runner (A script that can also be run in IDE‘s Unit Test View). Example - Run xxxx.sikuli with three arguments: PATH-TO-SIKULIsikuli-ide.exe -r xxxx.sikuli —args a1 a2 a3 Running sikuli-script.jar from Command line Run a script without IDE java -jar %SIKULI_HOME%sikuli-script.jar path-to-your-scriptyourScript.sikuli Run a script from Command line using the Sikuli contained Jython This option might be helpful in some cases, where you want to have access to the Jython layer, before any Sikuli feature is touched. Be aware, that a from sikuli.Sikuli import * is needed to have access to Sikuli. You might have to take care somehow, that images are found, since bundle path is not set. set SCRIPT=path-to-your-scriptyourScript.sikuliyourScript.py java -cp %SIKULI_HOME%sikuli-script.jar org.python.util.jython %SCRIPT% Interactive Sikuli Jython Session from Command Line (Option -i) sikuli-script.jar -i Start an interactive Jython session with the Sikuli environment already in place. This might be helpful to do some testing on commandline, without having to start the Sikuli IDE. A specified script and other options will be ignored. java -jar %SIKULI_HOME%sikuli-script.jar -iCreate Unit Testing Scripts for GUI:
  23. 23. Sikuli Satish Gorripotu Sikuli integrates with jUnit and supports unit testing for Graphical User Interfaces (GUI). The unit testing panel can be opened by clicking the menu View ‣ Unit Test or by the hot key ⌘ -U on Mac or Ctrl-U on Windows/Linux. Sikuli IDE aims to minimize the effort of writing code. With Sikuli IDE, a Python class inherited from junit.framework.TestCase is automatically generated to wrap your unit testing script. A typical unit testing script consists of two constructing and destructing methods, setUp() and tearDown(), and a bunch of methods named with a prefix test. The basic structure of a script is given as following: TestJEdit.sikuli def setUp(self): openApp("/Applications/jEdit.app") wait( , 20000) close = ( ) # if find(close): # click(close) def tearDown(self): closeApp("jEdit.app") untilNotExist( , 20000) def test_textarea_add_del_by_key(self): type("hello world") assertExist( ) type("a",KEY_CTRL) type("n") assertNotExist( ) def test_textarea_add_del_by_menu(self): type("hello world") assertExist( ) click( ) click( ) type("n") assertNotExist( ) def test_new_tab_by_key(self):
  24. 24. Sikuli Satish Gorripotu type("n",KEY_CTRL) assertExist(Pattern( ).similar(0.95)) def test_new_tab_by_menu(self): click( ) click( ) assertExist(Pattern( ).similar(0.95)) def test_new_tab_by_toolbar(self): click( ) assertExist(Pattern( ).similar(0.95)) def _verify_find(self): sleep(0.5) type("hello") type("n") click(Pattern( ).similar(0.90)) assertExist(Pattern( ).similar(0.94)) type("b") assertExist( ) def test_find_by_key(self): type("a long hello worldn") type("f",KEY_CTRL) self._verify_find() def test_find_by_menu(self): type("a long hello worldn") click( ) click( ) self._verify_find()
  25. 25. Sikuli Satish Gorripotu def test_find_by_toolbar(self): type("a long hello worldn") click( ) self._verify_find() def test_textfield_on_toolbar(self): type("a long hello worldn") assertExist( ) type("f",KEY_CTRL) assertExist(Pattern( ).similar(0.85)) type("hello") assertExist( ) def test_toolbar_print_dialog(self): click( ) assertExist( ) click( ) def test_menu_submenu(self): click( ) click( ) assertExist( ) type("test") click( ) click( )
  26. 26. Sikuli Satish Gorripotu click( ) click( ) click( ) click( ) assertExist( ) def test_scrollbar(self): for i in range(1,50): type("line %dn" % i) tail = Pattern( ).similar(0.92) wait(tail, 10000) assertExist(tail) thumb_at_top = Pattern( ).similar(0.98) assertNotExist(thumb_at_top) dragDrop( , [0, -1000]) assertExist(thumb_at_top) assertNotExist(tail) assertExist(Pattern( ).similar(0.92)) dragDrop(Pattern( ).similar(0.90), [0, 1000]) assertExist(tail) def test_quit(self): type("test") type("e",KEY_CTRL) type("q",KEY_CTRL) click( )
  27. 27. Sikuli Satish Gorripotu assertExist(Pattern( ).similar(0.90)) To run a unit testing script, you need to click on the Run button in the unit testing panel instead of the ordinary button. IMPORTANT: Before you try to run your script in this test mode the first time, it has to be saved. Everytime you change something, you have to save it again, before the next test run.Sikuli Script in JAVA programs: The core of Sikuli Script is written in Java, which means you can use Sikuli Script as a standard JAVA library in your program. This document lets you know how to do that. After having installed Sikuli on your system, as recommended on the download page, you have to do the following: 1. Get sikuli-script.jar from your Sikuli IDE installation path. Sikuli-IDE/sikuli-script.jar 2. Make the native libraries available 3. Include sikuli-script.jar in the CLASSPATH of your Java project. 4. Import the Sikuli classes you need You can simply ―import org.sikuli.script.*‖ or import the classes you need. In most cases, you would need at least Region or Screen. the package name was edu.mit.csail.uid.Sikuli together with other IDE’s - Eclipse: Install Python plugin (PyDev) and Jython Since the top level language used by Sikuli script is Python, you need the Python plugin PyDev. Different from NetBeans, though PyDev is prepared to use Jython as interpreter, you have to install Jython on your own seperately. So first install Jython, by downloading the version you want to use from the http://www.jython.org/downloads.html Sikuli currently is based on Jython 2.5.1, but you may choose Jython 2.5.2 as well. Install it according to Jythons installation HowTo. After installation, make a test from a commandline by typing jython.bat (Windows) or jython (Mac/Linux) to open an interactive Jython session. You might type the following (<enter> means pressing the enter/return key) import os <enter> import sys <enter> for e in sys.path: print e <enter> <enter> type exit() <enter> (to leave interactive Jython)
  28. 28. Sikuli Satish Gorripotu This shows the current Jython configuration of the Python path. Remember the place where Jython is installed. Now we install the PyDev plugin from inside Eclipse: Menu Help -> Eclipse Marketplace. Either search it or find it on the Popular tab‘s lower part. Simply click the Install button and do what you are asked to do ;-) The next step is to tell Eclipse PyDev, where it can find the Jython interpreter. Navigate to the Eclipse Preferences pane and open the category PyDev and inside the subcategory Interpreter-Jython. First try to Auto Config by clicking the appropriate button. If this does not work, click the button New, name the entry and navigate to the folder, where Jython was installed and select jython.jar. After clicking ok, a window Selection needed might come up: click Select all and ok to finalize this step. Other options with PyDev are available, but not relevant for our actual matter http://pydev.org/manual_101_root.html Configure for using Sikuli script features at runtime Again we have a difference to Netbeans: The PyDev plugin does not allow library references to folders inside of jar-files in the respective configuration dialog (it does not insert them to the Python path), though Jython itself accepts them, when specified on the Python path. So if you want to run and debug your script in Eclipse, you have to extract the folder Lib/sikuli from sikuli-script.jar Menu File -> New -> Project -> PyDev -> (open sublist) -> PyDev Project -> click Next button. On the configuration pane name your project, select Jython as Project type, grammar version 2.5 (higher Python language versions are not supported by Jython 2.5.x) and click the Finish button. Your project is created. Add at least one you-name-it.py file to the source folder and put from sikuli.Sikuli import * as the first line. In the last step, we tell PyDev, where to find the Sikuli libraries. Goto Menu Project -> Properties -> select category PyDev - PYTHONPATH and go to the tab External Libraries. We need a reference to path-to/sikuli-script.jar and another one to the extracted folder Lib containing the folder sikuli. reference to path-to/sikuli-script.jar o click button Add zip/jar/egg and select sikuli-script.jar from the Sikuli installation. reference to the extracted folder Lib containing the folder sikuli o click Add source folder and select the folder Lib in the place you had it extracted. o this is not needed, if you have moved the extracted folder sikuli to a folder, that is already on the Python path (e.g. jython-intallation/Lib/site-packages). Now you are able to run your first script. Remember, that in every script including the main script, that you are editing now, as the first line you need from sikuli.Sikuli import *, to have access to the Sikuli features at runtime.
  29. 29. Sikuli Satish Gorripotu Everytime later on you might come back to the project‘s preferences with Project -> Properties. Code Completion works from the start without any further configuration and even steps into the Java classes where appropriate.Configuration Eclipse:1) Open the Eclipse by double clicking on eclipse.exe, Click on OK for workspace launcher. Click onworkbench, it will allow you to move inside the eclipse.2) Click on File->New->Select Java Project.
  30. 30. Sikuli Satish Gorripotu 3) Enter the Project name and then click on ―Configue JRE‘s‖
  31. 31. Sikuli Satish GorripotuImprove the performance of Sikuli from org.sikuli.script.natives import Vision Vision.setParameter("MinTargetSize", 6)Import Packages/ FilesAll files must be in Sys path. So Sikuli will find the files automatically, if we give it as from sikuli import *Example: myScriptPath = "c:SikuliFilesmyLibrary" if not myScriptPath in sys.path: sys.path.append(myScriptPath) # supposing there is a myLib.sikuli import myLib # supposing myLib.sikuli contains a function "def myFunction():" myLib.myFunction() # makes the callAs Python load the file at once and will continue the library to entire script. If we modify the code in libraryfile, then Python wouldn‘t reflect the changes. So, # instead of: import module
  32. 32. Sikuli Satish Gorripotu import module reload(module) # instead of: from module import * import module reload(module) from module import * myLib.myFunction() or myLib import * # which integrates all names from myLib into your current namespace. myFunction() # We can use directly Importing from the directory: import os # get the directory containing your running .sikuli myPath = os.path.dirname(getBundlePath()) if not myPath in sys.path: sys.path.append(myPath) # now you can import every .sikuli in the same directory import myLib Scripts in same directory are found automatically # nothing else needed # now you can import every .sikuli in the same directory import myLib load(jar-file) : with path or no path if no path is specified, Sikuli first looks into the bundle (the Sikuli folder of the running script) and then into the extensions folder.Controlling Sikuli Scripts and their Behavior: setShowActions(False | True) True, when a script is run, Sikuli shows a visual effect (a blinking double lined red circle) on the spot. exit([value]) Stops the script gracefully at this point. The value is returned to the calling environment.Setting is a class available in Sikuli Package.
  33. 33. Sikuli Satish GorripotuThis class has several methodsClass Setting { 1. ActionLogs (True/False) will display the log messages in console 2. InfoLogs (True/False) will display the Info log messages in console 3. DebugLogs (True/False) will display the Debug log messages in console 4. MinSimilarity Sikuli will search for the minimal similarity of the object in the screen. Like smart Identification 5. MoveMouseDelay Control the mouse move animation. mmd = Settings.MoveMouseDelay # save default/actual value click(image1) # implicitly wait 0.5 seconds before click Settings.MoveMouseDelay = 3 click(image2) # give app 3 seconds time before clicking again Settings.MoveMouseDelay = mmd # reset to original value 6. DelayAfterDrag Specifies the waiting time after mouse down at the source location as a decimal value (seconds). 7. DelayBeforeDrop Specifies the waiting time before mouse up at the target location as a decimal value (seconds). Above DelayAfterDrag, DelayBeforeDrop can be used as alternative of Wait Time. # you may wish to save the actual settings before Settings.DelayAfterDrag = 1 Settings.DelayBeforeDrop = 1 Settings.MoveMouseDelay = 3 dragDrop(source_image, target_image) # time for complete dragDrop: about 5 seconds + search times 8. SlowMotionDelay Control the duration of the visual effect (seconds). 9. WaitScanRate 10.ObserveScanRate Specify the number of times actual search operations are performed per second while waiting for a pattern to appear or vanish. Example: Region.wait(), Region.exists(), Region.waitVanish(), Region.observe()
  34. 34. Sikuli Satish Gorripotu def myHandler(e): print "it happened" # you may wish to save the actual settings before Settings.ObserveScanRate = 0.2 onAppear(some_image, myHandler) observe(FOREVER, background = True) # the observer will look every 5 seconds # since your script does not wait here, you # might want to stop the observing later on ;-) 11.ObserveMinChangedPixels The minimum size in pixels of a change to trigger a change event. Ex: Region.onChange() without specifying this value. The default value is 50.}Controlling Applications and their Windows:Note: backslashes and double apostrophes ―In a Sikuli script in normal strings enclosed in ‖ (double apostrophes), these special characters and ‖have to be escaped using a backslash, when you have them inside the string. So for one backslash youneed and for one ‖ you need ‖. In a string enclosed in ‗ (single apostrophes), a ‗ has to be ‘ and a ‖ istaken as such.To avoid any problems, it is recommended to use the raw string rsome text with and " ..., since thereis no need for escaping. This is especially useful, when you have to specify Windows path‘s or want tosetup command lines for use with App.open(), openApp(), os.popen or Jythons Subprocess module.a fictive command line example: cmd = rc:Program Filesmyapp.exe -x "c:Some Placesome.txt" >..log.txt openApp(cmd)Open an application : os.open --> openApp() –> App.open()Switch to an application : switchApp() –> App.focus()Close an application : closeApp() –> App.close()
  35. 35. Sikuli Satish GorripotuopenApp(Application) Deprecated. Now it is App.open()This function may switch to an already opened application or may open a new instance of the application. # opens command prompt (found through PATH) openApp("cmd.exe") # with parameters (no sense, only to show ;-) openApp(rcmd.exe /c start c:Program Filesmyapp.bat) # opens Firefox (full path specified) openApp("c:Program FilesMozilla Firefoxfirefox.exe")switchApp(application) Deprecated. Now it is App.focus() This function switches the focus to the specified application and brings its windows to the front. The window is identified by the application string. This string (needs not be an application‘s name) is used to search the title text of all the opened windows for any part of the title matching the string. Example: # switches to an existing command prompt or starts a new one switchApp("cmd.exe") # opens a new browser window switchApp("c:Program FilesMozilla Firefoxfirefox.exe") # switches to the frontmost opened browser window (or does nothing if no browser window is currently opened) switchApp("mozilla firefox")closeApp(application) Deprecated. Now it is App.close(). Close the specified application. Parameters: application – the name of an application or part of a window title This function closes the application indicated by the windows whose titles contain the string application. Example: # closes an existing command prompt closeApp("cmd.exe") # does nothing, since text cannot be found in the window title closeApp("c:Program FilesMozilla Firefoxfirefox.exe") # stops firefox including all its windows
  36. 36. Sikuli Satish Gorripotu closeApp("mozilla firefox")run(command) Run command in the command line Parameters: command – a command that can be run from the command line. This function executes the command and the script waits for its completion.Note: Windows: Sikuli‘s strategy on these systems in the moment is to rely on implicit or explicit pathspecifications to find an application that has to be started.Running ―applications‖ can either be identified using their PID (process ID) or by using the window titles.So using a path specification will only switch to an open application, if the application internally handlesthe ―more than one instance‖ situation‖.We usually use App.open("c:Program FilesMozilla FirefoxFirefox.exe") to start Firefox. This mightopen an additional window.And you can use App.focus("Firefox") to switch to the front most Firefox windowNote: When specifying a path in a string, we have to use (double backslash) for each (backslash)e.g. myPath = "c:Program FilesSikuli-IDELib" )App is a class available in Sikuli Package.This class has several methods.We can use App class methods in two ways 1. class methods 2. instance methodsClass App { 1. Open Open the specified application. It opens the specified application and brings its window the front. Whether this operation switches to an already opened application or opens a new instance of the application depends on the system and application 1. Class Method : open(application) Usage : App.open("application-identifier") 2. Instance method : open() # App instance someApp was created before Usage : myApp = App("application-identifier") myApp.open() 2. Focus Switch the focus to an application. Parameters are the name of an application or part of a window title 1. Class Method : focus (application) Usage : App.focus("application-identifier")
  37. 37. Sikuli Satish Gorripotu 2. Instance method : open() # App instance someApp was created before Usage : myApp = App("application-identifier") myApp. focus () 3. Close It closes the given application or the matching windows. It does nothing if no opened window or running application can be found. Whether the application itself is closed depends on whether all open windows are closed Or a main window of the application is closed, that in turn closes all other opened windows. 1. Class Method : close (application) Usage : App. close ("application-identifier") 2. Instance method : close () # App instance someApp was created before Usage : myApp = App("application-identifier") myApp. close() 4. focusedWindow Identify the currently focused or the front most window and switch to it. Sikuli does not tell you, to which application this window belongs. Region object representing the window or None if there is no such window. This method always returns a region. When there is no window opened on the desktop, the region may refer to a special window such as the task bar or an icon in the system tray. 1. Class Method : focusedWindow (application) Usage : App. focusedWindow ("application-identifier") 2. Instance method : close () # App instance someApp was created before Usage : myApp = App("application-identifier") myApp. focusedWindow () Example: # highlight the currently fontmost window for 2 seconds App.focusedWindow().highlight(2) # save the windows region before firstWindow = App.focusedWindow() firstWindow.highlight(2) 5. window([n])
  38. 38. Sikuli Satish Gorripotu Get the series of windows with the matching title. 1. Class Method : App(application).window([n])#an App instance is created on the fly. Usage : App(application).window([n]) 2. Instance method : close () # App instance someApp was created before Usage : myApp = App("application-identifier") myApp.window([n]) Parameters: n – 0 or a positive integer number. If ommitted, 0 is taken as default. Returns : the region on the screen occupied by the window, if such window exists and None if otherwise. Example : #opens a Firefox browser window and switches to the address field. # using an existing window if possible myApp = App("Firefox") if not myApp.window(): # no window(0) - Firefox not open App.open("c:Program FilesMozilla FirefoxFirefox.exe") wait(2) myApp.focus() wait(1) type("l", KEY_CTRL) # switch to address field Afterwards, it focuses on the Firefox application, uses the window () method to obtain the region of the front most window, Applies some operations within the region, And finally closes the window: # using a new window firefox = App.open("c:Program FilesMozilla FirefoxFirefox.exe"); wait(2) firefox.focus() wait(1) # now your just opened new window should be the frontmost with firefox.window(): # see the general notes below # some actions inside the window(0)s region click("somebutton.png") firefox.close() # close the window - stop the process Below is another example that highlights all the windows of an application by looping through them # not more than 100 windows should be open ;-) myApp = App("Safari") for n in range(100): w = myApp.window(n) if not w: break # no more windows w.highlight(2) # window highlighted for 2 second
  39. 39. Sikuli Satish Gorripotu }Interacting with the User popup(text[, title]) Display a dialog box with an OK button and text as the message. The script then waits for the user to click the OK button. Parameters: text – text to be displayed as message title – optional title for the messagebox Example: popup("Hi satish….n How are you") input([text][, default]) Display a dialog box with an input field, a Cancel button, and an OK button. The optional text can be displayed as a caption. The script then waits for the user to click either the Cancel or the OK button. Parameters: text – optional text to be displayed as message default – optional preset text for the input field Returns: the text, contained in the input field, when the user clicked OK None, if the user pressed the Cancel button Example: name = input("Please enter your name:") name = input("Please enter your name to log in:", "anonymous") # a preset input text When using the parameter default, the text input field will be pre-populated with the given text, So we click OK/Cancel or edit the content of the input field
  40. 40. Sikuli Satish GorripotuListening to Global Hotkeys Env.addHotkey(key, modifiers, handler) Register the specified key + modifiers as a global hotkey. When the hotkey is pressed, the specified function handler will be called. Parameters: key – a character or a constant value defined in Key. modifiers – Key modifiers, which can be one or multiple constants defined in KeyModifier. def openAppleMenu(event): click(Any Image) # When we pressed Ctrl+Alt+F1, click the top-left apple icon. Env.addHotkey(Key.F1, KeyModifier.ALT+KeyModifier.CTRL, openAppleMenu) Env.removeHotkey(key, modifiers) Unregister the registered global hotkey key + modifiers. Parameters: key – a character or a constant value defined in Key. modifiers – Key modifiers, which can be one or multiple constants defined in KeyModifier.General Settings and Access to Environment InformationSikuli Level Sikuli internally uses the class Settings to store globally used settings. Publicly available attributes accessed by using Settings.[name-of-an-attribute] To get it‘s value and Settings.attribute = value to set it.Jython/Python Level We can use all settings defined in standard Python/Jython which are available in our system environment. The modules sys and time are already imported, so you can use their methods without the need for an import statement. sys.path may be one of the most valuable settings, since it is used by Python/Jython to locate modules, that are referenced using import module. myPath = "some-absolute-path" if not myPath in sys.path: sys.path.append(myPath)Java Level import java
  41. 41. Sikuli Satish Gorripotu # get a value val = java.lang.System.getProperty("key-of-property") # set a propertys value java.lang.System.getProperty("key-of-property", value)Image Search Path Sikuli maintains a list of locations to search for images when they are not found in the current .sikuli folder (a.k.a. bundle path). This list is maintained internally but can be inspected and modified. getImagePath() Get a list of paths where Sikuli will search for images. # getImagePath() returns a Java array of unicode strings imgPath = list(getImagePath()) # makes it a Python list # to loop through for p in imgPath: print p addImagePath(a-new-path) Add a new path to the list of image search paths removeImagePath(a-path-already-in-the-list) Remove a path from the list of image search paths Note: paths must be specified using the correct path separators as double blackslashes .Default bundle path can accessed and modified. setBundlePath(path-to-a-folder) Set the path for searching images in all Sikuli Script methods. Sikuli IDE sets this automatically to the path of the folder where it saves the script (.sikuli). getBundlePath() Get a string containing a path to a folder containing our images used for finding patterns. Sikuli IDE sets this automatically to the path of the folder where it saves the script (.sikuli). We can use this function to package our private files together with the script or to access the picture files in the .sikuli bundles for other purposes. Sikuli only gives access to the path name, We need other python modules for I/O or other purposes.Other Environment Information Env.getOS() Env.getOSVersion() Env.getSikuliVersion() Example: if not Env.getSikuliVersion() == "X-1.0rc2":
  42. 42. Sikuli Satish Gorripotu print "This script needs Sikuli X-1.0rc2" exit(1) Env.getClipboard() Get the content of the clipboard if it is text, otherwise an empty string. Env.getClipboard().paste() clipboard will contain what we just pasted. Env.getClipboard().strip() To get rid of surrounding white spaces. Env.isLockOn(key-constant) Get the current status ( on / off ) off the respective key. Only one key can be specified. key-constant – one of the key constants Key.CAPS_LOCK, Key.NUM_LOCK, Key.SCROLL_LOCK Env.getMouseLocation() Get the current location of the mouse cursor.Advanced Settings for Tuning Vision Algorithm Vision.setParameter(param, value) Vision.getParameter(param) Get/Set the parameter param of the vision algorithm. Parameters: param – a string that indicates the parameter to get. Returns: The float value of the specified parameter. MinTargetSize MinTargetSize is the minimum image size to which Sikuli can resize. Sikuli resizes the screen images to a smaller scale for faster matching. This scaling process speeds up the matching process, but also increases the possibility of false matching. The default value of MinTargetSize is 12, < 12. May increase speed but may returns a match that is not what we expect. Example Jython code. from org.sikuli.script.natives import Vision Vision.setParameter("MinTargetSize", 6) # the default is 12. Setting the size to a smaller value would make the matching algorithm be faster.
  43. 43. Sikuli Satish GorripotuRegion Region is a class available in Sikuli Package. This class has several methods Class Region { class Region Region is a rectangular area on a screen. It knows nothing about its visual content (windows, pictures, graphics, text, ...). It only knows the position on the screen and its dimension. Creating a Region, Setting and Getting Attributes Region(x, y, w, h) Region(region) Region(Rectangle)
  44. 44. Sikuli Satish Gorripotu Parameters: x – x position of a rectangle. #upper left corner of the screen (0, 0) y – y position of a rectangle. #upper left corner of the screen (0, 0) w – height of a rectangle. h – width of a rectangle. region – an existing region object. rectangle – an existing object of Java class Rectangle Returns: a new region object. region can also be created in run-time using Region.selectRegion(). You can create a region based on another region. This just duplicates the region (new object). Another way to create a region is to specify a rectangle object or to extend an existing region. Note: topLeft = Location(reg.x, reg.y) # equivalent to topLeft = reg.getTopLeft() theWidth = reg.w # equivalent to theWidth = reg.getW() reg.h = theWidth # equivalent to reg.setH(theWidth) We can use selectRegion() to interactively create a new region. setX(number) setY(number) setW(number) setH(number) Set the respective attribute of the region to the new value. This effectively moves the region around and/or changes its dimension. moveTo(location) Set the position of this region regarding it‘s top left corner to the given location (the x and y values are modified). Parameters : location – location object becomes the new top left corner Returns : the modified region object Example: reg.moveTo(anotherLocation) # equivalent to reg.setX(anotherLocation.x); reg.setY(anotherLocation.y) setROI(x, y, w, h) setROI(rectangle) setRect(x, y, w, h)
  45. 45. Sikuli Satish Gorripotu setRect(rectangle) Both methods do the same: setting position and dimension to new values. setROI() is intended to restrict the search to a smaller area to speed up processing setRect() should be used to redefine a region (which could be enlarging it). Parameters: x, y, w, h (all) – the attributes of a rectangle rectangle – a rectangle object . morphTo(region) Set the position and dimension of this region to the corresponding values of the region given as parameter. Parameters:region – a region object Returns: the modified region object Note: reg.morphTo(anotherRegion) # equivalent to r = anotherRegion; reg.setX(r.x); reg.setY(r.y); reg.setW(r.w); reg.setH(r.h) getX() getY() getW() getH() Get the respective attribute integer value of the region. getCenter() Get the center of the region or Location as an object. getTopLeft() getTopRight() getBottomLeft() getBottomRight() Get the location object of the region‘s respective corner getScreen() Returns the screen object that contains this region. This method makes sense in Multi Monitor Environments getLastMatch() getLastMatches() Returns the best match as a Match object or one or more match objects as an Iterator object respectively setAutoWaitTimeout(seconds) Set the maximum waiting time for all subsequent find operations.
  46. 46. Sikuli Satish Gorripotu This method enables all find operations to wait for the given pattern to appear until the specified amount of time has elapsed. The default is 3.0 seconds. This method is intended for users to override this default setting. Region.find() work like Region.wait(), without being able to set an individual timeout value for a specific find operation. getAutoWaitTimeout() Get the current value of the maximum waiting time for find operations.Extending a Region Note: In all cases the current region remains unchanged. In all cases the new region does not extend beyond any boundary of the screen that contains the given region. offset(location) Returns a new Region object, whose upper left corner is relocated adding the location‘s x and y value to the respective values of the given region. Width and height are the same. So this clones a region at a different place. Parameters : location – a Location object Returns :a new Region object Example: new_reg = reg.offset(Location(xoff, yoff)) # same as new_reg = Region(reg.x + xoff, reg.y + yoff, reg.w, reg.h) inside() Returns the same Region object. Retained for upward compatibility. region.inside().find() is totally equivalent to region.find(). nearby([range]) Returns a new Region object that includes the nearby neighbourhood of the the current region. The new region is defined by extending the current region‘s dimensions in all directions by range number of pixels. The center of the new region remains the same.
  47. 47. Sikuli Satish Gorripotu Parameters: range – a positive integer indicating the number of pixels (default = 50). above([range]) Returns a new Region object that is defined above the current region‘s top border with a height of range number of pixels. It does not include the current region. If range is omitted, it reaches to the top of the screen. The new region has the same width and x- position as the current region. Parameters:range – a positive integer defining the new height below([range]) Returns a new Region object that is defined below the current region‘s bottom border with a height of range number of pixels. It does not include the current region. If range is omitted, it reaches to the bottom of the screen. The new region has the same width and x-position as the current region. Parameters:range – a positive integer defining the new height left([range]) Returns a new Region object that is defined left of the current region‘s left border with a width of range number of pixels. It does not include the current region. If range is omitted, it reaches to the left border of the screen. The new region has the same height and y-position as the current region. Parameters:range – a positive integer defining the new width right([range]) Returns a new Region object that is defined right of the current region‘s right border with a width of range number of pixels. So it does not include the current region. If range is omitted, it reaches to the right border of the screen. The new region has the same height and y-position as the current region. Parameters:range – a positive integer defining the new widthFinding inside a Region and Waiting for a Visual Event Besides acting on visual objects, finding them is one of the core functions of Sikuli. PS: means, that either a Pattern or a string (path to an image file or just plain text) can be used as parameter. find(PS) Find a particular GUI element, which is seen as the given image or just plain text. The given file name of an image specifies the element‘s appearance. It searches within the region and returns the best match, which shows a similarity greater than the minimum similarity given by the pattern. Parameters:PS – a Pattern object or a string (path to an image file or just plain text) Returns: a Match object that contains the best match or fails if not found findAll(PS)
  48. 48. Sikuli Satish Gorripotu Repeatedly find ALL instances of a pattern, until no match can be found anymore, that meets the requirements for a single Region.find() with the specified pattern. Parameters:PS – a Pattern object or a string (path to an image file or just plain text) Returns :one or more Match objects as an iterator object or fails if not found By default, the returned matches are sorted by the similiarty. If you need them ordered by their positions, we have to use Python‘s sorted function. Example: def by_y(match): return match.y icons = findAll(―Image screen shot‖) # sort the icons by their y coordinates and put them into a new variable sorted_icons sorted_icons = sorted(icons, key=by_y) # another shorter version is using lambda. sorted_icons = sorted(icons, key=lambda m:m.y) for icon in sorted_icons: pass # do whatever you want to do with the sorted icons wait([PS][, seconds]) a Match object that contains the best match or fails if not found Parameters: PS – a Pattern object or a string (path to an image file or just plain text) seconds – a number, which can have a fraction, as maximum waiting time in seconds. If not specified, the auto wait timeout value set by Region.setAutoWaitTimeout() is used. Use the constant FOREVER to wait for an infinite time. If PS is not specified, the script just pauses for the specified amount of time. It is still possible to use sleep(seconds) instead, but this is deprecated. If PS is specified, it keeps searching the given pattern in the region until the image appears ( would have been found with Region.find()) or the specified amount of time has elapsed. At least one find operation is performed, even if 0 seconds is specified.) waitVanish(PS[, seconds]) Wait until the give pattern PS in the region vanishes. Returns True if the pattern vanishes within the specified waiting time, or False if the pattern stays visible after the waiting time has elapsed. This method keeps searching the given pattern in the region until the image vanishes (can not be found with Region.find() any longer) or the specified amount of time has elapsed. At least one find operation is performed, even if 0 seconds is specified Parameters: PS – a Pattern object or a string (path to an image file or just plain text) seconds – a number, which can have a fraction, as maximum waiting time in seconds. The internal granularity is milliseconds. If not specified, the auto wait timeout value set by Region. setAutoWaitTimeout() is used. Use the constant FOREVER to wait for an infinite time.
  49. 49. Sikuli Satish Gorripotu . exists(PS[, seconds]) Check whether the give pattern is visible on the screen. And returns a Match object that contains the best match. None is returned, if nothing is found within the specified waiting time Does exactly the same as Region.wait(), but no exception is raised in case of FindFailed. It can be used to symplify scripting in case that we want to know wether something is there or not to decide how to proceed in your workflow. It is typically used with an if statement. At least one find operation is performed, even if 0 seconds is specified. Parameters: PS – a Pattern object or a string (path to an image file or just plain text) seconds – a number, which can have a fraction, as maximum waiting time in seconds. The internal granularity is milliseconds. If not specified, the auto wait timeout value set by Region.setAutoWaitTimeout() is used. Use the constant FOREVER to wait for an infinite time.Observing Visual Events in a Region handler: as a parameter in the following methods, you have to specify the name of a function, which will be called by the observer, in case the observed event happens. The function itself has to be defined in your script before using the method that references the function. The existance of the function will be checked before starting the script. So to get your script running, you have to have at least the following statements in your script: def myHandler(event): # you can choose any valid function name # event: can be any variable name, it references the SikuliEvent object pass # add your statements here onAppear("path-to-an-image-file", myHandler) # or any other onEvent() observe(10) # observes for 10 seconds onAppear(PS, handler) With the given region you register an observer, that should wait for the pattern to be there or to appear and is activated with the next call of observe(). In the moment the internal find operation on the given pattern is successful during observation, your handler is called and the observation is paused until you return from your handler. Parameters: PS – a Pattern object or a string (path to an image file or just plain text. handler – the name of a handler function in the script
  50. 50. Sikuli Satish Gorripotu onVanish(PS, handler) With the given region you register an observer, that should wait for the pattern to be not there or to vanish and is activated with the next call of observe(). In the moment the internal find operation on the given pattern fails during observation, your handler is called and the observation is paused until you return from your handler. Parameters: PS – a Pattern object or a string (path to an image file or just plain text. handler – the name of a handler function in the script onChange([minChangedSize], handler) Parameters: minChangedSize – the minimum size in pixels of a change to trigger a change event. If omitted: 50 is used handler – the name of a handler function in the script minChangedSize With the given region we register an observer, that should wait for the visual content of the given region to change and is activated with the next call of observe(). In the moment the visual content changes during observation, our handler is called and the observation is paused until you return from your handler. Below example highlights all changes in an observed region. def changed(event): print "something changed in ", event.region for ch in event.changes: ch.highlight() # highlight all changes sleep(1) for ch in event.changes: ch.highlight() # turn off the highlights with selectRegion("select a region to observe") as r: # any change in r larger than 50 pixels would trigger the changed function onChange(50, changed) observe(background=True) wait(30) # another way to observe for 30 seconds r.stopObserver() observe([seconds][, background = False | True]) Begin observation within the region. For each region object, only one observation can be running at a given time.
  51. 51. Sikuli Satish Gorripotu Parameters: seconds – a number, which can have a fraction, as maximum observation time in seconds. Omit it or use the constant FOREVER to tell the observation to run for an infinite time (or until stopped by a call of stopObserve()). background – a flag indicating whether observation is run in the background. when set to True, the observation will be run in the background and processing of your script is continued immediately. Otherwise the script is paused until the completion of the observation. stopObserver() Stop observation within the region. This must be called on a valid region object. The source region of an observed visual event is available as one of the attributes of the event parameter that is passed to the handler function when the function is invoked. Example: to stop observation within a handler function, call event.region.stopObserver() inside the handler function. def myHandler(event): event.region.stopObserver() # stops the observation onAppear("path-to-an-image-file", myHandler) observe(FOREVER) # observes until stopped in handler }SikuliEvent SikuliEvent is a class available in Sikuli Package. This class has several methods Class SikuliEvent {
  52. 52. Sikuli Satish Gorripotu When processing an observation in a region, a handler function is called, when one of the registered events Region.onAppear(), Region.onVanish() or Region.onChange() One parameter, we have access to in handler function is an instance of Class SikuliEvent. Following attributes of the event, might help to identify the cause of the event and act on the resulting matches. Usage: event.attribute where event is the parameter from the definition of handler function: def myHandler(event): type SikuliEvent.Type.APPEAR, SikuliEvent.Type.VANISH, or SikuliEvent.Type.CHANGE. pattern The source pattern that triggered this event. This is only valid in APPEAR and VANISH events. region The source region of this event. match In an APPEAR event, match saves the top Match object that appeared in the observed region. In a VANISH event, match saves the last Match object that was in the observed region but vanished. This attribute is not valid in a CHANGE event. changes This attribute is valid only in a CHANGE event. This changes attribute is a list of Match objects that changed and their sizes are at least minChangedSize pixels. } Acting on a Region PSMRL: which means, that either a Pattern object or a string (path to an image file or just plain text) or a Match or a Region or a Location can be used as parameter. P: pattern: a Pattern object. An implicit find operation is processed first. If successful, the center of the resulting matches rectangle is the click point. If the pattern object has a target offset specified, this is used as click point instead. S: string: a path to an image file or just plain text. An implicit find operation with the default minimum similarity 0.7 is processed first. If successful, the center of the resulting match object is the click point.
  53. 53. Sikuli Satish Gorripotu M: match: a match object from a previous find operation. If the match has a target specified it is used as the click point, otherwise the center of the matches rectangle. R: region: a region object whose center is used as click point. L: location: a location object which by definition represents a point on the screen that is used as click point.Region Region is a class available in Sikuli Package. This class has several methods continuation to above methods. Class Region { click(PSMRL[, modifiers]) Perform a mouse click on the click point using the left button. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. modifiers – one or more key modifiers Returns: Number of performed clicks (actually 1). A 0 (integer null) means that because of some reason, no click could be performed (in case of PS may be not Found). doubleClick(PSMRL[, modifiers]) Perform a mouse double-click on the click point using the left button. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. modifiers – one or more key modifiers Returns: Number of performed double-clicks (actually 1). A 0 (integer null) means that because of some reason, no click could be performed (in case of PS may be not Found). rightClick(PSMRL[, modifiers]) Perform a mouse click on the click point using the right button. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. modifiers – one or more key modifiers Returns: Number of performed right cilicks (actually 1). A 0 (integer null) means that because of some reason, no click could be performed (in case of PS may be not Found). highlight([seconds]) Highlight the region for some period of time. Parameters:seconds – a decimal number taken as duration in seconds The region is highlighted showing a red colored frame around it. If the parameter seconds is given, the script is suspended for the specified time. If no time is given, the highlighting is started and the script continues.
  54. 54. Sikuli Satish Gorripotu When later on the same highlight call without a parameter is made, the highlighting is stopped (behaves like a toggling switch). Example: m = find(some_image) # the red frame will blink for about 7 - 8 seconds for i in range(5): m.highlight(1) wait(0.5) # a second red frame will blink as an overlay to the first one m.highlight() for i in range(5): m.highlight(1) wait(0.5) m.highlight() # the red frame will grow 5 times for i in range(5): m.highlight(1) m = m.nearby(20) hover(PSMRL) Move the mouse cursor to hover above a click point. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. modifiers – one or more key modifiers Returns: The number 1 if the mousepointer could be moved to the click point. A 0 (integer null) returned if no move could be performed (in case of PS may be not Found). dragDrop(PSMRL, PSMRL[, modifiers]) Perform a drag-and-drop operation from a starting click point to the target click point indicated by the two PSMRLs respectively. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. modifiers – one or more key modifiers If one of the parameters is PS, the operation might fail due to not Found. drag(PSMRL) Start a drag-and-drop operation by dragging at the given click point. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
  55. 55. Sikuli Satish Gorripotu Returns: Number 1 if the operation could be performed. A 0 (integer null) returned if no move could be performed (in case of PS may be not Found). dropAt(PSMRL[, delay]) Complete a drag-and-drop operation by dropping a previously dragged item at the given target click point. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. delay – time to wait after in seconds as decimal value Returns: Number 1 if the operation could be performed. A 0 (integer null) returned if no move could be performed (in case of PS may be not Found). type([PSMRL], text[, modifiers]) Type the text at the current focused input field or at a click point specified by PSMRL. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. modifiers – one or more key modifiers Returns: Number 1 if the operation could be performed, 0 (integer null) if it was not possible or the click couldn‘t be performed (in case of PS may be not Found). paste([PSMRL], text) Paste the text at a click point. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. Modifiers – one or more key modifiers Returns: Number 1 if the operation could be performed, 0 (integer null) if not possible or the click couldn‘t be performed (in case of PS may be not Found).Extracting Text from a Region text() Extract the text contained in the region using OCR.
  56. 56. Sikuli Satish GorripotuLow-level Mouse and Keyboard Actions mouseDown(button) Press the mouse button down. Parameters: button – one or a combination of the button constants Button.LEFT, Button.MIDDLE, Button.RIGHT. Returns: Number 1 if the operation is performed successfully, and zero if otherwise. mouseUp([button]) Release the mouse button previously pressed. Parameters: button – one or a combination of the button constants Button.LEFT, Button.MIDDLE, Button.RIGHT. Returns: Number 1 if the operation is performed successfully, and zero if otherwise. mouseMove(PSRML) Move the mouse pointer to a location indicated by PSRML. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
  57. 57. Sikuli Satish Gorripotu Returns: Number 1 if the operation could be performed. If using PS (which invokes an implicity find operation), find fails and you have switched off FindFailed exception, a 0 (integer null) is returned. Otherwise, the script is stopped with a FindFailed exception. wheel(PSRML, WHEEL_DOWN | WHEEL_UP, steps) Move the mouse pointer to a location indicated by PSRML and turn the mouse wheel in the specified direction by the specified number of steps. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. WHEEL_DOWN|WHEEL_UP – one of the two constants denoting the wheeling direction. steps – an integer indicating the amoung of wheeling. keyDown(key | list-of-keys) Press and hold the specified key(s) until released by a later call to Region.keyUp(). Parameters: key|list-of-keys – one or more keys (use the consts of class Key). A list of keys is a concatenation of several key constants using ―+‖. Returns: Number 1 if the operation could be performed and 0 if otherwise. keyUp([key | list-of-keys]) Release given keys. If no key is given, all currently pressed keys are released. Parameters: key|list-of-keys – one or more keys (use the consts of class Key). A list of keys is a concatenation of several key constants using ―+‖. Returns: Number 1 if the operation could be performed and 0 if otherwise.Exception FindFailed These are the possibilities to handle ―not found‖ situations: •generally abort a script, if not handled with try: ... except: ... (the default setting or using setThrowException(True) or setFindFailedResponse(ABORT)) •generally ignore all ―not found‖ situations (using setThrowException(False) or setFindFailedResponse(SKIP)), •want to be prompted in such a case (using setFindFailedResponse(PROMPT)) •advise Sikuli to wait forever (be careful with that!) (using setFindFailedResponse(RETRY)) Using PROMPT:
  58. 58. Sikuli Satish Gorripotu Example: # --- nice and easy if exists("path-to-image"): # no exception, returns None when not found pass # it is there else: pass # we miss it # --- using exception handling # every not found in the try block will switch to the except block try: find("path-to-image") pass # it is there except FindFailed: pass # we miss it # --- using setFindFailedResponse setFindFailedResponse(SKIP) # no exception raised, not found returns None if find("path-to-image"): setFindFailedResponse(ABORT) # reset to default pass # it is there else: setFindFailedResponse(ABORT) # reset to default pass # we miss it # --- using setThrowException setThrowException(False) # no exception raised, not found returns None if find("path-to-image"): setThrowException(True) # reset to default pass # it is there else: setThrowException(True) # reset to default pass # we miss it setFindFailedResponse(ABORT | SKIP | PROMPT | RETRY) For the specified region set the option, how Sikuli should handle ―not found‖ situations. The option stays in effect until changed by another setFindFailedResponse().
  59. 59. Sikuli Satish Gorripotu Parameters: ABORT – all subsequent find failed operations (explicit or implicit) will raise exception FindFailed (which is the default when a script is started). SKIP – all subsequent find operations will not raise exception FindFailed. Instead, explicit find operations such as Region.find() will return None. Implicit find operations (action functions) such as Region.click() will do nothing and return 0. PROMPT – all subsequent find operations will not raise exception FindFailed. Instead you will be prompted for the way to handle the situation (see using PROMPT). Only the current find operation is affected by your response to the prompt. RETRY – all subsequent find operations will not raise exception FindFailed. Instead, Sikuli will try to find the target until it gets visible. This is equivalent to using wait( ... , FOREVER) instead of find() or using setAutoWaitTimeout(FOREVER). getFindFailedResponse() Get the current setting in this region. Returns: ABORT or SKIP or PROMPT or RETRY Usage: val = getFindFailedResponse() if val == ABORT: print "not found will stop script with Exception FindFailed" elif val == SKIP: print "not found will be ignored" elif val == PROMPT: print "when not found you will be prompted" elif val == RETRY: print "we will always wait forever" else: print val, ": this is a bug :-(" Note: It is recommended to use set/getFindFailedResponse() instead of set/getThrowException() since the latter ones might be deprecated in the future. setThrowException(False | True) By using this method you control, how Sikuli should handle not found situations in this region. Parameters: True – all subsequent find operations (explicit or implicit) will raise exception FindFailed (which is the default when a script is started) in case of not found. False – all subsequent find operations will not raise exception FindFailed. Instead, explicit find operations such as Region.find() will return None. Implicit find operations (action functions) such as Region.click() will do nothing and return 0. getThrowException() Returns: True or False
  60. 60. Sikuli Satish GorripotuGrouping Method Calls ( with Region: ) Instead of: # reg is a region object if not reg.exists(image1): reg.click(image2) reg.wait(image3, 10) reg.doubleClick(image4) you can group methods applied to the same region using Python‘s with syntax: # reg is a region object with reg: if not exists(image1): click(image2) wait(image3, 10) doubleClick(image4) All methods inside the with block that have the region omitted are redirected to the region object specified at the with statement. Note for Sikuli X versions up to rc3 This should work for Regions and Region-like objects (Region, Screen, App.window()). Match objects have to be made a Region explicitly using Region(some_match). This is a general workaround when facing any problems using with some_region:. Special Methods for Developers getRegionFromPSRM(PSRM) Returns a new Region object derived from the given parameter. In case of PS, internally a find() is done inside this region. If found, the match is returned. In case RM, just a copy of the given region is returned. Params PSRM: Pattern, String, Region or Match object getLocationFromPSRML(PSRML) Returns a new Location object derived from the given parameter. In case of PS, internally a find() is done inside this region. If found, the match‘s target offset position is returned. In case RM, just a copy of the given region‘s respective location (center or target offset) is returned. Params PSRML: Pattern, String, Region, Match or Location object }Screen
  61. 61. Sikuli Satish Gorripotu Screen is a class available in Sikuli Package. This class has several methods continuation to above methods. Since Screen extends class Region, all methods of class Region can be used with a screen object. Class Screen { Screen: Setting, Getting Attributes and Information Screen([id]) Create a new Screen object Parameters: id – an integer number indicating which monitor in a multi-monitor environment. Returns: a new screen object. It creates a new screen object, that represents the default/primary monitor (whose id is 0), if id is omitted. Numbers 1 and higher represent additional monitors that are available at the time, the script is running. Using numbers that do not represent an existing monitor, will stop the script with an error. So we may either use getNumberScreens() or exception handling, to avoid this. Note: If we want to access the default/primary monitor ( Screen(0) ) without creating a new screen object, use the constant reference SCREEN, that is initiated when your script starts: SCREEN=Screen(0). getNumberScreens() Get the number of screens in a multi-monitor environment at the time the script is running getBounds() Get the dimensions of monitor (a rectangle object) represented by the screen object. Capturing Capturing is the feature, that allows to grab a rectangle from a screenshot, to save it for later use. At each time, a capturing is initiated, a new screenshot is taken. There are two different versions: Screen.capture() saves the content of the selected rectangle in a file and returns its file name, Screen.selectRegion() returns the position and dimension of the selected rectangle. capture([region | rectangle | text]) capture(x, y, w, h) Returns the path to the file, where the captured image was saved. In interactive mode, the user may cancel the capturing, in which case None is returned. Parameters: region – an existing region object. rectangle – an existing rectangle object (e.g., as a return value of another region method). text – text to display in the middle of the screen in the interactive mode. x – x position of the rectangle to capture y – y position of the rectangle to capture w – width of the rectangle to capture h – height of the rectangle to capture
  62. 62. Sikuli Satish Gorripotu Interactive Mode: The script enters the screen-capture mode like when clicking the button in the IDE, enabling the user to capture a rectangle on the screen. If no text is given, the default ―Select a region on the screen‖ is displayed. selectRegion([text]) Select a region on the screen interactively. Returns a new Region object or None, if the user cancels the capturing process Text is displayed for about 2 seconds in the middle of the screen. If text is omitted, the default ―Select a region on the screen‖ is displayed. The interactive capture mode is entered and allows the user to select a region the same way as using the selection tool in the IDE. Parameters: text – Text to display in the middle of the screen. } Multi-Monitor Environments If more than one monitor is available, Sikuli is able to manage regions and click points on these monitors. getNumberScreens() returns the number of available screens. getBounds() returns the rectangle covered by the default/primary monitor. Screen.getBounds() returns the rectangle covered by a screen object created using Screen(id). The monitor, that is the first one based on hardware mapping (e.g. the laptop monitor), will always be Screen(0). In the Windows settings it is possible to place the taskbar on one of the secondary monitors, which makes it the primary monitor getting the base coordinates (0,0). The other available monitors are mapped around based on your settings. But the Sikuli internal mapping is not changed, so the primary monitor might be any of your Screen() objects. Sikuli takes care for that and maps SCREEN always to the primary monitor (the one with the (0,0) coordinates). So for example you have a laptop with an external monitor, that shows the taskbar (is primary monitor): SCREEN maps to Screen(1) Screen(0) is your laptop monitor With its rectangle, a screen object is always identical with the monitor it was created using Screen(id). Using Region.setROI() to restrict the region of interest for find operations has no effect on the base rectangle of the screen object. On the other hand region objects and location objects can be positioned anywhere in the coordinate system. Only when a find operation or a click action has to be performed, the objects rectangle or point has to be inside the rectangle of an existing monitor (basically repersented by

×