Sikuli

Sikuli Satish Gorripotu

ome

SIKULI
SATISH GORRIPOTU
Gsatish_leela@yahoo.com

9885121381


TOPICS AND STATUS

Topic Comments Page Number
Sikuli Documented
Sikuli and Other Supporting Softwares Required Documented
Installing/Configuring JDK or JRE / Setting Class Path Documented
Installing Sikuli Documented
Starting Sikuli IDE Documented
Stop the script during run time: Documented
Sikuli Working Architecture: Documented
Check Box: Documented
dragDrop: Documented
Sliders Documented
Testing on Skype Application: Documented
Meeting Request Arrival Notifier: Documented
Images Dictionary/Repository: Documented
Open any application in run time. Documented
Empty Recycle Bin: Documented
Run Sikuli from Command Line: Documented
Create Unit Testing Scripts for GUI: Documented
Sikuli Script in JAVA programs: Documented
Sikuli together with other IDE‘s - Eclipse: Documented
Import Packages/ Files Documented
Controlling Sikuli Scripts and their Behavior: Documented
Setting Documented
Controlling Applications and their Windows: Documented
App Documented
Interacting with the User Documented
Listening to Global Hotkeys Documented
General Settings and Access to Environment Information Documented
Region Documented
Extending a Region Documented
Finding inside a Region and Waiting for a Visual Event Documented
Observing Visual Events in a Region Documented
SikuliEvent Documented
Low-level Mouse and Keyboard Actions Documented
Exception FindFailed Documented
Screen Documented
Match Documented
Pattern Documented
Finder Documented
Key Documented
Constants related to Mouse and Keyboard Actions Documented


Extensions Documented
Summary Documented
Java Docs Documented
Jython Basics Documented
References Documented

Excel Handling in Sikuli Yet to Complete
Object Repository Yet to Complete
Handling Synchronization Problems Yet to Complete
Methods that are mostly used Yet to Complete
Maintaining the Test Results Yet to Complete
Frameworks Yet to Complete
Database Testing using Sikuli Yet to Complete
Sending the test results to email by using Sikuli Yet to Complete
Challenges faced Yet to Complete


Sikuli

Sikuli is an open-source research project. This is a visual technology to automate and test graphical
user interfaces (GUI) using images (screenshots).
It includes
1. Sikuli Script,
2. Visual scripting API for Jython
3. Sikuli IDE.
Sikuli Script automates anything you see on the screen without internal API's support.
We can programmatically control a web page, a Windows/Linux/Mac OS X desktop application, or
even an iphone or android application running in a simulator or via VNC.

Download and unzip the Sikuli Using following link

http://sikuli.org/download.shtml

Sikuli and Other Supporting Softwares Required

1. Sikuli-X-1.0rc2-win32.exe

2. Java Runtime Environment (JRE) 6+

Make sure to have only one instance of each software is available on machine.
You are provided with Admin access to configure the paths in the Environmental variables.

Installing/Configuring JDK or JRE / Setting Class Path

Install the JDK package and update the JAVA_HOME environment variable (User) in the Advanced
tab. Of System Properties. In system variable edit 'PATH' by adding C:Program
FilesJavajdk1.6.0_14bin;
I . Right click on my computer->Properties->Advanced->Environment variables
II .
1. Click on "New" in system variables
2. Create a new variable as shown below
Variable name = JAVA_HOME
Variable value = C:Program FilesJavajdk1.6.0_14 // [path of jdk is installed]
III. Select the existing path variable and edit it as shown below
PATH = % JAVA_HOME%bin; % JAVA_HOME%lib;
IV. Click on OK


V . START-> RUN ->type CMD
VI. Now enter "javac" to check whether the java correctly installed or not.


Installing Sikuli

Sikuli-X-1.0rc2-win32.exe is self extracting file. Just double click and click next … next … until
finish whilst follow the instructions.
1. It will automatically install the software.
2. It will configure the environment variables.
3. In desk top or program files we can see the short cut for Sikuli IDE as

Starting Sikuli IDE

Double click on the Sikuli IDE will give the below IDE

There are two ways to capture the screen images

1. Click on in the IDE
2. Press a hot-key (Ctrl + Shift + 2)


Screen will look darker and freeze momentarily. The entire desktop becomes like a canvas where we
can draw a rectangle around the target to capture an image.
Now that image will be captured and inserted in IDE.

We can use Auto List command capture to enable the basic commands in the IDE
instead of typing the commands manually.
1. Find
a. find
b. find all
c. wait
d. waitvanish
e. exists

2. Mouse Actions
a. Click
b. doubleClick
c. rightClick
d. hover
e. dragDrop

3. Keyboard Actions
a. type
b. paste

4. Event Observation
a. onAppear
b. OnVanish
c. onChange
d. observe

Below script will open the Keane web site ―www.keane.com‖ and search with the keyword ―Testing‖.


Type:
type ("http://sikuli.org" + Key.ENTER)

miscellanous keys
ENTER - TAB - ESC - BACKSPACE - DELETE - INSERT
function keys
F1 - F2 - F3 - F4 - F5 - F6 - F7 - F8 - F9 - F10 - F11 - F12 - F13 - F14 - F15
navigation keys
HOME - END - LEFT - RIGHT - DOWN - UP - PAGE_DOWN - PAGE_UP
key modifiers
ALT - CMD - CTRL - META - SHIFT - WIN


Stop the script during run time:
Press alt-shift-c


Sikuli Working Architecture:

Sikuli Script

Sikuli Script is a Jython and Java library that automates GUI interaction using image patterns to direct
keyboard/mouse events. The core of Sikuli Script is a Java library that consists of two parts:

java.awt.Robot, which delivers keyboard and mouse events to appropriate locations, and a C++ engine
based on OpenCV, which searches given image patterns on the screen. The C++ engine is connected to
Java via JNI and needs to be compiled for each platform.

On top of the Java library, a thin Jython layer is provided for end-users as a set of simple and clear
commands. Therefore, it should be easy to add more thin layers for other languages running on JVM, e.g.
JRuby, Scala, Javascript, etc.

The Structure of a Sikuli source/executable script (.sikuli, .skl)
A Sikuli script (.sikuli) is a directory that consists of a Python source file (.py), and all the image files
(.png) used by the source file.
All images used in a Sikuli script are simply a path to the .png file in the .sikuli bundle. Therefore, the
Python source file can also be edited by any text editor.
While saving a script using Sikuli IDE, an extra HTML file is also created in the .sikuli directory so that
users can share the scripts on the web easily.
A Sikuli executable script (.skl) is simply a zipped file of all files in the .sikuli directory. When a script is
passed to Sikuli IDE as a command line argument, Sikuli IDE recognizes its type by check its filename
extension. If a .skl is seen, Sikuli IDE runs it without showing the IDE window. If a .Sikuli is seen, Sikuli
IDE opens it in a source code editor.


Sikuli IDE
Sikuli IDE edits and runs Sikuli source scripts. Sikuli IDE integrates screen capturing and a custom text
editor (SikuliPane) to optimize the usability of writing a Sikuli script.

To show embedded images in the SikuliPane, all string literals that ends with ".png" are replaced by a
custom JButton object, ImageButton. If a user adjusts the image pattern's similarity, a Pattern() is
automatically constructed on top of the image.

To execute a Sikuli script, Sikuli IDE creates a org.python.util.PythonInterpreter and automatically passes
a few lines of headers (e.g. to import Sikuli's Jython modules, and to set the path to .sikuli directory) to
the interpreter.
Once these headers are set, the .py script is simply executed by
PythonInterpreter.execfile().

Working with Check Box:
1. Check the all check boxes:
findAll() : searches the screen for all the matching visual patterns and returns a list of locations.
Then we can use click() function to check the all check box.

2. Check the particular check box:


Step 1: we have to select the region

Step 2: Then identify check box

Below is the code for that.


dragDrop:

The above script will drag and drop all the word files into specified folder.


Working with Sliders
Same procedure as check boxes. But we can use dragDrop() function to move slide.

Way 1:

Way 2:
Need to work with Coordinates. Moving to Null. Need to resolve by Ram Raju.
Find Below Screen shot. There x, Y are the coordinates.
Main Scenario is: Zoom in and Zoom Out of the the word document.


hover(find( ).nearby(200).right().find( ).below().find( ).getCenter().offset(3,18))


Testing on Skype Application:

Now status of the ―Family – Bujji‖ is . We have to wait up to status as .

We can use any of the below way to

Way 1:

while not exists( ):


sleep(5)

Way 2:

wait( ,FOREVER)

Way 3:

wait( ,3600)#wait for an hour

Way 4:

while exists( ):
sleep(5)

Way 5:

waitVanish( , FOREVER)

popup(―Oyee Satti !!!!!! Your Wife Online‖)


Meeting Request Arrival Notifier:
Define a region and focus only on that region to search for a particular visual pattern. So tool don‘t have to
scan the whole screen and we can resize the browser to show just that portion of the map, while leaving a
lot of screen space to do something else.
To do so, click on the ―Create region‖ button.

The entire screen will freeze and turn darker, similar to what happen in the screen capture mode. Simply
draw a rectangle to cover the entire neighborhood map. The region covered by this rectangle is visually
represented as a thumbnail image of the entire desktop where the region is shaded in red.


Next, we capture the screenshot image of the meeting symbol so that we can tell Sikuli Script to watch for

its appearance.
We can write the following script to check the mails and watch for the appearance of a meeting Request
symbol:

Way 1:

Way 2:

Images Dictionary/Repository:

1. Store Images as Predefined.


Filename = VDict()

Filename [ ]= 'Recycle'

Filename [ ] = 'Skype'

2. Store Images in runtime
img = capture()
Filename [img] = input("Please enter Name to Captured image")

3. Retrieve name of the Image on popup.
Way 1:
query = capture()
gotit = Filename[query][0]
popup(gotit)

Way 2:

popup("select a icon in the ")
query = capture()
Result = Filename [query]
if result:
popup(Result[0])
else:
popup("unknown Image")

Open any application in run time.

setThrowException(True)
setAutoWaitTimeout(10000)
switchApp("C:WINDOWSsystem32calc.exe")


Empty Recycle Bin:

Run Sikuli from Command Line:
PATH-TO-SIKULI/sikuli-ide.exe [options] or
PATH-TO-SIKULI/sikuli-ide.bat [options]

Command Line Options

Sikuli-IDE

--args <arguments>
specify the arguments passed to Jython‘s sys.argv

-h,--help
print this help message


usage:
Sikuli-IDE [--args <arguments>] [-h] [-r <sikuli-file>] [-s] [-t <sikuli-file>]
--args <arguments> specify the arguments passed to Jython's sys.argv
-h,--help print this help message
-r,--run <sikuli-file> run .sikuli or .skl file
-s,--stderr print runtime errors to stderr instead of popping up a message box
-t,--test <sikuli-file> run .sikuli as a unit test case with junit's text UI runner

-r,--run <sikuli-file>
run .sikuli or .skl file

-s,--stderr
print runtime errors to stderr instead of popping up a message box

-t,--test <sikuli-file>
run a .sikuli (.skl not possible) as a unit test with junit‘s text UI runner (A script that can also be
run in IDE‘s Unit Test View).

Example - Run xxxx.sikuli with three arguments:
PATH-TO-SIKULIsikuli-ide.exe -r xxxx.sikuli —args a1 a2 a3

Running sikuli-script.jar from Command line

Run a script without IDE
java -jar %SIKULI_HOME%sikuli-script.jar path-to-your-scriptyourScript.sikuli

Run a script from Command line using the Sikuli contained Jython

This option might be helpful in some cases, where you want to have access to the Jython layer,
before any Sikuli feature is touched. Be aware, that a from sikuli.Sikuli import * is needed to have
access to Sikuli. You might have to take care somehow, that images are found, since bundle path is
not set.

set SCRIPT=path-to-your-scriptyourScript.sikuliyourScript.py

java -cp %SIKULI_HOME%sikuli-script.jar org.python.util.jython %SCRIPT%

Interactive Sikuli Jython Session from Command Line (Option -i)

sikuli-script.jar

-i
Start an interactive Jython session with the Sikuli environment already in place.

This might be helpful to do some testing on commandline, without having to start the Sikuli IDE. A
specified script and other options will be ignored.

java -jar %SIKULI_HOME%sikuli-script.jar -i

Create Unit Testing Scripts for GUI:


Sikuli integrates with jUnit and supports unit testing for Graphical User Interfaces (GUI). The unit
testing panel can be opened by clicking the menu View ‣ Unit Test or by the hot key ⌘ -U on Mac
or Ctrl-U on Windows/Linux.

Sikuli IDE aims to minimize the effort of writing code. With Sikuli IDE, a Python class inherited
from junit.framework.TestCase is automatically generated to wrap your unit testing script.

A typical unit testing script consists of two constructing and destructing methods, setUp() and
tearDown(), and a bunch of methods named with a prefix test.

The basic structure of a script is given as following:

TestJEdit.sikuli

def setUp(self):
openApp("/Applications/jEdit.app")
wait( , 20000)
close = ( )
# if find(close):
# click(close)

def tearDown(self):
closeApp("jEdit.app")
untilNotExist( , 20000)

def test_textarea_add_del_by_key(self):
type("hello world")
assertExist( )
type("a",KEY_CTRL)
type("n")
assertNotExist( )

def test_textarea_add_del_by_menu(self):
type("hello world")
assertExist( )
click( )
click( )
type("n")
assertNotExist( )

def test_new_tab_by_key(self):


type("n",KEY_CTRL)

assertExist(Pattern( ).similar(0.95))

def test_new_tab_by_menu(self):
click( )
click( )


def test_new_tab_by_toolbar(self):

click( )


def _verify_find(self):
sleep(0.5)
type("hello")
type("n")

click(Pattern( ).similar(0.90))

type("b")

assertExist( )

def test_find_by_key(self):
type("a long hello worldn")
type("f",KEY_CTRL)
self._verify_find()

def test_find_by_menu(self):
click( )
click( )
self._verify_find()


def test_find_by_toolbar(self):

click( )
self._verify_find()

def test_textfield_on_toolbar(self):

assertExist( )
type("f",KEY_CTRL)

type("hello")

assertExist( )

def test_toolbar_print_dialog(self):

click( )

assertExist( )

click( )

def test_menu_submenu(self):
click( )
click( )

assertExist( )
type("test")

click( )
click( )


click( )

click( )
click( )
click( )
assertExist( )

def test_scrollbar(self):
for i in range(1,50):
type("line %dn" % i)

tail = Pattern( ).similar(0.92)
wait(tail, 10000)
assertExist(tail)

thumb_at_top = Pattern( ).similar(0.98)
assertNotExist(thumb_at_top)

dragDrop( , [0, -1000])
assertExist(thumb_at_top)
assertNotExist(tail)


dragDrop(Pattern( ).similar(0.90), [0, 1000])
assertExist(tail)

def test_quit(self):
type("test")
type("e",KEY_CTRL)
type("q",KEY_CTRL)

click( )



To run a unit testing script, you need to click on the Run button in the unit testing panel instead of
the ordinary button.
IMPORTANT: Before you try to run your script in this test mode the first time, it has to be saved.
Everytime you change something, you have to save it again, before the next test run.
Sikuli Script in JAVA programs:

The core of Sikuli Script is written in Java, which means you can use Sikuli Script as a standard
JAVA library in your program. This document lets you know how to do that.

After having installed Sikuli on your system, as recommended on the download page, you have to
do the following:

1. Get sikuli-script.jar from your Sikuli IDE installation path.
Sikuli-IDE/sikuli-script.jar

2. Make the native libraries available

3. Include sikuli-script.jar in the CLASSPATH of your Java project.

4. Import the Sikuli classes you need

You can simply ―import org.sikuli.script.*‖ or
import the classes you need. In most cases, you would need at least Region or Screen.
the package name was edu.mit.csail.uid.

Sikuli together with other IDE’s - Eclipse:
Install Python plugin (PyDev) and Jython
Since the top level language used by Sikuli script is Python, you need the Python plugin PyDev.
Different from NetBeans, though PyDev is prepared to use Jython as interpreter, you have to install
Jython on your own seperately.
So first install Jython, by downloading the version you want to use from the
http://www.jython.org/downloads.html
Sikuli currently is based on Jython 2.5.1, but you may choose Jython 2.5.2 as well. Install it
according to Jythons installation HowTo.
After installation, make a test from a commandline by typing jython.bat (Windows) or jython
(Mac/Linux) to open an interactive Jython session.
You might type the following (<enter> means pressing the enter/return key)
import os <enter>
import sys <enter>
for e in sys.path: print e <enter> <enter>
type exit() <enter> (to leave interactive Jython)


This shows the current Jython configuration of the Python path. Remember the place where Jython is
installed.
Now we install the PyDev plugin from inside Eclipse: Menu Help -> Eclipse Marketplace. Either
search it or find it on the Popular tab‘s lower part. Simply click the Install button and do what you are
asked to do ;-)
The next step is to tell Eclipse PyDev, where it can find the Jython interpreter. Navigate to the Eclipse
Preferences pane and open the category PyDev and inside the subcategory Interpreter-Jython. First
try to Auto Config by clicking the appropriate button. If this does not work, click the button New,
name the entry and navigate to the folder, where Jython was installed and select jython.jar. After
clicking ok, a window Selection needed might come up: click Select all and ok to finalize this step.
Other options with PyDev are available, but not relevant for our actual matter
http://pydev.org/manual_101_root.html

Configure for using Sikuli script features at runtime
Again we have a difference to Netbeans: The PyDev plugin does not allow library references to
folders inside of jar-files in the respective configuration dialog (it does not insert them to the Python
path), though Jython itself accepts them, when specified on the Python path. So if you want to run
and debug your script in Eclipse, you have to extract the folder Lib/sikuli from sikuli-script.jar

Menu File -> New -> Project -> PyDev -> (open sublist) -> PyDev Project -> click Next button.
On the configuration pane name your project, select Jython as Project type, grammar version 2.5
(higher Python language versions are not supported by Jython 2.5.x) and click the Finish button.
Your project is created. Add at least one you-name-it.py file to the source folder and put from
sikuli.Sikuli import * as the first line.
In the last step, we tell PyDev, where to find the Sikuli libraries.
Goto Menu Project -> Properties -> select category PyDev - PYTHONPATH and go to the tab
External Libraries. We need a reference to path-to/sikuli-script.jar and another one to the
extracted folder Lib containing the folder sikuli.
reference to path-to/sikuli-script.jar
o click button Add zip/jar/egg and select sikuli-script.jar from the Sikuli installation.

reference to the extracted folder Lib containing the folder sikuli
o click Add source folder and select the folder Lib in the place you had it extracted.
o this is not needed, if you have moved the extracted folder sikuli to a folder, that is
already on the Python path (e.g. jython-intallation/Lib/site-packages).
Now you are able to run your first script. Remember, that in every script including the main script,
that you are editing now, as the first line you need from sikuli.Sikuli import *, to have access to the
Sikuli features at runtime.


Everytime later on you might come back to the project‘s preferences with Project -> Properties.
Code Completion works from the start without any further configuration and even steps into the
Java classes where appropriate.

Configuration Eclipse:
1) Open the Eclipse by double clicking on eclipse.exe, Click on OK for workspace launcher. Click on
workbench, it will allow you to move inside the eclipse.

2) Click on File->New->Select Java Project.


3) Enter the Project name and then click on ―Configue JRE‘s‖


Improve the performance of Sikuli

from org.sikuli.script.natives import Vision
Vision.setParameter("MinTargetSize", 6)

Import Packages/ Files

All files must be in Sys path. So Sikuli will find the files automatically, if we give it as
from sikuli import *

Example:

myScriptPath = "c:SikuliFilesmyLibrary"

if not myScriptPath in sys.path: sys.path.append(myScriptPath)

# supposing there is a myLib.sikuli
import myLib

# supposing myLib.sikuli contains a function "def myFunction():"
myLib.myFunction() # makes the call

As Python load the file at once and will continue the library to entire script. If we modify the code in library
file, then Python wouldn‘t reflect the changes. So,

# instead of: import module


import module
reload(module)

# instead of: from module import *
import module
reload(module)

from module import *

myLib.myFunction()

or

myLib import * # which integrates all names from myLib into your current namespace.
myFunction() # We can use directly

Importing from the directory:

import os
# get the directory containing your running .sikuli
myPath = os.path.dirname(getBundlePath())
if not myPath in sys.path: sys.path.append(myPath)

# now you can import every .sikuli in the same directory
import myLib

Scripts in same directory are found automatically

# nothing else needed
# now you can import every .sikuli in the same directory
import myLib

load(jar-file) : with path or no path

if no path is specified, Sikuli first looks into the bundle (the Sikuli folder of the running script) and
then into the extensions folder.
Controlling Sikuli Scripts and their Behavior:

setShowActions(False | True)
True, when a script is run, Sikuli shows a visual effect (a blinking double lined red
circle) on the spot.

exit([value])
Stops the script gracefully at this point. The value is returned to the calling environment.

Setting is a class available in Sikuli Package.


This class has several methods
Class Setting {
1. ActionLogs
(True/False) will display the log messages in console
2. InfoLogs
(True/False) will display the Info log messages in console
3. DebugLogs
(True/False) will display the Debug log messages in console
4. MinSimilarity
Sikuli will search for the minimal similarity of the object in the screen. Like smart
Identification
5. MoveMouseDelay
Control the mouse move animation.

mmd = Settings.MoveMouseDelay # save default/actual value
click(image1) # implicitly wait 0.5 seconds before click
Settings.MoveMouseDelay = 3
click(image2) # give app 3 seconds time before clicking again
Settings.MoveMouseDelay = mmd # reset to original value

6. DelayAfterDrag

Specifies the waiting time after mouse down at the source location as a decimal value
(seconds).
7. DelayBeforeDrop
Specifies the waiting time before mouse up at the target location as a decimal value
(seconds).

Above DelayAfterDrag, DelayBeforeDrop can be used as alternative of Wait Time.

# you may wish to save the actual settings before
Settings.DelayAfterDrag = 1
Settings.DelayBeforeDrop = 1
Settings.MoveMouseDelay = 3
dragDrop(source_image, target_image)
# time for complete dragDrop: about 5 seconds + search times

8. SlowMotionDelay
Control the duration of the visual effect (seconds).
9. WaitScanRate
10.ObserveScanRate
Specify the number of times actual search operations are performed per second while waiting
for a pattern to appear or vanish.
Example: Region.wait(), Region.exists(), Region.waitVanish(), Region.observe()


def myHandler(e):
print "it happened"

# you may wish to save the actual settings before
Settings.ObserveScanRate = 0.2
onAppear(some_image, myHandler)
observe(FOREVER, background = True)
# the observer will look every 5 seconds
# since your script does not wait here, you
# might want to stop the observing later on ;-)

11.ObserveMinChangedPixels
The minimum size in pixels of a change to trigger a change event.

Ex: Region.onChange() without specifying this value. The default value is 50.
}

Controlling Applications and their Windows:

Note: backslashes and double apostrophes ―
In a Sikuli script in normal strings enclosed in ‖ (double apostrophes), these special characters and ‖
have to be escaped using a backslash, when you have them inside the string. So for one backslash you
need and for one ‖ you need ‖. In a string enclosed in ‗ (single apostrophes), a ‗ has to be ‘ and a ‖ is
taken as such.
To avoid any problems, it is recommended to use the raw string r'some text with and " ...', since there
is no need for escaping. This is especially useful, when you have to specify Windows path‘s or want to
setup command lines for use with App.open(), openApp(), os.popen or Jythons Subprocess module.
a fictive command line example:

cmd = r'c:Program Filesmyapp.exe -x "c:Some Placesome.txt" >..log.txt'
openApp(cmd)

Open an application : os.open --> openApp() –> App.open()
Switch to an application : switchApp() –> App.focus()
Close an application : closeApp() –> App.close()


openApp(Application) Deprecated. Now it is App.open()

This function may switch to an already opened application or may open a new instance of the application.

# opens command prompt (found through PATH)
openApp("cmd.exe")

# with parameters (no sense, only to show ;-)
openApp(r'cmd.exe /c start c:Program Filesmyapp.bat')

# opens Firefox (full path specified)
openApp("c:Program FilesMozilla Firefoxfirefox.exe")

switchApp(application) Deprecated. Now it is App.focus()

This function switches the focus to the specified application and brings its windows to the front.
The window is identified by the application string.
This string (needs not be an application‘s name) is used to search the title text of all the opened
windows for any part of the title matching the string.

Example:

# switches to an existing command prompt or starts a new one
switchApp("cmd.exe")

# opens a new browser window
switchApp("c:Program FilesMozilla Firefoxfirefox.exe")

# switches to the frontmost opened browser window (or does nothing if no browser window is
currently opened)
switchApp("mozilla firefox")

closeApp(application) Deprecated. Now it is App.close().

Close the specified application.

Parameters: application – the name of an application or part of a window title

This function closes the application indicated by the windows whose titles contain the string
application.

Example:

# closes an existing command prompt
closeApp("cmd.exe")

# does nothing, since text cannot be found in the window title
closeApp("c:Program FilesMozilla Firefoxfirefox.exe")

# stops firefox including all its windows


closeApp("mozilla firefox")

run(command)
Run command in the command line

Parameters: command – a command that can be run from the command line.

This function executes the command and the script waits for its completion.

Note: Windows: Sikuli‘s strategy on these systems in the moment is to rely on implicit or explicit path
specifications to find an application that has to be started.
Running ―applications‖ can either be identified using their PID (process ID) or by using the window titles.
So using a path specification will only switch to an open application, if the application internally handles
the ―more than one instance‖ situation‖.
We usually use App.open("c:Program FilesMozilla FirefoxFirefox.exe") to start Firefox. This might
open an additional window.
And you can use App.focus("Firefox") to switch to the front most Firefox window
Note: When specifying a path in a string, we have to use (double backslash) for each (backslash)
e.g. myPath = "c:Program FilesSikuli-IDELib" )
App is a class available in Sikuli Package.

This class has several methods.
We can use App class methods in two ways
1. class methods
2. instance methods

Class App {
1. Open
Open the specified application. It opens the specified application and brings its
window the front. Whether this operation switches to an already opened application or
opens a new instance of the application depends on the system and application

1. Class Method : open(application)
Usage : App.open("application-identifier")

2. Instance method : open() # App instance someApp was created before
Usage : myApp = App("application-identifier")
myApp.open()

2. Focus
Switch the focus to an application. Parameters are the name of an application or part of a
window title
1. Class Method : focus (application)
Usage : App.focus("application-identifier")


2. Instance method : open() # App instance someApp was created before
myApp. focus ()

3. Close
It closes the given application or the matching windows.
It does nothing if no opened window or running application can be found.
Whether the application itself is closed depends on whether all open windows are closed
Or a main window of the application is closed, that in turn closes all other opened windows.

1. Class Method : close (application)
Usage : App. close ("application-identifier")

2. Instance method : close () # App instance someApp was created before
myApp. close()

4. focusedWindow
Identify the currently focused or the front most window and switch to it.
Sikuli does not tell you, to which application this window belongs.

Region object representing the window or None if there is no such window.
This method always returns a region.
When there is no window opened on the desktop, the region may refer to a special window
such as the task bar or an icon in the system tray.

1. Class Method : focusedWindow (application)
Usage : App. focusedWindow ("application-identifier")

myApp. focusedWindow ()
Example:

# highlight the currently fontmost window for 2 seconds
App.focusedWindow().highlight(2)

# save the windows region before
firstWindow = App.focusedWindow()
firstWindow.highlight(2)

5. window([n])


Get the series of windows with the matching title.

1. Class Method : App(application).window([n])#an App instance is created on the fly.
Usage : App(application).window([n])

myApp.window([n])
Parameters: n – 0 or a positive integer number.
If ommitted, 0 is taken as default.
Returns : the region on the screen occupied by the window,
if such window exists and None if otherwise.
Example :

#opens a Firefox browser window and switches to the address field.
# using an existing window if possible
myApp = App("Firefox")
if not myApp.window(): # no window(0) - Firefox not open
App.open("c:Program FilesMozilla FirefoxFirefox.exe")
wait(2)
myApp.focus()
wait(1)
type("l", KEY_CTRL) # switch to address field

Afterwards, it focuses on the Firefox application, uses the window () method to obtain the region of
the front most window,
Applies some operations within the region,
And finally closes the window:

# using a new window
firefox = App.open("c:Program FilesMozilla FirefoxFirefox.exe");
wait(2)
firefox.focus()
wait(1)
# now your just opened new window should be the frontmost
with firefox.window(): # see the general notes below
# some actions inside the window(0)'s region
click("somebutton.png")
firefox.close() # close the window - stop the process

Below is another example that highlights all the windows of an application by looping through them

# not more than 100 windows should be open ;-)
myApp = App("Safari")
for n in range(100):
w = myApp.window(n)
if not w: break # no more windows
w.highlight(2) # window highlighted for 2 second


}

Interacting with the User

popup(text[, title])

Display a dialog box with an OK button and text as the message.
The script then waits for the user to click the OK button.

Parameters: text – text to be displayed as message
title – optional title for the messagebox
Example:

popup("Hi satish….n How are you")

input([text][, default])

Display a dialog box with an input field, a Cancel button, and an OK button.
The optional text can be displayed as a caption.
The script then waits for the user to click either the Cancel or the OK button.

Parameters: text – optional text to be displayed as message
default – optional preset text for the input field

Returns:
the text, contained in the input field, when the user clicked OK
None, if the user pressed the Cancel button

Example:

name = input("Please enter your name:")
name = input("Please enter your name to log in:", "anonymous") # a preset input text

When using the parameter default, the text input field will be pre-populated with the given text,
So we click OK/Cancel or edit the content of the input field


Listening to Global Hotkeys

Env.addHotkey(key, modifiers, handler)
Register the specified key + modifiers as a global hotkey.
When the hotkey is pressed, the specified function handler will be called.

Parameters:

key – a character or a constant value defined in Key.
modifiers – Key modifiers, which can be one or multiple constants defined in KeyModifier.

def openAppleMenu(event):
click(Any Image)
# When we pressed Ctrl+Alt+F1, click the top-left apple icon.
Env.addHotkey(Key.F1, KeyModifier.ALT+KeyModifier.CTRL, openAppleMenu)

Env.removeHotkey(key, modifiers)
Unregister the registered global hotkey key + modifiers.

Parameters:

key – a character or a constant value defined in Key.
modifiers – Key modifiers, which can be one or multiple constants defined in KeyModifier.

General Settings and Access to Environment Information

Sikuli Level

Sikuli internally uses the class Settings to store globally used settings.
Publicly available attributes accessed by using Settings.[name-of-an-attribute]
To get it‘s value and Settings.attribute = value to set it.

Jython/Python Level

We can use all settings defined in standard Python/Jython which are available in our system
environment.
The modules sys and time are already imported, so you can use their methods without the need for
an import statement.

sys.path may be one of the most valuable settings, since it is used by Python/Jython to locate
modules, that are referenced using import module.

myPath = "some-absolute-path"
if not myPath in sys.path:
sys.path.append(myPath)

Java Level

import java


# get a value
val = java.lang.System.getProperty("key-of-property")

# set a property's value
java.lang.System.getProperty("key-of-property", value)

Image Search Path

Sikuli maintains a list of locations to search for images when they are not found in the current
.sikuli folder (a.k.a. bundle path).
This list is maintained internally but can be inspected and modified.

getImagePath()
Get a list of paths where Sikuli will search for images.

# getImagePath() returns a Java array of unicode strings
imgPath = list(getImagePath()) # makes it a Python list
# to loop through
for p in imgPath:
print p

addImagePath(a-new-path)
Add a new path to the list of image search paths

removeImagePath(a-path-already-in-the-list)
Remove a path from the list of image search paths

Note: paths must be specified using the correct path separators as double blackslashes .

Default bundle path can accessed and modified.
setBundlePath(path-to-a-folder)
Set the path for searching images in all Sikuli Script methods.
Sikuli IDE sets this automatically to the path of the folder where it saves the script (.sikuli).

getBundlePath()
Get a string containing a path to a folder containing our images used for finding patterns.
Sikuli IDE sets this automatically to the path of the folder where it saves the script (.sikuli).
We can use this function to package our private files together with the script or to access the
picture files in the .sikuli bundles for other purposes.
Sikuli only gives access to the path name, We need other python modules for I/O or other
purposes.

Other Environment Information
Env.getOS()
Env.getOSVersion()

Env.getSikuliVersion()

Example:
if not Env.getSikuliVersion() == "X-1.0rc2":


print "This script needs Sikuli X-1.0rc2"
exit(1)

Env.getClipboard()
Get the content of the clipboard if it is text, otherwise an empty string.

Env.getClipboard().paste()
clipboard will contain what we just pasted.

Env.getClipboard().strip()
To get rid of surrounding white spaces.

Env.isLockOn(key-constant)
Get the current status ( on / off ) off the respective key. Only one key can be specified.

key-constant – one of the key constants Key.CAPS_LOCK, Key.NUM_LOCK, Key.SCROLL_LOCK

Env.getMouseLocation()
Get the current location of the mouse cursor.

Advanced Settings for Tuning Vision Algorithm

Vision.setParameter(param, value)
Vision.getParameter(param)
Get/Set the parameter param of the vision algorithm.

Parameters:
param – a string that indicates the parameter to get.
Returns:
The float value of the specified parameter.

MinTargetSize

MinTargetSize is the minimum image size to which Sikuli can resize.

Sikuli resizes the screen images to a smaller scale for faster matching. This scaling process speeds
up the matching process, but also increases the possibility of false matching.
The default value of MinTargetSize is 12,
< 12. May increase speed but may returns a match that is not what we expect.

Example Jython code.

from org.sikuli.script.natives import Vision
Vision.setParameter("MinTargetSize", 6) # the default is 12. Setting the size to a smaller
value would make the matching algorithm be faster.


Region

Region is a class available in Sikuli Package.
Class Region
{

class Region Region is a rectangular area on a screen.
It knows nothing about its visual content (windows, pictures, graphics, text, ...).
It only knows the position on the screen and its dimension.

Creating a Region, Setting and Getting Attributes

Region(x, y, w, h)
Region(region)
Region(Rectangle)


Parameters:
x – x position of a rectangle. #upper left corner of the screen (0, 0)
y – y position of a rectangle. #upper left corner of the screen (0, 0)
w – height of a rectangle.
h – width of a rectangle.
region – an existing region object.
rectangle – an existing object of Java class Rectangle

Returns: a new region object.

region can also be created in run-time using Region.selectRegion().

You can create a region based on another region. This just duplicates the region (new object).
Another way to create a region is to specify a rectangle object or to extend an existing region.

Note:

topLeft = Location(reg.x, reg.y) # equivalent to
topLeft = reg.getTopLeft()

theWidth = reg.w # equivalent to
theWidth = reg.getW()

reg.h = theWidth # equivalent to
reg.setH(theWidth)

We can use selectRegion() to interactively create a new region.

setX(number)
setY(number)
setW(number)
setH(number)

Set the respective attribute of the region to the new value.
This effectively moves the region around and/or changes its dimension.

moveTo(location)
Set the position of this region regarding it‘s top left corner to the given location (the x and y values
are modified).

Parameters : location – location object becomes the new top left corner
Returns : the modified region object

Example:
reg.moveTo(anotherLocation) # equivalent to
reg.setX(anotherLocation.x); reg.setY(anotherLocation.y)

setROI(x, y, w, h)
setROI(rectangle)
setRect(x, y, w, h)


setRect(rectangle)

Both methods do the same: setting position and dimension to new values.
setROI() is intended to restrict the search to a smaller area to speed up processing
setRect() should be used to redefine a region (which could be enlarging it).

Parameters: x, y, w, h (all) – the attributes of a rectangle
rectangle – a rectangle object

.
morphTo(region)
Set the position and dimension of this region to the corresponding values of the region given as
parameter.

Parameters:region – a region object
Returns: the modified region object

Note:
reg.morphTo(anotherRegion) # equivalent to
r = anotherRegion; reg.setX(r.x); reg.setY(r.y); reg.setW(r.w); reg.setH(r.h)

getX()
getY()
getW()
getH()
Get the respective attribute integer value of the region.

getCenter()
Get the center of the region or Location as an object.

getTopLeft()
getTopRight()
getBottomLeft()
getBottomRight()
Get the location object of the region‘s respective corner

getScreen()
Returns the screen object that contains this region.
This method makes sense in Multi Monitor Environments

getLastMatch()
getLastMatches()
Returns the best match as a Match object or one or more match objects as an Iterator
object respectively

setAutoWaitTimeout(seconds)
Set the maximum waiting time for all subsequent find operations.


This method enables all find operations to wait for the given pattern to appear until the specified
amount of time has elapsed. The default is 3.0 seconds.
This method is intended for users to override this default setting. Region.find() work like
Region.wait(), without being able to set an individual timeout value for a specific find operation.

getAutoWaitTimeout()
Get the current value of the maximum waiting time for find operations.

Extending a Region

Note: In all cases the current region remains unchanged.
In all cases the new region does not extend beyond any boundary of the screen that contains
the given region.

offset(location)
Returns a new Region object, whose upper left corner is relocated adding the location‘s x and y
value to the respective values of the given region. Width and height are the same. So this clones a
region at a different place.

Parameters : location – a Location object
Returns :a new Region object

Example:
new_reg = reg.offset(Location(xoff, yoff)) # same as
new_reg = Region(reg.x + xoff, reg.y + yoff, reg.w, reg.h)

inside()
Returns the same Region object. Retained for upward compatibility.
region.inside().find() is totally equivalent to region.find().

nearby([range])
Returns a new Region object that includes the nearby neighbourhood of the the current region.
The new region is defined by extending the current region‘s dimensions in all directions by range
number of pixels.
The center of the new region remains the same.


Parameters: range – a positive integer indicating the number of pixels (default = 50).

above([range])
Returns a new Region object that is defined above the current region‘s top border with a height of
range number of pixels.
It does not include the current region.
If range is omitted, it reaches to the top of the screen. The new region has the same width and x-
position as the current region.

Parameters:range – a positive integer defining the new height

below([range])
Returns a new Region object that is defined below the current region‘s bottom border with a height
of range number of pixels.
If range is omitted, it reaches to the bottom of the screen. The new region has the same width and
x-position as the current region.

Parameters:range – a positive integer defining the new height

left([range])
Returns a new Region object that is defined left of the current region‘s left border with a width of
range number of pixels.
If range is omitted, it reaches to the left border of the screen. The new region has the same height
and y-position as the current region.

Parameters:range – a positive integer defining the new width

right([range])
Returns a new Region object that is defined right of the current region‘s right border with a width
of range number of pixels. So it does not include the current region. If range is omitted, it reaches
to the right border of the screen. The new region has the same height and y-position as the current
region.

Parameters:range – a positive integer defining the new width

Finding inside a Region and Waiting for a Visual Event

Besides acting on visual objects, finding them is one of the core functions of Sikuli.

PS: means, that either a Pattern or a string (path to an image file or just plain text) can be used as
parameter.

find(PS)
Find a particular GUI element, which is seen as the given image or just plain text. The given
file name of an image specifies the element‘s appearance. It searches within the region and returns
the best match, which shows a similarity greater than the minimum similarity given by the pattern.

Parameters:PS – a Pattern object or a string (path to an image file or just plain text)
Returns: a Match object that contains the best match or fails if not found

findAll(PS)


Repeatedly find ALL instances of a pattern, until no match can be found anymore, that meets the
requirements for a single Region.find() with the specified pattern.

Parameters:PS – a Pattern object or a string (path to an image file or just plain text)
Returns :one or more Match objects as an iterator object or fails if not found

By default, the returned matches are sorted by the similiarty.
If you need them ordered by their positions, we have to use Python‘s sorted function.

Example:
def by_y(match):
return match.y

icons = findAll(―Image screen shot‖)
# sort the icons by their y coordinates and put them into a new variable sorted_icons
sorted_icons = sorted(icons, key=by_y)
# another shorter version is using lambda.
sorted_icons = sorted(icons, key=lambda m:m.y)
for icon in sorted_icons:
pass # do whatever you want to do with the sorted icons

wait([PS][, seconds])
a Match object that contains the best match or fails if not found

Parameters:
PS – a Pattern object or a string (path to an image file or just plain text)
seconds – a number, which can have a fraction, as maximum waiting time in seconds.
If not specified, the auto wait timeout value set by Region.setAutoWaitTimeout() is used.
Use the constant FOREVER to wait for an infinite time.

If PS is not specified, the script just pauses for the specified amount of time. It is still possible to
use sleep(seconds) instead, but this is deprecated.

If PS is specified, it keeps searching the given pattern in the region until the image appears ( would
have been found with Region.find()) or the specified amount of time has elapsed. At least one find
operation is performed, even if 0 seconds is specified.)

waitVanish(PS[, seconds])
Wait until the give pattern PS in the region vanishes. Returns True if the pattern vanishes within
the specified waiting time, or False if the pattern stays visible after the waiting time has elapsed.

This method keeps searching the given pattern in the region until the image vanishes (can not be
found with Region.find() any longer) or the specified amount of time has elapsed.
At least one find operation is performed, even if 0 seconds is specified

Parameters:
seconds – a number, which can have a fraction, as maximum waiting time in seconds.
The internal granularity is milliseconds.
If not specified, the auto wait timeout value set by Region.
setAutoWaitTimeout() is used. Use the constant FOREVER to wait for an infinite time.


.

exists(PS[, seconds])
Check whether the give pattern is visible on the screen. And returns a Match object that contains
the best match. None is returned, if nothing is found within the specified waiting time

Does exactly the same as Region.wait(), but no exception is raised in case of FindFailed.
It can be used to symplify scripting in case that we want to know wether something is there or not
to decide how to proceed in your workflow.
It is typically used with an if statement. At least one find operation is performed, even if 0 seconds
is specified.

Parameters:
seconds –
a number, which can have a fraction, as maximum waiting time in seconds. The
internal granularity is milliseconds.
If not specified, the auto wait timeout value set by Region.setAutoWaitTimeout() is
used. Use the constant FOREVER to wait for an infinite time.

Observing Visual Events in a Region

handler: as a parameter in the following methods, you have to specify the name of a function,
which will be called by the observer, in case the observed event happens.
The function itself has to be defined in your script before using the method that references the
function.
The existance of the function will be checked before starting the script. So to get your script
running, you have to have at least the following statements in your script:

def myHandler(event): # you can choose any valid function name
# event: can be any variable name, it references the SikuliEvent object
pass # add your statements here

onAppear("path-to-an-image-file", myHandler) # or any other onEvent()
observe(10) # observes for 10 seconds

onAppear(PS, handler)

With the given region you register an observer, that should wait for the pattern to be there or to
appear and is activated with the next call of observe().
In the moment the internal find operation on the given pattern is successful during observation,
your handler is called and the observation is paused until you return from your handler.

Parameters: PS – a Pattern object or a string (path to an image file or just plain text.
handler – the name of a handler function in the script


onVanish(PS, handler)

With the given region you register an observer, that should wait for the pattern to be not there or
to vanish and is activated with the next call of observe().
In the moment the internal find operation on the given pattern fails during observation, your
handler is called and the observation is paused until you return from your handler.

Parameters: PS – a Pattern object or a string (path to an image file or just plain text.

onChange([minChangedSize], handler)

Parameters: minChangedSize – the minimum size in pixels of a change to trigger a change event.
If omitted: 50 is used

minChangedSize

With the given region we register an observer, that should wait for the visual content of the given
region to change and is activated with the next call of observe().
In the moment the visual content changes during observation, our handler is called and the
observation is paused until you return from your handler.

Below example highlights all changes in an observed region.

def changed(event):
print "something changed in ", event.region
for ch in event.changes:
ch.highlight() # highlight all changes
sleep(1)
for ch in event.changes:
ch.highlight() # turn off the highlights
with selectRegion("select a region to observe") as r:
# any change in r larger than 50 pixels would trigger the changed function
onChange(50, changed)
observe(background=True)

wait(30) # another way to observe for 30 seconds
r.stopObserver()

observe([seconds][, background = False | True])
Begin observation within the region.
For each region object, only one observation can be running at a given time.


Parameters:

seconds – a number, which can have a fraction, as maximum observation time in seconds.
Omit it or use the constant FOREVER to tell the observation to run for an infinite time (or until
stopped by a call of stopObserve()).

background – a flag indicating whether observation is run in the background. when set to True, the
observation will be run in the background and processing of your script is continued immediately.
Otherwise the script is paused until the completion of the observation.

stopObserver()
Stop observation within the region.

This must be called on a valid region object. The source region of an observed visual event is
available as one of the attributes of the event parameter that is passed to the handler function
when the function is invoked.

Example: to stop observation within a handler function, call event.region.stopObserver() inside the
handler function.

def myHandler(event):
event.region.stopObserver() # stops the observation

onAppear("path-to-an-image-file", myHandler)
observe(FOREVER) # observes until stopped in handler

}

SikuliEvent
SikuliEvent is a class available in Sikuli Package.
Class SikuliEvent
{


When processing an observation in a region, a handler function is called, when one of the
registered events
Region.onAppear(),
Region.onVanish() or
Region.onChange()

One parameter, we have access to in handler function is an instance of Class SikuliEvent.
Following attributes of the event, might help to identify the cause of the event and act on the
resulting matches.

Usage: event.attribute
where event is the parameter from the definition of handler function: def myHandler(event):

type
SikuliEvent.Type.APPEAR,
SikuliEvent.Type.VANISH, or
SikuliEvent.Type.CHANGE.

pattern
The source pattern that triggered this event. This is only valid in APPEAR and VANISH events.

region
The source region of this event.

match
In an APPEAR event, match saves the top Match object that appeared in the observed region.
In a VANISH event, match saves the last Match object that was in the observed region but
vanished.

This attribute is not valid in a CHANGE event.

changes
This attribute is valid only in a CHANGE event.

This changes attribute is a list of Match objects that changed and their sizes are at least
minChangedSize pixels.

}

Acting on a Region

PSMRL: which means, that either a Pattern object or a string (path to an image file or just plain
text) or a Match or a Region or a Location can be used as parameter.
P: pattern: a Pattern object. An implicit find operation is processed first. If successful, the center
of the resulting matches rectangle is the click point. If the pattern object has a target offset
specified, this is used as click point instead.
S: string: a path to an image file or just plain text. An implicit find operation with the default
minimum similarity 0.7 is processed first. If successful, the center of the resulting match object is
the click point.


M: match: a match object from a previous find operation. If the match has a target specified it is
used as the click point, otherwise the center of the matches rectangle.
R: region: a region object whose center is used as click point.
L: location: a location object which by definition represents a point on the screen that is used as
click point.

Region
Region is a class available in Sikuli Package.
This class has several methods continuation to above methods.
Class Region
{

click(PSMRL[, modifiers])
Perform a mouse click on the click point using the left button.

Parameters:
PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
modifiers – one or more key modifiers

Returns:
Number of performed clicks (actually 1). A 0 (integer null) means that because of some
reason, no click could be performed (in case of PS may be not Found).

doubleClick(PSMRL[, modifiers])
Perform a mouse double-click on the click point using the left button.

Parameters:

Returns:
Number of performed double-clicks (actually 1). A 0 (integer null) means that because of
some reason, no click could be performed (in case of PS may be not Found).

rightClick(PSMRL[, modifiers])
Perform a mouse click on the click point using the right button.

Parameters:

Returns:
Number of performed right cilicks (actually 1). A 0 (integer null) means that because of
some reason, no click could be performed (in case of PS may be not Found).

highlight([seconds])
Highlight the region for some period of time.

Parameters:seconds – a decimal number taken as duration in seconds

The region is highlighted showing a red colored frame around it.
If the parameter seconds is given, the script is suspended for the specified time.
If no time is given, the highlighting is started and the script continues.


When later on the same highlight call without a parameter is made, the highlighting is stopped
(behaves like a toggling switch).

Example:

m = find(some_image)

# the red frame will blink for about 7 - 8 seconds
for i in range(5):
m.highlight(1)
wait(0.5)

# a second red frame will blink as an overlay to the first one
m.highlight()
for i in range(5):
m.highlight(1)
wait(0.5)
m.highlight()

# the red frame will grow 5 times
for i in range(5):
m.highlight(1)
m = m.nearby(20)

hover(PSMRL)
Move the mouse cursor to hover above a click point.

Parameters:

Returns:
The number 1 if the mousepointer could be moved to the click point.
A 0 (integer null) returned if no move could be performed (in case of PS may be not Found).

dragDrop(PSMRL, PSMRL[, modifiers])
Perform a drag-and-drop operation from a starting click point to the target click point
indicated by the two PSMRLs respectively.

Parameters:

If one of the parameters is PS, the operation might fail due to not Found.

drag(PSMRL)
Start a drag-and-drop operation by dragging at the given click point.

Parameters:


Returns:
Number 1 if the operation could be performed.

dropAt(PSMRL[, delay])
Complete a drag-and-drop operation by dropping a previously dragged item at the given target
click point.

Parameters:
delay – time to wait after in seconds as decimal value

Returns:
Number 1 if the operation could be performed.

type([PSMRL], text[, modifiers])
Type the text at the current focused input field or at a click point specified by PSMRL.

Parameters:

Returns:
Number 1 if the operation could be performed,
0 (integer null) if it was not possible or the click couldn‘t be performed (in case of PS may
be not Found).

paste([PSMRL], text)
Paste the text at a click point.

Parameters:
Modifiers – one or more key modifiers

Returns:
Number 1 if the operation could be performed,
0 (integer null) if not possible or the click couldn‘t be performed (in case of PS may be not
Found).

Extracting Text from a Region

text()
Extract the text contained in the region using OCR.


Low-level Mouse and Keyboard Actions

mouseDown(button)
Press the mouse button down.

Parameters:
button – one or a combination of the button constants
Button.LEFT,
Button.MIDDLE,
Button.RIGHT.

Returns:
Number 1 if the operation is performed successfully, and zero if otherwise.

mouseUp([button])
Release the mouse button previously pressed.

Parameters:
button – one or a combination of the button constants
Button.LEFT,
Button.MIDDLE,
Button.RIGHT.
Returns:
Number 1 if the operation is performed successfully, and zero if otherwise.

mouseMove(PSRML)
Move the mouse pointer to a location indicated by PSRML.

Parameters:


Returns:
Number 1 if the operation could be performed. If using PS (which invokes an implicity find
operation), find fails and you have switched off FindFailed exception, a 0 (integer null) is returned.
Otherwise, the script is stopped with a FindFailed exception.

wheel(PSRML, WHEEL_DOWN | WHEEL_UP, steps)
Move the mouse pointer to a location indicated by PSRML and turn the mouse wheel in the
specified direction by the specified number of steps.

Parameters:
WHEEL_DOWN|WHEEL_UP – one of the two constants denoting the wheeling direction.
steps – an integer indicating the amoung of wheeling.

keyDown(key | list-of-keys)
Press and hold the specified key(s) until released by a later call to Region.keyUp().

Parameters:
key|list-of-keys – one or more keys (use the consts of class Key).
A list of keys is a concatenation of several key constants using ―+‖.

Returns:
Number 1 if the operation could be performed and 0 if otherwise.

keyUp([key | list-of-keys])
Release given keys. If no key is given, all currently pressed keys are released.

Parameters:
key|list-of-keys – one or more keys (use the consts of class Key).
A list of keys is a concatenation of several key constants using ―+‖.

Returns:
Number 1 if the operation could be performed and 0 if otherwise.

Exception FindFailed

These are the possibilities to handle ―not found‖ situations:

•generally abort a script, if not handled with try: ... except: ...
(the default setting or using setThrowException(True) or setFindFailedResponse(ABORT))

•generally ignore all ―not found‖ situations
(using setThrowException(False) or setFindFailedResponse(SKIP)),

•want to be prompted in such a case
(using setFindFailedResponse(PROMPT))

•advise Sikuli to wait forever (be careful with that!)
(using setFindFailedResponse(RETRY))

Using PROMPT:


Example:

# --- nice and easy
if exists("path-to-image"): # no exception, returns None when not found
pass # it is there
else:
pass # we miss it

# --- using exception handling
# every not found in the try block will switch to the except block
try:
find("path-to-image")
pass # it is there
except FindFailed:
pass # we miss it

# --- using setFindFailedResponse
setFindFailedResponse(SKIP) # no exception raised, not found returns None
if find("path-to-image"):
setFindFailedResponse(ABORT) # reset to default
pass # it is there
else:
setFindFailedResponse(ABORT) # reset to default
pass # we miss it

# --- using setThrowException
setThrowException(False) # no exception raised, not found returns None
if find("path-to-image"):
setThrowException(True) # reset to default
pass # it is there
else:
setThrowException(True) # reset to default
pass # we miss it

setFindFailedResponse(ABORT | SKIP | PROMPT | RETRY)
For the specified region set the option, how Sikuli should handle ―not found‖ situations. The option
stays in effect until changed by another setFindFailedResponse().


Parameters:

ABORT – all subsequent find failed operations (explicit or implicit) will raise exception FindFailed
(which is the default when a script is started).

SKIP – all subsequent find operations will not raise exception FindFailed. Instead, explicit find
operations such as Region.find() will return None. Implicit find operations (action functions) such
as Region.click() will do nothing and return 0.

PROMPT – all subsequent find operations will not raise exception FindFailed. Instead you will be
prompted for the way to handle the situation (see using PROMPT). Only the current find operation
is affected by your response to the prompt.

RETRY – all subsequent find operations will not raise exception FindFailed. Instead, Sikuli will try
to find the target until it gets visible. This is equivalent to using wait( ... , FOREVER) instead of
find() or using setAutoWaitTimeout(FOREVER).

getFindFailedResponse()
Get the current setting in this region.

Returns:
ABORT or SKIP or PROMPT or RETRY

Usage:
val = getFindFailedResponse()
if val == ABORT:
print "not found will stop script with Exception FindFailed"
elif val == SKIP:
print "not found will be ignored"
elif val == PROMPT:
print "when not found you will be prompted"
elif val == RETRY:
print "we will always wait forever"
else:
print val, ": this is a bug :-("

Note: It is recommended to use set/getFindFailedResponse() instead of set/getThrowException()
since the latter ones might be deprecated in the future.

setThrowException(False | True)
By using this method you control, how Sikuli should handle not found situations in this region.

Parameters:
True – all subsequent find operations (explicit or implicit) will raise exception FindFailed (which is
the default when a script is started) in case of not found.

False – all subsequent find operations will not raise exception FindFailed. Instead, explicit find
operations such as Region.find() will return None. Implicit find operations (action functions) such
as Region.click() will do nothing and return 0.

getThrowException()

Returns:
True or False


Grouping Method Calls ( with Region: )

Instead of:
# reg is a region object
if not reg.exists(image1):
reg.click(image2)
reg.wait(image3, 10)
reg.doubleClick(image4)

you can group methods applied to the same region using Python‘s with syntax:
# reg is a region object
with reg:
if not exists(image1):
click(image2)
wait(image3, 10)
doubleClick(image4)

All methods inside the with block that have the region omitted are redirected to the region object
specified at the with statement.

Note for Sikuli X versions up to rc3

This should work for Regions and Region-like objects (Region, Screen, App.window()). Match
objects have to be made a Region explicitly using Region(some_match).

This is a general workaround when facing any problems using with some_region:.

Special Methods for Developers

getRegionFromPSRM(PSRM)
Returns a new Region object derived from the given parameter. In case of PS, internally a find() is
done inside this region. If found, the match is returned. In case RM, just a copy of the given region
is returned.

Params PSRM:
Pattern, String, Region or Match object

getLocationFromPSRML(PSRML)
Returns a new Location object derived from the given parameter. In case of PS, internally a find()
is done inside this region. If found, the match‘s target offset position is returned. In case RM, just a
copy of the given region‘s respective location (center or target offset) is returned.

Params PSRML:
Pattern, String, Region, Match or Location object

}
Screen


Screen is a class available in Sikuli Package.
Since Screen extends class Region, all methods of class Region can be used with a screen object.

Class Screen
{

Screen: Setting, Getting Attributes and Information

Screen([id])
Create a new Screen object

Parameters: id – an integer number indicating which monitor in a multi-monitor environment.
Returns: a new screen object.

It creates a new screen object, that represents the default/primary monitor (whose id is 0), if id is
omitted. Numbers 1 and higher represent additional monitors that are available at the time, the
script is running.

Using numbers that do not represent an existing monitor, will stop the script with an error.
So we may either use getNumberScreens() or exception handling, to avoid this.

Note: If we want to access the default/primary monitor ( Screen(0) ) without creating a new
screen object, use the constant reference SCREEN, that is initiated when your script starts:
SCREEN=Screen(0).

getNumberScreens()
Get the number of screens in a multi-monitor environment at the time the script is running

getBounds()
Get the dimensions of monitor (a rectangle object) represented by the screen object.

Capturing
Capturing is the feature, that allows to grab a rectangle from a screenshot, to save it for later use.
At each time, a capturing is initiated, a new screenshot is taken.

There are two different versions:
Screen.capture() saves the content of the selected rectangle in a file and returns its file name,
Screen.selectRegion() returns the position and dimension of the selected rectangle.

capture([region | rectangle | text])
capture(x, y, w, h)

Returns the path to the file, where the captured image was saved.
In interactive mode, the user may cancel the capturing, in which case None is returned.

Parameters:
region – an existing region object.
rectangle – an existing rectangle object (e.g., as a return value of another region method).
text – text to display in the middle of the screen in the interactive mode.
x – x position of the rectangle to capture
y – y position of the rectangle to capture
w – width of the rectangle to capture
h – height of the rectangle to capture


Interactive Mode: The script enters the screen-capture mode like when clicking the button in the
IDE, enabling the user to capture a rectangle on the screen. If no text is given, the default ―Select
a region on the screen‖ is displayed.

selectRegion([text])
Select a region on the screen interactively.
Returns a new Region object or None, if the user cancels the capturing process

Text is displayed for about 2 seconds in the middle of the screen. If text is omitted, the default
―Select a region on the screen‖ is displayed.

The interactive capture mode is entered and allows the user to select a region the same way as
using the selection tool in the IDE.

Parameters: text – Text to display in the middle of the screen.
}

Multi-Monitor Environments
If more than one monitor is available, Sikuli is able to manage regions and click points on these
monitors.

getNumberScreens() returns the number of available screens.
getBounds() returns the rectangle covered by the default/primary monitor.
Screen.getBounds() returns the rectangle covered by a screen object created using Screen(id).

The monitor, that is the first one based on hardware mapping (e.g. the laptop monitor), will always
be Screen(0). In the Windows settings it is possible to place the taskbar on one of the secondary
monitors, which makes it the primary monitor getting the base coordinates (0,0). The other
available monitors are mapped around based on your settings. But the Sikuli internal mapping is
not changed, so the primary monitor might be any of your Screen() objects. Sikuli takes care for
that and maps SCREEN always to the primary monitor (the one with the (0,0) coordinates). So for
example you have a laptop with an external monitor, that shows the taskbar (is primary monitor):

SCREEN maps to Screen(1)
Screen(0) is your laptop monitor

With its rectangle, a screen object is always identical with the monitor it was created using
Screen(id). Using Region.setROI() to restrict the region of interest for find operations has no effect
on the base rectangle of the screen object.

On the other hand region objects and location objects can be positioned anywhere in the
coordinate system. Only when a find operation or a click action has to be performed, the objects
rectangle or point has to be inside the rectangle of an existing monitor (basically repersented by


Screen(0), Screen(1), ...). When this condition is met, everything works as expected and known
from a single monitor system.

With finding and acting there are the following exceptions:

Point Outside: a click point is outside any monitor rectangle. The clickpoint will be mapped to the
edges or corners of the primary monitor according to the relative position:
to the edges if its x or y value is in the range of the respective edge (right, left, above, below)i
to the corners, if x and y are outside any range of any edge (left/above -> upper left corner, ...)
Region Outside: a region is completely outside any monitor
a click action is handled in the same way as Point Outside
a find operation will always fail
Region Partially Outside: a region is partially outside a monitor but not overlapping another
monitor
a find operation will be carried out only in the part of region within the bounds of the monitor,
excluding the area outside the monitor.
Region Across Monitors: a region lies across multiple monitors:
a find operation will be restricted to the region within the bounds of the monitor that has a smaller
id.

An interactive capture (the user is asked to select an image or a rectangle via Screen.capture() or
Screen.selectRegion()) will automatically be restricted to the monitor, where it was started.

A scripted capture using a rectangle or a region (i.e. Screen.capture( region | rectangle )), will be
handled accordingly:

Region Outside: no image is captured, None is returned
Region Partially Outside: the returned image will only cover the part inside the monitor
Region Across Monitors: the returned image will only cover the part inside the monitor with the
smallest id.

Location
Location is a class available in Sikuli Package.
Handle single points on the screen directly by its position (x, y).

Class Location
{

Location(x, y)
Returns a new location object representing the position (x,y) on the screen
Parameters: x – x position
y – y position

getX()
getY()
Get the x or y value of a location object


It is possible to get the values directly by location.x or location.y.
It is also possible to set these values directly by location.x = value or location.y = value.

setLocation(x, y)
Set the location of this object to the specified coordinates.

offset(dx, dy)
Get a new location which is dx and dy pixels away horizontally and vertically from the
current location.

above(dy)
Get a new location which is dy pixels vertically above the current location.

below(dy)
Get a new location which is dy pixels vertically below the current location.

left(dx)
Get a new location which is dx pixels horizontally to the left of the current location.

right(dx)
Get a new location which is dx pixels horizontally to the right of the current location.

}

Match
Match is a class available in Sikuli Package.
class Match extends class Region, all methods of class Region can be used with a match object.

An object of class Match represents the result of a successful find operation.
It has the rectangle dimension of the image, that was used to search.
It knows the point of its upper left corner on an existing monitor, where it was found.


Class Match
{

Creating a Match, Getting Attributes

A match object is created as the result of an explicit find operation. It can be saved in a variable
for later use with actions like click().

# m is a reference to a match object, if found
m = find(Some Image)
print m # message area: Match[10,0 30x22] score=1.00, target=center

Sikuli

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Sikuli

Similar to Sikuli (20)

Recently uploaded

Recently uploaded (20)

Sikuli