Sikuli
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share
Uploaded on

This Content is directly copied from http://sikuli.org/ site. ...

This Content is directly copied from http://sikuli.org/ site.

This is not an updated document.

All the credit goes to my friends
1. Sundar Ram Raju
2. Shiva Prasad Raju
for knowing me about this tool

Thanks to my manager Srinivasa Movva for encouraging me to learn this tool.

© Copyright 2009-2010 Sikuli.org • All Rights Reserved with Sikuli org

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
9,112
On Slideshare
9,108
From Embeds
4
Number of Embeds
3

Actions

Shares
Downloads
598
Comments
1
Likes
9

Embeds 4

http://www.docseek.net 2
http://www.docshut.com 1
https://twitter.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Sikuli Satish Gorripotu ome SIKULI SATISH GORRIPOTU Gsatish_leela@yahoo.com 9885121381
  • 2. Sikuli Satish Gorripotu TOPICS AND STATUS Topic Comments Page NumberSikuli DocumentedSikuli and Other Supporting Softwares Required DocumentedInstalling/Configuring JDK or JRE / Setting Class Path DocumentedInstalling Sikuli DocumentedStarting Sikuli IDE DocumentedStop the script during run time: DocumentedSikuli Working Architecture: DocumentedCheck Box: DocumenteddragDrop: DocumentedSliders DocumentedTesting on Skype Application: DocumentedMeeting Request Arrival Notifier: DocumentedImages Dictionary/Repository: DocumentedOpen any application in run time. DocumentedEmpty Recycle Bin: DocumentedRun Sikuli from Command Line: DocumentedCreate Unit Testing Scripts for GUI: DocumentedSikuli Script in JAVA programs: DocumentedSikuli together with other IDE‘s - Eclipse: DocumentedImport Packages/ Files DocumentedControlling Sikuli Scripts and their Behavior: DocumentedSetting DocumentedControlling Applications and their Windows: DocumentedApp DocumentedInteracting with the User DocumentedListening to Global Hotkeys DocumentedGeneral Settings and Access to Environment Information DocumentedRegion DocumentedExtending a Region DocumentedFinding inside a Region and Waiting for a Visual Event DocumentedObserving Visual Events in a Region DocumentedSikuliEvent DocumentedLow-level Mouse and Keyboard Actions DocumentedException FindFailed DocumentedScreen DocumentedMatch DocumentedPattern DocumentedFinder DocumentedKey DocumentedConstants related to Mouse and Keyboard Actions Documented
  • 3. Sikuli Satish GorripotuExtensions DocumentedSummary DocumentedJava Docs DocumentedJython Basics DocumentedReferences DocumentedExcel Handling in Sikuli Yet to CompleteObject Repository Yet to CompleteHandling Synchronization Problems Yet to CompleteMethods that are mostly used Yet to CompleteMaintaining the Test Results Yet to CompleteFrameworks Yet to CompleteDatabase Testing using Sikuli Yet to CompleteSending the test results to email by using Sikuli Yet to CompleteChallenges faced Yet to Complete
  • 4. Sikuli Satish Gorripotu Sikuli Sikuli is an open-source research project. This is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots). It includes 1. Sikuli Script, 2. Visual scripting API for Jython 3. Sikuli IDE. Sikuli Script automates anything you see on the screen without internal APIs support. We can programmatically control a web page, a Windows/Linux/Mac OS X desktop application, or even an iphone or android application running in a simulator or via VNC. Download and unzip the Sikuli Using following link http://sikuli.org/download.shtml Sikuli and Other Supporting Softwares Required 1. Sikuli-X-1.0rc2-win32.exe 2. Java Runtime Environment (JRE) 6+ Make sure to have only one instance of each software is available on machine. You are provided with Admin access to configure the paths in the Environmental variables. Installing/Configuring JDK or JRE / Setting Class Path Install the JDK package and update the JAVA_HOME environment variable (User) in the Advanced tab. Of System Properties. In system variable edit PATH by adding C:Program FilesJavajdk1.6.0_14bin; I . Right click on my computer->Properties->Advanced->Environment variables II . 1. Click on "New" in system variables 2. Create a new variable as shown below Variable name = JAVA_HOME Variable value = C:Program FilesJavajdk1.6.0_14 // [path of jdk is installed] III. Select the existing path variable and edit it as shown below PATH = % JAVA_HOME%bin; % JAVA_HOME%lib; IV. Click on OK
  • 5. Sikuli Satish Gorripotu V . START-> RUN ->type CMD VI. Now enter "javac" to check whether the java correctly installed or not.
  • 6. Sikuli Satish Gorripotu Installing Sikuli Sikuli-X-1.0rc2-win32.exe is self extracting file. Just double click and click next … next … until finish whilst follow the instructions. 1. It will automatically install the software. 2. It will configure the environment variables. 3. In desk top or program files we can see the short cut for Sikuli IDE as Starting Sikuli IDE Double click on the Sikuli IDE will give the below IDE There are two ways to capture the screen images 1. Click on in the IDE 2. Press a hot-key (Ctrl + Shift + 2)
  • 7. Sikuli Satish Gorripotu Screen will look darker and freeze momentarily. The entire desktop becomes like a canvas where we can draw a rectangle around the target to capture an image. Now that image will be captured and inserted in IDE.We can use Auto List command capture to enable the basic commands in the IDEinstead of typing the commands manually.1. Find a. find b. find all c. wait d. waitvanish e. exists2. Mouse Actions a. Click b. doubleClick c. rightClick d. hover e. dragDrop3. Keyboard Actions a. type b. paste4. Event Observation a. onAppear b. OnVanish c. onChange d. observeBelow script will open the Keane web site ―www.keane.com‖ and search with the keyword ―Testing‖.
  • 8. Sikuli Satish Gorripotu Type: type ("http://sikuli.org" + Key.ENTER) miscellanous keys ENTER - TAB - ESC - BACKSPACE - DELETE - INSERT function keys F1 - F2 - F3 - F4 - F5 - F6 - F7 - F8 - F9 - F10 - F11 - F12 - F13 - F14 - F15 navigation keys HOME - END - LEFT - RIGHT - DOWN - UP - PAGE_DOWN - PAGE_UP key modifiers ALT - CMD - CTRL - META - SHIFT - WIN
  • 9. Sikuli Satish Gorripotu Stop the script during run time: Press alt-shift-c
  • 10. Sikuli Satish GorripotuSikuli Working Architecture:Sikuli ScriptSikuli Script is a Jython and Java library that automates GUI interaction using image patterns to directkeyboard/mouse events. The core of Sikuli Script is a Java library that consists of two parts:java.awt.Robot, which delivers keyboard and mouse events to appropriate locations, and a C++ enginebased on OpenCV, which searches given image patterns on the screen. The C++ engine is connected toJava via JNI and needs to be compiled for each platform.On top of the Java library, a thin Jython layer is provided for end-users as a set of simple and clearcommands. Therefore, it should be easy to add more thin layers for other languages running on JVM, e.g.JRuby, Scala, Javascript, etc.The Structure of a Sikuli source/executable script (.sikuli, .skl)A Sikuli script (.sikuli) is a directory that consists of a Python source file (.py), and all the image files(.png) used by the source file.All images used in a Sikuli script are simply a path to the .png file in the .sikuli bundle. Therefore, thePython source file can also be edited by any text editor.While saving a script using Sikuli IDE, an extra HTML file is also created in the .sikuli directory so thatusers can share the scripts on the web easily.A Sikuli executable script (.skl) is simply a zipped file of all files in the .sikuli directory. When a script ispassed to Sikuli IDE as a command line argument, Sikuli IDE recognizes its type by check its filenameextension. If a .skl is seen, Sikuli IDE runs it without showing the IDE window. If a .Sikuli is seen, SikuliIDE opens it in a source code editor.
  • 11. Sikuli Satish GorripotuSikuli IDESikuli IDE edits and runs Sikuli source scripts. Sikuli IDE integrates screen capturing and a custom texteditor (SikuliPane) to optimize the usability of writing a Sikuli script.To show embedded images in the SikuliPane, all string literals that ends with ".png" are replaced by acustom JButton object, ImageButton. If a user adjusts the image patterns similarity, a Pattern() isautomatically constructed on top of the image.To execute a Sikuli script, Sikuli IDE creates a org.python.util.PythonInterpreter and automatically passesa few lines of headers (e.g. to import Sikulis Jython modules, and to set the path to .sikuli directory) tothe interpreter.Once these headers are set, the .py script is simply executed byPythonInterpreter.execfile().Working with Check Box:1. Check the all check boxes: findAll() : searches the screen for all the matching visual patterns and returns a list of locations. Then we can use click() function to check the all check box.2. Check the particular check box:
  • 12. Sikuli Satish Gorripotu Step 1: we have to select the region Step 2: Then identify check box Below is the code for that.
  • 13. Sikuli Satish GorripotudragDrop:The above script will drag and drop all the word files into specified folder.
  • 14. Sikuli Satish GorripotuWorking with Sliders Same procedure as check boxes. But we can use dragDrop() function to move slide.Way 1:Way 2: Need to work with Coordinates. Moving to Null. Need to resolve by Ram Raju.Find Below Screen shot. There x, Y are the coordinates.Main Scenario is: Zoom in and Zoom Out of the the word document.
  • 15. Sikuli Satish Gorripotuhover(find( ).nearby(200).right().find( ).below().find( ).getCenter().offset(3,18))
  • 16. Sikuli Satish GorripotuTesting on Skype Application:Now status of the ―Family – Bujji‖ is . We have to wait up to status as .We can use any of the below way toWay 1: while not exists( ):
  • 17. Sikuli Satish Gorripotu sleep(5)Way 2: wait( ,FOREVER)Way 3: wait( ,3600)#wait for an hourWay 4: while exists( ): sleep(5)Way 5: waitVanish( , FOREVER) popup(―Oyee Satti !!!!!! Your Wife Online‖)
  • 18. Sikuli Satish GorripotuMeeting Request Arrival Notifier:Define a region and focus only on that region to search for a particular visual pattern. So tool don‘t have toscan the whole screen and we can resize the browser to show just that portion of the map, while leaving alot of screen space to do something else.To do so, click on the ―Create region‖ button.The entire screen will freeze and turn darker, similar to what happen in the screen capture mode. Simplydraw a rectangle to cover the entire neighborhood map. The region covered by this rectangle is visuallyrepresented as a thumbnail image of the entire desktop where the region is shaded in red.
  • 19. Sikuli Satish GorripotuNext, we capture the screenshot image of the meeting symbol so that we can tell Sikuli Script to watch forits appearance.We can write the following script to check the mails and watch for the appearance of a meeting Requestsymbol:Way 1:Way 2:Images Dictionary/Repository: 1. Store Images as Predefined.
  • 20. Sikuli Satish Gorripotu Filename = VDict() Filename [ ]= Recycle Filename [ ] = Skype 2. Store Images in runtime img = capture() Filename [img] = input("Please enter Name to Captured image") 3. Retrieve name of the Image on popup. Way 1: query = capture() gotit = Filename[query][0] popup(gotit) Way 2: popup("select a icon in the ") query = capture() Result = Filename [query] if result: popup(Result[0]) else: popup("unknown Image")Open any application in run time. setThrowException(True) setAutoWaitTimeout(10000) switchApp("C:WINDOWSsystem32calc.exe")
  • 21. Sikuli Satish GorripotuEmpty Recycle Bin:Run Sikuli from Command Line: PATH-TO-SIKULI/sikuli-ide.exe [options] or PATH-TO-SIKULI/sikuli-ide.bat [options] Command Line Options Sikuli-IDE --args <arguments> specify the arguments passed to Jython‘s sys.argv -h,--help print this help message
  • 22. Sikuli Satish Gorripotu usage: Sikuli-IDE [--args <arguments>] [-h] [-r <sikuli-file>] [-s] [-t <sikuli-file>] --args <arguments> specify the arguments passed to Jythons sys.argv -h,--help print this help message -r,--run <sikuli-file> run .sikuli or .skl file -s,--stderr print runtime errors to stderr instead of popping up a message box -t,--test <sikuli-file> run .sikuli as a unit test case with junits text UI runner -r,--run <sikuli-file> run .sikuli or .skl file -s,--stderr print runtime errors to stderr instead of popping up a message box -t,--test <sikuli-file> run a .sikuli (.skl not possible) as a unit test with junit‘s text UI runner (A script that can also be run in IDE‘s Unit Test View). Example - Run xxxx.sikuli with three arguments: PATH-TO-SIKULIsikuli-ide.exe -r xxxx.sikuli —args a1 a2 a3 Running sikuli-script.jar from Command line Run a script without IDE java -jar %SIKULI_HOME%sikuli-script.jar path-to-your-scriptyourScript.sikuli Run a script from Command line using the Sikuli contained Jython This option might be helpful in some cases, where you want to have access to the Jython layer, before any Sikuli feature is touched. Be aware, that a from sikuli.Sikuli import * is needed to have access to Sikuli. You might have to take care somehow, that images are found, since bundle path is not set. set SCRIPT=path-to-your-scriptyourScript.sikuliyourScript.py java -cp %SIKULI_HOME%sikuli-script.jar org.python.util.jython %SCRIPT% Interactive Sikuli Jython Session from Command Line (Option -i) sikuli-script.jar -i Start an interactive Jython session with the Sikuli environment already in place. This might be helpful to do some testing on commandline, without having to start the Sikuli IDE. A specified script and other options will be ignored. java -jar %SIKULI_HOME%sikuli-script.jar -iCreate Unit Testing Scripts for GUI:
  • 23. Sikuli Satish Gorripotu Sikuli integrates with jUnit and supports unit testing for Graphical User Interfaces (GUI). The unit testing panel can be opened by clicking the menu View ‣ Unit Test or by the hot key ⌘ -U on Mac or Ctrl-U on Windows/Linux. Sikuli IDE aims to minimize the effort of writing code. With Sikuli IDE, a Python class inherited from junit.framework.TestCase is automatically generated to wrap your unit testing script. A typical unit testing script consists of two constructing and destructing methods, setUp() and tearDown(), and a bunch of methods named with a prefix test. The basic structure of a script is given as following: TestJEdit.sikuli def setUp(self): openApp("/Applications/jEdit.app") wait( , 20000) close = ( ) # if find(close): # click(close) def tearDown(self): closeApp("jEdit.app") untilNotExist( , 20000) def test_textarea_add_del_by_key(self): type("hello world") assertExist( ) type("a",KEY_CTRL) type("n") assertNotExist( ) def test_textarea_add_del_by_menu(self): type("hello world") assertExist( ) click( ) click( ) type("n") assertNotExist( ) def test_new_tab_by_key(self):
  • 24. Sikuli Satish Gorripotu type("n",KEY_CTRL) assertExist(Pattern( ).similar(0.95)) def test_new_tab_by_menu(self): click( ) click( ) assertExist(Pattern( ).similar(0.95)) def test_new_tab_by_toolbar(self): click( ) assertExist(Pattern( ).similar(0.95)) def _verify_find(self): sleep(0.5) type("hello") type("n") click(Pattern( ).similar(0.90)) assertExist(Pattern( ).similar(0.94)) type("b") assertExist( ) def test_find_by_key(self): type("a long hello worldn") type("f",KEY_CTRL) self._verify_find() def test_find_by_menu(self): type("a long hello worldn") click( ) click( ) self._verify_find()
  • 25. Sikuli Satish Gorripotu def test_find_by_toolbar(self): type("a long hello worldn") click( ) self._verify_find() def test_textfield_on_toolbar(self): type("a long hello worldn") assertExist( ) type("f",KEY_CTRL) assertExist(Pattern( ).similar(0.85)) type("hello") assertExist( ) def test_toolbar_print_dialog(self): click( ) assertExist( ) click( ) def test_menu_submenu(self): click( ) click( ) assertExist( ) type("test") click( ) click( )
  • 26. Sikuli Satish Gorripotu click( ) click( ) click( ) click( ) assertExist( ) def test_scrollbar(self): for i in range(1,50): type("line %dn" % i) tail = Pattern( ).similar(0.92) wait(tail, 10000) assertExist(tail) thumb_at_top = Pattern( ).similar(0.98) assertNotExist(thumb_at_top) dragDrop( , [0, -1000]) assertExist(thumb_at_top) assertNotExist(tail) assertExist(Pattern( ).similar(0.92)) dragDrop(Pattern( ).similar(0.90), [0, 1000]) assertExist(tail) def test_quit(self): type("test") type("e",KEY_CTRL) type("q",KEY_CTRL) click( )
  • 27. Sikuli Satish Gorripotu assertExist(Pattern( ).similar(0.90)) To run a unit testing script, you need to click on the Run button in the unit testing panel instead of the ordinary button. IMPORTANT: Before you try to run your script in this test mode the first time, it has to be saved. Everytime you change something, you have to save it again, before the next test run.Sikuli Script in JAVA programs: The core of Sikuli Script is written in Java, which means you can use Sikuli Script as a standard JAVA library in your program. This document lets you know how to do that. After having installed Sikuli on your system, as recommended on the download page, you have to do the following: 1. Get sikuli-script.jar from your Sikuli IDE installation path. Sikuli-IDE/sikuli-script.jar 2. Make the native libraries available 3. Include sikuli-script.jar in the CLASSPATH of your Java project. 4. Import the Sikuli classes you need You can simply ―import org.sikuli.script.*‖ or import the classes you need. In most cases, you would need at least Region or Screen. the package name was edu.mit.csail.uid.Sikuli together with other IDE’s - Eclipse: Install Python plugin (PyDev) and Jython Since the top level language used by Sikuli script is Python, you need the Python plugin PyDev. Different from NetBeans, though PyDev is prepared to use Jython as interpreter, you have to install Jython on your own seperately. So first install Jython, by downloading the version you want to use from the http://www.jython.org/downloads.html Sikuli currently is based on Jython 2.5.1, but you may choose Jython 2.5.2 as well. Install it according to Jythons installation HowTo. After installation, make a test from a commandline by typing jython.bat (Windows) or jython (Mac/Linux) to open an interactive Jython session. You might type the following (<enter> means pressing the enter/return key) import os <enter> import sys <enter> for e in sys.path: print e <enter> <enter> type exit() <enter> (to leave interactive Jython)
  • 28. Sikuli Satish Gorripotu This shows the current Jython configuration of the Python path. Remember the place where Jython is installed. Now we install the PyDev plugin from inside Eclipse: Menu Help -> Eclipse Marketplace. Either search it or find it on the Popular tab‘s lower part. Simply click the Install button and do what you are asked to do ;-) The next step is to tell Eclipse PyDev, where it can find the Jython interpreter. Navigate to the Eclipse Preferences pane and open the category PyDev and inside the subcategory Interpreter-Jython. First try to Auto Config by clicking the appropriate button. If this does not work, click the button New, name the entry and navigate to the folder, where Jython was installed and select jython.jar. After clicking ok, a window Selection needed might come up: click Select all and ok to finalize this step. Other options with PyDev are available, but not relevant for our actual matter http://pydev.org/manual_101_root.html Configure for using Sikuli script features at runtime Again we have a difference to Netbeans: The PyDev plugin does not allow library references to folders inside of jar-files in the respective configuration dialog (it does not insert them to the Python path), though Jython itself accepts them, when specified on the Python path. So if you want to run and debug your script in Eclipse, you have to extract the folder Lib/sikuli from sikuli-script.jar Menu File -> New -> Project -> PyDev -> (open sublist) -> PyDev Project -> click Next button. On the configuration pane name your project, select Jython as Project type, grammar version 2.5 (higher Python language versions are not supported by Jython 2.5.x) and click the Finish button. Your project is created. Add at least one you-name-it.py file to the source folder and put from sikuli.Sikuli import * as the first line. In the last step, we tell PyDev, where to find the Sikuli libraries. Goto Menu Project -> Properties -> select category PyDev - PYTHONPATH and go to the tab External Libraries. We need a reference to path-to/sikuli-script.jar and another one to the extracted folder Lib containing the folder sikuli. reference to path-to/sikuli-script.jar o click button Add zip/jar/egg and select sikuli-script.jar from the Sikuli installation. reference to the extracted folder Lib containing the folder sikuli o click Add source folder and select the folder Lib in the place you had it extracted. o this is not needed, if you have moved the extracted folder sikuli to a folder, that is already on the Python path (e.g. jython-intallation/Lib/site-packages). Now you are able to run your first script. Remember, that in every script including the main script, that you are editing now, as the first line you need from sikuli.Sikuli import *, to have access to the Sikuli features at runtime.
  • 29. Sikuli Satish Gorripotu Everytime later on you might come back to the project‘s preferences with Project -> Properties. Code Completion works from the start without any further configuration and even steps into the Java classes where appropriate.Configuration Eclipse:1) Open the Eclipse by double clicking on eclipse.exe, Click on OK for workspace launcher. Click onworkbench, it will allow you to move inside the eclipse.2) Click on File->New->Select Java Project.
  • 30. Sikuli Satish Gorripotu 3) Enter the Project name and then click on ―Configue JRE‘s‖
  • 31. Sikuli Satish GorripotuImprove the performance of Sikuli from org.sikuli.script.natives import Vision Vision.setParameter("MinTargetSize", 6)Import Packages/ FilesAll files must be in Sys path. So Sikuli will find the files automatically, if we give it as from sikuli import *Example: myScriptPath = "c:SikuliFilesmyLibrary" if not myScriptPath in sys.path: sys.path.append(myScriptPath) # supposing there is a myLib.sikuli import myLib # supposing myLib.sikuli contains a function "def myFunction():" myLib.myFunction() # makes the callAs Python load the file at once and will continue the library to entire script. If we modify the code in libraryfile, then Python wouldn‘t reflect the changes. So, # instead of: import module
  • 32. Sikuli Satish Gorripotu import module reload(module) # instead of: from module import * import module reload(module) from module import * myLib.myFunction() or myLib import * # which integrates all names from myLib into your current namespace. myFunction() # We can use directly Importing from the directory: import os # get the directory containing your running .sikuli myPath = os.path.dirname(getBundlePath()) if not myPath in sys.path: sys.path.append(myPath) # now you can import every .sikuli in the same directory import myLib Scripts in same directory are found automatically # nothing else needed # now you can import every .sikuli in the same directory import myLib load(jar-file) : with path or no path if no path is specified, Sikuli first looks into the bundle (the Sikuli folder of the running script) and then into the extensions folder.Controlling Sikuli Scripts and their Behavior: setShowActions(False | True) True, when a script is run, Sikuli shows a visual effect (a blinking double lined red circle) on the spot. exit([value]) Stops the script gracefully at this point. The value is returned to the calling environment.Setting is a class available in Sikuli Package.
  • 33. Sikuli Satish GorripotuThis class has several methodsClass Setting { 1. ActionLogs (True/False) will display the log messages in console 2. InfoLogs (True/False) will display the Info log messages in console 3. DebugLogs (True/False) will display the Debug log messages in console 4. MinSimilarity Sikuli will search for the minimal similarity of the object in the screen. Like smart Identification 5. MoveMouseDelay Control the mouse move animation. mmd = Settings.MoveMouseDelay # save default/actual value click(image1) # implicitly wait 0.5 seconds before click Settings.MoveMouseDelay = 3 click(image2) # give app 3 seconds time before clicking again Settings.MoveMouseDelay = mmd # reset to original value 6. DelayAfterDrag Specifies the waiting time after mouse down at the source location as a decimal value (seconds). 7. DelayBeforeDrop Specifies the waiting time before mouse up at the target location as a decimal value (seconds). Above DelayAfterDrag, DelayBeforeDrop can be used as alternative of Wait Time. # you may wish to save the actual settings before Settings.DelayAfterDrag = 1 Settings.DelayBeforeDrop = 1 Settings.MoveMouseDelay = 3 dragDrop(source_image, target_image) # time for complete dragDrop: about 5 seconds + search times 8. SlowMotionDelay Control the duration of the visual effect (seconds). 9. WaitScanRate 10.ObserveScanRate Specify the number of times actual search operations are performed per second while waiting for a pattern to appear or vanish. Example: Region.wait(), Region.exists(), Region.waitVanish(), Region.observe()
  • 34. Sikuli Satish Gorripotu def myHandler(e): print "it happened" # you may wish to save the actual settings before Settings.ObserveScanRate = 0.2 onAppear(some_image, myHandler) observe(FOREVER, background = True) # the observer will look every 5 seconds # since your script does not wait here, you # might want to stop the observing later on ;-) 11.ObserveMinChangedPixels The minimum size in pixels of a change to trigger a change event. Ex: Region.onChange() without specifying this value. The default value is 50.}Controlling Applications and their Windows:Note: backslashes and double apostrophes ―In a Sikuli script in normal strings enclosed in ‖ (double apostrophes), these special characters and ‖have to be escaped using a backslash, when you have them inside the string. So for one backslash youneed and for one ‖ you need ‖. In a string enclosed in ‗ (single apostrophes), a ‗ has to be ‘ and a ‖ istaken as such.To avoid any problems, it is recommended to use the raw string rsome text with and " ..., since thereis no need for escaping. This is especially useful, when you have to specify Windows path‘s or want tosetup command lines for use with App.open(), openApp(), os.popen or Jythons Subprocess module.a fictive command line example: cmd = rc:Program Filesmyapp.exe -x "c:Some Placesome.txt" >..log.txt openApp(cmd)Open an application : os.open --> openApp() –> App.open()Switch to an application : switchApp() –> App.focus()Close an application : closeApp() –> App.close()
  • 35. Sikuli Satish GorripotuopenApp(Application) Deprecated. Now it is App.open()This function may switch to an already opened application or may open a new instance of the application. # opens command prompt (found through PATH) openApp("cmd.exe") # with parameters (no sense, only to show ;-) openApp(rcmd.exe /c start c:Program Filesmyapp.bat) # opens Firefox (full path specified) openApp("c:Program FilesMozilla Firefoxfirefox.exe")switchApp(application) Deprecated. Now it is App.focus() This function switches the focus to the specified application and brings its windows to the front. The window is identified by the application string. This string (needs not be an application‘s name) is used to search the title text of all the opened windows for any part of the title matching the string. Example: # switches to an existing command prompt or starts a new one switchApp("cmd.exe") # opens a new browser window switchApp("c:Program FilesMozilla Firefoxfirefox.exe") # switches to the frontmost opened browser window (or does nothing if no browser window is currently opened) switchApp("mozilla firefox")closeApp(application) Deprecated. Now it is App.close(). Close the specified application. Parameters: application – the name of an application or part of a window title This function closes the application indicated by the windows whose titles contain the string application. Example: # closes an existing command prompt closeApp("cmd.exe") # does nothing, since text cannot be found in the window title closeApp("c:Program FilesMozilla Firefoxfirefox.exe") # stops firefox including all its windows
  • 36. Sikuli Satish Gorripotu closeApp("mozilla firefox")run(command) Run command in the command line Parameters: command – a command that can be run from the command line. This function executes the command and the script waits for its completion.Note: Windows: Sikuli‘s strategy on these systems in the moment is to rely on implicit or explicit pathspecifications to find an application that has to be started.Running ―applications‖ can either be identified using their PID (process ID) or by using the window titles.So using a path specification will only switch to an open application, if the application internally handlesthe ―more than one instance‖ situation‖.We usually use App.open("c:Program FilesMozilla FirefoxFirefox.exe") to start Firefox. This mightopen an additional window.And you can use App.focus("Firefox") to switch to the front most Firefox windowNote: When specifying a path in a string, we have to use (double backslash) for each (backslash)e.g. myPath = "c:Program FilesSikuli-IDELib" )App is a class available in Sikuli Package.This class has several methods.We can use App class methods in two ways 1. class methods 2. instance methodsClass App { 1. Open Open the specified application. It opens the specified application and brings its window the front. Whether this operation switches to an already opened application or opens a new instance of the application depends on the system and application 1. Class Method : open(application) Usage : App.open("application-identifier") 2. Instance method : open() # App instance someApp was created before Usage : myApp = App("application-identifier") myApp.open() 2. Focus Switch the focus to an application. Parameters are the name of an application or part of a window title 1. Class Method : focus (application) Usage : App.focus("application-identifier")
  • 37. Sikuli Satish Gorripotu 2. Instance method : open() # App instance someApp was created before Usage : myApp = App("application-identifier") myApp. focus () 3. Close It closes the given application or the matching windows. It does nothing if no opened window or running application can be found. Whether the application itself is closed depends on whether all open windows are closed Or a main window of the application is closed, that in turn closes all other opened windows. 1. Class Method : close (application) Usage : App. close ("application-identifier") 2. Instance method : close () # App instance someApp was created before Usage : myApp = App("application-identifier") myApp. close() 4. focusedWindow Identify the currently focused or the front most window and switch to it. Sikuli does not tell you, to which application this window belongs. Region object representing the window or None if there is no such window. This method always returns a region. When there is no window opened on the desktop, the region may refer to a special window such as the task bar or an icon in the system tray. 1. Class Method : focusedWindow (application) Usage : App. focusedWindow ("application-identifier") 2. Instance method : close () # App instance someApp was created before Usage : myApp = App("application-identifier") myApp. focusedWindow () Example: # highlight the currently fontmost window for 2 seconds App.focusedWindow().highlight(2) # save the windows region before firstWindow = App.focusedWindow() firstWindow.highlight(2) 5. window([n])
  • 38. Sikuli Satish Gorripotu Get the series of windows with the matching title. 1. Class Method : App(application).window([n])#an App instance is created on the fly. Usage : App(application).window([n]) 2. Instance method : close () # App instance someApp was created before Usage : myApp = App("application-identifier") myApp.window([n]) Parameters: n – 0 or a positive integer number. If ommitted, 0 is taken as default. Returns : the region on the screen occupied by the window, if such window exists and None if otherwise. Example : #opens a Firefox browser window and switches to the address field. # using an existing window if possible myApp = App("Firefox") if not myApp.window(): # no window(0) - Firefox not open App.open("c:Program FilesMozilla FirefoxFirefox.exe") wait(2) myApp.focus() wait(1) type("l", KEY_CTRL) # switch to address field Afterwards, it focuses on the Firefox application, uses the window () method to obtain the region of the front most window, Applies some operations within the region, And finally closes the window: # using a new window firefox = App.open("c:Program FilesMozilla FirefoxFirefox.exe"); wait(2) firefox.focus() wait(1) # now your just opened new window should be the frontmost with firefox.window(): # see the general notes below # some actions inside the window(0)s region click("somebutton.png") firefox.close() # close the window - stop the process Below is another example that highlights all the windows of an application by looping through them # not more than 100 windows should be open ;-) myApp = App("Safari") for n in range(100): w = myApp.window(n) if not w: break # no more windows w.highlight(2) # window highlighted for 2 second
  • 39. Sikuli Satish Gorripotu }Interacting with the User popup(text[, title]) Display a dialog box with an OK button and text as the message. The script then waits for the user to click the OK button. Parameters: text – text to be displayed as message title – optional title for the messagebox Example: popup("Hi satish….n How are you") input([text][, default]) Display a dialog box with an input field, a Cancel button, and an OK button. The optional text can be displayed as a caption. The script then waits for the user to click either the Cancel or the OK button. Parameters: text – optional text to be displayed as message default – optional preset text for the input field Returns: the text, contained in the input field, when the user clicked OK None, if the user pressed the Cancel button Example: name = input("Please enter your name:") name = input("Please enter your name to log in:", "anonymous") # a preset input text When using the parameter default, the text input field will be pre-populated with the given text, So we click OK/Cancel or edit the content of the input field
  • 40. Sikuli Satish GorripotuListening to Global Hotkeys Env.addHotkey(key, modifiers, handler) Register the specified key + modifiers as a global hotkey. When the hotkey is pressed, the specified function handler will be called. Parameters: key – a character or a constant value defined in Key. modifiers – Key modifiers, which can be one or multiple constants defined in KeyModifier. def openAppleMenu(event): click(Any Image) # When we pressed Ctrl+Alt+F1, click the top-left apple icon. Env.addHotkey(Key.F1, KeyModifier.ALT+KeyModifier.CTRL, openAppleMenu) Env.removeHotkey(key, modifiers) Unregister the registered global hotkey key + modifiers. Parameters: key – a character or a constant value defined in Key. modifiers – Key modifiers, which can be one or multiple constants defined in KeyModifier.General Settings and Access to Environment InformationSikuli Level Sikuli internally uses the class Settings to store globally used settings. Publicly available attributes accessed by using Settings.[name-of-an-attribute] To get it‘s value and Settings.attribute = value to set it.Jython/Python Level We can use all settings defined in standard Python/Jython which are available in our system environment. The modules sys and time are already imported, so you can use their methods without the need for an import statement. sys.path may be one of the most valuable settings, since it is used by Python/Jython to locate modules, that are referenced using import module. myPath = "some-absolute-path" if not myPath in sys.path: sys.path.append(myPath)Java Level import java
  • 41. Sikuli Satish Gorripotu # get a value val = java.lang.System.getProperty("key-of-property") # set a propertys value java.lang.System.getProperty("key-of-property", value)Image Search Path Sikuli maintains a list of locations to search for images when they are not found in the current .sikuli folder (a.k.a. bundle path). This list is maintained internally but can be inspected and modified. getImagePath() Get a list of paths where Sikuli will search for images. # getImagePath() returns a Java array of unicode strings imgPath = list(getImagePath()) # makes it a Python list # to loop through for p in imgPath: print p addImagePath(a-new-path) Add a new path to the list of image search paths removeImagePath(a-path-already-in-the-list) Remove a path from the list of image search paths Note: paths must be specified using the correct path separators as double blackslashes .Default bundle path can accessed and modified. setBundlePath(path-to-a-folder) Set the path for searching images in all Sikuli Script methods. Sikuli IDE sets this automatically to the path of the folder where it saves the script (.sikuli). getBundlePath() Get a string containing a path to a folder containing our images used for finding patterns. Sikuli IDE sets this automatically to the path of the folder where it saves the script (.sikuli). We can use this function to package our private files together with the script or to access the picture files in the .sikuli bundles for other purposes. Sikuli only gives access to the path name, We need other python modules for I/O or other purposes.Other Environment Information Env.getOS() Env.getOSVersion() Env.getSikuliVersion() Example: if not Env.getSikuliVersion() == "X-1.0rc2":
  • 42. Sikuli Satish Gorripotu print "This script needs Sikuli X-1.0rc2" exit(1) Env.getClipboard() Get the content of the clipboard if it is text, otherwise an empty string. Env.getClipboard().paste() clipboard will contain what we just pasted. Env.getClipboard().strip() To get rid of surrounding white spaces. Env.isLockOn(key-constant) Get the current status ( on / off ) off the respective key. Only one key can be specified. key-constant – one of the key constants Key.CAPS_LOCK, Key.NUM_LOCK, Key.SCROLL_LOCK Env.getMouseLocation() Get the current location of the mouse cursor.Advanced Settings for Tuning Vision Algorithm Vision.setParameter(param, value) Vision.getParameter(param) Get/Set the parameter param of the vision algorithm. Parameters: param – a string that indicates the parameter to get. Returns: The float value of the specified parameter. MinTargetSize MinTargetSize is the minimum image size to which Sikuli can resize. Sikuli resizes the screen images to a smaller scale for faster matching. This scaling process speeds up the matching process, but also increases the possibility of false matching. The default value of MinTargetSize is 12, < 12. May increase speed but may returns a match that is not what we expect. Example Jython code. from org.sikuli.script.natives import Vision Vision.setParameter("MinTargetSize", 6) # the default is 12. Setting the size to a smaller value would make the matching algorithm be faster.
  • 43. Sikuli Satish GorripotuRegion Region is a class available in Sikuli Package. This class has several methods Class Region { class Region Region is a rectangular area on a screen. It knows nothing about its visual content (windows, pictures, graphics, text, ...). It only knows the position on the screen and its dimension. Creating a Region, Setting and Getting Attributes Region(x, y, w, h) Region(region) Region(Rectangle)
  • 44. Sikuli Satish Gorripotu Parameters: x – x position of a rectangle. #upper left corner of the screen (0, 0) y – y position of a rectangle. #upper left corner of the screen (0, 0) w – height of a rectangle. h – width of a rectangle. region – an existing region object. rectangle – an existing object of Java class Rectangle Returns: a new region object. region can also be created in run-time using Region.selectRegion(). You can create a region based on another region. This just duplicates the region (new object). Another way to create a region is to specify a rectangle object or to extend an existing region. Note: topLeft = Location(reg.x, reg.y) # equivalent to topLeft = reg.getTopLeft() theWidth = reg.w # equivalent to theWidth = reg.getW() reg.h = theWidth # equivalent to reg.setH(theWidth) We can use selectRegion() to interactively create a new region. setX(number) setY(number) setW(number) setH(number) Set the respective attribute of the region to the new value. This effectively moves the region around and/or changes its dimension. moveTo(location) Set the position of this region regarding it‘s top left corner to the given location (the x and y values are modified). Parameters : location – location object becomes the new top left corner Returns : the modified region object Example: reg.moveTo(anotherLocation) # equivalent to reg.setX(anotherLocation.x); reg.setY(anotherLocation.y) setROI(x, y, w, h) setROI(rectangle) setRect(x, y, w, h)
  • 45. Sikuli Satish Gorripotu setRect(rectangle) Both methods do the same: setting position and dimension to new values. setROI() is intended to restrict the search to a smaller area to speed up processing setRect() should be used to redefine a region (which could be enlarging it). Parameters: x, y, w, h (all) – the attributes of a rectangle rectangle – a rectangle object . morphTo(region) Set the position and dimension of this region to the corresponding values of the region given as parameter. Parameters:region – a region object Returns: the modified region object Note: reg.morphTo(anotherRegion) # equivalent to r = anotherRegion; reg.setX(r.x); reg.setY(r.y); reg.setW(r.w); reg.setH(r.h) getX() getY() getW() getH() Get the respective attribute integer value of the region. getCenter() Get the center of the region or Location as an object. getTopLeft() getTopRight() getBottomLeft() getBottomRight() Get the location object of the region‘s respective corner getScreen() Returns the screen object that contains this region. This method makes sense in Multi Monitor Environments getLastMatch() getLastMatches() Returns the best match as a Match object or one or more match objects as an Iterator object respectively setAutoWaitTimeout(seconds) Set the maximum waiting time for all subsequent find operations.
  • 46. Sikuli Satish Gorripotu This method enables all find operations to wait for the given pattern to appear until the specified amount of time has elapsed. The default is 3.0 seconds. This method is intended for users to override this default setting. Region.find() work like Region.wait(), without being able to set an individual timeout value for a specific find operation. getAutoWaitTimeout() Get the current value of the maximum waiting time for find operations.Extending a Region Note: In all cases the current region remains unchanged. In all cases the new region does not extend beyond any boundary of the screen that contains the given region. offset(location) Returns a new Region object, whose upper left corner is relocated adding the location‘s x and y value to the respective values of the given region. Width and height are the same. So this clones a region at a different place. Parameters : location – a Location object Returns :a new Region object Example: new_reg = reg.offset(Location(xoff, yoff)) # same as new_reg = Region(reg.x + xoff, reg.y + yoff, reg.w, reg.h) inside() Returns the same Region object. Retained for upward compatibility. region.inside().find() is totally equivalent to region.find(). nearby([range]) Returns a new Region object that includes the nearby neighbourhood of the the current region. The new region is defined by extending the current region‘s dimensions in all directions by range number of pixels. The center of the new region remains the same.
  • 47. Sikuli Satish Gorripotu Parameters: range – a positive integer indicating the number of pixels (default = 50). above([range]) Returns a new Region object that is defined above the current region‘s top border with a height of range number of pixels. It does not include the current region. If range is omitted, it reaches to the top of the screen. The new region has the same width and x- position as the current region. Parameters:range – a positive integer defining the new height below([range]) Returns a new Region object that is defined below the current region‘s bottom border with a height of range number of pixels. It does not include the current region. If range is omitted, it reaches to the bottom of the screen. The new region has the same width and x-position as the current region. Parameters:range – a positive integer defining the new height left([range]) Returns a new Region object that is defined left of the current region‘s left border with a width of range number of pixels. It does not include the current region. If range is omitted, it reaches to the left border of the screen. The new region has the same height and y-position as the current region. Parameters:range – a positive integer defining the new width right([range]) Returns a new Region object that is defined right of the current region‘s right border with a width of range number of pixels. So it does not include the current region. If range is omitted, it reaches to the right border of the screen. The new region has the same height and y-position as the current region. Parameters:range – a positive integer defining the new widthFinding inside a Region and Waiting for a Visual Event Besides acting on visual objects, finding them is one of the core functions of Sikuli. PS: means, that either a Pattern or a string (path to an image file or just plain text) can be used as parameter. find(PS) Find a particular GUI element, which is seen as the given image or just plain text. The given file name of an image specifies the element‘s appearance. It searches within the region and returns the best match, which shows a similarity greater than the minimum similarity given by the pattern. Parameters:PS – a Pattern object or a string (path to an image file or just plain text) Returns: a Match object that contains the best match or fails if not found findAll(PS)
  • 48. Sikuli Satish Gorripotu Repeatedly find ALL instances of a pattern, until no match can be found anymore, that meets the requirements for a single Region.find() with the specified pattern. Parameters:PS – a Pattern object or a string (path to an image file or just plain text) Returns :one or more Match objects as an iterator object or fails if not found By default, the returned matches are sorted by the similiarty. If you need them ordered by their positions, we have to use Python‘s sorted function. Example: def by_y(match): return match.y icons = findAll(―Image screen shot‖) # sort the icons by their y coordinates and put them into a new variable sorted_icons sorted_icons = sorted(icons, key=by_y) # another shorter version is using lambda. sorted_icons = sorted(icons, key=lambda m:m.y) for icon in sorted_icons: pass # do whatever you want to do with the sorted icons wait([PS][, seconds]) a Match object that contains the best match or fails if not found Parameters: PS – a Pattern object or a string (path to an image file or just plain text) seconds – a number, which can have a fraction, as maximum waiting time in seconds. If not specified, the auto wait timeout value set by Region.setAutoWaitTimeout() is used. Use the constant FOREVER to wait for an infinite time. If PS is not specified, the script just pauses for the specified amount of time. It is still possible to use sleep(seconds) instead, but this is deprecated. If PS is specified, it keeps searching the given pattern in the region until the image appears ( would have been found with Region.find()) or the specified amount of time has elapsed. At least one find operation is performed, even if 0 seconds is specified.) waitVanish(PS[, seconds]) Wait until the give pattern PS in the region vanishes. Returns True if the pattern vanishes within the specified waiting time, or False if the pattern stays visible after the waiting time has elapsed. This method keeps searching the given pattern in the region until the image vanishes (can not be found with Region.find() any longer) or the specified amount of time has elapsed. At least one find operation is performed, even if 0 seconds is specified Parameters: PS – a Pattern object or a string (path to an image file or just plain text) seconds – a number, which can have a fraction, as maximum waiting time in seconds. The internal granularity is milliseconds. If not specified, the auto wait timeout value set by Region. setAutoWaitTimeout() is used. Use the constant FOREVER to wait for an infinite time.
  • 49. Sikuli Satish Gorripotu . exists(PS[, seconds]) Check whether the give pattern is visible on the screen. And returns a Match object that contains the best match. None is returned, if nothing is found within the specified waiting time Does exactly the same as Region.wait(), but no exception is raised in case of FindFailed. It can be used to symplify scripting in case that we want to know wether something is there or not to decide how to proceed in your workflow. It is typically used with an if statement. At least one find operation is performed, even if 0 seconds is specified. Parameters: PS – a Pattern object or a string (path to an image file or just plain text) seconds – a number, which can have a fraction, as maximum waiting time in seconds. The internal granularity is milliseconds. If not specified, the auto wait timeout value set by Region.setAutoWaitTimeout() is used. Use the constant FOREVER to wait for an infinite time.Observing Visual Events in a Region handler: as a parameter in the following methods, you have to specify the name of a function, which will be called by the observer, in case the observed event happens. The function itself has to be defined in your script before using the method that references the function. The existance of the function will be checked before starting the script. So to get your script running, you have to have at least the following statements in your script: def myHandler(event): # you can choose any valid function name # event: can be any variable name, it references the SikuliEvent object pass # add your statements here onAppear("path-to-an-image-file", myHandler) # or any other onEvent() observe(10) # observes for 10 seconds onAppear(PS, handler) With the given region you register an observer, that should wait for the pattern to be there or to appear and is activated with the next call of observe(). In the moment the internal find operation on the given pattern is successful during observation, your handler is called and the observation is paused until you return from your handler. Parameters: PS – a Pattern object or a string (path to an image file or just plain text. handler – the name of a handler function in the script
  • 50. Sikuli Satish Gorripotu onVanish(PS, handler) With the given region you register an observer, that should wait for the pattern to be not there or to vanish and is activated with the next call of observe(). In the moment the internal find operation on the given pattern fails during observation, your handler is called and the observation is paused until you return from your handler. Parameters: PS – a Pattern object or a string (path to an image file or just plain text. handler – the name of a handler function in the script onChange([minChangedSize], handler) Parameters: minChangedSize – the minimum size in pixels of a change to trigger a change event. If omitted: 50 is used handler – the name of a handler function in the script minChangedSize With the given region we register an observer, that should wait for the visual content of the given region to change and is activated with the next call of observe(). In the moment the visual content changes during observation, our handler is called and the observation is paused until you return from your handler. Below example highlights all changes in an observed region. def changed(event): print "something changed in ", event.region for ch in event.changes: ch.highlight() # highlight all changes sleep(1) for ch in event.changes: ch.highlight() # turn off the highlights with selectRegion("select a region to observe") as r: # any change in r larger than 50 pixels would trigger the changed function onChange(50, changed) observe(background=True) wait(30) # another way to observe for 30 seconds r.stopObserver() observe([seconds][, background = False | True]) Begin observation within the region. For each region object, only one observation can be running at a given time.
  • 51. Sikuli Satish Gorripotu Parameters: seconds – a number, which can have a fraction, as maximum observation time in seconds. Omit it or use the constant FOREVER to tell the observation to run for an infinite time (or until stopped by a call of stopObserve()). background – a flag indicating whether observation is run in the background. when set to True, the observation will be run in the background and processing of your script is continued immediately. Otherwise the script is paused until the completion of the observation. stopObserver() Stop observation within the region. This must be called on a valid region object. The source region of an observed visual event is available as one of the attributes of the event parameter that is passed to the handler function when the function is invoked. Example: to stop observation within a handler function, call event.region.stopObserver() inside the handler function. def myHandler(event): event.region.stopObserver() # stops the observation onAppear("path-to-an-image-file", myHandler) observe(FOREVER) # observes until stopped in handler }SikuliEvent SikuliEvent is a class available in Sikuli Package. This class has several methods Class SikuliEvent {
  • 52. Sikuli Satish Gorripotu When processing an observation in a region, a handler function is called, when one of the registered events Region.onAppear(), Region.onVanish() or Region.onChange() One parameter, we have access to in handler function is an instance of Class SikuliEvent. Following attributes of the event, might help to identify the cause of the event and act on the resulting matches. Usage: event.attribute where event is the parameter from the definition of handler function: def myHandler(event): type SikuliEvent.Type.APPEAR, SikuliEvent.Type.VANISH, or SikuliEvent.Type.CHANGE. pattern The source pattern that triggered this event. This is only valid in APPEAR and VANISH events. region The source region of this event. match In an APPEAR event, match saves the top Match object that appeared in the observed region. In a VANISH event, match saves the last Match object that was in the observed region but vanished. This attribute is not valid in a CHANGE event. changes This attribute is valid only in a CHANGE event. This changes attribute is a list of Match objects that changed and their sizes are at least minChangedSize pixels. } Acting on a Region PSMRL: which means, that either a Pattern object or a string (path to an image file or just plain text) or a Match or a Region or a Location can be used as parameter. P: pattern: a Pattern object. An implicit find operation is processed first. If successful, the center of the resulting matches rectangle is the click point. If the pattern object has a target offset specified, this is used as click point instead. S: string: a path to an image file or just plain text. An implicit find operation with the default minimum similarity 0.7 is processed first. If successful, the center of the resulting match object is the click point.
  • 53. Sikuli Satish Gorripotu M: match: a match object from a previous find operation. If the match has a target specified it is used as the click point, otherwise the center of the matches rectangle. R: region: a region object whose center is used as click point. L: location: a location object which by definition represents a point on the screen that is used as click point.Region Region is a class available in Sikuli Package. This class has several methods continuation to above methods. Class Region { click(PSMRL[, modifiers]) Perform a mouse click on the click point using the left button. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. modifiers – one or more key modifiers Returns: Number of performed clicks (actually 1). A 0 (integer null) means that because of some reason, no click could be performed (in case of PS may be not Found). doubleClick(PSMRL[, modifiers]) Perform a mouse double-click on the click point using the left button. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. modifiers – one or more key modifiers Returns: Number of performed double-clicks (actually 1). A 0 (integer null) means that because of some reason, no click could be performed (in case of PS may be not Found). rightClick(PSMRL[, modifiers]) Perform a mouse click on the click point using the right button. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. modifiers – one or more key modifiers Returns: Number of performed right cilicks (actually 1). A 0 (integer null) means that because of some reason, no click could be performed (in case of PS may be not Found). highlight([seconds]) Highlight the region for some period of time. Parameters:seconds – a decimal number taken as duration in seconds The region is highlighted showing a red colored frame around it. If the parameter seconds is given, the script is suspended for the specified time. If no time is given, the highlighting is started and the script continues.
  • 54. Sikuli Satish Gorripotu When later on the same highlight call without a parameter is made, the highlighting is stopped (behaves like a toggling switch). Example: m = find(some_image) # the red frame will blink for about 7 - 8 seconds for i in range(5): m.highlight(1) wait(0.5) # a second red frame will blink as an overlay to the first one m.highlight() for i in range(5): m.highlight(1) wait(0.5) m.highlight() # the red frame will grow 5 times for i in range(5): m.highlight(1) m = m.nearby(20) hover(PSMRL) Move the mouse cursor to hover above a click point. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. modifiers – one or more key modifiers Returns: The number 1 if the mousepointer could be moved to the click point. A 0 (integer null) returned if no move could be performed (in case of PS may be not Found). dragDrop(PSMRL, PSMRL[, modifiers]) Perform a drag-and-drop operation from a starting click point to the target click point indicated by the two PSMRLs respectively. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. modifiers – one or more key modifiers If one of the parameters is PS, the operation might fail due to not Found. drag(PSMRL) Start a drag-and-drop operation by dragging at the given click point. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
  • 55. Sikuli Satish Gorripotu Returns: Number 1 if the operation could be performed. A 0 (integer null) returned if no move could be performed (in case of PS may be not Found). dropAt(PSMRL[, delay]) Complete a drag-and-drop operation by dropping a previously dragged item at the given target click point. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. delay – time to wait after in seconds as decimal value Returns: Number 1 if the operation could be performed. A 0 (integer null) returned if no move could be performed (in case of PS may be not Found). type([PSMRL], text[, modifiers]) Type the text at the current focused input field or at a click point specified by PSMRL. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. modifiers – one or more key modifiers Returns: Number 1 if the operation could be performed, 0 (integer null) if it was not possible or the click couldn‘t be performed (in case of PS may be not Found). paste([PSMRL], text) Paste the text at a click point. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. Modifiers – one or more key modifiers Returns: Number 1 if the operation could be performed, 0 (integer null) if not possible or the click couldn‘t be performed (in case of PS may be not Found).Extracting Text from a Region text() Extract the text contained in the region using OCR.
  • 56. Sikuli Satish GorripotuLow-level Mouse and Keyboard Actions mouseDown(button) Press the mouse button down. Parameters: button – one or a combination of the button constants Button.LEFT, Button.MIDDLE, Button.RIGHT. Returns: Number 1 if the operation is performed successfully, and zero if otherwise. mouseUp([button]) Release the mouse button previously pressed. Parameters: button – one or a combination of the button constants Button.LEFT, Button.MIDDLE, Button.RIGHT. Returns: Number 1 if the operation is performed successfully, and zero if otherwise. mouseMove(PSRML) Move the mouse pointer to a location indicated by PSRML. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
  • 57. Sikuli Satish Gorripotu Returns: Number 1 if the operation could be performed. If using PS (which invokes an implicity find operation), find fails and you have switched off FindFailed exception, a 0 (integer null) is returned. Otherwise, the script is stopped with a FindFailed exception. wheel(PSRML, WHEEL_DOWN | WHEEL_UP, steps) Move the mouse pointer to a location indicated by PSRML and turn the mouse wheel in the specified direction by the specified number of steps. Parameters: PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point. WHEEL_DOWN|WHEEL_UP – one of the two constants denoting the wheeling direction. steps – an integer indicating the amoung of wheeling. keyDown(key | list-of-keys) Press and hold the specified key(s) until released by a later call to Region.keyUp(). Parameters: key|list-of-keys – one or more keys (use the consts of class Key). A list of keys is a concatenation of several key constants using ―+‖. Returns: Number 1 if the operation could be performed and 0 if otherwise. keyUp([key | list-of-keys]) Release given keys. If no key is given, all currently pressed keys are released. Parameters: key|list-of-keys – one or more keys (use the consts of class Key). A list of keys is a concatenation of several key constants using ―+‖. Returns: Number 1 if the operation could be performed and 0 if otherwise.Exception FindFailed These are the possibilities to handle ―not found‖ situations: •generally abort a script, if not handled with try: ... except: ... (the default setting or using setThrowException(True) or setFindFailedResponse(ABORT)) •generally ignore all ―not found‖ situations (using setThrowException(False) or setFindFailedResponse(SKIP)), •want to be prompted in such a case (using setFindFailedResponse(PROMPT)) •advise Sikuli to wait forever (be careful with that!) (using setFindFailedResponse(RETRY)) Using PROMPT:
  • 58. Sikuli Satish Gorripotu Example: # --- nice and easy if exists("path-to-image"): # no exception, returns None when not found pass # it is there else: pass # we miss it # --- using exception handling # every not found in the try block will switch to the except block try: find("path-to-image") pass # it is there except FindFailed: pass # we miss it # --- using setFindFailedResponse setFindFailedResponse(SKIP) # no exception raised, not found returns None if find("path-to-image"): setFindFailedResponse(ABORT) # reset to default pass # it is there else: setFindFailedResponse(ABORT) # reset to default pass # we miss it # --- using setThrowException setThrowException(False) # no exception raised, not found returns None if find("path-to-image"): setThrowException(True) # reset to default pass # it is there else: setThrowException(True) # reset to default pass # we miss it setFindFailedResponse(ABORT | SKIP | PROMPT | RETRY) For the specified region set the option, how Sikuli should handle ―not found‖ situations. The option stays in effect until changed by another setFindFailedResponse().
  • 59. Sikuli Satish Gorripotu Parameters: ABORT – all subsequent find failed operations (explicit or implicit) will raise exception FindFailed (which is the default when a script is started). SKIP – all subsequent find operations will not raise exception FindFailed. Instead, explicit find operations such as Region.find() will return None. Implicit find operations (action functions) such as Region.click() will do nothing and return 0. PROMPT – all subsequent find operations will not raise exception FindFailed. Instead you will be prompted for the way to handle the situation (see using PROMPT). Only the current find operation is affected by your response to the prompt. RETRY – all subsequent find operations will not raise exception FindFailed. Instead, Sikuli will try to find the target until it gets visible. This is equivalent to using wait( ... , FOREVER) instead of find() or using setAutoWaitTimeout(FOREVER). getFindFailedResponse() Get the current setting in this region. Returns: ABORT or SKIP or PROMPT or RETRY Usage: val = getFindFailedResponse() if val == ABORT: print "not found will stop script with Exception FindFailed" elif val == SKIP: print "not found will be ignored" elif val == PROMPT: print "when not found you will be prompted" elif val == RETRY: print "we will always wait forever" else: print val, ": this is a bug :-(" Note: It is recommended to use set/getFindFailedResponse() instead of set/getThrowException() since the latter ones might be deprecated in the future. setThrowException(False | True) By using this method you control, how Sikuli should handle not found situations in this region. Parameters: True – all subsequent find operations (explicit or implicit) will raise exception FindFailed (which is the default when a script is started) in case of not found. False – all subsequent find operations will not raise exception FindFailed. Instead, explicit find operations such as Region.find() will return None. Implicit find operations (action functions) such as Region.click() will do nothing and return 0. getThrowException() Returns: True or False
  • 60. Sikuli Satish GorripotuGrouping Method Calls ( with Region: ) Instead of: # reg is a region object if not reg.exists(image1): reg.click(image2) reg.wait(image3, 10) reg.doubleClick(image4) you can group methods applied to the same region using Python‘s with syntax: # reg is a region object with reg: if not exists(image1): click(image2) wait(image3, 10) doubleClick(image4) All methods inside the with block that have the region omitted are redirected to the region object specified at the with statement. Note for Sikuli X versions up to rc3 This should work for Regions and Region-like objects (Region, Screen, App.window()). Match objects have to be made a Region explicitly using Region(some_match). This is a general workaround when facing any problems using with some_region:. Special Methods for Developers getRegionFromPSRM(PSRM) Returns a new Region object derived from the given parameter. In case of PS, internally a find() is done inside this region. If found, the match is returned. In case RM, just a copy of the given region is returned. Params PSRM: Pattern, String, Region or Match object getLocationFromPSRML(PSRML) Returns a new Location object derived from the given parameter. In case of PS, internally a find() is done inside this region. If found, the match‘s target offset position is returned. In case RM, just a copy of the given region‘s respective location (center or target offset) is returned. Params PSRML: Pattern, String, Region, Match or Location object }Screen
  • 61. Sikuli Satish Gorripotu Screen is a class available in Sikuli Package. This class has several methods continuation to above methods. Since Screen extends class Region, all methods of class Region can be used with a screen object. Class Screen { Screen: Setting, Getting Attributes and Information Screen([id]) Create a new Screen object Parameters: id – an integer number indicating which monitor in a multi-monitor environment. Returns: a new screen object. It creates a new screen object, that represents the default/primary monitor (whose id is 0), if id is omitted. Numbers 1 and higher represent additional monitors that are available at the time, the script is running. Using numbers that do not represent an existing monitor, will stop the script with an error. So we may either use getNumberScreens() or exception handling, to avoid this. Note: If we want to access the default/primary monitor ( Screen(0) ) without creating a new screen object, use the constant reference SCREEN, that is initiated when your script starts: SCREEN=Screen(0). getNumberScreens() Get the number of screens in a multi-monitor environment at the time the script is running getBounds() Get the dimensions of monitor (a rectangle object) represented by the screen object. Capturing Capturing is the feature, that allows to grab a rectangle from a screenshot, to save it for later use. At each time, a capturing is initiated, a new screenshot is taken. There are two different versions: Screen.capture() saves the content of the selected rectangle in a file and returns its file name, Screen.selectRegion() returns the position and dimension of the selected rectangle. capture([region | rectangle | text]) capture(x, y, w, h) Returns the path to the file, where the captured image was saved. In interactive mode, the user may cancel the capturing, in which case None is returned. Parameters: region – an existing region object. rectangle – an existing rectangle object (e.g., as a return value of another region method). text – text to display in the middle of the screen in the interactive mode. x – x position of the rectangle to capture y – y position of the rectangle to capture w – width of the rectangle to capture h – height of the rectangle to capture
  • 62. Sikuli Satish Gorripotu Interactive Mode: The script enters the screen-capture mode like when clicking the button in the IDE, enabling the user to capture a rectangle on the screen. If no text is given, the default ―Select a region on the screen‖ is displayed. selectRegion([text]) Select a region on the screen interactively. Returns a new Region object or None, if the user cancels the capturing process Text is displayed for about 2 seconds in the middle of the screen. If text is omitted, the default ―Select a region on the screen‖ is displayed. The interactive capture mode is entered and allows the user to select a region the same way as using the selection tool in the IDE. Parameters: text – Text to display in the middle of the screen. } Multi-Monitor Environments If more than one monitor is available, Sikuli is able to manage regions and click points on these monitors. getNumberScreens() returns the number of available screens. getBounds() returns the rectangle covered by the default/primary monitor. Screen.getBounds() returns the rectangle covered by a screen object created using Screen(id). The monitor, that is the first one based on hardware mapping (e.g. the laptop monitor), will always be Screen(0). In the Windows settings it is possible to place the taskbar on one of the secondary monitors, which makes it the primary monitor getting the base coordinates (0,0). The other available monitors are mapped around based on your settings. But the Sikuli internal mapping is not changed, so the primary monitor might be any of your Screen() objects. Sikuli takes care for that and maps SCREEN always to the primary monitor (the one with the (0,0) coordinates). So for example you have a laptop with an external monitor, that shows the taskbar (is primary monitor): SCREEN maps to Screen(1) Screen(0) is your laptop monitor With its rectangle, a screen object is always identical with the monitor it was created using Screen(id). Using Region.setROI() to restrict the region of interest for find operations has no effect on the base rectangle of the screen object. On the other hand region objects and location objects can be positioned anywhere in the coordinate system. Only when a find operation or a click action has to be performed, the objects rectangle or point has to be inside the rectangle of an existing monitor (basically repersented by
  • 63. Sikuli Satish Gorripotu Screen(0), Screen(1), ...). When this condition is met, everything works as expected and known from a single monitor system. With finding and acting there are the following exceptions: Point Outside: a click point is outside any monitor rectangle. The clickpoint will be mapped to the edges or corners of the primary monitor according to the relative position: to the edges if its x or y value is in the range of the respective edge (right, left, above, below)i to the corners, if x and y are outside any range of any edge (left/above -> upper left corner, ...) Region Outside: a region is completely outside any monitor a click action is handled in the same way as Point Outside a find operation will always fail Region Partially Outside: a region is partially outside a monitor but not overlapping another monitor a click action is handled in the same way as Point Outside a find operation will be carried out only in the part of region within the bounds of the monitor, excluding the area outside the monitor. Region Across Monitors: a region lies across multiple monitors: a click action is handled in the same way as Point Outside a find operation will be restricted to the region within the bounds of the monitor that has a smaller id. An interactive capture (the user is asked to select an image or a rectangle via Screen.capture() or Screen.selectRegion()) will automatically be restricted to the monitor, where it was started. A scripted capture using a rectangle or a region (i.e. Screen.capture( region | rectangle )), will be handled accordingly: Region Outside: no image is captured, None is returned Region Partially Outside: the returned image will only cover the part inside the monitor Region Across Monitors: the returned image will only cover the part inside the monitor with the smallest id.Location Location is a class available in Sikuli Package. Handle single points on the screen directly by its position (x, y). This class has several methods continuation to above methods. Class Location { Location(x, y) Returns a new location object representing the position (x,y) on the screen Parameters: x – x position y – y position getX() getY() Get the x or y value of a location object
  • 64. Sikuli Satish Gorripotu It is possible to get the values directly by location.x or location.y. It is also possible to set these values directly by location.x = value or location.y = value. setLocation(x, y) Set the location of this object to the specified coordinates. offset(dx, dy) Get a new location which is dx and dy pixels away horizontally and vertically from the current location. above(dy) Get a new location which is dy pixels vertically above the current location. below(dy) Get a new location which is dy pixels vertically below the current location. left(dx) Get a new location which is dx pixels horizontally to the left of the current location. right(dx) Get a new location which is dx pixels horizontally to the right of the current location. }Match Match is a class available in Sikuli Package. class Match extends class Region, all methods of class Region can be used with a match object. An object of class Match represents the result of a successful find operation. It has the rectangle dimension of the image, that was used to search. It knows the point of its upper left corner on an existing monitor, where it was found. This class has several methods continuation to above methods. Class Match { Creating a Match, Getting Attributes A match object is created as the result of an explicit find operation. It can be saved in a variable for later use with actions like click(). # m is a reference to a match object, if found m = find(Some Image) print m # message area: Match[10,0 30x22] score=1.00, target=center
  • 65. Sikuli Satish Gorripotu # m is a reference to a match object, if found m = find(Pattern(Some Image).similar(0.5).targetOffset(100,0)) print m # message area: Match[10,0 30x22] score=1.00, target=(105,11) getScore() Get the similarity score the image or pattern was found. The value is between 0 and 1. getTarget() Get the location object that will be used as the click point. If no offset was specified by Pattern.targetOffset(),click point is the center of the matched region. If an offset was given, the click point is the offset relative to the center. Iterating over Matches after findAll() find operation Region.findAll() returns an iterator object that can be used to fetch all found matches as match objects one by one. A reference to the iterator is stored in the respective region and can be accessed using Region.getLastMatches(). Note: per definition, an iterator can be stepped through only once - it is empty afterwards To save contained matches for later use, you can convert them to list. findAll() # find all matches mm = list(getLastMatches()) Example: using while: with default screen findAll() # find all matches mm = SCREEN.getLastMatches() while mm.hasNext(): # loop as long there is a first and more matches print "found: ", mm.next() # access the next match in the row print mm.hasNext() # is False, because mm is empty now print mm.next() # is None, because mm is empty now print SCREEN.getLastMatches().hasNext() # is False also ;-) Example: using with: with default screen with findAll() as mm: while mm.hasNext(): # loop as long there is a first and more matches print "found: ", mm.next() # access the next match # mm will be None afterwards (destroyed automatically)
  • 66. Sikuli Satish GorripotuPattern Pattern is a class available in Sikuli Package. A pattern is used, to associate an image file with additional attributes used in find operations and when acting on a match object. Normally when clicking on a match, the center pixel of the associated rectangle is used. With a pattern object, we can define a different click point relative to the center using targetOffset(). Class Pattern { Pattern(string) This will initialize a new pattern object without any additional attributes. As long as no pattern methods are used additionally, it is the same as just using the image file name itself in the find operation Parameters: string – a path to an image file similar(similarity) Return a new Pattern object containing the same attributes (image, click point) with the minimum similarity set to the specified value. Parameters: similarity – the minimum similarity to use in a find operation. The value should be between 0 and 1. exact() Return a new Pattern object containing the same attributes (image, click point) with the minimum similarity set to 1.0, which means exact match is required. targetOffset(dx, dy)
  • 67. Sikuli Satish Gorripotu Return a new Pattern object containing the same attributes (image, similarity), but a different definition for the click. By default, the click point is the center of the found match. By setting the target offset, it is possible to specify a click point other than the center. dx and dy will be used to calculate the position relative to the center. Parameters: dx – x offset from the center dy – y offset from the center getFilename() Get the filename of the image as a string contained in the Pattern object. getTargetOffset() Get a Location object as the target offset of the Pattern object. }Finder Finder is a class available in Sikuli Package. A Finder object implements an iterator of matches and allows to search for a visual object in an image file that you provide (e.g. a screenshot taken and saved in a file before). After setting up the finder object and doing a find operation, you can iterate through the found matches if any. Note: 1. per definition, an iterator can be stepped through only once - it is empty afterwards 2. it has to be destroyed using finder.destroy(), especially when used with for: or while: 3. When used in a with: construct, it is destroyed automatically Compared with the region based find operation, no exception FindFailed is raised in case nothing is found at all (use hasNext() to check). The finder object can be compared to what you get with region.getLastMatches() when using findAll(). Note: With this version, there is no chance, to get the number of matches in advance. If you would iterate through to count, afterwards your finder would be empty. So in this case, you have to save your matches somehow The workflow always is, that you first do a find operation and afterwards go through the matches found. After a complete iteration, the finder object is empty. So we could start a new find operation again. Class Finder { Finder(path-to-imagefile) Create a new finder object. Parameters: path-to-imagefile – filename to a source image to search within find(path-to-imagefile[, similarity])
  • 68. Sikuli Satish Gorripotu Find a given image within a source image previously specified in the constructor of the finder object. Parameters: path-to-imagefile – the target image to search for similarity – the minimum similarity a match should have. If omitted, the default is used. hasNext() Check whether there are more matches available that satisfy the minimum similarity requirement. next() Get the next match object. The returnd reference to a match object is no longer available in the finder object afterwards. So if it is needed later on, it has to be saved to another variable. Example 1: basic operations using a Finder # create a Finder with your saved screenshot f = Finder() img= # the image you are searching f.find(img) # find all matches while f.hasNext(): # loop as long there is a first and more matches print "found: ", f.next() # access the next match in the row print f.hasNext() # is False, because f is empty now f.destroy() # release the memory used by finder Example 2: we want to know how many matches (based on the previous example). # create a Finder with your saved screenshot f = Finder() img= # the image you are searching f.find(img) # find all matches mm = [] # an empty list while f.hasNext(): # loop as long there is a first and more matches mm.append(f.next()) # access next match and add to mm print f.hasNext() # is False, because f is empty now f.destroy() # release the memory used by finder # now we have our matches saved in the list mm print len(mm) # the number of matches # we want to use our matches for m in mm: print m }
  • 69. Sikuli Satish GorripotuKey Constants class Key { Applicable usage situations for these predefined constants of special keys and key modifiers can be found in Acting on a Region and Low Level Mouse and Keyboard Actions.Special Keys The methods supporting the use of special keys are type(), keyDown(), and keyUp(). Key.CONSTANT (where CONSTANT is one of the following key names). String concatenation with other text or other key constants is possible using ―+‖. type("some text" + Key.TAB + "more text" + Key.TAB + Key.ENTER) # or eqivalent type("some texttmore textn") miscellanous keys ENTER, TAB, ESC, BACKSPACE, DELETE, INSERT,SPACE function keys F1, F2, F3, F4, F5, F6, F7, F8, F9, F10, F11, F12, F13, F14, F15 navigation keys HOME, END, LEFT, RIGHT, DOWN, UP, PAGE_DOWN, PAGE_UP special keys PRINTSCREEN, PAUSE, CAPS_LOCK, SCROLL_LOCK, NUM_LOCK Note: The status ( on / off ) of the keys Key.CAPS_LOCK, Key.NUM_LOCK and Key.SCROLL_LOCK can be evaluated with the method Env.isLockOn(). numpad keys NUM0, NUM1, NUM2, NUM3, NUM4, NUM5, NUM6, NUM7, NUM8, NUM9 SEPARATOR, ADD, MINUS, MULTIPLY, DIVIDE modifier keys ALT, CMD, CTRL, META, SHIFT, WIN
  • 70. Sikuli Satish Gorripotu These modifier keys cannot be used as a key modifier with functions like type(), rightClick(), etc. They can only be used with keyDown() and keyUp(). If you need key modifiers, use KeyModifier instead.Key Modifiers Methods where key modifiers can be used include: click(), dragDrop() , doubleClick() , rightClick(), type(). system specific Win KEY_WIN, KEY_CMD KEY_META (a synonym for KEY_WIN or KEY_CMD on Windows). The old modifiers with a KEY_ prefix are deprecated. Use KeyModifier.CTRL, KeyModifier.ALT, KeyModifier.SHIFT, KeyModifier.META instead.KeyModifier class KeyModifier KeyModifier.CONSTANT (where CONSTANT is one of the following key names). CTRL equivalent to the old KEY_CTRL SHIFT equivalent to the old KEY_SHIFT ALT equivalent to the old KEY_ALT META equivalent to the old KEY_META CMD equivalent to the old KEY_CMD (and KEY_META) WIN equivalent to the old KEY_WIN (and KEY_META) The modifier constants can be combined to the modifier parameter by either using ―+‖ or ―|‖, if more than one key modifier is needed. type(Key.ESC, KeyModifier.CTRL + KeyModifier.ALT) # or equivalent type(Key.ESC, KeyModifier.CTRL | KeyModifier.ALT) They should only be used in the modifiers parameter with functions like type(), rightClick(), etc. They should never be used with keyDown() or keyUp().Constants related to Mouse and Keyboard Actions
  • 71. Sikuli Satish Gorripotu Mouse related Class Button : Button.LEFT - Button.MIDDLE - Button.RIGHT related to mouse wheel actions: WHEEL_DOWN - WHEEL_UP Keyboard related Key Modifiers KEY_ALT - KEY_CMD - KEY_CTRL - KEY_META - KEY_SHIFT - KEY_WIN A combination of these keys has to be set up using "+" or "|", e.g. KEY_SHIFT + KEY_ALT or KEY_SHIFT | KEY_ALT | KEY_CMD (used with action methods) Special Keys To be used with type() or keyDown()/keyUp(): ADD - ALT - BACKSPACE - CAPS_LOCK - DIVIDE - DOWN - END - ENTER - ESC - DELETE - F1 - F2 - F3 - F4 - F5 - F6 - F7 - F8 - F9 - F10 - F11 - F12 - F13 - F14 - F15 - HOME - INSERT- LEFT - MINUS - MULTIPLY - NUM_LOCK - NUM0 - NUM1 - NUM2 - NUM3 - NUM4 - NUM5 - NUM6 - NUM7 - NUM8 - NUM9 - PAGE_DOWN - PAGE_UP - PAUSE - PRINTSCREEN - RIGHT - SCROLL_LOCK - SEPARATOR - TAB – UP To be used with keyDown()/keyUp() only: CMD - CTRL - META - SHIFT - TAB- WIN To use these constants, write e.g. Key.ENTER. String concatenation with "+" is possible, e.g. "some text" + Key.ENTER Constants FOREVER: can be used with exists - observe - wait - waitVanish SCREEN : a constant reference to a screen object created using Screen(0) at startup, The default/primary monitor used as the default region for methods not qualified by an instance/object/class.Extensions Download an Extension The download of an extension is supported by the IDE through the menu Tools -> Extensions.
  • 72. Sikuli Satish Gorripotu We get a popup, that lists the available and already installed extensions and allows to download new packages or updates for installed ones. This popup shows a new package not yet installed: Use an Extension: To use the features of an installed extension in one of your scripts, just say from extension-name import *. Technical Details Extensions are Java JAR files containing some Java classes (usually the core functions) and/or Python modules, which define the API to be used in a script. Sikuli maintains a local extensions directory, where downloaded extensions are stored together with a hidden list of the installed extensions %APPDATA%Sikuliextensions Once an extension is imported using import extension-name, Sikuli automatically searches and loads the JAR file of that extension into the current context with load(path-to-jar-file). Develop an extension The source structure of an extension named extension-name looks like this: Java - org/com -- your-organization-or-company --- extension-name ---- yourClass1.java ---- yourClass2.java ---- .... more classes python - extension-name -- __init__.py -- extension-name.py The final structure of a JAR (filename extension-name-X.Y where X.Y is the version string) looks like this: org/com - your-organization-or-company -- extension-name --- yourClass1.class --- yourClass2.class --- .... more classes extension-name - __init__.py
  • 73. Sikuli Satish Gorripotu - extension-name.py META-INF - MANIFEST.MFThe file __init__.py contains at least from extension-name import * to avoid one qualification level. So in a script you might either use: import extension-name extension-name.functionXYZ() or: from extension-name import * functionXYZ() The second case requires more investement in a naming convention, that avoids naming conflicts. The file extension-name.py contains the classes and methods, that represent the API, that one might use in a Sikuli script. Name the extensions Sikuli extensions can be Python/Jython modules or Java classes. For Java classes, following the reverse URL convention of Java is a good idea (for example, org.foo.your-extension). However, DO NOT use Java‘s convention for Python/Jython modules. You need to come up with a unique extension name that does not conflict with existing Python modules and other Sikuli extensions. Test the extension While developing your extensions, you can put the JAR file in Sikuli‘s extension directory or in the same .sikuli folder as your test script. The JAR file should not have a version number in its file name, e.g. extension-name.jar. Because Sikuli starts to search extensions in the .sikuli folder of the running script and then in the Sikuli extensions folder, it is usually a good idea to put your developing extensions in the .sikuli folder of your test script. Another option is to use the load() function with an absolute path to your extension-name.jar. If this fails, Sikuli goes on searching in the current .sikuli folder and then in the Sikuli extensions folder. If load() succeeds, it returns True and puts absolute-path-to-your-extension-name.jar into sys.path, so you can use import extension-name afterwards.Guide:
  • 74. Sikuli Satish Gorripotu Example: from guide import * text(<Some Image on the screen>, "This is Sikulis logo") show(5) Adding Multiple Annotations The script uses the function tooltip(text) to add tooltips to three links in addition to the text annotation from guide import * text(, "This is Sikulis logo") tooltip(,"Previous") tooltip(,"Next") tooltip(,"Index") show(5) from guide import *
  • 75. Sikuli Satish Gorripotu while True: tooltip(,"Previous") show(3) tooltip(,"Next") show(3) tooltip(,"Index") show(3) Adding Interaction from guide import * text(,"Use these to jump to other parts") dialog("Part 1: Navigation Links") show() text(,"Use this to go back to Home") dialog("Part 2: Logo") show() Making a Region clickable from guide import * logo = find() text(logo, "To proceed click this red ...") clickable(logo) index = logo.above().right().find() text(index, "... or click this red") clickable(index) show() Static Annotations guide.rectangle(PSRM) Add a rectangular overlay in red on the specified target‘s region. Parameters: PSRM – a pattern, string, region or match guide.circle(PSRM) Add a red circle around the specified target‘s region. Parameters: PSRM – a pattern, string, region or match guide.text(PSRM, text) Add some text (white large letters on dark grey background) left justified below the specified target‘s region, which is additionally highlighted. Parameters: PSRM – a pattern, string, region or match text – a string as text to display
  • 76. Sikuli Satish Gorripotu guide.tooltip(PSRM, text) Add a tooltip (small text in a light yellow box) left justified below the specified target‘s region. Parameters: PSRM – a pattern, string, region or match text – a string as text to display Interactive Elements guide.dialog(text) Add a dialog box displaying the given text in the middle of the screen above all othe windows. Parameters: text – a string as text to display guide.clickable(PSRM) Add a clickable element corresponding to the specified target‘s region. Parameters: PSRM – a pattern, string, region or match Control guide.show([seconds]) Show static and interactive components added so far for the specified amount of time. Parameters: seconds – a decimal number as display duration in seconds The default duration is 10 seconds. If interactive elements (either one or more clickable elements or a dialog box) were previously added, it waits until the user interacts with one of these elements. At this time all elements vanish and are discarded. Note: If a dialog() element is present, other interactive elements are visible, but not clickable. If the dialog element is clicked, all elements vanish and are discarded.Summary of Class and Methods Region Methods of Region above - below - click - drag - dragDrop - dropAt - doubleClick - exists - find - findAll - getCenter - getH - getLastMatch - getLastMatches - getRect - getROI - getScreen - getW - getX - getY - highlight - hover - inside - keyDown - keyUp - left - mouseDown - mouseMove - mouseUp - nearby - observe - onAppear - onVanish - onChange - paste - right - rightClick - selectRegion - setAutoWaitTimeout - setH - setRect - setROI - setThrowException - setW - setX - setY - stopObserver - text - type - wait – waitVanish Creating a Region, Setting and Getting Attributes getCenter - getH - getLastMatch - getLastMatches - getRect - getROI - getScreen - getW - getX - getY - selectRegion - setAutoWaitTimeout - setH - setRect - setROI - setW - setX - setY Extending a Region above - below - inside - left - nearby - right Finding inside a Region and Waiting for a Visual Event exists - find - findAll - wait - waitVanish Observing Visual Events in a Region
  • 77. Sikuli Satish Gorripotu observe - onAppear - onVanish - onChange - stopObserver Acting on a Region click - drag - dragDrop - dropAt - doubleClick - highlight - hover - paste - rightClick - type Extracting Text from a Region text Low Level Mouse and Keyboard Actions getMouseLocation - keyDown - keyUpmouseDown - mouseMove - mouseUp - wheel Screen click - drag - dragDrop - dropAt - doubleClick - exists - find - findAll - getCenter - getH - getLastMatch - getLastMatches - getROI - getW - getX - getY - hover - keyDown - keyUp - mouseDown - mouseMove - mouseUp - observe - onAppear - onVanish - onChange - paste - rightClick - setAutoWaitTimeout - setH - setROI - setThrowException - setW - setX - setY - stopObserver - text - type - wait – waitVanish capture - getBounds - getNumberScreens - selectRegion Setting, Getting Attributes and Information getBounds - getNumberScreens Screen as (Default) Region Capturing capture - selectRegion Location above - below - getsetL - getsetL - left - offset - right - getsetL - getsetL Match above - below - click - drag - dragDrop - dropAt - doubleClick - exists - find - findAll - getCenter - getH - getLastMatch - getLastMatches - getScreen - getW - getX - getY - hover - inside - left - mouseMove - nearby - observe - onAppear - onVanish - onChange - paste - right - rightClick - selectRegion - setAutoWaitTimeout - setThrowException - stopObserver - type - wait – waitVanish
  • 78. Sikuli Satish Gorripotu getScore - getTarget Finder find - hasNext - next Pattern exact - getFilename - getTargetOffset - similar - targetOffset VDict vdict[ ] - vdict[ ] = - [ not in vdict] - del vdict[ ] - get -get1 - get_exact - len( ) - keys Env getOS() returns information about the system, the script is running on ( -> General Information and Settings ) getOSVersion() returns the systems version number, the script is running on ( -> General Information and Settings ) getMouseLocation() returns the current location of the mouse pointer ( -> Low Level Mouse and Keyboard Actions ) getClipboard() returns the content of the Clipboard if it is text, otherwise an empty string. ( -> General Information and Settings )
  • 79. Sikuli Satish Gorripotu Sikuli – Class and methodsClass Method Parameter Return value Usage 1 Usage 2Region Create a region Region(x, y, w, h) object Create a region Region(region) object Create a region Region(Rectangle) object setX(number) setY(number) setW(number) setH(number) moveTo(location) setROI(x, y, w, h) setROI(rectangle) setRect(x, y, w, h) setRect(rectangle) morphTo(region) getX() getY() getW() getH() getCenter() getTopLeft() getTopRight() getBottomLeft() getBottomRight() getScreen() getLastMatch() getLastMatches() setAutoWaitTimeout(seconds) setAutoWaitTimeout(seconds) getAutoWaitTimeout() offset(location) inside()
  • 80. Sikuli Satish Gorripotu nearby([range]) above([range]) below([range]) left([range]) right([range]) find(PS) findAll(PS) wait([PS][, seconds]) waitVanish(PS[, seconds]) exists(PS[, seconds]) onAppear(PS, handler) onVanish(PS, handler) onChange([minChangedSize], handler) observe([seconds][, background = False | True]) stopObserver() click(PSMRL[, modifiers]) doubleClick(PSMRL[, modifiers]) rightClick(PSMRL[, modifiers]) highlight([seconds]) hover(PSMRL) dragDrop(PSMRL, PSMRL[, modifiers]) drag(PSMRL) dropAt(PSMRL[, delay]) type([PSMRL], text[, modifiers]) paste([PSMRL], text) text() mouseDown(button) mouseUp([button]) mouseMove(PSRML) wheel(PSRML, WHEEL_DOWN | WHEEL_UP, steps) keyDown(key | list-of-keys) keyUp([key | list-of-keys]) setFindFailedResponse(ABOR T | SKIP | PROMPT | RETRY) getFindFailedResponse() setThrowException(False | True) getThrowException() getRegionFromPSRM(PSRM) getLocationFromPSRML(PSRM L)SikuliEvent type pattern region match changes
  • 81. Sikuli Satish GorripotuScreen Screen([id]) getNumberScreens() getBounds() capture([region | rectangle | text]) capture(x, y, w, h) selectRegion([text])Location Location(x, y) getX() getY() setLocation(x, y) offset(dx, dy) above(dy) below(dy) left(dx) right(dx)Match getScore() getTarget()Pattern Pattern(string) similar(similarity) exact() targetOffset(dx, dy) getFilename() getTargetOffset()Finder Finder(path-to-imagefile) find(path-to-imagefile[, similarity]) hasNext() next() Button left middle rightApp (Application)Env (Environment)Key (Key Constants)VDict (Visual Dictionary) Sikuli consolidated Methods:
  • 82. Sikuli Satish Gorripotu S.No Methods Examples Page Numbers command line option --args <arguments> -h,--help "-I" -r,--run <sikuli-file> -s,--stderr -t,--test <sikuli-file> A 1 above() (Location method) 2 Above (Region method) 3 ActionLogs (Settings attribute) 4 addHotkey() (Env method) 5 addImagePath() (built-in function) 6 ALT (built-in variable) 7 App (built-in class), [1] B 8 below() (Location method) 9 below (Region method) 10 Bundle Path C 11 capture() (Screen method) 12 changes (SikuliEvent attribute) 13 circle() (in module guide) 14 click() (Region method) Done 15 clickable() (in module guide) 16 close() (App class method) 17 Close (App method) 18 closeApp() (built-in function) 19 CMD (built-in variable) 20 --args <arguments> 21 -h,--help 22 "-I" 23 -r,--run <sikuli-file> 24 -s,--stderr 25 -t,--test <sikuli-file> 26 CTRL (built-in variable) D 27 DebugLogs (Settings attribute) 28 DelayAfterDrag (Settings attribute)
  • 83. Sikuli Satish Gorripotu 29 DelayBeforeDrop (Settings attribute) 30 dialog() (in module guide) 31 doubleClick() (Region method) Done 32 drag() (Region method) 33 dragDrop() (Region method) Done 34 dropAt() (Region method) destroy E 35 exact() (Pattern method) 36 exists() (Region method) 37 exit() (built-in function) F 38 find() (Finder method) Done 39 find (Region method) 40 findAll() (Region method) Done 41 Finder (built-in class), [1] 42 Finder() (Finder method) 43 focus() (App class method) 44 focus (App method) 45 focusedWindow() (App class method) G 46 getAutoWaitTimeout() (Region method) 47 getBottomLeft() (Region method) 48 getBottomRight() (Region method) 49 getBounds() (Screen method) 50 getBundlePath() (built-in function) 51 getCenter() (Region method) 52 getClipboard() (Env method) 53 getFilename() (Pattern method) 54 getFindFailedResponse() (Region method) 55 getH() (Region method) 56 getImagePath() (built-in function) 57 getLastMatch() (Region method) 58 getLastMatches() (Region method) 59 getLocationFromPSRML() (Region method) 60 getMouseLocation() (Env method) 61 getNumberScreens() (Screen method) 62 getOS() (Env method) 63 getOSVersion() (Env method) 64 getParameter() (Vision method) 65 getRegionFromPSRM() (Region method) 66 getScore() (Match method) 67 getScreen() (Region method)
  • 84. Sikuli Satish Gorripotu 68 getSikuliVersion() (Env method) 69 getTarget() (Match method) 70 getTargetOffset() (Pattern method) 71 getThrowException() (Region method) 72 getTopLeft() (Region method) 73 getTopRight() (Region method) 74 getW() (Region method) 75 getX() (Location method) 76 getX(Region method) 77 getY() (Location method) 78 getY(Region method) 79 guide (module) getRect getROI H 80 hasNext() (Finder method) 81 highlight() (Region method) 82 hover() (Region method) I 83 Image Search Path 84 SIKULI_IMAGE_PATH 85 import .sikuli 86 InfoLogs (Settings attribute) 87 input() (built-in function) Done 88 inside() (Region method) 89 isLockOn() (Env method) K 90 Key (built-in class) 91 keyDown() (Region method) 92 KeyModifier (built-in class) 93 keyUp() (Region method) L 94 left() (Location method) 95 Left (Region method) 96 load() (built-in function) 97 Location (built-in class) 98 Location() (Location method) M 99 Match (built-in class), [1] 100 match (SikuliEvent attribute) 101 META (built-in variable) 102 MinSimilarity (Settings attribute)
  • 85. Sikuli Satish Gorripotu 103 morphTo() (Region method) 104 mouseDown() (Region method) 105 mouseMove() (Region method) 106 mouseUp() (Region method) 107 MoveMouseDelay (Settings attribute) 108 moveTo() (Region method) N 109 nearby() (Region method) 110 next() (Finder method) O 111 observe() (Region method) 112 ObserveMinChangedPixels (Settings attribute) 113 ObserveScanRate (Settings attribute) 114 offset() (Location method) 115 offset(Region method) 116 onAppear() (Region method) 117 onChange() (Region method) 118 onVanish() (Region method) 119 open() (App class method) 120 open(App method) 121 openApp() (built-in function) P 122 paste() (Region method) 123 Pattern (built-in class), [1] 124 pattern (SikuliEvent attribute) 125 Pattern() (Pattern method) 126 popup() (built-in function) Done R 127 rectangle() (in module guide) Region (built-in class), [1], [2], [3], [4], [5], [6], [7], 128 [8], [9] 129 region (SikuliEvent attribute) 130 Region() (Region method) 131 removeHotkey() (Env method) 132 removeImagePath() (built-in function) 133 right() (Location method) 134 right(Region method) 135 rightClick() (Region method) Done 136 run() (built-in function) S 137 Screen (built-in class), [1], [2] 138 Screen() (Screen method) 139 selectRegion() (Screen method)
  • 86. Sikuli Satish Gorripotu 140 setAutoWaitTimeout() (Region method) 141 setBundlePath() (built-in function) 142 setFindFailedResponse() (Region method) 143 setH() (Region method) 144 setLocation() (Location method) 145 setParameter() (Vision method) 146 setRect() (Region method) 147 setROI() (Region method) 148 setShowActions() (built-in function) 149 setThrowException() (Region method) 150 Settings (built-in class) 151 setW() (Region method) 152 setX() (Region method) 153 setY() (Region method) 154 SHIFT (built-in variable) 155 show() (in module guide) 156 SIKULI_IMAGE_PATH 157 Image Search Path 158 SikuliEvent (built-in class) 159 similar() (Pattern method) 160 SlowMotionDelay (Settings attribute) 161 stopObserver() (Region method) 162 switchApp() (built-in function) setX (Location) setY (Location) T 163 targetOffset() (Pattern method) 164 text() (in module guide) 165 text(Region method) 166 tooltip() (in module guide) 167 type (SikuliEvent attribute) 168 type() (Region method) Done W 169 wait() (Region method) Done 170 WaitScanRate (Settings attribute) 171 waitVanish() (Region method) 172 wheel() (Region method) 173 WIN (built-in variable) 174 window() (App method)
  • 87. Sikuli Satish Gorripotu Java Docs http://sikuli.org/doc/java-x/S.No All Classes 1 Animator 2 App 3 AppearEvent 4 AppNotFound 5 Button 6 CapturePrompt 7 ChangeEvent 8 Clipboard 9 Clipboard.Charset 10 Clipboard.TextType 11 Clipboard.TransferType 12 Constants 13 Debug 14 DesktopRobot 15 DummyOSUtil 16 Env 17 EventManager 18 ExtensionManager 19 Finder 20 FindFailed 21 FindFailedResponse 22 HotkeyEvent 23 HotkeyListener 24 ImageLocator 25 IRobot
  • 88. Sikuli Satish Gorripotu 26 IRobot.KeyMode 27 IScreen 28 Key 29 KeyModifier 30 LinuxUtil 31 Location 32 MacUtil 33 Match 34 Observer 35 OpenCV 36 OS 37 OSUtil 38 Pattern 39 Region 40 ResourceExtractor 41 Screen 42 ScreenHighlighter 43 ScreenImage 44 ScriptRunner 45 Settings 46 SikuliAction 47 SikuliActionListener 48 SikuliActionManager 49 SikuliEvent 50 SikuliEvent.Type 51 SikuliEventAdapter 52 SikuliEventObserver 53 SikuliException 54 SikuliScript 55 Subject 56 TextRecognizer 57 TextRecognizer.ListTextMode 58 TransparentWindow 59 UnionScreen 60 Util 61 VanishEvent 62 VDictProxy 63 Win32Util
  • 89. Sikuli Satish Gorripotu Basics of Jython – Python for java Platform Sikuli Using IDE as Eclipse then, we have two choices. 1. Using Java as a scripting language 2. Using Python as scripting language by downloading Python or Jython plugIn to Eclipse. If we use default Sikuli Ide then we have to use Python or Jython as scripting language. http://jythonpodcast.hostjava.net/jythonbook/en/1.0/ Declare Variable Java – int x = 0; Python x=0 Reserved Words and assert break class continue def del elif else except exec finally for from global or pass print raise return try while with yield Coding Structure Java if-statement x = 100; if(x > 0) { System.out.println("Wow, this is Java"); } else { System.out.println("Java likes curly braces"); } Python if-statement x = 100 if x > 0: print Wow, this is elegant else: print Organization is the key
  • 90. Sikuli Satish Gorripotu Functions Example 1: def my_function_name(parameter_list): implementation Example 2: def my_simple_function(): ... print This is a really basic function my_simple_function() Example 3: def multiply_nums(x, y): ... return x * y multiply_nums(25, 7) 175 Example 4: def perform_math(oper): ... return oper(5, 6) perform_math(multiply_nums) 30 Classes Java - this Python - self >>> class my_object: ... def __init__(self, x, y): ... self.x = x ... self.y = y ... ... def mult(self): ... print self.x * self.y ... ... def add(self): ... print self.x + self.y ...
  • 91. Sikuli Satish Gorripotu >>> obj1 = my_object(7, 8) >>> obj1.mult() 56 >>> obj1.add() 15 Statements Statement Keywords if-elif-else for while continue break try-except-finally assert def print del raise import if-elif-else Statement if <an expression to test>: perform an action else: perform a different action >>> x = 3 >>> y = 2 >>> if x == y: ... print x is equal to y ... elif x > y: ... print x is greater than y ... else: ... print x is less than y ... x is greater than y print Statement Java Print Output Example System.out.println("This text will be printed to the command line"); Python Print Output Example print This text will be printed to the command line >>> my_value = Satish Gorripotu >>> print my_value Satish Gorripotu >>> print My Name is Satish, but people called me as + my_value My Name is Satish, but people called me as Satish Gorripotu >>> z = 10 >>> print I am a fan of the number: + z Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: cannot concatenate str and int objects
  • 92. Sikuli Satish Gorripotu To resolve, we have to use below formatting operators * %s - String * %d - Decimal * %f - Float print String of text goes here %d %s %f % (decimalValue, stringValue, floatValue) >>> string_value = hello world >>> float_value = 3.998 >>> decimal_value = 5 >>> print Here is a test of the print statement using the values: %d, %s, and %f % (decimal_value, string_value, float_value) Here is a test of the print statement using the values: 5, hello world, and 3.998000 >>> x = 1 >>> y = 2 >>> print The value of x + y is: %d % (x + y) The value of x + y is: 3 >>> x = 2.3456 >>> print %s % x 2.3456 >>> print %d % x 2 >>> print %f % x 2.345600 try-except-finally >>> # Suppose weve calculated a value and assigned it to x >>> x 8.97 >>> y = 0 >>> try: ... print Just Print: %f % (x/y) ... except: ... print ‗Satti, we have a problem.... Satti, we have a problem. raise Statement the raise statement is used to throw or ―raise‖ an exception in Python >>> raise NameError Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError
  • 93. Sikuli Satish Gorripotu >>> raise Exception(Custom Exception) Traceback (most recent call last): File "<stdin>", line 1, in <module> Exception: Custom Exception import Statement # Import a module named TipCalculator import TipCalculator # Import a function tipCalculator from within a module called ExternalModule.py from ExternalModule import tipCalculator To avode over riding import tipCalculator as tip Iteration Index start from zero >>> my_numbers = [1, 2, 3, 4, 5] >>> my_numbers [1, 2, 3, 4, 5] >>> my_numbers[1] 2 >>> for value in my_numbers: ... print value ... 1 2 3 4 5 >>> x = 0 >>> while x < len(my_numbers): ... print my_numbers[x] ... x = x + 1 ... 1 2 3 4 5 While Loop
  • 94. Sikuli Satish Gorripotu while True perform operation Example of a Java While Loop int x = 9; int y = 2; int z = x – y; while (y < x) { System.out.println("y is " + z + " less than x"); y = y++; } Example of a Python While Loop >>> x = 9 >>> y = 2 >>> while y < x: ... print y is %d less than x % (x-y) ... y += 1 ... y is 7 less than x y is 6 less than x y is 5 less than x y is 4 less than x y is 3 less than x y is 2 less than x y is 1 less than x For Loop for each value in this defined set: perform suite of operations Example of Java For Loop for (x = 0; x <= 6; x++){ System.out.println(x); } Example of Python For Loop >>> for x in range(10): ... print x ... 0 1 2 3 4 5
  • 95. Sikuli Satish Gorripotu Basic Keyboard Input 1. raw_input() 2. input() # The text within the function is optional, and it is used as a prompt to the user >>> name = raw_input("Enter Your Name:") Enter Your Name:satish >>> print name satish # Use the input function to evaluate an expression entered in by the user >>> val = input (Please provide an expression: ) Please provide an expression: 9 * 3 >>> val 27 # The input function raises an error if an expression is not provided >>> val = input (Please provide an expression: ) Please provide an expression: My Name is Satti Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 1 My Name is Satti ^ SyntaxError: invalid syntax String MethodsMethod Description of Functionalitycapitalize() Returns a capitalized copy of string Returns a repositioned string with specified width and providecenter (width[,fill]) optional padding filler character Count the number of distinct times the substring occurs within thecount(sub[,start[,end]]) stringdecode([encoding[,errors]]) Decodes and returns Unicode stringencode([encoding[,errors]]) Returns an encoded version of a stringendswith(suffix[,start[,end]]) Returns a boolean to state whether the string ends in a given patternexpandtabs([tabsize]) Converts tabs within a string into spaces Returns the index of the position where the first occurrence of thefind(sub[,start[,end]]) given substring begins Returns the index of the position where the first occurrence of the given substring begins. Raises a ValueError with the substring is notindex(sub[,start[,end]) found. Returns a boolean to state whether the string contain only alphabeticisalnum() and numeric characters Returns a boolean to state whether the string contains all alphabeticisalpha() characters Returns a boolean to state whether the string contains all numericisdigit() characters Returns a boolean to state whether a string contains all lowercaseislower() characters
  • 96. Sikuli Satish Gorripotu Returns a boolean to state whether the string consists of allisspace() whitespace Returns a boolean to state whether the first character of each word inistitle() the string is capitalized Returns a boolean to state whether all characters within the string areisupper() uppercase Returns a copy of sequence joined together with the original stringjoin(sequence) placed between each element Returns a string of the specified width along with a copy of the original string at the leftmost bit. (Optionally padding empty spaceljust(width[,fillchar]) with fillchar) Returns a copy of the original string with all characters in the stringlower() converted to lowercase Removes the first found characters in the string from the left that match the given characters. Also removes whitespace from the left.lstrip([chars]) Whitespace removal is default when specified with no arguments. Returns a partitioned string starting from the left using the providedpartition(separator) separator Returns a copy of the original string replacing the portion of stringreplace(old,new[,count]) given in old with the portion given in new Searches string from right to left and finds the first occurrence of therfind(sub[,start[,end]]) given string and returns highest index where sub is found Searches string from right to left and finds the first occurrence of the given string and either returns highest index where sub is found orrindex(sub[,start[,end]]) raises an exceptionrjust(width[,fillchar]) Returns copy of string Aligned to the right by width Returns a copy of stringPartitioned starting from the right using therpartition(separator) provided separator object Returns list of words in string and splits the string from the right side and uses the given separator as a delimiter. If maxsplit is specifiedrsplit([separator[,maxsplit]]) then at most maxsplit splits are done (from the right). Returns copy of string removing the first found characters in the string from the right that match those given. Also removesrstrip([chars]) whitespace from the right when no argument is specified. Returns a list of words in string and splits the string from the left sidesplit([separator[,maxsplit]]) and uses the given separator as a delimiter. Splits the string into a list of lines. Keepends denotes if newlinesplitlines([keepends]) delimiters are removed. Returns the list of lines in the string. Returns a boolean to state whether the string starts with the givenstartswith(prefix[,start[,end]]) prefix Returns a copy of string with the given characters removed from thestrip([chars]) string. If no argument is specified then whitespace is removed. Returns a copy of the string the case of each character in the stringswapcase() converted. Returns a copy of the string with the first character in each wordtitle() uppercase. Returns a copy of the string using the given character translation table to translate the string. All characters occurring in optionaltranslate(table[,deletechars]) deletechars argument are removed. Returns a copy of string with all of the characters in the stringupper() converted to uppercase Returns a numeric string padded from the left with zeros for thezfill(width) specified width.
  • 97. Sikuli Satish Gorripotu Examples: our_string=python is the best language ever # Capitalize first character of a String >>> our_string.capitalize() Python is the best language ever # Center string >>> our_string.center(50) python is the best language ever >>> our_string.center(50,-) ---------python is the best language ever--------- # Count substring within a string >>> our_string.count(a) 2 # Count occurrences of substrings >>> state = Mississippi >>> state.count(ss) 2 # Partition a string returning a 3-tuple including the portion of string # prior to separator, the separator # and the portion of string after the separator >>> x = "Hello, my name is Josh" >>> x.partition(n) (Hello, my , n, ame is Josh) # Assuming the same x as above, split the string using l as the separator >>> x.split(l) [He, , o, my name is Josh] # As you can see, the tuple returned does not contain the separator value # Now if we add maxsplits value of 1, you can see that the right-most split is # taken. If we specify maxsplits value of 2, the two right-most splits are taken
  • 98. Sikuli Satish Gorripotu >>> x.split(l,1) [He, lo, my name is Josh] >>> x.split(l,2) [He, , o, my name is Josh] Lists Defining Lists # Define an empty list my_list = [] my_list = list() # rarely used # Single Item List >>> my_list = [1] >>> my_list # note that there is no need to use print to display a variable in the >>> # interpreter [1] # Define a list of string values my_string_list = [Hello, Jython ,Lists] # Define a list containing mulitple data types multi_list = [1, 2, three, 4, five, six] # Define a list containing a list combo_list = [1, my_string_list, multi_list] # Define a list containing a list inline >>> my_new_list = [new_item1, new_item2, [1, 2, 3, 4], new_item3] >>> print my_new_list [new_item1, new_item2, [1, 2, 3, 4], new_item3]
  • 99. Sikuli Satish Gorripotu Accessing a List # Obtain elements in the list >>> my_string_list[0] Hello >>> my_string_list[2] Lists # Negative indexes start with the last element in the list and work back towards the first # item >>> my_string_list[-1] Lists >>> my_string_list[-2] Jython # Using slicing (Note that slice includes element at starting index and excludes the end) >>> my_string_list[0:2] [Hello, Jython] # Create a shallow copy of a list using slice >>> my_string_list_copy = my_string_list[:] >>> my_string_list_copy [Hello, Jython, Lists] # Return every other element in a list >>> new_list=[2, 4, 6, 8, 10, 12, 14, 16, 18, 20] # Using a third parameter in the slice will cause a stepping action to take place # In this example we step by one >>> new_list[0:10:1]
  • 100. Sikuli Satish Gorripotu [2, 4, 6, 8, 10, 12, 14, 16, 18, 20] # And here we step by two >>> new_list[0:10:2] [2, 6, 10, 14, 18] # Leaving a positional index blank will also work as the default is 0 for the start, and the length of the string for the end. >>> new_list[::2] [2, 6, 10, 14, 18] Modifying a List # Modify an element in a list. In this case well modify the element in the 9th position >>> new_list[9] = 25 >>> new_list [2, 4, 6, 8, 10, 12, 14, 16, 18, 25] Boolean Expressions Example: Testing a String >>> mystr = >>> if mystr: ... Now I contain the following: %s % (mystr) ... else: ... I do not contain anything ... I do not contain anything >>> mystr = Now I have a value >>> if mystr: ... Now I contain the following: %s % (mystr) ... else: ... I do not contain anything ... Now I contain the following: Now I have a value Using Expressions to Control Program Flow if-elif-else Statement
  • 101. Sikuli Satish Gorripotu Example: Standard if-elif-else # terminal symbols are left out of this example so that you can see the precise indentation pi =3.14 x = 2.7 * 1.45 if x == pi: print The number is pi elif x > pi: print The number is greater than pi else: print The number is less than pi Example: Evaluate Empty List # Use an if-statement to determine whether a list is empty # Suppose mylist is going to be a list of names >>> mylist = [] >>> if mylist: ... for person in mylist: ... print person ... else: ... print The list is empty ... The list is empty while Loop Python while Statement >>> x = 0 >>> y = 10 >>> while x <= y: ... print The current value of x is: %d % (x) ... x += 1 ... else: ... print Processing Complete... ... The current value of x is: 0 The current value of x is: 1 The current value of x is: 2 The current value of x is: 3 The current value of x is: 4 The current value of x is: 5 The current value of x is: 6 The current value of x is: 7 The current value of x is: 8 The current value of x is: 9 The current value of x is: 10 Processing Complete...
  • 102. Sikuli Satish Gorripotu esetting Counter Using with-else >>> total = 0 >>> x = 0 >>> y = 20 >>> while x <= y: ... total += x ... x += 1 ... else: ... print total ... total = 0 ... 210 continue Statement # Iterate over range and print out only the positive numbers >>> x = 0 >>> while x < 10: ... x += 1 ... if x % 2 != 0: ... continue ... print x ... 2 4 6 8 10 break Statement >>> x = 10 >>> while True: ... if x == 0: ... print x is now equal to zero! ... break ... if x % 2 == 0: ... print x ... x -= 1 ... 10 8 6 4 2 x is now equal to zero! for Loop
  • 103. Sikuli Satish Gorripotu Comparing Java and Python for-loop Example of Java for-loop for(x = 0; x <= myList.size(); x++){ // processing statements iterating through myList System.out.println("The current index is: " + x); } Example of Python for-loop my_list = [1,2,3,4,5] >>> for value in my_list: # processing statements using value as the current item in my_list ... print The current value is %s % (value) ... The current value is 1 The current value is 2 The current value is 3 The current value is 4 The current value is 5 Enumerate() Functionality >>> myList = [jython,java,python,jruby,groovy] >>> for index, value in enumerate(myList): ... print index, value ... 0 jython 1 java 2 python 3 jruby 4 groovy If we do not require the use of an index, it can be removed and the syntax can be cleaned up a bit. >>> myList = [jython, java, python, jruby, groovy] >>> for item in myList: ... print item ... jython java python jruby groovy Exception Handling - Need to work on it JDBC - Need to work on Data Base conncetions
  • 104. Sikuli Satish Gorripotu from __future__ import with_statement from com.ziclix.python.sql import zxJDBC # for example jdbc_url = "jdbc:postgresql:test" username = "postgres" password = "jython25" driver = "org.postgresql.Driver" # obtain a connection using the with-statment with zxJDBC.connect(jdbc_url, username, password, driver) as conn: with conn: with conn.cursor() as c: c.execute("select name from country") c.fetchone() Excel Handling - Need to work on Test tada and Test resultsReferenced and consolidated from below sites: http://sikuli.org/trac/wiki http://sikuli.org/trac/wiki/TitleIndex http://sikuli.org/trac/wiki/IPAddressMac-0.10?action=history http://sikuli.org/trac/wiki/WikiStart http://sikuli.org/trac/wiki/reference-0.10 http://sikuli.org/trac/wiki/RaiMansExamples http://sikuli.org/trac/wiki/The%20Complete%20Guide%20To%20Sikuli%20X http://sikuli.org/trac/wiki/A%20Quick%20Tour%20of%20Sikuli%20Script%20IDE?action=diff&ver sion=7 http://sikuli.org/trac/prefs http://sikuli.org/docx/ http://sikuli.org/documentation.shtml http://script.sikuli.org/documentation.shtml#doc/pythondoc- python.edu.mit.csail.uid.Sikuli.html#python.edu.mit.csail.uid.Sikuli.type-function http://sikuli.csail.mit.edu/trac/wiki/TitleIndex http://www.csail.mit.edu/search/node/sikuli
  • 105. Sikuli Satish Gorripotu http://www.makeuseof.com/tech- fun/search/?cx=009717636731598800244%3Aqhe4rh7wuxs&cof=FORID%3A11&q=sikuli#773 https://launchpad.net/sikuli/trunk/0.10 http://jythonpodcast.hostjava.net/jythonbook/en/1.0/ http://www.jython.org/jythonbook/en/1.0/LangSyntax.html#while-loop http://www.jython.org/jythonbook/en/1.0/LangSyntax.html#for-loop http://www.jython.org/jythonbook/en/1.0/InputOutput.html#file-i-o AutoHotkey : same like Sikuli, But takes Keyboard actions. http://www.autohotkey.com/docs/Tutorial.htm AutoHotkey supports image recognition and macro recording. I dont see much here that is new. The bus proximity and lasso search examples are both great ideas, but could each be easily implemented using either AutoHotkey or AutoIt.