This PDF illustrates how to use Sikuli tool for web page automation

1. Sikuli
Mohd Shahabuddin Page 1
What is Sikuli?
 Automation scripting tool
 Supports Python, Ruby and Java language
 Runs on Mac, Linux/Unix and Windows
Why Sikuli?
 Automates anything you see on the screen of your desktop computer running
Windows, Mac or Linux/Unix.
 It uses image recognition to identify and control GUI components.
 Sikuli is the only option when there is no way to identify web elements
 It can run the mouse and the keyboard to interact with the identified GUI
elements.
History:
 Open-source research project.
 Started at MIT
 Started by Tsung-Hsiang Chang and Tom Yeh
What are the prerequisite to learn Sikuli?
 Any scripting language with selenium web driver
How to install Sikuli in maven project?
Step-1:
 Add the following dependency in pom.xml file
<dependency>
<groupId>org.sikuli</groupId>
<artifactId>sikuli-java</artifactId>
<version>1.0.1</version>

2. Sikuli
Mohd Shahabuddin Page 2
</dependency>
 If not downloaded automatically during maven clean install then manually
download 'sikuli-java.jar' and save it on specific location say ‘c:sikulisikuli-
java.jar’
i) Open command prompt.
ii) Run the below command.
mvn install:install-file -Dfile= c:sikulisikuli-java.jar -DgroupId=org.sikuli -
DartifactId=sikuli-java -Dversion=1.0.1 -Dpackaging=jar -DgeneratePom=true
iii) If you don’t see 'build successful' then re-trace your steps.
Step-2
 Go to the local repository of the user and verify the following files present:
How to type a text in a text field when using element locator?
Step-1

3. Sikuli
Mohd Shahabuddin Page 3
 Go to the google screen shown below:
Step-2
 Create a test to type text ‘Hello World’ in google search text field.
@Test()
public void sikuliTest() throws Exception {
Screen s = new Screen();
s.type("Hello World");
}
 Run as TestNG test

4. Sikuli
Mohd Shahabuddin Page 4
Step-3
 See text is typed in browser’s URL instead of google search text field because
Skilui captures entire screen and find a first text field anywhere on the screen and
perform the action on it.
How To distinguish one image from other and perform action on it?
Step-1
 Let us create a test to type a text ‘Hello World’ in Google’s search text field
 Copy the portion of Google’s screen where we have search text field as shown
below in a .png file using MS Paint. Let file is saved in location
c:google‘GoogleTextField.png’
 Saved file loacation

5. Sikuli
Mohd Shahabuddin Page 5
v
Step-2
 Write a test to first verify the desired text field presence and then type a text into
it.
@Test()
public void sikuliTest() throws Exception {
String googleSearchTextFieldFilePath="C:/google/GoogleTextField.png";
Thread.sleep(2000);
Region r = new Screen();
//saved pattern on drive
Pattern savedPattern = new Pattern(googleSearchTextFieldFilePath);
//set the percentage of matching intensity. 0.8 is the maximum match value
savedPattern = savedPattern.similar(0.8f);
//check whether saved pattern is present on the current screen with expected
matching intensity( i.e 8).
//If match not found Null is returned
Match match = r.exists(savedPattern, 8);
logger.info("Pattern " + googleSearchTextFieldFilePath + "->" + match);
Assert.assertNotNull(match, "pattern "+googleSearchTextFieldFilePath+" not
found");
//to set the focus to google search text field
r.keyDown(KeyEvent.VK_ENTER);
//wait a bit
Thread.sleep(2000);
//type the desired text into matching pattern
r.type("Hello World");
}
 Right click and run as TextNG. Below is the output where test is typing text into
desired location

6. Sikuli
Mohd Shahabuddin Page 6
Further practices
 There are many APIs that can be used as per the requirement. Below screen
shows few of them:

7. Sikuli
Mohd Shahabuddin Page 7
***