The presentation contains brief introduction to a variety of algorithms which can be used for Image Recognition. It focuses on assessing the different tools and algorithms in terms of its reliability to identify desktop/web elements. The criterion is identified as such because the results of the research will be used for desktop automation. To know more about RAX Automation Suite, visit www.raxsuite.com

1. Image Recognition:
For RAX Studio Citrix Automation

2. Fast.ai
Deep Learning Algorithm
Convolutional Neural Network (CNN)
1

3. How it works:
Dog
3
Cat
System
Identifies:
What is this?
…..
…..

4. Feature Detection
Feature Matching
Specific patterns which are unique and can be
easily compared and tracked.
SIFT - Scale-Invariant Feature Transform
SURF - Speeded-Up Robust Features
ORB - Oriented FAST and Rotated BRIEF
2

5. How it works:
5

6. Template Matching
Searching for an object in an image3

7. How it works:
7

8. Calculation Methods:
8
SQDIFF - calculates square difference
CCORR - calculates correlation
COEFF - calculates correlation coefficient

9. Optical Character
Recognition
Electronic conversion of images to machine-
encoded text
4

10. 10
How it should work:
- Use OCR on the whole
screen
- Find the nth occurence
of word
- Get the image position
of that word

11. “
- We cannot train fast.ai to identify all
icons/text that the user will use for
automation.
- Feature matching is good for finding
scene images but an image of a website
,for example, could be full of text.
Therefore confusing the features to be
matched.
11
Assessments:

12. “
- Template Matching is the most reliable
one for citrix automation especially if it
is for matching images with static
graphical user interface.
- OCR is good for web content which
contains dynamic changes to its
graphical design that may be hard for
template matching to track.
12
Assessments:

13. Proposal:
Template
Matching
13
Make use of 2 different methods to broaden the
option of the users.
OCR

14. Proposal:
Template
Matching
14
- Template Matching would be used for content
which is unique and those with minimal
changes to the GUI.
- The user should be able to choose the size of
the cropped template. Larger template means
more unique elements included.

15. Proposal:
15
- If template matching still fails, then the user
should opt in to OCR.
- OCR should be able to find the nth occurrence
of the word and get its position.
- Then it could click anywhere on the screen
relative to the text’s position.
OCR

16. Suggestion:
16
- The keyboard keys & shortcut keys would be a
great tool for navigating and getting the cursor
to the different textboxes/links.
Ex. Tab, Page up, Page Down
Win + Down = Minimize
etc.

17. Thanks!
Any questions?
17

18. References &
Resources:
◇ https://docs.fast.ai/
◇ http://cs231n.github.io/convolutional-networks/
◇ https://docs.opencv.org/
◇ https://www.kaggle.com/wesamelshamy/tutorial-image-feature-
extraction-and-matching
◇ http://www.aishack.in/tutorials/template-matching/
◇ http://scikit-image.org/docs
◇ https://www.slidescarnival.com/
◇ Google Images
18