Thesis

Applying Sequential Space Relativity for Image Recognition
By
Roby Chhokar
A thesis submitted in partial fulfillment of the requirements
For the degree of
Master of Science in Computer and Information Sciences
Approved by: ___________________________________________________________
Professor George F. Schmoll, III, Thesis Advisor
Date: ___________________________________________________________
Knowledge Systems Institute
Skokie, Illinois
May 2006

Abstract
In this research we attempt to perform image recognition using some of the known
properties that humans possess. In particular, we use the property of the eye performing
saccades on an image. We correlated space and time as the essence of this type of behavior. We
described space as the amount of space the current image portion in the visual was taking during
a saccade, and time was described as the sequence of saccades. From intuition, we introduced
the notion of relativity from one saccade to another, and used this idea to build a sequence.
A system was developed to implement this theory that consisted of 100 training images,
and 400 distorted images to test the system's recognition ability. The distorted images were
created from the training images and had either been: rotated/mirrored; scaled (larger or smaller);
partially removed; or noise-added. The system presented a 95% accuracy rate for partial
images, 62% for scaled images, 60% for noise-added images, and 55% for flipped images.
We found that the theory’s applicability may be promising for industrial applications, if it
can exhibit sequence path changes and saccadic distance changes dynamically during
recognition.
ii

Acknowledgements
Trying to create an algorithm and/or mathematical formula(s) that will model the
intelligence humans and other living species is a very difficult task, and perhaps to others,
impossible. I am pleased to have had the opportunity to use some of the known features of
intelligence to perform image recognition.
I would like to thank Professor Fred Thulin for initially assisting me with the directions
to take for my Thesis and for pointing me to Professor George Schmoll, who has helped me
tremendously. Professor Schmoll introduced me to the concept of humans making saccadic eye
movements to recognize images correlating to the essence of time, which is the basis of this
research.
Finally I would like to thank my parents, two daughters and wife for supporting me in my
very ambitious research and everything else.
iii

Table of Contents
Abstract……………………………………………………………………………………………ii
Acknowledgements………………………………………………………………………………iii
Table of Contents…………………………………………………………………………………iv
List of Figures…………………………………………………………………………………….vi
List of Tables…………………………………………………………………………………….vii
Chapter 1 Introduction…………………………………………………………………………….1
Background………………………………………………………………………………..1
Neurons……………………………………………………...…………………….1
Saccadic Eye Movements…………………………………………………………5
Overview………………...………………………………………………………………...5
Chapter 2 Building the Theory……………………………………………………………………7
Foundation………………………………………………………………………………...7
The Relativity Function……………..…………………………………………………….7
The Score Function………………………………………………………………………..8
Chapter 3 The Program…………………………………………………………………………..10
User Interface…………………………………………………………………………….10
Implementation…………………………………………………………………………..16
iv

Chapter 4 Image Recognition……………………………………………………………………41
Test Results………………………………………………………………………………42
Analysis of test Results…………………………………………………………………..46
Chapter 5 Conclusion…………………………………………………………………………….48
Appendix A Program Code………………………………………………………………………49
References………………………………………………………………………………………..61
v

List of Figures
Figure 1.1: Schematic diagram of a neuron……………………………………………………….1
Figure 1.2: Varying input to neuron ……………………………..……………………………….2
Figure 1.3: Summary of the neuro-computational properties……………………………………..4
Figure 1.4: Saccadic Eye Movement on Human Face…………………………………………….5
Figure 3.1: Initial Program Screen from version 2.……………………………………………...10
Figure 3.2: Program Screens from version 1 Training…………………………………………..14
Figure 3.3: Program Screen from version 1 Recognition………………………………………..15
Figure 3.4: Program Screen from version 2 Results……………………………………………..16
Figure 3.5: Train Method………………………….……………………………………………..17
Figure 3.6: Recognize Method…………………………………………………………………...18
Figure 3.7: Score Method………………………………………………………………………..19
Figure 3.8: Saccade Path…………………………………………………………………………20
vi

List of Tables
Table 4.1: Image Recognition Results…………………………………………………………...22
Table 4.2: Image Types with number of Rank 1………………………………………………...46
vii

Chapter 1 Introduction
Background
Neurons
To understand the motivation and basis of this research, we will need to appreciate some
of the known workings of the brain. How does the brain work? More specifically, how does the
brain recognize the same object despite its variance through the sense(s) it is perceived?
Furthermore, how is it able to perform this remarkable task merely based off axons firing over
time? Despite the tremendous research that has been performed on this subject, it is still
unanswered and probably one of the greatest mysteries known to human beings.
A neuron is the most basic processing unit in the brain. A typical neuron receives input
from more than 10,000 other neurons through the contacts on its dentritic tree called synapses. If
a particular threshold is met through its input (dendrites), then the neuron will fire, this is sent
through its axon. There is one and only one axon for any neuron. The axon can connect with
other dendrites via synapses, which will then be input to other neurons. Please see Figure 1.1 for
a schematic image of a neuron with its associated connections.
1

Figure 1.1: Schematic diagram of a neuron. The Terminal buttons are the synapses. [1]
What is the threshold? The threshold potential can be as low as -55mV or as high as
-40mV. This means when the neuron’s threshold potential is met, then the neuron will fire. The
resting membrane potential is lower than the threshold potential and is typically from -70mV to
-60mV [2]. When the neuron fires (A.K.A. spikes), the membrane potential reaches +30mV for
1 to 2 ms. It is agreed upon that the neuron does fire when the membrane potential reaches the
threshold potential for that neuron. But what makes the membrane potential reach the threshold
potential?
The classic view is that if enough input is applied to a neuron then the membrane
potential will reach the threshold potential and fire. However experiments have been performed
that contradict this notion [2]. As it can be seen in figure 1.2, the neuron fires on
Figure 1.2: Varying input to neuron. Here is the resonant response of the mesencephalic V
neuron of rat brainstem to pulses of injected current having 10ms period (in vitro). [2]
varying input. Please note the “inhibitory burst” figure in 1.2, the neuron fires to bursts of
negative input, which contradicts the idea that a neuron will only fire if enough positively input
is applied. It has been argued that neurons are dynamical systems [2] which we agree with. The
Spiking Model by Izhikevich [3] (biologically plausible as the Hodgkin-Huxley model) can
2

exhibit firing patterns of all known types of cortical neurons with the choice of parameters a, b,
c, and d, which demonstrates the dynamical systems of neurons.
Here variable v represents the membrane potential of the neuron and u represents a membrane
recovery variable, which accounts for K+ ionic currents and inactivation of Na+ ionic currents,
and it provides negative feedback to v. After the spike reaches +30mV, the membrane voltage
and the recovery variable are reset according to (3). If v skips over 30, then it first is reset to 30,
and then to c so that all spikes have equal magnitudes. Figure 1.3 demonstrate the various neuro-
computational properties of biological spiking neurons by adjusting the parameters a, b, c, and d.
Please note the varying behavior of the neuron spikes with varying input. It appears the neuron
fires based off its current and past input.
3

Figure 1.3: Summary of the neuro-computational properties of biological spiking neurons using
the Spiking Model by Izhikevich [3][4].
4

Saccadic Eye Movement
Saccadic eye movements occur when the eyes fixate from one point to another. One
reason why the eye saccades from one point to another is to keep the image of interest in the area
of the eye that has the highest concentration of cone cells, the Optic fovea [7]. Exactly how the
brain uses the information from the saccades, and knowing when and where to saccade is
currently unknown. But it is generally agreed upon that the eye makes saccade movements at
least during recognition.
Figure 1.4: The picture on the right shows the gaze of a subject exploring the portrait on the left.
[6]
Overview
Although this research is to apply sequential space relativity for image recognition, the
reader must note this theory can be applied to anything that can be broken down to space and
time, which will be defined in the next chapter. This is analogous to the brain in that it is not
5

concerned with the actual sense producing the stimulus (for instance, stimulus from the cochlea);
rather it is concerned with axons firing over time.
Chapter 2 will build the theory; Chapter 3 will apply the theory with implementation.
Chapter 4 will present the test results with analysis; and Chapter 5 will have the conclusions.
6

Chapter 2 Building the Theory
Foundation
In this section we build the foundation for our theory.
• Let e1, e2, …, en represent a sequence of events.
• Let r(1,2), r(2,3), …, r(n-1,n) represent the sequence of relativities derived from the
sequence of events, that is, r(n-1,n) is the relativity derived from the sequence en-1, en.
• Let the set R be a one-to-one function, where the domain and range are the events in E.
The Relativity Function
From our intuition we use the concept of relativity as the basis for sequence analysis. From
the definitions in the previous section we have, R: E → E, where R is the relativity one-to-one
function that maps an event from E to another event in E. Although there are no hard set of rules
7

to follow to determine which relativity function to use, we however, want to choose a function
that fits our perception of the relativity from one event to another. So in this research, the events
represent the amount of space being used from one saccade to another. We define the relativity
function as:
r(n) = e(n+1) / e(n) , where e(n), e(n+1) ∈ and r(n) ∈
The Score Function
The score function is used to determine the score of how close one sequence matches
another. The higher the score, the closer the match, which is somewhat similar to the BLAST
theory used in DNA sequence analysis [8].
• Let i represent a relativity of the sequence of relativities I.
• Let j represent a relativity of the sequence of relativities J.
• We will define the score function as:
S(i,j) = i / (i + E(i,j) ), where E(i,j) = |i - j| and i,j > 0
We will now justify how this function was derived. When determining the score outcome from
comparing two relativities, i and j, we defined the upper bound to be 1, which will be reached if
the relativities match exactly. The lower bound to be 0, which will be approached as the error
(difference) between the relativities increases.
lim S(i,j) = 1
E(i,j) → 0
lim S(i,j) = 0
E(i,j) → ∞
8

In order to achieve this type of behavior, we intuitively started with f: x → 1/x and modified it to
better suit our needs, which gave us the score function defined.
To achieve the total score when comparing two sequences, we sum the comparative result
from each score function per alignment. Since we ensured every saccade would have at least 1
unit of space (1 black pixel), we always received a result from the score function in the range
(0,1] for each alignment. All sequences had to be the same length (training and recognition), so
the maximum score any two sequences could have when compared to each other is the sequence
length (predefined) multiplied by the maximum score (1) per alignment comparison.
9

Chapter 3 The Program
User Interface
The program contained two versions. The first version was more graphical where the
user could observe the saccades on each image that was manually chosen. The second version
was developed for speed where the user could not see the saccades and all the images to train
were automatically trained from a predefined directory. The user interface of the program
required the user to first train the system on a set of images.
Figure 3.1: Initial Program Screen from version 2.
10

11

12

13

Figure 3.2: Program Screen from version 1 Training. Each snapshot is a saccade on the letter
“A”.
After the user had trained the system on the set of images, a distorted image that
resembled one of the training images was tested to determine the system’s recognition ability,
which was determined by the training image that received the highest score. See Figure 3.4.
14

Figure 3.3: Program Screen from version 1 Recognition. A distorted “A”.
15

Figure 3.4: Program Screen from version 2 Results. The results after presenting a distorted “A”.
Implementation
The program was written in Microsoft C# 2005 Express Edition [5]. We trained and
performed the recognition on 32 by 32 bitmap (black & white) pixels. Each image was
represented by black pixels. When the system was instructed to be in ‘train’ mode, it read from
every bitmap image within a predefined directory (version 2). For each image, it determined the
total amount of black pixels that existed for each saccade, which was then written to memory (a
16

file – memory.txt). Each line in the file corresponded to an image’s sequence of space (number
of black pixels), for instance the letter “A”,
“A.bmp -97,96,95,93,91,89,86,84,81,78,76,73,71,69,66, …”.
Figure 3.5: Train Method. The primary method used during the training mode.
In the recognition mode, the system went through every image it was trained on and
generated the sequence of relativities accordingly. Each sequence was put in the score function
with the image to recognize, which also was transformed to a sequence of relativities.
17

Figure 3.6: Recognize Method. The primary method used during the recognition mode.
18

Figure 3.7: Score Method. The method used to determine the score for the two sequences.
To ensure no sequence had a space in it, every saccade had to have at least 1 pixel of the
image. Also, every image had to be centered in the bitmap because we wanted to ensure the
saccades were performed with respect to the center of the image.
There were a total of 256 saccades with one predefined path that was used during training
and recognition. Figure 3.8 shows the saccadic path. The start position was from the center of
the visual field. There were 16 saccades for each direction (arrow in diagram).
19

Figure 3.8: Saccade Path. The direction of the saccades.
20

Chapter 4 Image Recognition
Test Results
The following table contains the test results.
• The system was trained on 100 images (see “Training Image” column)
• The system performed recognition on 100 flipped images: horizontally, vertically and
rotated (see “Flipped Image” column).
• The system performed recognition on 100 noise-added images (see “Noise-added Image”
column), which meant additional black pixels were marked on the image screen.
• The system performed recognition on 100 partial images (see “Partial Image” column),
which meant black pixels were removed from the image.
• The system performed recognition on 100 scaled images (see “Scaled Image” column),
which meant the image was scaled to a greater or smaller size overall or just certain parts.
• The row above each image indicates the rank the recognized image had with respect to
the training image (note: the image to recognize is the training image modified as
described above). A rank 1 means the system recognized the corresponding training
image (in same row in the table) with the highest probability from the other training
images. And of course, as the rank number increases (up to a maximum of 100), the
system’s probability of recognition decreased with respect to the other training images. If
the system did not have rank 1 for the training image during recognition, then the training
image that was rank 1 was indicated.
21

Table 4.1: Image Recognition Results
Image to Recognize
Training Image Flipped Image Noise-added
Image
Partial Image Scaled Image
0.bmp Rank 1 Rank 1 Rank 1 Rank 1
1.bmp Rank 7
Rank 1 – A.bmp
Rank 1 Rank 1 Rank 1
1Over4.bmp Rank 2
Rank 1 –
AtSign.bmp
22

2Over4.bmp Rank 4
Rank 1 –
baseball.bmp
Rank 2
Rank 1 –
X.bmp
Rank 1 Rank 1
3.bmp Rank 26
Rank 1 – 8.bmp
Rank 6
Rank 1 –
X.bmp
Rank 1 Rank 1
3Over4.bmp Rank 7
Rank 1 –
AtSign.bmp
Rank 3
Rank 1 –
X.bmp
Rank 1 Rank 1
4.bmp Rank 1 Rank 2
Rank 1 –
X.bmp
Rank 1 Rank 1
23

5.bmp Rank 1 Rank 2
Rank 1 –
X.bmp
Rank 1 Rank 6
Rank 1 –
crown.bmp
Rank 1 – 8.bmp
7.bmp Rank 62
Rank 1 – S.bmp
Rank 25
Rank 1 –
X.bmp
Rank 1 Rank 1
8.bmp Rank 1 Rank 3
Rank 1 –
X.bmp
Rank 1 Rank 7
Rank 1 – X.bmp
24

A.bmp Rank 1 Rank 1 Rank 1 Rank 1
a_small.bmp Rank 1 Rank 5
Rank 1 –
X.bmp
Rank 1 Rank 32
Rank 1 – Q.bmp
Ampersand.bmp Rank 1 Rank 1 Rank 1 Rank 1
AtSign.bmp Rank 1 Rank 19
Rank 1 –
X.bmp
Rank 1 Rank 1
25

B.bmp Rank 1 Rank 1 Rank 1 Rank 2
Rank 1 – 8.bmp
baseball.bmp Rank 1 Rank 2
Rank 1 –
X.bmp
Rank 1 Rank 1
BigWheelJeep.bmp Rank 1 Rank 2
Rank 1 –
X.bmp
Rank 1 Rank 1
bowl.bmp Rank 34
Rank 1 –
FaceWithBeird.bmp
Rank 1 – X.bmp
box3d.bmp Rank 1 Rank 2
Rank 1 –
FaceWithBeird.bm
p
Rank 1 Rank 33
Rank 1 – X.bmp
26

building.bmp Rank 1 Rank 1 Rank 1 Rank 1
Bush.bmp Rank 1 Rank 2
Rank 1 –
X.bmp
Rank 1 Rank 2
Rank 1 – X.bmp
C.bmp Rank 4
Rank 1 – Q.bmp
Calculator.bmp Rank 27
Rank 1 –
Camera.bmp
Rank 9
Rank 1 –
X.bmp
Rank 1 Rank 2
Rank 1 -
copyright.bmp
Camera.bmp Rank 1 Rank 1 Rank 1 Rank 3
27

Rank 1 –
Dress.bmp
Car.bmp Rank 1 Rank 1 Rank 1 Rank 1
Continent.bmp Rank 3
Rank 1 – O.bmp
Rank 1 –
copyright.bmp
Copyright.bmp Rank 1 Rank 1 Rank 1 Rank 1
Crown.bmp Rank 15
Rank 1 –
FaceWithBeird.bmp
Rank 10
Rank 1 –
X.bmp
Rank 1 Rank 1
28

D.bmp Rank 1 Rank 2
Rank 1 –
X.bmp
Rank 1 Rank 1
DollarSign.bmp Rank 2
Rank 1 – X.bmp
Door.bmp Rank 1 Rank 27
Rank 1 –
FaceWithBeird.bm
p
Rank 1 Rank 1
Dress.bmp Rank 1 Rank 1 Rank 1 Rank 1
E.bmp Rank 32
Rank 1 – Z.bmp
Rank 1 – Z.bmp
29

E_small.bmp Rank 1 Rank 1 Rank 1 Rank 3
Rank 1 – 0.bmp
Eye.bmp Rank 21
Rank 1 –
AtSign.bmp
F.bmp Rank 73
Rank 1 – H.bmp
Rank 1 – P.bmp
Face1.bmp Rank 1 Rank 1 Rank 1 Rank 1
30

FaceAlien.bmp Rank 16
Rank 1 –
RegisteredTradeMa
rk .bmp
Rank 37
Rank 1 –
X.bmp
Rank 1 Rank 1
FaceChubby.bmp Rank 1 Rank 1 Rank 1 Rank 1
FaceDog.bmp Rank 1 Rank 1 Rank 3
Rank 1 –
e_small.bmp
Rank 2
Rank 1 – X.bmp
FaceWithBeird .bmp Rank 1 Rank 1 Rank 1 Rank 9
Rank 1 –
FaceWoman.bmp
31

FaceWithGlasses .bmp Rank 3
Rank 1 –
FaceDog.bmp
Rank 1 –
FaceWoman.bmp
FaceWithHat.bmp Rank 1 Rank 1 Rank 1 Rank 1
FaceWithMostache
.bmp
Rank 17
Rank 1 –
RegisteredTradeMa
rk. bmp
Rank 1 – X.bmp
FaceWoman.bmp Rank 9
Rank 1 –
FaceWithBeird.bmp
Rank 2
Rank 1 –
X.bmp
Rank 1 Rank 1
Flower.bmp Rank 1 Rank 2
Rank 1 -
FaceWithBeird.bm
p
Rank 1 Rank 1
32

G.bmp Rank 1 Rank 1 Rank 1 Rank 1
g.bmp Rank 1 Rank 2 Rank 3 Rank 4
Ghost.bmp Rank 1 Rank 1 Rank 1 Rank 1
Guitar.bmp Rank 2
Rank 1 –
Wrench.bmp
Rank 5
Rank 1 –
X.bmp
Rank 1 Rank 7
Rank 1 –
Ampersand.bmp
33

H.bmp Rank 90
Rank 1 –
PoundSign.bmp
Rank 3
Rank 1 – 0.bmp
Rank 1 Rank 1
Hammer.bmp Rank 3
Rank 1 –
ScrewDriver.bm
p
Hand.bmp Rank 1 Rank 1 Rank 1 Rank 1
HardDisk.bmp Rank 2
Rank 1 –
RegisteredTradeMa
rk .bmp
HatBirthday.bmp Rank 3
Rank 1 –
Wrench.bmp
Rank 5
Rank 1 –
X.bmp
Rank 2
Rank 1 –
Snowman.bmp
Rank 7
Rank 1 –
Snowman.bmp
34

Helicopter.bmp Rank 1 Rank 1 Rank 1 Rank 1
House.bmp Rank 4
Rank 1 –
Ampersand.bmp
J.bmp Rank 16
Rank 1 –
DollarSign.bmp
Rank 1 – Z.bmp
K.bmp Rank 37
Rank 1 – 3.bmp
Rank 1 –
RegisteredTradeMark .
bmp
35

LightSwitch.bmp Rank 95
Rank 1 –
PhoneCell.bmp
M.bmp Rank 38
Rank 1 – W.bmp
Rank 2
Rank 1 –
X.bmp
Rank 1 Rank 1
M_small.bmp Rank 4
Rank 1 –
FaceWithBeird
.bmp
Rank 2
Rank 1 –
eye.bmp
Rank 1 Rank 1
MountainsWithLake
.bmp
Rank 2
Rank 1 – X.bmp
Rank 2
Rank 1 –
X.bmp
Rank 2
Rank 1 -
Car.bmp
Rank 2
Rank 1 – X.bmp
N.bmp Rank 4
Rank 1 – X.bmp
36

O.bmp Rank 3
Rank 1 –
MountainsWithLak
e .bmp
P.bmp Rank 1 Rank 1 Rank 1 Rank 1
PaperWithWriting
.bmp
Rank 56
Rank 1 –
FaceWithBeird .
bmp
Rank 3
Rank 1 –
FaceDog.bmp
Rank 1 Rank 1
People.bmp Rank 1 Rank 2
Rank 1 –
X.bmp
Rank 1 Rank 1
37

PercentSign.bmp Rank 1 Rank 1 Rank 1 Rank 2
Rank 1 – eye.bmp
PhoneCell.bmp Rank 1 Rank 3
Rank 1 –
X.bmp
Rank 1 Rank 1
Plug.bmp Rank 98
Rank 1 –
Snowman.bmp
Rank 7
Rank 1 –
X.bmp
Rank 1 Rank 1
PoundSign.bmp Rank 1 Rank 2
Rank 1 –
copyright.bmp
Rank 1 Rank 47
Rank 1 – O.bmp
38

Q.bmp Rank 1 Rank 1 Rank 1 Rank 3
Rank –
Ampersand.bmp
R.bmp Rank 1 Rank 1 Rank 1 Rank 1
Rainman.bmp Rank 1 Rank 2
Rank 1 –
X.bmp
Rank 1 Rank 3
Rank 1 – X.bmp
RegisteredTradeMark .
bmp
Rank 1 Rank 2
Rank 1 –
X.bmp
Rank 1 Rank 1
39

Rocket.bmp Rank 1 Rank 9
Rank 1 –
X.bmp
Rank 1 Rank 17
Rank 1-
MountainWithLake
.bmp
S.bmp Rank 3
Rank 1 – 0.bmp
Sailboat.bmp Rank 51
Rank 1 -
FaceWithGlasse
s .bmp
Rank 21
Rank 1 –
bush.bmp
Rank 1 Rank 8
Rank 1 –
Ampersand.bmp
ScrewDriver.bmp Rank 1 Rank 1 Rank 1 Rank 1
40

Snowman.bmp Rank 1 Rank 16
Rank 1 –
X.bmp
Rank 1 Rank 16
Rank 1 – eye.bmp
Star.bmp Rank 1 Rank 1 Rank 1 Rank 36
Rank 1 – eye.bmp
Steps.bmp Rank 95
Rank 1 –
Camera.bmp
StickFgureHuman
.bmp
Rank 4
Rank 1 –
HatBirthday.bm
p
Rank 1 –
MountainsWithLake.b
mp
41

Suitcase.bmp Rank 1 Rank 40
Rank 1 –
X.bmp
Rank 1 Rank 21
Rank 1 –
FaceWithBeird .bmp
Sword.bmp Rank 1 Rank 3
Rank 1 –
X.bmp
Rank 1 Rank 34
Rank 1 – X.bmp
Table.bmp Rank 18
Rank 1 –
1Over4.bmp
Rank 14
Rank 1 –
X.bmp
Rank 1 Rank 1
Tree.bmp Rank 4
Rank 1 –
Ampersand.bmp
42

TreePine.bmp Rank 23
Rank 1 –
AtSign.bmp
Rank 1 Rank 6
Rank 1 –
FaceWoman
.bmp
Rank 1
TruckSemi.bmp Rank 1 Rank 53
Rank 1 –
X.bmp
Rank 1 Rank 12
Rank 1 – Q.bmp
U.bmp Rank 1 Rank 1 Rank 1 Rank 2
Rank 1 – G.bmp
V.bmp Rank 3
Rank 1 –
AtSign.bmp
Rank 1 –
FaceWithMostache
.bmp
43

W.bmp Rank 51
Rank 1 – M.bmp
Rank 3
Rank 1 –
X.bmp
Rank 1 Rank 2
Rank 1 – X.bmp
WindowWithCurtain.b
mp
Rank 3
Rank 1 –
FaceWithBeird .
bmp
Wrench.bmp Rank 17
Rank 1 –
HatBirthday.bmp
X.bmp Rank 1 Rank 1 Rank 1 Rank 1
44

Z.bmp Rank 1 Rank 1 Rank 1 Rank 1
Table 4.1: Image Recognition Results
Analysis of Test Results
In this section we will analyze the test results.
Image Type Recognition (number of Rank 1 out of the 100 images)
Flipped 55
Noise-added 60
Partial 95
Scaled 62
Table 4.2: Image Types with number of Rank 1
As it is illustrated the Partial images had the most amount of Rank 1 (95/100), where
flipped had the lowest (55/100).
45

Since we isolated the system to only saccade in a particular sequence, it is no surprise
that flipped images had the least amount of recognition. When an image is flipped or rotated, the
sequence must be relative to the way it was during training. So if the image during recognition
was rotated 90 degrees, then the sequence of saccades needs to start from 90 degrees relative to
the training image. Additionally if the image is mirrored, then the system would have to
sequence the mirrored image relatively to how it sequenced the training image.
Another type of adjustment that may need to occur during recognition is the relative
distance that occurs from one saccade to another. This applies to scaled images. Since the visual
window is the same size during training and recognition, the distance traveled from one saccade
to another during recognition would need to be increased (if image portion is larger) or decreased
(if image portion is smaller) to match the same amount of saccades used during training. For
example, if a human face image was trained and the system performed one saccade from the
nose to the left ear moving a distance of say 10 pixels, then during recognition if the face is twice
as large as in training, then the system would still need to perform one saccade from the nose to
the ear, but with a distance increase of 20 pixels.
Although the two types of adjustments just mentioned would dramatically increase the
recognition accuracy for flipped and scaled images, there is no saccadic adjustment (speaking in
the realm of this research) that could have been done to increase the partial and noise-added
images recognition accuracy. This is because the partial and noise-added images maintained the
exact orientation and size as the training images.
Chapter 5 Conclusion
46

In this research it was demonstrated that very high accuracy can be obtained in image
recognition by taking an image over time, which was accomplished by taking the sequential
space relativity from one saccade to another. For this theory to be practical in the real world,
dynamic saccadic adjustments would have to be made by the system during recognition to
accommodate image scaling or rotation/mirroring as discussed in the previous chapter.
Furthermore, the system must only saccade on parts of the visual field that are of interest, that is,
if the system is looking for a dog’s face, it must saccade in the relative manner as done in
training.
As mentioned in the introduction of this research, although the theory is applied
for image recognition, the theory can be applied to anything that can be broken down to space
and time. More specifically, at any particular time (sequence) there must be particular space,
where space can be anything that is guaranteed to occur. In this research the amount of space
(pixels) occupied from one saccade to another was guaranteed to occur.
Appendix A Program Code
The following is the program code used to make the system in this research. It is organized in
files.
Program.cs
using System;
using System.Collections.Generic;
using System.Windows.Forms;
namespace Thesis3
{
static class Program
{
/// <summary>
/// The main entry point for the application.
/// </summary>
[STAThread]
47

static void Main()
{
Application.EnableVisualStyles();
Application.SetCompatibleTextRenderingDefault(false);
Application.Run(new Form1());
}
}
}
Form1.Designer.cs
namespace Thesis3
{
partial class Form1
{
/// <summary>
/// Required designer variable.
/// </summary>
private System.ComponentModel.IContainer components = null;
/// <summary>
/// Clean up any resources being used.
/// </summary>
/// <param name="disposing">true if managed resources should be
disposed; otherwise, false.</param>
protected override void Dispose(bool disposing)
{
if (disposing && (components != null))
{
components.Dispose();
}
base.Dispose(disposing);
}
#region Windows Form Designer generated code
/// <summary>
/// Required method for Designer support - do not modify
/// the contents of this method with the code editor.
/// </summary>
private void InitializeComponent()
{
this.buttonTrain = new System.Windows.Forms.Button();
this.buttonRecognize = new System.Windows.Forms.Button();
this.openFileDialog1 = new System.Windows.Forms.OpenFileDialog();
this.labelImageName = new System.Windows.Forms.Label();
this.dataGridViewInfo = new System.Windows.Forms.DataGridView();
((System.ComponentModel.ISupportInitialize)
(this.dataGridViewInfo)).BeginInit();
this.SuspendLayout();
//
// buttonTrain
//
this.buttonTrain.Location = new System.Drawing.Point(132, 22);
this.buttonTrain.Name = "buttonTrain";
48

this.buttonTrain.Size = new System.Drawing.Size(75, 23);
this.buttonTrain.TabIndex = 0;
this.buttonTrain.Text = "Train";
this.buttonTrain.UseVisualStyleBackColor = true;
this.buttonTrain.Click += new
System.EventHandler(this.button1_Click);
//
// buttonRecognize
//
this.buttonRecognize.Location = new System.Drawing.Point(34, 22);
this.buttonRecognize.Name = "buttonRecognize";
this.buttonRecognize.Size = new System.Drawing.Size(75, 23);
this.buttonRecognize.TabIndex = 1;
this.buttonRecognize.Text = "Recognize";
this.buttonRecognize.UseVisualStyleBackColor = true;
this.buttonRecognize.Click += new
System.EventHandler(this.buttonRecognize_Click);
//
// openFileDialog1
//
this.openFileDialog1.FileName = "openFileDialog1";
//
// labelImageName
//
this.labelImageName.AutoSize = true;
this.labelImageName.Location = new System.Drawing.Point(40, 68);
this.labelImageName.Name = "labelImageName";
this.labelImageName.Size = new System.Drawing.Size(0, 13);
this.labelImageName.TabIndex = 2;
//
// dataGridViewInfo
//
this.dataGridViewInfo.ColumnHeadersHeightSizeMode =
System.Windows.Forms.DataGridViewColumnHeadersHeightSizeMode.AutoSize;
this.dataGridViewInfo.Location = new System.Drawing.Point(225,
12);
this.dataGridViewInfo.Name = "dataGridViewInfo";
this.dataGridViewInfo.Size = new System.Drawing.Size(529, 542);
this.dataGridViewInfo.TabIndex = 3;
//
// Form1
//
this.AutoScaleDimensions = new System.Drawing.SizeF(6F, 13F);
this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;
this.ClientSize = new System.Drawing.Size(792, 566);
this.Controls.Add(this.dataGridViewInfo);
this.Controls.Add(this.labelImageName);
this.Controls.Add(this.buttonRecognize);
this.Controls.Add(this.buttonTrain);
this.Name = "Form1";
this.Text = "Form1";
this.Load += new System.EventHandler(this.Form1_Load);
((System.ComponentModel.ISupportInitialize)
(this.dataGridViewInfo)).EndInit();
this.ResumeLayout(false);
this.PerformLayout();
49

}
#endregion
private System.Windows.Forms.Button buttonTrain;
private System.Windows.Forms.Button buttonRecognize;
private System.Windows.Forms.OpenFileDialog openFileDialog1;
private System.Windows.Forms.Label labelImageName;
private System.Windows.Forms.DataGridView dataGridViewInfo;
}
}
Form1.cs
using System;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
namespace Thesis3
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
this.Text = "Knowledge Systems Institute Thesis Program Version 2
- Roby Chhokar";
}
private void button1_Click(object sender, EventArgs e)
{
this.buttonTrain.Enabled = false;
new Brain().train();
this.buttonTrain.Enabled = true;
}
private void buttonRecognize_Click(object sender, EventArgs e)
{
this.openFileDialog1.ShowDialog();
if (this.openFileDialog1.CheckFileExists)
{
this.labelImageName.Text =
this.openFileDialog1.FileName.Substring(this.openFileDialog1.FileName.LastInd
exOf("") + 1);
}
new Brain().recognize(this.labelImageName.Text, ref
this.dataGridViewInfo);
}
private void Form1_Load(object sender, EventArgs e)
50

{
}
}
}
Grid.cs
using System;
using System.Text;
using System.Data;
using System.Text;
namespace Thesis3
{
public class Grid
{
private bool[,] gridCells; //true if an image cell
private Rectangle visualWindow; //top-left corner of visual field is
with respect to the grid Cells.
private bool[,] visualCellsPrev; //true if an image cell
private bool[,] visualCellsCurr; //true if an image cell
private int SaccadeCount = 0;
private int saccadeAmount;
private const byte SACCADE_LEFT = 1;
private const byte SACCADE_RIGHT = 2;
private const byte SACCADE_UP = 3;
private const byte SACCADE_DOWN = 4;
public static int NumSaccades = 256;
private int currentOverlap;
public int getSaccadeCount()
{
return this.SaccadeCount;
}
public Grid(Bitmap image)
{
this.gridCells = new bool[image.Width * 3, image.Height * 3];
this.visualCellsPrev = new bool[image.Width, image.Height];
this.visualCellsCurr = new bool[image.Width, image.Height];
for (int i = image.Width; i < image.Width * 2; i++)
{
for (int j = image.Height; j < image.Height * 2; j++)
{
if (image.GetPixel(i - image.Width, j -
image.Height).Name != "ffffffff")
{
gridCells[i, j] = true;
51

visualCellsCurr[i - image.Width,j - image.Height] =
true;
}
}//for j
}//for i
this.visualWindow = new Rectangle(new Point(image.Width,
image.Height), new Size(image.Width, image.Height));
saccadeAmount = this.visualWindow.Width / 32;
}
public int getCurrentOverlap()
{
int overlap = 0;
for (int i = 0; i < visualWindow.Width; i++)
{
for (int j = 0; j < visualWindow.Height; j++)
{
if (visualCellsPrev[i, j] && visualCellsCurr[i, j])
{
overlap++;
}
}//for j
}//for i
return overlap;
}
public int getCurrentSpace()
{
int space = 0;
for (int i = visualWindow.Location.X; i < visualWindow.Location.X
+ visualWindow.Size.Width; i++)
{
for (int j = visualWindow.Location.Y; j <
visualWindow.Location.Y + visualWindow.Size.Height; j++)
{
if (gridCells[i, j])
{
space++;
}
}//for j
}//for i
return space;
}
private void saccade(byte saccadeDirection)
{
Point newLocation = new Point(this.visualWindow.Location.X,
this.visualWindow.Location.Y);
switch (saccadeDirection)
{
case SACCADE_DOWN: newLocation.Y = newLocation.Y +
saccadeAmount; break;
case SACCADE_UP: newLocation.Y = newLocation.Y -
52

case SACCADE_RIGHT: newLocation.X = newLocation.X +
case SACCADE_LEFT: newLocation.X = newLocation.X -
}
this.visualWindow.Location = newLocation;
copyCurrentToPrevVisualCells();
setCurrentVisualCells();
}
private void setCurrentVisualCells()
{
for (int i = visualWindow.Location.X; i < visualWindow.Location.X
+ visualWindow.Size.Width; i++)
{
for (int j = visualWindow.Location.Y; j <
visualWindow.Location.Y + visualWindow.Size.Height; j++)
{
if (gridCells[i, j])
{
visualCellsCurr[i - visualWindow.Location.X, j -
visualWindow.Location.Y] = true;
}
else
{
visualCellsCurr[i - visualWindow.Location.X, j -
visualWindow.Location.Y] = false;
}
}//for j
}//for i
}
private void copyCurrentToPrevVisualCells()
{
for (int i = 0; i < visualWindow.Width; i++)
{
for (int j = 0; j < visualWindow.Height; j++)
{
visualCellsPrev[i, j] = visualCellsCurr[i, j];
}//for j
}//for i
}
public void saccade()
{
SaccadeCount++;
/* 256 */
if (SaccadeCount >= 1 && SaccadeCount <= 16)
{
this.saccade(SACCADE_DOWN);
}
else if (SaccadeCount >= 17 && SaccadeCount <= 32)
53

{
this.saccade(SACCADE_RIGHT);
}
{
this.saccade(SACCADE_UP);
}
{
this.saccade(SACCADE_LEFT);
}
{
}
{
}
{
}
{
}
{
}
{
}
{
}
{
}
{
}
{
}
{
}
{
54

}
}
}
}
Brain.cs
using System;
using System.Text;
using System.Configuration;
using System.IO;
using System.Data;
using System.Text;
namespace Thesis3
{
class Brain
{
private String memoryFile =
ConfigurationSettings.AppSettings.Get("memory_file");
private int[] currentSpaceSequence = new int[Grid.NumSaccades];
private int[] currentSpaceOverlap = new int[Grid.NumSaccades];
public void recognize(String imageName_, ref DataGridView outputInfo)
{
setImageInfo(imageName_);
outputInfo.Columns.Clear();
outputInfo.Columns.Add("Image", "Image");
outputInfo.Columns.Add("Score", "Score");
outputInfo.Columns.Add("Rank", "Rank");
if (System.IO.File.Exists(memoryFile))
{
System.IO.StreamReader file = new
System.IO.StreamReader(memoryFile);
String currentLine = null;
int currentLineNum = 0;
double spaceError = 0;
double score = 0;
while (!file.EndOfStream)
{
currentLine = file.ReadLine();
outputInfo.Rows.Add();
outputInfo.Rows[currentLineNum].Cells[0].Value =
currentLine.Split('-')[0];
55

score =
getScore(convertSequenceToArray(currentLine.Split('-')[1]));
outputInfo.Rows[currentLineNum].Cells[1].Value = score;
currentLineNum++;
}
file.Close();
outputInfo.Sort(outputInfo.Columns[1],
System.ComponentModel.ListSortDirection.Descending);
for (int i = 0; i < outputInfo.Rows.Count; i++)
{
outputInfo.Rows[i].Cells[2].Value = i + 1;
}
}
}
private double getScore(int[] spaceSequence)
{
double score = 0.0;
double currRel = 0.0;
double rel = 0.0;
double curScore = 0.0;
for (int i = 1; i < spaceSequence.Length; i++)
{
currRel = 0.0;
rel = 0.0;
currRel = (double)currentSpaceSequence[i] /
(double)currentSpaceSequence[i - 1];
rel = (double)spaceSequence[i] / (double)spaceSequence[i - 1];
curScore = currRel / (currRel + getError(currRel, rel));
//if (curScore < 0.0) curScore = 0.0;
score = score + curScore;
}
return score;
}
private double getSpaceError(int [] spaceSequence)
{
double spaceError = 0.0;
double rel = 0.0;
double error = 0.0;
for (int i = 1; i < spaceSequence.Length; i++)
{
currRel = 0.0;
rel = 0.0;
currRel = (double)currentSpaceSequence[i] /
(double)currentSpaceSequence[i - 1];
rel = (double)spaceSequence[i] / (double)spaceSequence[i - 1];
56

spaceError = spaceError + getError(currRel, rel);
}
return spaceError;
}
private double getError(double trainRel, double recRel)
{
double error = 0.0;
//error = Math.Abs(trainRel - recRel) / trainRel;
error = Math.Abs(trainRel - recRel);
return error;
}
private double getOverlapError(int[] overlapSequence)
{
double overlapError = 0.0;
double rel = 0.0;
double error = 0.0;
for (int i = 1; i < overlapSequence.Length; i++)
{
currRel = 0.0;
rel = 0.0;
currRel = (double)currentSpaceOverlap[i] /
(double)currentSpaceOverlap[i - 1];
rel = (double)overlapSequence[i] / (double)overlapSequence[i
- 1];
overlapError = overlapError + getError(currRel, rel);
}
return overlapError;
}
private void setImageInfo(String imageName) {
Grid grid = new Grid(new Bitmap(imageName));
for (int i = 0; i < Grid.NumSaccades; i++)
{
currentSpaceSequence[i] = grid.getCurrentSpace();
grid.saccade();
currentSpaceOverlap[i] = grid.getCurrentOverlap();
}//for i
}
public void train()
{
Grid grid = null;
int[] spaceSequence = new int[Grid.NumSaccades];
int[] overlapSequence = new int[Grid.NumSaccades];
String[] trainImages =
Directory.GetFiles(ConfigurationSettings.AppSettings.Get("train_images"));
if (System.IO.File.Exists(memoryFile) && (new
System.IO.FileInfo(memoryFile).Length > 0))
{
System.IO.File.Copy(memoryFile, memoryFile + "_bak", true);
57

}
System.IO.StreamWriter file = new
System.IO.StreamWriter(memoryFile);
foreach (String trainImage in trainImages)
{
grid = new Grid(new Bitmap(trainImage));
for (int i = 0; i < Grid.NumSaccades; i++)
{
spaceSequence[i] = grid.getCurrentSpace();
grid.saccade();
overlapSequence[i] = grid.getCurrentOverlap();
}//for i
file.WriteLine(trainImage.Substring(ConfigurationSettings.AppSettings.Get("tr
ain_images").Length) + " - " + convertSequenceToString(spaceSequence) + " - "
+ convertSequenceToString(overlapSequence));
}//for images
file.Close();
}
private String convertSequenceToString(int [] sequence)
{
String seqStr = "";
foreach (int i in sequence)
{
seqStr = seqStr + "," + i;
}
return seqStr.Substring(1); //remove first comma
}
private int [] convertSequenceToArray(String sequence)
{
String [] seqArrTemp = sequence.Split(',');
int[] seqArr = new int[seqArrTemp.Length];
for (int i = 0; i < seqArrTemp.Length; i++ )
{
seqArr[i] = Int32.Parse(seqArrTemp[i]);
}
return seqArr; //remove first comma
}
}
}
58

References
[1] Fraser, Neil. (1998). The Biological Neuron. Retrieved November 30, 2005, from
http://www.virtualventures.ca/~neil/neural/neuron-a.html
[2] Izhikevich, M., Eugene. (2006). Dynamical Systems in Neuroscience: The Geometry of
Excitability and Bursting (December 2005 Version). Cambridge, MA: The MIT Press,
[3] Izhikevich, M., Eugene. (2004). Which Model to Use for Cortical Spiking Neurons?:
IEEE Transactions on Neural Networks, Vol. 15, NO. 5. New York, New York: IEEE
[4] Izhikevich, M., Eugene. (2003). Simple Model of Spiking Neurons. IEEE Transactions
on Neural Networks, Vol. 14, NO. 6. New York, New York: IEEE
[5] Visual C# 2005 Express Edition [Computer Software]. Redmond, WA: Microsoft
Corporation.
59

[6] Smeraldi, Fabrizio. (1999). Saccadic Eye Movements. Retrieved December 15, 2005,
from http://diwww.epfl.ch/lami/team/smeraldi/saccadic/yarbus.html
[7] Wikipedia, Optic Fovea. Retrieved November 11, 2005, from
http://en.wikipedia.org/wiki/Optic_fovea
[8] Ewens, J., Warren, & Grant, R., Gregory. (2001). Statistical Methods in Bioinformatics:
An Introduction. New York, New York: Springer-Verlag.
60

61

Thesis

More Related Content

What's hot

Viewers also liked

Similar to Thesis

Thesis