Project report on Eye tracking interpretation system

A Project Report on
EYE TRACKING INTERPRETATION SYSTEM
Submitted by
Name Seat No
GAVHALE NAVLESH B120283824
GAVHANE DEVENDRA B120283825
KURKUTE SIDDHESHWAR B120283843
A Project report submitted as a partial fulﬁllment towards Project for term-II of
Bachelor of Electronics Engineering,
2015-16
Under the guidance of
Mrs.P.S.Kasliwal
Department of Electronics Engineering
MIT Academy of Engineering, Alandi (D),
Pune 412 105
Savitribai Phule Pune University.
2015-2016

CERTIFICATE
This is to certify that
Name Seat No
NAVLESH GAVHALE B120283824
DEVENDRA GAVHANE B120283825
SIDDHESHWAR KURKUTE B120283843
of
MIT Academy of Engineering, Alandi (D), Pune have submitted Project report on
EYE TRACKING INTERPRETATION SYSTEM as a partial fulﬁllment of term II for award
of degree of Bachelor of Electronics Engineering, from Savitribai Phule Pune University, Pune,
during the academic year 2015-16.
Project Guide Head of Dept
Mrs.P.S.Kasliwal Dr.M.D.Goudar
External Examiner

Acknowledgement
We take this opportunity to thank certain people without whom this endeavor would not have been
possible. We would also express our thanks to the head of Department of Electronics engineering
Dr.M.D.Goudar. We would like to express our sincere gratitude to our guide Mrs.P.S.Kasliwal
for constant encouragement, help and guidance without which this project would not have been
completed.
We would like to express our sincere gratitude towards Mr.S.A.Khandekar, Mr.P.R.Ubare,
Mr.G.R.Vyawhare, Mr.P.P.Kumbhar for their constant support and valuable advice throughout
the progress of the project. Last but not the least, We express our heartiest acknowledgement to
our parents, friends and colleagues who directly or indirectly helped us in completing the project.

ABSTRACT
With a growing number of computer devices around,the increasing time we spend for interacting
with such devices, we are interested in finding new methods of interaction which ease the use
of computers or increase interaction efficiency. Eye tracking looks to be a promising technology
to achieve this goal.The aim of this project is to create a low cost Eye Tracking Interpretation
System.This system will penetrate to masses and using this system poor people will also get a
quality life. In our project the camera is mounted on a cheap sunglass frame at a distance of 5-10
cm.We will track and interpret eye movements with Human Computer Interaction(HCI) by using
Infrared Pupil-Corneal Reflection/pupil-centre method.The outcome of this project will shape fu-
ture projects where we try to integrate such a system to develop assistant systems for cars,language
reading, music reading, human activity recognition, the perception of advertising etc.

INDEX
1 INTRODUCTION 2
1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Necessity of project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 LITERATURE SURVEY 5
2.1 Eye Tracking Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.1 Videooculography (VOG) . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.2 Video-based infrared (IR) pupil-corneal reﬂection (PCR) . . . . . . . . . . 8
2.1.3 Electrooculography (EOG) . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3 SYSTEM DESCRIPTION 12
3.1 Related work component selection . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.1.1 Selection of Single Board Computer . . . . . . . . . . . . . . . . . . . . . 12
3.2 IR Sensitive Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2.1 The Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2.2 Three Kinds of Infrared Lights . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2.3 How It Is Processed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3 Raspberry Pi2 Model B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.3.1 Speciﬁcations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.3.2 Connectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.4 Costing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Department of Electronics Engineering, MIT Academy of Engineering, Pune i

4 SOFTWARE DESCRIPTION 25
4.1 The NOOBS installer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.2 Operating System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.3 Boot Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.4 Open CV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.5 Image Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.5.1 Image Thresholding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.5.2 Image Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.5.3 Tracking Algorithm’s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.6 Virtual Network Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5 METHODOLOGY 38
5.1 Block Diagram and Description . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.2 Flowchart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.3.1 USB Camera interface with the Raspberry Pi . . . . . . . . . . . . . . . . 42
5.3.2 Pupil detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.3.3 Output of pupil focus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.3.4 Execution on Raspberry pi . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.3.5 Project as a standalone system . . . . . . . . . . . . . . . . . . . . . . . . 44
6 RESULT 46
7 APPLICATIONS 48
8 CONCLUSION AND FUTURE SCOPE 50
9 REFERENCES 51
Department of Electronics Engineering, MIT Academy of Engineering, Pune ii

List of Figures
1.1 Eye Anatomy[5]
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1 VOG based tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 The relationship between the pupil center and the corneal reflection when the user
fixates on different locations on the screen . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Wearable EOG goggles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.1 Comparison of Single Board Computers(SBC’s) . . . . . . . . . . . . . . . . . . . 13
3.2 IR Imaging Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 Blocks on Raspberry Pi2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.4 Block Diagram of BCM2836 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.5 Block Diagram of ARM Cortex-A7 . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.6 Pin out of Raspberry Pi 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.7 RCA Video Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.1 Raspbian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.2 Boot process of Raspberry Pi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.3 Otsus Thresholding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.4 Averaging filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.5 Gaussian filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.6 Median filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.7 MeanShift . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Department of Electronics Engineering, MIT Academy of Engineering, Pune iii

4.8 VNC Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.1 Block Diagram[9]
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.2 Flowchart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.3 Camera image without IR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.4 Camera image with IR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.5 Project as a standalone system . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.1 Samples of different subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Department of Electronics Engineering, MIT Academy of Engineering, Pune iv

List of Tables
3.1 Component Cost Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
6.1 Comparison of MSE of various ﬁlters . . . . . . . . . . . . . . . . . . . . . . . . 46
9.1 Project Schedule Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Department of Electronics Engineering, MIT Academy of Engineering, Pune v

Chapter 1
INTRODUCTION
An eye has a lot of communicative power. Eye contact and gaze direction are central and very im-
portant factors in human communication.The eye has also been said to be a mirror to the soul or
window into the brain.Gaze behavior reflects cognitive processes that can give hints of our think-
ing and intentions.[4] We often look at things before acting on them. To see an object we have to
fixate our gaze at it long enough for the brains visual system to perceive it. Fixations are typically
between 200 and 600 ms. During any fixation, we only see a fairly narrow area visual scene with
high acuity. To perceive the scene accurately, we need to scan constantly it with rapid movement
of an eye, so-called saccades. Saccades are quick, ballistic jumps of 2 Degrees or longer that take
about 30 to 120 ms each.In addition to saccadic movements, the eyes can follow a moving target;
this is known as (smooth) pursuit movement.[1]
The size of the fovea, subtends at an angle of about one degree from the eye. The diameter of this
region corresponds to area of two degrees, which is about the size of a thumbnail when viewed
with the arm extended. Everything inside the fovea is perceived with high acuity but the acuity
decreases rapidly towards the periphery of the eye. The cause for this can be seen by examining
the retina(see Fig. 1.1). The lens focuses the reflected light coming from the pupil on the center
of the retina. The fovea contains cones, photoreceptive cells that are sensitive to color and provide
acuity. In reverse case, the peripheral area contains mostly rods, i.e. cells that are sensitive to light,
shade and motion.For example, a sudden movement in periphery can quickly attract the viewers
Department of Electronics Engineering, MIT Academy of Engineering, Pune 1

Figure 1.1: Eye Anatomy[5]
attention.We only see a small fraction of the visual scene in front of us with high acuity at any
point in time. The user need to move his/her eyes toward the target is the basic requirement for eye
tracking: it is possible to deduct the gaze vector by observing the line-of-sight.
Eye tracking is a technique where an individual users pupil(eye) movements are measured so that
we come to know that both where a user is looking at any given instant and the sequence in
which his/her eyes get shifted from one location to another.Eye tracking is the process of tracking
pupil(eye) movements or the absolute point of gaze (POG)referring to the point where users gaze
is focused at in the visuals. Eye tracking is being used in a no. of application areas, from psycho-
logical research and medical diagnostic to usability studies and interactive gaze-controlled applica-
tions.We are focused on the use of real-time data from human pupil(eye) movements.Just as speech
and other technologies requires accurate interpretation of user speech and related parameters,eye-
movement data analysis requires accurate interpretation of user eye movements i.e.mapping ob-
served, to the user intentions that produced them.[12]

Eye movement recordings can provide an objective source of interface-evaluation data that can in-
form the design of improved interfaces. Eye movements can also be captured and used as control
signals to enable people to interact with interfaces directly without the need for mouse or keyboard
input, which can be a major advantage for certain populations of users such as disabled individuals.
1.1 Problem Statement
As Paralyzed/physically disabled people could not convey their messages to others, hence build a
low cost system to communicate themselves to the outside world.
1.2 Necessity of project
The main objective of this project is to Provide a useful system to paralyzed/physically disabled
peoples which will be easy to conﬁgure and handle.

Chapter 2
LITERATURE SURVEY
During the course of this project, there were many surveys carried out regarding the various com-
ponents,techniques of eye tracking they might be implemented in our project along with the various
systems already existing. Many websites were visited to get ideas about the current working mod-
ules for controlling of the parameters. Books on micro controller as well as various eye behaviours
were read to understand exactly how much it will be able to sustain our requirements and if any
other options could be used.
Initially, eye movements were mainly studied by the physicological inspection and observation.
Basic eye movements were categorized and their duration estimated long before the eye tracking
technology enabled precise measurement of eye movements.The first generation of eye tracking
devices was highly uncomfortable. A breakthrough in eye tracking technology was the develop-
ment of the first contact-free eye tracking apparatus based on photography and light reflected from
the cornea. It can be considered as the first invention in video-based eye tracking, corneal reflec-
tion eye tracking systems. The development of unobtrusive camera-based systems and the increase
of computing power enabled gathering of eye tracking data in real time, enabling the use of pupil
movement as a control method for people with disabilities.[11]
As far as camera is concerned it is better to use the camera with high resolution not less than 640 x
480 and camera must be IR sensitive. As far as sensors are concerned, there are many IR sensors
that are particularly used in such systems to accurately track the eye movements. Some of them

are Sharp GP2Y0A02YK0F,Phidgets 1103,Dagu Compound,Fairchild QRB1134 etc.But all of of
them are too much bigger in size to mount around the camera so here we will be using IR LED’S
to do the job.
2.1 Eye Tracking Techniques
While a large number of different techniques are being deployed to track eye movements in the
past, three of them have emerged as the predominant ones and are commonly used in research and
commercial applications in today’s world.These techniques are
(1) Videooculography (VOG)-video based tracking using remote visible light video cameras.
(2) Video-based infrared pupil-corneal reflection (PCR).
(3) Electrooculography (EOG).
While the first two video-based techniques have so many common properties, all techniques have
wide range of application areas where they are mostly used.Video-based eye tracking relies on
video cameras and therefore can be used for developing interfaces that do not require accurate
POG tracking (e.g. about 4Degrees). In contrast, video based PCR provides highly accurate point
of gaze measurements of up to 0.5Degrees of visual angle and has therefore it is a preferred tech-
nique in scientific domains, such as reading or pupil(eye) movement based interaction, and com-
mercial applications, such as in car safety research. Finally, EOG has been used for decades for
ophthalmological studies as it can be used for measuring movements of the eyes with high ac-
curacy. In addition to different application areas, each of these measurement techniques also has
specific technical advantages and disadvantages.
2.1.1 Videooculography (VOG)
A video-based eye tracking that can be either used in a remote or head-mounted configuration. A
typical setup consists of a video camera that records the movements of the eye(s) and a computer
that saves and analyses the gaze data. In remote systems, the camera is typically based below
the computer screen (Fig. 2.1) while in head-mounted systems, the camera is attached either on

a frame of eyeglasses or in a separate helmet. Head-mounted systems often also include a scene
camera for recording the users point of view, which can then be used to map the users gaze to the
current visual scene.
The frame rate and resolution of the video camera have a signiﬁcant effect on the accuracy of
tracking; a low-cost web camera cannot compete with a high-end camera with high-resolution and
high sample rate. The focal length of the lens, the angle, as well as the distance between the eye
and the camera have an effect on the working distance and the quality of gaze tracking. With
large zooming (large focal length), it is possible to get a close-up view of the eye but it narrows
the working angle of the camera and requires the user to sit fairly still (unless the camera follows
the users movements). In head-mounted systems, the camera is placed near the eye, which means
a bigger image of the eye and thus more pixels for tracking the eye. If a wide angle camera is
used, it allows more freedom of movement of the user but also requires a high-resolution camera
to maintain enough accuracy for tracking the pupil.
Since tracking of an eye is based on video images, it requires clear view of the eye. There are so
many issues that may affect the quality of eye tracking,such as rapidly changing light conditions,
reﬂections from eyeglasses, continuously dropping of eyelids,squinting the eyes while smiling,
or even heavy makeup.The video images are the basis for estimating the gaze position on the
computer screen: the location of the eye(s) and the center of the pupil are detected. Changes
in the position is tracked, analyzed and mapped to gaze coordinates. For a detailed survey of
video-based eye tracking techniques,detection of pupil and estimation of gaze position. If only one
reference point is used and no other reference points are available, the user has to stay still for an
accurate calculation of the gaze vector (the line of sight from the users eye to the point of view on
the screen). Forcing the user to sit still may be uncomfortable,thus various methods for tracking
and compensating the head movement have been implemented. Head tracking methods are also
required for head-mounted systems, if one wishes to calculate the point of gaze in relation to the
users eye and the environment.

Figure 2.1: VOG based tracking
2.1.2 Video-based infrared (IR) pupil-corneal reflection (PCR)
Systems only based on visible light and pupil center tracking tend to be inaccurate and sensitive to
head movement. To address this problem, a reference point, a so called corneal reflection or glint,
can be added. Such a reference point can be added by using an artificial infrared (IR) light source
aimed on- or off-axis at the eye. An on-axis light source will result in a bright pupil effect, making
it easier for the analysis software to recognize the pupil in the image. The effect is similar to the
red-eye effect caused by flash in a photograph. The off-axis light results in dark pupil images. Both
will help in keeping the eye area well lit but they do not disturb viewing or affect pupil dilation
since IR light is invisible to the human eye.
By measuring the corneal reflection(s) from the IR source relative to the center of the pupil, the
system can compensate for inaccuracies and also allow for a limited degree of head movement.
Gaze direction is then calculated by measuring the changing relationship between the moving pupil
center of the eye and the corneal reflection (see Fig. 2.2). As the position of the corneal reflection

remains roughly constant during eye movement, the reflection will remain static during rotation
of the eye and changes in gaze direction, thus giving a basic eye and head position reference. In
addition, it also provides a simple reference point to compare with the moving pupil, and thus
enables calculation of the gaze vector.
Figure 2.2: The relationship between the pupil center and the corneal reflection when the user
fixates on different locations on the screen
While IR illumination enables fairly accurate remote tracking of the user it does not work well
in changing ambient light, such as in outdoors settings. There is an ongoing research that tries
to solve this issue. In addition, according to our personal experience, there seems to be a small
number of people for whom, robust/accurate eye tracking does not seem to work even in laboratory
settings. Electrooculography is not dependent or disturbed by lighting conditions and thus can
replace VOG-based tracking in some of these situations and for some applications.

2.1.3 Electrooculography (EOG)
The human eye can be modeled as a dipole with its positive pole at the cornea and its negative
pole at the retina. Assuming a stable cornea-retinal potential difference,the eye is the origin of a
steady electric potential field. The electrical signal that can be measured from this field is called the
electrooculogram (EOG). The signal is measured between two pairs of surface electrodes placed
in periorbital positions around the eye (see Fig. 2.3) with respect to a reference electrode (typically
placed on the forehead). If the eyes move from the center position towards one of these electrodes,
the retina approaches this electrode, while the cornea approaches the opposing one. This change
in dipole orientation causes a change in the electric potential field, which in turn can be measured
to track eye movements. In contrast to video-based eye tracking, recorded eye movements are
typically split into one horizontal and one vertical EOG signal component. This split reflects the
discretisation given by the electrode setup.
One drawback of EOG compared to video-based tracking is the fact that EOG requires electrodes
to be attached to the skin around the eyes. In addition, EOG provides lower spatial POG tracking
accuracy and is therefore better suited for tracking relative eye movements. EOG signals are
subject to signal noise and artifacts and prone to drifting, particularly if recorded in mobile settings.
EOG signalslike other physiological signalsmay be corrupted with noise from the residential power
line, the measurement circuitry, electrodes, or other interfering physiological sources.
One advantage of EOG compared to video-based eye tracking is that changing lighting conditions
have only little impact on EOG signals, a property that is particularly useful for mobile recordings
in daily life settings. As light falling into the eyes is not required for the electric potential field to
be established; EOG can also be measured in total darkness or when the eyes are closed. It is for
this reason that EOG is a well known measurement technique for recording eye movements during
sleep e.g. to identify REM phases or diagnosing sleep disorders . The second major advantage of
EOG is that the signal processing is computationally light-weight and particularly does not require
any complex video and image processing.Consequently, while EOG has traditionally been used in
stationary settings, it can also be implemented as a low power and fully embedded on body system
for mobile recordings in daily life settings. While state-of-the art video-based eye trackers require

Figure 2.3: Wearable EOG goggles
additional equipment for data recording and storage, such as laptops, and are limited to recording
times of a few hours, EOG allows for long term recordings, allowing capture of peoples everyday
life.[11]

Chapter 3
SYSTEM DESCRIPTION
3.1 Related work component selection
1. Camera = IR sensitive and minimum resolution of 640 x 480
2. Sunglasses =Cheap and strong enough to sustain the weight of camera
3. Input to the system = Reﬂections from Human eye.
• a.IR Sensor 4-IR LEDS.
4. Power Requirement
For Single Board Computer=5V.
3.1.1 Selection of Single Board Computer
Selecting the proper microcontroller unit is one of the critical decisions which is more likely to be
responsible for the success or failure of the project. There are numerous criteria’s which should
be considered while choosing a microcontroller.While selecting the microcontroller the ﬁrst ques-
tion which should be asked is ”What does the microcontroller need to do in your project?” Others
factors those should be considered include features, pricing, availability, development tools, man-
ufacturer support, stability, and solesourcing. While Selecting microcontroller for our project we

compared various Single Board Computers (SBC’s) those are Shown in below Figure:
Figure 3.1: Comparison of Single Board Computers(SBC’s)
3.2 IR Sensitive Camera
3.2.1 The Basics
Infrared imaging cameras allow you to do more than just see in the dark they work regardless of
the lighting conditions, giving you a sharp set of eyes even in absolutely zero light.An infrared
imaging camera basically detects light. The quantity of energy found in a light wave is correlated
to its wavelength. The longer the wavelength is, the lower energy it has. This kind of energy can
only be detected by an infrared thermal camera. Aside from light waves, there are visible lights
and these are better known to us as colors.Among all the colors that we have, violet has the most

energy and red has the least energy. Because of this, the terms Ultraviolet and Infrared came to be.
All lights that are below red or are darker than red are considered infrared which is not visible to
the human eye but is visible through imaging cameras.[6]
Infrared rays can also travel through fog, dust and smoke no matter how thick. It can even travel
through some materials. Although infrared imaging cameras are often referred to as night vision
cameras, dont confuse them with day and night cameras. A day and night camera has a very
sensitive imaging chip that lets the camera capture a viewable picture even in less light conditions.
It does not, however, use infrared technology. A day and night camera is a good option for areas
that have a constant source of light, such as a street light or security light, but they will not work
if that light is switched off (either accidentally or knowingly). When light is available, infrared
cameras will give you a color picture. As it gets darker, the camera automatically switches to
infrared mode. In this mode, the camera records in black and white.
Figure 3.2: IR Imaging Camera

3.2.2 Three Kinds of Infrared Lights
The infrared lights can be categorized into 3 groups. They are the near infrared, mid infrared and
thermal infrared. Near infrared ray is the nearest to visible light and has the wavelength between
0.7 to 1.3 mns. Mid infrared ray has a wavelength between 1.3 to 3.1 mns. Both the near infrared
rays and the mid infrared are being utilized in common electronic devices in our current world.
Some of them are remote controls, cellular phones and electronic switches. Thermal infrared is
the one that has the largest part of the Spectrum of Light, while near infrared is between 0.7 to 1.3
mns and mid infrared ray is between 1.3 to 3.1 mns (the thermal infrared is between 3 to 30 mns).
3.2.3 How It Is Processed
An infrared imaging camera uses a special lens that is focused mainly of infrared light emitted
by all the objects that it can see. The infrared light emitted is processed by an array of infrared
detectors, which then creates a temperature pattern that is also known as thermogram. The entire
process from obtaining the infrared light to making a thermogram takes approximately 0.033 ms.
After the thermogram is created, it is then made into an electrical impulse. The electric impulse
that was made from the thermogram is then sent to a single processing unit or a circuit board that
is mainly focused on sending the signal into a data for display. Once the data has been sent to the
display, people are be able to see various colors depending on the intensity of the emission of the
infrared light. Through the different combinations that came from the impulses made by different
objects, an infrared thermal image is then created.Sometimes, infrared cameras are best utilized
when people want to see the things that cannot be seen by the naked eye. They are also being
utilized by the military because the infrared imaging camera allows them to see in the dark.
To check how good is your IR imaging camera, One way is measuring the camera’s lux. Lux refers
to the amount of light required to give a good picture. Obviously, the lower the lux, the lower the
light the camera needs. A true IR camera will have a 0.0 lux in infrared mode, which means they
can see in complete, utter, total darkness with no light at all. You can also compare IR cameras
according to how far they can see in complete darkness.Some long range cameras can see up to

150 feet...in total darkness! Depending on your requirement, you can select short range or long
range cameras that will keep you covered.[4]
3.3 Raspberry Pi2 Model B
Figure 3.3: Blocks on Raspberry Pi2
The Raspberry Pi 2 delivers 6 times the processing capacity of previous models. This second
generation Raspberry Pi has an upgraded Broadcom BCM2836 processor, which is a powerful
ARM Cortex-A7 based quad-core processor that runs at 900MHz. The board also features an
increase in memory capacity to 1Gbyte.
3.3.1 Speciﬁcations
Chip - Broadcom BCM2836 SoC
Core architecture - Quad-core ARM Cortex-A7
CPU - 900 MHz
GPU - Dual Core VideoCore IV Multimedia Co-Processor, Provides Open GL ES 2.0, hardware-
accelerated OpenVG, and 1080p30 H.264 high-proﬁle decode

Capable of 1Gpixel/s, 1.5Gtexel/s or 24GFLOPs with texture filtering and DMA infrastructure
Memory - 1GB LPDDR2
Operating System - Boots from Micro SD card, running a version of the Linux operating system
Dimensions - 85 x 56 x 17mm
Power - Micro USB socket 5V, 2A
3.3.2 Connectors
Ethernet - 10/100 BaseT Ethernet socket
Video Output - HDMI (rev 1.3 and 1.4)
Audio Output - 3.5mm jack, HDMI
USB - 4 x USB 2.0 Connector
GPIO Connector - 40-pin 2.54 mm (100 mil) expansion header: 2x20 strip, Providing 27 GPIO
pins as well as +3.3 V, +5 V and GND supply lines
Camera Connector - 15-pin MIPI Camera Serial Interface (CSI-2)
JTAG - Not populated
Display Connector -Display Serial Interface (DSI) 15 way flat flex cable connector with two data
lanes and a clock lane
Memory Card Slot - Micro SDIO
1.Processor
Broadcom BCM2836:
This second generation Raspberry Pi has an upgraded Broadcom BCM2836 processor, which is a
powerful ARM Cortex-A7 based quad-core processor that runs at 900MHz.The processor is built
with four ARM cores, compared to the earlier generation BCM2835, which has a single ARM core
that was used in the original Raspberry Pi. It has twice the amount of memory (RAM) on board
and has the same pocket-sized form factor.

Figure 3.4: Block Diagram of BCM2836
ARM Cortex-A7:
The Cortex-A7 MPCore processor is a high performance, low power processor that implements
the ARMv7A architecture. The Cortex-A7 MPCore processor has one to four processors in a sin-
gle multiprocessor device. It provides up to 20percent more single thread performance than the
Cortex-A5.The Cortex-A7 processor builds on the energy-efﬁcient 8-stage pipeline.Performance
and power optimized L1 caches combine minimal access latency techniques to maximize perfor-
mance and minimize power consumption.It also beneﬁts from an integrated L2 cache designed for
low-power, with lower transaction latencies and improved OS support for cache maintenance.NEON
technology in it can accelerate multimedia and signal processing algorithms such as video en-
code/decode, 2D/3D graphics, gaming, audio and speech processing, image processing, telephony,
and sound synthesis.Figure below shows the block diagram of ARM Cortex-A7:

Figure 3.5: Block Diagram of ARM Cortex-A7
2.Power source
The device is powered by a 5V micro USB supply. Exactly how much current (mA) the Rasp-
berry Pi requires is dependent on what you connect to it.Typically, the model B uses between
700-1000mA depending on what peripherals are connected.The maximum power the Raspberry
Pi can use is 1 Amp. If you need to connect a USB device that will take the power requirements
above 1 Amp, then you must connect it to an externally-powered USB hub.
The power requirements of the Raspberry Pi increase as you make use of the various interfaces
on the Raspberry Pi. The GPIO pins can draw 50mA safely, distributed across all the pins; an
individual GPIO pin can only safely draw 16mA. The HDMI port uses 50mA, the camera module
requires 250mA, and keyboards and mouse can take as little as 100mA or over 1000mA.Just hav-
ing the Pi 2 Model B running idle (no HDMI, graphics, ethernet or wiﬁ just console cable) the Pi 2
draws 200mA and with WiFi running, that adds another 170mA and if you have Ethernet instead,
that adds about 40mA.
BACKPOWERING:
Backpowering occurs when USB hubs do not provide a diode to stop the hub from powering against
the host computer. Other hubs will provide as much power as you want out each port.some hubs
may backfeed the Raspberry Pi. This means that the hubs will power the Raspberry Pi through its

USB cable input cable, without the need for a separate micro-USB power cable, and bypass the
voltage protection. If you are using a hub that backfeeds to the Raspberry Pi and the hub experi-
ences a power surge, your Raspberry Pi could potentially be damaged.
3.SD Card
The Raspberry Pi does not have any onboard storage available. The operating system is loaded on
a SD card which is inserted on the SD card slot on the Raspberry Pi. The operating system can be
loaded on the card using a card reader on any computer.In Raspberry Pi the data is written and read
a lot more frequently, and from differing locations on the card.Please note that it is not necessary
that a SD card with high write speed rating is always required.
4.GPIO
GPIO General Purpose Input Output
General-purpose input/output (GPIO) is a generic pin on an integrated circuit whose behaviour,
including whether it is an input or output pin, can be controlled by the user at run time.GPIO pe-
ripherals vary widely. In some cases, they are simple group of pins that can switch as a group to
either input or output. In others, each pin can be set up to accept or source different logic volt-
ages, with configurable drive strengths and pull ups/downs. Input and output voltages are typically
though not always limited to the supply voltage of the device with the GPIOs, and may be damaged
by greater voltages. GPIO pins have no special purpose defined, and go unused by default. The
idea is that sometimes the system designer building a full system that uses the chip might find it
useful to have a handful of additional digital control lines, and having these available from the chip
can save the hassle of having to arrange additional circuitry to provide them. GPIO capabilities
may include:
1.GPIO pins can be configured to be input or output.
2.GPIO pins can be enabled/disabled.
3.Input values are readable (typically high=1, low=0).

4.Output values are writable/readable.
5.Input values can often be used as IRQs (typically for wake up events).
The Raspberry Pi2 Model B board has a 40-pin 2.54 mm expansion header arranged in a 2x20
strip. They provide 27 GPIO pins as well as +3.3 V, +5 V and GND supply lines.
Figure 3.6: Pin out of Raspberry Pi 2

5.DSI Connector
The Display Serial Interface (DSI) is a speciﬁcation by the Mobile Industry Processor Interface
(MIPI) Alliance aimed at reducing the cost of display controllers in a mobile device. It is com-
monly targeted at LCD and similar display technologies. It deﬁnes a serial bus and a communica-
tion protocol between the host (source of the image data) and the device (destination of the image
data).A DSI compatible LCD screen can be connected through the DSI connector.
6.RCA Video
RCA Video outputs (PAL and NTSC) are available on all models of Raspberry Pi. Any television
or screen with a RCA jack can be connected with the RPi.
Figure 3.7: RCA Video Connector
7.USB 2.0 Port
The Raspberry Pi 2 Model B is equipped with four USB2.0 ports. These are connected to the
LAN9512 combo hub/Ethernet chip IC3, which is itself a USB device connected to the single
upstream USB port on BCM2836.The USB ports enable the attachment of peripherals such as
keyboards, mouse, webcams that provide the Pi with additional functionality. The USB host port
inside the Pi is an On-The-Go (OTG) host as the application processor powering the Pi, BCM2836,

was originally intended to be used in the mobile market: i.e. as the single USB port on a phone
for connection to a PC, or to a single device. In essence, the OTG hardware is simpler than the
equivalent hardware on a PC.OTG in general supports communication to all types of USB device,
but to provide an adequate level of functionality for most of the USB devices that one might plug
into a Pi, the system software has to do more work. The USB specification defines three device
speeds - Low, Full and High. Most mouse and keyboards are Low-speed, most USB sound devices
are Full-speed and most video devices (webcams or video capture) are High-speed.
General Limitations:
The OTG hardware on Raspberry Pi has a simpler level of support for certain devices, which may
present a higher software processing overhead. The Raspberry Pi also has only one root USB port:
all traffic from all connected devices is funneled down this bus, which operates at a maximum
speed of 480mbps.The software overhead incurred when talking to Low- and Full-speed devices
means that there are soft limitations on the number of simultaneously active Low- and Full-speed
devices. Small numbers of these types of devices connected to a Pi will cause no issues.
8.Ethernet
Ethernet port is available on Model B. It can be connected to a network or internet using a standard
LAN cable on the Ethernet port. The Ethernet ports are controlled by Microchip LAN9514 LAN
controller chip.Experience shows that under ideal circumstances close to 99Mbps of useful data
can be transferred on a 100Mbps Ethernet segment.We can get higher than 100Mbps of throughput
as long as we add a gigabit USB3.0 Ethernet adapter.
9.HDMI
HDMI High Definition Multimedia Interface
HDMI1.3and1.4 - a type A port is provided on the RPi to connect with HDMI screens.The HDMI
port on the Raspberry Pi supports the Consumer Electronics Control(CEC) Standard.

3.4 Costing
Sr.No. Component Cost
1) Raspberry Pi 2Model B 2500
2) iBall ROBO K20 Camera 1020
3) Sunglass Frame 200
4) 4-IR LED’s(Transmitter) 100
Table 3.1: Component Cost Table

Chapter 4
SOFTWARE DESCRIPTION
4.1 The NOOBS installer
The Raspberry Pi package only comes with the main board and nothing else. It does not come
shipped with an operating system. Operating systems are loaded on a SD card from a computer
and then the SD card is inserted in the Pi which becomes the primary boot device.
Installing operating system can be easy for some enthusiasts, but for some beginners working with
image files of operating systems can be difficult. So the Raspberry Pi foundation made a software
called NOOBS New Out Of Box Software which eases the process of installing an operating sys-
tem on the Pi.
The NOOBS installer can be downloaded from the official website. A user only needs to connect a
SD card with the computer and just run the setup file to install NOOBS on the SD card. Next, insert
the card on the Raspberry Pi. On booting the first time, the NOOBS interface is loaded and the user
can select from a list of operating systems to install. It is much convenient to install the operating
system this way. Also once the operating system is installed on the card with the NOOBS installer,
every time the Pi boots, a recovery mode provided by the NOOBS can be accessed by holding the
shift key during boot. It also allows editing of the config.txt file for the operating system.

4.2 Operating System
The Raspberry Pi primarily uses Linux kernel-based operating systems. The ARM11 is based on
version 6 of the ARM which is no longer supported by several popular versions of Linux, including
Ubuntu. The install manager for Raspberry Pi is NOOBS. The OSs included with NOOBS are:
1.Archlinux ARM
2.OpenELEC
3.Pidora (Fedora Remix)
4.Raspbmc and the XBMC open source digital media center
5.RISC OS The operating system of the first ARM-based computer
6.Raspbian Maintained independently of the Foundation; based on ARM hard-float (armhf)-
Debian 7 ’Wheezy’ architecture port, that was designed for a newer ARMv7 processor whose
binaries would not work on the Rapberry Pi, but Raspbian is compiled for the ARMv6 instruction
set of the Raspberry Pi making it work but with slower performance. It provides some available deb
software packages, pre-compiled software bundles. A minimum size of 2 GB SD card is required,
but a 4 GB SD card or above is recommended. There is a Pi Store for exchanging programs. The
’Raspbian Server Edition (RSEv2.4)’, is a stripped version with other software packages bundled
as compared to the usual desktop computer oriented Raspbian.
About Raspbian
Figure 4.1: Raspbian
Raspbian is an unofficial port of Debian Wheezy armhf with compilation settings adjusted to pro-
duce optimized ”hard float” code that will run on the Raspberry Pi. This provides significantly
faster performance for applications that make heavy use of floating point arithmetic operations.

All other applications will also gain some performance through the use of advanced instructions of
the ARMv6 CPU in Raspberry Pi. Although, Raspbian is primarily the efforts of Mike Thompson
and Peter Green, it has also benefited greatly from the enthusiastic support of Raspberry Pi com-
munity members who wish to get the maximum performance from their device.
Raspbian is a free operating system based on Debian optimized for the Raspberry Pi hardware. An
operating system is the set of basic programs and utilities that make your Raspberry Pi run. How-
ever, Raspbian provides more than a pure OS: it comes with over 35,000 packages, pre-compiled
software bundled in a nice format for easy installation on your Raspberry Pi. The initial build of
over 35,000 Raspbian packages, optimized for best performance on the Raspberry Pi, was com-
pleted in June of 2012. However, Raspbian is still under active development with an emphasis on
improving the stability and performance of as many Debian packages as possible. Note: Raspbian
is not affiliated with the Raspberry Pi Foundation. Raspbian was created by a small, dedicated team
of developers that are fans of the Raspberry Pi hardware, the educational goals of the Raspberry Pi
Foundation and, of course, the Debian Project.
What is Debian?
Debian is a free operating system for your computer and includes the basic set of programs and
utilities that make your computer run along with many thousands of other packages. Debian has
a reputation within the Linux community for being very high-quality, stable and scalable. Debian
also has an extensive and friendly user community that can help new users with support for practi-
cally any problem. This makes Debian an ideal operating system for the Raspberry Pi that will be
used by children and many others using Linux for the first time.
What is Raspbian?
Raspbian is an unofficial port of Debian wheezy armhf with compilation settings adjusted to pro-
duce code that uses ”hardware floating point”, the ”hard float” ABI and will run on the Raspberry
Pi. The port is necessary because the official Debian wheezy armhf release is compatible only with
versions of the ARM architecture later than the one used on the Raspberry Pi (ARMv7-A CPUs
and higher, vs the Raspberry Pi’s ARMv6 CPU). The Debian squeeze image issued by the Rasp-
berry Pi foundation was based on debian armel which uses software floating point and the ”soft

float” ABI. The foundation used the existing Debian port for less capable ARM devices. There-
fore, it did not use of the Pi’s processor’s floating point hardware - reducing the Pi’s performance
during floating point intensive applications - or the advanced instructions of the ARMv6 CPU.
4.3 Boot Process
The Raspberry Pi does not boot as a traditional computer. The Video Core i.e. the Graphics pro-
cessor actually boots before the ARM CPU.
The boot process of the Raspberry Pi can be explained as follows:
1.When the power is turned on, the first bits of code to run is stored in a ROM chip in the SoC and
is built into the Pi during manufacture. This is the called the first-stage bootloader.
2.The SoC is hardwired to run this code on startup on a small RISC Core (Reduced Instruction Set
Computer). It is used to mount the FAT32 boot partition in the SDCard so that the second-stage
bootloader can be accessed. So what is this second-stage bootloader stored in the SD Card? Its
bootcode.bin. This file can be seen while mount process of an operating system on the SD Card in
windows.
3.Now heres something tricky. The first-stage bootloader has not yet initialized the ARM CPU
(meaning CPU is in reset) or the RAM. So, the second-stage bootloader also has to run on the
GPU. The bootloader.bin file is loaded into the 128K 4 way set associative L2 cache of the GPU
and then executed. This enables the RAM and loads start.elf which is also in the SD Card. This is
the third-stage bootloader and is also the most important. It is the firmware for the GPU, meaning
it contains the settings or in our case, has instructions to load the settings from config.txt which is
also in the SD Card. We can think of the config.txt as the BIOS settings.
4.The start.elf also splits the RAM between the GPU and the ARM CPU. The ARM only has
access the to the address space left over by the GPU address space. For example, if the GPU
was allocated addresses from 0x000F000 0x0000FFFF, the ARM has access to addresses from
0x00000000 0x0000EFFF.
5.The physical addresses perceived by the ARM core is actually mapped to another address in the

VideoCore (0xC0000000 and beyond) by the MMU (Memory Management Unit) of the Video-
Core.
6.The config.txt is loaded after the split is done so the splitting amounts cannot be specified in the
config.txt. However, different .elf files having different splits exist in the SD Card. So, depending
on the requirement, the file can be renamed to start.elf and boot the Pi. In the Pi, the GPU is King!
7.Other than loading config.txt and splitting RAM, the start.elf also loads cmdline.txt if it exists.
It contains the command line parameters for whatever kernel that is to be loaded. This brings us
to the final stage of the boot process. The start.elf finally loads kernel.img which is the binary
file containing the OS kernel and releases the reset on the CPU. The ARM CPU then executes
whatever instructions in the kernel.img thereby loading the operating system.
8.After starting the operating system, the GPU code is not unloaded. In fact, start.elf is not just
firmware for the GPU, It is a proprietary operating system called VideoCore OS (VCOS). When
the normal OS (Linux) requires an element not directly accessible to it, Linux communicates with
VCOS using the mailbox messaging system.
Figure 4.2: Boot process of Raspberry Pi

4.4 Open CV
About OpenCV
Open Source Computer Vision Library is an open source computer vision and machine learning
software library. OpenCV was built to provide a common infrastructure for computer vision ap-
plications and to accelerate the use of machine perception in the commercial products. Being a
BSD-licensed product, OpenCV makes it easy for to modify the code.The library has more than
2500 optimized algorithms, which includes a comprehensive set of both classic and state-of-the-
art computer vision and machine learning algorithms. These algorithms can be used to detect and
recognize faces, identify objects, classify human actions in videos, track camera movements, stitch
images together to produce a high resolution image of an entire scene, find similar images from an
image database, remove red eyes from images taken using flash,etc.
Along with well-established companies like Google, Yahoo, Microsoft, Intel, IBM, Sony, Honda,
Toyota that employ the library. OpenCVs deployed uses span the range from stitching street view
images together, detecting intrusions in surveillance video in Israel, monitoring mine equipment
in China, helping robots navigate and pick up objects at Willow Garage, detection of swimming
pool drowning accidents in Europe, running interactive art in Spain and New York, checking run-
ways for debris in Turkey, inspecting labels on products in factories around the world on to rapid
face detection in Japan. It has C++, C, Python, Java and MATLAB interfaces and supports Win-
dows, Linux, Android and Mac OS. OpenCV leans mostly towards real-time vision applications
and takes advantage of MMX and SSE instructions when available.OpenCV is written natively in
C++. OpenCV has a modular structure, which means that the package includes several shared or
static libraries. The following modules are available:
1.core - a compact module defining basic data structures, including the dense multi-dimensional
array Mat and basic functions used by all other modules.
2.imgproc - an image processing module that includes linear and non-linear image filtering, geo-
metrical image transformations (resize, affine and perspective warping, generic table-based remap-
ping), color space conversion, histograms, and so on.
3.video - a video analysis module that includes motion estimation, background subtraction, and

object tracking algorithms.
4.calib3d - basic multiple view geometry algorithms, single and stereo camera calibration, object
pose estimation, stereo correspondence algorithms, and elements of 3D reconstruction.
5.features2d - salient feature detectors, descriptors, and descriptor matchers.
6.objdetect - detection of objects and instances of the predefined classes.
7.highgui - an easy to use interface to video capturing, image and video codecs, as well as simple
UI capabilities.
8.gpu - GPU accelerated algorithms from different OpenCV modules.
9.some other helper modules, such as FLANN and Google test wrappers, Python bindings, and
others.
4.5 Image Processing
Images may suffer from the following degradations:
1)Poor contrast due to poor illumination or finite sensitivity of the imaging device
2)Electronic sensor noise or atmospheric disturbances leading to broad band noise
3)Aliasing effects due to inadequate sampling
4)Finite aperture effects or motion leading to spatial
In order to avoid all above one may think of options like Image filtering,Thresholding etc.
4.5.1 Image Thresholding
There are various types of Thresholding Some of them are:
1)Simple Thresholding
2)Adaptive Thresholding
3)Otsus Thresholding
1. Simple Thresholding
In this type if pixel value is greater than a threshold value, it is assigned one value (may be white),

else it is assigned another value (may be black). First argument will be the source image, which
should be a grayscale image. Second argument will be the threshold value which is used to classify
the pixel values. Third argument will be the max. value which represents the value to be given
if pixel value is more than (sometimes less than) the threshold value.It is also called as Global
Thresholding As the the threshold value given will be global.
2. Adaptive Thresholding
when image has different lighting conditions in different areas. In that case, we go for adaptive
thresholding. In this, we have to calculate the threshold for a small regions of the image. So we get
different thresholds for different regions of the same image and it gives us better results for images
with varying illumination.[13]
3. Otsus Thresholding
It automatically calculates a threshold value from image histogram for a bimodal image (In simple
words, bimodal image is an image whose histogram has two peaks). For images which are not
bimodal, Otsus Thresholding wont be accurate.
Figure 4.3: Otsus Thresholding

4.5.2 Image Filtering
We will mainly focus on two types of filters:
1)Smoothing (low-pass)
2)Sharpening (high-pass)
1.Smoothing filters
Some of the Smoothing filters are listed and explained below:
a)Averaging filtering
b)Gaussian filtering
c)Median filtering
a)Averaging filtering
It simply takes the average of all the pixels under kernel area and replaces the central element with
this average. This type of filtering is also called Blur Filtering. Fig below Shows the effect of
averaging filter on input image as well as the updated value of matrix and Mean Square Error:
Figure 4.4: Averaging filtering

b)Gaussian filtering
The smoothing depends on standard deviation in the X and Y directions, sigmaX and sigmaY re-
spectively. Fig below Shows the effect of gaussian filter on input image as well as the updated
value of matrix and Mean Square Error:
Figure 4.5: Gaussian filtering
c)Median filtering
In this the median of all the pixels under the kernel window and the central pixel is replaced with
this median value. This is highly effective in removing salt-and-pepper noise. One interesting
thing to note is that, in the Gaussian and box filters, the filtered value for the central element can be
a value which may not exist in the original image. However this is not the case in median filtering,
since the central element is always replaced by some pixel value in the image. This reduces the
noise effectively. Fig below Shows the effect of Median filter on input image as well as the updated
value of matrix and Mean Square Error:

Figure 4.6: Median filtering
2.Sharpening filters
Some of them are listed below:
a)Unsharp masking
b)High Boost filter
c)Gradient (1st derivative)
d)Laplacian (2nd derivative)
4.5.3 Tracking Algorithm’s
In Opencv there are so many algorithms are available out of which we are going to use two of them
those are explained below:
1)MeanShift
2)CamShift
MeanShift
The aim of meanshift is simple. Consider you have a set of points(It can be a pixel distribution).You
are given a small window ( may be a circle) and you have to move that window to the area of max-

imum pixel density (or maximum number of points). It can be explained using the simple image
given below:
Figure 4.7: MeanShift
The initial window is shown in blue circle with the name C1. Its original center is marked in
blue rectangle, named C1-o. But if you find the centroid of the points inside that window, you
will get the point C1-r (marked in small blue circle) which is the real centroid of window. Surely
they dont match. So move your window such that circle of the new window matches with previous
centroid. Again find the new centroid. Most probably, it wont match. So move it again, and con-
tinue the iterations such that center of window and its centroid falls on the same location (or with a
small desired error). So finally what you obtain is a window with maximum pixel distribution. It is
marked with green circle, named C2. As you can see in image, it has maximum number of points.
CamShift
As seen in mean shift the size of the window remains constant which is not good for the mov-
ing object,So in order to adapt the window size with size and rotation of the target, CAMshift
(Continuously Adaptive Meanshift) is invented by Gary Bradsky. It applies meanshift first. Once
meanshift converges, it updates the size of the window. It also calculates the orientation of best
fitting ellipse to it. Again it applies the meanshift with new scaled search window and previous

window location. The process is continued until required accuracy is met.[13]
4.6 Virtual Network Computing
Figure 4.8: VNC Connections
The aim is to install the VNC server software on Pi and the VNC viewer software on the host
computer.conﬁgure the Pi to give out an IP address.Raspberry Pi will now be directly connected
to a host computer using a single Ethernet cable, thus making a completely isolated point to point
network between the two and therefore your network becomes stronger.
Note: (you don’t need a cross over cable for connection,a standard cable will work because the
Pi Ethernet port automatically switches the transmit and receive pins.) It makes the Pi Ethernet
port behave in a similar way to a home router. This means assigning a static IP address to it and
installing a DHCP service that will respond to address requests from the host computer. It is good
to use an IP address range that is very different to your main network.Using this internet can be
shared from laptops WiFi over Ethernet. This also lets you access internet on the pi and connect
raspberry pi to laptop display.

Chapter 5
METHODOLOGY
5.1 Block Diagram and Description
Block Diagram:
Figure 5.1: Block Diagram[9]

Description:
The block diagram mainly consist of six parts
1) User
2) Sunglass frame
3) IR Module
4) Micro CMOS cam
5) Processor
6) Computational Software
7) Monitor
1)User:The user will be one who will provide the input to the system through his/her eyes.
2)Sunglass frame: Cheap and strong enough to sustain the weight of camera.
3)IR Module: Four IR LEDs mounted below the camera which will propagate the ir light on hu-
man eye which will help in tracking the eye movements more precisely.
4)Micro CMOS cam:IR sensitive and minimum resolution of 640 x 480 which will be mounted
on the eyeglass frame.
5)Processor:Raspberry pi which will process the images/pictures captured by camera.
6)Computational Software:Raspbian,Open CV.The output from processor is then computed in
open cv.
7)Monitor:It will be used to display the output generated from Computational software.

5.2 Flowchart
Figure 5.2: Flowchart

5.3 Implementation
Visible spectrum imaging is a passive approach of taking ambient light reflected from the eye. it
is often the case that the most effective feature to track in visible spectrum analysis is the contour
between the iris and the sclera known as the limbus. The 3 most relevant features of the eyes
are the pupil - the aperture that lets light into the pupil (eye), the iris - the colored muscle group
that controls the diameter of the pupil(eye), and the sclera, the white protective tissue that covers
the remaining portion of the pupil(eye). Visible spectrum eye tracking is complicated by the fact
that uncontrolled adequate light conditions is being used as the source, which can contain multiple
specular and diffuse components. IR imaging eliminates uncontrolled specular reflection by illu-
minating the eye with a uniform and controlled IR light not perceivable by the user.
A further benefit of IR imaging is that the pupil, rather than the limbus,it is the strongest feature
contour in the image both the sclera and iris strongly reflect IR light while only the sclera strongly
reflects visible light. Tracing the pupil contour is preferable given that the pupil contour is small
and more sharply defined than the limbus. Further more, due to its size, the pupil is less likely to
be occluded by the eyelids. The primary disadvantage of IR imaging techniques is that they cannot
be used outdoors during daytime due to the ambient IR illumination.
Infrared eye tracking typically utilizes either bright-pupil or dark-pupil techniques. Bright-pupil
techniques illuminate the eye with a source that is on or very near the axis of the camera. The
result of such illumination is that the pupil is clearly demarcated as a bright region due to the photo
reflective nature of the back of the eye. Dark-pupil techniques illuminate the eye with an off axis
source such that the pupil is the darkest region in the image, while the sclera, iris and eye lids all
reflect relatively more illumination. In either method, the first surface specular reflection of the
illumination source off of the cornea (the outer most optical element of the eye) is also visible.
This vector between the pupil center and the corneal reflection is typically used as the dependent
measure rather than the pupil center alone. This is because the vector difference is insensitive to
slippage of the head gear - both the camera and the source move simultaneously.[10]

5.3.1 USB Camera interface with the Raspberry Pi
Speciﬁcations of Camera Used:
1)Image Sensor - High quality 1/6 CMOS sensor
2)Sensor resolution - 300K pixels
3)Video Format - 24-Bit True Color
4)Max. Video Resolution - 1600 x 1200 pixels
5)Lens - High quality 5G wide angle lens
6)Max. Image Resolution - 5500 x 3640 pixels
7)Interface - USB 2.0. Backward compatible with USB 1.1
8)Focus - 5 cm to Inﬁnity
9)Night Vision - Yes
10)Power Supply - USB bus powered
11)White Balance - Auto
12)Frame Rates - 18 frames per second
USB Cameras are imaging cameras that use USB 2.0 to transfer image data. USB Cameras are
designed to easily interface with dedicated computer systems by using the same USB technology
that is found on most computers. It is on raspberry pi also. The accessibility of USB technology
in computer systems as well as the 480 Mb/s transfer rate of USB 2.0 makes USB Cameras ideal
for many imaging applications. Minimum resolution Of 640x480 pixels is required. The distance
between camera and eye is 5 to 10 cm.In this project use iBall ROBO-K20 PC webcam as a USB
camera. There is a lot of recent motivation to do image processing and computer vision tasks on
the Rasberry Pi. Then opencv is installed on raspberry Pi.We installed properly USB camera with
the raspberry pi then USB camera will use the driver to work properly on raspberry pi board.UV4L
driver will be used for raspberry pi.

Figure 5.3: Camera image without IR
Figure 5.4: Camera image with IR
5.3.2 Pupil detection
The pupil is a hole located in the center of the iris of the eye that allows light to enter the retina.
It appears black because light rays entering the pupil are either absorbed by the tissues inside the
eye directly, or absorbed after diffuse reﬂections within the eye that mostly miss exiting the narrow
pupil. Pupil - the opening in the center of the iris- it changes size as the amount of light changes

(the more light, the smaller the hole). By measuring the corneal reﬂection(s) from the IR source
relative to the center of the pupil, the system can compensate for inaccuracies and also allow for
a limited degree of head movement. Gaze direction is then calculated by measuring the changing
relationship between the moving pupil center of the eye and the corneal reﬂection.
5.3.3 Output of pupil focus
the eye and pupil of eye will be detected by the camera by the opencv code. USB Camera will
be interface to the Rasberry Pi. This result shown in raspbian OS.Then different value of X, Y
coordinates will be obtained and then these coordinates can be used for any purpose depending on
applications.
5.3.4 Execution on Raspberry pi
USB Camera interfaced with raspberry pi. Rassberry pi with the use of SD card, will then install
raspbian OS and opencv on rassberry pi. Fist image will be captured by USB Camera. Focus of
camera is on eye in an image and detection of the center position of pupil by opencv code. Take
the position value of pupil as reference, and then the different value of X, Y coordinates will be set
for particular command.[5]
5.3.5 Project as a standalone system
Our project can be implemented as a standalone system. The cost of the system can be considerably
reduced using only those features of the raspberry pi which are required like Ethernet, One USB
2.0, SD-CARD, HDMI, etc. For the purpose of displaying the result any kind of display unit like
LCD or VGA can be used. As a further development the result can be transferred to anywhere
using IoT like technologies. Because of this there no need to sit beside the patient continuously.

Figure 5.5: Project as a standalone system

Chapter 6
RESULT
1. The objective of project is to interpret the eye movements for relevant information.In view of
this it is necessary to retrieve an image with no noise or else the noise introduced will change the
interpretation.Thus a study of different filters was carried out on the image.The table below shows
that the gaussian filter is the right choice in such application least MSE.This ca be further improved
by measuring the PSNR of an image.
Sr.No. Name of the Filter Mean Square Er-
ror (MSE)
1) Averaging(Blur) 2.3945
2) Bilateral 2.7921
3) Gaussian 1.5961
4) Median 2.0907
Table 6.1: Comparison of MSE of various filters

2. As our eye i.e. pupil moves in such a way that it will draw a letter one at a time similarly a
word then a sentence can be drawn. In our project we took some samples of the different subjects
as shown in below table and then calculated the accuracy which came to be around 65-70percent.
Figure 6.1: Samples of different subjects

Chapter 7
APPLICATIONS
1.Explicit eye input is utilized in applications that implement gaze-based command and control.
Here, people use voluntary eye movements and consciously control their gaze direction, for exam-
ple, to communicate or control a computer. Gaze-based control is especially useful for people with
severe disabilities for whom eyes may be the only ordue to developments in technologya reckon-
able option to interact with the world.
2.In its simplest form, the eye can be used as a switch. For example, the user may blink once or
twice, or use simple vertical or horizontal eye movements as an indication of agreement or dis-
agreement (obtainable even with a low-cost web camera based tracker). The most common way to
implement gaze-based control is to use the eyes capability to point at the desired target (requiring
a more accurate tracker).
3.Gaze-based user modeling provides a way to better understand the users behavior, cognitive pro-
cesses, and intentions.[4]
4.Passive eye monitoring is useful for diagnostic applications in which the users visual behavior
is only recorded and stored for later ofﬂine processing and analysis with no immediate reaction or
effect on the users interaction with the world.
5.computer interaction is to move beyond such purely descriptive analyses of a small set of speciﬁc
eye movement characteristics toward developing holistic computational models of a users visual
behavior. These models typically rely on computational methods from machine learning and pat-

tern recognition. The key goal of these efforts is to gain a better understanding of and to be able to
perform automatic predictions about user behavior.
For example, As evidenced by research that a variety of visual and non-visual human activities,
such as reading or common office activities, could be spotted and recognized automatically by ana-
lyzing features based solely on eye movement patternsindependent of any information on gaze.[4]
6.It can be used for automatic annotation and filtering in life logging applications.
7.Can be used by paralyzed/needy people for communication purpose.
8.Marketing research and Medical research (neurological diagnosis).
9.The car industry does research in this field too, with the aim of developing assistant systems for
cars. For example, an eye tracker in the car could warn the driver when she or he falls asleep while
driving the car.
10.Specific applications include the tracking eye movement in language reading, music reading,
human activity recognition, the perception of advertising.

Chapter 8
CONCLUSION AND FUTURE SCOPE
Conclusion
The idea of eye control is of great use to not only the future of natural input but more importantly
the handicapped and disabled. One of the main goals for Eye Tracking Interpretation System is
to enable completely paralyzed patients to make their life more accessible and to provide them
opportunity of independence and movement.According the our studies this project is feasible.
Future Scope
1.In addition to gaze direction and eye movement patterns, also other eye-related measurements
such as the pupil size and even microsaccades can contribute to the interpretation of the users emo-
tional and cognitive state.
2.Gaze behavior can also be combined with other measurements from the users face and body,
enabling multimodel physiological computing.
3.Gaze-based user modeling may offer a step toward truly intelligent interfaces that are able to
facilitate the user in a smart way that complements the users natural behavior.[12]

Chapter 9
REFERENCES
1.De Luca A. Weiss R., and Drewes H. Evaluation of Eye-Gaze Interaction Methods for Security
Enhanced PIN-Entry In Proceedings of the 19th Australasian Conference on Computer-Human
interaction, OZCHI 2007. vol. 51. ACM Press (2007), 199 202.
2.Ashdown M., Oka K., and Sato Y. Combining Head Tracking and Mouse Input for a GUI on
Multiple Monitors. In Extended Abstracts on Human Factors in Computing Systems, CHI ’05.
ACM Press (2005), 1188 1191.
3.Atterer R. Schmidt A., and Wnuk M. A. Proxy-Based Infrastructure for Web Application Shar-
ing and Remote Collaboration on Web Pages, In Proceedings of the 11th IFIP TC13 International
Conference on Human-Computer Interaction, INTERACT 2007, Springer (2007), 74 87.
4.Abrams R.A., Meyer D.E., and Kornblum S. Speed and Accuracy of Saccadic Eye Movements:
Characteristics of Impulse Variability in the Oculomotor System, Journal of Experimental Psychol-
ogy: Human Perception and Performance (1989), Vol. 15, No. 3, 529 543.
5.Santella A., Agrawala M., DeCarlo D., Salesin D., and Cohen M. Gaze-Based Interaction for
Semi-Automatic Photo Cropping, In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems CHI ’06. ACM Press (2006), 771 780.
6.Albert W. (2002). Do web users actually look at ads A case study of banner ads and eye-tracking
technology, In Proceedings of the Eleventh Annual Conference of the Usability Professionals As-
sociation.

7.Cowen L., Ball L. J., and Delin J. (2002). An eye-movement analysis in X. Faulkner, J. Finlay,
and F. Dtienne (Eds.), People and Computers XVI Memorable yet Invisible: Proceedings of HCI
2002 (pp. 317-335). London: Springer- Verlag Ltd.
8.Dongheng Li, David Winﬁeld, Derrick J. Parkhurst. Starburst: A hybrid algorithm for video-
based eye tracking combining feature-based and model-based approaches,Human Computer Inter-
action Program Iowa State University, Ames, Iowa, 50010
9.Brigham FJ, Zaimi E, Matkins JJ et al (2001) The eyes may have it: reconsidering eye-movement
research in human cognition, In: Scruggs TE, Mastropieri MA (eds) Technological applications.
Advances in learning and behavioral disabilities, vol 15. Emerald Group Publishing Limited, Bin-
gley, pp 3959
10.Heathcote A. Brown S., and Mewhort D. J. The Power Law repealed: The case for an Expo-
nential Law of Practice, In Pyschonomic Bulletin and Review Vol. 7, Issue 2, (2000), 185 207.
11.Majaranta P. and Raiha K. Twenty Years of Eye Typing: Systems and Design Issues, In Proceed-
ings of the 2002 Symposium on Eye Tracking Research and Applications. ETRA ’02. ACM Press
(2002), 15 22.
12.Alex Poole and Linden J. Ball.Eye Tracking in Human-Computer Interaction and Usability Re-
search: Current Status and Future Prospects.
13.Alexander Mordvintsev and Abid KOpenCV-Python Tutorials Documentation Release 1.

• PROJECT SCHEDULE PLAN
Sr.No. Activity Plan(period) Execution
1) Literature survey August Completed
2) Coding and Software Development September,October and
November
Completed
3) Testing January and February Completed
4) Implementation February and March Completed
5) Report Submission March Completed
6) Project Submission April
Table 9.1: Project Schedule Plan
Project Guide Project Co-ordinator Head of Dept
Mrs.P.S.Kasliwal Mr.S.A.Khandekar Dr.M.D.Goudar

Project report on Eye tracking interpretation system

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Project report on Eye tracking interpretation system

Similar to Project report on Eye tracking interpretation system (20)

Recently uploaded

Recently uploaded (20)

Project report on Eye tracking interpretation system