This document summarizes a research paper titled "EyePhone: Activating Mobile Phones With Your Eyes". It discusses the following key points: 1. The paper proposes a system called EyePhone that allows users to control their mobile phone with eye movements and blinks detected by the front-facing camera. EyePhone tracks the user's eye on the display and detects blinks to emulate mouse clicks. 2. EyePhone works in four phases - eye detection, open eye template creation, eye tracking, and blink detection. It uses template matching and thresholding techniques to detect eyes, track eye movements, and determine when the user blinks. 3. The system was evaluated for accuracy of eye tracking and blink

1. EYEPHONEEYEPHONE
Emiliano Miluzzo, Tianyu Wang, Andrew T. Campbell,
"EyePhone:Activating Mobile Phones WithYour Eyes,"
In Proc. of the ACM SIGCOMM Workshop on Networking,
Systems, and Applications on Mobile Handhelds
(MobiHeld'10),August 30, 2010
Class Code: EEL 6788 Instructor: Dr. Damla
Turgut
Date: 03-16-2011 Presenter: Taranjeet Singh Bhatia

2. Content
 Technology introduction
 HPI Evolution
 Terminologies
 Paper Abstract
 Eye Phone Design
 Evaluation
 Application
 Demo
 Future work
 Improvement on Paper
 Related work

3. Technology Introduction
 Human Computer Interaction(HCI)
“HCI (human-computer interaction) is the study of how people interact with
computers and to what extent computers are or are not developed for
successful interaction with human beings”
Most HCI technology addresses the interaction
between people and computers in “ideal”
environments, i.e., where people sit in front of a
desktop machine with specialized sensors and
cameras centered on them

4. Technology Introduction
 Human Phone Interaction(HI)
“Human-Computer Interaction (HCI) researchers and phone vendors are
continuously searching for new approaches to reduce the effort users exert
when accessing applications on limited form factor devices such as mobile
phones.”
Human-phone interaction (HPI) extends the challenges not typically found in HCI
research, more specially related to the phone and how we use it. In order to
address these goals HPI technology should be less intrusive; that is,
i) it should not rely on any external devices other than the mobile phone itself;
ii) it should be readily usable with minimum user dependency as possible;
iii) it should be fast in the inference phase;
iv) it should be lightweight in terms of computation;
v) it should preserve the phone user experience, e.g., it should not deplete the
phone battery over normal operations.

5. HPI Evolution - Keypad
 1983 –

6. HPI Evolution – Touch Screen
 1993 –

7. HPI Evolution – Trackball
 2006 –

8. HPI Evolution – Voice Operated
 2010 –

9. HPI Evolution – What Next ?
Eye Phone – Controlling Phone by Eyes

10. Terminologies
 Intrusive Method:
 which requires direct contact with the eyes
 Non- Intrusive methods:
 which avoid any physical contact with the user
 Model based method
 Appearance based method
 Feature based method

11. Paper Abstract
EyePhone tracks the user’s eye movement
across the phone’s display using the camera
mounted on the front of the phone
i) Track the eye and infer its position on the mobile
phone display as a user views a particular
application
ii) Detect eye blinks that emulate mouse clicks to
activate the target application under view.

12. EyePhone Design
1) An eye detection phase
2) An open eye template creation phase
3) An eye tracking phase
4) A blink detection phase

13. Eye detection phase
 By applying a motion analysis technique which operates on
consecutive frames, this phase consists on finding the contour of
the eyes. The eye pair is identified by the left and right eye
contours.
Results of original algorithm
running on a desk-top with a USB
camera
Ref: EyeTrackingandBlinkDetectionLibrary.
(http://tinyurl.com/yb9v4f2)
Results of EyePhone running
on a Nokia N810.
The smaller dot s are
erroneously int erpret ed as
eye cont ours

14. Eye detection
Based on previous experimental observations, they modify the original
algorithm
i) Reducing the image resolution, which reduces the eye detection
error rate
ii) Adding two more criteria to the original heuristics that filter out the
false eye contours
widthmin ≤ width ≤ widthmax
heightmin ≤ height ≤ heightmax
widthmin, widthmax, heightmin, heightmax are thresholds, which identify the possible
sizes for a true eye contour, are determined under various experimental
conditions (e.g., bright, dark, moving, not moving) and with different people. This
design approach boosts the eye tracking accuracy considerably

15. Open eye template creation
EyePhone creates a template of a user’s open
eye once at the beginning when a person uses
the system for the first time using the eye
detection algorithm. The template is saved in
the persistent memory of the device and
fetched when EyePhone is invoked
 Downsideofthisoff-linetemplatecreationapproachisthatatemplatecreatedin
certainlightingconditionsmightnotbeperfectlysuitableforotherenvironments

16. Eye tracking
The eye tracking algorithm is based on template matching.
The template matching function calculates a correlation
score between the open eye template, created the first
time the application is used, and a search window.
Outer box around the left eye encloses the region where the correlation score
is calculated. If the normalized correlation coefficient equals 0.4 we conclude
that there is an eye in the search window. This threshold has been verified
accurate by means of multiple experiments under different conditions (e.g.,
bright, dark, moving, not moving)

17. Eye tracking
Correlation:
1. Correlationisameasureofthedegreetowhichtwovariablesagree, notnecessaryinactualvaluebutingeneralbehavior.
2. Thetwovariablesarethecorrespondingpixelvaluesintwoimages, templateandsource.
Thevaluecorisbetween– 1 and+1, withlargervaluesrepresentingastrongerrelationshipbetweenthetwoimages
( )
( ) ( )∑ ∑
∑
−
=
−
=
−
=
−⋅−
−⋅−
=
1
0
1
0
22
1
0
)(
N
i
N
i
ii
i
N
i i
yyxx
yyxx
cor
x is the template gray level image
x is the average grey level in the template
image
y is the source image section
y is the average grey level in the source
image
N is the number of pixels in the section
image
(N= template image size = columns * rows)

18. Blink Detection
To detect blinks they apply a thresholding technique for the
normalized correlation coefficient returned by the
template matching function
Problem: QualityofthemobilecameraisnotthesameasagoodUSBcamera, andthephone’scameraisgenerallyclosertotheperson’s
facethanisthecaseofusingadesktopandUSBcamera. Becauseofthislattersituationthecameracanpickupirismovements,
i.e., theinterioroftheeye, duetoeyeballrotation. Inparticular, whentheirisisturnedtowardsthecorneroftheeye, upwardsor
downwards, ablinkisinferredeveniftheeyeremainsopen. Thisoccursbecauseinthiscasethemajorityoftheeyeballsurface
turnswhitewhichisconfusedwiththecoloroftheskin
Four thresholds:
T1
min
= 0.64
T1
max
=0.75
T2
min
=-0.53
T2
max
=-0.45
If we indicate with cmin and cmax, respectively, the minimum
and maximum normalized correlation coefficient values
returned by the template matching function,
the eye is inferred to be closed if
T1
min
≤ cmax ≤ T1
max
and T2
min
≤ cmin ≤ T2
max
It is inferred to be open otherwise.

19. EVALUATION
DS = eye tracking accuracy measured in daylight exposure and being steady;
AS =eye tracking accuracy measured in artificial light exposure and being
steady;
DM = eye tracking accuracy measured in daylight exposure and walking;
BD = blink detection accuracy in daylight exposure
DeviceUsed: NokiaN810, 400MHzprocessor, 128MBofRAM, Maemo4.1 OS, OpenCV(OpenSourceComputerVision)
library, algorithmscompiledontheMaemoscratchbox, videoframesfromthecamerausingGStreamer

20. EVALUATION
Average CPU usage, RAM usage, and computation time for one video frame.
The front camera supports up to 15 frames per second. The last column
reports the percentage of used battery by EyePhone after a three hour run
of the system

21. Applications
 EyeMenu
 Car Driver Safety
EyePhone could also be used to detect drivers drowsiness
and distraction in cars.

22. Demo
http://sensorlab.cs.dartmouth.edu/eyephone/demo.m4v

23. FUTURE WORK
 Creation of the open eye template and the
filtering algorithm for wrong eye contours
 Improvement over variations of lighting
conditions or movement of the phone in a
person’s hand
 Using a learning approach instead of a fixed
thresholding policy

24. Improvement On The Paper
 Very Basic Approach used
 High speed and better DSP libraries are
available
 Need to implement on I-phone and android
phones
 Can be incorporated with face detection
 Iris tracking
 Improvement for various light condition and
face orientation
 Online template creation

25. RELATED WORK
There are a number of developments in HCI related to mobile phones
over the last several years
 Use of accelerometers on the phone in order to infer gestures
 PhonePoint Pen project : phone being transformed into a pen to
write in the air in order to quickly take small notes without needing
to either write on paper or type on a computer. [4]
 uWave project exploits a 3-D accelerometer on a Wii remote-based
prototype for personalized gesture recognition. [7]
 Eye movement has recently been used for activity recognition. By
tracking the eye researchers show that it is possible to infer the
actions of watching a video, browsing the web or reading notes. [24]
 The eyeSight technology exploits the phone camera to control the
mobile phone by simple hand gestures over the camera. [5]

26. PAPER REFERENCES
[4] S. Agrawal, I. Constandache, S. Gaonkar, and R.R.Choudhury. PhonePoint Pen: Using Mobile Phones to Write in
Air. In MobiHeld ‘09,pages1–6.ACM,2009.
[5] eyeSight. http://www.eyesight-tech.com
[7] J. Liu, L. Zhong, J. Wickramasuriya, and V. Vasudevan. uWave: Accelerometer-Based Personalized Gesture
Recognition and its Applications. In Pervasive and Mobile Computing, vol. 5, issue 6, pp. 657-675, December
2009.
[8] Accelerometer-Based Gesture Recognition with the iPhone. http://tinyurl.com/yjddr8q.
[9] The Mobile Phone that Could “Read Lips”. http://www.cellular-news.com/story/42211.php.
10] Nokia Mobile Radar. http://tinyurl.com/yfo2dtd.
[11] The Eye Mouse project. http://www.arts.ac.uk/research/eyemouse/index.htm.
[12] F. Karray, M. Alemzadeh, J.A. Saleh, and M.N. Arab.Human-Computer Interaction: Overview on State of the Art.
In International Journal on Smart Sensing and Intelligent Systems, Vol. 1, No. 1, March 2008.
[13] M. Chau and M. Betke. Real Time Eye Tracking and Blink Detection with USB Cameras. In Boston University
Computer Science Technical Report No.2005-12,2005.
[14] E. Miluzzo, C.T. Cornelius, A. Ramaswamy,T. Choudhury, Z. Liu, A.T. Campbell. Darwin Phones: The Evolution of
Sensing and Inference on Mobile Phones. In Eighth International ACM Conference on Mobile Systems,
Applications, and Services (MobiSys ’10), San Francisco, CA, USA, June 15-18, 2010.
[15] EyePhone demo video. http://sensorlab.cs.dartmouth.edu/eyephone/demo.m4v
[16] Nokia n900. http://maemo.nokia.com/n900.
[17] Eye Tracking and Blink Detection Library. http://tinyurl.com/yb9v4f2.

27. Thank You
Questions?