Mouse Cursor Control Hands Free Using Deep Learning

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 10 Issue: 05 | May 2023 www.irjet.net p-ISSN: 2395-0072
© 2023, IRJET | Impact Factor value: 8.226 | ISO 9001:2008 Certified Journal | Page 1521
Mouse Cursor Control Hands Free Using Deep Learning
G. Abhishek1, M. Bhanu Teja2, S. Mohammed Hafeez3, N. Latha Madhuri4
1UG student, Dept. of Electronics and Computers Engineering, Sreenidhi Institute of Science and Technology,
Telangana, India
Telangana, India
Telangana, India
4Asst. Professor, Dept. of Electronics and Computers Engineering, Sreenidhi Institute of Science and Technology,
Telangana, India
---------------------------------------------------------------------***---------------------------------------------------------------------
Abstract – Mouse-free cursor control using facial
movements has emerged as an innovative technology that
revolutionizes the way we interact with digital interfaces. By
harnessing the power of facial expressions, users cannavigate
menus, click, scroll, and perform various actions seamlessly,
mimicking natural interaction. This technologyeliminatesthe
need for physical input devices such as mice and trackpads,
offering a more intuitive and accessible approach to human-
computer interaction. Facial recognition and tracking
technologies have played a crucial role in the development of
mouse-free cursor control.. By predictingfaciallandmarksand
analyzing features like eye aspect ratio and mouth aspect
ratio, the system can detect actions such as winks, blinks,
yawns, and pouts, which act as triggers for controlling the
mouse cursor. We have used SVM and HOG algorithms in this
project to predict cursor movement from facial features
extracted and HOG for obtaining facial landmarks. The
applications of mouse-free cursor control extend beyond
personal computing devices. Industries such as healthcare,
gaming, virtual reality, and design can greatly benefit from
this technology.Inhealthcare, itenableshands-free navigation
of medical records, improving workflow efficiency and
reducing the risk of contamination. In gaming and virtual
reality, it provides an immersive and intuitive experience by
allowing users to control their virtual avatars with facial
expressions
Key Words:Mouse-freecursorcontrol,facial movements,
digital interfaces, natural interaction, SVM, HOG, facial
recognition, tracking technologies, intelligent systems,
facial landmarks, eye aspect ratio, mouth aspect ratio.
1.INTRODUCTION
In today's digital age, where computers and devices play an
integral and essential role in our daily lives, the way we
interact with technology continues to evolve.Thetraditional
cursor control methods that are prevailing in the market are
primarily reliant on manual input devices such as mice and
trackpads, are being challenged by innovative solutions
aimed at enhancing user experience and accessibility. One
such groundbreaking advancement is the advent of mouse-
free cursor control using facial movements. This technology
allows users to navigate and interact with digital interfaces
through natural facial expressions. Facial recognition and
tracking technologies have witnessed significantprogress in
recent years, finding applications in various domains, from
security systems to augmented reality. Building upon these
advancements, researchers and engineers have devised
intelligent systems capable of accurately interpreting facial
movements and translating them into precise cursor
movements on the screen. This breakthrough opens up a
world of possibilities, empowering individuals with limited
mobility, eliminating the need for physical peripherals, and
revolutionizing the way we interact with digital interfaces.
The concept of this project is to make facial movements
embodies a user-centered approach, emphasizing the
importance of intuitive and natural interaction. By
harnessing the power of facial expressions, users can
navigate menus, click, scroll, and perform a wide range of
actions seamlessly, mimicking the way we instinctively
interact with the physical world. This technology will not
just enhance the accessibility butalsoimprovesproductivity,
as it eliminates the physical constraints linked with
traditional input devices, allowing users to interact with
their devices in a more dynamic and flexible manner.
Moreover, the potential applications of mouse-free cursor
control extend beyond personal computing devices.
Industries such as healthcare, gaming, virtual reality, and
design can greatly benefit from this technology. In
healthcare, for instance, it can facilitate hands-free
navigation of medical records, reducing the risk of
contamination and improving workflow efficiency. In the
gaming and virtual reality realms, it can provide an
immersive and intuitive experience by enabling users to
control their virtual avatars.
2. LITERATURE SURVEY
The primary objective of this eye-controlled system is to
monitor cursor movement byidentifyingthecentral position
of the pupil. To achieve this, a combination of Raspberry Pi
and OpenCV is employed. The Raspberry Pi serves as the
hardware platform and is equipped with an SD/MMC card

slot. The SD card contains the necessary operating system
for the Raspberry Pi to begin operating. Oncetheapplication
program is installed on the Raspberry Pi, it can activate the
eye-tracking feature and carry out the required
operations.[1]
In this project, the authors aimed to assist individuals with
hand disabilities that hinder their ability to use a mouse. To
address this, we developed an application that utilizes facial
features such as the nose tip and eyes to interact with a
computer. Our approach for detection involves quickly
extracting potential face candidates using a Six-Segmented
Rectangular (SSR) filter. We then verify these candidates
using a support vector machine to ensure accurate face
detection. To avoid mistakenly selecting background
elements, we incorporate a straightforward motion cue.
Additionally, for face tracking purposes, we track the
patterns between the eyes by continuously updating
template matching techniques.[2]
The system utilizes Dlib and the Haar Cascade algorithm for
cursor control, relying on eye and facial movements. This
system can be further enhanced by incorporating additional
techniques, such as clicking events and interfaces that
respond to eye movement and blinking. By leveraging these
methods, users can interact with the cursor and perform
actions through their eyes and facial expressions.[3]
This paper presents a real-time method for detecting eye
blinks and winks using video footage andMATLABsoftware.
The approach includes localizing the eyes and facial pairs, as
well as applying optical flow classification and analysis. The
methodology in this study has been verified and tested in
various environments, demonstrating a high level of
accuracy in detecting blinks and winks. Furthermore, the
system is capable of performing mouse-like functions,
providing users with an alternative means of interacting
with their computers.[4]
3. METHODOLOGY
The procedure entails utilizing a Face Detection Model to
capture a facial image from a live video feed obtained
through a camera or webcam. Following that, the Landmark
Detection Model is applied to identify different facial
components such as the mouth, left and righteyes,andother
facial characteristics fromtheobtainedimage.Subsequently,
the Head Pose Estimation Model is employed to analyze the
movements of the head, which correspond to the motion of
the computer mouse cursor. The accurate orientation of the
computer mouse cursor reliesonthecollaborativeoutcomes
of both the Landmark Detection Model and the Head Pose
Estimation Model, which are integrated by the Gaze
Estimation Model. The Gaze Estimation Model plays a vital
role in deciphering facial expressions to ensure precise
positioning of the mouse pointer.
Fig 3.1: Architecture of Proposed Work
4. WORKING
This system that we developed seamlessly integrates
advanced machine learning algorithms like Support Vector
Machines (SVM)andHistogramofOrientedGradients(HOG),
alongside facial ratiossuchasMouthAspectRatio(MAR)and
Eye Aspect Ratio (EAR). Essential machine learninglibraries
like OpenCV, dlib, and pyautoGUI are instrumental in the
implementation of this cutting-edge technology. By
effectively detecting and tracking facial landmarks,
extracting pertinent features, and employing SVM
classification with the MAR and EAR ratios, the system
precisely interprets facial movements to accurately predict
the desired cursor direction. The steps involved in the
working of this project are:-
Step 1: The system starts by using a facedetectionalgorithm
from OpenCV or dlib to locate and extract the face region
from a video stream captured by a camera or webcam.
Step 2: Once the face is detected, a landmark detection
algorithm, such as dlib's shape predictor or OpenCV's facial
landmark detector, is employed to identify key facial
landmarks such as the eyes, nose, mouth, and eyebrows.
Step 3: The system computes facial ratios,suchastheMouth
Aspect Ratio (MAR) and Eye Aspect Ratio (EAR). MAR is
calculated by measuring the ratio of the distance between
the upper and lower lips to the vertical distancebetween the
mouth and the chin. EAR is calculated bymeasuringthe ratio
of the distance between the inner and outer corners of the
eyes to the distance between the vertical midpoints of the
eyes and the eye corners. These ratios provide valuable
information about the mouth and eye movements.
Step 4: To enable machine learning, a trainingdatasetneeds
to be created. This dataset includes samples of facial
movement patterns, such as moving the cursor up, down,

left, or right. The dataset is labeled with the corresponding
cursor movement.
Step 5: A Support Vector Machine (SVM) classifier is trained
using the extracted features, includingMAR andEAR,andthe
corresponding cursor movement labels. SVM isa supervised
learning algorithm that can classifyandpredictnewsamples
based on the learned patterns.
Step 6: Once the SVM model is trained, the system utilizes
the real-time facial movement data obtained from the facial
landmarks to calculate the MAR and EAR ratios.Theseratios
are then fed into the trained SVM model to predict the
corresponding cursor movement. The SVM model classifies
the facial movement features and determines the
appropriate cursor direction.
Step 7: Finally, the pyautogui library is employed to
translate the predicted cursor movement into actual mouse
movements. Pyautogui provides functions to control the
mouse cursor's position and simulate mouse clicks.
Eye Region Detection: It is evident that the Eye-Aspect-
Ratio (EAR) is a straightforward yet effective feature that
effectively utilizes facial landmarks. EAR proves valuable in
detecting both blinks and winks. It Begins with the left
corner of the eye as you were looking at the person and it is
working clockwise around the left out region. Each eye is
represented with coordinates (x, y). Onlookinganimage, we
should consider the key points of an eye: As there is a
relation between the height and width of the points
(coordinates). This is called Eye Aspect Ratio(EAR).
Fig 4.1 Eye Aspect Ratio
Mouth-Aspect-Ratio (MAR): Drawing significant
inspiration from the EAR feature, we made slight
modifications to the formula to create a metric capable of
detecting the state of an open or closed mouth. While not
entirely original, the approach proves effective in achieving
the desired outcome. This feature along with EAR both are
taken into consideration to calculate the result.
Fig 4.2 Mouth Aspect Ratio
5. RESULTS
By capturing and analyzing facial movements, the system
accurately interprets the user's intentions and translates
them into precise cursor movements on the screen.Theuser
can control the cursor's direction by simply moving their
head, and basic mouse actions such as left-clicking, right-
clicking, and scrolling can be performed through intuitive
facial gestures such as winking and squinting.
Fig 5.1: Moving cursor to left
Fig 5.2: Moving Cursor to Right
Fig 5.3: Enabling scrolling mode

6. CONCLUSIONS
The concept of mouse-free cursor control using facial
movements presents an innovative and promising solution
for revolutionizing ourcomputerinteractions.Byharnessing
the power of facial movements, this technology opens up
new frontiers of accessibility, enabling individuals with
disabilities to effortlessly engage with digital devices.
Moreover, its implications span across industries, boosting
productivity and effectiveness in education, entertainment,
and healthcare. With continued research and development,
we can expect even more sophisticated and user-friendly
solutions in the future.Mouse-freecursorcontrol usingfacial
movements holds the potential to transform the way we
interact with technology, making it more intuitive, natural,
and accessible to all users, ultimately enhancing our digital
experiences
REFERENCES
[1] A. Sivasangari., D. Deepa., T. Anandhi.,A.PonrajandM.S.
Roobini., "Eyeball based Cursor Movement Control,"
2020 International Conference on Communication and
Signal Processing (ICCSP), Chennai, India, 2020, pp.
1116-1119, doi: 10.1109/ICCSP48568.2020.9182296.
[2] Sunita Barve, Dhaval Dholakiya, Shashank Gupta,
Dhananjay Dhatrak / International Journal of
Engineering Research and Applications (IJERA) ISSN:
2248-9622 www.ijera.comVol. 2, Issue 2,Mar-Apr2012,
pp.1406-1410
[3] Satyanarayana B.,etal."Eyeball MovementbasedCursor
Control using Raspberry Pi and OpenCV." 2020
International Journal of Innovative Technology and
Exploring Engineering (IJITEE). ISSN: 2278-3075
[4] Sovit Ranjan Rath, et al. “Machine Learning and Deep
Learning, Face detection withDlibusingHOGandLinear
SVM”.

Mouse Cursor Control Hands Free Using Deep Learning

More Related Content

Similar to Mouse Cursor Control Hands Free Using Deep Learning

More from IRJET Journal

Recently uploaded

Mouse Cursor Control Hands Free Using Deep Learning