Gesture recognition

Gesture Recognition
Created By: - Pathan Mariya A. Page 1
Introduction
Since the inception of graphical computer systems in April 1981,
interaction with graphical computer systems has evolved over the past few years to
include such interface metaphors as the mouse and keyboard, pen computing, touch,
and recently multi touch. These enabling developments have allowed interaction to
become more accessible and natural.
In gesture recognition technology, a camera reads the movements of
the human body and communicates the data to a computer that uses the gestures as
input to control devices or applications.
The importance of gesture recognition lies in building efficient
human machine interaction. Its applications range from sign language recognition
through medical rehabilitation to virtual reality.
Gesture recognition 35 soft computing tools pose another promising
application to static hand gesture identification.
Gesture recognition promises wide-ranging applications in fields
from photo-journalism through medical technology to biometrics.
Gesture recognition can be conducted with techniques from
computer vision and image processing.

Gesture Recognition
2. What is Gesture Recognition?
Figure 1 : Gesture Recognition
Gesture recognition is a topic in computer science and language
technology with the goal of interpreting human gestures via mathematical algorithms.
Gestures can originate from any bodily motion or state but commonly originate from
the face or hand. Current focuses in the field include emotion recognition from face
and hand gesture recognition. Users can use simple gestures to control or interact with
devices without physically touching them. Many approaches have been made using
cameras and computer vision algorithms to interpret sign language. However, the
identification and recognition of posture, gait, proxemics, and human behaviors is also
the subject of gesture recognition techniques. Gesture recognition can be seen as a
way for computers to begin to understand human body language, thus building a
richer bridge between machines and humans than primitive text user interfaces or
even GUIs (graphical user interfaces), which still limit the majority of input to
keyboard and mouse.
Gesture recognition enables humans to communicate with the machine
(HMI) and interact naturally without any mechanical devices. Using the concept of
gesture recognition, it is possible to point a finger at the computer screen so that
the cursor will move accordingly. This could make conventional input devices such
as mouse, keyboards and even touch-screens redundant.

Gesture Recognition
What is meant by gesture control?
 Gesture control is the ability to recognize and interpret movements of the
human body in order to interact with and control a computer system without
direct physical contact.
 Gesture Recognition Features:
 More Accurate
 High Stability
 Time saving to unlock a device
 The major application areas of gesture recognition in the current scenario
are:
 Automotive sector
 Consumer Electronics sector
 Transit sector
 Gaming sector
 To unlock smartphones
 Gesture Type
In computer interfaces, two types of gestures are distinguished.
We consider online gestures, which can also be regarded as direct manipulations like
scaling and rotating. In contrast, offline gestures are usually processed after the
interaction is finished; e. g. a circle is drawn to activate a context menu.
 Offline gestures: Those gestures that are processed after the user interaction with
the object. An example is the gesture to activate a menu.
 Online gestures: Direct manipulation gestures. They are used to scale or rotate a
tangible object.

Gesture Recognition
 Gesture Recognition Type
1) Hand and Arm Gesture Recognition: - Hand gesture recognition
consists of hand poses and sign languages. Hand gesture technology
allows operations of complex machines using only a series of fingers
and hand movements, eliminating the need for physical contact
between operator and machine.
2) Body Gesture Recognition: - Body gesture involves full body motion.
Recognizing body gestures, and recognizing human activity. Such as
tracking movement of two people interacting outdoors, recognizing
human gaits for medical rehabilitation and athletic training.
Why develop gesture technology?

Gesture Recognition
Why search for a 'more natural way to communicate with computers'. There are
perhaps as many answers as there are people involved in these developments. Yet,
here are some of the most common answers:
 To enable the transformation of the computer from a tool to an assistant, or a
robot, be it mr. Data or Marvin the Paranoid Android,
 To enable a hands-free or device-free interaction with a computer, expanding
the range of possibilities for computer usage,
 Because under special circumstances it would be very handy, think VR
 To get off our lazy buts and start being physical again, think Eye-toy,
 To allow those who can only use their voice, or eyes, or mouth to use
computers, think RSI or Stephen Hawking,
 Because of a personal disliking of keyboards and mice,
 Because we can.
These answers aren’t mutually exclusive, nor are the list exhaustive. But if
you work on speech or gesture recognition you'll probably resonate well with several
of the answers above.
3. Input devices

Gesture Recognition
The ability to track a person's movements and determine
what gestures they may be performing can be achieved through various tools.
The kinetic user interfaces (KUIs) are an emerging type of user interfaces that allow
users to interact with computing devices through the motion of objects and bodies.
Examples of KUIs include tangible user interfaces and motion-aware games such
as Wii and Microsoft's Kinect, and other interactive projects.
Although there is a large amount of research done in image/video based gesture
recognition, there is some variation within the tools and environments used between
implementations.
 Wired gloves: - These can provide input to the computer about the position and
rotation of the hands using magnetic or inertial tracking devices. Furthermore,
some gloves can detect finger bending with a high degree of accuracy (5-10
degrees), or even provide haptic feedback to the user, which is a simulation of the
sense of touch. The first commercially available hand-tracking glove-type device
was the DataGlove, a glove-type device which could detect hand position,
movement and finger bending. This uses fiber optic cables running down the back
of the hand. Light pulses are created and when the fingers are bent, light leaks
through small cracks and the loss is registered, giving an approximation of the
hand pose.
Figure 2:- Wired gloves

Gesture Recognition
 Depth-aware cameras:- Using specialized cameras such as structured
light or time-of-flight cameras, one can generate a depth map of what is being
seen through the camera at a short range, and use this data to approximate a 3d
representation of what is being seen. These can be effective for detection of hand
gestures due to their short range capabilities.
Figure 3:- Depth-aware cameras
 Stereo cameras:-Using two cameras whose relations to one another are known, a
3d representation can be approximated by the output of the cameras. To get the
cameras' relations, one can use a positioning reference such as a lexian-
stripe or infrared emitters. In combination with direct motion measurement (6D-
Vision) gestures can directly be detected.
Figure 4:- Stereo cameras

Gesture Recognition
 Gesture-based controllers: - These controllers act as an extension of the body so
that when gestures are performed, some of their motion can be conveniently
captured by software. An example of emerging gesture-based motion capture is
through skeletal hand tracking, which is being developed for virtual reality and
augmented reality applications. An example of this technology is shown by
tracking companies uSens and Gestigon, which allow users to interact with their
surrounding without controllers.
Figure 5:- Gesture Based Controllers
 Single camera: - A standard 2D camera can be used for gesture recognition
where the resources/environment would not be convenient for other forms of
image-based recognition. Earlier it was thought that single camera may not be
as effective as stereo or depth aware cameras, but some companies are
challenging this theory. Software-based gesture recognition technology using
a standard 2D camera that can detect robust hand gestures.
Figure 6:- Single Camera

Gesture Recognition
Another example of this is mouse gesture trackings, where the motion of the mouse is
correlated to a symbol being drawn by a person's hand, as is the Wii Remote or
the Myo armband or the mForce Wizard wristband, which can study changes in
acceleration over time to represent gestures. Devices such as the LG Electronics
Magic Wand, the Loop and the Scoop use Hillcrest Labs' Freespace technology,
which uses MEMS accelerometers, gyroscopes and other sensors to translate gestures
into cursor movement. The software also compensates for human tremor and
inadvertent movement. AudioCubes are another example. The sensors of these smart
light emitting cubes can be used to sense hands and fingers as well as other objects
nearby, and can be used to process data. Most applications are in music and sound
synthesis, but can be applied to other fields.

Gesture Recognition
Uses of Gesture Recognition
1) Socially assistive robotics: - The field of socially assistive robotics is growing
but has not yet been properly defined and circumscribed. There has been
significant attention given to and great progress made in contact assistive
robotics. Yet it is crucial to note that hands-on assistive robotics is only part of
the total composition of assistive robotics. Currently there is no clear
definition of robots that provide assistance through interaction and without
physical contact, namely socially assistive robotics. We begin by
distinguishing these categories.
Figure 7:- Socially assistive robotics
2) Sign language recognition: - A gesture may be defined as a movement,
usually of hand or face that expresses an idea, sentiment or emotion e.g. rising
of eyebrows, shrugging of shoulders is some of the gestures we use in our day
to day life. Sign language is a more organized and defined way of
communication in which every word or alphabet is assigned some gesture. In
American Sign Language (ASL) each alphabet of English vocabulary, A-Z, is
assigned a unique gesture. Sign language is mostly used by the deaf, dumb or
people with any other kind of disabilities. With the rapid advancements in
technology, the use of computers in our daily life has increased manifolds. Our
aim is to design a Human Computer Interface (HCI) system that can

Gesture Recognition
understand the sign language accurately so that the signing people may
communicate with the non signing people without the need of an interpreter.
Figure 8:- Sign language recognition
3) Remote Control: - Among the rising age of technology in the field of gesture
recognition for hand gesture or human computer interaction many research are
done. Here the Handmote is referred to as use of hand gesture recognition to
control the home or office gadgets that are operated through an infrared
remote control in general. Simple remote controlled gadgets can be operated to
change a TV channel or to tune radio by finding the key on hand held remote
control and pressing it. But in this paper author puts effort to control the same
but using hand gesture. Simply a hand gesture or showing number of fingers
TV channel can be changed or it can be On/Off . By showing a cross fingered
gesture to the camera, TV can be muted, rotating the hand in clockwise or
counter clockwise can change the volume level or TV channels.
Figure 9:- Remote control

Gesture Recognition
4) Aid to physically challenge: - People who are visually impaired or have some
other complexity in their motor functions can take help of gesture based input
devices so that there is no discomfort while they access computers. Also, these
days machine wheel chairs are coming with gesture based systems. All that is
required from the user in here is to lightly move hands on the panel at the arm
rest of the wheel chair. The movements of the hands will act as a controller
and speed as well as direction can be easily controlled.
Figure 10:- Aid to physically challenged
5) Video Game Controllers:- With the arrival of 6th generation video game
consoles such as Microsoft X-Box with Kinect sensor, Sony PS3 with motion
sensor controller, gesture recognition was widely implemented. In X-Box,
often the user is the controller and has to perform all the physical movements
that they desire the character in the game to do. For instance, one has to
imitate kicking a football if he is playing football on any of the above listed
gaming console. The Kinect sensor has a camera that catches the motions and
processes it so that the character exactly does it.
In Sony PS3, users have to move the controller in such a
way so that it imitates the action the user wants the character in the game to
perform.

Gesture Recognition
Figure 11:- Video Game Controllers
6) Control through facial gesture :-A system tracks human head and facial
features over time by analyzing a sequence of images. The system provides
descriptions of motion of both head and facial features between two image
frames. These descriptions of motion are further analyzed by the system to
recognize facial movement and expression. The system analyzes motion
between two images using parameterized models of image motion. Initially, a
first image in a sequence of images is segmented into a face region and a
plurality of facial feature regions. A planar model is used to recover motion
parameters that estimate motion between the segmented face region in the first
image and a second image in the sequence of images.
Figure 12:-Control through facial gesture

Gesture Recognition
Algorithms
Figure 13:-Gesture Recognition Algorithms
Different ways of tracking and analyzing gestures exists. And
some basic layout is given is in the diagram above. For example, volumetric models
convey the necessary information required for an elaborate analysis, however they
prove to be very intensive in terms of computational power and require further
technological developments in order to be implemented for real time analysis. On the
other hand, appearance-based models are easier to process but usually lack the
generality required for Human-Computer interaction.
The taxonomy that seems very appropriate for Human-Computer
Interaction has been proposed by Quek in “Toward a vision-Based Hand Gesture
Interface”. He presents several interactive gesture systems in order to capture the
whole space of the gestures:
1. Manipulative;
2. Semaphoric;
3. Conversational.
Some literature differentiates 2 different approaches in gesture
recognition: a 3D model based and an appearance-based. The foremost

Gesture Recognition
method makes use of 3D information of key elements of the body parts in
order to obtain several important parameters, like palm position or joint
angles. On the other hand, Appearance-based systems use images or videos for
direct interpretation.
3D Model-basedalgorithms:-
A more interesting approach would be to map
simple primitive objects to the person’s most important body parts
Figure 14:-3-D Model based algorithms
(For example cylinders for the arms and neck, sphere for the head) and analyse the
way these interact with each other. Furthermore, some abstract structures like super-
quadrics and generalised cylinders may be even more suitable for approximating the
body parts.
A real hand (right)is interpreted as a collection of vertices and lines in the
3D mesh version (left) , and the software uses their relative position and interaction in
order to infer the gesture.
Skeletalbasedalgorithms:-
Instead of using intensive processing of the 3D
models and dealing with a lot of parameters, one can just use a simplified version of
joint angle parameters along with segment lengths. This is known as a skeletal

Gesture Recognition
representation of the body, where a virtual skeleton of the person is computed and
parts of the body are mapped to certain segments.
Figure 16:-Skeletal based algorithms
The skeletal version (right) is effectively modelling the hand (left).
This has fewer parameters than the volumetric version and it’s easier to compute,
making it suitable for real-time gesture analysis systems.
Advantages of using skeletal models:
 Algorithms are faster because only key parameters are analyzed.
 Pattern matching against a template database is possible.
 Using key points allows the detection program to focus on the significant parts
of the body.
Appearance-basedAlgorithms:-
These models don’t use a spatial representation of the
body anymore, because they derive the parameters directly from the images or video
using a template database. Some are based on the deformable 2D templates of
human parts of the body, particularly hands. Deformable templates are sets of points
on the outline of an object, used as interpolation nodes for the objects outline
approximation. These template-based models are mostly used for hand-tracking but
could also be use for simple gesture classification.

Gesture Recognition
Figure 16:-Appearance based algorithms
Second approach is image sequences. Parameters for this method
are either the images themselves, or certain features derived from these.
These binary silhouette(left) or contour(right) images represent
typical input for appearance-based algorithms. They are compared with
different hand templates and if they match, the correspondent gesture is
inferred.

Gesture Recognition
Architecture
The solution should act as middleware to from glue
between paltform and application. A sensor collect and sends raw data to the solution.
The solution is a middleware that sits between the platform and application. It
receives raw data from the sensor , processes and send high level data to appliaction
variouos vision based gesture application for market such as TV, Automobile,
healthcare and many others can be built on top of the solution.
Figure 17:-Architecture Gesture

Gesture Recognition
Application Development
The solution effectiveness is judge by the solution usage. The
aim of the solution will be to allow developers to easily create a variety of natural
gesture control applications such as games, consumer electronics, and car
infotainment solution.
Figure 18:-Application development gesture recognition
Let us consider the application example of car infotainment
solution to illustrate how effectively this ideal solution will be used. Automobile
manufacturers are benefitted with gesture recognition as the technology is adding
more value to their offerings. Intuitive car infotainment solutions enable the user to
explore maps, toggle menus and radio stations using simple gesture control.

Gesture Recognition
Challenges
There are many challenges associated with the accuracy and usefulness of
gesture recognition.
 For image- based gesture recognition there are limitations on the equipment
used and image noise.
 Images or video may not be under consistent lighting, or in the same location.
 Items in the background or distinct features of the users may make recognition
more difficult.
 The amount of background noise also causes tracking and recognition
difficulties, especially when partial and full occur.
 Furthermore, the distance from the camera, and the camera’s resolution and
quality, also cause variations in recognition accuracy.
 In order to capture human gesture by visual sensors, robust computer vision
methods are also required, for example for hand tracking and hand posture
recognition or for capturing movements of the head, facial expressions or gaze
direction.

Gesture Recognition
Advantages
 Speed and sufficient reliable for recognition system. Good performance
system with complex background.
 The system successfully recognized static and dynamic gestures. Could be
applied on a mobile robot control.
 Simple, fast, and easy to implement. Can be applied on real system and play
games.
 No training is required.
 Replace mouse and keyboard.
 Pointing gesture
 Navigate in virtual environment
 Pick up and manipulate virtual objects
 Interact with 3D world
 No physical contact with computer
 Communicate at a distance
Disadvantages
 Irrelevant object might overlap with the hand. Wrong object extraction
appeared if the objects larger than the hand.
 Performance recognition algorithm decreases when the distance is greater than
1.5 meters between the user and the camera.
 System limitations restrict the applications such as the arm must be vertical,
the palm is facing the camera and finger colour must be basic colour such as
either red or green or blue.
 Ambient light affects the colour detection threshold.

Gesture Recognition
Upcoming New Technologies
The Sixth Sense Device:-
Sixth Sense is a wearable gestural interface device
developed by Pranv Mistry, a PhD student in the Fluid interfaces Group at the MIT
Media Lab. IT is similar to Telepointer, a neckworm projector/camera system
developed by Media Lab student Steve Mann (which Mann originally referred to as
“Synthetic Synesthesia of the sixth sense”).
The Sixth Sense Prototype is comprised of a pocket
projector , a mirror and a camera. The hardware components are coupled in a pendent
like mobile wearable device. Both the projector and the camera are connected to the
mobile computing device in the user’s pocket. The projector projects visual
information enabling surfaces, walls and physical objects around us to be used as
interface; while the camera recognizes and tracks user’s hand gesture and physical
objects using computer-vision based techniques. The software program processes the
video stream data captured by the camera and tracks the locations of the colored
markers at the tip of the user’s fingers using simple computer-vision techniques. The
movements and arrangements of these fiduclals are interpreted into gesture s that acts
as interaction instructions for the projected application interfaces. The maximum
number of tracked fingers only constrained by the number of unique fiducials, thus
Sixth Sense also supports multi-touch and multi-user interaction. The Sixth Sense
prototype implements several applications that demonstrate the usefulness, viability
and flexibility of the system. The map application lets the user navigate a map
displayed on a nearby surface using hand gestures, similar to gestures supported by
Multi-Touch based systems, letting the user zoom in , zoom out or pan using intuitive
hand movements.
The drawing application lets the user draw on any
surface by tracking the fingertip movements of the user’s index finger. Sixth Sense
also recognizes user’s freehand gestures. For example, the Sixth Sense system
implements a gestural camera that takes photos of the scene the user is looking at by
detecting the framing gesture. The user can stop by any surface or wall and flick

Gesture Recognition
through the photos he/she has taken. Sixth Sense also lets the user draw icons or
symbols in the air using the movement of the index finger and recognizes those
symbols as interaction instructions. For example, drawing a magnifying glass symbol
takes the user to the map application or drawing an ,,@” symbol lets the user check
his mail. The Sixth Sense system also augments physical objects the user is
interacting with by projecting more information about these objects projected on
them.
Construction and working:- The Sixth Sense prototype comprises a pocket
projector, & mirror and a camera contained in a pendant like, wearable device.
Both the projector and camera are connected to a mobile computing device in the
user’s pocket. The projector projects visual information enabling surfaces, walls and
physical objects around us to be used as interfaces; while the camera recognizes and
tracks user’s hand gesture and physical objects using computer-vision based
techniques. The software program process the video stream data captured by the
camera and tracks the locations of the colored markers at the tips of the user’s fingers.
The movements and arrangements of these fiducials are interpreted into gestures that
act as interaction instructions for the projected application interfaces. Sixth Sense
supports multi-touch and multi-user interaction.
Advantage Sixth sence
 Replaces large I/P devices.
 Also use full for physically handicapped person.
 It provides a simple, usable and interesting user interface and satisfies the need
for more freedom in a human computer interaction environment.
 It is considered as a powerful tool for computers to begin to understand human
body language.
 It is widely used in various application areas since it gives the user a new
experience of feeling.
Disadvantage Sixth sence
 Very Costly.

Gesture Recognition
Conclusion
According to the Markets and Markets analysis, the growth of gesture recognition is
going to be huge. So, we have a huge opportunity to play in this technology. The end
user is now moving towards a whole new path of human machine interaction. This is
creating a demand for enabling gesture recognition in every facet of market.
The solution we are proposing will have a mammoth place in gesture recognition
market. Using the solution, we can develop easily and quickly gesture recognition
applications for various industries. The solution will aim to develop business tie-ups
with major OEMs [original equipment manufacturer].

Gesture Recognition
References
 http://www.marketsandmarkets.com/Market-Reports/touchless-sensing-
gesturing-market-369.html
 https://www.slideshare.net/search/slideshow?searchfrom=header&q=gesture+r
ecognition
 https://scholar.google.co.in/scholar?q=gesture+recognition+details&hl=en&as
_sdt=0&as_vis=1&oi=scholart&sa=X&ved=0ahUKEwjJyp63lvvSAhWDs48
KHdplDYwQgQMIFzAA
 http://www.iosrjournals.org/iosr-jece/papers/sicete-volume5/57.pdf

Gesture recognition

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Gesture recognition

Similar to Gesture recognition (20)

Recently uploaded

Recently uploaded (20)

Gesture recognition