lips reading

3.  Lip reading, known as speechreading, is a
technique of understanding speech by visually
interpreting to analysis of the moving lips when
normal sound is not available, where speech is
validated by both the shape and movement of the
lips. This thesis investigates various issues faced by
an visual lips reading system and proposes a novel
“visual words” based approach to visual lip
reading.
 Lip reading is used to understand or interpret speech without hearing it, a
technique especially mastered by people with hearing difficulties. The
ability to lip read enables a person with a hearing impairment to
communicate with others. Recent advances in the fields of computer
vision, pattern recognition, and imageprocessing has led to a growing
interest in this challenging task of lip reading.

4.  Is it possible to build an a system for lip reader comparable to or even better
than a human lip reader.
 The human mouth is one of the most deformable parts of the human body,
leading to different appearances such as mouth opened, closed, widely
opened, so there are problems related to extracting the shape and edge of
the lips with accuracy, which the proposed system depends on these
features for tracking lips movement using different technique.
 Which best method can be Chosen for lips feature extraction.
 When designing the system for lip reading, we will need to address
limitations Revolve around; there are no perspicuous rules used to
determine spoken Arabic words, no steady dictionary that can translate the
sequential of the video frames to a corresponding word, as well as, there is
no visual speech Arabic dataset.

5.  Recognizing speech is a very basic task for human beings, highlighting a
significant gap between the possibilities offered by current technology and
user requirements. A fundamental motivation is to contribute to bridge this
gap, allowing future users to use speech technologies without current
limitations and constraints
 Proposing a system that study should have the ability to accurately extract
the visual features of movement lips that the system later relies on it for lip
tracking and speech recognition.
 Recording training data has been an integral part. To train a Arabic visual
speech recognizer large quantities of speaker video data are required
 Vwords can be applied efficiently for speaker identification through the
person’s utterance, depending on his/her different (unique to some extent)
way of speech.

6.  The study presented by this work contributes to the lips reading research;
proposed Arabic visual word recognition methods, which add techniques for
localizing lips, extracting visual features, and tracking for recognizing lips
motion.
 For both types of lip traits (physiological and behavioral) a comprehensive
study performed to discover the underlying mechanism of the
discriminatory power of lip biometrics, and Presenting a detailed analysis on
the role of the various physiological and behavioral features of the lips in
analyzing the way the speaker pronounces the Arabic word and the degree
of its convergence between the speakers.
 Proposed polynomial motion feature for lip reading.
 The new-recorded Arabic database for lip-reading purposes can be used in
other biometrics and image processing research studies.
 The central contribution of this study to the research community
(particularly the VSR community) is the development of a accurate and
efficiency Arabic VSR system using the proposed Vwords approach,

7. 5
conclusion
1
Polynomial
Tracking
2
Geometrical
Feature
3 VGG16_vsr
4
Deep_vsr
results

8.  System structure for this work. In the first phase: the pre-processing
operation takes place, in which the face area is localized, as all the
information in tracking the visual speech is located in the mouth region.
Later, the region of the ​​interest (ROI), represented by the mouth extracted.
 The second phase is the process of extracting those features related to
visual speech from the region of the ​​interest. In the proposed model, the
features extracted from the movement and tracking of the lips contour are
classified as physiological features that depend on the shape of the lips to
recognize the spoken word.

10.  The lips motion for utterance word is then kept
as a coefficient for the polynomial function,
which interprets the movement of the lips by
using the polynomial equation and represents
this movement through Drawing curves.
 The curve can be applied to any lip model
because it is an adaptive curve without being
restricted by the size of the lip model;
 polynomial Equation:
2
1
3
4
5
Y= A + Bx + Cx2 + Dx3 + Ex4 +…..

11. 2
1
3
4
5
 Synthesis lips motion based on geometric
features doing by using facial points that
correspond to the lip region which represent
thought points from 49 to 68.
 Lip features geometry can be extracted by
measuring the distance between the upper and
lower lip, corners.
 Calculated MAR_out for extracting geometrical
lips shape using formula:

18. 2
1
3
4
5
2
1
3
4
5  Deep learning techniques provide perfect
solutions to the problems of automatically
extracting features, the proposed model was
mainly based on the VGG16 network.
 The model consists of two main parts: in the first
part, it is used to extract visual features, which is
the information that represents the spoken
word, and the second part is the classification
which is based on those features extracted for
the purpose of recognizing the spoken word.

20. 2
1
3
4
5
2
1
3
4
5

21. 2
1
3
4
5
2
1
3
4
5

23. 2
1
3
4
5
2
1
3
4
5  Using pre-trained VGG16_vsr in lip reading at
the level of a word in the Arabic language to
increase accuracy in predicting the word. The
proposed method based on image processing
and transfer learning, in addition to fine-tuning
and data augmentations technology, provided a
high efficiency and accuracy of system
performance.

24.  The proposed “visual words” (Vwords) scheme uses lips geometric
measurements tackle the VSR problem, where the system recognizes the whole
word In this approach, a word is represented by a signature that consists of
several feature vectors. Each signal is constructed by temporal measurements
of its associated feature. for instance, the mouth height feature measured over
the time period of a spoken word.
 Using MAR_out and inner with three key point on the lips counter leads to
increase the lips tracking accuracy for recognized visual words.
 The slopes occur along the line of movement of the lips of the frames at
different times; the variation in the slopes may vary depending on the peak in
the speech movement at that time, so when implementing our VSR model
based on a polynomial mathematical equation, we got to conclude that high
order is used in the polynomial function to simulate the curve of the
movements of the control points, so that the high order of the polynomial
function leads to the possibility of analyzing the visual speech and storing the
visual information in the function parameters, which are later represented as a
curve, and that the obtained curve is less curvy and wobbly.

25. “thank for all of you”