This document discusses a new system to convert visual-speech to text for deaf individuals using telephone communication. The system aims to automatically recognize Cued Speech gestures and convert them to text. Researchers extracted lip shapes and hand coordinates from video recordings of individuals performing Cued Speech. They used Hidden Markov Models and feature fusion to integrate lip shapes and hand gestures for isolated word and continuous phoneme recognition. The system achieved 86-94% accuracy for isolated words and 82-89% for continuous phonemes, indicating it can effectively convert visual-speech to text.

1. Visual-speech to text
conversion applicable
to telephone
communication for deaf
individuals
30TH APRIL 2013

2. Visual-speech to text conversion applicable to telephone communication for deaf individuals
INTRODUCTION
 Lip-reading technique,
 speech can be understood by interpreting
movements of lips, face and tongue.
 not one-to-one
 Impossible to distinguish phonemes using
visual information alone

3. Visual-speech to text conversion applicable to telephone communication for deaf individuals
 the Cued Speech system
 developed by Cornett
 contains two components:
the hand shape the hand position relative to the
face.
 Hand shapes- consonant phonemes
 hand positions -vowel phonemes.
 improves speech perception to a large extent

4. Visual-speech to text conversion applicable to telephone communication for deaf individuals
the Cued Speech system

5. Visual-speech to text conversion applicable to telephone communication for deaf individuals
AIM OF NEW SYSTEM
 To investigate the designing of a system able to
automatically recognize Cued Speech and convert it
to text.
 Possible for deaf or speech-impaired individuals to
communicate with each other and also with normal-hearing
persons
 Using gestures
 captured by devices equipped by a camera

6. Visual-speech to text conversion applicable to telephone communication for deaf individuals
METHODS
 Corpus, feature extraction, and
statistical modeling
 The speakers’ lips were painted blue, and color
marks were placed on the speakers’ fingers. .
 The data were derived from a video recording of
the cuers pronouncing and coding in Cued
Speech
 landmarks with different colors were placed on
the fingers

7. Visual-speech to text conversion applicable to telephone communication for deaf individuals
 faster and more accurate image processing
stage.
 The audio part of the video recording was
synchronized with the image.
 An automatic image processing method was
appliedli pt ow idththe ( Av)i,d eo
 lip aperture (B),
 lip area (S).
 pinching of the upper lip (Bsup)
 lower (Binf) lip

8. Visual-speech to text conversion applicable to telephone communication for deaf individuals
 Concatenative feature fusion
 Tracks and extracts the xy coordinates
each time frame,
 uses those values as features in the
HMM modeling.
 uses the concatenation of the
synchronous lip shape and hand features
as the joint feature vector given by,

9. Visual-speech to text conversion applicable to telephone communication for deaf individuals
Joint lip hand
feature vector,
Lip shape
feature vector,
Hand feature
vector,
Dimensionality of the
joint feature vector
 Parameters used for lip
shape modeling.

10. Visual-speech to text conversion applicable to telephone communication for deaf individuals
RESULTS
 Isolated word recognition
1. Recognition in normal-hearing subject

11. Visual-speech to text conversion applicable to telephone communication for deaf individuals
2. Recognition in deaf subject

12. Visual-speech to text conversion applicable to telephone communication for deaf individuals
3. Multi-speaker isolated word recognition:
 investigate whether it is possible to train speaker-independent
HMMs for Cued Speech recognition.
 The training data consisted of 750 words from the
normal-hearing subject, and 750 words from the
deaf subject.
 For testing 700 words from normal-hearing subject
and 700 words from the deaf subject were used,
respectively.
 Each state was modeled with a mixture of 4
Gaussian distributions.
 For lip shape and hand shape integration,
concatenative feature fusion was used.

13. Visual-speech to text conversion applicable to telephone communication for deaf individuals

14. Visual-speech to text conversion applicable to telephone communication for deaf individuals
4. Continuous phoneme recognition
 Phoneme correct for continuous phoneme word
recognition in the case of a normal-hearing subject.

15. Visual-speech to text conversion applicable to telephone communication for deaf individuals
Phoneme correct for continuous phoneme word
recognition in the case of a deaf subject.

16. Visual-speech to text conversion applicable to telephone communication for deaf individuals
CONCLUSION
 Hand shapes and lips shape were integrated
using concatenative feature fusion and HMM-based
automatic recognition was conducted.
 For continuous phoneme recognition, a 86%
phoneme correct was achieved for the normal-hearing
cuer and a 82.7% phoneme correct for
the dead cuer were achieved, respectively.
 Speech in both normal-hearing and deaf
subjects were also conducted obtaining a
94.9% and a 89% accuracy, respectively.
.

17. Visual-speech to text conversion applicable to telephone communication for deaf individuals
CONCLUSION
 A multi-speaker experiment using data
from both normal-hearing and deaf subject
showed a 89.6% word accuracy, on
average.
 This result indicates that training speaker-independent
HMMs for Cued Speech using
a large number of subjects should not face
particular difficulties

18. Visual-speech to text conversion applicable to telephone communication for deaf individuals
REFERENCES
 G. Potamianos, C. Neti, G. Gravier, A. Garg, and A.W. Senior,
“recent Advances in the automatic recognition of audiovisual
speech,” in Proceedings of the IEEE, vol. 91, issue 9, pp.
1306–1326, 2003.
 S. Nakamura, K. Kumatani, and S. Tamura, “Multi-modal
temporal asynchronicity modeling by product hmms for
robust audio-visual speech recognition,” in Proceedings of
Fourth IEEE International Conference on Multimodal
Interfaces (ICMI’02), p. 305, 2002.
 R. O. Cornett, “Cued speech,” American Annals of the Deaf,
vol. 112, pp. 3–13, 1967.
 J. Leybaert, “Phonology acquired through the eyes and
spelling in deaf children,”Journal of Experimental Child
Psychology, vol. 75, pp. 291– 318, 2000

19. Thank you!

20. Visual-speech to text conversion applicable to telephone communication for deaf individuals
ANY
QUESTION
S?