The document discusses a lipreading model that uses a spatiotemporal front-end with ResNet and a temporal convolutional backend followed by a bidirectional LSTM backend to predict words from video input. It provides statistics on the model's top predictions and worst predictions on a test set of 50 examples with 500 labels each. Future work proposed includes improving the preprocessing step and using the concept of sentences.

1. Omer Cahana
Lipreading
from
video
Vlad Katsman | Omer Cahana

2. Viseme
is any several speech sounds that look the
same for lip reading , such as K and G, pet
and bell
Phoneme
is one of the units sound that distinguish one
word from another in a particular language
such as thumb and dumb
Creating common language

3. 02
03
01
Preprocessing

4. Model
ResNet50
maxPooling3d
3D - conv
Temporal convolutional backendBidirectional Lstm - backend
conv
maxPooling
conv
averagePooling
linear
linear
lstm
lstm
linear
concatenate
lstm
lstm
Spatiotemporal front - end

5. A Spatiotemporal front-end + ResNet
conv3D
Max pooling 3D
Input (1,29,112,112)
b. Resnet 50

6. B Temporal convolutional back-end
Max pooling
Conv - 1D
Conv - 1D
Linear
Average pooling

7. C Bidirectional LSTM back-end
Linear layer(256)
lstm(256)
Linear layer(500)
lstm(256)
lstm(256)lstm(256)

8. D Results
TOP 1 PREDICTION
TOP 5 PREDICTION
Afternoon Access Between Everything Small
Afternoon
Africa
Difficult
Levels
Evidence
Access
Cases
Editor
Anything
Asking
Between
Doing
Story
Three
During
Everything
Think
Evidence
Living
Giving
Small
Support
Tomorrow
Report
Football

9. D1 Statistics (based on test set, 50 examples x 500 labels
0
25
50
75
100
Top predictions

10. D2 Statistics (based on test set, 50 examples x 500 labels
0
20
40
60
Worst predictions

11. D2 Statistics (based on test set, 50 examples x 500 labels
0
20
40
60
Worst predictions

12. D2 Statistics (based on test set, 50 examples x 500 labels
0
20
40
60
Worst predictions

13. Future work
● Preprocessing step
improvements
● Use concept of the
sentence

14. Thank
you