My portfolio project presentation for Data Science Retreat Batch 09 Demo Day: Utilising convolutional neural networks and transfer learning for translating American Sign Language (fingerspelling) from images/videos into text in real-time Further details & code available on GitHub: https://github.com/BelalC/sign2text

1. Belal Chaudhary
Real-time translation of
Sign Language into text
DSR Demo Day Symposium

2. Objectives:
• Translate American Sign Language (ASL) from
images into text (Sign2Text)
• Focus on the ASL alphabet (A-Z)
Real-time translation of
Sign Language into text

3. Pipeline
Pillow

4. ASL BU Corpus
1555 images
Crowdsourced
300 images

5. T
NM
B
A A

6. U V W

7. Raw image
Padding
+
BGR channel editing
• ~2000 images
• ~70 / class (unbalanced numbers)
244 x 244
Re-size for NN

8. Model Accuracy
Time per
prediction
(CPU)
VGG16
+ classification block
99.1% ~300ms
ResNet50
+ classification block
87.2% ~235ms
Transfer learning + CNN

9. DATA
AUGMENTATION
applied every epoch

10. LIVE DEMO
Public repo: https://github.com/BelalC/sign2text

11. Thanks
• Data science
• Deep learning
• Text, images, sensors
• Biotechnology
Let’s talk…
/BelalC
/belalchaudhary
• Volunteer signers,
interpreters, signing
community
• Boston University

12. Appendix
VGG16 Architecture
1000classes

13. Appendix
My Model architecture Flatten
FC (256)
Dropout
FC (26)
+ softmax activation
Probabilities / class

14. Appendix
VGG16 Architecture

15. Appendix
Convolutional neural networks, transfer learning & more
• http://setosa.io/ev/image-kernels/
• http://cs231n.github.io/
• http://cs231n.github.io/transfer-learning/