This document describes a handwriting recognition project that uses a convolutional neural network model to identify handwritten digits captured from a webcam. The model was trained on the MNIST dataset and uses TensorFlow and Keras for the deep learning components. It takes input from the webcam using OpenCV, processes the image, and returns the predicted digit as output. Some challenges included data cleaning, selecting model parameters, integrating the dynamic input method from the webcam, and image processing. The project provided learning opportunities around label encoding, activation functions, padding techniques, and combining computer vision and deep learning.

1. Handwriting Reader
Project Report
Abhinav Rawat

2. Acknowledgement
The project could not have been completed without the people who gave me support and guidance which
makes me honored to express my gratitude to all such people. I would like to thank Meenakshi
Devara, my mentor who gave the necessary tools and help to help me complete this project. I would
also like to thank my parents who encouraged me to pursue this field and gave me the motivation to
work hard

3. GitHub files link
Link - https://github.com/Abh-Raw/MNIST-Digit-Recognition

4. Index
● Problem Statement
● Model Abstract
● Requirements
● Constraints
● Data Organization/Cleaning
● Model
● Application
● Challenges Faced
● Learning Opportunities
● Video
● Summary

5. Problem Statement
With the advancement made in fields such as computer vision, it became necessary to accompany this
progress with equally advanced deep learning tools to solve complex problems and to evolve the field of
neural networks to integrate it with computer vision. This integration has potential to open new fields
from simple objectives such as recognizing handwriting to more complex activities such as self driving
cars

6. Model Abstract
The handwriting reader is a model based on the concepts of Deep Learning, more specifically
Convolutional Neural Networks. The model is trained using the mnist dataset for digits. The model has
been made utilising libraries from Tensorflow such as Keras and NumPy libraries. The user’s dynamic
input is taken implementing libraries of computer vision (cv2) and processed to send as an input the
model to give an output.

7. Requirements
● Python
● Tensorflow
● Cv2
● GPU(optional)
● Web Cam
● Blue colored small object (for eg a blue bottle cap)

8. Constraints
● Works only on digits from 0-9
● No blue or nearly blue colored objects in background
● Web cam with a clear feed

9. Data Organization and cleaning
The data has been taken from the mnist dataset, which includes 60000 images in 28 x 28 pixel format.
Before using the data to train the model, the data had to be processed. The labels had to be categorical to
enable softmax activation and the feature matrix was reshaped by adding another dimension which
handled the color channel. The arrays have to be in float and they are converted.

10. Model
After data processing, it is sent to the model. Due to the simple nature of the problem, Sequential mode
was used. The model had 2 convolution and pooling layers (same padding). Activation used for all layers
was relu except the output layer which is softmax. Used a 60 percent dropout to help in regularization to
prevent overfitting. The model was compiled with ‘adam’ optimizer and saved in a .h5 file format. After
that the training data is fit into the model and cross validated using test. Due to system constraints,
epochs used are just 2 with a batch size of 64. The model details such as loss and number of parameters
are printed in the end

11. Application
The model is loaded here. Outputs are stored in dictionary. Computer Vision library (cv2) is used to take
dynamic input from web cam using contours and the image is sent to be resized and processed to be sent
as an input to the model. Softmax activation enables the model to give the output with maximum
probability and that output is mapped to the dictionary for the final output

12. Video

13. Challenges Faced
● Data cleaning
● Model Parameters selection
● Dynamic input method
● Image processing

14. Learning Opportunities
● Learned more about label encoders
● Gained more knowledge on different types of activation functions and types of padding
● Learned about cv integration with deep learning
● Learned about new image processing techniques
● Learned more about contours and thresh
● Learned utilising web cam for direct input feed

15. Summary
Deep learning and computer vision is the future of IT industry. The future holds the solutions to many
limitations we may be facing in these fields right now and once those solutions are found, many more
discoveries will be made possible since the potential these fields hold are practically limitless. This basic
application can be developed into a much more sound application and used for purposes such as text-to-
speech, writing recognition and much more