phase- 1 Face Detection. Facial Landmark detection. phase- 2 Neural Network Training and Testing. validation and implementation. phase - 1 has been completed successfully.

1. Phase - 1
Seminar Presentation on
Emotion Detection of Babies’ Emotional Face using Deep Learning
Presented By-
Takrim Ul Islam Laskar
Roll No. 170302005
M.Tech 3rd Sem
Gauhati University Institute of Science And Technology
Dept. of Information Technology,Gauhati University
Under guidance of Dr. Parishmita Sarma
Assistant Professor, Dept. of IT,
Gauhati University.

2. Contents
• Introduction
• Computer Vision
• Face Detection
• Facial Landmark
• Machine Learning
• Artificial Neural Network
• Deep Learning
• Tools and Environment
• Proposed Methodology
• Result
• Conclusion and future work

3. Introduction.
• Emotion is a psychological and physiological state which is subject to
different conditions of mind. Broadly classifying happiness, sadness,
anger, disgust, surprise and fear are some of them.
• Human being is well capable of identifying emotions by observing the
actions of the subject. We are to build a system that makes a
computer capable to do the same.
• Pose, speech, facial expression, behaviour, etc convey emotions of
individuals but facial expression is the most precise to detect.

4. Computer Vision.
• Human beings are capable of seeing the world and objects with the
help of the eyes and act accordingly.
• The computers should be availed a system through which the
aforementioned task can be accomplished.
• Therefore, the system through which a computer undergoes a similar
task of seeing is referred to as computer vision.

5. Face Detection.
• To be able to identify the facial emotion, the fast and
foremost task is to locate the area in the image
where the face is available.
• There are many approaches to detect faces in a
digital image. Some of them are face detection with
controlled background, colour, motion, etc.[7]
Fig. Face Detection

6. Facial Landmark Detection
• Localization of Facial features like eyes, eyebrows, nose, mouth,
jawline can be achieved through Histogram of oriented gradients
(HOG), Haar wavelets, etc
• Applications of facial landmark detection[16]:
• Face recognition.
• Head pose estimation.
• Face morphing.
• Virtual makeup.
• Face replacement.

7. Machine Learning
• ML algorithms build a mathematical model of sample data known as
training data with the help of which it makes predictions and
decisions without explicitly running certain commands.
• More the size of samples, better is the performance and more is the
accuracy.
• Machine Learning uses two types of techniques: Supervised Learning
and Unsupervised Learning.

8. Supervised Learning[12] [17].
• Supervised Learning trains a model on known input and output data so that it
can predict future outputs.
• Supervised learning uses classication and regression techniques:
• Classification techniques predict discrete responses. Classification models
classify input data into categories. Typical applications include medical imaging,
speech recognition, etc.
• Regression techniques predict continuous responses like changes in
temperature or fluctuations in power demand. Typical applications include
electricity load forecasting and algorithmic trading.
Fig. Types of Machine Learning

9. Unsupervised Learning.
• Unsupervised Learning finds hidden patterns or intrinsic structures in data. It
is used to draw inferences from datasets consisting of input data without
labeled responses.
• Clustering is the most common unsupervised learning technique. It is used for
exploratory data analysis to find hidden patterns or groupings in data.
Applications for cluster analysis include gene sequence analysis, market
research, and object recognition
Fig. Clustering of patterns

10. Artificial Neural Network[14] [18]
• Artificial Neural Networks are computer systems designed based on
biological neural networks which constitute the animal brain.
• Biological Neurons
• A nerve cell (neuron) is a special biological cell that processes information.
• According to an estimation, there are huge number of neurons, approximately
1011
with numerous interconnections, approximately 1015
.

11. Working of Biological Neural network
• Dendrites- They are tree-like branches, responsible for receiving the
information from other neurons it is connected to.
• Soma- It is the cell body of the neuron and is responsible for processing of
information, they have received from dendrites.
• Axon- It is just like a cable through which neurons send the information.
• Synapses- It is the connection between the axon and other neuron
dendrites.
Fig. Biological Neural Network

12. Analogy of ANN and BNN
Biological Neural Network
(BNN)
Articial Neural Network
(ANN)
Dendrites Input
Soma Node
Axon Output
Synapse Weights or Interconnections
Fig. Analogy of ANN and BNN.

13. Model of Artificial Neural network
• The following diagram represents the general model of ANN followed by its
processing.
For the above general model of articial neural network, the net input can be calculated as follows-
Net Input, 𝑦𝑖𝑛 = 𝑥1. 𝑤1 + 𝑥2. 𝑤2 + 𝑥3. 𝑤3…… 𝑥 𝑚. 𝑤 𝑚
i.e., 𝑦𝑖𝑛 = 𝑖
𝑚
𝑥𝑖 𝑤𝑖
The output can be calculated by applying the activation function over the net input.
Y = F( 𝑦𝑖𝑛)

14. Deep Learning.[13]
• Deep Learning is a class of machine learning algorithms that use a cascade
of multiple layers of parallel processing units.
• Each successive layer uses the output from the previous layer as input.
• It learns in supervised or/and unsupervised manner.
Fig. Multiple layer NN in Deep Learning

15. Tools and Environment.
• Tools
• Python
• openCV
• Dlib
• Camera
• Dependencies
• Shape_predictor_68_face_landmarks.dat
• Imutils
• Environment
• Operating System- Ubuntu 16.04 (Linux based plateform)
• RAM- 4GB
• Processor- Intel core i5 7th Gen.

16. Proposed Methodology
• The proposed system enable a computer to detect the seven
universal emotions of human baby through detection of facial
expressions.
• The system may be useful in health care centres for automated
surveillance of psychological and physiological state of babies through
detection of facial expressions in real-time.

17. Proposed Methodology(cont.)
• Image Acquisition - The frame from the real-
time video would be the image.
• Face Detection - Dlib's frontal face detector
is used for detection of face in the image.
• Landmark Detection - Facial feature or
landmark is then extracted. These features
include ears, eyebrows, nose, mouth,
jawline.
• Neural Network - The data from the feature
points is then fed to the NN for classication
of emotions.
• After classication, the result of the
expression is then reflected to the image
using image tagging.
Fig. Block Diagram proposed methodology

18. Flow of the project
Fig. Flowchart of the project

19. Image Acquisition
The images have to be loaded first from the storage for training and testing the
Neural Network Classifiers.
To do so, openCV function cv2.imread() is used.
• Colour- cv2.imread(`image',1)
• Grayscale- cv2.imread(`image',0)
• Unchanged- cv2.imread(`image',-1)
To read image as CLI argument, function cv2.imread(args["image"]) is used.
To read image from directory:
directry=os.path.join(os.getcwd(), `dir')
cv2.imread(directory+'/'+image)
Fig. Image Acquisition.

20. Preprocessing.
After loading the image, pre-processing is done by resizing the image to 500
pixels and converting the resized image to grayscale.
• Resizing
• Scaling down the images allow faster processing.
• Moreover the images have to be of the same size to train a neural
network.
• To resize the images imutils function imutils.resize(image, width=500) is
used.
• Grayscale Conversion
• The openCV supports BGR colour model and dlib supports RGB colour
model. And grayscale color model is supported in both.
• Grayscale colour model is used for 2D image processing.
• Edges are better depicted in grayscale images.
• To convert the image to grayscale openCV function cv2.cvtColor(image,
cv2.COLOR _BGR2GRAY) is used.
Fig. Image Resizing.
Fig. Grayscale Conversion

21. Face Detection
• We used frontal_face_detector which comes with dlib
library.
• It is based on Histogram of Oriented Gradiants(HOG)
features and Support Vector Machine(SVM).
• The model is built out of 5 HOG filters which are front
looking, left looking, right looking, front looking but
rotated left and front looking but rotated right.
• The dataset used for training consisted of 2825
images, obtained from LFW dataset and manually
annotated by Davis King, the author of dlib.
Fig. Face detection.

22. Facial landmark extraction
• We have used the dlib's shape predictor and analysed
the dataset images with a pretrained file,
‘shape_predictor_68_face_landmarks.dat’ to get the
facial landmarks detected.
• An ensemble of regression trees is used to train the
file for estimation the facial landmark positions
through pixel intensities themselves.
• The facial landmark detector is used to estimate the
location of 68 (x , y)-coordinates that map to facial
structure on the face.
• The facial landmarks are thereby detected and are
ready to be fed to the the Convolutional Neural
Network.
Fig. Detection of Facial
Landmark.

23. Annotations based on 68 point iBUG 300-W
dataset.
Facial Landmark Points
Jaw 1-17
Right Eyebrow 18-22
Left Eyebrow 23-27
Nose 28-36
Right Eye 37-42
Left Eye 43-48
Mouth 49-68
Fig. Annotations based on 68 point iBUG 300-W dataset.

24. Training Neural network Classifier
• In the phase-2 of the project, a good amount of images of babies with
different emotions will be collected. These datasets would be used to
train Convolutional Neural Network(CNN) Classifiers.
• This would be done by feeding the coordinates of the landmarks
extracted from the images.
• Each emotion would generate particular mathematical instances after
training. These instances would be used to detect the emotions of
facial expression.
• 90% of the images would be used as training dataset.

25. Testing Neural network Classifier
• The trained Classifiers are then tested with a dataset of images which
were not used for training the classifiers.
• 10% of the images would be used as a testing dataset. The true
positive and false positive of the classification is then used to
calculate the accuracy of the System.

26. Result
• In the system, face detection and facial landmark detection have been
done successfully.
Fig. Face and landmark Detection on Image with Multiple Faces.

27. Conclusion and future work
• In this paper we have developed a system for facial landmark detection to detect
seven human emotions through detection of facial expressions.
• For that, we first detected the faces in the image using dlib's face detector. Then,
we extracted the 68 facial landmarks like eyes, eyebrow, nose, mouth, jawline
using dilb's shape predictor with the help of
shape_predictor_68_face_landmarks.dat file.
• In the next phase of the project the objective is to use a Convolutional Neural
Network to train classifiers to detect the emotions.
• The classifier is a statistical model which is created by the neural network based
on the image data used to train the NN classifier. The NN classifier would be then
tested with test dataset to calculate the accuracy of detection.
• The final output would be a system capable of detecting the emotions based on
facial expressions of babies.

28. References.
[1] D. R. Frischholz. (2018) Face detection algorithms techniques. [Online]. Available:
https://facedetection.com/algorithms/
[2] V. Gupta. (2018) Face detection { opencv, dlib and deep learning ( c++ / python ). [Online].
Available: https://www.learnopencv.com/face-detection-opencv-dlib-and-deep-learning-c-python/
[3] G. Hemalatha and C. Sumathi, A study of techniques for facial detection and expression classication,"
International Journal of Computer Science and Engineering Survey, vol. 5, no. 2, p. 27, 2014.
[4] D. E. King. (2018) Dlib c++ library. [Online]. Available: http://dlib.net/
[5] S. Mallick. (2015) Facial landmark detection. [Online]. Available: https://www.learnopencv.com/facial-
landmark-detection/
[6] openCV team. (2018) Opencv documentation. [Online]. Available: https://opencv.org/
[7] . M. Pantic, Facial expression recognition," Imperial College London, London, UK; University of
Twente, AE Enschede, The Netherlands, 2008.
[8] M. Pantic, L. Rothkrantz, and H. Koppelaar, Automation of non-verbal communication of facial
expressions," in Proc. Conf. Euromedia, 1998, pp. 86{93.

29. References.
[9] A. Pascu, B. A. Intelligence, and R. King, Facial expression recognition system," University of Manchester,
2015.
[10] T. V. Pham, M. Worring et al., Face detection methods, a critical evaluation," 2000.
[11] T. V. Pham, M. Worring, and A. W. Smeulders, Face detection by aggregated bayesian network classiers,"
Pattern Recognition Letters, vol. 23, no. 4, pp. 451{461, 2002.
[12] A. Savoiu and J. Wong, Recognizing facial expressions using deep learning."
[13] M. team. (2016) Machine learning. [Online]. Available: https://in.mathworks.com/discovery/machine-
learning.html
[14] T. team. (2015) Articial neural network - basic concepts. [Online]. Available:
https://www.tutorialspoint.com/articial neural network/articial neural network basic concepts.htm
[15] W. N. Widanagamaachchi, Facial emotion recognition with a neural network approach,“ University of
Colombo, 2009.
[16] Wikipedia. (2016) Articial neural network. [Online]. Available: https://en.wikipedia.org/wiki/Articial neural
network
[17] Wikipedia. (2016) Machine learning. [Online]. Available: https://en.wikipedia.org/wiki/Machine learning
[18] Wikipedia.. (2017) Deep learning. [Online]. Available: https://en.wikipedia.org/wiki/Deep learning

30. Thank You.