The thesis discusses detecting facial expressions and key facial coordinates using deep learning techniques. The objectives are to detect key facial coordinates and recognize seven universal facial expressions with high accuracy. The methodology uses the FER2013 dataset and a key facial points dataset, applying preprocessing like resizing and data augmentation. A residual network model is trained on this data, achieving accuracy scores that outperform previous state-of-the-art models in recognizing both facial expressions and key coordinates.

1. MS .Thesis Defense
Facial expressions and key facial coordinates detection using deep learning
techniques
Syed Aienullah Agha (MS IT)
Supervisor: Dr. Syed Attique Shah (FICT)
Co-Supervisor: Dr. Mehmood biryalai (FICT)
Department of Information Technology, Balochistan
University of Information Technology, Engineering, and
Management Sciences (BUITEMS)

2. Contents
2
1. Introduction
2. Objectives
3. Litrature Review
4. Methodology
5. Results
6. Conclusion
7. References

3. Acknowledgments
3
Explicitly, I am greatly thankful to all the concerned souls who have been
provided me inclusive guidelines and productive suggestions in order to
utter my research. From the core of my heart, I impart special thanks to
my decisive supervisors Dr Attique Shah and Dr Mehmood Baryalai. No
doubt, my research journey could not be possible to reach its peak without
their special assistance. Apart from others, such experiences with them
have educated me and will also enlighten my soul ahead.

4. Facial Expressions
• Facial expressions are the non verbal
cues which we usually use in our
interpersonal communication.
• Many researchers believes that 93%
of communication occurs through non
verbal cues and only 7% of
communication take place through the
use of words.
• There are 7 universal facial
expressions that are common across
all the cultures.
4
Figure 1: Seven universal facial expressions

5. Facial key coordinates
• Facial key coordinates are the special points
in a facial structure which that maps the
expression of an emotion.
• It is a critical step in face identification and is
described as the process of finding certain
areas, points and landmarks.
5
Figure 2: Facial key coordinates.

6. Facial expression recognition
6
• Facial expression recognition (FER) is a research field that aimed to identify human emotions based
on facial expressions.
• It can be utilized in biometric authentication, interactive human-computer interaction, robotics, and
clinical care to treat schizophrenia, anxiety, stress, and psychological issues.

7. Problem Statement
7
• It has often been said that the eyes are the "window to the soul." This statement
may be carried to a logical assumption that not only the eyes but the entire face
may reflect the "hidden" emotions of the individual.
• As we better know that conceiving a human's expression is easy for us. Certainly,
it is still a difficult task for a computer to recognize human emotions accurately.
• Over the last few years, various state-of-the-art machine learning-based facial
expression models proposed by different researchers. However, traditional
machine learning models are brittle and have a low recognition rate due to lack of
image information and noise intrusion.

8. Objectives
9
• Detection of key feature coordinates. These are coordinated points in a facial structure that maps
the expression of an emotion.
• Recognition of facial expressions using the proposed model, e.g., happiness, sadness, anger,
surprise, disgust, stoic, fear with a remarkable accuracy score.

9. Literature Review
10
Studies Dataset Classifier Accuracy
Giannopoulos et al. [1] FER2013 ALEXNET 73%
Jain et al. [2] JAFFE CNN 80%
Zhenhua. [3] JAFFE CNN 64%
Kuan Li. [4] CK+ &
JAFFE
CNN 80%
Moises Garcia. [5] JAFFE DNN 80%
Nessrine Abbassi. [6] FER2013 VGG19 72%
Shekhar Singh et al. [7] FER2013 CNN 75.2%

10. Methodology
11
Dataset Availability:
FER2103 and key facial points dataset which has been adopted for the
experiment in this research work, consisting of 28,709 training images,
3,589 test images, and 3,589 validation images were collected from
wolfram.com using Google search. The size of all images is 48*48, and
there are seven categories of emotions present in the FER2013 dataset.
 Moreover, for facial key coordinates detection we used the key
facial points detection dataset. The dataset consists of x-coordinates
and y-coordinates of 15 facial key points of 2140 images (96 X 96)
in a grayscale.
Figure 10: Dataset Images

11. Dataset distribution
Graphical representation of dataset segmentation
12

12. Pre-processing
14
Figure 11: Augmented Pictures

13. Architecture
16
Figure 12: Proposed model architecture

14. Results
Figure 14: Accuracy comparison after augmentation
Figure 13: Accuracy comparison before augmentation

15. 18

16. Loss comparison
19
Figure 15: Loss comparison before augmentation Figure 16: Loss comparison after augmentation

17. 20

18. Confusion Matrix
21
Figure 19: Confusion matrix of facial expressions Figure 20: Confusion matrix of key facial points

19. Classification Report
22

20. Model’s Prediction
24
Figure 21: Proposed model prediction

21. 25

22. Conclusion
26
• This thesis discussed how to recognize facial expressions and key coordinates using the
residual network model.
• For feature extraction, we have used a state-of-the-art pre-trained ResNet model. By which
we can save computational resources as its already trained weights don't need to train again.
• Pre-processing techniques were also helpful in this regard where we first re shape our
images from 48x48 to 96x96 pixels and data augmentation process provides us more variety
for all the images.
• the proposed model's result showed us that the Residual networks perform tremendously on
huge image datasets and can attain good accuracy scores on different computer vision tasks.
however, the proposed model recognized facial expression and key coordinates with
considerably high accuracy and outperformed previously proposed, state-of-the-art models.

23. References
• [1] Panagiotis Giannopoulos, Isidoros Perikos, and Ioannis Hatzilygeroudis. Deep learn-ing approaches for
facial emotion recognition: A case study on fer-2013. InAdvancesin hybridization of intelligent methods,
pages 1–16. Springer, 2018.
• [2] Deepak Kumar Jain, Pourya Shamsolmoali, and Paramjit Sehdev. Extended deepneural network for facial
emotion recognition.Pattern Recognition Letters, 120:69–74, 2019.
• [3] Zhenhua Nie. Research on facial expression recognition of robot based on cnn con-volution neural
network. In2020 IEEE international conference on power, intelligentcomputing and systems (ICPICS), pages
1067–1070. IEEE, 2020.
• [4] Kuan Li, Yi Jin, Muhammad Waqar Akram, Ruize Han, and Jiongwei Chen. Facialexpression recognition
with convolutional neural networks via a new face croppingand rotation strategy.The visual computer,
36(2):391–404, 2020.
• [5] Moises Garcia Villanueva and Salvador Ram ́ırez Zavala. Deep neural network archi-tecture: Application
for facial expression recognition.IEEE Latin America Transac-tions, 18(07):1311–1319, 2020.
• [6] Nessrine Abbassi, Rabie Helaly, Mohamed Ali Hajjaji, and Abdellatif Mtibaa. A deeplearning facial
emotion classification system: a vggnet-19 based approach. In202020th International Conference on Sciences
and Techniques of Automatic Control and Computer Engineering (STA), pages 271–276. IEEE, 2020.
27

24. Thank You