Face Recognition System Using Deep Convolutional Neural Networks

1. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Face Recognition System Using Deep
Convolutional Neural Networks
Supervised By -
Rizoan Toufiq.1
Assistant Professor.
Presented By -
Humayun Ahmed Ashik.2
Roll: 133068
1,2Department of Computer Science and Engineering
Rajshahi University of Engineering and Technology
Thesis Presentation, 2018
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 1 /31

2. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Outlines
Introduction
Background Study
Implementation
Result
Conclusion
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 2 /31

3. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Introduction
Face recognition is a biometric recognition process
passive
non-intrusive1
Face Recognition has drawn plenty of attention-
Biometrical verification
Search for a person through cameras
Face spoofing and anti-spoofing
...
1
AK Jain, A Ross, S Prabhakar, An introduction to biometric recognition IEEE Transactions on circuits and
systems for video technology 14 (1), 4-20, 2004
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 3 /31

4. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Introduction
Types of Biometrics-
Figure 1: Types of biometrics.2
2
Available at - https://www.elprocus.com/different-types-biometric-sensors/. Accessed: 16 november,2049
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 4 /31

5. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Problem Statement
Performance degrades because of -
low inter-subject distance
high intra-subject distance
Figure 2: (a)(b)(c) Extreme case of low inter-subject distance.(d) High
intra-subject distance
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 5 /31

6. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Constraints & Challenges
Performance degrades3 because of -
Facial Expressions
Figure 3: Variations of faces due to different facial expressions
3
M. Hassaballah, S. Aly, ”Face recognition: challenges achievements and future directions”, IET Computer
Vision, vol. 9, no. 4, pp. 614-626, 2015.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 6 /31

7. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Constraints & Challenges
Performance degrades3 because of -
Growing and Aging Process
Figure 4: Variations of faces due different ages
3
M. Hassaballah, S. Aly, ”Face recognition: challenges achievements and future directions”, IET Computer
Vision, vol. 9, no. 4, pp. 614-626, 2015.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 7 /31

8. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Constraints & Challenges
Performance degrades3 because of -
Occlusions
Figure 5: Occlusions given by different unusual objects
3
M. Hassaballah, S. Aly, ”Face recognition: challenges achievements and future directions”, IET Computer
Vision, vol. 9, no. 4, pp. 614-626, 2015.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 8 /31

9. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Constraints & Challenges
Performance degrades3 because of -
Viewing Geometry
Figure 6: Full set of different faces of the same subject with different pose
3
M. Hassaballah, S. Aly, ”Face recognition: challenges achievements and future directions”, IET Computer
Vision, vol. 9, no. 4, pp. 614-626, 2015.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 9 /31

10. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Constraints & Challenges
Performance degrades3 because of -
Lighting Conditions
Figure 7: Faces of the same subject under varying light conditions
3
M. Hassaballah, S. Aly, ”Face recognition: challenges achievements and future directions”, IET Computer
Vision, vol. 9, no. 4, pp. 614-626, 2015.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 10 /31

11. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Objective
To develop a robust face recognition system invariant of -
Pose
Illumination
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 11 /31

12. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Background
Two of the most successful applications of CNNs in the FR
problem
DeepFace4 and FaceNet5
These two have provided state-of-art results in recent years
Although there are other methods providing close results, we
decided to focus on CNN.
The reasons were not only result driven, but also interest
driven
4
Taigman, Yaniv et al. (2014). DeepFace: Closing the Gap to Human-Level Performance in Face Verification.
In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 14.
Washington, DC, USA: IEEE Computer Society, pp. 17011708. ISBN: 978-1-4799-5118- 5. DOI:
10.1109/CVPR.2014.220.
5
Schroff, Florian, Dmitry Kalenichenko, and James Philbin (2015). FaceNet: A Unified Embedding for Face
Recognition and Clustering. In: CoRR abs/1503.03832. URL: http://arxiv.org/abs/1503.03832
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 12 /31

13. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Proposed Approach
Figure 8: A generic diagram of proposed face recognition system .
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 13 /31

14. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Frontalization
Figure 9: Face frontalization process.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 14 /31

15. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Frontalization- Convert to YUV
RGB images to YUV color model6:


Y
U
V

 =


+0.257 +0.507 +0.098
−0.148 −0.291 +0.439
+0.439 −0.368 −0.071

 ∗


R
G
B

 +


16
128
128


Figure 10: Enhancement of input images.
6
Ahmed, E., Crystal, M., Dunxu H: Skin Detection-a short Tutorial, Encyclopedia of Biometrics, pp.12181224
,Springer-Verlag Berlin, Heidelberg,(2009).
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 15 /31

16. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Face Detection
Procedure
Using dlibs face detector7
based on HOG features with a linear classifier
and sliding window detection scheme.
Figure 11: Face detection using dlib’s HOG features
7
Available at: https://dlib.net/face detector.py.html.Accessed: November 15, 2018
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 16 /31

17. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Face frontalization
Procedure
Locate a set of 46 fiducial points
Consider the same points in a 3D pre-defined model
Generate a projection matrix to map from 2D input to the 3D
reference
(Apply vertical similarity to fill in empty spots)← Discarded
Figure 12: Face frontalization Gradients(HOG)8
.
8
Hassner, Tal et al. (2015). Effective Face Frontalization in Unconstrained Images. In: IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR).URL:http://www.openu.ac.il/home/hassner/projects/ frontalize
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 17 /31

18. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Feature Extraction
CNN structure:
Figure 13: Proposed CNN network architecture used in this face recognition
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 18 /31

19. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Feature Extraction
Table of CNN architecture:
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 19 /31

20. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Data and Training
Table Face databases - 1. LFW9
2. AT&T10.
9
LFW. Labeled faces in the wild. http://vis-www.cs.umass.edu/lfw/, 2016. Accessed: 2018-11-08. Cited on
pages 9 and 20.
10
AT&Tfaces.https://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html,2018. Accessed:
08.11.2018
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 20 /31

21. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Loss function
Cross entropy loss function is used to measure the network error -
L = 1
N
N
i Li
Li = − N
i yi log ˆyi
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 21 /31

22. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Outcome
Some of the true class and predicted class on LFW database:
Figure 14: Result that showing the true class vs predicted class
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 22 /31

23. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Outcome
5 classes are selected among 5749 classes that contain
minimum of 100 faces
Train on 1026 samples, validate on 114 samples
Figure 15: Average confusion matrix in the LFW dataset
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 23 /31

24. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Outcome
Figure 16: Classification report on LFW database
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 24 /31

25. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Outcome
Figure 17: Test accuracy of face recognition per epochs
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 25 /31

26. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Outcome
Figure 18: Loss vs no. of epochs
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 26 /31

27. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Outcome
Some of the true class and predicted class on AT&T database:
Figure 19: Result that showing the true class vs predicted class
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 27 /31

28. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Findings/Result
Result on LFW:
Train Accuracy: 97.45%
Test Accuracy: 92.11%
Result on AT & T :
Train Accuracy: 95.31%
Test Accuracy: 93%
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 28 /31

29. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Lackings & Future work
Shortcomings
Result is very low compared to the original DeepFaces’s work
which is 97.35%
Lack of computation power
In future-
Data augmentation will be performed for each subjects
Will try for a different classifier/ logistic regression
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 29 /31

30. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Conclusion
To conclude...
We have developed a functional Face Recognition System
using CNNs
Works in uncontrolled environment,
Compared with state of art methods, it underperforms
There is still room for improvement.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 30 /31

31. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
The End
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 31 /31

32. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Paper references at a glance
1. AK Jain, A Ross, S Prabhakar, An introduction to biometric recognition IEEE Transactions on circuits and
systems for video technology 14 (1), 4-20, 2004
2. M. Hassaballah, S. Aly, ”Face recognition: challenges achievements and future directions”, IET Computer
Vision, vol. 9, no. 4, pp. 614-626, 2015.
3. Taigman, Yaniv et al. (2014). DeepFace: Closing the Gap to Human-Level Performance in Face Verification. In:
Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 14. Washington,
DC, USA: IEEE Computer Society, pp. 17011708. ISBN: 978-1-4799-5118- 5. DOI: 10.1109/CVPR.2014.220.
4. Schroff, Florian, Dmitry Kalenichenko, and James Philbin (2015). FaceNet: A Unified Embedding for Face
Recognition and Clustering. In: CoRR abs/1503.03832. URL: http://arxiv.org/abs/1503.03832.
5. Ahmed, E., Crystal, M., Dunxu H: Skin Detection-a short Tutorial, Encyclopedia of Biometrics, pp.12181224
,Springer-Verlag Berlin, Heidelberg,(2009).
6. Hassner, Tal et al. (2015). Effective Face Frontalization in Unconstrained Images. In: IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR).URL:http://www.openu.ac.il/home/hassner/projects/ frontalize
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 32 /31

33. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - HOG
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 33 /31

34. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - HOG
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 34 /31

35. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - HOG
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 35 /31

36. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - HOG
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 36 /31

37. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - CNN
Why CNN?
Dealing with a 1 megapixel image, the total number of
features in that picture is 3 million (=1,000 x 1,000 x 3 color
channels).
Passing this through a neural network with just 1,000 hidden
units and weights of 3 billion parameters. Too big to be
managed !
perfect solution: Convolutional neural networks (ConvNets).
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 37 /31

38. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - CNN
Terms associated with CNN -
Kernel size: the width and height of the receptive field for the
neurons in that stack.
Stride: Each neuron processes a region of the input space. As
these regions can overlap, the stride indicates the distance
between their centers.
Padding: In some cases, the neurons in the borders of the
layer cannot process a whole receptive field. This may happen
due to the stride.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 38 /31

39. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - CNN
Layers of CNN -
Convolution(CONV)
Pooling(POOL)
Fully connected(FC)
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 39 /31

40. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - CNN
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 40 /31

41. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - CNN
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 41 /31

42. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - CNN
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 42 /31

43. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - CNN
What is a gradient descent?
Gradient descent is an optimization algorithm used in machine
learning to learn values of parameters that minimize the cost
function.
Its an iterative algorithm, in every iteration, we compute the
gradient of the cost function with respect to each parameter
and update the parameters of the function
Padding: In some cases, the neurons in the borders of the
layer cannot process a whole receptive field. This may happen
due to the stride.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 43 /31

44. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - CNN
gradient descent: batch, stochastic and mini-batch
Stochastic Gradient Descent - Uses only single training
example to calculate the gradient and update parameters.
Batch Gradient Descent - Calculate the gradients for the
whole dataset and perform just one update at each iteration.
Mini-batch Gradient Descent - Mini-batch gradient is a
variation of stochastic gradient descent where instead of
single training example, mini-batch of samples is used. Its one
of the most popular optimization algorithms.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 44 /31

45. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - CNN
What is the role of the activation function?
The goal of an activation function is to introduce non-linearity
into the neural network so that it can learn more complex
function.
Without it, the neural network would be only able to learn
function which is a linear combination of its input data.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 45 /31

46. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - CNN
Hyperparameters -
Learning rate - It determines how fast we want to update the
weights during optimization.
Number of epochs - Epoch is defined as one forward pass and
one backward pass of all training data.
Batch size - The number of training examples in one
forward/backward pass.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 46 /31

47. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - CNN
What is a dropout?
Dropout is a regularization technique for reducing overfitting
in neural networks.
At each training step we randomly drop out (set to zero) set
of nodes,
Thus we create a different model for each training case, all of
these models share weights. Its a form of model averaging.
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 47 /31

48. Introduction
Implementation
Result & Discussion
RUET, Rajshahi, Bangladesh
Notes - Evaluation
133068 17/11/2018 Face Recognition System Using Deep Convolutional Neural Networks 48 /31