This document summarizes research on developing an AI to detect facial emotions in images. The researchers tested several models including KNN, logistic regression, decision trees, neural networks, CNNs, and transfer learning. Their most accurate models were a pre-trained VGG model using transfer learning at 68.2% accuracy and their own trained CNN model at 68.0% accuracy. Potential applications of the emotion detection AI include helping children with autism and improving online learning environments. If given more time, the researchers would develop a live camera feature for the AI.

1. AI Facial Emotion
Detection
By: Ellie Pierson, Richard Thomas, Emily Joseph,
Renold Thomas, Luca Netter, Bergen Cloninger
Instructor: Laikh Tewari

2. What we are doing?
● Creating an AI to read emotions
○ Helps kids with autism
○ Driving assistance/safety
○ Help teachers in online teaching

3. Can you label the images below using these 5
emotions?
● Happy
● Sad
● Surprised
● Angry
● Neutral
1 2 3
4 5 6

4. Can you label the images below using these 5
emotions?
● Happy
● Sad
● Surprised
● Angry
● Neutral
✓
1 2 3
4 5 6
Happy Angry Surprised
Sad Angry Neutral

5. How can we use AI to identify these
emotions?

6. Setup and Approach
● Use the best model that predicts
emotions most accurately
Landmarks and distances
Models
Evaluating
Results!
1 2
3
4
● Calculate distances between
each facial landmark
● Find landmarks on different face
images
● Training models using data
consisting of faces and labels (knn,
lr, dt, neural networks)
● Evaluate model performance using
confusion matrix, epoch/accuracy
graphs, and accuracy scores

7. Model Input Data
EXTRACTED FEATURES
Each image has 64 facial landmarks
The # of distances between landmarks:
64+63+62+61… + 1 = 2278
Then we condense those 2278 distances into
just 20 features!
INPUT:
20 distance values for each
OR
RAW PIXELS
Each image in the data set is 48*48 pixels (in
grayscale)
48*48 = 2304
INPUT:
2,304 pixels for each image

8. KNN example
● We used a KNN model (K nearest neighbors) to establish a baseline
accuracy
● A KNN model is a type of classification where an algorithm tries to
predict what a new data point will look like based off of existing data
● KNN classification tends to be fast but inaccurate
● With this model, the accuracy comes out to be 43.55%

9. Log Reg Example
● Logistic regression is a model
that classifies based off of a
probability.
● Logistic regression creates a
curve shaped like an S.
● An ‘S-Curve’ cannot extend the
class number above or below 0-1.
● With this model, the accuracy
comes out to be 36.8%

10. Neural Network
● A Neural Network is a model based off of
how the human brain works, and uses
input, hidden, and output layers.
● Each layer consists of some amount of
neurons, which are used to test and train
information by sending the information
through every possible path between the
layers. It filters the information which in
this case is images, and attempts to
correctly predict the emotion
● With this model, the accuracy comes out
to be 51.7%, which is almost as good as a
human, but just shy.

11. Neural Network Example
● Here is an example of a
Neural Network:
● Ours looks similar to
this, but with many
more neurons in each
layer.

12. CNN Example
● Takes an image
● Filters through a set of layers
● Predicts a label
Through different variables:
Our best model showed: Validation Accuracy = 68% , Loss = 0.85

13. Transfer Learning
● CNN Data Set →Model →Test
Result
● Val Acc: 68.2%
● Better than most tested previous
models

14. Eye + Eyebrow Model
We wanted to see how our
models would do if we were
only given landmarks and
distances above the nose
as if we had a mask on.
Our models will only receive
inputs of eyes and
eyebrows landmarks.
Can you tell what
emotion this is?
Happy!
K Nearest Neighbors:
38.35%
Logistic Regression:
38.15%
Decision Tree:
34.35%
Neural Network:
42.65%

15. Comparison Table
The most accurate model was a pre
trained VGG model using Transfer
Learning at 68.2% accuracy. Our trained
CNN model was close behind at 68.0%
accuracy.

16. Future Applications
Emotion Detection could be used for:
● Help kids with autism
● In learning environments (i.e. Zoom,
Google Meet)

17. If we had 1 more week
● Create a live camera feature

18. Thank You