A Comparison of Human and Machine Learning-based Accuracy for Valence Classification of Subjects in Video Fragments. John Schavemaker, Yorick Holkamp

1. A Comparison of Human and
Machine Learning-based
Accuracy for Valence
Classification of Subjects in
Video Fragments
Yorick Holkamp, John Schavemaker

6. PLAY
WATCH
SENSEANNOTATE
RECOMMEND
MIME cycle

10. Machine vs. human?
• to determine the
performance of human
annotators
• humans have good
experience in assessing
the emotions of their
peers
• what are the practical
limitations of facial
expressions?

11. MAHNOB-HCI Dataset
• Soleymani, M., Lichtenauer, J.,
Pun, T., & Pantic, M. (2012). A
multimodal database for affect
recognition and implicit tagging.
Affective Computing, IEEE Trans.
on, 3(1), 42-55.
• created for affect recognition
and implicit tagging applications
• contains face video recordings,
EEG data and more
• 24 subjects
• Inter-rater agreement 0.66
(‘substantial agreement’)
setup
example

12. -1
0
1
valence

13. -1
0
1
Valence

14. Positive valence

15. Negative valence
Fragment
from
Hannibal
(censored)

16. Baseline Machine Method
Sensor data
Data
processing
Machine
learning
Performance
evaluation
Application
Overview

17. Machine Features
Time Action Unit
00:00 6. Raise cheeks
00:00 12. Pull lip corners
00:01 6. Raise cheeks
...
Noldus
FaceReader
Data aggregation
Combination # Starts # Stops
6+12 3 3
1+4+15 5 4
... ...
Facial expression-based method with onset and offset counting by Koelstra and Patras:
Koelstra, S., & Patras, I. (2013). Fusion of facial expressions and EEG for implicit affective
tagging. Image and Vision Computing, 31(2), 164-174.

18. Machine Training
Machine learning
Combination Onsets Offsets
06+12 3 3
1+4+15 5 4
... ...
Combination Onsets Offsets
06+12 3 3
1+4+15 5 4
... ...
Combination # Starts # Stops
6+12 3 3
1+4+15 5 4
... ...
Subject x’s history

19. Machine Classification
Machine learning
Combination Onsets Offsets
6+12 3 3
1+4+15 5 4
... ...

20. Our Extensions (FEI)
• Facial Expression Intensity
• Different aggregation methods
• Average activation level
• Standard deviation in activation level
• Alternative training method
• Train using 23 subjects, predict for 1

21. Annotation

22. Experiment

23. human
Human vs.
machine

24. Machine versus human annotator
accuracy
0%
50%
100%
Percentageofcorrectratings
Video fragments
Machine - human accuracy
Machine Human

25. Human accuracy per fragment
0%
25%
50%
75%
100%
Percentageofcorrectvotes
Video fragments
Human accuracy per fragment Positive Negative

26. conclusions
• In this paper we present the results of a comparison
between classification accuracy of humans and
machine-learning classifiers.
• For this we used the MAHNOB-HCI affective computing
dataset and we have reproduced and extended the
facial expression-based method by Koelstra and Patras.
• Our results show that both humans and machine
classifiers agree to a large portion on the appropriate
class for video fragments.
• In our experiments, we found that human annotators
obtained higher accuracy than the automatic
classification methods.

27. QUESTIONS?