This presentation discusses developing personalized emotion models using associative classifiers. It begins with background on adaptive user interfaces and computational emotions. A new corpus was created with video recordings of people expressing basic emotions and affects. Initial analysis of Kinect data for happy expressions across subjects is shown. The approach explores generating individualized sets of rules linking facial action units using tree/rule classifiers like M5, with the goal of personalized emotion detection models.

1. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Cognitive Reasoning and Inferences through
Psychologically based Personalised Modelling of
Emotions Using Associative Classifiers
Aladdin Ayesh∗
Miguel Arevalillo-Herr´aez+
Francesc J. Ferri+
∗
De Montfort University
+
University of Valencia
Presentation given at ICCI*CC 2014
August 18, 2014
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

2. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Agenda
1 Context
Adaptive User Interfaces
Computational Emotions
2 Towards Dynamic Personalised Emotion Models
Corpus Development
Data Analysis
Personalized Model
3 Critical Review
Findings
Future Work
4 Conclusion
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

3. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
eLearning and Adaptive User Interfaces
eLearning includes:
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

4. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
eLearning and Adaptive User Interfaces
eLearning includes:
Managed Learning Environments (MLE)
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

5. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
eLearning and Adaptive User Interfaces
eLearning includes:
Managed Learning Environments (MLE)
Intelligent Tutoring Systems (ITS)
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

6. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
eLearning and Adaptive User Interfaces
eLearning includes:
Managed Learning Environments (MLE)
Intelligent Tutoring Systems (ITS)
Virtual Classrooms and Social Networking
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

7. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
eLearning and Adaptive User Interfaces
eLearning includes:
Managed Learning Environments (MLE)
Intelligent Tutoring Systems (ITS)
Virtual Classrooms and Social Networking
In the first two the prime interaction happens between human and
computer. Whilst other humans will facilitate the interaction in the
third case of virtual classrooms and social networking, the system
has to have emotional intelligence in the first two cases of MLE and
ITS.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

8. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
eLearning and Adaptive User Interfaces
eLearning includes:
Managed Learning Environments (MLE)
Intelligent Tutoring Systems (ITS)
Virtual Classrooms and Social Networking
In the first two the prime interaction happens between human and
computer. Whilst other humans will facilitate the interaction in the
third case of virtual classrooms and social networking, the system
has to have emotional intelligence in the first two cases of MLE and
ITS.
but why ...
and would it be possible to have emotionally intelligent systems?
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

9. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
Emotion Model
Modelling emotions is the first step in the growing field of computational
emotions research and enables us to develop further processes such as
emotion detection and expression.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

10. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
Emotion Model
Modelling emotions is the first step in the growing field of computational
emotions research and enables us to develop further processes such as
emotion detection and expression.
Preliminaries
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

11. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
Emotion Model
Modelling emotions is the first step in the growing field of computational
emotions research and enables us to develop further processes such as
emotion detection and expression.
Preliminaries
Most common emotion model is the Darwinian 6-basic emotions
Helped by Ekman FACS system (Ekman basic emotions)
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

12. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
Emotion Model
Modelling emotions is the first step in the growing field of computational
emotions research and enables us to develop further processes such as
emotion detection and expression.
Preliminaries
Most common emotion model is the Darwinian 6-basic emotions
Helped by Ekman FACS system (Ekman basic emotions)
FACS system novelty is in the standardisation of the different muscle
movements into recognisable Action Units (AU).
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

13. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
Emotion Model
Modelling emotions is the first step in the growing field of computational
emotions research and enables us to develop further processes such as
emotion detection and expression.
Preliminaries
Most common emotion model is the Darwinian 6-basic emotions
Helped by Ekman FACS system (Ekman basic emotions)
FACS system novelty is in the standardisation of the different muscle
movements into recognisable Action Units (AU).
FACS system AUs are more than what is necessary to produce an
automated system for facial expressions and emotion detection.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

14. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
Emotion Model
Modelling emotions is the first step in the growing field of computational
emotions research and enables us to develop further processes such as
emotion detection and expression.
Preliminaries
Most common emotion model is the Darwinian 6-basic emotions
Helped by Ekman FACS system (Ekman basic emotions)
FACS system novelty is in the standardisation of the different muscle
movements into recognisable Action Units (AU).
FACS system AUs are more than what is necessary to produce an
automated system for facial expressions and emotion detection.
Few researchers have already noticed this observation and attempted
to identify a small set of sufficient AUs.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

15. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
Emotion Model
Modelling emotions is the first step in the growing field of computational
emotions research and enables us to develop further processes such as
emotion detection and expression.
Preliminaries
Most common emotion model is the Darwinian 6-basic emotions
Helped by Ekman FACS system (Ekman basic emotions)
FACS system novelty is in the standardisation of the different muscle
movements into recognisable Action Units (AU).
FACS system AUs are more than what is necessary to produce an
automated system for facial expressions and emotion detection.
Few researchers have already noticed this observation and attempted
to identify a small set of sufficient AUs.
Also, facial expressions alone are not sufficient means to determine a
given user’s emotions.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

16. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
Emotion Detection: Kinect and FACS
Kinect SDK can detect to a high degree of accuracy a small set of 6
AUs only.
It uses CANDIDE model 1
to map the face, thus detects AUs, head
postures, etc.
1CANDIDE model itself implements a subset of FACS system AUs.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

17. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Adaptive User Interfaces
Computational Emotions
Emotion Detection: Kinect and FACS
Kinect SDK can detect to a high degree of accuracy a small set of 6
AUs only.
It uses CANDIDE model 1
to map the face, thus detects AUs, head
postures, etc.
Relating Kinect AUs to FACS AUs
FACS AUs Kinect AUs
AU10 AU0
AU26 AU1
AU20 AU2
AU4 AU3
AU15 AU4
AU2 AU5
1CANDIDE model itself implements a subset of FACS system AUs.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

18. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Why a new corpus?
Requirements
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

19. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Why a new corpus?
Requirements
We aim for a system that works with minimum constraints in
natural user environment.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

20. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Why a new corpus?
Requirements
We aim for a system that works with minimum constraints in
natural user environment.
The system success rate expected to be low but
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

21. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Why a new corpus?
Requirements
We aim for a system that works with minimum constraints in
natural user environment.
The system success rate expected to be low but
Sufficient for an intelligent adaptation of user’s interface.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

22. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Why a new corpus?
Requirements
We aim for a system that works with minimum constraints in
natural user environment.
The system success rate expected to be low but
Sufficient for an intelligent adaptation of user’s interface.
Developed Corpus consists of two parts:
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

23. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Why a new corpus?
Requirements
We aim for a system that works with minimum constraints in
natural user environment.
The system success rate expected to be low but
Sufficient for an intelligent adaptation of user’s interface.
Developed Corpus consists of two parts:
The first part includes a video recording of 5 volunteers expressing
the 6 basic emotions of Happy, Sad, Anger, Disgust, Surprise and
Fear.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

24. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Why a new corpus?
Requirements
We aim for a system that works with minimum constraints in
natural user environment.
The system success rate expected to be low but
Sufficient for an intelligent adaptation of user’s interface.
Developed Corpus consists of two parts:
The first part includes a video recording of 5 volunteers expressing
the 6 basic emotions of Happy, Sad, Anger, Disgust, Surprise and
Fear.
The second part was developed in similar manner for detecting
expressions of affects.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

25. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Initial Analysis of Kinect Data
Happy expression across 3 subjects
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

26. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Classifiers Approach
What if ...
we can generate a set of associative rules of AUs per individual, i.e. an
individualised expert system?
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

27. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Classifiers Approach
What if ...
we can generate a set of associative rules of AUs per individual, i.e. an
individualised expert system?
we explored some popular tree/rule classifiers.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

28. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Classifiers Approach
What if ...
we can generate a set of associative rules of AUs per individual, i.e. an
individualised expert system?
we explored some popular tree/rule classifiers.
they are mostly regression algorithms (evolving around M5 classifier)
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

29. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Classifiers Approach
What if ...
we can generate a set of associative rules of AUs per individual, i.e. an
individualised expert system?
we explored some popular tree/rule classifiers.
they are mostly regression algorithms (evolving around M5 classifier)
number of rules generated differ from algorithm to the next but ...
general trends are maintained
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

30. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Classifiers Approach
What if ...
we can generate a set of associative rules of AUs per individual, i.e. an
individualised expert system?
we explored some popular tree/rule classifiers.
they are mostly regression algorithms (evolving around M5 classifier)
number of rules generated differ from algorithm to the next but ...
general trends are maintained
concrete numbers limit the generalisation of the generated rules
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

31. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Generated Rules: Examples
Two rules generated for AU0 and AU5 from subject-1 Anger data
LMnum : 46 AU0 = 0.1579 ∗ AU1 − 0.053 ∗ AU2 − 0.115 ∗ AU3 −
0.306 ∗ AU4 − 0.3503 ∗ AU5 + 0.0975
LMnum : 47 AU0 = 0.2874 ∗ AU1 − 0.0072 ∗ AU2 − 0.1478 ∗ AU3 −
0.2311 ∗ AU4 − 0.0311 ∗ AU5 + 0.3731
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

32. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Corpus Development
Data Analysis
Personalized Model
Generated Rules: Examples
Two rules generated for AU0 and AU5 from subject-1 Anger data
LMnum : 46 AU0 = 0.1579 ∗ AU1 − 0.053 ∗ AU2 − 0.115 ∗ AU3 −
0.306 ∗ AU4 − 0.3503 ∗ AU5 + 0.0975
LMnum : 47 AU0 = 0.2874 ∗ AU1 − 0.0072 ∗ AU2 − 0.1478 ∗ AU3 −
0.2311 ∗ AU4 − 0.0311 ∗ AU5 + 0.3731
Examples of AU5 rules
LMnum : 40 AU5 = −0.3099 ∗ AU0 + 0.534 ∗ AU1 − 0.0305 ∗ AU2 −
0.2681 ∗ AU3 − 0.2012 ∗ AU4 − 0.0643
LMnum : 41 AU5 = −0.2827 ∗ AU0 + 0.4997 ∗ AU1 − 0.0314 ∗ AU2 −
0.2681 ∗ AU3 − 0.2012 ∗ AU4 − 0.0724
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

33. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Findings
Future Work
Findings
There is clear inter-relationships between AUs that could allow us to
determine the presence and intensity of a given AU from raw data.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

34. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Findings
Future Work
Findings
There is clear inter-relationships between AUs that could allow us to
determine the presence and intensity of a given AU from raw data.
AUs relationships are not reflexive in connection to emotions, i.e.
the contribution of AU5 in the rules of AU0 are not necessary of the
same value and significance as of AU0 in AU5 rules.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

35. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Findings
Future Work
Findings
There is clear inter-relationships between AUs that could allow us to
determine the presence and intensity of a given AU from raw data.
AUs relationships are not reflexive in connection to emotions, i.e.
the contribution of AU5 in the rules of AU0 are not necessary of the
same value and significance as of AU0 in AU5 rules.
Number of rules required for each AU to determine a given emotion
differs for each user and for each emotion.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

36. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Findings
Future Work
Findings
There is clear inter-relationships between AUs that could allow us to
determine the presence and intensity of a given AU from raw data.
AUs relationships are not reflexive in connection to emotions, i.e.
the contribution of AU5 in the rules of AU0 are not necessary of the
same value and significance as of AU0 in AU5 rules.
Number of rules required for each AU to determine a given emotion
differs for each user and for each emotion.
M5 and M5P are more likely candidates to enable us in generating a
personified rule-based system for emotion detection.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

37. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Findings
Future Work
Findings
There is clear inter-relationships between AUs that could allow us to
determine the presence and intensity of a given AU from raw data.
AUs relationships are not reflexive in connection to emotions, i.e.
the contribution of AU5 in the rules of AU0 are not necessary of the
same value and significance as of AU0 in AU5 rules.
Number of rules required for each AU to determine a given emotion
differs for each user and for each emotion.
M5 and M5P are more likely candidates to enable us in generating a
personified rule-based system for emotion detection.
The rules generated will require partial generalisation to reduce the
number of rules and widen applicability.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

38. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Findings
Future Work
Future plans
Explore classifier algorithms further and generalise the generated
rules
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

39. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Findings
Future Work
Future plans
Explore classifier algorithms further and generalise the generated
rules
Automate the process to dynamically generate personified set of
rules for emotion detection.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

40. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Findings
Future Work
Future plans
Explore classifier algorithms further and generalise the generated
rules
Automate the process to dynamically generate personified set of
rules for emotion detection.
Explore alternative rule representations, e.g. rules based on fuzzy
and multi-valued logic, which may also help in generalisation.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised

41. Context
Towards Dynamic Personalised Emotion Models
Critical Review
Conclusion
Conclusion
Dr. Aladdin Ayesh
aayesh@dmu.ac.uk – dr.aladdin.ayesh@ieee.org
www.aladdin-ayesh.info
Referencing paper
A. Ayesh, M. Arevalillo-Herr´aez, and F. J. Ferri, “Cognitive
reasoning and inferences through psychologically based personalised
modelling of emotions using associative classifiers,” in ICCI*CC 2014
Proceedings. London: IEEE, 2014, pp. 67–72.
Aladdin Ayesh∗ Miguel Arevalillo-Herr´aez+ Francesc J. Ferri+ Cognitive Reasoning and Inferences through Psychologically based Personalised