2. Topic 9: Emotion & Cognition
in Smart Systems
• Emotional AI
• Emotion & Cognition in Games
• Machine Learning for Cognitive Behaviour
Assessment
• Modelling Behaviour using Wearable Devices
2
3. Emotional AI
• Emotional AI is a term to refer to affective
computing techniques, machine learning and
Artificial Intelligence (AI)
10. Discussion Time
• What questions would you ask to address a
number of concerns in Emotional AI (for example
the validity of the training data, mental health in
users, opt out or opt in policies, or people
reactions?)
12. Virtual Agent Acceptability
Paradigm
• Efficiency
• Believability
• Social Acceptability
• a behaviour that fits into game
rules
• dynamic behaviour adjustments
• an environmental perception
that resembles the one of a
player
• personal characteristics that
allow to determine a particular
character
• a social interaction between
agents
13. Decision Making Systems
• Planned behaviour
• Finite State Machine
• Behaviour Trees
• Machine Learning Methods
• Utility AI
14. Machine Learning for
Cognitive Behaviour
• Cognitive impairments in early childhood may lead
to poor academic performance
• Research shows that a traditional game of Head-
Shoulders-Knees-Toes can provide psychometric
information leading to behavioural self-regulation
• Visual observation of HSKT can lead to predicting
cognitive behaviour
15. Method
• Use of Microsoft Kinect V2 Camera
• UI for recording and observing HSKT
• Machine learning techniques on pose estimation
from RGB video streams
19. Experiment
• 15 participants (18-30 years as pilot test beds)
• 60,000 frames of RGB data collected, 4443 frames
annotated
• Dataset available: http://vlm1.uta.edu/˜srujana/
HTKS/CogniLearn_HTKS_Dataset.html
20. Modelling Intelligence using
Wearable Devices
• Social Signal Processing techniques used to
analyse human behaviour
• Training a computational model to provide
feedback to a public speaker about his co-verbal
communication
• Using wearable devices: smart watch, smart
phone, eye tracking device with microphone.
21.
22. Social Intelligence Modelling
• Dynamic Bayesian Networks to model complex
temporal relationships between variables
• Machine learning techniques are used to associate
the cognitive state of the public speaker to the
annotated feedback
23. • Cognitive state influences multimodal behaviour
• Variables include: volume, intonation, speech, gaze fixations, hand gesture
energy, body energy
• Appropriate feedback is a direct consequence of multimodal scores of non-
verbal behaviour
• Appropriate feedback is influenced by the mental state of the user
• Temporal correlation between CS at a certain time, and the previous state of
the user
24. • For the case studies mentioned, think about the
following questions:
• How can we place the human at the centre of
every day's interaction and task activity?
• How can an interactive system adapt to human
cognitive and emotional factors with the aim to
deliver a personalised and more usable
interface?
• What models, architectures and frameworks, do
these case studies use? Discuss them