This document discusses topics related to emotion and cognition in smart systems, including emotional AI, emotion and cognition in games, and using machine learning and wearable devices to model cognitive behavior and intelligence. It describes several case studies, including using Microsoft Kinect to predict cognitive behavior by observing a game of Head-Shoulders-Knees-Toes, and using wearable devices and social signal processing to provide feedback to public speakers.

1. Intelligent User
Interfaces
ICS2208
Dr Vanessa Camiller
i
Department of AI,
University of Malta

2. Topic 9: Emotion & Cognition
in Smart Systems
• Emotional AI
• Emotion & Cognition in Games
• Machine Learning for Cognitive Behaviour
Assessment
• Modelling Behaviour using Wearable Devices
2

3. Emotional AI
• Emotional AI is a term to refer to affective
computing techniques, machine learning and
Artificial Intelligence (AI)

5. FACS

6. Theories of Emotion
• Adaptation
• Dimensional
• Motivation
• Circuit
• Discrete
• Lexical
• Social constructionist

8. Discussion Time
• In which fields or domains would you see the
practicality of applying Emotional AI?

9. Discussion Time
• Would Emotional AI be necessary in education?
Why and why not?

10. Discussion Time
• What questions would you ask to address a
number of concerns in Emotional AI (for example
the validity of the training data, mental health in
users, opt out or opt in policies, or people
reactions?)

11. Social Dynamics in a Virtual
Game Environment

12. Virtual Agent Acceptability
Paradigm
• Efficiency
• Believability
• Social Acceptability
• a behaviour that fits into game
rules
• dynamic behaviour adjustments
• an environmental perception
that resembles the one of a
player
• personal characteristics that
allow to determine a particular
character
• a social interaction between
agents

13. Decision Making Systems
• Planned behaviour
• Finite State Machine
• Behaviour Trees
• Machine Learning Methods
• Utility AI

14. Machine Learning for
Cognitive Behaviour
• Cognitive impairments in early childhood may lead
to poor academic performance
• Research shows that a traditional game of Head-
Shoulders-Knees-Toes can provide psychometric
information leading to behavioural self-regulation
• Visual observation of HSKT can lead to predicting
cognitive behaviour

15. Method
• Use of Microsoft Kinect V2 Camera
• UI for recording and observing HSKT
• Machine learning techniques on pose estimation
from RGB video streams

16. Framework
• Deep Learning Architecture exploiting a
Convolutional Neural Network (CNN)

18. Interface

19. Experiment
• 15 participants (18-30 years as pilot test beds)
• 60,000 frames of RGB data collected, 4443 frames
annotated
• Dataset available: http://vlm1.uta.edu/˜srujana/
HTKS/CogniLearn_HTKS_Dataset.html

20. Modelling Intelligence using
Wearable Devices
• Social Signal Processing techniques used to
analyse human behaviour
• Training a computational model to provide
feedback to a public speaker about his co-verbal
communication
• Using wearable devices: smart watch, smart
phone, eye tracking device with microphone.

22. Social Intelligence Modelling
• Dynamic Bayesian Networks to model complex
temporal relationships between variables
• Machine learning techniques are used to associate
the cognitive state of the public speaker to the
annotated feedback

23. • Cognitive state influences multimodal behaviour
• Variables include: volume, intonation, speech, gaze fixations, hand gesture
energy, body energy
• Appropriate feedback is a direct consequence of multimodal scores of non-
verbal behaviour
• Appropriate feedback is influenced by the mental state of the user
• Temporal correlation between CS at a certain time, and the previous state of
the user

24. • For the case studies mentioned, think about the
following questions:
• How can we place the human at the centre of
every day's interaction and task activity?
• How can an interactive system adapt to human
cognitive and emotional factors with the aim to
deliver a personalised and more usable
interface?
• What models, architectures and frameworks, do
these case studies use? Discuss them