Affective User Modeling to Improve Human-Computer Interaction Cristina Conati Department of Computer Science University of British Columbia

Research Context (1) Adaptive User Interfaces (AUI) AUI Fascinating interdisciplinary field aiming to Create user interfaces that can better support individual users By autonomously and intelligently adapting to their specific needs Cognitive Science HCI AI

Adaptive User Interfaces (AUI)

Fascinating interdisciplinary field aiming to

Create user interfaces that can better support individual users

By autonomously and intelligently adapting to their specific needs

Research Context (2) User Modeling : how to efficiently infer, represent and reason about non-trivial user features relevant for adaptivity . User Model Adaptation Inference Representation Inference

User Modeling : how to efficiently infer, represent and reason about non-trivial user features relevant for adaptivity .

Long-term Research Goal Extend the range of features in a user model: from cognitive elements to meta-cognitive abilities and affective states . Affective States Emotions Moods Motivation… From Adaptation Cognitive Elements User Model Meta Cognitive Abilities Learning from examples Reasoning by analogy Self-monitoring… Cognitive elements Knowledge Goals, Beliefs… User Model

Extend the range of features in a user model: from cognitive elements to meta-cognitive abilities and affective states .

Affective States

Emotions

Moods

Motivation…

Meta Cognitive Abilities

Learning from examples

Reasoning by analogy

Self-monitoring…

Cognitive

elements

Knowledge

Goals,

Beliefs…

Challenge Limited information bandwidth (amount and quality of information available to build the model) It can be difficult to unobtrusively capture the relevant traits from simple interaction events High level of uncertainty

Limited information bandwidth (amount and quality of

information available to build the model)

It can be difficult to unobtrusively capture the relevant traits from simple interaction events

High level of uncertainty

How to Assess a User’s Emotions? Emotions can be assessed by Reasoning about possible causes (i.e. the interface keeps interrupting the user, so she is probably frustrated) Looking at the one’s reactions (i.e. the user punches the screen, so she is probably frustrated) However, the mapping between emotions, their causes and their effects can be highly ambiguous Very hard to build models of user affect

Emotions can be assessed by

Reasoning about possible causes (i.e. the interface keeps interrupting the user, so she is probably frustrated)

Looking at the one’s reactions (i.e. the user punches the screen, so she is probably frustrated)

However, the mapping between emotions, their causes and their effects can be highly ambiguous

Very hard to build models of user affect

Why do We Care? Assumption: understanding user affect may enable an interface to better meet the user’s needs Especially in emotionally-charged context such as E-health E-games Computer-based education We have been working on affective user modeling for an educational computer game

Assumption: understanding user affect may enable an interface to better meet the user’s needs

Especially in emotionally-charged context such as

E-health

E-games

Computer-based education

We have been working on affective user modeling for an educational computer game

Outline Educational Computer Games The Prime Climb Game An Affective Student Model for Prime Climb Future work and Conclusions

Educational Computer Games

The Prime Climb Game

An Affective Student Model for Prime Climb

Future work and Conclusions

Educational Games Educational systems designed to teach via game-like activities Pros: generate high level of emotional engagement and motivation. Cons: Often possible to play the game without understanding the underlying knowledge Suitable only for certain types of learners

Educational systems designed to teach via game-like activities

Pros: generate high level of emotional engagement and motivation.

Cons:

Often possible to play the game without understanding the underlying knowledge

Suitable only for certain types of learners

Example: The Prime Climb Educational Game Designed by EGEMS group at UBC to teach number factorization to students in 6 th and 7 th grade (11 and 12 year old)

Designed by EGEMS group at UBC to teach number

factorization to students in 6 th and 7 th grade (11 and 12 year old)

Our Solution Emotionally Intelligent Pedagogical Agents that Monitor how students learn from a game Generate tailored interventions to trigger constructive reasoning… … while maintaining a high level of student emotional engagement Crucial to model student affect in addition to learning

Emotionally Intelligent Pedagogical Agents that

Monitor how students learn from a game

Generate tailored interventions to trigger constructive reasoning…

… while maintaining a high level of student emotional engagement

The Prime Climb Pedagogical Agent Provides hints to help students learn from the game Hints based on A model of student learning (Manske and Conati 2005) - for now AND a model of student affect – in the future

Provides hints to help students learn from the game

Hints based on

A model of student learning (Manske and Conati 2005) - for now

AND a model of student affect – in the future

Challenge Difficulty of modeling affect enhanced by the fact that players often experience Multiple emotions Possibly overlapping Rapidly changing For instance: Happy with a successful move but upset with the agent who tells them to reflect about it Ashamed immediately after because of a bad fall

Difficulty of modeling affect enhanced by the fact that players often experience

Multiple emotions

Possibly overlapping

Rapidly changing

For instance:

Happy with a successful move but upset with the agent who tells them to reflect about it

Ashamed immediately after because of a bad fall

Previous Approaches Reduce the uncertainty in modeling affect by Modeling one relevant emotion in a restricted situation [e.g., Healy and Picard, 2000; Hudlicka and McNeese, 2002] Modeling only intensity and valence of emotional arousal [e.g., Ball and Breeze, 2000] Not ideal for precise affective interventions in the complex emotional context triggered by edu-games

Reduce the uncertainty in modeling affect by

Modeling one relevant emotion in a restricted situation [e.g., Healy and Picard, 2000; Hudlicka and McNeese, 2002]

Modeling only intensity and valence of emotional arousal [e.g., Ball and Breeze, 2000]

Not ideal for precise affective interventions in the complex emotional context triggered by edu-games

Our solution Handle the inherent uncertainty in modeling via formal methods for probabilistic reasoning: Bayesian networks and their extensions Integrate information on both causes and effects of emotional reaction Based model design on existing, well-established theories from emotional psychology

Handle the inherent uncertainty in modeling via formal methods for probabilistic reasoning: Bayesian networks and their extensions

Integrate information on both causes and effects of emotional reaction

Based model design on existing, well-established theories from emotional psychology

The Prime Climb Affective Model Player Reactions Predictive Assessment Emotional State Game-based Causes Based on the OCC Theory of Emotions (Ortony Clore and Collins, 1998) Diagnistic Assessment

OCC Theory action OUTCOME Goals e.g., Have Fun Avoid Falling Defines 22 different emotions are the result of evaluating ( appraising ) the current circumstances with respect of one’s goals Joy/Distress Admiration/Reproach Pride/Shame Joy/Distress action OUTCOME Goals

The Prime Climb Affective Model Player Reactions Predictive Assessment Emotional State Game-based Causes Infers player goals at runtime (e.g., Have Fun, Learn Math, Avoid Falling…) Has information to assess which game states satisfy/dissatisfy the goals Diagnistic Assessment 6 of the 22 emotions in the OCC theory Joy/Regret toward the game Admiration/Reproach toward the agent Pride/Shame toward oneself

Infers player goals at runtime (e.g., Have Fun, Learn Math, Avoid Falling…)

Has information to assess which game states satisfy/dissatisfy the goals

6 of the 22 emotions in the OCC theory

Joy/Regret toward the game

Admiration/Reproach toward the agent

Pride/Shame toward oneself

Diagnistic Assessment We use an Electromiogram (EMG) sensor on the forehead to detects activity in the corrugator muscle Previous studies (e.g., Cacioppo 1993) show that greater muscle activity is a reliable indicator of negative affect reduced activity is an indicator of positive affect

We use an Electromiogram (EMG) sensor on the forehead to detects activity in the corrugator muscle

Previous studies (e.g., Cacioppo 1993) show that

greater muscle activity is a reliable indicator of negative affect

reduced activity is an indicator of positive affect

The Prime Climb Affective Model Player Reactions Predictive Assessment Emotional State Game-based Causes Infers player goals at runtime (e.g., Have Fun, Learn Math, Avoid Falling…) Has information to assess which game states satisfy/dissatisfy the goals Diagnistic Assessment Includes 6 of the 22 emotions in the OCC theory Joy/Regret toward the game Admiration/Reproach toward the agent Pride/Shame toward oneself EMG to detect corrugator muscle activity Helps the model understand if the player is feeling a positive or negative activity

Infers player goals at runtime (e.g., Have Fun, Learn Math, Avoid Falling…)

Has information to assess which game states satisfy/dissatisfy the goals

Includes 6 of the 22 emotions in the OCC theory

Joy/Regret toward the game

Admiration/Reproach toward the agent

Pride/Shame toward oneself

EMG to detect corrugator muscle activity

Helps the model understand if the player is feeling a positive or negative activity

Very Encouraging Results

Lots of Exciting Future Work Add more diagnostic elements to improve model accuracy (e.g., more expression recognition, speech/intonation patterns) Integrate model of affect and model of learning Create emotionally intelligent agent that takes into account both student affect and learning to decide how to act Prove that it works better than agent with no affect!

Add more diagnostic elements to improve model accuracy (e.g., more expression recognition, speech/intonation patterns)

Integrate model of affect and model of learning

Create emotionally intelligent agent that takes into account both student affect and learning to decide how to act

Prove that it works better than agent with no affect!

Conclusions Affective Computing has great potential to improve Human Computer interaction Exciting multi-disciplinary field that brings computer science together with disciplines traditionally more appealing to women (Cognitive Science, Psychology) Not sure if this means that women may be privileged in the developments and use of the next generation ICTs But I have been privileged to be working in this field and I really hope that more people will join

Affective Computing has great potential to improve Human Computer interaction

Exciting multi-disciplinary field that brings computer science together with disciplines traditionally more appealing to women (Cognitive Science, Psychology)

Not sure if this means that women may be privileged in the developments and use of the next generation ICTs

But I have been privileged to be working in this field and I really hope that more people will join