V Jornadas eMadrid sobre “Educación Digital”. Jesús G. Boticario, Universidad Nacional de Educación a Distancia: Sistemas recomendadores afectivos basados en modelado de usuario. 2015-06-30

1. aDeNu-UNED 2015 ©
Jesus G. Boticario
jgb@dia.uned.es
aDeNu Research Group
UNED

2. USER MODELING (LEARNING)
Student Centered
AI-ED / ML / EDM /
LAK / ITS / CSCL /
L@S… ??

3. Context & Motivation
Different User Agents
Varied AssistiveTechnologies
Multiple contents format
Negotiation requirements
¿ ?
¿ ?

4. Context & Motivation
Observable features and AMI
RFID tracking
Movements tracking
Face tracking

5. Context & Motivation
Standards support transferability and interoperability

6. Research Questions
Standards usage
& extension
Adaptive interaction support (run-time affect oriented)
Student’s centered learning (UCD affect oriented)
ADAPTIVE &
AFFECTIVE &
INCLUSIVE &
LEARNING
SYSTEMS
Affect detection: Accurately detect
meaningful features in LEARNING
New inputs & outputs:
modality fusion
Educational Issues
Pedagogical interventions
New learning opportunities
Transferability
Domain independent
features
Transferrable processes

7. 2. aDeNu Background

8. http://adenu.ia.uned.es/
adenu@ia.uned.es

9. Accessibility Guidelines &
Standards
W3CWeb Accessibility Initiative: WCAG
(contents), ATAG (authoring tools),
UAAG (user agents)
ISO 24751, IMS-LIP, IMS-AfA, IMS-MD,
IMS-CP, IMS-MD, IEEE-LOM, IMS-
RDCEO, ISO/IECTR 24763, IMS-QTI,
SCORM,W3C CC/PP
User Modeling
Individual & CSCL (needs, preferences,
learning styles, CSCL features...)
Monitoring dynamic information from
interactions
aDeNu projects

10. Standards-based Modeling
<imsld:locpersproperty identifier="resourcetest4.score">
<imsld:datatype datatype="integer" />
<imsld:initialvalue>0</imsld:initialvalue>
</imsld:locpersproperty>
<imsld:learning-activity identifier="LA-task41">
<imsld:title>4.1. Impressionism Evaluation</imsld:title>
<imsld:activity-description>
<imsld:item identifierref="resource-test4" />
</imsld:activity-description>
<imsld:complete-activity>
<imsld:when-property-value-is-set>
<imsld:property-ref ref="impeval-good-enough"/>
<imsld:property-value>true</imsld:property-
value>
</imsld:when-property-value-is-set>
</imsld:complete-activity>
</imsld:learning-activity>
<imsld:conditions>
<imsld:if>
<imsld:greater-than>
<imsld:property-ref ref="resource-test4.score"/>
<imsld:property-value>5</imsld:property-value>
</imsld:greater-than>
</imsld:if>
<imsld:then>
<imsld:change-property-value>
<property-ref ref="impeval-good-enough" />
<property-value>true</property-value>
</imsld:change-property-value>
</imsld:then>
Practice
Contact Tutor
QTI re-test
Remediation
QTI test
Inductive/visual Deductive/verbal
Yes
No
Yes
No
Learning Style
Property value?
Mastered QTI test?
Mastered
Re-test?
Practice
Contact Tutor
QTI re-test
Remediation
QTI test
Inductive/visual Deductive/verbal
Yes
No
Yes
No
Learning Style
Property value?
Mastered QTI test?
Mastered
Re-test?

11. Standards-based Modeling
aLFanet

12. CSCL & OLM

13. CSCL:The CLFA
 User profiles
 Participative, Insightful, Useful, Non-
collaborative,With-initiative,
Communicative
 Thinker-out, Unsecure, Gossip,
Inspirable, Inspiring,Thorough
Forum conversations started Forum messages sent Replies to student interactions
N_thrd = ∑i
n(xi); x number of
threads started on day i and n a
set of days in the experience
N_msg = ∑i
n(xi); x number of
messages sent on day i and n a
set of days in the experience
N_r_thrd = number of
messages in the thread started
by user
M_thrd = average (N_thrd) =
(1/N)( ∑i
n(xi)); N number of
days in the experience
M_msg = average (N_msg) M_r_thrd = N_r_thrd / N_thrd
V_thrd = variance (N_thrd) V_msg = variance (N_msg) N_r_msg = number of replies
L_thrd = N_thrd /√V_thrd L_msg = N_msg /√V_msg M_r_msg = N_r_msg / N_msg

14. Accesibilidad y Adaptación
Agente Usuario
Tecnologías de Apoyo

15. 12/06/2008 ALPE Workshop - EDeAN Congress

16. aLFanet, ADAPTAPlan, FAA, ALPE, EU4ALL

17. LMS-
EVAApoyo
autoría
Valoración
Necesidades
Accessibility
support
Upload and tagging
Accesibilidad
Recurso
Comunicación y soporte
Descripción de
necesidades
Evaluación
Adaptación
Recursos
Needs
Related
Guidance
Accessibility
Evaluation
& Guidance
Adaptation
Request &
Supervision
Trans-
formation
Tagging
Supervision
Feedback
Resource-Course
Feedback
Profesor
Estudiante
Estudiante
Personalised
resource
Bibliotecario
Técnico de
Transformación
Resource
Feedback
Resource
Feedback
UM
MR
CP
RS
Accesibilidad
Curso
Senior
Manager
Decision
Taking
DM
Framework
component
EU4ALL
eService
Técnico
Atención
discapacidad

18. UC-AI→ Accessibility / Context??
User who requests textual captions for the
audio on videos. ISO PNP
<accessForAllUser>
<content>
<adaptationPreference>
<adaptationType=”caption”/>
<originalAccessMode=”auditory/”>
<usage=”required”/>
<language=”eng”/>
</adaptationPreference>
</content>
</accessForAllUser>

19. UC-AI → Interoperability / scale-up??
CONTENT PERSONALIZATION
CP Service
UM
DM
MR
LMS
IMS-LIP
ISO-AfA
ISO-
DRD
CC/PP
Rules
File
SOAP
SOAP
SOAP
RESACCINFO
Service
SOAP
SOAP
• Isa
– Daisy as an alternative
to the SCORM
– Transcript for video
• Leo
– Sign language
– Subtitles for video
EU4ALL architecture components
CP: Content Personalization
UM: User Modeling
MR: Metadata Repository
DM: Device Model
LMS: Learning Management System (Moodle, dotLRN, Sakai)

20. UC-AI: Recommendation delivery

21. Recommenders
The SERS approach
 Extend e-learning services
with ANS
 actions (read / contribute) on
platform objects
 standard-based service oriented
architecture (IMS,W3C, ISO)
 Recommendations model
 Elements: type, content,
runtime information,
justification, recommendation
features
 Support for eliciting
educational oriented Rec-s
 TORMES methodology
▪ UCD based (ISO 9241-210)
TORMES methodology
Service oriented architecture

22. Example of Rec
Recommendation 1: Read the tutorial on how to use the platform
Object: tutorial Action: read
Content (text + link): “Visit the platform tutorial of the platform”
Title: “Access to the page with the platform tutorial”
Applicability conditions:
The learner is new to the platform
The learner has interacted with the platform several times
The learner has not accessed the tutorial
The learner has not contributed in any of the platform services
Restrictions:
There is a tutorial in the platform
Category:
technical
support
Stage:
getting used to the
platform
Origin:
tutor
Relevance:
4.2
Rationale: make the learner get familiarised with the platform
Explanation: “Since you are new to the platform and you have not yet used the
services available in the platform, you can access this tutorial to get familiarised
with the platform operation”

23. Recommenders
UGIs

24. Recommenders
SERS Experiences
ID Description
R1 Choose a lesson to review
R2 Start the review of the concepts
R3 Review the concept estimated as less known
R4 Use the forum to share a doubt
R5 Read a thread of the forum with many posts
R6 Read the educators’ welcome message
R7 Change the avatar that represents
R8 Change the avatar that represents the learner
R9 Look at the learner conceptual model
R10 Look at the conceptual model of the class
R11 Log in to start the course for the first time
R12 Log in to keep reviewing the contents
Indicators Description p
avg_sessions participants who received recommendations spent more sessions in average 0.0362
avg_hits participants who received recommendations made more hits (visited more
pages) in average
0.0241
ratio<2days_period less participants from those who received recommendations entered less
than 2 days
0.0136
ratio_all_days_connected more participants from those who received recommendations had sessions
every day
0.0299
0.0150
avg_connection_window the connection window (number of days between the first day and the last
day) is larger in average for those who received recommendations
0.0019
avg_days_to_enter participants who received recommendations waited less days in average to
enter the module
0.0206
avg_days_to_end participants who received recommendations spent more days in average 0.0169
avg_days_connected participants who received recommendations connected in average more
days
0.0357
P values for Engagement
Indicators Description R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12
ratio_all_correct_answers
Percentage of learners who
answered all questions
correctly
+ + + + + + + +
avg_correct_answers
Average of questions
answered correctly by the
learners
+ + + + + + + + + + +
Knowledge acquisition confidence level 95%

25. Modeling Affect
aDeNu-UNED 2015 ©
MAMIPEC Project (TIN2011-29221-C03-00)
Partners Funded by

26. TORMES elicitation cycle
Understand Context of
Use
Interviews & Questionnaires:
educators’ best practices on
affective issues
DM analysis: affective data
processing complement educators’
descriptions
Specify requirements
Scenario based approach: problem
(situations demanding affective
support) and solution (affective recs
in terms of semantic affective rec
model) scenarios
Evaluate designs against
requirements
Running prototype / Wizard of Oz:
educators’ & learners’ evaluation of
affective recs.
Produce design solutions
Focus group (educators with
affective computing experience)
validate model-formulated affective
recs
eliciting
affective
recs
validated
semantically
described
affective
recs
Understanding recommendations needs
in intelligent educational contexts that
consider multimodal approaches
for affective modelling
[@ RecSysTEL 201
ISO 9241-210
iterative design
cycle

27. Motivation
Relation between
emotions & cognitive
processes
Advances in affective
computing
AIED 2015 Doctoral Consortium24 June 2015, Madrid, Spain 27

28. Wrap up: State of the Art
→ Identify research gaps
Psychology Computer Science
Learning
technologies
Parasympathetic
Nervous System
Physiological signals
Emotion theories
Signal Processing
Interaction indicators
AffectiveComputing
Data Mining
Learner Modelling
IntelligentTutoring
Systems
Ubiquitous learning

29. Large Scale Experience (≈ 80 users / 4Tb)
Sensor belt with the following
sensors: Electrocardiogram
(ECG), Galvanic Skin Response
(GSR), Respiratory Rate (RR),
and Blood Pressure (BP)
Kinect for Windows for face
features extraction
Webcam (with integrated
microphone) and infrared-light
webcam
Keyboard and mouse (via a
keylogger and a mouse tracker)
Questionnaires: General Self-
Efficacy Scale (GSE), PANAS,
BFI, SAM

30. Large Scale Experience (≈ 80users / 4Tb)

31. Desktop / Mouse / Keboard tracking
Mouse:
17:24:29:734-154,183 [mov] [154, 183]
17:24:29:781-154,184 [mov] [154, 184]
17:24:29:812-154,185 [mov] [154, 185]
17:24:29:968-154,185 [pre] [left] [154, 185]
17:24:30:093-154,185 [rel] [left] [154, 185]
17:24:30:250-156,185 [mov] [156, 185]
Keyboard:
16:59:32:484-48 [down]->0
16:59:32:546-48 [up]->0
16:59:42:062-32 [down]->SPACE
16:59:42:156-32 [up]->SPACE

32. Labelling emotions: SAM
Valence / Arousal

33. Task-1: math problems (lack of time)

34. Task 2: Logic series (easy task)

35. Affect Reports
Me he divertido resolviendo los ejercicios
Me he sentido orgulloso/a por haber sido
capaz de resolver los ejercicios
Me he sentido enfadado/a por la dificultad de
los ejercicios
Me he puesto nervioso al enfrentarme a la
resolución de los ejercicios
Me he sentido avergonzado por no ser capaz
de resolver los ejercicios
Me he desesperado tratando de hallar la
solución de los ejercicios
Me he aburrido haciendo los ejercicios
SC:
Al realizar esta tarea me he sentido ...
Al realizar la tarea he pensado ...
Las dificultades que he encontrado para resolver la tarea han sido..
Y para superar estas dificultades he ...
Puedes usar: ABURRIDO, ACTIVO, ADMIRADO, AGOBIADO,
ALEGRE, AVERGONZADO, CABREADO, DEFRAUDADO,
DEPRIMIDO, DESANIMADO, DESESPERADO…

36. Minería de datos (4TB): Diseño
Puntuación
Sentiment
analysis
Interacciones
de usuario
Medidas
fisiológicas
Puntuación
de la tarea
Cuestionarios
Algoritmos
de DM
SAM del
usuario
Etiquetado
emocional de
los reportes
emocionales
Etiquetado de registros
Predicción de la
puntuación de
valencia afectiva

37. Inclusive scenarios: Experimental design
Physiological sensors, Kinect, Mouse, Webcam
Questionnaires
Sensor
placement
Initial Baseline
Sensor
calibration
questions
Task 1
(Problem solving)
Emotional
Report 1
Task 2
(Problem solving
with time limit)
Emotional
Report 2
Task 3
Logical series
Emotional
Report 3
Final Baseline
Physiological sensors, Kinect, Mouse, Webcam
Physiological sensors, Kinect, Mouse, Webcam
Physiological sensors
SAM
SAM
SAM
SAM
Keystrokes
Keystrokes
Keystrokes
Personality
traits
Adaptationsforinclusiveness
[@ HCI 2013]

38. Inclusive scenarios: Physiological Signals

39. DESAFÍOS ANTE LA DIVERSIDAD
FUNCIONAL
 Adaptaciones en logística, tareas y técnicas de interacción.
 Ajustes en tamaño de fuentes en pantalla y papel.
 Ajustes en luminosidad.
 Reducción distancia a la pantalla (20ctms) que interfirió en el
dispositivo de Kinect.
 Filtrado por parte de Kinect de movimientos estereotipados
relacionados con discapacidad visual.
 Estudiar de manera diferencial el patrón de interacción
mediante teclado durante navegación/introducción de
datos.
 Cambios fisiológicos y conductuales en función de la
dificultad y la limitación en la tarea
 Sonidos con componente afectivos provocaron cambios
fisiológicos.

40. Research question
How ERS can take advantage of
affective computing
to improve the personalized support
in educational scenarios with
emotional and affective issues?

41. Approach
User Device
e-Learning
Platform
User
Model
• Learning outcomes
• Questionnaires scoresbio-feedback
devices
sensor &
interaction
data
Learning
Interactions
Personality
Traits
Data
Mining
Affective
information
+Learner
Affective
Model
Affective
Feedback
[@ CAEPIA 2013]

42. Some Recommendations identified
 R1: Provide course instructions for newbie learners,
so they do not get lost in the course space
 R2: Carry out self-assessment questionnaires to
foster learners’ meta-cognitive issues
 R3: Review related course concepts to help
progressing in the course contents
 R6: Propose strategies to cope with temporal failures
of the platform, especially when deadlines are
approaching
 R8: Change task type to keep motivation and
engagement in the tasks

43. Validation criteria
 C1: need to deliver the recommendation
 C2: recommendation content suitability
 C3: timely delivery of the recommendation
 C4: benefit of the recommendation
 C5: suitability of the recommendation
presentation mode

44. Validation results
C1 C2 C3 C4 C5
Avg Std. Avg Std. Avg Std. Avg Std. Avg Std.
R1 7,83 1,17 7,83 1,17 8,33 1,21 8,17 0,75 7,00 1,90
R2 8,17 0,75 7,83 1,17 8,17 1,17 7,33 1,21 7,50 1,38
R3 8,17 0,75 7,83 1,17 8,00 1,10 8,00 1,67 7,50 1,76
R4 8,33 0,52 8,50 0,55 7,83 1,60 8,50 0,84 8,00 1,26
R5 8,00 1,10 7,67 1,51 7,67 1,86 7,67 1,86 8,50 0,55
R6 8,67 0,52 8,17 0,75 8,17 0,98 8,67 0,52 8,00 1,10
R7 8,33 0,52 8,00 1,55 8,50 0,55 8,17 1,17 7,17 3,13
R8 7,83 1,17 8,33 0,52 8,00 1,10 8,17 0,75 8,17 0,98
R9 8,67 0,52 8,50 0,55 8,33 1,21 8,50 0,84 8,33 1,03
R10 8,00 1,55 8,17 1,17 8,17 1,17 8,17 1,17 8,17 0,98

45. Selecting a Rec. for the pilot
 R2  appropriate to provide personalized support while
participants are carrying out the mathematical self-assessment
tasks proposed.
 situations to be encounter ed: focusing attention on relevant data,
reviewing mistakes and results, analyzing the information provided in
the problem wording, avoiding a lack of motivation as consequence of
wrong results, etc.
 Texts & Emotion-aware recommendation rules
 prepared by the 2 educational researchers involved in activity 2
 adapted to the tasks particular context
 consider lessons learnt compiled in reviews regarding formative
feedback [Shute, 2008] and affective feedback [Girad et al., under
review]
 Different metacognitive strategies (2different moments)
▪ Reading and planning how to solve the problem
▪ Reviewing the results obtained

46. Texts proposed for R2
 Text-A: “Some of the exercises can be a bit confusing.
Thus, you should read the wording in detail”
 Text-B: “Take your time and read the different
alternative options in detail to solve the exercise”
 Text-C: “Focusing is very important to solve
mathematical tasks. If you focus on the wording and
the options, you will solve it”
 Text-D: “Don’t worry about the results obtained so
far.The most important is to keep motivated to try to
solve the next ones the best you can”
 Text-E: “We learn from our mistakes.Thus, if you
review in which issues you have failed, this will help
you to do better next time”

47. Some Emotion-aware Recommenation rules (R2)
 Rule 1: If there are 2 wrong responses in a row,
deliverText-A
 Rule 2: If the learners’ face shows confusion
when reading the wording of an exercise, deliver
Text-A
 Rule 3: If the learner cannot decide which option
to select and moves from one to another, deliver
Text-B
 Rule 5: If the learner is distracted, looks away
the screen, deliverText-C
 Rule 6: If the learner increase her facial and body
movements’ rate, deliverText-C

48. Rec. delivery in the pilot

49. Works on Emotions Detection & ERS
 2012: Experimental design for collecting affective data at
Madrid Science Week in 2012: 75 participants
 2013: Machine learning techniques in an incremental way,
considering a subset of the collected input sources.
 2014: Detailed analysis on keyboard and mouse features
 Related papers:
SWJ´14, SCP’14, SWJ´14, AIED’13, HCI’13, UMAP’13-14, EDM’13-
14, SCP14, ERS&T12, IJAIT’13, ESWA11, EXSY13, IJWBC12,
UMUA’11, … IJAIED’16
 Springer volume on Recommender Systems forTechnology
Enhanced Learning (13)
49

50. ITS UV-UNED
aDeNu-UNED 2015 ©

51. Experiment
 Real context
 High School
 14-year old students
 Using the high school’s computers room
 Using an ITS
 Teaching the resolution of story problems in an
arithmetic way
 6 problems were proposed
AIED 2015 22-29 June 2015 Madrid 51

52. Experiment
 Data collected
 Physiological data
 Video data
 Interaction data
 Task Data
AIED 2015 22-29 June 2015 Madrid 52

53. ITS:Tipos de problemas (I)
Uno de los mayores problemas al aprender álgebra es la traducción
de los problemas del mundo real a notación simbólica.
Problemas que suelen aparecer:
 El estudiante puede utilizar un número arbitrario de letras o
símbolos para designar las cantidades del problema
 El tipo de ecuaciones planteadas
 El camino elegido para solucionar el problema

54. ITS:Tipos de problemas (II)
 Algunos ejemplos de problemas:
 Luis, Juan y Roberto ganaron 960 € por pintar una casa. Debido a que no
trabajaron el mismo tiempo, Luis recibió 24 € menos que Juan y la tercera
parte de lo que ganó Roberto. ¿Cuánto ganaron cada uno?
 Una cesta tiene 60 piezas de fruta, entre manzanas y peras. En concreto
tiene 10 veces más manzanas que peras. ¿Cuántas manzanas hay en la cesta?
 El padre de Miguel es 3 veces mayor que Miguel. Hace 4 años era 4 veces
mayor. ¿Cuál es la edad de Miguel?

55. GUI ITS MAMIPEC

56. Domain-specific knowledge representation
• Allows:
• Representing all potential solutions to the problem
• Representing the current state of the resolution process
• Determining all potentially valid user actions
Basedonhypergraphs
With semantic annotations provided in a XML file
Mike's father is 3 times as old as Mike. 4 years ago, he was 4 times
older. How old is Mike?

57. Ayudas personalizadas
Mensajes predefinidos para
los errores más comunes y
frecuentes
Mensajes Adhoc que usan
anotación semántica

58. Ayudas personalizadas (solicitadas)
Anotaciones semánticas usadas para
producir ayudas bajo demanda

59. Experiment
 Data collected
 Affective data
▪ At the beginning of the experiment
▪ AttributionalAchievement Motivation Scale
▪ At the end of each exercise
▪ SelfAssessment Manikin scale
▪ At the end of the problem series
▪ a descriptive self-report
▪ At the end of the experiment
▪ A recorded visualization of the experiment was made by the
student with a psychologist who had followed the experiment
AIED 2015 22-29 June 2015 Madrid 59

60. Facial expressions and
body movement detection
AU3. Brow Lowerer
Head pose . Roll
User Task Time Duration Location Movement type Comments Affective State
act2usr1s
es10d09
m11
T2 0:10:27 5 sec Brow furrow brow
reasoning
about why
he failed a
problem
Confused
,,,
act2usr2s
es10d09
m11
T3 0:10:50 8 sec Head
Tilt head to one
side
rereading
the problem
statement
Concentrated

62. Ejemplo
Operaciones
Cantidades para operar
Explicación
de lo que hago
Operar
Ayuda
Enunciado del Problema a resolver

63. Operaciones
Cantidades para operar
Explicación
de lo que hago
Operar
Ayuda
Enunciado del Problema a resolver

64. Operaciones
Cantidades para operar
Explicación
de lo que hago
Operar
Ayuda
Enunciado del Problema a resolver

65. Resultado de la
operación
realizada
Se añade en la lista de cantidades para poder utilizarse
Ayuda
Enunciado del Problema a resolver

68. Valencia
Valora si te ha gustado el problema

69. Activación
Da igual si te ha gustado o no.
Valora cuánta energía has puesto para
resolver el problema

70. Objective
 Improve emotion detection
 Following a multimodal approach
 Using non-intrusive devices
 Exploring new approaches
AIED 2015 22-29 June 2015 Madrid 70

71. Exploring new approaches
 Current methods:
▪ Try to classify the emotion directly
 A two-class classifier to identify relevant time slots
▪ Adding a new layer
▪ A simpler initial problem (2 values to predictVS a list of emotions)
▪ Using that information for the generating a more
balanced dataset and more detailed predictions.
AIED 2015 22-29 June 2015 Madrid 71

72. 2-step emotion detection approach
Emotion? No
Emotion? Yes
Emotion? Yes
Boredom
Engageme
nt
First
Step
Second
Step
AIED 2015 22-29 June 2015 Madrid 72

73. Data preparation
 Video
 Synchronizing Facial and desktop videos
▪ During the experiment
▪ Reviewing the experiment with the students
 Movement labelling
▪ Following an already proposed methodology
▪ Taking into account:
Type of movement Body part moved Movement
duration
AIED 2015 22-29 June 2015 Madrid 73

74. Data preparation
 Physiological signals
 Features had to be generated…
▪ Which time window to use?
▪ A recursive analysis was performed with different time windows (1
minute, 30 seconds, 20 seconds)
▪ Finally, the 20 second time window was used in all the data
 …According to a baseline
▪ Initial baseline was discarded
▪ Some signals were still not completely stabilized
▪ Possible reactions to the experimental situation
▪ The final baseline was used as a reference for each student
AIED 2015 22-29 June 2015 Madrid 74

75. Data preparation
 Physiological variations according to the
baseline
 An ANOVA was carried out, looking for significant
differences between the baseline and each of the
time windows.
▪ Those temporal windows (per subject and signal)
significantly different from the final baseline (p <0.001)
were labeled as “activated”.
AIED 2015 22-29 June 2015 Madrid 75

76. Features used
 Features generated for every 20 seconds time
window:
 Physiological signal significance flag.
 Number of Significant physiological signals.
 Number of wrong actions.
 Number hints requested.
 Movement indicators:
▪ Body part moved.
▪ Type of movement.
▪ Movement duration
AIED 2015 22-29 June 2015 Madrid 76

77. Emotional labeling
 We have the SAM scores for each problem
 An expert labeled the emotions seen in the videos
following a categorical approach
▪ The expert’s labeling was supported by the emotional
reports and experiment review done by the student.
AIED 2015 22-29 June 2015 Madrid 77

78. Results
 Emotion indicator  2-step approach Traditional
approach
Boredom
Frustration
Boredom
Frustration
None
Less than 60%
accuracy
Kappa: 0.31
Emotion
? No
62.6% accuracy
Kappa: 0.31
Emotion? Yes
Emotion? No
Emotion? Yes
74.8% accuracy
Kappa: 0.49
Emotion?
Yes
Emotion?
Yes
AIED 2015 22-29 June 2015 Madrid 78

79. Recommendations  textual

80. Responding to emotions
 Open issues in detecting emotions do NOT
stop research of affective interventions
 Runtime access to signals collected
 Rapid prototyping
 Open hardware

81. Goal
Identify Ambient Intelligent
Recommendations that provide
Interactive Context-Aware
Affective Educational support
Publication:
Santos, O.C., Saneiro, M., Rodriguez-Sanchez, M.C. and Boticario,
J.G. (2015)
Towards Interactive Context-Aware Affective Educational
Recommendations in Computer Assisted Language Learning.
New Review of Hypermedia and Multimedia, in press.

82. Responding to emotions
 Open issues in detecting emotions do NOT
stop research of affective interventions
Wizard of Oz:
human detects user
behaviour & reacts
(simulates system response)

83. Eliciting AmI Recommendations
 Educational scenario:
 Preparing for the oral examination in a foreign
language learning course
TORMES Methodology
Santos & Boticario, 2015
Computers & Education, vol.
 Recommendation:
 Suggest the learner to
breathe slowly to calm her
down when nervious
▪ without interrupting her
activity

86. Detecting emotions with Arduino
E-Health Platform:
• Pulse
• GSR
• T
• ECG
• Airflow
+ Adaptation &
Integration of
a piezoelectric
breathe belt

87. Integrating piezoelectric belt

88. Deciding affective response (rec)
 Rule-based approach
 weighted physiological states: Current vs. Baseline
▪ if HR_now > 20%HR_bl  trigger Rec1
 Wizard of Oz decides
▪ physiological signals
▪ participant’s facial & body movements
▪ task progress
▪ …

89. Open issues
1. How to deliver the recommendations
2. When to provide recommendations
3. Learners’ features
4. Social aspects (when collaboration)

90. Multisensorial actuation channels

91. First try (unary): on  relax
1 blue led
(lights)
1 buzzer
(sounds & ‘vibrates’)

92. Second try: modulate breathing
 Acompany learner’s desired breathing
behaviour
 Fixed rate:
▪ 4 breathes/minute (inhalation/exhalation = 7.5 segs)
 Dynamic rate:
▪ Relative to breathing at Baseline (or other signal)

93. Second try: modulate breathing
 Approaches considered per sensorial channel
Flashlights (-)
Array of leds
Ambient light (+)
Speaker Vibrator (+)

94. Second try: modulate breathing I
 On-Off (binary)  till user relaxed or stoped
 on for 7.5 segs  inhalate
 off for 7.5 segs  exhalate
 Speaker (pure tone 440Hz = ‘la’)
 2 flashlights (red/white)  Array of leds
(intensity manually controlled)

95. Second try: modulate breathing II
 Progressive
 Signal increases for 7.5 segs  inhalate
 Signal stops  hold breath
 Signal decreases for 7.5 segs  exhalate
 Array leds (12 blue)
 Speaker: cromatic musical scale 1/8 (‘la’’la’)
On going: dynamic rate  modulation depends on BL

96. On-going works
 Ambient light
 Sourrounding light that changes of colour
▪ Green: calm
▪ Red: stress
▪ Blue: measuring baseline
▪ White: reporting emotional state
 Moves from Green to Red

97. On-going works
 Vibrator (touch sense)
 Alerts before the rec is necessary
 On skin / table / chair
e.g., extend an intelligent
cushion developed by R. Barba
in his Final Career project to
detect user’s movements on the
chair → IJDSN (Accepted)

98. Pilot studies
first try  tell
unary (on= relax) second try  modulate
binary (on-off)
Pilot 1: 6 participants (1 blind)Pilot 2: 4 participants

99. Evaluation approach
 System Usability Scale (Brooke, 1996)
 Ad-hoc questions:
 Q1: Did you feel relaxed during the experience?
 Q2: Do you consider that the recommendations provided
during the experience had an impact on your
performance?
 Q3: Do you think that the recommendations had been
provided at the right time?
 Q4: Do you consider that the recommendation format is
appropriate?Would you prefer any other alternative
format?
 Q5: Do you consider that the recommendation was
effective in modifying or improving your performance?

100. Evaluation outcomes
 SUS  system not usable (agree!)
 Unary approach
 Light hardly perceived
 Sound too strong (distracts / warning)
 Binary approach
 Array of leds  visible
 Progressive (light) more intuitive
 Placement is important
 Keyboard vs. top of the screen
 Alternative formats?Textual / Speach
 Posibility of selecting the channel

101. learner
features
recs
selected
Services in the architecture
data
processed
device
capabilities
learner
features
device
data
learner
data
request
recommendation
with context data
Emotional
Data
Processor
Multimodal
Emotional
Detector
SAERS
server
Emotional
Delivery
Component
SAERS
admin
Recs
modelled
TORMES methodology
User
Model
Device
Model
environment
data
collected
emotions
detected
recs
recs
affective
personalized
educational
recommendations
sense of arrows:
Initiator of info flow
Learner accessing the e-learning system with a certain device
in an environment with Sensors and Actuators

102. Discussion
AIED 2015 22-29 June 2015 Madrid
 Importance of labeling on the results
 Who labels the emotions
▪ User itself
▪ External viewer
 How the emotions are labeled
▪ Numeric scale
▪ Categorical values
▪ Which values are shown?
▪ Free labeling
▪ Which machine learning techniques can be used?
 What time window length should be considered
for emotion detection?
102

103. Discussion
AIED 2015 22-29 June 2015 Madrid
 Emotions not only depend on what you are
doing, the context around you also may
affect:
▪ Music, noise, etc
▪ Light conditions
▪ People around
▪ Other things you migh have in mind
103

104. Discussion
 Computational costs
 2 different prediction problems
▪ The second one with a smaller dataset
 Importance of FP and FN in the first step
prediction
 FP (Predicting an emotional change when there is no
emotion):
▪ System reaction may change the student’s flow (if the state
was the right one).
 FN: (Not predicting emotions when there are some
affective changes):
▪ Being late on reacting to negative state changes.
AIED 2015 22-29 June 2015 Madrid 104

105. Discussion
AIED 2015 22-29 June 2015 Madrid
 Data sources dependencies
 Task to be solved
 User interface & skill
 User special needs
105

106. Trabajos Futuros / Diseminación
 Áreas de trabajo
Colegios / laboratorios
 Algunos artículos relacionados:
C&E’15, NRHM’15, IJDSN’15, EDM’15, AIED’15, UMAP
-PALE’15, SWJ´14, SCP’14, SWJ´14, AIED’13, HCI’13,
UMAP’13-14, EDM’13-14, SCP14,, ERS&T12, IJAIT’13,
ESWA11, EXSY13, IJWBC12, UMUA’11, … IJAIED’16
Springer volume on Recommender Systems for
Technology Enhanced Learning (13)
 Experiencias:
intra- / inter-sujeto
colegios / lab

107. References & Acknowledgements
aDeNu research group
https://adenu.ia.uned.es
Background related projects
https://adenu.ia.uned.es/web/projects
o European:
aLFanet, ALPE, EU4ALL, ALTER-NATIVA
o National:
FAA, CISVI, AMI4INCLUSION, ATODOS, A2UN@,
ADAPTAPlan, MAMIPEC, BIG-AFF
‘