LEARNER CENTRED DESIGN
INTELLIGENT SYSTEMS
and
Rosella Gennari
http://www.inf.unibz.it/~gennari
Distinguished Speakers 
Oxford Women in CS
Setting
technology
enhanced learning
Climax
TERENCE case
study
Resolution
reflections
Story outline
Setting
technology
enhanced learning
Climax
TERENCE case
study
Resolution
reflections
Story outline
TECHNOLOGY ENHANCED LEARNING
Technology Enhanced Learning
(TEL) is the usage of technology
for supporting a learning
experience
Herby we take a narrow view:
intelligent TEL =
Artificial Intelligence (AI)
technology based products
for supporting a learning
experience
TEL 4 LEARNING EXPERIENCE
How can we design
technological products
that supports their users’
learning experience?
LET'S SEE EDUCATORS' VIEWPOINT...
Maria Montessori (1870-1952)
Paraphrasing her words, adequate
tasks that come in a prepared
environment, designed on top of the
learner characteristics can
effectively support the learner’s
learning
was the first Italian woman physician
and educator, best known for
Montessori pedagogy
ousability of technology
learning products
opedagogical effectiveness
of technology
learning products
TEL 4 LEARNING EXPERIENCE
Adequate tasks that come in a prepared
environment designed on top of the learner
characteristics can effectively support learning
HOW TO DESIGN USABLE AND
PEDAGOGICALLY EFFECTIVE TEL
Based on UCD process diagram (© Tom Wellings)
requirement
specification
designevaluation
plan
models +
prototypes
intermediate
product
final
product
HOW TO DESIGN USABLE AND
PEDAGOGICALLY EFFECTIVE TEL
USABILITY +
P. EFFECTIVENESS
Setting
technology
enhanced learning
Climax
TERENCE case
study
Resolution
reflections
Story outline
TERENCE
DESIGN
Based on UCD process diagram (© Tom Wellings)
requirement
specification
designevaluation
plan
models +
prototypes
intermediate
products
final
products
TERENCE was an FP7 TEL project
blending user centred and evidence based
design
USABILITY +
P. EFFECTIVENESS
THE PROBLEM
‣ TERENCE developed an adaptive learning system (ALS) that,
via a learner GUI, recommends poor comprehenders
- its learning material, i.e., books of stories and games
- its learning tasks, i.e., reading and playing
‣ so as to stimulate their reading comprehension
‣ More than 10% of primary school children, older than 8,
are diagnosed with deep text comprehension problems
‣ They are referred to as poor comprehenders
THE TERENCE WORLD
a d e q u a t e
b o o k o f 
s t o r i e s
s i g n i n
a d e q u a t e
s m a r t
g a m e s
r e w a r d
TERENCE INTELLIGENT TEL
PRODUCTS
ALS LayerGUI Layer
Learner
Educator
Expert
Learner GUI
Expert GUI
Persistence Layer
OpenRDF
User
Manager
OpenRDF
Story
Manager
OpenRDF
Game
Manager
OpenRDF
Visualisation
Manager
illustrations
NPL
Reasoner
Adaptive
Engine
Visualisation
Reasoning
Module
Annotation
Module
Visualisation
Module
game
generation
adaptation to
learners
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE
DESIGN
TERENCE
DESIGN
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
DATA GATHERING METHODS
‣ Data for designing the learning
material and tasks were from
- contextual inquiries with
‣ IT & UK diagnosis
‣ as well as IT & USA evidence-
based medicine therapy experts
- field studies with educators and
primary school children
4510
~ 500
C
H
A
R
A
C
T
E
R
I
S
T
I
C
S
Persona Name: Carol
Age: 8
Classroom: year 4
RC levels: low reading levels
Rural/Urban: urban
Deaf/hearing: deaf
First Language: Italian Sign Language
Cochlear Implantation (if deaf): yes
Degree of hearing loss (if deaf): profound
Motor skills (if deaf): average
Summary of the class
represented by this
persona
A younger deaf girl who is very enthusiastic about using new
technology (such as iPhone and IPad) and who adores her
Nintendo DS. Her reading RC levels are very low, but she reads
together with her parents to learn new words and spelling. She
also likes to do many other things such as drawing, taking care of
her pets and going to the park.
Quote “I really love Mario and Luigi. And I would love to have an iPhone
and an IPad, like my dad.”
Personality Open
Role in classroom Active
Role out of the
classroom
Active
Console/Technology Carol and her sister watch TV after school. They like Tom & Jerry,
Ben 10, Hello Kitty and Mickey Mouse Clubhouse.
Carol sometimes uses the computer, but only to play minilab
games. Her computer is in her bedroom, but her parents don’t
allow her to use it all of the time. She can use it only one hour per
day. Carol’s dad has an iPhone and an IPad, and Carol would
really like to use those as well, but her dad tells her she is a bit
too young. Carol likes watching him with his IPad and iPhone
though.
Carol doesn’t use a mobile phone.
Carol plays games on the computer and on her Nintendo DS.
She plays by herself. She likes the mini-clip games on the
computer, and Mario Kart and brain training games on her DS.
She likes games with non-photorealistic human avatars, and
prefers fantasy avatars to animal avatars.
Socio-Cultural Level of
his/her own family
Medium
School performance Carol has sever reading problems. In her class she is below
average in all activities but drawing, where she feels she can truly
express her intimate feelings.
Homework After school, Carol does her homework together with her mum.
L
I
F
E
S
T
Y
L
E
Outdoors Activities Carol often goes to the park with her mum.
Indoors Activities Games on the DS
Carol reads sometimes. Her mum and dad help her reading in the
evening. She likes some of the stories they read together, but
mostly, she wants to read because she has to learn new words
and spelling.
Carol likes drawing and taking care of her pets.
Her mum often plays with her.
Home activities Carol also likes to help her mum in the kitchen or in the garden.
Sport activities Carol practices no specific sport.
.
SMART GAME REQUIREMENTS
What for Description
Difficulty levels Macro levels for learners:
- entry: character games;
- intermediate: time games;
- top: causality games.
Scheduling of reading
and playing
1st silent reading; 2nd playing smart games; 3rd playing
relaxing games
Constraints on actions Learners should get faster, hence a game has a maximal
resolution time
Progress and feedback Monitor and give learners (1) visible idea of progress, (2)
explanatory feedback, (3) recall their attention and solicit
them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via
visual representation and reasoning
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE
DESIGN
SMART GAME REQUIREMENTS
What for Description
Difficulty levels Macro levels for learners:
- entry: character games;
- intermediate: time games;
- top: causality games.
Scheduling of reading
and playing
1st silent reading; 2nd playing smart games; 3rd playing
relaxing games
Constraints on actions Learners should get faster, hence a game has a maximal
resolution time
Progress and feedback Monitor and give learners (1) visible idea of progress, (2)
explanatory feedback, (3) recall their attention and solicit
them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via
visual representation and reasoning
who is the actor of … ? what does (a main) character do?
when does … happen in relation to a central
event?
why does the central event happen?
SMART GAME REQUIREMENTS
What for Description
Difficulty levels Macro levels for learners:
- entry: character games;
- intermediate: time games;
- top: causality games.
Scheduling of reading
and playing
1st silent reading; 2nd playing smart games; 3rd playing
relaxing games
Constraints on actions Learners should get faster, hence a game has a maximal
resolution time
Progress and feedback Monitor and give learners (1) visible idea of progress, (2)
explanatory feedback, (3) recall their attention and solicit
them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via
visual representation and reasoning
points for each smart
coins for all smartunlocked if read+play
visual feedback
Instructions Questions Motivational Interaction
Choices Choices for learner Fixed event
Solutions Choices that are either correct (c) or wrong (w)
Feedback Interaction Consistency Explanatory Solution
Smart points Proportional to the learner’s ability in the game level
Relaxing
points
Constant
Avatar Happy/sad states
Time solution constant interaction constant
Rules States of the system, actions of the learner, constraints
What for Description
Difficulty levels Macro levels for learners:
- entry: character games;
- intermediate: time games;
- top: causality games.
Scheduling of
reading and
playing
1st silent reading; 2nd playing smart games; 3rd
playing relaxing games
Constraints on
actions
Learners should get faster, hence a game has a
maximal resolution time
Progress and
feedback
Monitor and give learners (1) idea of progress, (2)
explanatory feedback, (3) recall their attention and
solicit them to give a resolution (in time)
Representation Production can be impaired hence promote resolution
via visual representation and reasoning
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE
DESIGN
EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS WHEN
analytical
HMI experts or
domain experts
heuristic evaluation
formative,
summative
expert evaluation
cognitive walk-through
small-scale learners
observations
formative
think aloud
large-scale learners field studies summative
EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS HOW
analytical
HMI experts or
domain experts
heuristic evaluation
formative,
summative
expert evaluation
cognitive walk-through
small-scale learners
observations
formative
think aloud
large-scale learners field studies summative
G1. interfaces
follow general
design
guidelines
G2. interfaces
support the
user’s next step
to achieve a
task
G3. interfaces
provide users
with timely
feedback
Instructions are not
under focus and
cannot be easily read
Game question and
possible resolutions should
be proximally close
Game question and
possible resolutions should
be proximally close
Evaluation of interfaces
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE
DESIGN
problem:
256 stories,
each with ~12 games
Smart game design
how
can we automatise the
development of smart games via AI (and,
hopefully, be efficient)?
Smart game design
enriched annotations
story
annotations
ALS LayerGUI Layer
Learner
Educator
Expert
Learner GUI
Expert GUI
illustrations
NPL
Reasoner
Adaptive
Engine
Visualisation
Reasoning
Module
Annotation
Module
Visualisation
Module
Semi-automated generation
enriched annotations
story

text

text

text

text
annotations
Semi-automated generation
ALS LayerGUI Layer
Learner
Educator
Expert
Learner GUI
Expert GUI
illustrations
NPL
Reasoner
Adaptive
Engine
Visualisation
Reasoning
Module
Annotation
Module
Visualisation
Module
text

text

text

text
image
image image image
enriched annotations
story
annotations
Semi-automated generation
games
template
visual

text

text

text

text
image
image image image
enriched annotations
story
annotations
Semi-automated generation
GUI Layer
Learner
Educator
Expert
Learner GUI
Expert GUI
Visualisation
Visualisation
Module
text
story
text + visual
games
AUTOM. MANUAL AUTOM.
Semi-automated generation
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE
DESIGN
APPROACH WITH WHOM EXAMPLE METHODS WHEN
analytical
HMI experts or
domain experts
heuristic evaluation
formative,
summative
expert evaluation
cognitive walk-through
small-scale learners
observations
formative
think aloud
large-scale learners field studies summative
EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS WHEN
analytical
HMI experts or
domain experts
heuristic evaluation
formative,
summative
expert evaluation
cognitive walk-through
small-scale learners
observations
formative
think aloud
large-scale learners field studies summative
EXAMPLE METHODS IN TERENCE
LARGE-SCALE STUDY DESIGN
Common design of the intervention with TERENCE:
‣how: pretest/posttest design, with experimental and control groups
‣hypothesis: TERENCE improves reading comprehension measured
with standardized text comprehension tests
ControlExperimental
A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
-and requires (1) reading
A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
-and requires (1) reading (2) playing smart
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
-and requires (1) reading (3) playing relaxing(2) playing smart
A 3-PHASE INTERVENTION
‣ Post-test (pedagogical only) for re-assessing text comprehension
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
-and requires (1) reading (3) playing relaxing(2) playing smart
A 3-PHASE INTERVENTION
The experimental group is of 344 learners:
‣ Avezzano: 270 learners:
- 7-9 years old: 118
- 9-11 years old: 152
‣ Pescina: 74 learners:
- 7-9 years old: 37
- 9-11 years old: 37
‣ They were tested (January-February), stimulated (March-May),
and re-tested (may-June)
EXPERIMENTAL GROUP IN IT
Pre-post performances for text comprehension
(dependent variable) were as follows:
‣ Pescina:
- pre: 14 poor comprehenders (20.59%)
- post: 6 poor comprehenders (8.82%)
‣ Avezzano:
- pre: 15 poor comprehenders (5.95%)
- post: 2 poor comprehenders (0.79%)
MAIN RESULTS IN IT
Pre
Pescina Avezzano
5,95%
20,59%
‣ Wilcoxon signed-rank test supports that differences
are statistically significant
- Pescina: z=-4.904, p<0.0001
- Avezzano: z=-2.266, p=0.0234
EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS HOW
analytical
HMI experts or
domain experts
heuristic evaluation
formative,
summative
expert evaluation
cognitive walk-through
small-scale learners
observations
formative
think aloud
large-scale learners field studies summative
EXPERT EVALUATION
Experts of pedagogy: 1 coordinator; 9
evaluators
Sophie'comes'down'the'steps
He had never been beaten before, since he
only ever raced with kids who were
smaller and slower than him.
He wanted a rematch, so the two boys set
off again. Ben was paddling as fast as he
could, still he didn’t make it to the wall
before Luke. It was completely unfair, he
thought. Luke was so much faster. No
sooner had they climbed out of the water,
than he saw his sister coming down the
steps. She was smiling at Ben and gave
him a playful pat on the shoulder. She also
gave Ben a friendly speech about winners
and losers.
revise selection of
solutions
revise selection of
central event
How-to:
1. each pair of evaluators read a story,
and edited its games
2. the coordinator revised their work
3. a pair of evaluators was blindly
assigned revised games, and another
the manually created games
Main edit tasks:
(1) creation of missing games (~recall)
(2) revision of games (~precision)
From D4.2 and D4.3 technical annex
overall assessment of
generation
text
story
text + visual
games
revision of Automated
Reasoning (AR) selection
of central events and
solutions
revision of Natural
Language
Processing (NLP) of
text
text
text text text
Edit tasks in details
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE
DESIGN
From D4.2 and D4.3 technical annex
overall generation
text
story
text + visual
games
AR selection of central
events and solutions
revision of NLP text
text
text text text
Analyses of evaluation results
AR selection of central events for games:
>Results: only in 15 out of 250 cases (6%), it was necessary
to select a different central event than the automatically
generated one
From D4.2 and D4.3 technical annex
>Implications for AR:
none picked up
Automated part evaluation-based re-design
AR selection of plausible solutions:
>Results: out of 140 changes of selection of solutions, the majority
was for wrong solutions
- generate a wrong solution from
correct one by changing participants,
e.g.,
<correct_sentence id="2">

The man ran and fell on the ground.
</correct_sentence>
<wrong_sentence id="2wh1">

Peter ran and fell on the ground.
</wrong_sentence>
>Implications for WP4: new heuristics for wrong plausible solutions in
the last part of Y3,
From D4.2 and D4.3 technical annex
Automated part evaluation-based re-design
Overall generation: development
times:
>Results for revision time:
- 12’6” per game instance:
↑ 12’8” for time games
↓ 10’6” for who games
>Results for creation time:
- avg. 23” per game instance
text
story
text + visual
games
From D4.2 and D4.3 technical annex
>Implications for AR: the semi-automated development process seems to
be promising for optimising development times
Automated part evaluation-based re-design
Game over
1st 2nd 3
Sep. 2011 December 2012 September 2013
Sophie'comes'down'the'steps
He had never been beaten before, since he
only ever raced with kids who were
smaller and slower than him.
He wanted a rematch, so the two boys set
off again. Ben was paddling as fast as he
could, still he didn’t make it to the wall
before Luke. It was completely unfair, he
thought. Luke was so much faster. No
sooner had they climbed out of the water,
than he saw his sister coming down the
steps. She was smiling at Ben and gave
him a playful pat on the shoulder. She also
gave Ben a friendly speech about winners
and losers.
revise selection of
solutions
revise selection of
central event
Requirements+for Description
Dif$iculty*levels Macro*levels*for*learners:
4*entry:*character*games;
4*intermediate:*time*games;
4*top:*causality*games.*
Scheduling*of*
reading*and*playing
1st*silent*reading;* 2nd* playing* smart*games;*3rd*playing*
relaxing*games
Constraints*on*
actions
Learners* should* get*faster,* hence* a* game* has* a* maximal*
resolution+time
Progress*and*
feedback
Monitor* and* give* learners* (1)* idea* of* progress,* (2)*
explanatory*feedback,*(3)*recall*their*attention*and*solicit+
them*to*give*a*resolution*(in*time)
Representation Production*can*be* impaired*hence*promote*resolution*via*
visual*representation+and+reasoning
Instruc(ons Ques%onsQues%ons Mo%va%onalMo%va%onalMo%va%onalMo%va%onal Interac%onInterac%on
Choices Choices3for3learnerChoices3for3learnerChoices3for3learnerChoices3for3learnerChoices3for3learner 3Fixed3event3Fixed3event3Fixed3event
Solu(ons Choices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%ons
Feedback Interac%on Consistency3(c/w)Consistency3(c/w)Consistency3(c/w) ExplanatoryExplanatoryExplanatory Solu%on
Smart6points Propor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3level
Relaxing6points ConstantConstantConstantConstantConstantConstantConstantConstant
Avatar Happy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3states
Time solu%on3constantsolu%on3constantsolu%on3constant interac%on3constantinterac%on3constantinterac%on3constantinterac%on3constantinterac%on3constant
Rules States3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraints
data structures
NLP+ AR1 for
stories
AR1 for txt games
framework
AR2 + NLP1 for
txt games
AR2 for stories
AR3 + NLP2 for
txt games
requirements
Setting
technology
enhanced learning
Climax
TERENCE case
study
Resolution
reflections
Story outline
Till 2007
CategoryAxis
AR
HMI
TEL
Game
0 4 8 12 16
Response: work areas
Amsterdam U. and CWI
FBK-irst
Free U. of Bolzano
From 2007
CategoryAxis
AR
HMI
TEL
Game
0 4 8 12 16
Response: work areas
Free U. of Bolzano
Possible explanation?
co-designgamification cooperative learning
HOW
WHY engagement design together inclusion
childrendesigners teachersWHO
WHAT
GACOCO
trees
tree puzzle
Gamification of protocol (tasks, subtasks and types of feedback)
Gamification (competition for cooperation)
MISSIONS
CHALLENGES
REWARDS
well
done!
Acknowledgments to
TERENCE colleagues and
schools
Current
colleagues and schools
DIARY FOR PRESENT
THE TERENCE BOOK
Setting
technology
enhanced learning
Climax
TERENCE case
study
Resolution
reflections
Story outline
?

Intelligent systems and learning centred design

  • 1.
    LEARNER CENTRED DESIGN INTELLIGENTSYSTEMS and Rosella Gennari http://www.inf.unibz.it/~gennari Distinguished Speakers Oxford Women in CS
  • 2.
  • 3.
  • 4.
    TECHNOLOGY ENHANCED LEARNING TechnologyEnhanced Learning (TEL) is the usage of technology for supporting a learning experience Herby we take a narrow view: intelligent TEL = Artificial Intelligence (AI) technology based products for supporting a learning experience
  • 5.
    TEL 4 LEARNINGEXPERIENCE How can we design technological products that supports their users’ learning experience?
  • 6.
    LET'S SEE EDUCATORS'VIEWPOINT... Maria Montessori (1870-1952) Paraphrasing her words, adequate tasks that come in a prepared environment, designed on top of the learner characteristics can effectively support the learner’s learning was the first Italian woman physician and educator, best known for Montessori pedagogy
  • 7.
    ousability of technology learningproducts opedagogical effectiveness of technology learning products TEL 4 LEARNING EXPERIENCE Adequate tasks that come in a prepared environment designed on top of the learner characteristics can effectively support learning
  • 8.
    HOW TO DESIGNUSABLE AND PEDAGOGICALLY EFFECTIVE TEL
  • 9.
    Based on UCDprocess diagram (© Tom Wellings) requirement specification designevaluation plan models + prototypes intermediate product final product HOW TO DESIGN USABLE AND PEDAGOGICALLY EFFECTIVE TEL USABILITY + P. EFFECTIVENESS
  • 10.
  • 11.
    TERENCE DESIGN Based on UCDprocess diagram (© Tom Wellings) requirement specification designevaluation plan models + prototypes intermediate products final products TERENCE was an FP7 TEL project blending user centred and evidence based design USABILITY + P. EFFECTIVENESS
  • 12.
    THE PROBLEM ‣ TERENCEdeveloped an adaptive learning system (ALS) that, via a learner GUI, recommends poor comprehenders - its learning material, i.e., books of stories and games - its learning tasks, i.e., reading and playing ‣ so as to stimulate their reading comprehension ‣ More than 10% of primary school children, older than 8, are diagnosed with deep text comprehension problems ‣ They are referred to as poor comprehenders
  • 13.
    THE TERENCE WORLD ad e q u a t e b o o k o f s t o r i e s s i g n i n a d e q u a t e s m a r t g a m e s r e w a r d
  • 14.
    TERENCE INTELLIGENT TEL PRODUCTS ALSLayerGUI Layer Learner Educator Expert Learner GUI Expert GUI Persistence Layer OpenRDF User Manager OpenRDF Story Manager OpenRDF Game Manager OpenRDF Visualisation Manager illustrations NPL Reasoner Adaptive Engine Visualisation Reasoning Module Annotation Module Visualisation Module game generation adaptation to learners
  • 15.
    Requirements Prototypes Analytic+ small ev. Int. prod. Analytic+large Fin. prod. TERENCE DESIGN
  • 16.
    TERENCE DESIGN Requirements Prototypes Analytic+ small ev. Int. prod. Analytic+large Fin. prod.
  • 17.
    DATA GATHERING METHODS ‣Data for designing the learning material and tasks were from - contextual inquiries with ‣ IT & UK diagnosis ‣ as well as IT & USA evidence- based medicine therapy experts - field studies with educators and primary school children 4510 ~ 500
  • 18.
    C H A R A C T E R I S T I C S Persona Name: Carol Age:8 Classroom: year 4 RC levels: low reading levels Rural/Urban: urban Deaf/hearing: deaf First Language: Italian Sign Language Cochlear Implantation (if deaf): yes Degree of hearing loss (if deaf): profound Motor skills (if deaf): average Summary of the class represented by this persona A younger deaf girl who is very enthusiastic about using new technology (such as iPhone and IPad) and who adores her Nintendo DS. Her reading RC levels are very low, but she reads together with her parents to learn new words and spelling. She also likes to do many other things such as drawing, taking care of her pets and going to the park. Quote “I really love Mario and Luigi. And I would love to have an iPhone and an IPad, like my dad.” Personality Open Role in classroom Active Role out of the classroom Active Console/Technology Carol and her sister watch TV after school. They like Tom & Jerry, Ben 10, Hello Kitty and Mickey Mouse Clubhouse. Carol sometimes uses the computer, but only to play minilab games. Her computer is in her bedroom, but her parents don’t allow her to use it all of the time. She can use it only one hour per day. Carol’s dad has an iPhone and an IPad, and Carol would really like to use those as well, but her dad tells her she is a bit too young. Carol likes watching him with his IPad and iPhone though. Carol doesn’t use a mobile phone. Carol plays games on the computer and on her Nintendo DS. She plays by herself. She likes the mini-clip games on the computer, and Mario Kart and brain training games on her DS. She likes games with non-photorealistic human avatars, and prefers fantasy avatars to animal avatars. Socio-Cultural Level of his/her own family Medium School performance Carol has sever reading problems. In her class she is below average in all activities but drawing, where she feels she can truly express her intimate feelings. Homework After school, Carol does her homework together with her mum. L I F E S T Y L E Outdoors Activities Carol often goes to the park with her mum. Indoors Activities Games on the DS Carol reads sometimes. Her mum and dad help her reading in the evening. She likes some of the stories they read together, but mostly, she wants to read because she has to learn new words and spelling. Carol likes drawing and taking care of her pets. Her mum often plays with her. Home activities Carol also likes to help her mum in the kitchen or in the garden. Sport activities Carol practices no specific sport. .
  • 19.
    SMART GAME REQUIREMENTS Whatfor Description Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games. Scheduling of reading and playing 1st silent reading; 2nd playing smart games; 3rd playing relaxing games Constraints on actions Learners should get faster, hence a game has a maximal resolution time Progress and feedback Monitor and give learners (1) visible idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time) Representation Production can be impaired hence promote resolution via visual representation and reasoning
  • 20.
    Requirements Prototypes Analytic+ small ev. Int. prod. Analytic+large Fin. prod. TERENCE DESIGN
  • 21.
    SMART GAME REQUIREMENTS Whatfor Description Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games. Scheduling of reading and playing 1st silent reading; 2nd playing smart games; 3rd playing relaxing games Constraints on actions Learners should get faster, hence a game has a maximal resolution time Progress and feedback Monitor and give learners (1) visible idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time) Representation Production can be impaired hence promote resolution via visual representation and reasoning
  • 22.
    who is theactor of … ? what does (a main) character do? when does … happen in relation to a central event? why does the central event happen?
  • 23.
    SMART GAME REQUIREMENTS Whatfor Description Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games. Scheduling of reading and playing 1st silent reading; 2nd playing smart games; 3rd playing relaxing games Constraints on actions Learners should get faster, hence a game has a maximal resolution time Progress and feedback Monitor and give learners (1) visible idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time) Representation Production can be impaired hence promote resolution via visual representation and reasoning
  • 24.
    points for eachsmart coins for all smartunlocked if read+play visual feedback
  • 25.
    Instructions Questions MotivationalInteraction Choices Choices for learner Fixed event Solutions Choices that are either correct (c) or wrong (w) Feedback Interaction Consistency Explanatory Solution Smart points Proportional to the learner’s ability in the game level Relaxing points Constant Avatar Happy/sad states Time solution constant interaction constant Rules States of the system, actions of the learner, constraints What for Description Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games. Scheduling of reading and playing 1st silent reading; 2nd playing smart games; 3rd playing relaxing games Constraints on actions Learners should get faster, hence a game has a maximal resolution time Progress and feedback Monitor and give learners (1) idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time) Representation Production can be impaired hence promote resolution via visual representation and reasoning
  • 26.
    Requirements Prototypes Analytic+ small ev. Int. prod. Analytic+large Fin. prod. TERENCE DESIGN
  • 27.
    EXAMPLE METHODS INTERENCE APPROACH WITH WHOM EXAMPLE METHODS WHEN analytical HMI experts or domain experts heuristic evaluation formative, summative expert evaluation cognitive walk-through small-scale learners observations formative think aloud large-scale learners field studies summative
  • 28.
    EXAMPLE METHODS INTERENCE APPROACH WITH WHOM EXAMPLE METHODS HOW analytical HMI experts or domain experts heuristic evaluation formative, summative expert evaluation cognitive walk-through small-scale learners observations formative think aloud large-scale learners field studies summative
  • 29.
    G1. interfaces follow general design guidelines G2.interfaces support the user’s next step to achieve a task G3. interfaces provide users with timely feedback Instructions are not under focus and cannot be easily read Game question and possible resolutions should be proximally close Game question and possible resolutions should be proximally close Evaluation of interfaces
  • 30.
    Requirements Prototypes Analytic+ small ev. Int. prod. Analytic+large Fin. prod. TERENCE DESIGN
  • 31.
    problem: 256 stories, each with~12 games Smart game design
  • 32.
    how can we automatisethe development of smart games via AI (and, hopefully, be efficient)? Smart game design
  • 33.
    enriched annotations story annotations ALS LayerGUILayer Learner Educator Expert Learner GUI Expert GUI illustrations NPL Reasoner Adaptive Engine Visualisation Reasoning Module Annotation Module Visualisation Module Semi-automated generation
  • 34.
    enriched annotations story text text text text annotations Semi-automated generation ALSLayerGUI Layer Learner Educator Expert Learner GUI Expert GUI illustrations NPL Reasoner Adaptive Engine Visualisation Reasoning Module Annotation Module Visualisation Module
  • 35.
    text text text text image image image image enrichedannotations story annotations Semi-automated generation
  • 36.
    games template visual text text text text image image image image enrichedannotations story annotations Semi-automated generation GUI Layer Learner Educator Expert Learner GUI Expert GUI Visualisation Visualisation Module
  • 37.
    text story text + visual games AUTOM.MANUAL AUTOM. Semi-automated generation
  • 38.
    Requirements Prototypes Analytic+ small ev. Int. prod. Analytic+large Fin. prod. TERENCE DESIGN
  • 39.
    APPROACH WITH WHOMEXAMPLE METHODS WHEN analytical HMI experts or domain experts heuristic evaluation formative, summative expert evaluation cognitive walk-through small-scale learners observations formative think aloud large-scale learners field studies summative EXAMPLE METHODS IN TERENCE
  • 40.
    APPROACH WITH WHOMEXAMPLE METHODS WHEN analytical HMI experts or domain experts heuristic evaluation formative, summative expert evaluation cognitive walk-through small-scale learners observations formative think aloud large-scale learners field studies summative EXAMPLE METHODS IN TERENCE
  • 41.
    LARGE-SCALE STUDY DESIGN Commondesign of the intervention with TERENCE: ‣how: pretest/posttest design, with experimental and control groups ‣hypothesis: TERENCE improves reading comprehension measured with standardized text comprehension tests ControlExperimental
  • 42.
    A 3-PHASE INTERVENTION ‣Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
  • 43.
    A 3-PHASE INTERVENTION ‣Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE ‣ Stimulation phase for experimental group with usage sessions so that each -lasts < 45 minutes for attention needs
  • 44.
    A 3-PHASE INTERVENTION ‣Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE ‣ Stimulation phase for experimental group with usage sessions so that each -lasts < 45 minutes for attention needs -and requires (1) reading
  • 45.
    A 3-PHASE INTERVENTION ‣Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE ‣ Stimulation phase for experimental group with usage sessions so that each -lasts < 45 minutes for attention needs -and requires (1) reading (2) playing smart
  • 46.
    ‣ Pre-test for(1) assessing txt comprehension, (2) initialising TERENCE ‣ Stimulation phase for experimental group with usage sessions so that each -lasts < 45 minutes for attention needs -and requires (1) reading (3) playing relaxing(2) playing smart A 3-PHASE INTERVENTION
  • 47.
    ‣ Post-test (pedagogicalonly) for re-assessing text comprehension ‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE ‣ Stimulation phase for experimental group with usage sessions so that each -lasts < 45 minutes for attention needs -and requires (1) reading (3) playing relaxing(2) playing smart A 3-PHASE INTERVENTION
  • 48.
    The experimental groupis of 344 learners: ‣ Avezzano: 270 learners: - 7-9 years old: 118 - 9-11 years old: 152 ‣ Pescina: 74 learners: - 7-9 years old: 37 - 9-11 years old: 37 ‣ They were tested (January-February), stimulated (March-May), and re-tested (may-June) EXPERIMENTAL GROUP IN IT
  • 49.
    Pre-post performances fortext comprehension (dependent variable) were as follows: ‣ Pescina: - pre: 14 poor comprehenders (20.59%) - post: 6 poor comprehenders (8.82%) ‣ Avezzano: - pre: 15 poor comprehenders (5.95%) - post: 2 poor comprehenders (0.79%) MAIN RESULTS IN IT Pre Pescina Avezzano 5,95% 20,59% ‣ Wilcoxon signed-rank test supports that differences are statistically significant - Pescina: z=-4.904, p<0.0001 - Avezzano: z=-2.266, p=0.0234
  • 50.
    EXAMPLE METHODS INTERENCE APPROACH WITH WHOM EXAMPLE METHODS HOW analytical HMI experts or domain experts heuristic evaluation formative, summative expert evaluation cognitive walk-through small-scale learners observations formative think aloud large-scale learners field studies summative
  • 51.
    EXPERT EVALUATION Experts ofpedagogy: 1 coordinator; 9 evaluators Sophie'comes'down'the'steps He had never been beaten before, since he only ever raced with kids who were smaller and slower than him. He wanted a rematch, so the two boys set off again. Ben was paddling as fast as he could, still he didn’t make it to the wall before Luke. It was completely unfair, he thought. Luke was so much faster. No sooner had they climbed out of the water, than he saw his sister coming down the steps. She was smiling at Ben and gave him a playful pat on the shoulder. She also gave Ben a friendly speech about winners and losers. revise selection of solutions revise selection of central event How-to: 1. each pair of evaluators read a story, and edited its games 2. the coordinator revised their work 3. a pair of evaluators was blindly assigned revised games, and another the manually created games Main edit tasks: (1) creation of missing games (~recall) (2) revision of games (~precision)
  • 52.
    From D4.2 andD4.3 technical annex overall assessment of generation text story text + visual games revision of Automated Reasoning (AR) selection of central events and solutions revision of Natural Language Processing (NLP) of text text text text text Edit tasks in details
  • 53.
    Requirements Prototypes Analytic+ small ev. Int. prod. Analytic+large Fin. prod. TERENCE DESIGN
  • 54.
    From D4.2 andD4.3 technical annex overall generation text story text + visual games AR selection of central events and solutions revision of NLP text text text text text Analyses of evaluation results
  • 55.
    AR selection ofcentral events for games: >Results: only in 15 out of 250 cases (6%), it was necessary to select a different central event than the automatically generated one From D4.2 and D4.3 technical annex >Implications for AR: none picked up Automated part evaluation-based re-design
  • 56.
    AR selection ofplausible solutions: >Results: out of 140 changes of selection of solutions, the majority was for wrong solutions - generate a wrong solution from correct one by changing participants, e.g., <correct_sentence id="2">
 The man ran and fell on the ground. </correct_sentence> <wrong_sentence id="2wh1">
 Peter ran and fell on the ground. </wrong_sentence> >Implications for WP4: new heuristics for wrong plausible solutions in the last part of Y3, From D4.2 and D4.3 technical annex Automated part evaluation-based re-design
  • 57.
    Overall generation: development times: >Resultsfor revision time: - 12’6” per game instance: ↑ 12’8” for time games ↓ 10’6” for who games >Results for creation time: - avg. 23” per game instance text story text + visual games From D4.2 and D4.3 technical annex >Implications for AR: the semi-automated development process seems to be promising for optimising development times Automated part evaluation-based re-design
  • 58.
    Game over 1st 2nd3 Sep. 2011 December 2012 September 2013 Sophie'comes'down'the'steps He had never been beaten before, since he only ever raced with kids who were smaller and slower than him. He wanted a rematch, so the two boys set off again. Ben was paddling as fast as he could, still he didn’t make it to the wall before Luke. It was completely unfair, he thought. Luke was so much faster. No sooner had they climbed out of the water, than he saw his sister coming down the steps. She was smiling at Ben and gave him a playful pat on the shoulder. She also gave Ben a friendly speech about winners and losers. revise selection of solutions revise selection of central event Requirements+for Description Dif$iculty*levels Macro*levels*for*learners: 4*entry:*character*games; 4*intermediate:*time*games; 4*top:*causality*games.* Scheduling*of* reading*and*playing 1st*silent*reading;* 2nd* playing* smart*games;*3rd*playing* relaxing*games Constraints*on* actions Learners* should* get*faster,* hence* a* game* has* a* maximal* resolution+time Progress*and* feedback Monitor* and* give* learners* (1)* idea* of* progress,* (2)* explanatory*feedback,*(3)*recall*their*attention*and*solicit+ them*to*give*a*resolution*(in*time) Representation Production*can*be* impaired*hence*promote*resolution*via* visual*representation+and+reasoning Instruc(ons Ques%onsQues%ons Mo%va%onalMo%va%onalMo%va%onalMo%va%onal Interac%onInterac%on Choices Choices3for3learnerChoices3for3learnerChoices3for3learnerChoices3for3learnerChoices3for3learner 3Fixed3event3Fixed3event3Fixed3event Solu(ons Choices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%ons Feedback Interac%on Consistency3(c/w)Consistency3(c/w)Consistency3(c/w) ExplanatoryExplanatoryExplanatory Solu%on Smart6points Propor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3level Relaxing6points ConstantConstantConstantConstantConstantConstantConstantConstant Avatar Happy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3states Time solu%on3constantsolu%on3constantsolu%on3constant interac%on3constantinterac%on3constantinterac%on3constantinterac%on3constantinterac%on3constant Rules States3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraints data structures NLP+ AR1 for stories AR1 for txt games framework AR2 + NLP1 for txt games AR2 for stories AR3 + NLP2 for txt games requirements
  • 59.
  • 60.
    Till 2007 CategoryAxis AR HMI TEL Game 0 48 12 16 Response: work areas Amsterdam U. and CWI FBK-irst Free U. of Bolzano
  • 61.
    From 2007 CategoryAxis AR HMI TEL Game 0 48 12 16 Response: work areas Free U. of Bolzano
  • 62.
  • 63.
    co-designgamification cooperative learning HOW WHYengagement design together inclusion childrendesigners teachersWHO WHAT GACOCO trees tree puzzle
  • 64.
    Gamification of protocol(tasks, subtasks and types of feedback)
  • 65.
    Gamification (competition forcooperation) MISSIONS CHALLENGES REWARDS well done!
  • 66.
    Acknowledgments to TERENCE colleaguesand schools Current colleagues and schools DIARY FOR PRESENT THE TERENCE BOOK
  • 67.