What are intelligent systems for enhancing learning? How do we design them so as to enhance a certain learner's experience? This talk starts with pedagogy theories focusing on designing learning experiences according to learner characteristics. Then it boldly connects them to specific design methodologies, which aims at producing usable and pedagogically effective intelligent systems for delivering enhanced learning material and experiences. Then the real pulp starts: how we designed the intelligent system of the TERENCE FP7 EU project for designing adequate learning material, specifically, smart games for reasoning about stories and adequate to the needs of poor comprehenders. The talk concludes reflecting upon the speaker's change in research areas possibly due to several factors, such as personal reasons, interests and project-dependent requirements. The on-going work is also briefly illustrated, that is, how to design smart games not only for learners but also with learners, engaging and including all.

1. LEARNER CENTRED DESIGN
INTELLIGENT SYSTEMS
and
Rosella Gennari
http://www.inf.unibz.it/~gennari
Distinguished Speakers
Oxford Women in CS

2. Setting
technology
enhanced learning
Climax
TERENCE case
study
Resolution
reflections
Story outline

3. Setting
technology
enhanced learning
Climax
TERENCE case
study
Resolution
reflections
Story outline

4. TECHNOLOGY ENHANCED LEARNING
Technology Enhanced Learning
(TEL) is the usage of technology
for supporting a learning
experience
Herby we take a narrow view:
intelligent TEL =
Artificial Intelligence (AI)
technology based products
for supporting a learning
experience

5. TEL 4 LEARNING EXPERIENCE
How can we design
technological products
that supports their users’
learning experience?

6. LET'S SEE EDUCATORS' VIEWPOINT...
Maria Montessori (1870-1952)
Paraphrasing her words, adequate
tasks that come in a prepared
environment, designed on top of the
learner characteristics can
effectively support the learner’s
learning
was the first Italian woman physician
and educator, best known for
Montessori pedagogy

7. ousability of technology
learning products
opedagogical effectiveness
of technology
learning products
TEL 4 LEARNING EXPERIENCE
Adequate tasks that come in a prepared
environment designed on top of the learner
characteristics can effectively support learning

8. HOW TO DESIGN USABLE AND
PEDAGOGICALLY EFFECTIVE TEL

9. Based on UCD process diagram (© Tom Wellings)
requirement
specification
designevaluation
plan
models +
prototypes
intermediate
product
final
product
HOW TO DESIGN USABLE AND
PEDAGOGICALLY EFFECTIVE TEL
USABILITY +
P. EFFECTIVENESS

10. Setting
technology
enhanced learning
Climax
TERENCE case
study
Resolution
reflections
Story outline

11. TERENCE
DESIGN
Based on UCD process diagram (© Tom Wellings)
requirement
specification
designevaluation
plan
models +
prototypes
intermediate
products
final
products
TERENCE was an FP7 TEL project
blending user centred and evidence based
design
USABILITY +
P. EFFECTIVENESS

12. THE PROBLEM
‣ TERENCE developed an adaptive learning system (ALS) that,
via a learner GUI, recommends poor comprehenders
- its learning material, i.e., books of stories and games
- its learning tasks, i.e., reading and playing
‣ so as to stimulate their reading comprehension
‣ More than 10% of primary school children, older than 8,
are diagnosed with deep text comprehension problems
‣ They are referred to as poor comprehenders

13. THE TERENCE WORLD
a d e q u a t e
b o o k o f
s t o r i e s
s i g n i n
a d e q u a t e
s m a r t
g a m e s
r e w a r d

14. TERENCE INTELLIGENT TEL
PRODUCTS
ALS LayerGUI Layer
Learner
Educator
Expert
Learner GUI
Expert GUI
Persistence Layer
OpenRDF
User
Manager
OpenRDF
Story
Manager
OpenRDF
Game
Manager
OpenRDF
Visualisation
Manager
illustrations
NPL
Reasoner
Adaptive
Engine
Visualisation
Reasoning
Module
Annotation
Module
Visualisation
Module
game
generation
adaptation to
learners

15. Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE
DESIGN

16. TERENCE
DESIGN
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.

17. DATA GATHERING METHODS
‣ Data for designing the learning
material and tasks were from
- contextual inquiries with
‣ IT & UK diagnosis
‣ as well as IT & USA evidence-
based medicine therapy experts
- field studies with educators and
primary school children
4510
~ 500

18. C
H
A
R
A
C
T
E
R
I
S
T
I
C
S
Persona Name: Carol
Age: 8
Classroom: year 4
RC levels: low reading levels
Rural/Urban: urban
Deaf/hearing: deaf
First Language: Italian Sign Language
Cochlear Implantation (if deaf): yes
Degree of hearing loss (if deaf): profound
Motor skills (if deaf): average
Summary of the class
represented by this
persona
A younger deaf girl who is very enthusiastic about using new
technology (such as iPhone and IPad) and who adores her
Nintendo DS. Her reading RC levels are very low, but she reads
together with her parents to learn new words and spelling. She
also likes to do many other things such as drawing, taking care of
her pets and going to the park.
Quote “I really love Mario and Luigi. And I would love to have an iPhone
and an IPad, like my dad.”
Personality Open
Role in classroom Active
Role out of the
classroom
Active
Console/Technology Carol and her sister watch TV after school. They like Tom & Jerry,
Ben 10, Hello Kitty and Mickey Mouse Clubhouse.
Carol sometimes uses the computer, but only to play minilab
games. Her computer is in her bedroom, but her parents don’t
allow her to use it all of the time. She can use it only one hour per
day. Carol’s dad has an iPhone and an IPad, and Carol would
really like to use those as well, but her dad tells her she is a bit
too young. Carol likes watching him with his IPad and iPhone
though.
Carol doesn’t use a mobile phone.
Carol plays games on the computer and on her Nintendo DS.
She plays by herself. She likes the mini-clip games on the
computer, and Mario Kart and brain training games on her DS.
She likes games with non-photorealistic human avatars, and
prefers fantasy avatars to animal avatars.
Socio-Cultural Level of
his/her own family
Medium
School performance Carol has sever reading problems. In her class she is below
average in all activities but drawing, where she feels she can truly
express her intimate feelings.
Homework After school, Carol does her homework together with her mum.
L
I
F
E
S
T
Y
L
E
Outdoors Activities Carol often goes to the park with her mum.
Indoors Activities Games on the DS
Carol reads sometimes. Her mum and dad help her reading in the
evening. She likes some of the stories they read together, but
mostly, she wants to read because she has to learn new words
and spelling.
Carol likes drawing and taking care of her pets.
Her mum often plays with her.
Home activities Carol also likes to help her mum in the kitchen or in the garden.
Sport activities Carol practices no specific sport.
.

19. SMART GAME REQUIREMENTS
What for Description
Difficulty levels Macro levels for learners:
- entry: character games;
- intermediate: time games;
- top: causality games.
Scheduling of reading
and playing
1st silent reading; 2nd playing smart games; 3rd playing
relaxing games
Constraints on actions Learners should get faster, hence a game has a maximal
resolution time
Progress and feedback Monitor and give learners (1) visible idea of progress, (2)
explanatory feedback, (3) recall their attention and solicit
them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via
visual representation and reasoning

20. Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE
DESIGN

21. SMART GAME REQUIREMENTS
What for Description
Difficulty levels Macro levels for learners:
- entry: character games;
- intermediate: time games;
- top: causality games.
Scheduling of reading
and playing
1st silent reading; 2nd playing smart games; 3rd playing
relaxing games
Constraints on actions Learners should get faster, hence a game has a maximal
resolution time
Progress and feedback Monitor and give learners (1) visible idea of progress, (2)
explanatory feedback, (3) recall their attention and solicit
them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via
visual representation and reasoning

22. who is the actor of … ? what does (a main) character do?
when does … happen in relation to a central
event?
why does the central event happen?

23. SMART GAME REQUIREMENTS
What for Description
Difficulty levels Macro levels for learners:
- entry: character games;
- intermediate: time games;
- top: causality games.
Scheduling of reading
and playing
1st silent reading; 2nd playing smart games; 3rd playing
relaxing games
Constraints on actions Learners should get faster, hence a game has a maximal
resolution time
Progress and feedback Monitor and give learners (1) visible idea of progress, (2)
explanatory feedback, (3) recall their attention and solicit
them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via
visual representation and reasoning

24. points for each smart
coins for all smartunlocked if read+play
visual feedback

25. Instructions Questions Motivational Interaction
Choices Choices for learner Fixed event
Solutions Choices that are either correct (c) or wrong (w)
Feedback Interaction Consistency Explanatory Solution
Smart points Proportional to the learner’s ability in the game level
Relaxing
points
Constant
Avatar Happy/sad states
Time solution constant interaction constant
Rules States of the system, actions of the learner, constraints
What for Description
Difficulty levels Macro levels for learners:
- entry: character games;
- intermediate: time games;
- top: causality games.
Scheduling of
reading and
playing
1st silent reading; 2nd playing smart games; 3rd
playing relaxing games
Constraints on
actions
Learners should get faster, hence a game has a
maximal resolution time
Progress and
feedback
Monitor and give learners (1) idea of progress, (2)
explanatory feedback, (3) recall their attention and
solicit them to give a resolution (in time)
Representation Production can be impaired hence promote resolution
via visual representation and reasoning

26. Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE
DESIGN

27. EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS WHEN
analytical
HMI experts or
domain experts
heuristic evaluation
formative,
summative
expert evaluation
cognitive walk-through
small-scale learners
observations
formative
think aloud
large-scale learners field studies summative

28. EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS HOW
analytical
HMI experts or
domain experts
heuristic evaluation
formative,
summative
expert evaluation
cognitive walk-through
small-scale learners
observations
formative
think aloud
large-scale learners field studies summative

29. G1. interfaces
follow general
design
guidelines
G2. interfaces
support the
user’s next step
to achieve a
task
G3. interfaces
provide users
with timely
feedback
Instructions are not
under focus and
cannot be easily read
Game question and
possible resolutions should
be proximally close
Game question and
possible resolutions should
be proximally close
Evaluation of interfaces

30. Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE
DESIGN

31. problem:
256 stories,
each with ~12 games
Smart game design

32. how
can we automatise the
development of smart games via AI (and,
hopefully, be efficient)?
Smart game design

33. enriched annotations
story
annotations
ALS LayerGUI Layer
Learner
Educator
Expert
Learner GUI
Expert GUI
illustrations
NPL
Reasoner
Adaptive
Engine
Visualisation
Reasoning
Module
Annotation
Module
Visualisation
Module
Semi-automated generation

34. enriched annotations
story
text
text
text
text
annotations
Semi-automated generation
ALS LayerGUI Layer
Learner
Educator
Expert
Learner GUI
Expert GUI
illustrations
NPL
Reasoner
Adaptive
Engine
Visualisation
Reasoning
Module
Annotation
Module
Visualisation
Module

35. text
text
text
text
image
image image image
enriched annotations
story
annotations
Semi-automated generation

36. games
template
visual
text
text
text
text
image
image image image
enriched annotations
story
annotations
Semi-automated generation
GUI Layer
Learner
Educator
Expert
Learner GUI
Expert GUI
Visualisation
Visualisation
Module

37. text
story
text + visual
games
AUTOM. MANUAL AUTOM.
Semi-automated generation

38. Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE
DESIGN

39. APPROACH WITH WHOM EXAMPLE METHODS WHEN
analytical
HMI experts or
domain experts
heuristic evaluation
formative,
summative
expert evaluation
cognitive walk-through
small-scale learners
observations
formative
think aloud
large-scale learners field studies summative
EXAMPLE METHODS IN TERENCE

40. APPROACH WITH WHOM EXAMPLE METHODS WHEN
analytical
HMI experts or
domain experts
heuristic evaluation
formative,
summative
expert evaluation
cognitive walk-through
small-scale learners
observations
formative
think aloud
large-scale learners field studies summative
EXAMPLE METHODS IN TERENCE

41. LARGE-SCALE STUDY DESIGN
Common design of the intervention with TERENCE:
‣how: pretest/posttest design, with experimental and control groups
‣hypothesis: TERENCE improves reading comprehension measured
with standardized text comprehension tests
ControlExperimental

42. A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE

43. A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs

44. A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
-and requires (1) reading

45. A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
-and requires (1) reading (2) playing smart

46. ‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
-and requires (1) reading (3) playing relaxing(2) playing smart
A 3-PHASE INTERVENTION

47. ‣ Post-test (pedagogical only) for re-assessing text comprehension
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
-and requires (1) reading (3) playing relaxing(2) playing smart
A 3-PHASE INTERVENTION

48. The experimental group is of 344 learners:
‣ Avezzano: 270 learners:
- 7-9 years old: 118
- 9-11 years old: 152
‣ Pescina: 74 learners:
- 7-9 years old: 37
- 9-11 years old: 37
‣ They were tested (January-February), stimulated (March-May),
and re-tested (may-June)
EXPERIMENTAL GROUP IN IT

49. Pre-post performances for text comprehension
(dependent variable) were as follows:
‣ Pescina:
- pre: 14 poor comprehenders (20.59%)
- post: 6 poor comprehenders (8.82%)
‣ Avezzano:
- pre: 15 poor comprehenders (5.95%)
- post: 2 poor comprehenders (0.79%)
MAIN RESULTS IN IT
Pre
Pescina Avezzano
5,95%
20,59%
‣ Wilcoxon signed-rank test supports that differences
are statistically significant
- Pescina: z=-4.904, p<0.0001
- Avezzano: z=-2.266, p=0.0234

50. EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS HOW
analytical
HMI experts or
domain experts
heuristic evaluation
formative,
summative
expert evaluation
cognitive walk-through
small-scale learners
observations
formative
think aloud
large-scale learners field studies summative

51. EXPERT EVALUATION
Experts of pedagogy: 1 coordinator; 9
evaluators
Sophie'comes'down'the'steps
He had never been beaten before, since he
only ever raced with kids who were
smaller and slower than him.
He wanted a rematch, so the two boys set
off again. Ben was paddling as fast as he
could, still he didn’t make it to the wall
before Luke. It was completely unfair, he
thought. Luke was so much faster. No
sooner had they climbed out of the water,
than he saw his sister coming down the
steps. She was smiling at Ben and gave
him a playful pat on the shoulder. She also
gave Ben a friendly speech about winners
and losers.
revise selection of
solutions
revise selection of
central event
How-to:
1. each pair of evaluators read a story,
and edited its games
2. the coordinator revised their work
3. a pair of evaluators was blindly
assigned revised games, and another
the manually created games
Main edit tasks:
(1) creation of missing games (~recall)
(2) revision of games (~precision)

52. From D4.2 and D4.3 technical annex
overall assessment of
generation
text
story
text + visual
games
revision of Automated
Reasoning (AR) selection
of central events and
solutions
revision of Natural
Language
Processing (NLP) of
text
text
text text text
Edit tasks in details

53. Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE
DESIGN

54. From D4.2 and D4.3 technical annex
overall generation
text
story
text + visual
games
AR selection of central
events and solutions
revision of NLP text
text
text text text
Analyses of evaluation results

55. AR selection of central events for games:
>Results: only in 15 out of 250 cases (6%), it was necessary
to select a different central event than the automatically
generated one
From D4.2 and D4.3 technical annex
>Implications for AR:
none picked up
Automated part evaluation-based re-design

56. AR selection of plausible solutions:
>Results: out of 140 changes of selection of solutions, the majority
was for wrong solutions
- generate a wrong solution from
correct one by changing participants,
e.g.,
<correct_sentence id="2">
The man ran and fell on the ground.
</correct_sentence>
<wrong_sentence id="2wh1">
Peter ran and fell on the ground.
</wrong_sentence>
>Implications for WP4: new heuristics for wrong plausible solutions in
the last part of Y3,
From D4.2 and D4.3 technical annex
Automated part evaluation-based re-design

57. Overall generation: development
times:
>Results for revision time:
- 12’6” per game instance:
↑ 12’8” for time games
↓ 10’6” for who games
>Results for creation time:
- avg. 23” per game instance
text
story
text + visual
games
From D4.2 and D4.3 technical annex
>Implications for AR: the semi-automated development process seems to
be promising for optimising development times
Automated part evaluation-based re-design

58. Game over
1st 2nd 3
Sep. 2011 December 2012 September 2013
Sophie'comes'down'the'steps
He had never been beaten before, since he
only ever raced with kids who were
smaller and slower than him.
He wanted a rematch, so the two boys set
off again. Ben was paddling as fast as he
could, still he didn’t make it to the wall
before Luke. It was completely unfair, he
thought. Luke was so much faster. No
sooner had they climbed out of the water,
than he saw his sister coming down the
steps. She was smiling at Ben and gave
him a playful pat on the shoulder. She also
gave Ben a friendly speech about winners
and losers.
revise selection of
solutions
revise selection of
central event
Requirements+for Description
Dif$iculty*levels Macro*levels*for*learners:
4*entry:*character*games;
4*intermediate:*time*games;
4*top:*causality*games.*
Scheduling*of*
reading*and*playing
1st*silent*reading;* 2nd* playing* smart*games;*3rd*playing*
relaxing*games
Constraints*on*
actions
Learners* should* get*faster,* hence* a* game* has* a* maximal*
resolution+time
Progress*and*
feedback
Monitor* and* give* learners* (1)* idea* of* progress,* (2)*
explanatory*feedback,*(3)*recall*their*attention*and*solicit+
them*to*give*a*resolution*(in*time)
Representation Production*can*be* impaired*hence*promote*resolution*via*
visual*representation+and+reasoning
Instruc(ons Ques%onsQues%ons Mo%va%onalMo%va%onalMo%va%onalMo%va%onal Interac%onInterac%on
Choices Choices3for3learnerChoices3for3learnerChoices3for3learnerChoices3for3learnerChoices3for3learner 3Fixed3event3Fixed3event3Fixed3event
Solu(ons Choices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%ons
Feedback Interac%on Consistency3(c/w)Consistency3(c/w)Consistency3(c/w) ExplanatoryExplanatoryExplanatory Solu%on
Smart6points Propor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3level
Relaxing6points ConstantConstantConstantConstantConstantConstantConstantConstant
Avatar Happy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3states
Time solu%on3constantsolu%on3constantsolu%on3constant interac%on3constantinterac%on3constantinterac%on3constantinterac%on3constantinterac%on3constant
Rules States3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraints
data structures
NLP+ AR1 for
stories
AR1 for txt games
framework
AR2 + NLP1 for
txt games
AR2 for stories
AR3 + NLP2 for
txt games
requirements

59. Setting
technology
enhanced learning
Climax
TERENCE case
study
Resolution
reflections
Story outline

60. Till 2007
CategoryAxis
AR
HMI
TEL
Game
0 4 8 12 16
Response: work areas
Amsterdam U. and CWI
FBK-irst
Free U. of Bolzano

61. From 2007
CategoryAxis
AR
HMI
TEL
Game
0 4 8 12 16
Response: work areas
Free U. of Bolzano

62. Possible explanation?

63. co-designgamification cooperative learning
HOW
WHY engagement design together inclusion
childrendesigners teachersWHO
WHAT
GACOCO
trees
tree puzzle

64. Gamification of protocol (tasks, subtasks and types of feedback)

65. Gamification (competition for cooperation)
MISSIONS
CHALLENGES
REWARDS
well
done!

66. Acknowledgments to
TERENCE colleagues and
schools
Current
colleagues and schools
DIARY FOR PRESENT
THE TERENCE BOOK

67. Setting
technology
enhanced learning
Climax
TERENCE case
study
Resolution
reflections
Story outline
?