Presentation at UMAP 2017 - Fine-Grained Open Learner Models: Complexity Versus Support

1. Fine-­‐Grained
Open
Learner
Models:
Complexity
Versus
Support
Julio
Guerra-­‐Hollstein
Ins>tuto
de
Informá>ca,
Universidad
Asutral
de
Chile
School
of
Informa>on
Sciences,
University
of
PiHsburgh
Jordan
Barría-­‐Pineda
School
of
Informa>on
Sciences
University
of
PiHsburgh
Chris>an
D.
Schunn
Learning
Research
and
Development
Center,
University
of
PiHsburgh
Susan
Bull
Ins>tute
of
Educa>on,
University
College
London
Peter
Brusilovsky
School
of
Informa>on
Sciences
University
of
PiHsburgh

2. OLMs
2

3. Mastery
Grids
OLM
3

4. Mastery
Grids
OLM
4
Which
ac>vity
is
about
for
loop
using
+=
operator?

5. Learner,
Domain
and
Content
Models
5
Topics
Knowledge
Components
(KC)
Ac>vi>es
concatena>on
int
+=
AND
++
Opera>ons
Variables
Loops
...
for

6. General
topics
Knowledge
Components (KCs)
or Concepts
Rich-­‐
OLM
Goal
Finding
knowledge
holes
Support
ac>vity
naviga>on
Simple
and
easy
to
use
Limited
guidance
Limited
metacogni>ve
support
Complex
Complement
Mastery
Grids
with
a
fine-­‐grained
view
of
the
LM,
balancing
support
and
complexity

7. Study
1
7

8. Study
1
(N=42)
8

9. Rich-­‐OLM
9
hHps://youtu.be/lJZG4WEF4-­‐8

10. Study
2
10
Basic
Rich-­‐OLM
(KC)
Adds
interpreta>on
aid
-­‐
gauge
(KCG)
Adds
more
informa>on
–Social
Comparison
(KCS)

11. Study
2
• Within-­‐subjects design
• Pretest
• Task: select the best activity to maximize
mastery of the target topic
• Task Questionnaire: usefulness, satisfaction
• NASA TLX (mental demand, performance,
frustration, effort).
• Final Questionaire: rank designs, preferences
• Log variables: diRiculty of activity selected,
activities browsed, times.
11

12. Study
2
-­‐
Results
(N=29)
12
(lower
is
beHer)

13. Study
2
-­‐
Results
(N=29)
13
?

14. Conclusions
• More
details
=>
more
complexity
• Complexity
maHers,
as
shown
by
adding
Social
Comparison
features
in
Rich-­‐OLM
• Complexity
problem
can
be
addressed
with
visual
elements
aiding
the
interpreta>on
of
the
data
(Gauge)
14

15. 15

16. Study
1
-­‐
Results
(N=42)
16
Motivates to explore further!
Awareness of the content of topic!
Understand content relations!
Easy to understand!

17. Study
1
-­‐
Results
(N=42)
17

18. Study
2
-­‐
Results
(N=29)
18
Survey
(low
)
1
-­‐
7
(high)
higher
is
beHer
TLX:
(low)
0
-­‐
1
(high)
lower
is
beHer

19. Study
2
-­‐
Results
(N=29)
• Factor analyses of task survey grouped answers in
3 factors:
– USEFUL (to choose relevant, avoid easier, to learn)
– CRITICAL (criticalto do task right and quick)
– HELPLESS (reverse measure: not helpful, led to useless)
• USEFUL is lower in KCS (with Social features) and
tend to be higher in KCG (gauge)
19

20. Study
2
-­‐
Results
(N=29)
20

21. Study
2
-­‐
Results
(N=29)
21

22. Study
2
-­‐
Results
(N=29)
Associa'ons
behaviour
–
percep'ons
more
mouseover
~
low
confidence
in
task
~
system
less
helpful
to
avoid
hard
ac>vi>es
~
lower
frustra>on
selec>on
of
more
difficult
ac>vi>es
~
not
useless
(helpful)
more
aHempts
to
select
an
ac>vity
~
less
frustra>on
opening
more
ac>vi>es
~
lower
perce>on
of
failure
(peformance
TLX)
22

23. Study
2
-­‐
Results
(N=29)
23
(lower
is
beHer)

24. Conclusions
• More
details
=>
more
complexity
• Complexity
maHers,
as
shown
by
adding
Social
Comparison
features
in
Rich-­‐OLM
• Complexity
problem
can
be
addressed
with
visual
elements
aiding
the
interpreta>on
of
the
data
(Gauge)
24

25. 25

26. Study
2
–
Task
Ques>onnaire
26

27. Study
2
–
Task
Ques>onnaire
27
NASA
TLX

28. Study
2
–
Final
Ques>onnaire
28

29. Study
2
–
Final
Ques>onnaire
29

30. 30
Rich
OLM

31. Rich
OLM
31

32. Rich-­‐OLM
32

33. Gauge
33

34. Social
Comparison
for
KC
level
34