Presentation for EC-TEL 2015 paper: Loboda, T., Guerra, J., Hosseini, R., and Brusilovsky, P. (2014) Mastery Grids: An Open Source Social Educational Progress Visualization. In: S. de Freitas, C. Rensing, P. J. Muñoz Merino and T. Ley (eds.) Proceedings of 9th European Conference on Technology Enhanced Learning (EC-TEL 2014), Graz, Austria, September 16-19, 2014, pp. 235-248.

1. Mastery Grids: An Open Source
Social Educational Progress
Visualization
Tomasz D. Loboda, Julio Guerra,
Roya Hosseini, Peter Brusilovsky
PAWS Lab,
University of Pittsburgh

2. Overview
• The past
– Why we are doing it?
• The paper
– Mastery Grids and its evaluation
• Today’s state
– What we have done since submitting the paper?
• The future
– What are the plans and invitation to collaborate

3. The Past
• Why?
–Increase user performance
–Increase motivation and retention
• How?
–Adaptive Navigation Support
–Topic-based Adaptation
–Open Social Student Modeling

4. Adaptive Link Annotation: InterBook
1. Concept role
2. Current concept state
3. Current section state
4. Linked sections state
4
3
2
1
√

5. Questions of
the current
quiz, served
by QuizPACK
List of annotated
links to all quizzes
available for a
student in the
current course
Refresh
and help
icons
QuizGuide = Topic-Based ANS

6. Topic-Based Adaptation
Concept
A
Concept
B
Concept
C
 Each topic is associated with a number of
educational activities to learn about this topic
 Each activity classified under 1 topic

7. QuizGuide: Adaptive Annotations
• Target-arrow abstraction:
– Number of arrows – level of
knowledge for the specific
topic (from 0 to 3).
Individual, event-based
adaptation.
– Color Intensity – learning
goal (current, prerequisite
for current, not-relevant,
not-ready). Group, time-
based adaptation.
 Topic–quiz organization:

8. QuizGuide: Success Rate

9. QuizGuide: Motivation
Average activity
0
50
100
150
200
250
300
2002 2003 2004
Average num. of
sessions
0
5
10
15
20
2002 2003 2004
Average course
coverage
0%
10%
20%
30%
40%
50%
60%
2002 2003 2004
 Within the same class QuizGuide session were much
longer than QuizPACK sessions: 24 vs. 14 question
attempts at average.
 Average Knowledge Gain for the class rose from 5.1 to 6.5

10. • Topic-Based interface organization is
familiar, matches the course
organization, and provides a
compromise between too-much and
too-little
• Two-way adaptive navigation
support guides to the right topic
• Open student model provides clear
overview of the progress
Topic-Based ANS: Success Recipes

11. Concept-based student modeling
Example 2 Example M
Example 1
Problem 1
Problem 2 Problem K
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N
Examples
Problems
Concepts

12. NavEx = Concept-Based ANS

13. • To investigate possible influence of concept-based
adaptation in the present of topic-based adaptation we
developed two versions of QuizGuide:
Topic-based Topic-based+Concept-Based
Concept-based vs Topic-based ANS

14. Social Guidance
• Concept-based and topic-based navigation support
work well to increase success and motivation
• Knowledge-based approaches require some
knowledge engineering – concept/topic models,
prerequisites, time schedule
• In our past work we learned that social navigation –
“wisdom” extracted from the work of a community
of learners – might replace knowledge-based
guidance
• Social wisdom vs. knowledge engineering

15. Open Social Student Modeling
• Key ideas
– Assume simple topic-based design
– Show topic- and content- level knowledge progress of
a student in contrast to the same progress of the class
• Main challenge
– How to design the interface to show student and class
progress over topics?
– We went through several attempts…

16. QuizMap
16

17. Parallel Introspective Views
17

18. Progressor
18

19. • Topic organization should follow the
natural progress or topics in the
course
• Clear comparison between “me” and
“group”
• Ability to compare with individual
peers, not only the group
• Privacy management
OSLM: Success Recipes

20. The Value of OSLM
205.73
113.05
80.81
125.5
0
50
100
150
200
250
Attempts
Progressor
QuizJET+IV
QuizJET+Portal
JavaGuide
68.39%
71.35%
42.63%
58.31%
0.00%
20.00%
40.00%
60.00%
80.00%
Success Rate
Progressor
QuizJET+IV
QuizJET+Portal
JavaGuide

21. The Secret

22. MasteryGrids
• Adaptive Navigation Support
• Topic-based Adaptation
• Open Social Student Modeling
• Social Educational Progress Visualization
• Multiple Content Types
• Open Source
• Concept-Based Recommendation
• Multiple Groups

28. The Study
• Fall 2013, 3 classes
• Java Programming
– 19 topics,75 examples and 94 QiuzJet problems
• Databases
– 19 topics, 64 examples, 46 SQL-Knot questions.
• Offered as an addition to the traditional portal
access (called “Links” in the paper)
• Incentive: 5 points out of 100 to solve 15+
problems – either Links or MG

29. Accessing MG interface from Links

30. Usage

31. Usage Groups in Java Class
• (Z) 4 students did not use either of the tools
• (L1) 2 students used Links only and never used
visualization
• (L2) 3 students used Links for content access, loaded the
visualization, did not use it
• (L.MG1) 16 students used Links for content access,
interactively explored MG, did not use it for content
access
• (L.MG2) 6 students used both Links and Mastery Grids
for content access
• (MG) 5 students used exclusively Mastery Grids for
content access.

32. Group-Based Analysis
• 5 (L1+L2) had little to no use of Mastery Grids
• 26 (L.MG1+L.MG2+MG) used it considerably
• Students who used the visualization seemed to
be more engaged with self-study content
– answered more questions
– tried more examples
– inspected more example line comments
– got a higher correct question answer ratio.

34. Subjective Responses
• High level of “good” responses
• A number of “poor” responses to specific
features which guided our work in 2014

35. More MG Activity = Better Grade?
• How?
– Pooled data from all three courses
– Only students who logged in at least once
– Fitted four linear mixed models
• Separately considered content and interface
– Only content access through Links or MG
– All activity with MG

36. What we are doing now?
• Enhanced Interface after two semesters of
learning and student feedback
• Easy authoring to define “your course”
• Exploring more advanced guidance and
modeling approaches based on large volume of
social data
• Interface and cultural studies in a wide variety of
classes from US to Nigeria
– Interested to be a pilot site? Write to peterb@pitt.edu

37. MG flexibility
• Parameters to set the visualization:
– show hide toolbar or any of its elements
– set the (sub) groups: top N, other sub groups
– preset values (for example load individual view by
default)
– enable/disable recommendation
• Parameters can be specified by group or
by user

39. Course Authoring Interface
A label showing that
you are the creator
of the course
domain
Institution
code
Course
code Course
title
Number of
Groups
using this
course
Creator
name

40. Acknowledgements
• Past work on ANS and OSLM
– Sergey Sosnovsky
– Michael Yudelson
– Sharon Hsiao
• Pitt “Innovation in Education” grant
• NSF Grants
– EHR 0310576
– IIS 0426021
– CAREER 0447083
• ADL “PAL” grant to build MasteryGrids

41. Try It!
• GitHub link Twitted
• Brusilovsky, P., Sosnovsky, S., and Yudelson, M. (2009)
Addictive links: The motivational value of adaptive link annotation.
New Review of Hypermedia and Multimedia 15 (1), 97-118.
• Hsiao, I.-H., Sosnovsky, S., and Brusilovsky, P. (2010) Guiding
students to the right questions: adaptive navigation support in an E-
Learning system for Java programming. Journal of Computer Assisted
Learning 26 (4), 270-283.
• Hsiao, I.-H., Bakalov, F., Brusilovsky, P., and König-Ries, B.
(2013) Progressor: social navigation support through open social
student modeling. New Review of Hypermedia and Multimedia [PDF]
Read About It!