An adaptive learning system called Mastery Grids was created to increase student engagement with online educational content by incorporating personalized and social adaptive features. Mastery Grids uses open learner modeling to display a student's knowledge progress compared to their peers, adaptive navigation support to guide students to relevant activities, and concept-based recommendations of content. A study found that Mastery Grids significantly increased student success rates, time spent engaging with content, and learning compared to non-adaptive systems. Further research added direct recommendations to Mastery Grids and found they increased transparency and led to more efficient learning when explanations of recommendations were provided through the open learner model visualizations.

1. Personalized Online Practice
Systems for Learning
Programming
Peter Brusilovsky with:
Sergey Sosnovsky, Michael Yudelson, Sharon Hsiao,
Julio Guerra, Yun Huang, Roya Hosseini, Jordan
Barria-Pineda, Kamil Akhuseyinoglu
School of Computing and Information,
University of Pittsburgh

2. Why adaptive
practice?

3. • Online learning system
should engage student
in meaningful learning
activities
– Interact with worked
examples
– Explore a simulation
– Work on solving
problems
– Receive feedbackImage credit: http://merchandisingblog.inspire.ca/find-the-hidden-treasure/
People Learn through Activities

4. • Each student need
different activities
(kind, amount, order)
• An adaptive learning
system could use data
about each student
(knowledge, goals, …)
to guide them to the
next most relevant
activity
People Learn Differently

5. • Assessment-based
• Same for all
• Not enough doing
• Weak feedback loop
• High threshold
– From reading to
complex problems
– Many are not ready
Labs/Homework Don’t Cover the Need

6. CS Education Research to the Rescue
• Improving homework
– Better IDEs, Autograding, Extended feedback
• Beyond homework: “smart” practice content
– Program visualization (i.e., Python Tutor)
– Practice problems (i.e, Coding Bat)
– Worked examples (i.e., WebEx)

7. What is Missing?
EngagementPersonalization

8. The Problem of Engagement
• Great free content and top teachers are not
enough to engage students
• Peter Norvig: Motivation and engagement are
key problems for MOOCs
• A lot of great practice content
– Works perfectly in lab studies, great gains
– Released to students free use to enhance learning
– No impact – students do not use it

9. MOOC Completion Rate
Classic loop user modeling - adaptation in adaptive systems
http://www.katyjordan.com/MOOCproject.html

10. Recipes for Personalized Engaging Practice
• Adaptive navigation support
• Open learner modeling
• Social comparison
• Knowledge/opportunity visualization
• Content recommendation
– Proactive
– Remedial
– Explainable

11. QuizPACK: Code Tracing Exercises
• QuizPACK: Quizzes for
Parameterized Assessment of
C Knowledge
• Each question is a pattern of a
simple C program. When it is
delivered to a student the
special parameter is
dynamically instantiated by a
random value within the pre-
assigned borders.
• Used mostly as a self-
assessment tool in two C-
programming courses

12. QuizPACK: Value and Problems
• Good news:
– activity with QuizPACK significantly correlated with
student performance in classroom quizzes
– Knowledge gain rose from 1.94 to 5.37
• But:
– Low success rate - below 40%
– The system is under-used (used less than it deserves)
• Less than 10 sessions at average
• Average Course Coverage below 40%
– We need personalization and engagement!

13. Engaging Known Recepies: OLM + ANS
• Open Learner Modeling
– Increases motivations
– Support self-organized learning
• Adaptive navigation support
– Lower navigation overhead
• Access the content at the right time
• Find relevant information faster
– Better learning outcomes

14. Questions of
the current
quiz, served
by QuizPACK
List of annotated
links to all quizzes
available for a
student in the
current course
Refresh
and help
icons
QuizGuide = QuizPACK+ANS

15. QuizGuide: OLM+ANS
• Target-arrow abstraction:
– Number of arrows – level of
knowledge for the specific
topic (from 0 to 3).
Individual, event-based
adaptation.
– Color Intensity – learning
goal (current, prerequisite
for current, not-relevant,
not-ready). Group, time-
based adaptation.
 Topic–quiz organization:

16. QuizGuide: Success Rate
 It works!
 Mean success value for
QuizGuide is significantly
larger then the one for
QuizPACK:
F(1, 43) = 5.07
(p-value = 0.03).

17. QuizGuide: Motivation
• Adaptive navigation support increased student's
activity and persistence of using the system
Average activity
0
50
100
150
200
250
300
2002 2003 2004
Average num. of
sessions
0
5
10
15
20
2002 2003 2004
Average course
coverage
0%
10%
20%
30%
40%
50%
60%
2002 2003 2004
Active students
0%
20%
40%
60%
80%
100%
2002 2003 2004
 Within the same class QuizGuide session were much longer than
QuizPACK sessions: 24 vs. 14 question attempts at average.
 Average Knowledge Gain for the class rose from 5.1 to 6.5

18. WebEx - Code Examples

19. NavEx = WebEx + ANS

20. Concept-based student modeling
Example 2 Example M
Example 1
Problem 1
Problem 2 Problem K
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N
Examples
Problems
Concepts

21. Does it Work?
• The increase of the amount of work for the
course
Clicks - Overall
0
50
100
150
200
250
300
Non-adaptive Adaptive
Examples
Quizzes
Lectures - Overall
0
2
4
6
8
10
12
Non-adaptive Adaptive
Examples
Quizzes
Learning Objects - Overall
0
5
10
15
20
25
30
Non-adaptive Adaptive
Examples
Quizzes

22. Is It Really Engaging?
• Are they coming more often? A bit…
• But when they come, they stay… like in an
engaging game
Clicks - Per Session
0
5
10
15
20
Non-adaptive Adaptive
Examples
Quizzes
Learning Objects - Per
Session
0
1
2
3
4
Non-adaptive Adaptive
Examples
Quizzes

23. Why It Is Working?
• Progress-based adaptive annotation
– Displays the progress achieved so far
– Open Learner Modeling
• State-based adaptive annotation
– Not useful, ready, not ready
– Access activities in the right time
– Appropriate difficulty, keep motivation

24. Checking in Another Domains…
• Is it something relevant to C programming or to
special kind of content?
• Near transfer
– Java instead of C, complex problems
• Far transfer
– SQL Programming instead of C
– Programming problems (code writing) instead of
questions (code evaluation)
– Give students a chance to choose how to access

25. QuizJET: Tracing Problems for Java

26. Navigation Area Presentation Area
JavaGuide

27. JavaGuide
(Fall 2008)
QuizJET
(Spring 2008)
parameters (n=22) (n=31)
Overall User
Statistics
Attempts 125.50 41.71
Success Rate 58.31% 42.63%
Distinct Topics 11.77 4.94
Distinct Questions 46.18 17.23
Average
User Session
Statistics
Attempts 30.34 21.50
Distinct Topics 2.85 2.55
Distinct Questions 11.16 8.88
Findings Confirmed

28. • SQL-KnoT delivers online SQL problems, checks student’s
answers and provides a corrective feedback
• Every problem is dynamically generated using a template
and a set of
databases
• All problems have
been assigned to 1
of the course
topics and
indexed with
concepts from the
SQL ontology
SQL Knowledge Tester

29. • To investigate possible influence of concept-based
adaptation in the present of topic-based adaptation we
developed two versions of QuizGuide:
Topic-based Topic-based+Concept-Based
SQL-Guide

30. • Total number of attempts made by all students:
in adaptive mode (4081), in non-adaptive mode (1218)
• Students in general were much more willing to access
the adaptive version of the system, explored more
content with it and to stayed with it longer:
Questions
0
25
50
75
100
Quizzes
0
5
10
15
20
25
Topics
0
1
2
3
4
5
6
Sessions
0
1
2
3
4
5 Session Length
0
5
10
15
20
25
Adaptive
Non-adaptive
Confirmed… and Students Prefer It

31. Social Comparison and Navigation
• OLM and adaptive navigation support work well to
increase success and motivation
• Knowledge-based approaches require some
knowledge engineering – concept/topic models,
prerequisites, time schedule
• In our past work we learned that social navigation –
guidance extracted from the work of a community of
learners – might replace knowledge-based guidance
• Social wisdom vs. knowledge engineering

32. Open Social Learner Modeling
• Key ideas
– Show topic- and content- level knowledge progress of
a student in contrast to the same progress of the class
– Use social comparison to engage and guide students
• Main challenge
– How to design the interface to show student and class
progress over topics?
– We went through several attempts

33. QuizMap
39

34. Progressor
40

35. Parallel Introspective Views
41

36. Class vs. Peers
• Peer progress was important, students
frequently accessed content using peer models
• The more the students compared to their peers,
the higher post-quiz scores they received (r=
0.34 p=0.004)
• Parallel IV didn’t allow to recognized good peers
before opening the model
• Progressor added clear peer progress
comparison

37. Progressor+ OSLM for two types of content
• macro- and micro- comparisons (group or peers)
47

38. Students Spent More Time in Progressor+
Quiz =: 5 hours
Example : 5 hours 20 mins48
60.04
150.19
224.7
296.9
69.52
121.23
110.66
321.1
0
50
100
150
200
250
300
350
400
QuizJET JavaGuide Progressor Progressor+
Total time spent (minutes)
Quiz
Example

39. Students Achieved Higher Success Rate
49
42.63%
58.31%
68.39%
71.20%
0.00%
20.00%
40.00%
60.00%
80.00%
QuizJET JavaGuide Progressor Progressor+
Success Rate
p<.01

40. MasteryGrids
• Adaptive Navigation Support
• Topic-based Adaptation
• Open Social Learner Modeling
• Social Educational Progress Visualization
• Multiple Content Types
• Concept-Based Recommendation
• Open Source

41. Open Social Learer Modeling
Interactive Demo YouTube Demo

42. Progress Over Muliple Content Types

43. Group and Peer Social Comparison
53

44. Mastery Grids in a Database Course
• A classroom study in a graduate Database Course
• Two sections of the same class. Same teacher, same
lectures, etc.
• The students were able to access non-mandatory
database practice content (exercises, examples) through
Mastery Grids
• 47 students worked with OSM interface and 42 students
worked with OSSM interface
Brusilovsky, P., Somyurek, S., Guerra, J., Hosseini, R., Zadorozhny, V., and Durlach, P. (2016) The Value of
Social: Comparing Open Student Modeling and Open Social Student Modeling. IEEE Transactions on
Emerging Topics in Computing 4 (3), 450-461.

45. Does OSLM Motivates Free Practice?
Variable
OSM OSSM
U
Mean Mean
Sessions 3.93 6.26 685.500*
Topics coverage 19.0% 56.4% 567.500**
Total attempts to problems 25.86 97.62 548.500**
Correct attempts to problems 14.62 60.28 548.000**
Distinct problems attempted 7.71 23.51 549.000**
Distinct problems attempted correctly 7.52 23.11 545.000**
Distinct examples viewed 18.19 38.55 611.500**
Views to example lines 91.60 209.40 609.000**
MG loads 5.05 9.83 618.500**
MG clicks on topic cells 24.17 61.36 638.500**
MG click on content cells 46.17 119.19 577.500**
MG difficulty feedback answers 6.83 14.68 599.500**
Total time in the system 5145.34 9276.58 667.000**
Time in problems 911.86 2727.38 582.000**
Time in MG (navigation) 2260.10 4085.31 625.000**

46. Impact on Learning
• Student knowledge significantly increased in both
groups
• Number of attempted problems significantly
predicts the final grade (SE=0.04,p=.017).
• We obtained the coefficient of 0.09 for number of
attempts on problems, meaning attempting 100
problems increases the final grade by 9
• The mean learning gain was higher for both weak and
strong students in OSSM group
• The difference was significant for weak students
(p=.033)

47. Does OSLM increase Efficiency?
• Time per line, time per example and time per activity
scores of students in OSSM group are significantly lower
than in the other group.
• Students who used OSLM interface worked more
efficiently.
Variable
OSM OSSM
U
Mean Mean
Time per line 22.93 11.61 570.000**
Time per
example 97.74 58.54 508.000*
Time per
problem 37.96 29.72 242.000
Time per
activity 47.92 34.33 277.000*

48. Does OSSM Increase Student Retention?
0
20
40
60
80
100
0+ 10+ 20+ 30+ 40+ 50+
%Studentsinclass
Problem attempts
OSSM
OSM
• OSSM group had much higher
student usage
• Looking much more interesting to
students at the start (compare
#students after the first login)
• At the level of 30+, serious
engagement with the system, the
OSSM group still retained more
than 50% of its original users
while OSM engagement was below
20%.
0
20
40
60
80
100
0+ 10+ 20+ 30+ 40+ 50+
Problem attempts
OSSM
OSM

49. Personalized Visual Support for
Activity Selection with Rich-OLM
Guerra, J., C. Schunn, S. Bull, J.
Barria-Pineda and P. Brusilovsky
(2018). Navigation support in
complex open learner models:
assessing visual design
alternatives. New Review of
Hypermedia and
Multimedia 24(3): 160-192.

50. Mousing over this
activity
Concepts in the selected
activity are highlighted
This gauge estimates the
how much you can learn
in the selected activity.
You will probably learn
more in activities that
have more new concepts
Guerra-Hollstein, J., Barria-Pineda, J., Schunn, C., Bull, S.,
and Brusilovsky, P. (2017) Fine-Grained Open Learner
Models: Complexity Versus Support. In: Proceedings of the
25th Conference on User Modeling, Adaptation and
Personalization, Bratislava, Slovakia, ACM, pp. 41-49.

51. ATEC Worksho
2 0 1 9
Los Angeles
61
Explanations with OLM
Barria-Pineda,Jordan,andPeterBrusilovsky.2019.
"ExplainingEducationalRecommendationsThrougha
Concept-levelKnowledgeVisualization."InProceedingsof
the24thInternationalConferenceonIntelligentUser
Interfaces:Companion,103--04.NewYork,NY,USA:ACM.

52. Adding Direct Recommendations
• Stronger guidance than Adaptive
Navigation Support
• Proactive recommendations
– Expand knowledge
• Remedial recommendation
– Address problems
• Explanations
– Visual explanations with OLM
– Text-based explanations

53. Proactive Recommendations

54. BarriaPineda,J.andBrusilovsky,P.(2019).MakingEducational
RecommendationsTransparentthroughaFine-GrainedOpenLearner
Model.In:ProceedingsoftheWorkshoponIntelligentUserInterfacesfor
AlgorithmicTransparencyinEmergingTechnologiesatthe24thACM
ConferenceonIntelligentUserInterfaces,IUI2019,LosAngeles,USA

56. Remedial Recommendations
Remedial visual
explanations
Related concepts highlighted
Knowledge estimates as bar-chart
Recent success rate as bar-color
Warning sign on “struggled”
concepts
68
Barria-Pineda, J., Akhuseyinoglu, K., and Brusilovsky, P. (2019) Explaining Need-based Educational
Recommendations Using Interactive Open Learner Models. Proceedings of International Workshop on
Transparent Personalization Methods based on Heterogeneous Personal Data, ExHUM at the 27th
ACM Conference On User Modelling, Adaptation And Personalization, UMAP '19, Larnaca, Cyprus

57. Remedial Recommendations
Explained

58. What we are doing now?
• Connecting mandatory work (labs and
homeworks) with free practice
• Being able to see your knowledge and your
target, i.e., lab, exam, homework, you could get
ready for the challenge
• If you have troubles in your lab, the system
records it (as well as success)
• You could run a remedial adaptive practice after
the lab.

59. Acknowledgements
• Joint work with
– Sergey Sosnovsky, Michael Yudelson
– Rosta Farzan, Sharon Hsiao, Tomek Loboda
– Yun Huang, Julio Guerra, Roya Hosseini
– Jordan Barria, Kamil Akhuseyinoglu
• U. of Pittsburgh “Innovation in Education” awards
• NSF Grants
– CAREER 0447083
– DUE 0310576
– DUE 0633494
– IIS 0426021
– DLR 1740775
• ADL.net support for OSLM work

60. Visit us in Pittsburgh to Learn More!

61. … or Read our Papers
• http://www.pitt.edu/~peterb/papers.html
• https://www.researchgate.net/profile/Peter_Br
usilovsky