By Roya Hosseini, I-Han Hsiao, Julio Guerra, Peter Brusilovsky
. In Proceedings of EC-TEL 2015.

1. What Should I Do Next?
Adaptive Sequencing in the Context of
Open Social Student Modeling
Roya Hosseini, I-Han Hsiao,
Julio Guerra, Peter Brusilovsky
PAWS Lab
University of Pittsburgh

2. Overview
• Motivation
– why do we care about guidance?
• Past work
– how to guide students to the right content?
• Current work
– adaptive sequencing combined with social guidance
– what we learned from the classroom study
• Work in progress & future work
2

3. Motivation
Goal
– personalized guidance to the most appropriate educational
content for each learner
!
Why personalized guidance?
– helps students acquire knowledge faster
– improves learning outcomes
– reduces navigational overhead
– increases student motivation to work with content
3

4. Existing Guidance Technologies
1. Knowledge-based approaches
• decide the most appropriate content for an individual with
respect to the domain model, student model, and course goal
• adaptation type:
• fine-grained concept-based (ELM-ART, NavEx)
• coarse-grained topic-based (QuizGuide)
!
2. Social guidance
4

5. Concept-Based Adaptation
Example 2 Example M
Example 1
Problem 1
Problem 2 Problem K
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N
Examples
Problems
Concepts
5

6. ELM-ART: Adaptive Link Annotation in LISP
6
green bullet indicates a
recommended page
red bullet indicates a
page user is not ready for
G. Weber And P. Brusilovsky, IJAIED 2001. Elm-Art: An
Adaptive Versatile System For Web-Based Instruction

7. NavEx: Concept-Based Adaptive Navigation Support
bullet is filled based on
progress
font style denotes the
relevance of example
a relevant example with
no progress
an example not ready to
be browsed
7
M. Yudelson And P. Brusilovsky, AIED 2005. Navex: Providing Navigation Support For
Adaptive Browsing Of Annotated Code Examples.

8. Topic-Based Adaptation
• each topic is associated with a number of educational activities
!
• each activity is classified under 1 topic
8
Topic
A
Topic
B
Topic
C

9. QuizGuide : Topic-Based Adaptive Navigation Support
Current quiz
number of arrows:
knowledge in the topic (0-3)
color Intensity: learning goal
P. Brusilovsky, S. Sosnovsky And O. Shcherbinina, E-Learn 2004. Quizguide: Increasing The Educational Value Of
Individualized Self-Assessment Quizzes With Adaptive Navigation Support.
9
current
prerequisite
not-relevant
not-ready

10. Knowledge Maximizer Paradigm
10
Hosseini, R., Brusilovsky, P., & Guerra, J. (AIED 2013, January). Knowledge Maximizer: Concept-based Adaptive
Problem Sequencing for Exam Preparation.
Learn maximum knowledge from next
activity while controlling prerequisites

11. Existing Guidance Technologies
1. Knowledge-based approaches
2. Social guidance
• uses Open Social Student Modeling (OSSM)
• students can view each others’ or class knowledge model
• almost as efficient as knowledge-based guidance
- higher success rates & engagement
- much less knowledge engineering overhead
• drawback: make students more conservative in their work
!
! 11

12. Mastery Grids: Topic-based Navigation Support in OSSM Platform
anonymized ranked
list of peers and
their topic-based
progress
position of current
student in class
topic-based progress of student
topic-based
progress of class
Loboda, T. D., Guerra, J., Hosseini, R., & Brusilovsky, P. (EC-TEL 2014). Mastery
Grids: An Open Source Social Educational Progress Visualization. 12

13. • combines social guidance with knowledge-based
guidance
• enhances the approach to maximize student
knowledge
• implements the guidance in context of Mastery
Grids OSSM
• reports the results from the classroom study
Sequencing + Open Social Student Modeling
13

14. Greedy Sequencing (GS)
• aims at maximizing student knowledge in domain concepts
• concept-based adaptation:
- uses prerequisite and outcome concepts in content items
14
User%
Modeling%
database%
Greedy%
Sequencing%
Knowledge%
Report%Service%
Rank%C1%
Prerequisites%
Outcomes%
Content%C1:%Concepts%

15. Greedy Sequencing: Content Ranking by
Knowledge Maximization
15
amount of known
prerequisites
amount of
unknown outcomes
rank of the content, [0-1]
number of outcomes
np:number of prerequisites
ki: knowledge of concept i
wi: weight of concept i, log(tf-idf value)

16. 16
• marked top three recommendations generated by GS
• size of star shows relative rank of content
- bigger star —> higher priority

17. The Study
143 undergraduates in ASU (Fall 2014)
Java Programming & Data Structure course
‣ 111 problems — 103 examples — 19 topics
!
Study had 2 main Parts
(1) no sequencing (Aug. 21 – Sep. 25)
(2) with sequencing (Sep. 26 – Oct. 21)
• 86 subjects logged into the system
• we considered 53 subjects with problem attempts >= 30
17

18. Navigational Pattern Analysis
GS breaks out the common path of social guidance
0.08
0.08
0.16
0.68
0.06
0.05
0.12
0.78
0.17
0.17
0.2
0.47
Jump−Backward
Jump−Forward
Next−Topic
Within−Topic
Part 1 Part 2−N Part 2−R
when following GS, “groupthink” stay on
the current topic shortens considerably
!
students moved to next topic more quickly
& expanded their non-sequential navigation

19. Value of GS on Amount of Learning & Speed
19
Learning gain:
• no significant differences in the learning gain
- non-followers (M = 0.50, SD = 0.27)
- followers (M=0.44, SD=0.23)
!
Learning speed: (learning gain/number of problem attempts)×100
! • speed of learning was higher among the followers
- non-followers (M = 0.54%, SD = 0.27%)
- followers (M = 0.97%,SD = 0.88%) speed increased about twice
- p = .083, using a Welch t-test

20. Value of GS on Learning & Speed: Weak vs. Strong Students
20
0.00#
0.20#
0.40#
0.60#
0.80#
1.00#
1.20#
1.40#
1.60#
1.80#
2.00#
Weak#students# Strong#students#
%#Learning#speed##
Non;followers# Followers#
0"
0.1"
0.2"
0.3"
0.4"
0.5"
0.6"
0.7"
0.8"
0.9"
Weak"students" Strong"students"
Normalized"learning"gain"
Non?followers" Followers"
• no significant differences in learning gain
• followers with high prior knowledge learn faster (p=.039)

21. Value of GS on Problem Solving Performance
21
Correctness is more frequent in recommended problems
• odds of correct answer in a problem offered by GS was 1.59
(SE = 0.19) times more than a not-recommended problem
How:
• data collected from part 1 and 2 of study
(5760 problem attempts: 5275 not-recommended, 485 offered by GS)
• fitted a logistic mixed effects model
• fixed effect: attempt type (recommended, not-recommended)
• response variable: correctness of attempt (0/1)

22. Value of GS on Class Performance
22
An attempt on a GS recommendation was associated with
higher grade
• attempting a recommended content (problem/example) was associated
with 0.56 increase in final grade (SE=0.24, p=.017)
~ 9 times greater than the effect of a not-recommended content
How:
• data of 40 students (had exam score + used system)
• fitted regression model to predict exam grade using number of attempts on contents

23. • 6 questions (5-point Likert scale)
• data collected from 51 students (answered questionnaire + used the system)
M:4.1 M:3.9 M:3.1 M:3.8 M:4.2M:2.4
Subjective Feedback
23
like
star
useful
clear
!
reason
distractive

24. Wrap Up
adaptive sequencing + social guidance:
!
✓encouraged non-sequential navigation patterns
✓increased learning speed of stronger students
‣ more optimal content navigation
✓was positively related to student performance
‣ higher exam score
‣ more success in problems

25. Work in Progress & Future Work
๏ running study with over 200 students in ASU
- GS vs. probabilistic approach based on FAST
!
๏ what is the best way to visualize student/class data?
- alternatives to topic-based guidance (2D content maps )
!
๏ how to increase students’ awareness of recommendations?
- adding annotations, …

26. References
Knowledge Maximizer: Hosseini, R., Brusilovsky, P., & Guerra, J. (2013, January). Knowledge Maximizer:
Concept-based Adaptive Problem Sequencing for Exam Preparation. In Artificial Intelligence in Education (pp.
848-851). Springer Berlin Heidelberg.
!
Mastery Grids: Loboda, T. D., Guerra, J., Hosseini, R., & Brusilovsky, P. (2014). Mastery Grids: An Open
Source Social Educational Progress Visualization. In Open Learning and Teaching in Educational Communities
(pp. 235-248). Springer International Publishing.
!
QuizGuide: P. Brusilovsky, S. Sosnovsky And O. Shcherbinina, 2004. Quizguide: Increasing The Educational
Value Of Individualized Self-Assessment Quizzes With Adaptive Navigation Support. In: J. Nall And R.
Robson, Eds., World Conference On Elearning, E-Learn 2004 Aace, Washington, Dc, Usa, 1806-1813.
!
NavEx: M. Yudelson And P. Brusilovsky, 2005. Navex: Providing Navigation Support For
Adaptive Browsing Of Annotated Code Examples. In: C.-K. Looi, G. Mccalla, B. Bredeweg And J. Breuker,
Eds., 12Th International Conference On Artificial Intelligence In Education, Ai-Ed'2005 Ios Press, Amsterdam,
The Netherlands, 710-717.
!
ELM-ART: G. Weber And P. Brusilovsky, 2001. Elm-Art: An Adaptive Versatile System For Web-Based
Instruction. International Journal Of Artificial Intelligence In Education, 12 (4), 351-384
26

27. Thank You!
Intelligent Systems Program
Roya Hosseini
roh38@pitt.edu
Peter Brusilovsky
peterb@pitt.edu
I-Han (Sharon) Hsiao
sharon.hsiao@asu.edu
Julio Guerra
jdg60@pitt.edu
Try it! adapt2.sis.pitt.edu/kt/mg-gs.html
https://www.youtube.com/watch?v=Kak8F2y5GkU