Parameterized Exercises in Java Programming: using Knowledge Structure for Performance Prediction

2Shaghayegh Sahebi (Sherry)
• Programming questions
– Java problems
• Can be designed with parameterized exercises
– One question with multiple parameter sets
– Can be repeated multiple times by one student
• Authoring tool for Java questions
– Create and modify questions
– Indexing service to define concepts inside the
question

Each question is generated from a template, and
students can try multiple attempts.
Students give values for specified variable, or give
the output of the code.
A question for practicing skill
nested loops

Aggregate
(MasteryGrids
Services)
Aggregate
UM2
Other
(content
specific)
PAWS
UM Services
Content apps
Server side apps
(Apache Tomcat)
Databases (MySQL)
Client interface
MasteryGrids Interface
Content popout iframe
QuizJet
WebEx
SQLKnot
(a)
GUI calls MG
services
direct link
services calls
(b)
Aggregate uses
UM services
cbum login
Overall view of the architecture
Mastery Grids

• Students can choose what question to solve
– Using social navigation support
• Adding guidance to the question
– Use the whole set of data to develop personalized
guidance
– Predict how likely the problem will be solved
– Avoid too simple and too complex problems

• Predicting the student’s capability to perform
an educational task
• Assumption: the student can learn by
practicing over time by repeating
– Time sequence modeling effect on PSP
• Will present at the Problem Solving & Strategies session
on Monday
– Knowledge structure effect on PSP
• Today’s talk

• Questions related to topics, concepts, or skills
– many dimensions in the data
– Structure in the data (knowledge structure)
• Traditional methods: mostly consider student’s
past performance
– Only consider correct/incorrect attempts of students
(ignoring the multidimensionality of the data)
– Bayesian Knowledge Tracing (BKT)
– Performance Factor Analysis (PFA)

• Considering knowledge structure in PSP
– Feature-Aware Knowledge Tracing (FAST) [González-Brenes
et al., ‘13]
– Our suggestion: Tensor Factorization Methods

• Study the effect of knowledge structure
modeling in PSP for parameterized questions
• Compare five approaches:
– Bayesian Knowledge Tracing (BKT)
– Performance Factor Analysis (PFA)
– Feature-Aware Knowledge Tracing (FAST)
– 3D and 4D Tensor Factorization (3D-BPTF, 4D-
BPTF)

• Markov Model with two states
• No knowledge structure: Only one type of
knowledge component
• Guess, slip, learning, and initial knowledge
parameters
Knowledge
Structure

• Regression model
• No knowledge structure
m(i, j Î KCs
,k Î items,s,f) = bk
+ (gj
si,j
+ rj
fi,j
)
jÎKCs
å
Knowledge
Structure

• Extension of BKT
• Can include knowledge structure as regression
variables
fq,l,t
features L
yq,t
kq,t
timesteps T
# of skills Q
fq,e,t
features E
Knowledge
Structure

• Tensors: n-dimensional arrays
• Used in collaborative-filtering recommender
systems
– Estimates each tensor as the sum of multiple rank-1
tensors
• Can be extended to as many dimensions
– Can include the data structure
– Each dimension of the data ≈ one dimension of the
tensor

• Used successfully in PSP for traditional PSP
• No knowledge structure
• We use Bayesian Probabilistic Tensor
Factorization Model (3D-BPTF) [Xiong et al., 2010]
Students
Questions/ topics
…
Knowledge
Structure

• Used for the first time in PSP
• Adds knowledge structure modeling
• Can be extended to more dimensions if
needed
Students
Questions Students Questions
Students
Questions
Attempt 1 Attempt 2 Attempt 3
Knowledge
Structure

• From QuizJET system
• Java programming questions
• Six semesters
• 166 students
• 103 questions
• 69.04% majority class (successes)

• Here, each topic can have multiple questions and
each question is related to one topic
– Two dimensions: questions and topics
• Study 1: traditional approach
– Question as knowledge unit
• Study 2: considering knowledge structure
– Topic added as knowledge unit
• 5-Fold cross validation
– 80% of students in train data, rest in test data
– User-stratified

Study 1: comparing traditional approaches
(no knowledge structure)

71
71.5
72
72.5
73
73.5
74
74.5
75
75.5
76
FAST with no
additional
parameters
BKT PFA 3D-BPTF
Accuracy of Traditional Models

0
200
400
600
800
1000
1200
1400
FAST with no
additional
parameters
BKT PFA 3D-BPTF
False Positive

0
100
200
300
400
500
600
700
800
FAST with no
additional
parameters
BKT PFA 3D-BPTF
False Negative

0
10
20
30
40
50
60
70
80
90
FAST with
no
additional
parameters
BKT PFA 3D-BPTF
Majority Precision
Minority Precision
BKT predicts failure better

Study 2: comparing approaches
including knowledge structure

64
66
68
70
72
74
76
78
FAST 4D-BPTF BKT PFA 3D-BPTF
Accuracy of Approaches with Additional Knowledge
Structure

0
200
400
600
800
1000
1200
1400
1600
1800
False Positive

0
100
200
300
400
500
600
False Negative

0
10
20
30
40
50
60
70
80
90
100
Majority Precision
Minority Precision
3D-BPTF predicts failure better

66
67
68
69
70
71
72
73
74
75
76
Question as KC (No Structure)
Topic as KC (with Question
Structure)
3D-BPTF
Accuracy

66
67
68
69
70
71
72
73
74
75
76
Question as KC (No Structure)
Topic as KC (with Question
Structure)
Accuracy

• Accuracy in predicting students performance
depends on the input of the method
– When ignoring the topic of questions as KCs, all
models perform similarly
– When including topic information, in addition to
the question information, the methods that can
leverage it perform better

• Adding the extra topic data in the methods
that cannot model this information decreases
the method’s accuracy
• Knowledge structure can add to the accuracy
of PSP in parameterized questions
• Tensor factorization methods are as good, or
better than the pioneers PSP methods

• Include additional structure into tensor
factorization using more dimensions
• Use of other collaborative filtering methods
for PSP
• Test on other programming courses

Thank You!

• EM algorithm for BKT and set the initial
parameters as follows: p(L0) = 0:5 , p(G) = 0:2 ,
p(S) = 0:1 , p(T) = 0:3 . For running PFA, we use
• the implementation of logistic regression in
WEKA [3]. For BPTF and BPMF,
• we utilize the Matlab code prepared by Xiong et.
al. We experimented with different latent space
dimensions for BPTF and BPMF (5, 10, 20 and 30)
and chose the best one, which has the latent
space dimension of 10

• From collaborative filtering
• Used successfully in PSP for static questions
• No attempt sequence modeling
• We use Bayesian Probabilistic Matrix
Factorization (BPMF)
1 0 0 0
1 1 0 1
0 0 1 1
0 0 0 1
Students
Questions/ topics
0.9 0
1.5 0.4
0 1.4
0 0.9
Students
KCs
0.8 0.5 0 0.3
0 0 0.5 0.8
KCs
Questions/ topics

Accuracy of all models is very close to each other

BKT over estimates the student’s performance

FAST tends to predict more failures for the students

If FAST predicts a success for a student and if BKT predicts a failure
for students, their prediction is more likely to be true compared to
the other methods.

FAST and 4D-BPTF perform significantly better than all other
approaches

BKT and PFA perform similarly to their results in Study 1 and 3D-BPTF
on topics is slightly weaker than 3D-BPTF on questions in terms of
accuracy.

• Visual, interactive, adaptive E-learning
platform
– Multi-facet social comparison
– Multi-type learning materials support
– Social navigation
– Personalized guidance
• Integration with other systems with little set
up and modification

Parameterized Exercises in Java Programming: using Knowledge Structure for Performance Prediction

More Related Content

Similar to Parameterized Exercises in Java Programming: using Knowledge Structure for Performance Prediction

Recently uploaded

Parameterized Exercises in Java Programming: using Knowledge Structure for Performance Prediction

Editor's Notes