A General Approach to Data-Driven Redesign of Tutoring Systems Using Multiple Methods

1
A General Multi-method Approach to
Data-Driven Redesign of Tutoring Systems
Yun Huang, Nikki G. Lobczowski, J. Elizabeth Richey
Elizabeth A. McLaughlin, Michael W. Asher*, Judith M. Harackiewicz*,
Vincent Aleven, Kenneth R. Koedinger
Carnegie Mellon University (USA) *University of Wisconsin-Madison (USA)
April, 2021, LAK ‘21
LAK 2021, Apr 15

Outline
● Introduction
● Method
● Results
● Discussion and Conclusion
2

Data-Driven Redesign
3
● Why is it important?
○ Earlier version may be suboptimal
due to expert blind spot
○ Analytics of data can help uncover
design deﬁciencies
○ A requirement of the learning
analytics cycle (Clow, ‘12)
[1] Clow, D. (2012). The learning analytics cycle: closing the loop effectively. In Proceedings of the 2nd international
conference on learning analytics and knowledge (pp. 134-138).
Learners
Data
Metrics/
Analytics
Interventions
● What is it? Use analytics of data from previous iterations to
improve the design of courses or systems

Prior Work on Data-Driven Redesign
● Course Signals (Arnold & Pistilli, ‘12)
● The Loop tool (Bakharia et al., ‘16)
● Geometry tutoring system (Liu & Koedinger, ‘17)
○ Added visual cues to draw attention to applying a square root, etc.
○ Led to signiﬁcant learning gains relative to the control tutor
(p=.027, Cohen’s d=.47, N=91)
4
[1] Arnold, K. E., & Pistilli, M. D. (2012, April). Course signals at Purdue: Using learning analytics to increase student success. In Proceedings of
the 2nd international conference on learning analytics and knowledge (pp. 267-270).
[2] Bakharia, A., Corrin, L., De Barba, P., Kennedy, G., Gašević, D., Mulder, R., ... & Lockyer, L. (2016, April). A conceptual framework linking
learning design with learning analytics. In Proceedings of the sixth international conference on learning analytics & knowledge (pp. 329-338).
[3] Liu, R., & Koedinger, K. R. (2017). Closing the Loop: Automated Data-Driven Cognitive Model Discoveries Lead to Improved Instruction and
Learning Gains. Journal of Educational Data Mining, 9(1), 25-41.

There’s still a lot of room for methodological improvement
● Applied a single analytics method considering limited aspects, e.g.,
adding visual cues for an identiﬁed hard skill (Liu & Koedinger, ‘17)
○ Is it good to always provide ﬁxed scaffolding?
● Focused on adapting to students’ differences (Mostafavi & Barnes, ‘17;
Zhou et al., ‘17)
○ What if the content itself is not good enough?
● Many learning analytics or data mining methods improve predictive
performance (González-Brenes et al., ‘14; Piech et al., ‘15)
○ How to convert analytics into better system design? Do they lead
to better student learning?
5
Mostafavi, B., & Barnes, T. (2017). Evolution of an intelligent deductive logic tutor using data-driven elements. In IJAIED; Zhou et al. (2017).
Towards Closing the Loop: Bridging Machine-Induced Pedagogical Policies to Learning Theories. In EDM; González-Brenes et al. (2014). General
features in knowledge tracing to model multiple subskills, temporal item response theory, and expert knowledge. In EDM; Piech et al. (2015).
Deep knowledge tracing. In advances in neural information processing systems.

Our Contributions
● #1: Provide general, explicit guidance on combining multiple
methods from different aspects to maximumly improve a
system, achieving three kinds of adaptivity
● #2: Provide general, explicit guidance on converting analytics
into better system design and instructional design
6

Our Approach Addresses Three kinds of Adaptivity
7
Adapt to students’
differences
● When to show scaffolding?
● Which task to give next?
Adapt to students’ similarities
(general demands of task domain)
● Discover hidden hard skills (e.g., skill 3)
● Reﬁne scaffolding (e.g., Task 1 Step 3)
● Create new problems (e.g., Task 10-11)
Step-Loop
Adaptivity
Task-Loop
Adaptivity
Step 1
Step 2
Task 1
Step 1
Task 5
Step 2
Task 10
Step 1 (Skill 3) Step 1 (Skill 1)
Task 1
Step 2 (Skill 2)
Step 3 (Skill 3)
Task 11
Step 1 (Skill 3)
Design-Loop
Adaptivity
Aleven, V., McLaughlin, E. A., Glenn, R. A., & Koedinger, K. R. (2016). Instruction based on adaptive learning technologies.
Handbook of research on learning and instruction, 522-560.
Following the Adaptivity Grid Framework (Aleven et al., ‘16)

Our Contributions
#3: Provide empirical evidence and comprehensive evaluation of
the effectiveness and generality of our approach
● Our past work provided initial evidence (Huang et al., 2020)
● Our current work extended our approach, redesigned new units,
conducted a larger-scale and longer-span classroom study
● Compared to the original tutor, the redesigned tutor
○ Led to higher learning outcomes (p=.046; Cohen’s d=.31)
○ Reduced over- and under-practice
○ Yielded a more effective practice experience
○ Selected skills progressing from easier to harder to a greater degree
8
Huang, Y., Aleven, V., McLaughlin, E., & Koedinger, K. (2020). A General Multi-method Approach to Design-Loop Adaptivity in Intelligent
Tutoring Systems. In International Conference on Artificial Intelligence in Education (pp. 124-129). Springer, Cham.

Outline
● Introduction
● Method
● Results
9

A General Multi-method Approach to Data-Driven
Redesign of Tutoring Systems (MADDRED)
Three general challenges:
1. How to create an accurate skill model that explains students’
performance and learning transfer well?
2. How to design content that facilitates learning hard skills?
3. How to effectively distribute practice time across skills for
individual students?
10

A General Multi-method Approach to Data-Driven
Redesign of Tutoring Systems (MADDRED)
Three general goals:
1. Reﬁne the skill model (i.e., knowledge component model)
2. Redesign content: adapting to students’ similarities or the
general demands of the task domain
3. Optimize individualized learning: adapting to students’
differences
Combine existing and new learning analytics methods and
instructional design methods to reach each goal
11

A General Multi-method Approach to Data-Driven Redesign of Tutoring Systems
12
Optimize
individualized learning
● Optimize student
model parameters
● Optimize adaptive
task selection
● Data-tuning
parameters
● Adaptive tutoring
simulation
Refine the knowledge
component (KC) model
● Identify task factors that
cause difficulties for KCs
● Hypothesize and compare
alternative KC models
● Difficulty Factor Effect
Analysis
Redesign content
● Create focused tasks that target
hard KCs with better scaffolding
● Add more content to ensure a
sufficiency of content for mastery
● Probability-Propagated Practice
Estimation
● Learning curve guided error analysis
● Focused Practice Task Design
Subgoals
Methods
Goals

Mathtutor
● https://mathtutor.web.cmu.edu
● A free online tutor with comprehensive content for
middle-school mathematics [1]
● Designed based on best practices and prior instructional
design research[2]
, but had not been data-tuned
● Three units, log data from 499 students with 53,596 txs.
13
[1] Aleven, V., & Sewall, J. (2016). The frequency of tutor behaviors: a case study. In International Conference on
Intelligent Tutoring Systems (pp. 396-401). Springer, Cham.
[2] Koedinger, K. R., & Anderson, J. R. (1998). Illustrating principled design: The early evolution of a cognitive tutor
for algebra symbolization. Interactive Learning Environments, 5(1), 161-179.

Original Modeling Unit
14
(Cells ﬁlled in correctly; hint and skill windows excluded)

Original Explanation Units (2-operator)
15

Original Explanation Unit (3-operator)
16

17
Optimize
model parameters
task selection
● Data-tuning
parameters
simulation
Analysis
Redesign content
Estimation
Subgoals
Methods
Goals

Difficulty Factor Effect Analysis (DFEA)
● Key idea: capture task distinctions important for novices
○ Uncover hard KCs, differentiate easy and hard KCs
● Difficulty factor (DF): a property that makes some steps more difficult
○ “Involve a negative number or not”
● DFEA: an efficient regression search process we created
○ Input: automatically coded step features that might impact difficulty (e.g.,
“Require parentheses or not”)
○ Examine main effect and interaction effects of DFs for each KC
18
Write an
expression
# of operators?
require parentheses?
repeated in a problem?

An Example of an Original KC
19
Write an
expression
Write an
expression

New KCs after splitting “Write an expression” in a 1-oper Problem
20
Write an expression
(1-oper ﬁrst)
Write an expression
(1-oper repeat)

New KCs after splitting “Write an expression” in a 2-oper Problem
21
Write an expression
(2-oper ﬁrst)
Write an expression
(2-oper repeat)
Write an expression
(2-oper par. ﬁrst)
Write an expression
(2-oper par. repeat)

22
Optimize
model parameters
task selection
● Data-tuning
parameters
simulation
Analysis
Redesign content
Estimation
Subgoals
Methods
Goals

Practice Estimation
Two central questions:
● How much and where did under- and over-practice occur in the
original tutor?
○ Over-practice: A student kept practicing a KC, although he/she is
estimated to had mastered the KC (post hoc estimation)
○ Under-practice: A student stopped practice a KC, although the
estimated knowledge is far from mastery (post hoc estimation)
● How many practice opportunities did a student need for
mastering a KC in the original tutor?
○ Did hard KCs need an extremely high # of opp. to reach mastery?
○ Were there enough problems (or opp.) for reaching mastery?
23

Probability-Propagated Practice Estimation
● General idea:
○ Mastery: P(Known) (probability of a student knowing a KC) > 0.95
○ Obtain P(Known) per transaction by a statistical model of learning
(e.g., BKT) with the new KC model and parameters ﬁt to the data
○ Use above estimates as “ground truth”, compare with actual # of opp.
● Difference with prior work:
○ Propagate the probability of succeeding not simulated outcomes,
much more eﬃcient than main existing method (Lee & Brunskill ‘12)
○ Inform content redesign rather than comparing learner models
24
BKT: Bayesian Knowledge Tracing (Corbett & Anderson ‘96)
Corbett, A. T., & Anderson, J. R. (1994). Knowledge tracing: Modeling the acquisition of procedural knowledge. In UMAP
Lee, J. I., & Brunskill, E. (2012). The Impact on Individualizing Student Models on Necessary Practice Opportunities. In EDM

A KC estimated to be highly over-practiced
25
Enter a given x value
KC
# of
stu
Under-Practice Over-Practice Avg # of opp.
(to mastery)
% of stu Avg # % of stu Avg # Actual BKT
enter-giv-x 353 14 20 84 21 26 11

26
Optimize
model parameters
task selection
● Data-tuning
parameters
simulation
Analysis
Redesign content
Estimation
Subgoals
Methods
Goals

Focused Practice Task Design
● A novel data-driven instructional design method
● Create focused tasks that target hard KCs, following
○ Content redesign strategies derived from analytics of KCs
○ Informed by prior cognitive and instructional design research
● Two types of focused tasks
○ Focused whole tasks: larger application context
○ Focused part tasks: smaller application context
27

Analytics about the original tutor Design strategies for focused tasks
Inappropriate amount of practice on KCs:
● Many students over-practiced easier KCs
● Many students under-practiced hard KCs
● Different KCs needed different # of opp.
to mastery
Reduce over-practice on easier KCs and
under-practice on hard KCs, e.g.,
● Eliminate fixed steps of untargeted KCs (i.e.,
easier KCs or other hard KCs)
● Provide dynamic scaffolding
Inadequate scaffolding for hard KCs:
● Required too many # of opp. to mastery
on hard KCs
● No explicit practice on hidden hard KCs
Provide effective scaffolding for hard KCs
informed by prior research, e.g.,
● Composition scaffolding
● Explicit practice on hidden hard KCs
● Multiple-choice questions for enhancing
understanding
Common errors persistent across
opportunities and KCs
Provide error feedback and hint messages
to address common errors early on
28

Analytics about the original tutor Design Strategies for focused tasks
to mastery
on hard KCs
understanding
29

Original Full Task
30
Over-practiced Under-practiced

Original Full Task
31
Need much more
practice on steps
requiring paren.
than steps not
requiring paren.

to mastery
on hard KCs
understanding
32

Analytics about the original tutor Design Strategies for focused tasks
to mastery
on hard KCs
understanding
35

Original Full Task
36
>= 58 practice opp. (steps) needed
for each hard KC involved here

Main diﬃculty lies in composition
● Prior cognitive research[1]
: The main diﬃculty is from a composition KC,
expression embedding (e.g., putting 800-y and 40x together into 800-40x)
○ Our analytics in this data were consistent with the cognitive research
37
[1] Heffernan, N. T., & Koedinger, K. R. (1997). The composition effect in symbolizing: The role of symbol production vs. text comprehension.
In Proceedings of the nineteenth annual conference of the cognitive science society (pp. 307-312).
[2] Koedinger, K., & McLaughlin, E. (2010). Seeing language learning inside the math: Cognitive analysis yields transfer. In Proceedings of the
Annual Meeting of the Cognitive Science Society (Vol. 32, No. 32).
● Prior instructional design research[2]
: Substitution tasks (“Substitute 500-x
for y in y/3”) are effective for learning this KC
○ Original tutor didn’t provide explicit practice on this KC
Much harder than

to mastery
on hard KCs
understanding
38

Focused Whole Task
39
Composition scaﬀolding

Focused Part Task
40
Explicit practice on a hidden hard KC
(composition / expression embedding)

Focused Part Task
41
Multiple-choice for enhancing understanding
for common errors (e.g., missing paren.)

to mastery
on hard KCs
understanding
practice opportunities and KCs
42

43
Optimize
model parameters
task selection
● Data-tuning
parameters
simulation
Analysis
Redesign content
Estimation
● Learning curve guided error
analysis
Subgoals
Methods
Goals

Adaptive tutoring simulation
● Key idea:
○ Simulates the practice sequence that would be provided by a
redesigned tutor for each simulated student
○ Estimates the time required to master a set of KCs
● Past work:
○ Used simulation to evaluate learner models
○ Used simulation to evaluate problem selection algorithm[1]
● We devised simulation to evaluate the tutor as a whole, and
conduct final tuning of various aspects:
○ Refine the problem selection algorithm
○ Refine model parameters for KCs
○ Examine whether there is sufficient content for mastery
44
[1] Doroudi, S., Aleven, V., & Brunskill, E. (2017, April). Robust evaluation matrix: Towards a more principled offline
exploration of instructional policies. In Proceedings of the fourth (2017) ACM conference on learning@ scale (pp. 3-12).

Outline
● Introduction
● Method
● Results
45

● High school Algebra I classes (8 class periods from 3 teachers)
○ 49% females, 27% free or reduced lunch, 11% Black, Latinx, or multiracial
● 1 month, two 40-minute per week, during normal class periods
● Pretest → Practice → Posttest
● Random assignment to 1 of 2 conditions within each class period
Classroom Experiment Setup
Data-tuned Adaptive (DA) condition
(redesigned tutor)
60 students
Control Condition
(Original tutor)
69 students
46

Comparison between two conditions (tutors)
47
High-bar control condition ...

Analysis Schema
● Overall effectiveness by comparing posttest scores (controlling for
pretest scores and other factors)
● Drill-down analyses with different focuses to understand the
processes leading to the overall effect
● Research questions:
○ RQ1: Did the redesigned tutor yield higher learning outcomes?
○ RQ2: Did the redesigned tutor reduce over- and under-practice?
○ RQ3: Did the redesigned tutor lead to a more effective practice
experience?
○ RQ4: Did the redesigned tutor select skills to practice progressing
from easier to harder to a greater degree?
48

Learning Outcomes
49
Compared to the Control condition, the DA condition:
● Produced signiﬁcantly higher learning outcomes (p=.046; Cohen’s d=.31)
● Replacing much of the full task practice with focused task practice
(Error bars represent 95% CIs)

Over- and Under-Practice
50
Compared to the Control condition, the DA condition
● Reduced over-practice for easy skills & high pretest students (p=.21, d=.51; p=.16, d=.40)
● Reduced under-practice for hard skills & low pretest students (p=.04, d=.86; p=.03, d=.62)

Over- and Under-Practice
51
Compared to the Control condition, the DA condition
● Reduced over-practice for easy skills & high pretest students (p=.21, d=.51; p=.16, d=.40)
● Reduced under-practice for hard skills & low pretest students (p=.04, d=.86; p=.03, d=.62)
● Reduced the amount of over-practice by
half on average;
● The sum of reduced amount over skills
roughly the equivalence of opp. for at
least one more mastered skill
● The sum of reduced under-practiced
opp. over skills were non-trivial (94 for
hard skills and 153 for low pretest
students on average).

Practice Experience
52
● We utilized learning curves to inspect practice experience, since they were
often used in prior research as a subtle way to measure learning outcomes.
● A learning curve depicts the error rates (averaged over students) over
successive practice opportunities for a KC or aggregated over KCs

Practice Experience
53
● We utilized learning curves to inspect practice experience, since they were
often used in prior research as a subtle way to measure learning outcomes.
● A learning curve depicts the error rates (averaged over students) over
successive practice opportunities for a KC or aggregated over KCs
DA led to faster
decrease in error rates
DA led to faster
decrease on hard KCs
DA led to faster decrease for
low and high pretest students
Low pretest students in
DA were more prepared
when facing a new skill

Diﬃculty Progression
54
Compared to Control condition, the DA
condition selected skills progressing
from easier to harder to a greater degree
(p=.002, d=.57)
DA: 20% frequency difference vs.
Control: 10% frequency difference
The frequency difference (EH-HE)
was a signiﬁcant predictor for
posttest scores (controlling for
pretest scores and other factors)
for each condition and overall:

Outline
● Introduction
● Method
● Results
55

Conclusions & Contributions
● Demonstrate a multi-method approach (MADDRED) to data-driven
redesign of tutoring systems
○ Adapt to both students’ differences and similarities
○ Key idea: identify hard skills, design focused tasks (e.g., target hard
KCs, fewer steps, better scaffolding) and optimize task selection
● Provide empirical evidence of its effectiveness and generality
○ Extending our prior approach and evaluation on a single unit
○ Provide indirect evidence for the effectiveness of our focused tasks
● Help deﬁne and enhance data-driven learning engineering processes
56

Future Work & Promising Directions
● More iterations
○ Redesigned tutor still led to some amount of over-practice and a
non-trivial amount of under-practice
○ Data-driven redesign is intended as an iterative process
● Other task domains, systems or environments
○ Intended to be applicable to systems with learning-by-doing activities &
designed and organized based on a KC model
● Not just maximizing learning outcomes for system redesign
○ Engagement analytics, motivation, meta-cognition
● Data-driven + human-centered learning analytics
○ Answer design questions that data analytics alone cannot answer
57

Takeaways
● A systematic multi-method approach to system redesign could lead to
significant improvements in learning and learning experiences.
● It’s worth the effort to design and organize activities and instructions
based on a good KC (skill) model
○ It’s powerful to redesign based on a refined KC model and
analytics regarding difficulties, over-/under-practice on these KCs
○ Focused task: not simply shorter/smaller problems, but problems
that focus on the critical KC(s).
● Analytics still need to be combined with cognitive / learning sciences
/ instructional design research to reach design decisions
58

Acknowledgements
● This work was supported by Bill and Melinda Gates
Foundation Prime Award #OPP1196889.
● Thank reviewers for their valuable feedback
● Thank collaborating teachers
● Thank all the team members
● Thank co-PI (Vincent) and PI (Ken)
59

Our scripts
Will be available shortly at:
http://learnsphere.org
https://github.com/MADDRED
60

Thank you for your attention!
Feedback & Questions ?
61

A General Approach to Data-Driven Redesign of Tutoring Systems Using Multiple Methods

Recommended

Recommended

More Related Content

Similar to A General Approach to Data-Driven Redesign of Tutoring Systems Using Multiple Methods

Similar to A General Approach to Data-Driven Redesign of Tutoring Systems Using Multiple Methods (20)

More from Yun Huang

More from Yun Huang (7)

Recently uploaded

Recently uploaded (20)

A General Approach to Data-Driven Redesign of Tutoring Systems Using Multiple Methods