2. What do we do in the Advanced
Computing & Data Science Lab?
Prototyping
● Adaptive learning capabilities
● Authoring and tagging tools with machine
intelligence
● Exploration of new technology
Research and Development
● Direct support of a product or prototype
● Exploratory research into new capabilities
2
4. Data Context and IRT
Online learning systems provide students with instruction, homework, and summative
tests for an entire course.
The Book is organized hierarchically into Chapters and Sections. Organization is
called the Table of Contents (TOC). Items are within a single Section.
4
Traditional (unidimensional) IRT models can work well at both
● Local-level: Section or Chapter specific models
● Book-level: placing the entire course on a single latent trait
We need to turn to multidimensional models to help understand the relationships
between the structures in the book.
5. Explore the use of multivariate
and multidimensional techniques
to make inferences about the
structure of content in online
learning systems
5
7. Multivariate Section-Level IRT
1. IRT model specified at section level
○ Can be any specification, 1PL, 2PL, 3PL polytomous, etc
2. Section-level covariance matrix
○ Jointly estimate all section-level IRT models
○ Simple structure. All items within a section only load on the section-level
latent trait. All section-level latent traits can freely covary
3. Secondary analysis of covariance matrix
○ Plug covariance into any EFA, PCA, SEM
○ EFA to explore book structures analysis
○ SEM to verify TOC structures or “aggregate” covariance up the TOC
7
8. A Few Equations
8
1. IRT model specified at section level
2. Section-level covariance matrix
3. Secondary analysis of covariance matrix
9. 1. Section-level IRT
All the usual unidimensional
psychometric results are available
• ability
• difficulty
• discrimination
• guessing
• ...
Now we have psychometric results but only inside in context of the item’s
section. Next, we look at the covariance between all objectives.
10. 2. Section-level covariance matrix
Section 1 Section 2 Section 3 Section 4
Item 1 X
Item 2 X
Item 3 X
Item 4 X
Item 5 X
Item 6 X
Item 7 X
Item 8 X
Item 9 X
Item 10 X
Item 11 X
Item 12 X
Item 13 X
Item 14 X
Item 15 X
Item 16 X
15. Not your usual data...
15
● 5,000 items
● Item cloning, each item can have up to thousands of instantions
● Instructor controlled learning aids, scoring policies, and settings
● A semester can have 20,000 students and 6,000,000 responses
● As a person by item response matrix, that’s 95% missing data
● Missingness do to an ensemble of effects
○ Instructor customizations
○ Variety of courses and institutions using the same book
16. Table of Contents and Multidimensional Models
16
● Books have can have 10 to 30 Chapters
● Each Chapter has about 4 to 8 Sections
So if we want to do what we said...
…that implies a 40 to 250 dimensional model.
How do we do that?
18. How do we estimate high dimensional models?
Pairwise.
● Problem grows quadratically, not exponentially.
● Instead of fitting one 40-dimensional model with, for example, 10^40 latent evaluation points,
we fit (40^2 - 40)/2 = 780 2-dimensional models each with 10^2 latent evaluation points
● Pairwise models are easily parallelized, CPU-limited, and chunk the data, allowing the method to
scale with appropriate computational resources
18
19. The Obvious Criticism
● Secondary (post-hoc) analysis of covariance matrix does not correctly account
for standard errors.
● It would be better to jointly estimate the model on the covariance matrix
simultaneous with its estimation.
○ ...but the pairwise estimation is hard part, requiring significant
computational resources and time. Once we get that, the secondary
analysis is trivial.
○ ...and there are larger threats to the inference and standard errors
(non-ignorable missing data, student growth over time, etc).
○ ...even still, the value of the results justify its use.
19
28. Screen Book for Areas for
Expert Review
28
Ch A
Ch B
Ch C
● Flag areas where data do not match
expectations
● Can be thought of as taking the TOC as
the expert domain model, and then
validating that model with the data
● Target human and expert reviews to
areas most likely in need
Odd Section
