The document discusses conditions for effectively deriving a Q-Matrix from assessment data using Non-Negative Matrix Factorization (NMF). It describes a study that used NMF to interpret a Q-Matrix and derive the underlying skills assessed by test questions in mathematics, biology, history and French. The study analyzed response data from 297 respondents to a 40 question test. NMF was able to effectively cluster test questions that assessed well delineated skills.

1. Conditions for effectively
deriving a Q-Matrix from data
with Non-Negative Matrix
Factorization
MICHEL DESMARAIS 2011

2. Research Questions
• What are the factors and assumptions that NMF can effectively derive the
underlying high level skills behind assessment results?

3. Context of the study
• The purpose of the study was Q-Matrix Interpretation with NMF

4. Method
• Exploratory Research

5. Method used for Analytical Inferences
• Nonnegative Matrix Factorization

6. Data
• SubjectTest Data (SAT)
• Mathematics
• Biology
• World History
• French
• 297 respondents completed 40 question items test over the internet
• Respondents unknown, probably university student community

7. Tool or Program Used
• Statistical Software R
• Lots of algorithms, parameters and formulas

8. Part of a Project
• In this paper, parts of the study byWinters (2005) replicated, which focus on
one of the cluster algorithms NMF

9. Result
• That is not the case in general, however, the visualization technique used
throughout this paper show that for well delineated topic skills, clustering
with NMF is easily perceived through the human eye.

10. What Do Students Know?
An Outcomes Based Assessment System
Winters,T., & Payne,T. (2005)

11. Research Questions
• How much of each competence has each student achieved, and how
difficult and/or discriminating is each item of assessment (test question or
homework problem) in the curriculum?

12. Context of the Study
• This paper describes a project to develop an outcomes-based assessment
system that mines per-item scores to track each student’s skills and
knowledge

13. Methods
1. Item Evaluation
Item Response Theory is the study of test and item scores based on assumptions concerning the
mathematical relationship between abilities and item responses.
*Using this technique, it is possible to identify question types or individual questions that are
particularly good or bad given the instructor’s teaching and grading style.
• Calculating α and β (difficulty&discrimination parameters)
• Assumptions
• CrossTerm Consistency
• Item Assessment Result
• Estimate Missing Data

14. Method
2. Objective Identification
• Collaborative Filtering (how much does each factor affect a given user)
• Nonnegative Matrix Factorization (approximates the matrix of interest M by the
product of two nonnegative matrices U H ≈M)
• Known Relevance (encoding a known value)
• Topic Identification Results

15. Method Used for Analytical Inferences
• Nonnegative Matrix Factorization
Also,
• Statistical inference techniques from both educational statistics and data mining
will quantitatively determine each student’s acquired competency, with minimal
input from faculty.

16. Data
• University of California– Computer Science and Engineering faculty members
• Two major tools have been developed for the gathering of per-item score
data, Agar and Marksense. (CAA)
• Between these tools, it is generally faster and easier to grade an instrument
and get item-level data output and stored in the correct format than it would
be to grade by hand.

17. Tool & Program
• Lots of algorithms, parameters and formulas

18. Part of a Project
• Outcomes-based assessment system that mines per-item scores to track
each student’s skills and knowledge

19. Result
• NMF Algorithms, the domain knowledge of the instructors, and a moderate
amount of computational time are sufficient to generate a detailed
quantitative evaluation of how much graded exposure the students were
given to each topic,
• and how successful they were on those items.