In this presentation, we discuss our initial approaches to learning analytics at the transaction level.

1. Using Transaction-Level Data to Diagnose
Knowledge Gaps and Misconceptions
Randy Davies, Rob Nyland, John Chapman, Gove Allen
Brigham Young University
LAK15, Marist College, Poughkeepsie, NY
@robnyland @chapmjs

2. Introduction
• Does assessment data give us an
accurate picture of student knowledge?
• Do assessments leave room for possible
student misconceptions?
• What could be the possible problems with
these student misconceptions?

3. Types of Data
System Level Data
• Macro-level data: Data for groups of students,
universities. Typically the realm of academic
analytics.
Individual Level Data
• Assessment Data: Data about student performance in
class activities
Transaction Level Data
• The individual transactions that create an
assessment. Step level data.

4. Research Questions
1. How can we identify misconceptions from
student log data?
2. Does student log data tell a different story
than final answer data?

5. Progress Diagram
Individualize
Remediation
Process
Automate
Process
Actionable
Information
Manual Analysis
Phase 1 Phase 2 Phase 3

6. Data Collection
• Data collected from online Intro to Excel
Class
– www.myeducator.com
• Assessments are situated and task-based
• Step-level data for each task is captured

7. Example Student Log

8. Knowledge Components
• Syntax
• Cell Referencing
• Calculation
• Absolute
References ($)

9. Types of Errors
Optimal
Solution
TEXT TEXT TEXT
Used $ when
not needed
=$C11*$C$8
=C11*$C$8
Major Issue
Failed to use
$ when it was
needed
=C11*C8
=$C8*$C11
Used $
incorrectly
=C$11*C8
=C8*C$11
Type in value
to avoid $ use
=C11*0.0675
0.50.05
Error Weighting
Optimal Solution: =C11*C$8
0.5 0.6
Minor Issue

10. Transaction Level Data Example
D11
Error
rating D20
Error
ratin
g
Optimal Solution =C11*C$8 =F19*C$13/12
Step 1 =D11*C8 .5 =(F19*C13)/12 .5
Step 2 =C11*C8 .5
=($F$19*$C$13)/1
2
.55
Step 3 =C11*$C$8 .05 =(F19*C15)/12 .5
Step 4 =(F19*C13)/12 .5
Step 5 =F19*($C$13/12) .05
Final Solution =C11*$C$8 .05 =F19*($C$13/12) .05

11. Knowledge Gap Analysis Results
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Solution Process Final Answer
First Attempt
2nd Attempt
Amount of Error Detected
Major Errors Only

12. Knowledge Gap Analysis Results
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Persisted Resolved Emerged No Error
Solution Process
Final Answer
Only Major Errors

13. Knowledge Gap Analysis Results
0%
10%
20%
30%
40%
50%
60%
70%
80%
Persisted Resolved Emerged No Error
Solution Process
Final Answer
Including Minor Errors

14. Future Work
• Automating the process of discovering
patterns in student answers
• Give feedback to the student based on
their responses

15. Questions?
Randy.Davies@byu.edu
robnyland@byu.edu
John.Chapman@byu.net