This is the slide deck for a presentation I made at AMCIS 2018.

1. Video Tutorials
Improve Students’
Ability to
Recover from Errors
Nathan Garrett, PhD
Woodbury University, Nathan.Garrett@Woodbury.edu
11.SIGED, IS in Education General II.
Friday, 8:30am – 10:00am

2. Abstract
This experiment compared three forms of tutorials: text, video, and segmented
videos. It asked participants to follow along with a step-by-step Excel conditional
formatting tutorial, and then to transfer this knowledge to a new problem.
Participants assigned to the text tutorial were much slower than those assigned
the video tutorials. This performance difference was mostly due to the difficulty
text participants had in detecting and recovering from errors. Error detection and
recovery may explain performance differences found in earlier studies comparing
text and video tutorials. Participants reported no difference in cognitive load
between the three instructional formats. But, those with low pre-existing Excel
skills reported higher cognitive load and made more errors on the knowledge
transfer task. This study also found that self-reported Excel competency is only
weakly correlated with actual performance on an Excel assessment.

4. Why study video tutorials?
• Few empirical studies for
procedural knowledge (Höffler
et al. 2007)
• Mixed results comparing text &
video (Meij and Meij 2014)
• Findings
 Video is harder to skim/search
 Video design affects results
 Expertise may moderate

5. Working Memory
Theory Background
• Mayer & Moreno’s Multimedia
Theory
 Words & pictures processed
separately
 Integrated in working memory
• Measure effort with Cognitive
Load Theory (CLT)
 Intrinsic load, extraneous,
germane
 Transient information effect when
info is presented too fast
• Segmentation reduces CLT
 Good for highly-interactive or
difficult material
 Reversal effect for experts
https://en.wikiversity.org/wiki/Cognitive_Theory_of_Multimedia_Learning
Language
Processing
Visual
Processing
Long-term Memory
Visual Input Audio Input
Limited & lossy
Unlimited

6. Evidence-based Video Best Practices
1. Use an appropriate and clear title
2. Use animations with voice-over narration.
3. Enable interactivity through careful pacing and enable users
to control the video
4. Preview the task in the conversational style and showing use
in context
5. Provide procedural rather than conceptual information
6. Keep tasks clear and simple, helping users build a plan, see
the interconnection between actions and system reactions,
and use highlighting to guide attention.
7. Keep videos short
8. Strengthen demonstrations with practice.
(van der Meij and van der Meij 2013)

7. Research Questions
1) What is the impact of instructional format on learning?
Does text, video, and segmented video => learning + CL
2) What underlying variables explain the impact of
instructional format on learning?
Moderating: format, prior expertise

8. Experimental Condition
Study Design
• Excel Conditional Formatting
• Experimental design
 One-on-one
 Random assignment
 Screen recording
• Participants (n=49 59)
 Business students
 14 months, 7 sections
 54 surveys
 39 usable videos
Pre-tests Post-tests
Survey Excel test Tutorial
Step-by-
step
Application CL Survey

9. Pre-tests
• Survey
 Overall
 Excel skills
• Excel test

10. Experimental Conditions
Passive Tutorial => Active Step-by-Step
Video Auto-paused videoText
All three conditions used identical words

11. Post-tests
• Excel Application (test)
• Survey on CL

12. Results

13. Validation of Excel Test/Survey

14. Self skill ratings are almost worthless
Successfully
Completed
in Excel
Performance Correlation with
Self-Reported Confidence (p>0.05)
Task Yes No 1 2 3 4 5 6 7 8 9 10
1. Use currency format 48 0 NA
2. Apply cell fill 46 2 NA
3. Change font color 42 6 NA
4. Apply red border 38 10 0.41
5. Conditional Formatting 0 48 NA
6. Add numbers 30 18 NA
7. Use Now() 28 20 NA
8. Use Sum() 42 6 0.37
9. Insert line chart 41 7 NA
10. Sort numbers 38 10 NA
Average Confidence (1-7) 6.5 6.6 6.7 6.5 4.5 6.0 5.6 6.2 6.2 6.3

15. Self skill ratings are almost worthless
Successfully
Completed
in Excel
Performance Correlation with
Self-Reported Confidence (p>0.05)
Task Yes No 1 2 3 4 5 6 7 8 9 10
1. Use currency format 48 0 NA
2. Apply cell fill 46 2 NA
3. Change font color 42 6 NA
4. Apply red border 38 10 0.41
5. Conditional Formatting 0 48 NA
6. Add numbers 30 18 NA
7. Use Now() 28 20 NA
8. Use Sum() 42 6 0.37
9. Insert line chart 41 7 NA
10. Sort numbers 38 10 NA
Average Confidence (1-7) 6.5 6.6 6.7 6.5 4.5 6.0 5.6 6.2 6.2 6.3

16. Self skill ratings are almost worthless
Successfully
Completed
in Excel
Performance Correlation with
Self-Reported Confidence (p>0.05)
Task Yes No 1 2 3 4 5 6 7 8 9 10
1. Use currency format 48 0 NA
2. Apply cell fill 46 2 NA
3. Change font color 42 6 NA
4. Apply red border 38 10 0.41
5. Conditional Formatting 0 48 NA
6. Add numbers 30 18 NA
7. Use Now() 28 20 NA
8. Use Sum() 42 6 0.37
9. Insert line chart 41 7 NA
10. Sort numbers 38 10 NA
Average Confidence (1-7) 6.5 6.6 6.7 6.5 4.5 6.0 5.6 6.2 6.2 6.3

17. Self skill ratings are almost worthless
Successfully
Completed
in Excel
Performance Correlation with
Self-Reported Confidence (p>0.05)
Task Yes No 1 2 3 4 5 6 7 8 9 10
1. Use currency format 48 0 NA
2. Apply cell fill 46 2 NA
3. Change font color 42 6 NA
4. Apply red border 38 10 0.41
5. Conditional Formatting 0 48 NA
6. Add numbers 30 18 NA
7. Use Now() 28 20 NA
8. Use Sum() 42 6 0.37
9. Insert line chart 41 7 NA
10. Sort numbers 38 10 NA
Average Confidence (1-7) 6.5 6.6 6.7 6.5 4.5 6.0 5.6 6.2 6.2 6.3

18. Self skill ratings are almost worthless
Successfully
Completed
in Excel
Performance Correlation with
Self-Reported Confidence (p>0.05)
Task Yes No 1 2 3 4 5 6 7 8 9 10
1. Use currency format 48 0 NA
2. Apply cell fill 46 2 NA
3. Change font color 42 6 NA
4. Apply red border 38 10 0.41
5. Conditional Formatting 0 48 NA
6. Add numbers 30 18 NA
7. Use Now() 28 20 NA
8. Use Sum() 42 6 0.37
9. Insert line chart 41 7 NA
10. Sort numbers 38 10 NA
Average Confidence (1-7) 6.5 6.6 6.7 6.5 4.5 6.0 5.6 6.2 6.2 6.3

19. T-test shows slightly different means
Task
Successfully
Completed in Excel
n
Survey Avg.
Confidence
(1-7)
Difference between
success and failure
t-test (assuming equal
variance)
1. Use currency format
Yes 54
NA NA
No 0 NA
2. Apply cell fill
Yes 53
NA NA
No 1 NA
3. Change font color
Yes 46 6.9
0.2 p=0.58
No 8 6.7
4. Apply red border
Yes 46 6.7
1.1 p=0.00
No 8 5.6
5. Conditional Formatting
Yes 0 NA
NA NA
No 54
6. Add numbers
Yes 36 6.4
1.0 p=0.02
No 18 5.4
7. Use Now()
Yes 33 6.0
1.0 p=0.02
No 21 5.0
8. Use Sum()
Yes 49 6.3
2.1 p=0.00
No 5 4.2
9. Insert line chart
Yes 49 6.1
-0.3 p=0.60
No 5 6.4
10. Sort numbers
Yes 45 6.5
0.9 p=0.03
No 9 5.6

20. T-test shows slightly different means
Task
Successfully
Completed in Excel
n
Survey Avg.
Confidence
(1-7)
Difference between
success and failure
t-test (assuming equal
variance)
1. Use currency format
Yes 54
NA NA
No 0 NA
2. Apply cell fill
Yes 53
NA NA
No 1 NA
3. Change font color
Yes 46 6.9
0.2 p=0.58
No 8 6.7
4. Apply red border
Yes 46 6.7
1.1 p=0.00
No 8 5.6
5. Conditional Formatting
Yes 0 NA
NA NA
No 54
6. Add numbers
Yes 36 6.4
1.0 p=0.02
No 18 5.4
7. Use Now()
Yes 33 6.0
1.0 p=0.02
No 21 5.0
8. Use Sum()
Yes 49 6.3
2.1 p=0.00
No 5 4.2
9. Insert line chart
Yes 49 6.1
-0.3 p=0.60
No 5 6.4
10. Sort numbers
Yes 45 6.5
0.9 p=0.03
No 9 5.6

21. T-test shows slightly different means
Task
Successfully
Completed in Excel
n
Survey Avg.
Confidence
(1-7)
Difference between
success and failure
t-test (assuming equal
variance)
1. Use currency format
Yes 54
NA NA
No 0 NA
2. Apply cell fill
Yes 53
NA NA
No 1 NA
3. Change font color
Yes 46 6.9
0.2 p=0.58
No 8 6.7
4. Apply red border
Yes 46 6.7
1.1 p=0.00No 8 5.6
5. Conditional Formatting
Yes 0 NA
NA NA
No 54
6. Add numbers
Yes 36 6.4
1.0 p=0.02No 18 5.4
7. Use Now()
Yes 33 6.0
1.0 p=0.02No 21 5.0
8. Use Sum()
Yes 49 6.3
2.1 p=0.00No 5 4.2
9. Insert line chart
Yes 49 6.1
-0.3 p=0.60
No 5 6.4
10. Sort numbers
Yes 45 6.5
0.9 p=0.03No 9 5.6

22. T-test shows slightly different means
Task
Successfully
Completed in Excel
n
Survey Avg.
Confidence
(1-7)
Difference between
success and failure
t-test (assuming equal
variance)
1. Use currency format
Yes 54
NA NA
No 0 NA
2. Apply cell fill
Yes 53
NA NA
No 1 NA
3. Change font color
Yes 46 6.9
0.2 p=0.58
No 8 6.7
4. Apply red border
Yes 46 6.7
1.1 p=0.00No 8 5.6
5. Conditional Formatting
Yes 0 NA
NA NA
No 54
6. Add numbers
Yes 36 6.4
1.0 p=0.02No 18 5.4
7. Use Now()
Yes 33 6.0
1.0 p=0.02No 21 5.0
8. Use Sum()
Yes 49 6.3
2.1 p=0.00No 5 4.2
9. Insert line chart
Yes 49 6.1
-0.3 p=0.60
No 5 6.4
10. Sort numbers
Yes 45 6.5
0.9 p=0.03No 9 5.6

23. Confidence != capability

24. Confidence != capability

25. Self overall ratings somewhat useful
• Weak 0.326 correlation with performance on the Excel test (p=0.029)

26. Question 1. Impact of instructional format
• Format
 Video most efficient format
 Similar cognitive load
 No evidence for interaction
between Excel skill and CL
• Higher Excel skills decreased some time & cognitive load
 CL correlation -0.289
 Time correlation
 With knowledge transfer -0.31
 With step-by-step NA
Condition N Avg. Time (s)
Text 17 318
Video 16 200
Paused Video 15 271

27. Question 2. Explanatory Variables?
• Video analysis showed error detection
and recovery explain time on task
• 14/31 made an error
 Paper errors more common and took
longer to recover
Condition
Passive
Tutorial Time
Avg.
Active
Step-by-step
Time Avg.
Step-by-step
Off-track Time
Avg.
Knowledge
Transfer Time
Avg.
Knowledge
Transfer Off-track
Time Avg.
Text 45 200 68 74 12
Paused Video 61 108 12 102 16
Video 53 71 0 77 14
Total 54 123 24 86 14

28. Condition Excel Competency Tutorial Time Step-by-step Time
Step-by-step Off-
track Time
Knowledge Transfer
Time
Knowledge
Transfer Off-track
Time
9 36 200 117 41 -
9 23 73 - 43 -
9 23 104 - 46 -
9 52 233 141 71 -
8 48 111 - 57 -
8 31 104 - 62 -
7 37 183 40 71 -
7 34 99 - 57 -
7 59 187 40 71 -
6 84 297 140 119 28
6 28 217 122 128 74
4 47 288 131 214 14
9 61 90 - 58 -
9 60 81 - 50 -
9 61 78 - 80 -
9 66 108 - 54 -
9 56 61 - 37 -
8 60 75 - 46 -
8 60 98 - 100 30
8 75 123 - 110 -
8 61 100 25 49 -
8 56 61 - 53 -
7 57 59 - 285 55
6 59 223 43 149 -
3 60 140 17 232 158
3 61 128 - 63 -
9 54 62 - 32 -
9 56 61 - 36 -
9 66 169 - 62 -
9 60 98 - 54 -
8 57 82 - 72 -
8 57 93 - 61 -
6 56 61 - 52 -
6 57 61 - 48 -
6 58 59 - 94 -
6 57 61 - 66 -
5 57 77 - 177 121
5 57 60 - 117 67
Measure
Scale1-10, 10 being
higher
Seconds Seconds Seconds Seconds Seconds
Video
Paper
PausedVideo
Higher competencyTime on tutorialStep-by-step timeKnowledge transfer timeStep-by-step off timeKnowledge transfer off time

29. Condition Excel Competency Tutorial Time Step-by-step Time
Step-by-step Off-
track Time
Knowledge Transfer
Time
Knowledge
Transfer Off-track
Time
9 36 200 117 41 -
9 23 73 - 43 -
9 23 104 - 46 -
9 52 233 141 71 -
8 48 111 - 57 -
8 31 104 - 62 -
7 37 183 40 71 -
7 34 99 - 57 -
7 59 187 40 71 -
6 84 297 140 119 28
6 28 217 122 128 74
4 47 288 131 214 14
9 61 90 - 58 -
9 60 81 - 50 -
9 61 78 - 80 -
9 66 108 - 54 -
9 56 61 - 37 -
8 60 75 - 46 -
8 60 98 - 100 30
8 75 123 - 110 -
8 61 100 25 49 -
8 56 61 - 53 -
7 57 59 - 285 55
6 59 223 43 149 -
3 60 140 17 232 158
3 61 128 - 63 -
9 54 62 - 32 -
9 56 61 - 36 -
9 66 169 - 62 -
9 60 98 - 54 -
8 57 82 - 72 -
8 57 93 - 61 -
6 56 61 - 52 -
6 57 61 - 48 -
6 58 59 - 94 -
6 57 61 - 66 -
5 57 77 - 177 121
5 57 60 - 117 67
Measure
Scale1-10, 10 being
higher
Seconds Seconds Seconds Seconds Seconds
Video
Paper
PausedVideo
Paper participants
struggled to follow
directions
Expertise didn’t
help!

30. Condition Excel Competency Tutorial Time Step-by-step Time
Step-by-step Off-
track Time
Knowledge Transfer
Time
Knowledge
Transfer Off-track
Time
9 36 200 117 41 -
9 23 73 - 43 -
9 23 104 - 46 -
9 52 233 141 71 -
8 48 111 - 57 -
8 31 104 - 62 -
7 37 183 40 71 -
7 34 99 - 57 -
7 59 187 40 71 -
6 84 297 140 119 28
6 28 217 122 128 74
4 47 288 131 214 14
9 61 90 - 58 -
9 60 81 - 50 -
9 61 78 - 80 -
9 66 108 - 54 -
9 56 61 - 37 -
8 60 75 - 46 -
8 60 98 - 100 30
8 75 123 - 110 -
8 61 100 25 49 -
8 56 61 - 53 -
7 57 59 - 285 55
6 59 223 43 149 -
3 60 140 17 232 158
3 61 128 - 63 -
9 54 62 - 32 -
9 56 61 - 36 -
9 66 169 - 62 -
9 60 98 - 54 -
8 57 82 - 72 -
8 57 93 - 61 -
6 56 61 - 52 -
6 57 61 - 48 -
6 58 59 - 94 -
6 57 61 - 66 -
5 57 77 - 177 121
5 57 60 - 117 67
Measure
Scale1-10, 10 being
higher
Seconds Seconds Seconds Seconds Seconds
Video
Paper
PausedVideo
Low expertise led
to high error rates
on the knowledge
transfer task
For all formats!

31. Limitations
Objection Response
Small n
Meta-analysis of instructional designs studies on
animation/images found mean sample of 55, median
of 40
Another found an average of 13 participants per
experimental condition
Learning style
moderator?
No experimental evidence of learning styles (Pashley
et al. 2008)
Badly designed
instruction?
Applies to all conditions

32. Conclusions
• Don’t trust self-reported Excel skills!
• Format effectiveness
 Paper increases step-by-step errors for all participants
 Video is most successful format due to error detection & recovery
 Segmented video had no benefit over traditional video. Pacing?
• Low-skill participants showed higher error rates and CL
• Participants had significant difficulty scrubbing video
• Future work
 Expertise is a moderating variable for curriculum adoption
 Error detection and recovery is a major issue