Evaluating the Effectiveness of
Web-Based Learning Tools for
Online Learning
Dr. Robin Kay
UOIT
Oshawa, Canada
Website: faculty.uoit.ca/kay/home

UOIT
• Started in 2003
• 6000+ Students
• Laptop based university
• Masters and PhD
programs online

Overview
• What is a Web-Based Learning Tool?
• Why do we need to Evaluate WBLTs?
• Previous Research
• WBLT Evaluation Scale
• Using the Scale
• Study
• Results
• Conclusions

What is a Web-Based Learning Tool?
• Sample 1 – Tool - Boyle’s Law
• Sample 2 – Practice - Animal Cells
• Sample 3 – Demonstration – Trigonometry
Which one was the best tool?

Brief Definition of WBLT
“Interactive tool that supports the
learning of specific concepts by enhancing,
amplifying, and/or guiding the cognitive
processes of learners”

Why Evaluate WBLTs?
1. Quality assurance
2. Reduces search time
3. Encourages effective reuse

Previous Evaluation Research
Typical Pattern
• Informal & anecdotal
• Limited number of WBLTs evaluated
• Reliability and validity missing
Evaluation Studies
• Reviewed research and key models
• List of Criteria

WBLT Evaluate Scale
Construct Criteria
Interactivity • Meaningful Interactions
• Control
• Multimedia that adds to learning
Design • Consistency
• Layout
• Labeling
• Readability
Engagement • Quality of feedback
• Attractive
• Graphics (not video)
• Learning modes
• Motivation
Usability • Natural to use
• Orientation
• Navigation cues
• Instructions
• Language level

Using the Scale
• Sample 1 – Tool - Boyle’s Law
• Use the Hand out to evaluate:
▫ Interactivity
▫ Design
▫ Engagement
▫ Usability

LOEM Study
4 Raters
• 4 teachers – 3 days of training using scale
• Inter-rater reliability – 94% to 96%
Students
• 1113 students (588 males, 525 females)
• Secondary School
• 64 different classrooms
• 44 different WBLTs

Results - Reliability
Construct Number of Items Reliability
Interactivity 3 0.70
Design 4 0.74
Engagement 5 0.77
Usability 5 0.80

Results – Validity

Results – Student Evaluations
Construct S-Learn S-Quality S-Engagement
Interactivity 0.06 0.08 -0.08
Design 0.25 * 0.30 * 0.20 *
Engagement 0.24 * 0.32 * 0.27 *
Usability 0.28 * 0.27 * 0.15 **

Results – Teacher Evaluations
Construct T-Learn T-Quality T-Engagement
Interactivity 0.06 0.19 * -0.10
Design 0.15** 0.19 * 0.06
Engagement 0.33 * 0.20 * 0.29 *
Usability 0.12 0.40 * -0.13

Results – Student Performance
Construct Student Performance
Interactivity 0.28 *
Design 0.43 *
Engagement 0.37 *
Usability 0.40 *

Conclusions
• WBLT Evaluation Scale
▫ Fair reliability
▫ Good validity
▫ Agreed with student and teacher perceptions
▫ Predicted student performance

Practical Implications
• When evaluating WBLTs
▫ Interactivity
▫ Design
▫ Engagement
▫ Usability
• Can be used to evaluate WBLTs ahead of time to
save educators time
• Starts a more rigorous discussion about quality
of WBLTs

Future Research - Resources
Future Research
▫ Instructional Design
▫ Clark & Mayer (2008)
Website
▫ faculty.uoit.ca/kay/wblt
Contact Information
▫ robin.kay@uoit.ca