AECT 2019

1. Evaluating Open Educational Resources:
Results of a Collaborative Evaluation of
Crowdsourced Lesson Plans
2019 AECT International Convention
Tuesday, October 22nd
1:00 Convention Center | Pavilion 11

2. Our Google Drive Folder
bit.ly/aect19eval

3. Session Overview
Project
Background
Evaluation
Findings
Evaluation
Protocols
Effects on
Practice

4. Project | Phase 02
Expert Design TeamOverview
Project Background

5. Our mission is to provide service-
learning opportunities for
instructional designers to help
nonprofit organizations create the
solutions necessary for effecting
positive change.
Designers for Learning’s
Charitable Mission

6. Multifaceted Project
Current Project
Phase 01 | Expert Review
Current Project
Phase 02 | Expert Design Teams
Research GSA Observers
Prior Years
Instructional Design Service-Learning Projects

7. 4:00 - 5:30 Today | Pavilion
Phase 01 | Expert Review
Phase 02 | Expert Design Teams
Research
GSA
Observers
D&D
Showcase

8. 10:00 Wednesday | Ballroom B
Phase 01 | Expert Review
Phase 02 | Expert Design Teams
Research
GSA
Observers
Expert
Design
Team
&
Research

9. Expert Review Team
● Managed by Designers for Learning
○ Coordinated Call for Volunteers
○ Volunteers selected based on needed expertise
● Team included:
○ Subject Matter Experts in adult basic skills education
○ College faculty and students with interest and expertise in:
■ Instructional design
■ Open educational resources
■ Evaluation

10. Expert Review Team Members
● Jennifer Englund
University of Minnesota
● Royce Kimmons
Brigham Young University
● Jennifer Maddrell
Designers for Learning
● Cherise Moore
American Institutes of Research
● Jessica Resig
Penn State University
● Emily Tolzmann
Chinese Mutual Aid Association

11. Project | Phase 02
Expert Design TeamProject | Phase 01
Expert Review

12. Examine the efficacy,
efficiency, and usability of
crowdsourced OER
developed by service-learning
volunteers
Evaluation Goal

13. Evaluation
Target
106 lesson plans for adult basic
skills education developed by
our service-learning volunteers
on OER Commons

14. Evaluation Aims
Efficacy
Examined the extent to which the
instructional presentation, learner
practice opportunities, and
embedded guidance or feedback
within the OER designed by service
learners are likely to positively
influence student learning.
Efficiency & Usability
Considered the ease with which
learners should be able to interact
with the instruction, including
issues of accessibility, user
instructions & support.

15. Evaluation Categories
Lesson
Descriptions
Accessibility
Lesson Content
Elements
Instructional
Strategies and
Assessments
Overall
Quality

16. Evaluation Protocols

17. Evaluation
Sprint
Evaluation
Pilot
Evaluation Process
1
2
BYU
Virtual &
Crowdsourced
BYU Students
Volunteer
Evaluators*
Completed
online
evaluations
Trial of
protocols &
instruments
* Including Cara North, evaluation rock star!

18. Crowdsourced Evaluation Sprint
5,533 emailed +
Social Media
Call for
Volunteers
170 people
responded
Responded to
the Call
205 evaluations
by 70 volunteers
Completed
Evaluations
November 2018 December 2018 January 2019
Mailchimp +
Facebook, Twitter &
LinkedIn
Google Forms Google Forms

19. Evaluation Methods
Machine
Analysis
Online
Evaluation
Form
Evaluator
Note
Taking

20. Evaluation
Instructions
Google Doc with instructions to
select a lesson plan from list and
how to complete the online
evaluation form. Sent to
volunteers via Mailchimp.

21. List of
Lesson Plans
Google Sheets roster of lesson
plans with evaluator signup and
link to online evaluation form

22. Evaluation
Sprint Form
Google Form with lesson plan title
and ID populated

23. Evaluation
Sprint Form
Example of review prompts

24. Evaluation
Sprint Form
Example of open response
prompts

25. Volunteer Badges via Badgr

26. Evaluation Findings

27. Accessibility
WAVE lesson plan average results (+system / -system):
18.8 / 0.7 errors
64.3 / 7.2 alerts

28. Lesson Description Evaluation
● Machine analysis of lesson descriptions versus requirements:
○ 16.7% didn’t include any of the required headings,
○ 21.3% were missing more than half of required headings, and
○ 64.8% were missing two or more.

29. Evaluation Range
1
Unacceptably
3
Acceptably
5
Very Well
42
Described / Developed

30. Lesson Description Evaluation

31. Lesson Content Evaluation
● Generally positive, ranging from 3.4 to 3.9.

32. Instructional Strategies & Assessments
● Generally positive, ranging from 3.5 to 3.8.

33. Lesson Plan Quality
● Lesson plan quality was rated with a 3.4 average (1 = major
revisions; 5 = no revisions; SD = 1.1), with a mode and median
of 4 (or 28.6% of all lesson plans), suggesting that most were
of good quality but required some minor attention in focused
areas.
● Only 6 lesson plans (5.7%) yielded an average overall rating of
5, and 17 (16%) had an average score of 4.5 or higher.

34. Overall
Quality
The service-learning
crowdsourced lesson plans
were of generally good
quality but would benefit
from some focused
attention in specific areas.
Evaluation findings suggest ...

35. Effects on Practice

36. The findings informed ...
Future
evaluation
protocols
Design of
service-
learning
OER
redesign
efforts

37. OER (re)Design by Expert Design Teams
Internet
Searches
Interview
Skills
Money
Management
for Adults
Ratios
Main Idea and
Details

38. Questions or Comments?