Evaluation Of Evidence Based Practices In Online LearningPresentation Transcript
Evaluation of Evidence-Based Practices in Online Learning A Meta-Analysis and Review of Online Learning Studies U.S. Department of Education http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf
A meta-analysis of on-line learning. Looking specifically at
How does the effectiveness of online learning (OL) compare with that of face-to-face (F2F) instruction?
Mean effect = +0.14, p < 0.05
Does supplementing F2F instruction with OL instruction enhance learning?
Mean effect = +0.35, p < 0.0001
What practices are associated with more effective OL learning?
Mean effect = various, p-values = various
What conditions influence the effectiveness of OL learning?
None of them!
The really want to find results for K-12, but alas-- (See caveats, page xii)
Greatest effect for undergraduate and older students, but not K-12, secondary or graduate students. Weird.
Hard to measure since time spent, pedagogy and curriculum are by definition different.
“ [T]he studies . . . do not demonstrate that online learning is superior as a medium.” Page xvii
“ Studies in which learners in the online condition spent more time on task than to students in the F2F condition found a greater benefit for online.” Duh.
I’m seeing a creepy amount of p-values of 0.05. Trying too hard?
Buried deeply is the definition of their effect sizes. Most as “smallish”.
Differentiation of Studies
OL as replacement or enhancement of F2F
Pedagogy: Expository, Interactive, active
Synchronous or asynchronous
Can’t control conditions: Year, demographics, teacher credentials, state systems, etc.
Scaling for progress? Yes, sort of . . .
Best is F2F + OL (Page xv) Contradicts!
Purely OL = F2F + OL (Blended) Page 38
Seven of eight studies found no significant differences among media combinations. Page 40
Best if students given a choice of resources. Page 41
Online Quizzes don’t work! Hoorah! (page xvi)
More media is not always better. (page xvi)
Learner Reflection is good . (page 44)
Instructors as on-line moderators are unnecessary. (page 46)
Don’t script online interaction (page 46)
One study found that synchronous OL had a negative effect, whereas asynchronous OL was significantly positive. Suggested that the negative synchronous result was due to old-fashioned techniques. Page 7
Year published was not statistically significantly! Page 30
Although earlier meta-analysis of distance education found it equivalent to classroom instruction several reviewers have suggested that this pattern may chance. They argue that online learning as practiced in the 21 st century can be expected to outperform earlier forms of distance education in terms of effects on learning.
Truth in Advertising
“Finally, readers should be cautioned that the literature on alternative online learning practices has been conducted for the most part by professors and others instructors who are conducting research using their own courses. Moreover, the combinations of technology, content and activities used in different experimental conditions have often been ad hoc rather than theory based.
Authors Title Effect Size 95% Con Int Null (2-tail) Retention Rate (%) #Units
g SE Lower Limit Upper Limit Z -Value Online Face-to-Face
LaRose(1998) Audiographic telecourses for the Web: An experiment
+0.070 0.281 -0.481 0.621 0.25 Unknown Unknown 49 students
Lowry (2007) Effects of online versus face-to-face professional development with a team-based learning community approach on teachers’ application
of a new instructional practice
-0.281 0.335 -0.937 0.370 -0.84 80 93.55 53 students
Mentzer & Teclehaimanot (2007) A comparison of face-to-face and Web-based classrooms
-0.796 0.339 -1.460 -0.131 -2.35* Unknown Unknown 36 students
Nguyen (2008) Randomized controlled trial of an Internet-based versus face-to-face dyspnea self-management program for patients with chronic obstructive
Schmeeckle (2003) Online training: An evaluation of the effectiveness and efficiency of training law enforcement personnel over the Internet -0.106 0.198 -0.494 0.282 -0.53 Unknown Unknown 101 students
Schoenfeld-Tacher, McConnell and Graham (2001) Do no harm: A comparison of the effects of online vs. traditional delivery media on a science course +0.800 0.459 -0.100 1.700 1.74 100 99.94 Unknown
Exhibit 4a. Purely Online Versus Face-to-Face (Category 1) Studies Included in the Meta-Analysis (continued) Notice that negative effect sizes were from studies with broad titles, whereas positive effect sizes were from studies of OL learning of specific tasks.