Evaluation Of Evidence Based Practices In Online Learning


Published on

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

Evaluation Of Evidence Based Practices In Online Learning

  1. 1. Evaluation of Evidence-Based Practices in Online Learning A Meta-Analysis and Review of Online Learning Studies U.S. Department of Education http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf
  2. 2. Overview <ul><li>A meta-analysis of on-line learning. Looking specifically at </li></ul><ul><ul><li>How does the effectiveness of online learning (OL) compare with that of face-to-face (F2F) instruction? </li></ul></ul><ul><ul><ul><li>Mean effect = +0.14, p < 0.05 </li></ul></ul></ul><ul><ul><li>Does supplementing F2F instruction with OL instruction enhance learning? </li></ul></ul><ul><ul><ul><li>Mean effect = +0.35, p < 0.0001 </li></ul></ul></ul><ul><ul><li>What practices are associated with more effective OL learning? </li></ul></ul><ul><ul><ul><li>Mean effect = various, p-values = various </li></ul></ul></ul><ul><ul><li>What conditions influence the effectiveness of OL learning? </li></ul></ul><ul><ul><ul><li>None of them! </li></ul></ul></ul>
  3. 3. Observations <ul><li>The really want to find results for K-12, but alas-- (See caveats, page xii) </li></ul><ul><li>Greatest effect for undergraduate and older students, but not K-12, secondary or graduate students. Weird. </li></ul><ul><li>Hard to measure since time spent, pedagogy and curriculum are by definition different. </li></ul><ul><li>“ [T]he studies . . . do not demonstrate that online learning is superior as a medium.” Page xvii </li></ul><ul><li>“ Studies in which learners in the online condition spent more time on task than to students in the F2F condition found a greater benefit for online.” Duh. </li></ul><ul><li>I’m seeing a creepy amount of p-values of 0.05. Trying too hard? </li></ul><ul><li>Buried deeply is the definition of their effect sizes. Most as “smallish”. </li></ul>
  4. 4. Differentiation of Studies <ul><li>OL as replacement or enhancement of F2F </li></ul><ul><li>Pedagogy: Expository, Interactive, active </li></ul><ul><li>Synchronous or asynchronous </li></ul><ul><li>Can’t control conditions: Year, demographics, teacher credentials, state systems, etc. </li></ul>
  5. 5. Questions <ul><li>Quasi-Experimental </li></ul><ul><li>Scaling for progress? Yes, sort of . . . </li></ul>
  6. 6. Findings <ul><li>Best is F2F + OL (Page xv) Contradicts! </li></ul><ul><li>Purely OL = F2F + OL (Blended) Page 38 </li></ul><ul><li>Seven of eight studies found no significant differences among media combinations. Page 40 </li></ul><ul><li>Best if students given a choice of resources. Page 41 </li></ul><ul><li>Online Quizzes don’t work! Hoorah! (page xvi) </li></ul><ul><li>More media is not always better. (page xvi) </li></ul><ul><li>Learner Reflection is good . (page 44) </li></ul><ul><li>Instructors as on-line moderators are unnecessary. (page 46) </li></ul><ul><li>Don’t script online interaction (page 46) </li></ul>
  7. 7. Surprises <ul><li>One study found that synchronous OL had a negative effect, whereas asynchronous OL was significantly positive. Suggested that the negative synchronous result was due to old-fashioned techniques. Page 7 </li></ul><ul><li>Contradicts </li></ul><ul><li>Year published was not statistically significantly! Page 30 </li></ul>
  8. 8. Best Quote <ul><li>Although earlier meta-analysis of distance education found it equivalent to classroom instruction several reviewers have suggested that this pattern may chance. They argue that online learning as practiced in the 21 st century can be expected to outperform earlier forms of distance education in terms of effects on learning. </li></ul>
  9. 9. Truth in Advertising <ul><li>“Finally, readers should be cautioned that the literature on alternative online learning practices has been conducted for the most part by professors and others instructors who are conducting research using their own courses. Moreover, the combinations of technology, content and activities used in different experimental conditions have often been ad hoc rather than theory based. </li></ul>
  10. 10. <ul><ul><li>Authors Title Effect Size 95% Con Int Null (2-tail) Retention Rate (%) #Units </li></ul></ul><ul><li>g SE Lower Limit Upper Limit Z -Value Online Face-to-Face </li></ul><ul><li>LaRose(1998) Audiographic telecourses for the Web: An experiment </li></ul><ul><li>+0.070 0.281 -0.481 0.621 0.25 Unknown Unknown 49 students </li></ul><ul><li>Lowry (2007) Effects of online versus face-to-face professional development with a team-based learning community approach on teachers’ application </li></ul><ul><li>of a new instructional practice </li></ul><ul><li>-0.281 0.335 -0.937 0.370 -0.84 80 93.55 53 students </li></ul><ul><li>Mentzer & Teclehaimanot (2007) A comparison of face-to-face and Web-based classrooms </li></ul><ul><li>-0.796 0.339 -1.460 -0.131 -2.35* Unknown Unknown 36 students </li></ul><ul><li>Nguyen (2008) Randomized controlled trial of an Internet-based versus face-to-face dyspnea self-management program for patients with chronic obstructive </li></ul><ul><li>pulmonary disease: Pilot study </li></ul><ul><li>+0.292 0.316 -0.327 0.910 0.93 Unknown Unknown 39 participa </li></ul><ul><li>Ocker and Yaverbaum (1999) Asynchronous computer-mediated communication versus face-to-face collaboration: Results on student learning, quality and satisfaction </li></ul><ul><li>-0.030 0.214 -0.449 0.389 -0.14 Unknown Unknown 43 students </li></ul><ul><li>Padalino and Peres (2007) E-learning: A comparative study for knowledge apprehension among nurses </li></ul><ul><li>0.115 0.281 -0.437 0.666 0.41 Unknown Unknown 49 participa </li></ul><ul><li>Peterson and Bond (2004) Online compared to face-to-face teacher preparation for learning standards-based planning skills </li></ul><ul><li>+0.100 0.214 -0.320 0.520 0.47 Unknown Unknown 4 sections </li></ul><ul><li>Schmeeckle (2003) Online training: An evaluation of the effectiveness and efficiency of training law enforcement personnel over the Internet -0.106 0.198 -0.494 0.282 -0.53 Unknown Unknown 101 students </li></ul><ul><li>Schoenfeld-Tacher, McConnell and Graham (2001) Do no harm: A comparison of the effects of online vs. traditional delivery media on a science course +0.800 0.459 -0.100 1.700 1.74 100 99.94 Unknown </li></ul>Exhibit 4a. Purely Online Versus Face-to-Face (Category 1) Studies Included in the Meta-Analysis (continued) Notice that negative effect sizes were from studies with broad titles, whereas positive effect sizes were from studies of OL learning of specific tasks.