This document summarizes the findings of a systematic review and meta-analysis of research on effective distance, online, and blended teaching and learning. The review identified five prominent themes in the literature: comparisons of distance/online and face-to-face instruction; factors affecting effectiveness; institutional adoption; research methods; and topic analysis. Key findings included that distance education can be as effective as traditional instruction, especially when incorporating interaction and collaboration. Blended learning showed particularly positive results. Effectiveness has increased over time with improvements in technology and pedagogy. However, more support is still needed for instructors to develop new competencies for online facilitation.

1. What does effective
distance/online/blended
teaching look like?
Vitomir Kovanović,
School of Informatics
THE UNIVERSITY of EDINBURGH
v.kovanovic@ed.ac.uk
@vkovanovic
EASS Teaching Academics Network
The University of South Australia,
Adelaide, SA, Australia
9 March 2016

2. Preparing for the Digital University
Download: http://bit.ly/pdu-report
● Authors
○ George Siemens, UTA
○ Dragan Gašević, UoE
○ Shane Dawson, UNISA
○ Charles Lang, NYU
○ Srećko Joksimović, UoE
○ Oleksandra Poquet, UNISA
○ Vitomir Kovanović, UoE
Coverage by the Chronicle of Higher Education
http://bit.ly/digital-uni
2

3. Study objectives
● Many changes in educational landscape
○ Mobile technologies
○ Personalization & feedback
○ Learning analytics
○ MOOCs
What is the current state of distance education research, as reflected through
meta-analyses and systematic reviews?
Goals:
1. Identify themes that have framed current research,
2. Summarize the current state of research and practice, and
3. Reveal prospective directions for the future
3

4. An umbrella review
Given the large number of studies, we focused on second-order studies:
● Reduces the number of studies to look
● Increases the validity of evidence
● Two types of studies investigated:
○ Meta-analyses:
■ Integrate statistical results from several studies
■ Can investigate consistency of findings
○ Systematic reviews:
■ Literature reviews with a clear inclusion criteria
4

5. Plethora of terms & modalities
● Distance/Online education/learning
● E-Learning
● Technology-enhanced learning
● Computer-aided/supported instruction/training
● Internet-based learning/training
● Blended/Hybrid learning
● Web-based learning
● Mobile learning
● M-Learning
● MOOCs (cMOOCs, xMOOCs)
● Intelligent tutoring systems
5

6. Plethora of terms & modalities
● Distance/Online education/learning
● E-Learning
● Technology-enhanced learning
● Computer-aided/supported instruction/training
● Internet-based learning/training
● Blended/Hybrid learning
● Web-based learning
● Mobile learning
● M-Learning
● MOOCs (cMOOCs, xMOOCs)
● Intelligent tutoring systems
6
One term to rule them all...
Digital Learning

7. Dataset
7
Focus on 1984-2014 period
74 studies for DE & OL
20 studies for BL

8. Five prominent themes
1. Topic analysis of the published literature,
2. Comparison of DE/OL and F2F instruction,
3. Factors affecting effectiveness of DE/OL,
4. Institutional adoption of DE/OL, and
5. DE/OL research methods.
8

9. Comparison of DE/OL and F2F instruction
Comparing student learning outcomes between DL/OL and traditional modes of
instructions prominent theme in policy discourse (Arbaugh et al. 2009);
● Pros:
○ flexibility,
○ alleviation of overcrowded classrooms,
○ increased enrollment,
○ reduced cost, and
○ increased profit
● Cons:
○ instructor training cost,
○ feelings of isolation, and
○ technology gaps (Clardy, 2009; Grandzol & Grandzol, 2006)
9

10. Comparison of DE/OL and F2F instruction
● Distance education can be at minimum as effective as traditional classroom
instruction (Arbaugh et al., 2009).
● No difference in effect size for student academic performance (Arbaugh et al.,
2009; Bernard et al., 2004a; Lou, Bernard, and Abrami, 2006)
● Importance of context (Cook et al., 2010b)
10

11. Even better than F2F?
● better mode of delivery (Shachar and Neumann, 2003; Allen et al. (2004)
● highly effective for instruction delivery to adult learners (Roberts, 2011)
● better for health professionals (Williams, 2006)
However,
● traditional modes of instruction yield higher student satisfaction levels
(Chipps et al., 2012b; Tomlinson et al., 2013; Allen et al., 2002; Bernard,
Abrami, Wade, Borokhovski, and Lou, 2004b)
11

12. Blended is the way to go
● Online learning did not produce significantly different learning outcomes than
face-to-face learning (Means et al., 2013).
○ Blended learning showed significant positive effects.
● Online learning did not outperform other educational methods
○ Improved scores on multiple-choice tests (Chumley-Jones, Dobbie, &
Alford, 2002).
12

13. Factors affecting effectiveness of DE/OL
Three categories of factors (Minnaar, 2011):
● Infrastructure and technology,
● Pedagogy, and
● Human factors.
13

14. Mode of delivery (Sync vs. Async)
Results still not conclusive, buuut:
● Asynchronous delivery >> classroom delivery >> synchronous delivery.
(Bernard, Abrami, Lou, et al., 2004; Lou et al., 2006; Allen et al., 2004)
Online discussions enable for rich student-student interactions that stimulate the
development of deep and critical thinking skills (San, 2006).
● literature being focused on graduate-level students
● lack of involvement from course instructors and the research from their
perspective.
14

15. How to improve?
● Synchronous:
○ More personalized contact between students and instructors
■ make it similar to face-to-face education (Bernard, Abrami, Wade, et
al. 2004).
● Asynchronous:
○ use of problem-based learning
■ Positive effects in both achievement and attitude (Bernard, Abrami,
Wade, et al., 2004).
○ use of media to enable collaborative learning (Lou et al. 2006)
15

16. ● “... pedagogically rich strategies that include the instructor’s involvement
and participation in the discussion; monitoring and moderating the
discussion; regular interaction with students; and facilitating learners’
interaction, collaboration, and teamwork” (Darabi et al., 2013, p. 13).
● Organizing discussions in small groups which help students to develop a
community of learners and ”feel comfortable engaging in relevant discourse”
(Ravenna, Foster, and Bishop, 2012, p. 22).
● Requirement for instructors’ participation (Ravenna et al., 2012):
○ “model respondent”,
○ to share the knowledge and
○ promote community among students
Strategies for fostering online discussions
16

17. Course domain
DE/OL was far more effective for workspace & adult professional learners than for
graduate and undergraduate students (Machtmes and Asher, 2000; Williams,
2006).
Largest effect in favor of distance education was achieved for foreign language
courses (Allen et al., 2004)
Bernard, Abrami, Lou, et al. (2004):
● best DE/OL: business, military, and computing courses
● best F2F: mathematics, science, and engineering courses.
17

18. Communication technology & media
89% of all courses made use of written materials, while 55% used multimedia
materials. 32% of all courses used asynchronous discussions, and 9%
synchronous discussions (Cook, et al., 2010a).
No significant differences arising from adoption of different technologies
(Machtmes and Asher, 2000).
Pedagogy, rather than the adopted technology, plays a dominant role in the
effectiveness of DE/OL (Bernard, Abrami, Wade, et al., 2004).
18

19. Cultural considerations
Students’ cultural background had important implications on their academic
success.
Challenges for students unfamiliar with institutional culture:
● Problems with course expectations,
● Social presence, and
● Building relationships with instructors (Uzuner, 2009).
Current focus on Asian students
19

20. Course interactions (S-S, S-T, S-C)
Importance of support for meaningful interactions among students and between
students and instructors (Wong, Greenhalgh, and Pawson, 2010; George, 2014).
All three forms of interaction produced positive effect sizes on academic
performance (Bernard et al., 2009):
● S-S and S-C had higher effect than S-T interactions.
Importance of the strong course design that guides students to deeply engage
with the content!
20

21. Course interactions (S-S, S-T, S-C)
Most effective S-S interactions:
● those designed to support collaboration between students (Borokhovski,
Tamim, Bernard, Abrami, and Sokolovskaya, 2012)
S-T interactions: Single most important course-related characteristic
● influence learners’ motivation
● overall course experience
● timely feedback and formal support particularly important
(Carroll, Booth, Papaioannou, Sutton, & Wong, 2009;
Cook et al., 2010c; Minnaar, 2011; Peterson, 2008; Styer, 2007)
21

22. Course organization and design
High interactivity (Cook et al. 2010c). Best outcomes:
● practice exercises,
● feedback,
● repetition of study materials, and
● use of online discussions
Planning of course interactions (Borokhovski et al., 2012)
● designed and planned interactions leading to higher academic performance.
22

23. Course organization and design
Positive effect (Means et al., 2013):
● Collaborative or instructor-directed instruction, rather than individual learning,
● individualized instruction, simulations, and support for students’ reflection,
No effect:
● various media elements (e.g., images and videos),
● student control,
● online quizzes,
● interactions of groups of learners, and
● scripted collaboration did not contribute to learning performance.
23

24. Student perspective
● Students value:
○ well-designed, frequently updated courses
○ courses that incorporate extrinsic motivating factors, with
tasks/examples immediately relevant for their practice,
○ a reasonable level of control and flexibility (primarily deadlines),
○ support to collaborate with their peers, and
○ high level of instructor involvement
○ summative and timely feedback (Carroll et al., 2009; Styer, 2007).
24

25. Learner control?
Learner control (ability to decide when, where, and who to learn with)
Conflicting evidence:
● Positive (Sitzmann et al., 2006; Carroll et al., 2009; Peterson, 2008)
● Negative (Landers, 2009)
25

26. Effectiveness increasing over time!
26
Machtmes and Asher (2000):
● 1960s: 0.09 SD in favor of traditional instruction
● 1990s: 0.23 SD in favor of DE/OL
Roberts (2011):
● Before 2006: 0.61 Hedges’ g
● After 2009: 0.83 Hedges’ g

27. Effectiveness increasing over time!
27
Reasons (Machtmes & Asher, 2000; Roberts, 2011):
● Overall maturation of the DE/OL field,
● Maturation of DE/OL pedagogies,
● Modes of instruction,
● Learners gaining experience and understanding of DE/OL and,
● New and more flexible technologies .

28. 28
Conceptual model

29. Challenges:
● Amount of participation in course design and facilitation seems hardly
sustainable for instructors (Moallem, 2003).
● Competencies (Koch, 2014):
○ technical and didactical competencies,
○ ability to deliver engaging and interactive course content,
○ ability to anticipate students’ need and facilitate asynchronous learning,
○ use of innovative teaching methods,
○ ability to project teaching presence within an online environment,
○ ability to mediate asynchronous interaction and create a community,
○ high degree of self-organization, and
○ ways of assessing students’ learning and participation.
29

30. Challenges
● Real challenge for instructors is not the “mastery of these competencies, but
rather in using them to create a high quality e-learning experience with added
value for the student” (Koch, 2014, p.4).
● Instructor's role also changes “from that of the content expert to that of a
facilitator of learning processes, who guides and coaches students to help
them achieve learning objectives” (Koch, 2014, p.3).
30

31. Summary
Effective learning will not occur if online courses “are designed in such a way that
traditional F2F methods (textbook readings, lectures, examinations) are published
on the Internet without considering social isolation, de-individualized instruction,
and using technology for the sake of technology” (Cohen, Carbone, and Beffa-
Negrini, 2011, p.9).
The real question is not “how does effective learning looks like?” but
“how to support instructors to teach more effectively? “
31

32. Thank you
32

33. References I
Arbaugh, J. B., Godfrey, M., Johnson, M., Pollack, B. L., Niendorf, B., & Wresch, W. (2009). Research in online and blended
learning in the business disciplines: Key findings and possible future directions. The Internet and Higher Education, 12(2),
71 – 87. http://doi.org/http://dx.doi.org/10.1016/j.iheduc.2009.06.006
Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., … Huang, B. (2004a). How Does Distance
Education Compare with Classroom Instruction? A Meta-Analysis of the Empirical Literature. Review of Educational
Research, 74(3), 379–439.
Bernard, R. M., Abrami, P. C., Wade, A., Borokhovski, E., & Lou, Y. (2004b). The Effects of Synchronous and Asynchronous
Distance Education: A Meta-Analytical Assessment of Simonson’s “Equivalency Theory.” In 2004 Annual proceedings of
selected research and development papers presented at the national convention of the Association for Educational
Communications and Technology (pp. 102–110).
Borokhovski, E., Tamim, R., Bernard, R. M., Abrami, P. C., & Sokolovskaya, A. (2012). Are contextual and designed student–
student interaction treatments equally effective in distance education? Distance Education, 33(3), 311–329. http://doi.
org/10.1080/01587919.2012.723162
Carroll, C., Booth, A., Papaioannou, D., Sutton, A., & Wong, R. (2009). UK health-care professionals’ experience of on-line
learning techniques: A systematic review of qualitative data. Journal of Continuing Education in the Health Professions, 29
(4), 235–241. http://doi.org/10.1002/chp.20041
33

34. References II
Chipps, J., Ramlall, S., & Mars, M. (2012b). Videoconference-based education for psychiatry registrars at the University of
KwaZulu-Natal, South Africa. African Journal of Psychiatry, 15(4). http://doi.org/10.4314/ajpsy.v15i4.32
Chumley-Jones, H. S., Dobbie, A., & Alford, C. L. (2002). Web-based learning: Sound educational method or hype? A review of the
evaluation literature. Academic Medicine, 77(Suppl10), S86–S93. http://doi.org/10.1097/00001888-200210001-00028
Cook, D. A., Levinson, A. J., & Garside, S. (2010b). Time and learning efficiency in Internet-based learning: a systematic review
and meta-analysis. Advances in Health Sciences Education, 15(5), 755–770. http://doi.org/10.1007/s10459-010-9231-x
Darabi, A., Liang, X., Suryavanshi, R., & Yurekli, H. (2013). Effectiveness of Online Discussion Strategies: A Meta-Analysis.
American Journal of Distance Education, 27(4), 228–241. http://doi.org/10.1080/08923647.2013.837651
George, B. (2014). Identification of the Constraints and Barriers to the Adoption of Distributed Design Education.
Landers, R. N. (2009). Traditional, web-based, and hybrid instruction: A comparison of training methods. University of
Minnesota.
Lou, Y., Bernard, R. M., & Abrami, P. C. (2006). Media and Pedagogy in Undergraduate Distance Education: A Theory-Based
Meta-Analysis of Empirical Literature. Educational Technology Research and Development, 54(2), 141–176. http://doi.org/10.
1007/s11423-006-8252-x
34

35. References III
Machtmes, K., & Asher, J. W. (2000). A meta‐analysis of the effectiveness of telecourses in distance education. American
Journal of Distance Education, 14(1), 27–46.
Means, B., Toyama, Y., Murphy, R. F., & Baki, M. (2013). The Effectiveness of Online and Blended Learning: A Meta-Analysis of
the Empirical Literature. Teachers College Record, 115(3).
Minnaar, A. (2011). Student support in e-learning courses in higher education – insights from a metasynthesis “A pedagogy of
panic attacks.” Africa Education Review, 8(3), 483–503. http://doi.org/10.1080/18146627.2011.618664
Peterson, D. S. (2008). A meta-analytic study of adult self-directed learning and online nursing education: A review of research
from 1995 to 2007. Capella University.
Ravenna, G., Foster, C., & Bishop, C. (2012). Increasing Student Interaction Online: A Review of the Literature in Teacher
Education Programs. Journal of Technology and Teacher Education, 20(2), 177–203.
Shachar, M., & Neumann, Y. (2003). Differences Between Traditional and Distance Education Academic Performances: A Meta-
Analytic Approach. The International Review of Research in Open and Distance Learning, 4(2).
San, M. M. P. (2006). Looking for critical thinking in online threaded discussions. E-JIST : E-Journal of Instructional Science and
Technology, 9(2).
35

36. References IV
Styer, A. J. (2007). A grounded meta-analysis of adult learner motivation in online learning from the perspective of the learner.
Capella University.
Tomlinson, J., Shaw, T., Munro, A., Johnson, R., Madden, D. L., Phillips, R., & McGregor, D. (2013). How does tele-learning
compare with other forms of education delivery? A systematic review of tele-learning educational outcomes for health
professionals. New South Wales Public Health Bulletin, 24(2), 70–75. http://doi.org/10.1071/NB12076
Uzuner, S. (2009). Questions of Culture in Distance Learning: A Research Review. The International Review of Research in Open
and Distance Learning, 10(3).
Williams, S. L. (2006). The Effectiveness of Distance Education in Allied Health Science Programs: A Meta-Analysis of
Outcomes. American Journal of Distance Education, 20(3), 127–141. http://doi.org/10.1207/s15389286ajde2003_2
Wong, G., Greenhalgh, T., & Pawson, R. (2010). Internet-based medical education: a realist review of what works, for whom and
in what circumstances. BMC Medical Education, 10(1), 12.
36

37. Questions?
email: v.kovanovic@ed.ac.uk
twitter: @vkovanovic
37