This chapter discusses evidence-based practice in instructional design. It defines evidence-based practice as using instructional techniques supported by research findings and theory. It discusses different research methods like experimental comparisons and observational analysis to test instructional features. It provides guidance on identifying relevant research studies, including focusing on similar learners, experimental designs, replication of results, application-based measurement, and consideration of practical significance. It notes that meta-analyses of multiple experiments can help determine when strong instructional effects are most likely but that continued e-learning research growth is still needed.

1. Chapter 3: Evidence-Based Practice
Nicole Beauchamp
EDT 598
Summer 2018

2. What is Evidence-Based Practice?
• Instructional techniques should be based on research findings and
research-based theory.

4. Research Method Choice:
Use what’s appropriate for the research question
WHAT?
• Experimental Comparison: Compare the test performance of people who
learned with or without the instructional feature
WHEN?
• Factorial Experimental Comparison: Compare the test performance of people who
learned with or without the instructional feature but you also vary the type of
learner, the type of learning objective or the type of learning environment for each
instructional feature
HOW?
• Observational Analysis: Carefully observe what the learner does during
learning or ask the learner to tell you about the learning principles

5. What to look for in Experimental Comparisons
Select studies that…
1. Focus on the instructional method you’re interested in
2. Are based on the same learners, materials and learning
environments as yours
3. Use the appropriate research method
4. Meet the criteria of good research methodology

6. Good Research Methodology
• Experimental control: only one variable is being tested at a time
• Random assignment: test subjects should be randomly assigned to
groups (or treatment conditions)
• Appropriate measures: the research report should tell you the mean,
standard deviation and sample size for each group on a relevant
measure of learning

7. How to Interpret No Effect in
Experimental Comparison
Consider the following six factors…

8. How to Interpret Research Statistics
• Standard deviation tells you how spread out the scores are, or how
much variation there is in the results. Powerful instructional methods
should yield high averages and low standard deviation.
• Statistical Significance: Probability (P)
• P < .05 This means there is less than a 5% chance that you are wrong and
more than a 95% chance that you are right.
• Practical Significance: Effect Size (ES)
• When the ES is less than .2 the practical impact of the experiment treatment
is a bit too small to worry about; an effect size of .5 is moderate and when it’s
.8 or above you have a large effect.

9. How Can You Identify Relevant Research?
5 questions to consider:
1. How similar are the learners in the research study of your learners?
2. Are the conclusions based on an experimental research design?
3. Are the experimental results replicated?
4. Is learning measured by tests that measure application?
5. Does the data analysis reflect practical significance as well as
statistical significance?

10. What we Don’t Know About
Evidence-Based Practice
• Meta-analysis comprised of many experiments that all test the
effectiveness of the same instructional method.
• In a meta-analysis you would record the effect size for each study and
compute an average effect size across all the studies.
• Meta-analysis can help pinpoint the conditions under which strong effects are
most likely to occur.
• Continued growth within the e–learning research base is needed for
valid meta-analysis to be conducted.

11. Conclusion: