The document summarizes a meta-analysis that compared simulation-based instruction to non-simulation instruction and simulation alone to simulation with modifications. For the first comparison, simulation was found to significantly outperform non-simulation, with younger students benefiting more. For the second comparison, simulation with modifications significantly outperformed simulation alone, with no differences found by grade level or study design. The analysis rejected the null hypothesis that there is no difference between the instructional methods in both comparisons.

1. Sim, no sim; sim, sim plus

2. (1) Simimlation vs no simulation and (2) simulation vs simulation
plus
Two research questions:

3. Forty-eight articles were coded.
Random control experiments and quasi-experimental studies were included.
In addition, it was necessary for the articles to measure achievement gains and
compare means under one of two possible control conditions:
Simulation vs traditional instructional treatment
Simulation vs simulation plus modification
The coding of articles

4. Rejected articles
Liu (2006) Used a counterbalanced repeated measures design
and did not report results after each intervention, therefore,
there is no clear sim vs. no sim that can be included in the
meta-analyis
Son, Robert, Goldstone (2009) The experimental and control
groups receive the same sim- what is being tested is the
language used in the sim (content vs intuitive descriptors)
Sheehy, Wylie & Orchard (2000) No control group-pre-
experimental design
LambAnnetta (2010)
Sierra-Fernandez & Perales-Palacios (2003) no achievement
gains were measured quantitatively
Jimoyiannis & Komis (2003) Did not compare group means

5. |QUESTION 1: Sim vs. no
sim|
How do simulation instructional treatments compare to
non-simulation instructional treatments?
Fixed effect size = .59 (truncated), df= k-1= (26), z-value
12.43, p = .00 (results are significant)
q-value = 311.29, df = k-1 = (26), p = .00 (sum of squares
of within study variance)
i-squared = 91.65 (measure of heterogeneity of studies)
*Using a random effect size is necessary because the
percentage of between studies heterogenity is quite large:
92%; for example, in terms of technology choices, design
and sample size.

6. Two kinds of variance
Within-study variance derives mainly from the sample size of the
study as reflected in the standard deviation (SD). The within-study
variance (V) is the SD squared.
Large studies will have smaller within-study variances and small
studies will have larger within-study variances.
Between-study variance can only be calculated when the studies are
synthesized.
It is expressed as a sum of squares (SS of within study variance).
When the V for each study is summed, the result is Q.
Slide authored
by Bob

7. Fixed effect model and random effects model
Fixed effect model assumes that there is one
fixed average effect size in the population and that
between-study variance is sampling error.
A Q-value that exceeds chance (using the Chi-
squared distribution with df = k -1) indicates that
the distribution is heterogeneous.
Random effects model does not assume
homogeneity and applies when effect sizes are
assumed to estimate different populations that
may vary one from another.
Between-study variance represents differences
among populations, not between-study variation
within populations.
Slide authored
by Bob

8. Question 1: Forest plot & effect sizes of individual studies
Large studies with negative results
are skewing the fixed effect model.

9. Question 1: Funnel plot of standard error by Hedge's g
The funnel plot looks at publication
bias.
The scale on the bottom demonstrates
the distribution of effect sizes and the
scale on the side shows standard error
(sample size).
When you read this chart, you want to
determine if the distribution is biased
across effect size and sample size.
In this instance, while there are too
many positive studies, the balance for
sample sizes is reasonable.

10. Tau -squared value
0.69 Tau squared value

11. Truncated values
We truncated three g-values to 2.5 because they were too
high (one was 2.7, one was over 3, and one was over 4)
fixed: .591 (.65 original)
random: .859 (.99 original)

12. Random effect size:
Random effect size = .89, k = df-1 = (26), z-value
5.19, p = .00 (results are significant)
*The unweighted average effect size is close to the
random effect size (approx. 1.02)
Note:
Fixed effect size = .59, k= df-1= (26), z-value 12.43,
p = .00 (results are significant)

13. Moderator variables:
Grade level and research design

14. Moderating grade
level
K-5, 2: 6-8, 3: 9-12, 4: (multiple ranges) *
A mixed effects analysis q-value= 7.469, df =(2), p-value
= 0.024 demonstrates that grade level is a significant
moderator
From the data, one can conclude that simulations work
better for younger learners than for older learners **
*The "4" multiple range was removed because there was only one study or effect and it
wouldn't mean anything
*NOTE: the younger learner category has only 2 main effects compared to 6 and 18
effect sizes for the other 2 ranges.

15. Moderating research design
RCT or Quasi-Experimental
A mixed effects analysis, revealed a Q-between
value = 0.01, df=1, p=.91 (results not significant)
Research design was not a significant moderator
These data are collapsed

16. Results and implications:
The meta-analysis of research studies comparing simulation
instructional treatments to non-simulation instructional treatments
rejects the null hypothesis that there is no difference in knowledge
acquisition between the two treatments. The students in the bulk of the
studies who received simulation training received significantly higher
scores when tested for learning or mastery of the material.
Instructors who provide training through simulation will most likely
improve learning outcomes over those who follow traditional
instructional methods. Furthermore, grade level was found to be a
moderating variable, however, the sample size in the elementary age
group makes reaching any definitive conclusion unwarranted. Research
design was found not to be a significant moderating variable.

17. QUESTION 2: Sim+modification vs.
sim
How do simulation alone compare with simulations
with instructional enhancements.
Fixed effect size = .766, df= k-1= (16), z-value 14.63, p =
.00 (results are significant)
q-value (sum of squares of within study variance) = 380.87,
df= k-1= (16), p =.00 (sum of squares of variance)
i-squared = 95.80 (measure of heterogeneity of studies)*
*Using a random effect size is necessary because the
percentage of between studies heterogenity is quite large:
96%; for example, in terms of technology choices, design
and sample size.

18. Question 2: Forest plot and effect sizes of individual
studies
Large studies are not skewing the
results since they fall into the
middle range of effect sizes.

19. Question 2: Funnel plot of standard error by Hedge's g
The funnel plot looks at publication
bias.
The scale on the bottom
demonstrates the distribution of
effect sizes and the scale on the
side shows standard error (sample
size).
When you read this chart, you want
to determine if the distribution is
biased across effect size and sample
size.
In this instance, there is a balanced
distribution of effect size and

20. Tau -squared value
1.09 Tau squared value

21. No truncation
Some effect sizes were found with large weights but they were in the
middle range.

22. Random effect size
Random effect size = 0.638, df = k-1 = (16), z-value 5.19,
p =.00 (results are significant)
Fixed effect size = .766, df= k-1= (16), z-value 14.63, p
=.00 (results are significant)
The fixed and random effect sizes are closer together
than for Q1, with no large effect sizes skewing the fixed
effect size
Some effect sizes were found with large weights, but they
were in the middle range of the forest plot and, therefore,
did not dramatically skew the fixed effect size.

23. Moderator variables:
Grade level and research design

24. Moderating grade level
K-5, 2: 6-8, 3: 9-12, (multiple ranges)
A mixed effects analysis reveals a q-value = 5.13, df=2, p=.08
This results demonstrates that grade level is not a significant
moderator
From the data, simulations plus modification does not work better
for younger learners
These data are collapsed

25. Moderating research design
RCT or Quasi-Experimental
A mixed effects analysis reveals a Q-between value =
0.02, df=1, p=0.87
Research design was not a significant moderator
These data are collapsed

26. Results and implications:
Our meta-analysis of research studies comparing modified
simulation, simulation combined with additional instructional
treatment, to simulation only instruction rejected the null
hypothesis that there is no difference in knowledge acquisition
between the two treatment methods. The students in the bulk of
the studies who received simulation plus supplemental training
groups received significantly higher scores when tested for
learning or mastery of the material over the simulation only
groups.
Instructors who provide training through simulation and with
additional modifications will most likely improve learning
outcomes over those who follow simulation only instructional
methods. Furthermore, no moderating variables were identified
in terms of grade level or research design.

27. A extremidade, the end, FIN