• Like
Meta analysis presentation-sim vs. no sim
Upcoming SlideShare
Loading in...5
×

Meta analysis presentation-sim vs. no sim

  • 146 views
Uploaded on

meta-analysis statistics tutorial training

meta-analysis statistics tutorial training

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
146
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
2
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Sim, no sim; sim, sim plus
  • 2. (1) Simimlation vs no simulation and (2) simulation vs simulation plus Two research questions:
  • 3. Forty-eight articles were coded. Random control experiments and quasi-experimental studies were included. In addition, it was necessary for the articles to measure achievement gains and compare means under one of two possible control conditions: Simulation vs traditional instructional treatment Simulation vs simulation plus modification The coding of articles
  • 4. Rejected articles Liu (2006) Used a counterbalanced repeated measures design and did not report results after each intervention, therefore, there is no clear sim vs. no sim that can be included in the meta-analyis Son, Robert, Goldstone (2009) The experimental and control groups receive the same sim- what is being tested is the language used in the sim (content vs intuitive descriptors) Sheehy, Wylie & Orchard (2000) No control group-pre- experimental design LambAnnetta (2010) Sierra-Fernandez & Perales-Palacios (2003) no achievement gains were measured quantitatively Jimoyiannis & Komis (2003) Did not compare group means
  • 5. |QUESTION 1: Sim vs. no sim| How do simulation instructional treatments compare to non-simulation instructional treatments? Fixed effect size = .59 (truncated), df= k-1= (26), z-value 12.43, p = .00 (results are significant) q-value = 311.29, df = k-1 = (26), p = .00 (sum of squares of within study variance) i-squared = 91.65 (measure of heterogeneity of studies) *Using a random effect size is necessary because the percentage of between studies heterogenity is quite large: 92%; for example, in terms of technology choices, design and sample size.
  • 6. Two kinds of variance Within-study variance derives mainly from the sample size of the study as reflected in the standard deviation (SD). The within-study variance (V) is the SD squared. Large studies will have smaller within-study variances and small studies will have larger within-study variances. Between-study variance can only be calculated when the studies are synthesized. It is expressed as a sum of squares (SS of within study variance). When the V for each study is summed, the result is Q. Slide authored by Bob
  • 7. Fixed effect model and random effects model Fixed effect model assumes that there is one fixed average effect size in the population and that between-study variance is sampling error. A Q-value that exceeds chance (using the Chi- squared distribution with df = k -1) indicates that the distribution is heterogeneous. Random effects model does not assume homogeneity and applies when effect sizes are assumed to estimate different populations that may vary one from another. Between-study variance represents differences among populations, not between-study variation within populations. Slide authored by Bob
  • 8. Question 1: Forest plot & effect sizes of individual studies Large studies with negative results are skewing the fixed effect model.
  • 9. Question 1: Funnel plot of standard error by Hedge's g The funnel plot looks at publication bias. The scale on the bottom demonstrates the distribution of effect sizes and the scale on the side shows standard error (sample size). When you read this chart, you want to determine if the distribution is biased across effect size and sample size. In this instance, while there are too many positive studies, the balance for sample sizes is reasonable.
  • 10. Tau -squared value 0.69 Tau squared value
  • 11. Truncated values We truncated three g-values to 2.5 because they were too high (one was 2.7, one was over 3, and one was over 4) fixed: .591 (.65 original) random: .859 (.99 original)
  • 12. Random effect size: Random effect size = .89, k = df-1 = (26), z-value 5.19, p = .00 (results are significant) *The unweighted average effect size is close to the random effect size (approx. 1.02) Note: Fixed effect size = .59, k= df-1= (26), z-value 12.43, p = .00 (results are significant)
  • 13. Moderator variables: Grade level and research design
  • 14. Moderating grade level K-5, 2: 6-8, 3: 9-12, 4: (multiple ranges) * A mixed effects analysis q-value= 7.469, df =(2), p-value = 0.024 demonstrates that grade level is a significant moderator From the data, one can conclude that simulations work better for younger learners than for older learners ** *The "4" multiple range was removed because there was only one study or effect and it wouldn't mean anything *NOTE: the younger learner category has only 2 main effects compared to 6 and 18 effect sizes for the other 2 ranges.
  • 15. Moderating research design RCT or Quasi-Experimental A mixed effects analysis, revealed a Q-between value = 0.01, df=1, p=.91 (results not significant) Research design was not a significant moderator These data are collapsed
  • 16. Results and implications: The meta-analysis of research studies comparing simulation instructional treatments to non-simulation instructional treatments rejects the null hypothesis that there is no difference in knowledge acquisition between the two treatments. The students in the bulk of the studies who received simulation training received significantly higher scores when tested for learning or mastery of the material. Instructors who provide training through simulation will most likely improve learning outcomes over those who follow traditional instructional methods. Furthermore, grade level was found to be a moderating variable, however, the sample size in the elementary age group makes reaching any definitive conclusion unwarranted. Research design was found not to be a significant moderating variable.
  • 17. QUESTION 2: Sim+modification vs. sim How do simulation alone compare with simulations with instructional enhancements. Fixed effect size = .766, df= k-1= (16), z-value 14.63, p = .00 (results are significant) q-value (sum of squares of within study variance) = 380.87, df= k-1= (16), p =.00 (sum of squares of variance) i-squared = 95.80 (measure of heterogeneity of studies)* *Using a random effect size is necessary because the percentage of between studies heterogenity is quite large: 96%; for example, in terms of technology choices, design and sample size.
  • 18. Question 2: Forest plot and effect sizes of individual studies Large studies are not skewing the results since they fall into the middle range of effect sizes.
  • 19. Question 2: Funnel plot of standard error by Hedge's g The funnel plot looks at publication bias. The scale on the bottom demonstrates the distribution of effect sizes and the scale on the side shows standard error (sample size). When you read this chart, you want to determine if the distribution is biased across effect size and sample size. In this instance, there is a balanced distribution of effect size and
  • 20. Tau -squared value 1.09 Tau squared value
  • 21. No truncation Some effect sizes were found with large weights but they were in the middle range.
  • 22. Random effect size Random effect size = 0.638, df = k-1 = (16), z-value 5.19, p =.00 (results are significant) Fixed effect size = .766, df= k-1= (16), z-value 14.63, p =.00 (results are significant) The fixed and random effect sizes are closer together than for Q1, with no large effect sizes skewing the fixed effect size Some effect sizes were found with large weights, but they were in the middle range of the forest plot and, therefore, did not dramatically skew the fixed effect size.
  • 23. Moderator variables: Grade level and research design
  • 24. Moderating grade level K-5, 2: 6-8, 3: 9-12, (multiple ranges) A mixed effects analysis reveals a q-value = 5.13, df=2, p=.08 This results demonstrates that grade level is not a significant moderator From the data, simulations plus modification does not work better for younger learners These data are collapsed
  • 25. Moderating research design RCT or Quasi-Experimental A mixed effects analysis reveals a Q-between value = 0.02, df=1, p=0.87 Research design was not a significant moderator These data are collapsed
  • 26. Results and implications: Our meta-analysis of research studies comparing modified simulation, simulation combined with additional instructional treatment, to simulation only instruction rejected the null hypothesis that there is no difference in knowledge acquisition between the two treatment methods. The students in the bulk of the studies who received simulation plus supplemental training groups received significantly higher scores when tested for learning or mastery of the material over the simulation only groups. Instructors who provide training through simulation and with additional modifications will most likely improve learning outcomes over those who follow simulation only instructional methods. Furthermore, no moderating variables were identified in terms of grade level or research design.
  • 27. A extremidade, the end, FIN