Chapter 7 Effect Size and Power Making Sense of Statistical Significance
An effect can be statistically significant without having much practical significance. Effect Size It is a measure of the difference between populations. It tells us how much something changes after a specific intervention. It indicates the extent to which two populations do not overlap. how much populations are separated due to the experimental procedure With a smaller effect size, the populations will overlap more. Effect Size
Raw Score Effect Size calculated by taking the difference between the Population 1 mean and the Population 2 mean Standardized Effect Size calculated by dividing the raw score effect size for each study by each study’s population standard deviation This standardizes the difference between means in the same way a Z-score gives us a way to compare two scores on different measures. Figuring The Effect Size
Effect Size = Population 1 M – Population 2 M Population SD Population 1 M = the mean for the population that receives the experimental manipulation Population 2 M = the mean of the known population (the basis for the comparison distribution) Population SD = the standard deviation of the population of individuals A negative effect size would mean that the mean of Population 1 is lower than the mean of Population 2. Formula for Calculating the Effect Size
Standard rules about what to consider a small, medium, and large effect size based on what is typical in behavioral and social science research Cohen’s effect size conventions for mean differences: Effect Size Conventions
A More General Importance of Effect Size Knowing the effect size of a study lets you compare results with effect sizes found in other studies, even when the other studies have different population standard deviations. Knowing what is a small or a large effect size helps you evaluate the overall importance of a result--- PRACTICAL SIGNIFICANCE! A result may be statistically significant without having a very large effect. Meta-Analysis a procedure that combines results from different studies, even results using different methods or measurements This is a quantitative rather than a qualitative review of the literature. Effect sizes are a crucial part of this procedure.
Statistical Power-The Ability to Achieve Your Goals! Probability that the study will produce a statistically significant result when the research hypothesis is really true When a study has only a small chance of being significant even if the research hypothesis is true, the study has low power. When a study has a high chance of being significant when the study hypothesis is actually true, the study has high power.
If there is a is a mean difference in the population, you have more chance of getting a significant result in the study. Since the difference between population means is the main component of effect size, the bigger the effect size, the greater the power. Effect size is also determined by the standard deviation of a population. The smaller the standard deviation, the bigger the effect size. The smaller the standard deviation, the greater the power. What determines the Power of a Study?Effect Size and Power
The more people there are in the study, the greater the power is. The larger the sample size, the smaller the standard deviation of the distribution of means becomes. The smaller the standard deviation of the distribution of means, the narrower the distribution of means—and the less overlap there is between distributions leading to higher power. Remember that though sample size and effect size both influence power, they have nothing to do with each other. Sample Size
Statistical Significance vs. Practical Significance It is possible for a study with a small effect size to be significant. Though the results are statistically significant , they may not have any practical significance. e.g., if you tested a psychological treatment and your result is not big enough to make a difference that matters when treating patients Evaluating the practical significance of study results is important when studying hypotheses that have practical implications. e.g., whether a therapy treatment works, whether a particular math tutoring program actually helps to improve math skills, or whether sending mailing reminders increases the number of people who respond to the Census Statistical Significance vs. Practical Significance
With a small sample size, if a result is statistically significant, it is likely to be practically significant. In a study with a large sample size, the effect size should also be considered. More things to think about….
A nonsignificant result from a study with low power is truly inconclusive. A nonsignificant result from a study with high power suggests that: the research hypothesis is false or there is less of an effect than was predicted when calculating power Role of Power When a Result is Not Statistically Significant