Small Sample vs Large Sample Formula Even with a really small sample size of 10, using either formula gave us exactly the same p values.
Unpaired t test vs Welch’s test That’s our starting point, we will then stress the difference in sample size and standard deviation (variance) between Pilots and Controls.
Unpaired t test vs Welch’s test stressing differences in variance and sample size The multiples were both applied on the Controls . Notice how both effects are in the same direction. They both show that the t test slightly understates the P value relative to the Welch’s test. But, the effects counter each other. This is because the key metric is the Standard Error = Standard Deviation/SQRT(n).
Unpaired t test vs Welch’s test stressing differences in variance and sample size. 2 nd try In an attempt to stress the difference between the two test P values, we applied the sample multiple to the Controls and the Standard deviation multiple to the Pilots . It really did not do that much.
Conclusion <ul><li>The unpaired t test (large sample formula) is pretty robust as it does not deviate much if at all from the other mentioned tests even when samples are very small, unequal, and variance and standard deviations are very different. </li></ul><ul><li>Given that this unpaired t test is far more intuitive than the others, it makes sense to use it more frequently than the others. </li></ul>
Far more important considerations… <ul><li>… are whether the samples come from populations that are normally distributed. If not, you have to use a nonparametric test instead (i.e. Mann-Whitney test). </li></ul><ul><li>… how material the difference between Pilots and Controls really is. This leads you away from p values and towards Effect Size and Confidence Intervals. </li></ul>