1. Unit IX – EDUC 201
Testing Statistical Significance
2. Statistical significance
is used to provide evidence concerning the plausibility of the
null hypothesis, which hypothesizes that there is nothing more
than random chance at work in the
data. Statistical hypothesis testing is used to determine
whether the result of a data set is statistically significant.
3. Tests for statistical significance are used to
address the question:
What is the probability that what we think is a
relationship between two variables is really just a
chance occurrence?
4. Tests for statistical significance
tell us what the probability is
that the relationship we think
we have found is due only to
random chance.
They tell us what the
probability is that we would be
making an error if we assume
that we have found that a
relationship exists.
5. We can never be completely 100% certain that a
relationship exists between two variables.
There are too many sources of error to be controlled,
for example, sampling error, researcher bias, problems
with reliability and validity, simple mistakes, etc.
6. But using probability theory and the normal
curve, we can estimate the probability of being
wrong, if we assume that our finding a
relationship is true.
If the probability of being wrong is small, then we
say that our observation of the relationship is a
statistically significant finding.
7. Statistical significance means that there is a good
chance that we are right in finding that a relationship
exists between two variables. But statistical
significance is not the same as practical significance.
We can have a statistically significant finding, but the
implications of that finding may have no practical
application.
The researcher must always examine both the
statistical and the practical significance of any
research finding.
8. For example, we may find that there is a statistically
significant relationship between a citizen's age and
satisfaction with city recreation services. It may be that
older citizens are 5% less satisfied than younger
citizens with city recreation services. But is 5% a large
enough difference to be concerned about?
Often times, when differences are small but
statistically significant, it is due to a very large sample
size; in a sample of a smaller size, the differences
would not be enough to be statistically significant.
9. Steps in Testing for Statistical
Significance
1) State the Research Hypothesis
2) State the Null Hypothesis
3) Select a probability of error level (alpha level)
4) Select and compute the test for statistical
significance
5) Interpret the results
10. 1) State the Research Hypothesis
A research hypothesis states the expected relationship between two variables. It
may be stated in general terms, or it may include dimensions of direction and
magnitude. For example,
General: The length of the job training program is related to the rate of job
placement of trainees.
Direction: The longer the training program, the higher the rate of job placement
of trainees.
Magnitude: Longer training programs will place twice as many trainees into jobs
as shorter programs.
11. 2) State the Null Hypothesis
A null hypothesis usually states that there is no relationship
between the two variables. For example,
There is no relationship between the length of the job
training program and the rate of job placement of
trainees.
Graduate assistant pay is not influenced by gender.
A null hypothesis may also state that the relationship
proposed in the research hypothesis is not true.
12. Researchers use a null hypothesis in research
because it is easier to disprove a null hypothesis
than it is to prove a research hypothesis.
The null hypothesis is the researcher's "straw man."
That is, it is easier to show that something is false
once than to show that something is always true.
It is easier to find disconfirming evidence against
the null hypothesis than to find confirming evidence
for the research hypothesis.
13. 3) TYPE I AND TYPE II ERRORS
Even in the best research project, there is always a
possibility (hopefully a small one) that the researcher will
make a mistake regarding the relationship between the
two variables. There are two possible mistakes or
errors.
The first is called a Type I error. This occurs when the
researcher assumes that a relationship exists when in fact
the evidence is that it does not. In a Type I error, the
researcher should accept the null hypothesis and reject
the research hypothesis, but the opposite occurs. The
probability of committing a Type I error is called alpha.
14. The second is called a Type II error. This occurs when the researcher
assumes that a relationship does not exist when in fact the evidence is
that it does. In a Type II error, the researcher should reject the null
hypothesis and accept the research hypothesis, but the opposite occurs.
The probability of committing a Type II error is called beta.
Generally, reducing the possibility of committing a Type I error
increases the possibility of committing a Type II error and vice versa,
reducing the possibility of committing a Type II error increases the
possibility of committing a Type I error.
Researchers generally try to minimize Type I errors, because when a
researcher assumes a relationship exists when one really does not, things
may be worse off than before. In Type II errors, the researcher misses an
opportunity to confirm that a relationship exists, but is no worse off than
before.
15. 4) The Chi Square Test
For nominal and ordinal data, Chi Square is used
as a test for statistical significance. For example,
we hypothesize that there is a relationship
between the type of training program attended
and the job placement success of trainees.
16. 5) Interpret the results
If the computed value for Chi Square equals
or exceeds the value indicated in the table
for the given level of alpha and degrees of
freedom, then the researcher can assume
that the observed relationship between the
two variables exists (at the specified level of
probability of error, or alpha), and reject the
null hypothesis. This gives support to the
research hypothesis.
17. Summary
1) state the research hypothesis:
There is a relationship between the type of training program
attended and the job placement success of trainees
2) state the null hypothesis:
There is no relationship between the type of training program
attended and the job placement success of trainees
3) calculate the test for statistical significance
4) calculate the degrees of freedom of the contingency table
5) select the level of alpha
6) look up the Chi Square
7) interpret the result
18. Tests for statistical significance are used to estimate the
probability that a relationship observed in the data
occurred only by chance; the probability that the variables
are really unrelated in the population. They can be used to
filter out unpromising hypotheses.
Tests for statistical significance are used because they
constitute a common yardstick that can be understood by
a great many people, and they communicate essential
information about a research project that can be
compared to the findings of other projects.
19. However, they do not assure that the research has
been carefully designed and executed. In fact, tests for
statistical significance may be misleading, because they
are precise numbers. But they have no relationship to
the practical significance of the findings of the research.
Finally, one must always use measures of association
along with tests for statistical significance. The latter
estimate the probability that the relationship exists;
while the former estimate the strength (and sometimes
the direction) of the relationship. Each has its use, and
they are best when used together.
