This presentation talks about different threats involved in a experimental process or study. Threats to internal and external validity of an experimental work is also discussed. Also a few techniques of controlling the threats have been also presented.

1. Concept of Experimental Research-Part-II
Threats to Experimentation
© V. Singh
Vikramjit Singh, PhD

2. Potential Challenges in Conducting
Experimental Research
•Lack of exposure
• Clear/operational definition of IV
•Internal Validity
•External Validity
Am I am conducting
Activity Based Class or
Cooperative Learning
Class???
© V. Singh

3. Principle of Control in Experimentation.
The main function of the experimental research design
is to control variance which can be done in three ways.
• Principle of Control
• maximize systematic variance,
• control extraneous systematic variance,
• and minimize error variance.
© V. Singh

4. In Experimental Work
The researcher works on maximizing the variance of the
variable (s) under investigation.
The investigator studies the maximum or optimum possible
difference in the outcome (dependent variable) caused by
the treatment (independent variable) working at optimum
level or maximum level.
<a href='https://www.freepik.com/photos/school'>School photo created by stockking - www.freepik.com</a>
<a href='https://www.freepik.com/vectors/school'>School vector created by pch.vector - www.freepik.com</a>
© V. Singh

5. In Experimental Work
In addition to that the researcher needs to
remove or minimize the error or random
variance (i.e., avoid unreliable measurement
instruments which have high errors of
measurement also the experimentation
process has to be well defined and carried).
I was knowing that
this measuring tape
has some measuring
error
© V. Singh

6. In Experimental Work
The investigator has to control the
variance of extraneous or "unwanted”
variables that may have an effect on the
experimental outcomes, thus he has to
limit the variables other than IV causing
difference in the outcome(DV)
Image Courtesy-https://images.indianexpress.com/2018/12/Gujarat-education.jpg
© V. Singh

7. Control of Extraneous Variables
• Matching: Here we try to match both the groups based on certain set-up
criteria which are likely to influence the study like IQ, Age etc. so that
both the groups have similar types of participants.
• Randomization: It is another method of control where we assign and
distribute participants in experimental and control group randomly.
• Eliminate the variables : Another way of controlling the extraneous variables
is to eliminate those variable which may abrupt the study findings. For
example if we are worried of gender differences among the groups try to
include a particular gender in the study.
© V. Singh

8. Control of Extraneous Variables
• Analysis of Covariance: We can also statistically equate the
groups using ANCOVA (Analysis of Covariance) techniques
which try to control the initial difference existing in the
groups before the treatment.
• Build into the Group: Suppose we are worried about gender
influence in the study , we can make it a moderator variable
and by using Two way ANOVA(Analysis of Variance) we can
do the study and justify the findings.
© V. Singh

9. Threats to Experimentation.
• Internal validity: is the degree to which observed differences
on the dependent variable are a direct result of manipulation of
the independent variable, not some other variable.
• External validity: also called Ecological validity & Population
validity is the degree to which study results are generalizable, or
applicable, to groups and environments outside the experimental
setting.
Truth in
the Study
Truth in
Real Life
Internal Validity External Validity
Generalizability
© V. Singh

10. Threats to Internal Validity
• History: The incidences during the treatment period
experienced by the experimental group or control
group may influence the study and thus is a threat to
internal validity of the findings.
© V. Singh

11. Threats to Internal Validity
•Maturation: Changes occurring in the participants
like growing older, wiser, bored, tired, more
experienced etc. during the study effects the sensitivity
of the study.
© V. Singh

12. Threats to Internal Validity
•Testing/Pre-test: Testing the participants
before the treatment may lead towards
alteration in the actual test result after the
treatment to observe the IV effect.
Photo by Dominika Roseclay from Pexels
© V. Singh

13. Threats to Internal Validity
• Instrumentation: The testing instruments may not
be scores similarly. Perhaps the person grading
the posttest is fatigued and pays less attention to
the last set of papers reviewed.The testing tool or
instrument should be error free or it should have
the required validity and reliability or else the
internal validity of the experiment is under threat.
© V. Singh

14. Threats to Internal Validity
• Statistical regression: As we know extreme score both
towards higher or towards the lower end tends to regress
towards the mean. Thus we need to be very careful while
interpreting the post-test results.
For Ex- An individual with extreme tall height will be having a
comparatively shorter height child and similarly a extremely short heighted
person will be having comparatively taller child.
© V. Singh

15. Threats to Internal Validity
• Differential Selection of the Participants:
Participants in the experimental and control
groups have different characteristics that may
affect the dependent variable differently.
Difference between the two groups may be
deliberate/biased or naturally and thus it needs
to be checked too.
© V. Singh

16. Threats to Internal Validity
• Subject Mortality: Participants in the
experimental and control groups may
drop out at different time of the
treatment duration and thus it may
influence the effect created on DV by
the IV.
Exposure 1
Exposure 2
Exposure 3
Exposure 4
Exposure 5
© V. Singh

17. Threats to Internal Validity
Hawthorne Effect
The subjects may only respond differently after knowing the
fact that they(participants) are under observation . This
name comes from a very old study in which researchers were
studying the effect of lighting on worker productivity. As the
intensity of the factory lights increased, so did the worker
productivity. One researcher suggested that they reverse the
treatment and lower the lights. The productivity of the
workers continued to increase. It may be interpreted that
merely the fact that they are being observed, the productivity
of the workers is increasing and not due to the intensity of
the lights.
© V. Singh

18. Threats to Internal Validity
•John Henry Effect
One group may view that it is in competition with the other group and
may work harder than they would under normal circumstances. This
generally is applied to the control group "taking on" the treatment
group.
The terms refers to the classic old story of John Henry laying railroad
track.
I will definitely lift it I
don’t need coaching
My coach has told me to
practice lifting like this
© V. Singh

19. Threats to Internal Validity
• Resentful Demoralization of the Control
Group
Here the control group may feel discouragement as they are not
receiving any treatment and thus play at lower level than usual
and thus the results may be influenced and interpretation will be
not valid.
I don’t have any coach I
should give up weight lifting
My coach has told me to
practice lifting like this
© V. Singh

20. Threats to Internal Validity
• Compensatory Equalization of Treatment
The researcher may feel sad that the control group is not getting
some treatment and give them some special treatment. For
example suppose we want to study effect of ABL on students
attitude towards science and while observing that the control
group is not getting attention he may show the control group a
science fiction, which may in turn influence the expected results.
© V. Singh

21. Threats to Internal Validity
• Implementation
The process of manipulation has to be very well
understood and accordingly done . If we say that
we are studying the effect of flipped learning on
the students the researcher has to prepare his
instructional plan accordingly implement it if fails
to plan the lesson and also to implement it then the
study fails to give a valid finding.
Am I am
conducting Activity
Based Class or
Cooperative
Learning Class???
© V. Singh

22. Threats to Internal Validity
There are several other threats to internal Validity –
• Data Collectors/ Researchers Characteristics
•Data collectors biasedness
•Location
•Experimental –treatment diffusion
© V. Singh

23. Threats to
External Validity
© V. Singh

24. Threats to External Validity
• Pretest- Sensitisation: It may be there that
because the groups are tested before the
experimental treatment, the group , specially
the experimental group has acted differently
and thus the findings may not be replicated in
general. We have appeared the test
before we know how to
respond and do better
© V. Singh

25. Threats to External Validity
Multiple treatment sensitisation: If a researcher were to apply
several treatments, it is difficult to determine how well each
of the treatments would work individually. It might be that
only the combination of the treatments is effective.
When participants receive more than one treatment it may be
there that the prior treatment limit the effect of the current
treatment and thus exact effect can be misunderstood and
may not be generalized.
© V. Singh

26. Threats to External Validity
Explicit description of the experimental treatment: If
the researcher has not clearly or specifically defined
the independent variable and also has not adequately
describe how he or she conducted a study, it is
difficult to determine whether the results are
applicable to other settings.
© V. Singh

27. Threats to External Validity
Selection treatment interaction: If the researcher
has selected the participant using non- random
process or if the participants have voluntarily
participated in the study then it is sure that the
generalisation ability of the study shall be effected.
© V. Singh

28. Threats to External Validity
Experimenter Effect: It is also true that the specific
characteristics of the researcher/experimenter might
have caused different effect which may not be
replicated in other settings. I can see
Yellow
I can see
brown
© V. Singh

29. Threats to External Validity
Reactive Agents: The fact that they(subjects) are being
study or not being the part of experimental treatment
affects the participants to behave in way which is
different from a normal behaviour. Hawthrone and
John Henry Effect are example of reactive responses
of being in a study.
© V. Singh

30. Thanks!
Any questions?
You can find me at:
singhvikramjit@hotmail.com
Mob: +91-9438574139