Issues in Experimental Design by Muhammad Rawaha Saleem

1. Issues in
experimental design

2. Introduction
Experimental design is the design of any task that aims to
describe or explain the variation of information under
conditions that are hypothesized to reflect the variation.
It is mostly used in natural sciences, psychological and
educational research; however, such designs are less
used in social sciences. We try to explore strengths and
weaknesses of experimental design.

3. Methodological issues
 The problem of explanatory narrowness
The strength of experimental design is that it
describes well ho much direct causal impact a
variable has, but it is not well suited to provide an
explanation of the results and does not allow to
build a picture of complex set of pictures but only
one or two factors.

4.  Issue of randomization
Randomization makes it difficult to establish the
role of other factors. In removing the effect of
numerous variables, it removes too much
information. This can be explained with the
example of the impact of school type on student
academic achievement. At the end of study we
could compare the academic achievement of
students in the government funded schools with
that of students in private schools.

5. However, in real world many other factors apart
from the type of school itself may contribute to
performance of students like home background
but we have eliminated that impact by
randomizing.
Randomization can underestimate the total causal
effect. We can only look at the direct effect of the
experimental variable not at the real world effects
of a variable when working indirectly. E.g. gender
to income, gender via occupation to income,
gender via hours of work to income

6.  The problem with internal validity
The purpose of the design is to structure the
collection and analysis of data to reach a clear
conclusion. When the logic and structure of a
design are faulty then the design lacks internal
validity.
All designs face threats to internal validity.
Campbell and Stanley have described them as;

7.  History
If events take place between pre-test and post-test
then these events can threaten the internal
validity to the research. The lack of control of time
is a great danger to the research as the time
between pre-test and post-test increases.
 Maturation
The maturation threat can operate when changes
occur within subjects. Both history and
maturation are a concern in studies that involve
repeated observations of the same variables.

8.  Testing
Testing participants several times can
contaminate results.
 Instrument decay
Instruments are devices used by researchers to
collect information. Examples are; questionnaires,
surveys, tests, observation, participation.
Instrument decay can be a problem to internal
validity if the nature of the instrument is changed
over time.

9.  Statistical regression
Statistical regression can be a threat to internal
validity. When these scores are particularly high
or low (i.e., they are extreme scores), there is a
tendency for these scores to move (regress)
towards the mean (the average score); so an
individual with an extremely high score during the
pre-test measurement of an experiment gets a
lower score on the post-test measurement and an
individual with an extremely low score during the
pre-test measurement gets a higher score on the
post-test measurement.

10.  Selection
Selection bias can occur when the groups that are
being compared are not similar. They differ along
a wide range of factors, such in age, behavior,
gender, height, intelligence, and so forth.
 Mortality/Dropout
Loss of subjects is threat to internal validity. Due
to illness, family relocation or death, some
individuals may drop out of the study. Subjects
may be absent during the collection of data or fail
to complete the study.

11.  Problems with external validity
Campbell and Stanley identify a number of factors
that can undermine our ability to generalize
the results of a study beyond the study itself.
 Reactive or interaction effect of pre-testing
A pre-test might increase or decrease a subject's
sensitivity or responsiveness to the experimental
variable.

12.  Unrepresentativeness
How can we be confident that the study
represents the whole population? When people or
groups are not selected randomly, representation
lacks validity.
 Artificiality
Being selected for a study can make people feel
special and produce artificial results. When study
is artificial, we learn nothing of any use about real
people in real social contexts.

13. Practical issues
Many practical questions arise when designing social
experiment.
 How much do you tell participants?
While the ethical principle suggests that we
should tell participants everything, but doing so
could make the study worthless. If participant
know what we are looking for they might behave
in such a way as to either confirm or sabotage
one’s experiment. In such cases ethical guidelines
must be adhered to.

14.  How many participants?
Different methods require different sample sizes.
If our concern is external validity, the sample
size is both randomly selected and big enough and
if our concern is internal validity, the sample size
is usually smaller.
 How should participants be recruited?
For a larger sample size we select people
randomly. The sample is best when that is
representative. But even if we use random
sampling we will not end up with a representative

15. sample because of ethical guidelines as we
cannot force anyone to participate or
withdraw from the study.
 Gaps between tests and interventions
The length of the gap should depend on the time it
would take for an effect to be detectable. It
depends on whether we are looking for short-term
or long-term effects. The gap between a pre-test
and an intervention should be should be short to
minimize the chance of events taking place.

16.  Which method of data collection?
The data collection method we adopt depend on
the nature of the observations we need to make.
What is critical is that the method of data
collection produces reliable, valid and meaningful
data. Our data collection method must produce
valid information rather than reliable data but
wrong information. There is little merit of data
when the method of data collection is getting
people to express socially desirable views rather
than their own real views.

17.  Problems with randomized assignment
The practice of randomly assigning participants is
one of the distinguishing characteristics of
experimental design.
 Refuse to participate in the intervention
When people, allocated to a given intervention,
refuse to participate, it causes problems for
intervention.
 Dropouts
Since experimental design examine change over
time, they face the danger of dropout from

18. different groups. this can be a serious problem in
long term for field based social research.
 Unevenness of interventions
When the intervention is unevenly carried out, we
do not necessarily have a good test of impact of
the intended intervention.
 The self-fulfilling prophecy
In any research there is always the danger that the
researcher’s expectations and values distort the
way he or she collects and interprets data. A
researcher’s expectation can affect what they see,

19. how they interpret it and how hard they look for
evidence contrary to their expectations.

20. Ethical issues
Social research should conform to four broad ethical
principles.
 Voluntary participation
The researcher needs to be careful to stress that
participation is truly voluntary. Participation
should not be forced and they participants should
know they can withdraw from study any time. But
voluntary participation can also produce problems
in cases of dropout or mortality.

21.  Informed consent
Participants should be informed about;
I. the purpose of the study
II. The identity of the researcher
III. The use to which the data might be put
IV. An outline of foreseeable risks
V. Description of likely benefit of study
VI. An offer to answer any questions
VII. A statement that participation is voluntary

22. While the issue of informed consent seems
entirely reasonable and desirable, but it is not
always straightforward. The problem is how fully
informed the participants be? The detailed
technical information may confuse and distract
rather than inform. Providing details about
study can distort the way people answer
questions and undermine the validity of the
findings.

23.  No harm to participants
In some experimental studies participants are
potentially exposed to harm. Foe example, in
medical experiment in which a new drug is trialed,
participants are potentially endangered. In
psychological experiment, participants are told to
behave in particular ways which can be later
regretful and disgustful for them. In 1971, Dr.
Zimbardo, a psychology professor at Stanford
University conducts a controversial experiment of
prisoners and guards to know the effects of

24. torture on prisoners. Recently a film was made
under the name of ‘The Stanford Prison
Experiment’ to show the real life event of study.
Where there is any danger of harm to participants,
they should be told before participation. s
 Anonymity and confidentiality
Any obvious way in which participants can be
harmed is by failure to honor promise of
confidentiality. People participating in
experiments are entitled to expect that they
cannot be identified as the source of any particular

25. information. Personal information, if made public,
could be embarrassing or cause harm to
participants. It is essential therefore that
information be collected in such a way that
confidentiality can be guaranteed. People are
more likely to participate in the study especially
when it is about private matters, if they are
ensured full confidentiality. If a person feels their
answers are truly confidential they will be more
likely to provide frank and honest answers. For
example in a study of school children, you might
learn the identity of a drug dealer or a teacher

26. who sexually molests children but your promise
still remains intact to provide no harm to anyone
in any condition in the study. I