This document discusses research design methods in survey and experimental research. It describes key aspects of the survey method, including that it is a descriptive design that collects self-reported data through questions administered via interviews or questionnaires. Steps in survey design are outlined, including defining objectives, sampling, distribution of questionnaires, and follow-ups. Experimental design is described as the only method that can establish cause-and-effect through manipulation of independent variables and measurement of dependent variables. Types of experimental designs and threats to internal and external validity are summarized.

1. 1. Research Design in Survey &
Experimental Research
2. Internal & External Threats to
Validity of Research
Ms. Zehra Habib
MPhil, ETE (3751)
30th May, 2022

2. Research Design: Survey Method
• With respect to data collection method, survey method of
research falls under category of Descriptive Design
• Survey method is a method of gathering information in form of
self-report with the help of questions.
• Questions may be administered as
1. Interview approach: researcher asks questions and also may get
additional information
2. Questionnaire approach: researcher collect quick responses from larger
number of participants (face to face/email)
• This method is used to study opinion, attitudes and social facts.

3. Steps to follow in survey design
1) Define objectives.
2) It is necessary to make decisions about the total amount of the time , money and
personnel available before conducting survey.
3) Choose and develop tools for gathering data i.e. questionnaire and interview
4) Sampling - The random sampling is used. It ensures adequate representation of
the population.
5) Either e-mailed questionnaire or distributed in hardcopy along with a brief cover
letter which should establish credibility of the researcher and the study.
6) Follow-up: It is seen that only 40% -60% respondents return questionnaire, so
after 2-4 weeks follow up letters should be sent to those who have not responded.
7) Non-respondents: In most surveys there will be a percentage of people who fail to
return the questionnaire, called non-respondents. If possible then they should be
interviewed.

4. Research Design: Experimental Method
• This is the only research which establishes the cause-and-effect relationship
among variables.
• Most powerful research method is experiment
• This research is unique from two respect:
a) It directly attempts to influence a particular variable
b) It is the best type for testing hypotheses about cause-effect relationships.
• Independent variable (manipulated)
• Dependent variable (measured)
• Extraneous variables are those variables that may have some effect on a
dependent variable yet are not independent variables.

5. Group in Experimental Design
Control groups are essential to experimental design.
When researchers are interested in the impact of a new treatment,
they randomly divide their study participants into at least two
groups:
• The treatment group (also called the experimental group) receives the
treatment whose effect the researcher is interested in.
• The control group receives either no treatment, a standard treatment
whose effect is already known.

6. Types of Experimental Design
1. Pre-Experimental Design:
• A group kept under observation after implementing factors of cause and effect. This
research is conducted to understand whether further investigation is necessary for these
particular groups.
2. True- experimental design:
• relies on statistical analysis to prove or disprove a hypothesis
• This experimental research method commonly occurs in the physical
sciences.
3. Quasi-experimental design:
• the participants are not randomly assigned and they are used in settings where
randomization is difficult or impossible.
• Uses in educational research, where administrators are unwilling to allow the random
selection of students for experimental samples.

9. Internal Validity
• It refers to the degree of confidence that the causal relationship being tested is trustworthy
and not influenced by other factors or variables.
• Extraneous effects lower the internal validity of an experimental study.
• Following are the threats to internal validity that results from extraneous variable
1. History 5. Statistical Regression 9. Experimenter effects
2. Maturation 6. Differential Selection 10. Subject effects
3. Testing 7. Attrition 11. Diffusion
4. Instrumentation 8. Selection-maturation

10. Threats to Internal Validity
1. History
• Unexpected events that are not part of the experimental manipulation but influence
the dependent variable.
• It refers to events occur during the study, e.g., participant watch science show or visit
to science exhibition/field trip/zoo, may affect the variable of researcher i.e. comparing
the effectiveness of different methods of Science Instruction
2. Maturation
• It refers passage of time in which participants naturally changes their behavior over
time.
• During the six-month experiment, employees become more experienced and better at
their jobs. Therefore, job satisfaction may improve.
3. Testing
• Improvement in performance resulting from familiarity with a test, when dependent
variable focuses on factual knowledge, concrete skills, etc.
• develop two different test paper that are comparably same or comparable version of a
test and can be used as pretest and posttest.

11. Threats, Continue ….
4. Instrumentation
• Changes in performance of participants resulting from changes in the measurement
used.
• For instance, two moderator may give the instructions differently, OR pretest/posttest
differs in length, difficulty or interest to the participants may affect results of
intervention
5. Statistical Regression
• Tendency for the participants who make extreme scores on pretest measures to score
nearer to the mean at posttest.
• Participants with the highest pretest scores have nowhere to go down, whereas
participants with the lowest scores can only improve.

12. Continue ….
6. Differential selection
• Pre-existing differences between experimental and control groups such as ability level,
difference in experience and age.
• Threat may be the change in posttest result is due to initial group differences rather
than intervention of independent variable
7. Attrition
• Withdrawal or loss of participants from a study due to illness, declining motivation, or
resentment at being assigned to a particular condition.
• Validity will be compromised if mortality is extensive, or if there is a selective loss of
participants across group.
8. Selection-maturation
• Differential selection effects that result specifically from maturation.
• Selection-driven participants change the result. For example, the two different teachers
selected for two different classes represents an initial group difference (a differential
selection effect) that creates a difference in student progress (maturation effect)

13. Continue …
9. Experimenter effects
• Unwanted influence or expectation, the researcher has on a study as well as
experimenter characteristics and behavior that affect the way participants respond.
• Presence of researcher during the study
10. Subject effects
• Changes in participant attitudes and behavior towards experimenter
• Hawthorne effect: any change in behavior that occur when individuals are aware of
receiving special attention by researchers during their participation in experiment.
11. Diffusion
• Communication between groups results in sharing of information and resources
between them.
• Sharing of resources of information between participants of control and experimental
group

14. External Validity
• It refers to the extent to which
experimental findings/conclusion
can be generalized beyond the
original study i.e. other situations,
people, settings
• There are two types:
1. Population
• Results can be generalized to a larger group
of individuals.
2. Ecological
• Results can be generalized to a broader set of
environmental condition.

15. Threats to External Validity
Following are three types of interactions involving the experimental manipulation
(Cool & Campbell, 1979; Parker, 1993):
1. Selection-treatment interaction
• It refers to interaction between participant characteristics and experimental treatment.
• Effects depends on the nature of participants. (E.g. parenting intervention benefit low-
income families might not work for middle-class families)
2. Setting-treatment interaction
• It refers to interaction between the experimental setting and experimental treatment.
• Effects depends on the nature of experimental setting. (e.g. academic enrichment program
may effective for students with facilities rather than without facilities)
3. Pretest-treatment interaction
• It refers to interaction between participation in the pre-test and experimental treatment.
• Exposure to a pretest may change response of participants to experimental treatment.
(e.g. due to appearing in pre-test of behavioral issues of high-school students, during
intervention group change their attitude rather than those who do not appear)

16. Thank you!