1. The document discusses survey research methods in psychology. It covers topics like survey research designs, sampling methods, questionnaire construction, reliability and validity. 2. Repeated measures designs are experimental designs where each participant experiences each condition. This within-subjects design reduces error variation and needs fewer participants compared to between-subjects designs. 3. Counterbalancing is used to control for practice and order effects in repeated measures designs. Complete counterbalancing balances effects within each participant through randomization or ABBA orders. Incomplete designs balance effects across participants.

1. Research Methods in Psychology
Survey Research
1
Survey Research
Survey research
Describe thoughts, opinions, feelings
Allows predictions based on correlations
Questionnaires
Predetermined set of questions
Sample represents a population
Examine survey procedures and analyses for sources of bias
2
Sampling in Survey Research
Use sample to represent the larger population
“Representative”: similar to
Requires careful selection of a sample
Goal: Generalize survey findings from representative sample to
the population
3

2. Basic Terms of Sampling (p. 141)
Population
Set of all cases of interest (e.g., all students on a college
campus)
Sampling Frame
List of the members of a population (e.g., registrar’s list of
enrolled students)
Sample
Subset of population drawn from sampling frame
Element
Each member of the population
4
Biased Samples
A biased sample
Characteristics of the sample differ systematically from those of
the population
Sample over-represents or under-represents segment(s) of a
population
Population is 50% urban, 30% suburban, and 20% rural
In a sample of 200 people, which one of the following would be
representative? (p. 142)
50 rural, 70 suburban, and 80 urban
20 rural, 85 suburban, and 105 urban
40 rural, 60 suburban, and 100 urban
5

3. Biased Samples
Two sources
Selection bias
Researcher’s procedures for selecting sample cause bias
Response-rate bias
Individuals selected for the sample do not complete the survey
6
Biased or Unbiased Sample Selection?
A graduate student interested in sports psychology investigates
fan reactions to a proposed change in the name of a local sports
team to make it more culturally sensitive. As fans exit a game,
he selects every 10th person to ask about the proposed name
change. From his findings, he concludes: “Local sports fans do
not want to see a name change.”
Biased or Unbiased Sample Selection?
A psychology professor offers extra credit for students in her
class who attend a campus movie on the topic of adolescent
depression and suicide. At the end of the movie a graduate
student assistant asks if they liked this way of earning extra
credit. Nine out of 10 students raise their hands, and the
graduate student reports to the professor: “Almost everyone in
your class likes this way of earning extra credit.”
Approaches to Sampling
“Sampling”

4. Procedures used to obtain a sample
Two basic approaches
Probability sampling
Nonprobability sampling
9
Probability sampling
All members (elements) of population have an equal chance of
being selected for the survey
Simple random sample (p. 146)
Random selection, random-digit dialing
Stratified random sample
Divide population into strata and sample proportionally (e.g.,
freshman, sophomore, etc.)
Improves representativeness of sample
Approaches to Sampling, continued
10
Nonprobability sampling
No guarantee each member of population has an equal chance of
being in the sample
“Convenience sampling”
Individuals are available and willing to respond to the survey
Example: magazine surveys, call-in radio/TV surveys (see page

5. 145), person on the street
Sample likely not representative of population
Approaches to Sampling, continued
11
Probability vs. Nonprobability Sampling
If we want to know the views of UG students on the college’s
recycling efforts
Go to cafeteria and sample 50 students (Probability or
nonprobability? Why?)
Randomly select students from the registrar’s list (Probability
or nonprobability? Why?)
Survey Methods
Four methods for obtaining survey data
Mail surveys
Pros--easy to administer and complete, no interviewer bias,
good for personal topics
Cons—no control over order of completing, respondents can’t
ask questions, low response rate
Personal interviews
Pros--control over questioning, asking for clarification, follow-
up on ambiguous open-ended answers, higher response rate
Cons--cost and interviewer bias (e.g., how follow-up questions
are asked)

6. 13
Survey Methods (continued)
Telephone interviews
Pros—same pros of personal interviews, reach more people
(better access), interviewers can be better supervised (if from
one site)
Cons—selection bias (not everyone has a phone, some have
multiple phones), interviewer bias, differential responding
(faceless voice), low response rate (voicemail, caller ID)
Internet surveys
Pros--efficiency/cost, ability to survey cross culturally
Cons--response rate bias (lower than for mail and telephone),
selection bias (have computer access), lack of control
(Understand instructions? Answering conscientiously? Multiple
submissions?)
Survey Research Designs
“Research design”
A plan for conducting a research project
Choose method best suited for answering a particular question
Three types of survey research designs
Cross-sectional design
Successive independent samples design
Longitudinal design
15

7. Cross-sectional survey design
Select sample from one or more populations at one time
Survey responses are used to
Describe population (descriptive statistics)
Compare responses across two or more populations
Make predictions for the population (correlations)
at that one point in time
Cannot assess change over time
Survey Research Designs, continued
16
Survey Research Designs, continued
Successive independent samples design (presidential approval;
college student values survey)
A series of cross-sectional surveys over time
A different sample from the population completes the survey
each time.
Each sample is selected from the same population.
Responses from each sample are used to describe changes in the
population over time.
Problem: noncomparable successive samples
17
Survey Research Designs, continued
Longitudinal survey design (Ex: teachers)
Same sample of individuals completes the survey at different

8. points in time
Assess how individuals change over time
Responses from the sample are generalized to describe changes
over time in the population.
Problems: attrition and reactivity (try to look consistent; trying
to hide problems)
18
Measures in Survey Research
Questionnaires
Most frequently used to collect survey data
Measure different types of variables
Demographic variables using checklists (e.g., ethnicity, age,
SES)
Preferences, opinions, and attitudes
Self-report rating scales (assume interval level of measurement)
All measures must be reliable and valid.
19
Reliability and Validity
Reliability refers to consistency of measurement.
Test-retest reliability
Administer measure two times to same sample
High correlation between the two sets of scores indicates good
reliability (r > .80)

9. 20
Reliability and Validity, continued
How to improve reliability?
More items
Greater variability among individuals on the factor being
measured (soccer example)
Testing situation free of distractions
Clear instructions
Unambiguously worded items
A measure can be reliable but not valid.
21
Reliability and Validity, continued
Validity refers to the truthfulness of a measure. (class test item
not addressing material covered in class—not valid?)
Assesses what it is intended to measure
Construct validity
Instrument measures the theoretical construct it was designed to
measure.
22
Reliability and Validity, continued
Establishing construct validity:

10. Convergent validity
Extent to which two measures of the same construct are
correlated (go together)
Discriminant validity
Extent to which two measures of different constructs are not
correlated (do not go together)
23
Constructing a Questionnaire
Best choice for selecting a measure
Use measure already shown to be reliable and valid in previous
research.
If no suitable measure is found, create a questionnaire or
measure.
Creating a reliable and valid questionnaire is not easy.
24
Constructing a Questionnaire, continued
Important first steps
Decide what information should be sought.
Decide how to administer the questionnaire.
Write a first draft of the questionnaire.
Reexamine and revise questionnaire based upon expert advice
(survey and content experts).
Pretest the questionnaire.
Review results and edit the questionnaire.

11. 25
Constructing a Questionnaire, continued
Next steps: Establish reliability and validity
Reliability
Test and re-test questionnaire using sample and conditions
similar to planned survey.
Validity
Convergent: Administer questionnaire with measures of
theoretically related constructs
Discriminant: Administer questionnaire with measures of
theoretically unrelated constructs
26
Constructing a Questionnaire, continued
Guidelines for Writing Survey Questions
Choose how participants will respond
Free-response (more expressive, difficult to score) or closed
questions (easy to score, responses don’t capture one’s view)
Use simple, familiar vocabulary
Write clear, specific questions
Avoid double-barreled questions
Place conditional phrases at the beginning of sentences (If you
were forced to leave your job…)
Avoid leading questions and loaded questions
Avoid response bias (positive and negative wording)

12. 27
Constructing a Questionnaire, continued
Ordering of questions
Self-administered questionnaires
Place most interesting questions first
Personal and telephone interviews
Demographic questions first
Use funnel questions and filter questions as needed.
28
Correspondence Between Reported and Actual Behavior
Survey responses may not be truthful.
Reactivity
Social desirability
Accept people’s responses as truthful unless there’s reason to
suspect otherwise.
Use a multimethod approach to answering research questions
(check records/archives to see if survey results match behavior).
Thinking Critically About Survey Research
29
Thinking Critically About Survey Research, continued
Correlation and causality

13. “Correlation does not imply causality”
Three possible causal inferences for any correlation
A causes B
B causes A
Variable C causes both A and B
30
Research Methods in Psychology
Repeated Measures Designs
1
Repeated Measures Designs
Each individual participates in each condition of the experiment
Completes the DV measure with each condition
Hence “repeated measures”
Also called “within-subject” design
Entire experiment is conducted “within” each subject
2
Repeated Measures Designs, continued
Advantages
No need to balance individual differences across conditions of

14. experiment
Fewer participants needed
Convenient and efficient
Measure changes in participants’ behaviors over time
More sensitive design
3
Sensitivity
A sensitive experiment
Can detect effects of IV even when IV has small effect
“Error variation” is reduced
Same people participate in each condition
Variability due to individual differences eliminated
4
Practice Effects
Disadvantage: practice effects
People change as they are tested repeatedly.
Performance may improve over time.
People may become bored or tired as number of “trials”
increases.
Practice effects become a potential confounding variable if not
controlled.
5
Practice Effects, continued

15. Example
Suppose a researcher compares two different study methods, A
and B.
Condition A: Participants use a highlighter while reading a text,
then take a test on the material.
Condition B: Participants read a text and make up sample test
questions and answers, then take a test on the material.
6
Practice Effects, continued
Suppose
All participants first experience Condition A and then Condition
B
Results indicate test scores are higher in Condition A compared
to Condition B
Confounding of IV with order of presentation
Can’t determine effect of IV
Practice effects (boredom, fatigue) may explain poorer
performance in Condition B
7
Practice Effects, continued
Balance practice effects across conditions.
Counterbalance the order of conditions
Half of the participants do Condition A, then B
The remaining participants do Condition B, then A
Distribute practice effects equally across conditions.
Practice effects aren’t eliminated.
Balance, or average, practice effects across the conditions of the
experiment.

16. 8
Counterbalancing Practice Effects
Two types of repeated measures designs
Complete and Incomplete
Purpose of each type: counterbalance practice effects
9
Complete Repeated Measures Design
Balance practice effects within each participant.
Each participant experiences each condition several times.
Each participant forms a “complete experiment.”
Use different orders each time
Use when each condition is brief
e.g., simple judgments about stimuli
10
Complete Design, continued
Two methods for generating orders of conditions
Block randomization
ABBA counterbalancing
11
Complete Design, continued

17. Block randomization
Block = all conditions of an IV
e.g., 4 conditions: A, B, C, D (e.g., control; hand-held,
hands-free, passenger)
Generate a random order of the block (ACBD)
Participant completes condition A, then C, then B, then D
Generate new random order for each time participant completes
conditions of experiment
12
Complete Design, continued
Block randomization
In general, the number of blocks is equal to the number of times
each condition is administered, and the size of each block is
equal to the number of conditions.
Practice effects are averaged across the many presentations of
the conditions
Requires many presentations to balance practice effects
13
Complete Design, continued
ABBA counterbalancing
Use when conditions are presented only a few times to each
participant
Use random sequence of conditions (e.g., DACB)
Then present opposite of sequence (BCAD)
Repeat with new random sequence and opposite, etc.
Each condition has same amount of practice effects.

18. 14
Complete Design, continued
Use ABBA counterbalancing only if practice effects are
“linear”
Linear practice effects
Participants change in the same way with each presentation of a
condition.
Nonlinear practice effects
Participants change dramatically with the administration of a
condition (e.g., “aha”)
Confounding between practice effects and IV
Use block randomization
15
Complete Design, continued
Do not use ABBA counterbalancing when anticipation effects
can occur.
Participants form expectations about which condition will
appear next in sequence.
Responses may be influenced by expectations (not IV).
If anticipation effects are likely (e.g., conditions are
predictable), use block randomization.
16
Incomplete Repeated Measures Design
Each participant experiences each IV condition once.
Balance practice effects across participants (not within).

19. General rule for balancing practice effects
Each IV condition must appear in each ordinal position (1st,
2nd, 3rd) equally often.
17
Incomplete Design, continued
Two techniques for balancing practice effects
All possible orders
Selected orders
18
Incomplete Design, continued
All possible orders
Use with 4 or fewer IV conditions (e.g., control; hand-held,
hands-free, passenger)
2 conditions (A, B) → 2 possible orders: AB, BA
Randomly assign half of the participants to do condition A first,
then B; other half: B then A
3 conditions (A, B, C) → 6 possible orders
Randomly assign participants to one of the six orders
4 conditions (A, B, C, D) → 24 possible orders
Need at least 1 participant randomly assigned to each order
19
Incomplete Design, continued

20. Selected orders
Select particular orders of conditions to balance practice effects
Two methods
Latin Square
Random starting order with rotation
Each IV condition appears in each ordinal position exactly
once.
Randomly assign each participant to one of the orders of
conditions.
20
Latin Square (example)
We select the number of orders that is some multiple of the
number of conditions.
Each condition appears in each ordinal position once to balance
practice effects.
Order of Conditions
1st 2nd 3rd 4th
A B D C
B C A D
C D B A
D A C B
Incomplete Design, continued
21
Incomplete Design, continued
Another advantage of Latin Square:

21. 1st 2nd 3rd 4th
A B D C
B C A D
C D B A
D A C B
Each condition precedes and follows every other condition once
(e.g., AB and BA, BC and CB)
This helps control for potential order effects (p. 258)
22
Incomplete Design, continued
Random starting order with rotation
Generate random order of conditions (ABCD)
Rotation: move each condition one position
1st 2nd 3rd 4th
A B C D
B C D A
C D A B
D A B C
Each condition appears in each ordinal position
Order of conditions is not balanced
23
The Problem of Differential Transfer
Do not use repeated measures designs when differential transfer
is possible.
Effects of one condition persist and affect participants’
experience of subsequent conditions (problem solving
experiment)

22. Use independent groups design instead
24
All answers are worth 3 points unless otherwise noted.
1. A researcher wants to study the developmental changes in
social and political attitudes that occur during the time a person
is in college. The researcher is interested in both overall trends
and in the changes that individual students undergo. Which
survey research design will this researcher need to use?
A. observational design
B. cross-sectional design
C. successive independent samples design
D. longitudinal design
2. In a study that investigates the effects of two different doses
of a drug on memory performance, memory performance
represents the ____ variable and doses of the drug represent the
____ variable.
A. correlational; confounding
B. experimental; control
C. dependent; independent
D. independent; dependent
3. The validity of a questionnaire concerns

23. A. if it measures what it was designed to measure.
B. how consistently it measures.
C. removing any confounds.
D. whether or not it causes response bias.
4. The problem of _____ in a repeated measures design occurs
when the effects of a condition persist or carry over to affect
performance in the subsequent conditions.
A. sensitivity
B. anticipation effects
C. differential transfer
D. counterbalancing
5. Researchers have studied changes in the values and goals of
college freshman from 1966 to the present by drawing random
samples from each year's incoming class. Which survey research
design describes this research?
A. stratified random sampling design
B. cross-sectional design
C. successive independent samples design
D. longitudinal design
6. A health psychologist conducts an experiment to test the
effectiveness of three techniques for helping a person to relax.
The psychologist has a limited number of people available to
participate in the experiment, and each relaxation technique
takes a long time to complete. The psychologist has decided,
therefore, to use the incomplete repeated-measures design with
all possible orders to balance practice effects. What is the
minimum number of participants the psychologist will need for
this experiment?
A. 4 B. 6 C. 8 D. 24

24. 7. To determine which candidate for governor residents will
vote for, a survey study was conducted by phone by a polling
company. Ten percent of the state’s population makes over
$200,000; 20% makes between $100,000 and $200,000; 50%
makes between $50,000 and $100,000; and 20% makes below
$50,000. Which sampling strategy makes sure all segments of
the population are adequately surveyed?
A. random B. non-probability C. prorated random D.
stratified random
8. Each of the three techniques that are used to balance practice
effects in the incomplete repeated measures design conforms to
a general rule that can be stated as
A. each condition must appear in only one ordinal position.
B. each condition must appear in the first ordinal position
equally often.
C. each condition must appear in each ordinal position at least
twice.
D. each condition must appear in each ordinal position equally
often.
9. Match the definitions below with the proper terms.
__ Sampling frame __Population __ Sample __
Element
a) each member of a population
b) list of the members of a population
c) set of all cases of interest
d) subset of population drawn from a sampling frame
10. Which of the following samples would be representative of
a population that is 75% women and 25%
men?

25. A. 60 women, 20 men
B. 155 women, 25 men
C. 225 women, 75 men
D. all of these
E. none of these
11. Survey researchers are most interested in
A. the responses of individuals in their sample.
B. obtaining a very large sample size.
C. generalizing responses from a sample to the population.
D. predicting the size of the sampling frame based on the
sample.
12. A researcher chose to administer a survey using the
internet. This sample likely overrepresents people who own a
computer and underrepresents people who don't own a
computer. This problem in the researcher's survey is best
described as
A. response bias.
B. a stratified sample.
C. selection bias.
D. a spurious relationship.
13. Which of the following is NOT one of the reasons
researchers choose to use the repeated measures design?
A. Repeated measures designs are generally less sensitive than
are independent groups designs.
B. Repeated measures designs are more convenient and
efficient.
C. Repeated measures designs require fewer subjects.
D. Repeated measures designs are needed when the
experimental procedures require participants to compare two or

26. more stimuli.
14. In a survey of 500 randomly selected respondents, 360
respondents (72%) said they want the United Nations (U.N.)
headquarters to remain in the United States. A TV news show
conducted a call-in survey and asked the same question. Of the
186,000 who phoned in their response, 124,620 people (67%)
said they want the U.N. out of the United States. The most
reasonable statement about these different findings is that
A. the call-in survey results should be believed because of the
larger number of people who responded they want the U.N. out
of the United States.
B. the large sample size for the call-in survey indicates it is a
more representative sample.
C. the findings for the randomly selected sample more likely
represent the views of the population.
D. call-in surveys lead to more reliable findings.
15. In a nationwide study conducted in 2017, samples of
students were asked their opinions about the quality of their
college education. The results of the survey were analyzed for
differences among students at the various colleges and
universities. What survey design was used in this study?
A. correlated samples design
B. cross-sectional design
C. successive independent samples design
D. longitudinal design
16. A student is considering doing a complete repeated
measures design experiment involving motor skills. The
student's advisor has told him that people show a large initial
improvement on the task followed by slow steady improvement
after this initial change. The student must choose a technique
for balancing practice effects. Which technique should the
student NOT use?

27. A. ABBA counterbalancing
B. block randomization
C. Latin Square
D. all possible orders
17. A researcher designs an experiment with six conditions, but
each condition takes only a very short amount
of time to administer. The researcher has the opportunity,
therefore, to administer many trials of each condition in the
experiment. Which of the following techniques is likely to be
most effective for balancing practice effects?
A. ABBA counterbalancing
B. block randomization
C. stratified randomization
D. all possible orders of conditions
18. A researcher plans to conduct a complete repeated-
measures design in which the independent variable is
manipulated using four conditions that vary the level of
violence depicted in a photograph (none, low, medium, high).
There are 10 photographs in each of the four conditions. Each
participant will rate each photo once. If the researcher wishes to
have 80 ratings for each photo, how many participants should
the researcher recruit?
A. 2 B. 10 C. 40
D. 80
19. Balancing the order of conditions in a repeated measures
design
A. equates practice effects with those found in random groups
designs.
B. eliminates practice effects in the conditions of the
experiment.
C. averages practice effects across the conditions of the

28. experiment.
D. balances individual differences variables across the
conditions of the experiment.
20. A marketing researcher wants to test the effectiveness of
three different types of advertisement. Each participant will
respond to 50 examples of each type of advertisement so that
the researcher can determine each person's preference. What
type of design is this researcher using?
A. conditional repeated measures design
B. partial repeated measures design
C. complete repeated measures design
D. incomplete repeated measures design
21. A researcher compares students' performance using a new
learning strategy to their performance using the old strategy.
Students' performance is first tested with the old strategy,
followed by the new strategy. The results indicate that students
perform better with the new strategy. These results
A. show that the order of the two learning strategies does not
matter.
B. are uninterpretable due to the confounding with practice
effects.
C. indicate that teachers should use the new strategy.
D. all of these
22. (5 points)Distinguish between probability sampling and
nonprobability sampling (explain the difference).
Name two types of probability sampling and one type of
nonprobability sampling.
22. (8 points)A researcher was interested in developing a
reliable and valid measure of friendliness. That is,
she wanted to distinguish between friendly and unfriendly
people using her questionnaire. To determine whether she
developed a good measure (called "Friend"), she administered
her "Friend" questionnaire to a group of 300 people twice

29. (Friend-Time 1 and Friend-Time 2), separated by six months. At
the second testing session, she also administered a similar
questionnaire called "Nice," which previous research has shown
to be a valid measure of people's ability to get along with
people and to make friends. She also administered a
questionnaire designed to measure art ability, which she
predicted to be unrelated to friendliness. She observed the
following correlation coefficients among the measures:
Friend-Time1/Friend Time2 correlation: .90
Friend Time1/Nice correlation: .80
Friend Time2/Nice correlation: .85
Friend Time1/Art Ability correlation: .10
Friend Time2/Art Ability correlation: .15
Which correlation coefficient is an indicator of whether the
"Friend" measure is reliable?
Which correlation(s) test(s) convergent validity?
Which correlation(s) test(s) discriminant validity?
Do these correlations suggest her questionnaire is reliable and
valid? Why or why not?
23. (8 points)A researcher wishes to use stratified random
sampling to select a representative sample of 100
people from a population that has the following characteristics:
Describe the sample in terms of how many single and married
women and men the researcher would need to represent the
population.

30. Suppose the researcher mails the survey to the 100 people in the
representative sample and only 50 of the people return a
completed survey. Can you state that this sample of 50
represents the population? Why or why not?
24. (8 points) A psychologist wishes to have participants rate
two different sets of inkblots (labeled A and B)
for their "emotionality." The inkblots differ in their use of
color. Ten inkblots in set A are created in shades of gray; ten
inkblots in set B use shades of red, blue, and yellow. The
psychologist hypothesizes that color will influence participants'
ratings of emotionality. Each participant will rate all twenty
inkblots.
What is the independent variable in this experiment?
What is the dependent variable in this experiment?
What potential confounding variable must be balanced in this
research?
Explain what balancing procedure would be best for this
experiment.

31. 25. (4 points) Explain one concept from Chapter 5 or 7 that you
know that I did not ask a question about on
this test.