This document summarizes key concepts from chapters 4 and 5 of a research methods textbook. It discusses reliability and validity, describing reliability as the consistency of measurements and validity as how accurately a study measures its intended concept. It also outlines different types of reliability like test-retest and internal consistency, as well as types of validity like content, criterion, and construct validity. Finally, it reviews observational research methods like naturalistic observation, participant observation, and field experiments.

1. PSBE2-08
Research Methods
Week 4
Tassos Sarampalis 1

2. The week’s reading
Chapter 4 (pp. 87-122)
Chapter 5 (pp. 123-159)
2

3. RELIABILITY AND VALIDITY
3

4. Reliability
The consistency or repeatability of the
measurements
Validity
The degree of accuracy to which a study reflects
or assesses the specific concept that the
researcher is attempting to measure
4

5. All measurements are aggregates of:
The ‘true’ value
+
Random (chance) error
5

6. Stability and Equivalence
• Test-retest reliability
• Alternate-form reliability
Version Version Reliability
• Internal-consistency reliability
1 1
Equivalence
Version Version
2 2
time 6

7. Internal Consistency
How related are individual items within the
measure?
• Spearman-Brown
• Kuder and Richardson Formula 20 (KR20)
• Cronbach’s alpha
7

8. Replication in Research
• What do we mean by ‘replication’?
• Three factors to consider:
– When
– How is the replication is conducted?
– By whom
8

9. Replication in Research
• What do we mean by ‘replication’?
• Three factors to consider:
– When
– How is the replication is conducted?
– By whom
Early replications are more valuable than later
ones 9

10. Replication in Research
• What do we mean by ‘replication’?
• Three factors to consider:
– When
– How is the replication is conducted?
– By whom
The difference between precise and varied
replication 10

11. Replication in Research
• What do we mean by ‘replication’?
• Three factors to consider:
– When
– How is the replication is conducted?
– By whom
Independence of replicators
11

12. Validity
• Content-related
– How well a measure represents all dimensions
• Criterion-related
– Concurrent vs. predictive validity
• Construct-related
– Convergent vs. discriminant validity
12

13. OBSERVATION
13

14. Observational Methods
• Categories based on intervention by
researcher
– Observation without Intervention
– Observation with Intervention
• Categories based on methods for recording
behavior
– comprehensive record
– selected behaviors

15. Observation without Intervention
• Naturalistic Observation
– observation in natural (real-world) setting without
attempt to intervene or change situation
– use when ethical considerations prevent
experimental manipulation
• Goals
– describe “normal” behavior, examine relationships
among naturally occurring variables
– establish external validity of lab findings

16. Observation with Intervention
• Most psychological research involves
intervention
• Three methods in natural settings
– participant observation
– structured observation
– field experiment

17. Observation with Intervention
• Participant observation
– observer is active participant in the natural setting
he or she observes
• undisguised: people know they’re being observed
(e.g., Kitchen Stories, 2003 Norwegian film)
• disguised: people don’t know they’re being observed

18. Observation with Intervention
• Problems with participant observation
– Reactivity
• when people change their usual behavior because
they’re being observed
• disguised participant observation controls reactivity
– Observers lose objectivity or become too involved
in situation
– Observers influence behavior of people they’re
observing (e.g., Hawthorne Effect)

19. Observation with Intervention
• Structured observation
– set up (structure) specific situation in order to
observe behavior
– used when behavior is difficult to observe as it
naturally occurs
– researchers use confederates to structure
situations
– problems: when observers don’t follow same
procedures across observations

20. Observation with Intervention
• Example of structured observation
– Simons and Levin (1998): “change blindness”
http://www.youtube.com/watch?v=FWSxSQsspiQ

21. Observation with Intervention
• Field Experiment
– manipulate independent variable in natural
setting and observe behavior (dependent variable)
• two or more conditions to compare (IV)
• often use confederates to create conditions
• strive for control in natural setting

22. On Being Sane in Insane Places
Original Article:
http://cooley.libarts.wsu.edu/soc3611/Documents/Being
_Sane_in_Insane_Places.pdf
Video:
http://www.youtube.com/watch?v=Kq-7uvVOoyk
In Fiction:
Simpson’s Episode S03E01 – “Stark Raving Dad“
Dr. David Rosenhan
22

23. Maximising Serendipity
• With
– Common sense
– Hard work
– The right info
– And a bit of luck
Raising the flag on Iwo Jima - 1945
23

24. Errors in Observational Research
• Non-interactional artifacts
– Interpreter biases
– Observer biases
• Interactional artifacts
24

25. Non-reactive Measures
• Archival research
• Physical Traces
• Unobtrusive Observation
– Simple
– Contrived
25

26. Selecting Judges
• Identify the type of judge that is most likely to
produce accurate judgments
– Search the literature
– Test them
26

27. Rating Biases
• Errors of Leniency
– e.g., Halo Effect
• Severity Errors
• Errors of Central Tendency
• Logical Errors
27

28. Thank you
28