This document provides an overview of key concepts in research design and methodology. It discusses the scientific attitude, theories of proof and scientific progress, research questions, variables, conceptual models, reliability and validity, research designs (fixed vs. flexible), and threats to validity. The document emphasizes developing testable hypotheses from theoretical frameworks and choosing appropriate research strategies and methods to answer specific research questions.

1. General
Research Design
Issues
PYC 5040: Advanced Research
Grant M. Heller, Ph.D.

2. The Scientific Attitude (Robson, 2002)
•Systematic
•Skeptical
•Ethical

3. Proof, Disproof & Scientific
Progress (Leary, 2004)
• The logical impossibility of proof
o Theories cannot be proved b/c obtaining empirical support
for a hypothesis does not necessarily mean that the theory
from which the hypothesis was derived is true.
• The practical impossibility of disproof
o Unlike proof, disproof is a logically valid operation
o Absence of evidence is not necessarily evidence of
absence.
• If not proof or disproof, then what?
o The scientific filter

4. Scientific filter
adapted by Leary (2004) from Bauer (1992)
1. All ideas: Scientific training, concern for professional
reputation, availability of resources (filters out
nonsense)
2. Initial Research Projects: Self-judgment of viability
(filters out dead ends & fringe topics)
3. Research programs: Peer review (filters methodological
biases & errors, unimportant contributions).
4. Published research: Use, replicability & extension by
others (filters out nonreplication, uninteresting &
nonuseful stuff)
5. Secondary Research Literature – Established
Knowledge

5. "Real World" vs.
"Academic" Research
Real World Emphasis
1. Solving problems
2. Robust results
3. Finding basis for
action
4. Often "in the field"
(e.g., hospital, busine
ss, school)
5. Constraints of
funding & time
Academic Emphasis
1. Advancing the
discipline (aka "basic
research")
2. Est. relationships
3. Developing theory
4. Often "in the lab"
5. Potentially less
funding & time
constraints?
(Robson, 2002)

6. "Real World" vs.
"Academic" Research (cont.)
Real World Emphasis
1. High consistency of
topic from 1 study to
another
2. Generalist researcher
3. Oriented to client
needs
4. Viewed as dubious by
many academics
5. Need highly developed
social skills
Academic Emphasis
1. Little consistency of
topic from 1 study to
another
2. Specialist researcher
3. Oriented to academic
peers
4. High academic
prestige
5. Need some social
skills
(Robson, 2002)

7. Main steps when carrying out a
research project (Robson, 2002)
1) Start a research journal
2) Determine focus of project
3) Develop research questions
4) Choose research design
5) Select method(s)
6) Arrange practicalities
7) Collect data
8) Prep data for analysis
9) Analyze & Interpret data
10) Report & disseminate findings

8. General Design Issues
• Focus on turning research questions into projects.
• Strategies & tactics based on questions you want
to answer (Manstead & Semin, 1988).
o Research focus, questions, strategy & tactics
o River crossing analogy
• Hakim (1987):
o think like an architect

9. A framework for research
design
•Components to consider: (Robson, 2002)
1. Purpose(s)
2. Theory
3. Research questions
4. Methods
5. Sampling strategy

10. A framework for research
design
Research
Questions
Purpose(s)
Conceptual
framework
Sampling
strategy
Methods

11. Research Questions
• Specific
questions, framed in
testable ways, that
provide impetus for
studies.
• Direct comparisons
are used;
conceptual models
are implied by these
comparisons

12. Types of Research
Questions
1. Descriptive (what, when)
2. Explorative (correlations, descriptive)
3. Evaluative (applied, outcomes research)
4. Predictive (underlying causal model, correlations)
5. Explanatory (explicit causal model, experimental
regressions)
6. Control (change cause, observe effect)
To formulate research questions, focus on variables
(not measures), and focus on predictive or
explanatory or control questions.

13. Theory
• Selection of research problem & design
should be based on theory
• THEORY: a conceptualization & explanation
of the phenomenon of interest (usually a
causal model)
o A theoretical framework guides the interpretation of
research results and guides the generation of further
studies
o Explains a set of relationships among concepts

14. Importance of Theory
(Kazdin, 2003)
• Can bring order to areas
where findings are diffuse or
multiple
• Can explain the basis of
change & unite diverse
outcomes
• Can direct our attention to
which moderators to study
• Application and extension of
knowledge to the world
beyond the laboratory

15. Theory building
Black, 1999

16. Cyclic life & evolution of a
theory
Black, 1999

17. Operational Definitions
(Kazdin, 2003)
• Defining a concept on the basis of the
specific operations used in the study.
• Allows us to measure & quantify
• Concept: anxiety
• Operational definition: participants scoring at
or above 75th percentile on measure of
anxiety (e.g., GAD-7)

18. Operational Definitions:
Limitations (Kazdin, 2003)
• May oversimplify the concept of interest or
limit scope/focus
• May include features that are irrelevant or not
central to the original concept
o Unnecessary error or noise
• Use of single measures to define a concept
o May impede drawing general relationships among
concepts

19. Hypotheses (Leary, 2004)
• A Hypothesis is a specific proposition that
logically follows from the theory.
• Deriving hypotheses from theory involves
deduction.
• Logical implications of a theory
• If the theory is true, what would we expect to
observe?
• Virtually all hypotheses can be reduced to an
if-then statement.

20. A well written hypothesis
• Stated in declarative form
• Posit a relationship between variables
• Reflect a theory or body of literature upon
which they are based
• Be brief & to the point
• Be testable
• Null vs. Alternative (research) hypotheses

21. Choosing a research strategy
(Robson, 2011, pgs. 74 – 77)
A. FIXED, FLEXIBLE, or MULTI-STRATEGY?
B. Is your proposed study an EVALUATION?
A. Focus on:
A. Outcome: Fixed
B. Process: Flexible
C. Both: Multi-strategy
C. Do you wish to carry out ACTION RESEARCH?
A. Focus of improvement of practice, increased
understanding of practice & improvement of situation
A. If so, Flexible approach almost always used
D. E. F. What design strategy is most appropriate?

22. Choosing a research strategy
(Robson, 2011, pgs. 74 – 77) cont.
G. The purposes(s) helps
in selecting a strategy
H. The research
questions have a
strong influence on
the strategy
I. Specific methods
need not be tied to
particular research
strategies

23. Research Nuts & Bolts:
Variables
• Anything that varies
• A concept that can be measured
• Represents a class of outcomes that can take
on more than one value
• Types
o Independent
o Dependent

24. Difference between
concepts & variables
Concepts
• Subjective impression
• No uniformity as to its
understanding among
different people
o as such, cannot be
measured
Examples: effectiveness,
satisfaction, impact,
excellence, achievement,
domestic violence, self
esteem, etc.
Variables
• Measurable though the
degree of precision varies
from scale to scale &
variable to variable
Examples: Gender
(male/female), age, income,
weight, height, religion
attitude (subjective), etc.

25. Levels of measurement
• Nominal
o categorical (names or categories)
• Ordinal
• Interval
o equal distance between points
• Ratio
o equal distance + absolute zero point
 e.g., degrees Kelvin, distance, mass,
time

26. Research Design: Key
Concepts
• Independent Variable (IV)
• Dependent Variable (DV)
• What effect does ___ (IV) have on ___ (DV) ?
• Experimental groups vs. Control groups
Independent
Variable (IV)
Dependent
Variable (DV)

27. Mediating Variables
Ice Cream
Sales
(IV)
Violent
Crime
(DV)
Temperature
(mediator)

28. Moderating Variables
Therapy
(IV)
Tx vs. Wait
List
Symptom
Reduction
(DV)
Intelligence
(moderator)

29. Conceptual Models
• Graphical representation of a theory
o diagram of proposed relationships among theoretical
concepts
Concept (abstract, theoretical)
Variable (more detailed, specifically defined
meaning)
Measure (operational definition)
• Conceptual models are useful for clarifying a
research question, aims & hypotheses
• Distinct from research design (later step in planning)

30. Mediating & Moderating
Effects
• Moderating variable(s)
o age
• Mediating variable(s)
o Motivation
o Coping Skills (Wykes & Spaulding, 2011)

31. Law of Parsimony /
Occam's Razor
• Are we looking for zebras?
• Can we explain the data
with concepts & models we
already know?
• Adopting the simpler of 2
solutions that account
equally well for the data.

32. Plausible Rival Hypotheses
• Other plausible explanations for our
results?
• Example: imipramine (antidepressant)
vs. St. John’s Wort (herbal) in tx of
moderate depression.
o Woelk, 2000 found equal improvement
after tx
• Equal effect?
• What would you want to know?
• What do you think is really going on?
o Placebo effect

33. Correlation & Causation
Correlation does not mean causation...

34. Correlation & Causation
...but is a necessary component of a causal
model.
• Inferring causality (Leary, 2004):
1. Covariation (correlation)
2. Temporal precedence (A then B)
3. All extraneous factors that might influence
the relationship between the variables of
interest are controlled or eliminated.

35. Basic Statistics Review
• Descriptive Statistics
o Measures of central tendency
 e.g., mean, median, mode
o Measures of variability
 e.g., range, standard deviation, standard
 error to the mean (SEM)
• Inferential statistics
o Goal: to make inferences about population from a sample
 Parametric
• e.g., t-tests, ANOVA, regression
 Nonparamentric
• e.g., chi square

36. Population vs. Sample

37. The Normal Distribution

38. Null Hypothesis Significance
Testing (NHST)
•If p < .05, then…
o Reject the null hypothesis in favor
of the alternative hypothesis.
•If p > .05, then…
o Fail to reject the null hypothesis
 or retain the null

39. What does NHST tell us??
• Assuming that the null hypothesis is true, p
would the odds of having attained results this or
more extreme by chance alone (assuming we have
a normal distribution).
• Tells us if our results are statistically significant
(different that practically or clinically significant).
o E.g., difference between two groups reaches a level of
statistical significance.
• Outcome is either or, does not inform us of strength
of effect.
o p = .001 is not more significant that p = .01.

40. NHST: Outcomes

41. Types of error

42. Threats to Statistical
Conclusion Validity (Kazdin, 2003)
• Low statistical power
• Variability in the procedures
• Subject heterogeneity
• Unreliability of the measures
• Multiple comparisons & error rates

43. Null Hypothesis Significance Testing
(NHST)
Cohen, "p < .05, the world is flat"

44. alpha inflation &
Type I Error (false +)

45. Statistical Power
• Probability of accepting a hypothesis (the null
hypothesis) when in actuality it is false
(aka, Type 2 error, false negative
• Sensitivity to effects of IV
• Probability is 1 – Beta
• Conventionally set at 0.80 (Cohen, 1988)
• Powerful designs are able to detect effects of
the IV more easily than less powerful
designs.

46. Methods to Increase Power
(Shadish, Cook & Campbell, 2002)
• Use matching, stratifying, blocking
• Measure & correct for covariates
• Use larger sample sizes
• Use equal cell sample sizes
• Improve measurement
• Increase the strength of treatment
• Increase the variability of treatment
• Use within-participants design
• Use homogenous participants selected to be responsive to tx
• Reduce random setting irrelevancies
• Ensure powerful statistical tests are used & assumptions met

47. Effect Size (E.S.)
• Way of expressing the difference between
conditions (e.g., treatment vs. control).
• A common metric that can be used between
studies
• Magnitude of effect
o Often classified into small, medium, large

48. 4 Categories of E.S.
(Ferguson, 2009)
1. Group differences indices
a. magnitude of difference between 2 or more variables
b. e.g., Cohen's d
2. Strength of association indices
a. magnitude of shared variance between variables
b. e.g., Pearson's r
3. Corrected estimates
a. estimates correcting for sampling error
b. e.g., Adjusted R2
4. Risk estimates
a. more commonly used in medical outcome research
b. e.g., relative risk (RR) & odds ratio (OR).

49. Suggested E.S.
Interpretation (Ferguson, 2009)
Type of E.S.
Est.
Included
Indices
RMPE "Moderate"
effect
"Strong"
effect
Group
Difference
Cohen's d,
Glass' delta,
Hedges g
0.41 1.15 2.70
Strength of
Association
r, R, partial
r, rh
tau
0.2 0.5 0.8
Squared
Association
Indices
r2, R2, eta
squared,
adj. R2
0.04 0.25 0.64
Risk Estimates RR, OR 2.0 (interpret
w/ caution)
3.0 4.0

50. Reliability
A reliable measure is consistent or repeatable.
more of different types of reliability later...

51. but is reliability enough?
phrenelogy was reliable...
but was it valid?

52. Validity
• Internal Validity
o extent to which the changes in the study DV can be
attributed to changes in the IV
• External Validity
o extent to which the results can be generalized

53. Validity (cont)
• Example
o Progress in therapy
• Did therapy cause the improvement?
Measure
at
Intake
Measure
at
Termination
Therapy

54. Threats to Internal Validity
• History
• Maturation
• Testing
• Instrumentation
• Statistical
regression
• Differential
selection
• Experimental
mortality (attrition)
• Selection X
maturation interaction
• Statistical conclusion
validity (lack of
power)
• Subject heterogeneity

55. Threats to Internal Validity
• Remember the acronym: MRS SMITH
o Maturation
o Regression to the mean
o Selection of subjects
o Selection by maturation interaction
o Mortality
o Instrumentation
o Testing
o History

56. Threats to External Validity
• Reactive effect of testing
• Reactive effect of experimental
arrangements
• Interaction between selection bias and the IV
• Multiple treatment interference

57. Defense against threats to
validity
• for External Validity
o Random selection of subjects
• for Internal Validity
o Random assignment to conditions
• Various research designs have
stronger internal or external validity
o often a balancing act: External vs. Internal
o Mook (1983) "In defense of External
Invalidity"

58. Regression to the Mean
(RTM)
How to Reduce RTM
1. Random allocation to comparison groups
2. Selection of Ss based on multiple
measurements
3. Estimate size of RTM
a. can be subtracted from observed change to give an
adjusted estimate
4. Statistical control of covariates (i.e., ANCOVA)
(Barnette, van der Pols & Dobson, 2005).

59. Other forms of Validity
• Face Validity
• Content Validity
• Construct Validity
o Convergent
o Discriminant
• Criterion Validity
o Predictive Validity

60. Range Restrictions
• Beware of range
restrictions in data
• You can miss the big
picture
• Beware of floor and
ceiling effects as well

61. Fixed or Flexible design?
• Some projects using social research
methods are pre-planned in detail: they have
FIXED designs (commonly referred to as
quantitative research).
• Others expect the plan to change or evolve
while the project is underway: their design is
FLEXIBLE (commonly referred to as
qualitative research).

62. Fixed Designs
• Pre-specify exactly what you plan to happen
BEFORE (a priori) the main data collection.
• Examples are experiments and surveys.
• They typically rely almost exclusively on
quantitative data collection (and are often
referred to as quantitative research).
• more to come...

63. Flexible Designs
• Initial planning is limited to the focus of the research and
(possibly) to setting out some general research
questions.
• Details of the design change depending on the initial
findings.
• Examples are grounded theory and ethnographic
studies.
• They typically rely largely on the collection of qualitative
data (and are often referred to as qualitative research)
though some quantitative data is often also collected.
• More to come...