Research Validity & Threats to Validity
Upcoming SlideShare
Loading in...5

Research Validity & Threats to Validity



Lecture from a research methods class.

Lecture from a research methods class.



Total Views
Views on SlideShare
Embed Views



6 Embeds 62 22 17 13 8 1 1



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-NonCommercial LicenseCC Attribution-NonCommercial License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • It is important to consider both external and content validity. External validity is the extent to which the results of a study can be generalized from a sample to a population. An instrument that is externally valid helps obtain population generalizability, or the degree to which a sample represents the population. Content validity refers to the appropriateness of the content of an instrument. In other words, do the measures (questions, observation logs, etc.) accurately assess what you want to know
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Research Validity & Threats to Validity Research Validity & Threats to Validity Presentation Transcript

  • Graziano and Raulin
    Research Methods: Chapter 8
    Research Methods:Validity and Threats to Validity
  • Generating Research Hypotheses(Review)
    Ideas lead to
    library research
    Statement of problem
    Problem statements become research hypotheses when constructs are operationalized
  • Characteristics of a Good Study Question
    F Feasible
    I Interesting
    N Novel
    E Ethical
    R Relevant
  • Actually testing three sets of hypotheses
    The null hypothesis
    The confounding variable hypotheses
    The causal hypothesis
    Accept causal hypothesis only if you
    reject null hypothesis (statistical analysis)
    rule out each potential confounding variable hypothesis (based on appropriate controls)
    Testing Research Hypotheses
  • Criteria for Nomothetic Causality
    Correlation (also called covariation)
    Relationship found between variables
    Time order
    Cause must occur before result
    Alternative explanations must be eliminated from possibility
    Experiments are intended to reduce or rule out alternative explanations and confoundingvariables
  • Asking “the Question”
    The PICO format:
    P Population
    I Intervention or Interest area
    C Comparison intervention or status
    O Outcome
  • Applying the “PICO” format
    “What is the usefulness or accuracy of the current 1-10 pain scale assessment in treating a patient’s pain, and what are other options that may prove more useful?”
    Does a 10 point pain Visual Analog Scale (____, ____) accurately assess pain in the first day postop abdominal total hysterectomy patient when compared with the Faces Pain Scale (Pasaro, 1997)?
  • Evaluating Hypotheses
  • Statistical Validity – carrying out the actual statistical analysis properly
    Construct Validityrefer most often to a characteristic of an instrument but also to the whole study
    External Validity refers to the generalizability of study findings
    Internal Validity refers to a characteristic of a study’s design
    Types of Validity
  • Statistical Validity
    Are the statistical tests accurate?
    Threatened by
    Unreliable measures
    Violations of statistical assumptions
    How do we detect these problems???
    Strengthened by
    Using well validated measures
    Having approximately equal sample sizes in each group)
  • Is our theory the best explanation for the results?
    Threatened by
    Any alternative explanation for the results
    HOW do we locate these alternative explanations?
    Strengthened by
    Using well-validated constructs to build the theoretical predictions for the study
    Construct Validity
  • Do the results apply to the broader population?
    Threatened by
    Unrepresentative samples
    Generalizing beyond the limits of the sample
    HOW do we know when this problem is present???
    Strengthened by
    Gathering a representative sample (if possible)
    Clearly describing sample, so that other researchers will know the limits of generalization
    External Validity
  • Is the independent variable responsible for theobserved changes in the dependent variable?
    Threatened by
    Confounding variables
    HOW do we detect the presence of confounding variables????
    Strengthened by
    Adding adequate controls to reduce or eliminate confounding
    Internal Validity
  • Confounding and internal validity
    Many sources for confounding (covered next)
    With proper controls, confounding can be virtually eliminated (see Chapter 9)
    Confounding and construct validity
    Make sure that you have considered alternative theoretical explanations for the anticipated phenomenon
    Avoiding Confounding
  • Pretest-Posttest Research Design
    Single-group, pretest-posttest design compares pre-treatment and post-treatment scores to determine improvement
    Fails to control most sources of confounding
  • Sources of Internal Invalidity
    Historicalevents may occur during the course of the experiment.
    Remember Pygmalion effect & its story
    Maturationof the subjects.
    Testingand retesting can influence awareness of variables or behavior
    Learn Hawthorne effect & its story.
    Instrumentation– measurement methods or procedures may not be equivalent
  • Sources of Internal Invalidity
    Statistical regression of subjects starting out in extreme positions.
    Selection biases (we will see several types)
    Experimental mortality(a.k.a. attrition) – subjects drop out of the study before it's finished.
    Sequence effects – Performance on one measure is related to previous experience with other measures. Outcome depends on the sequence of measures.
  • Social threats to validity
    Demoralization subjects incontrol group find out, loseinterest in study, stop trying
    Diffusionof treatment (those who get the experimental stimulus spread it to controls)
    Rivalry(controls change behavior to try to beat the experimental group)
    Equalizationof treatment (researcher compensates controls for not getting treatment)
  • Social Psychology Experiment:Conformity to Norms(file)
  • Participants are not passive
    They try to understand the study to help them to know what they “should do” (termed subject effects)
    Respond to subtle cues about what is expected (termed demand characteristics)
    Placebo effect: treatment effect due to expectations that the treatment will work
    Subject Effects
  • Based on the expectations of the researcher
    Can affect the outcome of studies if not controlled
    May be due to the experimenter providing demand characteristics to the participant
    Not the same as scientific fraud (which is deliberate)
    Experimenter Effect
  • Validity, Control, and Constraint
    Three closely-tied concepts
    The accuracy of the study or procedure
    Increased by using appropriate control procedures
    The more controls we employ, the higher the level of constraint
    Controls may increase some types of validity while, by their unnatural aspect, decreasing other types of validity.
  • Risk is balanced by reward
    A poorly designed study will provide no useful information; therefore, any risk would be unacceptable
    Informed Consent
    Virtually guarantees that you will have confounding due to selection because some people will refuse to participate
    A small price to pay to maintain ethical standards
    Ethical Principles
  • Start by building a research hypothesis
    Testing the research hypothesis is actually testing three hypotheses
    (1) null; (2) confounding-variable; (3) causal
    Several types of validity
    Many potential confounding variables
    Subject and experimenter effects can also affect the outcome of the study