Threats to internal and external validity

55,202 views

Published on

from educational research

Published in: Education, Technology
3 Comments
23 Likes
Statistics
Notes
No Downloads
Views
Total views
55,202
On SlideShare
0
From Embeds
0
Number of Embeds
226
Actions
Shares
0
Downloads
1,018
Comments
3
Likes
23
Embeds 0
No embeds

No notes for slide

Threats to internal and external validity

  1. 1. ByBieger and Gerlach(A RodSaz Report)
  2. 2.  The results of a research study are only useful to the extent that they can be accurately and confidently interpreted. The issue of accurate and confident interpretation of results is at the center of any discussion of validity.
  3. 3.  From validus, meaning “strong” Refers to the degree with which correct inferences can be made from the results of a research study.
  4. 4. •INTERNAL VALIDITY•EXTERNAL VALIDITY
  5. 5.  Refers to the extent to which the results obtained in a research study are a function of the variables that were systematically manipulated, measured, and /or observed in the study.
  6. 6.  For example, a researcher in interested in determining which of two instructional methods is superior for teaching a history concept. Suppose further that the researcher asked two teachers to each use one of the methods of instruction and then compared the mean test scores of each class following the instructions.
  7. 7.  Teachers’ teaching effectiveness and enthusiasm Class interest or preparation One class may have interruptions (fire drills)
  8. 8.  Refers to the occurrence of events that could alter the outcome or the results of a study.
  9. 9.  Refers to the occurrence of events that could alter the outcome or the results of a study.  Previous history – occurs before the study  A study of the effectiveness of a new method for teaching a unit on the biology of a cell, many watched a television documentary about “The Cell”
  10. 10.  Refers to the occurrence of events that could alter the outcome or the results of a study.  Concurrent history – occurs during the study  Eg. Studying the effectiveness of using musical activities to teach mathematics concepts. While one teacher uses the standard curriculum, another teacher is using the musical activities curriculum
  11. 11.  Pertains to any changes that occur in the subjects during the course of the study that are not part of the study and that might affect the results of the study.
  12. 12.  Pertains to any changes that occur in the subjects during the course of the study that are not part of the study and that might affect the results of the study.  Biological ( growth processes)  Eg. Weight gain or increase in height due to breakfast or lunch program
  13. 13.  Pertains to any changes that occur in the subjects during the course of the study that are not part of the study and that might affect the results of the study.  Psychological (learning or development)  Eg. Effects of certain instructional techniques on concept learning of sixth graders, attainment of certain operational thought during that period has to be considered
  14. 14.  Relates to the possible effects of a pretest on the performance of participants in a study on the posttest.  May alert subjects to the fact that they are being studied  May affect performance on later administrations
  15. 15.  Is concerned with the effects on the outcome of a study of the inconsistent use of a measurement instrument (what the instrument is measuring changes during the duration of the study). Eg. The effects of fatigue on an achievement test
  16. 16.  Refers to the tendency of extreme scores to move (or regress) toward the mean score on subsequent retesting.  Eg. Students scoring below 25% (lowest extreme) on an IQ test are given a posttest. A higher posttest score is expected.
  17. 17.  Refers to the loss of subjects from a study due to their initial nonavailability or subsequent withdrawal from the study.  Eg. More high-scoring people drop out from the experimental group than from the control group
  18. 18.  Pertains to the possibility that groups in a study may possess different characteristics and that those differences may affect the results.  Differences in age, ability, gender or racial/ethnic composition, or any of an almost unlimited number of ways.
  19. 19. 1. Standardization of the conditions under which the research study is carried out will help minimize threats to internal validity from history and instrumentation.
  20. 20. 2. Obtaining as much information as possible about the participants in the research study aids in minimizing threats to internal validity from mortality and selection.
  21. 21. 3. Obtaining as much information as possible about the procedural details of the research study, for example, where and when the study occurs, minimizes threats to internal validity from history and instrumentation.
  22. 22. 4. Choosing an appropriate research design can help control most other threats to internal validity.
  23. 23.  The use of a control group, selected from the same population as the experimental group(s) and which experiences the same concurrent history as the experimental group(s), can help eliminate most of the effects of history. Also, the shorter the duration of an experiment, the less likely history will be a threat.
  24. 24.  The effects of maturation, like the effects of history, can be minimized by the use of a control group, selected from the same population as the experimental group(s). Also, like the effects of history, the effects of maturation tend to be minimized in studies of short duration.
  25. 25.  The use of a research design that does not include a pretest can eliminate testing as a potential threat to internal validity. If baseline or pretreatment or data are needed, the use of unobtrusive measures (data collection techniques about which the experimental participant is unaware) may minimize the effects of testing. It also may help for a researcher to use different equivalent forms of a test for pretesting and posttesting.
  26. 26.  Careful specification and control of the measurement procedures can eliminate most instrumentation threats. Standardized instruments, administration or data collection procedures, and the training or observers are among the procedures that help control the instrumentation threat.
  27. 27.  Avoiding the use of extreme scorers, when average scorers are excluded, will minimize the threat.
  28. 28.  Choosing large groups and ensuring that they are representative of the population from which they were selected can minimize mortality threats. The use of follow-up procedures with a portion of those who leave the study or who were initially unavailable can further minimize mortality as a threat
  29. 29.  Random selection and random assignment of subjects minimize selection as a threat to internal validity. If random selection and assignment are not possible, the use of certain statistical techniques, used as part of a careful quasi-experimental design, can adjust for group differences and thereby minimize selection as a threat.
  30. 30.  Refers to the extent to which the results of a research study are able to be generalized confidently to a group larger than the group that participated in the study. (Bracht & Glass)
  31. 31.  Refers to the extent to which the results of a research study are able to be generalized confidently to a group larger than the group that participated in the study. (Bracht & Glass) The researcher must have reason to believe that the variables used in the study are similar to those aspects as they exist in the larger population.
  32. 32.  The extent to which a sample is representative (or not representative) of the population from which it was selected.
  33. 33.  The extent to which characteristics of the setting or context of the research study are representative (or not representative ) of the setting and context to which the results are to be generalized.
  34. 34.  Refers to the fact that the administration of a test (for example, a pretest) may affect the responses or the performance of the participants in a research study. Its results may not be generalizable to situations where pretesting will not occur.
  35. 35.  It pertains to the situation in which participants in a study receive more than one treatment. In such a case, the effects of the multiple treatment may interact.
  36. 36.  It is concerned with the possibility that some characteristic of the participants selected for the study interacts with some aspect of the treatment.  May include prior experiences, learning, personality factors, or any traits that might interact with the effect of the treatment
  37. 37.  It pertains to situations where participants become aware that they are involved in a study, and , as a result of that awareness, their response or performance is different from what it would have been otherwise.
  38. 38.  It pertains to situations where participants become aware that they are involved in a study, and , as a result of that awareness, their response or performance is different from what it would have been otherwise.  “Novelty effect” (newness) or belief on the part of the participants that they are receiving some “special” treatment (sometimes known as Hawthorne Effect.)  “Placebo effect” or to the participants’ belief in the effectivenesss of the treatment
  39. 39.  They refer to the possibility that an experimenter may sometimes unintentionally influence the performance of participants in a study.
  40. 40.  They refer to the possibility that an experimenter may sometimes unintentionally influence the performance of participants in a study.
  41. 41.  They refer to the possibility that an experimenter may sometimes unintentionally influence the performance of participants in a study.  PASSIVE  Gender, race, or personal attributes of the researcher
  42. 42.  They refer to the possibility that an experimenter may sometimes unintentionally influence the performance of participants in a study.  PASSIVE  Gender, race, or personal attributes of the researcher  ACTIVE  The expectations of the researcher or observer are communicated to the participant in a manner that affects performance
  43. 43.  It is concerned with the extent to which the variables in a study are adequately described and operationally defined. Variables can be defined too specifically.
  44. 44.  Use research designs that do not include pretests. Solomon four-group design is useful in determining the extent to which pretesting may have influenced the results of the study.
  45. 45.  The researcher should try to choose a design in which only one treatment is assigned to each subject. If such a design is practical, the researcher should try to control and/or measure the effects of all relevant treatments and incorporate them into a multiple-treatment design.
  46. 46.  Use random selection and assignment of participants. If the above is not practical, use statistical techniques such as analysis of covariance, used in conjunction with a quasi-experimental design, can take into account differences due to measurable attributes of the individual.
  47. 47.  Have a control group and a placebo group Both should be treated the same in all respects, except in regard to the treatment itself All of the participants should believe they are being treated comparably
  48. 48.  Use “blind” data collection procedures. This means that the researcher does not collect data or make observations but instead trains a “naïve” observer to do so. The person collecting the data or making the observations should be unaware of the purpose of the study and should be unaware of which participants are receiving the experimental treatment.
  49. 49.  Must operationally define variables carefully in a way that is meaningful in settings beyond that in which the study is being conducted. Use of widely agreed upon definitions or multiple competing definitions should be considered.

×