Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Prof. dr. Rolf Fasting

  • Be the first to comment

Prof. dr. Rolf Fasting

  1. 1. Validity and validation in qualitativeand quantitative research:A battle of truth?Rolf B. Fasting ( of Special Education NeedsOslo and Akershus University College of Applied sciences
  2. 2. 2
  3. 3. Validity: epistemological considerations — Positivism - the “objective” truth — Post-positivism - interpretivism — Epistemological relativism: „From a moderate constructivist position, realism assumes that knowledge of both nature and other minds depend on a reiterated sequence of perceptions, cognitions, and inferences, all of which may be questioned, rejected, and revised.’ (Jensen, 2002: 268 f.)3
  4. 4. Validity: origin and emergence Statistical Methods for Research Workers (Ronald Fisher 1925) The Reliability and Validity of Tests: derivation and interpretation of fundamental formulae concerned with reliability and validity of tests and illustrative problems (Louis Leon Thurstone 1931). “Technical standards for evaluating tests and the contents of test manuals” (APA 1950). “Validity will be evaluated in terms of two major criteria. First, and as a basic minimum, is what can be called internal validity: did in fact the experimental stimulus make some significant difference in this specific instance? The second criterion is that of external validity, representativeness, or generalizability: to what populations, settings, and variables can this effect be generalized.” (Campbell, 1957, p.297, emphasis in the original)4
  5. 5. Diverging perspectives… The concept of validity used in quantitative research is rooted in positivism and consequently the concept used in qualitative research needs a different content and a different basis (Altheide & Johnson 1994)5
  6. 6. A legitimation crisis… (Morse, Barret, Mayan, Olson, & Spiers 2002; Denzin & Lincoln 2003) • successor validity • neopragmatic validity • catalytic validity • rhizomatic validity • interrogated validity • communicative validity • transgressive validity • concurrent validity • imperial validity • pragmatic validity • simulacra/ ironic validity • descriptive validity • situated validity • interpretive validity • voluptuous validity • theoretical validity • reflexive validity • evaluative validity • trustworthiness: – credibility – dependability – transferability – confirmability (Guba and Lincoln 1994)6
  7. 7. Validity: a battle of truth, or a need of appropriate strategies to face the threats of trustworthiness?7
  8. 8. Validity and validation… Validity: the approximate truth of an inference; Shadish, Cook & Campbell 2002) [D]ata in themselves cannot be valid or invalid; what is at issue are the inferences drawn from them Hammersley & Atkinson (1983: 191) Validity refers to the inferences drawn, not to the processes in use8
  9. 9. Cook and Campbell’s validity system (Shadish, Cook & Campbell, 2002).  Construct validity  Internal validity  Statistical validity  External validity9
  10. 10. Construct validity…  validity of inferences: - from theoretical framework or interest to indicators (from what we have seen to what we “call” what we have seen) “[T]o what extent are the constructs of theoretical interest successfully operationalized in the research” (Judd, Smith & Kidder 1991: 29) In qualitative research indicators generally are first observed, and constructs are «constructed» through the process of analysis. How well does the concept represent the indicators?10
  11. 11. Validating construct validity Possible threats: —systematic measurement errors —random measurement errors (Crocker & Algina 1986).11
  12. 12. Systematic measurement errors (Messick 1995) Construct underrepresentation and Construct irrelevance12
  13. 13. Validating construct validity Random measurement errors — To what extent would another observer (with the same theoretical position, using the same tools, and at the same time) observe the same things and make the same interpretations? — To what extent will we observe the same things if we observe at another point of time?13
  14. 14. Assessment of Writing Proficiency, grade 5 and 8: 16 schools; 320 scripts; 82 raters. (Oct. 2011) Pair of raters – pair A Pair of raters – pair B Pair of raters – pair x Assessed domains: Assessed domains: Assessed domains: Communication Communication Communication Content Content Content Text structure Text structure Text structure Language Language Language Spelling Spelling Spelling Punctuation Punctuation Punctuation 36 scripts 36 scripts 36 scripts + 4 scripts from pair B + 4 scripts from pair A + 4 scripts from pair A + 4 scripts from pair x + 4 scripts from pair x + 4 scripts from pair B14
  15. 15. Internal validity….  validity of inferences: - from an observed covariation to a causal interpretation (to the interpretation that something is influenced by another thing) —no basis for causal interpretation until alternative causal interpretations are eliminated or at least shown to be unlikely Is there any causal relationship between the included variables?15
  16. 16. Validating internal validity Quantitative designs: — Molar causation: the overall relation between a “treatment package” and its effects -- -- Qualitative designs: — Molecular causation: which parts of a “treatment package” are more or less responsible for which parts of the effects through which mediational processes. — Causal [molecular] descriptions: “[R]eferring to the actual causal mechanisms and processes that are involved in particular events and situations” (Maxwell, 2004: 9).16
  17. 17. Statistical [conclusion] validity…  validity of inferences - about covariation between variables or phenomena (trivial or worthy of a substantial interpretation?) Is a tendency substantial enough to be worthy of an interpretation?17
  18. 18. Validating statistical [conclusion] validity In quantitative studies: — if a tendency should be considered substantial enough to be worthy of an interpretation (p. value) — is there any reasonable evidence from which to infer that the presumed cause and effect covary? — If so; how strongly (effect size - f.x. Choen’s d)? In qualitative studies — Is the children’s change of behaviour in the classroom from time A to time B a real change, or just a trivial fluctuation?18
  19. 19. 19
  20. 20. External validity  validity of inferences - inferences from the context of the study to a wider context or to other contexts - Generalization - Transferability A study has good external validity to the extent it makes possible non-statistical generalizations to or over relevant individuals, situations and time with reasonable certainty (Lund, 2002: 121)20
  21. 21. External validity: Non-statistical generalization Non-statistical generalization: Generalisation over situations; individuals and groups: mainly depending on similarities and differences between situations and persons actually studied, and the situations and persons we draw our inferences about Generalisation over time: mainly a question about nearness in time, and what important changes may have taken place in the meantime (Lund, 2002: 121) -- -- -- “Thorough” or “Thick” descriptions (Geertz 1973; Ryle 1968) …qualitative tools when judgments are made about transferring results to other situations Cronbach 1975)21
  22. 22. External validity Cronbach’s (1975) advice concerning generalizations: 1. Consider the results (knowledge claims) to be context-bound 2. Consider generalizations as work hypotheses rather than conclusions 3. Study the same phenomenon in other contexts to see whether we get the same results there 4. Pay attention to exceptions as well as to cases confirming «the rule», as exceptions may indicate context-specific conditions22
  23. 23. Conclusion: The relevance of the various aspects of validity depends on what kind of inferences are drawn, not on what kind of data used as basis for the inferences (
  24. 24. Further reading: — Altheide, D.L. & Johnson, J.M. (1994). Criteria for assessing interpretive validity in qualitative research. In N.K. Denzin & Y.S. Lincoln (eds.): Handbook of Qualitative Research. Thousand Oaks, CA: SAGE Publications. — Cook, T.D. & Campbell, D.T. (1979). Quasi-Experimentation. Design and Analysis Issues for Field Settings. Boston, MA: Houghton Mifflin Company. — Crocker, L. & Algina, J. (1986). Introduction to Classical and Modern Test Theory. Fort Worth, TX: Harcourt Brace Jovanovich College Publishers. — Denzin, N.K. & Lincoln, Y.S. (2003). Paradigmatic controversies, contradictions, and emerging confluences. In N.K. Denzin & Y.S. Lincoln (eds.): The Landscape of Qualitative Research. Theories and Issues. Thousand Oaks, CA: SAGE Publications. — Fisher, R. A. (1935). The Design of Experiments. Endiburgh, Oliver & Boyd. — Geertz, C. (1973). Thick Description: Toward an Interpretive Theory of Culture. In The Interpretation of Cultures: Selected Essays. Clifford Geertz. pp 3–30. New York: Basic Books. — Guba, E.G. & Lincoln, Y.S. (1994). Competing paradigms in qualitative research. In N.K. Denzin & Y.S. Lincoln (eds.): Handbook of Qualitative24 Research. Thousand Oaks, CA: SAGE Publications.
  25. 25. — Hammersley, M. & Atkinson: (1983). Ethnography: Principles in Practice. London: Tavistock Publications. — Jensen, K.B. (2002). The complementarity of qualitative and quantitative methodologies in media and communication research. In K.B. Jensen (ed.): A Handbook of Media and Communication Research. Qualitative and Quantitative Methodologies. London: Routledge. — Judd, C.M., Smith, E.R. & Kidder, L.H. (1991). Research Methods in Social Relations. Fort Worth, TX: Holt, Rinehart and Winston, Inc. — Kvale, S. (1989). To validate is to question. In S. Kvale (ed.): Issues of Validity in Qualitative Research. Lund: Studentlitteratur. — Lund, T. (2005a). The qualitative-quantitative distinction: some comments. Scandinavian Journal of Educational Research, 49(2), 115–132. — Maxwell, J.A. (2004). Causal explanation, qualitative research and scientific inquiry in education. Educational Researcher, 33(2), 3–11. — Messick, S. (1995). Validity of psychological assessment. American Psychologist, 50(9), 741–749. — Shadish, W.R., Cook, T.D. & Campbell, D.T. (2002). Experimental and Quasi- experimental Designs for Generalized Causal Inference. Boston, MA: Houghton Mifflin Company.25
  26. 26. 26
  27. 27. 27
  28. 28. 28
  29. 29. —History —Maturation —Testing —Instrumentation —Statistical Regression —Selection —Mortality Cook and Campbell 1979)29
  30. 30. —Epistemological relativism: For all X-judgements, some feature, F, of an X- judgement is relative to Y30
  31. 31. Aims —ontological and epistemological fundation of validity —validity theory and correspondence theory —validation validity.31
  32. 32. Aims —Validity: ontological and epistemological consederations —validity theory and correspondence theory —validity and validation.32
  33. 33. Validity - ontologial and epistemological considerations Validity within quantitative methodology: … a post-positivist critical realist perspective. Validity within qualitative methodolog: —constructionist position (Guba and Lincoln 1994) —critical realism (Maxwell 1994; Miles & Huberman 1994) —subtle realism (Hammersley 1992) —pragmatic realism (Alvesson & Sköldberg 1994)33
  34. 34. Ontological and epistemological positions of critical realism (Klaus Bruhn Jensen 2002; 268) Ontological realism: Rejecting skepticist and nominalist positions which variously have held that no knowledge of the empirical world is possible, or that reality is nothing but the sum of our descriptions of it, realism reverses the burden of proof, in a sense. A realist would argue that we must assume the existence of reality as a limit condition or regulatory ideal in order to account for the sorts of natural and cultural phenomena one encounters in science as well as in everyday life.34
  35. 35. The moderate posisjon «Weak relativists believe that both the ontological world and the worlds of ideology, interests, values, hopes, and wishes play a role in the construction of scientific knowledge.» Shadish, Cook and Campbell (2002: 36)35
  36. 36. Cook and Campbell’s validity system (Shadish, Cook & Campbell, 2002). Construct validity: validity of inferences from indicators to constructs (from what we have seen to what we call what we have seen) Statistical validity: validity of inferences about covariation between variables (trivial or worthy of a substantial interpretation?) Internal validity: validity of inferences from an observed covariation to a causal interpretation (to the interpretation that something is influenced by another thing) External validity: validity of inferences from the context of the study to a wider context or to other contexts36
  37. 37. Summing up37
  38. 38. Systematic measurement errors (Messick 1995) Construct underrepresentation and Construct irrelevance38

    Be the first to comment

    Login to see the comments

  • justinmwai1

    Mar. 21, 2018


Total views


On Slideshare


From embeds


Number of embeds