“Measurement Validity: A Shared Standard for Qualitative and Quantitative Research.” by Robert Adcock and David Collier

1. Review of Measurement Validity: A Shared
Standard for Qualitative and Quantitative
Research
Student Led Article Critique
by Michael-Paul “Jack” James

2. Four reasons to study measurement validity
1. To create unified standards in quantitative and qualitative research.
2. To aid disputes resolution between measurement validity and conceptual
meaning.
3. To address contextual validation and invalidation through sensitivity to disparate
situations.
4. To clarify vernacular and establish language norms for measurement validation
discussion.

3. “Valid measurement is achieved when scores
meaningfully captures the ideas contained in
the corresponding concept”
When analyzing measurement validity, it is important to ignore issues
over the background concept.

4. Four Relational Levels Between Concepts and Observations
● Background Concept
○ The general assumptions of a diversity meaning.
● Systemized Concept
○ The explicit articulation of the theory by the researchers
● Indicators
○ The measures used to test the theory.
● Scores
○ The results of measuring observation.

5. Iterative Diagram of Relational Levels of Concept and Measurement
Level 1:
Background
Concept
Level 2:
Systematized
Concept
Level 3:
Indicators
Level 4:
Scores for
Cases
Task:
Conceptualization
Task:
Operationalization
Task:
Scoring Cases
Task:
Refining Indicators
Task: Modifying
Systemized Concept
Task: Revisiting
Background Concept
FEEDBACK LOOP
OBSERVATION
MEASUREMENT

6. Validity and Reliability
Systematic measurement error, or bias, is generally referred to as validity
Non-systematic, or random error, is generally referred to as reliability.
Some researchers argue that validity must be generally absent of both for measurement
validity

7. "Proper concepts are needed to formulate a good theory, but we need a good theory to
arrive at the proper concepts"
Three Common Traps
1. The concepts accepted meanings binds inherent flexibility of exploration.
2. The more encompassing the claim the more difficult the defense.
3. Always completely flesh-out systemized concepts, avoiding easy dismissal.
Kaplan’s Paradox

8. The concern that contextual variations may invalidate measurement.
● Areas of concern: time periods, response styles, idiosyncrasies between subgroups
● Assess methods for creating equivalence
○ Carefully consider the domain of application.
○ Context-specific indicators
■ Find similiar systems with parallel functions, internally comparisons
■ Score each method differently to achieve close equivalence.
○ Adjusted common indicators
■ Utilize scalability, weighted measures, and standardizing methods
● Create contextually sensitive measures.
Contextual Specificity

9. Trinity v. Monotheistic Modes of Validity
● Content Validity
○ The degree a indicator completely contains and appraises the systemized concept
○ Encompassing abstract phenomena in social science is arduously evaluated and impossibly defined
● Criterion Validity: Convergent/Discriminant
○ The degree a indicator determines the outcome
○ Elusive variables seldom found for evaluative measurement. Use when available.
● Construct Validity: Nomological
○ The degree a indicator achieves the researchers stated goals; appropriateness of the measure.
● A Unified Model: Construct Emergence
○ Nomological validity supersedes others.: the degree a construct works as proposed within the model

10. Proposed System
● Content Validation
○ The degree an indicator encompasses the content of the systemized concept: Evaluate logic
○ Focus on conceptual concerns; maintaining the internal the validity
○ Critique: Necessary but not sufficient condition for validation
● Convergent/Discriminant Validation
○ The empirical association of the indicator with the systematized concept: Measure relational effects
○ Convergent: what should be observed is. Discriminate: what should not be observed is not.
○ Critique: Deception in correlation, endogeneity concerns, narrowly confined,
● Nomological/Construct Validity
○ The degree indicators function in confirming the systemized concept: Corroboration, performance.
○ AHEM: Assume the Hypothesis, Evaluate the Measure
○ Critique: Using the product to validate the source is circular. Insufficient hypothesis weakens claims

11. Questions
1. Does agreement on communication metrics warrant a strong theoretical
foundation?
a. Doesn’t any agreement resolve the issues?
b. Is french, english, or chinese a better language?
2. Is procedure methods likened to a language, tradition, or a theory?
3. Is the primary root of this article an attempt to manage the academic
communities expectations?
a. Is this a masked complaint of those who operate too loosely for the authors tastes?
4. In a field with a plethora of contextual exceptions, is the structuring of processes
even relevant?
a. What could possibly be considered ordinary in human behavior?
b. Aren’t exceptions the norm of academic inquiry? If it fits nicely into a system, isn’t it likely known?