2. Overview Extrapolation: an old problem Extrapolation in social science: the background Cook & Campbell Guala and Steel Virtues and vices of mechanism-based approaches to extrapolation How to improve the mechanism-based approach Modelling vs. using mechanisms Role of populational properties 2
3. The evergreen riddle of induction Goodman’s new riddle Things are GRUE If they are green before a certain time t If they are blue and not examined before time t. All Emeralds are green … but are they also grue? Distinguishing between projectible and non-projectible properties The new riddle is a form of extrapolation 3
4. The riddle of extrapolation Extrapolation, or external validity, is the inference by which the results of one study, e.g. an experiment, are extended to a larger or a different population or to a different setting Distinguishing between projectible and non-projectible results animal models and human models; experimental economics and real world economic situations; demographic models across different countries; … 4
5. Extrapolation in social science:Cook & Campbell Validity: the best available approximation of the truth of causal statements Types of validity Internal: confidence in causal relation within population External: confidence in generalising to other populations 5
6. Exporting results:an issue of external validity Generalisingto and across populations External validity refers to the approximate validity with which we can infer that the presumed causal relationship can be generalized to and across alternate measures of the cause and the effect and across different types of persons, settings, and times. (Cook and Campbell, 1979, p. 37) Generalizing to well-explicated target populations should be clearly distinguished from generalizing across populations. Each is germane to external validity: the former is crucial for ascertaining whether any research goal that specified populations have been met, and the latter is crucial for ascertaining which different populations (or subpopulations) have been affected by a treatment, i.e., for assessing how far one can generalize. (Cook and Campbell, 1979, p. 71) 6
7. The assessment of external validity When? How? Results of test of statistical interactions Between the selection of individuals to take part in the study and the treatment / intervention Representativeness of sample Between history (= particular conditions of the study/experiment) and treatment Possibility to replicate studies 7
9. Guala: analogical reasoning External validity rests an empirical problem Solution: wisely combine field and laboratory evidence in analogical reasoning 9
10. The structure of such an inference can be reconstructed as follows: 1. If all directly observable features of the target and the experi- mental system are similar in structure; 2. If all the indirectly observable features have been adequately controlled in the laboratory; 3. If there is no reason to believe that they differ in the target system; 4. And if the outcome of the two systems at work (the data) is similar; 5. Then, the experimental and target systems are likely to be structurally similar mechanisms (or data-generating processes). (Guala, 2005, p. 180) 10
11. Steel: comparative process tracing The extrapolator’s circle Challenge of successfully exporting information from model population knowing that it is limited and partial The problem of difference Challenge of providing successful methods in presence of differences between the model and the target population Solution: Comparing the mechanisms in the model and in the target at the points in which they are more likely to differ 11
12. Thus, efficient applications of comparative process tracing can focus on likely sources of difference in downstream stages of the mechanism. (Steel, 2008, p. 90, emphasis in the original) 12
13. Mechanism-based extrapolation is a step beyond Cook & Campbell tradition What’s the scope of mechanism-based approaches? Is all extrapolation practice mechanism-based? Many extrapolation practices aren’t mechanism-based, although arguably they should 13
15. Beyond ‘nichilist’ stanceà la LaFollette & Schanks Causal Analogue Models: can grant extrapolation Hypothetical Analogue Models: can only suggest hypotheses to test Animal models can only be HAM Extrapolation possible only if there are no differences … but that’s exactly the problem! 15
16. Field knowledge is important Beyond statistically-minded tradition of Cook and Campbell Stress only representativeness of samples and possibility to replicate studies 16
17. Against detractors of extrapolation … Even by Cook and Campbell! The priority among validity types varies with the kind of research being conducted. For persons interested in theory testing it is almost as important to show that the variables involved in the research are constructs A and B (construct validity) as it is to show that the relationship is causal and goes from one variable to another (internal validity). Few theories specify crucial target settings, population, or times to or across which generalization is desired. Consequently, external validity is of relatively little importance.[…] For investigators with theoretical interests our estimate is that the types of validity, in order of importance, are probably internal, construct, statistical conclusion, and external validity. Cook and Campbell (1979, p. 83) 17
18. Luckily, contrary voices exist: A primary goal in all sciences, including the social sciences, is the production of general knowledge. General knowledge is knowledge that is not confined to the particulars of time and place. Lucas (2003, p. 236) 18
20. Guala: Sometimes external validity takes the form of “the in vitro–in vivo problem” (biochemistry), sometimes it is called “ecological validity” (psychology), and sometimes it is called “parallelism” (economics), but the issue is always the same. (Guala, 2005, p.160) 20
21. Steel: The best way to introduce to topic of this book is with a few examples. Studies find that a particular substance is carcinogenic in rats. We would like to know whether it is also such in humans. A randomized controlled experiment has found that a pilot welfare-to-work program improved the economic prospects of welfare recipients. It is desired to know whether the program will be similarly effective in other locations and when implemented on a larger scale. On the basis of a controlled experiment concerning outcomes resulting from initiating anti-retroviral therapies earlier or later among HIV+ patients, a physician wishes to decide the best time to initiate this therapy for the patients she treats. [. . . ] I will use the term extrapolation to refer to inferences of this sort. (Steel, 2008, p. 3) 21
22. Too much in the same basket? Are external validity issues / methods the same in domains as different as economics and biology? An old issue arises again Are the natural and social sciences completely apart? Or is it a problem of having more direct and independent access to social or biological mechanisms? 22
24. External validities Internal/external: within/outside the sample Population: different populations of subjects Ecological: same subjects, different settings Temporal: same subjects, same settings, different times Do different types of validity require different methods of extrapolation? What population are we talking about? Cook & Campbell: maximise representativeness of sample Mechanism-based extrapolation marks step beyond, yet, not enough emphasis on the role of socio-demo-political characteristics of populations 24
29. Modelling and using mechanisms Modelling Find out what the mechanism is: what it is made of, how it functions Infer the mechanism from observations and experiments Using For explanation Mechanisms carry explanatory power because they display how the phenomenon was brought about For external validity Mechanisms used in various ways 29
30. What population are we talking about? Recall the many external validities on the market: Internal/external: within/outside the sample Population: different populations of subjects Ecological: same subjects, different settings Temporal: same subjects, same settings, different times But what counts as ‘same’ or ‘different’ population? 30
31. The overlooked role ofpopulational properties E.g. demographic, socio-political-economic characteristics for the choice of variables or of proxies for some properties or of the statistical model, for the interpretation of results … The possibility to extrapolate strongly depends on the properties of the populations The process of extrapolation itself requires comparing the properties of the populations This has been overlooked in both the statistically-minded and the mechanism-based approach 31
32. Thus, efficient applications of comparative process tracing can focus on likely sources of difference in downstream stages of the mechanism. A few important qualifications about the emphasis on downstream stages should be noted. The strategy could lead to mistaken conclusions if there is a path that bypasses the downstream stage. [. . . ] Second, the mark that upstream stages leave upon the downstream stages must be distinctive in the sense that it could not have resulted from some independent causes. (Steel, 2008, p. 90) 32
33. Downstream … where? Appealing to the properties of the population does not exactly coincide with “the sources of difference in downstream stages of the mechanism” identified by Steel. is finding downstream differences about the populations themselves. 33
34.
35. Selected bibliography Cook, T. and Campbell, D. (1979). Quasi-Experimentation. Design and Analysis Issues for Field Settings. Rand MacNally, Chicago. Godfrey-Smith, P. (2003). Goodman's problem and scientic methodology. The Journal of Philosophy, 100(11):573-590. Goodman, N. (1955). Facts, Fiction, and Forecast. Cambridge University Press, Harvard. Guala, F. (2005). The methodology of experimental economics. Cambridge University Press. LaFollette, H. and Schanks, N. (1995). Two models of models in biomedical research. Philosophical Quarterly, 45:141-160. Lucas, J. W. (2003). Theory-testing, generalization, and the problem of external validity. Sociological Theory, 21(3):236-253. Steel, D. (2008). Across the boundaries. Extrapolation in biology and social science. Oxford University Press. 35
