On extrapolation<br />Federica Russo<br />Philosophy, Kent<br />
Overview<br />Extrapolation: an old problem<br />Extrapolation in social science: the background<br />Cook & Campbell<br /...
The evergreen riddle of induction<br />Goodman’s new riddle<br />Things are GRUE <br />If they are green before a certain ...
The riddle of extrapolation<br />Extrapolation, or external validity, is the inference<br />by which the results of one st...
Extrapolation in social science:Cook & Campbell<br />Validity:<br />the best available approximation<br />of the truth of ...
Exporting results:an issue of external validity<br />Generalisingto and across populations<br />External validity refers t...
The assessment of external validity<br />When? How?<br />Results of test of statistical interactions<br />Between the sele...
Extrapolation entersthe philosophical debate<br />An emergent awareness<br />For once, from the right angle:<br />Analogic...
Guala: analogical reasoning<br />External validity rests an empirical problem<br />Solution: wisely combine field and<br /...
The structure of such an inference can be reconstructed as follows:<br />1. If all directly observable features of the tar...
Steel: comparative process tracing<br />The extrapolator’s circle<br />Challenge of successfully exporting information fro...
Thus, efficient applications of comparative process tracing can focus on likely sources of difference in downstream stages...
Mechanism-based extrapolation is a step<br />beyond Cook & Campbell tradition<br />What’s the scope of mechanism-based ap...
Virtues<br />14<br />
Beyond ‘nichilist’ stanceà la LaFollette & Schanks<br />Causal Analogue Models:  can grant extrapolation<br />Hypothetical...
Field knowledge is important<br />Beyond statistically-minded tradition<br />of Cook and Campbell<br />Stress only represe...
Against detractors of extrapolation<br />… Even by Cook and Campbell!<br />The priority among validity types varies with t...
Luckily, contrary voices exist:<br />A primary goal in all sciences, including the social sciences, is the production of g...
Scope of mechanism-based extrapolation<br />19<br />
Guala:<br />	Sometimes external validity takes the form of “the in vitro–in vivo problem” (biochemistry), sometimes it is ...
Steel:<br />The best way to introduce to topic of this book is with a few examples.<br />Studies find that a particular su...
Too much in the same basket?<br />Are external validity issues / methods the same in domains as different as economics and...
Varieties of external validity inferences<br />
External validities<br />Internal/external: within/outside the sample<br />Population: different populations of subjects<b...
Knowledge of the mechanism<br />Steel / Guala<br />Knowledge of the functioning mechanism<br />in the model and in the tar...
Role of the mechanism<br />IARC: use evidence from animal models<br />This evidence should be explicitly mechanistic<br />...
Direction of the inference<br />To a larger population<br />e.g. RCTs<br />To the experimental setting<br />e.g. to test t...
How to improvethe mechanism-based approach<br />
Modelling and using mechanisms<br />Modelling<br />Find out what the mechanism is:<br />what it is made of, how it functio...
What population are we talking about?<br />Recall the many external validities on the market:<br />Internal/external: with...
The overlooked role ofpopulational properties<br />E.g. demographic, socio-political-economic characteristics<br />for the...
Thus, efficient applications of comparative process tracing can focus on likely sources of difference in downstream stages...
Downstream … where?<br />Appealing to the properties of the population<br />does not exactly coincide with “the sources of...
To sum up and conclude<br />The Cook & Campbell tradition<br />Mechanism-based extrapolation<br />Goesbeyond Cook & Campb...
Selected bibliography<br />Cook, T. and Campbell, D. (1979). Quasi-Experimentation. Design and Analysis Issues for Field S...
Upcoming SlideShare
Loading in...5
×

Extrapolation Kent Feb10

1,240

Published on

Published in: Technology, Economy & Finance
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,240
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Extrapolation Kent Feb10

  1. 1. On extrapolation<br />Federica Russo<br />Philosophy, Kent<br />
  2. 2. Overview<br />Extrapolation: an old problem<br />Extrapolation in social science: the background<br />Cook & Campbell<br />Guala and Steel<br />Virtues and vices of mechanism-based<br />approaches to extrapolation<br />How to improve the mechanism-based approach<br />Modelling vs. using mechanisms<br />Role of populational properties<br />2<br />
  3. 3. The evergreen riddle of induction<br />Goodman’s new riddle<br />Things are GRUE <br />If they are green before a certain time t<br />If they are blue and not examined before time t.<br />All Emeralds are green … but are they also grue?<br />Distinguishing between projectible and<br />non-projectible properties<br />The new riddle is a form of extrapolation <br />3<br />
  4. 4. The riddle of extrapolation<br />Extrapolation, or external validity, is the inference<br />by which the results of one study, e.g. an experiment,<br />are extended to a larger or a different population<br />or to a different setting<br />Distinguishing between projectible and<br />non-projectible results <br />animal models and human models;<br />experimental economics and real world economic situations;<br />demographic models across different countries; …<br />4<br />
  5. 5. Extrapolation in social science:Cook & Campbell<br />Validity:<br />the best available approximation<br />of the truth of causal statements<br />Types of validity<br />Internal:<br />confidence in causal relation within population<br />External:<br />confidence in generalising to other populations<br />5<br />
  6. 6. Exporting results:an issue of external validity<br />Generalisingto and across populations<br />External validity refers to the approximate validity with which we can infer that the presumed causal relationship can be generalized to and across alternate measures of the cause and the effect and across different types of persons, settings, and times. (Cook and Campbell, 1979, p. 37)<br />Generalizing to well-explicated target populations should be clearly distinguished from generalizing across populations. Each is germane to external validity: the former is crucial for ascertaining whether any research goal that specified populations have been met, and the latter is crucial for ascertaining which different populations (or subpopulations) have been affected by a treatment, i.e., for assessing how far one can generalize. (Cook and Campbell, 1979, p. 71)<br />6<br />
  7. 7. The assessment of external validity<br />When? How?<br />Results of test of statistical interactions<br />Between the selection of individuals to take part in the study<br />and the treatment / intervention<br />Representativeness of sample<br />Between history (= particular conditions of the study/experiment)<br />and treatment<br />Possibility to replicate studies<br />7<br />
  8. 8. Extrapolation entersthe philosophical debate<br />An emergent awareness<br />For once, from the right angle:<br />Analogical reasoning and mechanisms<br />8<br />
  9. 9. Guala: analogical reasoning<br />External validity rests an empirical problem<br />Solution: wisely combine field and<br />laboratory evidence in analogical reasoning<br />9<br />
  10. 10. The structure of such an inference can be reconstructed as follows:<br />1. If all directly observable features of the target and the experi- mental system are similar in structure;<br />2. If all the indirectly observable features have been adequately controlled in the laboratory;<br />3. If there is no reason to believe that they differ in the target system;<br />4. And if the outcome of the two systems at work (the data) is similar;<br />5. Then, the experimental and target systems are likely to be structurally similar mechanisms (or data-generating processes). <br />(Guala, 2005, p. 180)<br />10<br />
  11. 11. Steel: comparative process tracing<br />The extrapolator’s circle<br />Challenge of successfully exporting information from<br />model population knowing that it is limited and partial<br />The problem of difference<br />Challenge of providing successful methods in presence<br />of differences between the model and the target population<br />Solution:<br />Comparing the mechanisms in the model and in the target<br />at the points in which they are more likely to differ<br />11<br />
  12. 12. Thus, efficient applications of comparative process tracing can focus on likely sources of difference in downstream stages of the mechanism.<br />(Steel, 2008, p. 90, emphasis in the original)<br />12<br />
  13. 13. Mechanism-based extrapolation is a step<br />beyond Cook & Campbell tradition<br />What’s the scope of mechanism-based approaches?<br />Is all extrapolation practice mechanism-based?<br />Many extrapolation practices aren’t mechanism-based,<br />although arguably they should<br />13<br />
  14. 14. Virtues<br />14<br />
  15. 15. Beyond ‘nichilist’ stanceà la LaFollette & Schanks<br />Causal Analogue Models: can grant extrapolation<br />Hypothetical Analogue Models: can only suggest hypotheses to test<br />Animal models can only be HAM<br />Extrapolation possible only if there are no differences …<br />but that’s exactly the problem!<br />15<br />
  16. 16. Field knowledge is important<br />Beyond statistically-minded tradition<br />of Cook and Campbell<br />Stress only representativeness of samples<br />and possibility to replicate studies<br />16<br />
  17. 17. Against detractors of extrapolation<br />… Even by Cook and Campbell!<br />The priority among validity types varies with the kind of research being conducted. For persons interested in theory testing it is almost as important to show that the variables involved in the research are constructs A and B (construct validity) as it is to show that the relationship is causal and goes from one variable to another (internal validity). Few theories specify crucial target settings, population, or times to or across which generalization is desired. Consequently, external validity is of relatively little importance.[…] For investigators with theoretical interests our estimate is that the types of validity, in order of importance, are probably internal, construct, statistical conclusion, and external validity.<br />Cook and Campbell (1979, p. 83)<br />17<br />
  18. 18. Luckily, contrary voices exist:<br />A primary goal in all sciences, including the social sciences, is the production of general knowledge. General knowledge is knowledge that is not confined to the particulars of time and place.<br />Lucas (2003, p. 236)<br />18<br />
  19. 19. Scope of mechanism-based extrapolation<br />19<br />
  20. 20. Guala:<br /> Sometimes external validity takes the form of “the in vitro–in vivo problem” (biochemistry), sometimes it is called “ecological validity” (psychology), and sometimes it is called “parallelism” (economics), but the issue is always the same.<br />(Guala, 2005, p.160)<br />20<br />
  21. 21. Steel:<br />The best way to introduce to topic of this book is with a few examples.<br />Studies find that a particular substance is carcinogenic in rats. We would like to know whether it is also such in humans.<br />A randomized controlled experiment has found that a pilot welfare-to-work program improved the economic prospects of welfare recipients. It is desired to know whether the program will be similarly effective in other locations and when implemented on a larger scale.<br />On the basis of a controlled experiment concerning outcomes resulting from initiating anti-retroviral therapies earlier or later among HIV+ patients, a physician wishes to decide the best time to initiate this therapy for the patients she treats.<br />[. . . ]<br />I will use the term extrapolation to refer to inferences of this sort.<br />(Steel, 2008, p. 3)<br />21<br />
  22. 22. Too much in the same basket?<br />Are external validity issues / methods the same in domains as different as economics and biology?<br />An old issue arises again<br />Are the natural and social sciences completely apart?<br />Or is it a problem of having more direct and independent access<br />to social or biological mechanisms?<br />22<br />
  23. 23. Varieties of external validity inferences<br />
  24. 24. External validities<br />Internal/external: within/outside the sample<br />Population: different populations of subjects<br />Ecological: same subjects, different settings<br />Temporal: same subjects, same settings, different times<br />Do different types of validity require different methods of extrapolation?<br />What population are we talking about?<br />Cook & Campbell: maximise representativeness of sample<br />Mechanism-based extrapolation marks step beyond,<br />yet, not enough emphasis on the role of<br />socio-demo-political characteristics of populations<br />24<br />
  25. 25. Knowledge of the mechanism<br />Steel / Guala<br />Knowledge of the functioning mechanism<br />in the model and in the target<br />Ethnographers<br />Knowledge of mechanism in the model<br />‘Translate’ the mechanism in the target<br /><ul><li>A suitable less quantitative oriented</li></ul> mechanism-based extrapolation to be developed<br />25<br />
  26. 26. Role of the mechanism<br />IARC: use evidence from animal models<br />This evidence should be explicitly mechanistic<br />Verificationist strategies<br />during the process of research<br />Mechanistic considerations should come<br />at the ‘theory development stage’<br />through ‘modelling mechanisms’<br /><ul><li>Both should involve mechanistic considerations</li></ul>26<br />
  27. 27. Direction of the inference<br />To a larger population<br />e.g. RCTs<br />To the experimental setting<br />e.g. to test theoretical explanations /<br />general theories<br /><ul><li>They should involve mechanistic considerations</li></ul>27<br />
  28. 28. How to improvethe mechanism-based approach<br />
  29. 29. Modelling and using mechanisms<br />Modelling<br />Find out what the mechanism is:<br />what it is made of, how it functions<br />Infer the mechanism<br />from observations and experiments<br />Using<br />For explanation<br />Mechanisms carry explanatory power because they display<br />how the phenomenon was brought about<br />For external validity<br />Mechanisms used in various ways<br />29<br />
  30. 30. What population are we talking about?<br />Recall the many external validities on the market:<br />Internal/external: within/outside the sample<br />Population: different populations of subjects<br />Ecological: same subjects, different settings<br />Temporal: same subjects, same settings, different times<br />But what counts as ‘same’ or ‘different’ population?<br />30<br />
  31. 31. The overlooked role ofpopulational properties<br />E.g. demographic, socio-political-economic characteristics<br />for the choice of variables or of proxies for some properties or<br />of the statistical model, for the interpretation of results …<br />The possibility to extrapolate strongly depends<br />on the properties of the populations<br />The process of extrapolation itself requires<br />comparing the properties of the populations<br />This has been overlooked in both the statistically-minded<br />and the mechanism-based approach<br />31<br />
  32. 32. Thus, efficient applications of comparative process tracing can focus on likely sources of difference in downstream stages of the mechanism. A few important qualifications about the emphasis on downstream stages should be noted. The strategy could lead to mistaken conclusions if there is a path that bypasses the downstream stage. [. . . ] Second, the mark that upstream stages leave upon the downstream stages must be distinctive in the sense that it could not have resulted from some independent causes.<br />(Steel, 2008, p. 90)<br />32<br />
  33. 33. Downstream … where?<br />Appealing to the properties of the population<br />does not exactly coincide with “the sources of difference<br />in downstream stages of the mechanism”<br />identified by Steel.<br />is finding downstream differences<br />about the populations themselves.<br />33<br />
  34. 34. To sum up and conclude<br />The Cook & Campbell tradition<br />Mechanism-based extrapolation<br />Goesbeyond Cook & Campbell<br /><ul><li>Shouldn’t treat so fast biology and economics and alike</li></ul>Shouldbe adopted in extrapolation procedures<br />that do not involve mechanistic considerations<br /><ul><li>Should incorporate a thorough analysis of</li></ul>populational properties<br />34<br />
  35. 35. Selected bibliography<br />Cook, T. and Campbell, D. (1979). Quasi-Experimentation. Design and Analysis Issues for Field Settings. Rand MacNally, Chicago.<br />Godfrey-Smith, P. (2003). Goodman&apos;s problem and scientic methodology. The Journal of Philosophy, 100(11):573-590.<br />Goodman, N. (1955). Facts, Fiction, and Forecast. Cambridge University Press, Harvard.<br />Guala, F. (2005). The methodology of experimental economics. Cambridge University Press.<br />LaFollette, H. and Schanks, N. (1995). Two models of models in biomedical research. Philosophical Quarterly, 45:141-160.<br />Lucas, J. W. (2003). Theory-testing, generalization, and the problem of external validity. Sociological Theory, 21(3):236-253.<br />Steel, D. (2008). Across the boundaries. Extrapolation in biology and social science. Oxford University Press.<br />35<br />

×