1) The document discusses the differences between explanatory and predictive modeling in scientific research. 2) Explanatory models are used to test causal theories, while predictive models are used to predict new records or scenarios. 3) Explanatory power and predictive accuracy are different and one cannot be inferred from the other. The best explanatory model is often not the best predictive model and vice versa.

1. Galit ShmuéliGeorgetown UniversityOctober 30, 2009To Explain or To Predict?Explanatory vs. Predictive Modeling in Scientific Research

2. The path to discovery

3. ExplainPredict

4. What are “explaining”? “predicting”?

5. Statistical modeling in social science researchPurpose: test causal theory (“explain”)Association-based statistical models Prediction nearly absent

6. Lesson #1:Whether statisticians like it or not,in the social sciences,association-based statistical models are used for testing causal theory.Justification: a strong underlying theoretical model provides the causality.

7. Definition: Explanatory ModelA statistical model used for testing causal theory(“proper” or not)

8. Definition: Predictive ModelAn empirical model used for predicting new records/scenarios

10. Multi-page sections with theoretical justifications of each hypothesis

11. Concept operationalizationPovertyTrustAngerEconomic stabilityWell-being4 pages of such tables

12. Statistical model (here: path analysis)

13. “Statistical” conclusions

14. Research conclusions

15. Lesson #2In the social sciences,empirical analysis is mainly used for testing causal theory.Empirical prediction is considered un-academic.Some statisticians share this view: The two goals in analyzing data... I prefer to describe as “management” and “science”. Management seeks profit... Science seeks truth.Parzen, Statistical Science 2001

16. Prediction in the Information Systems literature

17. Predictive goal stated?Predictive power assessed?

18. 1072 articles of which52 empirical with predictive claims“Examples of [predictive] theory in IS do not come readily to hand, suggesting that they are not common”Gregor, MISQ 2006

19. Breakdown of the 52 “predictive” articles

20. Why Predict?Scientific use of empirical modelsTo PredictTo Explaintest causal theory(utility)relevancenew theorypredictability

21. Why are statistical explanatory models different than predictive models?

22. Theory vs. its manifestation?

23. “The goal of finding models that are predictively accurate differs from the goal of finding models that are true.”

24. Given the research environment in the social sciences, two critically important points are:Explanatory power and predictive accuracy cannot be inferred from one another.The “best” explanatory model is (nearly) never the “best” predictive model, and vice versa.

25. Point #1Explanatory PowerPredictive Power≠Cannot infer one from the other

26. What is R2 ?

27. In-sample vs. out-of-sample evaluation

28. out-of-sampleinterpretationprediction accuracyp-valuesPerformance EvaluationR2costsgoodness-of-fitrun timeDanger: type I,II errorsDanger: over-fitting

29. Suggestion for social scientists:Report predictive accuracy in addition to explanatory power

30. Predictive PowerExplanatory Power

31. Point #2Best explanatory model≠Best predictive model

32. Predict ≠ Explain“We should mention that not all data features were found to be useful. For example, we tried to benefit from an extensive set of attributes describing each of the movies in the dataset. Those attributes certainly carry a significant signal and can explain some of the user behavior. However, we concluded that they could not help at all for improving the accuracy of well tuned collaborative filtering models.” Bell et al., 2008 +?

33. Predict ≠ ExplainThe FDA considers two products bioequivalent if the 90% CI of the relative mean of the generic to brand formulation is within 80%-125%“We are planning to… develop predictive models for bioavailability and bioequivalence”Lester M. Crawford, 2005Acting Commissioner of Food & Drugs

34. Let’s dig in

35. Explanatory goal: minimize model biasPredictive goal: minimize MSE(model bias + sampling variance)

36. What isOptimized?BiasPrediction MSEorVar(Y)= uncontrollablebias2 = model misspecificationestimation (sampling variance)

37. Linear Regression ExampleUnderspecified modelEstimated modelTrue modelEstimated modelMSE2 < MSE1 when: σ2 large |β2| small corr(x1,x2) high limited range of x’s

38. China's Diverging Paths, photo by Clark SmithTwostatistical modeling paths

39. Design & CollectionData PreparationGoal DefinitionEDAVariables? Methods?Model Use & ReportingEvaluation, Validation & Model Selection

40. Hierarchical dataStudy design & data collectionObservational or experiment?Primary or secondary data?Instrument (reliability+validity vs. measur accuracy) How much data? How to sample?

41. reduced-feature modelspartitioningData preparationmissing

42. summary statsplotsoutlierstrendsInteractive visualizationPCASVD

43. Which variables?Multicollinearity?A, B, A*B?theoryassociationsex-post availability

44. Methods / ModelsbiasvarianceBlackbox / interpretableMapping to theoryridge regressionensemblesboostingPLSPCR

45. Model fit ≠ValidationExplanatory powerEmpirical modelTheoretical modelDataEvaluation, Validation& Model SelectionTraining dataEmpirical modelOver-fitting analysisHoldout dataPredictive power

46. Model UseInferenceTest causal theoryNull hypothesisPredictions(utility)RelevanceNew theoryPredictabilityPredictive performanceOver-fitting analysisNaïve/baseline

47. Design & CollectionData PreparationGoal DefinitionEDAVariables? Methods?Model Use & ReportingEvaluation, Validation, & Model Selection

48. How does all this impact research in the (social) sciences?

49. Three Current ProblemsPrediction underappreciatedDistinction blurredInappropriate modeling/assessment“While the value of scientific prediction… is beyond question… the inexact sciences [do not] have…the use of predictive expertise well in hand.”Helmer & Rescher, 1959

50. Why?What can be done?Statisticians should acknowledge the difference and teach it!

51. It’s time for ChangeTo PredictTo Explain