This paper studies an identification problem that arises when clinicians seek to personalize patient care by predicting health outcomes conditional on observed patient covariates. Let y be an outcome of interest and let (x = k, w = j) be observed patient covariates. Suppose a clinician wants to choose a care option that maximizes a patient's expected utility conditional on the observed covariates. To accomplish this, the clinician needs to know the conditional probability distribution P(y|x = k, w = j). It is common to have a trustworthy evidence-based risk assessment that predicts y conditional on a subset of the observed covariates, say x, but not conditional on (x, w). Then the clinician knows P(y|x = k) but not P(y|x = k, w = j). Research on the ecological inference problem studies partial identification of P(y∣x, w) given knowledge of P(y|x) and P(w|x). Combining this knowledge with structural assumptions yields tighter conclusions. A psychological literature comparing actuarial predictions and clinical judgments has concluded that clinicians should not attempt to subjectively predict patient outcomes conditional on covariates that are not utilized in evidence-based risk assessments. I argue that formalizing clinical judgment through analysis of the identification problem can improve risk assessments and care decisions.

1. CREDIBLE ECOLOGICAL INFERENCE FOR MEDICAL DECISIONS
WITH PERSONALIZED RISK ASSESSMENT
Quantitative Economics, 2018, forthcoming
Charles F. Manski
Department of Economics and Institute for Policy Research
Northwestern University

2. Background: My Research Program
I consider how limited ability to predict illness and treatment response may affect the
welfare achieved in patient care.
I call attention to questionable methodological practices in the research that supports
evidence-based medicine.
I perform basic research on identification whose objective is to yield credible prediction of
patient outcomes.
I apply statistical decision theory to suggest reasonable decision criteria with well-
understood welfare properties.

3. Sources
Horowitz, J., and C. Manski (2000), “Nonparametric Analysis of Randomized Experiments with
Missing Covariate and Outcome Data,” Journal of the American Statistical Association, 95, 77-84.
Manski, C. (2009), "Diversified Treatment under Ambiguity," International Economic Review, 50,
1013-1041.
Manski, C. (2010), "Vaccination with Partial Knowledge of External Effectiveness," Proceedings of
the National Academy of Sciences, 107, 3953-3960.
Manski, C. (2013), “Diagnostic Testing and Treatment under Ambiguity: Using Decision Analysis to
Inform Clinical Practice,” Proceedings of the National Academy of Sciences, 110, 2064-2069.
Manski, C. and A. Tetenov (2016), "Sufficient Trial Size to Inform Clinical Practice," Proceedings
of the National Academy of Sciences, 113, 10518-10523.
Manski, C. (2017), "Mandating Vaccination with Unknown Indirect Effects," Journal of Public
Economics Theory, Vol. 19, No. 3, 2017, pp. 603-619.
Manski, C. (2018), "Credible Ecological Inference for Medical Decisions with Personalized Risk
Assessment," Quantitative Economics, forthcoming.
Manski, C. (2018), "Reasonable Patient Care under Uncertainty," Health Economics, forthcoming.

4. My concern is improvement of empirical analysis that seeks to inform patient care.
I focus on decision making under uncertainty.
By "uncertainty," I do not just mean that clinicians make probabilistic rather than
deterministic predictions of patient outcomes.
I mean that available evidence and medical knowledge may not suffice to yield precise
probabilistic predictions.
A patient may ask:
"What is the chance that I will develop disease X in the next five years?"
"What is the chance that treatment Y will cure me?"
A credible response may be a range, say "20 to 40 percent" or "at least 50 percent."

5. Decision theorists may use the terms "deep uncertainty" and "ambiguity."
I encompass them within the broader term "uncertainty."
Uncertainty has sometimes been acknowledged verbally, but it has generally not been
addressed in research on evidence-based medicine.
I think this a huge omission.

6. I have no formal training in medicine.
The contributions that I might make concern the methodology of evidence-based medicine.
This lies within the expertise of econometricians, statisticians, and decision analysts.
Research on treatment response and risk assessment shares a common objective:
Probabilistic prediction of patient outcomes conditional on patient attributes.
Development of methodology for probabilistic conditional prediction is a core concern of
econometrics and statistics.
(synonyms: regression, actuarial prediction, machine learning)

7. Statisticalimprecisionandidentification problems affectempiricalresearch thatuses sample
data to predict population outcomes.
Statistical theory characterizes the inferences that can be drawn about a study population by
observing the outcomes of a finite sample.
Identification analysis studies inferential difficulties that persist when sample size grows
without bound.
Identification problems often are the dominant difficulty.
I particularly study problems of partial identification.

8. Identification problems are given little formal attention in the textbooks and articles that
haveprovidedthestandardmethodologicalframework for evidence-based medicalresearch.
These sources focus on statistical imprecision.
As a result, medical researchers, guideline developers, and clinicians may have little
quantitative sense of how identification problems affect prediction.
In contrast, study of identification has been a core concern of research in econometrics from
the beginnings of the field in the 1930s through the present.
I summarize here my most recent study of partial identification in medicine.

9. Partial Personalized Risk Assessment
Evidence-based risk assessments condition on a subset of clinically observable patient
attributes.
Example: The CDC life tables predict life span y conditional on x = (age, sex, race) but not
conditional on attributes w characterizing a patient's health status.
Example: The NCI Breast Cancer Risk Assessment predicts the probability that a woman
will develop invasive breast cancer (y = 0 or 1) conditional on
x = (history of breast cancer, presence of BRCA mutation, age, age of first menstrual
period, age of first live birth of a child, number of first-degree female relatives with
breast cancer, number of breast biopsies, race/ethnicity),
but not on other clinically observable attributes w.

10. Clinicians have used subjective judgment to predict outcomes conditional on all observed
patient attributes, but psychologists have found these judgments fallible.
Manski (QE, 2018) shows that this identification problem can be mitigated if the predictions
made by evidence-based studies are combined with auxiliary data that reveal the distribution
of the additional attributes across the patient population.
The findings are bounds on personalized probabilities of disease development.

11. The Ecological Inference Problem in Personalized Medicine
Let each member of a population be characterized by a triple (y, x, w) whose value lies in
a set Y × X × W. Let P(y, x, w) denote the population distribution.
A well-known problem is identification of the long conditional distribution P(y*x,w) given
knowledge of the short conditional distributions P(y*x) and P(w*x).
The Law of Total Probability relates the short and long predictive distributions:
P(y*x = k) = P(w = j*x = k)P(y*x = k, w = j) + P(w … j*x = k)P(y*x = k, w … j).
Knowledge of P(y|x) alone reveals nothing about P(y*x, w). Partial conclusions may be
drawn if one knows P(y|x) and P(w|x). This is the ecological inference problem.

12. In patient care,
y is a patient health outcome of interest.
(x, w) are clinically observable patient attributes.
The clinician wants to know P(y|x, w).
It is common to have a trustworthy risk assessment that predicts y conditional on a subset
of the observed attributes, say x, but not conditional on (x, w).
Then the clinician knows P(y|x) but not P(y|x, w).

13. Evidence-Based Care and Clinical Judgment
How should the clinician assess risk?
One option is to ignore w and base care only on x. Another is to make subjective Bayesian
predictions conditional on (x, w). This is called prediction with clinical judgment.
Psychological research comparing evidence-based predictions with ones made by clinical
judgment has concluded that the former consistently outperforms the latter when the
predictions are made using the same patient attributes.
The gap in performance persists even when clinical judgment uses additional attributes as
predictors. See Dawes, Faust, and Meehl (Science, 1989).

14. They attribute the weak performance of clinical judgment to clinician failure to adequately
grasp the logic of the prediction problem and to their use of decision rules that place too
much emphasis on w relative to x.
They caution against use of clinical judgment to predict disease risk or treatment response.

15. Prediction Formalizing Clinical Judgment
Perhaps psychologists are correct to advise against use of subjective clinical judgment to
predict patient outcomes.
This does not foreclose the possibility of making well-grounded predictions that combine
evidence with judgment.
Analysis of the ecological inference problem uses knowledge of P(y|x) and P(w|x).
This knowledge is often available in practice.

16. Basic Bounds without Structural Assumptions
Assume that the evidence-based assessment tool correctly reveals P(y|x).
Consider a patient with attributes (x = k, w = j).
The joint identification region for P(y|x = k, w = j) and P(y*x = k, w … j) given knowledge
of P(y|x) and P(w|x) is the set of pairs of long distributions that satisfy the Law of Total
Probability, restated here:
P(y*x = k) = P(w = j*x = k)P(y*x = k, w = j) + P(w … j*x = k)P(y*x = k, w … j).

17. Predicting Binary Outcomes
When y is binary, the identification region is
P(y = 1|x = k, w = j) 0 [0, 1]
P(y = 1|x = k) ! P(w … j|x = k) P(y = 1|x = k)
1 [————————————–, ——————].
P(w = j|x = k) P(w = j|x = k)
This result was sketched by Duncan and Davis (ASR, 1953). The first formal proof appears
to be in Horowitz and Manski (ECMA, 1995).
The interval is interior to [0, 1], with width P(w … j|x = k)/P(w = j|x = k), when
P(w … j|x = k) < P(y = 1|x = k) < P(w = j|x = k).

18. Predicting Mean and Quantile Outcomes
When y is real-valued, there is no characterization of the identification region for P(y|x, w)
of comparable simplicity.
Horowitz and Manski (ECMA, 1995) derive relatively simple expressions for the
identification regions of the mean and quantiles of P(y|x, w).
Consider E(y*x = k, w = j) or a quantile Qá(y*x = k, w = j), where á 0 (0, 1).
Horowitz and Manski prove a general result for the class of parameters that respect
stochastic dominance and apply this result to the mean and quantiles.

19. Let p / P(w … j|x = k).
The identification region for P(y*x = k, w = j) contains a “smallest” distribution Lp that is
stochastically dominated by all feasible values of P(y*x = k, w = j) and a “largest”
distribution Up that stochastically dominates all feasible P(y*x = k, w= j).
Lp right-truncates P(y|x = k) at its (1 ! p)–quantile and Up left-truncates P(y|x = k) at its
p–quantile.
If D(@) is a parameter that respects stochastic dominance, the smallest feasible value of
D[P(y*x = k, w = j)] is D(Lp) and the largest feasible value is D(Up).
Sharp lower and upper bounds on E(y*x = k, w = j) are the means of Lp and Up. Sharp
bounds on Qá(y*x = k, w = j) are the á–quantiles of Lp and Up.

20. Illustration
A common problem in risk assessment is to predict remaining life span conditional on
observed patient attributes. Let y denote remaining life span.
CDC life tables provide actuarial predictions of life span in the U. S. conditional on (age,
sex, race). The life tables do not predict life span conditional on other patient attributes that
clinicians commonly observe.
Let x = 50-year-old male in one of two races, non-Hispanic (NH) black or white.
Let w = classification of persons into those with or without high blood pressure (HBP).

21. The life tables show that
E(y|age 50, NH black male) = 26.6, E(y|age 50, NH white male) = 29.7.
Data in the National Health and Nutrition Examination Survey provide a basis for estimation
of P(w|x). I use the age-aggregated probabilities
P(HBP|NH black male) = 0.426, P(HBP|NH white male) = 0.334.
The data yield these sharp bounds on E(y|age, race, sex, blood pressure):
E(y|age 50, NH black male, not HBP) 0 [18.1, 35.4]
E(y|age 50, NH black male, HBP) 0 [14.3, 38.5]
E(y|age 50, NH white male, not HBP) 0 [23.8, 36.4]
E(y|age 50, NH white male, HBP) 0 [15.6, 42.0]

22. Risk Assessment with Strong Structural Assumptions
Tighter inferences maybe feasiblewith structuralassumptions. The literature has developed
two approaches imposing assumptions that point-identify P(y|x, w).
Instrumental Variables
Goodman (ASR, 1953) considered inference on E(y*x, w).
The Law of Iterated Expectations gives
E(y*x = k) = 3 E(y*x = k, w = j)P(w = j*x = k), k 0 X.
j 0 W
This a system of *X* linear equations in the *X*×*W* unknowns E(y*x = k, w =j), (k, j) 0
X × W.

23. Goodman uses x as an instrumental variable, assuming that
E(y*x = k, w = j) = E(y*w = j), all (k, j) 0 X × W.
Then
E(y*x = k) = 3 E(y*w = j)P(w = j*x = k), k 0 X.
j 0 W
This is a system of *X* equations in the *W* unknowns E(y*w = j), j 0 W.
The equations have a unique solution if *X* $ *W* and if matrix [P(w = j*x = k), (k, j) 0 X
× W] has full rank *W*.
The IV assumption is refutable. The equation system may have no solution, or its solution
may lie outside the basic bound on E(y*x = k, w = j).

24. Although Goodman demonstrated the identifying power of the IV assumption, he did not
advocate its regular use in practice.
He cautioned that the assumption holds "in very special circumstances."
If the assumption holds, the attributes x used by assessment tools have no predictive power
when one conditions prediction on the additional attributes w.
It is difficult to conjecture instances in assessment of health risk where the assumption may
be credible.
It may be credible when health risk is genetically determined, x measures a phenotype
statistically associated with the underlying genetic determinant of risk, and w is the finding
of a DNA test that directly measures the genetic determinant.

25. Illustration
Consider the life-table and blood-pressure illustration, with the IV assumption
E(y|age 50, NH black male, not HBP) = E(y|age 50, NH white male, not HBP),
E(y|age 50, NH black male, HBP) = E(y|age 50, NH white male, HBP).
There are two linear equations in two unknowns, whose unique solution is
E(y|age 50, NH black male, not HBP)
= E(y|age 50, NH white male, not HBP) = 41.0,
E(y|age 50, NH black male, HBP) = E(y|age 50, NH white male, HBP) = 7.3.
This solution is outside the basic bounds. Hence the IV assumption is invalid.

26. Parametric Models
The second approach used to point-identify P(y|x, w) asserts a parametric model that places
these conditional distributions in a finite-dimensional family.
Let È be a subset of L-dimensional real space. Let F(@, @, @) be a function mapping X × W
× È into distributions on Y. Assume there exists a è 0 È such that
P(y|x = k, w = j) = F(k, j, è), all (k, j) 0 X × W.
Then the Law of Total Probability yields
P(y*x = k) = 3 F(k, j, è)P(w = j*x = k), k 0 X.
j 0 W
è solves this system of equations.

27. Analysis of distributional equations can be difficult.
Progress can be made by considering the implications for prediction of mean outcomes.
Let e(k, j, è) denote the mean of the random variable with distribution F(k, j, è).
Insertion of e(k, j, è) into the Law of Iterated Expectations yields
E(y*x = k) = 3 e(k, j, è)P(w = j*x = k), k 0 X.
j 0 W
This system of equations is similar to Goodman's system except that e(k, j, è) generally
varies nonlinearly with è.
A specific parametric model was proposed by King(1997), who asserted that he had found
“a solution to the ecological inference problem.” His assumptions drew strong criticism.

28. Bounded-Variation Assumptions
There is a substantial middle ground between making no structural assumptions and making
assumptions strong enough to yield point identification.
Bounded-variation assumptions flexibly restrict the magnitudes of risk assessments and the
degree to which they vary with patient attributes, enabling clinicians to express quantitative
judgments in a structured way.
Bounded-variation assumptions have previously been used to provide identifying power in
other settings. See Manski and Pepper (ECMA, 2000; JQC, 2013; REStat, 2018).

29. Binary Outcomes
Recall the Law of Total Probability
P(y*x = k) = P(w = j*x = k)P(y*x = k, w = j) + P(w … j*x = k)P(y*x = k, w … j).
Assume that
a(k, …j) # P(y = 1*x = k, w … j) # b(k, …j),
a(k, j) # P(y = 1*x = k, w = j) # b(k, j).
These bounded-variation assumptions yield this identification region:
P(y = 1*x = k, w = j) 0 [a(k, j), b(k, j)] 1
P(y = 1|x = k) ! b(k, …j)@P(w … j|x = k) P(y = 1|x = k) ! a(k, …j)@P(w … j|x = k)
[ ————————————–———, ————————————————].
P(w = j|x = k) P(w = j|x = k)

30. Illustration: Monotone Risk Variation
Assume that risk of illness increases with the value of a binary attribute w. Thus,
0 # P(y = 1*x = k, w = 0) # P(y = 1*x = k),
P(y = 1*x = k) # P(y = 1*x = k, w = 1) # 1.
Then
P(y = 1*x = k, w = 0) 0 [0, P(y = 1*x = k)]
P(y = 1|x = k) ! P(w = 1|x = k)
1 [ —————————–———, P(y = 1|x = k)],
P(w = 0|x = k)
P(y = 1*x = k, w = 1) 0 [P(y = 1*x = k), 1]
P (y = 1|x = k)
1 [P(y = 1|x = k), ——————].
P (w = 1|x = k)

31. Bounds on Mean Outcomes
A clinician may find it credible to impose various bounds on E(y|x, w). Leading cases are
1. a(kN, jN) # E(y*x = kN, w = jN) # b(kN, jN).
2. a(kN, kO, j) # E(y*x = kN, w = j) ! E(y*x = kO, w = j) # b(kN, kO, j),
This weakens the Goodman IV assumption, where a(kN, kO, j) = b(kN, kO, j) = 0.
3. a(k, jN, jO) # E(y*x = k, w = jN) ! E(y*x = k, w = jO) # b(k, jN, jO).
The identification region for E(y*x = k, w = j) is the interval whose lower (upper) bound
minimizes (maximizes) E(y*x = k, w = j) subject to the bounded-variation assumptions and
the restrictions implied by knowledge of P(y|x) and P(w|x).

32. Cross and Manski (ECMA, 2002) proves that, for each value of kN, knowledge of P(y|x) and
P(w|x) confines the |W|-vector [E(y*x = kN, w] to a bounded convex set whose extreme
points are the expectations of certain |W|-tuples of stacked distributions.
The identification region for E(y|x, w) without use of bounded-variation assumptions is the
Cartesian Product of these convex sets across the values of x.
Computation of the identification region for E(y*x = k, w = j) is generally difficult when
bounded-variation assumptions are imposed.

33. It is useful to obtain an informative bound that may not be sharp but is easy to compute.
The identification region for E(y*x = k, w = j) given only knowledge of P(y|x) and P(w|x)
is the interval [E(Ljk), E(Ujk)].
It follows that E(y|x, w) is contained in the |X|×|W|-rectangle × (kN, jN) 0 X × W [E(LjNkN), E(UjNkN)],
which is easy to compute. This is the smallest rectangular set that encloses the convex
identification region studied by Cross and Manski.
Combinethe bounded-variation assumptions with knowledgethatE(y|x,w)satisfiestheLaw
of Iterated Expectations and is contained in × (kN, jN) 0 X × W [E(LjNkN), E(UjNkN)]. The result is a
simple but possibly non-sharp bound on E(y*x = k, w = j).
The lower and upper bounds solve linear programming problems.

34. Illustration
Consider the life-table and blood-pressure illustration.
Assume that persons with HBP have less life expectancy than those without HBP:
0 # E(y|age 50, NH white male, not HBP) ! E(y|age 50, NH white male, HBP),
0 # E(y|age 50, NH black male, not HBP) ! E(y|age 50, NH black male, HBP).
Assume that black males have up to 2.5 years less life expectancy than white males
conditional on blood pressure:
0 # E(y|age 50, NH white male, not HBP) ! E(y|age 50, NH black male, not HBP) # 2.5,
0 # E(y|age 50, NH white male, HBP) ! E(y|age 50, NH black male, HBP) # 2.5.

35. Combining these assumptions with the bounds on E(y|x, w) using only knowledge of P(y|x)
and P(w|x) yields the bounded-variation bounds
E(y|age 50, NH black male, not HBP) 0 [29.4, 35.4],
E(y|age 50, NH black male, HBP) 0 [14.7, 22.9],
E(y|age 50, NH white male, not HBP) 0 [31.9, 36.4],
E(y|age 50, NH white male, HBP) 0 [16.3, 25.4].

36. Patient Care with Partial Personalized Risk Assessment
The basic lesson of the above analysis is that one may often draw some credible conclusions
about P(y|x, w) but one can rarely learn it precisely.
A clinician with partial knowledge may or may not have sufficient information to choose
an optimal treatment for a patient (e.g., one that maximizes objective expected utility).
How might a clinician choose patient care when credible risk assessment is not sufficiently
informative to optimize?

37. Bayesian decision theorists suggest maximization of subjective expected utility, using
clinical judgment to make a subjective probabilistic risk assessment.
Bayesian patient care may be attractive if subjective probabilistic risk assessment has a
credible foundation, but it may be harmful otherwise.
A clinician who acts without making a subjective probabilistic risk assessment is said to face
a problem of decision making under ambiguity.
There exists no optimal strategy for decision making under ambiguity, but one can usefully
pose alternative decision criteria and compare their properties.

38. A broad idea is to use a criterion that achieves uniformly satisfactory results, whatever the
truth may be.
There are multiple ways to formalize the idea of uniformly satisfactory results.
Two that have long been prominent are the maximin and minimax-regret criteria.
I have previously applied these criteria to medical decision making in Manski (IER, 2009;
PNAS, 2010; PNAS, 2013; JPET, 2017).
In (QE, 2018), I consider choice under ambiguity between surveillance of a patient and
aggressive treatment. An example is choice between periodic screening for breast cancer
and aggressive drug treatment.

39. Directions for Further Applications and Methodological Research
This paper reports new analysis of the ecological inference problem and its implications for
decision making.
It adds to previous analysis of partial identification by studying ecological inference with
bounded variation assumptions.
It initiates patient-centric analysis of treatment under ambiguity by considering choice
between surveillance and aggressive treatment using partial personalized risk assessment.

40. These contributions notwithstanding, I do not view the primary significance of the paper to
be development of new methodology per se.
It is rather that the paper calls attention to a difficult identification problem that arises in
personalized assessment of health risks and shows how analysis of partial identification and
decision making under ambiguity may potentially be used to improve patient care.
The paper uses two important questions in risk assessment to illustrate, prediction of breast
cancer conditional on multiple patient attributes and prediction of life span conditional on
demographic covariates and blood pressure.

41. There is considerable scope for new methodological research that would usefully extend the
present analysis.
I assume complete knowledge of P(y|x) and P(w|x) throughout the paper. In practice, a
decision maker may have only finite-sample estimates of these distributions and thus have
to cope with sampling imprecision as well as the ecological inference problem.
Another open question is characterization of the identification region for P(y|x, w) and study
of decision making under ambiguity when the outcome is multi-dimensional. The analysis
restricted attention to settings in which y is binary or real-valued. This suffices for some
applications but not for others.