CHE Economic evaluation seminar 16th October 2020. Presented by David Manheim, School of Public Health, University of Haifa

1. The Value of Value of Information
When and How to Use Simpler or
Heuristic VoI approaches
David Manheim
Oct 15, 2020
Economic Evaluation
and Health Technology
Assessment Seminar

2. Table of contents
1 Preliminaries: Who am I, and what is VoI?
2 When to Use VoI: Two Easy Examples, and One Difficult One
3 A Path Forward for VoI
4 The Joys of Computational Approaches to VoI
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 1 / 14

3. Introduction
I’m David Manheim
This presentation will be about Value of Information (VoI).
It will be somewhat informal.
I’m hoping for feedback about how valuable this seems, whether
there is anything missing, and who it’s most useful for / where it
might lead.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 2 / 14

4. Introduction
I’m David Manheim
PhD from RAND Corporation in policy and decision analysis
Dissertation on "Value of Information for Policy Analysis"
Currently working on long term biological risks and policy, and global
catastrophic risk reduction across threats. (And currently, COVID.
Like everyone else.)
This presentation will be about Value of Information (VoI).
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 2 / 14

5. Introduction
I’m David Manheim
This presentation will be about Value of Information (VoI).
I’m going to give a very short introduction. I am hoping everyone
knows what VoI is, at least in general.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 2 / 14

6. Introduction
I’m David Manheim
This presentation will be about Value of Information (VoI).
This is a decision tree:
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 2 / 14

7. Introduction
I’m David Manheim
This presentation will be about Value of Information (VoI).
This is a decision tree when we know if the patient is sick:
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 2 / 14

8. Introduction
I’m David Manheim
This presentation will be about Value of Information (VoI).
This is a decision tree when we could collect information:
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 2 / 14

9. Introduction
I’m David Manheim
This presentation will be about Value of Information (VoI).
Here’s a more typical policy question where VoI would be useful:
Should we develop a new test type / information source, given
various decision options, given value is realized over time?
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 2 / 14

10. Introduction
I’m David Manheim
This presentation will be about Value of Information (VoI).
Here’s a more typical policy question where VoI would be useful:
Should we develop a new test type / information source, given
various decision options, given value is realized over time?
This doesn’t look like a tree.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 2 / 14

11. Introduction
I’m David Manheim
This presentation will be about Value of Information (VoI).
Here’s a more typical policy question where VoI would be useful:
Should we develop a new test type / information source, given
various decision options, given value is realized over time?
This doesn’t look like a tree.
(I’ll be talking about these types of question.)
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 2 / 14

12. Does VoI matter?
VoI is fundamental for almost every decision.
What impact will our research have?
What should we research?
Is this research valuable enough to fund?
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 3 / 14

13. Does VoI matter?
VoI is fundamental for almost every decision.
What impact will our research have?
What should we research?
Is this research valuable enough to fund?
Also, lots of applied / practical questions.
And all of these questions look like the last example.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 3 / 14

14. Does VoI matter?
VoI is fundamental for almost every decision.
What impact will our research have?
What should we research?
Is this research valuable enough to fund?
Also, lots of applied / practical questions.
And all of these questions look like the last example.
Shouldn’t we use VoI everywhere?
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 3 / 14

15. Does VoI matter?
VoI is fundamental for almost every decision.
What impact will our research have?
What should we research?
Is this research valuable enough to fund?
Also, lots of applied / practical questions.
And all of these questions look like the last example.
Shouldn’t we use VoI everywhere?
No.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 3 / 14

16. Rejecting VoI for Simple decisions
Where is VoI clearly overkill?
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 4 / 14

17. Rejecting VoI for Simple decisions
Where is VoI clearly overkill?
When the question is:
Simple enough that VoI analysis is unneeded.
Low value enough for VoI not to matter.
Clearly over-determined.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 4 / 14

18. Rejecting VoI for Simple decisions
Where is VoI clearly overkill?
When the question is:
Simple enough that VoI analysis is unneeded.
Low value enough for VoI not to matter.
Clearly over-determined.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 4 / 14

19. Rejecting VoI for Simple decisions
Where is VoI clearly overkill?
When the question is:
Simple enough that VoI analysis is unneeded.
Should I eat food instead of starving?
Low value enough for VoI not to matter.
What beer should I order?
Clearly over-determined.
A free diagnostic test with perfect accuracy.
An new treatment more effective and cheaper than extant ones.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 4 / 14

20. Rejecting VoI for Simple decisions
Where is VoI clearly overkill?
When the question is:
Simple enough that VoI analysis is unneeded.
Should I eat food instead of starving?
Low value enough for VoI not to matter.
What beer should I order?
Clearly over-determined.
A free diagnostic test with perfect accuracy.
An new treatment more effective and cheaper than extant ones.
But even simple questions need to be answered.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 4 / 14

21. Rejecting VoI for Simple decisions
Where is VoI clearly overkill?
When the question is:
Simple enough that VoI analysis is unneeded.
Should I eat food instead of starving?
Yes, if you can.
Low value enough for VoI not to matter.
What beer should I order?
Whatever you liked last time.
Clearly over-determined.
A free diagnostic test with perfect accuracy.
It should be used.
An new treatment more effective and cheaper than extant ones.
It should be moved to clinical trials.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 4 / 14

22. Rejecting VoI for Complex Decisions
Some questions are deeply uncertain in ways that make
probabilistic approaches difficult, so that VoI isn’t an appropriate
tool.
1 Water policy in California requires long-term decisions which have
payoffs that depend on future relative levels of rainfall in different
parts of the United States due to climate change.
2 Politically fraught decisions involving human lives, economic growth,
and government spending are often better supported with more
qualitative or multi-outcome decision making tools.
Others can be answered easily based on expert knowledge, or are
too time sensitive to do analysis.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 5 / 14

23. A Hard Example for VoI - Biosurveillance
Government spends vast sums to monitor seasonal influenza.
For the first-order decision problem, want to decide:
Observations about the influenza season
Posterior distribution of influenza prevalence and severity
Outcomes for each response action over the distribution.
What action should be taken in each case.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 6 / 14

24. A Hard Example for VoI - Biosurveillance
Government spends vast sums to monitor seasonal influenza.
Given:
Observations about the influenza season
Posterior distribution of influenza prevalence and severity
Outcomes for each response action over the distribution.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 6 / 14

25. A Hard Example for VoI - Biosurveillance
Government spends vast sums to monitor seasonal influenza.
Given:
Observations about the influenza season
Posterior distribution of influenza prevalence and severity
Outcomes for each response action over the distribution.
For a VoI analysis, we can also ask:
Should they collect a larger (or smaller) sample?
Should they use additional, complementary data sources?
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 6 / 14

26. A Hard Example for VoI - Biosurveillance
Government spends vast sums to monitor seasonal influenza.
Given:
Observations about the influenza season
Posterior distribution of influenza prevalence and severity
Outcomes for each response action over the distribution.
For a VoI analysis, we can also ask:
Should they collect a larger (or smaller) sample?
Should they use additional, complementary data sources?
So we also need:
A prior on frequency of difference cases.
A prior on the data generation characteristics for data sources.
...to find the value of each action given possible information, to find the
expected value of gathering information over different cases.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 6 / 14

27. Biosurveillance, Continued
I built a Bayesian Markov-Chain Monte-Carlo model to simulate
observable data from a surveillance system, given a compartmental
model of influenza1.
1
David Manheim. A Generative Bayesian Approach for Incorporating
Biosurveillance Sources into Epidemiological Models. bioRxiv 328518,
https://doi.org/10.1101/328518
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 7 / 14

28. Biosurveillance, Continued
I built a Bayesian Markov-Chain Monte-Carlo model to simulate
observable data from a surveillance system, given a compartmental
model of influenza1.
This was a bad idea.
We have very few useful interventions to significantly reduce
spread of influenza except in extreme cases.
We can identify extreme cases without much biosurveillance.
1
David Manheim. A Generative Bayesian Approach for Incorporating
Biosurveillance Sources into Epidemiological Models. bioRxiv 328518,
https://doi.org/10.1101/328518
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 7 / 14

29. Biosurveillance, Continued
I built a Bayesian Markov-Chain Monte-Carlo model to simulate
observable data from a surveillance system, given a compartmental
model of influenza1.
This was a bad idea.
We have very few useful interventions to significantly reduce
spread of influenza except in extreme cases.
We can identify extreme cases without much biosurveillance.
(Also, numeric integration + Markov-chain Monte Carlo = very slow)
1
David Manheim. A Generative Bayesian Approach for Incorporating
Biosurveillance Sources into Epidemiological Models. bioRxiv 328518,
https://doi.org/10.1101/328518
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 7 / 14

30. What I should have done.
I should have;
Clarified the decision problem and the intervention set.
Checked EVPI.
Checked where else information comes from. (Extreme cases
identify themselves.)
Focused on my other case study.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 8 / 14

31. What I should have done.
I should have;
Clarified the decision problem and the intervention set.
Checked EVPI.
Checked where else information comes from. (Extreme cases
identify themselves.)
Focused on my other case study.
Which leads to the general question:
What should a VoI decision analysis look like?
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 8 / 14

32. Staged VoI Analysis Process
Start simple. Only use more complex tools if needed.
1 Structure the problem;
2 Consider the set of potential data sources and interventions;
3 Consider value of information for key interventions using heuristics;
4 Use expert input and/or structured elicitation;
5 Consider computational modeling.
Each stage informs the following stages, and at each stage, there is a
chance to consider VoVoI, and whether to continue.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 9 / 14

33. Staged VoI Analysis Process
Start simple. Only use more complex tools if needed.
1 Structure the problem; This is just the basic decision analysis.
What decision(s) are being made?
What are the goals?
What are the alternatives / interventions?
Are there relevant uncertainties?
Ask: Is further work justified?
2 Consider the set of potential data sources and interventions;
3 Consider value of information for key interventions using heuristics;
4 Use expert input and/or structured elicitation;
5 Consider computational modeling.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 9 / 14

34. Staged VoI Analysis Process
Start simple. Only use more complex tools if needed.
1 Structure the problem;
2 Consider the set of potential data sources and interventions;
What uncertainties exist?
What information might we collect?
What interventions could be used?
Is there a relationship between choice of intervention and the
uncertainty?
Could the information change our decision?
Ask: Is the answer obvious yet?
3 Consider value of information for key interventions using heuristics;
4 Use expert input and/or structured elicitation;
5 Consider computational modeling.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 9 / 14

35. Staged VoI Analysis Process
Start simple. Only use more complex tools if needed.
1 Structure the problem;
2 Consider the set of potential data sources and interventions;
3 Consider value of information for key interventions using heuristics;
What specific cases have critical uncertainties?
What information would resolve the uncertainty?
What additional uncertainties remain?
Are there alternatives that are acceptable which do not depend on
the uncertainties?
How valuable is a good (better) decision?
Ask: Was that enough? (We hope so.)
4 Use expert input and/or structured elicitation;
5 Consider computational modeling.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 9 / 14

36. Staged VoI Analysis Process
Start simple. Only use more complex tools if needed.
1 Structure the problem;
2 Consider the set of potential data sources and interventions;
3 Consider value of information for key interventions using heuristics;
4 Use expert input and/or structured elicitation;
Build a simplified decision tree
Ask experts to predict input probabilities and values.
Plug in values to find VoI.
If needed: Do sensitivity tests.
Ask: Is the answer clear now? We really hope so.
5 Consider computational modeling.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 9 / 14

37. Staged VoI Analysis Process
Start simple. Only use more complex tools if needed.
1 Structure the problem;
2 Consider the set of potential data sources and interventions;
3 Consider value of information for key interventions using heuristics;
4 Use expert input and/or structured elicitation;
5 Consider computational modeling.
First: Will this actually impact the decision?
What type of computational model? (Discrete Bayesian Network or
Markov-Chain Monte-Carlo?)
This will require several components:
Causal model of phenomenon
Model of decision alternatives
Utility / preference model
Bayesian data source / information model Note: This must relate all
observable data (not just that evaluated for VoI) to the phenomenon
Priors for all uncertainties
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 9 / 14

38. A few desiderata for Computational VoI
The phenomenon needs to be predictable with a model.
Relationships should be causal, not correlative.
Interventions need to relate to the predictive model inputs.
Costs / decision procedures should depend only on the model and
interventions.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 10 / 14

39. A few desiderata for Computational VoI
The phenomenon needs to be predictable with a model.
The worse the predictive model is, the less valuable information is.
(If we can’t predict what will happen, we can’t decide what to do.)
Relationships should be causal, not correlative.
Interventions change the regime, and invalidate correlations.
Interventions need to relate to the predictive model inputs.
The model should have different outputs if we intervene. If the
interventions aren’t in the model, we can’t include them in the VoI
calculation.
Costs / decision procedures should depend only on the model and
interventions.
The decision needs to be predicted in the model in order to evaluate
the decision given information.
You can’t accurately model a decision which is based on an ad-hoc
"once it happens we’ll decide." (Many political decisions look like
this.)
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 10 / 14

40. Computational VoI Process (1/3)
1 Build a probabilistic model for all observable and unobservable
quantities;
Including both the outcome model with representation of
interventions, and the information model with a generative
relationship
Containing appropriate priors for the events and data source
relationships
Consistent with problem knowledge, including causal representation
of both the outcomes and interventions, and the relationship of
observable data to other parts of the model.
2 Condition on observed data to calculate a posterior estimate of:
The parametrization of data sources (for VoI)
The distributions of unobservable factors (for simulation)
The conditional distribution of outcomes based on interventions (for
validating the model and causal relationships used for later
simulation, if such data is available.)
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 11 / 14

41. Computational VoI Process (2/3)
3 Validate the model, and evaluate the fit of the posteriors, including
the data source model and the phenomenological model.
Validation and verification of a models is often assisted by evaluating
posterior fit
Non-identifiabilities can often be found at this stage, and are
important to eliminate where possible
The evaluation of the posterior differs by model type;
1 For BNs, likelihood and other diagnostics.
2 For statistical Bayesian representation, statistical measures of the fit of
a model
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 12 / 14

42. Computational VoI Process (3/3)
4 Simulate data sources for the range of cases
For BNs, this can be done without simulation (and avoids step 5).
1 Use utility nodes in a Bayesian decision network formulation
For statistical models, simulate fictitious observable data, and fit the
simulated data;
1 This requires a set of samples across the full range of the prior
distribution for observed and unobserved quantities, and for each data
source and intervention.
2 Plug outputs from the model of the simulated observable data into the
outcome model to approximate what the decisionmaker would know.
5 For statistical simulation, evaluate the expected outcomes of each
potential decision.
Generate simulated observable data for each case in the distribution.
Simulate choices based on a modeled estimate using the simulated
available data, and outcomes based on the simulated true case.
David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 13 / 14

43. David Manheim VoVoI - The Use and Abuse of VoI 16th October 2020 14 / 14