Many, if not all portfolio managers spend a great deal of time worrying about the quality of their economic input data. These concerns cover the gamut from systemic over- or under-estimation, to project or region-specific bias, to simple sloppy and inconsistent data gathering. Many postpone initiating advanced decision-making techniques “until the data are in better shape”. However, investment decision data quality rarely improves substantially until that data starts being used consistently, systematically, and seriously. How then are we initiate use of advanced decision making techniques if we don’t trust the data? Business decisions are made every day, using the data that is available at the time. If we think of Portfolio Management and Optimization as techniques to improve the quality of our decisions, instead of as ways to find “the right answer”, we can begin to use these techniques to improve those decisions in spite of reservations we may have about the quality of data currently at hand. Once a system is in place and functioning, data quality improvement measures can start to take hold with sufficient feedback to guide the process.

1. Portfolio Management Using
Questionable Quality Data
SPE 90395
Jim DuBois
Portfolio Decisions Inc.
September 28, 2004

2. Opening Thoughts
This paper is not about improving data, but
about how to work with the data you havetoday.
Until you work with your data in a holistic
manner, it is very unlikely that it will improve.
Treat data quality as another unknown, like
reservoir size or Ps. Learn what can and
what can’t hurt you, and to what extent.
May result in choosing a portfolio that is
slightly less “optimal”, but also less sensitive
to perceived data issues.
SPE 90395

3. Avoid “Answerism”
Many delay using advanced decision methods
because “without perfect data, I won’t get the
right answer”. But:
There is no “right”
There is no “answer”
There are, however, better insights, which
lead to better decisions.
SPE 90395

4. Better Decisions
The question facing you is not, “Can I
make perfect decisions”, but can I make
better decisions than I am making now?
You are going to make decisions.
Improve in cycles as you learn.
SPE 90395

5. Procedure
Define the suspected data problem.
Describe the decision process.
Plan and execute the analysis.
Draw conclusions and communicate.
SPE 90395

6. Step One- Define What You Don’t
like About Your Data
Systemic Over or Under Estimation
Specific Bias
Random Error
– Competence/Training/Consistency
– Resources
– Emphasis/Management Pressure
SPE 90395

7. Step Two- Describe How the Data
is Currently Used for Decisions
Used in PM, we just don’t trust it.
Used in another decision process (Hurdles, Rank
and Cut, etc.)
Different data for approva, planning, budgeting,
etc.
Used as a starting point, then “massaged” by
decision makers, intermediaries.
– Massaged how?
Used as a smoke screen to legitimize intuitive
decision making.
Ignored
SPE 90395

8. Step 2 Assumption
For now, for today’s decision, this
is the best the data is going to get.
(Always try to improve for the next decision,
but don’t abdicate today’s decision.)
SPE 90395

9. Step Three- Plan Your Analysis Based
on Answers to Two Previous Questions
Two types of consequence from questionable
data:
– Performance- If you act on a set of data which is
flawed, the range of possible portfolio outcomes
will be different (usually worse) than you expect.
– Lost Opportunity- Had you had “correct” data, you
would have made different decisions. This is often
seen as a fairness issue.
Both types of consequence will always exist.
SPE 90395

10. Step Three- Plan Your Analysis
Based on Answers
Basic Procedure:
– Analyze honoring current data.
– Adjust data by increments in the direction
feared.
Ps
Reserve Size
Capital Cost
Timing, etc.
– (May be possible to use previous corporate
behavior as a guide). Do not reoptimize yet.
(Examples to follow).
SPE 90395

11. Step Three- Plan Your Analysis
Based on Answers
Basic Procedure (continued):
– Describe portfolio implications on two fronts:
Metric performance
Value
– Reoptimize at an interesting sensitivity point
– Describe implications on three fronts:
Metric performance
Selections
Value
(con’t)
SPE 90395

12. Step Three- Plan Your Analysis
Based on Answers
Basic Procedure (continued):
– Examine implication of new selections when
paired with original dataset.
– Continue analysis, seeking a portfolio with
acceptable performance if the data is “right”,
but which has resilience if the data is “wrong”.
– Can combine factors, but don’t overwhelm
your ability to understand what is going
on.Examine the impact of each factor
separately first, then in combination.
SPE 90395

13. Step Four- Draw Conclusions and
Communicate Them
Did the feared data problem have the
magnitude of impact you assumed?
In what time frame was the impact?
At what point did the impact become
significant?
If there are other decision methods being
used, use them with the same data variations.
Are the impacts more or less than using the
preceding analysis?
SPE 90395

14. Step Four- Draw Conclusions and
Communicate Them
If appropriate, reexamine at a coarser
granularity.
– Impactful changes vs. incidental
– Changes that result in major shifts of capital, jobs,
etc., obviously require more investigation
What was constant through the cases?
At what point (amount of bias) did impactful
changes start?
Communicate impacts in both directions.
SPE 90395

15. Examples
Three Examples in the Paper:
Ps Sensitivity Analysis
Capital Sensitivity Analysis
Prod and Reserve Bias Analysis
SPE 90395

16. Examples
WE want to understand:
The effect of possible under-estimation on
performance results
The effect of possible under-estimation on
project selection
The level of under-estimation at which these
effects become critical
If there exists a portfolio that performs better if
there is under-estimation, but still performs well
if there is not.
SPE 90395

17. Base Data Metrics
NPV = $18427 M
SPE 90395

18. Base Data Selections
SPE 90395

19. Ps –5% Metrics
NPV = $17516 M
SPE 90395

20. Ps –10% Metrics
NPV = $16604 M
SPE 90395

21. Ps –20% Metrics
NPV = $14782 M
SPE 90395

22. Ps –30% Metrics
NPV = $12959 M
SPE 90395

23. Ps –20% Metrics Reopt
SPE 90395

24. Ps –20% Metrics (Compare)
NPV = $14782 M
SPE 90395

25. Ps-20% Selections
SPE 90395

26. Ps –20% Selection Comparison
SPE 90395

27. Ps –20% Selections- Original Ps
SPE 90395

28. Ps Decrease Vs. NPV
SPE 90395

29. Ps Sensitivity Conclusions
Not as sensitive to Ps as suspected, largely
because of extra room in late years.
Largest effect is in late reserves- makes sense
since this was a critical metric.
A 20% across the board reduction in exploratory
Ps results in Reserves violations 2007-2009,
minor early production violations.
The 20% reduction case can be reoptimized to
honor all the constraints
SPE 90395

30. Ps Sensitivity Conclusions
Reoptimized solution moves away from DW and
Int’l, and towards the mid-continent.
If we use the original PS with the 20% reduction
selections, there is little loss of performance, but
we do exceed capital constraints in 2009.
Reserve performance is better.
Overall, loss of Value is more severe than loss
of needed performance.
SPE 90395

31. Some General Data
Considerations
In PM, frequency is usually more important than
amplitude. In ranking and hurdling, amplitude is
more important than frequency.
Beware the specific case. The general case is often
more telling, and more subject to common sense QC.
Hurdles cause bias.
Any decision is valid. Manipulating data to justify a
decision already made is not.
A vast amount of our “experience” has to do with
generated data, not with actual data.
Beware unchecked internal limits- i.e. limiting only by
internal cash flow- multiplies the effects of bias.
SPE 90395

32. Finally
Bottom Line:
– Shift from data paralysis to
understanding the impact of data
shortcomings.
– Understand what does and doesn’t
depend upon improving the data.
– Be an analyst, not merely a compiler.
SPE 90395