On the pre-conference day of PBIRG's Annual General Meeting 2012 in Chicago, SKIM presented about how to better deal with uncertainty in forecasts. Gerard Loosschilder, Jemma Lampkin and Eelke Roos explored Monte Carlo simulations for scenario planning and addressed 'what if' questions, building a Monte Carlo simulation from the ground up. Participants left with better ideas of how to deal with the certainty of uncertainty in forecasting, understanding how to just deal with it - turning uncertainty into a useful, and even playful, approach.

1. expect great answers
Uncertainty?
Just deal with it!
Jemma Lampkin | Eelke Roos | Gerard Loosschilder
For PBIRG | Chicago | May 2012

2. At the end of this workshop, we hope you are …
… even more accepting of uncertainty in your
forecasts, actually turning it into an integrated
part of your scenario thinking.

3. The purpose of a forecast is
to support business planning
Determine …
How much you are going to
sell.
If you will have a positive
return on your investment.

4. Your forecast … not a point estimate
Performance 
Your annual
peak sales
is $1 Billion
Time 
4

5. Your forecast … a range estimate
Your annual peak
Performance 
sales are
 100% sure to
be $800 million
 80% sure to be
$1 billion
Time   10% sure to be
At a likelihood of x% At moment tx $1.5 billion
5

6. The output is a range estimate
of likely outcomes Probable revenue
Cumulative revenue test treatment range: 90% chance
of revenue falling
1200 within this range
Cumulative revenue treatment (million USD)
based on Monte
Carlo simulation
1000 Maximum cumulative
revenue
800
Average cumulative
revenue
600
90% likelihood range
400
200
Minimum cumulative revenue
0
2012 2013 2014 2015 2016 2017 2018 2019 2020 2021

7. Sources of uncertainty
can be categorized in two clusters:
The accuracy of metrics The likelihood of events
• Metrics collected in our • Market conditions that
studies may change
• Metrics available in the • Competitive actions
public domain, and reactions,
syndicated data and preempting and trailing
with the client
That is why we prefer to talk about scenario thinking
instead of forecasting, to properly focus the attention on
the question “what if?”.
7

8. To deal with uncertainty and risk, we suggest using …
Monte Carlo Simulation
An alternative way to support scenario thinking
8

9. Monte Carlo Simulation is an extension
of your modeling practice
Stochastic
Δ Input Δ Output
Not deterministic
Inputs and
outputs follow a
Uniform if distribution
uncertainty 60
is high 50
40
30
20
Normal if 10
0
uncertainty 1 2 3 4 5 6 7 8 9 10
is low
9

10. A normal distribution
if uncertainty is low
Compliance The input variable
of “compliance”
600
Number of simulations at this value (#)
500 assumes a
400 normal
distribution with a
300
mean of 50% and
200
a standard
100 deviation of 8%.
0
0% 20% 40% 60% 80% 100%
Compliance value (%)
Input
10

11. A uniform distribution
if uncertainty is high
Uptake after the 1st year The input variable
of “uptake”
600
Number of simulations at this value (#)
500 assumes a
400 uniform
distribution with
300
an equal
200
likelihood of all
100 values between
0
40% and 60% to
1% 11% 21% 31% 41% 51% 61% 71% 81% 91%
Uptake value (% of peak share)
happen.
Input
11

12. The likelihood of events
are inserted as discrete variables
Launch scenario Efficacy scenario
100% 100%
80% 80%
60%
60% 50% 60%
40% 40%
25% 25% 20% 20%
20% 20%
0% 0%
Worst Base Best
These events have discrete
probabilities of happening
Input
12

13. The likelihood of events
are inserted as discrete variables
Launch scenario Efficacy scenario
100% 100%
80% 80%
60% 50% 60% 50%
40%
40% 40% 30%
20%
20% 10% 20%
0% 0%
Worst Base Best
These events have discrete
probabilities of happening
Input
13

14. Probability distribution of sales forecast
if uncertainties in continuous inputs are high
Probability distribution of sales
100% The distribution of
Probability of making the sales (%)
90% forecasted sales
80%
values shows a
70%
60%
gradual decline
50% as a result of
40%
higher
30%
20%
uncertainties in
10% continuous input
0%
0.5 0.8 1.1 1.4 1.7 2.0 2.3 2.6 2.9 3.2 3.5 3.8 4.1 4.4 4.7 5.0 5.3 5.6 5.9 variables.
Sales in billion USD
Output
14

15. Probability distribution of sales forecast
if uncertainties in continuous inputs are low
Probability distribution of sales
100% The distribution of
Probability of making the sales (%)
90% forecasted sales
80%
values shows a
70%
60% steep decline as
50% a result of lower
40%
uncertainties in
30%
20%
continuous input
10% variables.
0%
0.5 0.8 1.1 1.4 1.7 2.0 2.3 2.6 2.9 3.2 3.5 3.8 4.1 4.4 4.7 5.0 5.3 5.6 5.9
Sales in billion USD
Output
15

16. Probability distribution of sales forecast
if critical input variables have higher values
Probability distribution of sales
100% The distribution of
Probability of making the sales (%)
90% forecasted sales
80%
values shifts to
70%
60% the right as a
50% result of higher
40%
values for the
30%
20%
input variables.
10%
0%
0.5 0.8 1.1 1.4 1.7 2.0 2.3 2.6 2.9 3.2 3.5 3.8 4.1 4.4 4.7 5.0 5.3 5.6 5.9
Sales in billion USD
Output
16

17. Probability distribution of sales forecast if
strongly impacted by discrete input variables
Probability distribution of sales
100% The distribution of
Probability of making the sales (%)
90% forecasted sales
80%
values assumes a
70%
60% step-wise shape
50% as a result of a
40%
higher impact of
30%
20%
discrete input
10% variables.
0%
0.5 0.8 1.1 1.4 1.7 2.0 2.3 2.6 2.9 3.2 3.5 3.8 4.1 4.4 4.7 5.0 5.3 5.6 5.9
Sales in billion USD
Output
17

18. Working with uncertainties works best
if we also manage our expectations
That is why we work with
action standards.
An action standard is a
threshold value that a key
performance indicator needs
to exceed at an acceptable
risk, before we to decide to
pursue the initiative.
I.e., we want to be 80% sure
to make $1 billion or more.
18

19. We met the action standard
Probability distribution of Action standard
sales
100%
We want to be 80%
Probability of making the sales (%)
90% sure to make $1
80%
70%
billion or more.
60%
50%
40% Result
30%
The probability of
20%
10% making $1 billion worth
0%
of sales is 84%, so we
2.0
5.9
0.5
0.8
1.1
1.4
1.7
2.3
2.6
2.9
3.2
3.5
3.8
4.1
4.4
4.7
5.0
5.3
5.6
Sales in billion USD have exceeded the
action standard.
19

20. Introduction into the business case
How Ducendi Inc. wants to build a business case
for its in-licensing agreement with Novus
pharmaceuticals

21. Novus is developing an oral type 2
diabetes drug with a novel mode of action
Novus pharmaceuticals is a
biotechnology company on the rise.
In order to raise new funding, Novus
has offered the new treatment in an
in-license agreement to Ducendi, a
big pharmaceutical corporation.
Ducendi wants to know how likely it
is for Periculum to get a positive
ROI.
21

22. Novus claims a high likelihood of success
Ducendi is not so sure
Upside Downside
• Survey sponsored by • A likelihood that efficacy is
Novus: 80% of physicians only moderate
are positive; 60% are likely • Competitive treatments in
to prescribe it clinical development are
• Advantages: safety and expected to have
tolerability profile, interaction with Periculum
risk/benefit profile and • Competitive treatments may
Mode of Action be launched sooner than
Periculum
22

23. How well do you deal with the uncertainty?
Your 5-year revenue will surely be $1.5 billion
Your 5-year revenue will have a 80% likelihood of being
$1.5 billion
Your 5-year revenue will have a 80% likelihood of being
$1.5 billion
It also has a 99% likelihood of being $300 million
and 30% of being $2 billion
23

24. Setting the action standard for this case
• What would be the accepted
amount of risk you are
willing to take?
• How would you set the
action standard?
• Would setting an action
standard like this fit with your
business practice and
resonate with your team?
24

25. Exercise – set the action standard for Ducendi’s
$1.5 billion investment in Novus’ Periculum
• Senior management has asked you to assess the likelihood of a
positive ROI 5 years post-launch
• Ducendi has calculated a positive ROI to equal $1.5 billion in 5 years
• This investment includes the development, production, launch and
maintenance of Periculum
How certain do we need to be of
5 year revenue reaching this revenue point?
$ 1.5 billion At %
$ 2.0 billion At %
We will use these numbers in the business case.
25

26. Introduction into the Monte Carlo Simulator
for scenario thinking
The return on investment of Periculum
launched in two major markets

27. Ducendi wants to forecast the potential in two
crucial markets, the United States and Elbonia
United States of America Elbonia
Strategically important Strategically important
established market emerging market
• Largest T2D market in the world Big opportunity but …
in terms of revenue • Market characterized by high out-
• Health insurance provided by the of-pocket expenses
both public and private entities • High use of branded generics
• Complex payer dynamics • Aggressive low cost local
competitors
• T2D data available from many • Not many data available. High
sources at high precision, quality uncertainty and low quality.
and certainty levels Based on qualitative impressions
 High risk, low uncertainty  High risk, high uncertainty
Accounts for ~70% of revenue Accounts for ~30% of revenue
27

28. With Periculum being launched in 2016, Ducendi
wishes to break even in 5 years
30 Let us assume the
Million T2D patients
25 size of the patient
population is a
20
given at a lower
15 and upper bound.
10
5
0
2016 2017 2018 2019 2020
Year
US, Minimum US, Maximum
Elbonia, Minimum Elbonia, Maximum
28

29. Ducendi uses conjoint methodology to measure
demand for Periculum under various scenarios
Ducendi’s conjoint study replicates the following launch scenarios:
Efficacy (phase III) of Periculum
• Higher than phase II data (best case)
• Similar to phase II data (base case)
• Lower than phase II data (worst case)
Competitive launch
• Before Periculum
• At the same time as Periculum
• After Periculum
29

30. Competition is expected to launch a similar drug.
However, who goes first?
First-mover advantage: the first mover preempts the follower, and
gets a lasting advantage throughout this 5 year period.
The first mover advantage is modeled as a likelihood in the scenarios:
what’s the likelihood of:
P C E.g.,
Periculum first,
30%
competitor second
2015 2016 2017 2018 2019 2020
PC E.g.,
Periculum and competitor
40%
at the same time
2015 2016 2017 2018 2019 2020
C P E.g.,
Competitor first,
30%
Periculum second
2015 2016 2017 2018 2019 2020
30

31. Now we need your input!
What are the ranges we put in,
and what level of uncertainty do we assume?

32. First, we look at the accuracy of market data:
compliance/persistence and uptake
Uptake Compliance x Persistence
What do you expect the uptake of the new drug to be What do you expect the patient compliance
by the physician population? and persistence with the new drug to be?
Uptake is influenced by satisfaction with current Compliance is the patient’s adherence to the
products, awareness/”buzz,” access/price, opportunity, prescribed dose per day
competition and the quality of the product
Shape Persistence is the proportion of patients
United States Elbonia (uncertainty) persisting with the prescribed therapy
Min: 35 % Min: %
2016
Max: 40 % Max: % Uniform (high) United Shape
States Elbonia (uncertainty)
Min: 65 % Min: %
2017
Max: 75 % Max: % 75
Lower % %
Min: 95 % Min: % Bound Uniform (high)
2018
Max: 100 % Max: % Normal (low)
Normal (low)
80
Min: 100 % Min: % Upper % %
2019 Bound
Max: 100 % Max: %
Min: 100 % Min: %
2020
Max: 100 % Max: %

33. Second, we look at the likelihood of events:
efficacy and a competitive launch
Efficacy Competitive launch
Coming out of phase III, what is What is the likelihood of the competitor
the likelihood of Periculum to be drug to be launched before or after
less, equally, or more efficacious Periculum, or at the same time?
than measured in phase II?
Higher United States Elbonia
___%
(best case)
Before 20_
_ _% 60
___%
Similar
___% Same time 50_
_ _% 30
___%
(base case)
After 30_
_ _% 10
___%
Lower
___%
(worst case)
33

34. Now let us plug in the numbers and …
See what happens in the business case
34

35. So, did we make it?
Target Actual
Revenue % of risk Revenue % of risk
Total $ 1.5 billion At __ % $ 1.5 billion At %
Total $ 2.0 billion At % $ 2.0 billion At %
Do you want to go back and change a few parameters
to see what happens?
Set action standard Set market data Set launch data

36. That is all nice,
but my business cannot deal with uncertainty.
My business needs to make a decision!
So, how can we help the business
make a decision while dealing with uncertainty?

37. Eventually, the business needs to make a
few decisions to overcome the uncertainty
Yes
Did we
meet or
exceed the
action
standard?
No
37

38. First, the business needs to decide
if it finds enough reason to continue
Continue with
Yes
the initiative
Did we Not meeting the action
meet or
standard usually
exceed the
action results in more
standard? questions and
uncertainty. The
business needs to
No Now what? decide what to do next.
38

39. If not, the business needs to decide if it is
due to the quality and accuracy of the data
Continue with
Yes
the initiative
Did we
meet or
exceed the
action Yes
standard?
Did we
have the
No best data
we could
have had?
No
39

40. If not, the business needs to decide if it is
due to the quality and accuracy of the data
Continue with
Yes
the initiative
Deciding that the
data were not
Did we
accurate is the
meet or
exceed the easiest way out.
action Yes Now what? But what if the
standard? data were the best
Did we we could have?
have the
No best data
we could
have had?
Invest in more
No
accurate data
40

41. Last, the business needs to decide what is
in its power to meet the action standard
Continue with
Yes Yes
the initiative
Can the
Did we business Some parameters
meet or invest to can be in control
exceed the have a of the business,
action Yes higher like investments in
standard? probability compliance or time
Did we of meeting
have the
to market.
the action
No best data standard?
we could
have had? No
Invest in more
No
accurate data
41

42. Last, the business needs to decide what is
in its power to meet the action standard
Continue with Revise the
Yes Yes
the initiative business case
Can the
Did we business
meet or invest to
exceed the have a
action Yes higher
standard? probability
Did we of meeting
have the the action
No best data standard?
we could
Stop the
have had? No
initiative
Invest in more
No
accurate data
42

43. We hope that by now, you’re even more
accepting of uncertainty in your forecasts
Turning it into an integrated part of scenario thinking
• Working with a Monte Carlo based simulator, thinking
in terms of ranges instead of point estimates
• Setting action standards in consultation with the
business, representative of their appetite to risk
43

44. Any great questions?
Jemma Lampkin | Senior Project Manager
j.lampkin@skimgroup.com | +1 201 963 8430
Eelke Roos | Project Manager
e.roos@skimgroup.com | +1 201 963 8430
Gerard Loosschilder | Chief Methodology Officer
g.loosschilder@skimgroup.com | +31 10 282 3535