How to select between two consultants, each differing in the accuracy of their predictions, each charging a different fee (hint: it's not intuitive).

1. Strategy
Implementation.

2. Selecting from Competing Services
The Audience for this Deck:
• Those charged with writing, or managing the implementation of, a quantitative, transparent business plan
• or developing governing policy for organizational effectiveness
How to evaluate and choose between two competing consultants.
• Each charge a different amount
• Each has a different history of predictions
Which one do you choose, and why?
Marc Vandenplas, mjvandenplas@gmail.com

3. Example
You are determining whether or not to make a product and sell it. There is a question of the Sales Volume, i.e. how much of your product
you will sell. Your own expert has estimated the probabilities of certain sales volumes.
Two consulting market research firms have approached you to provide their services. Firm A will charge a fee of $10,000 to predict the
Sales Volume for your product. However, Firm B wants to charge a fee that is a whopping $700,000 to predict the Sales Volume for your
product. You ask the firms for their respective history of predictions and they readily provide it.
The firms’ two histories are similar, but different, and you need to evaluate which option to select:
• A or B,
• or none,
• or possibly both A & B.
in view of their wildly different prices they charge, your own expert’s previous estimates (which we do not want to ignore), and other factors
such as the intended price and total costs.
You must evaluate the firms in view of their respective history and fees, your own expert’s estimates, and your intended pricing and
expected total costs.
The following deck illustrates how this is accomplished and results in
• Selection of the firm most appropriate for our needs
• A Policy addresses the circumstances in which we will, and will not, make our Product
• A Risk Profile for our business

4. Your Own Expert Has Estimated the Probable Volume of Sales
Sales may be low, nominal, or high.
50% of 1 unit
(High Volume)
20% of 14 units
(Low Volume)
30% of 6 units
(Med Volume)

5. Firm A says They Predict Volume with this Accuracy:
Their Accuracy is not 100%.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Low in fact Med in fact High in fact
Firm A’s Accuracy
Predicted "Low" Predicted "Med" Predicted "High"
Firm A Costs $100,000

6. 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
"Low" in fact "Med" in fact "High" in fact
Firm B’s Accuracy
Predicted "Low" Predicted "Med" Predicted "High"
Firm B Costs $700,000
Firm B says They Predict Volume with this Accuracy:
Their Accuracy is not 100%.

7. The Question: Which Firm (A or B, or Both, or None) Do You Select?
• Firm A charges $100K and has a good history of predictions
• Firm B charges $700K and also has a good history of predictions
• Hiring Both A & B will cost $800K
• Not Hiring Either will save you their fees (at least $100k)
Doing which of the above adds the most value to your business plan? We enter the information into an appropriate Decision Model.
Price
Cost
Decision
Model

8. Make Product
0
[37000000]None
Firm A's Prediction
-100000
[36900000]Firm A
Sales Volume
[-11065854]Yes
[-700000]No
Make Product
[-700000]Low
Sales Volume
[48014286]Yes
[-700000]No
Make Product
[48014286]Med
Sales Volume
[100133333]Yes
[-700000]No
Make Product
[100133333]High
Firm B's Prediction
-700000
[40550000]Firm B
Firm A's Prediction
-800000
[40450000]Both A & B
Which Research Firm
[40550000]
Selecting Firm B adds net $3.55
million more value
The Answer: Firm B, costing $700K, should be selected.
Firm B $ 40,550,000
None $ (37,000,000)
$ 3,550,000
Decision
Model

9. Make Product
0
[37000000]None
Firm A's Prediction
-100000
[36900000]Firm A
Sales Volume
[-11065854]Yes
[-700000]No
Make Product
[-700000]Low
Sales Volume
[48014286]Yes
[-700000]No
Make Product
[48014286]Med
Sales Volume
[100133333]Yes
[-700000]No
Make Product
[100133333]High
Firm B's Prediction
-700000
[40550000]Firm B
Firm A's Prediction
-800000
[40450000]Both A & B
Which Research Firm
[40550000]
We should not make the Product if
Firm B predicts “Low”
Combining your own Expert’s Estimate with Firm B’s,
Decision
Model

10. Our Policy has less chance of losing
money.
This Policy is Better at Managing Risk:
Decision
Model

11. What We’ve Just Demonstrated
Which market research firm (or any costly observation) is not an obvious selection
• You compared two firms, each with a different cost and predictive history
• Of the two, it turned out that the most costly one was the best firm to select
Combining all our predictions of the Sales Volume into a decision model gave us a Policy
• You kept the prediction made by your own expert
• You added Firm A’s predictive accuracy and its $100,00 cost
• And Firm B’s predictive accuracy and its $700,000 cost
• Plus the Price and the Total Cost (Sales Volume * Price – Total Cost = Profit)
The resulting Policy is to
• Not make the Product if Firm B’s prediction is for “Low” Sales Volume, otherwise make the Product
• The selection of Firm B, although they cost $700,000, raises your risk-weighted expected value by $3,55
million
• And better reduces your Risk of losing money