Risk-Adjusted Expected Development Outcome of Private Sector Investment Projects, Presentation at the 2011 Evaluation Week

1. Adjusting Development
Outcomes by Risk
Risk-Adjusted Expected Development Outcome of
Private Sector Investment Projects
IEG Evaluation Week
October 24, 2011
Hiroyuki Hatashima
IEG Private Sector Evaluation
1

2. Outline of the presentation
• Performance measurement vs risk
• Drivers of development outcome for
IFC investment projects
• Model based on drivers
• Two applications of the model
‒ Performance assessment
‒ Expected development outcomes
2

3. Performance measurement vs risk
• Risk is measurable possibilities of
losing or not gaining value
• Risk-return trade-off as the basis of
investment management
(if you don’t want risk, don’t expect return)
• For fair comparison, the return has to
be compared to the risk undertaken.
3

4. Private Sector Development Outcome
Development Outcome =
Overall effect on country’s development
Business Economic Envi. & Social Private Sector
Performance Performance Performance Development
Gov’t, taxpayers Demo. effects,
employees, linkages, corp. gov,
Neighbors &
customers, investment climate
Financiers Environment
suppliers & new entrants
4

5. Model based on Development
Outcome Results Drivers
DO = f + )+
(
5

6. Risk factors influencing
development outcomes
Coeff. Signif. at
Risk Factors Description against 5%
DO
Experience, financial capacity,
Sponsor Risk commitment and reputation of -0.21 Yes
sponsors (1=high risk, 0=low risk)
Business competitiveness in the
Market Risk market, distortions (1=high risk, 0=low -0.14 Yes
risk)
Changes in Changes in Country Risk indicators
Country between approval and evaluation (i.e. 0.009
Yes
Business 5 years) (score @ eval – score @
Climate approval)
“greenfield” (high risk) vs. expansion
Project Type -0.06 No
project (1=greenfield, 0=expansion)
Based on 2000-2010 XPSR Evaluation, N=655
Coefficient against DO in multiple variable regression. R2=0.09 6

7. Regression results of all drivers
Factors Coefficient Signif. at 5%
Sponsor Risk -0.11 Yes
Market Risk -0.12 Yes
Changes in Country Business Climate 0.005 Yes
Project type -0.07 No
Screening, appraisal & structuring work 0.33 Yes
quality
Supervision & administration work quality 0.28 Yes
IFC role and contribution 0.46 Yes
Based on 2000-2010 XPSR Evaluation, N=655
Coef. for multiple variable regression. R2=0.37 7

8. Two applications of the model
1. Performance assessment
• Outcomes with focus on IFC Work
Quality, taking risks
2. Likely outcomes
• Based on existing risks and work
quality scenarios
8

9. Use of model 1:
Performance Assessment
• Focusing on IFC controllable factors, given risks taken
• Which is the best performing region, adjusted by risk,
focusing on WQ improvement?
Region Development Outcome
high %
Southern Europe & Central Asia 87%
Middle East & North Africa 80%
Latin America & Caribbean 79%
Sub-Saharan Africa 74%
South Asia 72%
East Asia & Pacific 62%
Central & Eastern Europe 60%
IFC overall 73% 9

10. Two Work Quality
scenarios for comparison
DO= f (Risks + Work Quality) +
Scenarios: Potential Benchmark
WQ Perfect WQ = 100% High WQ ratings of previous
assumptions WQ across the board periods
Risks Actual project risks
Indicator Model-generated maximum expected success rate
means possible DO rating expected to outperform,
adjusted by the
embedded project risk
factors.
Comparison Indicate gaps to potential Reflecting WQ changes
with actual based on existing risks
outcome
10

11. Gaps between actual and potential/benchmark =
risk adjusted outcome indicators
Region Actual Potential Benchmark
Southern Europe 87% 90% 85%
& Central Asia
Middle East & 80% 86% 59%
North Africa
Latin America & 79% 92% 83%
Caribbean
Sub-Saharan 74% 88% 56%
Africa
South Asia 72% 90% 62%
East Asia & Pacific 62% 89% 55%
Central & Eastern 60% 90% 83%
Europe
IFC overall 73% 89% 72% 11

12. MENA & AFR outcomes stood out,
adjusted by risks/Work Quality.
MENA 25
SS AFR
20
S Asia 15
EAP 10
(Benchmark-Actual)
IFC 5
(Potential – Actual)
0
SECA
-40 -30 -20 -10 -5 0
LAC
-10
-15
CEE -20
-25
12
-30

13. INFRA (AFR/LAC/EMENA) and
FM (AFR/LAC) outcomes stood out
15
INF AFR/LAC
(Benchmark-Actual)
MAS Asia
10
INF Asia
INF EMENA
FM AFR/LAC
5
(Potential – Actual) IFC
MAS EMENA 0
-30 -25 -20 -15 -10 -5 0 5
FM Asia MAS AFR/LAC
-5
FM EMENA -10
-15
13

14. Use of model 2: likely outcomes
IDA – Non IDA gap in recent years: will it continue?
XPSR Development Outcome
90%
80%
70%
60%
50%
DO high %
40%
Non IDA
30%
IDA
20%
10%
0%
Evaluation Year 14

15. What’s behind the gap?
• Project Risks
– IDA more riskier than non IDA
Sponsor Risk Market Risk
70% 90%
60% 80%
50% 70%
High Risk %
High Risk %
40% 60%
50%
30%
40%
20%
30% NonIDA
10%
20% IDA
0%
10%
0%
Approval Year
Approval Year
15

16. What’s behind the gap?
• Changes in Country Business Climate
- IDA lower than non IDA
20
15
Changes in IICCR s cores
10
5 IDA
non-IDA
0
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
-5
-10 16
Evaluation year

17. What’s behind the gap?
Appraisal WQ and Roles dropped in IDA
Appraisal Work Quality Supervision Work Quality
90% 100%
80%
70% 80%
60% 60%
50%
40% 40%
30%
20% 20%
10% 0%
0%
Evaluation Year
Evaluation Year
Role and Contribution
100%
90%
80%
70%
60%
50%
40%
30% Non IDA
20% IDA
10%
0%
1996-98
1997-99
1998-00
1999-01
2000-02
2001-03
2002-04
2003-05
2004-06
2005-07
2006-08
2007-09
2008-10
17

18. Prospects for coming years
• Assume 08-10 XPSR WQ (IDA average)
• Actual risk of 2006-09 approvals (except
country risk – changes so far) – what will be
the likely results?
Sponsor Risk Market Risk
70% 90%
60% 80%
High Risk %
50% 70%
60%
40% 50%
30% 40%
20% 30%
20% NonIDA
10%
10%
0% 0% IDA
1995-97
1996-98
1997-99
1998-00
1999-01
2000-02
2001-03
2002-04
2003-05
2004-06
2005-07
2006-08
2007-09
Approval Year
Approval Year
18

19. Changes in country business
climate so far
• IDA – Not dropping sharply as non-IDA
Changes in IICCR Score between approval and
evaluation
(or approval and most recent score for 2006-2010
approvals = 2011-15 maturity)
20
15
Changes in IICCR score
10
5 IDA
non-IDA
0
-5
-10
Evaluation Year (approval + 5 years)
19

20. Future prospects: IDA-non IDA
gap will narrow
• If above WQ assumptions hold
Development Outcome (incl. projection)
90%
80%
Development Outcome High %
70%
60%
50%
40%
Non IDA
30%
IDA
20%
10%
0%
Evaluation year
= Predicted by the model 20

21. Summary
• Project performance = need to separate
factors controlled by institution vs risks
• Multi-factor Model can generate success
rates that
= can treat as benchmarks for performance
measurement;
= can indicate future outcomes, given risks and
work quality assumptions
21

22. Adjusting Development
Outcomes by Risk
Risk-Adjusted Expected Development Outcome of
Private Sector Investment Projects
IEG Evaluation Week
October 24, 2011
Hiroyuki Hatashima
IEG Private Sector Evaluation 22