Evaluation Approaches, Methods and Data in a Rapidly    Changing Context: Global Crisis Response Evaluations              ...
The Global Economic Crisis and the Challenge                     to WBG                     Rapid movement of economic eve...
Crisis Evaluation: Nature of the ChallengeDaily data show the decline in value of the UK   Complexity of CrisisFTSE index ...
Crisis Evaluation: Nature of the Challenge                       Scope of the present paper                         Quest...
Crisis Evaluation –Nature of Evaluative                   Questions                    Evaluative Questions – complex     ...
Crisis Evaluation: IEG’s Phased ResponseMonthly data on emerging markets showed                                           ...
1. Retrospective Evaluation - WB Response toPast Crises – 17 case studies                   Evaluative questions based on ...
2. WBG Response to the Global Crisis:                    Phase I Evaluation    Private Capital Flows during the crisis:   ...
2. WBG Response to the Global Crisis                   Phase I Evaluation    Alternative Scenarios – Poverty in      Phase...
WBG Response to the Global Crisis -Phase II             Scope of the Evaluation                        Utilization of high...
WB Response to the Global Crisis                Phase II Evaluation Methods     Use of High Frequency data to measure mult...
Evaluating the Lending Response relative to                                        stress     Incremental Lending Relative...
Volume and Distribution of Support -                                                            Other IFIs / MDBs     Incr...
Crisis Evaluation and High Frequency Data - EBRD     EBRD’s crisis response evaluation established a three phase typology ...
Crisis Evaluation and High Frequency Data -                         EBRD     EBRD’s evaluation then traces crisis response...
IMF – Evaluation of the Fund in the Run Up to                       the Crisis     The IMF evaluated the quality of its su...
Conclusions and Suggestions              Rapidly changing situations pose special challenges                 for evaluator...
Upcoming SlideShare
Loading in …5
×

Global Crisis Response Evaluations at the World Bank Group

1,219 views

Published on

This presentation describes lessons and recommendations from the evaluation of World Bank Group response to global financial crisis of 2009-2010.

  • Be the first to comment

  • Be the first to like this

Global Crisis Response Evaluations at the World Bank Group

  1. 1. Evaluation Approaches, Methods and Data in a Rapidly Changing Context: Global Crisis Response Evaluations at the World Bank Group Independent Evaluation Group World Bank / IFC / MIGA European Evaluation Society Helsinki, October 2012 Anjali Kumar, Andaleeb Alam and Ali Khadr IEG World Bank Group1
  2. 2. The Global Economic Crisis and the Challenge to WBG Rapid movement of economic events during the global crisis of 2008-2009  Some stock markets lost 50% of value in days  Global growth slowdown (3.9 % to -2.1% )  Unprecedented scale, contagion  Estimated 50-64 million more poor people Motivating strong response from WBG  117 countries received Bank loans during 2009- 10; tripling of IBRD lending to $100m  17 received crisis support during 1993–2003 Call for ‘real time’ evaluation of response2
  3. 3. Crisis Evaluation: Nature of the ChallengeDaily data show the decline in value of the UK Complexity of CrisisFTSE index from 5,600 to 3,800 in weeks Response Evaluations Events are sudden, unexpected and fast-evolving in character Evaluative lessons are needed in a compressed time frame Conventional evaluation techniques that use information based on a sequential logical framework are of limited use3 Uncertainty prevails
  4. 4. Crisis Evaluation: Nature of the Challenge Scope of the present paper Questions asked in crisis evaluation Data usable to answer such questions Use of high frequency data Crisis evaluations at other IFIs / MDBs Are such economic events becoming more frequent? ‘Black swan’ events may increase because they are probabilistic outliers Increased need for corresponding4 evaluative techniques
  5. 5. Crisis Evaluation –Nature of Evaluative Questions Evaluative Questions – complex evaluations under uncertainty  Skills demonstrated in dealing with complexity and change Speed of response; reflecting information on events as they occur Identification of risks involved Results framework, learning: Quality at entry5
  6. 6. Crisis Evaluation: IEG’s Phased ResponseMonthly data on emerging markets showed IEG prepared a series of ‘real time’declines in industrial production and credit evaluations on WBG Crisisgrowth not revealed in annual averages Response 12% 1.5 Review of WBG Response to Past Crises 10% (2009) 1.0 Industrial Production (sd units) Private Credit Growth (q-o-q) 17 Country Case studies – no current crisis data 8% 0.5 Phase I Evaluation of WBG Crisis 6% Response (2010) 4% 0.0 Real time evaluation focused on volume, speed, and early results – selective use of data to 2% -0.5 benchmark Bank actions 0% -1.0 Phase II Evaluation of WBG Crisis Response (2011) Jan-08 Jan-10 Jan-07 Jan-09 Jul-07 Jul-08 Jul-09 Jul-10 Oct-07 Oct-08 Oct-09 Oct-10 Apr-07 Apr-08 Apr-10 Apr-09 Used high frequency data to analyze patterns of Private Credit Growth (q-o-q) Industrial Production (sd units) Bank support relative to crisis incidence6
  7. 7. 1. Retrospective Evaluation - WB Response toPast Crises – 17 case studies Evaluative questions based on preceding principles Focus on: Scale and modality of the lending response Impact on the Bank’s own balance sheet Partnerships with other multilateral agencies Content of the WB crisis-response operations Macroeconomic trade and financial and fiscal Use of Data The crisis retrospective did not look at data on the current crisis, It did trace data on previous crises- –extent of World Bank’s response relative to its baseline lending, –and relative to other international financial institutions;7 –time taken to return to normal lending patterns.
  8. 8. 2. WBG Response to the Global Crisis: Phase I Evaluation Private Capital Flows during the crisis: Evaluative questions : January 2007–December 2010 Design aspects –readiness, relevance, poverty focus, Implementation aspects – Speed, internal organization, instruments, monitoring and evaluation Early Outcomes and Prospects- Meeting Objectives, additionality, debt sustainability Data Use: Annual economic data used to trace effects of crisis on Bank borrower / client countries8
  9. 9. 2. WBG Response to the Global Crisis Phase I Evaluation Alternative Scenarios – Poverty in Phase I Example: Developing Countries in 2015 and 2020 Data Use in Evaluation under Uncertainty: Scenario analysis Ten year projections of poverty impact of the crisis Based on a series of recent data on GDP growth, trade, capital flows and demand assumptions9
  10. 10. WBG Response to the Global Crisis -Phase II Scope of the Evaluation Utilization of high frequency data across a range of economic variables  To track multiple dimensions of stress  These multiple measures of stress were only partially correlated.  Analysed relative to other IFIs and MDBs Multidimensionality of crisis  Exchange rate and foreign exchange reserve stress  Financial Stress, including market, credit and banking system  Social indicators – unemployment, private consumption; and  Fiscal deficit and public debt as a percentage of GDP)10
  11. 11. WB Response to the Global Crisis Phase II Evaluation Methods Use of High Frequency data to measure multiple dimensions of stress that were not necessarily correlated11
  12. 12. Evaluating the Lending Response relative to stress Incremental Lending Relative to Levels of Simple and Composite stress measures Crisis: World Bank  Simple - changes in GDP;  Composite - principal factor analysis 700%  Peak to trough, period average 600% Illustrative bands ranking countries according to stress in diagram 500%  Are compared with the distribution of % Increase in Lending 400% incremental lending. 300% Underlying these data are regressions  That measure crisis intensity and 200% incremental Bank response on a 100% continuous basis Caveats and limitations of the analysis  Country demand, country performance, 0% other IFIs Growth Decline  Controls can however be introduced for some of these factors12
  13. 13. Volume and Distribution of Support - Other IFIs / MDBs Incremental Lending Relative to Levels of Increased WB lending patterns were also Crisis: Comparisons with Other Donors compared with other MDBs 12 1.2  Correlations of each institution’s incremental support to crisis intensity Increase in Other Major Donor Lending in client countries 10 1 Increase in World Bank Lending 8 0.8  Based on data obtained from relevant IFIs as % of GDP as % of GDP  Comparisons undertaken at multiple 6 0.6 4 0.4 levels  Response to changes in GDP alone, to 2 0.2 composite overall stress measures, and compared to specific measures of stress 0 0 Do not analyse other MDBs’ overall Other Major Donors (incl. IMF) Other Major Donors (excl. IMF/EIB/EU) response  Comparisons limited to countries that World Bank13 were common borrowers
  14. 14. Crisis Evaluation and High Frequency Data - EBRD EBRD’s crisis response evaluation established a three phase typology Based on two high frequency data series on interest rates  European Central Bank deposit facility rate  ‘FED funds rate14
  15. 15. Crisis Evaluation and High Frequency Data - EBRD EBRD’s evaluation then traces crisis response in each phase  Eg: evolution of EBRD loan pricing in response to market signals15
  16. 16. IMF – Evaluation of the Fund in the Run Up to the Crisis The IMF evaluated the quality of its surveillance in the run up to the crisis  Evolution of crisis traced also using high frequency data:  EMBI Global Spread (Left)  TED spread (right)16
  17. 17. Conclusions and Suggestions Rapidly changing situations pose special challenges for evaluators  Different questions  That can use high speed data for their response  In order to give real time feedback Financial and some macroeconomic data exist  In multiple dimensions and high frequencies  However more systematic monthly data on GDP, fiscal deficits etc are still to be produced More difficult are data on social dimensions  Eg integrated global databases on employment or consumption Greater global cooperation on such data would be a benefit  To policy makers and to evaluators17

×