Your SlideShare is downloading. ×
Beyond Experiments: General Equilibrium Simulation Methods for Impact Evaluation
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Beyond Experiments: General Equilibrium Simulation Methods for Impact Evaluation


Published on

“Beyond Experiments: General Equilibrium Simulation Methods for Impact Evaluation" presented by Xinshen Diao, IFPRI and Edward Taylor, University of California at the ReSAKSS-Asia Conference, Nov …

“Beyond Experiments: General Equilibrium Simulation Methods for Impact Evaluation" presented by Xinshen Diao, IFPRI and Edward Taylor, University of California at the ReSAKSS-Asia Conference, Nov 14-16, 2011, in Kathmandu, Nepal.

Published in: Education

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Beyond Experiments: General Equilibrium Simulation Methods for Impact Evaluation Xinshen Diao International Food Policy Research Institute J. Edward Taylor University of California, Davis Katmandu, Nepal, November 16-18, 2011
  • 2. Outline  Why simulation approach?  Why general equilibrium?  What is a simulation approach?  What is a local-economy general equilibrium simulation model?  Conclusions: simulation approach and its implication to FtF
  • 3. Why simulation approach? We need to look beyond experiments when… Planning a large scale intervention (such as FtF) often requires an ex-ante assessment of its potential impact (no pilot rollout is possible) Treatment and control groups are impracticable – Can’t randomize over large number of units – Investments (e.g., irrigation and rural road) to target certain areas instead of individuals Economic impacts are indirect; higher-level effects (e.g., poverty reduction and economic growth) We want to know “Why” & “if” there are impacts Multiple inputs and inter-related outcomes Impacts are heterogeneous, likely winners and losers
  • 4. Why general equilibrium? - Externalities and linkages  What experimentalists call “externalities” or “control-group contamination” …GE modelers call “linkages”  Linkages transmit impacts from the treatment group to others in the same location  Can also create higher-level impacts outside the targeted locations
  • 5. Transfer Policy
  • 6. Transfer Policy Treatment households adjust
  • 7. External Linkages Transfer Policy …affecting other households
  • 8. External Linkages Transfer Policy …which adjust
  • 9. External Linkages Transfer Policy …affecting still other households
  • 10. What is a simulation approach? Thinking about a flight simulator  Flight simulator contains good model of mechanics and aerodynamics – If not, don’t fly with that pilot!  If we have a good model of how the local economy works, we can use it to – Simulate impacts of project, policy changes – Do an local-economy GE cost-benefit analysis – Estimate the distribution of impacts, winners and losers, whom to compensate/provide adjustment assistance – Experiment with project designs w/ specific goals  Ex-post: We can use experimental results to see whether the plane really flew
  • 11. What is a local-economy simulation model? Recipe for simulation-based project evaluation  Understand the project or policy to be simulated – Elements of the project: E.g., cash transfer or input subsidy? Who’s the target (i.e., treatment group)?  Understand the actors and the economic system – How is the treatment group connected with others in the zone of influence (ZOI) of the project? – How do we model their behavior? – Sketch out a social accounting matrix (SAM) for each household group and/or locality to reflect this  Inventory existing data needs and availability to construct SAMs – Baseline surveys fill data gaps (can modify pre-treatment surveys)  Build the simulator: construct SAMs, use them to calibrate a general- equilibrium (GE) model encompassing treatment and control groups  Do simulations to evaluate high-level impacts of intervention  Use the simulation results as inputs into CB or impact analysis, project design  Use experimental results for validation, recalibration of models
  • 12. Examples of a local economy model: Malawi case Challenges of this (like most) impact evaluation  Three transfer mechanisms – Input subsidy (IS) • Malawi Agricultural Inputs Subsidy Program (MAISP) – Cash transfer (CT) • Malawi’s Social Cash Transfer Scheme, ­SCTS – Farm gate market price support (MPS) • Implemented historically Can’t do an experiment for each of them
  • 13. Challenges (continued)  Immediate indirect effects of transfers on the control group (linkages effect) – Experiments aren’t going to capture them  Heterogeneous treatment and control groups  Sensitivity of outcomes to market structures – E.g., will cash transfers create multiplier effects within households by loosening production constraints? – Ex-post experimental evidence can help us parameterize this in the simulation model
  • 14. Multiple goals of the analysis  To compare the effects of these three transfer mechanisms on incomes and welfare in rural areas – Including high-level effects, on non-beneficiary households  To assess differences in these effects across household groups and market scenarios – The structure of the economy shapes outcomes  To understand why different transfer mechanisms produce different outcomes
  • 15. Developing an economywide GE model in which…  A set of farm (and nonfarm) household models are defined  Each household model is representative of a group of households defined according to their eligibility for each transfer program  All these household models are embedded in an economywide GE model
  • 16. Data  Ideally, parameterize the model with data from a baseline (pre- project) survey  In this application, we had to rely on existing data… – IHS2 (Second Integrated Household Survey) • 2004, immediately preceding the first round of the MAISP – National agricultural production and consumption information available online from FAOSTAT • 2003, the last completed cropping season before the IHS2 was conducted  Constructing a social accounting matrix (SAM) for each household group from the data  Nest the households within a “meta-SAM” for the ZOI (in this case, the entire rural economy)  Includes market accounts that link together the household groups
  • 17. Simulations  Assumptions on market conditions matter 1. Perfect markets benchmark 2. With constrained input use 3. With unemployment 4. Combined 2 and 3  Under each type of market arrangements, simulating IS, MPS, CT separately at the given cost ($52 million)
  • 18. Simulation results at the household level (perfect market benchmark) (1) (2) (3) (4) (5) (6) Transfer Mechanism Ineligible, Non- farm households Ineligible, Small farms Ineligible, Large farms Eligible for CT (ultra-poor labor- constrained) Eligible for MAISP (poor small- holders) Eligible for both CT & MAISP Group's share of total households (%) 3 19 23 1 47 7 a) IS: Crop Inputs subsidies for eligible households Group’s share of transfer (%) - - 93.0 7.0 Welfare (CV), % change 0.80 0.00 -0.30 0.01 5.47 4.50 Household-level efficiency - - 0.69 0.78 b) MPS: Market Price Support for Maize Group’s share of transfer (%) 22.0 57.0 1.0 20.0 0.0 Welfare, % change -1.1 2.0 2.7 1.6 0.6 -1.9 Household-level efficiency 0.64 0.66 0.57 0.37 - c) CT: Cash Transfer to eligible households Group’s share of transfer (%) - 17.5 - 82.5 Welfare (CV), % change 0.0 0.0 0.0 50.8 0.0 69.7 Household-level efficiency - - - 1.00 - 1.00
  • 19. Total production effects and efficiency measure under alternative market conditions (a) (b) (c) (d) Perfect markets benchmark With constrained input use With unemployment With unemployment & constrained input use Production effects (% change in total agricultural output) Input Subsidy 4.0 2.3 13.4 5.0 MPS 1.0 -0.3 8.6 2.9 Cash transfer 0.0 0.8 0.0 2.0 Total transfer efficiency (welfare gain/transfer cost) Input Subsidy 0.66 0.60 2.59 1.59 MPS 0.57 0.04 2.29 1.30 Cash transfer 1.00 1.17 1.00 1.47 Input subsidy becomes most efficient when households face unemployment and liquidity constraints
  • 20. Which assumptions reflect reality?  Perfect markets benchmark seems to be overly optimistic  Effects of transfers depend on: – The elasticity of input supply – The responsiveness of wages to shifts in labor demand – The extent to which there are cash constraints on input demand  All are likely to vary across project settings
  • 21. Conclusions: Simulation approaches and FtF  Experiments have become the favored method of impact evaluation  Simulation methods will be increasingly important; and particularly important for FtF
  • 22. Advantages of experiments  Verifiability – Create random treatment and control groups – Simply compare averages of outcomes of interest to evaluate average effect of treatment on the treated
  • 23. Disadvantages  Experiments often are impracticable (cost, politics, ethics)  They almost never come out truly random (need for econometrics)  Control group contamination (due to GE linkages)  Difficulty comparing impacts of several different project designs  Non-structural: Generally don’t tell us why treatments have the impacts they do  GE feedbacks change impacts once programs are “ramped up”
  • 24. Simulation approaches  Designed to overcome these limitations of experiments  Ideal for – Capturing higher-level impacts – Comparing alternative mechanism designs – Understanding the “Why?” – Evaluating differences in project impacts across market environments  Can be implemented before projects
  • 25. The Simulation-experiment ideal  Simulations: Use to evaluate likely impacts of alternative project interventions ex-ante – Parameterize with data from baseline surveys  Carry out randomized experiment using most promising program designs  Use results of experiment ex-post to verify and (if needed) reparameterize simulation model  Use simulation model to provide a structural interpretation of experiment results (i.e., to answer the “Why?” question) – …and improve policy design
  • 26. Thank You