“Beyond Experiments: General Equilibrium Simulation Methods for Impact Evaluation" presented by Xinshen Diao, IFPRI and Edward Taylor, University of California at the ReSAKSS-Asia Conference, Nov 14-16, 2011, in Kathmandu, Nepal.

1. Beyond Experiments: General
Equilibrium Simulation Methods for
Impact Evaluation
Xinshen Diao
International Food Policy Research Institute
J. Edward Taylor
University of California, Davis
Katmandu, Nepal, November 16-18, 2011

2. Outline
 Why simulation approach?
 Why general equilibrium?
 What is a simulation approach?
 What is a local-economy general equilibrium
simulation model?
 Conclusions: simulation approach and its
implication to FtF

3. Why simulation approach?
We need to look beyond experiments when…
Planning a large scale intervention (such as FtF)
often requires an ex-ante assessment of its potential
impact (no pilot rollout is possible)
Treatment and control groups are impracticable
– Can’t randomize over large number of units
– Investments (e.g., irrigation and rural road) to target certain
areas instead of individuals
Economic impacts are indirect; higher-level effects
(e.g., poverty reduction and economic growth)
We want to know “Why” & “if” there are impacts
Multiple inputs and inter-related outcomes
Impacts are heterogeneous, likely winners and losers

4. Why general equilibrium?
- Externalities and linkages
 What experimentalists call “externalities” or
“control-group contamination”
…GE modelers call “linkages”
 Linkages transmit impacts from the
treatment group to others in the same
location
 Can also create higher-level impacts
outside the targeted locations

5. Transfer Policy

6. Transfer Policy
Treatment
households
adjust

7. External Linkages
Transfer Policy
…affecting
other
households

8. External Linkages
Transfer Policy
…which
adjust

9. External Linkages
Transfer Policy
…affecting
still other
households

10. What is a simulation approach?
Thinking about a flight simulator
 Flight simulator contains good model of
mechanics and aerodynamics
– If not, don’t fly with that pilot!
 If we have a good model of how the local
economy works, we can use it to
– Simulate impacts of project, policy changes
– Do an local-economy GE cost-benefit analysis
– Estimate the distribution of impacts, winners and
losers, whom to compensate/provide adjustment
assistance
– Experiment with project designs w/ specific goals
 Ex-post: We can use experimental results to
see whether the plane really flew

11. What is a local-economy simulation model?
Recipe for simulation-based project evaluation
 Understand the project or policy to be simulated
– Elements of the project: E.g., cash transfer or input subsidy? Who’s the target
(i.e., treatment group)?
 Understand the actors and the economic system
– How is the treatment group connected with others in the zone of influence (ZOI)
of the project?
– How do we model their behavior?
– Sketch out a social accounting matrix (SAM) for each household group and/or
locality to reflect this
 Inventory existing data needs and availability to construct SAMs
– Baseline surveys fill data gaps (can modify pre-treatment surveys)
 Build the simulator: construct SAMs, use them to calibrate a general-
equilibrium (GE) model encompassing treatment and control groups
 Do simulations to evaluate high-level impacts of intervention
 Use the simulation results as inputs into CB or impact analysis, project
design
 Use experimental results for validation, recalibration of models

12. Examples of a local economy model:
Malawi case
Challenges of this (like most) impact evaluation
 Three transfer mechanisms
– Input subsidy (IS)
• Malawi Agricultural Inputs Subsidy Program (MAISP)
– Cash transfer (CT)
• Malawi’s Social Cash Transfer Scheme, ­SCTS
– Farm gate market price support (MPS)
• Implemented historically
Can’t do an experiment for each of them

13. Challenges (continued)
 Immediate indirect effects of transfers on the control
group (linkages effect)
– Experiments aren’t going to capture them
 Heterogeneous treatment and control groups
 Sensitivity of outcomes to market structures
– E.g., will cash transfers create multiplier effects within
households by loosening production constraints?
– Ex-post experimental evidence can help us parameterize this
in the simulation model

14. Multiple goals of the analysis
 To compare the effects of these three
transfer mechanisms on incomes and
welfare in rural areas
– Including high-level effects, on non-beneficiary
households
 To assess differences in these effects across
household groups and market scenarios
– The structure of the economy shapes outcomes
 To understand why different transfer
mechanisms produce different outcomes

15. Developing an economywide GE model
in which…
 A set of farm (and nonfarm) household
models are defined
 Each household model is representative of a
group of households defined according to
their eligibility for each transfer program
 All these household models are embedded
in an economywide GE model

16. Data
 Ideally, parameterize the model with data from a baseline (pre-
project) survey
 In this application, we had to rely on existing data…
– IHS2 (Second Integrated Household Survey)
• 2004, immediately preceding the first round of the MAISP
– National agricultural production and consumption information available
online from FAOSTAT
• 2003, the last completed cropping season before the IHS2 was
conducted
 Constructing a social accounting matrix (SAM) for each
household group from the data
 Nest the households within a “meta-SAM” for the ZOI (in this
case, the entire rural economy)
 Includes market accounts that link together the household groups

17. Simulations
 Assumptions on market conditions matter
1. Perfect markets benchmark
2. With constrained input use
3. With unemployment
4. Combined 2 and 3
 Under each type of market arrangements,
simulating IS, MPS, CT separately at the
given cost ($52 million)

18. Simulation results at the household level (perfect market benchmark)
(1) (2) (3) (4) (5) (6)
Transfer Mechanism
Ineligible, Non-
farm households
Ineligible, Small
farms
Ineligible,
Large farms
Eligible for CT
(ultra-poor labor-
constrained)
Eligible for MAISP
(poor small-
holders)
Eligible for
both CT &
MAISP
Group's share of total
households (%)
3 19 23 1 47 7
a) IS: Crop Inputs subsidies for eligible households
Group’s share of transfer (%) - - 93.0 7.0
Welfare (CV), % change 0.80 0.00 -0.30 0.01 5.47 4.50
Household-level efficiency - - 0.69 0.78
b) MPS: Market Price Support for Maize
Group’s share of transfer (%) 22.0 57.0 1.0 20.0 0.0
Welfare, % change -1.1 2.0 2.7 1.6 0.6 -1.9
Household-level efficiency 0.64 0.66 0.57 0.37 -
c) CT: Cash Transfer to eligible households
Group’s share of transfer (%) - 17.5 - 82.5
Welfare (CV), % change 0.0 0.0 0.0 50.8 0.0 69.7
Household-level
efficiency - - - 1.00 - 1.00

19. Total production effects and efficiency measure under
alternative market conditions
(a) (b) (c) (d)
Perfect markets
benchmark
With constrained
input use
With
unemployment
With unemployment
& constrained input
use
Production effects (% change in total agricultural output)
Input Subsidy 4.0 2.3 13.4 5.0
MPS 1.0 -0.3 8.6 2.9
Cash transfer 0.0 0.8 0.0 2.0
Total transfer efficiency (welfare gain/transfer cost)
Input Subsidy 0.66 0.60 2.59 1.59
MPS 0.57 0.04 2.29 1.30
Cash transfer 1.00 1.17 1.00 1.47
Input subsidy becomes most efficient when households face
unemployment and liquidity constraints

20. Which assumptions reflect reality?
 Perfect markets benchmark seems to be
overly optimistic
 Effects of transfers depend on:
– The elasticity of input supply
– The responsiveness of wages to shifts in labor
demand
– The extent to which there are cash constraints
on input demand
 All are likely to vary across project settings

21. Conclusions: Simulation
approaches and FtF
 Experiments have become the favored
method of impact evaluation
 Simulation methods will be increasingly
important; and particularly important for
FtF

22. Advantages of experiments
 Verifiability
– Create random treatment and control groups
– Simply compare averages of outcomes of
interest to evaluate average effect of treatment
on the treated

23. Disadvantages
 Experiments often are impracticable (cost, politics,
ethics)
 They almost never come out truly random (need for
econometrics)
 Control group contamination (due to GE linkages)
 Difficulty comparing impacts of several different
project designs
 Non-structural: Generally don’t tell us why
treatments have the impacts they do
 GE feedbacks change impacts once programs are
“ramped up”

24. Simulation approaches
 Designed to overcome these limitations of
experiments
 Ideal for
– Capturing higher-level impacts
– Comparing alternative mechanism designs
– Understanding the “Why?”
– Evaluating differences in project impacts across
market environments
 Can be implemented before projects

25. The Simulation-experiment ideal
 Simulations: Use to evaluate likely impacts of
alternative project interventions ex-ante
– Parameterize with data from baseline surveys
 Carry out randomized experiment using most
promising program designs
 Use results of experiment ex-post to verify and (if
needed) reparameterize simulation model
 Use simulation model to provide a structural
interpretation of experiment results (i.e., to answer
the “Why?” question)
– …and improve policy design

26. Thank You