This document summarizes a study that evaluated the impact of providing working capital loans to farmer groups in Uganda. The study used a randomized controlled trial where some farmer groups received loans to make partial cash payments to farmers upon delivery of crops, allowing "cash on delivery." The goal was to evaluate if this increased the proportion of crops sold through the groups and the prices received by farmers by reducing the costs of selling to the groups. Data was collected through surveys before and after the intervention to assess the impact of the working capital loans on marketing outcomes for farmers.

1. Impact evaluation An application to farmers groups in Uganda
Improving market access of farmer groups in
Uganda: evaluating the role of working capital
Ruth Vargas Hill and Eduardo Maruyama
May 9, 2012

2. Impact evaluation An application to farmers groups in Uganda
Outline
Impact evaluation
Introduction
RCTs
An application to farmers groups in Uganda
Introduction
Implementation
Results
Concluding remarks

3. Impact evaluation An application to farmers groups in Uganda
Why evaluate?
• Evaluating interventions (policies or programs) helps:
• Understand the actual rather than the anticipated effects of
programs.
• Determine how to design new programs.
• Determine the most cost-effective approach to achieve a
desired goal.

4. Impact evaluation An application to farmers groups in Uganda
Estimating impact: introduction
• When we conduct impact evaluation we assess how a program
affects the well-being or welfare of individuals, households or
communities:
• Profitability of agricultural production
• Increased income or consumption (or other measures of
welfare) of rural households
• Poverty levels or growth rates at the community level

5. Impact evaluation An application to farmers groups in Uganda
Impact evaluation versus other M&E tools
• Impact evaluation is different from other M&E tools in that it
focuses on discerning the impact of the program from all
other confounding effects.
• The focus of impact evaluation is providing evidence of the
causal link between an intervention and an outcome.
• This is why impact evaluation is a powerful too, but also what
makes it difficult to implement in practice.

6. Impact evaluation An application to farmers groups in Uganda
Impact evaluation versus other M&E tools
DIFFICULTY OF
Low High
SHOWING CAUSALITY
Inputs Outputs Outcomes Impacts
Example: A program of providing advice on a new technology to farmers
Visits by
Increased
extensions
Knowledge of Use of the yields, higher
agents,
the new new farm profits,
physical inputs
technology technology improved
(such as
consumption
seeds)

7. Impact evaluation An application to farmers groups in Uganda
Essential component: counterfactual
• Difficulty is determining what would have happened to the
individuals or communities of interest in absence of the
project.
• We are interested in the difference in an outcome for an
individual with and without the intervention.
• Problem: can only observe people in one state of the world at
one time
• The key component to an impact evaluation is to construct a
suitable comparison group to proxy for the counterfactual.

8. Impact evaluation An application to farmers groups in Uganda
Before and after comparisons
• Why not collect data on individuals before and after
intervention (the Reflexive)? Difference in income, etc, would
be due to project
• Problem: many things change over time, including the project
• The country is growing and profits are rising. Is this due to the
program or would have occurred in absence of program?
• This is particularly a problem for agricultural interventions:
many factors affect yield (weather, availability of inputs) and
prices in a given year.

9. Impact evaluation An application to farmers groups in Uganda
Comparison groups
• Instead of using before/after comparisons, we need to use
comparison groups to proxy for the counterfactual
• Two core problems in finding suitable groups:
• Programs are targeted
• Recipients receive intervention for particular reason
• Participation is voluntary
• Individuals who participate differ in observable and
unobservable ways (selection bias)
• Hence, a comparison of participants and an arbitrary group of
non-participants can lead to misleading or incorrect results

10. Impact evaluation An application to farmers groups in Uganda
Randomizing to create a true comparison group
• We need a comparison group that is as identical in observable
and unobservable dimensions as possible, to those receiving
the program, and a comparison group that will not receive
spillover benefits.
• Number of techniques:
• Randomized control trials (the gold standard)
• Careful matching techniques: IV, propensity score matching,
regression discontinuity design

11. Impact evaluation An application to farmers groups in Uganda
Randomized Control Trials (RCTs)
• In RCTs, participation in a policy (or usually eligibility to
participate in a policy) is randomly assigned.
• This is done to ensure that the only difference between those
in and out of an intervention, is their participation, and as a
result any difference between participants and
non-participants can be attributed to the program alone.
• Because participation (treatment) is randomized, the
non-treatment outcomes between those that are not treated
and those that are treated is equal.

12. Impact evaluation An application to farmers groups in Uganda
Households or groups of households

13. Impact evaluation An application to farmers groups in Uganda
Households or groups of households
C
T C
C
C
T T C
C T C
T
C
T C
T
C T
C C C
C
T T
T C T

14. Impact evaluation An application to farmers groups in Uganda
Does randomization create a true comparison group?
• We can test that they are equal by collecting data on the two
groups before the intervention and checking that the average
characteristics of the two groups are the same.
• For the treatment and control groups to be statistically equal
you need a large number of each. Cannot have one treated
household and one control household.
• Means that you cannot use this method to answer questions
about country policy changes (e.g. fiscal policy changes).
• There are stragegies that can be used to ensure that the
treatment and control groups are equal (e.g. stratification).

15. Impact evaluation An application to farmers groups in Uganda
How do we estimate impact by randomizing?
• Identify the outcome we are interested in (e.g. yields, amount
of output marketed, price received)
• Estimate the average of the outcome in the treatment group.
• Estimate the average of the outcome in the control group.
• Calculate the difference of these averages and test to see if
the two averages are significantly different from each other.
• Average Treatment Effect
• Note: it is just differences in the AVERAGE outcome that are
estimated.

16. Impact evaluation An application to farmers groups in Uganda
Challenges to estimating impact
• Sometimes the effect of the program is small.
• Or there are many other factors affecting the outcome of
interest that it is hard to see if a difference is statistically
different between two groups.
• We try and control for this in two ways:
• Include a large number of households in treatment and control.
This increases our power to detect a small effect.
• Collect data on characteristics of the household that may
influence the outcome variable at baseline (including the
pre-intervention outcome of interest)

17. Impact evaluation An application to farmers groups in Uganda
Other challenges to estimating impact
• Are we sure that the intervention had no impact on the
control group? Are there no spillover effects? (E.g. on prices)
• Was there any attrition as a result of the program that means
we miss capturing some of the impact? For example did
people migrate as a result of the program? If so, we will miss
capturing the effect of the program on these people.
• We randomized to avoid selection bias, but some of it still
may remain:
• Did everyone in the treatment group participate as expected?
• Did anyone in the control group participate even if they were
not meant to?

18. Impact evaluation An application to farmers groups in Uganda
Selection bias
Not in
evaluation
Target
Population
Treatment Participants
group
Evaluation Random No-Shows
Sample Assignment
Non-
Control group Participants
Cross-overs
36

19. Impact evaluation An application to farmers groups in Uganda
Handling selection bias
• Intent to treat (ITT):
• Average impact of program in practice: treats all noncompliars
as treated, and treats all crossovers as remaining in the control
• Problem: power is reduced by noncompliance and does not
provide an idea of what the average impact of the program on
the treated is.
• Treatment on the treated (ToT):
• Instruments for take-up with assignment: gives an idea of the
average impact of the program for a specific group

20. Impact evaluation An application to farmers groups in Uganda
Summary of advantages and disadvantages
• Powerful method to identify causal impact of a policy or
program.
• Careful design is needed to ensure you are able to detect
changes
• Can be expensive: baseline and follow-up, a large number of
participants (especially if it is to be representative)
• Only valid in some circumstances: randomization over a
number of units needs to be possible.
• Provides information on the average outcome.
• Internally valid, repetition and a theory of change needed to
make predictions from results (external validity).

21. Impact evaluation An application to farmers groups in Uganda
External validity
• Tells us whether something worked in a specific context,
understanding whether an intervention would work again in a
different setting, external validity, is very difficult to know.
• Great benefit of few assumptions, comes with great cost
”narrowness of scope” (Cartwright 2007).
• Any external validity involves some assumptions about the
project working in different conditions.
• Yet replicating a project is almost impossible, replicating
triggers of mechanisms that produce the change is often more
possible.
• Result: we need to repeat impact evaluations and also have a
theory of change to know what will work in the future, i.e. to
really learn.
• But better than an approach that is not internally valid.
Predictions cannot be made from this either.

22. Impact evaluation An application to farmers groups in Uganda
Other concerns in learning from results
• General equilibrium effects.
• Corruption in implementing a large scale.
• Capacity to implement at a large scale.
• Overlap between new environment and old (example of
medicine)
• Not automatic to go from experiments to learning and policy
advice.

23. Impact evaluation An application to farmers groups in Uganda
Another approach
• We need a theory of change that guides us in going from
one-off impact evaluation to general lessons. This means
make assumptions and, ideally, use experiments to test and
refine these assumptions.
• When designing impact evaluation for this purpose, it often
looks quite different:
• Theory of change influences the design of the impact
evaluation.
• Often identifying the differential impact of different treatments,
rather than the impact of one treatment against baseline.

24. Impact evaluation An application to farmers groups in Uganda
Outline
Impact evaluation
Introduction
RCTs
An application to farmers groups in Uganda
Introduction
Implementation
Results
Concluding remarks

25. Impact evaluation An application to farmers groups in Uganda
Introduction
• Smallholder agriculture in Sub-Saharan Africa is largely
exposed to pervasive market failures, translating into missed
opportunities and sub-optimal economic behavior.
• These failures are often rooted in the importance of
economies of scale in procuring inputs and marketing produce.
• By engaging in markets collectively through a farmers group,
smallholders can overcome economies of scale.
• Despite the renewed interest from governments and donor
agencies in farmers groups as a means to overcome these
market failures, evidence shows that they have so far had
limited success.

26. Impact evaluation An application to farmers groups in Uganda
Ugandan context
• The majority of Ugandan farmers sell their (unprocessed)
produce at harvest time to itinerant traders at the farm-gate.
• Survey of farmers groups engaged in some form of output
marketing revealed that:
• Farmers get a higher price when they sell collectively.
• Yet few farmers sell through the marketing group of which
they are a member (only 47% make sales through group)
• Farmers are less likely to sell collectively when they are
liquidity constrained and in need of emergency money.
• Groups that offer cash on delivery of produce (rather than
payment some days later) have a higher proportion of
members selling through the group.

27. Impact evaluation An application to farmers groups in Uganda
Key impact question
• Would providing working capital loans to farmers groups so
that they can provide cash on delivery, improve marketing
outcomes for farmers?
• We cannot infer this from the baseline data: good groups may
be better at collective sales and better able to access finance
which allows payment on delivery.
• We would like to compare groups of similar quality and see if
working capital loans increase sales amongst those that
received them.

28. Impact evaluation An application to farmers groups in Uganda
Testing a theory of change
• Farmer groups can offer higher prices but because of the
waiting times involved in receiving payment, farmers find it
costly to sell though the group.
• Farmers are liquidity constrained and often sell coffee to meet
urgent financial needs, so even small delays in payment can be
problematic.
• Waiting for payment involves a high level of trust in the ability
of the group to market and transparency. There is a risk if the
groups cannot be trusted.
• Enabling groups to make payment on delivery through a
working capital loan will reduce the cost of selling through the
group.
• More farmers will sell through the group and receive higher
prices as a result.

29. Impact evaluation An application to farmers groups in Uganda
The impact of working capital credit
• Randomized provision of working capital credit to farmers
groups that had already been engaged in output marketing:
• Provide selected groups with a fund to make partial cash
payments to farmers upon delivery of produce. Once the group
makes a sale the fund is replenished and farmers are given the
remaining balance. “Cash on Delivery” (CoD)
• Assess the impact of this credit on the proportion of produce
sold through the group and on the price farmers received.
• Understand why this worked?
• Did this work for farmers likely to face liquidity constraints, or
only in groups where trust was already high?
• Implement an intervention on improved transparency to
randomly selected groups to improve trust in some groups. Is
the working capital intervention just as effective in those
groups with the transparency intervention?
• Information on Sales (IoS): SMS system to provide members
with specific information about transactions made by the
group (final sale price, fees deducted, etc.), plus reinforced
training on book-keeping.

30. Impact evaluation An application to farmers groups in Uganda
Coffee/maize group marketing structure
• Farmers groups (“PO”s for producer organizations) are
typically grouped under associations (DCs for district
committees).
• The PO handles bulking and coordination of transport with
members at the village level.
• The DC take care of collection and in some cases value
addition to the next stage of marketing.
• In most cases, a service organization offers support to DCs
and POs through lobbying, access to extension and additional
marketing services.

31. Impact evaluation An application to farmers groups in Uganda
Implementation
• The study was carried out in 9 DCs marketing coffee and
maize, containing 165 POs under them.
• March 2010, Baseline survey:
• A 3-tiered survey which collected detailed information on DCs,
POs, and member households.
• Full roster of members for each PO, and a complete household
survey for at least 2 members of each group.
• November 2010 September 2011, implemented intervention
in randomly selected groups:
• Provided working capital credit to randomly selected POs.
• Provided SMS information on deliveries to randomly selected
POs.
• October 2011, Follow-up survey.
• Collected detailed information on POs and member households.
• Collected administrative data from the DC records to obtain
more reliable delivery data.

32. Impact evaluation An application to farmers groups in Uganda
Randomization strategy
• We randomized the interventions at the PO level stratifying
the sample by DC, since the sample size is not large enough
at the DC level and the risk of spill-overs is too high at the
household level.
• POs in each DC are randomly assigned into 4 groups: (1)
CoD, (2) IoS, (3) CoD + IoS, and (4) none.
• The fund for the CoD was managed by the DC, and vouchers
were given to treated POs so their members could request
immediate partial payments for output deliveries.
• For the IoS intervention, a DC staff member was selected to
send the messages to key farmers in the treated POs.

33. Impact evaluation An application to farmers groups in Uganda
Are control and treatment groups equal?
Control CoD IoS Both
(mean)
Members 24.256 2.194 4.558 1.597
(3.106)∗∗∗ (4.365) (4.289) (4.339)
Years since foundation 4.400 1.039 0.460 1.014
(0.526)∗∗∗ (0.739) (0.731) (0.739)
Marketing services 0.825 -0.020 -0.081 -0.093
(0.066)∗∗∗ (0.093) (0.092) (0.093)
Output bulked (kgs.) 854.025 -240.708 -192.862 -325.440
(236.818)∗∗∗ (332.863) (329.018) (332.863)
Female leader 0.250 -0.006 -0.064 -0.030
(0.067)∗∗∗ (0.094) (0.093) (0.094)
Leader’s age 52.200 -4.639 -2.153 0.190
(1.874)∗∗∗ (2.634)∗ (2.604) (2.634)
Leader’s schooling 8.025 0.073 0.208 -0.562
(0.460)∗∗∗ (0.647) (0.639) (0.647)
POs 40 41 43 41

34. Impact evaluation An application to farmers groups in Uganda
• Implementing the interventions represented a major challenge:
1. The POs in our study are spread over many regions in the
country.
2. Implementation needed to be done by a 3rd party, to avoid
service organizations and DCs contaminating the PO-level
randomization strategy.
3. In order to avoid undesired heterogeneity in implementation,
training, and monitoring of the interventions, a single
implementing agency was favored over several regional
organizations.
4. Training and distribution of vouchers within the PO was
delegated to PO leaders in some DCs.
• Our own monitoring activities as well as the follow-up survey
indicate implementation was problematic.
• Some cross-over and no-shows for CoD intervention
• Overall implementation of IoS intervention.

35. Impact evaluation An application to farmers groups in Uganda
Empirical strategy
McKenzie (2011) shows that using baseline data on the outcome
variable of interest, allows more power to detect impact.
Therefore, for our analysis we estimate:
Yi,1 = α + γj Di,j + θYi,0 + εi,1
j

36. Impact evaluation An application to farmers groups in Uganda
Results
Table 2.1: Impact of interventions on produce
deliveries
PO Household
Kgs. P(Delivery) Kgs.
CoD only 747.826 0.186 162.700
(325.294)∗∗ (0.079)∗∗ (88.400)∗
IoS only 355.764 0.089 62.660
(320.386) (0.078) (87.060)
Both -584.566 0.101 122.000
(455.073) (0.077) (86.280)
Observations 165 244 243
R2 0.422 0.269 0.084

37. Impact evaluation An application to farmers groups in Uganda
Results
Table 2.2: Impact of selling through PO
on transaction features
Price Days
between
sale and
payment
Sold through PO 0.858 -6.540
(instrumented) (0.477)∗ (21.170)
Observations 193 192
R2 0.704 0.210

38. Impact evaluation An application to farmers groups in Uganda
Concluding remarks
• Despite implementation problems, the CoD intervention has a
significant impact on group marketing.
• CoD increases the probability a household will sell through the
group, how much each household will sell, and the total
amount sold by the group.
• By encouraging farmers to sell through the group, CoD has an
effect on increasing the price they receive.
•