ICAR-IFPRI: Methodologies in Agricultural Extension Research capacity development program from September 21-24, 2015 in New Delhi

1. Revisiting and Other issues
Devesh Roy (September 22, 2015)
IFPRI-ICAR training

2. Using Monitoring Data
• Monitoring data -a critical resource in an IE.
• Lets the evaluator verify which participants received the program,
• how fast the program is expanding,
• how resources are being spent, and
• whether activities are being implemented as planned. This information is
critical to implementing the En, for example, to ensure that baseline data are
collected before the program is introduced and to verify the integrity of the
treatment and comparison groups.
• In addition, M can provide information on the cost of implementing
the program, which is also needed for cost-benefit analysis.

3. Evaluation question
• What is the impact or causal effect of the program on an outcome of
interest?

4. Setting up an evaluation: The steps (Gertler et
al)
• (i) establishing the type of question to be answered by the evaluation,
(ii) constructing a theory of change that outlines how the project is
supposed to achieve the intended results (iii) developing a results
chain, formulating hypotheses to be tested by the evaluation, and
selecting performance indicators.
• All of these steps are best taken at the outset of the program,
engaging a range of stakeholders from policy makers to program
managers, to forge a common vision of the program’s goals and how
they will be achieved. This engagement builds consensus regarding
the main questions to be answered and will strengthen links between
the evaluation, program implementation, and policy.

5. Theories of change
• A theory of change is a description of how an intervention is supposed to
deliver the desired results. It describes the causal logic of how and why a
particular project, program, or policy will reach its intended outcomes.
• A theory of change is a key underpinning of any impact evaluation, given
the cause-and-effect focus of the research.
• A theory of change can specify the research questions.
• The best time to develop a theory of change for a program is at the
beginning of the design process, when stakeholders can be brought
together to develop a common vision for the program, its goals, and the
path to achieving those goals.
• Stakeholders can then start program implementation from a common
understanding of the program, how it works, and its objectives.

6. Theories of change: The results chain
• A basic results chain maps the following elements
• Inputs: Resources at the disposal of the project, including staff and budget
Activities: Actions taken or work performed to convert inputs into outputs
• Outputs: The tangible goods and services that the project activities produce (They
are directly under the control of the implementing agency.)
• Outcomes: Results likely to be achieved once the benefi ciary population
• uses the project outputs (They are usually achieved in the short-to-medium
• term.)
• Final outcomes: The fi nal project goals (They can be infl uenced by multiple
• factors and are typically achieved over a longer period of time.)
• The results chain has three main parts:

7. Results
• Results: Intended results consist of the outcomes and final outcomes,
which are not under the direct control of the project and are
contingent on behavioral changes by program beneficiaries.

8. Selecting performance indicators (Gertler et
al 2010)
• SMART is the rule
• Specific: to measure the information required as closely as possible
• Measurable: to ensure that the information can be readily obtained
• Attributable: to ensure that each measure is linked to the project’s
efforts
• Realistic: to ensure that the data can be obtained in a timely fashion,
with reasonable frequency, and at reasonable cost
• Targeted: to the objective population.

9. Intent to treat versus treatment- easier to
understand with randomized assessment example
• Program offering- Less than full compliance
• Non compliance possible from both sides beneficiaries as well as non-
beneficiaries
• Under these circumstances, a straight comparison of the group originally
assigned to treatment with the group originally assigned to comparison will
yield the “intent to-treat” estimate (ITT).
• We will be comparing those whom we intended to treat (those assigned to
the treatment group) with those whom we intended not to treat (those
assigned to the comparison group).
• It is not unimportant since most policy makers can only offer a program
and cannot force the program on their target population

10. What about treatment effects?
• In getting the treatment effect requires correcting for the fact that
some of the units assigned to the treatment group did not actually
receive the treatment, or that some of the units assigned to the
comparison group actually did receive it.
• In other words, we want to estimate the impact of the program on
those to whom treatment was offered and who actually enrolled. This
is the “treatment-on the-treated” estimate (TOT).

11. Example (Gertler et al 2010)
• Enroll-if-offered. These are the individuals who comply with their
assignment.
• If they are assigned to the treatment group (offered the program), they
take it up, or enroll; if they are assigned to the comparison group (not
offered the program), they do not enroll.
• Never. These are the individuals that never enroll in or take up the
program, even if they are assigned to the treatment group. They are
noncompliers in the treatment group.
• Always. These are the individuals who will find a way to enroll in the
program or take it up, even if they are assigned to the comparison group.
They are noncompliers in the comparison group.

12. Non-compliance issue: continued (Gertler et
al 2010)

14. ITT and ATT (Gertler et al 2010)
• If the average income (Y) for the treatment group is $110, and the average income for
the comparison group is $70, then the ITT is $40.
• Second, we need to recover the treatment-on-the-treated estimate (TOT) from the
intention-to-treat estimate. To do that, we will need to identify where the $40 difference
came from. Let us proceed by elimination. First, we know that the difference cannot be
caused by any differences between the Nevers in the treatment and comparison groups.
The reason is that the Nevers never enroll in the program, so that for them, it makes no
difference whether they are in the treatment group or in the comparison group.
Second,we know that the $40 difference cannot be caused by differences between the
Always people in the treatment and comparison groups because the Always people
always enroll in the program. For them, too, it makes no difference whether they are in
the treatment group or the comparison group.
• Thus, the difference in outcomes between the two groups must necessarily come from
the effect of the program on the only group affected by their assignment to treatment or
comparison, that is, the Enroll-if-offered group.

15. ITT and ATT
• Suppose that a doctor tells everyone in a treatment group to go home
and exercise for an hour per day and tell the control group nothing.
• After a month, if he evaluates the difference in their blood pressure.
• If just compare the difference in mean blood pressures between the
two groups, get the ITT.
• This doesn't tell the causal effect of exercise on blood pressure, but
the causal effect of telling people to exercise on blood pressure. We
would presume that this estimate would be smaller than the
treatment effect of exercise per se, as only a (small!) fraction of
people in the treatment group would follow the advice.

16. Retrieving ATT (Gertler et al 2010)
• We know that the entire impact of $40 came from a difference in
enrollment for the 80 percent of the units in our sample who are
Enroll-if-offered. Now if 80 percent of the units are responsible for an
average impact of $40 for the entire group offered treatment, then
the impact on those 80 percent of Enroll-if-offered must be 40/0.8, or
$50. Put another way, the impact of the program for the Enroll-if-
offered is $50, but when this impact is spread across the entire group
offered treatment, the average effect is watered down by the percent
that was noncompliant with the original randomized assignment.
• ATT=40/0.8=50