Presentation at "Impact Evaluation for Financial Inclusion" (January 2013) CGAP and the UK Department for International Development (DFID) convened over 70 funders, practitioners, and researchers for a workshop on impact evaluation for financial inclusion in January 2013. Co-hosted by DFID in London, the workshop was an opportunity for participants to engage with leading researchers on the latest research methods of impact evaluation and to discuss other areas on the impact evaluation agenda.

1. Acting on Evidence
The case of microfinance
Stefan Dercon
DFID(UK)

2. ns
Overview
1. Stating the Problem: Using Evidence to Inform Action
2. The Compelling Case
3. The Hard Core Case
4. The Religious Case
5. The Keynesian Case
With illustrations from microcredit and microfinance
1

3. ns
Stating the Problem
• Development business:
– driven by a passionate identification of need
– a commitment to spend cash and time sensibly on achieving
change
• The business in practice
– Those controlling the purse want the best ideas to spend the
marginal £ on
– A commitment to changing lives, including via financial
inclusion
– Variety of outcome indicators (incomes, financial security, stability, broader
development outcomes)
– Financial inclusion: a theory of change that this leads for many if not all
people to positive outcomes in terms of the above
© 2010 The Actuarial Profession Ÿ www.actuaries.org.uk
2

4. ns
My question
• How do we and can we use evidence, not least from impact
evaluation, to inform these decisions?
– Starting from how decision making typically works
– Acknowledging the state and nature of evidence
• Evidence has always mattered – it is the way we produce and
use it, that has changed
• and let’s not forget: the importance of evidence was not
invented in the last 1990s.
© 2010 The Actuarial Profession Ÿ www.actuaries.org.uk
3

5. ns
The Compelling Case
Develop the strategic case and need:
• “get a compelling story” with good foundations in theory and
experience
• get examples to prove the point
© 2010 The Actuarial Profession Ÿ www.actuaries.org.uk
4

6. ns
The compelling case
Feature Microcredit Microinsurance
Roots in Poor strapped of cash, including to Poor face vast array of risks,
experience invest in their own activities; such as drought, health, etc.
and evidence Existing credit sources restricted, or Informal risk sharing and
of problem even exploitative. management imperfect.
Theory of the Good reasons for its failure and for Good reasons for its failure and
market substantial consequences, for substantial consequences
justifying to act justifying to act
Narrative • Poor so provide cash • Disaster looms around the
• Encourage them to invest corner
• But ensure they repay • Protect against disaster
• So they do more positive things
Development Responsibility, Enterpreneurship Security, Rights, Protection
narrative = positive, easy sell = more negative, difficult sell
- Large take-up not difficult - Take –up problematic
- Much traction - Limited traction
- Evidence on problem and - Evidence on problem and
market failure market failure 5

7. ns
The compelling case
Feature Microcredit Microinsurance
Roots in Poor strapped of cash, including to Poor face vast array of risks,
experience invest in their own activities; such as drought, health, etc.
and evidence Existing credit sources restricted, or Informal risk sharing and
of problem even exploitative. management imperfect.
Theory of the Good reasons for its failure and for Good reasons for its failure and
market substantial consequences, for substantial consequences
justifying to act justifying to act
Narrative • Poor so provide cash • Disaster looms around the
• Encourage them to invest corner
• But ensure they repay • Protect against disaster
• So they do more positive things
Development Responsibility, Enterpreneurship Security, Rights, Protection
narrative = positive, easy sell = more negative, difficult sell
- Large take-up not difficult - Take –up problematic
- Much traction - Limited traction
- Evidence on problem and - Evidence on problem and
market failure market failure 6

8. ns
The compelling case
Feature Microcredit Microinsurance
Roots in Poor strapped of cash, including to Poor face vast array of risks,
experience invest in their own activities; such as drought, health, etc.
and evidence Existing credit sources restricted, or Informal risk sharing and
of problem even exploitative. management imperfect.
Theory of the Good reasons for its failure and for Good reasons for its failure and
market substantial consequences, for substantial consequences
justifying to act justifying to act
Narrative • Poor so provide cash • Disaster looms around the
• Encourage them to invest corner
• But ensure they repay • Protect against disaster
• So they do more positive things
Development Responsibility, Enterpreneurship Security, Rights, Protection
narrative = positive, easy sell = more negative, difficult sell
- Large take-up not difficult - Take –up problematic
- Much traction - Limited traction
- Evidence on problem and - Evidence on problem and
market failure market failure 7

9. ns
The compelling case
Feature Microcredit Microinsurance
Roots in Poor strapped of cash, including to Poor face vast array of risks,
experience invest in their own activities; such as drought, health, etc.
and evidence Existing credit sources restricted, or Informal risk sharing and
of problem even exploitative. management imperfect.
Theory of the Good reasons for its failure and for Good reasons for its failure and
market substantial consequences, for substantial consequences
justifying to act justifying to act
Narrative • Poor so provide cash • Disaster looms around the
• Encourage them to invest corner
• But ensure they repay • Protect against disaster
• So they do more positive things
Development Responsibility, Enterpreneurship Security, Rights, Protection
narrative = positive, easy sell = more negative, difficult sell
Result - Large take-up not difficult - Take –up problematic
- Much traction - Limited traction
- Evidence on problem and - Evidence on problem and
market failure market failure 8

10. ns
The compelling case
Feature Microcredit Microinsurance
Roots in Poor strapped of cash, including to Poor face vast array of risks,
experience invest in their own activities; such as drought, health, etc.
and evidence Existing credit sources restricted, or Informal risk sharing and
of problem even exploitative. management imperfect.
Theory of the Good reasons for its failure and for Good reasons for its failure and
market substantial consequences, for substantial consequences
justifying to act justifying to act
Narrative • Poor so provide cash • Disaster looms around the
• Encourage them to invest corner
• But ensure they repay • Protect against disaster
• So they do more positive things
Development Responsibility, Enterpreneurship Security, Rights, Protection
narrative = positive, easy sell = more negative, difficult sell
Result - Large take-up not difficult - Take –up problematic
- Much traction - Limited traction
- Evidence on problem and - Evidence on problem and
market failure market failure 9

11. ns
The Compelling Case
Develop the strategic case and need:
• “get a compelling story” with good foundations in theory and
experience
• get examples to prove the point
It is the world we still largely live in…
© 2010 The Actuarial Profession Ÿ www.actuaries.org.uk
10

12. ns
Overview
1. Stating the Problem: Using Evidence to Inform Action
2. The Compelling Case
3. The Hard Core Case
4. The Religious Case
5. The Keynesian Case
With illustrations from microcredit and microfinance
11

13. ns
The Hard Core Case
• Let’s decide all spending on the basis what works
• Get some help from bright sharp minds to help us decide what
works: doing the full cost-benefit analysis
• Get high quality methods to get at outcomes and impacts of our
interventions
• Ensure clear cost information
• Find out what ‘has high return’ in development terms,
• Invest in highest return for marginal £
© 2010 The Actuarial Profession Ÿ www.actuaries.org.uk
12

14. ns
How feasible?
Pro:
• Forces us to use all evidence to get average and distribution of
impact
• Humility and possibility of transparency about what we know
and don’t know
• Avoids ‘bias’ of example as in compelling case
• And huge incentives to invest in more evidence to improve
© 2010 The Actuarial Profession Ÿ www.actuaries.org.uk
13

15. ns
How feasible?
• The problem with evidence
– the gold in the gold standard is at times impure
– “external validity” of the experiments
• More crucially:
– Evidence is vastly incomplete
– Cost curves and impact curves and scale?
– Context specificity and complexity of interventions
– Evidence is by its nature “with margin of error”
– While pressures of need remain
© 2010 The Actuarial Profession Ÿ www.actuaries.org.uk
14

16. ns
Example of microcredit
• Microcredit is probably one of the most researched and
evaluated interventions
• When focused on impact of the poorest, hard to find strong
evidence of positive impact, in systematic way
– E.g. systematic reviews by DFID – but still below 30 studies
to be included
– Roodman’s book – sourced more widely
– Or CGAP recent notes
• But each microcredit intervention is a bit different, different
political economy, economy, institutions.
• Hard to include in CBA
© 2010 The Actuarial Profession Ÿ www.actuaries.org.uk
15

17. ns
Example of microinsurance
• Evidence much thinner
• Many studies suggesting uptake problems
• But enough ‘not to act’? Wait for another 10 years?
© 2010 The Actuarial Profession Ÿ www.actuaries.org.uk
16

18. ns
Overview
1. Stating the Problem: Using Evidence to Inform Action
2. The Compelling Case
3. The Hard Core Case
4. The Religious Case
5. The Keynesian Case
With illustrations from microcredit and microfinance
17

19. ns
The religious case
• The evidence from impact evaluation is flawed
– Let’s attack the evidence
• Evidence of no evidence is enough to justify?
• Policy making by example
– Compelling case squared
– Comms dominating evidence or caution
• “millions of clients cannot be wrong”
– Average impact, distribution of impact
© 2010 The Actuarial Profession Ÿ www.actuaries.org.uk
18

20. ns
Overview
1. Stating the Problem: Using Evidence to Inform Action
2. The Compelling Case
3. The Hard Core Case
4. The Religious Case
5. The Keynesian Case
With illustrations from microcredit and microfinance
19

21. ns
The Keynesian case
“when the facts change, I change my mind” J M Keynes
• Decision making as “error-correction”: use evidence to change
course
• When to act?
– Negative role of evidence: if you fail to find any impact in
various attempts, become more cautious
– Note this is average impact – direct resources to find out who
may be negatively affected (and don’t just look for the
positive stories and examples)
• Always be open about learning
© 2010 The Actuarial Profession Ÿ www.actuaries.org.uk
20

22. ns
conclusions
• Always a business where “compelling case” matters
• we cannot get full information on what works as guide to
development –
– evidence will always be less than perfect due to its nature
and nature of problems we want to address
– so hard core view is not quite achievable
• But we should guard us to “religious” fanaticism against reason
and the misuse of example-based evidence.
• Move towards an ‘error-correction’ model of evidence?
© 2010 The Actuarial Profession Ÿ www.actuaries.org.uk
21