Presentation at CGAP event, "The Evolving Role of Funders and DFIs in Advancing Financial Inclusion" (14 Nov 2012)

1. The Evolving Role of Donors and DFIs in
Advancing Financial Inclusion
How do we measure progress
in market development?
Presentation to Funder Consultation, Luxembourg
Sukhwinder Arora
Oxford Policy Management
Wednesday 14 November, 2012

2. Presentation Structure
 The environment of the market system
 Challenges in measuring progress for market
development projects
 Difficult context
 Measuring progress can promote market development!
 Measurement example - OPM’s VFM work for FSD
Kenya
Monday 14 November, 2012 © 2011 Oxford Policy Management Ltd 2

3. The environment of the market system
Note the change in the direction of the arrows as they enter the market –
a refraction that represents unintended consequences!
THE FINANCIAL
SERVICES MARKET
DONORS
OTHER MARKETS
e.g. Agriculture, Industry
Trade, Infrastructure, Land,
Government services
For example, demand for financial
services includes demand from
enterprises involved in every sector.
Similar externalities apply to almost
Some every element in the market
interventions miss
the target entirely
Source: Robert Stone
Monday 14 November, 2012 © 2011 Oxford Policy Management Ltd 3

4. Challenges in measuring progress for market
development projects
A market development approach
 does not result in any obvious ‘hard’ outputs that can be measured
 necessarily involves influencing other players (but attributing ‘success’
to any set of players would be judgemental/ difficult)
 involves complex impact pathways
– operating directly and indirectly and at every level of the financial
sector (Rules, Supporting functions and Core market)
– not easily amenable to impact assessment (with or without; time
frames etc.) esp. tracking changes in rules of the game/ market
support function)
 facilitates a process of change (but cannot really supply any detailed ex
ante statement of the precise results that will eventually emerge from that
process)
Monday 14 November, 2012 © 2011 Oxford Policy Management Ltd 4

5. Difficult context
 Increased scrutiny of aid budgets
 Limited relevant market data in developing markets
 Market development is not a science
 Impacts of M4P programmes not easily quantifiable
 Logframe indicators can not be aggregated
 Mixed signals from donors to market actors
 Getting the balance right between
– Short term market development vs systemic change (aligning incentives for
market actors, policy makers and donors)
– donors taking credit for contribution/ successes and national stakeholders
driving the market development process
Independent Commission for Aid Effectiveness summed up the dilemma
 “What matters is not always easy to quantify: we recognise that evaluating the achievement of many
objectives within the UK aid programme – such as increasing accountability and reducing conflict and
fragility – may rely more heavily on qualitative than quantitative analysis.”
 “We do not believe it is right to shy away from the difficult-to-measure impacts, as these are often the ones
that are most effective in the long term.”
Monday 14 November, 2012 © 2011 Oxford Policy Management Ltd 5

6. Measuring progress can promote market
development!
Why measure progress?
 Prove Impact
 Improve Practice
 Assess Value for Money
The process of prioritisation, generation and dissemination of
market data and insights
is also a critical part of market facilitation and development
For Example:
 Facilitate informed debates amongst key stakeholders
 Establish common vision/ road map and how to monitor progress
 Signal what is important
 Create evidence for setting and tracking policy priorities
 Identify market opportunities
Monday 14 November, 2012 © 2011 Oxford Policy Management Ltd 6

7. FSD Kenya – value for money assessment
Part of knowledge management by FSD Kenya (one of the impact
assessment tools)
Recognise two types of outcomes/ outputs
a) Can be quantified with
– plausible assumptions and
– reasonable costs for data gathering and analysis
b) Difficult to quantify
Measurement of four key attributable outcomes
 Increased number and penetration of bank accounts
 Money transfers – lower costs and greater volumes
 Increased availability of bank and non-bank financial outlets
 The lower risks of SACCO failure
Monday 14 November, 2012 © 2011 Oxford Policy Management Ltd 7

8. FSD Kenya – value for money assessment
Key takeaways
 Not measuring everything can work in well conceived projects where some of the
benefits are large enough on their own to justify the full costs of that project
 The whole result can be transformational - much greater than sum of its parts -
but dis-aggregation is not easy e.g.
– TA to MFIs plus more enabling rules of the game;
– SACCO regulation and increasing competition
 Outcomes not always correlated with direct expenditure
– FSDK advice sometime more valuable then the much larger investments
– Right advice on demand can deliver magnified results
 Success always has many parents (failure is often the orphan)
 Key difference between ex-post (looking back) and ex-ante (looking ahead)
 Time frame for measuring results
 Measurement time and costs proportionate to the programme/ project costs (drive
towards VFM of the measurement itself)
 Explicitly share data sources and assumptions for scrutiny by other stakeholders
Monday 14 November, 2012 © 2011 Oxford Policy Management Ltd 8

9. Distinguishing between Direct and Final
Impacts
Intermediate data Final Impact: Big Picture
on Direct Impact Assessments
linked to project
FSD operational data Fin Access-type data plus
Performance and project- landscapes, special studies
Indicators/ specific special (not project-specific) etc.
Targets studies/surveys
Growth, Job Creation,
Poverty Reduction etc.
Monday 14 November, 2012 © 2011 Oxford Policy Management Ltd 9

10. Distinguishing between Direct and Final
Impacts
 Define the theory of change
 Measuring Direct Impact: start with the project/portfolio data from FSD and its
implementing partners – capture FSD’s own impacts (attribution is relatively
easier but limited tracking of final impacts)
 Measuring Final Impact: Better at measuring WHAT has changed but weaker
on its own to clarify WHY and HOW
 Need for triangulation of evidence (to confirm attribution)
 Gathering the evidence for multiple purposes
– proving impact,
– improving practice,
– assessing VFM and
– facilitating market development
 Tracking short term trends and not just long term post mortems
 Ensuring VFM of the measurement itself
Monday 14 November, 2012 © 2011 Oxford Policy Management Ltd 10

11. References
Robert Stone, Susan Johnson and Janet Hayes, FSD Kenya: Impact Assessment,
Final Report, January 2010 (OPM and University of Bath).
Sukhwinder Arora, Alan Roe and Robert Stone, Assessing Value for Money: The
Case of Donor Support to FSD Kenya, OPM Working Paper, June 2012
Sukhwinder Arora, Alan Roe and Robert Stone, Valuing the unquantifiable? How
VFM can be applied to complex projects, OPM Briefing Paper, October 2012
Sukhwinder Arora, Alan Roe and Robert Stone, FSD Impact Research and Value for
Money Assessment: A Suggested Approach, Report to FSD Kenya, August 2012
Available at
www.opml.co.uk
www.fsdkenya.org
Monday 14 November, 2012 © 2011 Oxford Policy Management Ltd 11

12. Thank you