SlideShare a Scribd company logo
1 of 31
Session: The role of the evaluator in results based evaluation
ney
                                                                     xpay  ers’ mo
                                                          oun d of ta e of value.”
      'Aid mus
                  t show re                   for every p e 100 penc
     value for               al       re that          strat
                   money'       “Ensu     , we  demon
                                we spend
                                           “Overseas aid to be
“Interna  tional                                                  spent through new
                nt secretary               system of payment
                                                                 by results”
 de velopme t
            to ac
 pledges                ure
             ly' to ens                                           ‘THE £3.7billion h
  'ruthless oney for                                              overseas aid could
                                                                                     ike in
            r  m
   value fo                                                       paid for 200 new sc
                                                                                       have
              rs”
   taxpaye                                                        kept 123,333 cops
                                                                                       hools,
      Corr                                                                            on the
           uptio                                                 beat or locked up
                                                                                     an extra
     accu         n                                              97,368 criminals,..
          s
    Glob ations hi                                               can reveal.’’
                                                                                    The Sun
         al Fu        t the
                 nd
                                                                 ‘Corru
                                                                         p
                                                                UK ai tion 'ruinin
                                                                       d bu d          g
                                                                              get'

                                                 ‘Value of aid to war-torn
                                                 countries ‘cannot easily
                                                 be assessed’
Evaluating impact, maximising Value for Money (VfM) and being transparent
Why Results?
 Assessment - measure whether achieved what was planned
 Accountability - communicate the impact of development
  assistance to Parliament, taxpayers, civil society, & recipients
 Decision making - make evidence-based decisions by knowing
  what works and what doesn’t


Global shift to accountability and decision-making driven by
 demand for results and evidence of results
Longer-term shift towards Results-Based Management
Everyone’s at it…
“The Government has made a priority of
achieving better results from development
assistance
Ulla Tørnæs, Danish Minister for Development Cooperation




“The question we ask today is not whether our government is
too big or too small, but whether it works …those of us who
manage the public’s dollars will be held to account - to
spend wisely, reform bad habits, and do our business in the
light of day – because only then can we restore the vital trust
between a people and their government.”
…and without being hard-hearted, we will also be hard-
headed, and make sure our aid money is directed at
those things which are quantifiable and
measureable…
… so we really know we are getting results.

David Cameron, Lagos, 19th July 2011
• …Strengthen our efforts to achieve concrete and
  sustainable results. This involves better managing for results,
  monitoring, evaluating and communicating progress; as well as
  scaling-up our support, strengthening national capacities and
  leveraging diverse resources and initiatives in support of
  development results.

•    We will partner to implement a monitor progress, evaluate
    impact, ensure sound, results-focused public sector
    management, and highlight strategic issues for policy decisions
Results are…
 A describable or measurable change in state that is derived
  from a cause and effect relationship. Results are the same
  as outcomes, and are further qualified as immediate,
  intermediate, or ultimate.

•Sustained improvement in development outcomes at the
country level (e.g., families lifted out of poverty…).

•Results: The output, outcome or impact (intended or
unintended, positive and/or negative) of a development
intervention
The Key Achievements
section at the start of this
report sets out a number of
aggregate results based on
DFID’s standard indicator
set…
Opportunities/Threats
 with the Results agenda
          Opportunities                             Threats


Opportunity for evaluators to ‘step   Indicator driven development:
up’ to the plate...now is our time    doing what is easy to measure…
(in development)!
Focus on theory of change to          Results agenda dominated by
ensure program results are possible   ‘output’ measurement

Invest in developing innovative       Focus only on positive results
ways to measure results               (comms messages)
Needs very good results               Failure to show results = failure of
measurement systems (M&e)             evaluators rather than programmes
                                      (or the fact that doing this is
                                      hard!...)
The ‘Value for Money’ debate...
“    our commitment to reaching 0.7% of
    national income on aid by 2013, and
    enshrining this in law, imposes an even
    greater duty on us, more than any other
    Department in Whitehall, to get value for
    money, to bear down on waste, and to ensure
    that aid secures 100 pence of value for every
    hard-earned British taxpayer's pound we
    spend.

Andrew Mitchell, UK Secretary of State for
  International Development. Oct 2010
Value for Money
What is Value for Money (VfM)?
                  Often VfM is
Resources /      understood by
                                         Service & Wider
Investment     comparing unit costs         Outcomes


   Money                                    Economic


   People     Inputs           Outputs        Social


Environment    Real VfM is achieved       Environmental
              by comparing outcomes
                  with investment
Opportunities/Threats
with VFM agenda
          Opportunities                            Threats
(Re) promote an evaluation agenda    VFM agenda does not ‘learn’ from
(with new language!)                 the experience of the evaluation
                                     sector (ignore counterfactual,
                                     attribution etc)
Help clarify and operationalise      Multiple interpretations of VFM
                                     (e.g. technical v allocative VF)

Create innovative and useful tools   Simple, high-level definitions, with
to help people use VFM as            little guidance on measurement;
management tool; recognises
different ‘values’
X-Whitehall initiative               Focus on the easy to measure
                                     (economy & efficiency) - perverse
                                     incentives
Transparency
W is aid transparency important?
 hy

  ACCOUNTABILITY                         OWNERSHIP


        EFFICIENCY                  FEEDBACK
                       RESULTS


                                 PREDICTABILITY
        CORRUPTION

                                          EFFECTIVENESS
                     DIVERSION
  Traceability
                                     BUDGET PLANNING
     COORDINATION

                           Demonstrating Results
Evaluating impact, maximising Value for Money (VfM) and being transparent
Evaluating impact, maximising Value for Money (VfM) and being transparent
International Aid Transparency
                Initiative (IATI)
1.   A standard for data on aid flows - NOT a new database
2.   A common exchange format
3.   Open data as a platform
4.   Multi-stakeholder initiative
Background: Aid Information Today




      Lots of people want data                       There is a lot of data out there




www.aidtransparency.net   BUT... Data is not accessible to users
… the worst of all worlds
How does IATI work ?
 IATI is a standard, not a database or a reporting system:
        1) agreement on what will be published;
        2) common definitions for publication;
        3) a common, open, electronic data format;
        4) a Framework for Implementation.
 The approach:
 • Aid providers publish their aid information (HQ or country office)
      •Open Aid Register
 • Link to the IATI Registry (an index that points users searching for aid
 information to its location)

 The underlying principle is publish once, use many times
 www.aidtransparency.net
IATI: publish once, use often
IATI “Beast and the Beauty”
Evaluating impact, maximising Value for Money (VfM) and being transparent
Evaluating impact, maximising Value for Money (VfM) and being transparent
Conclusions and… more questions?
 Individual agency data of little use; collaborative data better value

 Exchange data to improve performance

 Increase public support?
   Set up a collaborative platform to tell the story of UKAid (in HIV),
    visualising ‘results’ (focusing on financials now is probably a
    tactical mistake)
Opportunities/Threats
 with the Transparency agenda

            Opportunities                                  Threats
Evaluation findings can (finally) get the   transparency can still be "gamed“ (just
exposure ‘we’ would want!                   as targets can...)

The focus on getting the data out is only Transparency can result in ‘damage’ to
the first part of the job...we still need development agenda (e.g. negative
experts to analyse and interpret the      results used to justify political agendas)
data (that’s our job!)

Evaluations should be easier (since core    Being ‘transparent’ requires significant
data should be available...)                technical and financial resources.
                                            Opportunity cost and who's funding this?

Chance to push transparency around
null/failed or uncompleted evaluations
agenda (e.g. IE registries of IE)
Evaluating impact, maximising Value for Money (VfM) and being transparent
New opportunities

New skills needed (in all three areas)

New partnerships (data people for Transparency)

The dangers of over-emphasis on results (&
 measurement) driving evaluation out of fashion [being
 pushed back]

Need to be better communicators – all three agendas
 converge around communicating to wider audiences
Thank
 you!

More Related Content

Similar to Evaluating impact, maximising Value for Money (VfM) and being transparent

Does"grease money"speed up the wheels of commerce?
Does"grease money"speed up the wheels of commerce?Does"grease money"speed up the wheels of commerce?
Does"grease money"speed up the wheels of commerce?Andy Dabydeen
 
Accounting Theorist &Amp; Economist Recommending
Accounting Theorist &Amp; Economist RecommendingAccounting Theorist &Amp; Economist Recommending
Accounting Theorist &Amp; Economist RecommendingCrystal Nelson
 
Vitold Henisz slides: “Financial Returns to Stakeholder Capital: Tying Commun...
Vitold Henisz slides: “Financial Returns to Stakeholder Capital: Tying Commun...Vitold Henisz slides: “Financial Returns to Stakeholder Capital: Tying Commun...
Vitold Henisz slides: “Financial Returns to Stakeholder Capital: Tying Commun...Sustainable Brands
 
Book services marketing, lovelock
Book services marketing, lovelockBook services marketing, lovelock
Book services marketing, lovelockMayuresh Patil
 
Making sense of sustainable development
Making sense of sustainable developmentMaking sense of sustainable development
Making sense of sustainable developmentClive Bates
 
Five Tips for Maximizing Grants
Five Tips for Maximizing GrantsFive Tips for Maximizing Grants
Five Tips for Maximizing GrantsAbila
 
There Has Been Great Debate On Whether Science Should Be
There Has Been Great Debate On Whether Science Should BeThere Has Been Great Debate On Whether Science Should Be
There Has Been Great Debate On Whether Science Should BeErin Adams
 
MBA economic notes
MBA economic  notesMBA economic  notes
MBA economic noteskoko20115
 
first data annual reports 2004
first data annual reports 2004first data annual reports 2004
first data annual reports 2004finance29
 
TVA p301 CORE CONCEPTS for TRUE VALUE ACCOUNTING
TVA p301 CORE CONCEPTS for TRUE VALUE ACCOUNTING TVA p301 CORE CONCEPTS for TRUE VALUE ACCOUNTING
TVA p301 CORE CONCEPTS for TRUE VALUE ACCOUNTING Peter Burgess
 
TVA p3-01 CORE CONCEPTS for TRUE VALUE ACCOUNTING
TVA p3-01 CORE CONCEPTS for TRUE VALUE ACCOUNTINGTVA p3-01 CORE CONCEPTS for TRUE VALUE ACCOUNTING
TVA p3-01 CORE CONCEPTS for TRUE VALUE ACCOUNTINGPeter Burgess
 
Benefits Management – a fool’s errand?
Benefits Management – a fool’s errand?Benefits Management – a fool’s errand?
Benefits Management – a fool’s errand?grantpn
 
Seratio Proof-of-Impact Transaction Platform (11 Feb 2017) [v3.05]
Seratio Proof-of-Impact Transaction Platform (11 Feb 2017) [v3.05]Seratio Proof-of-Impact Transaction Platform (11 Feb 2017) [v3.05]
Seratio Proof-of-Impact Transaction Platform (11 Feb 2017) [v3.05]Professor Olinga Taeed PhD FIoD
 

Similar to Evaluating impact, maximising Value for Money (VfM) and being transparent (20)

Does"grease money"speed up the wheels of commerce?
Does"grease money"speed up the wheels of commerce?Does"grease money"speed up the wheels of commerce?
Does"grease money"speed up the wheels of commerce?
 
Accounting Theorist &Amp; Economist Recommending
Accounting Theorist &Amp; Economist RecommendingAccounting Theorist &Amp; Economist Recommending
Accounting Theorist &Amp; Economist Recommending
 
Vitold Henisz slides: “Financial Returns to Stakeholder Capital: Tying Commun...
Vitold Henisz slides: “Financial Returns to Stakeholder Capital: Tying Commun...Vitold Henisz slides: “Financial Returns to Stakeholder Capital: Tying Commun...
Vitold Henisz slides: “Financial Returns to Stakeholder Capital: Tying Commun...
 
Book services marketing, lovelock
Book services marketing, lovelockBook services marketing, lovelock
Book services marketing, lovelock
 
Publicsector Insight 2
Publicsector Insight 2Publicsector Insight 2
Publicsector Insight 2
 
Making sense of sustainable development
Making sense of sustainable developmentMaking sense of sustainable development
Making sense of sustainable development
 
Five Tips for Maximizing Grants
Five Tips for Maximizing GrantsFive Tips for Maximizing Grants
Five Tips for Maximizing Grants
 
There Has Been Great Debate On Whether Science Should Be
There Has Been Great Debate On Whether Science Should BeThere Has Been Great Debate On Whether Science Should Be
There Has Been Great Debate On Whether Science Should Be
 
MBA economic notes
MBA economic  notesMBA economic  notes
MBA economic notes
 
Acca Notes
Acca NotesAcca Notes
Acca Notes
 
Amtelecom Group
Amtelecom GroupAmtelecom Group
Amtelecom Group
 
Indonesia NAMA success story
Indonesia NAMA success storyIndonesia NAMA success story
Indonesia NAMA success story
 
first data annual reports 2004
first data annual reports 2004first data annual reports 2004
first data annual reports 2004
 
Basic Concepts In Economy
Basic Concepts In EconomyBasic Concepts In Economy
Basic Concepts In Economy
 
TVA p301 CORE CONCEPTS for TRUE VALUE ACCOUNTING
TVA p301 CORE CONCEPTS for TRUE VALUE ACCOUNTING TVA p301 CORE CONCEPTS for TRUE VALUE ACCOUNTING
TVA p301 CORE CONCEPTS for TRUE VALUE ACCOUNTING
 
Unit 1 .pptx
Unit 1 .pptxUnit 1 .pptx
Unit 1 .pptx
 
TVA p3-01 CORE CONCEPTS for TRUE VALUE ACCOUNTING
TVA p3-01 CORE CONCEPTS for TRUE VALUE ACCOUNTINGTVA p3-01 CORE CONCEPTS for TRUE VALUE ACCOUNTING
TVA p3-01 CORE CONCEPTS for TRUE VALUE ACCOUNTING
 
Benefits Management – a fool’s errand?
Benefits Management – a fool’s errand?Benefits Management – a fool’s errand?
Benefits Management – a fool’s errand?
 
Seratio Proof-of-Impact Transaction Platform (11 Feb 2017) [v3.05]
Seratio Proof-of-Impact Transaction Platform (11 Feb 2017) [v3.05]Seratio Proof-of-Impact Transaction Platform (11 Feb 2017) [v3.05]
Seratio Proof-of-Impact Transaction Platform (11 Feb 2017) [v3.05]
 
Noise, Numbers and Cut-through: What is the future role of financial reporting?
Noise, Numbers and Cut-through: What is the future role of financial reporting?Noise, Numbers and Cut-through: What is the future role of financial reporting?
Noise, Numbers and Cut-through: What is the future role of financial reporting?
 

Evaluating impact, maximising Value for Money (VfM) and being transparent

  • 1. Session: The role of the evaluator in results based evaluation
  • 2. ney xpay ers’ mo oun d of ta e of value.” 'Aid mus t show re for every p e 100 penc value for al re that strat money' “Ensu , we demon we spend “Overseas aid to be “Interna tional spent through new nt secretary system of payment by results” de velopme t to ac pledges ure ly' to ens ‘THE £3.7billion h 'ruthless oney for overseas aid could ike in r m value fo paid for 200 new sc have rs” taxpaye kept 123,333 cops hools, Corr on the uptio beat or locked up an extra accu n 97,368 criminals,.. s Glob ations hi can reveal.’’ The Sun al Fu t the nd ‘Corru p UK ai tion 'ruinin d bu d g get' ‘Value of aid to war-torn countries ‘cannot easily be assessed’
  • 4. Why Results?  Assessment - measure whether achieved what was planned  Accountability - communicate the impact of development assistance to Parliament, taxpayers, civil society, & recipients  Decision making - make evidence-based decisions by knowing what works and what doesn’t Global shift to accountability and decision-making driven by demand for results and evidence of results Longer-term shift towards Results-Based Management
  • 5. Everyone’s at it… “The Government has made a priority of achieving better results from development assistance Ulla Tørnæs, Danish Minister for Development Cooperation “The question we ask today is not whether our government is too big or too small, but whether it works …those of us who manage the public’s dollars will be held to account - to spend wisely, reform bad habits, and do our business in the light of day – because only then can we restore the vital trust between a people and their government.”
  • 6. …and without being hard-hearted, we will also be hard- headed, and make sure our aid money is directed at those things which are quantifiable and measureable… … so we really know we are getting results. David Cameron, Lagos, 19th July 2011
  • 7. • …Strengthen our efforts to achieve concrete and sustainable results. This involves better managing for results, monitoring, evaluating and communicating progress; as well as scaling-up our support, strengthening national capacities and leveraging diverse resources and initiatives in support of development results. • We will partner to implement a monitor progress, evaluate impact, ensure sound, results-focused public sector management, and highlight strategic issues for policy decisions
  • 8. Results are…  A describable or measurable change in state that is derived from a cause and effect relationship. Results are the same as outcomes, and are further qualified as immediate, intermediate, or ultimate. •Sustained improvement in development outcomes at the country level (e.g., families lifted out of poverty…). •Results: The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention
  • 9. The Key Achievements section at the start of this report sets out a number of aggregate results based on DFID’s standard indicator set…
  • 10. Opportunities/Threats with the Results agenda Opportunities Threats Opportunity for evaluators to ‘step Indicator driven development: up’ to the plate...now is our time doing what is easy to measure… (in development)! Focus on theory of change to Results agenda dominated by ensure program results are possible ‘output’ measurement Invest in developing innovative Focus only on positive results ways to measure results (comms messages) Needs very good results Failure to show results = failure of measurement systems (M&e) evaluators rather than programmes (or the fact that doing this is hard!...)
  • 11. The ‘Value for Money’ debate... “ our commitment to reaching 0.7% of national income on aid by 2013, and enshrining this in law, imposes an even greater duty on us, more than any other Department in Whitehall, to get value for money, to bear down on waste, and to ensure that aid secures 100 pence of value for every hard-earned British taxpayer's pound we spend. Andrew Mitchell, UK Secretary of State for International Development. Oct 2010
  • 13. What is Value for Money (VfM)? Often VfM is Resources / understood by Service & Wider Investment comparing unit costs Outcomes Money Economic People Inputs Outputs Social Environment Real VfM is achieved Environmental by comparing outcomes with investment
  • 14. Opportunities/Threats with VFM agenda Opportunities Threats (Re) promote an evaluation agenda VFM agenda does not ‘learn’ from (with new language!) the experience of the evaluation sector (ignore counterfactual, attribution etc) Help clarify and operationalise Multiple interpretations of VFM (e.g. technical v allocative VF) Create innovative and useful tools Simple, high-level definitions, with to help people use VFM as little guidance on measurement; management tool; recognises different ‘values’ X-Whitehall initiative Focus on the easy to measure (economy & efficiency) - perverse incentives
  • 16. W is aid transparency important? hy ACCOUNTABILITY OWNERSHIP EFFICIENCY FEEDBACK RESULTS PREDICTABILITY CORRUPTION EFFECTIVENESS DIVERSION Traceability BUDGET PLANNING COORDINATION Demonstrating Results
  • 19. International Aid Transparency Initiative (IATI) 1. A standard for data on aid flows - NOT a new database 2. A common exchange format 3. Open data as a platform 4. Multi-stakeholder initiative
  • 20. Background: Aid Information Today Lots of people want data There is a lot of data out there www.aidtransparency.net BUT... Data is not accessible to users
  • 21. … the worst of all worlds
  • 22. How does IATI work ? IATI is a standard, not a database or a reporting system: 1) agreement on what will be published; 2) common definitions for publication; 3) a common, open, electronic data format; 4) a Framework for Implementation. The approach: • Aid providers publish their aid information (HQ or country office) •Open Aid Register • Link to the IATI Registry (an index that points users searching for aid information to its location) The underlying principle is publish once, use many times www.aidtransparency.net
  • 23. IATI: publish once, use often
  • 24. IATI “Beast and the Beauty”
  • 27. Conclusions and… more questions?  Individual agency data of little use; collaborative data better value  Exchange data to improve performance  Increase public support?  Set up a collaborative platform to tell the story of UKAid (in HIV), visualising ‘results’ (focusing on financials now is probably a tactical mistake)
  • 28. Opportunities/Threats with the Transparency agenda Opportunities Threats Evaluation findings can (finally) get the transparency can still be "gamed“ (just exposure ‘we’ would want! as targets can...) The focus on getting the data out is only Transparency can result in ‘damage’ to the first part of the job...we still need development agenda (e.g. negative experts to analyse and interpret the results used to justify political agendas) data (that’s our job!) Evaluations should be easier (since core Being ‘transparent’ requires significant data should be available...) technical and financial resources. Opportunity cost and who's funding this? Chance to push transparency around null/failed or uncompleted evaluations agenda (e.g. IE registries of IE)
  • 30. New opportunities New skills needed (in all three areas) New partnerships (data people for Transparency) The dangers of over-emphasis on results (& measurement) driving evaluation out of fashion [being pushed back] Need to be better communicators – all three agendas converge around communicating to wider audiences

Editor's Notes

  1. http://publishwhatyoufund.org/files/OUTCOME_DOCUMENT_-_FINAL_EN1.pdf
  2. "Value for Money in International Development: Deconstructing Some Myths to Promote More Constructive Discussion" "This paper seeks to deconstruct some myths around "value for money" and promote a more constructive discussion about the relevance and limitations of the concept to development co-operation. It takes as its starting point the fact that few people disagree that we want development funds to be used as effectively as possible. So it is surprising that the concept of value for money in development has caused so many waves. This paper argues that once we untangle the confusion around what value for money really means it is clear that, as a concept, value for money is relevant to development co-operation. The challenge then is in applying this concept in a productive and pragmatic way, so that it can be a tool for development co-operation and not a straitjacket."
  3. Yet another way to look at VFM – with quality (adapted from New Economics Foundation slide)…