1. Can we demonstrate the difference
that Norwegian Aid makes?
Evaluation of results measurement and how this can be
improved
Date: 4th April 2014
In association with the Chr. Michelsens Institute

2. Overview
Purpose of the assignment
Methodology
Key findings
Recommendations

3. Purpose of the assignment
• Norad’s Evaluation Department (EVAL) found ‘none of the
evaluations and studies commissioned by EVAL and
finalised in 2011 could report sufficiently on results at the
level of outcomes or impact’
• Why? What are the factors and dynamics ?
• This report:
– contributes to learning
– provide evidence-based recommendations for improvements
– responds to government’s demand for a strengthened focus on
results

4. Structure of the results measurement system
External evaluations commissioned and
managed by EVAL using independent
consultants and through collaboration with
peer agencies . EVAL provides external
oversight of the overall aid administration.
Grants managed by desk officers within
Norad, MFA and embassies with optional
technical guidance from Norad and external
advisers. The grant management system
generates the bulk of results data.
Grant
progress
reports
Grant
final reports
Grant
reviews &
evaluations
System-wide,
thematic &
country
evaluations
Results data generated through the grant
management system
Provides
oversight on
results
Internal grant management
system
The Evaluation Department
(EVAL)
Synthesis
studies
Joint
evaluations

5. Hypotheses – testing the system
1. Internal policies, systems and procedures … provide appropriate and
comprehensive guidance
2. Staff receive appropriate training and technical advice/support….
3. The policies, systems and procedures … are being correctly and
adequately implemented
4. The planning, commissioning and quality assurance of evaluations
places an emphasis on measuring results
5. Evaluators have adequate competencies to effectively measure results
and find/use evidence

6. Methodology
Assessment of grant management system
• Desk-based document assessment
• Assessment of courses, training and capacity building of staff
• Review of grant management processes including quality assurance
• Comparative review of approaches at the World Bank, DFID and Danida
• Review of current practice through a sample of recent grants
• Interviews and a survey of staff to explore current practice and opinions
Assessment of EVAL planning, commissioning and management of
evaluations
• An assessment of the results-focus in a sample of recent evaluation
reports
• An assessment of evaluators’ competencies in the sample

7. Analytical framework
Policies &
systems
Organisational
culture and
structures
Staff
Capacities
Results
measurement
Rules and
Regulations
Guidelines and
checklists
Standard
procedures
Knowledge and
skills
Training
Attitudes
Incentives
Sanctions
Leadership

8. KEY FINDINGS

9. Hypothesis 1 - Internal policies, systems and procedures to ensure evaluability
and results documentation in the grant management process provide
appropriate and comprehensive guidance
• There are gaps in the minimum standards required from grant
applicants about how they plan to measure results which make it
difficult for programme officers to make an informed judgement on
the quality of their approach and to ensure evaluability
• Development of grant scheme rules and a new Grant Management
Manual could have improved consistency and coherence around
results measurement. But the non-mandatory use of templates
means there are no clear standards
• There is an absence of a clear strategic approach to and guidance on
reviews and evaluations which means their use within the grant
management cycle is ad hoc and their quality variable

10. Hypothesis 2 - Staff receive appropriate training and technical advice/support
to effectively ensure evaluability and results documentation as part of the
grant management process
• Training on results measurement is generally of a good quality, with
high staff satisfaction. However:
– Gaps in content
– Short duration courses
– Low attendance levels
– Not a useful career move
– Little other practical guidance
• The institutional arrangements for technical support and QA on
result measurement are not set up in a way that ensures consistent
quality across the system
– Technical advice only mandatory in a minority of grant schemes and is little used
– Sector specialists often don’t have skills in results measurement.

11. Hypothesis 3 - The policies, systems and procedures that are in place (to ensure
interventions are evaluable and robust results data are being collected) are
being correctly and adequately implemented
• The current rules and guidance on results measurement in grant
management are not being adequately implemented
• Staff are under pressure and in many instances find themselves
unable to devote the attention necessary to improve specification
of results for a number of reasons:
– Lack of time
– Lack of prioritisation by leadership
– Lack of incentives and sanctions
• Concern to follow the post- Paris harmonisation agenda and use
partners’ own systems appears to have held back constructive dialogue
about results measurement

12. Hypothesis 4 - The planning, commissioning and quality assurance of
evaluations places an emphasis on measuring results
• Evaluations designed by EVAL often do not put sufficient emphasis
on gathering evidence about results, particularly during the
planning and commissioning phase
• EVAL does not have a systematic active management towards
ensuring evaluation consultants remain focused on measuring
results
• Evaluations directed towards outcomes and impacts do not have
the necessary design features to ensure an outcome/impact
assessment is delivered

13. Hypothesis 5 - Evaluators have adequate competencies to effectively measure
results and find/use evidence
• Necessary competencies are not expressed clearly in the ToRs
• A majority of the consultants have substantial experience with
evaluation, have formal qualifications in the discipline and have a
solid foundation in the application of core evaluation approaches
and tools. However there are large gaps between competency and
application
• Few evaluators have adequate competencies to measure results at
the outcome and impact levels, but evaluators have been assessed
as competent for the ToRs as tendered
– The evaluators lack skills and expertise in the application of more sophisticated evaluation
methodologies. Contracts are awarded through a competitive tendering process and
competency was accepted.

14. Conclusions for each hypothesis
Hypothesis 1 - Internal policies, systems and procedures to ensure
evaluability and results documentation in the grant management
process provide appropriate and comprehensive guidance
Rejected
Hypothesis 2 - Staff receive appropriate training and technical
advice/support to effectively ensure evaluability and results
documentation as part of the grant management process
Rejected
Hypothesis 3 - The policies, systems and procedures that are in place
(to ensure interventions are evaluable and robust results data are being
collected) are being correctly and adequately implemented
Rejected
Hypothesis 4 - The planning, commissioning and quality assurance of
evaluations places an emphasis on measuring results
Rejected
Hypothesis 5 - Evaluators have adequate competencies to effectively
measure results and find/use evidence
Unproved

15. KEY RECOMMENDATIONS

16. Two types of recommendations for improving results
measurement and evaluability in the aid administration
• Technical – to help resolve the shortcomings and
gaps in grant policies and procedures
– Target: Grant Management Unit in the Ministry of
Foreign Affairs, Norad's Department for Quality
Assurance, EVAL
• Structural – to address the challenges that are
more structural in nature raised in the evaluation
– Target: Norad and MFA senior management

17. Technical recommendations
Strengthen the quality of results measurement in the grant
management process
• Require partners to outline in greater detail how they plan to measure results.
• Develop guidance for staff on how to put results into practice and support partners in developing
effective measurement systems.
• Develop standard Quality Assurance checklists for staff to use
• Require partners to use the standard templates that have been developed
Strengthen the quality and use of reviews and evaluations at
grant level
• More strategic approach to the use of evaluations and reviews when planning grants that is
grounded in review of evidence (and its quality)
Strengthen the training support provided to staff on results
measurement
• Develop a more comprehensive training programme to support staff capacity in results
measurement
• An online resource hub should be developed that provides staff access to examples of good
practice in results measurement and pools sector-specific resources

18. EVAL recommendations
Designing an evaluation
• Tighten the design specifications for evaluations
• Keep evaluation questions focused
• Require evaluators to clearly describe the programme logic of the intervention
being evaluated
• Be more specific in ToRs about the required consultants’ skills
Managing an evaluation
• Monitor the progress of evaluations more closely
Conducting impact evaluations
• Develop a clear process for deciding and managing impact evaluations
• Ensure the appropriate competencies exist both among the EVAL staff managing
the evaluation and the consultants
• Ensure the specification of methodologies, data requirements and competencies
of evaluators are in line

19. Technical changes are not enough
• Two contrasting strategies are suggested to tackle
structural problems:
– by concentrating expertise or
– by broadening it
• The two approaches represent different strategies but
are not necessarily mutually exclusive. Both (or
elements of both) could be taken forward in tandem
• Cutting across each approach are a number of common
recommendations. They provide the foundations of
both approaches and we deal with them first.

20. Leadership : Incentives : Consistency
• More visible support at senior management level for
results measurement.
• Improve staff incentives for measuring results and
ensuring evaluability.
• The requirements on technical assistance and Quality
Assurance should be harmonised across all grant
scheme rules.
• Establish a consistent basis, such as size of grant, for
which technical assistance and Quality Assurance of
partners’ results frameworks, reporting and evaluation
plans is mandatory.

21. Structural recommendations –
Approach 1 (concentrating expertise)
• Resource AMOR to provide more comprehensive support
to all eligible grants on their results measurement
frameworks, and evaluation plans and reports at approval
and completion
– Need for additional resources
– Could be contracted out or managed as a task force
– Could provide information to feed into the Annual Results
Report
• This approach concentrates actions to improve quality. It
would bring a high degree of consistency, but has
substantial implications for staffing in the short time span
necessary to process grants

22. Structural recommendations –
Approach 2 (broadening expertise)
• Build a cadre of staff specialised in results measurement and
evaluation
– Small number of staff receive intensive training in evaluation and
results measurement
– Would bring advice and training closer to the ground
– More flexible and informal
– Sector specialisation possible
– Incentivise with career rewards
• This approach is designed more to work within current practices
and build through progressive change rather than radical
change. It broadens the approach by developing staff capacity
and working more closely with partners, which fits with local
ownership and capacity building of partners’ systems

23. ‘Can we demonstrate the difference that
Norwegian aid makes?’
There are some elements of good foundations for
better results measurement, but current arrangements
lack the strength of leadership, depth of guidance and
coherence of procedures necessary for effective
evaluation of Norwegian aid

24. Thank you for listening
Any Questions?