The needle in the haystack: external evaluaitons (Saul Guerro, ACF)
The Needle in the HaystackThe role of external evaluations in identifying and promoting good practices within NGOs Saul Guerrero Senior Evaluations, Learning & Accountability Advisor Action Against Hunger (ACF)
A simple story that may be of interest… First, because organisations may relate to the issues Second, because our experiences may help others in tackling these issuesThird, because this is the story of what happens when a consultant/evaluator infiltrates an organisation
In 2010No finalised/approved policy or guidelines for evaluations The impact of individual evaluations on organisational performance and practice was inconsistentEvaluations were something that field teams largely saw as someone else’s desire to “audit” a programme
But then an opportunity… ACF International Evaluation Policy & GuidelineA framework that could help us change ideas about, and contributions of, external evaluations
Diversity Terms of Reference ACF Evaluation Policy & Guidelines Best DACPractices Criteria Predictability
DAC CriteriaCriteria Rating Rationale (1 low, 5 high) 1 2 3 4 5ImpactSustainabilityCoherenceCoverageRelevance/AppropriatenessEffectivenessEfficiency But why the need to quantify?
For field programmesTrack changes from year to year (or evaluation to evaluation) For Headquarters Track changes from programme to programme BUT everyone understood that the numbers meant only so much. What matters is their comparative value, and the rationale behind them.
Today we can use this to show that programmes that arerelevant/appropriate, for example, are those that are good at recognising and adapting to changing needs Link it to actual programmes Promote it as an evidence-based value
Best Practices Not just a “feel good” exerciseA way of using an often underutilised dimension of workingwith external evaluators: their experience as an actual filter to distil the “special stuff” from a programme“The evaluation is expected to provide one (1) key exampleof Best Practice from the project/programme. This exampleshould relate to the technical area of intervention, either in terms of processes or systems, and should be potentially applicable to other contexts where ACF operates”
What is this “Best Practice”? What makes it good/different/stand out? How can we do more of it? Not just an external idea/processGiving external evaluators a designated role in triggering a much wider learning process
What is it? How does it work? Best Practices from all 46 missionsHow to move forward? Expanded goes into Learning Review Best Practice resulting in feeds into Field HQs Technical Techies Technical/Policy Publications Debate shared with which influence Best Practice which provide Field PracticesIdentified by evaluations
It is not perfect, and there is much to do still…But after two years, we are starting to see this process come to life So, what have we learned from it all?
Sort out the haystack first Organisations can change perceptions about the role/usefulness of evaluations – but you need to make ityour goal, and you need to work top-down and bottom-up Enlist help in searchingYou can cater for the learning interests of programmes, and the learning needs of an organisation– but evaluations need to strike a balance between diversity and predictability of information Do something with the needle! External evaluations can be given a key role in triggeringwider organisational learning – but its must be seen as part of a larger, more participatory process that involves different people at different levels