IDS Impact, Innovation and Learning Workshop March 2013: Day 2, Keynote 2 Patricia Rogers
Towards a Research Agenda for ImpactEvaluation of DevelopmentImpact, Innovation and Learning: Towards a Research andPractice Agenda for the Future conferenceInstitute of Development Studies, Brighton, UK26-27 March 2013Professor Patricia RogersRMIT University (Royal Melbourne Institute of Technology), AustraliaPatricia.Rogers@rmit.edu.au
Towards a research agendafor impact evaluation in development1. What it needs to cover2. What is needed to develop it3. Types of research needed4. Some burning research questions5. What is needed to develop and enact it
Filling in the mapof impact evaluation in development
1.What the research agendaneeds to coverTypes of development impact evaluationScale Individual evaluations, evaluations of multiple projects in aprogram, evaluation systemsPurpose Identify ‘best buys’, understand how to scale up and translateeffective programs, understand how to improve effectivenessQuestions Does it work? What works? What does it take for it to work? Whatworks for whom in what circumstances? Is it working?Users Donors, implementing agencies, policymakers, regionalassociations, communitiesDone by External evaluators, internal evaluators, managers and staff,communities
77MANAGEDEFINEFRAMEUNDERSTANDCAUSESSYNTHESIZEREPORT &SUPPORT USEDESCRIBECDC Evaluation Framework with BetterEvaluation components overlaid1.What the research agendaneeds to coverAspects of development impact evaluation
2.What is needed to developthe research agendaConsultations with Different parties involved in conducting, managing, usingand being affected by impact evaluationConsultations about Gaps in knowledge, issues, priorities, opportunitiesReview of Documentation and guidance for development impactevaluation – and impact evaluation generallyIssues and challenges in impact evaluation for developmentPrevious research into impact evaluationPotential methods, tools and approaches from other areas ofevaluation and researchPromising examples and recent innovations in developmentimpact evaluation
3.Types of research neededDocumenting practice Retrospectively, concurrently; good practice, problematicpractice; micro-interactions; decision-making heuristics;Positive deviance Learning by intended users from success casesTrials To address particular issues through varioustools/methods/strategiesTrials Of possible uses for particular tools/methods/strategiesLongitudinal studies Of the impact of impact evaluationSupporting interdisciplinary communities of practiceKnowledge translation To other contexts (sectors, organisations, roles)
OVERALL1. How do we do impact evaluation that actuallysupports development?2. How do we support all agents of development,including communities, to be reflective and empiricalabout the impact of their work?3. Why does so much development impact evaluationfail to be informed by what has been learned abouteffective evaluation?4. Some burning researchquestions
MANAGE an evaluation or evaluationsystem4. What are effective ways to support communities to havegenuine involvement in decision making aboutevaluations?5. How can an evaluation accommodate different ideas aboutwhat constitutes credible evidence among intendedusers?6. When should different strategies be used for developingan evaluation design (as part of the brief, as part of theproposal, as a separate project)?7. How can an evaluation design best accommodateemerging issues?8. How can organisations working in the same region shareinformation and data collection?8b What are options for funding public interest evaluationsnot under the control of the powerful?
DEFINE what is to be evaluated9. How can a theory of change/program theory effectivelyrepresent complicated aspects of interventions(multiple layers, components and partners andcomplex aspects (adaptability, emergence)?10.How can an organisation support projects to havelocally specific theories of change/program theory thatare still widely coherent?11.What are effective strategies for identifying possiblenegative impacts in advance?12.What investments and activities are the subjects ofevaluation? What examination is made of others?
FRAME the boundaries of the evaluation13. What are effective processes for developing goodKey Evaluation Questions – in terms of likely to beuseful and feasible?14. How can implicit values about results, processes anddistribution of benefits be made explicit?14b How can evaluations deal with ‘undiscussables’ – egactual unstated program objectives, unaddressedpoor performance?
DESCRIBE activities, outcomes,impacts, context15.When is purposeful sampling most appropriate, and howcan it be used validly and effectively?16.How can long-term results be followed up?17.How can unanticipated negative outcomes and impacts beidentified and addressed in data collection and reporting?18.How can reasonable intermediate outcomes be identifiedfor an evaluation that will end before impacts are evident?19.How can Big Data be used effectively for developmentimpact evaluation?19b What standard measures and indicators should be usedfor common outcomes and impacts of interest?
UNDERSTAND CAUSES of outcomes andimpacts20.What are credible methods and strategies for non-experimental causal inference in developmentimpact evaluations?
SYNTHESISE data from one or moreevaluations21.How can different values be accommodated indeveloping an overall evaluative judgement?22.How can systematic reviews which don’t excludematerials in terms of a hierarchy of evidence dealwith the large number of potentially relevant sources?
REPORT AND SUPPORT USE23.How can a development impact evaluation respond tosignificant changes in the intended users during thecourse of an evaluation?24.How can a development impact evaluation provide acoherent message without only focusing on averageeffects?25.What are effective strategies for supporting use ofdevelopment impact evaluation, especially in difficultsituations – eg fragile states, changing decisionmakers?25b What process and structures can be created toprotect those ‘speaking truth to power’?
5.What is needed to developand enact a research agendaLegitimacyResources, especially complementary to existing resourcesInterdisciplinary co-operation‘Creative abrasion’