Beyond Scaling Up: Approaches to Evaluation


Published on

This presentation was given at the 'Beyond Scaling Up: Pathways to Universal Access' workshop which was held at the Institute of Development Studies, Brighton on the 24-25 May, 2010. This event was co-sponsored by the Future Health Systems Research Programme Consortium and the STEPS Centre. Lucas presented on approaches to evaluation.

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Beyond Scaling Up: Approaches to Evaluation

  1. 1. Beyond Scaling Up Approaches to evaluation in complex and rapidly changing situations
  2. 2. Evidence based policy and practice <ul><li>Long history of clinical trials to decide: ‘what works?’ or even ‘what works best?’ </li></ul><ul><li>Experience with health-systems research much less encouraging – interesting individual studies but limited accumulated knowledge </li></ul><ul><li>Simple medical interventions involve complex social interventions </li></ul><ul><ul><li>The ‘implementation is the intervention’ </li></ul></ul><ul><ul><li>Context (including historical context – path dependency) </li></ul></ul><ul><ul><li>Outcomes dependent on detailed processes & pathways </li></ul></ul>
  3. 3. Who needs what evidence? <ul><li>Multiple stakeholders: implementers, providers, regulators, civil society, community groups, local governments, national governments, funding agencies </li></ul><ul><li>Need very different types of ‘evidence’ to address a variety of concerns: attribution, accountability, justification, learning, management, etc. </li></ul><ul><li>Many stakeholders may have limited understanding as to what types of information would be most useful to them . </li></ul>
  4. 4. Monitoring, learning and evaluation <ul><li>MLE process needs to: </li></ul><ul><ul><li>Understand priorities of different stakeholders </li></ul></ul><ul><ul><li>Make objectives transparent </li></ul></ul><ul><ul><li>Be aware of trade-offs between objectives </li></ul></ul><ul><ul><li>Make a realistic assessment of what can be achieved and allocate resources accordingly </li></ul></ul><ul><li>No single ‘right way’ to undertake MLE – depends on balancing multiple objectives. </li></ul><ul><li>Three main paradigms: experimental, theories of change, realistic </li></ul>
  5. 5. Experimental <ul><li>Randomised controlled trials provide the only scientific approach to the evaluation of an intervention. </li></ul><ul><li>Sometimes not possible to undertake RCTs, in which case we should approximate the RCT benchmark as closely as possible. </li></ul><ul><li>This requires the careful construction of a counterfactual which involves at least (1) the identification of intervention and control groups and (2) baseline and evaluation studies. </li></ul><ul><li>Recent debates consider value of: </li></ul><ul><ul><li>‘ explanatory’ RCTS – designed to test specific hypothesis in a highly controlled context </li></ul></ul><ul><ul><li>‘ practical’ RCTS – designed to identify interventions that might produce beneficial outcomes in practice </li></ul></ul>
  6. 6. Theories of Change <ul><li>Randomised controlled trials would be the best option in an ideal world but it will usually be impossible to employ them in practice. </li></ul><ul><li>With or without RCTs, it is essential that we focus not simply on whether an intervention succeeded or failed but why. </li></ul><ul><li>By devoting sufficient resources to developing a shared understanding as to how an intervention is intended to work, we can design monitoring systems that will allow us to evaluate how and to what extent observed outcomes can be plausible attributed to the intervention – implementation theory . </li></ul><ul><li>The use of a RCT or well constructed counterfactual may be very helpful in this task. </li></ul>
  7. 7. Realistic <ul><li>The interventions with which we are concerned are multi-faceted and highly malleable, with multiple components that will be modified by local contexts (path dependency, emergent behaviour) </li></ul><ul><li>Those who participate in the implementation process, including intended beneficiaries and intervention managers, will have a diverse range of characteristics, perceptions and attitudes that shape their responses to the various intervention components. </li></ul><ul><li>Placebo effects – positive or negative responses to the fact of the intervention – will typically be large and uncontrollable. </li></ul><ul><li>The external environment within which the intervention is made will also inevitably give rise to unforeseen effects that vary over the intervention period. </li></ul>
  8. 8. Realistic <ul><li>Given this reality, it is essentially irrational to seek for evidence that given types of intervention ‘work’. </li></ul><ul><li>The use of RCT or ‘quasi-experimental’ designs is a waste of time and resources in terms of systematic learning. </li></ul><ul><li>What can be usefully achieved is to seek out the most interesting specific elements of the intervention and explore how they have performed in relation to specific groups of individuals. </li></ul><ul><li>This will allow the construction of programme theories that genuinely advance our knowledge and can be used in the modification of the current intervention or design of the next. </li></ul>
  9. 9. Some questions <ul><li>Are the expectations of MLE too high (given typical level of resources)? </li></ul><ul><li>How should limited resources be allocated to different objectives: e.g. attribution versus learning? </li></ul><ul><li>How should objectives link to methodology (do multiple objectives imply multiple methodologies)? </li></ul><ul><li>Evaluator as expert or facilitator? </li></ul><ul><li>How do we balance the ‘objectivity’ of external evaluators against the in-depth knowledge of internal evaluators? </li></ul>