Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ia Clinic Penang


Published on

Impact Assessment 1

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

Ia Clinic Penang

  1. 1. Improving ICT4D Impact Assessment Pan-all Partners’ Conference Penang, Malaysia June 13, 2009
  2. 2. What is Impact Assessment? <ul><li>“ Impact assessment is the systematic analysis of lasting or significant changes - positive or negative, intended or not – in people’s lives brought about by a given action or series of actions” – Chris Roche (1999) </li></ul><ul><li>IA Concerns with lasting changes in people’s lives </li></ul><ul><li>The changes takes place in a context where many factors can contribute to the changes </li></ul><ul><li>The changes can be both intended and unintended </li></ul><ul><li>Citations: </li></ul><ul><li>Roche, Chris. 1999. Impact Assessment for Development Agencies: Learning to Value Change. London: Oxfam </li></ul>
  3. 3. Impact Assessment versus Project Monitoring & Evaluation <ul><li>Monitoring is a systematic and continuous assessment of progress over time (Oakley, Pratt and Clayton, 1998) </li></ul><ul><li>Evaluation is a periodic assessment of the relevance of performance, efficiency, and outputs against stated objectives (Bakewell, 2003) </li></ul><ul><li>Citations: </li></ul><ul><li>Oakley, P., Pratt, B. and Clayton, A. (1998) Outcomes and Impact: Evaluating Change in Social Development, Oxford: INTRAC </li></ul><ul><li>Bakewell, O. (2003) Sharpening the Development Process: A Practical Guide to Monitoring and </li></ul><ul><li>Evaluation. Oxford: INTRAC. </li></ul>
  4. 4. Impact Assessment versus Project Monitoring & Evaluation Source: Souter, D. 2008. BCO Impact Assessment Study: Final Report. After Implementation Lasting and sustainable change Impact Assessment At the end of implementation Performance against objectives Evaluation During Implementation Ongoing Activities Monitoring Primary Timing Measuring Activity
  5. 5. The ICT4D Value Chain Precursors -Data systems -Legal -Institutional -Human -Technological -Leadership -Drivers/Demand Inputs -Money -Labour -Technology -Values and Motivations -Political support -Targets Intermediates / Deliverables -Telecentres -Libraries -Shared telephony -Other public access systems Outputs -New Communication Patterns -New Information & Decisions -New Actions & Transactions Outcomes -Financial & other quantitative benefits -Qualitative benefits -Disbenefits Development Impacts -Public goals (e.g. MDGs) Strategy Implementation Adoption Use Exogenous Factors READINESS UPTAKE IMPACT AVAILABILITY Sustainability Scalability Heeks, R. and A. Molla. 2008. Compendium on Impact Assessment of ICT-for-Development Projects.
  6. 6. What Makes Impact Assessment Difficult?
  7. 7. Challenges Related Impact Measurements <ul><li>Complexity of change </li></ul><ul><li>Contextual challenge </li></ul><ul><li>Challenge of baseline </li></ul><ul><li>Challenge of attribution </li></ul><ul><li>Challenge of aggregation and disaggregation </li></ul><ul><li>Challenge of non-users </li></ul><ul><li>Challenge resulting from the unexpected </li></ul><ul><li>Challenge of perspective/perception </li></ul><ul><li>“ Longitudinal” problem </li></ul><ul><li>Source: Souter, D. 2008. BCO Impact Assessment Study: Final Report. </li></ul>
  8. 8. Complexity of Change <ul><li>Change often occurs within a complex system – not linear and straightforward </li></ul><ul><li>Change in sought by development intervention can be susceptible to unexpected externalities </li></ul><ul><li>Change often needs to be measure against a moving baseline </li></ul><ul><li>The velocity of change can fluctuate due to various factors – some which are unpredictable </li></ul><ul><li>Important considerations: </li></ul><ul><li>In-depth understanding of context where the intervention takes place is necessary </li></ul><ul><li>In-depth understanding of baseline in which change can be measured </li></ul><ul><li>Continuous monitoring of data is needed rather than finding indicators only at the beginning and end of a particular intervention </li></ul>
  9. 9. Contextual Challenges <ul><li>Contextual understanding requires the knowledge of social, cultural, gender, structural, economic, political, and environmental factors. </li></ul><ul><li>The complexity and influence of context increases along the ICT4D value chain, from output to development impact </li></ul><ul><li>In most cases, those who understand context best are those living within the communities concerned </li></ul><ul><li>Important Considerations: </li></ul><ul><li>Important to account for all target beneficieries and stakeholders </li></ul><ul><li>Participation of project/program users in the design of impact assessment studies </li></ul><ul><li>Don’t assume replicability of impact before understanding context </li></ul>
  10. 10. The Challenge of Baseline <ul><li>IA concerns with measurement of change - need a starting point from which change is to be measured </li></ul><ul><li>Ideal baseline data include: broad context of the intervention or phenomenon, most up to date data, and data obtained from qualitative sources (to complement quantitative data) </li></ul><ul><li>In some instances, baseline data continue to shift rapidly (e.g. number of cell phone owned per 100 households) </li></ul><ul><li>Important considerations: </li></ul><ul><li>Integrate IA into the project/research design and its monitoring plan </li></ul><ul><li>Building a discipline for data collection is important in any intervention </li></ul><ul><li>Consider using trends rather than static measure in light of a moving baseline </li></ul>
  11. 11. The Challenge of Attribution <ul><li>While attributing immediate results (i.e. outputs) and even intermediate results (i.e. outcomes) may be possible, longer-term results (i.e. impact) pose a greater challenge </li></ul><ul><li>Allocating responsibility of a particular result to a particular cause (or causes) is substantially more difficult within a complex system. </li></ul><ul><li>Attributing impact becomes more difficult as the size of the intervention decreases </li></ul><ul><li>Important considerations: </li></ul><ul><li>Consider approaches that contributes to the reduction of uncertainty rather than trying to “prove” attribution in all IA scenario </li></ul><ul><li>Don’t underestimate the richness of data over statistical rigor – IA is not only about precision but it is also about understanding and improving </li></ul><ul><li>Understand the nature and character of the intervention in order to determine the limits of randomization/experimental approach </li></ul>
  12. 12. Challenge of Aggregation and Disaggregation <ul><li>Aggregation challenge – in a complex system different activities often have influence on one another, thus changes are attributed to a collection of interventions that influence one another </li></ul><ul><li>Disaggregation challenge – impact measures need to be disaggregated to various categories in order for it to be properly understood </li></ul><ul><li>Important considerations: </li></ul><ul><li>Aggregation challenge - same considerations in addressing IA attribution challenge – know the nature of the intervention before conducting IA, compliment “conventional” IA approach with other methods </li></ul><ul><li>Disaggregation challenge - build capacity to monitor and assess impact at the lowest level of disaggregation into the project design </li></ul>
  13. 13. Challenge of the Non-Users <ul><li>To get a holistic picture of what is happening following an intervention, impact assessment needs to include a wide range of stakeholders, more so than short-term evaluation </li></ul><ul><li>Those not intended to receive benefits from the intervention (i.e. non-users) may still be affected by the intervention, because of possible interactions with the beneficiaries. </li></ul><ul><li>Important Considerations: </li></ul><ul><li>Since non-users can confound IA results, they need to have the same chance of being selected in the study sample. </li></ul><ul><li>Consider visually mapping important stakeholders within the area where the intervention takes place when designing IA studies </li></ul>
  14. 14. Challenge of the Unexpected <ul><li>It is unlikely we will be able to anticipate every possible outcomes of an intervention at the beginning </li></ul><ul><li>Unexpected outcomes can be both positive and negative, and sometimes it can be more significant than the intended outcomes </li></ul><ul><li>Important Considerations: </li></ul><ul><li>Always expect the “unexpected” and be honest about the results – there can be important lessons from the negative experiences </li></ul><ul><li>Use of participatory methods in the design of impact indicators can help mitigate the unexpected </li></ul>
  15. 15. Challenge of Perception <ul><li>Non-beneficiaries and target beneficiaries of an intervention are likely to perceive change differently from one another </li></ul><ul><li>Different stakeholders also have different perceptions of impact (e.g. some beneficiaries may perceive impact at a household level whereas the project manager may perceive it a level of a social group or community) </li></ul><ul><li>Important Considerations: </li></ul><ul><li>Diversity of experience and perceptions should be captured around an intervention </li></ul><ul><li>Use of participatory methods for capturing diverse stakeholder views in the design of IA </li></ul>
  16. 16. « Longitudinal » Problem <ul><li>Impact assessment is often taken immediately after the completion of a project – but how can IA offer insights into “lasting and sustainable change” before they actually occur? How do we know if they have occurred? </li></ul><ul><li>Important Considerations: </li></ul><ul><li>Using proxy indicators may be useful for identifying future impact (e.g. using attitude surveys) </li></ul><ul><li>Consider IA design using a longitudinal studies (or tracker studies), undertaken some time after the intervention has been completed </li></ul>