Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. Common Impact Assessment Approaches and Learning Summarised for EDIAIS Case Studies Report by: Angela Motsa, WISE Representative/ Consultant Updated October 2003 Introduction The tables below summarise some of the key Impact Assessment (IA) approaches and lessons that can be learnt from impact assessment case studies found on the Enterprise Development Impact Assessment Information System (EDIAIS) website. The summary includes: o Table 1: Case study summaries o Table 2: Common IA approaches o Table 3: Key policy lessons per case study o Table 4: Common issues across different IA case studies Abbreviations The following abbreviations are used in the report. AAP: Agribusiness Agent Programme ADV: Advocacy ApproTEC: Appropriate Technologies for Enterprise Creation BDS: Business Development Services BG: Bank Guarantees CA: Contract Association CB: Commercial Bank CD: Capacity Development CEEWA-UG: Council for Economic Empowerment for Women of Africa – Uganda Chapter CERUDEB: Centenary Rural Development Bank CO: Country Office Cons. Consultancy Services CRIMP: Central Region Infrastructure Maintenance Programme CRISP: Credit for the Informal Sector DCFU: Development Finance Company of Uganda DG: DCFU Group DM&E: Design Monitoring and Evaluation EC: Employment creation ED: Enterprise Development EQ: Equipment purchase or leasing ET: Enterprise training FEBDEV: Foundation for Entrepreneurial and Business Development FGD: Focus Group Discussion FSA: FSA International FT: Fair trade HIV: Programmes with a focus on HIV/ AIDS IA: Impact Assessment INFR: Infrastructure development IDPM: Institute for Development and Policy Management, University of Manchester IS: Input Supplier ILGP: ITDG’s Light Engineering Project ITDG: Intermediate Technology Development Group JX: Just-X KF: Kashf Foundation KK: Kuapo Kokoo K-Rep: K-Rep Holdings Ltd Kenya 1
  2. 2. LED: Local Economic Development MDA: Mineworkers Development Agency M&E: Monitoring and Evaluation MF: Microfinance MFI: Micro-finance institution MICROSAVE: MICROSAVE-AFRICA MIS: Management Information System ML: Micro Leasing MLO: Market linkages and opportunities/ Marketing Support MR: Market Research PLA: Participatory Learning in Action PRA: Participatory Rapid Appraisal PSSSEP: Port Sudan Small Scale Enterprise Programme PWR: Participatory Wealth Ranking OFT: Oxfam Fair Trade OI: Opportunity International OPR: Output to Purpose review R: Research RF: Rural Focus RI: Research International RMU: Regional Management Unit SBF: Small and Medium Business Finance SBP: Small Business Project SEF: Small Enterprise Foundation SELFINA: SERO Lease and Finance Company SFMC: SIBDI Foundation for Micro Credit SWARMU: Southern and West African Regional Management Unit TEEM: Training for Enterprise and Exports Malawi UGAFODE: Uganda Agency for Development Ltd (UGAFODE) ZHABS: Zambia HIV/ AIDS Business Sector Project Tables 1-4 follow below. 2
  3. 3. Table 1: Case study summaries Programme Type1 Description Key elements in IA approach include: Agribusiness Agent MLO, IS Improving access to agri-inputs for small • Donor required reviews and evaluations (OPR). Programme (AAP), holder farmers through: • Commitments to IA studies as a result of OPR. Zimbabwe • sale of inputs by agents • Baseline survey. • linking large input supply companies • MIS (as a result of OPR) – monitoring mechanisms. to small rural traders (agents) • Indicators (geographical and poverty indicators). • IA to provide future direction of programme. • Research studies on agents’ operations. Appropriate EQ Developing and promoting technologies • IA is integral to organisational strategy development. Technologies for (equipment) to ensure profitable small • IA as customer service and market research tool. Enterprise Creation scale enterprises. • Baseline of enterprise/ household levels – to deal with attribution. (ApproTEC), Kenya • Continuous internal IA. • Annual internal IA. • Occasional/ external reviews (required by donor). CARE International RMU Regional management unit for CARE’s • IA approach is being developed focusing on: Southern and West Southern and West African programmes o Building capacities of CO for DM&E, African Regional focused on strengthening the country o Generating MIS’s from log frames, Management Unit offices’ (CO) design, monitoring and o Participatory methods of information gathering, (CARE SWARMU), evaluation (DM&E) capacities. o Monitoring as part of ongoing project activities, South Africa o Data analysis and information management, and o Information sharing and experiential exchange. CASHE MF, CD, CARE India / DFID project focused on • Impact indicators set up per goal as required by funder RF improving the rural micro finance sector – Client satisfaction study based on key goals, activities and including dealing with institutional, measures collectively agreed log frame. Used the following IA methodological and regulatory issues. methodologies: document review, FGDs, questionnaires, statistical Promotes client owned and managed analysis savings and credit groups and then federated organisations representing these groups Centenary Rural MF, CB Commercial bank offering a full range of • External Reviews – recommendations have been incorporated into new Development Bank appropriate financial services including business plan. (CERUDEB), Uganda savings, credit and funds transfer services • Ongoing collection of financial data through MIS. 1 Refer to table of abbreviations 3
  4. 4. Programme Type Description Key elements in IA approach include: to economically-disadvantaged • Customer perception/ satisfaction surveys where IA is used for market Ugandans. research. • Use of customer satisfaction surveys for staff incentive programme Central Region CD, Community based road maintenance • Participatory livelihood assessment informs programme design. Infrastructure INFR through community based contract • Baseline study for wealth ranking and as basis for information system. Maintenance associations (CA), skills development and • Livelihood monitoring system using log frame. Programme (CRIMP), the establishment of a network of small • Quarterly data collection, analysis and policy evaluation/ review. Malawi scale community contractors. • Specific budget for M&E. Council for Economic Adv, R, CEEWA-UG is involved in advocacy, • Baseline studies on various sectors (microfinance, agricultural extension, Empowerment for CD research and training to increase information and communication technologies) have influenced Women of Africa – women’s access and control over key government policy Uganda Chapter economic resources such as land, • Use a variety of quantitative, qualitative and participatory methodologies (CEEWA-UG), Uganda agriculture, finance, markets and including key informant interviews, FGDs and questionnaire based information. surveys Credit for the Informal MF, CB Commercial Bank providing micro-finance • External reviews. Sector (CRISP) (savings and credit) to the economically • Regular assessment of profitability to determine feasibility of expansion. Zimbabwe active poor. • Concern is mainly outreach and business sustainability. • Donor interest in number of jobs created and sustained. DCFU Group (DG), SBF, MF, Development finance company offering • Internal concern is profitability and sustainability. Uganda ML small and medium business finance, • Donors (shareholders) require socio-economic assessment. (formerly Development wholesale finance for MFIs, and leasing • Annual report. Finance Company of products. • Client surveys as required by funders. Uganda) Foundation for ET, CD Focuses on providing and developing • IA approach is being developed. Baseline survey identified parameters, Entrepreneurial and entrepreneurial skills to individuals, data capture methods and standard baseline data and records required Business Development companies and educational institutions. for IA. Also identified social indicators, baseline information and on-going (FEBDEV), South Africa Main programme focus on schools, M&E data questionnaire. Currently track achievement of objectives colleges and enterprises. through progress reports and want to assess impact at 3 levels of educators, learners and the institution based on existing M&E tools. • Baseline snapshot included data collected through questionnaire, FGDs, case studies and educational institution staff workshops. • Challenge in reconciling donor vs. organisational needs for IA. A comprehensive range of indicators is required by the donor including impact on capacity, attitude, access and livelihoods. • IA approach varies across programmes and levels of assessment FSA International (FSA), MF Micro-finance services (shares, loans, • External evaluations of achievements required by funder (OPR) include 4
  5. 5. Programme Type Description Key elements in IA approach include: Uganda savings account) through village banking. wider socio-economic impact. • Monthly social audit report produced. • Quarterly random surveys of stakeholders. IDPM & EDA Rural R Research project to understand the • 2 research sites 1 urban 1 rural Systems “Money market for financial services among the • 3 different techniques used over a period of a year: Matters”, India poor in North India. Funded by DFID. – Financial diaries of 48 households compiled - record of financial transactions compiled through open and unstructured interviews – More detailed diaries for a further 92 households (2 rounds of interviews on financial needs and preferences) – Interviews with financial service providers which respondents had said were especially important to them. • Participatory Wealth Ranking used to identify a broad range of respondents ITDG’s Light MLO, Manufacturing and production equipment • Ongoing impact assessment within operations and the project Engineering Project EQ, ML hire and purchase for artisans. Linkages management cycle. (ILEP). Zimbabwe are also made between manufacturers • MIS to capture routine project data. and the market. • Regular and random impact assessment of buyers. • Surveys of wider impacts on livelihoods. • IA used for strategic planning. Just-X (JX), South FT, MLO Marketing support and fair trade • Evaluations through external consultants (some as a requirement of Africa (closed) organisation assisting small producers to main funder) assessing effectiveness, sustainability, outreach, and be more effective in local and export efficiency. marketing. Kashf Foundation (KF), MF, CD, Provides micro-finance (loans and • At the early stages IA is seen as a capacity building and learning Pakistan ADV deposits), social advocacy and capacity initiative to improve practice building for poor women. • Client exit surveys. • Client attitude survey. • Poverty Assessment Study. K-Rep Holdings Ltd (K- MF, Holding company for K-Rep Bank, which • In a learning organisation that innovates, tests and implements, IA is Rep), Kenya Cons, R provides a range of microfinance products measured both from an organisational and product performance and services (savings and credit to micro perspective using indicators related to profitability, portfolio quality, client and small enterprises) and K-Rep records, client incomes, and outreach. Development Agency which is an NGO • Methods used include: regular group meetings FSAs with staff doing microfinance product development monitoring and FSA monthly board meetings minutes; regular staff and experimentation. Has also set up 43 meetings, day-to-day programme performance indicators, external 5
  6. 6. Programme Type Description Key elements in IA approach include: FSAs reviews and evaluations, and research. Kuapo Kokoo (KK), FT, MLO, Member based fair trade cooperative • Donor funded impact assessment. Ghana MF dealing with marketing problems and • M&E system. increasing FT market share for cocoa • Framework for analysis including commodity chain analysis, livelihoods producers in Ghana. Also have 2 credit analysis, sustainability and cost effectiveness analysis. schemes. MicroSave-Africa CD, R MicroSave-Africa builds the capacity of • Market research (MR) with a focus is on improving (not proving) impact (MicroSave), Uganda (MF) African MFIs to provide secure, high of MFI services through seeking clients’ views on products and services. quality savings services for poor people MR toolkits use qualitative methods such as PRA and FGDs, quantitative through research, curriculum methods such as questionnaires. development, training and dissemination. • IA case studies used for comparative analysis and lesson learning. IA Activities include: uses FGD, PRA and individual interviews to assess client perceptions. • Impact assessment tools for use in • Part of 3-year East African microfinance impact assessment programme market research by MFIs which aims to improve MFI services through increased effectiveness of • Conducts IAs for MFIs as case impact assessment systems. Will include developing indicators for IA, studies improving MIS, qualitative research and comparative analysis. • Coordinates regional impact project as part of ImpAct Mineworkers CD, A trade union initiated Business • Client data base capturing quantitative information. Development Agency MLO, Development Services organisation • External gender appraisal. (MDA), South Africa EQ, IS focusing on assisting people affected by • First full IA donor driven, done by external consultant led to set up of large scale mine retrenchments to start M&E unit. income generating projects and small • Baseline surveys and forms for capturing data as part of M&E system. businesses. • Organisational review also followed the IA. Oxfam Fair Trade (OFT), FT, MLO Fair Trade programme targeting • Participatory methodologies link IA to organisational learning and UK disadvantaged producers to help them operations. build their capacities to access markets, • M&E and IA are seen are part of the project management cycle. both export and domestic markets • Participatory monitoring. • External pilot IA focusing on impact of FT on improving livelihoods, sustainability and the profile of beneficiaries. Port Sudan Small Scale MF An ACORD network project providing • 2 small exploratory gender impact assessment Enterprise Programme Islamic modes of credit to refugees and – gender review using participatory methods was undertaken in 1995, (PSSSEP) poor Sudanese living in the slum areas of – study of the impact of the training programmes in 1998. 18 women Port Sudan. Credit is mainly provided for in two groups were interviewed using qualitative interview self-employment and small business techniques as well as participant observation in training sessions. although sometimes for other purposes 6
  7. 7. Programme Type Description Key elements in IA approach include: such as housing and sanitation. • Following the potentially negative effects that the shift to financial sustainability had on gender a number of specific strategies were adopted - enhancing women's participation in credit administration, additional support to encourage group-based lending and networking activities among women, enhancing gender awareness, continuation of formal and informal community development activities, and informal support for women's centres Research International R, MF Research agency - case study focuses on • Impact of micro-finance assessed in terms of the effect on the wider (RI) East Africa, Kenya exploratory study based on case studies household and livelihoods. of clients of 6 MFIs to assess impact of • Attribution based on qualitative interpretation of evidence, rather than their services on businesses and statistical inference i.e. client perspective on impact. households. • Exploratory rather than conclusive research. SERO Lease and MF, Micro-finance and micro-leasing • Random Baseline study at the start of operations. Finance Company, ML, company making capital available to • Informal, spontaneous evaluations via weekly FGDs and loan officers’ Tanzania (SELFINA) BG, women’s and indigenous businesses visits to clients. • IA is closely linked to monitoring operations in terms of daily payment reports, weekly arrears reports, and a yearly financial report by external auditors. • Independent assessments of client satisfaction. SHARE Microfin Limited MF SML was part of an impact assessment • Used five AIMS (Assessing the Impact of Micro finance services) tools (SML) study of 21 MFIs in 14 countries initiated • Used quantitative (survey data collection) and qualitative tools (focus by Ford Foundation and Institute for group discussions) Development Studies • Approach incorporates tracking poverty reduction over time based on important indicators, plotting the position of households on a composite index of poverty at different points in time. SIBDI Foundation for MFI A foundation incorporating 31 micro • 5 MFIs representative of 31 MFIs partners representing different regions, Micro Credit (SFMC), finance partner institutions models of microfinance, age, outreach, range of services India • 2-4 sample areas at each MFI representing a typical area of the MFI • Within each sample area stratifies random sample of clients, non-clients and dropouts. Wealth ranking used for stratification • Techniques included focus group discussions, quantitative survey • Findings on impact in terms of outreach using wealth ranking categories, use of group vs. individual loans, credit needs and credit sources, impact of MFI on money lender, women’s empowerment, environmental issues Small Business Project MLO Corporate sector initiative focusing on • IA is a tool for reviewing and developing strategy (SBP), South Africa facilitating linkages between large • M&E is a basis for IA and includes ongoing (monthly) data collection and 7
  8. 8. Programme Type Description Key elements in IA approach include: companies and small firms and on LED. quarterly monitoring reporting • IA is an integral part of project log frame. • Staff involvement Small Enterprise MFI, Adv, MFI focusing on the provision of • Impact monitoring system with an emphasis on participation and Foundation (SEF), South CD sustainable microfinance services (credit organisational learning. Africa and savings promotion) with poverty • Focus on improving policy rather than proving impact. targeting and using group based • Elements include conventional IA studies, case study research and (Grameen) methodology. continual impact monitoring by staff • IA findings feed into programme policy; has led to changes in the programme (e.g. loan terms, sizes and increments). • Participatory data collection and analysis based on participatory wealth ranking (PWR) using indicators partly developed by clients / communities combining poverty, business and social indicators. • Specific IA studies on profiles of businesses, reasons for drop outs etc • Integrated into field staff responsibilities Training for Enterprise MLO Facilitates SME exports through the • Learning approach where IA is a tool for improving programme. and Exports (TEEM), provision of demand – led business • IA planned during project design Malawi counselling, product advice and export • Impact of each type of service is measured as well as overall impact on marketing services. poverty. Uganda Agency for MF, ET, Provides savings and credit (Trust Bank • M&E required by network partner (OI) as condition of membership. Development Ltd CD model, solidarity and individual loans), training • Emphasis on understanding clients’ needs and business conditions as a form of (UGAFODE) and counselling to micro-entrepreneurs. Loan market research. products include trust banks, solidarity group • Network partner supports establishment of M&E systems and IA studies. and individual loans. UGAFODE is a member • Ongoing monitoring includes monitoring repayments and arrears, and loan of Opportunity International (OI). officer’s routine assessments of client’s business performance, case studies tracked over time, weekly business unit staff meetings. • M&E and IA feeds into programme design and strategic planning. The IA tools are also seen as forming a basis for improvement in policies, work methods and product design in order to meet clients’ needs. • Recent IA study used both: qualitative (such as business practices, access to social facilities, quality of life, self esteem and decision making abilities, social relations and contribution towards community activities, political participation, gender equity, participation in religious activities) and quantitative indicators (enterprise resource and asset base, input supply, stock turnover and financial performance, household income, asset base, community progress in terms of jobs created • The following IA tools are being tested: FGD tool to test client satisfaction; in- depth individual interview tools to test client empowerment and use of loans, 8
  9. 9. Programme Type Description Key elements in IA approach include: profits and savings over time; quantitative client exit survey tool; and quantitative impact survey. Zambia HIV/ AIDS HIV, CD HIV/ AIDS information and awareness • Baseline behavioural surveillance survey to be follow up survey after a year. Business Sector raising for Zambian companies. • Data collected monthly on peer education sessions and at clinics. (ZHABS) Project • Quarterly follow up at clinics. • In depth clinic quality assessments • Involvement of community in investigating programme expansion. • External IA as required by funders as part of mid-term review (OPR) 9