Papa to do the welcome and introductions Explain also that this webinar will be followed up in the next few days with more detailed guidelines
Papa to lead
Papa to lead
Maureen to lead on this slide with Catherine adding in. Explain that all processes support each other but that they have different focus and purpose. The questions we ask are different. Go through the set. Use social housing example Will share two brief examples of successful programmes which have had negative impacts ... Tanzania and ??? As illustrations Point out that we evaluate impact ( and they will be doing this) to demonstrate the levels to which we achieved the impact we set out to achieve.. But with assessment we go further – instead of starting with the programme logic and plans, we start with changes in governance and transparency and the people this affects ... We explore what has changed for them ( good and bad) and then we assess what – if anything - our programme was able to contribute. WE ARE NOT TALKING ABOUT ATTRIBUTION
Catherine to lead on all of this component
A consistent stance in the evaluation that does not assume attribution of results to the Governance and Transparency Fund, but rather takes a critical approach and examines alternative explanations; Both the consultant in charge of the Impact Assessments and the consultant in charge of the Global Evaluation Report are available to support and advise individual national evaluation coordinators and consultants. Verification of evidence emerging through ongoing triangulation between the multiple data sources and methods employed; Step-by-step validation of evaluation results by national WaterAid teams (with peer review/ discussions as appropriate); Quality assurance processes that are built in to each national evaluation (as well as the preparation of the final global evaluation report) – should all meet the DAC Evaluation Quality Standards, UNEG Standards, or the comparable national or regional standards where these have been adopted; Prioritising the use of country systems to capitalise on existing data/literature including academia, universities, and civil society; and Using a set of agreed working definitions for key terms [and the WaterAid style Guide] to avoid confusion and inconsistent treatment.
Maureen to lead on this component
Build on what was said earlier.. In doing the evaluation, we have completed the accountability to the donor section and answered many questions around relevance and effectiveness etc. Now we really want to focus on the learning
Build on what we said earlier... This is really much more of an investigation... Really trying to find out what happened. I treat it as a murder mystery story... We know what’s changed but we don’t know how it happened. Your job is to find out.
As impact assessment is so focussed on change and our ability to influence change successfully, we need to be very clear about how we thought change would come about in different countries and contexts Each country programme should have developed its Theory of Change . We need to know about context, issues , key stakeholders. We especially need to understand the sequence of change that we thought would work ( what short term changes might lead to longer term change), what assumptions we made ( e.g. Moe press coverage will serve to influence both the wider public and eventually policy makers) We need to be clear abut other actors and factors who might either support or hinder progress in the areas we are working on
This is based on the specific outcomes that GTF hoped to influence. It is worded in neutral language to allow for you to explore negative and/or unintended changes as well as positive and expected changes Example - successful capacity building programme- staff plan and manage well, programmes are focussed and effective, staff are able to source funding etc... All good and lots of ticks. Bu the impact could be... They are so highly trained that they all leave to get jobs for the UN. We have to ask the so what question in order to understand what we should do differently next time
These are key questions ... You may have covered the positive and expected changes in the evaluation section, but this is an opp to really explore the “so what question” And find out if there have been unexpecxted and/or negative changes too. Baseline question – might be tricky to find the baseline, so you should build in questions that help you understand... E.g. Ask how networks have developed and changed over time the linking question is really important, as we want to know for example if it is worth investing efforts into capacity building – its not for itself... It should lead to changes in policy and practice... There might be bettre ways of making this happen To what extent did GTF contribute- really important question. The change might have happened but GTF might not have had anything to do with it. Example of girls education in Ethiopia - figures doubled in 4 years but it was because the big donors gave the Ethiopian gvt an ultimatum. Smaller orgs also working in advocacy in this area took credit for the change, but it wasn’t really as a result of their work. Remember – success has many parents but failure is an orphan! So try to find out who/what else might have been partly responsible for changes that you see Confidence is an important thing to think about. If you dont have concrete evidence for your claims, then the claims are weak. This is not always bad but you need to be clear about your levels of confidence in reporting change
Explain why we use areas of enquiry rather than indicators – to encourage answers that we don’t expect – remember open and probing questions We use words like shift and trends and levels. Where possible it is good to have checklists/scales to help you make sense of what has changed and for us to compare it across countries and programme. We’ll be sending checklists and ideas to support this. Will provide handout on characteristics of a functioning network Shifts in capacity - will be using the 7 s framework.. aspirations; strategy; organisational skills; human resources; systems and infrastructure; organisational structure; and culture. Each section is broken down into several indicators. Need to be aware that some partners may have stronger capacity that Wateraid, so there may be two way capacity building Wateraid advocacy scrap book make be able to help with trends in the way that organisations have been able to influence policy and practice
Awareness... Note to self – look at CAFOD voice and accountabilty tool – what has Wateraid got? See sustainability framework Need to look at all strategies and explore what differemce they have made –if any
Wiil find notes Need soemthing on media, Must read learning papers: Sustainability in Governancve programmes; Governance and Power analysis tools
Catherine to lead on this slide and following two
Maureen to lead on this slide and the following
Papa to lead Mention reading lists
Papa to lead
Transcript of "Final outline plan for webinar evaluation and impact assessment mof 2004 "
www.wateraid.orgWebinar on final evaluationand impact assessment ofGovernance and TransparencyFund ProgrammeTuesday 23rdApril 2013
www.wateraid.orgBackground and purpose of the twoexercises• The evaluation isprimarily forAccountability and theimpact assessment isprimarily for Learning• Complementaryexercises• CPs and partners areprimary users of results• Results forcommunication,fundraising etc• Progress against thebaseline data is critical
www.wateraid.orgOverview of all evaluation and learningprocesses and how they link together Mid Term Review – WHAT so far? Evaluation – WHAT? Impact assessment – So WHAT? Learning review – HOW? Most significant changeanalysis – the WHAT about the SOWHAT?
www.wateraid.orgKey Stakeholders in the processDFID/KPMG
www.wateraid.orgWhich country programme is doing what
www.wateraid.orgDifferent levels and how to deal withthis• 7 countries are doing a full scale evaluation• 9 countries are doing small scale evaluation• All countries are doing an impactassessment except from Kenya• Small scale = updating Mid Term Review• Full Scale = in depth assessment based onkey areas
www.wateraid.orgLength of the consultancy and how touse your time• Total number of days = 25 to be shared between twoexercises• Rough Guideline:Step 1: Understanding the context - understanding of theproblem in country that GTF is addressing• Background reading - 1 day• Working with country prog staff and key informants - reinforcingunderstanding of programme, stakeholders and intervention design,findings of MTR (if there was one) and conclusions - up to 3 daysStep 2: Enquiry – conducting self-assessment, semi-structured interviews, FGD…..8 daysStep 3: Analysis - Self-assessment collation of results,coding of qualitative data….2.5 accountability and 2.5 daysfor learning analysis;Step 4: Write the first draft of the report - 4 daysStep 5: Revisions and redraft of the report - 4 days
www.wateraid.orgTimelineDates Actions4thApril ToR for Evaluation and Impact Assessment sent out to all countries19thApril Each country to sign contracts with local consultants23th April Webinar with all consultantsMay 17thLocal consultants to submit draft Impact assessment section ofreportMay 31stLocal consultants to submit draft Evaluation reportsJune 24thLocal consultants to submit final Evaluation report including ImpactassessmentJuly 11thMoF to submit global consolidated impact assessmentWeek of July 22ndCM to submit draft Evaluation report and share report to KPMGWeek of July 29thLast Annual learning meetingMid Sept CM to submit Final Global Evaluation reportEnd of Sept /midOctoberPapa to share report with KPMGEnd of October Submission of WaterAid PCR to KPMG
www.wateraid.orgThe difference between anevaluation and impactassessment
www.wateraid.orgIn summary....Questions Monitoring Evaluation Impact AssessmentWhy do we doit?Measures on-goingactivitiesMeasures performanceagainst objectivesAssesses change in peopleslivesWhat is themain focus?Focus on programmeinterventionsFocus on programmeinterventionsFocus on stakeholdersAt what level? Outputs Outcomes/impact Impact and changeWhat are thekey questionsto ask?•What is being done?•Is our programmeprogressing asplanned?•What happened? Didwe achieve what we setout to achieve in termsof:•Effectiveness•Efficiency•Relevance•Sustainability•Impact•So what actuallychanged?•For whom?•How significant is it forthem?•Will it last?•What, if anything, did ourprogramme contribute?
www.wateraid.orgGTF Summative Evaluation• We are conducting a critical analysis ofthe GTF Programme in order to assesswhether or not it achieved its goals Whether the planned activities occurred Whether the activities led to achievement of goals; How effective the project was; How costly the project was; etc.• This is a summative or end of programmeevaluation
www.wateraid.orgPurpose of evaluation• For accountability- to enablebeneficiaries, board members, etc to knowhow funds have been used;• Our country evaluations will assess:– objectives against logframe targets andmilestones;– programme performance by OECD DACcriteria of effectiveness, efficiency, relevance,sustainability, replicability and impact
www.wateraid.orgA useful Global EvaluationReport• Top tips include:– i) the process of collation, analysis and write up– ii) enhanced rigour and comparability of results andreports…………...so• A consistent stance• Support and advice through online forum• Verification of evidence• Step-by-step validation of evaluation results• Quality assurance processes• Prioritise the use of country systems• Use a set of agreed working definitions for key terms• Use the WaterAid report template
www.wateraid.orgEvaluation QuestionsRelevance:• What we expect: Details of the programme’s significancewith respect to increasing voice, accountability andresponsiveness within the local context.Evaluation Questions:1. How well did the programme relate to governance priorities atlocal, national or internal levels? Please demonstrate withexamples in relation to: i)increasing voice; ii) accountability;and, iii) responsiveness within the local context.2. How well did the programme relate to the Country StrategyPaper aims and objectives? Of WaterAid and whereapplicable of the FAN network – ie regional secretariats and ofDFID.3. How logical is the current theory of change?
www.wateraid.orgEffectiveness• What we expect: An assessment of how far theintended outcomes were achieved in relation totargets set in the original logical framework.Evaluation Questions:1. Have interventions achieved the objectives? Atcountry regional and global level.2. How effective and appropriate was the programmeapproach? How effective was the MEL system andframework?3. With hindsight, how could it have been improved?
www.wateraid.orgPartnership• What we expect: How well did thepartnership and managementarrangements work and how did theydevelop over time? Please consider areassuch as monitoring, evaluation andlearning arrangements. If possible,consider from a regional perspective.
www.wateraid.orgAdvocacy• What we expect: To what extent has GTFcontributed to WaterAid influencingtargets?Evaluation Questions:1.How has the programme helpedimplement successful advocacystrategies? Are there any lessons learnedabout measuring influencing.2.How has the programme contributed tothe overall in country advocacy strategy?
www.wateraid.orgEquity• What we expect: Discussion of social differentiation (e.g. bygender, ethnicity, socio economic group, disability, etc) andthe extent to which the programme had a positive impact(from an accountability perspective) on the moredisadvantaged groups.Evaluation Questions:1. How did the programme actively promote gender equality?2. What was the impact of the programme on children, youthand the elderly?3. What was the impact of the programme on ethnic minorities?4. If the programme involved work with children, how were childprotection issues addressed?5. How were the needs of excluded groups, including peoplewith disabilities and people living with HIV/AIDS addressedwithin the programme?
www.wateraid.orgValue for Money• What we expect: Good value for money is the optimal use ofresources to achieve the intended outcome.Evaluation Questions:1. Has economy been achieved in the implementation ofprogramme activities?2. Could the same inputs have been purchased for less money?3. Were salaries and other expenditures appropriate to thecontext?4. What are the costs and benefits of this programme?5. Is there an optimum balance between Economy, Efficiencyand Effectiveness? Overall, did the programme representgood value for money?
www.wateraid.orgEfficiency• What we expect: How far funding, personnel, regulatory,administrative, time, other resources and procedurescontributed to or hindered the achievement of outputs.Evaluation Questions:1. Are there obvious links between significant expenditures andkey programme outputs? How well did the partnership andmanagement arrangements work and how did they developover time?2. How well did the financial systems work?3. Were the risks properly identified and well managed?4. For advice on measuring value for money in governanceprogrammes see DFID’s Briefing Note (July 2011)Indicators and VFM in Governance Programming, availableat: www.dfid.gov.uk
www.wateraid.orgSustainability• What we expect: Potential for thecontinuation of the impact achieved and ofthe delivery mechanisms following thewithdrawal of existing funding.Evaluation Questions:1.What are the prospects for the benefits of theprogramme being sustained after the fundingstops? Did this match the intentions?2.How have collaboration, networking andinfluencing of opinion support sustainability?
www.wateraid.orgInnovation & Replicability• What we expect: How replicable is theprocess that introduced the changes/impact?Refer especially to innovative aspects whichare replicable.Evaluation Questions:1.What aspects of the programme arereplicable elsewhere?2.Under what circumstances and/or in whatcontexts would the programme be replicable?
www.wateraid.orgExpected impact and change• What we expect: Details of the broader economic, social, and politicalconsequences of the programme and how it contributed to the overallobjectives of the Governance and Transparency Fund (increased capability,accountability and responsiveness) and to poverty reduction.Evaluation Questions:1. It is critical to demonstrate the progress in relation to the indicators included in theGTF programme logframe. The focus is on accountability for the impact.2. What was the programme’s overall impact and how does this compare with what wasexpected? Please demonstrate from an accountability perspective if the perceivedimpact was achieved and if not, why not.3. Did the programme address the intended target group and what was the actualcoverage? Again from an accountability perspective, was the coverage reached? Ifnot, why not, if yes, how?4. Who were the direct and indirect/wider beneficiaries of the programme? Again, theimportance here is to set out who these were for accountability purposes.5. What difference has been made to the lives of those involved in the programme?Describe the impact.6. As you are aware, the Consultant is also conducting more detailed critical analysis onImpact for learning purposes.
www.wateraid.orgVertical Logic of ProgrammeImpact is the higher levelsituation that the projectcontributes towardsachievingOutcome identifieswhat will change andwho benefits duringthe lifetime of theprojectOutputs are specificdeliverablesHuman Resource andfinancial inputsLEARNING:For the GTF GlobalConsultants thisrequires evidence of :‘so what’?LEARNING:For the GTF GlobalConsultants thisrequires evidence of :‘so what’?ACCOUNTABILITYFor the GTF GlobalConsultants this requiresevidence againstprogramme specificobjectivesACCOUNTABILITYFor the GTF GlobalConsultants this requiresevidence againstprogramme specificobjectives
www.wateraid.orgWhy conduct the impact assessmentcomponent?• To learn and improve:• To enable Country programme staff, stakeholders incountry, Wateraid staff and others to really understandwhat changed as a result of the programme and to applythis to future plans• To test and refine our understanding of how changehappens and how successful we have been in supportingpositive changes for our stakeholders:• To what extent did we work with the right people? In the right way? How didthis all link up ?• To what extent did the changes we expected to see along the way supportthe long term changes we were aiming to influence?• What does this tell us about the way we think we can influence change?• What should we do differently next time?
www.wateraid.orgThe Impact Assessment– Focus on the “so what question”• what’s changed?• For whom?• How significant/lasting are these changes fordifferent stakeholder groups?• In what ways did the programme contribute– Expect the unexpected - we are looking forevidence of positive/negative/ intended andunintended changes,– Prioritise analysis over gathering information– need for open and probing questions
www.wateraid.orgKey questions for theImpact Assessment
www.wateraid.orgBackground and context – what we needto know (Country Programme Theory ofChange):• The local and national context, including key social,political and environmental conditions and how theyhave changed over the life time of the programme• Key issues that the programme planned to address• The target groups who would ultimately benefit fromthe programme and how each would benefit?• The process or sequence of changes that would leadto the desired long-term goal• The assumptions that the programme made about theanticipated process of change• The other actors/factors who had the potential toinfluence the changes sought, both positively ornegatively.
www.wateraid.orgFour Domains of Change1. Changes in the ways in which CSOs function andnetwork, and their capacity to influence the design,implementation and evaluation of effective WASH policies atall levels2. Changes in the ways that CSOs, including thoserepresenting marginalised groups, are able to engage indecision-making processes affecting the WASH sector.3. Changes in the ways in which members of localcommunities demand accountability and responsivenessfrom governments and service providers in the WASH sector4. Changes in the ways that Governments and serviceproviders are accountable to citizens and end users in theWASH sector
www.wateraid.orgEach Domain is broken down further into“areas of enquiry”These are the key questions you need to explore across all Domains:1. What has actually changed for each of the different stakeholder groups,especially the poorest and most marginalized communities in relation toWASH (positive, negative, intended and/or unintended changes)2. How significant and/or sustainable are these changes for the differenttarget groups?3. To what extent do these changes compare with baselines and changesthat were planned and expected?4. How do they link together and/or influence each other?5. To what extent did the GTF programme contribute to these changes?How?6. Who or what else might have contributed to these changes? How?7. How confident are you in these findings (levels of evidence)?
www.wateraid.orgAreas of Enquiry Domain 1Domain 1 Key Areas of EnquiryChanges in the ways in whichCSOs function and network,and their capacity to influencethe design, implementationand evaluation of effectiveWASH policies at all levels• Ways in which networks have developedand function over time• Shifts in CSO capacity• How this capacity change has influencedpolicy and practice ato local levelso National levelNote: we will provide further guidelines on how to assess these areas of enquiry inthe next week
www.wateraid.orgAreas of Enquiry Domain 2• PDomain 2 Key Areas of EnquiryChanges in the ways thatCSOs, including thoserepresenting marginalizedgroups, are able to engagein decision-makingprocesses affecting theWASH sector.• Shift in awareness, knowledge and confidence ofmarginalized groups• Shifts in the ways that people have been able todemand their rights• The extent to which the voices of marginalizedpeople are making a difference to policy and practice• Ways in which different CSO strategies haveinfluenced change (e.g. budget tracking, participationin stakeholder reviews, etc…)Note: we will provide further guidelines on how to assess these areas ofenquiry in the next week
www.wateraid.orgAreas of Enquiry Domain 3Domain 3 Key Areas of EnquiryChanges in the ways in whichmembers of local communitiesdemand accountability andresponsiveness from governmentsand service providers in the WASHsector• Levels of awareness of rights in localcommunities• Ways in which media coveragesupports understanding of rights• Ways in which citizens are influencingpolicy and practice over time• Changes in community access to WASH• Changes in community influence overnatural resourcesNote: we will provide further guidelines on how to assess these areas of enquiry inthe next week
www.wateraid.orgAreas of Enquiry Domain 4Domain 4 Key Areas of EnquiryChanges in the ways thatGovernments and service providersare accountable to citizens and endusers in the WASH sector• Changing levels of governance,transparency and compliance• Changes in policy and regulation (e.g.new policies, laws, standards, politicaland institutional framework) – andthe consequences of these• Changes in practice relating to WASH(e.g. delivery of new services andsystems) and the consequencesNote: we will provide further guidelines on how to assess these areas of enquiry in thenext week
www.wateraid.orgMethodology for both componentsRestrict yourselves to using a few tried and testedtools. We suggest:– Facilitated self assessment: building on MTR which willsupport the evaluation component• How to do this and who should be involved• Note: we will be adding some more change questions this time– Follow up workshop to validate findings and focus on theimpact assessment element• How to do this and who should be involved– Other in depths interviews /FGD with key informants asrequired• This might enable a deeper understanding of e.g. how changesaffected particular target groups
www.wateraid.orgGuiding Principles for Methodology• Create an atmosphere where informantsfeel able to be honest and provide criticalfeedback. Use an appreciative enquiryapproach• Ensure a mix of both qualitative andquantitative data is gathered• For the impact assessment – ask openand probing questions for a deeperunderstanding of change• Findings must be backed up with evidenceand be set against the original baseline
www.wateraid.orgConsideration of sample sizeQuestions:• How many people to interview?• What is a “good enough sample?”Answers:• Be pragmatic (you have limited time but need to berepresentative).• Plan with in country focal point:– Include people/groups/interventions which represent• good/strong• Medium• Poor/weak• Explain your sampling decisions in the methodologysection of the report ( with an indication of the level ofrigour you believe this provides)
www.wateraid.orgThe ReportWe will provide more detailed guidance over the next week but this isthe guide35 -40 pages to include:• Executive Summaries x 2 - (4 pages in total)• Contents + Abbreviations (1 page)• Methodology + challenges and limitations (2 pages )• Country Context and introduction to the programme (3 pages )• Evaluation Report (10 pages) – findings and conclusions under thefollowing headings:– Relevance, Effectiveness, Partnership, Advocacy Equity, Value forMoney, Efficiency, Sustainability, Innovation and Replicability, ExpectedImpact and change.• Impact Assessment Report (10 pages) – findings and conclusions under thefollowing headings:– Changes under each Domain– Overall analysis of impact for different target groups– What difference the programme has made overall• Overall conclusions and learning for Country Programmes, for the sectorand globally (4 pages)• Annexes
www.wateraid.orgOther ways to present findings• Opportunity for Wateraid to share findings andlearning with a wide group of stakeholders -• Target groups:– Country programme staff– Networks in country– Partners– Wateraid donors– Sector specialists• Supplementary ways of presenting findings (optional)– Case studies– Video footage– Photos
www.wateraid.orgNext steps• Maureen and Catherine to send moredetailed guidelines by Friday April 26th• In country evaluation team to take stock ofthe outcomes of the webinar and• Prepare and send proposal to Catherine andMaureen copied to Marta and Papa byFriday April 26thwith a brief overview of yourplan including:– Time line– Your methodology– Key informants– Sample size and rationale for this
www.wateraid.orgSupport and assistance• Guidance notes to follow:– Some thematic guidance– Self-assessment format to use– Learning questions– Report format• Online forum: you will be able to post questions, debate issues andfindings with other consultants involved in the GTF final evaluation andimpact assessment.– Please expect an email in your inbox with a password and user details fromeither Catherine or her colleague Erica Packington Erica will manage access tothe private forum.– Please action the email immediately to guarantee access.• E mail contacts:– Technical advice• Catherine Currie firstname.lastname@example.org• Maureen O’Flynn email@example.com– Logistics:• Papa Diouf PapaDiouf@wateraid.org• Marta Barcelo MartaBarcelo@wateraid.orgPlease copy everything to Marta