Monitoring and evaluation of human rights projects

1,787 views

Published on

Monitoring a evaluace lidskoprávních projektů - Prezentace z workshopu Demas 3. 12. 2013

Published in: Education, Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,787
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
37
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • When to use logframe?ProblemAnalysis and ObjectiveSettingAnalysisofcauses and effects / prioritisationShows relevance to policies / contextChecksrisks and accountabilityforresultsEstablishesmeasurement systém and data collectionImplementationProvidesframeworkforResultsOriented monitoringEvaluationProvidesframeworkforevaluation as per OECD/DAC criteriaObservedIssuesToocomplicated, too many objectives / indicatorsTooambitiousobjectivesIndicatorsdifficult to measure / data not collectedordifficult to collectLogframe not usedforinternal monitoring / evaluation and decisionmaking
  • When to use logframe?ProblemAnalysis and ObjectiveSettingAnalysisofcauses and effects / prioritisationShows relevance to policies / contextChecksrisks and accountabilityforresultsEstablishesmeasurement systém and data collectionImplementationProvidesframeworkforResultsOriented monitoringEvaluationProvidesframeworkforevaluation as per OECD/DAC criteriaObservedIssuesToocomplicated, too many objectives / indicatorsTooambitiousobjectivesIndicatorsdifficult to measure / data not collectedordifficult to collectLogframe not usedforinternal monitoring / evaluation and decisionmaking
  • When to use logframe?ProblemAnalysis and ObjectiveSettingAnalysisofcauses and effects / prioritisationShows relevance to policies / contextChecksrisks and accountabilityforresultsEstablishesmeasurement systém and data collectionImplementationProvidesframeworkforResultsOriented monitoringEvaluationProvidesframeworkforevaluation as per OECD/DAC criteriaObservedIssuesToocomplicated, too many objectives / indicatorsTooambitiousobjectivesIndicatorsdifficult to measure / data not collectedordifficult to collectLogframe not usedforinternal monitoring / evaluation and decisionmaking
  • Examples of qualitative and quantitative indicatorshttp://www.smarttoolkit.net/?q=node/391Quantitative indicators for the ‘accessibility’ of an information service’:- Distance of rural communities from a low-cost telecentre with a helpdesk- Percentage change in the number of visitors to the telecentre over a defined periodQualitative indicators for the ‘improved knowledge’ resulting from an information service:- The telecentre is close enough to a range of communities for them to use it when they need to- Positive and negative experiences of the users of the information in solving the problem for which information was sought- Ability of the users to discuss the problem in questionReference points for comparison1. Trends (e.g., a consistent increase or decrease in requests for support; increasing feedback from readers of a publication)2. Thresholds (e.g., at least three districts covered by a database; the minimum number of students on a course)3. Targets (e.g., the number of documents distributed by a stipulated deadline; the proceedings of a conference available in printed form by a given date)
  • FromAssumptions – createRisksWeassumethat- Assumptionwillhappen – results/objectiveswillbeachievedRisk mighthappen, itthat case wehavepreparedmitigationmeasures so thatrisks do not happerprojectresults / objectives
  • Monitoring and evaluation of human rights projects

    1. 1. Workshop Monitoring a evaluace lidskoprávních projektů Monitoring and evaluation of human rights projects Inka Píbilová inka@evaluace.com 3. prosince 2013 1
    2. 2. Monitoring x evaluation x audit Evaluation • Assessment of project efficiency, effectiveness, impact, relevance and sustainability for the purpose of learning and accountability to stakeholders Monitoring • Ongoing analysis of project progress towards achieving planned results with the purpose of improving management decision making Audit • Assessment of (i) the legality and regularity of project expenditure and income i.e. compliance with laws and regulations and with applicable contractual rules and criteria; (ii) whether project funds have been used efficiently and economically i.e. in accordance with sound financial management;; and (iii) whether project funds have been used effectively i.e. for purposes intended. • Primarily a financial and financial management focus, with the focus of effectiveness being on project results. http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 2
    3. 3. Why to monitor and evaluate projects? • Accountability • Learning 3
    4. 4. Project cycle management Evaluation Implementation Financing Programming Identification Formulation http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 4
    5. 5. Stakeholder Analysis Matrix Example: River Pollution http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 5
    6. 6. Influence /Power Stakeholder Analysis – Power / Interest / Attitude KEEP SATISFIED INFLUENCE & ENGAGE Attitude Policy makers CSOs MONITOR Activists KEEP INFORMED Interest 6
    7. 7. Problem Tree Example: River Pollution http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 7
    8. 8. Objectives and strategy selection Example: River Pollution http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 8
    9. 9. Inputs – Activities - Outputs – Outcomes – Impacts Direct influence of implementers: Inputs I need for my work… (funds, people, time) Activities I do…. (research, conference) Outputs I produce … (research paper, 100 participants to conference) Indirect influence of implementers: Outcomes I strive to… (Results) (raise awareness on X) Impacts I aim to… (reduce human rights violations) 9 http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm
    10. 10. Logical framework http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 10
    11. 11. Logical framework Issues identified during project evaluations: • Logical frameworks too complicated, too many objectives / indicators • Too ambitious and/or too vague objectives • Indicators difficult to measure / data not collected or difficult to collect • Logframe not used for internal monitoring / evaluation and decision making Tips: • Numbering Objectives (1,2), Results (1.1., 1.2., ...), Activities (1.1.1., 1.1.2.) • Check source of verification – how will data be collected, by whom, when/how regularly? http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 11
    12. 12. Use both quantitative and qualitative indicators Compare using trends (increase), thresholds (min. 30%), targets (strategy by 12/Y1) Quantitative - SMARTER Specific / Simple (to understand, collect) Measurable Attainable/Available at cceptable costs Relevant to project / stakeholders Time-bound Evaluate/Engaging Reevaluate/Recordable Qualitative - SPICED Subjective Participatory Interpreted and communicable Cross-checked and compared Empowering Diverse / disaggregated (by gender) Min. 30 % of participants initiate a project aiming to address a local issue. Reasons why participants have (not) implemented a project to address a local issue http://www.smarttoolkit.net/?q=node/391 http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 12
    13. 13. Risks Analysis Assumptions in logframe http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 13
    14. 14. Monitoring: Why, for whom, what, how Why? • Timely identification of successes and problems during project implementation • Informed and timely decision making by project managers to support implementation • Accountability for the resources used and results achieved • Stakeholder awareness and participation • The evaluation of project achievements and audit of activities and finances 14
    15. 15. Use logframe to prepare Monitoring Plans Year 1 Indicator / Country 15.000 copies of publications distributed (5.000 each) 120 teachers trained (40 each) Min. 50% of teachers implement the new tools within 6 months • • • • CZ PL SK 90% 110% 100% 100% Achieved 45 60% 40 40% 120 40% 35 30% Total Note Achieved Not achieved Use indicators from the logframe Split by year and partner Use plan vs. actual, % or trends over time Develop adequate sources of verification • Distribution lists • Attendance sheets with contacts! • On-line feedback forms after 6 months (combined with supervision of teachers) • Media monitoring, public surveys... 15
    16. 16. Monitoring: Activity schedule with tasks and responsibilities 16
    17. 17. Monitoring Progress Report (Narrative) Monitoring Visit Checklist http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 17
    18. 18. Other monitoring tools 18 Forms at docs.google.com Geolocation at www.googlemaps.com, www.ushahidi.com
    19. 19. Monitoring tips • Keep the users of information clearly in mind (who needs what information?) • Build on local information systems and sources - costs and sustainability of new systems • Collect only the minimum amount of information required – keep it simple and practical, more information is not better information • Use Inception reports when team differs from the proposal writters, or when time passed by between identification and grants approval... usually 3 months after project start, revises original project proposal - updates logframe, sets annual action plan and monitoring plan
    20. 20. Evaluation: Why, for whom, what, how 20
    21. 21. Evaluator Selection Interviews Surveys Focus groups Case studies Final debriefing of all partners Draft evaluation report commented by all partners Final evaluation report Desk study Preliminary findings & conclusions Initial briefing and inception Terms of Reference – Objectives, scope, stakeholders, questions, budget, schedule, outputs, use. Plan external evaluation well ahead Communication with the Project Partners Inception phase Field research Reporting phase 1-3 months 1-3 months 1-2 months 21
    22. 22. Evaluation criteria 22
    23. 23. EC Evaluation Criteria 23
    24. 24. How to measure learning outcomes? http://leanlearning.wikispaces.com/learning_analytics 24
    25. 25. Evaluation tools 25
    26. 26. What does participation mean? 26
    27. 27. Reporting: Why, for whom, what, how 27
    28. 28. Where to learn more Road to Results https://openknowledge.worldbank.org/bitstrea m/handle/10986/2699/52678.pdf?sequence=1 EPDET - annual training in Evaluations by World Bank consultants http://www.dww.cz/english.php?page=epdet1 Evaluations of Czech development and humanitarian projects http://www.mzv.cz/jnp/cz/zahranicni_vztahy/rozvojova_spolupr ace/dvoustranna_zrs_cr/evaluace/index.html Different evaluation methods and approaches http://betterevaluation.org/ EC Project Cycle Management Manual http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm 28

    ×