Module 2rev

263 views
236 views

Published on

Published in: Business, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
263
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Module 2rev

  1. 1. EuropeAid ENPICROSS BORDER COOPERATION SEMINAR ON MONITORING METHODOLOGY 2. The EC ROM system: BASIC CONCEPTS BRUXELLES, 9-10 SEPTEMBER 2010
  2. 2. Origin of ROM EuropeAid• In May 1999 the Council of the European Union invited the European Commission to establish a performance- based monitoring system with the objective to strengthen Monitoring, Evaluation and Transparency• Monitoring is considered crucial for timely adjustments in programming.• Each Directorate General (DG) of the EC is called upon to “design monitoring arrangements that ensure that information on outcomes and use of resources is regularly collected”.• EuropeAid responded to this call with the establishment of the Results-Oriented Monitoring system (ROM). 2
  3. 3. What is ROM? EuropeAid A “snapshot” project by project According to a standardized methodology It essentially assesses results and outcomes and not just activities and use of resources
  4. 4. What ROM does? EuropeAid• ….provides the EC with a wide range of quantitative and qualitative data on the performance of the funded actions• …..serves as a tool for day-to-day project management by informing stakeholders about the performance of a specific project, but it also contributes to general EuropeAid policy articulation, implementation and review.• …..thanks to the monitoring data collected in EuropeAid database, provides an overview of the performance of the overall EC portfolio.• …thanks to data collected by independent experts through regular onsite missions through a consistent, highly structured methodology, it ensures the quality and the comparability of the collected data. 4
  5. 5. ROM in the Project Cycle EuropeAid 5
  6. 6. Purpose of ROM EuropeAid• ROM system gives an overview how the project portfoliois progressing towards results, helping the answers to themain questions: - are projects and programmes (likely) to deliver theintended improvements in the lives of people? - is the project, with all due flexibility pursuing its plannedcourse and will it attain its objectives? Is it on track?ROM can serve as a useful tool for identifying trendswhich should then be complemented by more in-depthanalysis such as evaluations. 6
  7. 7. Main features of ROM system EuropeAid• There are 5 main features characterizing the ROM system:The adoption of the accepted DAC evaluation criteriaThe use of external consultants to ensure independency and transparencyThe application of a common standard methodology throughout all missionsThe capacity to consolidate data and information to offer larger views on portfolio performanceThe utilization of standard PCM / LFA approach 7
  8. 8. 1. ROM assessment criteria EuropeAid• The DAC evaluation criteria are the basic points to assess the project performance.• Relevance describes how well a project addresses a real problem of the beneficiaries and how well it matches the EC development policies strategic objectives.• Efficiency stands for how well the inputs are transformed into output and outcomes.• Effectiveness measures the degree to which the project’s outputs have provided benefits and contributed to the project purpose.• Impact describes how and to which degree the project has contributed to the solution of the problem and to the achievement of the overall objective.• Sustainability measures to the likelihood of a continuation of the benefits produced by the project after the period of 8 external support has ended.
  9. 9. DAC concepts - 21 Assessment criteria EuropeAid•Relevance & Quality of design •Impact Prospects•1.1 Present level of relevance •4.1 Wider planned impact•1.2 Present design •4.2 Monitoring on wider impact•Efficiency •4.3 Follow up on•2.1 Availability of means / inputs recommendations•2.2 Implementation of activities •Potential Sustainability•2.3 Results achievement •5.1 Financial/ economic viability•2.4 Partner contribution •5.2 Ownership•Effectiveness •5.3 Policy Support•3.1 Access to project results / •5.4 Capacity developmentservices •5.5 Socio-cultural adequacy•3.2 Use of project results / services •5.6 Gender equality•3.3 Achievement of Project Purpose •5.7 Technological adequacy•3.4 Follow up on recommendations •5.8 Environmental adequacy 9
  10. 10. 2. Who are the monitors? EuropeAid• ROM Monitors are external independent experts recruited by the ROM contractors (private consulting companies) after approval by the EC.• Externality is required to avoid conflict of interest and ensure independence of judgment• Selection criteria:Sector/thematic experienceRegional/country knowledgeMix of Senior/Medium/Junior/LocalAvailabilityNo previous direct involvement in the project• Risks: costs, quality not always adequate, availability, dependence on each expert commitment 10
  11. 11. 3. Basics elements of the standard EuropeAidmethodology• The use of a common standardized methodology is one of the most decisive features of the ROM system• The standard methodology is based on four main instruments:A) common criteria to select the actions to be monitoredB) common procedures to implement the missionsC) use of common formats for collecting the information through standardized question and grading criteriaD) Common outputs 11
  12. 12. 3a. Selection of actions to be monitored EuropeAid• All projects with budgets exceeding 1 million €• All projects that have been in execution for at least six months (as of signature of the Financial Agreement etc.)• All projects that have at least six months remaining before closureStart Implementation End € 1 million 6 months 6 months Monitoring period- Monitoring visits are avoided when a mid-term review is taking place (period of six months before and after) - When AIDCO or Delegations specifically request it, projects below 1M€ can be included (special flexibility with thematic / NGO / SA budget lines) 12
  13. 13. 3b. Procedures to implement the missions EuropeAidTime allocation for ROM missions:• The monitoring exercise is to be speedy and quick.• Five working days are usually allocated for the field visit of an ongoing project.• Monitors have one day before the mission to get acquainted with the project and study the files and two more days after the mission to complete the reports.• So total working days per monitoring report are 8 plus travel (1 or 2 more)• Ex-post ROM incl. an adequate preparation time to identify interlocutors can require more days (up to 10 working days).• A ROM mission usually assigns two operations per monitor. 13
  14. 14. EuropeAid 14
  15. 15. 3c. Common formats EuropeAid• 3 special formats:Project Synopsis provides a concise overview on project background, logic of intervention and administrative data.Background Conclusion Sheet is the key methodological instrument for ROM providing the standard instrument through a set of questions for monitoring and ensure objectivity, consistency and comprehensiveness.Monitoring Report is the main document to present findings of the monitoring mission. It includes general and financial information on the project, grades for 5 ROM criteria (obtained via the embedded calculation in the BCS) and a summary of conclusions. 15
  16. 16. 3c. Common scoring system EuropeAid• The grading of a project/programme for each of the five evaluation criteria mentioned gives a quick overview of its performance. In addition, it allows for a comparison between different operations and for the aggregation of overall portfolio performance data.• The grade is meant to be a summary of the answer, rather than the answer being a justification for the grade• However, grades strongly reduce the complexity of the issues at hand and their context. They risk overshadowing the actual findings.• In the BCS, monitors should tick the performance conclusion checkboxes next to the questions after having given an answer in writing. 16
  17. 17. 3c. Common scoring system EuropeAid 17
  18. 18. 3c. Using the scoring system EuropeAid 18
  19. 19. 3.d Standard ROM Outputs EuropeAid1. A two page monitoring report: it provides, short report explanations and recommendations on quality improvement + important complementary information to internal monitoring2. In order to guarantee uniformity and consistency it is supported by the ‘Background Conclusion Sheets’, Sheets’ which ensure that all monitors address the same questions and issues. Each sub criteria has to be graded and the overall scores for each criterion are automatically calculated and then used in the MR.3. Project synopsis4. A response from the field on quality and usefulness of the report is also uploaded 19
  20. 20. EuropeAid 20
  21. 21. EuropeAid 21
  22. 22. Overview of EU ROM 10 years after EuropeAid• ROM: a useful instrument at “micro” level. To inform stakeholders on project performance and help project managers “to think in result oriented terms”. ROM’s main objective• ROM: provides statistics on overall portfolio performance. An overview for management decisions in AIDCO. ROM added value (quantitative analysis)• ROM reports (stored in CRIS) are one of the sources to inform the programming cycle. Lessons learnt and experiences: e.g. to the Quality Support Groups. ROM in the learning cycle 22
  23. 23. Who benefits from ROM ? EuropeAid• EuropeAid HQ – broad overview of project performance and identification of patterns of problems that need to be analyzed and supported.• EC Delegations and Task Managers – timely, independent review of portfolio of projects with identification of principal weaknesses and corresponding recommendations.• Project Management – provision of independent advice from appropriately qualified experts. Opportunity to assess progress and make necessary adjustments.• Final beneficiaries - opportunity to discuss progress and improved delivery of project services and outputs 23
  24. 24. ROM results after 10 years EuropeAidROM on ongoing projects:• More than 13,000 monitoring reports produced on 5500 project• More than 180 countries visited• In 2009 1,548 monitoring reports were produced• The total amount of investments monitored was €8.3 billion.• ROM covered 28% of the expenditures managed by EuropeAid and 21% of the projects and programmes ongoing in 2009ROM Ex post:• 700 projects/reports in 100 countriesROM on SPSP :• 34 missions in all regions from 2007 until the mid 2010 24
  25. 25. ROM results EuropeAidLarge majority of projects are performing well,Slight improvement in 2009
  26. 26. ROM results Performance by evaluation criteria 2009 EuropeAid2
  27. 27. Using ROM data: EuropeAidFactors for good and bad performanceROM data, especially those collected Ex-Post, can provide valuable lessons learnedfor future programming and implementation.A Study on “Causes underlyingEffectiveness and Impact of ECDevelopment Projects” identified 6 factorseach for good and bad performance: 27
  28. 28. Factors for poor performance: EuropeAidI. Weak Analysis Stage/Identification PhaseII. Strategic relevance (for EC or partner government) that does not correspond to real relevance for the target groups and final beneficiariesIII. Mismatch between allocated resources and planned objectivesIV. Inadequate risk managementV. Non-existent or very poor internal monitoring systems 28
  29. 29. Factors for good performance: EuropeAidI. Proactive managers and inclusive management approachesII. Implementing partners with strong organisational / technical capacityIII. Strong analysis stage to target real problems/priorities with well-suited strategiesIV. Relevance for beneficiaries addressed throughout the project’s lifetimeV. Demand-driven interventions resulting in widespread access and use of services and 29 high levels of ownership
  30. 30. Some final points for your consideration EuropeAid• ROM system is very effective in the production of large amount of information. However to be fully exploited it needs to be “re-processed” and “re-analyzed” in order to extract sound and credible lessons learnt for the future• ROM is a complex system based on a sophisticated methodology that needs highly qualified and professional human resources at every level (from general management to field missions)• This entails that the system could be relatively expensive: a single report cost is estimated now around 9/10,000 euros, raising then the question of the “value added” of the investment in case of project of minor size• Giving the cost and the needed overall structure, the frequency of the missions on same project (re-monitoring) 30 should then be decided with caution
  31. 31. Activity B – Identify theory / reality gaps EuropeAid• Now you have a basic idea of what Monitoring is, how is implemented , what can produce.• Try now to identify theory/reality ‘gaps with the environment of the ENPI CBC programme you are working in• Are there potential changes required and decision making implications for ENPI CBC management?• Think in terms of expected results, capacity building, availability of resources, human resources, inter-country agreement, etc. 31

×