This presentation has been prepared by Dr Malik Khalid Mehmood PhD for the Secretariat for use by SC members. It gives an overview of the PEFA program, PEFA Framework and PEFA Secretariat input. It may be shortened or more details may be needed on specific issues depending on audience and available time.
This section presents: The Program its stakeholders the Strengthened Approach (3 pillars0 The components The 6 critical dimensions of the FW Purpose of the FW Implementation modalities and typical stage of an assessment
This section presents: The Program its stakeholders the Strengthened Approach (3 pillars0
Objectives: based on the realization that former diagnostic tools did not enable measurement of change over time, led to conflicting conclusions and to duplication of efforts by individual development agencies with large transactions cost for the partner government Seven agencies : listed in next slide [#5] (diagram) These seven agencies constitute the PEFA Steering Committee. Committee meets semi-annually with rotating chair. OECD-DAC Task Force on PFM (formerly: Joint Venture on PFM): Set up in relation to the Paris Declaration and subsequent Accra Agenda for Action. Linked to Working Party on Aid Effectiveness – the Country Systems Cluster The Strengthened Approach : more details in slide 6
7 partners: 3 international agencies and 4 countries
Strengthened Approach developed 2004-2005 in parallel to work leading to the Paris Declaration 2005. Is completely consistent with the PD but focused only on PFM aspects.
This section presents: The purpose The components The 6 critical dimensions of the FW Implementation modalities and typical stages of an assessment
The framework was developed 2003-2005. Exposure draft February 2004. Was tested in 25 countries by means of desk and live assessments. Wide consultation Special guidance available for use at sub-national level
PFM systems enable governments to deliver on the budgetary outcomes, and in turn the fiscal/expenditure policy targets determined by political processes. These targets are important contributors to achieving the service delivery targets and eventually the high level development outcomes as expressed in the Political Manifestos, Millennium Development Goals, and Poverty Reduction Strategies. Other influencing factors that ultimate influence the achievement of goals include external shocks (such natural disasters, international commodity prices or refugee influx)
3 main budgetary outcomes are: Effective controls of the budget aggregates and management of fiscal risks contribute to maintain aggregate fiscal discipline . Planning and executing the budget in line with government priorities contributes to implementation of government’s objectives ( strategic resource allocation ) . Operational efficiency in managing the use of public resources contributes to efficient service delivery and value for money. [cf. Blue Book p.1] The Framework measures performance by looking at the outputs from the PFM systems , but not the reasons behind performance such as institutional, organizational and human resource aspects of capacity. [It can be considered as an annual medical with your GP. It will point to weaknesses, causes of which are easy to identify and remidy in some cases, whereas in other cases investigation by specialists will be needed] The Framework does not assess the government’s fiscal and expenditure policies, but only the performance of the systems which the government needs in order to implement such policies. It is intended to be widely applicable across countries with different levels of economic development.
The set of indicators includes: 28 Government PIs and + 3 Donors PIs. Each indicator may have 1-4 sub-indicators (or indicator dimensions) Indicators (and their dimensions) are scored on a 4-grade ordinal scale (A, B, C, D) The limited number of indicators make the assessments manageable in terms of resource demand/cost (average cost $125,000 per assessment) Donor indicators add an element of mutual accountability. Are not applicable in countries with negligible aid receipts. An additional indicator ‘HLG1’ required for sub national level government assessments The analytical summary provides the story line – based on the information in chapters 1-4.
The assessment shall look at each of the six dimensions in turn in relation to country context – It shall also look at how the six dimensions relate to each other. There are many ways to slice a cake and the same goes for PFM – the PEFA framework is based on many years of development and also extensive field trials – there is therefore a high degree of confidence that the indicator set captures the key dimensions of PFM performance. This organizing framework is used both for the performance indicator set and for the first level of summarizing the indicator assessments. Policy based budgeting – includes the involvement of the political and sector management levels in the budgeting process. Budget credibility is a result of the performance of the PFM systems i.e. performance on indicators #1-#4 is largely a function of indicators #5-#28 (or #5-31)
The PEFA tool appears as a ‘common good’ – available to any interested party There is no central planning or financing of PEFA asessments
The sheer number of steps in a typical process (in this simplified presentation) There is no “template” - each assessment should be driven by its own needs and requirements. This list illustrates important steps that should be considered in any assessment – but not be seen as a blueprint. This process does not include a government self-assessment and report, which may replace or run parallel to steps 5 and 6. The assessment team (consultants) may be needed in–country on 4 occasions (steps 4, 7, 10, 12). This is in practice - unfortunately - often cut down in order to save on the budget. It may be useful to arrange a validation workshop with government as introduction to the supplementary field mission, step 10.
The early assessments took place in Africa and the Pacific. Many Insular countries in the Pacific or the Caribbean also. Has picked up well in Latin America, Eastern Europe and Central/South Asia. Use is spreading at a slower rate in the Middle East and East Asia. Many sub-national assessments have taken place – especially in large federal states such as Brazil, Argentina, Nigeria, Ethiopia, Pakistan, India and Switzerland. But also municipalities such a Bogota, Dakar, Ouagadougou, Tblisi.
Repeat assessments typically take place after a period of 3 years from the baseline. Some countries have completed 3 comparative PEFA assessments. Most assessment have taken place in low income countries; but many MICs are also applying the Framework, as do a few HICs. HICs: Norway, Switzerland, Kuwait Large MICs: South Africa, Brazil, India, Ukraine, Morocco,, Philippines, Indonesia, Peru are final and publicly available, as are India (Himachal Pradesh) Thailand, Turkey and Kazakhstan are final but non public. Russian federation is in draft version but a final version will not be prepared.
Governments have expressed appreciation of the objective nature of the scoring criteria of indicators. Government is able to do its own assessment or challenge assessments prepared by others. Writing a fully evidenced assessment report is a challenge to governments in many countries where skilled professional staff is in very short supply. Where government leads, consultants are often drawn upon to write the report.
A PFM Reform program should be formulated from recommendations arising from an investigation having been undertaken to establish the weaknesses in the performance of PFM systems, processes and institutions. This high level performance overview of the PFM system can be obtained from the application of the PEFA framework. The diagram illustrate the coverage of the PFM report in the PFM reform cycle and the elements which it covers. The PFM report will embrace the high level performance overview and identify the main PFM weaknesses. This is a useful input to a reform dialogue with the aim of deciding reform priorities. Detailed investigation of underlying causes for poor performance may be needed in the selected priority areas for reform. Specialized ‘drill-down’ assessment tools exist for many sub-systems and will help in providing inputs needed to determine a detailed action plan. When reform action plans have been implemented for say 3 years it may be time to undertake a repeat assessment and measure change.
Information from 25 countries (Impact study and External Evaluation 2010/11) indicate widespread use of PEFA assessments for reform dialogue by both governments and development agencies. Government ownership of the assessment leads to government use of the report. Governments will typically need time digest the assessment findings, have an internal dialogue and formulate a position before they are ready to discuss with development partners. Indicators are often used to set medium – long term targets for performance improvements. However, other inputs are needed to formulate the reforms – political economy analysis, assessment of capacity factors, ongoing reforms. Reform does not have to focus any or all of the low scoring PFM features. A ‘D’-scored feature may not necessarily be of major significance in a particular country environment. Guidance on an approach to sequencing PFM reforms is being developed.
other development agencies using PEFA assessments : regional banks (AfDB, IADB), and bilateral agencies e.g. Netherlands, Germany, Sweden, CIDA (Canada), AusAid etc. internal guidelines : WB, EU, DFID, France. DFID specifically require that PEFA Secretariat has contributed a report review, before the report is used for further work. PEFA assessments – the indicator ratings – provide essential inputs to many processes determine aid allocations by development agencies or choice of aid modality. In most cases, however, other inputs are also required so that the impact of PEFA is diluted. This is reassuring since the incentives for assessment manipulation would be too great.
The monitoring report 2010 has proven that comparison of indicator rating over time is quite robust when the repeat assessment specifically was intended to compare to the baseline. It is easy to see where the indicator scores cannot be compared due to the baseline and/or repeat assessment assigning a ‘not used’, ‘not applicable’ or ‘not scored for lack of information’ to an indicator. Country comparison requires caution. Best to compare to reports by reading them in parallel and compare indicator by indicator. Many users wish to aggregate indicator ratings. Weighting of indicators and of countries pose problems. No method is perfect. PEFA program does not provide any method. Country contexts are different, structure of public sector may be different, different definitions may apply, assessment time may differ etc. The Program has wished to calculate country aggregate ratings and have a ’beauty contest’.
Three types of diagnostic instruments: broad PFM systems tools, speicalised drill-down tools, fiduciary risk assessment instruments PEFA has replaced HIPC assessments, most CFAAs and EC compliance audits . Broad based, overlapping assessment instruments : PEFA, OBI, Fiscal Transparency ROSC, Commonwealth PFM self asst tool kit etc. new ‘drill-down’ PFM tools : tax administration (EC tax administration blueprints) , procurement (MAPS), debt management (DeMPA), etc. Fiduciary instruments belong to individual development agencies.
PEFA Secretariat Located in Washington DC. Attached to the World Bank, which holds its funding in trust Head of Secretariat reports directly to the PEFA Steering Committee Staff of six full time
The Secretariat’s clients include: Development agency staff, Partner government officials, private sector consultants, training institutes, researchers
The first change to Framework was issued January 2011 – three indicators revised Clarification on the indicators are issued periodically. Quality review is a core function. Secretariat has limited capacity to deliver training. But often supporting others – e.g. co-facilitating More than 100 assessment reports are linked to the PEFA website by hyperlinks. No direct posting because the secretariat does not own any of the reports.
Semi-annual status lists are provides a full overview of assessments by country according to stage (planned, ongoing, full draft report, final report, Public report) and stakeholders involved (lead agency and others). Four major monitoring reports issue up to June 2011 (Early Experience Report, MR07, MR09, MR10) Ad hoc surveys include use of PEFA for internal procedures and reports on publication of assessments Secretariat provides advice to other developers of similar tools. Sharing experience. Encouraging clear linkages between tools. Several research reports have been undertaken on the basis of PEFA scoring data. Secretariat capacity to undertake research is very limited. Also used for evaluation studies of general budget support and of technical assistance to PFM reform
The Secretariat aims at providing comments on reports with ten business days of receiving request for review. For concept notes and terms of reference it is usually provided within a week. When acting as Reviewers the PEFA Secretariat does not undertake an audit to establish the authenticity or correctness of the data given in the report. Reliance is placed on the integrity and professional competence of the consultants or team performing the assessment. However, where conflicting data or evidence becomes apparent then this will be pointed out and clarification sought. The Secretariat’s review is undertaken by (i) appraising the application of the PFM performance indicators and (ii) considering whether the back-ground information as set out in chapters 1 & 2 of the framework has been given ; and whether the summary assessment brings out a clear message consistent with the indicator analysis. Important to have other quality reviewers with specific country knowledge – so that country data may be verified. Desirable to have reviewers from other agencies than the lead agency. Secretariat undertakes follow-up reviews where the response to each of the earlier provided comments by the Secretariat is evaluated to determine if it is adequate.
Pefa presentation by dr malik khalid mehmood ph_d
The PEFA Program- an Overview An Institutional Presentation. For use by SC members Prepared by Dr Malik Khalid Mehmood PhD
Content- Program Overview- The PEFA Framework- Use & Impact- The PEFA Secretariat 2
What is the PEFA program?l PEFA = Public Expenditure & Financial Accountability Programl Objective: § Results orientation in development of PFM systems § Harmonization of PFM analytical workl Established: § Established in 2001 by seven agencies. § Working in tandem with the OECD-DAC Task Force on PFM § Contributing to development effectiveness through: The Strengthened Approach to Supporting PFM Reform 4
The Strengthened Approach toSupporting PFM Reform1. A country-led PFM reform program § including a strategy and action plan reflecting country priorities; implemented through government structures2. A donor coordinated program of support § covering analytical, technical and financial support3. A common information pool § based on a framework for measuring performance and monitoring results over time § i.e. the PFM Performance Measurement Framework 6
The PEFA FrameworkPFM Performance Measurement Framework• Better known as ‘the PEFA Framework’• The ‘flagship’ product of the PEFA Program• Launched in June 2005Designed to measure performance of national PFM systems• Applicable to countries and different stages of development• Also applicable for Sub-National government systems 8
PFM Links to Development GoalsOther influencing factors MDGs, PRSP, Political Manifesto Dev Goals Budget deficit, Sector Service allocations, Investment, Delivery Goals Debt ratio, Tax burden etc Fiscal discipline, Budgetary Outcomes Strategic allocation, Fiscal/Exp Policy Goals Operational efficiency PEFA Framework PFM system performance
Purpose of the PEFA FrameworkObjective• To determine if a country has the tools to deliver three main budgetary outcomes (aggregate fiscal discipline; strategic resource allocation; efficient service delivery)?It provides:• a high level overview of all aspects of a country’s PFM systems performance• includes revenue, expenditure, procurement, financial assets/ liabilitiesIt does not provide:• Assessment of underlying causes for good / poor performance• An assessment of government fiscal/expenditure policies 10
Components of the PEFA Frameworkl A standard set of high level PFM performance indicators (PIs) § 31 performance indicators (with 76 sub-indicators) § Grouped in six critical dimensions § A compromise between simplicity and comprehensivenessl A standard report format § the PFM Performance Report (PFM-PR) § developed to provide country background, evidence on the indicators and an integrated, analytical summary. – 11
Dimensions of PFM system performance External scrutiny Budget credibility: & audit: Is the budget realistic, Are there effective & implemented as arrangements for intended? 6 critical scrutiny of public dimensions finances & follow up by the executive? of PFM Comprehensiveness system and transparency : performance Accounting, recording Are the budget & fiscal risk oversight comprehensive, & & reporting:is fiscal & budget information Are adequate records & accessible to the public? information produced, maintained & disseminated to meet decision-making, control, management & Policy-based Predictability & control in reporting purposes? budgeting: budget execution: Is the budget prepared Is the budget implemented in a with due regard to predictable manner & is control & government policy? stewardship exercised in the collection & use of public funds? 12
Implementation modalitiesl Country focus and decision § Application of the PEFA Framework is entirely decentralized to the country level (if, when, how to use Framework)l Inclusiveness § All stakeholders can be involved and any agency can in principle undertake any role in its implementation (managing, financing, quality review) § The Framework does not ‘belong’ to any single organization § Due process is crucial to acceptance and use - Joint work is encouraged 13
Stages in a typical process0. Agree the intention to undertake a PEFA based assessment 1. Agree purpose, scope and stakeholder roles 2. Prepare TOR 3. Mobilize assessment team 4. Introduction workshop for stakeholders 5. Review of existing information 6. Inception Report 7. Main field work 8. 1st Draft Report 9. Quality Review 10. Supplementary field work 11. Draft Final Report 12. Presentation seminar 13. Final report14. Use of the report for reform dialogue 14
PEFA Framework adoptionl Very good progress – globally § 220+ assessments substantially completed, covering 120+ countries. § Since 2010, mostly Repeat assessments and Sub-National assessmentsl High country coverage in many regions § Africa and Caribbean 90% of countries § Latin America, Eastern Europe, Asia, Pacific 50-80%l Used in many Middle Income Countries § Upper MICs: Brazil, Turkey, Belarus, Russian Federation, South Africa § Lower MICs: India, Kazakhstan, Ukraine, Morocco, Egypt, Thailand, Philippines, Indonesia, Colombia, Peru 17
Stakeholder involvementl Government leadership/self-assessment § Active government participation in assessment is key to the use in reform policy dialogue § Full self-assessment may be very demanding on govt. capacity § Increasing frequency of Self-assessment with expert assistance and external validation – but still a minority of casesl Development Agencies § About 25 development banks and donor agencies involved (leading, financing or in reference group) § World Bank and EU has taken the lead in 85% of all assessments § Increasing frequency of joint work and partner inclusivenessl 18
Coverage of PFM-PR in Reform Cycle Implement PFM reforms High level Formulate performance PFM reform overview program PFM-PR Identify main Recommend Recommend Identify main PFM reform PFM PFM PFM reform weaknesses measures weaknesses measures Investigate underlying causes 19
PEFA contribution to results orientationl Frequently used as basis for reform dialogue § Useful in deciding reform priorities § Ownership means government decisions on priorities § Indicators used to set targets for reform program results but: § PEFA assessment is one of several inputs § Do not use Indicator scores simplistically § Complementary analysis to PEFA required to identify root causes l Measuring performance change § Results from 33 repeat assessments indicate pace of progress in different areas of PFM – positive global trend but wide variation among PFM features l Peer learning § Used as basis for discussions in regional groups of government PFM experts (e.g. Eastern Europe, West Africa, Caribbean) 20
Use of PEFA for Development Agencies’Internal Processes§ Used by all seven PEFA partners for internal processes (GBS/DPL, FRA, CPIA, Debt Limits etc)§ Similarly used by several other development agencies§ Increasingly, PEFA assessments are directly mentioned in internal guidelines as key or preferred input§ PEFA assessment information is rarely considered the only source of information to a process§ Most of the processes have financial consequences for partner countries – though the impact of PEFA is usually indirect. 21
Assessment comparisonsPEFA Framework developed to measure progress over time in one country• ‘Summary assessment’ provides overview of strengths & weaknesses as basis for reform prioritization in a country• Comparison of assessments over time is robust where repeat assessments are explicitly comparing to baseline Country comparisons• Comparing scores alone can be misleading• Must be approached cautiously - ensure comparison of like with like• Aggregation - No method given for arriving at single measure of ‘overall performance’ – no attempt to create ‘league table’•• 22
Harmonization of PFM diagnostic toolsl Common information pool being created § Considerable use of PEFA assessments for many different purposes § Several earlier instruments replaced by PEFAl Transactions costs in PFM diagnostics appear not to decline, due to: § Still several overlapping assessment instruments § Many uncoordinated fiduciary assessments § Proliferation of new ‘drill-down’ PFM tools § Few countries have multi-year plans for analytical work 23
Role of the PEFA Secretariatl Neutral body § Supporting / advising any user of the Framework § does not represent a particular interest § does not undertake or finance assessments § support services are free of charge – 25
Activities of the PEFA Secretariat (1) l Custodian of the PEFA Framework § Development and maintenance of the Framework § Issues guidance notes and clarifications l Quality reviews of assessments § Both at concept note and report stage, on requestl Training § Develops and shares training materials § Supports training institutes in delivering PEFA courses § Delivers training selectively, mainly on a regional basisl Dissemination § Conference presentations, PFM blogs, PEFA Newsflash § Sharing PEFA assessment reports through website 26
Role of the PEFA Secretariat (2)l Monitoring § Semi-annual updates of PEFA assessment status list § Periodic monitoring reports § Ad hoc surveysl Promotes harmonization in assessment of PFM systems § Working with OECD-DAC Task Force on PFM § Stocktaking study of PFM diagnostic instruments § Supports developers of ‘drill-down’ assessment toolsl Support PFM research § Availability of PEFA indicators database 27
PEFA Secretariat quality review• On request, free of charge, quick feedback (10 business days)• For Concept Notes/TOR and as well as Assessment Reports• Appraises adequacy of background info (sections 1, 2 & 4)• Appraises application of performance indicators (section 3)• Review of each indicator: correctly interpreted, sufficient evidence, scoring method correctly applied?• Considers whether summary assessment brings out clear message consistent with indicator analysis and background• Follow-up reviews – evaluates responses to the initial reviewl 28
Stay in touch with PEFAl Thank you for your kind attention.l Dr Malik Khalid Mehmood, Chief Technical AdvisorEvaluation, Assessment and Monitoring Methodology