Use of PEFA assessments ICGFM Conference Miami – May 21, 2009 PEFA Secretariat

Harmonize information needs Common information pool Inform PFM reform formulation and priorities Compare to and learn from peers Monitor results of reform efforts What can countries use the PEFA Framework for?

Harmonize information needs

Common information pool

Inform PFM reform formulation and priorities

Compare to and learn from peers

Monitor results of reform efforts

Harmonize information needs Inform PFM reform formulation and priorities Compare to and learn from peers Monitor results of reform efforts What can countries use the PEFA Framework for?

Harmonize information needs

Inform PFM reform formulation and priorities

Compare to and learn from peers

Monitor results of reform efforts

PEFA reports for reform formulation (1) PEFA report is one – out of several - inputs Identification of main strengths and weaknesses – and potential impact on budgetary outcomes Other factors: political economy, institutional, culture, constitution/legal, resources, capacity at entry Ownership means government decisions on priorities Government to consider all factors in deciding priorities Reform dialogue with donors to allow ample space Do not use Indicator scores simplistically A low score is not sufficient justification for high priority reform Other factors: relative importance of subject, complexity/timeframe for improvement, interdependence with other elements, weakest links

PEFA report is one – out of several - inputs

Identification of main strengths and weaknesses – and potential impact on budgetary outcomes

Other factors: political economy, institutional, culture, constitution/legal, resources, capacity at entry

Ownership means government decisions on priorities

Government to consider all factors in deciding priorities

Reform dialogue with donors to allow ample space

Do not use Indicator scores simplistically

A low score is not sufficient justification for high priority reform

Other factors: relative importance of subject, complexity/timeframe for improvement, interdependence with other elements, weakest links

Implement PFM reforms Recommend PFM reform measures Identify main PFM weaknesses High level performance overview Coverage of PFM Performance Report in the PFM Reform Cycle Investigate underlying causes Formulate PFM reform program Identify main PFM weaknesses Recommend PFM reform measures PFM-PR

PEFA reports for reform formulation (2) Complementary analysis to PEFA required Detailed analysis of underlying causes needed for formulation of detailed action plan Limit such analysis to priority areas Drill-down tools – some exist, others under development PEFA is preparing guidance on use of PEFA reports in reform formulation Stock-taking of conceptual approaches used Country case studies illustrating good practice

Complementary analysis to PEFA required

Detailed analysis of underlying causes needed for formulation of detailed action plan

Limit such analysis to priority areas

Drill-down tools – some exist, others under development

PEFA is preparing guidance on use of PEFA reports in reform formulation

Stock-taking of conceptual approaches used

Country case studies illustrating good practice

Purposes of Standard Diagnostic Tools

Findings of Norad-managed self-assessment presented to OECD-DAC in December 2007 The assessment showed low scores for seven areas of PFM system performance Ministry of Finance reaction: Weaknesses in procurement practices and follow-up to external audit findings need to be addressed. Three areas of low scoring not considered priority at present (Multi-year program/sector budgeting, Limited extent of internal audit, no consolidated overview of risks from autonomous agencies and public corporations) Two indicators scored low but are municipal responsibilities; central government will not get involved. Country case - Norway

Findings of Norad-managed self-assessment presented to OECD-DAC in December 2007

The assessment showed low scores for seven areas of PFM system performance

Ministry of Finance reaction:

Weaknesses in procurement practices and follow-up to external audit findings need to be addressed.

Three areas of low scoring not considered priority at present (Multi-year program/sector budgeting, Limited extent of internal audit, no consolidated overview of risks from autonomous agencies and public corporations)

Two indicators scored low but are municipal responsibilities; central government will not get involved.

Harmonize information needs Inform PFM reform formulation, priorities Compare to and learn from peers Monitor results of reform efforts What can countries use the PEFA Framework for?

Harmonize information needs

Inform PFM reform formulation, priorities

Compare to and learn from peers

Monitor results of reform efforts

PEFA reports for country comparison (1) PEFA Framework developed for in-country use ‘ Summary assessment’ provides nuanced overview of strengths and weaknesses as basis for reform prioritization No method given for arriving at one measure for ‘overall performance level’ Wide interest in country comparison using aggregation Researchers - learning on determining factors Donors - aid allocations Governments - peer learning Aggregation requires three decisions Conversion from ordinal to numerical scale Weighting of indicators (generally and by country) Weighting of countries (for country cluster analysis)

PEFA Framework developed for in-country use

‘ Summary assessment’ provides nuanced overview of strengths and weaknesses as basis for reform prioritization

No method given for arriving at one measure for ‘overall performance level’

Wide interest in country comparison using aggregation

Researchers - learning on determining factors

Donors - aid allocations

Governments - peer learning

Aggregation requires three decisions

Conversion from ordinal to numerical scale

Weighting of indicators (generally and by country)

Weighting of countries (for country cluster analysis)

PEFA reports for country comparison (2) No scientifically correct or superior basis for deciding conversion and weights exists Each user takes those decisions on individual opinion PEFA program will not endorse any particular method Preferred method of country comparison: A nuanced comparison of two assessment reports Consider country context, ensure comparison of like with like In case aggregation is desired: Be transparent on aggregation methods used Discuss reasons for choice Sensitivity analysis to illustrate impact on findings

No scientifically correct or superior basis for deciding conversion and weights exists

Each user takes those decisions on individual opinion

PEFA program will not endorse any particular method

Preferred method of country comparison:

A nuanced comparison of two assessment reports

Consider country context, ensure comparison of like with like

In case aggregation is desired:

Be transparent on aggregation methods used

Discuss reasons for choice

Sensitivity analysis to illustrate impact on findings

Harmonize information needs Inform PFM reform formulation and priorities Compare to and learn from peers Monitor results of reform efforts What can countries use the PEFA Framework for?

Harmonize information needs

Inform PFM reform formulation and priorities

Compare to and learn from peers

Monitor results of reform efforts

Repeat Assessments As at March 2009, thirteen ‘real’ repeat assessment undertaken Expected to become common from 2008/2009 onward i.e. three years after the first series of baseline assessments

As at March 2009, thirteen ‘real’ repeat assessment undertaken

Expected to become common from 2008/2009 onward i.e. three years after the first series of baseline assessments

What we want to determine? Specific changes in system performance What has changed? How much?

Specific changes in system performance

What has changed?

How much?

Comparison of Indicator Scores Indicator scores will provide a crude overview of changes over time, but … Dimensions may change differently Performance may not always change enough to change the score (use of arrow) So more detailed explanation is required

Indicator scores will provide a crude overview of changes over time, but …

Dimensions may change differently

Performance may not always change enough to change the score (use of arrow)

So more detailed explanation is required

Other Possible Reasons for Change in Scores Changes in definitions Improved availability of or access to information Different information sampling & aggregation Scoring methodology mistakes in previous assessment

Changes in definitions

Improved availability of or access to information

Different information sampling & aggregation

Scoring methodology mistakes in previous assessment

Reporting on Progress Made Explain all factors that impact a change in rating indicator-by-indicator Identify the performance change Ensure that any reader can track the change from the previous assessment – what performance change led to the change in a rating

Explain all factors that impact a change in rating indicator-by-indicator

Identify the performance change

Ensure that any reader can track the change from the previous assessment – what performance change led to the change in a rating

Reporting on Progress Made Indicator Score 2005 Score 2007 Performance change Other factors PI-1 C B Performance appears improved based on 2005: last deviations 6%, 11%, 18% 2007: 5%, 11%, 6% Not clear if all external project funds were excluded from data for 2005 assessment but probably insignificant issue. PI-4 (i) A C Performance change is uncertain, despite reported arrears increase from 1% in 2005 to 6% in 2007. 2005 assessment used data on pending payment orders only, not overdue invoices.

Organizational Requirements Performance tracking to be clearly reflected in the TOR Use of same assessment team desirable but rarely possible The lead agency of the previous assessment, if different, should assist with access to previous assessors’ notes … and be part of the reference group for the repeat exercise

Performance tracking to be clearly reflected in the TOR

Use of same assessment team desirable but rarely possible

The lead agency of the previous assessment, if different, should assist with access to previous assessors’ notes

… and be part of the reference group for the repeat exercise

PEFA incorporated in PFM reform monitoring system baseline 2005, repeat assessment 2007, planned follow-up 2010 Important performance improvements 2005-2007 in budget execution revenue administration, cash management, internal controls Improvements resulted from: Reforms already well under way in 2005 (e.g. IFMIS/SISTAFE and revenue administration) Small managerial/admin changes (including quick-wins identified on the basis of 2005 assessment) New reform initiatives in pay-roll control (identified from the 2005 assessment as an important neglected area of reform) Country case - Mozambique

PEFA incorporated in PFM reform monitoring system

baseline 2005, repeat assessment 2007, planned follow-up 2010

Important performance improvements 2005-2007 in budget execution

revenue administration, cash management, internal controls

Improvements resulted from:

Reforms already well under way in 2005 (e.g. IFMIS/SISTAFE and revenue administration)

Small managerial/admin changes (including quick-wins identified on the basis of 2005 assessment)

New reform initiatives in pay-roll control (identified from the 2005 assessment as an important neglected area of reform)

Thank you for your attention