ISPMS Background, Purpose and Approach


Published on

What are Indicators of the Strength of Public Management Systems? ISPMS Background, Purpose and Approach

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • This is a tentative formulation intended to convey mechanisms which contribute to holding public actors to account directly, by providing information about their behaviors or their performance to the public, not mediated through public sector professional bodies, technical institutions or formal oversight bodies. The notion of providing information to assist in public accountability is consistent with the Demand for Good Governance approach of the Bank, but it excludes independent accountability institutions (IAIs) such as parliament, judiciary, ombudsman offices, corruption commissions, vigilance agencies, etc. that act as „checks and balance‟ institutions within the state.
  • ISPMS Background, Purpose and Approach

    2. 2. Initial motivation behind ISPMS 2 Lots of $ spent without much results Ambitious/costly initiatives not sustained Gaps exist in coverage of current indicators across themes, countries and time Failure of prior indicator development efforts Desire to provide overall view of what government looks like in a country New thinking shifted debate from form to function Better set of metrics would improve project effectiveness Many current indicators focus on de jure aspects World Bank PSM Approach 2011 World Bank IEG 2008 report
    3. 3. Main purposes of ISPMS 3 Provide a comprehensive and comprehensible set of indicators that:  Strengthen government ownership of reforms   Support donors in meeting Busan commitments   Help governments monitor and evaluate the results of reforms (see an example) Enable donor decisions to use country systems by providing information about their strengths and weaknesses (see an example) Improve project effectiveness    Help set time-bound targets for progress, which is particularly important for results-based lending approaches (see an example) Strengthen donor accountability by helping to track impact (see an example) Improve targeting of reforms   Enable research to build the evidence base on which institutions matter and in which contexts they are feasible (see an example) Track regional differences and changes over time  Help identify outliers (countries and systems that deserve a closer look)
    4. 4. ISPMS Approach 4 Overarching principle: Draw on existing efforts and avoid duplication Define a set of systems Identify strategy to measure conceptually difficult areas Define a set of criteria Apply criteria to available indicators Build consensus Are there gaps? What stories can we tell?
    5. 5. Setting the scope: Defining a set of public management systems and cross-cutting themes 5 The focus is on upstream institutions at the center of government. Public Financial Management Procurement Tax Admin. Public Admin. and Civil Service Public Information Transparency, Accountability & Participation *The scope of the project may be expanded to cover more public management systems.
    6. 6. ISPMS Criteria 6 ISPMS should meet 4 “utility” criteria… Criterion Definition 1. Action-worthy We know (or strongly believe) that they contribute to results 2. Actionable They are amenable to government action and project interventions 3. Behavioral Focus on function, not form, and performance, not design 4. Replicable Generated transparently and can be reproduced by others …and a “feasibility” criterion. Criterion Definition Sustainability Should be amenable to cost-effective and regular collection
    7. 7. The process: how the criteria were applied 7 Over 750 indicators 1. Identify potential datasets (see datasets) 2. Check if datasets are feasible, replicable and actionable 3. Apply “actionworthy” criteria to indicators (see details) 6. TEGs review results and refine (see details) 5. Apply “behavioral” criteria to indicators (see details) 4. Apply “actionable” criteria to indicators (see details) 50% aren’t action-worthy 5% aren’t actionable 55% aren’t behavioral About 100 indicators meet all criteria
    8. 8. Existing indicators that meet criteria 8 PFM Shortlist of most “action-worthy” Public Adminsitration and Civil Service Don’t yet meet feasibility criteria Tax Procurement Public Information Systems* 0 5 10 15 20 Number of indicators that meet ISPMS criteria * Public Information Systems are a subset of Public Accountability Mechanisms. 25
    9. 9. Building consensus: ISPMS/AGI Governance Global Steering Group (Donors, researchers, international organizations, international NGOs) Technical Expert Groups (World Bank staff, donors, data collectors) Procurement Tax Administration Public Financial Management Secretariat (World Bank) Public Administration and Civil Services Systems AGI/ Public Accountability Mechanism
    10. 10. Steering Group Members 10 Organization German Ministry for Economic Cooperation and Development (BMZ) UK Department for International Development (DFID) Australian Government Overseas Aid Programme (AUSAID) Swedish International Development Cooperation Agency (SIDA) Canadian International Development Agency Asian Development Bank European Commission Global Integrity International Budget Partnership Transparency International Organization for Economic Cooperation and Development Government of Bangladesh World Bank
    11. 11. Status and Next steps 11 Phase 1: 2012-2013   Launched Secretariat Developed dataset of existing indicators that met the criteria   Download the full list of indicators and data at: CSFR0 Financed by the German Ministry for Economic Cooperation and Development (BMZ) Phase 2: 2014-2015  Proposal with the following elements is under preparation:  sourcing new indicators and improving coverage through a “marketplace”  incubating indicators in areas that are conceptually difficult to measure  exploring the feasibility of administrative data sources
    12. 12. 12 Additional details • • Examples of the uses of ISPMS Details on the process used to identify existing indicators that meet the criteria
    13. 13. Purpose of ISPMS Strengthen government ownership of reform: example of PEFA 13 Mackie and Caprio (2011) surveyed 11 countries during 2009 and 2010.  9 out of 11 countries surveyed use PEFA to take control of their PFM reform agendas LICs use to benchmark status of PFM systems, becoming a centerpiece of dialogue with Budget Support donors  MICs use to inform national-level PFM reform processes 
    14. 14. Purpose of ISPMS Support donors in meeting Busan commitments 14    Ongoing efforts to revise indicators monitoring the strength and use of country financial and procurement systems. ISPMS work on PFM and procurement will feed into this process Research on action-worthiness can help deepen learning on determinants of success
    15. 15. Purpose of ISPMS Improve project effectiveness: setting time-bound targets What are reasonable expectations for how much a wage bill can be decreased over a 5 year period? -In most countries, the wage bill didn’t change much or even increased slightly. The top 20% of countries experienced a decrease of about 10%. -Donors and partner countries can use this information to set realistic targets for project outcomes. Upper quintile 15
    16. 16. Purpose of ISPMS Improve project effectiveness: hold donors accountable 16 Corporate Scorecard Tier II Indicators : Country Results Supported by the World Bank Success rate is derived by looking at ISPMS in projects Institutions and Governance Civil Service No of Countries with WB Projects Success Rate No of Countries with Tax Administration WB Projects Success Rate Public Financial Management No of Countries with WB Projects Success Rate 44 55% 12 67% 26 73%
    17. 17. Purpose of ISPMS Improve targeting of reforms: providing empirical evidence to test assumptions 17 Do countries with higher transparency of taxpayer obligations collect more taxes [as % of GDP]?
    18. 18. The Process Step 1-2: Identify potential datasets and check replicability, feasibility and actionability 18 Afro/Arab/Asian barometer Medium-Term Expenditure Framework Dataset Bertelsman Transformation Index Public Investment Management Country Policy and Institutional Assessments (CPIA) BEEPS Institutional Profiles Database Regional tax administration datasets Wage and Bill Pay Compression Doing Business International Budget Practices and Procedures Database Transparency International Global Corruption Barometer Quality of Government Public Accountability Mechanism Enterprise Surveys Methodology for Assessing Procurement Systems (MAPS) OECD Comparative Information Series (Tax) Public Expenditures and Financial Accountability Human Resource Management AGI World Bank GAC Diagnostic Surveys Global Integrity Indicators Open Budget Survey Evans and Rauch IMF Government Financial Statistics/ fiscal decentralization IAMTAX ILO Public Sector Employment Data Replicability: Does the dataset have an institutionalized review/cross-checking mechanisms? (100%) Feasibility: Have data been collected twice for at least 20 countries? (orange datasets don’t meet feasibility criterion) Actionable: Are disaggregated data available by country (i.e. not just composites)? (yellow don’t meet criterion)
    19. 19. The Process Step 3: Apply “action-worthy” criterion 19 1. At the first stage the ISPMS Secretariat mapped to CPIA standards  2. About 50% of indicators didn’t map to CPIA standards Technical expert group leaders reviewed list and compared to widely accepted conceptual framework (e.g. PEFA for PFM, MAPS for procurement), theory and empirical evidence
    20. 20. The Process Step 4: Apply “actionable” criterion 20  An indicator is “actionable” if it meets any of the following standards:  It has been used in World Bank or other donor projects before (and is therefore amenable to reform efforts)  It captures a well specified concept/is specifically worded  About 5% of indicators didn’t meet actionability criterion
    21. 21. The Process Step 5: Apply “behavioral” criterion 21  Two steps: Does the indicator measure de facto changes? 1.  Does the indicator measure performance rather than design features? 2.    De jure indicators measure the contents of government documents (laws, regulations, etc.), such as the Public Financial Management Law or Recruitment Regulations (e.g. policy requires an e-procurement portal) Design = a system is in place (e.g. IT systems and interface for eprocurement portal established) Performance = to what extent is the system being used (e.g. % of transactions conducted through portal OR (further down the results chain) cost savings resulting from use of e-portal) 55% of indicators didn’t meet the behavioral criterion
    22. 22. The Process Step 6. Technical Expert Groups (TEGS) review and refine 22 PFM TEG • Action-worthiness reviewed based on evidence, theory, PEFA framework, used to develop shortlist • List of existing indicators proposed with new indicators to fill gaps Tax TEG • Action-worthiness reviewed based on theory; consensus being built via TADAT (work in progress) • List of existing indicators proposed, new indicators likely to be developed by TADAT PACS TEG • Framework proposed based on intrinsic value, currently soliciting comments • Limited existing indicators meet criteria, expanded set to those that don’t yet meet feasibility criterion Procurement TEG • Currently drafting a discussion paper proposing a framework to determine action-worthiness and the application of criteria to develop a list of existing indicators AGI/PAM TEG •Suggested expanding indicator set to focus on: Transparency, Rule of Law, Non-executive accountability institutions, and costs of corruption •April 2013 in-person workshop to discuss methodologies for measuring difficult concepts •Analytical paper drafted with discussion of concepts, trade-offs, and action steps for measuring/assessing governance