What are Indicators of the Strength of Public Management Systems?ISPMS Background, Purpose and Approach

1. WHAT ARE INDICATORS OF
THE STRENGTH OF PUBLIC
MANAGEMENT SYSTEMS?
ISPMS BACKGROUND,
PURPOSE AND APPROACH

2. Initial motivation behind ISPMS
2
Lots of $ spent without
much results
Ambitious/costly
initiatives not sustained
Gaps exist in coverage of
current indicators across
themes, countries and time
Failure of
prior
indicator
development
efforts
Desire to
provide
overall view
of what
government
looks like in a
country
New thinking
shifted
debate from
form to
function
Better set of
metrics would
improve
project
effectiveness
Many current indicators
focus on de jure aspects
World Bank PSM
Approach 2011
World Bank IEG
2008 report

3. Main purposes of ISPMS
3
Provide a comprehensive and comprehensible set of indicators that:

Strengthen government ownership of reforms


Support donors in meeting Busan commitments


Help governments monitor and evaluate the results of reforms (see an example)
Enable donor decisions to use country systems by providing information about their strengths and
weaknesses (see an example)
Improve project effectiveness



Help set time-bound targets for progress, which is particularly important for results-based lending
approaches (see an example)
Strengthen donor accountability by helping to track impact (see an example)
Improve targeting of reforms


Enable research to build the evidence base on which institutions matter and in which contexts they are
feasible (see an example)
Track regional differences and changes over time

Help identify outliers (countries and systems that deserve a closer look)

4. ISPMS Approach
4
Overarching principle: Draw on existing efforts and avoid duplication
Define a set of
systems
Identify strategy
to measure
conceptually
difficult areas
Define a set of
criteria
Apply criteria to
available
indicators
Build consensus
Are there gaps?
What stories can
we tell?

5. Setting the scope: Defining a set of public
management systems and cross-cutting themes
5
The focus is on upstream institutions at the center of government.
Public
Financial
Management
Procurement
Tax Admin.
Public Admin.
and Civil
Service
Public
Information
Transparency, Accountability & Participation
*The scope of the project may be expanded to
cover more public management systems.

6. ISPMS Criteria
6
ISPMS should meet 4 “utility” criteria…
Criterion
Definition
1. Action-worthy
We know (or strongly believe) that they contribute to results
2. Actionable
They are amenable to government action and project interventions
3. Behavioral
Focus on function, not form, and performance, not design
4. Replicable
Generated transparently and can be reproduced by others
…and a “feasibility” criterion.
Criterion
Definition
Sustainability
Should be amenable to cost-effective and regular collection

7. The process: how the criteria were applied
7
Over 750 indicators
1. Identify potential
datasets
(see datasets)
2. Check if datasets
are feasible,
replicable and
actionable
3. Apply “actionworthy” criteria to
indicators
(see details)
6. TEGs review
results and refine
(see details)
5. Apply
“behavioral”
criteria to indicators
(see details)
4. Apply
“actionable” criteria
to indicators
(see details)
50% aren’t
action-worthy
5% aren’t
actionable
55% aren’t
behavioral
About 100 indicators meet all criteria

8. Existing indicators that meet criteria
8
PFM
Shortlist of most “action-worthy”
Public Adminsitration and Civil Service
Don’t yet meet
feasibility criteria
Tax
Procurement
Public Information Systems*
0
5
10
15
20
Number of indicators that meet ISPMS criteria
* Public Information Systems are a subset of Public Accountability Mechanisms.
25

9. Building consensus: ISPMS/AGI Governance
Global Steering
Group
(Donors, researchers,
international
organizations,
international NGOs)
Technical Expert Groups (World Bank staff, donors, data collectors)
Procurement
Tax Administration
Public Financial
Management
Secretariat (World Bank)
Public Administration and
Civil Services Systems
AGI/ Public
Accountability Mechanism

10. Steering Group Members
10
Organization
German Ministry for Economic Cooperation and Development (BMZ)
UK Department for International Development (DFID)
Australian Government Overseas Aid Programme (AUSAID)
Swedish International Development Cooperation Agency (SIDA)
Canadian International Development Agency
Asian Development Bank
European Commission
Global Integrity
International Budget Partnership
Transparency International
Organization for Economic Cooperation and Development
Government of Bangladesh
World Bank

11. Status and Next steps
11
Phase 1: 2012-2013


Launched Secretariat
Developed dataset of existing
indicators that met the criteria


Download the full list of
indicators and data at:
http://go.worldbank.org/99F3L
CSFR0
Financed by the German Ministry
for Economic Cooperation and
Development (BMZ)
Phase 2: 2014-2015

Proposal with the following
elements is under preparation:

sourcing new indicators and
improving coverage through a
“marketplace”

incubating indicators in areas
that are conceptually difficult to
measure

exploring the feasibility of
administrative data sources

12. 12
Additional details
•
•
Examples of the uses of ISPMS
Details on the process used to identify
existing indicators that meet the criteria

13. Purpose of ISPMS
Strengthen government ownership of reform: example of PEFA
13
Mackie and
Caprio
(2011)
surveyed 11
countries
during 2009
and 2010.

9 out of 11 countries surveyed use PEFA
to take control of their PFM reform
agendas
LICs use to benchmark status of PFM systems,
becoming a centerpiece of dialogue with
Budget Support donors
 MICs use to inform national-level PFM reform
processes


14. Purpose of ISPMS
Support donors in meeting Busan commitments
14



Ongoing efforts to revise indicators monitoring the strength
and use of country financial and procurement systems.
ISPMS work on PFM and procurement will feed into this process
Research on action-worthiness can help deepen learning on
determinants of success

15. Purpose of ISPMS
Improve project effectiveness: setting time-bound targets
What are reasonable
expectations for how
much a wage bill can
be decreased over a 5
year period?
-In most countries, the
wage bill didn’t change
much or even increased
slightly. The top 20% of
countries experienced a
decrease of about 10%.
-Donors and partner
countries can use this
information to set realistic
targets for project
outcomes.
Upper quintile
15

16. Purpose of ISPMS
Improve project effectiveness: hold donors accountable
16
Corporate Scorecard Tier II Indicators : Country Results
Supported by the World Bank
Success rate is derived by looking at ISPMS in projects
Institutions and Governance
Civil Service
No of Countries with
WB Projects
Success Rate
No of Countries with
Tax Administration WB Projects
Success Rate
Public Financial
Management
No of Countries with
WB Projects
Success Rate
44
55%
12
67%
26
73%

17. Purpose of ISPMS
Improve targeting of reforms: providing empirical evidence to test
assumptions
17
Do countries with
higher
transparency of
taxpayer
obligations collect
more taxes [as %
of GDP]?

18. The Process
Step 1-2: Identify potential datasets and check replicability,
feasibility and actionability
18
Afro/Arab/Asian
barometer
Medium-Term
Expenditure
Framework
Dataset
Bertelsman
Transformation
Index
Public Investment
Management
Country Policy and
Institutional
Assessments (CPIA)
BEEPS
Institutional
Profiles Database
Regional tax
administration
datasets
Wage and Bill
Pay Compression
Doing Business
International
Budget Practices
and Procedures
Database
Transparency
International
Global Corruption
Barometer
Quality of
Government
Public
Accountability
Mechanism
Enterprise Surveys
Methodology for
Assessing
Procurement
Systems (MAPS)
OECD
Comparative
Information Series
(Tax)
Public
Expenditures and
Financial
Accountability
Human Resource
Management AGI
World Bank GAC
Diagnostic Surveys
Global Integrity
Indicators
Open Budget
Survey
Evans and Rauch
IMF Government
Financial Statistics/
fiscal
decentralization
IAMTAX
ILO Public Sector
Employment Data
Replicability: Does the dataset have an institutionalized
review/cross-checking mechanisms? (100%)
Feasibility: Have data been collected twice for at least 20
countries? (orange datasets don’t meet feasibility criterion)
Actionable: Are disaggregated data available by country (i.e.
not just composites)? (yellow don’t meet criterion)

19. The Process
Step 3: Apply “action-worthy” criterion
19
1.
At the first stage the ISPMS Secretariat mapped to
CPIA standards

2.
About 50% of indicators didn’t map to CPIA standards
Technical expert group leaders reviewed list and
compared to widely accepted conceptual
framework (e.g. PEFA for PFM, MAPS for
procurement), theory and empirical evidence

20. The Process
Step 4: Apply “actionable” criterion
20

An indicator is “actionable” if it meets any of the
following standards:
 It
has been used in World Bank or other donor projects
before (and is therefore amenable to reform efforts)
 It captures a well specified concept/is specifically
worded

About 5% of indicators didn’t meet actionability
criterion

21. The Process
Step 5: Apply “behavioral” criterion
21

Two steps:
Does the indicator measure de facto changes?
1.

Does the indicator measure performance rather than design features?
2.



De jure indicators measure the contents of government documents (laws,
regulations, etc.), such as the Public Financial Management Law or
Recruitment Regulations (e.g. policy requires an e-procurement portal)
Design = a system is in place (e.g. IT systems and interface for eprocurement portal established)
Performance = to what extent is the system being used (e.g. % of
transactions conducted through portal OR (further down the results chain)
cost savings resulting from use of e-portal)
55% of indicators didn’t meet the behavioral
criterion

22. The Process
Step 6. Technical Expert Groups (TEGS) review and refine
22
PFM TEG
• Action-worthiness reviewed based on evidence, theory, PEFA framework, used to develop shortlist
• List of existing indicators proposed with new indicators to fill gaps
Tax TEG
• Action-worthiness reviewed based on theory; consensus being built via TADAT (work in progress)
• List of existing indicators proposed, new indicators likely to be developed by TADAT
PACS TEG
• Framework proposed based on intrinsic value, currently soliciting comments
• Limited existing indicators meet criteria, expanded set to those that don’t yet meet feasibility criterion
Procurement
TEG
• Currently drafting a discussion paper proposing a framework to determine action-worthiness and the
application of criteria to develop a list of existing indicators
AGI/PAM
TEG
•Suggested expanding indicator set to focus on: Transparency, Rule of Law, Non-executive accountability
institutions, and costs of corruption
•April 2013 in-person workshop to discuss methodologies for measuring difficult concepts
•Analytical paper drafted with discussion of concepts, trade-offs, and action steps for measuring/assessing
governance