This is compared to IEG ICR outcome data – compared to ieg looking at cpia
Other variables controlled forPSMNarrow==1 status==Not Free status==Partially Free PSMNarrow==1 & status==Not Free PSMNarrow==1 & status==Partially Free DPI Share of Programmatic Parties ICRG Bureaucracy Quality Rating [0-6] GDP per Capita Growth [annual %] GDP per Capita [const. PPP 2005 USD] Net ODA received (% of GNI) Committed Amount Supervision Costs [thousand USD] Lending (~Preparation) Costs [thousand USD] Project Duration (Appr. to Rev. Clos.) [days] Evaluation Lag (Rev. Clos. to Eval.) [days] Total share of PS components in project [%]
See paragraphs 65 and 66
See paragraphs 56 to 58
See paragraphs 59-61
PME Work in the Bank
1 Nick Manning Jurgen BlumMay 3nd, 2012
• The numbers suggest quantity more than quality• But we have some rebuttals• But, between us, we have some other problems• Where to from here?
Percent of IDA and IBRD countries with CPIA improvements 80 70 60 50 40 Any relevant lending no relevant lending 30 20 10 0CPIA 13 - Quality of budget CPIAfinancial managementadministration revneue mobilization and 15 - Quality of CPIA 14 - Efficiency of - Corruption, transparency and accountability public CPIA 16
Tax projects are the only projects that consistently perform above average(compared to non-PSM projects) at statistically significant levels (5 percentconfidence level) 20.00% 15.00% 10.00% 15.40%** 5.00% 4.00% 2.00% 0.00% -1.00% -7.00%* -5.00% -9.00%* -10.00% -15.00%
Tax is the only contents characteristic that remains a statistically significantpredictor of project performance (IEG outcome ratings), when controlling forother contextual and management variables Percentage point change in project success rates associated with a 10 percent increase in PS thematic share 12 10 8 6 9.74** 4 2 0.883 0.42 1.34 0 -0.963 -1.99 -2 -4
Compared to non-PSM projects, (upstream) PSM projects are particularlytargeted to :(i) the Africa (AFR, 30.5 percent) and Latin America and the Caribbean (LAC, 30.5 percent) region (see Table 12) and(ii) towards countries with significantly lower GDP per capita growth (1.6 percent for PSM versus 3.4 percent for non-PSM projects);(iii) with significantly higher aid flows than non-PSM projects (11.4 versus 6.3 percent); (iv) with a slightly higher degree of political and civil liberties (Freedom House ratings of political and civil liberties are about 0.25 to 0.3 points lower for countries with PSM projects, on a 7-point scale, see Table 10).(iv) By contrast, there is no conclusive evidence of a “needs-based” upstream PSM project targeting towards countries with lower initial administrative capability.
We know what We are formal uncertain changes we about what can make needs to happen in between!We know the sorts ofresults that we would like to see
OECD has a set of propositions around thesebehaviors/upstream performance competencies:1. Workforce planning and management2. Core values3. Staff performance and capacity4. Cooperation between levels of governmentThe HRM AGI work also had propositions:1. Attract and retain required staff2. Depoliticized, meritocratic HRM practices3. Performance-focusing HRM practices4. Fiscally sustainable wage bill5. Ethical behavior6. Effective working relationships with other cadres
1. Management of operations within the core administration a. HRM practices within the core administration are structured around education background and merit criteria clearly and publicly defined. [Q15a1] b. There are clear mechanisms within the public administration to ensure accountability of staff and avoid conflict of interest and abuse of power. [15a2, Q16c1, Q16c2] c. There is an effective arbitration system for violations of the HRM regime and its decisions are made publicly available in a timely manner. [Q15a3, Q16b1] d. The employment regime for the core administration is effective in attracting, retaining and motivating competent staff. [Q15a4, Q15a5, Q15a6]2. Quality management in policy and regulatory management a. The policy process is transparent and ensures credibility in social and sectoral policy pronouncements. [Q15b1] b. There is quality and credibility in the actions of sector regulators. [Q15b2, Q15b4] c. Sector regulators are held accountable for their decisions or lack of. [Q15b5] d. Decisions and regulations are made public available in a timely manner. [Q16b1]3. Coordination of the public sector HRM regime outside the core administration a. In the public sector outside of the core administration, HRM practices are structured around merit criteria that are clearly and publicly defined and the employment regime is effective in attracting, retaining and motivating competent staff. [Q15c1] b. There are clear mechanisms within the public administration to ensure accountability of staff and avoid conflict of interest and abuse of power. [Q15a2, Q16c1, Q16c2] c. The aggregate public sector wage bill is not high by comparison with other similar countries. [Q15c2]
We don’t know that they are correct!(although they are not unreasonable!)
Maybe replicate PEFA – balancing use with research 1. A rapid 50 expansion 40 7 18 since 2005: 19 30 14 13 135 17 5 20 9 countries, 32 13 5 10 5 10 3 15 23 12 23 11 assessments 0 8 in March 2005 2006 2007 2008 2009 2010 2012 Led by EC Led by Other Led by WB 2. Growing Nr. 135 first 52 second 8 third generation of repeat assessmentsNote: PEFA assessments are shown by lead agency. The year refers to the main mission when most of the interviews for the PEFA assessment were conducted. Thenumber of assessments reported includes both "finalized" and "draft assessments". While every effort was made to take stock of PEFA Assessmentscomprehensively, some assessments may have been omitted in the dataset.
1. IEs provide great value-added for evidence-based reform design, by rigorously identifying the effect of an intervention2. The PSM IE research agenda is lagging compared to other sectors.3. There is significant opportunity for expanding the IE research agenda on Public Sector Governance – and the Bank should play a leading role in this.4. This is true despite a typically small “n” in PSM reform. • By contrast to health or education reforms, that can be applied selectively to schools or health centers to construct a counterfactual, PSM reforms often focus on central agencies, such that n is small or a randomized roll-out difficult. • However, recent research demonstrates that IE methods can successfully be applied to learn about the effects of upstream reform elements – e.g. tax inspector incentives or the impact of performance- related pay schemes.
Move towards more standardized clusters of project indicators to encourage peer learning NOT: a straight-jacket! (not standardizing indicators) BUT: A way of better peer-learning from projects through standardized “menus” of indicators, e.g. organized by: 1. What is the specific reform area? (pay reform, regrading, agency creation etc.) ISPMS 2. Where in the results-chain does the indicator measure? 19
1. Focus on the functional problem, rather than the solution 2. Engage stakeholders in identifying functional problems and binding constraints3. Use political economy analysis prospectively 4. Use available evidence and accepted theory on whether20a reform will fix the problem
Benefit from the results-based-lending instrument (PforR) forPSM reform The Sierra Leone Experience: • A different, problem-focused engagement process • Results dialogue as a vehicle for encouraging MoF and line-ministries to engage in joint problem-solving • Balancing quantity and quality in indicator design is not easy • Planning to build in Impact Evaluation design 21
• The numbers suggest quantity more than quality• But we have some rebuttals• But, between us, we have some other problems• Where to from here? o Let’s find out more about what tends to work in general – new metrics, impact evaluation, standardized project indicator buckets o Let’s focus also on what is likely to work here – better diagnostics and more flexible instruments