This presentation was made by Paula Darville, Chile, at the 13th Annual Meeting of OECD Senior Budget Officials on Performance and Results held at the OECD Headquarters on 16-17 November 2017

1. Use of evidence in
performance budgeting: The
Chilean Experience
Paula Darville
Head of Management Control Division
Budget Directorate, Ministry of Finance, Chile
November 2017

2. 2
I. Ex post Evaluation System
II. Historical Statistics
III. Strengths
IV. Challenges
V. Lessons
Contents

3. 3
I. Ex post Evaluation System
• Objective
• Deliver evidence and information to improve allocation and use of
public resources, thus contributing to improve the quality of public
spending.
• Responsible Institution
• Budget Directorate, Ministry of Finance.
• Counterparts in ministries/services in charge of programs evaluated.
• Effects
• Commitments to incorporate recommendations from the evaluation.
• Follow up by formal agreements between Budget and Executing Unit.
• Re allocation of resources (in some cases).

4. Guiding Principles
a. Independency: Performed by independent evaluators
selected by public tendering (panels of experts or
universities and consulting firms).
b. Transparency: results are public.
c. Reliability: technical quality.
d. Relevance: focused on improving public expenditure.
e. Opportunity: deliver information within the budget cycle.
4
I. Ex post Evaluation System

5. Legal Framework:
• Legal obligation is defined in the State Financial Administration Law
• The Regulation of the law establishes:
- Guiding principles of the evaluation system (independence, transparency,
responsibility of Budget Directorate).
- Types of evaluation (components of the Ex post Evaluation System).
- Obligations for the evaluated institutions (provide information, comment on
the evaluation reports, establish improvement commitments resulting from
the evaluations, report the progress in complying those commitments).
- No direct budgetary effects of the evaluations are established.
5
I. Ex post Evaluation System

6. The role of Congress :
• Permanent Special Budget Commission: Senators and Deputies.
• The Special Commission recommends a list of programs and institutions to
be evaluated, through a protocol of agreement between Congress and
Government signed during the discussion of the Budget Project Law each
year.
• All information on results from evaluations, indicators of performance and
achievement of its goals is sent to the Congress along with the Budget
Project Law each year.
• Presentation of the general results of the evaluations, prior to the start of
the next budget discussion (Aug.-Sept. of each year).
• In cases that are requested, detailed presentations of evaluations of specific
programs.
6
I. Ex post Evaluation System

7. Components
a. Evaluation of Public Programs
• Created in 1997.
• Review consistency in design, execution and reporting, based on logical
framework methodology.
• Performed by panels of 3 independent experts, selected by public tendering.
• Final reports in 6 months. Directly connected to the budget cycle.
• Uses available information.
• Average Cost US$35.000.
b. Impact Evaluation
• Created in 2001.
• Assess program effectiveness on basis of impact measures.
• Methodology includes extensive data collection, more sophisticated evaluations
techniques (CBA, CEA), control groups.
• Performed by consulting firms, universities, selected by public tendering.
• Final reports in 1-11/2 years.
• Average Costs US$200.000. 7
I. Ex post Evaluation System

8. Components
c. Agency Expenditure Evaluation
• Created in 2002.
• Assess consistency of ministry/agency portfolio of programs in relation to agencies
mission and objectives. Search for duplications, inconsistencies, opportunities to
generate synergies and savings.
• Performed by consulting firms, universities, selected by public tendering.
• Final Report in 8 months, .
• Average costs US$100.000.
d. Evaluation of New Programs
• Created in 2009.
• Design of the evaluation at the beginning of each new program. Establishing control
groups for the evaluation, based on randomized trials whenever possible.
• Final Report in 24 months.
• Average Costs US$170.000.
8
I. Ex post Evaluation System

9. Components
e. Focused Evaluation
• Created in 2015.
• Review cost, implementation and diagnostic that creates the public intervention.
• Performed by consulting firms, universities, selected by public tendering.
• Final Report in 4 months, .
• Average costs US$50.000.
9
I. Ex post Evaluation System

10. Components 1997-1999 2000-2005 2006-2009 2010-2017 Total
a. Evaluation of Public
Programs
86 96 62 140 384
b. Impact Evaluation 0 30 40 54 124
c. Agency Expenditure
Evaluation
0 13 18 19 50
d. Evaluation of New
Programs
0 0 6 7 13
e. Focused Evaluation 0 0 0 12 12
Total 86 139 126 232 583*
*23 unfinished evaluations
10
II. Historical Statistics
• During 2006 – 2017 total budget evaluated has reached 49,4% of the 2017 total
central government budget. The programs and institutions evaluated in this
period reported in terms of budget over US$34.958 millions.
Number of Evaluated Programs and Institutions
1997-2017

11. Evaluation Results, by performance category
2011-2017
Category Total %
Good performance 5 4%
Adequate performance 40 28%
Under performance 83 58%
No Demonstrated Results 15 10%
Total 143 100%
11
II. Historical Statistics

12. Compliance with corporate agreements
1999-2016
12
II. Historical Statistics
Ministry
Category
Transferred
Number of total
commitments
30/06/2017
Achieved
Partially Not
Achieved Achieved
Ministry of Agriculture 99,4% 0,3% 0,3% - 652
Ministry of Defense 100% 0% 0% - 152
Ministry of Economy, Development and Tourism 97,4% 2,4% 0,2% - 382
Ministry of Education 97,4% 1,7% 0,9% - 1018
Ministry of Foreign Affairs 100% 0% 0% - 180
Ministry of Finance 97,7% 2,3% 0,0% - 128
Ministry of the Internal Affairs 99,2% 0,3% 0,0% 0,5% 378
Ministry of Justice 99,6% 0,4% 0,0% - 280
Ministry of Social Development 96,3% 1,9% 1,3% 0,5% 678
Ministry of Mining 100% 0% 0% - 49
Ministry of Public Works 87,8% 7,7% 4,5% - 156
Ministry of Health 94,4% 1,0% 4,6% - 197
Ministry General Secretariat of Government 91,9% 0,0% 0,0% 8,1% 273
Ministry General Secretariat of the Presidency 100% 0% 0% - 31
Ministry of Labor and Social Forecast 93,0% 1,9% 1,9% 3,2% 314
Ministry of Transportation and Communications 97,0% 2,0% 1,0% - 100
Ministry of Housing and Urban Planning 92,5% 5,5% 2,0% - 253
Ministry of National Assets 94,1% 2,0% 3,9% - 51
Ministry of Energy 92,7% 7,3% 0,0% - 41
Ministry of the Environment 100% 0% 0% 87
Total 96,8% 1,6% 0,9% 0,7% 5.400

13. Areas of Commitments %
Major reformulations 18%
Minor adjustments in the design 19%
Improvement of Information Systems 19%
Improvement of monitoring and evaluation
systems
25%
Improvement of the internal management of
the program
7%
Coordination actions 11%
Total 100%
Commitments of programs evaluated, Under
Performance Programs 2015 -2016
II. Historical Statistics

14. YEAR
PERFORMANCE CATEGORIES
Good
Performance
Adequate
Performance
Under
Performance
No
Demonstrated
Results
2011 - 5,60% -0,70% 15,90%
2012 11,50% 26,30% 12,30% 2,30%
2013 - 5,20% -63,00% 5,20%
2014 - 0,30% 28,10% -17,20%
2015 - 0,60% 6,00% -
2016 - 17,60% -4,40% -
2017 -107,60% 0,10% -19,00% -
Budgetary Effects, by performance categories
2011-2017
II. Historical Statistics

15. 15
Uses of Evaluations Results
(resources allocation)
II. Historical Statistics
2000-2003 2015-2016
Program Director 56% 53%
Head of Planning Unit 61%
Ministry Official 47% 56%

16. 16
Use of Evaluations
Use of the Information 2000-2003
2015-2016
Program
Director
Ministry
Officials
Justify current resources of the
program
30% 31% 33%
Reallocate resources in the program 32% 22% 22%
Redesign processes in the program
31% 29% -
II. Historical Statistics

17. • The system has been operating for 20 years, and has been perfected based on
its experience and the information requests received (including those from
the Congress), has created new lines of evaluation over time, and it is
recognized as good practice in the international level.
• The decision makers (the Executive and Legislative) know the existence and
have access to the System.
• The System has contributed to the development of an evaluation culture in
the public sector.
• The System has contributed to develop a critical mass of evaluators in the
country.
• The system is integrated into the budget process.
• Its results are public and easily accessible on the website of the Budget Office,
where there is a search engine by year, type of evaluation and institution
(www.dipres.gob.cl/595/w3-propertyvalue-23076.html).
IV. Strengths

18. 1. Achieve greater use of the information generated by the System:
• Formalize its use in budgetary decisions (both in the Executive and in the
Legislative).
• Increase citizen knowledge of the results of evaluations, to generate pressure on
their use in decision making.
• Institutionalize the information to the Congress in relation to the institutional
commitments that arise from the evaluations, as well as their subsequent
follow-up.
2. Move towards an evaluative approach of "policy areas" and not only
individual specific programs.
3. Increase the coverage of the evaluation system.
4. Move forward to improve the structure of the Budget Law according to the
actual programs carried by the Government .
5. Achieve balance between "evaluation as a learning process" and
"evaluation as a cause of budgetary decisions".
V. Challenges

20. 6. Improve institutional capacities:
• Ensure that institutional counterparts recognize evaluation as an inherent
part of the execution of a public policy program.
• Improve the management of evaluation concepts by institutional
counterparts.
7. Articulate ex ante and ex post evaluation processes to have well-
constructed programs from the beginning (with clear objectives,
indicators to measure results, etc.).
8. Design programs from the beginning in a way that allows
adequate methodological implementation of impact evaluations
(control groups, experimental methods, baselines).
9. Generate and provide access to the information necessary to
carry out a quality assessment (databases).
20
V. Challenges

21. 1. Developing an evaluation system linked to the budget is a process that
takes time.
2. It requires the cultural adaptation of public institutions to work for results.
3. For this, it is essential to generate trust in the stakeholders regarding the
quality and objectivity of the evaluation process.
4. The process is facilitated with the support of relevant actors: Ministry of
Finance, Congress, citizenship (who should understand that the evaluation
system is at their service).
5. For this it is fundamental to be transparent in the delivery of information.
6. It is necessary to develop institutional capacities: in the entity in charge of
the evaluations, in the evaluated counterparts, in the users of the
information (such as the Congress); and generate a critical mass of
external evaluators.
7. It is necessary to invest in generating quality information (without
information there is no possible evaluation).
21
VI. Lessons

22. Thank you