Independent Advisory Committee on Development Impact Independence in evaluation An assessment framework Robert Picciotto

Four questions Is independence in evaluation important? What is its precise meaning? Can it be evaluated? If so, how?

Is independence in evaluation important?

What is its precise meaning?

Can it be evaluated?

If so, how?

Is independence in evaluation important? Yes - but it is not everything: quality and transparency also matter. Independence without quality can do harm: it risks bestowing legitimacy on mistaken diagnostics Quality without independence lacks clout: it lacks credibility. Both independence and quality are enhanced by transparency: Transparency increases public pressures on decision makers to take evaluation seriously. Transparency also improves evaluation quality: it amplifies stakeholders’ voice.

Yes - but it is not everything: quality and transparency also matter.

Independence without quality can do harm: it risks bestowing legitimacy on mistaken diagnostics

Quality without independence lacks clout: it lacks credibility.

Both independence and quality are enhanced by transparency:

Transparency increases public pressures on decision makers to take evaluation seriously.

Transparency also improves evaluation quality: it amplifies stakeholders’ voice.

What is evaluation independence? Good evaluation just as good science calls for a frame of mind characterized by curiosity, skepticism and a hunger for evidence. But evaluators also need protection: they should be immune to capture by any of the parties that share in the responsibilities of operational management. DAC: independence is not isolation International Federation of Accountants: the perception of independence is as important as its reality. independence of mind independence of appearance

Good evaluation just as good science calls for a frame of mind characterized by curiosity, skepticism and a hunger for evidence.

But evaluators also need protection: they should be immune to capture by any of the parties that share in the responsibilities of operational management.

DAC: independence is not isolation

International Federation of Accountants: the perception of independence is as important as its reality.

independence of mind

independence of appearance

Optimum independence is not maximum independence . The challenge: combine independence with connectivity Full disengagement has costs: limited access to information weak intellectual leverage chilling effect on learning. External evaluation is not the solution: limited knowledge of the operating context. fee dependence need for protection from undue influence (UKES Guidelines)

The challenge: combine independence with connectivity

Full disengagement has costs:

limited access to information

weak intellectual leverage

chilling effect on learning.

External evaluation is not the solution:

limited knowledge of the operating context.

fee dependence

need for protection from undue influence (UKES Guidelines)

Independent evaluation needs self evaluation and vice versa Evaluation should promote learning as well as accountability. Just as financial auditing requires the prior production of accounts, independent evaluation is reliant on regular, timely and adequate self evaluation. Conversely, the quality of self evaluation hinges on its access to intellectual guidance and competent oversight by independent evaluation. Hence independent evaluation should enjoy a measure of authority over the design of self evaluation processes, programs and products .

Evaluation should promote learning as well as accountability.

Just as financial auditing requires the prior production of accounts, independent evaluation is reliant on regular, timely and adequate self evaluation.

Conversely, the quality of self evaluation hinges on its access to intellectual guidance and competent oversight by independent evaluation.

Hence independent evaluation should enjoy a measure of authority over the design of self evaluation processes, programs and products .

Is independence evaluable? Yes: guidelines have been issued by the Evaluation Cooperation Group (ECG) ECG is composed of the heads of evaluation of the multilateral development banks (MDBs), the United Nations evaluation working group, the DAC Evaluation Network and the International Monetary Fund. The guidelines draw on the good practice standards of official audit and evaluation agencies spanning government and the corporate sector https://wpqp1.adb.org/QuickPlace/ecg

Yes: guidelines have been issued by the Evaluation Cooperation Group (ECG)

ECG is composed of the heads of evaluation of the multilateral development banks (MDBs), the United Nations evaluation working group, the DAC Evaluation Network and the International Monetary Fund.

The guidelines draw on the good practice standards of official audit and evaluation agencies spanning government and the corporate sector

https://wpqp1.adb.org/QuickPlace/ecg

Specific criteria define evaluation independence Four basic criteria organizational independence; behavioral independence; protection from external influence; and avoidance of conflicts of interest The four criteria are interrelated. organizational independence enhances behavioral independence. behavioral independence is hindered by external interference undue external influence results from conflicts of interest conflicts of interest undermine organizational independence.

Four basic criteria

organizational independence;

behavioral independence;

protection from external influence; and

avoidance of conflicts of interest

The four criteria are interrelated.

organizational independence enhances behavioral independence.

behavioral independence is hindered by external interference

undue external influence results from conflicts of interest

conflicts of interest undermine organizational independence.

Organizational independence A proper mandate formally endorsed by the supreme governance authority A reporting relationship to the head or deputy head of the organization or its board An appropriate grade and title for the head evaluator position Adequate budget allocations Freedom to program evaluation activities without management interference or budget restrictions

A proper mandate formally endorsed by the supreme governance authority

A reporting relationship to the head or deputy head of the organization or its board

An appropriate grade and title for the head evaluator position

Adequate budget allocations

Freedom to program evaluation activities without management interference or budget restrictions

Behavioural independence Willingness and capacity to issue strong, high quality and uncompromising reports – and to do so more often than occasionally. Full protection to evaluators who deliver unvarnished and unwelcome assessments of performance. Selection, evaluation, remuneration and possible removal of the head of evaluation unit such as to guarantee independence. Unrestricted access to the staff and records of the organization. Capacity to report findings without fear of reprisal Willingness and capacity to stand up publicly and defend evaluation methods and findings. Freedom to publish findings

Willingness and capacity to issue strong, high quality and uncompromising reports – and to do so more often than occasionally.

Full protection to evaluators who deliver unvarnished and unwelcome assessments of performance.

Selection, evaluation, remuneration and possible removal of the head of evaluation unit such as to guarantee independence.

Unrestricted access to the staff and records of the organization.

Capacity to report findings without fear of reprisal

Willingness and capacity to stand up publicly and defend evaluation methods and findings.

Freedom to publish findings

Protection from outside influence Independence can be hindered through active or passive interference in the conduct of evaluations. Where delays are imposed by operational managers, independence is jeopardized. Safeguards are needed to avoid control by personnel regulations of hiring, firing, duration of office, compensation, rotation, performance reviews etc. Nor should evaluators’ reports be subject to mandatory clearances by program managers External funding tainted by special interests should be avoided.

Independence can be hindered through active or passive interference in the conduct of evaluations.

Where delays are imposed by operational managers, independence is jeopardized.

Safeguards are needed to avoid control by personnel regulations of hiring, firing, duration of office, compensation, rotation, performance reviews etc.

Nor should evaluators’ reports be subject to mandatory clearances by program managers

External funding tainted by special interests should be avoided.

Avoidance of conflicts of interest Providing a wide range of stakeholders with an opportunity to comment helps to correct evaluator bias. Participatory evaluation methods strengthen evaluation quality and transparency. BUT evaluation managers and staff should be excluded from evaluating programs, activities or entities that might involve a conflict of interest or create the perception that current or past associations and/or activities (whether personal or professional) could impair the integrity of the process.

Providing a wide range of stakeholders with an opportunity to comment helps to correct evaluator bias.

Participatory evaluation methods strengthen evaluation quality and transparency.

BUT evaluation managers and staff should be excluded from evaluating programs, activities or entities that might involve a conflict of interest or create the perception that current or past associations and/or activities (whether personal or professional) could impair the integrity of the process.

Conclusion: independence is critical to evaluation excellence Evaluation impact requires independence since lack of independence: constrains information so that evaluation products cannot have an adequate critical content; distorts the content of the evaluation program so that it does not contribute new knowledge; delays the evaluation process (or the disclosure of evaluation results) until after decisions are taken; induces evaluators to focus on irrelevant or marginal aspects of programs and policies distorts incentives so that evaluators are restrained from speaking truth to power

Evaluation impact requires independence since lack of independence:

constrains information so that evaluation products cannot have an adequate critical content;

distorts the content of the evaluation program so that it does not contribute new knowledge;

delays the evaluation process (or the disclosure of evaluation results) until after decisions are taken;

induces evaluators to focus on irrelevant or marginal aspects of programs and policies

distorts incentives so that evaluators are restrained from speaking truth to power