Managing for Quality and Use 质量控制以及使用

Managing for Quality and Use 质量控制以及使用 Shanghai International Program for Development Evaluation Training Asia-Pacific Finance and Development Center 200 Panlong Road-Shanghai, October 15, 2008 Linda Morra-Imas

Linda Morra-Imas Chief Evaluation Officer & Advisor Evaluation Capacity Building I nternational F inance C orporation 2121 Pennsylvania Avenue, NW Washington DC 20433 USA [email_address] www.ifc.org/ieg Dadang Solihin Indonesia Delegation Shanghai International Program for Development Evaluation Training

Introduction 介绍 Managing the Design Matrix 管理设计矩阵 Managing an Evaluation 管理评价 Managing Effectively 有效管理 Assessing Quality of an Evaluation 评估评价的质量 Using Evaluation Results 使用评价结果

Evaluation Design Matrix 评价设计矩阵 Key to success in evaluation is in planning 评价成功的关键因素在于计划 Evaluation Design Matrix, links descriptive, normative, and impact evaluation questions 评价设计矩阵，联系描述性，规范性，以及影响力评价问题 Helps to systematically map out the evaluation 协助系统地描绘出评价的全貌 Focuses the attention on each of the major components 注意力集中在每个重要的成分上

About the Design Matrix 关于设计矩阵 Takes time to fill in since you do not have all the information available at start 不用着急填写，因为一开始你不具备所有必要的信息 It is an iterative process 这是一个重复叠加的过程 Can change the format of the design matrix to fit your needs 可以改变设计矩阵的模式来适应你的需要

Guide for Using the Evaluation Design Matrix 使用评价设计矩阵的指南 For each question, complete all columns 完成每个问题的所有表栏 One row for each question 一个问题一行 if you have two main questions, the matrix should have two rows 如果你有 2 个主要问题，矩阵应该有 2 行 each row specifies how you plan to design your evaluation for each question 每一行都具体说明你是如何计划设计每个问题的评价 Comment column may help keep track of unresolved issues and/or concerns 评论栏能帮助你同没有解决的问题或焦点保持跟进

Guide for Using the Evaluation Design Matrix (Cont.) As mentioned before, the planning process is iterative and it will take time to determine the best way to conduct the evaluation. As you develop your design, you may find out that some of your assumptions were incorrect. Alternatively, you may be able to state very specifically where the data are: for example, agency report # 2001 issued May 1998 has the data you need. Even if you do not feel you have enough information to fill out all the columns, an evaluation design can still be enormously valuable for clarifying the main steps involved, and for communicating these to others.

Two Useful Tools 两个有用的工具 Evaluation Plans and Operations Checklist (Stufflebeam) 评价计划以及运行核查单（ Stufflebeam ） Checklist for Program Evaluation Planning (McNamara) 项目评价计划的核查单（麦克那马拉） You may want to adapt these to fit your needs 你可以更改这些以适应你的需求

Reviewing Your Design 检查你的设计 Make sure all of the pieces connect and will give you the best chance of obtaining the data necessary to answer the evaluation questions 保证所有部分之间都有连接，而且可以给你最好的机会来获得有用的数据来回答评价的问题

Pre-testing 开始前测试 Pre-testing is essential 开始前测试很重要 if survey, have several people use it in a real setting 如果调查，需要几个人在真实的环境下使用 if focus group, conduct as if real 如果是重点调查组，执行测试就好像是在真实的环境下 De-brief respondents, find out what worked, what did not 仔细检查回复，找出什么能发挥作用，什么不能 ask how they would fix the questions 找出怎么能找出问题 Have experts review your plans 找到专家来检查你的计划 Have cold reader review surveys for clarity, grammar, and typing 就明晰性，语法以及拼写进行咨询

Front-End Planning 从始至终的计划 Pay attention to: 注意： timing and time management 计划和时间管理 selection of actors and resources involved in the evaluation 对于要素的选择，以及涉及到评价的资源 role of program logic and program theory 项目逻辑的作用以及项目理论 design of the study 研究的设计 question to what extent data that are needed can be made available 询问能够得到多少程度的必需数据

Timing of the Evaluation 评价的时间控制 For an evaluation that emphasizes use, time it so your findings are available when decisions are being made or actions being taken 对于强调使用的评价，对它进行计时，这样你的发现能够在做出决定时或者做出行动时就准备好

Questions to Help Timing 帮助计时的问题 What decision, if any, is the evaluation finding expected to influence? 评价的发现能影响什么样的决策？ When will decisions be made? By whom? When, then, must the evaluations findings be presented to be timely and influential? 决策什么时候做出？由谁做出？之后，评价的发现什么时候被展示出来才是及时以及有影响力的？ What is the history and context of the decision-making process? 决策过程的历史及背景如何？ What other factors will affect the decision-making? 其他什么因素能影响决策？ How much influence do you expect the evaluation to have — realistically? 现实来说，你希望评价具有什么样的影响力？ (continued on next slide) 续下页

Questions to Help Timing (cont.) 帮助计时的问题（续） To what extent has the outcome of the decision already been determined? 决策的结果多大程度上已经被决定了？ What data and findings are needed to support decision making? 需要什么样的信息和发现来支持决策过程？ What needs to be done to achieve that level of influence? 达到这样的影响力需要做到什么？ How will we know afterward if the evaluation was used as intended? 我们之后如何知道是否评价的使用是同预期相同的？

The Evaluation Team 评价团队 Who 谁 client (commissioner of the evaluation) 客户（评价的委托人） Stakeholders 利益相关者 other evaluators 其他评价者 Participants 参与者 How 如何 Terms of Reference 任务大纲

Value of Terms of Reference (TOR) 任务大纲（ TOR ）的价值 Terms of Reference (TOR) describe the overall evaluation and establish the initial agreements prior to the work plan 任务大纲描述整个评价过程，并且建立工作计划开始之前的最初协议 TOR process: 任务大纲的过程： ensures all stakeholders are included in the discussion and decision-making 保证所有的利益相关者都被融入到了讨论以及决策过程 establishes the basic guidelines so everyone understands expectations and context of the evaluation 建立最基本的指导方针，这样每个人都知道评价的内容和预期

OECD, DAC Glossary Definition OECD, DAC 的词汇定义 Terms of Reference (TOR) are a written documentation that present: 任务大纲是一个书面文本，它展示了： the purpose and scope of the evaluation 评价的目的及范围 the methods to be used 使用的方法 the standard against which performance is to be assessed or analyses are to be conducted 评估表现或者进行分析的标准 the resources and time allocated 需要分配的时间和资源 reporting requirements 报告要求

TOR Typically Include: 任务大纲包括： Title: short and descriptive 标题：简短的，描述性的 Project or program description 项目描述 Reasons for the evaluation and expectations 评价以及预期的原因 Scope and focus of the evaluation: the issues to be addressed and questions to be answered 评价的范围以及焦点：需要谈到的和需要回答的问题 Stakeholder involvement: who will be involved, defined responsibilities, and accountability process 利益相关者的介入：谁会介入，定义责任以及负责过程 (continued on next slide) 续下页

TOR Typically Include: (cont.) 任务大纲通常包括：（续） Evaluation process: what will be done 评价过程：需要做什么 Deliverables: typically an evaluation work plan, interim report, final report and presentations 可提交的：通常是评价工作计划，临时报告，最终报告和展示 Evaluator qualifications: education, experience, skills and abilities required 评价者的资质：教育，经验，技能以及需要的能力 Cost projection based on activities, time, number of people, professional fees, travel, and any other related costs 基于活动，时间，人数，专业费用以及其他相关费用智商的成本预计

TOR Typically Include: (cont.) 任务大纲通常包括：（续） According to the Planning and Managing an Evaluation website from the UNDP: It is always good to have written TOR. The TOR serves as the basic tool for an evaluation manager to ensure the high quality of the exercise at different points - from the time the evaluation team is organized to the time that the exercise itself is conducted and the final report is prepared. Of course, the TOR has to be well written, emanating from consultations with evaluation stakeholders and clearly directed at some very specific issues [1] . [1] UNDP, Planning and managing an evaluation: http://www.undp.org/eo/evaluation_tips/evaluation_tips.html

Guidelines for Writing TOR 任务大纲的写作技巧 State clearly the objectives of the evaluation 明确说清楚评价的目的 identify the stakeholders of the evaluation 说明评价的利益相关者 the products expected from the evaluation 评价预期产生的产品 how the products are to be used 产品如何使用

Guidelines for Writing TOR (Cont.) the specific issues to be addressed 要谈到的具体问题 the methodology 方法论 arrangements for the evaluation 评价的准备 (continued on next slide) 续下页

Guidelines for Writing TOR (cont.) 任务大纲的写作技巧（续） Do not simply state the objectives in technical or process terms. Be clear on how the evaluation is expected to help the organization 不要仅仅用专业术语或过程来描述目的，清楚说明评价如何能帮助组织结构 Focus on key issues to be addressed by the evaluation 集中注意力于评价的主要问题 Avoid too many issues. It is better to have an evaluation that examines a few issues in-depth rather than one that looks into a broad range of issues superficially 避免太多问题。集中注意力集中的深入的检查几个问题比肤浅的涉及很多的问题要好

TOR Checklists 任务大纲的核查单 Evaluation Contracts Checklist (Stufflebeam) 评价合同核查单（ Stufflebeam ） Checklist for Developing and Evaluating Evaluation Budgets (Horn) 发展及评估评价预算的核查单（ Horn ） Key Evaluation Checklist (Scriven) 主要的评价核查单（ Scriven ）

Contracting Evaluations 承包评价 If human resources are not available in your organization, you may need to hire one or more other people to assist 如果你的机构人员不足，你需要雇佣其他人员进行协助 Can be brought in for the entire study or only parts 可以是全程参与，也可以是参与一部分

Two Main Steps for Hiring Contractor 雇佣承包者的两个主要步骤 Developing a request for proposal (RFP) 完成一个提交的要求表（ RFP ） Use a selection panel to choose a contract evaluator 组成一个委员会来挑选合同评价者

Items for Contracting 承包合同的条款 Purposes of the evaluation 评价的目的 Background and context for the study 研究的背景和内容 Key information requirements 主要的信息要求 Evaluation objectives 评价目的 Deliverables required 要求的可提交内容 Time frame 时间框架 Criteria for tender selection 细致选择的标准 Contract details for the project manager 项目经理的合同细节 Deadline for the proposal 议定书的截至日期 Budget and other resources 预算以及其他资源

Selection Process for Contracting 合同承包的选择过程 Select a panel comprising people with: 选取委员会包括的人员： evaluation knowledge and experience 评价知识以及经验 knowledge of the program area 项目领域的知识 knowledge of the culture 对于文化的知识 ownership of the findings and their uses 对于发现及其使用的所有权 Have the panel select the proposal using the criteria in the RFP 委员会利用 RFP 的要求来选择议定书 Keep a record of the selection process 记录选择过程

Criteria for Selecting the Contractor 选择承包人的标准 What is the contractor’s record of accomplishment? 承包人以前的业绩如何？ Has the RFP been adequately addressed? RFP 是否得到了充分的重视？ Is there a detailed explanation of implementation? 是否有对执行过程的仔细解释？ What is the communicating and reporting strategy? 交流以及报告的战略是什么？ Is there evidence of competencies? 是否有能胜任的证据？ What is the cost — is it specified in detail? 成本是什么？是否在细节中说明了？

Responsibilities of Purchaser, Once Hired 雇佣承包人后，采购者的责任 Keeping goals and objectives clear 保证目标明确 Maintaining ownership of the study 保持研究的所有权 Monitoring the work and providing timely feedback 监测工作，提供适时的反馈 Decision-making — in good time 在合适时做出决策 If changes are required to the contract, being open to negotiation with the contractor 如果合同需要改动，应该同承包人商讨

Managing Roles and Responsibilities 管理角色和责任 Clearly define and agree to who is responsible for what and when 明确定义并就何人就什么以及何时负责达成一致 Who needs roles and responsibilities defined: 谁需要定义角色和责任 evaluation managers 评价经理 Evaluators 评价者 Client 客户 providers of information (stakeholders) 信息提供者（利益相关者） consumers (impactees) 消费者（受影响者）

Evaluation Manager Roles 评价经理的角色 Helping evaluators do their work 帮助评价者工作 clarifying the TOR for the team 帮助团队明确任务大纲 answering questions 回答问题 checking on status of responsibilities 检查责任的归属及状态 asking if they need additional resources 询问他们是否需要额外的资源 helping team learn 帮助团队学习 facilitating team meetings 主持团队的会议 Choosing staff 挑选成员 Reviewing results and report 检查结果以及报告

Evaluation Manager Responsibilities 评价经理的责任 Detailed responsibilities in three areas: 在三个领域的责任细则： Preparation 准备 Implementation 执行 follow-up 跟进

Evaluators Skills and Abilities 评价者的技巧和能力 Expertise in the specific subject matter 在相关领域的专业能力 Knowledge of key development issues especially those relating to the main goals or the ability to see the “big picture” 重点发展领域的知识，尤其是那些跟主要目标以及需要见到全貌的能力有关的知识 Familiarity with organization’s business and the way such business is conducted 对组织业务以及组织业务开展方式的熟悉 Evaluation skills in design, data collection, data analysis, and preparing reports 设计中的评价技巧，数据收集，数据分析，以及准备报告 Skills in the use of information technology 信息技术中使用的技巧

Essential Competencies for Program Evaluators (ECPE) 项目评价者的主要资质（ ECPE ） See Table 12.1 in text 参见表 12.1 Includes these major areas: 包括一下主要领域： professional practice 专业经验 systematic inquiry 系统咨询 situational analysis 情景性分析 project management 项目管理 interpersonal competence 人际技能

Team Leader 团队领导 If multiple evaluators are working on an evaluation, may have a team leader 如果多个评价者在团队中工作，需要一个团队领导 Responsibilities 责任 supervise team members and manage day-to-day process 监督团队成员，并管理日常过程 act as mediator if there are dissenting voices within the evaluation team 如果评价团队中出现不同意见，领导应该是仲裁者 may write the report 撰写报告 make sure the team satisfies the TOR 保证团队对于任务大纲是满意的

Client 客户 The person who officially requests the evaluation 正式要求进行评价的人 Best if this is the same person to whom the person reports 最好需要给他提交报告的人也是他

Managing Projects 管理项目 Managing time 管理时间 Managing scope 管理领域 Managing people 管理人员 Managing money 管理资金 Managing resources 管理资源

Project Management Process 项目管理过程 Initiating 启动 Planning 计划 Executing 执行 Controlling 控制 Closing 结束

Phase I: Initiating 第一阶段：启动 Projects begin by confirming there is a need for the project 项目开始于存在对于这个项目的需求 Three Actions to be undertaken in this phase 这个阶段需要执行三个行动 Demonstrate project need and feasibility 表明项目的需要和可行性 Obtain project authorization 获得项目授权 Obtain authorization for the phase 获得这个阶段的授权

Phase I: Initiating 第一阶段：启动 In this first phase, Greer identifies three of the 20 actions. Demonstrate project need and feasibility. Obtain project authorization. Obtain authorization of the phase. Most projects begin by confirming that there is a need for the project. For evaluation projects, this is usually the beginning of the terms of reference (TOR). TOR were covered in Module 3, Initial Planning of an Evaluation . The second action in Greer’s initiation is authorization. In evaluations, you will probably need to get a “go or no go” decision by the stakeholders. You agree to the terms of reference and the stakeholders “sign off” on the TOR. The last action in this phase is to authorize that the initiation phase is complete and work can begin on the planning phase. For evaluations, this step might include a sign off on the initiation phase.

Phase II: Planning 第二阶段：计划 Managers spend much of their time planning 管理人员需要花很多时间来计划 13 actions in this phase 这个阶段有 13 个行动 Some steps are optional, depends on the size and scope of the project 一些步骤是可选的，这取决于项目的大小以及领域

Actions for Planning 项目的行动 Describe the project scope 描述项目领域 Define and sequence project activities 定义次序以及项目活动 Estimate durations for activities and resources required 估计活动的持续时间以及需要的资源 Develop a project schedule 完成一个项目计划表 Estimate costs 估计成本 Build a budget and spending plan 完成预算以及预算使用计划 (continued on next slide) 续下页

Actions for Planning (cont.) 项目的行动 Describe project scope. Statement of project scope Scope management plan Work breakdown structure Define and sequence project activities. An activity list (list of all activities that will be performed on the project) Updates to the work breakdown structure (WBS) A project network diagram Estimate durations for activities and resources required. Estimate of durations (time required) for each activity and assumptions related to each estimate Statement of resource requirements Updates to activity list

Actions for Planning (cont.) 项目的行动 Develop a project schedule. Project schedule in the form of Gantt charts, network diagrams, milestone charts, or text tables Supporting details, such as resource usage over time, cash flow projections, order/delivery schedules, etc. Estimate costs. Cost estimates for completing each activity Supporting detail, including assumptions and constraints Cost management plan describing how cost variances will be handled Build a budget and spending plan. A cost baseline or time-phased budget for measuring/monitoring costs A spending plan, telling how much will be spent on what resources at what time

Actions for Planning (cont.) 项目行动 ( 续 ) Create a formal quality plan 创造一个正式的质量计划 Create a formal project communication plan 创造一个正式的项目沟通计划 Organize and acquire staff 组织及获得人员 Identify risks and plan to respond 确认风险，以及应对计划 Plan for and acquire outside resources 计划以及获得外在的资源 Organize the project plan 组织项目计划 Close out the project planning phase 完成项目计划阶段

Actions for Planning (cont.) 项目行动 ( 续 ) Create a formal quality plan. (optional) Quality management plan, including operational definitions Quality verification checklists Create a formal project communications plan. (optional) Collection structure Distribution structure Description of information to be disseminated Schedules listing when information will be produced A method for updating the communications plan Organize and acquire staff. Role and responsibility assignments Staffing plan Organizational chart with detail as appropriate Project staff Project team directory

Actions for Planning (cont.) 项目行动 ( 续 ) Identify risks and plan to respond. (optional) A document describing potential risks, including their sources, symptoms, and ways to address them Plan for and acquire outside resources. (optional) Procurement management plan describing how contractors will be obtained Statement of work (SOW) or statement of requirements (SOR) describing the item (product or service) to be procured Bid documents, such as RFP (request for proposal), IFB (invitation for bid), etc. Evaluation criteria -- means of scoring contractor's proposals Contract with one or more suppliers of goods or services

Actions for Planning (cont.) 项目行动 ( 续 ) Organize the project plan. A comprehensive project plan that pulls together all the outputs of the preceding project planning activities Close out the project planning phase. A project plan that has been approved, in writing, by the sponsor A "green light" or okay to begin work on the project

Phase III: Executing 第三阶段：执行 Manager makes sure the tasks are being completed 管理人员保证任务完成 One action to be undertaken: 一个行动应该完成： Execute project activities 执行项目活动

Phase III: Executing (Cont.) Execute project activities Work results (deliverables) are created. Change requests (i.e., based on expanded or contracted project) are identified. Periodic progress reports are created. Team performance is assessed, guided, and improved if needed. Bids/proposals for deliverables are solicited, contractors (suppliers) are chosen, and contracts are established. Contracts are administered to achieve desired work results.

Phase IV: Controlling 第四阶段：控制 Manager keeps track of people, activities, money, and scope 经理对人员，活动，经费以及范畴保持沟通 One action to be undertaken 完成一个行动 Control project activities 控制项目活动

Phase IV: Controlling (Cont.) Control project activities Decision to accept inspected deliverables Corrective actions such as rework of deliverables, adjustments to work process, etc. Updates to project plan and scope List of lessons learned Improved quality Completed evaluation checklists (if applicable)

Phase V: Closing 第五阶段：结束 Manager finalizes activities, archives information 管理人员完成活动，对信息归类整理 One action to be undertaken 一个行动需要完成 Close out project activities 完成项目活动

Phase V: Closing (Cont.) Close out project activities Formal acceptance, documented in writing, that the sponsor has accepted the product of this phase or activity. Formal acceptance of contractor work products and updates to the contractor's files. Updated project records prepared for archiving. A plan for follow-up and/or hand-off of work products

Role of Evaluation Manager 评价经理的角色 Clearly describe the desired end project 清晰的描述被需要的完成项目 Describe what it was that you liked about the relevant previous efforts 描述你喜欢的有关的前期努力是什么 Involve the evaluator(s) in the planning 使评价人员参与到计划中 Monitor the progress of the evaluation 监测评价的过程

Role of Evaluation Manager (Cont.) Establish a timeline 建立时间轴 Motivate the evaluator’s to produce their best 激励评价人员发挥他们的最佳水平 Avoid micro-managing the evaluator(s) 避免评价人员自行其是 Thank the evaluator(s) for their work 感谢评价人员付出的努力

Managing People 人员管理 As a manager, you are managing people, NOT evaluations! 作为经理，你管理的是人，不是评价！ People are complicated. They are not machines. Their behavior will change from day to day. 人是负责的，他们不是机器，他们的行为会随时间改变 The manager is responsible for everything that happens within the manager’s scope of authority. 经理的指责是自己职权范围内所发生的一切

Managing People (Cont.) People are complicated. They are not machines any more than you are. Their behavior will change from day to day. Be sure to stay alert to what is going on with each person. At the beginning of each evaluation, sit down and get to know your staff. Find out what other projects they are doing, what their goals are, what they like to do in their free time, etc. You will find that you do not need to put your unique handprint on everything. Some things probably work just fine already. Also do not think or act like you know everything, nothing breeds resentment more than arrogance. You may be smart, but there are many who are smarter.

Managing People (Cont.) If you were promoted from an evaluator to a manager, you will need to “let go” of the doing the tasks of an evaluator. You have different responsibilities now. You, as the manager, are responsible for everything that happens in your scope of authority. Do not ever think that just because you may not be doing the actual work, you are not responsible---you are .

Working with Stakeholders 同利益相关者一起工作 Building trust 建立信任 Involving Directly Affected Stakeholders 融入直接的受影响利益相关者 Seeking Feedback 寻找反馈 Involving the Voiceless 让沉默的那些人参与进来 Involving the Opposition 融入反对意见

Managing Tasks 管理任务 May be easier than managing people 或许比管理人员容易 Task map 任务图 everyone’s assignments 每个人的任务 start and completion dates 开始以及完成时间 Gantt chart 甘特图表 chart showing interrelationships of projects and schedules 说明项目以及日程表之间关系的表格

Example of Task Map 任务图举例 Task 任务 Name 名字 Start Date 开始时间 Date due 截至日期 Review prior meetings 会议开始前检查 Linda 7/01 7/31 Schedule meeting with stakeholders 同利益相关者设定会议 Ed 7/15 7/31 Conduct stakeholder meetings 举行利益相关者会议 Linda and Ed 8/1 8/15 Design the evaluation 设计评价 Ray 7/1 8/31 Develop data collection instruments 信息收集工具的开发 Ray 8/1 8/31

Example of Gantt Chart 甘特图表举例 Task/ 任务 Month/ 月份 1 2 3 4 5 6 7 Review 检查  Meetings 会议  Design 设计  Implement 执行 ----- ----- ----- 

Scope of Work 工作范畴 Similar to TOR, they covers these tasks: 同任务大纲类似，他们包括的任务： identify what is to be evaluated 确认需要评价什么 provide a brief background on the intervention 对于干预提供简短的背景 identify existing data 确认已有的信息

Scope of Work (Cont.) state the purpose of the evaluation along with audience and use 与观众和使用一道说明评价的目的 identify questions 确认问题 specify the methods 确认方法 discuss the composition of the team 讨论团队的组成 sets out the schedule and budget 制订日程以及预算

A Good Evaluation: 一个好的评价： Meets stakeholder needs and requirements 达到利益相关者的需要和要求 Is relevant and realistic in scope 范畴上是否相关而实际 Uses appropriate methods 使用合适的方法 Produces reliable, accurate, and valid data 生产可靠的，精确的以及可靠的数据

A Good Evaluation: (Cont.) Includes appropriate and accurate analysis of data 包括金合适的以及精确的数据分析 Presents impartial conclusions 展示不偏不颇的结论 Conveys results clearly — in oral or written form 清楚的说明结果——以口头和书面形式 Meets professional standards 达到专业要求

Kusek and Rist’s Six Characteristics of Quality Evaluations Kusek 和 Rist 的质量评价的 6 个特征 Impartiality 没有偏见 Usefulness 实用 Technical adequacy 恰当的技术 Stakeholder involvement 利益相关者的介入 Feedback and dissemination 反馈与传播 Value for money 金钱价值

Kusek and Rist’s Six Characteristics of Quality Evaluations ( Cont.) Impartiality : The evaluation information should be free of political or other bias and deliberate distortions. The information should be presented with a description of its strengths and weaknesses. All relevant information should be presented, not just that which reinforces the views of the manager. Usefulness : Evaluation information needs to be relevant, timely, and written in an understandable form. It also needs to address the questions asked, and be presented in a form desired and best understood by the manager. Technical adequacy : The information needs to meet relevant technical standards – appropriate design, correct sampling procedures, accurate working of questionnaires and interview guides, appropriate statistical or content analysis, and adequate support for conclusions and recommendations, to name but a few.

Kusek and Rist’s Six Characteristics of Quality Evaluations ( Cont.) Stakeholder involvement : There should be adequate assurances that the relevant stakeholders have been consulted and involved in the evaluation effort. If the stakeholders are to trust the information, take ownership of the findings, and agree to incorporate what has been learned into ongoing and new policies, programs, and project, they have to be included in the political process as active partners. Creating a façade of involvement, or denying involvement to stakeholders, are sure ways of generating hostility and resentment toward the evaluation – and even toward the manager who asked for the evaluation in the first place.

Kusek and Rist’s Six Characteristics of Quality Evaluations ( Cont.) Feedback and dissemination : Sharing information in an appropriate, targeted, and timely fashion is a frequent distinguishing characteristic of evaluation utilization. There will be communication breakdowns, a loss of trust, and either indifference or suspicion about the findings themselves if: evaluation information is not appropriately shared and provided to those for whom it is relevant the evaluator does not plan to systematically disseminate the information and instead presumes that the work is done when the report or information is provided no effort is made to target the information appropriately to the audiences for whom it is intended.

Kusek and Rist’s Six Characteristics of Quality Evaluations ( Cont.) Value for money : Spend what is needed to gain the information desired, but no more. Gathering expensive data that will not be used is not appropriate – nor is using expensive strategies for data collection when less expensive means are available. The cost of the evaluation needs to be proportional to the overall cost of the initiative.

Checklists for Quality 质量的核查单 The Key Evaluation Checklist (Scriven) 主要的评价核查单（ Scriven ） Program Evaluations Meta-evaluation Checklist (Stufflebeam) 项目评价的广义评价核查单（ Stufflebeam ） Utilization Focused Evaluation Checklist (Patton) 关注于利用的评价核查单（ Patton ） Deliberative Democratic Evaluation Checklist (House & Lowe) 审慎而民主的评价核查单（ House & Lowe ） Guiding Principles Checklist (Stufflebeam) 核查单的指导原则（ Stufflebeam ）

Meta-evaluator 广义评价员 A person with evaluation expertise who is not involved in conducting the evaluation, but who you can use as a sounding board, advisor, and helpful critic 一个具有评价技能的人，他不会参与到评价的执行过程，但是他能够作为一个出色的咨询对象，并提供有益的批评

Quick Meta-evaluation 迅速的广义评价 Consider getting a “rapid assessment” meta-evaluator to quickly look over your evaluation plan (or report) 考虑获得一个“快速测评”的广义评价员，以能迅速的考察你的评价计划（或报告） Offer to act as a meta-evaluator/review for someone else 主动要求作为别人的广义评价员或检查

Quick Meta-evaluation (Cont.) Unable to afford a meta-evaluator? Here are some creative options available for those on a low budget. Perhaps you can think of some more to add to the list! Consider getting a “rapid assessment” meta-evaluator – get an expert to quickly look over your evaluation plan (or report) and identify any gaps and make suggestions. For a stronger enhancement, have two evaluators with complementary skills and perspectives to take a look at your work. The value of the feedback will be much more than double! Offer to act as meta-evaluator/reviewer for someone else, provided they will return the favor sometime. Even a quick look from a fresh set of eyes can add some real value.

Kusek and Rist: Kinds of Information from Evaluations Kusek 和 Rist ：评价中的各种信息 Strategy: are the right things being done? 战略：正确的事情做了吗？ Operations: are things being done right? 运行？事情做对了吗？ Learning: are there better ways? 教训：是否有更好的方法？

Kusek and Rist: Kinds of Information from Evaluations ( Cont.) Kusek and Rist [1] describe the following kind of information that evaluations can supply: Strategy: are the right things being done? rationale or justification clear theory of change Operations: are things being done right? effectiveness in achieving expected outcomes efficiency in optimizing resources client satisfaction Learning: are there better ways? Alternatives best practices lessons learned [1] Kusek and Rist. Ten steps to a results-based monitoring end evaluation system . p117.

Kusek and Rist: Pragmatic Uses for Evaluations Kusek 和 Rist ：评价的实际应用 Help make resource allocation decisions 帮助资源配置的决策 Help rethink the causes of a problem 帮助重新思考问题的原因 Identify emerging issues 确认出现的问题 Support decision making on competing or best alternatives 支持有竞争力的或最好的替代计划的决策 Support public sector reform and innovations 支持公共领域的改革和革新 Build consensus on the causes of a problem and how to respond 就问题的原因进行统计，并研究如何应对

Kusek and Rist: Pragmatic Uses for Evaluations ( Cont.) Kusek and Rist [1] describe the following kind of information that evaluations can supply: Strategy: are the right things being done? rationale or justification clear theory of change Operations: are things being done right? effectiveness in achieving expected outcomes efficiency in optimizing resources client satisfaction Learning: are there better ways? Alternatives best practices lessons learned [1] Kusek and Rist. Ten steps to a results-based monitoring end evaluation system . p117.

Patton: Uses for Findings from Evaluations Patton: 来自评价的发现的利用 Judge merit or worth 评价优势及价值 Improve programs 改进项目 Generate knowledge 产生知识

Patton: Uses for Findings from Evaluations ( Cont.) Use of Evaluation Judge merit or worth Examples Summative evaluation Accountability Audits Quality control Cost benefit decisions Decide a program’s future Accreditation/licensing

Patton: Uses for Findings from Evaluations ( Cont.) Improve programs Formative evaluation Identify strengths and weaknesses Continuous improvement Quality enhancement Being a learning organization Manage more effectively Adapt a model locally

Patton: Uses for Findings from Evaluations ( Cont.) Generate knowledge Generalizations about effectiveness Extrapolate principles about what works Theory building Synthesize patterns across programs Scholarly publishing Policy making

Patton: Primary Uses of Evaluation Logic and Processes Patton: 评价逻辑和过程的主要用途 Enhanced sharing understandings 改进的理解分享 Supporting and reinforcing the program intervention 支持及加强项目干预 Increasing engagement 不断增加的介入 self-determination, and ownership 自决以及所有权

Patton: Primary Uses of Evaluation Logic and Processes ( Cont.) Uses of Evaluation Logic and Processes Examples Enhanced sharing understandings Specifying intended uses to provide focus and general shared commitment Managing staff meetings around explicit outcomes Sharing criteria for equity/fairness Giving voice to different perspectives and valuing diverse experiences

Patton: Primary Uses of Evaluation Logic and Processes ( Cont.) Supporting and reinforcing the program intervention Building evaluation into program delivery processes Participants monitoring their own progress Specifying and monitoring outcomes as integral to working with program participants

Patton: Primary Uses of Evaluation Logic and Processes ( Cont.) Increasing engagement, self-determination, and ownership Participatory and collaborative evaluation Empowerment evaluation Reflective practice Self evaluation Program and organizational development Developmental evaluation Action research Mission oriented, strategic evaluation Evaluability assessment Model specification

Suggestions for Ways to Improve Use of Evaluations 评价用法改进的几点建议 Gain support from the top 获得来自高层的支持 Involve the stakeholders at the top, bottom, and sides 融入来自上层，底层以及周边的利益相关者 Make the evaluation a part of the institution 使得评价成为机制中的一部分 Plan your evaluations 规划你的评价 Consider your timing: timing is everything 考虑你的时间：时间就是一切 Communication is important 沟通是重要的

Suggestions for Ways to Improve Use of Evaluations (Cont.) The following are some of the suggestions for ways to improve the use of evaluations. Gain support from the top. Increase the awareness of upper level personnel of the role that evaluations can play and ways that the evaluation can help them. Help upper level personnel set realistic expectations. Involve the stakeholders at the top, bottom, and sides. Make the evaluation a part of the institution. Use formal mechanisms, carrots and sticks, and linking to budget processes. Plan your evaluations. Planning is a major success factor, the evaluation should be well-designed. Use methodological quality. Include identification and include likely user and meet their needs.

Suggestions for Ways to Improve Use of Evaluations (Cont.) Consider your timing: timing is everything. Evaluations are appropriate to the life of the program. Do not do an impact evaluation too soon. A good evaluation that arrives after the decision has been made is useless. A politically sensitive evaluation might be received better after an election. Communication is important. The use of the evaluation is increased by the credibility of the evaluation. Credibility is increased by competence and trust of the evaluator. Present an early draft of the evaluation to stakeholders for comment and revision. Make final reports available to the public. Include information with negative findings.

Influence and Effects of Evaluation 评价的影响和效果 How do you bring the information you learned to the attention of the decision-makers? 如何使得你获得的知识得到决策者的重视？ How can you be in the right place at the right time? 你何如在正确的时间出现在正确的地方？ Brief the decision makers on the evaluation results at the meeting where decisions will be made 在有决定权力的会议上对决策者简短说明评价结果 Ensure that decision makers have a package that summarizes the results, their implications, and recommendations if any 保证决策者有系统的知识能总结结果，他们的启示，以及可能的建议

Carol Weiss: To Get Use –Identify the Evaluation Users Carol Weiss: 利用——确认评价的使用者 Who are the primary intended users of the evaluation? 评价的主要预期使用者是谁？ Who are the target audiences of the evaluation? 评价的目标受众是谁？ Which groups or individuals are most likely to be affected by the evaluation? 哪个群体或者个体最容易被评价的结果影响到？

Carol Weiss: To Get Use –Identify the Evaluation Users ( Cont.) Whose actions and/or decisions will be influenced by their engagement with the evaluation process or evaluation findings? 谁的行为或决定会因为他们对于评价过程或评价发现的介入而受到影响 How can the intended users of the evaluation be involved? 如何能使得评价的预期使用者介入？ What challenges/barriers might you face in identifying and involving users? How to overcome them? 在确认并融入利用者的过程中，你会遇到什么样的挑战和障碍？如何来超越这些困难？

Carol Weiss: To Get Use –Identify the Evaluation Users ( Cont.) Once you have completed the evaluation, you face a major challenge. How do you to bring the information that you learned to the attention of the decision-makers, when they make the decisions? How can you be in the right place, at the right time, with the right information? An ideal way of accomplishing this task is to get a briefing on the evaluation results on the agenda in the meeting where the decision is to be made and making sure that the decision makers have a packet that summarizes the results and their implications for the decision(s) to be taken before the meeting. In other words, having completed the evaluation, you are now the advocate bringing to your decision-makers the information that you want them to use in making their decision. But keep in mind that in reality evaluation results are often only one important source of information among several that will be used in making a given decision.

Carol Weiss: To Get Use –Identify the Evaluation Users ( Cont.) Carol Weiss [1] suggests a primary step in getting the information from your evaluation used is to identify the evaluation users. Evaluation users are those with the willingness, authority, and ability to put what the learned from the evaluation to work in some way. Weiss offers the following questions to help determine the intended users of an evaluation. [1] Carol Weiss , Identifying the intended user(s) of an evaluation. (2004). International Development Research Center. http://www.idrc.ca/uploads/user-S/108739486317Guideline.pdf

Foster Evaluation Thinking 培养评价思考 Increased clarity, specificity, and focus 提升精确性，具体和焦点 Being systematic and making assumptions explicit 系统性，假设清晰 Translating program concepts, ideas, and goals into operational plans 将项目概念、想法以及目标解释为可操作的计划

Foster Evaluation Thinking (Cont.) Distinguishing inputs and processes from outputs 区分投入以及产出过程 Valuing empirical evidence 重视实证证据 Separating statements of fact from interpretations and judgments 把对于事实的解释和判断分开

Foster Evaluation Thinking (Cont.) Once you identify the evaluation users, you need to foster evaluative thinking among these users. Most evaluation users will not be familiar with evaluations and not know how to use them. You will need to introduce a general awareness and understanding of the practices and procedures of evaluation. Patton [1] suggests the following characteristics of evaluative thinking that you may try to foster. [1] Michael Quinn Patton (1997) in Weiss Identifying user(s) of an evaluation 2004.

Foster Evaluation Thinking (Cont.) When stakeholders participate in an evaluation, the process can: draw their attention to issues they have not considered create dialogues among the stakeholders This process can produce intended and unintended results long after the evaluation results are presented. [1] [1] Kirkhart, K. E. (2000). “Reconceptualizing evaluation use: An integrated theory of influence”. In V.J. Caracelli & H. Preskill (Eds.) The expanding scope of evaluation use. New Directions for Evaluation , No. 88. San Francisco: Josey-Bass.

Beside working as Assistant Professor at Graduate School of Asia-Pacific Studies, Waseda University, Tokyo, Japan, he also active as Associate Professor at University of Darma Persada, Jakarta, Indonesia. He got various training around the globe, included Shanghai International Program for Development Evaluation Training (2008) , Public Officials Capacity Building Training Program for Government Innovation, Seoul –Korea (2007), Advanced International Training Programme of Information Technology Management, at Karlstad City, Sweden (2005); the Training Seminar on Land Use and Management, Taiwan (2004); Developing Multimedia Applications for Managers, Kuala Lumpur, Malaysia (2003); Applied Policy Development Training, Vancouver, Canada (2002); Local Government Administration Training Course, Hiroshima, Japan (2001); and Regional Development and Planning Training Course, Sapporo, Japan (1999). He published more than five books regarding local autonomous. You can reach Dadang Solihin by email at [email_address] or by his mobile at +62812 932 2202 Dadang Solihin currently is Director for Regional D evelopment Performance Evaluation at Indonesian National Development Planning Agency (Bappenas). He holds MA degree in Economics from University of Colorado, USA. His previous post is Director for System and Reporting of Development Performance Evaluation at Bappenas. Dadang Solihin’s Profile

Managing for Quality and Use 质量控制以及使用

More Related Content

What's hot

Viewers also liked

Similar to Managing for Quality and Use 质量控制以及使用

More from Dadang Solihin

Recently uploaded

Managing for Quality and Use 质量控制以及使用