• Save
Developing  Evaluation Questions & Starting the Design Matrix 制定评价问题选择设计矩阵
Upcoming SlideShare
Loading in...5
×
 

Developing Evaluation Questions & Starting the Design Matrix 制定评价问题选择设计矩阵

on

  • 3,516 views

Shanghai International Program for Development Evaluation Training Asia-Pacific Finance and Development Center; 200 Panlong Road-Shanghai, October 13, 2008

Shanghai International Program for Development Evaluation Training Asia-Pacific Finance and Development Center; 200 Panlong Road-Shanghai, October 13, 2008

Statistics

Views

Total Views
3,516
Views on SlideShare
3,514
Embed Views
2

Actions

Likes
2
Downloads
0
Comments
1

2 Embeds 2

http://www.slideshare.net 1
https://learn.uark.edu 1

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Developing Evaluation Questions & Starting the Design Matrix 制定评价问题选择设计矩阵 Presentation Transcript

  • 1. Developing Evaluation Questions & Starting the Design Matrix 制定评价问题选择设计矩阵 Shanghai International Program for Development Evaluation Training Asia-Pacific Finance and Development Center 200 Panlong Road-Shanghai, October 13, 2008 Linda Morra-Imas
  • 2.
    • Linda Morra-Imas
    • Chief Evaluation Officer & Advisor Evaluation Capacity Building I nternational F inance C orporation
    • 2121 Pennsylvania Avenue, NW Washington DC 20433 USA
    • [email_address]
    • www.ifc.org/ieg
    Dadang Solihin Indonesia Delegation Shanghai International Program for Development Evaluation Training
  • 3. Introduction 引言
    • Sources of Questions
    • 问题来源
    • Three Types of Questions
    • 三类问题
    • Identifying and Selecting Questions
    • 提出和选择问题
    • Keys for Developing Good Evaluation Questions
    • 制定好的评价问题的关键因素
    • Suggestions for Developing Questions
    • 开发问题的建议
    • Evaluation Design
    • 评价设计
    www.dadangsolihin.com
  • 4. Evaluation Questions 评价问题
    • Give direction to an evaluation
    • 为一项评价指出方向
    • Critical element to help make decisions
    • 是帮助进行决策的关键因素
    • State what you want to learn
    • 表明你想要了解的东西
    • Must be in terms you can measure
    • 必须是你能够测量的
    www.dadangsolihin.com
  • 5. Sources 来源
    • Issues and questions of main client and key stakeholders 主要客户和利益相关者的问题
    • Program/project documents, e.g. decision papers, mid-term evaluation report 项目 / 计划文件,比如决议书,中期评价报告
    • Evaluation approaches or frameworks 评价方法或框架
    • Major issues identified in evaluation studies done by others of similar programs/projects 其他相似计划 / 项目在评价领域中明确的主要问题
    • Professional standards, checklists, guidelines, instruments, or criteria developed or used elsewhere 它地方制定的或使用过的专业标准、核查清单、指南、工具或准则。
    • Views and knowledge of expert consultants 咨询专家的观点和知识
    • The evaluator ’ s own professional judgment 评价人员自己的专业判断
    www.dadangsolihin.com
  • 6. Plus the Theory of Change: E.g. Logic Model to Questions 变革理论 问题的逻辑模型 www.dadangsolihin.com Inputs (Resources) 投入资源 Activities 活动 Outputs 产出 Short Term Results (Direct) 短期结果(直接) Intermediate Results (Indirect) 中期结果(间接) Long-Term Result 长期结果 Area of Control Internal to the Organization 组织内部的控制领域 Outputs Reach Direct Beneficiaries 直接受益人得到产出 Area of Influence External to the Organization 组织外部的影响领域 External Factors 外部因素 What aspects of our situation most shaped our ability to do the work we set out to do in our community 我们什么方面的情况最能够塑造我们执行既定工作的能力 What did our program accomplish in our community? 我们的计划在社会中取得了什么成就 What is our assessment of what resulted from our work in the community? 对于社会中我们工作所导致的事物我们如何评价? What have we learned about doing this kind of work in a community like yours? 关于在像你们这样的社区做这样的工作我们学到了什么 Formative Evaluation 形成性评价 Summative Evaluation 总结性评价 and/or 和 / 或
  • 7. Three Types 三类问题
    • Descriptive questions
    • 描述性问题
    • Normative questions
    • 规范性问题
    • Cause-Effect questions
    • 因果性问题
    www.dadangsolihin.com
  • 8. Descriptive Questions 描述性问题
    • Represent “ what is ”
    • 描述 “ 是什么 ”
    • Characteristics:
    • 特点:
    • seek to understand or describe a program or process
      • 试图了解或描述一个计划或过程
      • provide a snapshot of what is
      • 给出所要描述事情的 “ 快照 ”
      • are straightforward questions, such as:
      • 是直接了当的问题,例如:
      • who? what? where? when? how? how much/many?
      • 谁?什么?在哪里?什么时候?怎么样?多少?
      • can be used to describe:
      • 可以用于描述:
        • inputs, activities, outputs, outcomes 投入、活动、产出、成效
    www.dadangsolihin.com
  • 9. Examples of Descriptive Questions 描述性问题的例子
    • Who receives the program?
    • 计划的受施对象是谁?
    • What are the characteristics of the program?
    • 计划的特点是什么?
    • Where is the program delivered?
    • 计划在什么地方实施?
    • When was the program implemented?
    • 计划在什么时候实施?
    • How do the participants feel about the program?
    • 参与者对计划的感觉如何?
    • How much did the program cost?
    • 计划的成本是多少?
    • How many women participated in the program?
    • 参与计划的妇女有多少?
    • How were participants selected?
    • 是如何挑选计划参与者的?
    www.dadangsolihin.com
  • 10. Descriptive Questions 描述性问题
    • To what extent do stakeholders have different views of the goals of the program?
    • 利益相关者在多大程度上对计划的目标有不同的看法?
    • How did participants learn about the program?
    • 参与者如何获知该计划?
    • What are the eligibility requirements?
    • 资格要求是什么?
    • To what extent does program implementation vary across sites?
    • 计划的执行在不同地点之间有多大程度的不同?
    • What services does the program provide to men? To women?
    • 该计划提供什么服务给男性? 女性?
    • How do participants believe the program could be improved?
    • 参与者如何相信计划可以被改进?
    www.dadangsolihin.com
  • 11. Important Note 重要注释
    • Questions about the proportion of clients who find the program useful or the proportion that like the training are still descriptive questions*
    • 有关觉得项目有用的客户比例或者喜欢培训的客户比例的问题仍旧是描述性问题
    • Unless there is an established norm or standard
    • 除非存在一个已经建立的规范或者标准
    www.dadangsolihin.com
  • 12. Normative Questions 规范性问题
    • Compare “ what is ” to “ what should be ”
    • 将 “ 是什么 ” 与 “ 应该是什么 ” 进行比较
      • are we doing what we should be doing?
      • 我们是否正在做我们应该做的事情?
      • are we hitting our target?
      • 我们是否正在实现目标?
    • Similar to questions asked in performance auditing
    • 类似于在绩效审计中提出的问题
    • Measuring against a stated criteria
    • 根据阐明的标准进行衡量
    www.dadangsolihin.com
  • 13. Criteria for Normative Questions 规范性标准问题
    • Usually found in program authorizing documents e.g. legislation or Board approval documents
    • 通常可以再计划授权文件中找到, 比如立法或董事会批准文件
    • May be specified as indicators with specific targets in results-based monitoring systems
    • 在以结果为导向的系统中可能会具体为有特定目标的指标
    • May also come from accreditation systems, blue-ribbon panels, professional organizations, or other commissions
    • 可能同样出自认证体系,荣誉专家组,专业组织或其他机构
    www.dadangsolihin.com
  • 14. Examples of Normative Questions 规范性问题的例子
    • Did we spend as much as we had budgeted?
    • 我们的支出是否与预算的一样多?
    • Did we reach our goal of admitting 5,000 students per year?
    • 我们是否达到每年接收 5000 名学生的目标?
    • Did we vaccinate 80% of children as planned?
    • 我们是否按计划对 80% 的儿童接种疫苗?
    • Did we meet our objective of draining 100,000ha of land?
    • 我们是否实现灌溉 100,000 公顷土地的目标?
    • Was the process for selecting participants fair/equitable?
    • 选择参与者的过程是否公正 / 公平?
    www.dadangsolihin.com
  • 15. Important Notes 重要注释
    • The evaluator generally should refrain from setting new standards
    • 评价者通常会避免设立新标准
    • If a panel of experts is used, make sure it is balanced so no charge of bias
    • 如果聘请专家组,要保证小组的平衡,不会被指控有偏见
    • Otherwise, evaluator draws the fire!
    • 不然,评价者等于引火烧身
    www.dadangsolihin.com
  • 16. Cause-Effect Questions 因果性问题
    • Seek to determine what difference the intervention makes
    • 要确定干预活动起到了怎么样作用
    • Ask whether the desired results have been achieved AND whether it is the intervention that has caused the results
    • 询问是否取得了理想的结果以及是否是干预活动本身导致了这种结果
    • Imply before & after and with & without comparisons
    • 之前之后的暗示(有或没有比较)
    • Impact Evaluations focus on cause & effect question
    • 影响性评价注重因果性问题
    www.dadangsolihin.com
  • 17. Examples of Cause-Effect Questions 因果性问题的例子
    • As a result of the program, do participants have higher paying jobs than they otherwise would have?
    • 由于该项目,参与者是否是得到了比他们不参加计划更高工资的工作?
    • Do program graduates have higher literacy levels than before the program and can the gains be attributed to the program?
    • 项目毕业生是不是识字水平比参加前要更高,这样的提高能否归因于该项目?
    • Did draining the land result in crop production greater than those of surrounding areas?
    • 土地排水是否导致了作物产量比周围地区更高?
    • Did the new road increase traffic for trade and increase incomes more than would otherwise have been the case?
    • 新的道路修建是否比没有该路的时候增加了贸易并且增加了收入?
    www.dadangsolihin.com
  • 18. Determining Cause 明确原因
    • Should have the following:
    • 须要具备以下条件:
    • logic model or program theory: the connection between the intervention and outcomes should make sense
    • 逻辑模型或计划理论:必须清楚干预措施和成效之间的联系
    • time order: the interventions should come before the outcome
    • 时间顺序:干预措施的实施应该在出现成效之前
    • co-variation: both the intervention and the outcome should have the ability to change
    • 共变性:干预措施和成效都应该具备变化的能力
    • elimination of rival explanations: need to be able to establish if it is the intervention, rather than other factors that explain changes
    • 消除其它解释因素的影响:必须能够明确是干预措施而不是其它因素造成了这些变化
    www.dadangsolihin.com
  • 19. Choosing Questions 选择问题
    • Two phases:
    • 两个阶段:
      • divergent phase: develop a comprehensive list of questions
      • 发散阶段:提出一个全面的问题清单
      • convergent phase: narrow down the list
      • 收敛阶段:减少问题清单中的问题
    www.dadangsolihin.com
  • 20. Divergent Phase 发散阶段
    • Generate questions
    • 提出问题
    • Examine the questions
    • 检验问题
    • Organize the questions
    • 组织问题
      • classify each question as it fits into the labels of your model or framework
      • 对每个问题进行分类,使它符合你的模型或框架类别
    www.dadangsolihin.com
  • 21. Model to Question Type 问题类型模型
    • Questions about achievement of outputs are typically normative questions
    • 有关于产出成就的问题是典型的规范性问题
    • Questions about attainment of goals are typically cause and effect questions
    • 有关达成目标的问题是典型的因果性问题
    • Questions about impacts are typically cause and effect questions
    • 有关影响的问题是典型的因果性问题
    www.dadangsolihin.com
  • 22. Example of Classifying from Theory of Change 从逻辑模型进行问题分类的例子 Logic model for a micro-lending program 小额贷款计划的逻辑模型 Access to set-up funds for small businesses 获得小企业创业资金的机会 Income and employment for local people 为当地人们带来的收入和就业 Improved living conditions 生活条件的改善 Financial management advice and support 财务管理建议和支持 Skills in business & financial management 商业和财务管理的技能 Reduced family poverty 减少家庭贫困 www.dadangsolihin.com Questions about access to start up funds for small businesses 有关获得小企业创业资金的问题 Questions about income and employment for local people 有关对当地人们带来的收入和就业的问题
  • 23. Example of Classifying from Theory of Change 从计划成效模型进行问题分类的例子 Questions about resources 有关资源的问题 Questions about services 有关服务的问题 www.dadangsolihin.com Changes 变化 • money • 资金 • staff • 人员 • volunteers • 自愿者 • supplies • 物品供应 Benefits 效益 • new knowledge • 新知识 • increased skills • 技能提高 • changed attitudes • 态度变化 • new employment opportunities • 新的就业机会 Products 产品 • total # of classes • 班级数量 • hours of service • 服务时间 • number of participants completing course • 完成课程的参与者人数 Services 服务 • training • 培训 • education • 教育 • counseling • 咨询服务 Resources 资源 • money • 资金 • staff • 人员 • volunteers • 自愿者 • supplies • 物品供应 Impacts 影响 Outcomes 成效 Outputs 产出 Activities 活动 Inputs 投入
  • 24. Convergent Phase 收敛阶段
    • Be selective in identifying the study questions
    • 在确定研究问题的时候要有选择性
    • Eliminate interesting but not essential questions
    • 排除有趣但不是必须的问题
    www.dadangsolihin.com
  • 25. Suggested Criteria Narrow Down to Essential Questions 缩减到必要问题的建议准则
    • Who would use the information? Who wants to know? Who will be upset if this evaluation question is dropped?
    • 谁会使用这些信息?谁想了解这些信息?如果舍弃这个评价问题谁会反对?
    • Would an answer to the question reduce present uncertainty or provide information not now readily available?
    • 问题的答案是否会降低目前的不确定性或提供现在还没有的信息?
    • Would the answer to the question yield important information? Have an impact on the course of events?
    • 问题的答案是否会产生重要信息?是否对事件的过程产生影响?
    • (continued on next slide)
    • ( 转下页 )
    www.dadangsolihin.com
  • 26. Suggested Criteria (cont.) 建议使用的准则 ( 续 )
    • Is this question merely of passing interest to someone, or does it focus on critical dimensions or continued interest?
    • 这个问题只是仅仅想引起某人的兴趣,还是注重于关键方面或持续关注?
    • Would the scope or comprehensiveness of the evaluation be seriously limited if this question were dropped?
    • 如果舍弃这个问题,评价的范围和综合性是否会受到严重的限制?
    • Is it feasible to answer this question, given available financial and human resources, time, methods, and technology?
    • 在给定的财务和人力资源、时间、方法和技术的前提下,问答这个问题可行吗?
    www.dadangsolihin.com
  • 27. Matrix for Ranking and Selecting Questions 排列和选择问题的矩阵 Would the evaluation question … 评价问题能否 …… Question # 问题编号 1 2 3 4 5 6 7 8 … 1 be of interest to key audiences? 引起关键报告对象的兴趣? 2 reduce present uncertainty? 减小目前的不确定性? 3 yield important information? 产生重要的信息? 4 be of continuing interest? 具有持续的吸引力? 5 be critical to the study ’ s scope and comprehensiveness? 对研究范围和综合性是否关键? 6 have an impact on the course of events? 对事件过程产生影响? 7 be answerable in terms of … 以 …… 来回答?
  • 28. Keys for Developing Good Evaluation Questions 开发好的评价问题的关键要素
    • Begin by identifying the major issues
    • 从确认主要议题开始
    • Ask the questions that will help you learn if the issues have been affected by the policy or intervention
    • 提出的问题要有助于你了解该议题是否受到政策或干预措施的影响
    • Compound questions are not good, it is better to separate the issues
    • 复合的问题不好,把这些议题分开会更好
    • Questions about an issue can be addressed using all three question types by adjusting the wording
    • 通过调整措辞,有关一个议题的问题可以使用这三类问题来解决
    www.dadangsolihin.com
  • 29. Example: Questions about Issues 举例:提问和问题
    • What outreach methods were used?
    • 使用了什么外展方法
    • How effective were the outreach methods?
    • 外展方法有多有效?
    • What was the incidence of life-threatening infant diseases prior to and during the time the program was operational?
    • 在项目运作之前和之中的时候,致命的婴儿疾病的发病率是多少?
    • Did the program decrease mortality rates?
    • 项目是否降低了死亡率?
    www.dadangsolihin.com
  • 30. Write Better Questions 开发更好的问题
    • There should be a link between each evaluation question and the purpose of the study
    • 每个评价问题与研究的目的之间应该有联系
    • The issues of greatest concern should be addressed by the evaluation questions
    • 评价问题必须解决最受关注的议题
    • The questions should be answerable; if not, change the question or acknowledge the limitations
    • 问题应该能够回答,如果不能,改变问题或者声明其局限性
    • (continued on next slide)
    • ( 转下页 )
    www.dadangsolihin.com
  • 31. Write Better Questions (cont.) 开发更好的问题 ( 续 )
    • Be realistic about the number of questions that can be answered in a single evaluation
    • 在一个评价中要回答的问题的数量要切合实际
    • Focus on the important questions — the ones that must be answered as opposed to those that would be nice to know
    • 注重关键问题 —— 即必须回答的问题而不是轻易回答的问题
    • Consider the timing of the evaluation relative to the program cycle:
    • 考虑评价相对于计划周期的时机选择 :
      • Questions about impact are best answered after the intervention has been fully operational for a few years
      • 有关影响的问题最好是在干预措施全面实施后几年后才回答。
    www.dadangsolihin.com
  • 32. Evaluation Design: A Process for 评价设计:有关下列的流程
      • determining issues and questions to address
      • 明确要解决的问题
      • identifying measures
      • 确定度量
      • selecting information sources
      • 选择信息来源
      • choosing a sample
      • 选择一个样本
      • deciding on instrumentation
      • 决定工具的使用
      • collecting data
      • 选择数据
      • analyzing data
      • 分析数据
      • reporting results
      • 报告结果
      • getting the results used
      • 使用结果
    www.dadangsolihin.com
  • 33. Approach to Development Evaluation 发展评价方法 Focus the Evaluation 关注评价 ▪ Identification and meeting with stakeholders 确认会见利益相关者 ▪ Purpose – meeting with the client 会见客户 ▪ Research – other studies and program documentation 其他研究和计划文件 ▪ Theory of Change 变革理论 ▪ Specification of evaluation questions 详细说明评价问题 ▪ Terms of Reference 任务大纲 Design & Methodology 设计和方法 ▪ Evaluation questions 评价问题 ▪ Measurement strategy 测量策略 ▪ Data collection design 数据收集设计 ▪ Data collection strategy 数据收集策略 ▪ Sampling strategy 抽样策略 ▪ Develop data collection instruments 制定数据收集工具 ▪ Develop analysis plan 制定分析计划 Brief client and stakeholders 主要客户和利益相关者 Gather & Analyze Data 收集 & 分析数据 ▪ Test instruments 测试工具 ▪ Develop protocols 制定协议 ▪ Train as needed 按需求培训 ▪ Gather data according to protocols 根据协议收集数据 ▪ Prepare data for analysis 准备分析用的数据 ▪ Analyze data 分析数据 ▪ Interpret data 解释数据 ▪ Hold message conference 开展通气会 ▪ Draft statement of findings 草拟发现说明 Report Findings 报告发现 ▪ Write report 写报告 ▪ Review and quality checks 审核质量核查 ▪ Make recommendations Incorporate feedback/ 做出建议 包含建议 refine 完善 ▪ Deliver 提交 Use Evaluation 使用评价 ▪ Develop communication strategy ▪ Brief on Evaluation Design ▪ Update brief on evaluation process ▪ Communicate Findings ▪ Feed-back ▪ Decision-making ▪ Action Plan ▪ Follow-up ▪ Recommendations ▪ Tracking www.dadangsolihin.com
  • 34. Design Matrix 设计矩阵
    • An organizing tool to help plan an evaluation
    • 一个可以帮助规划评价的组织工具
    • Organizes questions and the plans for collecting information to answer questions
    • 为收集需要回答问题的信息,组织问题和方案
    www.dadangsolihin.com
  • 35. Matrix Elements 矩阵元素
    • Main evaluation issue
    • 主要评价问题
    • General approach 一般方法
    • Questions 问题
    • Sub-questions 次问题
    • Type of (sub)-question
    • 问题类型
    • Measure or Indicator
    • 测量或指标
    • Target or standard (if normative)
    • 目标或标准(如果是规范性问题)
    • Presence or Absence of Baseline data
    • 有无基准数据
    • Design Strategy 设计策略
    • Data Sources 数据来源
    • Sample or Census 抽样或普查
    • Data Collection Instrument 数据收集工具
    • Data Analysis and Graphics 数据分析和图表
    • Comments 注释
    www.dadangsolihin.com
  • 36. Matrix Elements (Cont.)
    • The purpose of the matrix is to organize the evaluation purpose and questions and match what is to be evaluated with appropriate data collection techniques. Although there is no hard and fast rule, a design matrix usually includes the following elements:
    • Although each evaluation question is unique, data collection activities are designed to address more than one question and several activities may address a single question. The matrix incorporates known sources of information and planned sources. As you move from planning to implementation, sources can be expanded and clarified.
    • Beyond its immediate usefulness as a planning tool, the matrix can serve as a tool to promote collaboration between evaluators and program staff. During later stages of the evaluation, evaluators and program staff can review the matrix, update it and use it as a guide for implementing the evaluation.
    www.dadangsolihin.com
  • 37. www.dadangsolihin.com Main Evaluation Question/Issue: 主要评价问题 Page 1 A Questions 问题 Sub-Questions 次问题 Type of (sub)question 问题类型 Measure or Indicator 测量或指标 Target or Standard (if normative) 目标或标准(如果是规范性问题) Baseline Data? 基准数据
  • 38. www.dadangsolihin.com General Approach: 一般方法 Page 1 B Design 设计 Data Sources 数据来源 Sample or Census 抽样或普查 Data Collection Instrument 数据收集工具 Data Analysis and Graphics 数据分析和图表 Comments 注释
  • 39. Thank you! 谢谢!
  • 40.
    • Beside working as Assistant Professor at Graduate School of Asia-Pacific Studies, Waseda University, Tokyo, Japan, he also active as Associate Professor at University of Darma Persada, Jakarta, Indonesia.
    • He got various training around the globe, included Shanghai International Program for Development Evaluation Training (2008) , Public Officials Capacity Building Training Program for Government Innovation, Seoul –Korea (2007), Advanced International Training Programme of Information Technology Management, at Karlstad City, Sweden (2005); the Training Seminar on Land Use and Management, Taiwan (2004); Developing Multimedia Applications for Managers, Kuala Lumpur, Malaysia (2003); Applied Policy Development Training, Vancouver, Canada (2002); Local Government Administration Training Course, Hiroshima, Japan (2001); and Regional Development and Planning Training Course, Sapporo, Japan (1999). He published more than five books regarding local autonomous.
    • You can reach Dadang Solihin by email at [email_address] or by his mobile at +62812 932 2202
    Dadang Solihin currently is Director for Regional D evelopment Performance Evaluation at Indonesian National Development Planning Agency (Bappenas). He holds MA degree in Economics from University of Colorado, USA. His previous post is Director for System and Reporting of Development Performance Evaluation at Bappenas. Dadang Solihin’s Profile www.dadangsolihin.com