The document explores the evaluation contexts and the importance of understanding the theory of change in program evaluation, emphasizing the roles of stakeholders and the necessity of front-end analysis. It discusses the balancing of costs and benefits in evaluations, the dangers of inappropriate planning, and the need for rigorous theory construction to identify critical program elements and assumptions. Key components also include stakeholder analysis and the implications of political and social environments on program success.

1. Understanding the Evaluation Context and Program Theory of Change 理解评价背景和项目变革理论 Shanghai International Program for Development Evaluation Training Asia-Pacific Finance and Development Center 200 Panlong Road-Shanghai, October 10, 2008 Linda Morra-Imas

2. Linda Morra-Imas Chief Evaluation Officer & Advisor Evaluation Capacity Building I nternational F inance C orporation 2121 Pennsylvania Avenue, NW Washington DC 20433 USA [email_address] www.ifc.org/ieg Dadang Solihin Indonesia Delegation Shanghai International Program for Development Evaluation Training

3. Introduction 引言 Front-end Analysis 前期分析 Identifying the Main Client and Key Stakeholders 明确主要客户和关键利益相关者 Understanding the Context 了解背景 Constructing, Working with, and Assessing a Theory of Change 建立，使用，评估变革理论 www.dadangsolihin.com

4. Front-End Analysis 前期分析 Client and stakeholders 客户和利益相关者 Timing and time management 时机和时间管理 Nature of resources 资源性质 Relevance of social science theory 和社会科学理论的相关性 Findings of similar evaluations 相似评价的发现 Role of theory of change 变革理论的作用 Existing data 现有数据 www.dadangsolihin.com

5. Balancing Costs and Benefits 成本收益间的平衡 Benefits 收益 strong knowledge 扎实的知识基础 Intended use by main client 主要客户有目的使用 Reauthorization or replenishment 再授权或者补充 Potential for learning what works 学习行之有效方法的潜力 context specification of what will work 对即将发生作用的事物的情况说明 Building local capacity 建立本地能力 answers to questions of stakeholder concern 利益相关者关心问题的答案 www.dadangsolihin.com

6. Balancing Costs and Benefits 成本收益间的平衡 Costs 成本 Evaluations are limited by time and money, some more than others 评价活动受时间和资金限制而有所不同 Examples of costs: 成本举例： cost of evaluation in relation to the cost of the program 与计划成本相关的评价成本 costs in terms of the social burden to program officials, evaluands, respondents 计划官员、评价对象、受访者所承担的社会成本 reputation costs to the evaluator and the evaluation community 评价人员和评价机构的信誉成本 www.dadangsolihin.com

7. Balancing Costs and Benefits (Cont) Costs of evaluations however are important too. Think of: cost of evaluation in relation to the cost of the program costs in terms of the social burden to program officials, evaluands ( ‘respondents’) as a consequence of participating in the study reputation costs to the evaluator and the evaluation community when results are disputed because of theoretical and or methodological criteria transaction costs www.dadangsolihin.com

8. Dangers or Pitfalls 危险或缺陷 The belief that everything should be covered up front 面面俱到的观念 The to-do-ism fixation 偏执的行动主义 McDonaldisation of society 社会的“麦当劳化” Truisms pop up while doing the front-end planning 进行前期规划时突然出现的“真理” Front-end planning does not prevent people to join in “group think” 前期规划没有阻止人们的“小团体思想” Power matters, never forget to look at the power position of the participants 在权力问题上决不要忘记考虑参与者的权力地位 www.dadangsolihin.com

9. Dangers or Pitfalls (Cont) The belief that everything should be covered up front including the belief that if front-end planning has been taken place, everything will be okay. The to-do-ism fixation : this is the danger that the principal investigator or somebody gets fixed on the original document of planning and can only do what has been planned before . Ritzer (199.) has coined the concept of McDonaldization of society. A danger of front-end planning that is particularly applicable is when checklists, to do-lists, frameworks and log-frames ‘take over’ and replace reflective thinking . www.dadangsolihin.com

10. Dangers or Pitfalls (Cont) Another danger is that truisms pop up while doing the front-end planning. “Randomized experiments…no way: ‘too complicated, too expensive, too conservative…’. Or: ‘a logical framework analysis is the least we can do and therefore we should do it.’ Front-end planning does not prevent people to join in ‘group think’ – agreeing with the group to remain a part of the group, criticism is not encouraged. Power matters. Never forget to look into the power position of the participants. www.dadangsolihin.com

11. Main Client 主要客户 One main client 一个主要客户 Key stakeholder sponsoring or requesting the evaluation 资助或者要求进行评价的关键利益相关者 May be 可能 : authorizing or funding the program 授权或资助该计划 authorizing or funding the evaluation 授权或资助评价 accountable for the public for the intervention 对于干预对公众负责 to whom the evaluators are accountable 评价者对谁负责 www.dadangsolihin.com

12. Stakeholders 利益相关者 People or representatives of organizations that have a “ stake ” in the intervention 与干预措施具有相关 “ 利益 ” 的个人或机构代表 Those affected by an intervention, in its lifetime, or in subsequent years 一直、或在未来几年内受到干预措施影响的人 Important to include those who would typically not be asked to participate 将那些没有被特别要求参与的人也包括在内，这一点很重要 www.dadangsolihin.com

13. Examples of Stakeholders 利益相关者实例 Participants 参与者 Direct beneficiaries 直接受益人 Indirect beneficiaries 间接受益人 Others impacted (directly or indirectly) 其他（直接或间接）受到影响的人 Donors 援助者 Government officials 政府官员 Program directors 计划负责人 Policy-makers 决策者 Community and interest groups or associations 社团、利益群体或协会 www.dadangsolihin.com

14. Stakeholder Roles 利益相关者的职能 Each evaluation is different, stakeholders have different roles 每次评价各不相同，利益相关者也扮演着不同的角色 Use a checklist to determine the role(s) of each stakeholder (example on next slide) 使用清单确定每一位利益相关者的职能（见下页） www.dadangsolihin.com

15. Checklist of Stakeholder Roles 利益相关者职能列表 Individuals, groups, or agencies 个人、团体或机构 To make policy 制定政策 To make operational decisions 制定经营决策 To provide input to evaluation 为评价提供资金 To react 作出反馈 For interest only 只为获取利益 Developer of program 计划开发者 Funder of program 计划资助者 Boards/agencies 委员会 / 机构 etc. 其他

16. Identifying and Involving Stakeholders 确认并使利益相关者参与进来 Identify the stakeholders 确定利益相关者 Have periodic stakeholder meetings 定期召开利益相关者会议 Involve early on so all have understanding of the intervention 尽早参与，以便于各利益相关方对干预措施有所了解 Describe the extent to which stakeholders are actively involved 描述利益相关者在多大程度上积极参与 www.dadangsolihin.com

17. Stakeholder Analysis 利益相关者分析 A technique to identify and assess the importance of key people, groups of people or institutions 这是一种用来确认、评价关键人物、群体或机构重要性的方法 Reasons to do: 分析目的： identify people, groups, and institutions 确认相关人士、群体和机构 anticipate the kind of influence – positive or negative – these groups will have on your initiative 预期这些群体将给你的提议带来怎样的正面或负面影响 develop strategies to get the most effective support possible 制定策略，尽可能赢得最有效的支持 www.dadangsolihin.com

18. Stakeholder Analysis “How To” 利益相关者分析方法 Stakeholder 利益相关者 Stakeholder Interest(s) in the Project 利益相关者在该项目中的利益 Assessment of Impact 影响评价 a) evaluation on stakeholder b) evaluation on stakeholder Potential Strategies for Obtaining Support or Reducing Obstacles 赢得支持、减少障碍的可能策略

19. Program Stages and the Broad Evaluation Question 计划各阶段与主要评价问题 Stage of program development 计划发展阶段 Evaluation question to be asked 提出的评价问题 Assessment of social problem and needs 对社会问题和需求的评价 To what extent are community needs and standards met? 社区的需求和生活水准能够得到多大程度的满足？ Determination of goals 目标的确定 What must be done to meet those needs and standards? 必须采取什么措施来满足这些需求和标准？ Design of program alternatives 计划备选方案的设计 What services could be used to produce the desired changes? 提供哪些服务可以产生预期的变化？ Selection of alternative 备选方案的选择 Which of the possible program approaches is best? 最佳的计划实施途径是什么？ Program implementation 计划实施 How should the program be put into operation? 如何将计划付诸实施？ Program operation 计划运行 Is the program operating as planned? 计划是否按照既定方案运行？ Program outcomes/effects/impact 计划成果 / 效果 / 影响 Is the program having the desired effects? 计划是否产生了预期的效果？ Program efficiency 计划的效率 Are the program effects attainted at a reasonable cost? 为取得计划效果所花费的成本是否合理？

20. Existing Theory and Knowledge 现有理论和知识 The underlying theory of change 重要的变革理论 Using existing evidence through research synthesis 通过分析合成使用现有证据 Interpreting a complex program as intervention chains: 将复杂计划作为干预链来分析解释 one set of stakeholders provide resources to other stakeholders 一组利益相关者向其他利益相关者提供资源 belief that behavior change will follow 相信行为变化会紧随其后 www.dadangsolihin.com

21. Sources of Theoretical and Empirical Knowledge 理论和经验知识的来源 Journals synthesizing the accumulation of explanatory knowledge 综合了解释性知识的期刊 evaluations and other social science research, including psychological and economic studies 评价和其它社会科学研究，包括心理研究和经济研究 problem-based research 基于问题的研究 repositories of randomized experiments 随机实验知识库 Knowledge Fund 知识储备 www.dadangsolihin.com

22. Sources of Theoretical and Empirical Knowledge (Cont) The third dimension we suggest is to get to know the existing theoretical and empirical knowledge about the project, program, or policy. The knowledge coming from evaluations and other social science research, including psychological and economic studies increases every day. Journals synthesizing the accumulation of explanatory knowledge are available. Problem-oriented research into how organizations function combines theories and research from such diverse disciplines like organizational sociology, cognitive psychology, public choice (economics) and law and economics (Scott, 2003; Swedberg, 2003). Repositories of randomized experiments in the field of criminal justice and crime prevention, social welfare programs, and health and educational programs indicate that more than 10.000 ‘experiments’ have been done (Campbell Society, 2003). See the following example from the field of criminal justice programs. www.dadangsolihin.com

23. Function of the Theory of Change 变革理论的功能 Depict a sequence of the 描述下列顺序 : inputs the program will use 计划会用到的投入 activities the inputs will support 投入将支持的活动 outputs to which the program is budgeting 计划中拨给预算的产出 results anticipated (outcomes and impacts) 预期结果（成果和影响） Depict other events or conditions 描述其他活动或情况 Identify assumptions for cause and effect 明确有关因果的假设 Identify critical assumptions that need to be examined by the evaluation 明确需要由评价来检验的关键假设 www.dadangsolihin.com

24. How “Theories” Work 理论是如何发挥作用的 Note the research that explains or predicts an event 关注解释或预测某一事件的研究 example: students do better when parents are involved in their studies 例如：当家长也参与到学习中时，学生会表现得更好 The program theory links what the research is showing to a “ theory ” of the effects of the project/program 计划会将研究结果和项目 / 计划效果 “ 理论 ” 联系在一起 www.dadangsolihin.com

25. Potential Influences on Program Results 对于计划结果的可能影响 www.dadangsolihin.com Inputs 投入  Activities 活动  Outputs 产出 Outcomes 成果  Impacts 影响 Black Box 黑箱 Political Environment (Governance, etc.) 政治环境（治理，等） Public Attitudes 公众态度 Environment 环境 Aid Players 援助参与者 Macro-economic Picture 宏观经济形势 Policy Context 政策背景

26. Why Use a Theory of Change 为什么使用变革理论 Helps identify elements of programs critical to success 有助于明确对于成功至关重要的计划元素 Helps build a common understanding of the program and expectations among stakeholders 有助于在利益相关这种建立计划和预期的共识 Identifies and measures progress on intervening variables 明确并衡量中介变量的进展 Identifies assumptions that may become the basis of evaluation questions 明确可能成为评价问题基础的假设 www.dadangsolihin.com

27. Review or Create 审查或制订 If the program/project already has a developed theory of change, carefully review it 如果计划 / 项目已经具备了成熟的变革理论，则需仔细审查： may need to refine or rework it 是否需要对其加以改进或重新修订 If there is no theory of change, need to create one 如果尚不具有变革理论，则需制订 www.dadangsolihin.com

28. Constructing a Theory of Change 建立变革理论 Is there research underlying the program? 有没有突出计划的研究？ What is the logic of the program? 计划的逻辑是什么？ What are the key assumptions being made? 做出了什么关键的假设？ www.dadangsolihin.com

29. Process for Constructing a Theory of Change 建立变革理论的流程 www.dadangsolihin.com Research 研究

30. Is There Research Underlying the Program? 有没有突出计划的研究 Evaluative studies within organization 组织内的评价性研究 OECD DAC repository of Publications and Documents or Information by Country 经合组织发展评价委员会出版物资料库以及按照国家来分得文件或信息 Evaluation studies done by development organizations 由发展组织完成的评价研究 Evaluation studies in related fields 在相关领域的评价研究 Research of theories of development 发展理论的研究 www.dadangsolihin.com

31. What is the Logic of the Program? 计划的逻辑是什么？ Identify probable occurrences surrounding the intervention: 明确干预活动周围的可能发生的事件 long-term goals 长期目标 specify the details about 明确说明下列有关细节 : target audience 目标受众 amount of change to signal success 标志成功的变化量 Timeframe 时间框架 specify the chain of events 明确说明一些列事件 www.dadangsolihin.com

32. Questions to Check Logic 检查逻辑的问题 Is this plausible ? Is it likely to lead to the long-term goal? 是否有道理？ 有没有可能达到长期目标？ Is this feasible ? Are the capabilities and resources to implement the strategies possible? 是否可行？是否有执行战略的能力和资源？ Is it testable ? Are the measurements of how success will be determined specific? 是否可测试？衡量成功与否的测量是否明确？ www.dadangsolihin.com

33. Mapping or Drawing the Logic 规划逻辑 Can be done in many different ways 可以用不同的方式完成 Most are not linear, but might be circular, or a storyboard 大多数不是直线的，但是有可能是循环的或是情节串连的 Show complex relationships 显示复杂关系 Usually have boxes and/or arrows that link back to earlier or ahead to later parts of the theory 通常有框和 / 或箭头指向之前或之后的理论部分 www.dadangsolihin.com

34. What are the Key Assumptions being Made? 做出了什么关键假设 Assertions about the connections between long-term, immediate, and early outcomes 对于长期，中期和短期成果之间联系的主张 Substantiation for the claim that all of the important preconditions for success have been identified 证实所有成功的先决条件已经明确的断言 Justifications supporting the links between program activities and outcomes they are expected to produce 支持项目活动和预期成果之间联系的证明 www.dadangsolihin.com

35. Theory of Change Template 变革理论模版 IPDET www.dadangsolihin.com Strategies 战略 Assumptions 假设 Influential Factors 有影响力的因素 Problem or Issue 问题 Community Needs/Assets 社区需求 / 资产 Desired Results (outputs, outcomes, and impact) 期望结果（产出，成果，影响） 1 2 3 4 5 6 Source: WK Kellogg Foundation, 2004

36. Simple Theory of Change Model 简单逻辑模型 By training 通过培训 www.dadangsolihin.com Training 培训 High Quality Evaluations 高质量的评价 Useful Information 有用的信息 Better Decisions 更好的决策 We should get better decisions 我们将获得更好的结果

37. Theory of Change with Assumptions 变革理论及其假设 www.dadangsolihin.com Resources given to follow what they learned 足够的资源 Training 培训 High Quality Evaluations 高质的培新 Useful Information 无用的信息 Better Decisions 更好的决定 Needs of trainees met 培训者的需求得到满足 Report writing skills adequate to communicate with government 报告写作技巧足够同政府交流 Enough time to learn 足够的时间来学习

38. Example of Program Theory Model 计划理论模型示例 Visits by teachers to students’ homes 老师对学生进行家访 www.dadangsolihin.com Sharing of views by parent and teacher 家长和老师交流各自的观点 Teachers’ understanding of the home culture 老师了解学生家庭文化 Teachers’ sympathy with children and their view of the world 老师体谅学生及其世界观 Teaching in terms comfortable and understandable to students 以轻松、易接受的方式教育学生 Student moral 学生道德 Achievement in reading 阅读方面的成绩 Parents’ knowledge of school’s expectations for students 家长了解学校对学生的期望 Parental support and encouragement with child’s homework and school assignments 家长支持并鼓励学生完成家庭作业及学校分配的任务 Conscientiousness of work by students 学生的工作责任心 Identification of special problems that retard student’s achievement ( health, emotional, etc.) 发现影响学生表现（健康、情绪等）的特殊问题 Parents’ knowledge of school’s expectations for students 家长了解学校对学生的期望 Student’s receipt of special help 学生接受特殊帮助 Improvement of condition (health, emotional) 状态（健康、情绪）的改善 Parental support for better attendance at school 家长支持学生保持高出勤率 Student attendance 学生出勤

39. Example Basic Model Theory of Change (Research Grant Proposal) 变革理论的基本模型举例 www.dadangsolihin.com Selection Criteria 选择标准 Selecting Applicants 选择申请者 Enhanced Selection Process 提升的选择过程 Improved Research Quality 提升了的研究质量 Increased Use of Research Findings 研究发现的利用 Activities 活动 Outputs 产出 Immediate Results 立即的结果 Intermediate Results 暂时的结果 Final Results 最终的结果

40. Flow Chart of Classic Logic Model 经典逻辑模型流程图 活动 产出 近期结果 中期结果 最终结果 www.dadangsolihin.com

41. Logic Model Emphasizing Assumptions 强调假设的逻辑模型 www.dadangsolihin.com Assumptions 假设 Inputs 投入 Activities 活动 Outputs 产出 Outcomes 产效 The Beginnings 开始 The Planned Work 计划的工作 The Intended Results 预计的结果 Source: Adapted from WK Kellogg, 2004

42. Results Chain 结果链 www.dadangsolihin.com Inputs (Resources) 投入（资源） Activities 活动 Outputs 产出 Short-Term Results (Direct) 短期结果（直接） Intermediate Results (Indirect) 中间结果（间接） Long Term Result 长远结果 Area of Control Internal to the Organization 对于组织的内部控制领域 Outputs Reach Direct Beneficiaries 产出直接联系到受益人 Area of Influence External to the Organization 外部对于组织的影响 External Factors 外部因素

43. www.dadangsolihin.com

44. www.dadangsolihin.com Impact nutrition, health, and clothing 对营养、健康和着装有影响 Women have limited access to economic opportunities due to lack of access to credit and other productive resources and to social control 由于缺少获得信贷和其它生产资源的机会，以及社会控制的原因，妇女很难获得经济机会 Project offers credit, technical assistance and group information 计划能够提供信贷、技术援助和大量信息 Women create business 妇女创办企业 Generate profits 盈利 Profits re-invested 利润再投资 Short-term improvement in household welfare 家庭福利的短期改善 Permanent improvement in household welfare 家庭福利的永久改善 Improved housing 居住条件得以改善 Improved education for girls 女孩的教育条件得到改善 Economic improvements 经济状况改善 Business sustained 企业持续发展

45. Example of Program Outcome Model 计划成效模型示例 www.dadangsolihin.com  Resources 资源 ▪ Money 资金 ▪ Staff 人员 ▪ Volunteers 志愿者 ▪ Supplies 物料供应 • Eligible participants 合格的参与者 Services 服务 ▪ Training 培训 ▪ Education 教育 ▪ Counselling 咨询服务 Products 产品 ▪ Total # of classes 班级总数 ▪ Hours of service 服务时间（小时） ▪ Number of participants completing course 完成课程的参加者人数 Benefits 效益 ▪ New knowledge 新知识 ▪ Increased skills 技能提升 ▪ Changed attitudes 态度改变 ▪ New employment opportunities 新的就业机会 Changes 改变 ▪ Trainers earn more over five years than those not receiving training 培训者在 5 年内比那些没有接受培训的人挣得更多 ▪ Trainees have higher standard of living than the control group 接受培训的人的生活水平高于控制组 Inputs 投入 Activities 活动 Outputs 产出 Impacts 影响 Outcomes 成效

46. Logical Framework (Logframe) 逻辑框架 Links the activities, results, purpose, and objectives of an intervention 连接一项干预措施的活动、结果、目的和目标 Is a specific logic model using a 4 x 4 matrix 是一个使用 4 x 4 矩阵的特定逻辑模型 For each component, the evaluator identifies the: 对于每一项内容，评价人员都要明确： indicators that are needed 必需的指标 sources 来源 assumptions 假设 www.dadangsolihin.com

47. Logframe Model 逻辑框架 Narrative Summary 描述性总结 Performance Indicators 绩效指标 M&E/Supervision/ Verification 监测与评价 / 监督 / 核实 Key Assumptions 关键假设 Program Goal: 计划目标： Project Development Objective: 计划发展目标： Outputs: 产出： Components: 组成：

48. IPDET www.dadangsolihin.com GOAL (general objective) Improve the economic and social welfare of women and their families INDICATORS Improvements in family income in x% of participating families Improvements in measures of health status, nutritional status, and educational participation VERIFICATION Household surveys of the economic, social, and health condition of all family members PURPOSE (specific objective) Provide women with opportunities to earn and learn while their children are cared for in home day care centers INDICATORS  day care homes functioning, providing accessible, affordable care of adequate quality during working hours and thus allowing shifts in women's employment and education activities VERIFICATION From surveys: changes in women's employment and education and their evaluations of the care provided Evaluations of quality of care provided based on observation ASSUMPTIONS Other family members maintain or improve their employment and earnings Economic conditions remain stable or improve OUTPUTS Trained caregivers, supervisors, and directors Day care homes upgraded and operating Materials developed Administrative system in place MIS in place INDICATORS  caregivers trained  homes upgraded and operating  materials created and distributed  a functioning MIS VERIFICATION Data from MIS on trainees, homes, and materials Evaluations of trainees after initial training and during course of continuous training ASSUMPTIONS Family conditions allow home day care mothers to carry through on their agreements to provide care ACTIVITIES Select caregivers and supervisors and provide initial training Upgrade homes Develop materials Develop administrative system Deliver home day care Provide continuous training and supervision Develop monitoring and evaluation system RESOURCES Budget Technology Human resources VERIFICATION Plan of action, budgets, and accounting records Studies showing that the chosen model and curriculum work Evaluations to see that the activities were not only carried out but done well Survey o

49. Assessing a Theory of Change: Viewpoint or Framework 评估变革理论：观点与框架 In relation to social needs 同社会需求的关系 Of logic and plausibility 有关的逻辑以及可行性 Through comparison with research and practice 通过研究和实践来进行比较 By confronting a program theory with one or more relevant scientific theories 通过一个或多个相关的科学理论来挑战一个项目理论 Via preliminary observation 通过前期观察 www.dadangsolihin.com

50. Theories of Change Models should Answer these Questions: 变革模型理论需要回答以下问题 ： Is the model an accurate depiction of the program ? 模型是否是计划的准确描述？ Are all elements well defined ? 所有的元素是否都准确的定义？ Are there any gaps in the logical chain of events ? 在事件的逻辑链中有没有任何缺口？ Is it realistic to assume that the program will result in the attainment of stated goals in a meaningful manner? 假设计划能以一种有意义的方式实现既定目标，这是否现实？ www.dadangsolihin.com

51. Thank you! 谢谢！

52. Beside working as Assistant Professor at Graduate School of Asia-Pacific Studies, Waseda University, Tokyo, Japan, he also active as Associate Professor at University of Darma Persada, Jakarta, Indonesia. He got various training around the globe, included Shanghai International Program for Development Evaluation Training (2008) , Public Officials Capacity Building Training Program for Government Innovation, Seoul –Korea (2007), Advanced International Training Programme of Information Technology Management, at Karlstad City, Sweden (2005); the Training Seminar on Land Use and Management, Taiwan (2004); Developing Multimedia Applications for Managers, Kuala Lumpur, Malaysia (2003); Applied Policy Development Training, Vancouver, Canada (2002); Local Government Administration Training Course, Hiroshima, Japan (2001); and Regional Development and Planning Training Course, Sapporo, Japan (1999). He published more than five books regarding local autonomous. You can reach Dadang Solihin by email at [email_address] or by his mobile at +62812 932 2202 Dadang Solihin currently is Director for Regional D evelopment Performance Evaluation at Indonesian National Development Planning Agency (Bappenas). He holds MA degree in Economics from University of Colorado, USA. His previous post is Director for System and Reporting of Development Performance Evaluation at Bappenas. Dadang Solihin’s Profile www.dadangsolihin.com