This document outlines the strategies for selecting and constructing data collection instruments for evaluations, emphasizing the importance of a structured or semi-structured approach depending on the research goals and context. It discusses various data collection methods, including surveys, focus groups, and participatory approaches, while highlighting key measurement considerations such as credibility, validity, reliability, relevancy, and precision of data. Additionally, it provides practical tools and guidelines for effective data collection, stressing the importance of understanding the context and resources available.

1. Selecting and Constructing Data Collection Instruments 选择和构建数据收集工具 Shanghai International Program for Development Evaluation Training Asia-Pacific Finance and Development Center 200 Panlong Road-Shanghai, October 14, 2008 Linda Morra-Imas

2. Linda Morra-Imas Chief Evaluation Officer & Advisor Evaluation Capacity Building I nternational F inance C orporation 2121 Pennsylvania Avenue, NW Washington DC 20433 USA [email_address] www.ifc.org/ieg Dadang Solihin Indonesia Delegation Shanghai International Program for Development Evaluation Training

3. Introduction 引言 Data Collection Strategies 数据收集策略 Data Collection General Rules 数据收集的一般规则 Key Issues about Measures 有关测量的关键问题 Quantitative and Qualitative Data 定量和定性数据 Introduction to The Toolkit 数据收集工具介绍 Techniques for Using Surveys 使用调查的技术 Techniques for Using Focus Groups 使用专题组的技术 www.dadangsolihin.com

4. No One Best Way 不存在最好的方法 Decision Depends On: 方法选择取决于： What you need to know 你须要知道什么 numbers or stories 数字或故事 Where the data reside 数据在什么地方 environment, files, people 环境、文件、人 Resources and time available 可用的资源和时间 Complexity of the data to be collected 所要收集的数据的复杂性 Frequency of data collection 数据收集频率 www.dadangsolihin.com

5. Structured Approach 结构化方法 All data collected in the same way 用相同的方法收集所有数据 Important for multi-site and cluster evaluations so you can compare 对多点评价和聚类评价是重要的，这样你能够进行比较 Important when you need to make comparisons with alternate interventions 当你需要在备选的干预措施之间进行比较时，结构化方法很重要 www.dadangsolihin.com

6. When to Use Structured Approach: 什么时候要使用结构化的方法： You wish to be very precise 你希望结果非常精确 You are working with a large sample or populations 你的研究样本或总体很大 You are sure of what you wish to measure 你确定了你希望测量的事物 You want to show your results analytically 你想要显示出结果的分析性 You need to make comparisons across different sites or interventions 你须要在不同实施地点或干预措施之间进行对比 www.dadangsolihin.com

7. Semi-structured Approach 半结构化方法 Systematic and follow general procedures but data are not collected in exactly the same way every time 系统的遵照通用的程序，但数据的收集并非每次都是用完全相同的方法 More open and fluid 更具开放性和灵活性 Does not follow a rigid script 没有严格按照文本而来 may ask for more detail 可能需要更多细节 people can tell what they want in their own way 人们可以用他们自己的方式来讲述自己所要讲的事情 www.dadangsolihin.com

8. Use Semi-structured Approach 半结构化方法的使用 when Evaluator : 当评价员： Is conducting exploratory work in a new development area 在新的发展领域内进行探索性的工作时 Is seeking understanding, themes, and/or issues 在寻找理解、主题和 / 或议题时 Want anecdotes, stories, or in-dept information 想知道轶事、故事或深入的信息时 Is not sure of what you wish to measure 不清楚你希望测量的是什么时 Has no need to qualify 不需具备评价资格时 www.dadangsolihin.com

9. Use available data if they already exist 如果有现成信息，则利用这些信息 If using available data, be sure to find out how they are: 如果利用这些信息，保证找到它们如何： collect the data 收集信息 defined the variables 界定变量 ensure accuracy of the data 保证数据的精确性 Data Collection General Rules 数据收集的一般规则 www.dadangsolihin.com

10. General Rules (cont.) 一般规则 ( 续 ) If you must collect original data: 如果你必须收集原始数据： establish procedures and follow them 制定数据收集程序并遵照它 maintain accurate records of definitions and coding 保持定义和编码的准确记录 pre-test, pre-test, pre-test 反复进行预检 verify accuracy of coding, data input 验证编码和数据输入的精确性 www.dadangsolihin.com

11. Key Issues about Measures 有关测量的关键问题

12. Issues: 问题： Are the measures credible? 测量可信吗？ Are the measures valid? 测量有效吗？ Are the measures relevant? 测量是否相关？ Are the measures reliable? 你的测量可靠吗？ Are the measures precise? 你的测量精确吗？ www.dadangsolihin.com

13. Credibility 可信性 How trustworthy or believable your data collection is 你的数据收集的可信性如何 Are the data you are collecting giving you information about the actual situation? 你正在收集的数据是否提供了有关真实情况的信息？ Make sure data you are collecting are relevant and most important information 确定你正在收集的数据是相关的并且是最重要的信息 www.dadangsolihin.com

14. Validity 有效性 Does the measurement actually measure what it is supposed to? 所进行的测量是否是真正想要测量的？ Two kinds of validity 两种有效性 face validity contents of the test or procedure look like they are measuring what they are supposed to measure 表面有效性 测试或程序的内容看起来它们正式在测量要求它们测量的一样 content validity content of the test or procedure represents all that is required for validity 内容有效性 测试或者程序的内容能够代表所有有效性所要求的 www.dadangsolihin.com

15. Reliability 可靠性 Term to describe the stability of your measurement 用于描述测量稳定性的术语 Measures the same thing, same way in repeated tests 测量相同的事物，在重复的检验中使用相同的方法 Examples: 示例： in sports, speed measured by stopwatch 在体育运动中，用秒表测量速度 birth weights of newborn infants 新生婴儿出生时的体重 attendance rates at schools 学校的出勤率 www.dadangsolihin.com

16. Relevancy 相关性 Does the measurement actually measure what they counts? 测试方法是否真的测试它们计算的东西？ The most important information 最重要的信息 Do not just measure what is needed or just trying to measure everything 不要只测试需要的或者试图测试所有的信息 Design matrix is a tool to guide relevance 设计矩阵是指导相关性的工具 www.dadangsolihin.com

17. Precision 精确性 How the language used in the data collection matches the measure 数据收集中使用的语言与测量的匹配度如何 Example: 示例： if the question is about countries, the measures must be at the national level 如果问题是有关国家的，那么就必须在国家层面上进行测量 if the question is about people, the measures must be on the individual level 如果问题是有关人的，那么就必须在个人层面上进行测量 www.dadangsolihin.com

18. Reliability 可靠性 Term to describe the stability of your measurement 描述方法可靠性的词汇 Measures the same thing, same way in repeated tests 测量同一个事物，以同样的方式重复测量 Examples: 例子 - in sports, speed measured by stopwatch 运动中，速度由运动表来测量 - birth weights of newborn infants 新出生婴儿的重量 - attendance rate at schools 学校中的出勤率 www.dadangsolihin.com

19. Quantitative and Qualitative Data 定量数据和定性数据

20. Quantitative Approach 定量方法 More structured 结构性更强 Attempts to provide precise measures 试图提供精确的测量结果 Reliable 可靠 Harder to develop 方法的开发比较难 Easier to analyze 分析起来比较容易 www.dadangsolihin.com

21. Qualitative Approach 定性方法 Less structured 结构性较差 Easier to develop 更容易开发 Can provide “ rich data ” — detailed and widely applicable 能够提供 “ 丰富的数据 ”—— 详细且广泛适用 Is challenging to analyze 对分析是个挑战 Is labor intensive to collect 收集数据要做很多工作 Usually generates longer reports 通常会产生比较长的报告 www.dadangsolihin.com

22. Which Data? 使用什么样的数据？ do not need to quantify 不需要量化 are not sure what you want to measure 还不确定你所要测量的是什么 定性数据 want anecdotes or in-depth information 想要了解没有记载的或深入的信息 want to cover a large group 想要包含一个大的群体 - know exactly what you want to measure 准确地了解你所想要测量的是什么 定量数据 want to do statistical analysis 想要做一些统计分析 Then Use: 那么就使用： If you: 如果你：

23. Obtrusive vs. Unobtrusive Methods 介入式 方法 vs. 非介入式方法 Obtrusive – observations are made of behavior with the participant ’ s knowledge 介入式 – 观察是由行为组成的，且是在参与者知道的情况下进行的 interviews, surveys, focus groups 面访、调查、专题组面谈 Unobtrusive – observations done without the knowledge of the participant 非介入式 – 在参与者不知道的情况下进行的观察 historical/document/archival data 历史数据 / 文件数据 / 档案数据 watching participants at a distance 远距离观察参与者 www.dadangsolihin.com

24. Introduction to The Toolkit 数据收集工具介绍

25. How to decide on data collection approach 如何决定数据收集方法 Choice depends on the situation 方法的选择视情况而定 Each technique is more appropriate in some situations than others 每项技术在某种情况下会比其它技术更合适 Caution: All techniques are subject to bias 注意：所以的技术都会有偏差 www.dadangsolihin.com

26. Combinations 各种方法的结合 Can use a variety of data collection approaches to answer different questions or for multiple sources 可以使用多种数据收集方法以回答不同的问题或针对多种数据来源 Triangulation: collect same information using different approaches 三元法：使用不同方法来收集相同的信息 www.dadangsolihin.com

27. Measurement Considerations 有关测量的考虑 If people know they are being studied, they may act differently 如果人们知道他们在被人研究，他们的行为会不同 women may respond differently to a male interviewer or if their spouse is present 如果女士的配偶在场的话，她们对于男性面谈者的态度会不同 Starting by specifying what you want to measure the choices become clearer about answering questions 一开始就说明你想要测量什么 www.dadangsolihin.com

28. Data Collection 数据收集 Participatory Data Collection 参与式数据收集 Available Data 现有数据 Observation 观察 Surveys 调查 Focus Groups 专题组 www.dadangsolihin.com

29. Data Collection (Cont.) Diaries, Journals, Self-reported Checklists 日记、日志、自报告核查清单 Expert Judgment 专家评判 Delphi Technique 德尔菲技术 Citizen Report Cards 公民报告卡 www.dadangsolihin.com

30. Toolkit 1: Participatory Data Collection 工具 1 ：参与式数据收集 Data that are collected when interacting with people 在与人们的互动中收集数据 Examples: 示例： community meetings 社区会议 transect walks 直接观察法 social mapping 社会映射 www.dadangsolihin.com

31. Community Meetings 社区会议 One of the most common participatory methods 最为普遍的一种参与方式 Must be well organized 必须组织良好 agree on purpose 就目的达成一致 establish ground rules 建立基本规则 who will speak 谁发言 time allotted for speakers 发言人的时间分配 format for questions and answers 问与答的形式 www.dadangsolihin.com

32. Mapping 映射 Collect and plot information on the distribution, access, and use of resources within a community 收集并划分有关社区内资源的分布、获取和使用的信息 Useful tool to involve stakeholders 使利益相关者参与进来的有用工具 provides a way to work together 提供一种在一起工作的方法 increases understanding of the community 增加对社区的了解 generates discussions, verifies secondary sources of information, perceived changes 引起讨论，核实辅助来源的信息 , 感知变化 www.dadangsolihin.com

33. Mapping Process 映射过程 Draws a picture of the community 描绘社区的整体情况 Places resources and assets in their locations 将资源和资产放在各自的地点 individual assets: skills, talent, networks, money, etc. 个人资产：技能、才干、网络、资金等 civic assets: faith associations, clubs, social groups, etc. 公民资产：宗教团体、俱乐部、社会团体等 institutional assets: businesses, schools, health services, public transportation, etc. 机构资产：企业、学校、健康服务机构、公共交通等 environmental assets: parks, roads, farmland, housing, clean air and water, etc. 环境资产：公园、公路、农田、住房、清洁的空气和水等 www.dadangsolihin.com

34. Transect Walks 直接观察法 Evaluators walk around community observing people, surroundings, and resources 评价者在社区走动，对人、四周以及资源进行观察 Need good observation skills 需要良好的观察技巧 Walk a transect line through a map of a community — line should go through all zones of the community 沿社区地图的横切面的线条走，线条应该穿过社区的所有区域 www.dadangsolihin.com

35. Toolkit 2: Available Data 工具 2 ：现有数据 Example sources: 数据来源举例： files/records 档案 / 记录 computer data bases 计算机数据库 government reports 政府报告 other reports or prior evaluations 其它报告或以前的评价 census data 普查数据 documents (budgets, organizational charts, policies and procedures, maps) 文件（预算、组织机构图、政策和程序、地图） www.dadangsolihin.com

36. Using Existing Records 使用机构的记录 Most organizations have already collected the data from clients, community, and internal information systems 大多数机构已经从客户、社区和内部信息系统收集了你需要的数据 May also have summaries and/or reports, such as: 可能已经有了一些总结和 / 或报告，例如： internal management reports 内部管理报告 budget documents 预算文件 reports to the public or funding agencies 向公众或出资机构提交的报告 Key issues: 关键问题： validity, reliability, accuracy 有效性、可靠性和准确性 www.dadangsolihin.com

37. Advantage/Challenge: Available Data 优点 / 挑战：现有数据 Advantages 优点 Often less expensive and faster than collecting the original data yourself 与自己收集原始数据相比成本较低速度较快 Challenges 挑战 There may be coding errors or other problems. Data may not be exactly what is needed. You may have difficulty getting access. You have to verify validity and reliability of data 可能存在编码错误或其它问题。数据可能不是需要的确切数据。你在获取数据方面可能有困难。你必须核实数据的有效性和可靠性。

38. Toolkit 3: Observation 工具 3 ：观察 See what is happening 观察正在发生什么 traffic patterns 交通模式 land use patterns 土地使用模式 layout of city and rural areas 城市和农村地区规划 quality of housing 住宅的质量 condition of roads 公路状况 conditions of buildings 建筑状况 who goes to a health clinic 谁到卫生诊所就诊 www.dadangsolihin.com

39. Ways to Observe 观察方法 Unobtrusive 非介入式 No one knows you are observing 没有人知道你正在进行观察 Participant 参与式 You actually participate in the activity 你实际上参与到活动当中去了 Obtrusive 介入式 The people being observed know you are there to observe them 正在被观察的人知道你在观察他们

40. Guidelines for Planning Observations 规划观察的指导纲要 For less formal approach, have a few key questions in mind when you arrive 对于不太正式的方法，当你到达观察地点时，脑子里准备一些问题 www.dadangsolihin.com 1. Develop a check list to rate your observations. 设计一个核查清单来给你的观察评级 2 Develop a rating scheme. 设计一个评级方案 3. Have more than one observer, if feasible. 如果可行的话，要有多个观察员 4. Train observers so they observe the same things. 对观察员进行培训，这样他们能够观察相同的事物 5. Pilot test the observation data collection instrument. 通过试点检验观察数据收集工具

41. Advantages and Challenges: Observation 观察的优点与挑战 Advantages 优点 Collects data on actual vs. self- reported behavior or perceptions. It is real-time vs. retrospective 收集有关实际行为或理解的数据与自报告数据作比较，实时数据与回顾数据的对比 Challenges 挑战 Observer bias, potentially unreliable; interpretation and coding challenges; sampling can be a problem; can be labor intensive ； low responses rate 观察者的偏见，可能不可靠；解释和编码方面的挑战；抽样可能是个问题；要花费大量的劳力；反应率低

42. Toolkit 4: Surveys 工具 4 ：调查 Excellent for asking people about: perceptions, opinions, ideas 是一种用来询问人们的对事物的理解、观点、想法的很好的工具 Less accurate for measuring behavior 测量行为的精确度较差 Sample should be representative of the whole 样本应该对总体具有代表性 Big problem with response rates 回复率是个问题 www.dadangsolihin.com

43. Methods for Surveys 调查方法 In-person interviews 亲自面谈 Mail /phone/Internet interviews or surveys 邮件 / 电话 / 网络面谈或调查 Self-administered questionnaires 自填式问卷 www.dadangsolihin.com

44. Structures for Surveys 调查的结构性 Structured: 结构化的： Precisely worded with a range of pre-determined responses that the respondent can select 用一系列预先设计好的受访者能够选择的回答选项来精确地设计调查 Everyone is asked exactly the same questions in exactly the same way, given exactly the same choices 用严格相同的方法向每个人询问严格相同的问题，并给出严格相同的答案选项 Semi-structured 半结构化的 Asks the same general set of questions but may leave many, it not all, of the answers open-ended 询问一组相同的问题，但可能包含许多（不是全部）开放式问题 www.dadangsolihin.com

45. Structured vs. Semi-structured Surveys 结构化调查与半结构化调查的对比 Structured 结构化 harder to develop 设计较难 easier to complete 完成较容易 easier to analyze 分析较容易 more efficient when working with large numbers 当在处理大量调查对象时效率更高 Semi-structured 半结构化 a little easier to develop 设计较为容易 labor intensive to conduct 实施起来耗费较多劳力 harder to analyze but provide a rich source of data 分析较为困难，但提供了丰富的数据 subject to bias in interpreting 在解释时会有偏差 / 受偏见的影响 burdensome for people to complete as a self-administrated questionnaire 给完成自填式问卷的人造成负担

46. Advantages and Challenges of Surveys 调查的优点与挑战 Advantages 优点 Best when you want to know what people think, believe, or perceive, only they can tell you that 当你想要知道人们在想什么、相信什么、感知什么的时候，这是最好的方法，只有这些方法能够告诉你 Challenges 挑战 People may not accurately recall their behavior or may be reluctant to reveal their behavior if it is illegal or stigmatized. What people think they do or say they do is not the same as measuring what they actually do . 人们可能无法准确地回忆起他们的行为，或者如果行为是违法的或受污蔑的，他们可能不愿意暴露这些行为。人们 认为他们做了什么 或 说他们做了什么 与测量他们 实际上做了什么 并不一样

47. In-person Interviews 亲自面谈 Useful for in-depth understanding of experiences, opinions, or descriptions 对于深入了解过程、观点或描述有用 Useful when other approaches do not work 当其它方法不起作用的时候有用 Should be conversational 应该是会话式的 Can be done individually or in groups 可以进行个人面谈或者小组面谈 Can be structured or semi-structured 可以是结构化的或半结构化的 www.dadangsolihin.com

48. Mail / Phone / Internet Interviews and Surveys 邮件 / 电话 / 网络面谈和调查 Literacy issues 读写能力问题 Consider accessibility 考虑可达性 reliability of postal service 邮寄服务的可靠性 turn-around time 回收时间 do respondents have telephone access? 受访者是否能够使用电话？ do they have Internet access? 受访者是否能使用互联网？ www.dadangsolihin.com

49. Self-administered Questionnaires 自填式问卷 Written surveys that the respondent completes 由受访者完成的书面调查 Can be structured, semi-structured, or a combination 可以是结构化的、半结构化的、或两者结合 Should be short (no more than 20 min.) 应该简短（不要超过 20 分钟） Usually include one or two open-ended questions 通常包括一个或两个开放式问题 help make respondents more comfortable 有助于使受访者觉得更舒适 www.dadangsolihin.com

50. Advantages of Interviews 面谈的优点 Can be structured, unstructured, or a combination 可以是结构化的、半结构化的、或两者结合 Can explore complex issues in depth 能够深入地探讨复杂的问题 Forgiving of mistakes: unclear questions can be clarified during the interview and changed for subsequent interviews 容许错误：不清楚的问题能够在面谈中得到澄清，并且在随后的面谈中进行改变 Can provide evaluators with an intuitive sense of the situation 能够向评价 人员提供对情况的直觉了解 www.dadangsolihin.com

51. Challenges of Interviews 面谈的挑战 Can be expensive, labor intensive, and time consuming 可能需要耗费较多的成本、劳力和时间 May not be able to explore why people have different viewpoints 可能无法探究为什么人们有不同的观点 Selective hearing on the part of the interviewer may miss information that does not conform to pre-existing beliefs 面谈者有选择的倾听有可能会漏掉与以前相信的东西不一致的信息 Cultural sensitivity: gender issues 文化敏感性：性别问题 www.dadangsolihin.com

52. Toolkit 5: Focus Groups 工具 5 ：专题组 Type of qualitative research where small homogenous groups of people are brought together to informally discuss specific topics under the guidance of a moderator 一种定性研究方法，将一小群具有共同特征的人集中起来，在主持人的指导下对具体问题进行非正式地讨论 Purpose: to elicit reliable data, not just interesting information 目的：得出可靠的数据而不仅仅是有趣的信息 www.dadangsolihin.com

53. Uses of Focus Groups 专题组的使用 Help develop a survey questionnaire 帮助设计调查问卷 Clarify sample selection 阐明样本的选取 Contextualize survey data 把调查数据放在背景当中 Be used in tandem with surveys 与调查一起使用 Be used as a separate data collection tool 作为一种单独的数据收集工具来使用 www.dadangsolihin.com

54. Focus Group Process 专题组过程 Step Process 步骤 程序 1 Introduce the focus group meeting 介绍专题组会议 2 Have the participants introduce themselves 让参与者进行自我介绍 3 Present the first question, it should be easy, an ice-breaker 提出第一个问题，问题应该简单，起到 “ 破冰者 ” 的作用 4 Ask the main questions 提问主要问题 5 Ask the last (summary) questions 提问最后一个（总结性）问题 6 Ask if there are other comments or questions 问一下是否还有其它的说明和问题 7 Write-up impressions, major issues and points of discussion 整理对讨论的印象、讨论的主要议题和内容

55. Advantages and Challenges of Focus Groups 专题组的优点和挑战 Advantages 优点 Relatively quick and easy, may take less staff time than in-depth, in-person interviews; provides flexibility to make changes in process and question; ability to explore different perspectives; it can be fun 相对快速和容易，可能比深入的亲自面谈要节省工作人员的时间；在过程和问题中可以进行灵活的改变；能够探讨不同的看法；可能较为有趣 Challenges 挑战 Analysis is time consuming, participants might be different from rest of population; risk of bias in interpreting data; risk of group being influenced by moderator or dominant members 分析很耗时，参与者可能与其余人群不同；在解释说明数据时存在偏见的风险；存在小组会受到主持人或权威成员影响的风险

56. Toolkit 6: Diaries, Journals, Self-Reported Checklists 工具 6 ：日记、日志、自报告核查清单 Use when you want to capture information about events in people ’ s daily lives 当你想要获取有关人们日常生活中发生的事件的信息时使用 Participants capture experiences in real-time not later in a questionnaire 参与者获得了实时经验，而不是之后从问卷中获得的经验 Used to supplement other data collection 作为其它数据收集的补充 www.dadangsolihin.com

57. Guidelines for Diaries or Journals 日记或日志方法使用指南 www.dadangsolihin.com Step Process 步骤 过程 1 Recruit people face-to-face 面对面地招募参与者 • encourage participation, appeal to altruism, assure confidentiality, provide incentive 鼓励参与，引起对利他主义的兴趣，确保机密性，提出激励措施 2 Provide a booklet to each participant 向每人提供一本小册子 • cover page with clear instructions, definitions, example 封面要写明清晰的指示说明、定义和示例 • short memory-joggers, calendar 简短的备忘录 , 日历 3 Consider the time-period for collecting data 考虑收集数据的时间周期 • if too long, may become burdensome 如果时间太长，可能成为负担 • if too short may miss the behavior or event 如果时间太短，可能错过一些行为或事件

58. Self-reported Checklists 自报告核查清单 Cross between a questionnaire and a diary 调查问卷和日记的交叉 The evaluator specifies a list of behaviors or events and asks the respondents to complete the checklist 评价 人员具体给出了一个行为或时间清单，并请受访者完成该清单 Done over a period of time to capture the event or behavior 在一个时期内进行以便获取事件或行为的信息 More quantitative approach than diary 是比日记更具有量化性的方法 www.dadangsolihin.com

59. Advantages and Challenges of Diaries, Journals … 日记、日志的优点和挑战 Advantages 优点 Rich data can capture the details that might be quickly forgotten over time 丰富的数据能够获知细节信息，而这些细节可能很快就会被忘掉 Good for information on how people use time 对于获得有关人们是如何使用时间的信息很有用 Helps in collecting sensitive information 有助于收集敏感信息 Supplements interviews 是对面谈的一种补充 Challenges 挑战 Requires literacy 要求有读写能力 May change behavior 可能会改变行为 Data may be incomplete or inaccurate 数据可能不完整或不精确 Poor handwriting, difficult to understand phrases 字迹潦草，理解困难

60. Toolkit 7: Expert Judgment 工具 7 ：专家评判 Interviews with experts, one-on-one or in a panel 与专家进行面谈，一对一或对一个专家组 Can be structured or unstructured 可以是结构化的或非结构化的 everyone answers a set of specific questions 每个人都回答一组特定的问题 or free flowing (focus group model) 或者自由发言（专题组模式） or formal presentations 或者正式报告 www.dadangsolihin.com

61. Selecting Experts 选择专家 Establish criteria for selecting experts based on: 基于以下考虑因素制定选择专家的准则： recognized expert 公认的专家 areas of expertise 专业知识领域 diverse perspectives 多样化的观点 diverse political views 多样化的政治观 diverse technical expertise 多样化的技术专业知识 www.dadangsolihin.com

62. Advantages and Challenges of Expert Judgment 专家评判的优点和挑战 Advantages 优点 Fast, relatively inexpensive 快速、成本相对较低 Communication of final results can give credibility 交流最终结果能够表明可信性 Challenges 挑战 Weak for impact evaluation 对于影响评价 而言显得不足 May be based mostly on perceptions 可能大多以感知为基础 Worth of data is only as good as the perceived credibility of the experts 数据的价值仅仅与专家感觉的可信性一样好

63. Toolkit 8: Delphi Technique 工具 8 ：德尔菲法 Enables experts who live in different locations to engage in a dialogue 使在不同地方的专家参与到对话中来 Experts asked specific questions 专家被提问特定的问题 Answers are returned to a central source for the evaluator to summarize and feed it back to the experts for further comments 答案被回收集中起来，以便评价人员进行总结并反馈给专家以获取进一步的意见 Answers are returned to a central source for the evaluator to summarize and feed it back to the experts for further comments 没有人知道谁说了些什么，因此避免了冲突 Experts can agree or argue with others ’ comments 专家可以同意其他人的意见或进行争辩 www.dadangsolihin.com

64. Advantages and Challenges of Delphi Technique 德尔菲法的优点及挑战 www.dadangsolihin.com Advantages 优点 Inexpensive 便宜 Conducive to independent thinking 有助于独立思考 Allows sharing of information 允许信息共享 Challenges 挑战 Judgments of a selected group only 只有选定小组的意见 Tendency to eliminate extreme positions 有消除极端立场的趋势 Time consuming and requires skill in communication 耗时且需要沟通技巧 Requires adequate time and participant commitment 需要充足的时间以及参与者的努力

65. Tool 9: Citizen Report (Score) Cards 工具 9 ：公民报告卡 Used to collect citizen feedback on public services from actual users of a service 用于收集公民对于公共服务的反馈 Can assess the performance of individual service providers and/or compare performance across providers 能测评单个服务提供商的绩效和 / 或比较各个提供商的服务 Can generate a database of feedback on services that can then be placed in the public domain 能生成一个关于服务反馈意见的数据库，之后可应用于公共领域 www.dadangsolihin.com

66. Advantages and Challenges of Citizen Report Cards 公民报告卡的优势及挑战 www.dadangsolihin.com Advantages 优势 Mixes focus groups and questionnaire data collection 混合了专题组及问卷调查的数据收集 Increases response rates 增加了回应率 Simple communication 简单的交流 Challenges 挑战 Local conditions must be conducive 必须具备有利的当地条件 Requires a large sample 需要一个大样本 Lack of predictability in how different players respond 不同的参与者如何回应，对此缺乏预测

67. Techniques for Using Surveys 使用调查的具体技术

68. Developing the Survey 开发设计调查 Define the purpose and objectives 定义目的和目标 Decide the important issues to be covered 决定所要包含的重要议题 Establish the relative weight of the different modules in the survey 确定调查中不同模块的相对权重 Identify important issues within sectors 确认各部分的重要议题 Question writers may need to learn more about how specific programs work 问题设计者可能须要了解更多有关特定计划是如何工作的知识 Once this background work is done, the actual writing of the survey may begin 一旦这个准备工作已经完成，真正的调查设计可以开始了 www.dadangsolihin.com

69. Six Types of Questions 六种问题 Experience and behavior 经验与行为 Opinion and value 观点与价值 Feeling 情感 Knowledge 知识 Sensory 感知 Background/Demographic 背景 / 人口 Each can be asked in present, past, or future 每种问题都能问现在、过去和未来情况如何 www.dadangsolihin.com

70. Six Types of Questions (Cont.) Many times, the step of communicating and consulting with policymakers is not given enough attention. Many policymakers are not familiar with surveys and may have difficulty reading complicated questionnaires. As well, they may not know how to imagine how the answers to the questions will be analyzed. For this reason, it is important to show policymakers and program managers examples of tables or other analyses that could be produced from the survey as well as the survey itself. Iterative Process According to the LSMS, the process of survey development is an iterative one. Once the initial version of the survey is drafted, the various interested parties should review the draft in detail. During their review, they should make notes and share their criticisms and suggestions. Revisions are then made to the original draft. This process may be repeated several times. www.dadangsolihin.com

71. Wording Questions 如何措辞 Use simple words 使用简单的词 Make questions and responses specific 使问题和回答都很具体 Avoid questions that use the words “and” and “or” 避免使用带有“和”以及“或者”的问题 Avoid questions that assume knowledge (may need to provide information) 避免使用需要知识的问题（需要提供信息） Avoid double negatives 避免双重否定 (continued on next slide) 下一页继续 www.dadangsolihin.com

72. Wording Questions (Cont.) LSMS points out that it is crucial for the team of question writers to have extensive local expertise when designing the questions. They bring the knowledge of the country’s society and of existing programs. They also know what issues should be emphasized. In some cases, they may also be familiar with earlier local surveys about some of the topics covered which may help them design pre-coded questions. In addition, they will know the network of people and institutions that should be contacted during the survey design process. www.dadangsolihin.com

73. Wording Questions 如何措辞 Make response options balanced 答案的选择应该平衡 Avoid objectionable, intrusive, and/or condescending 避免令人反感的、介入式或者诱导性的答案 If possible use an existing questionnaire as a guide 如果可能，利用已有的调查问卷 Make the survey easy to complete 确保问卷能很容易完成 www.dadangsolihin.com

74. Sequencing Survey Questions 如何进行次序调查 Move from easy and interesting to difficult and uncomfortable 从简单有趣的问题开始，逐渐转到难的、让人不舒服的问题 Move from general to specific 从一般问题到具体问题 Keep the flow logical (chronological or other order) 保证问题逻辑畅通（按照时间顺序或者其它顺序） Ask the most important questions by 2/3 through survey 整个调查中， 2/3 的地方问最重要的问题 www.dadangsolihin.com

75. Questionnaires Tips and Tricks 问卷调查的小技巧 Use simple, clear language, appropriate for audience 语言须简单、清楚、适当 Ask only one question at a time 一次仅问一个问题 Write your questions so that all feel their responses are acceptable 写下你的问题，这样所有的人都感到他们的答案是可接受的 When possible, write questions so that responses range from negative to positive 可能的话，写下问题，这样回答会从否定到肯定 (continued) www.dadangsolihin.com

76. Questionnaires Tips and Tricks (cont.) 问卷调查的小技巧 Avoid “yes” or “no” responses 避免 “是”或“否”的回答 Avoid absolutes at either end of the scale (always, never), consider using scales 避免出现极端（如使用“总是”、“从不”等词），平衡考虑 Ask questions about the current situation 问关于目前现状的问题 Leave exits (no opinion) 留下出口（非观点） Avoid using double negatives 避免使用双重否定 www.dadangsolihin.com

77. Field Testing 现场检验 Test with small number of subjects, diverse areas and socioeconomic groups 对少数几个题目、不同领域和社会经济组进行检验 Field test should look at three levels: 现场检验应该考察三个层次： as a whole 整体 each section 部分 individual questions 单个问题 Make revisions based on results of test 在测试基础上进行修改 www.dadangsolihin.com

78. Sequencing Interview Questions 对问题进行排序 Get the respondents involved in the interview as soon as possible 使受访者尽量参与到面谈中 Before asking about controversial matters, first ask about some facts 在提问有争议的问题前，先问一些事实 Intersperse fact-based questions throughout the interview 面向事实的问题要贯穿在整个面谈过程中 Ask questions about the present before questions about the past or future 先问有关当前情况的问题，再问有关过去或未来的问题 The last questions might be to allow respondents to provide any other information 最后一个问题应该允许受访者提供其它信息 www.dadangsolihin.com

79. General Guidelines for Conducting Surveys (1 of 3) 实施调查的一般指导原则 (3-1) Keep it simple, clear, easy, short 调查要简单、清晰、容易、简短 Locate other people who have done the kind of evaluation you are interested in and locate surveys similar to what you think you want to do 找到其他实施了你感兴趣的评价 的人并发现与你想做的类似的调查 Make sure people know why you are asking them to participate 使人们清楚你为什么邀请他们接受调查 Ask questions that are easy to answer and do not frustrate respondent ’ s desire to be clear in their responses 提出的问题要易于回答，切记不要影响受访者给出清晰回答的意愿 (continued on next slide 转下页 ) www.dadangsolihin.com

80. General Guidelines for Conducting Surveys (2 of 3) 实施调查的一般指导原则 (3-2) Do not ask respondents for information that requires them to go to a file or other source. If you must do this, you need to let them know in advance so the material can be assembled prior to administrating the survey. 不要向受访者要求他们须要查找文件或其它信息来源才能获得的信息。如果你必须这样做，你须要事先通知他们，以便他们在接受调查之前就整理好有关资料 Respect their privacy. Treat surveys confidentially and have procedures in place to assure privacy. Make sure you can insure confidentiality. Never promise confidentially unless it can be absolutely delivered. 尊重他们的隐私。做好调查的保密工作，并有保护隐私的程序。确信你能保密。除非能够做到绝对的保密，否则不要对保密性做出承诺。 (continued on next slide 转下页 ) www.dadangsolihin.com

81. General Guidelines for Conducting Surveys (3 of 3) 实施调查的一般指导原则 (3-3) Respect respondents ’ time and intelligence 尊重受访者的时间和知识 Tell them how they were selected and why their participation is important 让他们知道他们是如何被挑选出来的以及为什么他们的参与是重要的 Do no harm: keep responses confidential. For example, in your report, use aggregate responses; and assign an identification number to the data and destroy the link to the person ’ s name 不要有伤害行为：对回答结果进行保密。例如，在你的报告中，使用总结后的回答结果；对每个数据分配一个识别号码并使其他人无法据其联系到回答者的姓名 www.dadangsolihin.com

82. Developing an Interview 设计一个面谈 Step Procedure 步骤 程序 1 Define the purpose of the interview. Link your purpose to the evaluation objectives 明确面谈的目的。将你的目的与评价 目标联系起来 2 Decide on open-ended or close-ended questions 决定用开放式问题或封闭式问题 3 Draft interview questions and sequence the questions 草拟面谈问题并对其进行排序 4 Prepare an Introduction and Closure for the interview 准备一份面谈简介和结束语 5 Prepare to record responses 准备好记录问题的回答结果 6 Pre-test the instrument 对面谈中使用的工具进行预先测试

83. Interviewing 面谈 Use trained data collectors 数据收集人员要经过培训 Use as few data collectors as possible 数据收集人员的人数要尽可能地少 Establish a protocol for data collection 制定数据收集的方案 Do a “ walk through ” or trial run 进行试面谈 3-4 face-to-face interviews 3-4 次面对面面谈 10 questionnaires 10 份调查问卷 www.dadangsolihin.com

84. Conducting Interviews 实施面谈 Let the interviewees know: 使受访者知道 : why they are being interviewed 为什么他们会接受面谈 how they were selected 他们是如何被挑选出来的 how the data will be used 将如何使用数据 whether it is confidential 面谈是否是保密的 how long the interview will take 面谈的时间有多长 whether you might want to talk to them again 你是否还想与他们进行再次面谈 www.dadangsolihin.com

85. Conducting Interviews: Set Up 实施面谈：开始 Try to pick a time a place that is quiet and free of distractions 设法选择一个安静的不受干扰的面谈时间和地点 Ideally, have a second person to help take notes 理想的情况是有另外一个人帮助记录 Tape recording might be a possibility; make sure you check with interviewee and get permission before recording 可能使用录音设备，但事先要征得受访者的同意 might want to ask when setting up the appointment 可以询问什么时候可以开始约好的面谈 www.dadangsolihin.com

86. Interviewing Skills 面谈技巧 Stick to your script 依照你的稿件 if asking close-ended questions, ask exactly the way written 如果问的是封闭式问题，则严格按照已写好的问题来问 have a script for clarifications 有一个解释说明 if asking open-ended questions, go with the flow rather than directing it 如果问的是开放式问题，则自由交谈而不要给出任何指示 Be aware of cultural norms: eye contact, direct questions, gender issues 要注意文化规范：眼睛交流、直截了当的问题、性别问题 www.dadangsolihin.com

87. More Interviewing Skills 更多的面谈技巧 Balance: if you ask about what they thing are the major supports, follow with what you think are the major barriers 平衡：如果你问的是他们认为是主要支持方面的问题，接下来就要问你认为是主要障碍方面的问题 Try to avoid asking “ why ” questions, some may find this aggressive or critical 尽量避免提问 “ 为什么 ” 的问题，有些人可能会觉得这样做有些挑剔或侵犯性 Accept whatever they say with empathy and without judgment 全神贯注记下他们所说的事情并不加任何评判 www.dadangsolihin.com

88. Conducting Interviews: Added Touches 实施面谈：增加接触 Share interview questions ahead of time 事先交流面谈问题 no surprises 这样不会对问题感到惊奇 Offer to share a summary of what you understand from the interview 提议共享你从面谈中了解的情况的总结 this might be especially useful to give the interviewee (especially if a high ranking official) a greater feeling of control 这可能对于使被访问者（尤其是高级官员）有更强的控制感方面特别有用 Thank you note afterwards 面谈之后的致谢 www.dadangsolihin.com

89. Interviewing: Taking Notes 面谈：做记录 Take good notes without distracting from the conversation 做好面谈记录，在对话过程中不分心 maintain eye contact while writing 在做笔记时保持目光接触 write key words or phrases, not verbatim 记下关键字或短语，不要逐字地记录 but — if someone is saying something you want to capture, it is OK to ask them to repeat it or to finish what you are writing before asking the next question 但是，如果某个人说了一些你想要知道的事情，可以请他们重复或等你记完后再问下一个问题 may want to ask “ May I use your exact words? ” 可以问 “ 我可以使用你的原话吗？ ” www.dadangsolihin.com

90. Writing-up the Interview 整理面谈记录 Every word and idea is valuable 每个字每个想法都是有价值的 Take time to write up your notes as carefully and in-depth as possible 花时间尽量仔细和深入地整理你的笔记 It is best to do at least a brief clean-up of notes immediately afterwards (leave an hour between interviews) 面谈之后（在两个面谈之间留出 1 个小时）最好立即对面谈记录进行整理，至少有一个简短的概述 Write up full notes within a day of the interview: memory decay sets in quickly 在面谈当天要整理出一个完成的面谈记录：记忆会很快遗忘的 www.dadangsolihin.com

91. Planning and Logistics 计划和后勤保障 Need several weeks to plan 需要几星期进行计划 Focus group sessions usually 1 to 2 hours 专题组会议通常 1-2 个小时 Some up to 3-5 hours with planned break (executives, in-house management, high-level participants) 大约 3-5 个小时的有计划的中断（行政主管、内部管理人员、高级别参与者） Have clear starting and stopping times 有明确的开始和结束的时间 Consider need for food and transportation 考虑到用餐和交通的需要 www.dadangsolihin.com

92. Facilities and Materials 设施 和资料 Ideal – commercial facility designed specifically for focus groups 理想的情况 —— 专门为专题组设计的商业设施 Neutral, accessible, hotel meeting room, school or church meeting area 中立的、能够达到的、宾馆会议室、学校或教堂会议地区 Table and chairs (comfortable setting) 桌子和椅子（舒适的摆放） Name tents 桌签 Laptop computer for taking notes (if possible) 用于记录的笔记本电脑（如果可能的话） Consider need for video or audio tape recordings or have additional people take notes 考虑用录像机和录音磁带进行记录的需要或请另外的人进行记录 www.dadangsolihin.com

93. Number of Focus Group Sessions 专题组会议的数量 No fixed rule 没有固定规则 Generally, do sessions until no new issues emerge or when common themes are consistent 一般是会议开到没有出现新问题的时候或当已经取得一致的共同主题的时候结束 usually 3 to 6 group sessions 通常 3-6 个专题组会议 www.dadangsolihin.com

94. Ground Rules 程序 “ What is said here, stays here ” “ 在此所说的话就到此为止 ” Everyone is encouraged to participate but not everyone has to answer every question 鼓励每个人参加，但不要求每个人都回答所有的问题 Respect different viewpoints 尊重不同的观点 There are no right or wrong answers 答案不分正确或错误 One person speaks at a time 每次只让一个人说 www.dadangsolihin.com

95. Typical Questions 典型的问题 What did you learn at the teachers college that is most helpful to you in teaching primary school? 你在师范大学学到的什么东西对于你在教小学的时候最有帮助？ What are your greatest challenges in teaching primary school students? 当你在教小学生的时候最大的挑战是什么？ What should the college teach so that graduates are prepared to meet the current challenges? 大学里应该教授什么使得毕业生能够做好应对当前挑战的准备？ www.dadangsolihin.com

96. Facilitator/Moderator Role 主持人角色 Be familiar with the script, rather than reading it, so the session appears conversational 熟悉稿件，而不是照本宣科，这样会议才会显得像个对话 Make sure everyone is heard 确保能听到每个人的声音 Manage time, closing off discussion, moving to next topic 把握时间，结束讨论，进入下一个话题 Set ground rules 制定程序 www.dadangsolihin.com

97. Facilitator/Moderator Role (Cont.) Say as little as possible 尽量少说话 Keep personal views outside the room 排除个人观点 Use active listening 积极地倾听 Probe for elaboration (Tell me more) 询问详细的解释（告诉我更多） www.dadangsolihin.com

98. Write-up 整理 Write-up your impressions immediately after each focus group 在每个专题组会议结束之后立即写下你的印象 major issues, major points of discussion 主要的议题和讨论内容 anything unusual that happened 已经发生的任何不寻常的事情 www.dadangsolihin.com

99. Write-up (Cont.) Compare with partner 与合作伙伴进行比较 If recorded, make arrangements to have it transcribed, or to carefully watch/listen to it later 如果记录下来，安排转录，或者之后仔细地看或听这些记录 Identify themes and emerging issues 确认主题和新出现的问题 www.dadangsolihin.com

100. Data Collection Summary 数据收集小结 You can choose more than one data collection technique, multiple tools often help you meet the evaluation needs 可以选取不只一种信息收集方法，多种方法通常能让你满足评价需求 Do not let the tool drive your work 不要在工具的驱使下工作 Choose the correct tool to meet the needs of the evaluation 选择正确的工具来满足评价的需要 www.dadangsolihin.com

101. Thank you! 谢谢！

102. Beside working as Assistant Professor at Graduate School of Asia-Pacific Studies, Waseda University, Tokyo, Japan, he also active as Associate Professor at University of Darma Persada, Jakarta, Indonesia. He got various training around the globe, included Shanghai International Program for Development Evaluation Training (2008) , Public Officials Capacity Building Training Program for Government Innovation, Seoul –Korea (2007), Advanced International Training Programme of Information Technology Management, at Karlstad City, Sweden (2005); the Training Seminar on Land Use and Management, Taiwan (2004); Developing Multimedia Applications for Managers, Kuala Lumpur, Malaysia (2003); Applied Policy Development Training, Vancouver, Canada (2002); Local Government Administration Training Course, Hiroshima, Japan (2001); and Regional Development and Planning Training Course, Sapporo, Japan (1999). He published more than five books regarding local autonomous. You can reach Dadang Solihin by email at [email_address] or by his mobile at +62812 932 2202 Dadang Solihin currently is Director for Regional D evelopment Performance Evaluation at Indonesian National Development Planning Agency (Bappenas). He holds MA degree in Economics from University of Colorado, USA. His previous post is Director for System and Reporting of Development Performance Evaluation at Bappenas. Dadang Solihin’s Profile www.dadangsolihin.com