Title: Emerging directions and challenges in survey methods
Presented by: Jolene Smyth, Associate Professor, Department of Sociology,
University of Nebraska-Lincoln
Brown Bag co-sponsored by the Evaluation Institute and Washington Evaluators
Survey research is difficult in Afghanistan. Violence, illiteracy in both urban and rural areas, cultural constraints, and access to family and women in particular have all been faced by D3 Systems in the process of building a self-sustaining national survey operation in Afghanistan. Grown from an organization capable of simple urban polls of Kabul in 2003 to multistage, nationally representative random survey samples today, D3’s partially-owned subsidiary called the Afghan Center for Socio-Economic Research is a vibrant, busy company conducting research every day throughout Afghanistan. This paper focuses on the various challenges faced by ACSOR operating in Afghanistan. Findings from the 2006 and 2007 nationwide probability samples completed by ACSOR for the Asia Foundation’s Annual Reports on Afghanistan and D3’s research on women’s issues will be included. Particular emphasis will be placed on issues of education, armed violence, lack of familiarity with research, cultural restrictions on women, ethno-linguistic fragmentation, and outdated population data. General results of the D3 Women in Muslim Countries and Asia Foundation surveys are discussed with emphasis on trends across time related to international development issues as they relate to survey research. Among these are human security as Afghans perceive it, the status of women in Afghan society, and education and awareness of democratic practices like public opinion polling among Afghans nationwide. Trends are demonstrated empirically with the Asia Foundation tracking data and supplemented with findings from recent reporting by D3 and the Center for Strategic and International Studies.
Surveys are a common method for capturing public opinion through formatted questions administered via phone, mail, email or web. They can provide fast, low-cost results but have limitations in depth of understanding. Different types of surveys include cross-sectional, longitudinal, and panel surveys. Question format includes open-ended, dichotomous, multiple choice, ranking and Likert scales. Proper wording, pre-testing and improving response rates are important to validity. With appropriate sampling, survey results can be generalized to a wider population.
The 2017 Census Test has three main objectives:
1. To test the effect of including a question on sexual identity and different collection methods.
2. To evaluate methods for maximizing online response before field staff follow-up to reduce costs.
3. To test options for the field operation design to optimize overall response rates while minimizing variability between areas.
The test will involve two components - one with and one without field staff follow-up. This will help assess how online systems perform and evaluate follow-up strategies. Areas have been selected to represent a range of response likelihoods. The results will inform decisions around question content and collection methodology for the 2021 Census.
1) The study aims to investigate whether people who owned dogs as children have more empathy towards others as adults compared to those who did not own dogs as children.
2) Previous research on the relationship between pet ownership and empathy in children had mixed findings, with one study finding higher empathy in children with pets, and another finding differences only based on type of pet owned.
3) The proposed study would administer questionnaires to 1000 adults in Southwestern Ontario and conduct follow-up interviews to further examine the relationship between childhood dog ownership and adult empathy.
Creating a substance abuse communications strategy on a tight budgetErin Norvell
Substance abuse is a complex public health issue that leaves many of today’s community organizations struggling and overwhelmed. With more Americans dying each year from drug overdoses than motor vehicle crashes, there’s a huge need for policy and communication teams to affect change at the local level. However, too often tight budgets lead to generalized messaging and outreach tactics. While initially appealing, the approach of reaching as many people as possible often lacks the message tailoring strategies needed to elicit behavior change. Developing an effective communication strategy can be challenging for anyone, especially community organizations working with limited planning and implementation budgets.
This presentation is an excerpt of the full training and provides an overview of the key steps in developing a substance abuse communication strategy.
For more from Digital Edge Communications, visit our website: www.digitaledgecommunications.us
This document discusses various data collection methods used in research. It explains the difference between primary and secondary data, and lists common sources for each. Methods of collecting primary data include surveys, interviews, observations, and focus groups. It then provides details on personal interviews, telephone interviews, questionnaires, and maximizing response rates. Limitations of using secondary data and survey research are also covered. The document provides a comprehensive overview of quantitative and qualitative data collection techniques.
The Social Side of Behavioural EconomicsDavid Perrott
Understanding how deeply hardwired our brains are to be social gives us a better understand of how we make judgments and decisions, creating the right foundation for new forms of communication and design.
Longitudinal Study on the Generational Impacts of Habitat for Humanity: A Res...Karthikeyan Umapathy
This proposal outlines a longitudinal study to assess the generational impacts of affordable housing provided by HabiJax, a Habitat for Humanity affiliate in Jacksonville, FL, over the past 30 years. Researchers will conduct interviews every 5 years with a sample of 100 former HabiJax homeowners and their families to understand changes in areas like education, employment, wealth, health, and civic engagement. An initial pilot study involved interviews with 8 subjects but had challenges with outdated contact information. The proposal seeks feedback on increasing participation, such as offering a higher incentive or interviewing adult children.
Survey research is difficult in Afghanistan. Violence, illiteracy in both urban and rural areas, cultural constraints, and access to family and women in particular have all been faced by D3 Systems in the process of building a self-sustaining national survey operation in Afghanistan. Grown from an organization capable of simple urban polls of Kabul in 2003 to multistage, nationally representative random survey samples today, D3’s partially-owned subsidiary called the Afghan Center for Socio-Economic Research is a vibrant, busy company conducting research every day throughout Afghanistan. This paper focuses on the various challenges faced by ACSOR operating in Afghanistan. Findings from the 2006 and 2007 nationwide probability samples completed by ACSOR for the Asia Foundation’s Annual Reports on Afghanistan and D3’s research on women’s issues will be included. Particular emphasis will be placed on issues of education, armed violence, lack of familiarity with research, cultural restrictions on women, ethno-linguistic fragmentation, and outdated population data. General results of the D3 Women in Muslim Countries and Asia Foundation surveys are discussed with emphasis on trends across time related to international development issues as they relate to survey research. Among these are human security as Afghans perceive it, the status of women in Afghan society, and education and awareness of democratic practices like public opinion polling among Afghans nationwide. Trends are demonstrated empirically with the Asia Foundation tracking data and supplemented with findings from recent reporting by D3 and the Center for Strategic and International Studies.
Surveys are a common method for capturing public opinion through formatted questions administered via phone, mail, email or web. They can provide fast, low-cost results but have limitations in depth of understanding. Different types of surveys include cross-sectional, longitudinal, and panel surveys. Question format includes open-ended, dichotomous, multiple choice, ranking and Likert scales. Proper wording, pre-testing and improving response rates are important to validity. With appropriate sampling, survey results can be generalized to a wider population.
The 2017 Census Test has three main objectives:
1. To test the effect of including a question on sexual identity and different collection methods.
2. To evaluate methods for maximizing online response before field staff follow-up to reduce costs.
3. To test options for the field operation design to optimize overall response rates while minimizing variability between areas.
The test will involve two components - one with and one without field staff follow-up. This will help assess how online systems perform and evaluate follow-up strategies. Areas have been selected to represent a range of response likelihoods. The results will inform decisions around question content and collection methodology for the 2021 Census.
1) The study aims to investigate whether people who owned dogs as children have more empathy towards others as adults compared to those who did not own dogs as children.
2) Previous research on the relationship between pet ownership and empathy in children had mixed findings, with one study finding higher empathy in children with pets, and another finding differences only based on type of pet owned.
3) The proposed study would administer questionnaires to 1000 adults in Southwestern Ontario and conduct follow-up interviews to further examine the relationship between childhood dog ownership and adult empathy.
Creating a substance abuse communications strategy on a tight budgetErin Norvell
Substance abuse is a complex public health issue that leaves many of today’s community organizations struggling and overwhelmed. With more Americans dying each year from drug overdoses than motor vehicle crashes, there’s a huge need for policy and communication teams to affect change at the local level. However, too often tight budgets lead to generalized messaging and outreach tactics. While initially appealing, the approach of reaching as many people as possible often lacks the message tailoring strategies needed to elicit behavior change. Developing an effective communication strategy can be challenging for anyone, especially community organizations working with limited planning and implementation budgets.
This presentation is an excerpt of the full training and provides an overview of the key steps in developing a substance abuse communication strategy.
For more from Digital Edge Communications, visit our website: www.digitaledgecommunications.us
This document discusses various data collection methods used in research. It explains the difference between primary and secondary data, and lists common sources for each. Methods of collecting primary data include surveys, interviews, observations, and focus groups. It then provides details on personal interviews, telephone interviews, questionnaires, and maximizing response rates. Limitations of using secondary data and survey research are also covered. The document provides a comprehensive overview of quantitative and qualitative data collection techniques.
The Social Side of Behavioural EconomicsDavid Perrott
Understanding how deeply hardwired our brains are to be social gives us a better understand of how we make judgments and decisions, creating the right foundation for new forms of communication and design.
Longitudinal Study on the Generational Impacts of Habitat for Humanity: A Res...Karthikeyan Umapathy
This proposal outlines a longitudinal study to assess the generational impacts of affordable housing provided by HabiJax, a Habitat for Humanity affiliate in Jacksonville, FL, over the past 30 years. Researchers will conduct interviews every 5 years with a sample of 100 former HabiJax homeowners and their families to understand changes in areas like education, employment, wealth, health, and civic engagement. An initial pilot study involved interviews with 8 subjects but had challenges with outdated contact information. The proposal seeks feedback on increasing participation, such as offering a higher incentive or interviewing adult children.
Analytics Academy 2015 Presentation SlidesHarvardComms
This document discusses how analytics can inform digital strategies and operations at NPR. It begins by outlining some key goals of NPR's digital and social media presence, such as informing the public and tracking changing audience behaviors. It then discusses how NPR analyzes various metrics like website traffic, mobile usage, social media engagement, and email effectiveness to understand what content and platforms resonate best with audiences. Insights from analytics are used to optimize strategies, like prioritizing mobile-friendly content or creating local Facebook stories that drive higher engagement. The document emphasizes using data to continuously learn and improve offerings, rather than just focusing on short-term optimizations. Context is also important to make data meaningful.
This document provides information about research methods for a research methods course. It discusses elements of a research proposal such as the title, background, objectives, and methodology. For methodology, it outlines the research design, study population, sampling, and research framework including theoretical framework, conceptual framework, definition of variables, and data collection and analysis plans. It also discusses people-oriented research, surveys, research frameworks, and outlines for the discussion section.
The document evaluates the quality of life of residents living in the Bob and Judy Charles SmartHome run by Imagine!. Data was collected through phone interviews and the Supports Intensity Scale before and one year after moving in. Results found that most quality of life indicators like safety, choices, and relationships increased while support needs decreased. There were also some positive correlations between subjective and objective quality of life reports. In conclusion, living in the SmartHome enhanced residents' quality of life by providing greater independence, access, and interaction with their environment.
Steps to Writing a Grant Proposal Essay
Summary: A Modest Proposal Letter
Project Proposal: Effects of Climate Change
School Improvement Essay
A Research Proposal On Big Data
Sample Grant Proposal Essay
Writing A Grant Proposal Essay
How To Write A Community Teaching Proposal
Domestic Violence research proposal
How To Write A Bullying Proposal
Project Proposal Essay
Research Proposal
Grant Proposal Evaluation
A Sample Group Counseling Project Proposal
Sample Csr Proposal
CSV NNVIA Measuring Impact of Volunteering event - Iona Joy - 27.03.15CSV_UK
Iona Joy, Head of charity team at NPC, presented on NPC's approach to measurement at the CSV conference in March 2015. NPC works at the nexus between charities and funders to increase the impact of both. NPC uses a four pillar approach to measuring impact that involves mapping a theory of change, prioritizing outcomes to measure, choosing an appropriate level of evidence, and selecting data sources and tools. An effective measurement framework is developed through this process to help charities strategically plan, deliver, assess, and review their impact.
Slides used by Deb James and Tim Vanson (NCVO), at the Big Local ‘Reviewing Big Local plans 1' learning event, organised as part of the Local Trust programme of networking and learning events for Big Local residents. The event took place on Thursday 22 January 2014.
Poynter Lesson 13 - More Quantitative Market ResearchRay Poynter
This document provides an overview and agenda for a marketing research lesson focusing on quantitative research methods. The key topics covered include:
- Reviewing a previous quiz and answering questions from last week's lesson.
- Discussing quantitative data collection methods like surveys, samples, questionnaires, and data analysis.
- Explaining important quantitative research concepts such as representative samples, bias, correlation, and driver analysis.
- Outlining the survey and questionnaire design process.
- Providing examples of quantitative data sources and collection modes.
The lesson aims to help students understand quantitative marketing research techniques and applying them to solve business problems.
Measuring Mission Value of Digital Communications in the Public Sector and 9 ...Scott Burns
This is the complete version of February 3, 2011 presentation by Scott Burns (CEO, GovDelivery) and Steve Ressler (Founder/President, GovLoop) on driving mission value from digital communications int he public sector and the 9 rules of engagement.
Measuring and Capturing Value of Government CommunicationGovLoop
Measuring and Capturing Value of Government Communication describes a methodology of government communications to go from clicks to engagement to real mission results
The document provides information about collecting data, including defining key terms, describing different data collection methods, and explaining different sampling techniques. It discusses primary and secondary data sources, methods of collecting data like surveys, questionnaires, and interviews. It also compares census and sample surveys, defines random and non-random sampling, and explains sampling errors and non-sampling errors. The key agencies that collect data in India are also outlined.
A Journalist’s Guide to Survey Research and Election Polls by Cliff ZuskinFincher Consulting
This document provides an overview of survey research and election polling for journalists. It discusses key things journalists should know about polls, including who conducted the poll, when it was done, who was sampled, and how questions were worded and ordered. The document outlines important factors in determining the validity of a poll, such as whether it used a probability sample. It also discusses trends in survey methods, like the increasing use of cell phones and challenges in obtaining a representative sample. Journalists are advised to consider these methodological factors when evaluating and reporting on polls.
A solutions-based approach, illustrated by case studies, which show how inferences can be improved from surveys administered to biased, low response rate and non-probability samples.
It addresses how to improve the accuracy of the survey estimates we generate from poorer quality and non-probability samples.
An online survey allows respondents to complete a questionnaire over the Internet. It is created as a web form that stores answers in a database and provides analytics. There are several modes for administering surveys, including online, mobile, telephone, mail, and face-to-face. Each mode has advantages and disadvantages related to costs, coverage of populations, response flexibility and accuracy. Key methodological issues for online surveys include sampling bias, low response rates, and ensuring data quality through design and post-survey adjustments.
This document discusses various topics related to survey research methods. It covers forms of survey research like face-to-face, mail, telephone and internet surveys. It also discusses interviewing, validity, reliability, sampling, data collection and analysis procedures. Common survey design errors and ways to improve response rates are outlined. The advantages and disadvantages of different data collection modes are compared.
This document provides an overview of qualitative interviewing techniques. It discusses when qualitative interviewing is appropriate to use as a data collection tool. It describes the advantages and disadvantages of semi-structured versus unstructured interviews. It also discusses focus groups and how to properly approach, interact with, and analyze qualitative interview data to enhance the quality of information gathered. The overall purpose is to understand qualitative interviewing as a method to study complex human perspectives and processes.
Module 3 - Improving Current Business with External Data- Online caniceconsulting
The document discusses how to use external data to improve business. It defines external data as data generated outside an organization that can come from a variety of sources and serve nearly every industry. The document outlines different types of external data like primary data, secondary data, and open data. It provides examples of sources for primary data, which is original and reliable, and secondary data, which already exists. The benefits of using external data to supplement internal data and gain a more comprehensive view are also discussed.
Public Relations Practice 2014: Week 4Kane Hopkins
The document discusses the key steps in effective public relations planning: research, planning, communication, and measurement. It emphasizes that research should inform all aspects of planning and should account for 10% of the budget. Research involves understanding issues, publics, attitudes, and information needs. Planning establishes goals, objectives, strategies, tactics, timelines and budgets. The document outlines Freitag's four questions for creating a PR plan: analyzing the current situation, defining goals and objectives, developing strategies and tactics, and planning evaluation. Objectives should be SMART. Overall, the document provides an overview of best practices for research-driven PR planning and evaluation.
The document discusses survey design and data collection. It covers several key topics in 3 sentences or less:
1. What should be measured including characteristics, channels, outcomes and assumptions based on a theory of change. Accurate and precise indicators are important.
2. Methods of data collection such as surveys, qualitative methods, and tests. Good measures are accurate without bias and precise without random error.
3. Challenges in measurement including things people don't know well or want to talk about, abstract concepts, things not directly observable, and things best directly observed through protocols. Data collection requires reliability, validity, integrity, accuracy and timeliness.
- Online surveys have both advantages and disadvantages. They are low-cost and convenient to administer remotely, but respondents may not provide accurate or honest answers due to lack of accountability or memory. Data errors can also occur.
- The advantages of online surveys are that they are relatively easy and low-cost to administer, can reach a large number of respondents, and allow for extensive data collection and analysis. Standardized online surveys can reduce biases.
- The document discusses designing effective surveys, including using validated instruments, considering aesthetics and question order, and collecting data through various methods like interviews, telephone, or electronic surveys. The goal is to maximize response rates and obtain meaningful results.
The Washington Eval membership survey found:
- Most members joined to learn about evaluation theories/practice and make connections.
- Monthly brown bags, deep dives, and social events are most popular. Preferred times are on-demand, 12-2pm, and 5:30-6pm.
- The weekly digest is most useful for sharing events, jobs, and opportunities.
- Most support increasing dues to $30, offering a two-year option, and auto-renew with opt-in.
- Members are generally satisfied with WE's diversity efforts but want more training and DEI incorporation.
- Many members expressed interest in pro bono, mentoring, and volunteer opportunities.
Are you interested in supporting emerging evaluators and developing the evaluation profession in the Washington, DC area? Are you an emerging evaluator interested in improving your skills and understanding or moving into a different field? This presentation will provide information on ways that Washington Evaluators members can engage in Mentor Minutes.
Mentor Minutes is an initiative that aims to connect current WE members to experienced evaluation professionals in the WE community through short-term mentorship opportunities. The purpose of Mentor Minutes is to pair experienced evaluators (mentors) with aspiring, emerging, or seasoned evaluators (mentees) and establish mutually beneficial professional connections.
More Related Content
Similar to Emerging directions and challenges in survey methods
Analytics Academy 2015 Presentation SlidesHarvardComms
This document discusses how analytics can inform digital strategies and operations at NPR. It begins by outlining some key goals of NPR's digital and social media presence, such as informing the public and tracking changing audience behaviors. It then discusses how NPR analyzes various metrics like website traffic, mobile usage, social media engagement, and email effectiveness to understand what content and platforms resonate best with audiences. Insights from analytics are used to optimize strategies, like prioritizing mobile-friendly content or creating local Facebook stories that drive higher engagement. The document emphasizes using data to continuously learn and improve offerings, rather than just focusing on short-term optimizations. Context is also important to make data meaningful.
This document provides information about research methods for a research methods course. It discusses elements of a research proposal such as the title, background, objectives, and methodology. For methodology, it outlines the research design, study population, sampling, and research framework including theoretical framework, conceptual framework, definition of variables, and data collection and analysis plans. It also discusses people-oriented research, surveys, research frameworks, and outlines for the discussion section.
The document evaluates the quality of life of residents living in the Bob and Judy Charles SmartHome run by Imagine!. Data was collected through phone interviews and the Supports Intensity Scale before and one year after moving in. Results found that most quality of life indicators like safety, choices, and relationships increased while support needs decreased. There were also some positive correlations between subjective and objective quality of life reports. In conclusion, living in the SmartHome enhanced residents' quality of life by providing greater independence, access, and interaction with their environment.
Steps to Writing a Grant Proposal Essay
Summary: A Modest Proposal Letter
Project Proposal: Effects of Climate Change
School Improvement Essay
A Research Proposal On Big Data
Sample Grant Proposal Essay
Writing A Grant Proposal Essay
How To Write A Community Teaching Proposal
Domestic Violence research proposal
How To Write A Bullying Proposal
Project Proposal Essay
Research Proposal
Grant Proposal Evaluation
A Sample Group Counseling Project Proposal
Sample Csr Proposal
CSV NNVIA Measuring Impact of Volunteering event - Iona Joy - 27.03.15CSV_UK
Iona Joy, Head of charity team at NPC, presented on NPC's approach to measurement at the CSV conference in March 2015. NPC works at the nexus between charities and funders to increase the impact of both. NPC uses a four pillar approach to measuring impact that involves mapping a theory of change, prioritizing outcomes to measure, choosing an appropriate level of evidence, and selecting data sources and tools. An effective measurement framework is developed through this process to help charities strategically plan, deliver, assess, and review their impact.
Slides used by Deb James and Tim Vanson (NCVO), at the Big Local ‘Reviewing Big Local plans 1' learning event, organised as part of the Local Trust programme of networking and learning events for Big Local residents. The event took place on Thursday 22 January 2014.
Poynter Lesson 13 - More Quantitative Market ResearchRay Poynter
This document provides an overview and agenda for a marketing research lesson focusing on quantitative research methods. The key topics covered include:
- Reviewing a previous quiz and answering questions from last week's lesson.
- Discussing quantitative data collection methods like surveys, samples, questionnaires, and data analysis.
- Explaining important quantitative research concepts such as representative samples, bias, correlation, and driver analysis.
- Outlining the survey and questionnaire design process.
- Providing examples of quantitative data sources and collection modes.
The lesson aims to help students understand quantitative marketing research techniques and applying them to solve business problems.
Measuring Mission Value of Digital Communications in the Public Sector and 9 ...Scott Burns
This is the complete version of February 3, 2011 presentation by Scott Burns (CEO, GovDelivery) and Steve Ressler (Founder/President, GovLoop) on driving mission value from digital communications int he public sector and the 9 rules of engagement.
Measuring and Capturing Value of Government CommunicationGovLoop
Measuring and Capturing Value of Government Communication describes a methodology of government communications to go from clicks to engagement to real mission results
The document provides information about collecting data, including defining key terms, describing different data collection methods, and explaining different sampling techniques. It discusses primary and secondary data sources, methods of collecting data like surveys, questionnaires, and interviews. It also compares census and sample surveys, defines random and non-random sampling, and explains sampling errors and non-sampling errors. The key agencies that collect data in India are also outlined.
A Journalist’s Guide to Survey Research and Election Polls by Cliff ZuskinFincher Consulting
This document provides an overview of survey research and election polling for journalists. It discusses key things journalists should know about polls, including who conducted the poll, when it was done, who was sampled, and how questions were worded and ordered. The document outlines important factors in determining the validity of a poll, such as whether it used a probability sample. It also discusses trends in survey methods, like the increasing use of cell phones and challenges in obtaining a representative sample. Journalists are advised to consider these methodological factors when evaluating and reporting on polls.
A solutions-based approach, illustrated by case studies, which show how inferences can be improved from surveys administered to biased, low response rate and non-probability samples.
It addresses how to improve the accuracy of the survey estimates we generate from poorer quality and non-probability samples.
An online survey allows respondents to complete a questionnaire over the Internet. It is created as a web form that stores answers in a database and provides analytics. There are several modes for administering surveys, including online, mobile, telephone, mail, and face-to-face. Each mode has advantages and disadvantages related to costs, coverage of populations, response flexibility and accuracy. Key methodological issues for online surveys include sampling bias, low response rates, and ensuring data quality through design and post-survey adjustments.
This document discusses various topics related to survey research methods. It covers forms of survey research like face-to-face, mail, telephone and internet surveys. It also discusses interviewing, validity, reliability, sampling, data collection and analysis procedures. Common survey design errors and ways to improve response rates are outlined. The advantages and disadvantages of different data collection modes are compared.
This document provides an overview of qualitative interviewing techniques. It discusses when qualitative interviewing is appropriate to use as a data collection tool. It describes the advantages and disadvantages of semi-structured versus unstructured interviews. It also discusses focus groups and how to properly approach, interact with, and analyze qualitative interview data to enhance the quality of information gathered. The overall purpose is to understand qualitative interviewing as a method to study complex human perspectives and processes.
Module 3 - Improving Current Business with External Data- Online caniceconsulting
The document discusses how to use external data to improve business. It defines external data as data generated outside an organization that can come from a variety of sources and serve nearly every industry. The document outlines different types of external data like primary data, secondary data, and open data. It provides examples of sources for primary data, which is original and reliable, and secondary data, which already exists. The benefits of using external data to supplement internal data and gain a more comprehensive view are also discussed.
Public Relations Practice 2014: Week 4Kane Hopkins
The document discusses the key steps in effective public relations planning: research, planning, communication, and measurement. It emphasizes that research should inform all aspects of planning and should account for 10% of the budget. Research involves understanding issues, publics, attitudes, and information needs. Planning establishes goals, objectives, strategies, tactics, timelines and budgets. The document outlines Freitag's four questions for creating a PR plan: analyzing the current situation, defining goals and objectives, developing strategies and tactics, and planning evaluation. Objectives should be SMART. Overall, the document provides an overview of best practices for research-driven PR planning and evaluation.
The document discusses survey design and data collection. It covers several key topics in 3 sentences or less:
1. What should be measured including characteristics, channels, outcomes and assumptions based on a theory of change. Accurate and precise indicators are important.
2. Methods of data collection such as surveys, qualitative methods, and tests. Good measures are accurate without bias and precise without random error.
3. Challenges in measurement including things people don't know well or want to talk about, abstract concepts, things not directly observable, and things best directly observed through protocols. Data collection requires reliability, validity, integrity, accuracy and timeliness.
- Online surveys have both advantages and disadvantages. They are low-cost and convenient to administer remotely, but respondents may not provide accurate or honest answers due to lack of accountability or memory. Data errors can also occur.
- The advantages of online surveys are that they are relatively easy and low-cost to administer, can reach a large number of respondents, and allow for extensive data collection and analysis. Standardized online surveys can reduce biases.
- The document discusses designing effective surveys, including using validated instruments, considering aesthetics and question order, and collecting data through various methods like interviews, telephone, or electronic surveys. The goal is to maximize response rates and obtain meaningful results.
Similar to Emerging directions and challenges in survey methods (20)
The Washington Eval membership survey found:
- Most members joined to learn about evaluation theories/practice and make connections.
- Monthly brown bags, deep dives, and social events are most popular. Preferred times are on-demand, 12-2pm, and 5:30-6pm.
- The weekly digest is most useful for sharing events, jobs, and opportunities.
- Most support increasing dues to $30, offering a two-year option, and auto-renew with opt-in.
- Members are generally satisfied with WE's diversity efforts but want more training and DEI incorporation.
- Many members expressed interest in pro bono, mentoring, and volunteer opportunities.
Are you interested in supporting emerging evaluators and developing the evaluation profession in the Washington, DC area? Are you an emerging evaluator interested in improving your skills and understanding or moving into a different field? This presentation will provide information on ways that Washington Evaluators members can engage in Mentor Minutes.
Mentor Minutes is an initiative that aims to connect current WE members to experienced evaluation professionals in the WE community through short-term mentorship opportunities. The purpose of Mentor Minutes is to pair experienced evaluators (mentors) with aspiring, emerging, or seasoned evaluators (mentees) and establish mutually beneficial professional connections.
George Julnes: Humility in Valuing in the Public Interest - Multiple Methods ...Washington Evaluators
Roundtable: Contributions of Cost-Effectiveness Studies to Evidence-Based Policymaking
Washington Evaluators and the Bipartisan Policy Center's Evidence-Based Policymaking Initiative are pleased to co-sponsor a roundtable discussion about the contributions of cost-effectiveness studies to informing policy decisions in government. This panel discussion will explore approaches to conducting cost-effectiveness studies, their value and use in government decisions, and practical steps for improving their utility for decision-makers. The distinguished panelists have collectively experienced the generation and use of cost-effectiveness studies from a variety of academic, non-governmental, and governmental positions. We invite you to join us on Tuesday, December 5th at 2 PM for a lively discussion of the implications of cost-effectiveness research on government decision making.
Harry Hatry: Cost-Effectiveness Basics for Evidence-Based PolicymakingWashington Evaluators
Washington Evaluators and the Bipartisan Policy Center's Evidence-Based Policymaking Initiative are pleased to co-sponsor a roundtable discussion about the contributions of cost-effectiveness studies to informing policy decisions in government. This panel discussion will explore approaches to conducting cost-effectiveness studies, their value and use in government decisions, and practical steps for improving their utility for decision-makers. The distinguished panelists have collectively experienced the generation and use of cost-effectiveness studies from a variety of academic, non-governmental, and governmental positions. We invite you to join us on Tuesday, December 5th at 2 PM for a lively discussion of the implications of cost-effectiveness research on government decision making.
The panel discussion will be introduced and chaired by Nick Hart, Director of BPC's Evidence-Based Policymaking Initiative and the 2017 Washington Evaluators President.
Panelists:
Harry Hatry, Distinguished Fellow and Director of the Urban Institute's Public Management Program
George Julnes, Professor in the University of Baltimore's School of Public and International Affairs
Sandy Davis, Senior Advisor to BPC's Evidence-Based Policymaking Initiative
George Julnes: Humility in Valuing in the Public Interest - Multiple Methods ...Washington Evaluators
Washington Evaluators and the Bipartisan Policy Center's Evidence-Based Policymaking Initiative are pleased to co-sponsor a roundtable discussion about the contributions of cost-effectiveness studies to informing policy decisions in government. This panel discussion will explore approaches to conducting cost-effectiveness studies, their value and use in government decisions, and practical steps for improving their utility for decision-makers. The distinguished panelists have collectively experienced the generation and use of cost-effectiveness studies from a variety of academic, non-governmental, and governmental positions. We invite you to join us on Tuesday, December 5th at 2 PM for a lively discussion of the implications of cost-effectiveness research on government decision making.
The panel discussion will be introduced and chaired by Nick Hart, Director of BPC's Evidence-Based Policymaking Initiative and the 2017 Washington Evaluators President.
Panelists:
Harry Hatry, Distinguished Fellow and Director of the Urban Institute's Public Management Program
George Julnes, Professor in the University of Baltimore's School of Public and International Affairs
Sandy Davis, Senior Advisor to BPC's Evidence-Based Policymaking Initiative
The DC Consortium Student Conference on Evaluation and Policy (SCEP) is a collaboration of universities in the District of Columbia, Northern Virginia and Maryland regions, representing the interests of students aspiring to be evaluators and policy makers. This collaboration aims to provide students with a platform to present their research and engage with evaluation experts in the opportunity-rich region of Washington, D.C., thereby serving as a bridge between students, academia and other evaluation and policy agencies/organizations. In this presentation, students from the Organizing Committee discuss lessons learned from DC SCEP’s inaugural conference. Features of the conference include a keynote address, interdisciplinary panel, and about 30 student presentations. We will highlight lessons learned concerning how the conference served to broker knowledge towards its theme, ‘Advancing Social Justice in Evaluation and Policy Integration’ with Consortium graduate students in the region.
The document summarizes findings from three recent GAO reports on the use of evidence in federal decision making. It discusses the results of GAO's 2017 survey of federal managers which found no significant increase in the use of performance measures or information in decision making. It also summarizes the GAO's assessment of GPRAMA implementation and key findings about quarterly performance reviews and program evaluation from the manager survey. The document concludes with a recommendation that OMB direct each agency to prepare an annual evaluation agenda.
As evaluators, policy makers, and program managers, we want our efforts to solve the problems of the world to be based on the best possible knowledge. Too often, however, that knowledge is not organized in a way that makes it easy to use for decision-making and action. As a result, too many programs fail to meet their potential.
“Causal knowledge mapping” is a technique for integrating and measurably improving knowledge from a broad range of sources. In this webinar, we’ll use real-world examples and interactive conversations to show three kinds of causal knowledge maps that can benefit an evaluation: (1) Collaborative maps to design programs that fit the local situation; (2) Literature maps to identify and improve upon effective practices; (3) Evaluation findings maps for continual improvement.
Partnerships for Transformative Change in Challenging Political Contexts w/ D...Washington Evaluators
The document summarizes a 4-day course on transformative evaluation held in Santiago, Chile in September 2016. The course was attended by 35 evaluators from several South American countries and focused on how evaluators can contribute to social justice and human rights through their work. It covered the transformative paradigm and questions about incorporating social change into evaluation design. Participants discussed solutions like empowering marginalized communities and forming diverse evaluation teams. The course organizers were flexible in bringing transformative evaluation concepts to different universities and organizations in Chile.
Founded in 1984 with an initial membership of 12 evaluators, the Washington Evaluators (WE) has since grown to include a professional and student membership base of more than 200 in the nation's capitol. This presentation describes WE's experience in developing and maintaining a community of evaluation practitioners that include a diverse mix of government, private, and self-employed evaluators as well as prominent evaluators in academia. This presentation discusses the strategies WE uses to foster personal connections and sharing information about the evaluation profession for both new and long-time evaluators.
Transitioning from School to Work: Preparing Evaluation Students and New Eval...Washington Evaluators
Unlike some professions, there is no single path for making the leap from student to new professional to being an established member of the profession. In large part this is because of the trans-disciplinary nature of evaluation field and the many the broad number of professions and sectors (public, non-profit, private) in which evaluation and social science research skills may be useful. This panel will explore the many approaches used by universities in the Washington, DC area to train graduate and undergraduate students in the field of evaluation, and the transition strategies to help students and new evaluators establish themselves in the evaluation field. The seven distinguished panelists are all associated with Washington Evaluators, and have served in AEA and/or WE leadership positions. Panelists and our Discussant will be asked to address questions such as:
1. In which disciplines/schools at your university would we expect to find courses in evaluation or related to evaluation?
2. What are the components of the evaluation curricula? Do you offer a degree or major field in evaluation?
3. Do you offer hands-on experiences for your students to design and conduct evaluations?
4. Where have your former students worked in the evaluation field, and what kinds of careers have they had?
5. What advice do you have for new evaluators regarding making the shift from school to work in the evaluation field? What types of professional and networking activities would you recommend to further careers in evaluation?
Challenges and Solutions to Conducting High Quality Contract Evaluations for ...Washington Evaluators
Challenges and Solutions to Conducting High Quality Contract Evaluations for the U.S. Government
Washington Evaluators Brown Bag
July 7, 2015
Presenter: David J. Bernstein
Discussant: Kathryn E. Newcomer
Lessons from World Bank Support for Evidence-Based Policy Making, Presented by Nils Junge on Wednesday, June 17, 2015 from 12 - 1:30 pm in the George Washington University Marvin Center (Room 308).
Since the late 1990s the World Bank has placed greater and greater emphasis on evidence-based policy making, with a specific focus on how the poor and vulnerable are affected. A commonly used approach is ‘Poverty and Social Impact Analysis’ (PSIA), typically undertaken before development projects are approved. PSIAs are implemented with the express purpose of informing public sector reforms in order to mitigate negative distributional impacts. To identify winners and losers of a given policy reform, PSIAs may use or combine various kinds of analysis: statistical, econometric cost-benefit, social, stakeholder, political economy, etc. Strongly utilization-focused, the evaluation process is often as important as the analytical work itself. After introducing PSIA methods, the presenter will share practical lessons from 12 years conducting PSIAs and some of the challenges inherent in this exciting area of evaluation.
Nils Junge works internationally as an independent evaluator and policy advisor. In addition to advising the World Bank and government counterparts on addressing reform impacts, he has conducted evaluations for over 20 clients in Africa, Eastern Europe and the Middle East/North Africa. Multi-lingual, he has worked in 5 languages. He has an MA from Johns Hopkins – School of Advanced International Studies (SAIS).
This document outlines the current state of monitoring and evaluation (M&E) in Tajikistan. It discusses the country's background, M&E system and players, possibilities and limitations. It also describes Tajikistan's National M&E Network, which was established in 2008 and includes over 100 members. The Network aims to share information, expand partnerships, and build M&E capacity in Tajikistan through activities like attending international conferences and developing local language resources. Overall, the document provides an overview of M&E practice in Tajikistan and the goals of the National M&E Network to further develop the field.
The Kyrgyz Republic established a national monitoring and evaluation (M&E) system beginning in the 2000s. As strategic planning increased the need for M&E and non-governmental organization involvement, the government began including M&E sections in programs and strategies from 2000 onward. A National M&E Network was formed in 2007 by NGOs and individuals to support M&E system development. While M&E practices were adopted, implementation has faced challenges of disconnected data collection across agencies and a lack of public input. The Network works to strengthen professional evaluation through training, publications, and events to help address these challenges and further establish M&E in governance.
Washington Evaluators (WE) is a local affiliate of the American Evaluation Association (AEA). WE was founded over 30 years ago as a professional society devoted to fostering state-of-the-art knowledge and information sharing.
Ann K. Emery gave a brown bag presentation on visualizing evaluation results to the Washington Evaluators on September 15, 2014 at George Washington University. The presentation highlighted tips for creating effective data visualizations including using intentional color schemes, ensuring visuals are accessible on websites and social media, and using checklists to guide design. Emery emphasized the importance of visualizing both qualitative and quantitative evaluation findings to tell compelling stories with data.
This presentation by Juraj Čorba, Chair of OECD Working Party on Artificial Intelligence Governance (AIGO), was made during the discussion “Artificial Intelligence, Data and Competition” held at the 143rd meeting of the OECD Competition Committee on 12 June 2024. More papers and presentations on the topic can be found at oe.cd/aicomp.
This presentation was uploaded with the author’s consent.
This presentation by OECD, OECD Secretariat, was made during the discussion “Pro-competitive Industrial Policy” held at the 143rd meeting of the OECD Competition Committee on 12 June 2024. More papers and presentations on the topic can be found at oe.cd/pcip.
This presentation was uploaded with the author’s consent.
Why Psychological Safety Matters for Software Teams - ACE 2024 - Ben Linders.pdfBen Linders
Psychological safety in teams is important; team members must feel safe and able to communicate and collaborate effectively to deliver value. It’s also necessary to build long-lasting teams since things will happen and relationships will be strained.
But, how safe is a team? How can we determine if there are any factors that make the team unsafe or have an impact on the team’s culture?
In this mini-workshop, we’ll play games for psychological safety and team culture utilizing a deck of coaching cards, The Psychological Safety Cards. We will learn how to use gamification to gain a better understanding of what’s going on in teams. Individuals share what they have learned from working in teams, what has impacted the team’s safety and culture, and what has led to positive change.
Different game formats will be played in groups in parallel. Examples are an ice-breaker to get people talking about psychological safety, a constellation where people take positions about aspects of psychological safety in their team or organization, and collaborative card games where people work together to create an environment that fosters psychological safety.
This presentation by Katharine Kemp, Associate Professor at the Faculty of Law & Justice at UNSW Sydney, was made during the discussion “The Intersection between Competition and Data Privacy” held at the 143rd meeting of the OECD Competition Committee on 13 June 2024. More papers and presentations on the topic can be found at oe.cd/ibcdp.
This presentation was uploaded with the author’s consent.
This presentation by Thibault Schrepel, Associate Professor of Law at Vrije Universiteit Amsterdam University, was made during the discussion “Artificial Intelligence, Data and Competition” held at the 143rd meeting of the OECD Competition Committee on 12 June 2024. More papers and presentations on the topic can be found at oe.cd/aicomp.
This presentation was uploaded with the author’s consent.
The importance of sustainable and efficient computational practices in artificial intelligence (AI) and deep learning has become increasingly critical. This webinar focuses on the intersection of sustainability and AI, highlighting the significance of energy-efficient deep learning, innovative randomization techniques in neural networks, the potential of reservoir computing, and the cutting-edge realm of neuromorphic computing. This webinar aims to connect theoretical knowledge with practical applications and provide insights into how these innovative approaches can lead to more robust, efficient, and environmentally conscious AI systems.
Webinar Speaker: Prof. Claudio Gallicchio, Assistant Professor, University of Pisa
Claudio Gallicchio is an Assistant Professor at the Department of Computer Science of the University of Pisa, Italy. His research involves merging concepts from Deep Learning, Dynamical Systems, and Randomized Neural Systems, and he has co-authored over 100 scientific publications on the subject. He is the founder of the IEEE CIS Task Force on Reservoir Computing, and the co-founder and chair of the IEEE Task Force on Randomization-based Neural Networks and Learning Systems. He is an associate editor of IEEE Transactions on Neural Networks and Learning Systems (TNNLS).
This presentation by Nathaniel Lane, Associate Professor in Economics at Oxford University, was made during the discussion “Pro-competitive Industrial Policy” held at the 143rd meeting of the OECD Competition Committee on 12 June 2024. More papers and presentations on the topic can be found at oe.cd/pcip.
This presentation was uploaded with the author’s consent.
This presentation by Tim Capel, Director of the UK Information Commissioner’s Office Legal Service, was made during the discussion “The Intersection between Competition and Data Privacy” held at the 143rd meeting of the OECD Competition Committee on 13 June 2024. More papers and presentations on the topic can be found at oe.cd/ibcdp.
This presentation was uploaded with the author’s consent.
This presentation by Yong Lim, Professor of Economic Law at Seoul National University School of Law, was made during the discussion “Artificial Intelligence, Data and Competition” held at the 143rd meeting of the OECD Competition Committee on 12 June 2024. More papers and presentations on the topic can be found at oe.cd/aicomp.
This presentation was uploaded with the author’s consent.
This presentation by OECD, OECD Secretariat, was made during the discussion “The Intersection between Competition and Data Privacy” held at the 143rd meeting of the OECD Competition Committee on 13 June 2024. More papers and presentations on the topic can be found at oe.cd/ibcdp.
This presentation was uploaded with the author’s consent.
1.) Introduction
Our Movement is not new; it is the same as it was for Freedom, Justice, and Equality since we were labeled as slaves. However, this movement at its core must entail economics.
2.) Historical Context
This is the same movement because none of the previous movements, such as boycotts, were ever completed. For some, maybe, but for the most part, it’s just a place to keep your stable until you’re ready to assimilate them into your system. The rest of the crabs are left in the world’s worst parts, begging for scraps.
3.) Economic Empowerment
Our Movement aims to show that it is indeed possible for the less fortunate to establish their economic system. Everyone else – Caucasian, Asian, Mexican, Israeli, Jews, etc. – has their systems, and they all set up and usurp money from the less fortunate. So, the less fortunate buy from every one of them, yet none of them buy from the less fortunate. Moreover, the less fortunate really don’t have anything to sell.
4.) Collaboration with Organizations
Our Movement will demonstrate how organizations such as the National Association for the Advancement of Colored People, National Urban League, Black Lives Matter, and others can assist in creating a much more indestructible Black Wall Street.
5.) Vision for the Future
Our Movement will not settle for less than those who came before us and stopped before the rights were equal. The economy, jobs, healthcare, education, housing, incarceration – everything is unfair, and what isn’t is rigged for the less fortunate to fail, as evidenced in society.
6.) Call to Action
Our movement has started and implemented everything needed for the advancement of the economic system. There are positions for only those who understand the importance of this movement, as failure to address it will continue the degradation of the people deemed less fortunate.
No, this isn’t Noah’s Ark, nor am I a Prophet. I’m just a man who wrote a couple of books, created a magnificent website: http://www.thearkproject.llc, and who truly hopes to try and initiate a truly sustainable economic system for deprived people. We may not all have the same beliefs, but if our methods are tried, tested, and proven, we can come together and help others. My website: http://www.thearkproject.llc is very informative and considerably controversial. Please check it out, and if you are afraid, leave immediately; it’s no place for cowards. The last Prophet said: “Whoever among you sees an evil action, then let him change it with his hand [by taking action]; if he cannot, then with his tongue [by speaking out]; and if he cannot, then, with his heart – and that is the weakest of faith.” [Sahih Muslim] If we all, or even some of us, did this, there would be significant change. We are able to witness it on small and grand scales, for example, from climate control to business partnerships. I encourage, invite, and challenge you all to support me by visiting my website.
This presentation by OECD, OECD Secretariat, was made during the discussion “Artificial Intelligence, Data and Competition” held at the 143rd meeting of the OECD Competition Committee on 12 June 2024. More papers and presentations on the topic can be found at oe.cd/aicomp.
This presentation was uploaded with the author’s consent.