The document discusses a webinar on measuring impact qualitatively held by Susan Pietrzyk and colleagues from MEASURE Evaluation and ICF International. It provides an overview of the webinar topics which included reflecting on evaluative efforts to understand and measure impact, the role of qualitative methods, and insights from a review of 32 USAID HIV/AIDS evaluation reports. The document outlines the objectives and approach of the webinar and document review, and shares insights and ideas for understanding and assessing impact qualitatively including considering impact as a subjective concept, specifically describing methods, and thinking about the relevance of experience and unit of analysis when evaluating impact.
Monitoring is the continuous collection of data and information on specified indicators to assess the implementation of a development intervention in relation to activity schedules and expenditure of allocated funds, and progress and achievements in relation to its intended outcome.
Evaluation is the periodic assessment of the design implementation, outcome, and impact of a development intervention. It should assess the relevance and achievement of the intended outcome, and implementation performance in terms of effectiveness and efficiency, and the nature, distribution, and sustainability of impact.
A simple presentation about Monitoring and Evaluation prepared by Jubair Ahmad Musazay for interns from Kabul University who are undergoing their internship program in General Directorate of Policy, Monitoring and Evaluation of Afghanistan National Development Strategy (ANDS), in Ministry of Economy of Islamic Republic of Afghanistan.
Uploaded in Slideshare for the purpose of sharing and spreading knowledge.
Monitoring is the continuous collection of data and information on specified indicators to assess the implementation of a development intervention in relation to activity schedules and expenditure of allocated funds, and progress and achievements in relation to its intended outcome.
Evaluation is the periodic assessment of the design implementation, outcome, and impact of a development intervention. It should assess the relevance and achievement of the intended outcome, and implementation performance in terms of effectiveness and efficiency, and the nature, distribution, and sustainability of impact.
A simple presentation about Monitoring and Evaluation prepared by Jubair Ahmad Musazay for interns from Kabul University who are undergoing their internship program in General Directorate of Policy, Monitoring and Evaluation of Afghanistan National Development Strategy (ANDS), in Ministry of Economy of Islamic Republic of Afghanistan.
Uploaded in Slideshare for the purpose of sharing and spreading knowledge.
6 M&E - Monitoring and Evaluation of Aid ProjectsTony
A series of course modules on project cycle, planning and the logical framework, aimed at team leaders of international NGOs in developing countries.
This is part 6 of 11, beginning with 2 modules on leadership and conflict resolution, then 9 modules on project cycle management.
This module has 3 handouts and presenter notes as separate documents.
Sample Proposal: http://www.slideshare.net/Makewa/6-watsan-training-sample-proposal-09
Slides as a handout: http://www.slideshare.net/Makewa/6-me-handout
Presenter notes: http://www.slideshare.net/Makewa/6-module-6-presenter-notes
Policy monitoring and evaluation (M&E) has a critical role
to play in effectively design, implement and deliver public policies and services. Ensuring that policy making is informed by sound evidence on what works is essential to achieve key long-term objectives.
The slides provides an overview on the planning phase of the project plan development. It also provides an introduction on the contents and forms of a project work plan
6 M&E - Monitoring and Evaluation of Aid ProjectsTony
A series of course modules on project cycle, planning and the logical framework, aimed at team leaders of international NGOs in developing countries.
This is part 6 of 11, beginning with 2 modules on leadership and conflict resolution, then 9 modules on project cycle management.
This module has 3 handouts and presenter notes as separate documents.
Sample Proposal: http://www.slideshare.net/Makewa/6-watsan-training-sample-proposal-09
Slides as a handout: http://www.slideshare.net/Makewa/6-me-handout
Presenter notes: http://www.slideshare.net/Makewa/6-module-6-presenter-notes
Policy monitoring and evaluation (M&E) has a critical role
to play in effectively design, implement and deliver public policies and services. Ensuring that policy making is informed by sound evidence on what works is essential to achieve key long-term objectives.
The slides provides an overview on the planning phase of the project plan development. It also provides an introduction on the contents and forms of a project work plan
Strengths and Challenges in the Implementation of Women’s Justice and Empower...MEASURE Evaluation
Led by Shelah Bloom, Suzanne Maman, Elly Arnoff, and Lauren Hill.
The webinar addressed MEASURE Evaluation's evaluation of the implementation of the Women’s Justice and Empowerment Initiative (WJEI) by conducting intensive field studies in Benin and Kenya and document review with telephone interviews in South Africa and Zambia.
I gave this talk at a Nigeria Health Summit in March 2016. It was an introduction to impact evaluation: what it is, when it's a good idea, and some possible approaches.
TIHR (Olivia Joyner + Joe Cullen) and HSMC (Deborah Davidson) presented emerging findings from the service evaluation on accessing the potential and progress of web-based feedback for quality improvement in the Health Service at the prestigious 7th Biennial Conference in Organisational Behaviour in Health Care in April 2010.
An 8-step guide to help not-for-profit organizations develop a Project Impact Pathway or Logic Model for their programs - The first presentation in our Project Evaluation Series. By DevMetrica
An overview of impact evaluation for organizations based on a program's Theory of Change, highlighting the need for a counterfactual and randomization (when possible) in order to convincingly demonstrate the effect of the program.
Including AIDS-affected young people in OVC research: Challenges and opportu...MEASURE Evaluation
A Child Status Network webinar discussing how to involve young people (especially HIV-positive young people) in research about orphans and other vulnerable children. Dr. Lucie Cluver from the Young Carers Project and Oxford University led the November 2012 webinar.
Day 2 keynote: Sanjeev Sridharan, University of Toronto: “Research and evaluation in global health policy processes”
Workshop on Approaches and Methods for Policy Process Research, co-sponsored by the CGIAR Research Programs on Policies, Institutions and Markets (PIM) and Agriculture for Nutrition and Health (A4NH) at IFPRI-Washington DC, November 18-20, 2013.
This discussion, covened by the Dubai Future Foundation, focusses on identifying the significance of the concept of well-being for social-science and policy; and the opportunities to measure it at scale.
Using case studies to explore the generalizability of 'complex' development i...Barb Knittel
Discussion of the questions of internal and external validity and how case-based approaches are relevant for informing replication and scale up. Case studies can help to extrapolate key facts regarding context dynamics, process mechanisms, implementation capability, and trajectories of change (Michael Woolcock, World Bank).
Join us in launching the NCCMT’s new tools for assessing the quality of community evidence. These tools can be applied to evidence for local health issues and context, such as surveillance data and community health status reports, as well as evidence for community and political preferences and actions, such as the expressed needs and interests of community members and support from public or government officials.
Understanding the Dynamics of Successful Health System Strengthening Interven...HFG Project
Evidence is scarce, scattered, and not widely disseminated on how interventions to strengthen health system performance contribute to sustained improvements in health status, particularly toward ending preventable child and maternal deaths and fostering an AIDS-free generation. Without this evidence, decision-makers lack a sound basis for investing scarce health funds in health system strengthening (HSS) interventions in an environment of competing investment options. This evidence gap impedes support for HSS from numerous stakeholders, both within and outside of USAID. This study will help address this evidence gap by exploring the dynamics of successful HSS interventions in low-income countries. The study seeks to address four key questions:
How were a range of successful HSS interventions implemented in different countries?
What factors facilitated and constrained the successful implementation and documented outcomes of the interventions?
What were important factors about implementation that emerged across the different cases?
What are the implications of this study for implementing future HSS interventions?
The study will comprise three main activities:
Six qualitative, retrospective case studies of successful USAID-supported HSS interventions to explore what factors contributed to successful implementation
Qualitative cross-case analysis to identify patterns of policy processes, circumstances, relationships, and characteristics that may be associated with successful HSS reforms
Develop and propose a set of strategic recommendations for introducing and sustaining HSS reforms in low-income countries
The session presents gender analysis tools that can be used during project design, implementation and evaluation. The gender analysis tools will help to make the development intervention gender sensitive, so that the benefits of project reach both women and men.
Zahrah Nesbitt-Ahmed's presentation on UNICEF Innocenti's new Gender-Responsive Age-Sensitive Social Protection (GRASSP) research project at a brown bag session at UNICEF HQ (New York City) during the 64th Commission on the Status of Women.
Community engagement in public health interventions for disadvantaged groups:...Health Evidence™
Health Evidence hosted a 60 minute webinar examining the effectiveness of community engagement in public health interventions for disadvantaged groups. Click here for access to the audio recording for this webinar: https://youtu.be/tUZ-u7QbMCY.
Alison O'Mara-Eves, Senior Researcher, University College London, EPPI-Centre and Ginny Brunton, Senior Health Researcher, University College London, EPPI-Centre presented findings from their review:
O'Mara-Eves A., Brunton G., Oliver S., Kavanagh J., Jamal F., & Thomas J. (2015). The effectiveness of community engagement in public health interventions for disadvantaged groups: A meta-analysis . BMC Public Health, 15, 129.
Community engagement is becoming an increasingly popular component included in the development and implementation of public health interventions. Involved community members take on roles that range from merely being informed, to being consulted, to collaborating or leading on the design, delivery and evaluation of public health strategies. This review examines the use of public health interventions with a community engagement component, particularly for its use in reducing health inequities among disadvantaged populations. Findings of the review suggest community engagement in public health interventions has an effect on several health outcomes, including health behaviours and self-efficacy. This webinar will examine the effectiveness and components of public health interventions that include community engagement and the impact on health outcomes.
SOCIAL PROBLEM RESEARCH4Social Problem ResearchFebruar.docxrosemariebrayshaw
SOCIAL PROBLEM RESEARCH 4
Social Problem Research
February 24, 2020
Running head: SOCIAL PROBLEM RESEARCH 1
Social Problem Research
The National Association of Social Workers (NASW) boasts over 130,000 members, making it the largest professional social workers’ membership organization globally. The organization protects, develops, and promotes professional social work practice and the welfare of its members. It also deals with the creation and maintenance of professional standards and seeks to enhance the well-being of people, families, and communities within the society through advocacy practices. Furthermore, it aims to improve the quality of life of all people in the nation through the practical tackling of social problems such as homeless, HIV/AIDs and other infections, racism and sexism, etc. It enhances the well-being of individuals and groups through its charitable organization, National Association of Social Workers Foundation (NASWF), which was founded in 2001.
Working under the auspices of NASWF, NASW has the goal of rapidly assisting with social crises response, promotion of [practice-based and evidence-based research, supporting education and research, etc. In a nutshell, NASW provides intervention to various social problems, including tackling the issue of homelessness. Hence, this program will address the problem of homelessness in Michigan. The program evaluation will be conducted using both qualitative and quantitative social research methods.
Therefore, first, how do I know that I have a legitimate research problem in Michigan? Homeless people refers to those who lack regular, fixed, and adequate residences to spend in at night, or those whose primary residences are temporary, or any place that is not recommended for people to get sleeping accommodation or those who live in temporary and supervised shelter. Homeless people do not have stable housing in both rural and urban areas and come from various demographic backgrounds. They also face several health concerns. Hence, it poses a severe public healthcare concern that needs a quick and structured social work service.
Apart from the qualitative underpinning, homelessness is as widespread in Michigan as it is in the entire US. Over 744,313 people, including children, women, and men, were homeless in the US by 2005 (Social Work Policy Institute, 2010). Globally, over a billion people lack adequate housing, of which 100 million are entirely destitute of homes. About 5.4 million families are at risk of becoming homeless because of sufficient income to afford to pay their rent. It is highly probable that over 66,051individuals in Michigan are considered homeless, with the statistics including 41,338 unsheltered and 24,713 sheltered homeless people. The majority of the people considered homeless in Michigan are adults.
Delineating the subparts of the homelessness as a social problem in this research project would be a complicated process. The evaluation criteria follo.
SOCIAL PROBLEM RESEARCH4Social Problem ResearchFebruar.docxsamuel699872
SOCIAL PROBLEM RESEARCH 4
Social Problem Research
February 24, 2020
Running head: SOCIAL PROBLEM RESEARCH 1
Social Problem Research
The National Association of Social Workers (NASW) boasts over 130,000 members, making it the largest professional social workers’ membership organization globally. The organization protects, develops, and promotes professional social work practice and the welfare of its members. It also deals with the creation and maintenance of professional standards and seeks to enhance the well-being of people, families, and communities within the society through advocacy practices. Furthermore, it aims to improve the quality of life of all people in the nation through the practical tackling of social problems such as homeless, HIV/AIDs and other infections, racism and sexism, etc. It enhances the well-being of individuals and groups through its charitable organization, National Association of Social Workers Foundation (NASWF), which was founded in 2001.
Working under the auspices of NASWF, NASW has the goal of rapidly assisting with social crises response, promotion of [practice-based and evidence-based research, supporting education and research, etc. In a nutshell, NASW provides intervention to various social problems, including tackling the issue of homelessness. Hence, this program will address the problem of homelessness in Michigan. The program evaluation will be conducted using both qualitative and quantitative social research methods.
Therefore, first, how do I know that I have a legitimate research problem in Michigan? Homeless people refers to those who lack regular, fixed, and adequate residences to spend in at night, or those whose primary residences are temporary, or any place that is not recommended for people to get sleeping accommodation or those who live in temporary and supervised shelter. Homeless people do not have stable housing in both rural and urban areas and come from various demographic backgrounds. They also face several health concerns. Hence, it poses a severe public healthcare concern that needs a quick and structured social work service.
Apart from the qualitative underpinning, homelessness is as widespread in Michigan as it is in the entire US. Over 744,313 people, including children, women, and men, were homeless in the US by 2005 (Social Work Policy Institute, 2010). Globally, over a billion people lack adequate housing, of which 100 million are entirely destitute of homes. About 5.4 million families are at risk of becoming homeless because of sufficient income to afford to pay their rent. It is highly probable that over 66,051individuals in Michigan are considered homeless, with the statistics including 41,338 unsheltered and 24,713 sheltered homeless people. The majority of the people considered homeless in Michigan are adults.
Delineating the subparts of the homelessness as a social problem in this research project would be a complicated process. The evaluation criteria follo.
Using case studies to explore the generalizability of 'complex' development i...JSI
Discussion of the questions of internal and external validity and how case-based approaches are relevant for informing replication and scale up. Case studies can help to extrapolate key facts regarding context dynamics, process mechanisms, implementation capability, and trajectories of change (Michael Woolcock, World Bank).
Peter Levesque explores the critical areas of measuring, interpreting, and analyzing results to ensure continual improvement of KT activities to produce intended results.
The purpose of this paper is to explore the development of universal design and to consider its application to large scale assessments. Building on universal design principles presented by the Center for Universal Design, seven elements of universally designed assessments are identified and described in this paper. The seven elements are:
1. Inclusive assessment population
2. Precisely defined constructs
3. Accessible, non-biased items
4. Amendable to accommodations
5. Simple, clear, and intuitive instructions and procedures
6. Maximum readability and comprehensibility
7. Maximum legibility
Each of the elements is explored in this paper.
ICRB is a leading research and consulting firm that
provides data driven pragmatic and cost effective
innovative solutions
Our Design centric Human Connection framework
helps Development Sector agencies in project
implementation and monitoring, streamlining and
scaling up processes, improving visibility and Sharing
knowledge.
ICRB's Strong and widespread network of consultants
and domain experts enable it to operate in diverse
markets and consumer segments
Managing missing values in routinely reported data: One approach from the Dem...MEASURE Evaluation
This Data for Impact webinar was held in December 2020. Access the recording and learn more at https://www.data4impactproject.org/resources/webinars/managing-missing-values-in-routinely-reported-data-one-approach-from-the-democratic-republic-of-the-congo/
This Data for Impact webinar took place October 29, 2020. Learn more at https://www.data4impactproject.org/resources/webinars/use-of-routine-data-for-economic-evaluations/
Data for Impact hosted a one-hour webinar sharing guidance for using routine data in evaluations. More: https://www.data4impactproject.org/resources/webinars/routine-data-use-in-evaluation-practical-guidance/
Lessons learned in using process tracing for evaluationMEASURE Evaluation
Access the recording for this Data for Impact (D4I) webinar at https://www.data4impactproject.org/lessons-learned-in-using-process-tracing-for-evaluation/
Instructions for Submissions thorugh G- Classroom.pptxJheel Barad
This presentation provides a briefing on how to upload submissions and documents in Google Classroom. It was prepared as part of an orientation for new Sainik School in-service teacher trainees. As a training officer, my goal is to ensure that you are comfortable and proficient with this essential tool for managing assignments and fostering student engagement.
Macroeconomics- Movie Location
This will be used as part of your Personal Professional Portfolio once graded.
Objective:
Prepare a presentation or a paper using research, basic comparative analysis, data organization and application of economic information. You will make an informed assessment of an economic climate outside of the United States to accomplish an entertainment industry objective.
Honest Reviews of Tim Han LMA Course Program.pptxtimhan337
Personal development courses are widely available today, with each one promising life-changing outcomes. Tim Han’s Life Mastery Achievers (LMA) Course has drawn a lot of interest. In addition to offering my frank assessment of Success Insider’s LMA Course, this piece examines the course’s effects via a variety of Tim Han LMA course reviews and Success Insider comments.
Biological screening of herbal drugs: Introduction and Need for
Phyto-Pharmacological Screening, New Strategies for evaluating
Natural Products, In vitro evaluation techniques for Antioxidants, Antimicrobial and Anticancer drugs. In vivo evaluation techniques
for Anti-inflammatory, Antiulcer, Anticancer, Wound healing, Antidiabetic, Hepatoprotective, Cardio protective, Diuretics and
Antifertility, Toxicity studies as per OECD guidelines
Operation “Blue Star” is the only event in the history of Independent India where the state went into war with its own people. Even after about 40 years it is not clear if it was culmination of states anger over people of the region, a political game of power or start of dictatorial chapter in the democratic setup.
The people of Punjab felt alienated from main stream due to denial of their just demands during a long democratic struggle since independence. As it happen all over the word, it led to militant struggle with great loss of lives of military, police and civilian personnel. Killing of Indira Gandhi and massacre of innocent Sikhs in Delhi and other India cities was also associated with this movement.
Embracing GenAI - A Strategic ImperativePeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Synthetic Fiber Construction in lab .pptxPavel ( NSTU)
Synthetic fiber production is a fascinating and complex field that blends chemistry, engineering, and environmental science. By understanding these aspects, students can gain a comprehensive view of synthetic fiber production, its impact on society and the environment, and the potential for future innovations. Synthetic fibers play a crucial role in modern society, impacting various aspects of daily life, industry, and the environment. ynthetic fibers are integral to modern life, offering a range of benefits from cost-effectiveness and versatility to innovative applications and performance characteristics. While they pose environmental challenges, ongoing research and development aim to create more sustainable and eco-friendly alternatives. Understanding the importance of synthetic fibers helps in appreciating their role in the economy, industry, and daily life, while also emphasizing the need for sustainable practices and innovation.
1. Susan Pietrzyk, PhD
MEASURE Evaluation
ICF International
Email: susan.pietrzyk@icfi.com
Twitter: @susanpietrzyk
October 29, 2015
Measuring Impact Qualitatively
2. Aim of the webinar
To elicit discussion and share insights
regarding evaluative efforts to understand
and measure impact and the role of
qualitative methods in these efforts
3. Common ground
Impact evaluations are trendy
Qualitative work important to how impact
is examined and measured
USAID evaluations have long been
qualitative oriented
Manuals and toolkits about qualitative
research are plentiful
Interest is in reflection, the trend
4. About the team
ICF International
Research and Evaluation Staff
Susan Pietrzyk, PhD
Reeti Hobson, MPH
Lwendo Moonzwe, PhD
Debra Prosnitz, MPH
5. Set-up of the study
Observation & Investigation
• Projects, articles, social media, RfPs, webinars,
reports, proposals, presentations, etc.
Document Review & Bearings
• USAID HIV/AIDS-related evaluation reports
• Development Experience Clearinghouse (DEC)
• 2003-present, selected 32 documents
• USAID policy statements
• 3ie impact evaluation repository
Conversations & Mapping
• Within ICF, MEASURE partners, friends
• Talks, networking, meeting new people
• Who’s conducted USAID HIV/AIDS evaluations
6. Overview of the webinar
Why
Take a step back and assess the trend
Reflection, both critical and practical
Context
Shifts over time and the evaluation landscape
Understand the past to plan for the future
Document Review
Insights and concrete ideas, not a singular method
Tips for impact and qualitative evaluative work
Questions
Your ideas and thoughts
Develop some momentum
7. Why
The motivation for the Measuring Impact
Qualitatively study has been twofold:
• Impact evaluations are increasingly of interest
among international development policymakers and
practitioners
• Although the term “impact evaluation” signifies to
many a specifically rigorous quantitative exercise
and use of a counterfactual, impact is also both an
old and a subjective concept
9. Impact = an old concept
It is now recognized that social progress is essential to economic
development, and that without improvements in education, both
general and technical, health, sanitation, land utilization, tax
structures, and social justice, foreign development assistance
programs would be extremely limited in their impact on the economic
development of underdeveloped countries
It is striking in fact that, while the United States has engaged in
assistance programs of various types for decades, no systematic
analysis of this type has been made on such problems as, for
example, the impact of certain technological developments on the
economies of nations, or the transfer of technology from one society
to another
10. Impact = an old concept
It is now recognized that social progress is essential to economic development, and
that without improvements in education, both general and technical, health, sanitation,
land utilization, tax structures, and social justice, foreign development assistance
programs would be extremely limited in their impact on the economic development of
underdeveloped countries
-Wallace J. Campbell and David D. Lloyd, Report on the
Eighth Annual National Conference on International
Economic and Social Development June 1961
It is striking in fact that, while the United States has engaged in assistance programs of
various types for decades, no systematic analysis of this type has been made on such
problems as, for example, the impact of certain technological developments on the
economies of nations, or the transfer of technology from one society to another
-Henry R. Labouisse, Director, President’s Task Force on
Foreign Economic Assistance. The Act for International
Development: A Program for the Decade of Development
June 1961
11. Impact = subjective concept
32 HIV/AIDS-related evaluation reports reviewed
Total Average
Number of pages 3,383 106
Uses of the word impact 765 24
Uses of impact-related words
Specifically: achievements, effectiveness,
outcome, performance, success, sustainability
6,105 191
12. Impact = subjective concept
765 + 6,105 = 6,870 words (32 reports)
These are all words that work as part of
understanding and assessing impact
13. Impact = subjective concept
765 + 6,105 = 6,870 words (32 reports)
These are all words that work as part of
understanding and assessing impact
• Is the meaning always the same?
• Does location change the meaning?
• Might women and men understand
the words differently?
• Can we know if the experience was
always the same?
14.
15. Document review
Objectives
Undertake a practical and critical reflection of the details
surrounding the use of qualitative methods in USAID
HIV/AIDS-related evaluations
Productively use this critical reflection to establish
insights, ideas, and promising practices around ways to
understand and assess impact in qualitative terms
16. Document review
• Search for impact evaluations (inclusive of a
counterfactual) and impact-oriented qualitative
evaluations (often, performance evaluations)
• Development Experiences Clearinghouse (DEC)
• Keyword search on evaluation, impact evaluation,
assessment, HIV, AIDS
o Three people searching
o 2003-2008, 2009-2013, 2014- present
• Yield in the 1,000s, narrow based on document title
o Exclude mid-term evaluations, conference proceedings, trip
reports, project quarterly reports
Methodology
17. Document review
• Down to 224 evaluation reports
o Read the abstract, scan the report
o Specifically looking for impact evaluations
o In reality, found more impact oriented qualitative evaluations
o Check-in going from 224 to 168 to 144 to 104
• At 104 divide between impact evaluation (23) and
impact oriented qualitative evaluation (81)
• From 104 to 32, HIV/AIDS focus, strive for balance
• Map the selected documents, maintain anonymity
o 21 countries, 25 evaluating organizations, 26 implementers
o Years: 2004, 2005, 2006 (2), 2007, 2008 (3), 2009 (2), 2010,
2011 (3), 2012 (9), 2013 (4), 2014 (4), 2015
• Read all 32, use ATLAS.ti to code
Methodology (cont.)
18. Instances(n = 32)
Evaluation / Data Type Type of Primary Data Collection (Qualitative)
Mixed Methods Evaluation 23
Key Informant
Interview
FocusGroup
Discussion
In-Depth
Interview
Primarily Qualitative Evaluation 9
EvaluationsInclusive of a Control Group 13
Primary Data Collection (Quantitative) 15
Primary Data Collection (Qualitative) 31 23 19 8
Total Average Median Range
Characteristicsof the Evaluation Reports(including annexes)
Pages 3,383 106 93 21 - 339
Uses of the Word Impact 765 24 22 3 - 91
Uses of Impact Related Words 6,105 191 146 14 - 794
Uses of the Word Gender 1,027 32 10 1 - 369
CountriesRepresented Acrossthe Sample (21)
Cote D'Ivoire (2), Ethiopia(3), Ghana, Honduras, India (3), Kenya (2), Malawi (2), Multiple Country Evaluations (3),
Namibia, Nepal, Rwanda, South Africa(4), Swaziland, Tanzania, Uganda(3), Zambia(2) and Zimbabwe
Year of the Evaluation Report Acrossthe Sample
2004, 2005, 2006 (2), 2007, 2008 (3), 2009 (2), 2010, 2011 (3), 2012 (9), 2013 (4), 2014 (4) and 2015
Evaluating OrganizationsRepresented Acrossthe Sample (25)
Business Enterprise, University of Pretoria, CAMRIS, Care India, CHANGES2, Clacherty & Associates Education and
Social Developments (Pty) Ltd, Development & TrainingServices (dTS) (3), Engender Health, FARST Africa, Feedback
Research and Analytics, Inc., HORIZONS(2), Impact Consulting, John Snow International (JSI), Management Sciences
for Health (MSH), Management Systems International (MSI) (2), MEASUREEvaluation (2), MELA PLC, Mendez England
& Associates (ME&A), MIDEGO, Inc., The Mitchell Group (2), QED Group, LLC (3), Save the Children, Social Impact,
Inc., Social Scientific Systems, Synergy and USAID
Project ImplementersRepresented Acrossthe Sample (26)
Care India, CHANGES2, Chemonics, Childline Mpumalanga, Children in Distress Network, Development
Alternatives, Inc. (DAI), Elizabeth Glaser Pediatric AIDSFoundation (EGPAF) (2), Engender Health (2), FHI360,
GOAL, HORIZONS(2), HumanaPeople to People, IntraHealth, John Snow International (JSI) (2), John Snow
International Research & TrainingInstitute, Inc., Johns Hopkins University Center for Communications Programs,
Management Sciences for Health (MSH), National Association of Child Care Workers, Government of Ghana
National HIV Prevention Program, Pact, Pathfinder, Population Services International (PSI), Save the Children (2),
USAID and Zambia VCT Partnership
19. Insights and ideas
Document Review
What struck me. What I discovered
Choosing not to use terms such as findings or results
The goal with these insights and ideas is not to produce a
manual about the use of qualitative methods. There are of
these enough already
There is no single qualitative method or approach. There
are many methods, many families of methods
Embrace the diversity. Own it. Say what you mean
20. Method specificity
Capacity assessment tool
Case story
Case study
Case study documentations
Client interaction
Client interviews
Community-level group discussion
Comprehensive analysis session
Detailed discussions
Detailed narratives
Direct observations
Discussions
Email survey
Facility checklist
Facility-level group discussion
Field visits
Fieldwork
Focus group discussion
Focus group interview
Guided group discussion
Held discussions
In-depth discussion
In-depth interview
Insightful occurrences
Interactions with stakeholders
Interpretive phenomenology
Key actor interview
Key informant interview
Observational checklists
Observations
Open-ended interviews
Opportunistic group discussion
Organization interviews
Plenary group dialogue meetings
Regular group analysis
Site visits
Structured group discussions
Structured interviews
Success story
Telephone discussions
Triangulation
41 qualitative data collection methods terms
21. The limits of listing
The Team performed an intensive desk review of the documents,
including those provided by USAID, and data encountered through
Internet searches, site visits and discussions with local counterparts
(Appendix B). A Team Planning Meeting was held on XXXX to draft
the evaluation framework, followed by an orientation meeting with
USAID on XXXX. The framework was subsequently approved by
USAID in an email dated XXXX. Following that meeting, the Team
met in XXXX with the implementing agencies and other stakeholders
as identified by USAID (Appendix B). The Team finalized the
evaluation framework in accordance with USAID feedback and
guidance, and further tested and refined the discussion guides
(Appendix D). (P24)
22. Activities ≠ method
1) a team planning meeting between the team and USAID/XXXX;
2) extensive desk review of all project-related documents;
3) interviews with key informants;
4) field visits to three of the seven XXXX districts in XXXX, one XXXX
district, and one XXXX district (control districts), where team members
met select NGOs and community members and visited XXXX Clinics;
5) meetings with the XXXX and XXXX in XXXX and XXXX;
6) client briefings with XXXX and USAID through in-person meetings and
teleconferences; and
7) presentations and discussion of findings with members of the XXXX
team, USAID and XXXX.
To enhance the quantitative rigor of the evaluation, the team undertook a
separate epidemiological study to analyze health outcomes. The evaluation
team conducted field visits to verify data collection and to inform subsequent
findings qualitatively. Key informant interviews further enhanced the findings
of the XXXX Documentation team and provided additional insight (P4)
The methodology included
23. Method
Two overall method statements
The evaluation team used both qualitative and
quantitative methods to collect and analyze information
relevant to the objectives, the four outcomes of the
development hypothesis, and the research questions
outlined in the Scope of Work. (P3)
A sequential mixed methods design was used to combine
quantitative (survey design) and interpretive qualitative
aspects. (P2)
24. Method
A subtle but telling variation
The evaluation team used both qualitative and
quantitative methods to collect and analyze information
relevant to the objectives, the four outcomes of the
development hypothesis, and the research questions
outlined in the Scope of Work. (P3)
A sequential mixed methods design was used to
combine quantitative (survey design) and interpretive
qualitative aspects. (P2)
25. Mixed method
Denzin, Norman (2010) Qualitative Inquiry
Analysis traces history surrounding rise mixed method.
Persons who are less familiar with the rich traditions of
qualitative inquiry are telling others with the same lack of
experience how to do qualitative work (2010: 420)
This type of instruction exists, Denzin suggests, because the
call for mixed method work has largely come from those with
quantitative expertise not those with qualitative expertise.
26. Mapping
Evidence for Denzin’s assertion
Article: Tracing publications, tracing history, a compelling
analysis of paradigms and when, why, how they shift
Document review: Very rare that the bios of the
evaluators are included; thus, we can’t know the expertise
of the evaluation team members
Observationally: Often an approach to qualitative
evaluations relies on mapping quantitative methods and
words onto a qualitative study. Potentially an indication of
quantitative experts trying to map their expertise onto an
area (qualitative work) where they have less expertise
27. Unit of analysis
Conceptually, the team chose to focus on both the
provision of services, the “supply side” or referred to in
the PMP as “Access”, as well as the “demand side”,
mainly the utilization of those services by beneficiaries.
Three main units of analysis were: first, national level
leaders who had received training and other inputs from
XXXX; second, the district level, including the XX country
unit of local government and local level civil society
organizations, and third, HIV/AIDS affected Households,
as identified from People with HIV/AIDS (PHA)
registration lists. (P17)
28. Impact
Key informant interview = 23
What is the unit of analysis for a key informant interview?
Focus group discussion = 19
What is the unit of analysis for a focus group discussion?
In-depth interview = 8
What is the unit of analysis for an in-depth interview?
29. Impact
Key informant interview = 23
Did the key informant experience the impact?
Focus group discussion = 19
Did the focus group experience the impact?
In-depth interview = 8
Did the in-depth interviewee experience the impact?
30. Summary & tips
Impact & Qualitative Evaluative Work
Consider that impact is a subjective concept
Tailor the methods, descriptively title interview types
Say what you mean, state method specifically and in detail
Think before you list, methods deserve full sentences
Flush things out, activities are not methods
Both and mixed are not synonyms
Mixed method requires quantitative and qualitative expertise
When to / when not to map quantitative onto qualitative
Unit of analysis and its relevance to understanding impact
Experience and its relevance to feeling impact
31. MEASURE Evaluation is funded by the U.S. Agency
for International Development (USAID) under terms
of Cooperative Agreement AID-OAA-L-14-00004 and
implemented by the Carolina Population Center, University
of North Carolina at Chapel Hill in partnership with ICF
International, John Snow, Inc., Management Sciences for
Health, Palladium Group, and Tulane University. The views
expressed in this presentation do not necessarily reflect
the views of USAID or the United States government.
www.measureevaluation.org