Sentence completion tests are a class of semi-structured projective techniques.
Sentence completion tests typically provide respondents with beginnings of sentences, referred to as "stems", and respondents then complete the sentences in ways that are meaningful to them.
The responses are believed to provide indications of
Attitudes,
Beliefs,
Motivations, or other
Mental states.
Therefore, sentence completion technique, with such advantage, promotes the respondents to disclose their concealed feelings.
There is debate over whether or not sentence completion tests elicit responses from conscious thought rather than unconscious states.
This debate would affect its categorizing as projective tests
Survey Methodology and Questionnaire Design Theory Part IQualtrics
Do you know what's going on in your respondents' heads as they take your survey? How can you design your questionnaire to collect better data? Understanding the answers to these questions can help you design surveys that collect high quality insights you can depend on.
Dave Vannette, principal research scientist at Qualtrics, shares his best hacks for designing surveys that will help you get quality data. In this presentation, Dave also highlights what your respondents are thinking when they take your surveys, and how your survey design can affect the responses you collect.
Sentence completion tests are a class of semi-structured projective techniques.
Sentence completion tests typically provide respondents with beginnings of sentences, referred to as "stems", and respondents then complete the sentences in ways that are meaningful to them.
The responses are believed to provide indications of
Attitudes,
Beliefs,
Motivations, or other
Mental states.
Therefore, sentence completion technique, with such advantage, promotes the respondents to disclose their concealed feelings.
There is debate over whether or not sentence completion tests elicit responses from conscious thought rather than unconscious states.
This debate would affect its categorizing as projective tests
Survey Methodology and Questionnaire Design Theory Part IQualtrics
Do you know what's going on in your respondents' heads as they take your survey? How can you design your questionnaire to collect better data? Understanding the answers to these questions can help you design surveys that collect high quality insights you can depend on.
Dave Vannette, principal research scientist at Qualtrics, shares his best hacks for designing surveys that will help you get quality data. In this presentation, Dave also highlights what your respondents are thinking when they take your surveys, and how your survey design can affect the responses you collect.
Companies are increasingly giving jobseekers psychological assessment as part of their hiring process.
They are becoming more popular than ever and most candidates have strong dislike feelings toward this type of tests.
This presentation is to share with you what type of Assessment you will face and what they are measuring.
We hope that with this knowledge at hand, you are more equipped to be prepared for the Psychological Assessment
For more classes visit
www.snaptutorial.com
PSY 352 Week 1 Assignment Attention and Perception
Attention and Perception. Due by Day 7. Prior to completing this assignment, please read Chapter 3 carefully.
Cognitive testing and vignettes in the WEAI - IFPRI Gender Methods SeminarIFPRI Gender
Katie Sproule and Chiara Kovarik (IFPRI) present on using cognitive testing and vignettes in the Women's Empowerment in Agriculture Index (WEAI), for a Gender Methods Seminar, Dec. 12, 2014.
Audio recording of the presentation available here: http://bit.ly/1zG14dI
A mini workshop designed to prepare teams with the knowledge and practice they need to better understand their problems and project gaps, determine appropriate participants, ask the right qualitative questions, and gather information in an unbiased and thoughtful way.
Developing a good instrument is the most crucial part in conducting a research study. When an instrument is poorly designed, it may lead to inaccurate data that may produce misguided conclusions. However, developing an instrument takes a lot of work and requires advanced preparation. Therefore, researchers must be careful when developing their instrument.
A questionnaire is a research instrument wherein a researcher poses a set of predetermined questions. It typically has three parts:
Questionnairre desisgn-Advance Research MethodologyRehan Ehsan
This Presentation states the details of Questionnairre desisgn for students to get help in advance research methodology. Rearchers may also get help from this work.
Companies are increasingly giving jobseekers psychological assessment as part of their hiring process.
They are becoming more popular than ever and most candidates have strong dislike feelings toward this type of tests.
This presentation is to share with you what type of Assessment you will face and what they are measuring.
We hope that with this knowledge at hand, you are more equipped to be prepared for the Psychological Assessment
For more classes visit
www.snaptutorial.com
PSY 352 Week 1 Assignment Attention and Perception
Attention and Perception. Due by Day 7. Prior to completing this assignment, please read Chapter 3 carefully.
Cognitive testing and vignettes in the WEAI - IFPRI Gender Methods SeminarIFPRI Gender
Katie Sproule and Chiara Kovarik (IFPRI) present on using cognitive testing and vignettes in the Women's Empowerment in Agriculture Index (WEAI), for a Gender Methods Seminar, Dec. 12, 2014.
Audio recording of the presentation available here: http://bit.ly/1zG14dI
A mini workshop designed to prepare teams with the knowledge and practice they need to better understand their problems and project gaps, determine appropriate participants, ask the right qualitative questions, and gather information in an unbiased and thoughtful way.
Developing a good instrument is the most crucial part in conducting a research study. When an instrument is poorly designed, it may lead to inaccurate data that may produce misguided conclusions. However, developing an instrument takes a lot of work and requires advanced preparation. Therefore, researchers must be careful when developing their instrument.
A questionnaire is a research instrument wherein a researcher poses a set of predetermined questions. It typically has three parts:
Questionnairre desisgn-Advance Research MethodologyRehan Ehsan
This Presentation states the details of Questionnairre desisgn for students to get help in advance research methodology. Rearchers may also get help from this work.
An interview is a qualitative research method that relies on asking questions in order to collect data. Interviews involve two or more people, one of whom is the interviewer asking the questions. There are several types of interviews, often differentiated by their level of structure.
Design thinking is a problem solving process geared for ambiguous situations. There are four principles of design thinking: empathize, visualize, co-create and iterate. This presentation gives tips and techniques for empathizing includes how to interview and how to analyze research data.
Value Chains for Nutrition in Rural India: investigating barriers among womenAg4HealthNutrition
A4NH and IFPRI Gender Task Force seminar on “Value Chains for Nutrition in Rural India: Investigating barriers to fruit and vegetable consumption among women of reproductive age” presented by Sarah Kehoe, Senior Research Fellow Medical Research Council Lifecourse Epidemiology Unit (MRC LEU), University of Southampton, UK
Measuring Infant and Young Child Complementary Feeding PracticesAg4HealthNutrition
In this presentation, Dr. Marie Ruel, Director of the Poverty, Health, and Nutrition Division of the International Food Policy Research Institute (IFPRI) tackles the important issue of how to measure infant and young child feeding practices, as they relate to child nutrition. She explains indicators, current practices, and research gaps. Dr. Ruel also serves as leader of the research flagship on integrated nutrition programs and policies for the CGIAR Research Program on Agriculture for Nutrition and Health (A4NH), led by IFPRI. This presentation was delivered at the 87th Nestle Nutrition Institute Workshop in Singapore May 8-11, 2016.
For more information on child nutrition research, please visit ifpri.org or a4nh.cgiar.org.
Training Session 4 – Raneri – Adapting the Gender Norms Case Study for NutritionAg4HealthNutrition
Training Session 4: Global Study on Gender Norms and Capacities to Innovate in Agriculture
Jessica Raneri
Adapting the Gender Norms Case Study for Nutrition
Training Session 4 – Bergman-Lodin – Intro to Global Study on Innovation in Ag IAg4HealthNutrition
Training Session 4: Global Study on Gender Norms and Capacities to Innovate in Agriculture
Johanna Bergman-Lodin
Intro to Global Study on Innovation in Ag I
Many ways to support street children.pptxSERUDS INDIA
By raising awareness, providing support, advocating for change, and offering assistance to children in need, individuals can play a crucial role in improving the lives of street children and helping them realize their full potential
Donate Us
https://serudsindia.org/how-individuals-can-support-street-children-in-india/
#donatefororphan, #donateforhomelesschildren, #childeducation, #ngochildeducation, #donateforeducation, #donationforchildeducation, #sponsorforpoorchild, #sponsororphanage #sponsororphanchild, #donation, #education, #charity, #educationforchild, #seruds, #kurnool, #joyhome
Jennifer Schaus and Associates hosts a complimentary webinar series on The FAR in 2024. Join the webinars on Wednesdays and Fridays at noon, eastern.
Recordings are on YouTube and the company website.
https://www.youtube.com/@jenniferschaus/videos
A process server is a authorized person for delivering legal documents, such as summons, complaints, subpoenas, and other court papers, to peoples involved in legal proceedings.
Understanding the Challenges of Street ChildrenSERUDS INDIA
By raising awareness, providing support, advocating for change, and offering assistance to children in need, individuals can play a crucial role in improving the lives of street children and helping them realize their full potential
Donate Us
https://serudsindia.org/how-individuals-can-support-street-children-in-india/
#donatefororphan, #donateforhomelesschildren, #childeducation, #ngochildeducation, #donateforeducation, #donationforchildeducation, #sponsorforpoorchild, #sponsororphanage #sponsororphanchild, #donation, #education, #charity, #educationforchild, #seruds, #kurnool, #joyhome
Russian anarchist and anti-war movement in the third year of full-scale warAntti Rautiainen
Anarchist group ANA Regensburg hosted my online-presentation on 16th of May 2024, in which I discussed tactics of anti-war activism in Russia, and reasons why the anti-war movement has not been able to make an impact to change the course of events yet. Cases of anarchists repressed for anti-war activities are presented, as well as strategies of support for political prisoners, and modest successes in supporting their struggles.
Thumbnail picture is by MediaZona, you may read their report on anti-war arson attacks in Russia here: https://en.zona.media/article/2022/10/13/burn-map
Links:
Autonomous Action
http://Avtonom.org
Anarchist Black Cross Moscow
http://Avtonom.org/abc
Solidarity Zone
https://t.me/solidarity_zone
Memorial
https://memopzk.org/, https://t.me/pzk_memorial
OVD-Info
https://en.ovdinfo.org/antiwar-ovd-info-guide
RosUznik
https://rosuznik.org/
Uznik Online
http://uznikonline.tilda.ws/
Russian Reader
https://therussianreader.com/
ABC Irkutsk
https://abc38.noblogs.org/
Send mail to prisoners from abroad:
http://Prisonmail.online
YouTube: https://youtu.be/c5nSOdU48O8
Spotify: https://podcasters.spotify.com/pod/show/libertarianlifecoach/episodes/Russian-anarchist-and-anti-war-movement-in-the-third-year-of-full-scale-war-e2k8ai4
Jennifer Schaus and Associates hosts a complimentary webinar series on The FAR in 2024. Join the webinars on Wednesdays and Fridays at noon, eastern.
Recordings are on YouTube and the company website.
https://www.youtube.com/@jenniferschaus/videos
This session provides a comprehensive overview of the latest updates to the Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards (commonly known as the Uniform Guidance) outlined in the 2 CFR 200.
With a focus on the 2024 revisions issued by the Office of Management and Budget (OMB), participants will gain insight into the key changes affecting federal grant recipients. The session will delve into critical regulatory updates, providing attendees with the knowledge and tools necessary to navigate and comply with the evolving landscape of federal grant management.
Learning Objectives:
- Understand the rationale behind the 2024 updates to the Uniform Guidance outlined in 2 CFR 200, and their implications for federal grant recipients.
- Identify the key changes and revisions introduced by the Office of Management and Budget (OMB) in the 2024 edition of 2 CFR 200.
- Gain proficiency in applying the updated regulations to ensure compliance with federal grant requirements and avoid potential audit findings.
- Develop strategies for effectively implementing the new guidelines within the grant management processes of their respective organizations, fostering efficiency and accountability in federal grant administration.
What is the point of small housing associations.pptxPaul Smith
Given the small scale of housing associations and their relative high cost per home what is the point of them and how do we justify their continued existance
ZGB - The Role of Generative AI in Government transformation.pdfSaeed Al Dhaheri
This keynote was presented during the the 7th edition of the UAE Hackathon 2024. It highlights the role of AI and Generative AI in addressing government transformation to achieve zero government bureaucracy
Training Session 2 – Sproule and Kovarik – Using Cognitive Testing and Vignettes in the WEAI
1. Lessons from WEAI fieldwork in Uganda and Bangladesh
Presented by Katie Sproule & Chiara Kovarik (IFPRI)
A4NH Gender Methods Workshop, Rome
December 2-4, 2014
2. Why use cognitive testing and vignettes in the WEAI?
Introduction to cognitive testing
Introduction to vignettes
Applying these tools to the WEAI
The fieldwork
The results
Lessons learned
2
3. Why did the WEAI need these
tools?
After the 2012-2013 baselines, it became obvious
that the WEAI needed to undergo some revisions
and streamlining
Key indicators were identified as problematic
Decision was made to develop a second version of
the WEAI (WEAI 2.0)
Cognitive testing was conducted to ensure that
the questions were capturing the various
dimensions of empowerment and also to ensure
that the index remained standardized
Vignettes were included to see if they would be a
better way of getting at issues of autonomy
• The WEAI was developed by
IFPPRI, OPHI, and USAID in 2012
to measure women’s levels of
empowerment and inclusion in
the agricultural sector
• It was initially designed to be a
monitoring and evaluation tool
for USAID’s Feed the Future (FTF)
programming in the 19 FTF
countries
• It is composed of 2 sub-indexes:
the five domains of
empowerment (5DE) and the
Gender Parity Index (GPI).
• The 5 domains are: Production,
Resources, Income, Leadership,
and Time
3
4. • Cognitive testing is a qualitative
method that is paired with a
(quantitative) survey
• The purpose of cognitive testing
is to systematically identify and
analyze sources of response error
in surveys, and to use that
information to improve the
quality and accuracy of survey
instruments (Johnson, 2013)
• Cognitive testing can be
especially important for
new/revised instruments, or
those that will be used in
multiple country contexts
(Johnson, 2013)
• Generally conducted as a pre-test
before full field work begins
Cognitive process
Cognitive Stages Cognitive Stage
Definition
Problems Causes
1. Comprehension
2. Retrieval
3. Judgment
4. Response
Source: Johnson, 2013
Breakdown can occur in ANY of the four stages
4
Respondent
interprets the
question
Respondent
does not
understand
Unknown terms, ambiguous
concepts, long and overly
complex
Respondent
searchers memory
for relevant
information
Respondent does
not
remember/does
not know
Recall difficulty, questions
assume respondent has
information
Respondent
evaluates and/or
estimates response
Respondent does
not want to tell,
can’t tell
Biased or sensitive, estimation
difficulty
Respondent
provides
information in the
format requested
Respondent can’t
respond in the
format requested
Incomplete response options,
multiple responses necessary
5. How satisfied are you with your available time for leisure activities? Please give your
opinion on a scale of 1 to 10. 1 means you are not satisfied and 10 means you are very satisfied. If you
are neither satisfied or dissatisfied this would be in the middle or 5 on the scale.
Breakdown in comprehension:
Respondent may not
understand the concept of
“leisure”, or may understand
it differently from the
researcher
The concept of
“satisfaction” is ambiguous
and subjective
Breakdown in response:
Respondent may have never
answered a question in this
format. While questions with
ranking scales are familiar to
Western audiences, they may
not be to everyone
Response error!!
5
6. Back to our example question: How satisfied are you with your leisure time? Please
rank on a scale of 1-10, with 1 being completely unsatisfied and 10 being
completely satisfied
Some follow up cognitive testing questions might be:
1. Can you tell me in your own words what “leisure” means?
2. What does it mean to you to be “satisfied”?
3. What recall period did you use in your response? Were you thinking about your
leisure time in the past week? The past month?
4. Did you find this question difficult? If so, why?
5. Do you think others would find this question difficult? If so, why?
6
8. Conduct surveys with between 10-15 respondents per language group
Sampling should be done to maximize variance among respondents
At least two rounds of cognitive testing should be conducted
Enumerators need to be appropriately trained in cognitive interviewing
Audio-record the interviews
2 enumerators should be present for each individual interview
There is a large degree of flexibility in designing a cognitive testing that will depend
on the survey and the context of the testing
8
9. What are vignettes?
• Research method where respondents respond to a set of stories describing different
scenarios related to the topic for a hypothetical person/household
• The vignette provides enough context and information for participants to have an
understanding of the scenario being depicted, but needs to be vague in ways that
compel participants to ‘fill in’ detail
• Reveals perceptions and values, as well as social norms in the community
• Allows researchers to get at topics that might otherwise be challenging to ask about
• Can be used as an ice breaker, a way to close the interview, a stand-alone technique
or part of a multi-method approach
9
10. ENUMERATOR: This set of questions is very important. I am going to
give you some reasons why you act as you do in the aspects of
household life I just mentioned. You might have several reasons for
doing what you do and there is no right or wrong answer. Please tell
me how true it would be to say:
[If household does not engage in that particular activity, enter 98 and
proceed to next activity.]
My actions in [ASPECT] are
partly because I will get in
trouble with someone if I
act differently.
[READ OPTIONS: Always
True, Somewhat True, Not
Very True, or Never True]
Regarding [ASPECT] I do
what I do so others don’t
think poorly of me.
[READ OPTIONS: Always
True, Somewhat True, Not
Very True, or Never True]
Regarding [ASPECT] I do
what I do because I
personally think it is the
right thing to do.
[READ OPTIONS: Always
True, Somewhat True, Not
Very True, or Never True]
G5.03 G5.04 G5.05
A Getting inputs for agricultural production
B The types of crops to grow for agricultural production
C Taking crops to the market (or not)
D Livestock raising
G5.03/G5.04/G5.05: Motivation for activity
Never true …………………………………..1
Not very true …………………………………..2
Somewhat true …………………………………..3
Always true …………………………………..4
Household does not engage in activity/Decision not made……………98
10
11. “Now I am going to read you some stories about different farmers and their situations regarding different agricultural
activities. This question format is different from the rest so take your time in answering. For each I will then ask you how
much you are like or not like each of these people. We would like to know if you are completely different from them, similar
to them or somewhere in between. There are no right or wrong answers to these questions.”
STORY QUESTION RESPONSE
A
The types of
crops to grow
for
agricultural
production
G4.A1 “[PERSON’S NAME] can’t grow other
types of crops here for agricultural
production. These are the only things
that grow here.”
To what extent does [PERSON’S
NAME]’s story describe your
situation?
Completely different………………….1
Not very similar…………………………..2
Quite similar……………………………….3
Describes my situation too …………4
Don’t know…………………………………97
G4.A2 “[PERSON’S NAME] is a farmer and
grows – [INSERT LOCAL CROPS]–
because her spouse, or another person
or group in her community tell her she
must raise these crops. She does what
they tell her to do.”
Whatever crops you grow for your
production, are you like [PERSON’S
NAME], doing what you are told by
others to do?
Completely different…………………..1
Not very similar…………………………..2
Quite similar………………………………3
Describes my situation too…………4
Don’t know…………………………..….97
11
12. 1. Culturally and contextually appropriate
2. Focus
• Can make some parts more detailed or direct their attention to it (Braun & Clarke 2013)
• Vignettes should focus on “mundane occurrences” rather than disastrous events (Finch 1987, Hughes 1998)
3. Complexity
• Stay away from overly complex vignettes with too many characters (Braun & Clarke 2013)
• Ensure that the vignette is tapping a single one-dimensional concept (King 2014)
4. Ambiguity
Can intentionally make certain parts vague to explore assumptions (Braun & Clarke 2013)
5. Single vignette vs. staged vignettes
Presenting character or plot development in “stages” (Braun & Clarke 2013)
6. Number
Generally use 5-7 vignettes per concept to be measured (King 2014)
12
13. Question:
1. Anchoring questions: “a technique
designed to ameliorate problems that
occur when different groups of
respondents understand and use ordinal
response categories” (King & Wand 2006)
2. Using “should” versus “would”:
When asking about how a character might
react you may want to get at moral
aspects of the situation or the pragmatic
(Braun & Clarke 2013)
Response:
1. Open-ended response vs. close-ended: Asking
the respondent his/her thoughts on the vignette. Or
giving response options (use an even number of
categories)
2. Response categories relating to the hypothetical
situation vs/ relating to respondent: Responses
relate to how character in situation should or would act
or relate to how respondent should or would act if
he/she were in the same situation (WEAI 2.0)
13
Enumerators need to be well-trained and comfortable with technique
14.
15. Sites: Bangladesh & Uganda
Sample size: Consisted of 120
interviews in Uganda and 70 interviews
in Bangladesh
Questionnaire: A series of roughly 100
questions were developed based off
Johnson et al.’s (2013) paper on
cognitively testing the original WEAI in
Haiti
Photo credit: Chiara Kovarik
15
16. Cognitive testing revealed issues with
the following areas:
Distinction between different concepts
Time frame and recall issues
Abstract terms or concepts
Discrepancies between identifying
something as challenging versus saying
others would find it challenging
Photo credit: Katie Sproule 16
17. Original survey question: “Did you yourself
participate in [ACTIVITY] in the past 12 months (that
is, during the last [one/two] cropping seasons)?”
Cognitive question: “What timeframe did you include
in your response?”
Problem: 35% of respondents in Uganda either could
not come up with the recall period used or referred to
a timeframe other than 12 months
Modified survey question: “Did you yourself
participate in [ACTIVITY] in the past 12 months (that
is, during the last [one/two] cropping seasons), from
[PRESENT MONTH] last year to [PRESENT MONTH]
this year?”
Results of modification: Timeframe recall errors
dropped to just 6% in Uganda Photo credit: Katie Sproule
17
18. Pros from our experience:
Vignettes are fun and new!
Cons from our experience:
Some respondents found it challenging to understand the concept of a hypothetical situation
It was challenging for both enumerators and respondents to grasp what part of the story they
were trying to relate to
Other thoughts:
Ambiguous results
Responses were often much longer and more descriptive than anticipated, even when posed in
a close-ended manner (Bangladesh). This may have been due to not understanding the
question.
Vignettes made an ideal candidate to cognitively test
What we might do differently next time
Vignettes as part of qualitative work?
18
19. Lessons learned and ideas for future
research
• Vignettes and cognitive testing
are not for every questionnaire
• They take extra time, resources,
and enumerator training
• Cognitive testing was valuable in that it allowed us to understand
what is wrong with a question in a very specific way, rather than just
knowing the question is poor and should be changed; it answers the
how it should be changed
• Cognitive testing is not necessarily a stand alone technique; there
were areas where we are unsure what to make of the results (i.e.
effectiveness of vignettes vs traditional autonomy questions)
• While doing multiple iterations of testing may not always be feasible,
doing either a single iteration or a more extended pre-test could be
beneficial to survey designers (e.g. Haiti WEAI cognitive testing)
• It was especially important to cognitively test the WEAI, because it is
administered in 19 countries; similar testing should be considered
with other large multi-country surveys
19
20. Hopkins, D.J., King, G. (2010). Improving Anchoring Vignettes: Designing Surveys to
Correct Interpersonal Incomparability. Public Opinion Quarterly. pp. 1-22.
Johnson, K. (2014). “Cognitive Pretesting of Cross-nationally Comparable Survey
Instruments in a Developing Country Context Seminar.” International Food Policy
Research Institute. Washington, DC. 9 May 2014.
King, G. (2009). Anchoring Vignettes FAQs and Examples.
http://gking.harvard.edu/vign/eg/ [Accessed November 6, 2014].
Wand, J. (2007). Credible Comparisons Using Interpersonally Incomparable Data:
Ranking Self-Evaluations Relative to Anchoring Vignettes or Other Common Survey
Questions. Available at http://wand.standford.edu.
Willis, G. B. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire
Design. Sage, Thousand Oaks, CA.
20
21. Thank you!
Any questions?
Contact Katie Sproule (k.Sproule@cgiar.org)
or Chiara Kovarik (c.kovarik@cgiar.org)
21
22. Bargaining power: “Hope is a cassava farmer in a nearby village. She has her own small plot
that she works on, though her husband owns it. When it comes time to bring her cassava to
market, her husband demands that she give him at least 80 percent of whatever she earns.
What should Hope do?”
Mobility: “Sumi wants to visit her parents, who live in another village 20 kilometers away.
Her husband agrees, but only if he goes with her. How much power does Sumi have to travel
when and where she wants? Response categories: a lot; some; a little; none.”
Motivation for decisions-making: “Toko grows the crops for agricultural production that her
family or community expect. She wants them to approve of her as a good farmer. Are you
completely like, somewhat like, somewhat different or completely different from Toko?”
22
-Point out that cognitive testing and vignettes don’t necessarily go together; they just happen to be two methods used to deal with challenges that arose with the WEAI
After the 2012-2013 baselines, it became obvious that the WEAI needed to undergo some revisions and streamlining
Key indicators that were identified as problematic were: time use, autonomy in decision making, group membership, and miscellaneous questions on asset ownership and production decisions
Decision was made to develop a second version of the WEAI (WEAI 2.0)
Vignettes were included to see if they would be a better way of getting at issues of autonomy in decision making
Cognitive testing was conducted to ensure that the questions were capturing the various dimensions of empowerment, as intended by the research team, and also to ensure that the index remained standardized despite being implemented in various country contexts
Cognitive testing is a qualitative method that is paired with a (quantitative) survey
The purpose of cognitive testing is to systematically identify and analyze sources of response error in surveys, and to use that information to improve the quality and accuracy of survey instruments (Johnson, 2013)
Basically you’re checking to see whether the question is generating the intended information
Cognitive testing can be especially important for new/revised instruments, or those that will be used in multiple country contexts (Johnson, 2013)
-You can go through this process with each survey question you identify to have one or more cognitive breakdown issues. You don’t have to ask this many questions, these are just some examples that will help you to get at whether any comprehension, retrieval, judgment or response issues exist. For the WEAI, while we tested the entire revised version of the instrument, we did have more probing questions for modules that had been identified as problematic during the first round.
-Example page from the WEAI cognitive test
-1st part is observations for enumerators to fill out on difficulties respondent had with the questionnaire module just administered
-2nd part is questions asked to the respondent about the module they just completed
The way we did it used a semi-retrospective technique, so after each module we asked the cognitive questions; the other options would be to ask the cognitive questions immediately after the survey question (we thought that would be disruptive to the interview) or you can wait until the end of the survey to do the cognitive interview, but we were afraid people would forget by that point
Other things of note here, some questions are coded responses, others are left with blank space for the enumerator to write (verbatim) what the respondent said; these comments are helpful when you are trying to figure out how to change a question (why was this question difficult? Word they didn’t understand? Too long? Etc.)
Cognitive testing should be done for between 10-15 respondents per language group
Doing more than 15 interviews leads to diminishing marginal returns
Sampling should be done to maximize variance among respondents
Have young/old, men/women, educated/non-educated; you want to see if there are problems specific to any of the sub-groups within your sample
Ideally, at least two rounds of cognitive testing should be conducted
Between each round, you should make time for revisions of the instrument and clarifications to enumerators
Enumerators need to be appropriately trained in cognitive testing
Generally speaking, the more experienced the enumerator the better, especially with qualitative methods; but with good training any good enumerator can do this
If possible, audio-record the interviews
This is helpful if you need to go back to further understand the nuance of a problem; however, transcribing and analyzing this additional data obviously requires additional resources
Ideally 2 enumerators should be present for each individual interview
1 to ask the questions and the second to take notes and observe the behavior of the individual
There is a large degree of flexibility in designing a cognitive testing that will depend survey and context of the testing
For instance, you can ask standardized or improvised questions; you can ask concurrently after each question, wait until the end of the module or end of the survey; and you can use a thinking aloud technique, asking respondents to reveal their thought process in answering the question, or use a probing technique in which the enumerator guides the respondent to reveal certain information about their cognitive and response process; you can have a completely structured script, as we did, or allow the enumerators to fully or partially improvise (requires more skill – definitely good qual interviewer).
NOTE: we did not develop these vignettes. These were suggested by a colleague at OPHI.
This segways nicely into talking about vignettes, which from our experience with the WEAI, are very important to cognitively test and can help with refinement of this type of questioning
Vignettes can be described as carefully contrived stories about individuals and situations which make reference to important points in the study of perceptions, beliefs and attitudes (Hughes 1998)
Key requirements (KS: I need to understand these better):
Response consistency requires each individual to use the response categories for the assessment question in approximately the same way he/she uses them to evaluate hypothetical questions in the vignettes (King 2009)
One challenge is that it is abstracted from the real situation so it is hard to tell whether the respondent will respond how he/she would actually respond
Vignette equivalence “the level of the variable represented in the vignette is understood by all respondents in the same way” (King 2009; Wand 2007)
Can be open- or close-ended questions
Does not have to have standardized response codes – could be open-ended.
Should have an even number of response code options (if not people will tend to the middle/neutral option) (Alkire)
Enumerators need to be well-trained. Need to be good storytellers and able to explain the alternative question format well so respondents understand.
Make vignettes as culturally and contextually appropriate as possible (i.e. relevant crops, social norms, names of characters, etc.).
There are numerous way to design an anchoring vignette (research by Hopkins and King tests these different methods)
1) self assessment, then hypothetical assessment
2) hypothetical assessment, then self assessment
3) combined, as we did – more efficient because less questions (though research should be done to test the validity against other methods)
Sample size consisted of 120 interviews in Uganda and 70 interviews in Bangaldesh
Split in half for each round, then within each, about 2/3 of the interviews were with women and the other 1/3 with men; women were selected from both dual headed and female headed households and at various age ranges
A series of roughly 100 questions were developed based off Johnson et al.’s (2013) paper on cognitively testing the original WEAI in Haiti
Each cognitive question corresponded with a particular survey question
Used a semi-retrospective standardized probing technique: After each module of the WEAI 2.0, the set of cognitive questions that corresponded with those WEAI questions were asked.
Done so that WEAI questions were as fresh in respondent’s minds as possible when they answered follow-up questions
Testing concluded with a set of cognitive questions on the WEAI survey process in general
We’re going to go through each of these in more depth but broadly speaking, cognitive testing revealed issues with the following areas:
Distinction between different concepts
Time frame and recall issues
Abstract terms or concepts
Discrepancies between identifying something as challenging versus saying others would find it challenging
The original survey question in the productive decision making module asks: “Did you yourself participate in [ACTIVITY-food crop farming, livestock raising, etc] in the past 12 months (that is, during the last [one/two] cropping seasons)?”
When asked how much time respondents included in their response, answers in Uganda ranged anywhere from 3-12 months; 35% of respondents either could not come up with the recall period they used or referred to a timeframe other than 12 months.
Modification of the question to, “Did you yourself participate in [ACTIVITY] in the past 12 months (that is, during the last [one/two] cropping seasons), from [PRESENT MONTH] last year to [PRESENT MONTH] this year?”, resulted in much less recall error with only 6% of respondents in Uganda stating periods of less than one year.
Interestingly, this question was not nearly as challenging in the Bangladesh pre-test.
New autonomy section tested
Problems identified with the old autonomy section included:
A sensitive topic
Hard to get a via standardized survey questions
Many problems implementing in various countries – challenging both for enumerators and for respondents
Feedback from teams on the vignettes was mixed. The Uganda team preferred the original questions at first but then became more comfortable with the vignettes, while the Bangladesh team preferred the vignettes. However, they noted that even with the original autonomy questions they were doing some “storytelling”, so it was not that different of a technique for them (Nice thing about this is that they are standardized stories)
Problems that arose with the vignettes:
Some respondents found it challenging to understand the concept of a hypothetical situation. (i.e. – said they did not know the person in the story)
It was challenging for both enumerators and respondents to grasp what part of the story they were trying to relate to.
Other thoughts:
Responses were often much longer and more descriptive than anticipated. This provided lots of interesting information, but may not be what you are looking for if you are trying to get a quick module.
Cognitive testing was, for the most part, extremely revealing and beneficial to do in the context of the WEAI
It allows you to understand what is wrong with a question in a very specific way, rather than just knowing the question is poor and should be changed; it answers the how it should be changed
Cognitive testing is not necessarily a stand alone technique; there were areas where we are unsure what to make of the results; primarily the autonomy vignettes; analysis of the full data will help us to make this decision
While doing multiple iterations of testing may not always be feasible, doing either a single iteration or a more extended pre-test could be beneficial to survey designers (e.g. Haiti WEAI cognitive testing)
It was especially important to cognitively test the WEAI, because it is administered in 19 countries; similar testing should be considered with other large multi-country surveys