[282 met] 322
Upcoming SlideShare
Loading in...5
×
 

[282 met] 322

on

  • 313 views

 

Statistics

Views

Total Views
313
Views on SlideShare
313
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

[282 met] 322 [282 met] 322 Document Transcript

  • HEALTH POLICY AND PLANNING; 17(3): 322–330 © Oxford University Press 2002How to do (or not to do) . . .Methods for pre-testing and piloting survey questions:illustrations from the KENQOL survey of health-related qualityof lifeA BOWDEN,1 JA FOX-RUSHBY,1 L NYANDIEKA2 AND J WANJAU21Health Policy Unit, London School of Hygiene and Tropical Medicine, UK, and 2Centre for Public Health Research,Kenya Medical Research Institute, Nairobi, Kenya Part of the assertion that any survey researcher can make about the validity of their results needs to contain an analysis of questions and their responses from the respondent’s viewpoint. Claims concerning the valid- ity, reliability and sensitivity of health-related quality of life measures tend to be based on the quantitative approach of psychometrics, which fails to identify when respondents: misinterpret questions; do not recall the information requested; or give answers that present themselves in a better or worse light. The paper presents some approaches to pre-testing and piloting survey questionnaires to check the interpretation of survey questions, using illustrations from the KENQOL project. The paper describes: how the intended referential and connotative meaning of each question was established; the criteria to judge the appropriate- ness of each question; the methods used to make those judgements; and the process of reviewing questions based on findings. The role of piloting is highlighted, and further reading is suggested for readers wishing to develop a model for their own investigation.Key words: survey, pre-testing, piloting, Kenya, health-related quality of lifeIntroduction We were in the process of developing a survey tool to measure the self-perceived and locally defined ‘health’ ofQuestionnaire-based surveys are a very common approach to people in Makueni District (Kenya), and reviewed practicesdata collection used by researchers and the interpretation of in the development and translation of such measures as partfindings may be used to inform policy or public debate. It is of this process. We noticed that claims concerning the valid-not uncommon to find reports in the media saying ‘a recent ity, reliability and sensitivity of health-related quality of lifesurvey showed that X% of people think . . . .’ But do they? (HRQL) measures tended to be based on the quantitativeHow much trust can we place in the findings of surveys? approach of psychometrics. Such quantitative techniques areAnswering questions on a survey requires several thought unlikely to identify when respondents misinterpret ques-processes: respondents have to interpret the question, tions, fail to recall the information requested or give answersretrieve the necessary information and then decide what that present themselves in a better light. Indeed the interpre-answer to give – and all this before the researcher gives their tation of questionnaire items has often been overlookedown interpretation to the responses. (or at least not reported) in the development and trans- lation of HRQL measures (Fox-Rushby and Parker 1995;Part of the claim that any survey researcher can make about Bowden and Fox-Rushby 2000). It is also the case that well-the validity of their results needs to contain an analysis of the known texts in survey research often fail to address issuesquestions from the respondent’s viewpoint. This is a salient of how to pre-test survey questions in sufficient detailstarting point either for those developing their own question- (Moser and Kalton 1992; Fowler 1993; Grosh and Glewwenaire or for those using an existing questionnaire in a new 2000).context (either in a new population or new language). Failingto investigate the interpretation of questionnaire items may The aim of this paper is to present a variety of examples toresult in misinterpretations (by respondents and researchers), help researchers improve the validity,1 reliability2 and sensi-falsified answers, missing responses (to the questionnaire as a tivity3 of their survey instruments prior to undertaking awhole or to particular items), and possibly an offended household survey. The paper first provides a brief context torespondent who chases away the interviewer and encourages our research before proceeding to the range of methods usedothers to refuse interviews. to pre-test and pilot the survey instrument. The discussion
  • How to do (or not to do) . . . 323section focuses on the generalizability of our approach to Pre-testing the survey questionstesting questions for surveys and introduces the reader to a Pre-testing is the main chance for researchers to gauge therange of literature for further reading. meaning attributed to survey questions ‘before it is too late’, i.e. before a substantial investment is made in the wrongContext of research questions or in questions where the researcher cannot be sure about what is being asked. It is therefore imperative thatThe overall aim of our research is to develop a culturally the researchers undertake these interviews, rather thanrelevant generic measure of ‘health’ to measure the impact of assigning the task to field assistants, because they know theinterventions designed to reduce disease and/or improve aim of the research. The pre-testing involved a number ofhealth in Kenya. We have been working amongst the Kamba steps including:of Makueni District in Eastern Kenya and the Maragoli ofVihiga District in Western Kenya. The conceptualization and • establishing the intended referential and connotativeoperationalization of ‘health’ emerged from extensive meaning of each question;periods of participant observation by anthropologists, in • agreeing a set of criteria to judge the appropriateness ofaddition to qualitative research by others (see Fox-Rushby et survey questions;al. 1997; Fox-Rushby, for the KENQOL Group, 2000). The • selecting the methods for judging appropriateness anddefinition of health adopted focused on the subjective per- undertaking research;ceptions of the positive and negative aspects of health that • reviewing questions for inclusion, revising (the question orcomprised ‘contentment’, ‘cleanliness’, corporeal capacity’, intended meaning) or exclusion.‘co-operation’ and ‘completeness’ of an intending individualwithin the intentional worlds of Makueni and Vihiga (Fox-Rushby, for the KENQOL Group, 2000). Step 1: Establishing the intended referential and connotative meaning of each questionThe KENQOL survey questionnaire was developed to It is important to ascertain whether respondents are inter-access those aspects of health that a member of the com- preting the question as intended to ensure appropriatemunity could respond to in public4 to an interviewer who was interpretations are placed on the data. Therefore we pre-not necessarily known to them personally. This paper focuses pared descriptions of the intended referential and connota-on our work amongst the Kamba and reflects the develop- tive meaning for each of the survey questions drawn up.ment of the household survey in the Kamba language.5 The There were several advantages to this: it served as an account-aim of this part of our research was to investigate: how able approach to assessing content validity; it facilitatedrespondents understood the KENQOL survey questions; clarification of our own intentions in asking questions; itthe types of information retrieved and over what time ensured we assessed the same hypotheses as there was lessperiod; the types of socially acceptable answers that might variation in our own interpretation of the meaning of thebe given and why; in addition to evaluating whether they original questions (either over time or between researchers);performed the task the way the researcher expected. We and it was expected to be helpful in the future assessment oftested a total of 150 potential health measurement survey the conceptual equivalence of the Kamba language versionquestions. with other language versions (see Herdman et al. 1997, 1998). Box 1 provides an example of the type of description weThe field site was located in Makueni District in Eastern attached to a question.Kenya. It is approximately halfway between Nairobi andMombasa, just north of the main trunk road. The Kambapopulation (part of the Central Bantu group) is the majorityethnic group in this region (99%) and traditionally they arepastoralists. Currently 50% of the population are aged0–14 years but infant mortality rates are high, at 96 per 1000live births in 1995 (Kenya Government 2000). Rainfall is Box 1. Sample meaning for a KENQOL survey questionscarce in the area, with an annual precipitation of less than500 mm and this, combined with high daytime temperatures,leads to a hot, dry environment. Agriculture is unreliable and Do you feel that there are some people who could sabo-food deficits are often experienced. Health facilities are rela- tage the progress of people in your neighbourhood at thetively scarce, although they can be reached by matatu from moment?the main Nairobi/Mombasa highway; the walk to the road cantake up to 2 hours. We want to find out an individual’s current views about whether they feel that their ability to improve them-A day’s bus journey is required to reach hospital facilities in selves and their lives (collectively as a community)Nairobi. The major diseases are malaria, respiratory tract may be adversely affected by people within the com-infections, diarrhoeal diseases, skin ulcers, intestinal worms munity in some way. It is possible this may include aand urinary tract infections (Kenya Government 2000). range of ideas such as violence and witchcraft and weIncomes are low compared with many other areas in Kenya, leave it up to the respondent to decide if they are beingwith many people living below the poverty line and involved threatened.in a daily struggle for survival and development.
  • 324 How to do (or not to do) . . .Table 1. Criteria selected for judging appropriateness of survey questions and investigative method employed1 Criteria 2 Method Expanded Targeted Group Expert interviews methods methods reviews 1 No negative questions, and therefore no double negative answers √ √ 2 There are not two questions in one √ √ 3 Level of language is not too high/old fashioned/unusual in any sense √ √ 4 The question is simple and grammatically correct √ 5 The question is free from jargon √ 6 Singular and plural ‘you’ is clear √ √ √ 7 The same idea is not contained in another question √ √ 8 The question is sensitive to measure change over time √ √ 9 Captures current views √ √10 Reflects local issues √ √11 Meaning and interpretation of question is clear √ √ √12 The question makes sense to everyone √ √13 A ‘yes’ is clear and unambiguous in meaning √ √14 A ‘no’ is clear and unambiguous in meaning √ √15 Time period is clear √ √ √16 Conception(s) of health it represents is specified √ √17 How the question relates to: time of year; head of household; individual/ homestead/wider family/community √ √ √18 Understanding of how it relates to severity of HRQL √ √Step 2: Criteria to judge the appropriateness of survey organize the expanded interviews we divided the set of surveyquestions questions into groups of between three and five questions. In each interview6 the respondent was first asked the KENQOLHaving developed the survey question and agreed on the survey question and asked to respond either ‘yes’, ‘no’ orintended meaning, we developed a series of criteria for evalu- ‘don’t know’ as they chose. The probes used were divided intoating the appropriateness of the survey questions. This facili- three types and used in separate interviews:tated an agreed and standardized way of making decisionsabout which questions to include and enabled different (1) Comprehension of ‘key phrases’ within the question. Weresearchers to bring their own judgements together in a asked respondents to talk about the kinds of things theycomparable way. Column 1 of Table 1 outlines the range of thought about when answering the question. Examples ofcriteria we adopted. It can be seen that the issues focus largely key phrases included the time frame the respondenton the style of language, attributed meaning, clarity of the considered and who respondents included when thequestion and consistency of interpretations. word ‘friends’ or ‘neighbours’ was used (see Box 2 for an example). (2) Applicability of the questions: to people of different ages,Step 3: Methods for judging appropriateness of survey gender and socioeconomic groups (see Box 3 for anquestionsThere are a wide variety of techniques available toresearchers wanting to test survey questions. We adoptedfour principal types (described in Forsyth and Lessler 1991): Box 2. Comprehension of key phrases in the questionexpanded interview techniques, targeted methods, groupmethods and expert evaluation. Others approaches areaddressed in the discussion. Table 1 (column 2) indicates Do you feel safe at the moment?which technique(s) addresses which type of issue and thissection outlines each method. Of these techniques, the The question intended to access concerns about per-expanded interviews took most of our time. sonal safety and safety of belongings. The probe ques- tions asked respondents to give examples for why someone (a) will feel safe and (b) may not feel safe. ItExpanded interviews emerged that the question not only addressed notionsExpanded interviews begin by one or more of the intended of personal safety and safety of belongings, but alsosurvey questions being asked and answered by the respon- having enough food, being properly taken care of anddent, followed by a set of probe questions to investigate the having money for school fees.thought processes involved in answering the question(s). To View slide
  • How to do (or not to do) . . . 325Box 3. Applicability of questions for respondents of different ages We asked respondents who in the family can and cannot be asked the question, ‘At the moment are you always worry- ing about getting food?’ The text below demonstrates some of our findings with respect to this question. Interview with an elderly man: Interviewer: Can children worry about getting food? Respondent: No, since it is their father’s fault they are fed. Interviewer: Could a young unmarried man answer this question? Respondent: A young unmarried man isn’t responsible for that. You will find that in most cases he comes at night and gets in his room and sleeps. Maybe he has passed to his friends’ home and had something to eat. The old mamas can’t worry since they are just looked after by their children they brought up. Interviewer: Is there anyone who might say no to this question? Respondent: Old and young people. example); by level of sensitivity of the question; and you ‘clean’? – meaning in a hygienic and moral sense), social desirability of responses (see Box 4). but further probing highlights some potential problems. Box 4 has a particularly interesting response because it (3) The meaning of a ‘yes’ or ‘no’ response needed to be clear also shows how sensitive the questioning on sensitivity and as intended (see Box 5). needs to be. It might at first appear that there are no prob- lems with the question ‘Ivindani yii nutonya kwiyikalya The respondents for the expanded interviews were volun- wi mutheu undu utonya kwenda?’ (At the moment, are teers willing to expend more time than the data collectionBox 4. Addressing the sensitivity of the question We asked respondents directly about the sensitivity of questions. We needed to know whether a question was so sensi- tive that people might purposefully provide the wrong answer. Alternatively they may provide the right answer, but this makes the respondent or their family uncomfortable in some way. The transcript below is a translation from an inter- view that addressed sensitivity. Interviewer: At the moment, are you ‘clean’ [hygienically and morally]? Respondent: Yes Interviewer: When we ask this question in the interview, we want to be sure that we are not embarrassing the person we are asking. Do you think this question might cause embarrassment if it is asked in an interview? Respondent: The question is good and cannot cause embarrassment in an interview. Interviewer: Do you think people will feel they can answer this question truthfully in an interview? Respondent: Not all people can answer truthfully. This is because if one feels unclean or has done wrong, he wouldn’t want to be known to have wronged. Interviewer: If someone answers ‘no’ to this question, how will it reflect on the family of that person? Respondent: One who answers no would be thought to be unclean, but not the whole family. View slide
  • 326 How to do (or not to do) . . .Box 5. Testing the meaning of a ‘yes’ or ‘no’ response By undertaking a relatively simple interview we could quickly establish whether a ‘yes’ or ‘no’ response could be inter- preted clearly and as intended. This particular question was dropped because it became clear that we could not be sure that a ‘no’ meant worse ‘health’. Interviewer: Do you get enough information about current events in Kenya? Respondent: No. Interviewer: If someone answers ‘no’ to this question, is he/she in a better or worse position than someone who answers ‘yes? Respondent: The one who answers ‘yes’ is in a better position. Interviewer: Why is this? Respondent: Information about current events in Kenya can assist people in this area by helping them develop, such as knowing different methods of farming, knowing about disease and outbreaks and ways of combating them, and also improvement of educational facilities. Interviewer: What other reasons are there that someone could answer ‘yes’? Respondent: Those who own radios and are literate. People can get access to this information from radios, newspapers, meetings and friends. Interviewer: What other reasons are there that someone could answer ‘no’? Respondent: Those who don’t have radios and don’t read newspapers. Everyone doesn’t want to have access to infor- mation about Kenya. Those who do are mostly interested in politics and current affairs. Those who aren’t are the old, illiterate and those not interested in politics.itself in helping us learn about the questions. They were poor; and those with harmonious relationships comparedselected purposively to include equal proportions of males with those in disputes.and females, as well as being roughly representative of the • As the questionnaire was intended to be able to measuredifferent generations in the community. In total each ques- change we asked eight people for one of two types of hypo-tion was the focus of between four and eight interviews with thetical considerations of the questionnaire. For example,a range of people from the local population. members of a youth-based water and agriculture project were asked to imagine how they might have answered the questions before and after the project, and other keyTargeted methods informants were asked how they might have completedTargeted methods focus on a particular component of the the questionnaire during and after malaria. The task wasresponse process, to gain a detailed insight into the respon- followed up with a discussion of the answers most likely todent’s comprehension and preparation of a response and change and why.to evaluate whether they performed the task the way the • To address which conception of health people felt theresearcher intended. We used three types of targeted question represented, as well as helping further investi-methods: gations into the meaning and interpretation of the ques- tion, respondents were asked to sort the Kikamba• People with known (and opposite) characteristics of inter- questions into groups, either under the Kikamba words est were asked to complete specific questions to assess how used to translate the English word ‘health’7 that each ques- likely it was that the questionnaire would be interpreted in tion was most closely related to, or to devise their own the same way by people with particular characteristics. This categories and group them. Each question was placed on a was important in being able to interpret any future differ- separate card for ease of sorting. The task was followed up ences/similarities across groups. Therefore we contacted with an interview about the meanings of the Kikamba (through the help of our field assistants): people with and words used to translate the English word ‘health’ and without sickness/illness; the relatively wealthy and very reasons for groupings.
  • How to do (or not to do) . . . 327Box 6. Group discussion on ‘people in your homestead’ Many of the KENQOL questions ask respondents about people in their homestead,i for example, • Do people in your homestead listen to your advice/suggestions at the moment? • When you are sick, do people in your homestead help you in getting treatment? • At the moment is it easy for people in your homestead to find transport to go for treatment when they are sick? To check who people thought about when answering these questions, we provided several groups of respondents with an example family (see below) and initiated a discussion about who different people in that family would think about when they were asked specific questions. Example family: Headed by a married male (mzee) and female (mama), who had three sons. The first son is married to one wife with a child and they live just outside the compound. The second son is married with two wives (polygy- nous marriage) and both wives have children. The third son lives away in Nairobi. The discussion revealed that people would make reference to different people in their response. In the example above, the mzee would consider seven adults, whereas the first-born son thinks of his wife and his wife thinks only of her husband. To complicate matters further still, we discovered that a person could alter the group of people they thought about depending on the question that was being asked, even if each question used identical phrases for ‘people in your homestead’. We noted all the instances in which this occurred.i The phrase musyi kwaku/kwenyu for homestead was used after finding that the word family was interpreted verydifferently across people.Group methods questions and issues 1–6 in Table 1. The local field assistants also reviewed the written Kikamba questions and gave theirGroup methods also focus on a particular component of the views on issues 1–6 as well as 13–17. They were very helpfulresponse process, but a group of people are addressed, rather in their criticisms and suggestions, and once we had been ablethan an individual, to encourage greater discussion around to take down their advice, it was also easier for them to inter-the issues involved. There were a number of key issues within view others (as opposed to discussing their own ideas withthe survey questions that were repeated often, such as time interviewees).frames, the use of singular or plural ‘you’ and terms such as‘neighbours’, ‘friends’ and ‘family’. Individual interviewswere very helpful in understanding interpretations but did Step 4: Reviewing questions for inclusion, revising thenot give us the opportunity to see opposing suggestions question or intended meaning, or dropping questionsdebated. We therefore held group discussions on focused The field transcripts and results from each research methodissues. The group discussions lasted 1–2 hours, and were led were collated and summarized by the authors for each surveyby two moderators. Each interview was recorded and later question. Reviews of questions were supported by a dis-transcribed. Each group involved people with similar ages cussion between the authors and field assistants. Followingand gender, and the sum of groups captured the range of ages these discussions, decisions were made concerning the actionand gender in the area. Box 6 gives an example of our required for each question including:approach to finding out how people interpreted the word‘homestead’. • acceptance of the original question and meaning; • acceptance of the original question, with a change to the meaning;Expert evaluation • changing the question (slightly) but keeping the meaningExpert evaluation involved no interaction with potential the same;survey respondents, but rather those who could offer particu- • dropping the question;lar guidance on specific issues. The ‘experts’ included two • writing a new question (and new intended meaning).groups of people: external advisors including Kikamba Bibletranslators and Kamba university lecturers on sociology; and Any changes were tested through additional expandedthe field assistants living locally in Makueni who had assisted interviews and any new questions also went through variousthe KENQOL research group for several years. The external translation checks. Prior to accepting any new or changedlanguage advisors reviewed the written survey questions and questions all data on the question were once again collatedfocused particularly on their interpretation of the meaning of and reviewed prior to being taken forward to piloting.
  • 328 How to do (or not to do) . . .Piloting into other languages. There has been some use of qualitative techniques to assess the content validity of original (e.g.Moser and Kalton (1992) refer to piloting as the ‘dress Saxena et al. 1998), but more usually the adapted versions of,rehearsal’. Questions are placed together as it is expected that generic HRQL instruments (e.g. Chwalow et al. 1992; Lam etthey will appear on the final questionnaire and the dynamics of al. 1994), as well as the applicability of generic instruments tothe survey as a whole are investigated. The KENQOL group specific disease groups (e.g. Tazaki et al. 1998). These investi-undertook piloting in two stages. The first focused on ques- gations tend to be relatively unstructured, and can betions from each sub-section (n = 6) of the survey. The second described as cursory investigations. In general there isinvolved straight runs through the entire survey. One of the too much emphasis placed on quantitative assessments ofauthors was present at each of these interviews, and a field reliability, validity and sensitivity. The quantitative assess-assistant conducted the interview in Kikamba. At both stages ment is a rather closed approach, that ignores the respon-of piloting, the interviewer noted down the response to each dent’s own interpretation of questions.question (yes, no, don’t know, no response). They also notedany other information the respondent gave, such as asking for We therefore adopted a qualitative approach to the assess-clarification of the question, or the respondent providing ment of validity, reliability and sensitivity. This made both theadditional information after responding ‘yes’, ‘no’ or ‘don’t researcher’s and the respondent’s interpretation of questionsknow’. In addition, the authors noted any body language, more open and accountable, and ensured that the final surveypauses before a response, emotional reactions to questions etc. questions were more reliable and valid and able to detectAt the end of the pilot survey interviews, each respondent was change as revisions were made. Earlier we defined a reliableasked for their opinion about the questions, including: measure as one which yields the same results in repeated trials (Carmine and Zeller 1979). A question may be un-• what they thought about the questions in general; reliable due to bad wording, causing a person to understand• whether any of the questions seemed to be strange or the question differently on different occasions (De Vaus unusual; 1995). This paper has described procedures for assessing the• their opinion on the order of questions; reliability of questions during the pre-testing stage, for• how appropriate the response categories were; example identifying questions that cannot elicit a double• whether the correct procedures are used to address: negative for a response, do not contain high level or old fash- (a) married and unmarried people, ioned language, and are free from jargon. A valid measure is (b) old and young people; one which measures what is intended (Carmine and Zeller• whether any questions should not be asked in the survey; 1979), and methods described in this paper which helped to• to point out any questions they do not want to answer, or address this included ensuring that the questions capture think they should not be asked; current views, reflect local issues and the conception of health• any problems they thought we might encounter asking each question represents is clear. Similarly, during pre-testing people the questions: the sensitivity to questions over time was investigated. De (a) alone, Vaus (1995) explains that instead of this procedure some (b) with others listening (e.g. young people were asked people include a large number of indicators and at a later whether they might have inhibitions answering ques- stage eliminate those that do not seem to be working. tions in front of parents); However, he explains: ‘this seems to be the wrong way of• whether any questions appeared to be asking the same doing things since we will end up by defining the concept in thing; terms of the indicators which ‘worked’. If this is done then the• their opinion on what changes we could make to improve indicators may not represent the concepts of the theory we this survey in terms of: set out to test and as such the research can end up having (a) introduction, little relevance to the original research question’ (De Vaus (b) questions; 1995: 54).• which questions might be difficult for someone to talk about in front of an interviewer who comes from the same However, we recognize that the process was labour intensive area as the respondent; and took a considerable amount of time (around 6 months of• whether it matters who interviews people; researcher and 12 months of field assistant’s time). In our case• who would be a good interviewer in terms of age, gender we judged this to be an important part of the process and and any other characteristics. outcome of our research. First, we are intending that this measure becomes used across many different diseases asBased on feedback during piloting, several additional an outcome measure used to judge the effectiveness andrevisions were made to the KENQOL questions. Each efficiency of health-improving interventions. It is thereforechange was followed up by further piloting (and pre-testing important to us that the people whose health we are trying towhere necessary) until the final version of the KENQOL measure (and improve) are integrated into this process, andsurvey was agreed. it is often the case that generic HRQL instruments are devel- oped and used for years following their development. Sec- ondly, it provided valuable comparative data for explainingDiscussion any similarities and dissonances in measured health in VihigaWe have been concerned about the procedures used in the district, Kenya. Related to this is the possibility that othersdevelopment of HRQL measures as well as their translations would want to translate the instrument for use elsewhere, and
  • How to do (or not to do) . . . 329we wanted to improve the basis from which judgements are required information through household survey, readersmade about the equivalence of future translations. should consider some of the literature from demographic studies; some examples include Caldwell and Igun (1997),There are other techniques that are available to researchers Ewbank (1981), Bledsoe and Pison (1994). We found cogni-to test their survey questions that we did not use. The tive research assessment a particularly inspiring approach asKENQOL researchers adopted some cognitive assessment it captures approaches given to a variety of psychologicalsurvey methodologies, but there are others that may be more processes involved in information retrieval. Some of thisappropriate for other research groups (Jobe and Mingay literature has already been cited above, but readers might1989): also like to consider Lessler and Tourangeu (1989) and the proceedings from a seminar on the Cognitive Aspects of• Concurrent think-aloud interviews: respondents think Survey Methodology (Sirken et al. 1999). Researchers in the aloud when answering questions, and responses are field of marketing have also been considering these issues for probed. some time, consider for example Gendall and Hoek (1990)• Retrospective think-aloud interviews: respondents answer and Kumar et al. (1999). all question first and then are asked how they arrived at their answers.• Confidence ratings: respondents relate the degree of confi- Conclusions/recommendations dence they have in the accuracy of their answers. The investigations described in this paper should not be seen• Paraphrasing: respondents repeat the questions in their as an unusual step in the development and testing of survey own words. questionnaires – particularly those that will be used repeat-• Response latency: measurement of the elapsed time edly. The validity and reliability (and possibly sensitivity) of between the questions and respondent’s answers. any survey is important to establish prior to accepting results and conclusions. Each of these ideas can begin to be explored,The KENQOL group also adopted some methods recognized and improved, through the use of qualitative methods in pre-as participatory or rapid rural appraisal (PRA and RRA) testing and piloting questionnaires. Indeed, evidence that aapproaches, for example the use of anthropologists and local survey tool has ‘content validity’ is required, rather thanleaders as key informants (Chambers 1983). Other useful simply relying on unsubstantiated assertions by researchers,RRA and PRA approaches might include a well-being (or and only once this has been demonstrated is it appropriate toHRQL) ranking exercise of local households with local adopt quantitative assessment methods for further analysis.participants, and linking this to likely responses to questions Finally, the knowledge gained about the interpretation ofand analysis of difference based on responses to questions survey questions will also help in judging the equivalence(see Chambers 1997). of any future translations of the survey questionnaire.Whilst the approach we have taken to the qualitative assess-ment of meaning attached to the KENQOL survey questions Endnotesis unique amongst generic HRQL research, many of the ideas 1 Validity determines that an instrument is measuring what waswere drawn from a variety of sources, including survey intended. It concerns the crucial relationship between concept andresearch (e.g. Moser and Kalton 1992; Fowler 1993), trans- indicator. The objective is to ensure that the indicator represents thelation studies (e.g. Larson 1984; Barnwell 1992), the cognitive intended (and only the intended) concept (Carmines and Zellersciences (e.g. Jobe 1990; Forsyth and Lessler 1991) as well as 1979). 2 Reliability concerns the extent to which an experiment, testthe health measurement literature (e.g. Guillemin et al. 1993; and any measuring procedure yields the same results on repeatedHerdman et al. 1997, 1998). De Vaus (1995) is particularly trials (Carmines and Zeller 1979).useful because of the links made between investigating ques- 3 The ability of a measure to detect change.tionnaire interpretation and the reliability and validity of a 4 Because most household interviews happen outside the homequestionnaire. This gives an indication of the applicability of with many people around. To ask to interview people privately in aour approach to the design of survey questions more gener- household survey tends to arouse a lot of suspicion. 5 Our household survey eventually consisted of 112 healthally. As Tourangeau (1984) states ‘[in] the respondent’s task measurement questions and 50 socioeconomic and demographic. . . there is considerable room for error. Respondents may questions. We achieved a total of 550 interviews – a response rate ofmisunderstand the question or the response categories; they 81.3% and a mean item response rate of 98.1% (SD 3.6).may forget or misremember the crucial information; they may 6 All interviews were undertaken in Kikamba, the local lan-misjudge the information they do recall; and they may mis- guage of the Kamba population we were working amongst. Thereport their answer’ (Tourangeau 1984: 73–74). Kikamba versions of the KENQOL questions were used for all survey testing, although the English versions are reported here. The pre-test interviews were written up in full in English to facilitateFinally, to help researchers give greater attention to investi- participation by the non-Kikamba speaking researchers.gating whether respondents interpret questions in ways 7 wailu, wianie, utheu, useo, wiwa nesa, uima.researchers expect (or hope), we give a brief introduction toa variety of different types of source texts. Most socialresearch texts address questionnaire design, but few provide Referencesdetail on investigations into question meanings. One text that Barnwell K. 1992. Bible Translation: An introductory course in trans-is useful in this context is Oppenheim (2000). For an excellent lation principles (3rd edition). Dallas, TX: Summer Institute ofinsight into the many pitfalls associated with eliciting the Linguistics International.
  • 330 How to do (or not to do) . . .Bledsoe C, Pison G. 1994. Introduction. In: Bledsoe C, Pison G (eds). Lessler JT, Tourangeau R. 1989. Vital health statistics. Series 6.1. Nuptiality in Sub-Saharan Africa: Contemporary anthropo- Washington, DC: National Center for Health Statistics. logical and demographic perspectives. Oxford: Clarendon Press, Moser CA, Kalton G. 1992. Survey methods in social investigation pp. 1–22. (2nd edition). Aldershot, UK: Gower.Bowden A, Fox-Rushby J. 2000. A critical review of the adaptation Oppenheim AN. 2000. Questionnaire design, interviewing and and use of generic HRQL measures amongst populations attitude measurement. Buckingham, UK: Open University outside of North America, western/southern/northern Europe, Press. Australia and New Zealand. Quality of Life Research 9: 313. Saxena S, Chandiramani K, Bhargava R. 1998. WHOQOL-Hindi: ACaldwell JC, Igun AA. 1971. An experiment with census-type age questionnaire for assessing the quality of life in health care set- enumeration in Nigeria. Population Studies 25: 287–302. tings in India. National Medical Journal of India 11: 160–5.Carmines EG, Zeller RA. 1979. Reliability and validity assessment. Sirken M, Jabine T, Willis G, Martin E, Tucker C. 1999. A New London: Sage Publications. Agenda for Interdisciplinary Survey Research Methods: Pro-Chambers R. 1983. Rural development: putting the last first. Harlow, ceedings of the CASM II Seminar. Hyattsville, MD: National UK: Longman. Center for Health Statistics. [http://www.cdc.gov/nchs/data/Chambers R. 1997. Whose reality counts? Putting the first last. casm2pro.pdf, 2nd August 2001] London: Intermediate Technology. Tazaki M, Nakane Y, Endo T et al. 1998. Results of a qualitative andChwalow AJ, Lurie A, Bean K et al. 1992. A French version of the field study using the WHOQOL instrument for cancer patients. Sickness Impact Profile (SIP): stages in the cross-cultural vali- Japanese Journal of Clinical Oncology 28: 134–41. dation of a generic quality of life scale. Fundamental Clinical Tourangeau R. 1984. Cognitive sciences and survey methods. In: Pharmacology 6: 319–26. Jabine TB (ed). Cognitive aspects of survey methodology: build-De Vaus. 1995. Surveys in social research (4th edition). London: ing a bridge between disciplines. Washington, DC: National Routledge. Academy Press, pp. 73–100.Ewbank DC. 1981. Age misreporting and age-selective undernumer- ation: sources, patterns, and consequences for demographic analysis. Committee on Population and Demography. Report Acknowledgements No. 4. Washington, DC: National Academy Press.Forsyth BH, Lessler JT. 1991. Cognitive laboratory methods: a tax- We thank the following organizations for the funding of this survey: onomy. In: Biemer PP, Groves SM, Lyberg LE, Mathiowetz UNDP/World Bank/WHO Special Programme for Research and NA, Sudan S (eds). Measurement errors in surveys. New York: Training in Tropical Disease, the UK Economic and Social Research John Wiley, pp. 393–418. Council (Ref. H52427005994) and the Health Economics andFowler FJ. 1993. Survey research methods (2nd edition). London: Financing Programme (funded by the Department for International Sage Publications. Development). We also thank all our field assistants, especiallyFox-Rushby J, Parker M. 1995. Culture and the measurement of Justus Mutunga, Jeremiah Nyamai and Alex Muvea, for their help, health related quality of life. European Review of Applied Psy- guidance and enthusiasm. Finally we thank all the people who chology 45: 257–63. participated in the preparation for this survey and made us welcomeFox-Rushby J, Johnson K, Mwanzo I et al. 1997. Creating an instru- in their homes. ment to assess lay perceptions of HRQL: options and impli- cations. Quality of Life Research 6: 633.Fox-Rushby J, for the KENQOL Group. 2000. Operationalising con- Biographies ceptions of ‘health’ amongst the Wakamba and Maragoli of Annabel Bowden, Ph.D. (Health Geography), was a Research Fellow Kenya: the basis of the KENQOL instrument. Quality of Life at the London School of Hygiene and Tropical Medicine (LSHTM) Research 9: 316. at the time of this study. She was involved with the KENQOL projectGendall P, Hoek J. 1990. A question of wording. Marketing Bulletin for 5 years and was based in Kenya for 1 year during the household 1: 25–36. survey. She also has a wider interest in the applicability ofGrosh M, Glewwe P. 2000. Designing household survey question- adapted/translated versions of generic health-related quality of life naires for developing countries: lessons from 15 years of the measures. She is now based with the Lewin Group (UK). Living Standard Measurement Survey (Volume 1). Washington, DC: World Bank. Julia Fox-Rushby, Ph.D. (Economics), is a Senior Lecturer at theGuillemin F, Bombardier C, Beaton D. 1993. Cross-cultural adap- LSHTM. She has written numerous academic papers on the cost- tation of health-related quality of life measures: literature effectiveness of health interventions across the world, specializing review and proposed guidelines. Journal of Clinical Epidemiol- particularly in maternal and child health, malaria and, more recently, ogy 46: 1417–32. vaccine preventable disease. She has also been involved over the pastHerdman M, Fox-Rushby JA, Badia X. 1997. ‘Equivalence’ and the 15 years in developing a number of non-disease specific measures of translation and adaptation of health-related quality of life ques- health-related quality of life as a member of the EuroQol group, tionnaires. Quality of Life Research 6: 237–47. advisor to the WHOQOL group and as principal investigator of theHerdman M, Fox-Rushby JA, Badia X. 1998. A model of equival- KENQOL group. ence in the cultural adaptation of HRQL measures: a univer- salist approach. Quality of Life Research 7: 323–35.Jobe JB. 1990. Research on questionnaire design: perspectives from Lilian Nyandieka, MA (Sociology), is a Research Officer at the other disciplines. American Journal of Epidemiology 132: 824. Center for Public Health Research, a department of the KenyaJobe JB, Mingay DJ. 1989. Cognitive research improves question- Medical Research Institute (PO Box 20752, Nairobi, Kenya). She has naires. American Journal Public Health 79: 1053–5. been involved in the KENQOL project for 3 years and is alsoKenya Government. 2000. Makueni District Development Report researching social and economic aspects of malaria. 1997–2001. Nairobi: Office of the Vice President and Ministry of Planning and National Development. John Wanjau, BA (Anthropology), is an Assistant Research OfficerKumar V, Aaker DA, Day GS. 1999. Essentials of marketing at the Center for Public Health Research, Kenya Medical Research research. Chichester, UK: John Wiley. Institute. He has been involved in the KENQOL project for 3 years.Lam CL, Van Weel C, Lauder IJ. 1994. Can the Dartmouth COOP/ WONCA charts be used to assess the functional status of Correspondence: Julia Fox-Rushby Ph.D., Health Policy Unit, Chinese patients? Family Practice 11: 85–94. London School of Hygiene and Tropical Medicine, Keppel Street,Larson ML. 1984. Meaning based translation: a guide to cross-lan- London WC1E 7HT, UK. Tel: +44 (0) 20 7927 2267, Fax: +44 (0) 20 guage equivalence. Lanham, MD: University Press of America 7637 5391, Email: Julia.Fox-Rushby@lshtm.ac.uk