Designing Questionnaire 
and Survey Research (updated Jan 2011)


Published on

Designing Questionnaire 
and Survey Research. A series of six presentation, introduce scientific research in the areas of cross-cultural, using quantitative approach.

Published in: Education, Technology, Business
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Designing Questionnaire 
and Survey Research (updated Jan 2011)

  1. 1. Hangzhou, January 2011 Prof. Dr. Hora Tjitra, Zhejiang University Designing  Questionnaire   and  Survey  Research Quantitative Research Methodologies (4/6)
  2. 2. Designing  Ques*onnaire  and  Survey  Research 2 Designing Questionnaire and Survey Research Survey  research:  a  specific  type  of  field  study  that  involves  the  collec3on  of  data  from   a  sample  of  elements  drawn  from  a  well-­‐defined  popula3on  through  the  use  of  a  ques3onnaire. The reasons to conduct survey research… • Study  a  representa-ve  sample  through  field   research  is  rela-vely  easy  and  prac-cal. • Research  can  confidently  generalize  the  finding   from  selected  sample  to  en-re  popula-on. • It  provides  ideal  condi-ons  for  explora-on   of  process  ×  individual  difference  interac-on.
  3. 3. Designing  Ques*onnaire  and  Survey  Research 3 14  years  in  Germany 7  years  in  China Born  and  grew  up   in  Indonesia Prof.Dr.Hora Tjitra - Cross-cultural and Business Psychology Dipl.-Psych.,Technical University of Braunschweig Organizational Psychology and Human Resource Management Dr.Phil.,University of Regensburg Intercultural Psychology and Strategic Management Executive Education,INSEAD HR Management in Asia
  4. 4. Designing  Ques*onnaire  and  Survey  Research Outline 1 Doing Survey Research 4 2 Designing Questionnaire 13 3 Sampling Design in the Survey Research 19 4
  5. 5. Doing  Survey  Research
  6. 6. Designing  Ques*onnaire  and  Survey  Research 6 Types of Surveys …the  broad  area  of  survey  research  encompasses  any  measurement   procedures  that  involve  asking  ques3ons  of  respondents.   Ques%onnaires:   Ques-onnaires  are  usually  paper-­‐and-­‐pencil   instruments  that  the  respondent  completes.   Interviews  (constructed):   Interviews  are  completed  by  the  interviewer  based   on  the  respondent  says.  
  7. 7. Designing  Ques*onnaire  and  Survey  Research 7 The Issues in Survey Research Question Issues ✓ What types of questions can be asked? ✓ How complex will the questions be? ✓ Will screening questions be needed? ✓ Can question sequence be controlled? ✓ Will lengthy questions be asked? ✓ Will long response scales be used? Population Issues ✓ Can the population be enumerated? ✓ Is the population literate? ✓ Are there language issues? ✓ Will the population cooperate? ✓ What are the geographic restrictions? Bias Issues ✓ Can social desirability be avoided? ✓ Can interviewer distortion and subversion be controlled? ✓ Can false respondents be avoided? Sampling Issues ✓ What data is available? ✓ Can respondents be found? ✓ Who is the respondent? ✓ Can all members of population be sampled? ✓ Are response rates likely to be a problem? Content Issues ✓ Can the respondents be expected to know about the issue? ✓ Will respondent need to consult records? Administrative Issues ✓ Costs & Facilities ✓ Time & Personnel
  8. 8. Designing  Ques*onnaire  and  Survey  Research 8 Three Types of Measurement Scaling  is  the  assignment  of  objects  to  numbers  according  to  a  rule.   Likert or "Summative" scaling there are a variety possible response scales (1-to-7, 1-to-9, 0-to-4). Guttman or “Cumulative” scaling the answering item is “Yes” or “No” Thurstone or Equal- appearing interval scaling rate each statement on a 1-to-11 scale in terms, then choose “yes” or “No”
  9. 9. Designing  Ques*onnaire  and  Survey  Research 9 The Basic Steps in Developing a Likert or "Summative" Scale. Defining the Focus: define  what  it  is  you  are  trying  to   measure   Administering the Scale: reversal  items     1=  strongly  disagree   2=  disagree   3=  undecided   4=  agree   5=  strongly  agree   Generating the Items: create  the  set  of  poten-al  scale  items.   These  should  be  items  that  can  be  rated  on  a  1-­‐to-­‐5  or  1-­‐to-­‐7   Disagree-­‐Agree  response  scale.   Rating the Items:  have  a  group  of  judges  rate  the  items.  Usually   you  would  use  a  1-­‐to-­‐5  ra-ng  scale  where:1=  strongly  unfavorable  to   the  concept  ;  5=  strongly  favorable  to  the  concept   Selecting the Items.  compute  the  intercorrela-ons  between  all   pairs  of  items,  based  on  the  ra-ngs  of  the  judges.   •Throw  out  any  items  that  have  a  low  correla<on  with  the  total   (summed)  score  across  all  items   •  For  each  item,  get  the  average  ra<ng  for  the  top  quarter  of   judges  and  the  boFom  quarter.  Then,  do  a  t-­‐test  of  the   differences  between  the  mean  value  for  the  item  for  the  top   and  boFom  quarter  judges.  
  10. 10. Designing  Ques*onnaire  and  Survey  Research 10 Example: The Employment Self Esteem Scale INSTRUCTIONS: Please rate how strongly you agree or disagree with each of the following statements by placing a check mark in the appropriate box. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 1. I feel good about my work on the job. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 2. On the whole, I get along well with others at work. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 3. I am proud of my ability to cope with difficulties at work. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 4. When I feel uncomfortable at work, I know how to handle it. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 5. I can tell that other people at work are glad to have me there. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 6. I know I'll be able to cope with work for as long as I want. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 7. I am proud of my relationship with my supervisor at work. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 8. I am confident that I can handle my job without constant assistance. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 9. I feel like I make a useful contribution at work. Strongly Disagree Somewhat Disagree Somewhat Agree Strongly Agree 10. I can tell that my coworkers respect me.
  11. 11. Designing  Ques*onnaire  and  Survey  Research 11 Survey-Research Design Cross-Sectional Design the collection of data at a single point in time from a sample drawn from a specified population Successive Independent Samples Design a series of cross-sectional survey are conducted over time Longitudinal Design the same sample of respondents is surveyed more than once
  12. 12. Designing  Ques*onnaire  and  Survey  Research 12 Total survey error The ultimate goal of survey research is to accurately measure particular constructs within a sample of people who represent the population of interest • Coverage Error: the bias that can result when the pool of potential survey participants from which a sample is selected does not include some portions of the interest • Sampling Error: the random differences that invariably exist between any sample and the population from which it was selected • Nonresponse Error: the bias that can result when data are not collected from all of the members of a sample • Measurement Error: all distortions in the assessment of the construct of interest, including systematic biases and random variance that can be brought by respondents’ own behavior, interviewer behavior, and the questionnaire
  13. 13. Designing  Questionnaire
  14. 14. Designing  Ques*onnaire  and  Survey  Research 14 Questionnaire Design: The six steps for preparing the questionnaire 1.  Decide  what  informa0on  should  be  sought 2.  Decide  what  type  of  ques0onnaire  should  be  used   3.  Write  a  first  dra;  of  the  ques0onnaire 4.  Reexamine  and  revise  the  ques0onnaire 5.  Pretest  the  ques0onnaire 6.  Edit  the  ques0onnaire  and  specify  the  procedure
  15. 15. Designing  Ques*onnaire  and  Survey  Research 15 Question Content • Is the Question Necessary/ Useful? ♪ Do you need the age of each child or just the number of children under 16? ♪ Do you need to ask income or can you estimate? • Are Several Questions Needed? ♪ double-barreled question (eg. Teacher & parents) ♪ does not cover all possibilities ♪ does not give you enough context ♪ does not determine the intensity • Do Respondents Have the Needed Information? ♪ You should ask a filter question first (e.g., Have you ever watched the show ER?) before asking them their opinions about it. • Does the Question Need to be More Specific? ♪ How well did you like the book? ♪ Did you recommend the book to others? • Is Question Sufficiently General? • Is Question Biased or Loaded? ♪e.g. What do you see as the benefits of a tax cut? ♪e.g. What do you see as the disadvantages of eliminating welfare? • Will Respondent Answer Truthfully?
  16. 16. Designing  Ques*onnaire  and  Survey  Research 16 Question Words ♪    Can the Question be Misunderstood? e.g. What  kind  of  headache  remedy  do  you  use? One  of  the  major  difficulty  in  wri-ng  good  survey  ques-ons  is  geRng  the  wording  right   ♪  Is the time frame specified? Whenever  using  the  words  "will",  "could",  "might",  or  "may"  in  a  ques<on,  you  might                                                                                                                                                                                                                      suspect  that  the  ques<on  asks  a  <me-­‐related  ques<on.   ♪    What Assumptions Does the Question Make? ♪    Is the wording too direct? E.g. How did you feel about being in the war? How well did the equipment hold up in the field?                                                                                        How  well  were  new  recruits  trained? ♪    Does the question contain difficult or unclear terminology?…..  
  17. 17. Designing  Ques*onnaire  and  Survey  Research 17 Question Placement Considering following questions:            ♪  Is  the  answer  influenced  by  prior  ques<ons?              ♪  Does  ques<on  come  too  early  or  too  late  to  arouse  interest?              ♪  Does  the  ques<on  receive  sufficient  aFen<on? A Checklist of Considerations ✓ start  with  easy,  nonthreatening  ques<ons ✓ put  more  difficult,  threatening  ques<ons  near  end ✓ never  start  a  mail  survey  with  an  open-­‐ended  ques<on ✓ for  historical  demographics,  follow  chronological  order ✓ ask  about  one  topic  at  a  <me ✓ when  switching  topics,  use  a  transi<on ✓ reduce  response  set  (the  tendency  of  respondent  to  just  keep  checking  the  same   response) ✓ for  filter  or  con<ngency  ques<ons,  make  a  flowchart   The  Golden  Rule Do  unto  your  respondents  as  you  would  have  them  do  unto  you!
  18. 18. Designing  Ques*onnaire  and  Survey  Research 18 Other Important Issues Question Order ✓ Grouping Questions by topic may be useful ✓ It is easier for the respondents to think in the same topic   ❖ For bipolar scales ( e.g., running from positive to negative with neutral in the middle), reliability and validity are highest for about 7 point.( Matell & Jacoby, 1971) ❖ The reliability and validity of unipolar scales (e.g., running from no importance to very high importance) seem to be optimized for a bit shorter scales, approximately 5 points lon (Wikman & Waarneryd, 1990) ❖ A good number of studies suggest that data quality is better when all scales points are labeled with words than only some are. ( Krosnick & Berent, 1993) Ra#ng  Scale  Formats  
  19. 19. Sampling  Design  in  the   Survey  Research
  20. 20. Designing  Ques*onnaire  and  Survey  Research 20 Sampling in Survey Research
  21. 21. Designing  Ques*onnaire  and  Survey  Research 21 Statistical Sampling Terms Variable: is a specific measurement value that a sampling unit supplies Statistic: the  responses  that  we  get   for  our  en<re  sample Parameter:   the measurement the entire population
  22. 22. Designing  Ques*onnaire  and  Survey  Research 22 Different Approaches to Sampling Methods Probability Sampling Non-Probability Sampling Any method of sampling that utilizes some form of random selection ➡ Simple Random Sampling ➡ Stratified Random Sampling ➡ Systematic Random Sampling ➡ Cluster (Area) Random Sampling Methods do not involve random selection ➡ Convenience Sampling ➡ Purposive Sampling
  23. 23. Designing  Ques*onnaire  and  Survey  Research 23 Probability Sampling any method of sampling that utilizes some form of random selection Some Definitions ✓ N = the number of cases in the sampling frame ✓ n = the number of cases in the sample ✓ NCn = the number of combinations (subsets) of n from N ✓ f = n/N = the sampling fraction
  24. 24. Designing  Ques*onnaire  and  Survey  Research 24 Simple Random Sampling Objective:  To  select  n  units  out  of   N  such  that  each  NCn  has  an  equal   chance  of  being  selected.   Procedure: Use a table of random numbers, a computer random number generator, or a mechanical device to select the sample.
  25. 25. Designing  Ques*onnaire  and  Survey  Research 25 Stratified Random Sampling Objective: Divide the population into non- overlapping groups (i.e., strata) N1, N2, N3, ... Ni, such that N1 + N2 + N3 + ... + Ni = N. Then do a simple random sample of f = n/N in each strata. Procedure: Divide your population into homogeneous subgroups and then taking a simple random sample in each subgroup.
  26. 26. Designing  Ques*onnaire  and  Survey  Research 26 Systematic Random Sampling • number the units in the population from 1 to N • decide on the n (sample size) that you want or need • k = N/n = the interval size • randomly select an integer between 1 to k • then take every kth unit
  27. 27. Designing  Ques*onnaire  and  Survey  Research 27 Cluster (Area) Random Sampling • Divide population into clusters (usually along geographic boundaries) • Randomly sample clusters • Measure all units within sampled clusters Beijing Jinan Nanjing Shanghai Hangzhou Changsha Shenzhen Chongqing Tianjin   ShijiaZhuang (Hebei) Zhengzhou (Henan)  Qingdao  Hefei (Anhui)  Suzhou Chengdu   Shantou  Xiamen Wuhan Xi’an (Shaanxi)  Zhanjiang Guiyang   Kunming Eg. Measuring Chinese Personality
  28. 28. Designing  Ques*onnaire  and  Survey  Research 28 Non-Probability Sampling does  not  involve  random  selec7on • Modal  Instance  Sampling • Expert  Sampling   • Quota  Sampling   • Snowball  Sampling     Purposive Sampling: sample  with  a  purpose  in  mind.   or called Accidental, Haphazard sampling they are not representative of the populations Convenience Sampling: tradi0onal  "man  on  the  street"  
  29. 29. Designing  Ques*onnaire  and  Survey  Research 29 Advantage and Disadvantage of Sampling Approach Probability Sampling Non-Probability Sampling Advantage  Research can be confident that a selected is representative  It permits researchers to precisely estimate the amount of variance present in a given data set that is due to sampling error Disadvantage ✓ Calculation Advantage  Convenient  Flexible Disadvantage ✓ We may or may not represent the population well, ✓ It will often be hard for us to know how well we've done so
  30. 30. Thank  You Contact us via … Mail: Follow: twitter@htjitra Website: Zhejiang  University,  Hangzhou  (China)