Survey Questions and Responses:Art and Research in Survey DesignAmy A. Germuth, Ph.D.Prepared for the Research Triangle Pa...
2<br />
Why do we care?<br />Bias – decreases the reliability with which we can speak about results<br />Multiple types of bias in...
Survey Questions<br />Can we identify any rules for what <br />makes a good survey question?<br />4<br />
Let’s take a survey!<br />5<br />
What makes a good survey question?<br />The question needs to be consistently understood.<br />The question needs to be on...
Survey Responses<br />What can we say about best practices <br />regarding survey response options?<br />7<br />
8<br />
9<br />Never, Sometimes, Frequently<br />
10<br />The Agreement Continuum – 5 Point Scale<br />
11<br />The Agreement Continuum – 6 Point Scale<br />
If quantitatively anchored, would potential respondents be less confused about where response options lie on a scale?<br /...
What do Quantitative anchors provide us?<br />May more accurately describe qualitative descriptions<br />Provide scaling t...
14<br />Withdrawing Cash Question<br />
Restaurant Service Question<br />And our scale would be: <br />1 = Horrible/Very poor, 2 = Poor, 3 = Average, 4 = Good, 5 ...
What are problems with Qualitative anchors?<br />Often ambiguous – not viewed reliably by respondents = introduces bias<br...
How Many Response Options?<br />It depends:<br />Cognitive Abilities – provide fewer responses for children<br />Middle Op...
Odd-Numbered Scales<br />Given our state budget issues,  do you think that the state sales tax should be increased or decr...
Persons lean towards one or the other polar alternatives and thus legitimate to force a selection.<br />Persons really do ...
10-20% of respondents will choose that option<br />The decline appears to effect the polar alternative categories equally ...
Resources<br />DeVellis, R. F. (2003).  Scale development: Theory and applications, 2nd Edition,  Thousand Oaks, CA: Sage....
Amy A. Germuth, Ph.D.<br />President, EvalWorks LLC<br />150 Solterra way<br />Durham, NC 27705<br />(919) 401-5403<br />a...
Upcoming SlideShare
Loading in …5
×

Improving Survey Questions and Responses

3,864 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,864
On SlideShare
0
From Embeds
0
Number of Embeds
66
Actions
Shares
0
Downloads
34
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Improving Survey Questions and Responses

  1. 1. Survey Questions and Responses:Art and Research in Survey DesignAmy A. Germuth, Ph.D.Prepared for the Research Triangle Park Evaluation MeetingAugust 13, 2010<br />Copyright 2010<br />Amy A. Germuth, Ph.D.<br />1<br />
  2. 2. 2<br />
  3. 3. Why do we care?<br />Bias – decreases the reliability with which we can speak about results<br />Multiple types of bias in surveys – we are interested in reducing bias associated with Measurement Error<br />3<br />
  4. 4. Survey Questions<br />Can we identify any rules for what <br />makes a good survey question?<br />4<br />
  5. 5. Let’s take a survey!<br />5<br />
  6. 6. What makes a good survey question?<br />The question needs to be consistently understood.<br />The question needs to be one that can be consistently administered or communicated to respondents.<br />What constitutes a good answer needs to be communicated via the question.<br />All respondents should have access to the information needed to answer the question (unless the question is meant to measure knowledge).<br />The questions must be ones to which respondents are willing to provide the answer called for. <br />6<br />
  7. 7. Survey Responses<br />What can we say about best practices <br />regarding survey response options?<br />7<br />
  8. 8. 8<br />
  9. 9. 9<br />Never, Sometimes, Frequently<br />
  10. 10. 10<br />The Agreement Continuum – 5 Point Scale<br />
  11. 11. 11<br />The Agreement Continuum – 6 Point Scale<br />
  12. 12. If quantitatively anchored, would potential respondents be less confused about where response options lie on a scale?<br />12<br />
  13. 13. What do Quantitative anchors provide us?<br />May more accurately describe qualitative descriptions<br />Provide scaling to qualitative descriptions<br />May be used to indicate response options are equal interval in nature<br />Improves reliability  decreases bias<br />13<br />
  14. 14. 14<br />Withdrawing Cash Question<br />
  15. 15. Restaurant Service Question<br />And our scale would be: <br />1 = Horrible/Very poor, 2 = Poor, 3 = Average, 4 = Good, 5 = Very good/Excellent<br />15<br />
  16. 16. What are problems with Qualitative anchors?<br />Often ambiguous – not viewed reliably by respondents = introduces bias<br />Fail to convey equal intervals<br />If multiple response options (e.g., 7 or 9) hard to differentiate<br />16<br />
  17. 17. How Many Response Options?<br />It depends:<br />Cognitive Abilities – provide fewer responses for children<br />Middle Option (Neutral) – will determine if number of response options are even or odd<br />From findings above – is 7 too many? 5 or 6 appear to be optimal as should provide enough variance in responses and can be both qualitatively and quantitatively anchored.<br />17<br />
  18. 18. Odd-Numbered Scales<br />Given our state budget issues, do you think that the state sales tax should be increased or decreased?<br />Should more or less time be provided for Physical Education to students in North Carolina's public schools?<br /> versus<br />How true is this statement for you: “I support a woman’s right to choose”?<br />18<br />
  19. 19. Persons lean towards one or the other polar alternatives and thus legitimate to force a selection.<br />Persons really do favor the middle position and if forced to choose one of the alternatives will contribute to random or systematic error.<br />19<br />Middle Alternatives – Two Views<br />
  20. 20. 10-20% of respondents will choose that option<br />The decline appears to effect the polar alternative categories equally <br />Persons who have low "intensity" about a subject are most likely to choose a middle alternative . <br />Lack of or lower education does not appear to affect whether someone choose a middle response or not.<br />Cognition does affect choices which is why one should be careful how many choices children are provided on surveys. <br />20<br />If A Middle Alternative is Offered<br />
  21. 21. Resources<br />DeVellis, R. F. (2003). Scale development: Theory and applications, 2nd Edition, Thousand Oaks, CA: Sage.<br />Dillman, D. A. (1999). Mail and internet surveys: The tailored design method (2nd. Ed.). New York, NY: John Wiley & Sons. <br />Fowler, F.J. Jr. (1995). Improving survey questions: Design and evaluation. Thousand Oaks, CA: Sage<br />Schuman, H. & Presser, S. (1996). Questions and Answers in Attitude Surveys. Thousand Oaks, CA: Sage<br />21<br />
  22. 22. Amy A. Germuth, Ph.D.<br />President, EvalWorks LLC<br />150 Solterra way<br />Durham, NC 27705<br />(919) 401-5403<br />agermuth@mindspring.com<br />Website: www.EvalWorks.com<br />Blog: www.EvalThoughts.com<br />22<br />Contact Information<br />

×