10. • Are overall response rates comparable?
• Are the errors of commission and omission comparable?
• Is there a similar distribution of answers?
• Is item non-response an issue?
• Is straightlining an issue in the grid?
• Is their more variability with the grid?
Research Questions
11. Results from Study:
Overall Response Rates
No statistically
significant
difference
Version Number
Responded
Total
Surveys
Response
Rates
Grid 1,780 2,850 68%
Non-grid 1,820 2,870 66%
68% 66%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Grid Non-grid
Unit Response Rate
12. Grid
Non-Grid
*Statistically Significant at p<.005
Results from Study:
Errors of Commission and Omission
Child participates in after school activities?
Grid response Missing No* Yes*
Filled out grid 55% 14% 98%
Did not fill out grid 45% 86% 2%
Child participates in after school activities?
Non-grid response Missing No* Yes
Filled out non-grid 48% 10% 97%
Did not fill out non-grid 52% 90% 3%
13. Grid
Non-Grid
*Statistically Significant at p<.005
Results from Study:
Errors of Commission and Omission
Child participates in after school activities?
Grid response Missing No* Yes*
Filled out grid 55% 14% 98%
Did not fill out grid 45% 86% 2%
Child participates in after school activities?
Non-grid response Missing No* Yes
Filled out non-grid 48% 10% 97%
Did not fill out non-grid 52% 90% 3%
14. Grid
Non-Grid
*Statistically Significant at p<.005
Results from Study:
Errors of Commission and Omission
Child participates in after school activities?
Grid response Missing No* Yes
Filled out grid 55% 14% 98%
Did not fill out grid 45% 86% 2%
Child participates in after school activities?
Non-grid response Missing No* Yes
Filled out non-grid 48% 10% 97%
Did not fill out non-grid 52% 90% 3%
15. Results from Study:
Distribution of Answers – Question 1
Label Rated as Very Important Checked Ranked
Location 61% 70% 41%
Cost 54% 59% 36%
Adult supervision 84% 65% 32%
Safety 91% 83% 64%
Nurturing environment 77% 53% 25%
Knowing caretaker 35% 20% 8%
Transportation 44% 30% 12%
Children can be at same place 29% 20% 7%
Hours are convenient 67% 60% 27%
Quality of facilities 67% 49% 17%
Licensed provider 62% 33% 7%
Speaks first language 44% 11% 1%
Special needs 27% 7% 3%
Adequate number of staff 58% 30% 4%
No other programs available 13% 6% 2%
18. Results from Study:
Item non-response in Grid 1
Label
No
response
Location 0.4%
Cost 0.9%
Adult supervision 0.5%
Safety 0.5%
Nurturing environment 0.9%
Knowing caretaker 0.9%
Transportation 1.1%
Children can be at same place 1.1%
Hours are convenient 0.9%
Quality of facilities 1.1%
Licensed provider 1.2%
Speaks first language 1.1%
Special needs 1.2%
Adequate number of staff 1.4%
No other programs available 3.9%
0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
3.5%
4.0%
4.5%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Percentage
Order in System
Non-Response Rate in Grid 1
19. Results from Study:
Item non-response in Grid 2
Label
No
response
Enjoyment of child 0.7%
Similar ages 0.4%
Spends time with family 0.8%
Cultural diversity 0.9%
Music, art, culture 0.8%
Computer access 0.8%
Math and science 0.7%
Provides tutoring 0.7%
Helps child learn English 0.9%
Helps child learn other language 1.4%
Provides reading 1.0%
Physical activities 0.9%
Provides role model 1.0%
Family likes staff 0.9%
Religious affiliation 1.4%
0.0%
0.2%
0.4%
0.6%
0.8%
1.0%
1.2%
1.4%
1.6%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Percentage
Order in System
Non-Response Rate in Grid 2
20. Results from Study:
Item not marked in Non-Grid System
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Percentage
Order in Check System
Percentage Not Marked in
Non-Grid System 1
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Percentage
Order in Check System
Percentage Not Marked in
Non-Grid System 1
21. Results from Study: Straightlining
Straightlining in Grid 1
Respondents who answered at least one 1,400
Respondents that straightlined 120
Straightlining percentage 9%
Straightlining in Grid 2
Respondents who answered at least one 1,380
Respondents that straightlined 120
Straightlining percentage 9%
9% 9%
0%
2%
4%
6%
8%
10%
12%
14%
SL in Grid 1 SL in Grid 2
Straightlining in Grid Systems
22. Results from Study: Variance
Grid System 1
Used exactly one rating label 9%
Used exactly two rating labels 18%
Used exactly three rating labels 30%
Used exactly four rating labels 30%
Used exactly five rating labels 14%
Grid System 2
Used exactly one rating label 9%
Used exactly two rating labels 18%
Used exactly three rating labels 25%
Used exactly four rating labels 30%
Used exactly five rating labels 17%
Out of respondents that
rated all items.
0%
5%
10%
15%
20%
25%
30%
35%
Used exactly
one rating
label
Used exactly
two rating
label
Used exactly
three rating
label
Used exactly
four rating
label
Used exactly
five rating
label
Grid 1
0%
5%
10%
15%
20%
25%
30%
35%
Used exactly
one rating
label
Used exactly
two rating
label
Used exactly
three rating
label
Used exactly
four rating
label
Used exactly
five rating
label
Grid 2
23. • Response rates comparable
• Distributions less comparable
• Item non response low
• Hard to tell with check system
• Straightlining low
• Variability low
Conclusions
24. • Cognitive testing
• Order Randomization
• Challenge when mode changes to web
• How to handle mixed-mode formatting
Conclusions:
Future Research/Implications
27. References
Alwin, D.F. & Krosnick, J.A. (1985). The measurement of values in surveys: A
comparison of ratings and rankings. The Public Opinion Quarterly, 49(4), 535-552.
Cole, J. S., McCormick, A. C., & Gonyea, R. M. (2012). Respondent use of straight-
lining as a response strategy in education survey research: Prevalence and
implications. Paper presented at the annual meeting of the American Educational
Research Conference, Vancouver, BC.
