The document outlines the objectives and content of a survey design workshop. It discusses key topics like questionnaire design, levels of measurement, sampling, and implementation issues. The workshop aims to help participants understand rigorous survey planning, common survey methods, questionnaire design best practices, and critically reviewing example surveys.

1. Survey Design Workshop Inter-University Research Workshop Program Dr. James Neill Centre for Applied Psychology University of Canberra 1 February, 2011

2. Outline Objectives

3. Introductions

4. Logins & Resources

5. Research methods

6. Questionnaire design

7. Levels of measurement

8. Sampling

9. Evaluation

10. Objective 1 Understand the importance of a rigorous , step-by-step process in planning, developing & implementing research questionnaires

11. Objective 2 Consider the pros and cons of common methods for survey administration Face-to-face interview

12. Telephone survey

13. Mail survey

14. Internet/mobile survey

15. Objective 3 Examine nuts & bolts of questionnaire design e.g., Question style,

16. Response formats,

17. Layout, and

18. Pre- and pilot testing

19. Objective 4 Consider implementation issues Sampling methods

20. Sample size

21. Objective 5 Critically review example surveys. Existing examples

22. Student in-progress examples with a view towards planning, drafting, and/or revising of an initial draft (pilot) survey.

23. Resources Survey Design Workshop Notes (Wikiversity)

24. Readings

25. Books about surveys design and survey research (check library)

26. Introductions Introductions

27. Types of Research ( Research Methods ) There are 3 main research methods: Experimental

28. Quasi-experimental

29. Non-experimental

30. Types of Research - Experimental Characterised by: Random assignment

31. Control over extraneous variables

32. Types of Research - Quasi-experimental Characterised by: Non-random assignment

33. Control over some extraneous variables

34. Groups are “naturally occuring”

35. Types of Research - Non-experimental Characterised by: No “groups” or “conditions” are created or used (i.e., no full experimental or quasi-experimental groups)

36. Minimal control over extraneous variables

37. Survey Research Characteristics Surveys are widely used in non-experimental social science research.

38. Often use interviews or questionnaires .

39. Involve real-world samples .

40. Often quantitative , but can be qualitative .

41. History of Survey Research Survey research methodology was initially developed in the 1940's – 1960's.

42. Since the 1980's, theories and principles evolved to create a unified perspective on the design, conduct, and evaluation of surveys.

43. 8 Survey Research Characteristics Backstrom & Hursh-C é sar (1981, pp. 3-4) Systematic : follows a specific set of rules, a formal and orderly logic of operations

44. Impartial : selects units of the population without prejudice or preference

45. Representative : includes units that together are representative of the problem under study and the population affected by it

46. Theory-based : operations are guided by relevant principles of human behaviour and by mathematical laws of probability (chance).

47. Quantitative : assigns numerical values to nonnumerical character

48. Self-monitoring : procedures can be designed in ways that reveal any unplanned and unwanted distortions (biases) that may occur 8 Survey Research Characteristics Backstrom & Hursh-C é sar (1981, pp. 3-4)

49. Contemporary : it is current, more than historical, fact-finding

50. Replicable : other people using the same methods in the same ways can get essentially the same results 8 Survey Research Characteristics Backstrom & Hursh-C é sar (1981, pp. 3-4)

51. Advantages of Survey-Based Research Ecological validity

52. Access to wide range of participants

53. Potentially large amounts of data

54. May be more ethical (than experiments)

55. Disadvantages of Survey-Based Research Lack of control -> less internal validity

56. Data may be 'superficial'

57. Can be costly to obtain representative data

58. Self-report data only

59. Potentially low compliance rates

60. Example Surveys Unit Satisfaction Survey Community Library Use

61. The research process 1. Establish need for info/ research 2. Problem definition/ Hypotheses 3. Research design 4. Sampling/ Data collection 5. Data analysis 6. Reporting

62. The research process

63. Survey construction: Overview What is a survey?

64. Types of questionnaires

65. Questionnaire development

66. Writing questions

67. Types of questions

68. Response formats  LOM

69. Survey formatting

70. A standardised stimulus

71. A measuring instrument

72. A way of converting fuzzy psychological stuff into hard data for analysis What is a survey?

73. Types of surveys Types of surveys Self - administered Interview - administered Postal Delivered and collected Telephone Face to face structured interview Web-based

74. Questionnaire development 1. Formulate generic questionnaire 2. Expand the questionnaire Turn into separate sections based on study objectives. Draft qs & response formats 4. Finalise questionnaire & implement Question order & funnel qs 3. Pre-test, pilot test, & redraft

75. Formulate Generic Questionnaire Turn objectives into sections of the survey

76. Ensure all questions relate to research objectives

77. For explanatory objectives or hypotheses ensure both dependent and independent variables exist

78. Cover Letter & Ethics Statement Introduction or cover letter: Outline details of research project and allow for ethical informed consent.

79. Few will read it without good prompting and being easy-to-read

80. Instructions Provides consistency - helps to ensure standard conditions across different administrations

81. Explain how to do the survey in a user-friendly manner

82. Example: Life Effectiveness Questionnaire

83. Instructions: Example

84. Cover letter / ethics statement checklist Outline details of research project e.g.,: Who are you? Are you bona fide?

85. Purpose of survey?

86. What's involved?

87. Explain any risks/costs/rewards

88. How will results be used?

89. Human ethics approval #

90. How is consent given / not given?

91. Voluntary - can choose not to continue anytime

92. More info: Complaints, how to obtain results, contact details etc.

93. Font (type, size)

94. No. of pages

95. Margins

96. Double vs. single-siding

97. Colour, etc. Layout

98. Demographics single section, usually at beginning or end of questionnaire

99. only include relevant questions Layout

100. Space for comments?

101. Indicate the end

102. Say thanks! Layout

103. Flow/Structure Logical order of questions (use sections)

104. Ask screening questions first, rather than later. Does the participant qualify for the survey? (esp. for internet surveys)

105. Use funneling/branching questions to move respondents through survey

106. Start off with easy to answer and engaging questions

107. More controversial questions in middle section

108. Expanding the Survey

109. Personal Information Generally, researchers put personal information at beginning of survey (if required). However, this may put off respondents, so also consider uncluding at towards end.

110. Consider response format e.g, Income in categories or ranges

111. Personal Information More likely to respond to personal questions for anonymous or mail surveys as opposed to face to face or telephone

112. Show cards for responses may help for face to face interviews

113. Pre- & pilot-testing Pre-test items on convenient others - ask for feedback

114. Revise items e.g., Which don’t apply to everybody

115. Are redundant

116. Are misunderstood

117. Are non-completed Reconsider ordering & layout

118. Pilot test on a small sample from the target population, analyse, & revise

119. Types of Questions Be able to justify and defend your choices ...

120. Writing questions - Dos Define target constructs

121. Check related research & questionnaires

122. Draft items (for important, fuzzy constructs aim to have multiple indicators)

123. Pre-test & revise

124. Writing questions - Dos Focus directly on topic/issue

125. Be clear

126. Be brief

127. Avoid big words

128. Use simple and correct grammar

129. Writing questions – Don'ts Inapplicable – must apply to all respondents Over-demanding – e.g., recall of detail or time-consuming, unnecessary questioning Ambiguous – meaning must be clear to all respondents Double negative – e.g., Do you not disapprove of tax reforms?

130. Writing questions – Don'ts Double-barrelled - e.g., “Do you think speed limits should be lowered for cars & trucks?” Leading - e.g., “don’t you see some danger in the new policy?” Loaded – e.g., “Do you advocate a lower speed limit to save human lives?” vs “Does traffic safety require a lower speed limit?”

131. Response biases Social desirability

132. Acquiescence yea- and nay-saying Self-serving bias

133. Order effects

134. Demand characteristics Interview High demand characteristics

135. Can elicit richer information Questionnaire Lower demand characteristics

136. Information may be less rich

137. Types of Questions

138. Accuracy of recall decreases over time

139. Objective questions A verifiably true answer (i.e., factual information) exists for each unit.

140. The question could be accurately answered by an observer. e.g., How times during 2009 did you visit a G.P.? ______

141. Subjective questions Asks about fuzzy personal perceptions.

142. There is no “true”, factual answer.

143. Many possible answers per unit.

144. Can't be accurately answered by an observer. e.g., Think about the visits you made to a G.P. during 2010. How well did you understand the medical advice you received? perfectly very well reasonably poorly not at all

145. Objective vs. subjective questions Both types of questions may be appropriate; depends on the purpose of the study.

146. One criticism of this distinction: There is no such thing as “objective” and that all responses are subjective.

147. Open-ended Questions Rich information can be gathered