Carma internet research module survey design issues
Survey Design IssuesDillman’s Tailored Design Method CARMA Internet Research Module Jeff Stanton
Why should I?• Why are you contacting me? Who the heck are you? What do you want? Do you really care about my opinion, or am I just a number?• What’s this study about? Why should I care?• What’s in it for me? How will I feel if I respond honestly? How will I feel if I sabotage you or simply refuse to respond?• Who will benefit if I return this? Who can I punish if I actively choose not to return it or better yet give bogus data?• What if I do this later? What’s the rush? Why not procrastinate? Why do it now?• Why do you want this personal information? How do I know you’ll keep it confidential?May 15-17, 2008 Internet Data Collection Methods (Day 2-2)
Surveys Stink• Response rates have been falling for more than 20 years in all modalities, first phone, then mail, and, after an initial period of optimism, the web• InSite Consulting documented the four main reasons: – No time (19%) – Survey is too long (12%) – “Don’t feel like it” (10%) – Too boring (10%)• Yet a survey is still the most cost effective method of self report data collection available to the researcher• It is still possible for a well-designed and well-executed survey project to get good data, from an appropriate sample, and with an excellent response rate
A self report study…• … is a system of interactions with each participant designed to motivate prompt, honest, and complete response.• The motivation is governed by:May 15-17, 2008 Internet Data Collection Methods (Day 2-4)
Social Exchange Obligation Altruism Inclusion Ego Satisfaction Researcher offers social Participant trades time rewards and minimizes and effort, not for money, barriers to participation but for fulfillment.May 15-17, 2008 Internet Data Collection Methods (Day 2-5)
Providing Rewards “I Trust You!” Importance of Responses Support Key Values Ask for advice Uniqueness Validation Feedback Increase Soc Show Positive Regard: Politeness, concern, time, respect, gratitude, appreciation, convenienceMay 15-17, 2008 Internet Data Collection Methods (Day 2-6)
Reducing Costs “I Trust You!” Non-subordinating language Short response time No embarrassment No drudgery Readability Simplicity Decrease So Obtain commitment by showing the respondent you have put their interests foremost in your design decisionsMay 15-17, 2008 Internet Data Collection Methods (Day 2-7)
Increasing Trust: The long term expectation that reward exceeds cost• Each respondent is going to get $2. When do you give it to them? – In the first email contact when you request their participation the study? – After they have submitted their survey online? – After you have verified the completeness of their data?May 15-17, 2008 Internet Data Collection Methods (Day 2-8)
Active vs. Passive NR Active Non-Response Passive Non-Response A conscious decision not Failure to respond to the to comply with the research request because of research request. Generally forgetting, lack of time, occurs at first contact. or work overload. Can Difficult or impossible to convert convert; can be once decision is taken. time consuming to convert.May 15-17, 2008 Internet Data Collection Methods (Day 2-9)
Think Benefits and Rewards• Money can be an important motivator, but you must differentiate between money as a token of esteem and money as a payment for services• Standing/Status: Exclusivity of expertise or status and standing in the community; participation in the survey can affirm this• Social engagement: How does participation in your survey help someone feel more strongly affiliated with their group, profession, organization, neighborhood, or city?• Feedback Cravings: Use the survey results to tell someone how good they are, what are their strengths, where they need help, what are their weaknesses• Information is a benefit: What can someone learn from taking your survey, how will they be better off when they are done• The litmus test: After a person completes your survey, would he or she choose to recruit similar others into it?
Summary of DillmanIncrease Benefits Decrease Barriers Foster TrustAsk for advice Avoid subordinating Provide token of language appreciation inShow gratitude advance Avoid embarrassmentAlign with group values Avoid inconvenience Study sponsored byRespondent identifies Study is short and easy legitimate authority with sponsoring org Questions are not The task is importantGive tangible rewards redundant The study outcome will Minimize personal make a positiveGive social validation changeMake study interesting information Emphasize similarity to The relationship isEvoke curiosity other requests similar to one thatCommunicate scarcity already exists of response opportunity Internet Data Collection Methods (Day 2- May 15-17, 2008 11)
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.