Sam Ladner, PhD 2008 Online Surveys 101

What is survey research? Online surveys Telephone surveys Structured face-to-face interviews Asking structured questions

Surveys versus other qualitative methods When a you want to know… … you need What are users’ opinions of the site? What is the demographic profile of our users? What kinds of beliefs or attitudes do our users have about our… brand customer service How can we be sure that design changes changed users opinions of the site? A survey How usable is our Web site? What do our users generally dislike about our Web site? What kinds of features should we include in our new Web site? Qualitative research method

What are users’ opinions of the site?

What is the demographic profile of our users?

What kinds of beliefs or attitudes do our users have about our…

brand

customer service

How can we be sure that design changes changed users opinions of the site?

Is a survey the right research method? I know a lot about the topic I need to summarize the findings in numerical format I don’t need a lot of detail and nuance I need to generalize the results to the population I need to predict the likelihood of a certain thing happening I want to measure incremental change I don’t really know much about the topic I don’t need to summarize the findings in numerical format I need to communicate a lot of detail and nuance I’m interested in case study information I’m not planning on predicting the likelihood of anything I don’t need to measure any change Yes No

I know a lot about the topic

I need to summarize the findings in numerical format

I don’t need a lot of detail and nuance

I need to generalize the results to the population

I need to predict the likelihood of a certain thing happening

I want to measure incremental change

I don’t really know much about the topic

I don’t need to summarize the findings in numerical format

I need to communicate a lot of detail and nuance

I’m interested in case study information

I’m not planning on predicting the likelihood of anything

I don’t need to measure any change

Is it time to do a survey? Yes No Discovered questions and answers through Qual research? Clear idea of all possible questions and answers? Yes No Formulated all questions and possible answers? Do a survey Do more Qual research Yes Pilot test the questions No

Asking good questions Mutually exclusive and exhaustive No overlap between answers All possible answers No double-barreled questions Wrong: “What do you think about our response times and customer service?” Right: “What is your opinion of our response time?” Focus on opinions or beliefs of participants Wrong: “How usable is our Web site?” Right: “What are users’ opinions of the site’s look and feel?” Close-ended is best Cuts down on analysis time Demographic questions at the end

Mutually exclusive and exhaustive

No overlap between answers

All possible answers

No double-barreled questions

Wrong: “What do you think about our response times and customer service?”

Right: “What is your opinion of our response time?”

Focus on opinions or beliefs of participants

Wrong: “How usable is our Web site?”

Right: “What are users’ opinions of the site’s look and feel?”

Close-ended is best

Cuts down on analysis time

Demographic questions at the end

Sampling and its implications You can predict the likelihood of any single person being selected to participate Has some randomized element to it Allows for you to compare your findings to what you would expect with pure random events Allows for prediction of accuracy Allows for generalizing to the population at large Can use inferential techniques You cannot predict the likelihood of any person participating Has no random element to it Cannot be used to predict future likelihood of anything Cannot be used to generalize to the population at large Still provides some insight, particularly if you ask everyone in a group the same question Does not allow to estimate the accuracy of the results Cannot use inferential techniques Probability Sample Non-probability Sample

You can predict the likelihood of any single person being selected to participate

Has some randomized element to it

Allows for you to compare your findings to what you would expect with pure random events

Allows for prediction of accuracy

Allows for generalizing to the population at large

Can use inferential techniques

You cannot predict the likelihood of any person participating

Has no random element to it

Cannot be used to predict future likelihood of anything

Cannot be used to generalize to the population at large

Still provides some insight, particularly if you ask everyone in a group the same question

Does not allow to estimate the accuracy of the results

Cannot use inferential techniques

The lay of the land: descriptive statistics Summarizes the responses What is the most “typical” response? Mean Mode Median How much variation is there between responses? Standard deviation Inter-quartile range

Summarizes the responses

What is the most “typical” response?

Mean

Mode

Median

How much variation is there between responses?

Standard deviation

Inter-quartile range

Predicting behavior: inferential statistics Infers or predicts future behavior Which respondents are more likely to… Cross-tabulation Which factors most influence Behavior Satisfaction etc

Infers or predicts future behavior

Which respondents are more likely to…

Cross-tabulation

Which factors most influence

Behavior

Satisfaction

etc

The tools you need Online Survey Tool Statistical Analysis Software

The skills you need Knowledge of the topic Ability to write good questions Familiarity with the survey tool Some basic knowledge of statistics A framework for writing the report Important to write the outline of the final report before you start!

Knowledge of the topic

Ability to write good questions

Familiarity with the survey tool

Some basic knowledge of statistics

A framework for writing the report

Important to write the outline of the final report before you start!

The payoff you’ll get “ More men are dissatisfied with the checkout process.” “ Younger users are much more unhappy with customer service that older users.” “ Higher-income individuals are the most unlikely to recommend this site to friends and family.”

“ More men are dissatisfied with the checkout process.”

“ Younger users are much more unhappy with customer service that older users.”

“ Higher-income individuals are the most unlikely to recommend this site to friends and family.”

What you won’t learn… Why aren’t people coming to our site? How can we innovate on our current design? Why don’t people finish the checkout process? What are the main characteristics of the people who don’t come to our site?

Why aren’t people coming to our site?

How can we innovate on our current design?

Why don’t people finish the checkout process?

What are the main characteristics of the people who don’t come to our site?