Write better survey questions, run a survey from start to finish, survey tips from the expert survey methodologists. Workshop at JBoye Conference, Aarhus, Denmark, 2011.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Thesis PROPOSAL Defense Presentation - March 26 Hermes Huang
This is the Thesis Proposal Defense Presentation by Hermes Huang titled
Analyzing Impacts of Networks within the Maker Movement: The Case of DIYBio in Yogyakarta, Indonesia
Thesis PROPOSAL Defense Presentation - March 26 Hermes Huang
This is the Thesis Proposal Defense Presentation by Hermes Huang titled
Analyzing Impacts of Networks within the Maker Movement: The Case of DIYBio in Yogyakarta, Indonesia
Descriptive Research Design - Techniques and TypesSundar B N
This ppt includes Introduction to Descriptive Research, Meaning of Descriptive Research Design and Methods used in Descriptive Research, Types of Descriptive Research and DIFFERENCE B/W EXPLORATORY AND CONCLUSIVE RESEARCH.
Subscribe to Vision Academy
https://www.youtube.com/channel/UCjzpit_cXjdnzER_165mIiw
Introduction to quantitative and qualitative researchLiz FitzGerald
This presentation, delivered in an Open University CALRG Building Knowledge session, gives a preliminary introduction to both quantitative and qualitative research approaches. There has been widespread debate when considering the relative merits of quantitative and qualitative strategies for research. Positions taken by individual researchers vary considerably, from those who see the two strategies as entirely separate, polar opposites that are based upon alternative views of the world, to those who are happy to mix these strategies within their research projects. We consider the different strengths, weaknesses and suitability of different approaches and draw upon some examples to highlight their use within educational technology.
Surveys that work: using questionnaires to gather useful data, November 2010Caroline Jarrett
This presentation to the 22nd Australasian Computer-Human Interaction Conference, OZCHI 2010, compares survey processes and looks at some of the detail of designing surveys – including how to avoid survey error.
Descriptive Research Design - Techniques and TypesSundar B N
This ppt includes Introduction to Descriptive Research, Meaning of Descriptive Research Design and Methods used in Descriptive Research, Types of Descriptive Research and DIFFERENCE B/W EXPLORATORY AND CONCLUSIVE RESEARCH.
Subscribe to Vision Academy
https://www.youtube.com/channel/UCjzpit_cXjdnzER_165mIiw
Introduction to quantitative and qualitative researchLiz FitzGerald
This presentation, delivered in an Open University CALRG Building Knowledge session, gives a preliminary introduction to both quantitative and qualitative research approaches. There has been widespread debate when considering the relative merits of quantitative and qualitative strategies for research. Positions taken by individual researchers vary considerably, from those who see the two strategies as entirely separate, polar opposites that are based upon alternative views of the world, to those who are happy to mix these strategies within their research projects. We consider the different strengths, weaknesses and suitability of different approaches and draw upon some examples to highlight their use within educational technology.
Surveys that work: using questionnaires to gather useful data, November 2010Caroline Jarrett
This presentation to the 22nd Australasian Computer-Human Interaction Conference, OZCHI 2010, compares survey processes and looks at some of the detail of designing surveys – including how to avoid survey error.
A questionnaire is a research instrument consisting of a series of questions and other prompts for the purpose of gathering information from respondents.
The first part of a workshop on user experience surveys. Topics: (1) how to improve the questions in surveys and (2) how to assess UX using a survey.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Ten tips for creating a better survey - questions, process, and testing. Presented at UX Bristol, 15th July 2011.
This is an updated version of '10 tips for a better survey' presented at STC2011.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Ten tips for surveys: on questions, process, and testing your survey.
Books mentioned are listed here: http://rosenfeldmedia.com/uxzeitgeist/lists/cjforms/10-tips-for-a-better-survey-stc2011
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
UXPA 2023: UX research: Optimizing collaboration with project research sponsorsUXPA International
UX researchers can deliver more value by optimizing how they work with research sponsors at two key stages of a study: defining study questions and delivering results. When defining study questions (i.e., scoping and framing the study), researchers can improve upon initial input from sponsors by (1) enlarging the problem frame and (2) refining the questions posed to study participants. When delivering results, researchers can use two tactics: (1) preserving freedom of action and (2) adding breathing room between findings and recommendations. The recommended practices in this talk arose my idiosyncratic reflections and solutions to challenges I’ve encountered in conducting UX research with project teams. These practices have been validated in numerous engagements, and shared informally with colleagues in multiple organizations.
What Do Users Really Think? Surveying Users About Your Help Contentpatricia_gale
You explore the product. You interview SMEs. You write until your fingers cramp. You polish. You publish. And…then what?
Do users use your beautiful documentation? Do they like it? Do they find it useful? How do you know? Ask them! Learn how to conduct a user survey to understand customer satisfaction with your learning content. Who should attend: Technical communicators of all stripes who want to understand what users think of their content, with the goal of improving the content, its findability, and/or usability.
An evidence-based model to enhance programme-wide assessment using technology: TESTA to FASTECH . Presented by Tansy Jessop and Yaz El-Hakim (University of Winchester) and Paul Hyland (Bath Spa University). Facilitated by Mark Russell (University of Hertfordshire).
Jisc conference 2011
In this workshop for the Virtual SDinGov 2024 , Caroline takes participants through two sets of guidelines in search of advice on how to make a single forms question accessible. She then introduces her own question protocol as a method of scrutinising and improving any question.
A presentation for the the Content Wrangler's coffee and content session on how to design and run surveys and gain actionable insights from the survey data.
Some thoughts on good survey design delivered to students at Olin College of Engineering. Caroline's talk covers her survey process, survey goals and focusing on a specific decision, sample and sampling error, ditching rating scales, and losing fear of open answers.
The Phylogenetic Tree in forms design - making forms work for complex academ...Caroline Jarrett
How can we guide busy academics in specialist fields through application processes that are complex, vary greatly depending on the funder, and always seem to be extra urgent? Especially when the stakes are high: awards can be in the millions, and research income is important to fund work that we can all benefit from.
For this year's HE Connect conference, Cambridge University Senior Product Manager Karen Fernandes and forms expert Caroline Jarrett reflected on how current work at Cambridge, and government forms patterns, can help (or hinder) this sort of multi-person, multi-challenge process.
In this half day workshop for ~WebExpo2023 Caroline Jarrett shares four ways to improve your survey so that you get plenty of useful responses.
Goals: Ruthlessly focus your survey on an immediate decision.
Sample: Write an invitation that makes people want to answer.
Questions: Ditch the rating scales.
Responses: Lose your fear of open answers.
Two ways to improve your survey, webinar for Delib 2023.pptxCaroline Jarrett
In this webinar for Delib, Caroline shows you how to get better results from shorter, more frequent surveys - with a special emphasis on local government and the requirement to run statutory consultations. Understanding and identifying the Most Crucial Question and making space for the Burning Issue are both helpful techniques for creating shorter more focused surveys.
Did you love the form that you filled in most recently? Or did you hit some problems? Most of us find all sorts of small or major problems with lots of the forms we are forced to use.
In this talk for #WebExpo2023, Caroline turns that around. She points out the ways in which not fixing your forms is costing your organisation a lot of money. She then goes on to share plenty of practical tips for making improvements that will enable people to successfully complete your forms.
Two ways to improve your surveys: the Most Crucial Question and the Burning I...Caroline Jarrett
In this webinar for product managers, Caroline introduces two key concepts from her book on surveys: identifying the most crucial question as part of getting clear on your goals, and allowing respondents to tell you the things that they want to - their burning issue. The webinar was organised by Productboard and held on March 30, 2023.
In this member call for Boye & Co Caroline takes participants through her process for expert reviews of forms. She also shares some of her top tips for making them easier to use and more effective.
What is a service designer SDinGOV 22 with all stickies.pptxCaroline Jarrett
In this case study for the 2022 Service Design in Government conference Caroline challenges people to think about their own definitions and shares her own - which is based on her three-layer model for creating good forms.
Helping teenage boys to become responsible adults.pptxCaroline Jarrett
Teenage boys use our services but many of us know little about them. In this session, Bukola (Kiki) Jolugbo and Caroline Jarrett shared some facts about teenage boys and some principles for helping them to become responsible adults.
Some thoughts on surveys: Boye and Company member conference callCaroline Jarrett
Slides from a short presentation on creating effective surveys. The event was a conference call for members of a community network organised by Janus Boye of Boye & Company.
Plain language to improve your survey houston 2022Caroline Jarrett
Plain language skills are vital for surveys - and especially to writing good questions and creating them for your survey audience. This presentation was prepared for the University of Houston's 8th Biannual Forum on Plain English, 24 February 2022.
Surveys that work:training course for Rosenfeld Media, day 1Caroline Jarrett
Surveys seem easy: anyone can throw together a few questions, send them out, and hope that they are rewarded with a decent response. But we’ve all seen examples of poorly conceived surveys that couldn’t possibly deliver real insights for the organisation that sponsored them.
This highly participative three-session training - arranged by Rosenfeld Media as part of its Virtual Training with UX Industry Leaders programme - takes you through the whole process of creating an effective survey, from defining a goal through analysis of data and creating a presentation.
These slides come from day 1 of the course: goals and sample.
Surveys that work: training course for Rosenfeld media, day 2Caroline Jarrett
Surveys seem easy: anyone can throw together a few questions, send them out, and hope that they are rewarded with a decent response. But we’ve all seen examples of poorly conceived surveys that couldn’t possibly deliver real insights for the organisation that sponsored them.
This highly participative three-session training - arranged by Rosenfeld Media as part of its Virtual Training with UX Industry Leaders programme - takes you through the whole process of creating an effective survey, from defining a goal through analysis of data and creating a presentation.
These slides come from day 2 of the course: questions, questionnaire and fieldwork
Surveys that work: training course for Rosenfeld Media, day 3 Caroline Jarrett
Surveys seem easy: anyone can throw together a few questions, send them out, and hope that they are rewarded with a decent response. But we’ve all seen examples of poorly conceived surveys that couldn’t possibly deliver real insights for the organisation that sponsored them.
This highly participative three-session training - arranged by Rosenfeld Media as part of its Virtual Training with UX Industry Leaders programme - takes you through the whole process of creating an effective survey, from defining a goal through analysis of data and creating a presentation.
These slides come from day 3 of the course: responses and reports.
Surveys that work: an introduction to the Survey Octopus and Total Survey ErrorCaroline Jarrett
A presentation for Harvard University's User Research Community on some of the key issues in creating effective surveys, including: why run a survey, writing good questions, statistical significance and how to avoid errors.
Expert Accessory Dwelling Unit (ADU) Drafting ServicesResDraft
Whether you’re looking to create a guest house, a rental unit, or a private retreat, our experienced team will design a space that complements your existing home and maximizes your investment. We provide personalized, comprehensive expert accessory dwelling unit (ADU)drafting solutions tailored to your needs, ensuring a seamless process from concept to completion.
Hello everyone! I am thrilled to present my latest portfolio on LinkedIn, marking the culmination of my architectural journey thus far. Over the span of five years, I've been fortunate to acquire a wealth of knowledge under the guidance of esteemed professors and industry mentors. From rigorous academic pursuits to practical engagements, each experience has contributed to my growth and refinement as an architecture student. This portfolio not only showcases my projects but also underscores my attention to detail and to innovative architecture as a profession.
Maximize Your Content with Beautiful Assets : Content & Asset for Landing Page pmgdscunsri
Figma is a cloud-based design tool widely used by designers for prototyping, UI/UX design, and real-time collaboration. With features such as precision pen tools, grid system, and reusable components, Figma makes it easy for teams to work together on design projects. Its flexibility and accessibility make Figma a top choice in the digital age.
2. Agenda Introduction and getting started with surveys
Surveys in practice:
• Working on questions
• A practical survey process
Surveys in theory:
• What the experts tell us about an ideal process
• The four types of survey error
• Some tips from the experts on how to avoid error
2
3. Write down your answers to this questionnaire
1. How many surveys have you run?
NONE 1 to 5 6 to 10 more than 10
2. What is your top tip for a better survey, based on
experience of writing or answering?
__________________________________
__________________________________
3
4. Now try it as an interview
1. How many surveys have you run?
NONE 1 to 5 6 to 10 more than 10
2. What is your top tip for a better survey, based on
experience of writing or answering?
__________________________________
__________________________________
4
5. Now tell each other a story about your
experience relevant to these questions
1. How many surveys have you run?
NONE 1 to 5 6 to 10 more than 10
2. What is your top tip for a better survey, based on
experience of writing or answering?
__________________________________
__________________________________
5
6. Now let’s share data and stories
• What numbers have we collected?
• What open answers?
• What stories?
• Which of the three little surveys was the best?
6
8. We support the MS Society and want to help
• MS = Multiple Sclerosis
• Julie Howell
– Top PR person and accessibility campaigner
– Runs the MS online community ‘Jooly’s Joint’ (since 1995)
– Lives with MS
8
11. Agenda Introduction and getting started with surveys
Surveys in practice:
• Working on questions
• A practical survey process
Surveys in theory:
• What the experts tell us about an ideal process
• The four types of survey error
• Some tips from the experts on how to avoid error
11
12. Write good questions
• Mix question types: closed and open
• Use appropriate, unambiguous language
• Avoid leading questions
• Present one question at a time
• Keep questions concise
• Keep positive; negatives are harder to understand
12
13. Any problems with these questions
from our example?
Level 1 (easier): Find the problems
Level 2 (harder): Write better questions
13
14. The approximate curve of forgetting
Perfect
recall
Day
Gone
Major life event
Noticeable, occasional
Unremarkable,
repetitive
Week Month YearHour
14
15. Anatomy of a range question
Please rate the importance of ease of use:
Very high
Quite high
Neutral
Low
Very low
No answer
Other answer / comments
End
points
Topic
Categories
“No comment”
category
Label
Statement or
question
Response
16. Likert had several different types
of question in his scales
Likert, Rensis. (1932). A Technique for the Measurement of Attitudes.
Archives of Psychology, 140, 1–55. 16
17. You can find an academic paper to support
almost any number of points in a range
• Krosnick and Presser refer to ~87 papers on ranges
Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design.
Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier.
17
18. Two options that are probably OK
Best basic choice:
5 points + “no answer”
If you’re sure the user has an
opinion: 4 points + comment
18
20. Agenda Introduction and getting started with surveys
Surveys in practice:
• Working on questions
• A practical survey process
Surveys in theory:
• What the experts tell us about an ideal process
• The four types of survey error
• Some tips from the experts on how to avoid error
20
22. A basic survey process
Goals
• Establish
your goals
for the
survey
Questions
you need
answers to
Users
• Talk to users
about the
topics in
your survey
Questions
users can
answer
Build
• Write
questions
• Build the
questionnaire
Questionnaire
Deploy
• Run the
survey from
approach to
follow-up
Data
Analyse
• Analyse the
results
Insight
22
23. Goals
• Establish
your goals
for the
survey
What do you want to know?
Why do you want to know?
What will you do with the answers?
23
24. Our example: MS Society on diversity.
Write a short goal for the survey
24
25. Goals Users
• Talk to users
about the
topics in
your survey
• Who are they?
• How will you find them?
• Do they want to answer your questions?
• Do they understand your questions?
25
26. Users
Response relies on
effort, reward, and trust
Trust
Perceived
effort
Perceived
reward
Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability”
inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
26
28. Does your household have access
to running water?
Picture credits: James Mooney, Caroline Jarrett 28
29. Our example: MS Society
What does this question mean to you?
Any ideas about how to ask it better?
29
30. Goals Users Build
• Write
questions
• Build the
questionnaire
• Keep to one topic at a time
• Start with questions that are interesting
• Organize the questions to grab the user’s attention
• Minimize requests for personal information
30
31. Sorting out the topics: group these ones
1. What was your total family income in 2009?
2. Do you like to play golf?
3. What is your opinion on global warming?
4. Are you married?
5. Which political party does the best job of promoting economic growth?
6. How many times have you gone bowling in the last year?
7. What is your political party preference?
8. Do you favour or oppose higher tax on fuel as a measure
to reduce environmental pollution?
9. What is your occupation?
10.Please describe your favourite recreational activity.
11.How old are you?
Adapted from Dillman, D.A., Smyth, J.D. and Christian, L.M. (2009)
Internet, Mail and Mixed Mode Surveys: The Tailored Design Method 31
32. Goals Users Build Deploy
• Run the
survey from
approach to
follow-up• (Optional pre-notice?)
• Send the survey with token
incentive
• Send a reminder message (no
further incentive)
• Send a second copy of the
survey
• Thank participants
32
34. Goals Users Build Deploy Analyse
• Create
summaries and
comparisons
• Present the
results
• Tidy up the data
• Decide what to do about “item non-response”
• Remove the whole entry?
• Use the partial entry?
• Calculate an “inferred value”?
34
35. Analyse Compile responses
$ # answers
0 62
50 34
100 131
150 25
200 57
250 32
300 31
500 84
1000 44
2000 17
3000 8
5000 7
$250/class-up to 3 classes. $150 for tests-up to 2
depends upon my employers contribution
$250? initial, plus $50? each 'renewal'
42
$30 - unless my company agreed to pay it ;-)
don't know-I would want my emp'er to pay
$300 first time, $100 for renewal
Have no basis for comparison.
$300 for a 5-year certification
I can't say-depends entirely on the certification
35
39. Analyse Publish results - gently
• Don’t surprise people with bad news
• Make sure publication is timely
• Keep reports short
• It’s OK to have some gaps
in the results, “more work
needed”
39
40. Agenda Introduction and getting started with surveys
Surveys in practice:
• Working on questions
• A practical survey process
Surveys in theory:
• What the experts tell us about an ideal process
• The four types of survey error
• Some tips from the experts on how to avoid error
40
41. A basic survey process
Goals
• Establish
your goals
for the
survey
Questions
you need
answers to
Users
• Talk to users
about the
topics in
your survey
Questions
users can
answer
Build
• Write
questions
• Build the
questionnaire
Questionnaire
Deploy
• Run the
survey from
approach to
follow-up
Data
Analyse
• Analyse the
results
Insight
41
42. A complete process
Goals
• Establish
your goals
for the
survey
• Decide what
you will do
based on the
results
Users
• Talk to users
about the
topics in
your survey
• Design your
sampling
strategy
Build
• Write
questions
• Build the
questionnaire
• Usability test
• Pilot the
survey
Deploy
• Run the
survey from
approach to
follow-up
• Monitor your
response
rate
Analyse
• Analyse the
results
• Cross-check
the results
42
43. Typically, an expert will focus on just one topic
Goals Users Build
• Write
questions
• Build the
questionnaire
• Usability test
Deploy Analyse
Couper, M. (2008). “Designing effective Web surveys”.
Cambridge, Cambridge University Press.
43
44. Another example:
topics discussed by Dillman et al (2008)
Goals Users
• Talk to users
about the
topics in
your survey
• Design your
sampling
strategy
Build
• Write
questions
• Build the
questionnaire
• Usability test
• Pilot the
survey
Deploy
• Monitor your
response
rate
Analyse
Dillman, D. A., Smyth, J.D., Christian, L. M. (2008)
Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method
44
45. A tweet with a survey in it
Sunday 5th September
45
47. There are four types of survey error
• Sampling:
– Some respondents get left out; the ones left in are somewhat variable
• Non-response:
– The people who respond are different to the people who do not
• Coverage:
– Some of the population is not included in the sampling frame
• Measurement:
– Asking the wrong question, using the wrong technique
Groves, R.M. (2004) Survey errors and survey costs
47
48. If you have variable data,
you will have sampling error
Jarrett, C. (2011) “What is a confidence interval
and why would I want one?” http://www.uxmatters.com 48
49. Avoid non-response error
with a better response rate
• More respondents means fewer who do not respond:
– Less follow-up
– Less opportunity for them to share factors in common
(but different to the respondents)
– Statistical agencies like response rates 50%+
• BUT:
– A low response rate may be independent of any factor
– About 1% of people just like answering questions
– Sample representativeness is more important than raw response rate
50%+49
50. Dillman recommends mixed-mode surveys
to avoid coverage error
• Telephone surveys have coverage problems:
– Unlisted numbers
– ‘Do not call’ numbers
– Cell phone-only households
• Mail surveys have coverage problems:
– New delivery points
– Shared delivery points
• Internet surveys have coverage problems:
– People without internet access
– No directory of email addresses
– Spam filters
@
50
51. Adding steps to avoid coverage, sampling, and
non-response error
Users
• Design your
sampling
strategy
Deploy
• Monitor your
response rate
Analyse
• Cross-check
the results
Goals
• Establish
your goals
for the
survey
• Find out
what users
want to tell
you
Build
• Write
questions
• Build the
questionnaire
• Run the
survey from
approach to
follow-up
• Analyse the
results
51
52. Avoid measurement error by being careful
• Have clear goals for your survey
• Make sure your users want to answer your questions
• Keep testing everything, all the time
• Make sure that you analyse the data from your pilot
52
53. Adding steps to avoid measurement error
Goals
• Decide what
you will do
based on the
results
Build
• Usability test
• Pilot the
survey
• Establish
your goals
for the
survey
Users
• Find out
what users
want to tell
you
• Design your
sampling
strategy
• Write
questions
• Build the
questionnaire
Deploy
• Run the
survey from
approach to
follow-up
• Monitor your
response rate
Analyse
• Analyse the
results
• Cross-check
the results
53
54. Challenging the theory
• How many steps are practical?
• How much time do you have for a survey?
• Which of those things do you really do?
• Which might you try?
54
55. Acknowledgement and references
• Thanks to Karen Bachmann
– Workshop started as one that we presented together in 2002 and 2003
• Books on survey design
– Dillman, D.A., Smyth, J.D. and Christian, L.M. (2009)
Internet, Mail and Mixed Mode Surveys: The Tailored Design Method
• Very practical, informed by many years’ research. Written in a straightforward style.
• The earlier editions (1999 and 2007) are also good and you may be able to get one
cheaply second-hand
– Gilham, Bill (2008) "Developing a questionnaire" Continuum Books
• Thin, clear and practical book that takes you through the survey process.
• My edition is the 2000; there is a newer one in 2008 which I’m sure will be even better
– Schuman, H. and Presser, S. (1996) “Questions and Answers in Attitude
Surveys: Experiments on Question Form, Wording, and Context”
• Thorough analysis of all the ways in which questions can be slippery, based on many
years’ research.
55
56. Other references
• Krosnick, J. and Presser S. (2009) “Question and Questionnaire
Design” in Handbook of Survey Research (2nd Edition) James D.
Wright and Peter V. Marsden (Eds).
– http://comm.stanford.edu/faculty/krosnick/Handbook%20of%20Survey%20Resea
rch.pdf
– Literature review, comprehensively referenced
• Barnum, C. M. (2011) “Usability Testing Essentials: Ready Set, Test”
Morgan Kaufmann
– Thorough but approachable book on how to do usability testing
• Groves, R. M. (2004) “Survey errors and survey costs” Wiley
– More detail about this book here:
http://www.rosenfeldmedia.com/books/survey-
design/blog/survey_book_of_the_month_may_2/
56