Design tips for surveys UIE 2012
Upcoming SlideShare
Loading in...5
×
 

Design tips for surveys UIE 2012

on

  • 3,660 views

Ideas for extracting the maximum value from a survey that is going to happen anyway

Ideas for extracting the maximum value from a survey that is going to happen anyway

Statistics

Views

Total Views
3,660
Views on SlideShare
3,275
Embed Views
385

Actions

Likes
13
Downloads
107
Comments
4

7 Embeds 385

http://balcostics.com 141
http://www.rosenfeldmedia.com 112
http://rosenfeldmedia.com 112
http://us-w1.rockmelt.com 13
http://www.twylah.com 5
http://www.postano.com 1
https://blackboard.strayer.edu 1
More...

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • @cjforms You make a good point and I agree that the key is providing an incentive that seems commensurate, given the effort required, and creating trust that it will actually be awarded.
    Are you sure you want to
    Your message goes here
    Processing…
  • Thanks Jessica,

    I'm pretty sure that there must have been some not-small incentive research - away from my library right now so that's one for me to try to look up.

    Anecdotally, one of the participants in my survey workshop at OzCHI in 2010 reported that she'd got an excellent response from students in her university by offering a prize draw for an iPad. This is consistent with the idea of a perceived reward because the students knew that the pool of respondents was relatively small, so each student had a decent chance of winning, and also trusted the university to actually do the prize draw and award the prize. We can see a similar phenomenon at conferences where it's common for people to give their business cards up happily for a decent prize draw; I think that might be because they have some confidence that the prize will actually be awarded.
    Are you sure you want to
    Your message goes here
    Processing…
  • Great presentation Caroline, as always.

    Quick question: do you know if there's been any research into a 'not-small' incentive for online surveys? I'm thinking a $10 voucher or entry into $500 prize draw?

    Anything you know about, off the top of your head, would be useful.

    Cheers
    Jessica
    Are you sure you want to
    Your message goes here
    Processing…
  • For resources mentioned in this talk, see my blog post:
    http://www.rosenfeldmedia.com/books/survey-design/blog/survey_seminar_on_28th_februar/
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Design tips for surveys UIE 2012 Design tips for surveys UIE 2012 Presentation Transcript

  • Design tipsfor surveys2012Caroline JarrettUser Interface Engineering / Rosenfeld Media Seminar
  • We need lots of data. Your samples are too small. We‟re going to do a survey. But, but… er, er, … OK, you‟re the bossPlease make it one thatI‟ll enjoy answering 2
  • We need surveys in our toolbox1. You get lots of lovely numbers2. It‟s much easier to manage what you can measure3. They‟re going to do a survey anyway, so let‟s make it a good one And the more of them I do, the more I like them 3
  • Survey = Questionnaire + Process 4
  • Agenda Entice people to take a survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 5
  • Your answers to this survey areimportant for my business. But what‟s in it for me? And I‟m really busy today. 6
  • Response relies oneffort, reward, and trust Perceived Perceived effort reward TrustDiagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” 7inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
  • Reward, trust and effort for this invitation Intrinsic reward - but is it enough?Short is good,but do I trust Who are you?you enough to Will you spam me?believe you? 8
  • We‟ll offer a chance to win $100.That will get lots of responses. But I don‟t believe I‟ll be the winner 9
  • A small immediate payment works best for mail Response rates100% 50% 0% No incentive $300 lottery $2 bill Carlson, 1996 quoted in Dillman, D.A., Smyth, J. D. and Christian, L. M. (2009) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 10
  • „Small immediate payment‟ is difficult online Accessed first page100%50% 0% No incentive Prize draw (2x $2 via Pay Pal $2 via PayPal on $50,2x $25 prize) before survey completion Michael Bosnjak and Tracy L. Tute (2003) “Prepaid and Promised Incentives in Web Surveys : An Experiment” 11 Social Science Computer Review 2003 21: 20
  • Offer a meaningful,Tip 1 immediate reward Picture credit: 12 Flickr - Jessica Wilson (jek in the box)
  • Let‟s look again at trust Who are you? Will you spam me? 13
  • A well-known brand on a well-designed site I definitely know you And you won’t spam me 14
  • Be trustworthy andTip 2 look trustworthy 15
  • I really need theYou know that survey? I want „recommend to ato know where they live friend‟ question We have to ask them about navigation, obviouslyWhat about the new We have to ask aboutgraphics? the content! And finding us … we have to know about that Don‟t forget about the new feature 16
  • I respond to a lot of surveys 89 invitations 67 completed surveys 1387 questions 17
  • They wanted my opinions but it took 6 screensto get to the first opinion question 18
  • Could the BBC trim some of these?• Gender • What do you like about it• Region you live in • What do you dislike about it• Ethnic origin • Your main reason for visiting• Type of Internet the site today connection • And many more• Rate how it looks 19 Image credit: Flickr - Enokson
  • Focus on the questionsTip 3 that deliver the insights that you need the most 20 Image credit: Fraser Smith
  • Remember 1387 questions?This survey contributed 30 of them 21
  • On and on 22
  • On and on and on and on and on I love John Lewis! I shop there online and offline! Don‟t make me feel bad because I didn‟t read your buying guide. Just let me tell you I love you… 23
  • Your users may not be as interested in yourdesign decisions as you are. 24 “Wood for the trees” Image credit: Flickr - Paul & Aline Burland (P – A – S – T)
  • To get a good survey, talkTip 4 to lots of users 25
  • Agenda Entice people to take a survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 26
  • Back to the BBC. After I gave my initialopinions, they asked me some ratings 27
  • Most surveys I see have ratingsfrom 1 to 5 Wouldn‟t it be easier to have just two ? 7 points are more accurate! What about 1 to 10? 28
  • Likert had several different typesof question in his scalesLikert, Rensis. (1932). A Technique for the Measurement of Attitudes.Archives of Psychology, 140, 1–55. 29
  • You can find an academic paper to supportalmost any number of points in a range• Krosnick and Presser refer to ~87 papers on ranges • Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design. Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier• If you‟d like to track down the research yourself: http://comm.stanford.edu/faculty/krosnick/ Handbook%20of%20Survey%20Research.pdf 30
  • Users don‟t care much about the number ofpoints. They care more about the questions.What‟s the differencebetween those questions?This survey is too repetitive. 31
  • Don‟t stress about theTip 5 number of points in your rating scale 32 Picture credit: Flickr - Bill Soderman (BillsoPHOTO)
  • The BBC didn‟t stress about the number ofpoints in the rating scale. This one has 10 33
  • This one has 5. And I never noticed at the time. 34
  • Let‟s have another lookat one of Likert‟s questions Complex clause A rather big commitment Double-barrelled And what are the programs anyway?Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. 35Archives of Psychology, 140, 1–55.
  • Before- and after- for Likert‟s question?17a. The United States should join in the humanitarian work of the League of Nations17b. The United States should join in the economic work of the League of Nations 36
  • The BBC also asked me a complex question 37
  • Let‟s split it and simplify itIt is easy to select products on this serviceIt is easy to find suppliers on this service 38
  • Refine questions soTip 6 they‟re crystal clear Present one question at a time Use plain language http://www.plainlanguage.gov/howto/index.cfm  39
  • How much do you care about the economicprograms of the League of Nations?Likert, Rensis. (1932). A Technique for the Measurement of Attitudes.Archives of Psychology, 140, 1–55. 40
  • OK, so how much do you care about… 41
  • Two ways of asking the same question:One of them calls for imaginationWhat is your favourite meal? Imagine you are on death row and have to choose your last meal• Steak and chips • Scallops with black pudding and cream, rib eye steak with chips and a dolce latte cream sauce; stinking bishop (cheese) with 1960 port (year of my birth). Wine would have to be Chateau Lafite 1st Cru Pauillac 2000.Puleston, J. and Sleep, D (2011) “The Game Experiments” 42ESOMAR Congress
  • Could we make the question more interesting?Please rate the overallinformation contenton this web site Boring 43
  • Could we make the question more interesting?Please rate the overall Story:information content If you could wave aon this web site magic wand and make a new page for this web site, what would it do or say? Game: You‟ve asked us to improve the information content on this web site. Here are some different changes we thought of and what they would cost. If you had $10,000 to spend, which would you choose? 44
  • Cambridgeshire police tried the game idea 45
  • Write interesting questionsTip 7 46 Picture credit: Shutterstock.com
  • Agenda Entice people to take your survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 47
  • Does your household have accessto running water?Picture credits: Flickr - James Mooney; Caroline Jarrett 48
  • “Shared reference”:both sides interpret in the same way Reiss, E. (2000) “Practical Information Architecture: A Hands-On Approach to Structuring Successful Web Sites”Picture credits: Flickr - James Mooney; Caroline Jarrett 49
  • “Shared reference”:both sides interpret in the same way Service area? 50 Photo credit: Flickr - chaptstickaddict
  • Make sure that your usersTip 8 understand your questions in the same way that you do 51
  • Let‟s look at putting the answer on the page. 52
  • Here, the „agree‟ and „disagree‟ have floated 53
  • Not hard to fix.(And let‟s delete excess instructions while we‟re about it.) 54
  • Design the page so peopleTip 9 can answer easily http://www.slideshare.net/cjforms/labels-and-buttons-on-forms 55
  • Agenda Entice people to take your survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 56
  • We got so much great datathat I put every detail inthis report!!!! 57
  • Surveys can get you lots of lovely data 58
  • Cutting out some detail can help a lot 59
  • This graph could be good news or bad newsdepending on the question we ask of it 60
  • This graph could be good news or bad newsdepending on the question we ask of it How are we doing in the 35- 44 age group? Strong response from key target 61
  • This graph could be good news or bad newsdepending on the question we ask of it Disappointing results from seniors How are we doing with the seniors? 62
  • Think about the story theTip 10 data can tell Perceptual Edge has a Graph Design IQ test: http://www.perceptualedge.com/files/GraphDesignIQ.html 63
  • My best ever survey was very simple:needed data, got it, made decision 64
  • “Quality doesn‟t matter” We are focusing our time on technicalities of research, whereas the expectation from clients is actually in a totally different space. The clients are saying, ‘Inspire me, help me to take some transformational action’ Stan Sthanunathan Vice President Marketing Strategy & Insights The Coca-Cola Company“Research”, Oct 2009 http://www.research-live.com/4001230.articleImage credit: http://www.eltiempo.com/gente/ARTICULO-WEB- 65NEW_NOTA_INTERIOR-9736789.html
  • Insight = information that provokes actionIf information doesn’t change our,or our consumers’, behaviorthen we shouldn’t be doing it.Danny Russell,Marketing Strategy Director,BSkyB“Research”, January 2012 66http://www.research-live.com/features/who-are-you-talking-to?/4006763.article
  • Know what decision you‟llTip 11 make based on this survey 67 Image credit: Fraser Smith
  • We need to know oursatisfaction score Hmm… satisfaction has many aspects. Let‟s ask about aesthetics, desirability, navigation…. 68
  • Good news! We‟re rating 5 for aesthetics, 5 for desirability and 5 for navigation!But what aboutsatisfaction? 69
  • Tip 12 Know “The Most Crucial Question” and don‟t forget to ask it Image 70 credit: Shutterstock
  • Agenda Entice people to take your survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 71
  • Remember my definition? Survey = Questionnaire + Process 72
  • Bonus SuccessfulTip Survey = Questionnaire + Process That involves lots of testing 73
  • Agenda Entice people to take your survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 74
  • Caroline Jarretttwitter @cjformscaroline.jarrett@effortmark.co.ukSlides: http://www.slideshare.net/cjformsBlog posts and more resources:http://www.rosenfeldmedia.com/books/survey-design/ 75