Preference and Desirability Testing: Measuring Emotional Response to Guide Design

1.
Preference and DesirabilityTesting: Measuring Emotional Response to Guide DesignMichael HawleyChief Design Officer, Mad*Pow@hawleymichaelPaul DoncasterSenior User Experience Designer, Thomson Reuters

2.
AgendaWhy we shouldcareWhy it’s not always as simple as asking:“Which option do you prefer?”Methods to considerCase Study: Greenwich HospitalCase Study: WestlawNextSummary/Comparison

3.
Why we shouldcare3

4.
Impressions Count

5.
Visceral Emotions

6.
6Fogg’s Behavior ModelCoremotivators include:Pleasure/pain

7.
Hope/fear

8.
Acceptance/rejectionhttp://www.behaviormodel.org/

9.
7Positioning Desirability Studieshttp://www.xdstrategy.com/2008/10/28/desirability_studies/

10.
“Which do youprefer?”8

11.
Quantity, granularity breedapathy

12.
Poor articulation“It remindsme of…”“It’s nice and clean.”“There’s just something about it . . .”“I ordinarily don’t like red, but for some reason it works here . . .”“It’s better than the other ones.”

13.
What Stakeholders ShouldCare About“We should go with design C over A and B, because I feel it evokes the right kind of emotional response in our audience that is closer to our most important brand attributes.”

14.
Methods to Consider12

15.
Present three differentconcepts or ideas to participants, and ask them to identify how two of them are different from the third and why.13Triading

16.
Broad, experience-based questionnaires,that also include questions relating to visual appeal and aestheticsSUS (System Usability Scale)

17.
QUIS (Questionnaire forUser Interface Satisfaction)

18.
WAMMI (Website Analysisand Measurement Inventory)14Qualitative Questionnaires

19.
Show participants auser interface for a very brief moment, then take it away. Participants recall their first impression, then moderator probes for meaning.Helpful for layout decisions, prominence of content, labels

20.
www.fivesecondtest.com15Attention designers: Youhave 50 milliseconds to make a good first impressionQuick Exposure Memory Tests

21.
Electroencephalography (EEG): Brainactivity

22.
Electromyography (EMG): Musclesand ExcitementElectrodermal Activity (EDA): Sweat, Excitement

23.
Blood Volume Pressure(BVP): Arousal

24.
Pupil Dilation: Arousaland Mental Workload

25.
Respiration: Negative Valenceor Arousal16Physiological and Neurological

26.
17PrEmo Emotional MeasurementDr.Pieter Desmet, Technical University of Delfthttp://www.premo-online.com

27.
18http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.docProduct Reaction Cards

28.
Case Study: GreenwichHospital19

29.
Determine intended brandattributes (and their opposites)20Product Reaction Cards: Before You BeginLeverage existing marketing/brand materialsAlternatively, stakeholder brainstorm to identify key brand attributes/descriptors using full list of product reaction cards as a startTip: “If the brand was a person, how would it speak to your customers?”

30.
MethodologyInclude 60/40 splitof positive and negative wordsTarget 60 words, optimized to test brandSimple question: “Which of the following words do you feel best describe the site/design/product (please select 5):”One comp per participant, or multiple comps per participant (no more than 3)ParticipantsQualitative: Paired with usability testingQuantitative: Target minimum of 30 per option if possible21Product Reaction Cards: Conducting

31.
22Process - AnalyzingCalculatepercentage of positive and negative attributes per designVisualize overall sentiment of feedback using “word clouds” (see wordle.net)68% Positive32% Negative

32.
Align the websitewith the character of the Hospital

33.
Update the siteafter nearly 10 years

34.
Counter impressions thatGreenwich is more than just maternity and elder care

35.
Communicate that theyare long-standing members of the community23Case Study: Greenwich Hospital Website Redesign

36.
24Case Study: GreenwichHospital Website Redesign 3 visually designed comps

37.
50 people reactedto each comp (quantitative) via survey

38.
Additional feedback obtainedvia participant interviews (qualitative)Survey QuestionsHello, I am requesting feedback on a website I am working on. Your answers let me know if the site is conveying the right feel. 1. What are your initial reactions to the web site? 2. Which of the following words best do you feel best describe the site (select 5):

39.
25Three Different VisualDesigns

40.
26Results: Concept 112%Negative88% Positive

41.
27Results: Concept 287%Positive13% Negative

42.
28Results: Concept 35%Negative95% Positive

43.
Mix of qualitativeand quantitative is key

44.
Qualitative helps providecolor to the results

45.
Quantitative resonates withstakeholders and executives

46.
Position results asone form of input to decision-making process, not declaring a “winner”

47.
Simple, cost-efficient wayto assess audience’s emotional response to a design29Lessons Learned

48.
Case Study: WestlawNextUXResearch Team:Paul DoncasterDrew DrentlawShannon O’BrienBill QuieNovember Samnee30

50.
for Phase 1Use large sample sizes to establish a design “baseline,” from which to advance the design direction in subsequent iterations

51.
Isolate preference trendsfor specific page design aspects

52.
Determine tolerance formanipulation of the site “brand”

53.
Maintain tight securityGoals

54.
Sessions were heldin 4 cities over 5 daysSeattleDenverMemphisMinneapolis-St. Paul4 sessions were held per day, with a maximum of 25 participants per session1.5 hours allotted per study, most participants finished in less than 1 hour319 participants successfully completed their sessionsPhase 1: Logistics & Execution

55.
Participants completed thestudy at individual workstations at their own paceAll workstations included a 20” monitor, at 1024x768 resolutionPhase 1: Logistics & ExecutionMemphis, TN, May 2009

56.
Brief review ofWestlaw critical screens Positive/negative word selection to describe Westlaw 35Positive/negative product descriptors

57.
Each set ofElement variations were viewed in full screenParticipant selects “top choice” by dragging a thumbnail image to a drop area36Homepage: Design Elements

58.
37

59.
Homepage: Design Elements(1)All options viewed in full screenParticipant selects “top choice” by dragging a thumbnail image to a drop area

62.
Visual Weight (6options)Use of Imagery (8 options)Components (4 options)Search Area (4 options)Palette (10 options)Homepage: Design Elements

63.
19 HP designsviewed in full screen (randomized)All 19 options are presented again; participant assigns a rating using a 10-point slider.Top 5 and Bottom 2 choices are positioned in order of rating values on one long, scrollable page. Next to each design displayed, rates key aspects for each design on a 5-point scaleHomepage: Design Gallery

69.
Repeat the processfor Results List design:New Results ListDesign Elements

70.
Column Collapsing (4options)

71.
Column Separation (2options)

72.
Theme/Color (8 options)

73.
Design Gallery

74.
14 Results Listsdesigns (randomized)

75.
Key Aspects Rated

76.
Color scheme

77.
Global Header

78.
Summary and Excerpt(list contents)

79.
Filters design (leftcolumn)

80.
Overall look andfeelRepeat the process for Document Display design:New Document DisplayDesign Elements

81.
Tabs vs. Links(4 options)

82.
Background Separation (4options)

83.
Margin Width (3options)

84.
Font Size (12options)

85.
Locate (2 options)

86.
Design Gallery

87.
9 Document Displaydesigns (randomized)

88.
Key Aspects Rated

89.
Color scheme

90.
Layout of content

91.
Text formatting

92.
Overall look andfeel“Based on the designs I’ve liked most today . . .” 50Positive/negative design descriptors

93.
Results were analyzedacross 8 different sample filtersJob Title

94.
Age

95.
Testing Location

96.
Years of Experience

97.
Hours per WeekResearching

98.
Organization Size

99.
Role (decision-maker status) The top picks were surprisingly consistent across all of the ‘Top 5’ lists analyzedHigh-level Results

100.
52Top Homepage Designsby Job Title Job Title. Top 5 out of 19 possible. Overall (319)Associate (189)Librarian (37)Partner (81)Solo Practitioner (5)11111HP16HP1HP16HP5HP152HP8HP15HP15HP6HP16222233333HP8HP8HP16HP8HP104HP15HP14444HP5HP7HP555555HP10HP13HP8HP19HP14

101.
Home Page (19)HP16& HP15 designs consistently placed in the Top 5 across all filtersResults List (14)RL4 consistently placed in the Top 3 across all sample filters, and was the #1 choice for 80% of all participantsDocument Display (9)DD3 placed in the Top 5 across all sample filters and was the #1 choice for 77% of all participantsPhase 1: High-level Results

102.
Note, participants wereasked to describe the current Westlaw before being shown the new designs. 54Phase 1: Word Selection Results

103.
5 design themeswere derived from post-session discussions“New design(s) are better than current Westlaw”

104.
“Clean and Fresh”

105.
“Contrast is Important”

106.
“Prefer Westlaw Blue”

107.
“No Big FontsPlease”The study narrowed the list of potential designs, and we better understood what design elements that Westlaw users liked and disliked.Phase 1: High-level Results

108.
56Phase 2: September2009Kansas City, MO, Sept 2009

109.
GoalsRefine preferences forselected design directions

110.
Understand users personalreasons for liking their preferred choices

111.
Get closure onother design options for online and printed content

112.
Sustain tight securityToolSameas in Round 1, with some minor revisions to accommodate specialized input Phase 2: September 2009

113.
MethodView, Rate, andPick Top Choice for Homepage (3 options)Result List (2 options)Document Display (2 options)“Why?”Simple preference selection for two unresolved UI design issuesCiting References: Grid display or List display?Out of Plan Indication design (6 options)Type formatting preferences for 3 different content typesFont FaceFont SizeMargin Width Phase 2: September 2009

114.
Logistics3 cities (Philadelphia,Kansas City, Los Angeles)1 Day226 participantsAnalysisFilters (8 categories) were used to score the designs for each visual preferenceResultsClear choices for top designs in each of all categories“Why” feedback shed new light on designs under consideration and helped focus “homestretch” design activitiesPhase 2: September 2009

115.
Home Page (3)HP3ranked #1in 94% of filter groups (54% of total participants) Results List (2)RL5 ranked #1in 97% of filter groups (58% of total participants) Document Display (2)DD7 ranked #1in 94% of filter groups (61% of total participants)Phase 2: High-level Results

116.
The main concernsregarding Homepage Design HP3Search Box Too smallHow do I do a Terms-and-Connectors search?Browse SectionHow do I specify multiple or specific search content?Poor organizationPoor labelNeed access to “often-used” contentNeed better access to help61Participant Comments: Homepage

117.
GoalsGet feedback onbranding options from decision makers and those who influence purchase of the productGet closure on final outstanding design issuesToolSame as in Rounds 1 & 2, with some minor revisions to accommodate specialized input Phase 3: December 2009

118.
MethodWordmark/BrandingView wordmark colorcombinations and design elements against different backgrounds, pick top choice and provide commentsMake a final “Top Choice” from all selectionsSimple preference selection for outstanding UI design issuesHeader Space: Tile or No Tile?Notes DesignLocation: Inline or Column?State: Open or Closed?Headnote Icon design (4 variations)Phase 3: December 2009

119.
What color combinationdo you prefer? Please rank the 4 combinations below according to your preferences. To rank, click and drag an item from the left to a box on the right.Your Most Liked1234Your Least Liked

120.
Logistics3 cities (Seattle,Denver, Boston) 1 Day214 participantsAnalysisSimple preference, no advanced filters ResultsDecision-makers confirmed that critical brand elements should be retainedPhase 3: December 2009

121.
Decision Makers’ Picks(1 of 2)

122.
67

123.
MeasuringEmotional Responseto GuideDesignWhy it succeededQuantitative & qualitative data to identify preference trends

124.
“Slicing” across identifiablefilters

125.
Emphasis on “gut-level”reactions

126.
Intolerance for manipulationof product brand

127.
Rapid turnaround ofdata to all stakeholders

128.
Executive

129.
Design

130.
Development69Document DisplayMay 2009Sept2009Feb 2010

131.
At what cost(s)?Weheld off asking “why” until the second round

132.
If we hadasked why in the first round, we might have

133.
avoided some ofinternal design battles

134.
gotten more granularammunition for communicating the design vision to stakeholders

135.
“Need for speed”attained at the cost of detailed analysisRetrospective

136.
Recommendations for anyonethinking of undertaking something like thisProcure a “Matt” to create and administer your tool

137.
Get a goodtechnical vendor for on-site

138.
Report results inas close to real-time as possible on a wiki or other web-pageRetrospective

139.
Summary/Comparison72

140.
Both groups valuedsupport in design decision makingAlign methodology with needs of the projectResearch-inspired, not research-decided73Summary/Comparison

141.
Additional Reading andTools74

142.
Benedek, Joey andTrish Miner. “Measuring Desirability: New Methods for Evaluating Desirability in a Usability Lab Setting.” Proceedings of UPA 2002 Conference, Orlando, FL, July 8–12, 2002. http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.docLindgaard, Gitte, Gary Fernandes, Cathy Dudek, and J. Brown. "Attention Web Designers: You Have 50 Milliseconds to Make a Good First Impression!" Behaviour and Information Technology, 2006. http://www.imagescape.com/library/whitepapers/first-impression.pdfRohrer, Christian. “Desirability Studies: Measuring Aesthetic Response to Visual Designs.” xdStrategy.com, October 28, 2008. Retrieved February 10, 2010. http://www.xdstrategy.com/2008/10/28/desirability_studies75Additional Reading

143.
User Focus. "Measuringsatisfaction: Beyond the Usability Questionnaire." Retrieved February 10, 2010. http://www.userfocus.co.uk/articles/satisfaction.html UserEffect. "Guide to Low-Cost Usability Tools." Retrieved May 12, 2010.http://www.usereffect.com/topic/guide-to-low-cost-usability-toolsTullis, Thomas and Jacqueline Stetson. “A Comparison of Questionnaires for Assessing Website Usability.” Usability Professionals’ Association Conference, 2004.home.comcast.net/~tomtullis/publications/UPA2004TullisStetson.pdfWesterman, S. J., E. Sutherland, L. Robinson, H. Powell, and G. Tuck. “A Multi-method Approach to the Assessment of Web Page Designs.” Proceedings of the 2nd international conference on Affective Computing and Intelligent Interaction, 2007.http:// portal.acm.org/citation.cfm?id=142220076Additional Reading

144.
Five Second Testhttp://fivesecondtest.com/FeedbackArmy http://www.feedbackarmy.comWordlehttp://www.wordle.netPrEmohttp://www.premo-online.com77Additional Tools

Preference and Desirability Testing: Measuring Emotional Response to Guide Design

More Related Content

What's hot

Similar to Preference and Desirability Testing: Measuring Emotional Response to Guide Design

Recently uploaded

Preference and Desirability Testing: Measuring Emotional Response to Guide Design

Editor's Notes