Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Preference and Desirability Testing: Measuring Emotional Response to Guide Design<br />Michael Hawley<br />Chief Design Of...
Agenda<br />Why we should care<br />Why it’s not always as simple as asking:“Which option do you prefer?”<br />Methods to ...
Why we should care<br />3<br />
Impressions Count<br />
Visceral Emotions<br />
6<br />Fogg’s Behavior Model<br />Core motivators include:<br /><ul><li>Pleasure/pain
Hope/fear
Acceptance/rejection</li></ul>http://www.behaviormodel.org/<br />
7<br />Positioning Desirability Studies<br />http://www.xdstrategy.com/2008/10/28/desirability_studies/<br />
“Which do you prefer?”<br />8<br />
Quantity, granularity breed apathy<br />
Poor articulation<br />“It reminds me of…”<br />“It’s nice and clean.”<br />“There’s just something about it . . .”<br />“...
What Stakeholders Should Care About<br />“We should go with design C over A and B, because I feel it evokes the right kind...
Methods to Consider<br />12<br />
Present three different concepts or ideas to participants, and ask them to identify how two of them are different from the...
Broad, experience-based questionnaires, that also include questions relating to visual appeal and aesthetics<br /><ul><li>...
QUIS (Questionnaire for User Interface Satisfaction)
WAMMI (Website Analysis and Measurement Inventory)</li></ul>14<br />Qualitative Questionnaires<br />
Show participants a user interface for a very brief moment, then take it away. <br />Participants recall their first impre...
www.fivesecondtest.com</li></ul>15<br />Attention designers: <br />You have <br />50 milliseconds <br />to make a good <br...
<ul><li>Electroencephalography (EEG): Brain activity
Electromyography (EMG): </li></ul>Muscles and Excitement<br /><ul><li>Electrodermal Activity (EDA): Sweat, Excitement
Blood Volume Pressure (BVP): Arousal
Pupil Dilation: Arousal and Mental Workload
Respiration: </li></ul>Negative Valence or Arousal<br />16<br />Physiological and Neurological<br />
17<br />PrEmo Emotional Measurement<br />Dr. Pieter Desmet, Technical University of Delft<br />http://www.premo-online.com...
18<br />http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.doc<br />Product Reaction Cards<br />
Case Study: Greenwich Hospital<br />19<br />
Determine intended brand attributes (and their opposites)<br />20<br />Product Reaction Cards: Before You Begin<br />Lever...
Methodology<br />Include 60/40 split of positive and negative words<br />Target 60 words, optimized to test brand<br />Sim...
22<br />Process - Analyzing<br />Calculate percentage of positive and negative attributes per design<br />Visualize overal...
<ul><li>Align the website with the character of the Hospital
Update the site after nearly 10 years
Counter impressions that Greenwich is more than just maternity and elder care
Communicate that they are long-standing members of the community</li></ul>23<br />Case Study: Greenwich Hospital Website R...
24<br />Case Study: Greenwich Hospital Website Redesign <br /><ul><li>3 visually designed comps
50 people reacted to each comp (quantitative) via survey
Additional feedback obtained via participant interviews (qualitative)</li></ul>Survey Questions<br />Hello, I am requestin...
25<br />Three Different Visual Designs<br />
26<br />Results: Concept 1<br />12% Negative<br />88% Positive<br />
27<br />Results: Concept 2<br />87% Positive<br />13% Negative<br />
28<br />Results: Concept 3<br />5% Negative<br />95% Positive<br />
<ul><li>Mix of qualitative and quantitative is key
Qualitative helps provide color to the results
Quantitative resonates with stakeholders and executives
Position results as one form of input to decision-making process, not declaring a “winner”
Simple, cost-efficient way to assess audience’s emotional response to a design</li></ul>29<br />Lessons Learned<br />
Case Study: WestlawNext<br />UX Research Team:<br />Paul Doncaster<br />Drew Drentlaw<br />Shannon O’Brien<br />Bill Quie<...
for Phase 1 <br /><ul><li>Use large sample sizes to establish a design “baseline,” from which to advance the design direct...
Isolate preference trends for specific page design aspects
Determine tolerance for manipulation of the site “brand”
Maintain tight security</li></ul>Goals<br />
Sessions were held in 4 cities over 5 days<br />Seattle<br />Denver<br />Memphis<br />Minneapolis-St. Paul<br />4 sessions...
Participants completed the study at individual workstations at their own pace<br />All workstations included a 20” monitor...
Brief review of Westlaw critical screens <br />Positive/negative word selection to describe Westlaw <br />35<br />Positive...
Each set of Element variations were viewed in full screen<br />Participant selects “top choice” by dragging a thumbnail im...
37<br />
Homepage: Design Elements (1)<br />All options viewed in full screen<br />Participant selects “top choice” by dragging a t...
Visual Weight (6 options)<br />Use of Imagery (8 options)<br />Components (4 options)<br />Search Area (4 options)<br />Pa...
19 HP designs viewed in full screen (randomized)<br />All 19 options are presented again; participant assigns a rating usi...
Repeat the process for Results List design:<br />New Results List<br /><ul><li>Design Elements
Column Collapsing (4 options)
Column Separation (2 options)
Theme/Color (8 options)
Design Gallery
14 Results Lists designs (randomized)
Key Aspects Rated
Color scheme
Global Header
Upcoming SlideShare
Loading in …5
×

Preference and Desirability Testing: Measuring Emotional Response to Guide Design

11,475 views

Published on

(From UPA 2011-Atlanta) Usability practitioners have a variety of methods and techniques to inform interaction design and identify usability problems. However, these tools are not as effective at evaluating the visceral and emotional response generated by visual design and aesthetics. This presentation will discuss why studying visual design is important, review considerations for preference and desirability testing and present two alternative approaches to user studies of visual designs in the form of case studies.

Published in: Design, Technology
  • Be the first to comment

Preference and Desirability Testing: Measuring Emotional Response to Guide Design

  1. 1. Preference and Desirability Testing: Measuring Emotional Response to Guide Design<br />Michael Hawley<br />Chief Design Officer, Mad*Pow<br />@hawleymichael<br />Paul Doncaster<br />Senior User Experience Designer, Thomson Reuters<br />
  2. 2. Agenda<br />Why we should care<br />Why it’s not always as simple as asking:“Which option do you prefer?”<br />Methods to consider<br />Case Study: Greenwich Hospital<br />Case Study: WestlawNext<br />Summary/Comparison<br />
  3. 3. Why we should care<br />3<br />
  4. 4. Impressions Count<br />
  5. 5. Visceral Emotions<br />
  6. 6. 6<br />Fogg’s Behavior Model<br />Core motivators include:<br /><ul><li>Pleasure/pain
  7. 7. Hope/fear
  8. 8. Acceptance/rejection</li></ul>http://www.behaviormodel.org/<br />
  9. 9. 7<br />Positioning Desirability Studies<br />http://www.xdstrategy.com/2008/10/28/desirability_studies/<br />
  10. 10. “Which do you prefer?”<br />8<br />
  11. 11. Quantity, granularity breed apathy<br />
  12. 12. Poor articulation<br />“It reminds me of…”<br />“It’s nice and clean.”<br />“There’s just something about it . . .”<br />“I ordinarily don’t like red, but for some reason it works here . . .”<br />“It’s better than the other ones.”<br />
  13. 13. What Stakeholders Should Care About<br />“We should go with design C over A and B, because I feel it evokes the right kind of emotional response in our audience that is closer to our most important brand attributes.” <br />
  14. 14. Methods to Consider<br />12<br />
  15. 15. Present three different concepts or ideas to participants, and ask them to identify how two of them are different from the third and why.<br />13<br />Triading<br />
  16. 16. Broad, experience-based questionnaires, that also include questions relating to visual appeal and aesthetics<br /><ul><li>SUS (System Usability Scale)
  17. 17. QUIS (Questionnaire for User Interface Satisfaction)
  18. 18. WAMMI (Website Analysis and Measurement Inventory)</li></ul>14<br />Qualitative Questionnaires<br />
  19. 19. Show participants a user interface for a very brief moment, then take it away. <br />Participants recall their first impression, then moderator probes for meaning.<br /><ul><li>Helpful for layout decisions, prominence of content, labels
  20. 20. www.fivesecondtest.com</li></ul>15<br />Attention designers: <br />You have <br />50 milliseconds <br />to make a good <br />first impression<br />Quick Exposure Memory Tests<br />
  21. 21. <ul><li>Electroencephalography (EEG): Brain activity
  22. 22. Electromyography (EMG): </li></ul>Muscles and Excitement<br /><ul><li>Electrodermal Activity (EDA): Sweat, Excitement
  23. 23. Blood Volume Pressure (BVP): Arousal
  24. 24. Pupil Dilation: Arousal and Mental Workload
  25. 25. Respiration: </li></ul>Negative Valence or Arousal<br />16<br />Physiological and Neurological<br />
  26. 26. 17<br />PrEmo Emotional Measurement<br />Dr. Pieter Desmet, Technical University of Delft<br />http://www.premo-online.com<br />
  27. 27. 18<br />http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.doc<br />Product Reaction Cards<br />
  28. 28. Case Study: Greenwich Hospital<br />19<br />
  29. 29. Determine intended brand attributes (and their opposites)<br />20<br />Product Reaction Cards: Before You Begin<br />Leverage existing marketing/brand materials<br />Alternatively, stakeholder brainstorm to identify key brand attributes/descriptors using full list of product reaction cards as a start<br />Tip: “If the brand was a person, how would it speak to your customers?”<br />
  30. 30. Methodology<br />Include 60/40 split of positive and negative words<br />Target 60 words, optimized to test brand<br />Simple question: “Which of the following words do you feel best describe the site/design/product (please select 5):”<br />One comp per participant, or multiple comps per participant (no more than 3)<br />Participants<br />Qualitative: Paired with usability testing<br />Quantitative: Target minimum of 30 per option if possible<br />21<br />Product Reaction Cards: Conducting<br />
  31. 31. 22<br />Process - Analyzing<br />Calculate percentage of positive and negative attributes per design<br />Visualize overall sentiment of feedback using “word clouds” (see wordle.net)<br />68% Positive<br />32% Negative<br />
  32. 32. <ul><li>Align the website with the character of the Hospital
  33. 33. Update the site after nearly 10 years
  34. 34. Counter impressions that Greenwich is more than just maternity and elder care
  35. 35. Communicate that they are long-standing members of the community</li></ul>23<br />Case Study: Greenwich Hospital Website Redesign <br />
  36. 36. 24<br />Case Study: Greenwich Hospital Website Redesign <br /><ul><li>3 visually designed comps
  37. 37. 50 people reacted to each comp (quantitative) via survey
  38. 38. Additional feedback obtained via participant interviews (qualitative)</li></ul>Survey Questions<br />Hello, I am requesting feedback on a website I am working on. Your answers let me know if the site is conveying the right feel. <br />1. What are your initial reactions to the web site? <br />2. Which of the following words best do you feel best describe the site (select 5): <br />
  39. 39. 25<br />Three Different Visual Designs<br />
  40. 40. 26<br />Results: Concept 1<br />12% Negative<br />88% Positive<br />
  41. 41. 27<br />Results: Concept 2<br />87% Positive<br />13% Negative<br />
  42. 42. 28<br />Results: Concept 3<br />5% Negative<br />95% Positive<br />
  43. 43. <ul><li>Mix of qualitative and quantitative is key
  44. 44. Qualitative helps provide color to the results
  45. 45. Quantitative resonates with stakeholders and executives
  46. 46. Position results as one form of input to decision-making process, not declaring a “winner”
  47. 47. Simple, cost-efficient way to assess audience’s emotional response to a design</li></ul>29<br />Lessons Learned<br />
  48. 48. Case Study: WestlawNext<br />UX Research Team:<br />Paul Doncaster<br />Drew Drentlaw<br />Shannon O’Brien<br />Bill Quie<br />November Samnee<br />30<br />
  49. 49.
  50. 50. for Phase 1 <br /><ul><li>Use large sample sizes to establish a design “baseline,” from which to advance the design direction in subsequent iterations
  51. 51. Isolate preference trends for specific page design aspects
  52. 52. Determine tolerance for manipulation of the site “brand”
  53. 53. Maintain tight security</li></ul>Goals<br />
  54. 54. Sessions were held in 4 cities over 5 days<br />Seattle<br />Denver<br />Memphis<br />Minneapolis-St. Paul<br />4 sessions were held per day, with a maximum of 25 participants per session<br />1.5 hours allotted per study, most participants finished in less than 1 hour<br />319 participants successfully completed their sessions<br />Phase 1: Logistics & Execution<br />
  55. 55. Participants completed the study at individual workstations at their own pace<br />All workstations included a 20” monitor, at 1024x768 resolution<br />Phase 1: Logistics & Execution<br />Memphis, TN, May 2009<br />
  56. 56. Brief review of Westlaw critical screens <br />Positive/negative word selection to describe Westlaw <br />35<br />Positive/negative product descriptors<br />
  57. 57. Each set of Element variations were viewed in full screen<br />Participant selects “top choice” by dragging a thumbnail image to a drop area<br />36<br />Homepage: Design Elements<br />
  58. 58. 37<br />
  59. 59. Homepage: Design Elements (1)<br />All options viewed in full screen<br />Participant selects “top choice” by dragging a thumbnail image to a drop area<br />
  60. 60.
  61. 61.
  62. 62. Visual Weight (6 options)<br />Use of Imagery (8 options)<br />Components (4 options)<br />Search Area (4 options)<br />Palette (10 options)<br />Homepage: Design Elements<br />
  63. 63. 19 HP designs viewed in full screen (randomized)<br />All 19 options are presented again; participant assigns a rating using a 10-point slider.<br />Top 5 and Bottom 2 choices are positioned in order of rating values on one long, scrollable page. Next to each design displayed, rates key aspects for each design on a 5-point scale<br />Homepage: Design Gallery<br />
  64. 64.
  65. 65.
  66. 66.
  67. 67.
  68. 68.
  69. 69. Repeat the process for Results List design:<br />New Results List<br /><ul><li>Design Elements
  70. 70. Column Collapsing (4 options)
  71. 71. Column Separation (2 options)
  72. 72. Theme/Color (8 options)
  73. 73. Design Gallery
  74. 74. 14 Results Lists designs (randomized)
  75. 75. Key Aspects Rated
  76. 76. Color scheme
  77. 77. Global Header
  78. 78. Summary and Excerpt (list contents)
  79. 79. Filters design (left column)
  80. 80. Overall look and feel</li></li></ul><li>Repeat the process for Document Display design:<br />New Document Display<br /><ul><li>Design Elements
  81. 81. Tabs vs. Links (4 options)
  82. 82. Background Separation (4 options)
  83. 83. Margin Width (3 options)
  84. 84. Font Size (12 options)
  85. 85. Locate (2 options)
  86. 86. Design Gallery
  87. 87. 9 Document Display designs (randomized)
  88. 88. Key Aspects Rated
  89. 89. Color scheme
  90. 90. Layout of content
  91. 91. Text formatting
  92. 92. Overall look and feel</li></li></ul><li>“Based on the designs I’ve liked most today . . .” <br />50<br />Positive/negative design descriptors<br />
  93. 93. Results were analyzed across 8 different sample filters<br /><ul><li>Job Title
  94. 94. Age
  95. 95. Testing Location
  96. 96. Years of Experience
  97. 97. Hours per Week Researching
  98. 98. Organization Size
  99. 99. Role (decision-maker status) </li></ul>The top picks were surprisingly consistent across all of the ‘Top 5’ lists analyzed<br />High-level Results<br />
  100. 100. 52<br />Top Homepage Designs by Job Title <br />Job Title. Top 5 out of 19 possible. <br />Overall (319)<br />Associate (189)<br />Librarian (37)<br />Partner (81)<br />Solo Practitioner (5)<br />1<br />1<br />1<br />1<br />1<br />HP16<br />HP1<br />HP16<br />HP5<br />HP15<br />2<br />HP8<br />HP15<br />HP15<br />HP6<br />HP16<br />2<br />2<br />2<br />2<br />3<br />3<br />3<br />3<br />3<br />HP8<br />HP8<br />HP16<br />HP8<br />HP10<br />4<br />HP15<br />HP1<br />4<br />4<br />4<br />4<br />HP5<br />HP7<br />HP5<br />5<br />5<br />5<br />5<br />5<br />HP10<br />HP13<br />HP8<br />HP19<br />HP14<br />
  101. 101. Home Page (19)<br />HP16 & HP15 designs consistently placed in the Top 5 across all filters<br />Results List (14)<br />RL4 consistently placed in the Top 3 across all sample filters, and was the #1 choice for 80% of all participants<br />Document Display (9)<br />DD3 placed in the Top 5 across all sample filters and was the #1 choice for 77% of all participants<br />Phase 1: High-level Results<br />
  102. 102. Note, participants were asked to describe the current Westlaw before being shown the new designs. <br />54<br />Phase 1: Word Selection Results<br />
  103. 103. 5 design themes were derived from post-session discussions<br /><ul><li>“New design(s) are better than current Westlaw”
  104. 104. “Clean and Fresh”
  105. 105. “Contrast is Important”
  106. 106. “Prefer Westlaw Blue”
  107. 107. “No Big Fonts Please”</li></ul>The study narrowed the list of potential designs, and we better understood what design elements that Westlaw users liked and disliked.<br />Phase 1: High-level Results<br />
  108. 108. 56<br />Phase 2: September 2009<br />Kansas City, MO, Sept 2009<br />
  109. 109. Goals<br /><ul><li>Refine preferences for selected design directions
  110. 110. Understand users personal reasons for liking their preferred choices
  111. 111. Get closure on other design options for online and printed content
  112. 112. Sustain tight security</li></ul>Tool<br /><ul><li>Same as in Round 1, with some minor revisions to accommodate specialized input </li></ul>Phase 2: September 2009<br />
  113. 113. Method<br />View, Rate, and Pick Top Choice for <br />Homepage (3 options)<br />Result List (2 options)<br />Document Display (2 options)<br />“Why?”<br />Simple preference selection for two unresolved UI design issues<br />Citing References: Grid display or List display?<br />Out of Plan Indication design (6 options)<br />Type formatting preferences for 3 different content types<br />Font Face<br />Font Size<br />Margin Width <br />Phase 2: September 2009<br />
  114. 114. Logistics<br />3 cities (Philadelphia, Kansas City, Los Angeles)<br />1 Day<br />226 participants<br />Analysis<br />Filters (8 categories) were used to score the designs for each visual preference<br />Results<br />Clear choices for top designs in each of all categories<br />“Why” feedback shed new light on designs under consideration and helped focus “homestretch” design activities<br />Phase 2: September 2009<br />
  115. 115. Home Page (3)<br />HP3 ranked #1in 94% of filter groups (54% of total participants) <br />Results List (2)<br />RL5 ranked #1in 97% of filter groups (58% of total participants) <br />Document Display (2)<br />DD7 ranked #1in 94% of filter groups (61% of total participants)<br />Phase 2: High-level Results<br />
  116. 116. The main concerns regarding Homepage Design HP3<br />Search Box <br />Too small<br />How do I do a Terms-and-Connectors search?<br />Browse Section<br />How do I specify multiple or specific search content?<br />Poor organization<br />Poor label<br />Need access to “often-used” content<br />Need better access to help<br />61<br />Participant Comments: Homepage<br />
  117. 117. Goals<br />Get feedback on branding options from decision makers and those who influence purchase of the product<br />Get closure on final outstanding design issues<br />Tool<br />Same as in Rounds 1 & 2, with some minor revisions to accommodate specialized input <br />Phase 3: December 2009<br />
  118. 118. Method<br />Wordmark/Branding<br />View wordmark color combinations and design elements against different backgrounds, pick top choice and provide comments<br />Make a final “Top Choice” from all selections<br />Simple preference selection for outstanding UI design issues<br />Header Space: Tile or No Tile?<br />Notes Design<br />Location: Inline or Column?<br />State: Open or Closed?<br />Headnote Icon design (4 variations)<br />Phase 3: December 2009<br />
  119. 119. What color combination do you prefer? Please rank the 4 combinations below according to your preferences. To rank, click and drag an item from the left to a box on the right.<br />Your Most Liked<br />1<br />2<br />3<br />4<br />Your Least Liked<br />
  120. 120. Logistics<br />3 cities (Seattle, Denver, Boston) <br />1 Day<br />214 participants<br />Analysis<br />Simple preference, no advanced filters <br />Results<br />Decision-makers confirmed that critical brand elements should be retained<br />Phase 3: December 2009<br />
  121. 121. Decision Makers’ Picks (1 of 2)<br />
  122. 122. 67<br />
  123. 123. Measuring<br />Emotional Response<br />to Guide Design<br />Why it succeeded<br /><ul><li>Quantitative & qualitative data to identify preference trends
  124. 124. “Slicing” across identifiable filters
  125. 125. Emphasis on “gut-level” reactions
  126. 126. Intolerance for manipulation of product brand
  127. 127. Rapid turnaround of data to all stakeholders
  128. 128. Executive
  129. 129. Design
  130. 130. Development</li></li></ul><li>69<br />Document Display<br />May 2009<br />Sept 2009<br />Feb 2010<br />
  131. 131. At what cost(s)?<br /><ul><li>We held off asking “why” until the second round
  132. 132. If we had asked why in the first round, we might have
  133. 133. avoided some of internal design battles
  134. 134. gotten more granular ammunition for communicating the design vision to stakeholders
  135. 135. “Need for speed” attained at the cost of detailed analysis</li></ul>Retrospective <br />
  136. 136. Recommendations for anyone thinking of undertaking something like this<br /><ul><li>Procure a “Matt” to create and administer your tool
  137. 137. Get a good technical vendor for on-site
  138. 138. Report results in as close to real-time as possible on a wiki or other web-page</li></ul>Retrospective <br />
  139. 139. Summary/Comparison<br />72<br />
  140. 140. Both groups valued support in design decision making<br />Align methodology with needs of the project<br />Research-inspired, not research-decided<br />73<br />Summary/Comparison<br />
  141. 141. Additional Reading and Tools<br />74<br />
  142. 142. Benedek, Joey and Trish Miner. “Measuring Desirability: New Methods for Evaluating Desirability in a Usability Lab Setting.” Proceedings of UPA 2002 Conference, Orlando, FL, July 8–12, 2002. http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.doc<br />Lindgaard, Gitte, Gary Fernandes, Cathy Dudek, and J. Brown. "Attention Web Designers: You Have 50 Milliseconds to Make a Good First Impression!" Behaviour and Information Technology, 2006. http://www.imagescape.com/library/whitepapers/first-impression.pdf<br />Rohrer, Christian. “Desirability Studies: Measuring Aesthetic Response to Visual Designs.” xdStrategy.com, October 28, 2008. Retrieved February 10, 2010. http://www.xdstrategy.com/2008/10/28/desirability_studies<br />75<br />Additional Reading<br />
  143. 143. User Focus. "Measuring satisfaction: Beyond the Usability Questionnaire." Retrieved February 10, 2010. http://www.userfocus.co.uk/articles/satisfaction.html <br />UserEffect. "Guide to Low-Cost Usability Tools." Retrieved May 12, 2010.http://www.usereffect.com/topic/guide-to-low-cost-usability-tools<br />Tullis, Thomas and Jacqueline Stetson. “A Comparison of Questionnaires for Assessing Website Usability.” Usability Professionals’ Association Conference, 2004.home.comcast.net/~tomtullis/publications/UPA2004TullisStetson.pdf<br />Westerman, S. J., E. Sutherland, L. Robinson, H. Powell, and G. Tuck. “A Multi-method Approach to the Assessment of Web Page Designs.” Proceedings of the 2nd international conference on Affective Computing and Intelligent Interaction, 2007.http:// portal.acm.org/citation.cfm?id=1422200<br />76<br />Additional Reading<br />
  144. 144. Five Second Testhttp://fivesecondtest.com/<br />Feedback Army http://www.feedbackarmy.com<br />Wordlehttp://www.wordle.net<br />PrEmohttp://www.premo-online.com<br />77<br />Additional Tools<br />

×