Desirability and Preference Testing - UPA International 2011

Preference and Desirability Testing: Measuring Emotional Response to Guide Design
Michael Hawley
Chief Design Officer, Mad*Pow
Paul Doncaster
Senior User Experience Designer, Thomson Reuters

Agenda
Why we should care
Why it’s not always as simple as asking:“Which option do you prefer?”
Methods to consider
Case Study: Greenwich Hospital
Case Study: WestlawNext
Summary/Comparison

Why we should care
3

Impressions Count

An important role of visual design is to lead users through the hierarchy of a design as we intend
For interactive applications, a sense of organization can affect perceived usability and, ultimately, users’ overall satisfaction with the product
Functional Desirability

Visceral Emotions

7
Fogg’s Behavior Model
Core motivators include:
Pleasure/pain
Hope/fear
Acceptance/rejection
http://www.behaviormodel.org/

8
Positioning Desirability Studies
http://www.xdstrategy.com/2008/10/28/desirability_studies/

“Which do you prefer?”
9

Quantity, granularity breed apathy

Poor articulation
“It reminds me of…”
“It’s nice and clean.”
“There’s just something about it . . .”
“I ordinarily don’t like red, but for some reason it works here . . .”
“It’s better than the other ones.”

What Stakeholders Should Care About
“We should go with design C over A and B, because I feel it evokes the right kind of emotional response in our audience that is closer to our most important brand attributes.”

Methods to Consider
13

Present three different concepts or ideas to participants, and ask them to identify how two of them are different from the third and why.
14
Triading

Broad, experience-based questionnaires, that also include questions relating to visual appeal and aesthetics
SUS (System Usability Scale)
QUIS (Questionnaire for User Interface Satisfaction)
WAMMI (Website Analysis and Measurement Inventory)
15
Qualitative Questionnaires

Show participants a user interface for a very brief moment, then take it away.
Participants recall their first impression, then moderator probes for meaning.
Helpful for layout decisions, prominence of content, labels
www.fivesecondtest.com
16
Attention designers:
You have
50 milliseconds
to make a good
first impression
Quick Exposure Memory Tests

Electroencephalography (EEG): Brain activity
Electromyography (EMG):
Muscles and Excitement
Electrodermal Activity (EDA): Sweat, Excitement
Blood Volume Pressure (BVP): Arousal
Pupil Dilation: Arousal and Mental Workload
Respiration:
Negative Valence or Arousal
17
Physiological and Neurological

18
PrEmo Emotional Measurement
Dr. Pieter Desmet, Technical University of Delft
http://www.premo-online.com

19
http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.doc
Product Reaction Cards

Case Study: Greenwich Hospital
20

Determine intended brand attributes (and their opposites)
21
Product Reaction Cards: Before You Begin
Leverage existing marketing/brand materials
Alternatively, stakeholder brainstorm to identify key brand attributes/descriptors using full list of product reaction cards as a start
Tip: “If the brand was a person, how would it speak to your customers?”

Methodology
Include 60/40 split of positive and negative words
Target 60 words, optimized to test brand
Simple question: “Which of the following words do you feel best describe the site/design/product (please select 5):”
One comp per participant, or multiple comps per participant (no more than 3)
Participants
Qualitative: Paired with usability testing
Quantitative: Target minimum of 30 per option if possible
22
Product Reaction Cards: Conducting

23
Process - Analyzing
Calculate percentage of positive and negative attributes per design
Visualize overall sentiment of feedback using “word clouds” (see wordle.net)
68% Positive
32% Negative

Align the website with the character of the Hospital
Update the site after nearly 10 years
Counter impressions that Greenwich is more than just maternity and elder care
Communicate that they are long-standing members of the community
24
Case Study: Greenwich Hospital Website Redesign

25
Case Study: Greenwich Hospital Website Redesign
3 visually designed comps
50 people reacted to each comp (quantitative) via survey
Additional feedback obtained via participant interviews (qualitative)
Survey Questions
Hello, I am requesting feedback on a website I am working on. Your answers let me know if the site is conveying the right feel.
1. What are your initial reactions to the web site?
2. Which of the following words best do you feel best describe the site (select 5):

26
Three Different Visual Designs

27
Results: Concept 1
12% Negative
88% Positive

28
Results: Concept 2
87% Positive
13% Negative

29
Results: Concept 3
5% Negative
95% Positive

Mix of qualitative and quantitative is key
Qualitative helps provide color to the results
Quantitative resonates with stakeholders and executives
Position results as one form of input to decision-making process, not declaring a “winner”
Simple, cost-efficient way to assess audience’s emotional response to a design
30
Lessons Learned

Case Study: WestlawNext
UX Research Team:
Drew Drentlaw
Shannon O’Brien
Bill Quie
November Samnee
31

for Phase 1
Use large sample sizes to establish a design “baseline,” from which to advance the design direction in subsequent iterations
Isolate preference trends for specific page design aspects
Determine tolerance for manipulation of the site “brand”
Maintain tight security
Goals

Sessions were held in 4 cities over 5 days
Seattle
Denver
Memphis
Minneapolis-St. Paul
4 sessions were held per day, with a maximum of 25 participants per session
1.5 hours allotted per study, most participants finished in less than 1 hour
319 participants successfully completed their sessions
Phase 1: Logistics & Execution

Participants completed the study at individual workstations at their own pace
All workstations included a 20” monitor, at 1024x768 resolution
Phase 1: Logistics & Execution
Memphis, TN, May 2009

Brief review of Westlaw critical screens
Positive/negative word selection to describe Westlaw
36
Positive/negative product descriptors

Each set of Element variations were viewed in full screen
Participant selects “top choice” by dragging a thumbnail image to a drop area
37
Homepage: Design Elements

Homepage: Design Elements (1)
All options viewed in full screen
Participant selects “top choice” by dragging a thumbnail image to a drop area

Visual Weight (6 options)
Use of Imagery (8 options)
Components (4 options)
Search Area (4 options)
Palette (10 options)
Homepage: Design Elements

19 HP designs viewed in full screen (randomized)
All 19 options are presented again; participant assigns a rating using a 10-point slider.
Top 5 and Bottom 2 choices are positioned in order of rating values on one long, scrollable page. Next to each design displayed, rates key aspects for each design on a 5-point scale
Homepage: Design Gallery

Repeat the process for Results List design:
New Results List
Design Elements
Column Collapsing (4 options)
Column Separation (2 options)
Theme/Color (8 options)
Design Gallery
14 Results Lists designs (randomized)
Key Aspects Rated
Color scheme
Global Header
Summary and Excerpt (list contents)
Filters design (left column)
Overall look and feel

Repeat the process for Document Display design:
New Document Display
Design Elements
Tabs vs. Links (4 options)
Background Separation (4 options)
Margin Width (3 options)
Font Size (12 options)
Locate (2 options)
Design Gallery
9 Results Lists designs (randomized)
Key Aspects Rated
Color scheme
Layout of content
Text formatting
Overall look and feel

“Based on the designs I’ve liked most today . . .”
51
Positive/negative design descriptors

Results were analyzed across 8 different sample filters
Job Title
Age
Testing Location
Years of Experience
Hours per Week Researching
Organization Size
Role (decision-maker status)
The top picks were surprisingly consistent across all of the ‘Top 5’ lists analyzed
High-level Results

53
Top Homepage Designs by Job Title
Job Title. Top 5 out of 19 possible.
Overall (319)
Associate (189)
Librarian (37)
Partner (81)
Solo Practitioner (5)
1
1
1
1
1
HP16
HP1
HP16
HP5
HP15
2
HP8
HP15
HP15
HP6
HP16
2
2
2
2
3
3
3
3
3
HP8
HP8
HP16
HP8
HP10
4
HP15
HP1
4
4
4
4
HP5
HP7
HP5
5
5
5
5
5
HP10
HP13
HP8
HP19
HP14

Home Page (19)
HP16 & HP15 designs consistently placed in the Top 5 across all filters
Results List (14)
RL4 consistently placed in the Top 3 across all sample filters, and was the #1 choice for 80% of all participants
Document Display (9)
DD3 placed in the Top 5 across all sample filters and was the #1 choice for 77% of all participants
Phase 1: High-level Results

Note, participants were asked to describe the current Westlaw before being shown the new designs.
55
Phase 1: Word Selection Results

5 design themes were derived from post-session discussions
“New design(s) are better than current Westlaw”
“Clean and Fresh”
“Contrast is Important”
“Prefer Westlaw Blue”
“No Big Fonts Please”
The study narrowed the list of potential designs, and we better understood what design elements that Westlaw users liked and disliked.

57
Phase 2: September 2009
Kansas City, MO, Sept 2009

Goals
Refine preferences for selected design directions
Understand users personal reasons for liking their preferred choices
Get closure on other design options for online and printed content
Sustain tight security
Tool
Same as in Round 1, with some minor revisions to accommodate specialized input

Method
View, Rate, and Pick Top Choice for
Homepage (3 options)
Result List (2 options)
Document Display (2 options)
“Why?”
Simple preference selection for two unresolved UI design issues
Citing References: Grid display or List display?
Out of Plan Indication design (6 options)
Type formatting preferences for 3 different content types
Font Face
Font Size
Margin Width

Logistics
3 cities (Philadelphia, Kansas City, Los Angeles)
1 Day
226 participants
Analysis
Filters (8 categories) were used to score the designs for each visual preference
Results
Clear choices for top designs in each of all categories
“Why” feedback shed new light on designs under consideration and helped focus “homestretch” design activities

Home Page (3)
HP3 ranked #1in 94% of filter groups (54% of total participants)
Results List (2)
RL5 ranked #1in 97% of filter groups (58% of total participants)
Document Display (2)
DD7 ranked #1in 94% of filter groups (61% of total participants)

The main concerns regarding Homepage Design HP3
Search Box
Too small
How do I do a Terms-and-Connectors search?
Browse Section
How do I specify multiple or specific search content?
Poor organization
Poor label
Need access to “often-used” content
Need better access to help
62
Participant Comments: Homepage

Goals
Get feedback on branding options from decision makers and those who influence purchase of the product
Get closure on final outstanding design issues
Tool
Same as in Rounds 1 & 2, with some minor revisions to accommodate specialized input
Phase 3: December 2010

Method
Wordmark/Branding
View wordmark color combinations and design elements against different backgrounds, pick top choice and provide comments
Make a final “Top Choice” from all selections
Simple preference selection for outstanding UI design issues
Header Space: Tile or No Tile?
Notes Design
Location: Inline or Column?
State: Open or Closed?
Headnote Icon design (4 variations)

What color combination do you prefer? Please rank the 4 combinations below according to your preferences. To rank, click and drag an item from the left to a box on the right.
Your Most Liked
1
2
3
4
Your Least Liked

Logistics
3 cities (Seattle, Denver, Boston)
1 Day
214 participants
Analysis
Simple preference, no advanced filters
Results
Decision-makers confirmed that critical brand elements should be retained

Decision Makers’ Picks (1 of 2)

Measuring
Emotional Response
to Guide Design
Why it succeeded
Quantitative & qualitative data to identify preference trends
“Slicing” across identifiable filters
Emphasis on “gut-level” reactions
Intolerance for manipulation of product brand
Rapid turnaround of data to all stakeholders
Executive
Design
Development

70
Document Display
May 2009
Sept 2009
Feb 2010

At what cost(s)?
We held off asking “why” until the second round
If we had asked why in the first round, we might have
avoided some of internal design battles
gotten more granular ammunition for communicating the design vision to stakeholders
“Need for speed” attained at the cost of detailed analysis
Retrospective

Recommendations for anyone thinking of undertaking something like this
Procure a “Matt” to create and administer your tool
Get a good technical vendor for on-site
Report results in as close to real-time as possible on a wiki or other web-page
Retrospective

Summary/Comparison
73

Both groups valued support in design decision making
Align methodology with needs of the project
Research-inspired, not research-decided
74
Summary/Comparison

Additional Reading and Tools
75

Benedek, Joey and Trish Miner. “Measuring Desirability: New Methods for Evaluating Desirability in a Usability Lab Setting.” Proceedings of UPA 2002 Conference, Orlando, FL, July 8–12, 2002. http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.doc
Lindgaard, Gitte, Gary Fernandes, Cathy Dudek, and J. Brown. "Attention Web Designers: You Have 50 Milliseconds to Make a Good First Impression!" Behaviour and Information Technology, 2006. http://www.imagescape.com/library/whitepapers/first-impression.pdf
Rohrer, Christian. “Desirability Studies: Measuring Aesthetic Response to Visual Designs.” xdStrategy.com, October 28, 2008. Retrieved February 10, 2010. http://www.xdstrategy.com/2008/10/28/desirability_studies
76
Additional Reading

User Focus. "Measuring satisfaction: Beyond the Usability Questionnaire." Retrieved February 10, 2010. http://www.userfocus.co.uk/articles/satisfaction.html
UserEffect. "Guide to Low-Cost Usability Tools." Retrieved May 12, 2010.http://www.usereffect.com/topic/guide-to-low-cost-usability-tools
Tullis, Thomas and Jacqueline Stetson. “A Comparison of Questionnaires for Assessing Website Usability.” Usability Professionals’ Association Conference, 2004.home.comcast.net/~tomtullis/publications/UPA2004TullisStetson.pdf
Westerman, S. J., E. Sutherland, L. Robinson, H. Powell, and G. Tuck. “A Multi-method Approach to the Assessment of Web Page Designs.” Proceedings of the 2nd international conference on Affective Computing and Intelligent Interaction, 2007.http:// portal.acm.org/citation.cfm?id=1422200
77
Additional Reading

Five Second Testhttp://fivesecondtest.com/
Feedback Army http://www.feedbackarmy.com
Wordlehttp://www.wordle.net
PrEmohttp://www.premo-online.com
78
Additional Tools

Desirability and Preference Testing - UPA International 2011

by hawleymichael

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Desirability and Preference Testing - UPA International 2011 Presentation Transcript