This document discusses desirability testing, which assesses people's emotional responses to designs. It outlines various methods considered, including triading, questionnaires, quick exposure tests, and physiological measurements. The selected method, Product Reaction Cards, uses 60 descriptive words to elicit feedback on designs for a hospital website redesign. Three designs were tested quantitatively with 50 people each and qualitatively through interviews. Designs scored well on conveying caring, warmth and trustworthiness. Lessons included the value of mixed methods and positioning results as input, not declaring winners.

1. boston upa
Rapid Desirability Testing
ANALYZING EMOTIONAL RESPONSE TO A DESIGN (ON A BUDGET)
Prepared by:
Michael Hawley – VP Experience Design
Megan Grocki – Senior Experience Designer
June 9, 2009

2. boston upa
Agenda
• Introduction
• The Situation
• Desirability Testing Overview
• Methods Considered
• Our Selected Process
• Case Study
• Lessons Learned
2

3. boston upa
About Mad*Pow
3

4. boston upa
The Situation
4

5. boston upa
The Situation
Visual Designs Applied to Wireframe
5

6. boston upa
The Situation
Visual Designs Applied to Wireframe
6

7. boston upa
Desirability Testing Overview
7

8. boston upa
What Is Desirability Testing?
A collection of research methods intended to assess target audience’s
emotional response to a design or stimulus.
What It Is What It Is Not
• Measure of how closely a • Measure of how much people “like”
stimulus achieves the “desired” something
emotional response
• Figuring out which is the “best”
8

9. boston upa
Positioning Desirability Studies
http://www.xdstrategy.com/2008/10/28/desirability_studies/
9

10. boston upa
Why Is It Important
First impressions of a design to impact a product’s or application’s perceived
utility, usability, and credibility.
Functionality
Usability Aesthetics
10

11. boston upa
Methods Considered
11

12. boston upa
Triading
Definition
Present three different concepts or ideas to participants, and ask them to
identify how two of them are different from the third and why.
12

13. boston upa
Quantitative Questionnaires
Definition
Broad, experience-based questionnaires,
that also include questions relating to
visual appeal and aesthetics
• SUS (System Usability Scale),
• QUIS (Questionnaire for User Interface
Satisfaction)
• WAMMI (Website Analysis and
Measurement Inventory)
13

14. boston upa
Quick Exposure Memory Tests
Show participants a user interface
for a very brief moment, then take it Attention designers:
away. Participants recall their first
impression, then moderator probes
for meaning You have
50 milliseconds
• Helpful for layout decisions,
to make a good
prominence of content, even labels
first impression
• www.fivesecondtest.com
14

15. boston upa
Physiological and Neurological Measurements
Definition
• Sensors track participants’ physiological
measurements to particular designs. Changes in
suggest a particular emotional response.
• Paired with attitudinal and self-reporting surveys
measurements give a multifaceted view of
emotional reactions to a design
• Electroencephalography (EEG): Brain activity
• Electromyography (EMG): Muscles and Excitement
• Electrodermal Activity (EDA): Sweat, Excitement
• Blood Volume Pressure (BVP): Arousal
• Pupil Dilation: Arousal and Mental Workload
• Respiration: Negative Valence or Arousal
15

16. boston upa
PrEmo Emotional Measurement
http://www.premo-online.com
Dr. Pieter Desmet,Technical University of Delft
16

17. boston upa
Product Reaction Cards (Our Selected Approach)
http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.doc
17

18. boston upa
Product Reaction Cards Method
18

19. boston upa
Before You Begin
Determine intended brand attributes (and their opposites)
1. Leverage existing marketing/brand
materials
2. Alternatively, stakeholder brainstorm to
identify key brand attributes/descriptors
using full list of product reaction cards
as a start
3. Tip: “If the brand was a person, how
would it speak to your customers?”
19

20. boston upa
Process - Conducting
Methodology
1. Include 60/40 split of positive and negative words
2. Target 60 words, optimized to test brand
3. Simple question: “Which of the following words do you feel best describe the
site/design/product (please select 5):”
4. One comp per participant, or multiple comps per participant (no more than 3)
Participants
1. Qualitative: Paired with usability testing
2. Quantitative: Target minimum of 30 per option if possible
20

21. boston upa
Process - Analyzing
1. Calculate percentage of positive
and negative attributes per design
68% Positive
32% Negative
2. Visualize overall sentiment of
feedback using “word clouds” (see
wordle.net)
Tip: Use word list spreadsheet available at
http://www.userfocus.co.uk/articles/satisfaction.html
21

22. boston upa
Case Study: Greenwich Hospital Website Redesign
22

23. boston upa
Case Study: Greenwich Hospital Website Redesign
Background and Goals
• Align the website with the character of Greenwich Hospital
• “luxurious, approachable, friendly, capable, multi-cultural/inclusive, established”
• Update the site after nearly 10 years
• Counter impressions that Greenwich Hospital is more than just about
maternity and elder care, without damaging those notions
• Communicate that they are long-standing members of the community
23

24. boston upa
Case Study: Greenwich Hospital Website Redesign
Methodology
• 3 visually designed comps
• 50 people reacted to each comp (quantitative) via survey
• Additional feedback obtained via participant interviews (qualitative)
Survey Questions
Hello, I am requesting feedback on a website I am working on.
Your answers let me know if the site is conveying the right feel.
1. What are your initial reactions to the web site?
2. Which of the following words best do you feel best describe the site (please
select 5):
24

25. boston upa
Three Different Visual Designs
25

26. boston upa
Results: Concept 1
“My initial reaction to this web site is that it seems kind of
plain. There is not much going on in the page, and the 12% Negative
colors seem kind of drab.”
“This is a nice looking website. It is well designed, well
laid out, and is appealing to look at. It makes me want to 88% Positive
continue to navigate the site to learn more. “
26

27. boston upa
Results: Concept 2
“Men don`t really go with children… where`s a
baby, there must be a mother. “
87% Positive
“My initial reaction to the website is that it 13% Negative
seems very clean and modern. I like the layout,
it looks like its easy to find information. “
27

28. boston upa
Results: Concept 3
“I felt love. I saw a mother holding a child..
that`s pretty touchy. The site looks good, and it
makes the hospital trustworthy.”
5% Negative
“My initial reaction was that the Hospital is
represented by a caring, warm and friendly 95% Positive
website.”
28

29. boston upa
Lessons Learned
29

30. boston upa
Lessons Learned
Methodology
• Mix of qualitative and quantitative is key. Qualitative helps provide color to the
results, quantitative resonates with stakeholders and executives
• Position results as one form of input to decision-making process, not declaring
a “winner”
• Simple, cost-efficient way to assess audience’s emotional response to a design
30

31. boston upa
Key Take Aways
The Challenge:
• Measuring emotional responses to a design important, but complex.
Experiences of a visual design are multifaceted, and a number of design
aspects can impact their response to a product.
• There are a number of alternatives available to measure emotional response
Our Experience:
• Leveraging Product Reaction Cards provides a low-cost, low-effort means to
help us align aesthetics and general feel with desired brand attributes
31

32. boston upa
Thank You
Documentation Have a question?
Case Study results and full presentation slides: Michael Hawley
http://www.madpow.net mhawley@madpow.net
@hawleymichael
Megan Grocki
mgrocki@madpow.net
@megUX
603-436-7177
32

33. boston upa
Additional Reading
Benedek, Joey and Trish Miner. “Measuring Desirability: New Methods for
Evaluating Desirability in a Usability Lab Setting.” Proceedings of UPA 2002
Conference, Orlando, FL, July 8–12, 2002.
http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.doc
Lindgaard, Gitte, Gary Fernandes, Cathy Dudek, and J. Brown. "Attention Web
Designers: You Have 50 Milliseconds to Make a Good First Impression!" Behaviour
and Information Technology, 2006.
http://www.imagescape.com/library/whitepapers/first-impression.pdf
Rohrer, Christian. “Desirability Studies: Measuring Aesthetic Response to Visual
Designs.” xdStrategy.com, October 28, 2008. Retrieved February 10, 2010.
http://www.xdstrategy.com/2008/10/28/desirability_studies
33

34. boston upa
Additional Reading
User Focus. "Measuring satisfaction: Beyond the Usability Questionnaire." Retrieved
February 10, 2010.
http://www.userfocus.co.uk/articles/satisfaction.html
UserEffect. "Guide to Low-Cost Usability Tools." Retrieved May 12, 2010.
http://www.usereffect.com/topic/guide-to-low-cost-usability-tools
Tullis, Thomas and Jacqueline Stetson. “A Comparison of Questionnaires for
Assessing Website Usability.” Usability Professionals’ Association Conference, 2004.
home.comcast.net/~tomtullis/publications/UPA2004TullisStetson.pdf
Westerman, S. J., E. Sutherland, L. Robinson, H. Powell, and G. Tuck. “A Multi-
method Approach to the Assessment of Web Page Designs.” Proceedings of the 2nd
international conference on Affective Computing and Intelligent Interaction, 2007.
http:// portal.acm.org/citation.cfm?id=1422200
34

35. boston upa
Additional Tools
Five Second Test
http://fivesecondtest.com/
Feedback Army
http://www.feedbackarmy.com
Wordle
http://www.wordle.net
PrEmo
http://www.premo-online.com
35