The document outlines a research project conducted by InContext Design and SAP to develop a valid instrument for measuring product "coolness". Through qualitative consumer research, they identified seven constructs related to coolness. Further studies with business professionals validated these constructs and developed a survey with statements mapping to each construct. Extensive testing of the survey with consumers and business users confirmed the survey's ability to reliably differentiate coolness levels between products. The resulting 40-statement survey provides a repeatable way to measure and improve product coolness through a quick user assessment.

1. Measuring Product “Coolness” –
Developing a Measurement Instrument
Abstract
Cool products provide a leap in value and they increase
a company’s market share. The widespread adoption of
iOS and Android devices resulted in a radical change in
the role of technology in people’s lives. And anecdotally
people exclaimed about how “cool” their devices were;
“I can’t go back” to what I had before and “I can’t stop
talking about it—it’s so cool!”
In 2010 InContext Design launched The Cool Project1 to
understand the underlying principles, which make a
product “cool.” The research goal was to discover the
core drivers of the cool user experience. This consumer
research used qualitative research to identify seven
core constructs associated with “coolness” that
appeared independently of device or software or even
technology product. In 2011 SAP became interested in
developing a metric to measure “coolness” and in a
joint research effort with InContext Design validated
the constructs with business users and co-designed a
cool metric. The final result is a repeatable process for
measuring cool through a quick survey, the results of
which provide implications to product teams on how to
improve their products to increase “coolness”.
1 See Interactions Volume 18 Issue 6, November + December
2011, Pages 40-47, for overview of The Cool Project and the
Cool Concepts. See also http://bit.ly/vTBmEy
Carol Farnsworth
SAP Labs, LLC
3410 Hillview Avenue
Palo Alto, CA 94304 USA
carol.farnsworth@sap.com
Karen Holtzblatt
InContext Design
2352 Main Street, Suite 302
Concord, MA 01742 USA
karen@incontextdesign.com
Shantanu Pai
SAP Labs, LLC
3410 Hillview Avenue
Palo Alto, CA 94304 USA
shanritz@gmail.com
Kelley Wagg
InContext Design
2352 Main Street, Suite 302
Concord, MA 01742 USA
kelley.wagg@incontextdesign.com
Theo Held
SAP AG
Dietmar-Hopp-Allee 16 69190
Walldorf, Germany
theo.held@sap.com
Eli Wylen
InContext Design
2352 Main Street, Suite 302
Concord, MA 01742 USA
eli.wylen@incontextdesign.com
Sally Lawler Kennedy
SAP Labs, LLC
3410 Hillview Avenue
Palo Alto, CA 94304 USA
sally.lawler.kennedy@sap.com
Pallavi Kutty
SAP Labs, LLC
3410 Hillview Avenue
Palo Alto, CA 94304 USA
pallavi.kutty@sap.com
Copyright 2014 held by Owner/Author. Publication Rights
Licensed to ACM

2. This case study outlines the process of using qualitative
data to design a valid quantitative measure.
Author Keywords
Cool; innovation; transformative design; measures;
metrics; performance
ACM Classification Keywords
H.1.2 User/Machine Systems – Human Factors;
Software Psychology
Introduction
The world changed with the release of the iPhone and
the widespread adoption of iOS and Android devices.
The impact went far beyond the popularity of the
smartphone as a product—it affected the way people do
their work and live their lives. InContext Design wanted
to understand what was core to this cool user
experience and articulate it so that it could be used
systematically to develop transformative products.
If teams understood the underlying principles of what
makes a product cool, could those principles be used to
deliberately design cool into a product? To find out,
InContext Design conducted extensive field research
with consumers, then with business professionals
together with SAP.
This case study focuses on the process that was
followed and research conducted by SAP and InContext
Design to develop the cool measures instrument. It
provides an example of how reliable metrics may be
developed with qualitative and quantitative measures
grounded in a deep understanding of the phenomenon
being measured.
Consumer Research – Summer 2010
The consumer field research involved 65 US consumers
ages 15-60. Interviews were face-to-face in the
participants’ home. During the two-hour session the
discussion was focused around: 1) the most memorable
cool products from home and personal life; 2)
experience with products; and 3) a typical day in their
life. Asking about their own “cool” products focused us
on people’s self-defined experience of coolness,
ensuring we were accessing the correct phenomenon.
An affinity diagram of the qualitative data revealed the
core themes (constructs) that resulted in the
conceptual framework described as the Cool Concepts.
Four constructs, termed the Wheel of Joy, define the
way the product creates joy by affecting people’s lives
and fulfilling their core human motives.
§ Accomplishment: Allows people to get all the
activities of their life done while on the move.
§ Connection: Allows people to feel connected and
maintain their real personal and work relationships –
and participate in meaningful communities.
§ Identity: Allows people to define, celebrate, and
express their unique self.
§ Sensation: Provides people sensory delight
through stimulation of the senses and use of animation.
Three constructs, termed the Triangle of Design, define
the way people experience and use the product itself.
§ Direct into action: Intent is achieved in moments
delivering exactly what is expected, like magic.

3. § The Hassle Factor: Eliminates the pain of tool and
life hassle creating the joy of relief; No set up, no fuss.
§ The Delta: Eliminates learning by building
interaction on what people naturally know.
Questions generated from the Cool Concepts were
presented in an online survey to 800 US consumers,
recruited via Craigslist.com. Cool attribute statements
reflect the top five reasons their product was cool.
Findings from this study helped gauge product coolness
and confirmed the seven cool constructs.
Develop Initial Cool Measures – July 2011
InContext Design introduced the seven Cool Concepts
to SAP, and together we set out to extend the
consumer research into the business world to help us
understand what makes products cool for professionals.
During a 5-day immersion workshop, we developed the
initial set of cool measures to be used in the business
field study.
Professional Field Research – Sept 2011
The business professional field research employed the
same method as the 2010 consumer field research. In
addition, data were collected using the initial cool
measurement instrument. Twenty-eight (28) individuals
in Sales, Human Resources, Finance, and Procurement
participated.
The two-hour interview was focused around: 1) most
memorable cool work products; 2) experience with
current work products; 3) cool products from home and
personal life; 4) a typical day in their life. To conclude
the session, we asked the business professionals to
complete the cool measure and we discussed how the
measures reflect their actual experience.
Refine Cool Measures – December 2011
through March 2012
The prior research resulted in a set of 79 phrases
related to the seven Cool Concepts. We set out to
determine whether the measures were representative,
clustered together, and which dimensions most strongly
accounted for coolness. We also needed to reduce the
number of phrases by half so that the metric could be
administered in 5-8 minutes.
We conducted two extensive online studies, one in the
US, the other outside the US (United Kingdom,
Germany, India, and Singapore), to determine the
underlying factorial structure and to select the most
suitable measures. Our 79 measures were presented in
random order, while participants rated their coolest
device and business application.
The analysis of the data revealed that the phrases did
indeed cluster as predicted by the Cool Concepts, and
that they differentiated products in terms of “coolness”
both for devices and software products, including
business products. The “Connection” concept divided
into two cool factors (personal life and work
relationships). Also, the “Direct into Action” and
“Hassle” factors merged into one factor. The two
strongest factors for cool in business are
“Accomplishment” and “Direct into Action/Hassle”.
We used Confirmatory Factor Analysis (CFA) to help us:
1) Ensure that the emerging factors map to the cool
concepts; 2) Determine the loadings for each factor,
and thus understand the amount of variance accounted
by each factor and hence its importance; and 3)
Reduce redundancy by eliminating phrases associated
with factors with poor loadings.

4. A set of 40 phrases, down from 79, were selected as
statistically valid for a final version of the measures.
Validate Cool Measures – April 2012
To assess validity and reliability of the 40 measures, a
400-person validation study was conducted with
business professionals in HR, Sales, Finance and
Procurement within the USA. Statistical analysis
showed that both the overall measure of Cool and the
measure for each Cool Concept were valid. Overall, the
measures were tested with almost 900 consumers and
over 2000 business professional participants around the
world in various research activities conducted over 3
years. The measures differentiate between products,
and work for devices and software alike. Resulting in
measures grounded in data.
Cool Measures Instrument – Description and
Next Steps
The overall result of the project is a validated set of
statements, which can be used to measure the coolness
of user experiences. Various experiences can be
measured: 1) product already used and familiar, 2)
similar competitive products, 3) first time experiences
on products; and 4) product ideas in development
(concept, storyboard, prototype). A 7-minute Cool
Measures survey allows users to compare their
experience with the product against what they use now.
Users are shown statements in this form:
The user selects a point on a scale, reflecting how
strongly they feel the statement is true, with “Worse”
and “No change” on the left side of the scale and the
statement in question on the far right.
A cool score is derived from the average cool measures
across participants, applying statistically determined
correlation coefficients and weightings. The Cool
Measures instrument helps us understand if this
product, a new product and/or product idea is better
than what the participant is using today. Both SAP and
InContext Design are moving forward using the Cool
Metric with product teams. InContext has developed
several models to be used in the design process.
Acknowledgements
We thank Dan Rosenberg, SVP from SAP who had the
vision and confidence that we could develop a coolness
metric. He also provided resources to complete the
project. We thank the InContext Design support team
who helped prepare all of the data immersion and
workshop materials. We want to acknowledge the two
PhD statisticians who were skeptical in the beginning,
but after seeing the results of the analyses became
believers, enthusiasts, and promoters of the metric.
References
[1] Holtzblatt, Karen What Makes Things Cool?
Interactions Volume 18 Issue 6, November + December
2011, Pages 40-47, for overview of The Cool Project
and the Cool Concepts. See also http://bit.ly/vTBmEy
[2] Holtzblatt, Karen Rapid Contextual Design: a How-to
Guide to Key Techniques for User-Centered Design
Morgan Kauffmann Publishers, Elsevier, Inc. San
Francisco, CA, USA 2005