This document outlines a company's approach to research and testing at different stages of product development. It discusses the spectrum between minimal research and extensive evaluative research. It suggests combining approaches to save time and money while making informed decisions. Case studies are presented on MapQuest's redesign process, where mental models research provided insights into how users think about navigation tasks.

1. A Celebration of
Everything Entrepreneurial in Denver
September 28 – October 2, 2015
#DENStartupWeek

2. Title Sponsors

3. Designer Track Sponsor

4. Our Spectrum of Testing & Research
When do we research and test? When don’t we?
September 30, 2015
4

5. Hi!
Lindsay Moore
Lead UX Research
Scientist
SuAnne Hall
Director of Design

6. 01
Why are we talking to you about research?

7. 01
“Research is a luxury we don’t have time for.”

8. 8

9. 9
You can never. EVER. Guess how people will use
your product based on your own experiences and
assumptions.
— image by Whitney Hess

10. Assumption:
When a user looks up a POI, they’ll be delighted to get all of our
awesome POI data like hours, menus, amenities, reviews, tickets,
ability to book a nearby hotel, what’s nearby, etc
Reality:
When we literally put all this great info center stage, users were
very angry that we were blocking the map, as they could no
longer get the context for where the POI was!
Example 1 — Creating features based on partnerships,
revenue, data availability, etc

11. Assumption:
If a user doesn’t choose a result from predictive search, they
haven’t seen the result they wanted, so we should show them
different results
Reality:
Actually, many users just hit enter or search, bypassing search
ahead for “no good reason”
Example 2 — Designing based on common sense

12. Assumption:
We should optimize our public transit routing to minimize the
amount of walking.
Reality:
“If I'm not in a rush to get somewhere, I would think that a 30-45
min walk leg would be ok, beyond that and I would really have to
be wanting to walk to actually do it. Also, I personally am more
likely to walk in Manhattan as it's easy to hop on alternate
transportation if I get tired or have a change in plans.” —User
Quote
Example 3 — Engineering based on behavior assumptions

13. Assumption:
When users search, they search for a what & a where (Pizza,
Denver)
Reality:
Actually a minority of users search this way. Much more common
to just specify what and assume that we’ll find what’s nearby, or
to put the where at the beginning (eg. Anaheim Airport)
Example 4 — Letting assumptions become a cultural meme

14. 01
– Simon Sinek
“People don’t buy what you do, they buy why
you do it.”

15. We believe in saving the day by helping and
empowering our users, in both small and big ways.
It’s our job to remove the obstacles that make life
anything short of easy, fun — and flexible, too. When
the day requires a hundred little wins, we’re there.
15
We Believe…

16. 01
- Ken Jennings
“We’re accustomed to only using maps when
we’re frazzled and annoyed and when things
have gone wrong. It’s like a bottle of
Immodium. By the time you pull out a bottle
of Immodium, things have already gone very
wrong with your day.”

17. OK, so maybe research isn’t just a luxury…
17

18. 18
Ifwe ship quickly
without validating the
experience, we’ll end up
spending up to 100x more
to fix what we got wrong
than it would have cost in
the design stage.*
YOU COULD SPEND $200 AND 4 HOURS
FIXING THAT LEAKY PIPE, OR IT WILL
COST YOU $200,000 IN 2 MONTHS TO
REPAIR YOUR FLOODED BASEMENT.
HMMMM. I REALLY WANTED TO
TAKE THAT CAT SWEATER KNITTING
CLASS THIS WEEKEND. I THINK I’LL
JUST WAIT FOR MY BASEMENT TO
FLOOD AND PAY $200,000.
* IEEE Spectrum, Why Software Fails, http://spectrum.ieee.org/computing/software/why-software-fails
WallStreet & Technology, Can UX Deliver 1000% ROI? http://www.wallstreetandtech.com/wealth-management/can-ux-deliver-1000--roi/a/d-id/1268572?

19. 01
– CEO of Jaguar
“If you think good design is expensive, you
should look at the cost of bad design.”

20. 20
HOWEVER

21. 21
Ifwe spend time
researching, validating, and
testing every little decision
we make, we’ll become an
overly-cautious culture that
promotes the immobilization
of innovation and agility.
WE TEST NOW.
MMMM Nom nom
~burrrrrp
TIMELINE
EATO-MATIC 3000

22. 22
OK then. So WHEN do we research and test?

23. 23
There are two main schools of thought in which we tend to fall in one
bucket or the other. But we don’t have to take sides! When we
combine both schools, we can use an effective model that saves time
and money while still making smart, customer-centric decisions.
WOOT

24. We need a tactical answer
(like a button label)
We need a North
Star-type answer
(like a Product Narrative for
a new product)
WEEEEEEE
BAGOINK.
YES YES
Can we anwer our
questions quickly & easily?
If we get the answers wrong, will it be
expensive & time-consuming to fix post-dev?
Label or
button
changes
Major
changes to
the user’s
workflow
Brand new
product
Small
iterations to
existing
features
Lean MVP
date-driven
project
Project where
we’re not
reinventing
the wheel
Finding the
best
placement
or layout
Introducing
important
new data or
services
Redesign of
a product
Any updates
aren’t
feasible

25. We need a tactical answer
(like a button label)
We need a North
Star-type answer
(like a Product Narrative for
a new product)
WEEEEEEE
BAGOINK.
YES YES
Can we anwer our
questions quickly & easily?
If we get the answers wrong, will it be
expensive & time-consuming to fix post-dev?
GENERATIVE RESEARCH
EVALUATIVE RESEARCH
A/B TESTING
DESIGN EXPERTISE
QUANTITATIVE ASSESSMENT
CLICK/EYE-TRACKING TESTS
ETHNOGRAPHIC RESEARCH
MINI-TESTING
DIARY STUDIES
USABILITY TESTS
QUALITATIVE INTERVIEWS
HEURISTIC EVALUATION
COMPETITIVE ANALYSIS

26. • Instead of saying ‘we can’t afford to gather any customer
insight/research’, think about how much it could cost NOT to
do a bit of up-front research.
• Not too bad? Great! Let’s learn and iterate along the way.
• Would getting this implementation wrong be devastating to fix?
Would it alienate our users significantly?
Re-think that gamble.
How to use the spectrum

27. • Factor in the realities of your company and your situation.
• Time
• Team capacity
• Available funding
• Ease of dev change
• Existing userbase
• Skin in the game
Things to consider

28. • Minimal customer insight + dependence on a high ROI =
incredibly risky.
• Significant personal investment ($, time, emotional) can lead to
a loss of perspective on your product
Watch out for these!

29. • Don’t confuse your MVP (minimum viable product) with a MP
(minimal product)
• Confusing your MVP with your EP (the eventual product)
Watch out for these!

30. 01
– Laura Klein, VP of Product at Hint Health
“An MVP is not an excuse for a crappy
product. MVPs are all about learning, and you
don’t learn anything from a crappy product
except that people don’t like crappy
products.”

31. 01
Case Studies

32. 32
MapQuest.com Redesign

33. Web Revolution.
33

35. SERP, DERP
35
???

36. We need a tactical answer
(like a button label)
We need a North
Star-type answer
(like a Product Narrative for
a new product)
WEEEEEEE
BAGOINK.
YES YES
Can we anwer our
questions quickly & easily?
If we get the answers wrong, will it be
expensive & time-consuming to fix post-dev?

37. We need a tactical answer
(like a button label)
We need a North
Star-type answer
(like a Product Narrative for
a new product)
WEEEEEEE
BAGOINK.
YES YES
Can we anwer our
questions quickly & easily?
If we get the answers wrong, will it be
expensive & time-consuming to fix post-dev?
What do
we call
Search?
Mini-testing
Mental
Models
Cross-device
usability
testing of
resposive
designs
Search
testing:
where are
the
problems?

38. We need a tactical answer
(like a button label)
We need a North
Star-type answer
(like a Product Narrative for
a new product)
WEEEEEEE
BAGOINK.
YES YES
Can we anwer our
questions quickly & easily?
If we get the answers wrong, will it be
expensive & time-consuming to fix post-dev?
What do
we call
Search?
Mini-testing
Mental
Models
Cross-device
usability
testing of
resposive
designs
Search
testing:
where are
the
problems?

39. We need a tactical answer
(like a button label)
We need a North
Star-type answer
(like a Product Narrative for
a new product)
WEEEEEEE
BAGOINK.
YES YES
Can we anwer our
questions quickly & easily?
If we get the answers wrong, will it be
expensive & time-consuming to fix post-dev?
What do
we call
Search?
Mini-testing
Mental
Models
Cross-device
usability
testing of
resposive
designs
Search
testing:
where are
the
problems?

40. When can you stop researching?
40

41. 41
MapQuest.com’s Mental Models

42. 01
– Indi Young
“Mental models give you a deep
understanding of people’s motivations and
thought-processes, along with the
emotional and philosophical landscape in
which they are operating.”

43. Mental Model Example: Getting to Work
43

44. mapquest
Mental Spaces for users of MapQuest.com (Yogi)
MENTALSPACEMINDSETGOALSTASKSOBJECTS
+ I’m going someplace and need to get my bearings
+ I need to figure out the specifics of how to get from
point A to point B
+ I want to figure out a route between multiple places
+ I need to plan my transit: how long will it take me to
get there? how far away am I? what are my options
for getting there?
Finding a Place & Consuming Results can be
children of this mental space
+ I’m gathering info about my options so I can make a
decision
+ I’m applying my preferences and criteria to determine
the best route for me (explicitly or mentally)
+ I’m evaluating tradeoffs between routes
+ I want to scan an overview of the route/figure out
what’s in store
+ I want an overview and will dive in to details as needed
+ I need to understand the narrative step-by-step
+ I want to understand my backup options should
something go wrong
Find out the best route to get me
there. Familiarize myself with the
choices. Make a route decision.
Find out how to get somewhere
that I need to go.
See tasks
for ‘Finding
a Place’
See tasks for
‘Consuming
Results’
See tasks for
‘Browse an
Area/
Category’
Apply
criteria
(one time vs
sticky)
See tasks for
‘Consuming
Directions’
Take a
screenshot
Jot down
directions to
take with me
Configure
print page
(remove steps,
remove ads,
write notes,
mini maps)
Bookmark/
send my
directions
(to phone/
email/GPS)
Copy/paste
directions
into another
program
Close
browser
Print
directions
Apply
criteria &
knowledge
(one time vs
sticky, ‘I just
know’)
Select a
route
(suggested,
custom, make
mental
selection)
Change a
route
(dragging,
applying
options)
Read thru
narrative
Scan
map/route
for overview
Compare
routes
(time, distance,
road type, etc)
Set a
starting
location
Select a
route type
(driving, transit,
walking)
Set a
destination
Add stop(s)
to route
(actually or
mentally)
Set desired
info level
(ignore/hide
steps, show
more info, turn
on layers)
Interact with
map
(pan, zoom in for
detail, zoom out
for context, look
at turns, etc)
Drill further
into details
(zoom to step,
miles from one
step to another,
etc)
Re-route on
the fly
Reference
printed/
written
directions
Use
phone/GPS
to give
turn-by-turn
directions
Refine
search
(by typing
different words,
by selecting
addt’l criteria)
Type in place
(address, name,
name & area,
neighborhood,
district,
intersection,
city, state, etc)
Google for
address
Use geo-
targeting to
target
current
location
Select
choices from
search-
ahead
Interact with
map
(pan, zoom
in/out, click on
pins, turn on
layers, etc)
Confirm
desired
result found
(location/details
— is this the
right one?)
Scan list of
results/pins
on map for
overview
Print resultsScan list and
compare
results
(ratings, hours,
location, etc)
Select
individual
result to
view more
info
Take another
action on a
result
outside MQ
(call, visit
website, etc)
Get
directions to
a result
Share results
(mobile, URL,
email, social
media, etc)
+ I’ve decided on my route and now I need to get my
bearings before being in transit
+ I’ve decided on my route and I need to learn the
specifics before being in transit
+ I need to get this information out of mapquest.com
and into the type of tool/artifact I’ll use while in
transit
Consuming Directions can be revisited here
Prepare artifacts and knowledge to
take with into transit.
+ I’m referencing my previously-aquired information
and knowledge while in transit
+ I’m referencing my previously-prepared artifacts
while in transit
+ I need to be able to adjust on the fly, if possible
Use my previous activities and
artifacts to get there.
+ I want to find a specific place (out of curiosity or to
get more info/get bearings/etc)
+ I want to find something centered around a specific
place (targeted exploration)
+ I’m getting directions and don’t know the exact
destination so I’d better find that first
+ I want to verify that MQ can find this place before
getting my directions
+ I need to get directions and am just used to starting
with finding a place
Find the thing I’m searching for.
+ I’m verifying that the place I was looking for comes
up as a result
+ I need understand the context of where my result(s)
are on a map, in relation to one another, or in
relation to a specific place
+ I want to scan the set of results to compare and
figure out the best fit or best choice
+ I want to take action on my result(s) at MQ (get
more info, print, get directions, share, etc)
+ I want to take action on my result(s) outside of MQ
(visit website, call, etc)
Check if the results fit what I was
looking for. Make a decision about
a choice/set of choices. Take action.
Inputting My
Start & End
Consuming
Directions
Preparing for
Transit
Inputting a SearchIn Transit
Consuming
Results
ROUTE
SINGULAR PLACE SINGULAR PLACE
AREA
SET OF PLACES

45. mapquest
Mental Spaces (Cont.) for users of MapQuest.com (Yogi)
MENTALSPACE
Researching
Place Info
Browsing an
Area/Category
Exploring
Nearby
(find near a place)
Setting Up Tools
& Bookmarks
Trip Planning
(find near a route)
Utilizing Tools &
Bookmarks
MINDSETGOALSTASKSOBJECTS
+ I want to get more info on a result from my finding a
place inquiry
+ I’m researching a specific place out of curiosity
+ I’m researching a specific place that I am at/that I
will be at
+ I’m researching the individual places in a set of
results and comparing them
+ I want to take action on my result(s) at MQ (get
more info, print, get directions, share, etc)
+ I want to take action on my result(s) outside of MQ
(visit website, call, etc)
+ I’m looking for something to do or somewhere to go
+ I have a general area in mind, rather than a specific
place
+ I need to find something within a geographic
boundary (strict, loose, or unstated — could be a
neighborhood, district, city, state, etc)
+ I want to see what options are available within a
category of place (restaurants, activities, etc)
+ I want to do open-ended exploration
+ I want to browse a map to get context for what’s there
+ I want to browse a map or category out of curiosity
Find out what’s out there for me.
Gather a set of choices. Get the lay
of the land.
Get more detailed info about a
place. Make a decision about a
choice/set of choices. Take action.
Search for a
category/
keyword
nearby
Refine by
selecting
additional
criteria
(layers/filters)
Turn on
layers on a
map after
finding a
place
Expand
criteria to
trump
vincinity or
expand area
Type in
category/
keyword and
area
Pan/zoom
my default
map view
Type in area
(neighborhood,
district, city,
state, etc)
Refine
search terms
(Italian
restaurants,
banks open
now)
Turn on
layers on a
map
(category or
area, eg.
neighborhoods)
Type in
category &
assume area
(based on
current map
view)
Refine by
selecting
additional
criteria
(layers/filters)
Re-route to
stops/side
trips
(actually or
mentally)
Evaluate
tradeoffs
(stopping ealier
vs later, exit A
vs exit B, etc)
Search for a
category/
keyword or
turn on layers
(without wiping
out directions)
Utilize other
sites for
research
purposes
Utilize other
sites for
research
purposes
Expand area
to search
within
(”play areas”
along route)
Find best
stopping
points
(by time, available
services, distance,
convenient exits)
Intentionally
save a query
(search terms,
categories/
layers, addt’l
criteria/filters)
Have
MapQuest
remember
my recents
(passive saving of
places/settings )
Intentionally
save a place
(singular)
Intentionally
save a route
Share my
saved things
(places,
queries, routes,
custom maps)
Create
collections
of places/
custom
maps
Set up my
preferences
Set up my
home/work
locations
Have
MapQuest
default to
my settings
(sticky from last
time)
Pull up my
saved things
(places,
queries, routes,
custom maps)
Have
MapQuest
present my
recents
(places)
Continue a
task I
started on
.com on
mobile or in
app
Have
MapQuest
default to
my
preferences
+ I’m looking for something to do or somewhere to go
+ I have a specific place in mind that I want to look near
+ I want to see what options are available within a
category of place (restaurants, activites, etc) nearby a
specific place
+ I want to do targeted exploration
+ I want to browse a map to get context for what’s near
a place
+ I want to plan what to do at a destination (place or
area)
Find out what’s out there for me
near a given place. Gather a set of
choices. Get the lay of the land.
+ I’m looking for something to do or somewhere to go
along my way as I travel
+ I have a route in mind that I want to look near
+ I want to find things I’ll need along a route (gas, food, rest
stops, etc)
+ I want to plan sensible route stops (for convenience, time,
distance, availability of services, etc)
+ I want to do targeted exploration either before I go or
while in transit
+ I want to browse to get context for what’s along my route
+ I want to get a sense of how stops/side trips affect my
overall route
Find out what’s out there for me
along a route. Gather a set of
choices. Get the lay of the land.
+ I want to save/retain information for later use
+ I want to take an action now that will make me be
able to use MapQuest more efficiently in the future
Let me come back to what I’m
doing. Make me more efficient.
+ I want to recall/reference the information I found
earlier
+ I’m using MapQuest and it remembers things that
are useful to me
+ I’m using MapQuest and it remembers my prefer-
ences
+ I’m using MapQuest and it remembers my settings
from last time
Let me come back to what I’m
doing. Make me more efficient.
SET OF PLACES
ROUTE
SINGULAR PLACE SINGULAR PLACE SINGULAR PLACE
AREA AREA
Interact with
map
(pan, zoom
in/out, click on
pins, etc)
Consume
place details
(ratings, hours,
location, etc)
View nearby/
similar/related
things
(places, events,
activities,
lodging, etc)
Get
directions to
the place
Share place
(mobile, URL,
email, social
media, etc)
Print place
info
Take another
action on a
result
outside MQ
(call, visit
website, etc)
Take another
action at
MQ
(book hotel,
make
reservation, etc)
See tasks
for ‘Finding
a Place’
See tasks
for
‘Consuming
Results’
See tasks for
‘Researching
Place Info’
See tasks
for ‘Getting
Directions’
See tasks
for
‘Consuming
Directions’
See tasks for
‘Researching
Place Info’

46. mapquest
GETTING DIRECTIONS Mental Model for users of MapQuest.com (Yogi)
Inputting My
Start & End
Consuming
Directions
Preparing for
Transit In Transit
Inputting a Search
Inputting a
Search
Consuming
Results
Consuming
Results (in Search
Ahead)
Browsing an
Area/Category
EXAMPLE SCENARIOS
POSSIBLE NEXT STEPS
BEARINGS –OR– SPECIFICS BEARINGS –OR– SPECIFICS
EVALUATE &
ADJUST
TRY & REFINE
DECISION MADE!
“I need to drop off a floral delivery to a client at 10135 W. San Juan Way, Littleton, CO 80127 (Re/Max Professionals)”
“I need to go to my tennis match at Glenmoore Country Club.”
“I’m meeting friends at the bowling alley on 16th Street Mall.”
“I’ve got my route!”
Researching
Place Info
Trip Planning
Researching Place Info
ARRIVING FROM
ORGANIC
SEARCH RESULT
“I need to figure out how to get somewhere.”
I want to get my bearings for how to get there
—OR—
I want to figure out the specifics of how to get from point A to point B

47. Phew.
47

48. beta.mapquest.com

49. 49
MapQuest Compare

50. “We think we need to add public
transit to our apps…”
50

51. 01
“Hmmm, there’s something more here than
just public transit…”

52. Oh, dear.
52
Directions Scenario — Route Options
Page-level findings:
• Anything green looks
clickable, so people assumed
they could click the busses
here for more info
• In general people were able
to make a decision about
which route they’d want fairly
quickly
• But some wanted more info:
• the 41 is coming at 11:22
which is in 2 min. If I miss it,
is there another one right
behind it, or should I pick the
other route?
• Where exactly does that 41
bus pick up? Can I make it
there in time? Do I need to
run?
• Which busstop is closest?
• How many blocks is this
walk?
Not noticed by
some – they did
more of a manual
calculation
One person
interpreted this
as “1 minute”
Some wanted
intersection/stop
info all the way
up here in the
flow.
People felt they’d
be able to click here
for more info on
that bus line
People wanted to
be able to see the
next few departures
here, too
Generally noticed
People wondered why we’re
showing them a ridiculous
route with so many transfers.
But the icons help them
deduce # of transfers quickly.
‘Next’ was easily
found
Find Nearby Scenario — Bus-Stop Infosheet
Page
• Peo
ref
the
• Re
pe
ab
som
sho
un
wh
be
no
key
no
• Wi
to
mi
• So
fre
tim
These icons are
confusing, given the
bus-stop context.
Unclear Save To
Favs
Who is
this
phoning?
The transit
authority?
interpreted
as “…” or
“more”
Info
Difference is unclear
when this is interpreted
as “…”
Unclear to some how to get more
times or what to do next… Do
they hit ‘Go’? Do they tap on the
card for the N bus? Can they
swipe the card to show more
times? One person wanted a
‘More Times’ button
Blue highlight not clear
but label noticed
This needs to be
accurate based on
context: Downtown,
Uptown, Inbound,
Outbound
Find Nearby Scenario — Bus Line Details
Page-level findings:
• This view is not what people
expect to see when they get
more info from the previous
page. I think that’s because
their lens is “get me more
information about the N-bus
pickups at this bus stop”, not
“show me everything about
the N-bus line”
• With RT arrival info it’s useful
to communicate how many
min delayed, too
• Some people try to deduce
frequency from the listed
times, partially because
there’s an expectation that
RT is not accurate anyway
Highlight interpreted as
“leaving next” not as RT
Dot and arrow indicating the
stop you’re focused on is
understood by some, but it
takes a minute. Others miss
that altogether.
Unclear to some how to get more
times. Can they swipe to show
more times? Several people
wanted an arrow showing there
were more times or a button to
view later trains

53. Ok, so this Compare thing… Maybe there’s
something to that…
53

54. 14

55. “It’s raining but it’s cheaper to
take the train. I’m trying to see
how convenient it would be
since Ubers are more
expensive when it's raining. I
wasn't at home, but at a
restaurant, so I used google
maps to see how easy it would
be. Seemed like a pretty easy
route!”

56. Ah, yes, Compare. That’s more like it.
56

57. 57

59. 59
Takeaways

60. 60
Takeaway #1:
Figure out the right research, not
ask whether to do it at all.

61. 61
Takeaway #2:
Don’t take sides.

62. 62
Takeaway #3:
Build the right product without
becoming immobilized by
process.

63. 63
Takeaway #4…

64. 14
Stakeholder
Interviews
1
Quantitative
Assessment
2
Revenue
Analyses
1
Brand
Research
Review*
*FutureBrand
studies
300+
Customer
Service Logs
Reviewed
10
In-Depth
Qualitative
User
Interviews
(with concept testing)
April-May 2014
01

65. 8
Core Scenarios
Identified
12
Mental Spaces
Identified
3
Main Flows
Identified
10
Primary Page
States
Identified
416+
Survey
Responses
about Labeling
12
User
Interviews +
Concept
Testing
June 2014
01

66. 15
Rounds of
Wireframing
10
Interactive
Prototypes for
Responsive
Design Testing
11
Rounds of
Visual Design
Wireframe
Minitests in
Local Coffee
Shops
14
Visual Design
Minitests in
Local Coffee
Shops
July-October 2014
01
11
Usability
Sessions on
Responsive
Designs
31

67. 1
Card Sort
6
Usability
Sessions
34
Rounds of
Visual Design*
*We think??
We lost count!
Wireframe
Minitests in
Local Coffee
Shops
14
Visual Design
Minitests in
Local Coffee
Shops
November 2014-Today
01
5
Rounds of
Wireframes*
*We think??
We lost count!
17

68. 68
Takeaway #4:
If we can do it, you can do it too.

69. 69
Thanks!