Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014

Designing to Learn
Creating Eﬀective MVP Experiments
Melissa Perri
@lissijean

Consultant
Lean Product Manager
UX Designer
Teacher
Twitter addict (@lissijean)
Blogger (melissaperri.com)
Hi! I’m Melissa.

@lissijean
• What is an MVP (and why should we care)?
• Setting Up MVP Experiments
• Types of MVPs and How to Choose
• Iterating on MVPs
• Agile + MVP: How does it work together?
Agenda

@lissijean
“Match your taste with
the exceptional ﬁnds of
industry insiders.”

My way is faster.
This isn’t what
I needed.
I don’t know how to
use it.
The information I want to
see isn’t on here.
@lissijean

@lissijean
Validate
Before
Building

@lissijean
Build
Measure
Learn

@lissijean
Iterate your way to
Product/Market Fit
your users

@lissijean
MVPMinimum Viable Product

@lissijean
MVPSmallest amount of work you can do
to test your idea.

@lissijean
MVP = Minimum Feature Set
Agile Methodology MVPs

@lissijean
Focus on problem
not features.

@lissijean
Crappy products
Crappy products everywhere

@lissijean
Good MVPs oﬀer a great
experience to the customer.

@lissijean
Experiment Process
Set Up
!
Design
!
Run
!
Feedback
!
Iterate

@lissijean
Setting up MVP
Experiments
PART I

@lissijean
1. Deﬁne Customer & Problem
2. Investigate assumptions
3. Design a test
4. Measure customer behavior
5. Evaluate success
6. Pivot or persevere
Process

@lissijean
Customers want twitter-like
messages in their feed.

@lissijean
Customers want twitter-like
messages in their feed.
Not a problem.

@lissijean
Real Problem
Customers don’t think the
curators are authentic.

@lissijean
Assumptions
Shoppers care
that curators
are authentic.
Shoppers will
engage more
with real
messages on
site.
Shoppers
think tweets
are
authentic.

@lissijean
Riskiest Assumption
Shoppers care
that curators
are authentic.
Shoppers will
engage more
with curator
messages on
site.
Shoppers
think tweets
are
authentic.

@lissijean
Setting up a test
1. What do we want to learn?
2. How do we measure that?
3. What’s the least we can build to learn?

@lissijean
Will shoppers engage more
with curator messages?
What do we want to learn?

@lissijean
Engagement
Clicks, purchases, added to cart
How do we measure that?

@lissijean
What is the least we can build to learn?
Wizard of Oz
Looks real to the customer,
completely manual on the backend.

@lissijean
How do we measure
engagement?

@lissijean
Clicks on
product.
Purchases. Add to cart.

@lissijean
What do we consider
successful?

@lissijean
Minimum Success Criteria
What is the minimum number we need to
hit to consider this validated?

@lissijean
Time Salary# Peoplex x
Cost to build completely

@lissijean
Minimum
Success Criteria
Improvement needed
to justify cost to build.

@lissijean
25% increase in
engagement in A/B test.

@lissijean
0% change over other
products.

@lissijean
Why didn’t users engage?

@lissijean
User Testing Sessions
watch them use it.
How do we ﬁnd that out?

@lissijean
strategic change in direction
based on learning.
Pivot

@lissijean
Riskiest Assumption
Shoppers will
engage more
with curator
messages.

@lissijean
Will shoppers engage more
with curator messages
through email?

@lissijean
Engagement
Opens, Click throughs, purchases.
How do we measure that?

@lissijean
What is the least we can build to learn?

@lissijean
How do we determine success?
30% increase in engagement over other emails.

@lissijean
Success!
2x Open Rate
3x Purchases

@lissijean
Now, we automate.
Now, we invest.

@lissijean
workshop one:
Setting up MVP Experiments

@lissijean
We’ve recently received a ﬂurry of
complaints from Macy’s shoppers.

@lissijean
“Every time I come into Macy’s, I spend HOURS I
don’t have and can never seem to ﬁnd what I’m
looking for. I know exactly what type of item I want,
but options are all over the place. It’s daunting.”

@lissijean
“I needed a blue tie to go with my suit for a friend’s
wedding, so I went to ﬁnd it but there were too
many options. Where do I even start? I don’t even
know how to choose.”

@lissijean
Who is the customer?

@lissijean
“OpenSky shoppers who
visit the site at least once
a month, but rarely buy.”

@lissijean
What is their problem?

@lissijean
“Do not think the
curators are authentic.”

@lissijean
10 minutes
In your teams, write down your customer
and their problem for Macy’s.

@lissijean
Shoppers who are purchasing
something speciﬁc out of
necessity, not browsing.
Customer Hypothesis

@lissijean
Cannot ﬁnd what they
need eﬃciently.
Problem Hypothesis

@lissijean
What are we assuming about
the customer and the problem?

@lissijean
Riskiest Assumption
What are we the most uncertain about?

@lissijean
10 minutes
In your teams, write down your
assumptions and pick the riskiest one.

@lissijean
Assumptions
Shoppers are
overwhelmed
by the amount
of options.
Shoppers do
not want to
browse.
Shoppers
want help
ﬁnding items.

@lissijean
Now, we design a test for
that riskiest assumption.

@lissijean
Types of MVPs
PART II

@lissijean
WHAT SHOULD YOU
ALWAYS DO FIRST?

@lissijean
Interviews
Get out of the building and talk to people.

@lissijean
• First thing you should do.
• Talk to at least 10 people.
• Verify the problem and the customer hypothesis.
• Verify as many assumptions as you can.
When to do Interviews

@lissijean
Concierge
Delivering a service manually.
Delivering a service manually.
Concierge

@lissijean
The Good The Bad
• Very close to the
customer, easy to learn.
• Can adjust easily based on
feedback.
• Very easy and cheap to
get started.
• Not scalable.
• Time consuming to run.
Concierge

@lissijean
• If you are testing a service.
• Trying to determine what customers will respond to.
• Testing and reﬁning with a small number of people.
When to use Concierge.

@lissijean
Wizard of Oz
Looks real, but manual on the back end.

@lissijean
The Good The Bad
• Looks exactly real to
customers, so very
accurate response to if
you would build it.
• Need to code to get started.
• Can’t tell you why customers
don’t use it.
Wizard of Oz

@lissijean
• Running a test with a large audience.
• Trying to sell a feature or service before it is built,
but pretend it is already there.
• If you are concerned about sacriﬁcing brand.
When to use Wizard of Oz.

@lissijean
Landing Page
A pitch of your product, to gauge idea reception.

@lissijean
The Good The Bad
• Just a pitch, don't need to
reﬁne the whole idea
• Can launch in a matter of
minutes
• MOST PEOPLE FORGET
THEY NEED TO DIRECT
PEOPLE HERE THROUGH
MARKETING.
Landing Page

@lissijean
• Looking for initial customers to test.
• Trying to gauge how much interest there is
in the idea.
• Great way to pitch to a large audience.
When to use Landing Page.

@lissijean
Videos
Demo what your product can do through hacking
it together for a video.

@lissijean
The Good The Bad
• Can demonstrate
complicated things.
• Show a prototype before
producing them all.
• More complicated to put
together.
• Needs to be a certain
amount done before you can
make the video.
Videos

@lissijean
• If your product does something very easily.
• When your product sounds complicated.
• When people need to see it to believe it.
When to use Videos.

@lissijean
What if I’m not solving
a dire need?

@lissijean
workshop two:
Design your test.

@lissijean
You have already
interviewed your customer,
they have that problem.

@lissijean
Now how are you going
to solve that problem?

@lissijean
Choose an MVP strategy
and plan out a test.
30 minutes
• What you are trying to learn?
• Explain every little detail of the test.
• What are you measuring?
• Minimum success criteria.

@lissijean
Resources
Access to 2 Macy’s stores
Access to Macy’s online page
!
3 Store Interns $30,000 salary each
10 Sales Associates In Store $40,000 salary each
5 Developers $100,000 salary each
1 Designer $100,000 salary
Yourself $100,000 salary

@lissijean
Every time you start with
a minimum feature set,
a kitten dies.

@lissijean
Responding to
Constraints &
Feedback
PART III

@lissijean
The goal of an MVP is
to learn.

@lissijean
Restricts medium.
Restricts resources.

@lissijean
workshop three:
Alter your test based on constraints.

@lissijean
You learned that people only have
this problem online.
How do you make an online MVP?

@lissijean
You learned that people only have this
problem in store.
How do you make an in person MVP?

@lissijean
How would you change your MVP
experiment based on these new
constraints?
5 minutes

@lissijean
MVPs in Agile Teams
PART IV

@lissijean
Gather Requirements
Spec
features
Design Interfaces
Developm
ent
QA Process
Release
Waterfall

@lissijean
Gather Requirements
Spec
features
Design Interfaces
Developm
ent
QA Process
Release
Product Manager UX Designer Developer

@lissijean
Individuals and Interactions
over
Processes and Tools

@lissijean
Agile Sprints
Short, goal oriented sprints formed around a
problem running Build, Measure, and Learn.

@lissijean
Product Manager
UX Designer Developer
Small, cross-functional teams

@lissijean
monday
tuesday
wednesday
thursday
friday
Reflect & Define
Specify
Build & Refine
Build & Refine
Customer Feedback
Sprint Structure

@lissijean
Problem
Exploration
MVP
Experiments
Feature
Development

@lissijean
MVPs are only a test.

@lissijean
“My solution is too obvious, if I just build it
people will use it.”

@lissijean
Customers will only use your
features if it ﬁlls a need.
Customers will only use your
features if it ﬁlls a need.

@lissijean
“Lean is just for startups, my
company is too grown up.”

@lissijean
Lean applies to all companies.

@lissijean
Melissa Perri
!
Want an MVP Guide?
melissa@produxlabs.com
!
produxlabs.com
melissaperri.com
!
!
@lissijean

Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014

Similar to Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014 (20)

More from Melissa Perri

More from Melissa Perri (9)

Recently uploaded

Recently uploaded (16)

Designing To Learn: Creating Effective MVP Experiments - Lean UX NYC 2014