TESTING
YOUR
MVP
Tathagat Varma
@TathagatVarma
http://thoughtleadership.in
BUILD AND THEY WILL
COME…REALLY?
• Apple Newton, 1993-98, Invested $100m
• Webvan, 1999-01, Lost $800m
• Microsoft Zune, 2006-11,
• Amazon Fire, 2014, $170m written-off
• Target Canada, 2013-15, Over $7b lost
• …and many, many more!!!
WHY DID THEY “FAIL”?
Bad idea?
Incompetent team?
Lack of funding?
Dated technology?
Wrong market?
…?
90%
STARTUPS FAIL!
SO, WHAT’S A STARTUP’S
CONTEXT ?
• A bold experiment to the question
“should this business be built?”
• Startups search for their repeatable
and scalable business model.
• Early and frequent feedbacks allow
course correction and realignment
with the vision.
3 STAGES OF A STARTUP
Problem
Solution
Fit
Stage1
Product
Market
FitStage2
Scaling
Up
Stage3
“Do I have a
problem worth
solving?”
“Have I built
something
people want?”
“How do I
accelerate
growth?”
LEARNING IS PROGRESS!
https://blog.ycombinator.com/minimum-viable-product-process/
A STARTUP’S MOST
SCARCE RESOURCE…
TIME
BUILDING “100%
PRODUCT” IS TOO…
• Time-consuming
• Costly
• Risky
• …even boring :)
• So, what’s the alternative???
“MINIMUM VIABLE
PRODUCT”
Coined by Frank Robinson in 2001 as “that
unique product that maximizes return on
risk for both the vendor and the
customer.”
“MVP is a mindset of the
management and
development-team. It
says, think big for the long
term but small for the
short term.”
MVP…
“A(n) MVP helps entrepreneurs start the process of
learning as quickly as possible.
It is not necessarily the smallest product imaginable,
though; it is simply the fastest way to get through the
Build-Measure-Learn feedback loop with minimum
effort.
Unlike a prototype or concept test, an MVP is designed
not just to answer product design or technical questions.
Its goal is to test fundamental business hypotheses”
- Eric Ries
LIKE THIS…
MVP MYTHS…
• Tech = MVP
• Code = MVP
• Prototype = MVP
• Beta = MVP
• Half-baked = MVP
• MVP = Faster TTM / Revenue
• MVP = One MVP
MVP IS…
• The Mindset that we don’t have all the
answers, but the market does know them,
and if we can involve the market, we can
learn the answers.
• The Process of identifying riskiest
assumptions, designing the smallest and
cheapest experiments to acquire validated
learning, and correct the course, as often
as required.
WHY TEST THE MVP?
• Not so much to validate what we
already know, but to learn what we
don’t really know!
• There’s no such thing as the expected
output. We seek learning, not
validation.
• Whatever you get is the right output!
OK, SHOW MY HOW!
• Let’s deconstruct the startup process
• Our interest is to get to Product /
Market Fit (Stage 2)
• What kind of MVPs might help?
• How do we “test” them?
#1
IS THERE A PROBLEM
WORTH SOLVING?
START WITH PERSONAL
PAINPOINT…
TO SOLVING OTHER’S
PROBLEMS…
SEARCH TRENDS
SEARCH CAMPAIGNS
LANDING PAGE
• “Sell first, build later”
• Joel Gascoigne had an
idea for service to
queue up social
media / twitter
updates.
• Before building it, he
wanted to see if people
would need or use it?
AUDIENCE BUILDING
• Choose a low-cost mechanism to build a
virtual community tied to your product.
• Ryan Hoover started newsletter “Product
Hunt” in 20 minutes. After two weeks, he had
130 subscribers. Today, it is valued $22m
• Simplilearn started as blog, then grew into
video content for PMP aspirants…had over
300 courses, trained over 200,000
students…raised over $28m till now…
BLOGS / UPDATES
SLIDE DECKS
VIDEOS
AD CAMPAIGNS
#2
WHAT IS THE PAIN
POINT?
USER REVIEWS
SURVEYS
CUSTOMER INTERVIEWS
• Steve Blank - There are no facts
inside the building…get out of the
building!
• Exploratory conversations that allow
you to learn and understand about
customer problems
• Be an ethnographer, not a salesman!
INNOVATION GAMES
CUSTOMER JOBS
#3
WHO HAS THE PAIN
POINT?
EMPATHY MAP
Source: Gamestorming
USER PERSONAS
#4
HOW ARE THEY
DEALING TODAY?
JOURNEY MAPS
http://cargocollective.com/flaminiazannotti/filter/design-method/2-in-1-A-design-driven-strategy-for-Intellego-Technologies
CONCIERGE MVP
• Humans substitute for systems AND are
visible!
• When you want to learn deeply about how
potential users solve a problem
WIZARD OF OZ MVP
• Problem: In 1999, selling shoes online wasn’t so
obvious. Nick Swinmurm wanted to prove that the
idea was feasible without spending thousands of
dollars of inventory.
• Approach: Nick took pictures of shoes and created
visual catalog. When customers placed order, he
bought those shoes from stores.
• Results: He could prove the concept. After an
unsuccessful attempt to launch a competitive product,
Amazon bought Zappos for US$ 1.2 Billion in 2009.
Today Zappos annual revenues are ~US$ 2 Billion.
#5
WHAT WOULD
DELIGHT THEM?
PAPER & PHYSICAL PROTOTYPES,
STORYBOARDS, BODYSTORMING
SOME FAMOUS ONES…
TwitterSouthwestInternet
DIGITAL PROTOTYPES
POTEMKIN VILLAGE
FAKE DOORS
A/B TESTING
VALUE PROPOSITION
CANVAS
#6
WHAT WOULD THEY
PAY FOR IT?
LANDING PAGE
• After the initial
experiment with landing
page worked, Joel
experimented with
payment options
• Some people clicked on
paid plans!
• First paying customers in
4 days of launch of a
version with rough
edges!
PRE-ORDER PAGES
PAY WHAT YOU LIKE
SOFTWARE TESTING
• FiveSecondTest, QuickMVP,
OpenHallway
• Keynotopia
• bootstrap.js
• Google forms, wordpress, Google
Analytics, KISSmetrics,
PITFALLS OF MVP
TESTING
• Confirmation bias
• Starting too late
• Expedite time-to-market
• Accelerate revenues
• “Selling” the solution
• Consider “Fail Fast” as “failure”
• Ignore the feedback!
CHOOSING THE RIGHT
KIND OF MVP
• Validating MVPs use a worse product than what the final
version will be, so success proves your model but failure is
inconclusive. E.g. Dropbox
• Invalidating MVPs, however, have a better product than
the final business, so failure means the business model is
doomed but success is inconclusive. E.g. RentThe Runway
• Unfortunately, we usually pick the wrong kind of MVP: we
tend to choose validating MVPs, even though most
businesses fail because they persist too long in solving
the wrong problem — what invalidating MVPs are
designed to prevent.
https://hbr.org/2013/09/building-a-minimum-viable-prod
RECAP
• Startups exist to search for business
model.
• Building 100% product based simply on
assumptions is fatal!
• MVPs as a process allow systematic
validation of riskiest hypothesis.
• Testing the MVPs allows validated
learning to drive further decisions.
REFERENCES
• https://www.inc.com/steve-blank/how-test-prototype-minimum-viable-product.html
• https://www.sitepoint.com/7-new-ways-to-test-your-minimum-viable-product/
• https://envato.com/blog/straightforward-tactics-testing-minimum-viable-products/
• https://thenextweb.com/dd/2014/11/12/15-ways-test-minimum-viable-product/
#.tnw_1njw4EHg
• https://blog.ycombinator.com/minimum-viable-product-process/
• https://blog.kissmetrics.com/abcs-of-mvps/
• http://customerdevlabs.com/2016/04/06/what-does-minimum-viable-product-mean/
• https://spin.atomicobject.com/2017/02/14/mvp-minimal-enough/
• https://hackerchick.com/5-mvp-examples/
• https://productcoalition.com/using-data-from-an-mvp-c3bdcc2df72b
• https://www.typeform.com/blog/guides/minimum-viable-product/
• http://fresco.vc/build-an-mvp-not-a-beta/
• http://www.allencheng.com/concierge-mvp/
• https://design.org/blog/the-wrong-and-right-way-to-build-an-mvp/
• http://www.syncdev.com/minimum-viable-product/
REFERENCES…
• https://www.sitepoint.com/7-new-ways-to-test-your-minimum-viable-product/
• https://envato.com/blog/straightforward-tactics-testing-minimum-viable-products/
• https://www.kickstarter.com/projects/getpebble/pebble-e-paper-watch-for-iphone-and-android
• https://techcrunch.com/2011/10/19/dropbox-minimal-viable-product/
• https://hackerchick.com/how-to-build-an-mvp/
• https://medium.com/@joelgascoigne/how-to-successfully-validate-your-idea-with-a-landing-page-mvp-
ef3c2d02dc51
• https://blog.bufferapp.com/idea-to-paying-customers-in-7-weeks-how-we-did-it
• https://hbr.org/2013/09/building-a-minimum-viable-prod
• https://blog.kissmetrics.com/abcs-of-mvps/
• http://original.livestream.com/nytechmeetup/video?clipId=pla_df402e6b-fcb6-4ea0-9223-ed516884ea34
• https://vimeo.com/24749599
• https://vimeo.com/24744647
• https://techcrunch.com/2011/10/19/dropbox-minimal-viable-product/
Q&A
@TathagatVarma
tathagat.varma@gmail.com

Testing Your MVP

  • 1.
  • 2.
    BUILD AND THEYWILL COME…REALLY? • Apple Newton, 1993-98, Invested $100m • Webvan, 1999-01, Lost $800m • Microsoft Zune, 2006-11, • Amazon Fire, 2014, $170m written-off • Target Canada, 2013-15, Over $7b lost • …and many, many more!!!
  • 3.
    WHY DID THEY“FAIL”? Bad idea? Incompetent team? Lack of funding? Dated technology? Wrong market? …?
  • 5.
  • 7.
    SO, WHAT’S ASTARTUP’S CONTEXT ? • A bold experiment to the question “should this business be built?” • Startups search for their repeatable and scalable business model. • Early and frequent feedbacks allow course correction and realignment with the vision.
  • 8.
    3 STAGES OFA STARTUP Problem Solution Fit Stage1 Product Market FitStage2 Scaling Up Stage3 “Do I have a problem worth solving?” “Have I built something people want?” “How do I accelerate growth?”
  • 9.
  • 10.
    A STARTUP’S MOST SCARCERESOURCE… TIME
  • 11.
    BUILDING “100% PRODUCT” ISTOO… • Time-consuming • Costly • Risky • …even boring :) • So, what’s the alternative???
  • 12.
    “MINIMUM VIABLE PRODUCT” Coined byFrank Robinson in 2001 as “that unique product that maximizes return on risk for both the vendor and the customer.” “MVP is a mindset of the management and development-team. It says, think big for the long term but small for the short term.”
  • 13.
    MVP… “A(n) MVP helpsentrepreneurs start the process of learning as quickly as possible. It is not necessarily the smallest product imaginable, though; it is simply the fastest way to get through the Build-Measure-Learn feedback loop with minimum effort. Unlike a prototype or concept test, an MVP is designed not just to answer product design or technical questions. Its goal is to test fundamental business hypotheses” - Eric Ries
  • 14.
  • 15.
    MVP MYTHS… • Tech= MVP • Code = MVP • Prototype = MVP • Beta = MVP • Half-baked = MVP • MVP = Faster TTM / Revenue • MVP = One MVP
  • 16.
    MVP IS… • TheMindset that we don’t have all the answers, but the market does know them, and if we can involve the market, we can learn the answers. • The Process of identifying riskiest assumptions, designing the smallest and cheapest experiments to acquire validated learning, and correct the course, as often as required.
  • 17.
    WHY TEST THEMVP? • Not so much to validate what we already know, but to learn what we don’t really know! • There’s no such thing as the expected output. We seek learning, not validation. • Whatever you get is the right output!
  • 18.
    OK, SHOW MYHOW! • Let’s deconstruct the startup process • Our interest is to get to Product / Market Fit (Stage 2) • What kind of MVPs might help? • How do we “test” them?
  • 19.
    #1 IS THERE APROBLEM WORTH SOLVING?
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
    LANDING PAGE • “Sellfirst, build later” • Joel Gascoigne had an idea for service to queue up social media / twitter updates. • Before building it, he wanted to see if people would need or use it?
  • 25.
    AUDIENCE BUILDING • Choosea low-cost mechanism to build a virtual community tied to your product. • Ryan Hoover started newsletter “Product Hunt” in 20 minutes. After two weeks, he had 130 subscribers. Today, it is valued $22m • Simplilearn started as blog, then grew into video content for PMP aspirants…had over 300 courses, trained over 200,000 students…raised over $28m till now…
  • 26.
  • 27.
  • 28.
  • 29.
  • 30.
    #2 WHAT IS THEPAIN POINT?
  • 31.
  • 32.
  • 33.
    CUSTOMER INTERVIEWS • SteveBlank - There are no facts inside the building…get out of the building! • Exploratory conversations that allow you to learn and understand about customer problems • Be an ethnographer, not a salesman!
  • 34.
  • 35.
  • 36.
    #3 WHO HAS THEPAIN POINT?
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
    CONCIERGE MVP • Humanssubstitute for systems AND are visible! • When you want to learn deeply about how potential users solve a problem
  • 42.
    WIZARD OF OZMVP • Problem: In 1999, selling shoes online wasn’t so obvious. Nick Swinmurm wanted to prove that the idea was feasible without spending thousands of dollars of inventory. • Approach: Nick took pictures of shoes and created visual catalog. When customers placed order, he bought those shoes from stores. • Results: He could prove the concept. After an unsuccessful attempt to launch a competitive product, Amazon bought Zappos for US$ 1.2 Billion in 2009. Today Zappos annual revenues are ~US$ 2 Billion.
  • 43.
  • 44.
    PAPER & PHYSICALPROTOTYPES, STORYBOARDS, BODYSTORMING
  • 45.
  • 46.
  • 47.
  • 48.
  • 49.
  • 50.
  • 51.
  • 52.
    LANDING PAGE • Afterthe initial experiment with landing page worked, Joel experimented with payment options • Some people clicked on paid plans! • First paying customers in 4 days of launch of a version with rough edges!
  • 53.
  • 55.
  • 56.
    SOFTWARE TESTING • FiveSecondTest,QuickMVP, OpenHallway • Keynotopia • bootstrap.js • Google forms, wordpress, Google Analytics, KISSmetrics,
  • 57.
    PITFALLS OF MVP TESTING •Confirmation bias • Starting too late • Expedite time-to-market • Accelerate revenues • “Selling” the solution • Consider “Fail Fast” as “failure” • Ignore the feedback!
  • 58.
    CHOOSING THE RIGHT KINDOF MVP • Validating MVPs use a worse product than what the final version will be, so success proves your model but failure is inconclusive. E.g. Dropbox • Invalidating MVPs, however, have a better product than the final business, so failure means the business model is doomed but success is inconclusive. E.g. RentThe Runway • Unfortunately, we usually pick the wrong kind of MVP: we tend to choose validating MVPs, even though most businesses fail because they persist too long in solving the wrong problem — what invalidating MVPs are designed to prevent. https://hbr.org/2013/09/building-a-minimum-viable-prod
  • 59.
    RECAP • Startups existto search for business model. • Building 100% product based simply on assumptions is fatal! • MVPs as a process allow systematic validation of riskiest hypothesis. • Testing the MVPs allows validated learning to drive further decisions.
  • 60.
    REFERENCES • https://www.inc.com/steve-blank/how-test-prototype-minimum-viable-product.html • https://www.sitepoint.com/7-new-ways-to-test-your-minimum-viable-product/ •https://envato.com/blog/straightforward-tactics-testing-minimum-viable-products/ • https://thenextweb.com/dd/2014/11/12/15-ways-test-minimum-viable-product/ #.tnw_1njw4EHg • https://blog.ycombinator.com/minimum-viable-product-process/ • https://blog.kissmetrics.com/abcs-of-mvps/ • http://customerdevlabs.com/2016/04/06/what-does-minimum-viable-product-mean/ • https://spin.atomicobject.com/2017/02/14/mvp-minimal-enough/ • https://hackerchick.com/5-mvp-examples/ • https://productcoalition.com/using-data-from-an-mvp-c3bdcc2df72b • https://www.typeform.com/blog/guides/minimum-viable-product/ • http://fresco.vc/build-an-mvp-not-a-beta/ • http://www.allencheng.com/concierge-mvp/ • https://design.org/blog/the-wrong-and-right-way-to-build-an-mvp/ • http://www.syncdev.com/minimum-viable-product/
  • 61.
    REFERENCES… • https://www.sitepoint.com/7-new-ways-to-test-your-minimum-viable-product/ • https://envato.com/blog/straightforward-tactics-testing-minimum-viable-products/ •https://www.kickstarter.com/projects/getpebble/pebble-e-paper-watch-for-iphone-and-android • https://techcrunch.com/2011/10/19/dropbox-minimal-viable-product/ • https://hackerchick.com/how-to-build-an-mvp/ • https://medium.com/@joelgascoigne/how-to-successfully-validate-your-idea-with-a-landing-page-mvp- ef3c2d02dc51 • https://blog.bufferapp.com/idea-to-paying-customers-in-7-weeks-how-we-did-it • https://hbr.org/2013/09/building-a-minimum-viable-prod • https://blog.kissmetrics.com/abcs-of-mvps/ • http://original.livestream.com/nytechmeetup/video?clipId=pla_df402e6b-fcb6-4ea0-9223-ed516884ea34 • https://vimeo.com/24749599 • https://vimeo.com/24744647 • https://techcrunch.com/2011/10/19/dropbox-minimal-viable-product/
  • 62.