The document discusses product-market fit and lean startup methodology. It provides guidance on building a minimum viable product (MVP), measuring key metrics, iterating based on customer feedback, and achieving product-market fit. Key steps include clarifying the problem to solve with an MVP, building the MVP, conducting qualitative and quantitative analysis, iterating the product, and repeating until product-market fit is achieved. The document emphasizes measuring the right metrics, getting qualitative customer feedback, and using data to guide continuous improvement.
Special Purpose Vehicle (Purpose, Formation & examples)
Startup Science 2017 ⑥
1. Startup Science ⑥
"Making a successful startup is an Art.
Making a startup that doesn’t fail is a Science.”
Created by
Masa Tadokoro
2. Product-Market Fit
Deliver the MVP
to the first
customer
Clarify what to
learn with an
MVP
Figure out things to
learn using the MVP
Chose MVP type
and build
Learn by delivering
to an evangelist customer
Conduct Qualitative analysis by
interviews with customers and
Quantitative analysis by
innovation metrics
Examine the UX
with Hooked model
Improve UX to
Enhance
the customer
engagement
Iterate product to
bring more value
to customers
Iterate product/ launch
it
Conduct Quantitative
& Qualitative Analysis
Build MVP
Repeat the
process until
the PMF has
been achieved
Create
Highly adaptive teamPivot the business model
Pivot the
Business Model
Copyright 2017 Masayuki Tadokoro All rights reserved
4. Build
Ideas
Product
Mea
sure
Data
Learn
Use measured data
for product iteration
What is a Lean Startup?
With a lean startup, you build the product for a
new idea or concept quickly (build MVP) and then
iterate on a build-measure-learn cycle based on
the customer's feedback of your product.
Your hypothesis is validated during the iteration
cycles.
By using the lean start-up method, start-up
founders will be able to find a product-market-fit
before burning up all of the resources.
6. If you can’t measure it,
you can’t improve it.
P.F. Drucker
7. Question
❓
What to measure at the
early stage of a startup?
Copyright 2017 Masayuki Tadokoro All rights reserved
8. AARRR!
AARRR is a framework
to put users in a funnel
from acquisition to revenue
Acquisition: User acquisition
Activation: User sign-up
Retention: User comes back
Revenue: User pays for service
Referral: User introduces
other users
Introducing AARRR, pirate metrics.
9. Revenues increase as much as the
resources put into marketing.
5 times marketing costs leads to 5 times
customer acquisitions.
Copyright 2017 Masayuki Tadokoro All rights reserved
10. Before achieving product-market fit
(A state where you can create products what
customers need), customer acquisition by word of
mouth doesn’t work well; on the contrary, it will
lead a negative result.
Don’t stick to referral or customer
acquisition before achieving PMF.
Copyright 2017 Masayuki Tadokoro All rights reserved
11. Story Kanban Board
Backlog In Progress Completed Validation
Backlog story
Extract stories that
you want to learn in
the iteration
Build
Build
completed
launch
Quantitative
analysis (split
test, cohort
analysis)
Qualitative
analysis/introspection
by customer interview
MVP
story
Story
1-1
Story
1-2
Copyright 2017 Masayuki Tadokoro All rights reserved
Measure
Activation/
Retention/
Revenue
12. KGI
Acquisition Activation Retention Revenue
KPI Number of UU
accessing to the
sign up screen
Leaving rates
in the menu
screen
Percentage of
users starting
watching ads
Ad watch
completion
rates
Rates of
starting
using the
app
Factorize KGIs in each stage to actionable
KPIs
Leaving
rates in the
log in screen
Leaving
rates in the
menu
screen
Average number of ad
views per user
Sign up rates
Copyright 2017 Masayuki Tadokoro All rights reserved
Log in rates in 3days
after registering
Average number of ad
views per user
Percentage of users
who completed the
sequence
Sign up rates
Number of UU
accessing to the
sign up screen
First
Sign up
screen
Menu
screen
Start
watching
ad
Complete
watching
Wifi
usage
confirmation
Start
using wifi
Re-log in
screen
Menu
screen
Start
watching
ad
Complete
watching
13. KGI
Acquisition Activation Retention Revenue
KPI Number of UU
accessing to the
sign up screen
Leaving rates
in the menu
screen
Percentage of
users starting
watching ads
Ad watch
completion
rates
Rates of
starting
using the
app
Factorize KGIs in each stage to actionable
KPIs
Leaving
rates in the
log in screen
Leaving
rates in the
menu
screen
Average number of ad
views per user
Sign up rates
Copyright 2017 Masayuki Tadokoro All rights reserved
Log in rates in 3days
after registering
Average number of ad
views per user
Percentage of users
who completed the
sequence
Sign up rates
Number of UU
accessing to the
sign up screen
First
Sign up
screen
Menu
screen
Start
watching
ad
Complete
watching
Wifi
usage
confirmation
Start
using wifi
Re-log in
screen
Menu
screen
Start
watching
ad
Complete
watching
Result 335 ppl 302 ppl 93 ppl 23 ppl
4.2 times
( Total 96 tims)
14. Result of
measurement of
MVP (actual
number)
Result of
measurement of
MVP (rates)
Acquisition
(Visitors)
332 ppl 100%
Activation
(Registers) 305 ppl 90%
Activation
(completed
sequence)
93 ppl 30%
Retention
(Revisit in 3 days) 23 ppl 25%
Revenue
(Average ad views in a
day per user)
4.2 times 4.2 times
Show results quantitively using rates.
describe
the rates
Copyright 2017 Masayuki Tadokoro All rights reserved
15. Why is measurement needed?
Proper recognition
where you are heading
Recognizing a gap
between reality and goal may lead action
items to fill in the gap
KPIs will be
common language
among stakeholders
Copyright 2017 Masayuki Tadokoro All rights reserved
16. Recognizing a gap between reality and goal,
you may lead action items to fill in the gap
Copyright 2017 Masayuki Tadokoro All rights reserved
Result of
measurement of
MVP (actual
number)
Result of
measurement of
MVP (rates)
Goal
Acquisition
(Visitors)
332 ppl 100% 100%
Activation
(Registers) 305 ppl 90% 95%
Activation
(completed
sequence)
93 ppl 30% 80%
Retention
(Revisit in 3 days) 23 ppl 25% 80%
Revenue
(Average ad views in a
day per user)
4.2 times 4.2 times 10 times
Object to
achieve
17. What is a good indicator?
A good indicator
is measurable
A good indicator
Is easy to see
A good indicator
is actionable
A good indicator
is an indicator
of cause & effect
(An indicator
occurs changes
in other indicators)
Copyright 2017 Masayuki Tadokoro All rights reserved
18. Traps you may fall into
when setting KPIs.
See only result indicators:
UU, PV, revenue, CPA,
Etc.
See correlation indicators.
and repeater rates,
You mistakenly see a correlation,
you may not come up with the idea not
cause and effect relationships of what to do
next
See unactionable
indicators:
Indicating rough new user
rates
Copyright 2017 Masayuki Tadokoro All rights reserved
19. You won’t be able to set perfect KPIs for
your startup at the beginning.
You will continuously verify
whether you are measuring
appropriate KPIs at each stage.
20. Throw Vanity Metrics Away
Page view: Number of visitors to the page;
it is a vanity metric if you do not have
ad-sense or banner ads.
Number of visitors: It is a vanity metric since you do not
know what types of people are coming how frequently.
Unique visitors: It is a vanity metric since you do not know why
users came and what they have done
Visit duration: You have to measure which duration
for specific pages. If customers are staying on FAQ or
support pages for a long time, it indicates that usability is bad
Sign-up number:
It is meaningless if you can not
verify open rate of e-mails
Follower, Fan, Like: It is meaningless if it is just a number.
Key point is how many people will take an action
when you post something on social media stream
Copyright 2017 Masayuki Tadokoro All rights reserved
21. So
What ?
Vainglory indicators show a
realistic state but it would say, "so
what?" and are un-actionable.
Copyright 2017 Masayuki Tadokoro All rights reserved
22. With vanity metrics,
you will be reassured,
but it won't provide you
a clear guidance for
what you should do next.
- Eric Ries
Lean Startup
Copyright 2017 Masayuki Tadokoro All rights reserved
23. But
the revenue is flat
When you
measure
page views,
it goes up to right
time
Page
View
page view
revenue
You can easily find a metric which
is “goes up to right”.
24. We tend to only see what we
want.
People live in wrapped reality.
Copyright 2017 Masayuki Tadokoro All rights reserved
25. What you want to learn
Can customers gain wifi data volume and utilize it?
Verified story to learn
A user activates the app and signs up.
He/she watches add and utilize gained data
volume
What KPIs do you measure to verify?
What you build Cost & time to build
Quantitative result Qualitative result
What you learned
What you want to learn from the next sprint
Iteration Canvas
Copyright 2017 Masayuki Tadokoro All rights reserved
Activation: Sign up rates, rates of completion of the sequence
Retention: Rates of re-logging in in 3 days after registration
Revenue: Average # of ad views per a day
20 ppl per a day/2 weeks
Activation: 30%
Retention:25%
Revenue: 4 times in a day
26. Listen to the story behind the
numbers.
Copyright 2017 Masayuki Tadokoro All rights reserved
27. Story Kanban Board
Backlog In Progress Completed Validation
Backlog story
Extract stories that
you want to learn in
the iteration
Build
Build
completed
launch
Quantitative
analysis (split
test, cohort
analysis)
Qualitative
analysis/introspection
by customer interview
MVP
story
Story
1-1
Story
1-2
Copyright 2017 Masayuki Tadokoro All rights reserved
Get customer
interviews
28. A user story is built on the user's subjective view
through quantitative data only, you may not get
any subjective insight.
Copyright 2017 Masayuki Tadokoro All rights reserved
29. Interview with customers who have
actually touched the MVP.
Copyright 2017 Masayuki Tadokoro All rights reserved
30. ・Did you see any values on this product?
・What are top 3 features of value?
・Why did you feel that the features are valuable?
・What features didn’t you use and feel less valuable?
・Why these features them less valuable?
・Will you recommend this product to your family or
friends?
Question examples in an MVP interview.
Copyright 2017 Masayuki Tadokoro All rights reserved
31. Start-up Founder can speak directly to customers
while producing the product.
Competitive Advantage of Startups
Copyright 2017 Masayuki Tadokoro All rights reserved
32. Dig Out Insight from
The Customer
Copyright 2017 Masayuki Tadokoro All rights reserved
33. Customers’ viewpoint itself is superficial
or just an amateur analysis in many
cases.
Only listening to customers is
worthless.
Copyright 2017 Masayuki Tadokoro All rights reserved
36. Interviewer: Thank you for sparing your time today.
Catherin (customer): You're welcome.
Interviewer: You're using our product this 1 week, so how is
that?
Catherin (customer): I'm satisfied with it. Gaining wifi data
volume by watching ads is really convenient.
Interviewer: Well, then, why do you feel it is convenient?
Catherin (customer): I love to watch Youtube anywhere,
anytime, and then I can add data volume anytime when the
data volume becomes less.
Customer Interview
Copyright 2017 Masayuki Tadokoro All rights reserved
37. Interviewer: I see. You lose data volume while watching
Youtube and then you'd like to get data volume?
Catherin (customer): Yes, watching Youtube for 5 minutes
consumes 10MB and then I'd watch ads and add 5MB
repeatedly.
Interviewer: Well, that's interesting. What do you think of the
process?
Catherin: Honestly, it is a little tiresome. I think it is easier to
use if customers can add data volume at one time.
Copyright 2017 Masayuki Tadokoro All rights reserved
38. Voice of customer
:Really convenient
Insight:can continually
add data volume
to watch movies
Insight: watching adds
to gain data volume
is tiresome
More convenient
if customers can gain
more amounts of
data volume one time
Example of Anywhere Online
So what?
So what?
So what?
Copyright 2017 Masayuki Tadokoro All rights reserved
39. What you want to learn
Can customers gain wifi data volume and utilize it?
Verified story to learn
A user activates the app and signs up.
He/she watches add and utilize gained data
volume
What KPIs do you measure to verify?
What you build Cost & time to build
Quantitative result Qualitative result
Utilization is tiresome but customers use the
solution, feeling its value.
What you learned
What you want to learn from the next sprint
Iteration Canvas
Copyright 2017 Masayuki Tadokoro All rights reserved
Activation: Sign up rates, rates of completion of
the sequence Retention: Rates of re-logging in
3 days after registration
Revenue: Average # of ad views per a day
20 ppl per a day/2 weeks
Activation: 30%
Retention:25%
Revenue: 4 times in a day
41. It’s no good to give up an MVP quickly.
Hang up for a certain period
Attain an amount of qualitative data and
feedback.
Copyright 2017 Masayuki Tadokoro All rights reserved
42. What you want to learn
Can customers gain wifi data volume and utilize it?
Verified story to learn
A user activates the app and signs up.
He/she watches add and utilize gained data
volume
What KPIs do you measure to verify?
What you build Cost & time to build
Quantitative result Qualitative result
Utilization is tiresome but customers use the
solution, feeling its value.
What you learned
Customers feel its value and use it but there is less motivation to stick to the product because of its
fewer amounts of gained data volume.
What you want to learn from the next sprint
Iteration Canvas
Copyright 2017 Masayuki Tadokoro All rights reserved
Activation: Sign up rates, rates of completion of the sequence
Retention: Rates of re-logging in in 3 days after registration
Revenue: Average # of ad views per a day
20 ppl per a day/2 weeks
Activation: 30%
Retention:25%
Revenue: 4 times in a day
43. Verbalize and verify what you learned
through MVP with your team
Your team
Copyright 2017 Masayuki Tadokoro All rights reserved
” What we learned through MVP”
44. What to Verify with your Team
・What the reason customers use the product?
・Which features customers see valuable? Why?
・Why don’t customers use the product?
・Where is right and wrong in product hypothesis we built?
・Is our imagined appraisal standards of the product are
overlapped somehow with those of customers? Where are
overlapped or not?
・What is the most precious lesson from MVP?
・Which existing features are needed improvement? Which
existing features needs to do away? What kinds of features are
needed to add on?
Copyright 2017 Masayuki Tadokoro All rights reserved
45. Put organized knowledge
creation in a process.
Tacit
Knowledge
Tacit
Knowledge
TacitTacit
Formal
Knowledge
Formal
Knowledge
FormalFormal
Indiv
idual
Indiv
idual
G
I
I
I
I
I
I
Indiv
idual
G
G G
G
Acquiring tacit
knowledge through
body, five senses,
and direct
Experience
」
Creation of
hypothesis
of concept by
interview,
policy, metaphor
Systematization of
knowledge based
on composition of
formal knowledge
Formalize formal
knowledge
Understand it as
a new tacit
knowledge
46. New knowledge (= insight) is invented in a spiral
which is created by subjective ‘tacit knowledge’
based on one’s experience and objective ‘formal
knowledge’ that is able to be verbalized that are
transformed from and to each other.
Yujiro Nonaka
Emeritus professor
of Hitotsubashi University
Copyright 2017 Masayuki Tadokoro All rights reserved
49. No
Yes
Add it on Story Kanban board as
story
Launch MVP and conduct
qualitative and quantitative
analysis
Achieved PMF?
Can improve it if continuing to use
the same business model?
Conduct transition of the
organization to scale
Yes
Consider to pivot
No
Yes
No
Flow chart to decide the next step.
Yes
No
Add it on issue Kanban board as
feature
Takes amounts of
features/issues?
Yes
No
Takes an amounts of
features/issues?
Verify UX/function improvement of
existing features
Provide enough values with only
existing features?
Verify product iteration added
features
Copyright 2017 Masayuki Tadokoro All rights reserved
Add it on issue Kanban board as
feature
Add it on Story Kanban board as
story
51. No
Yes
Add it on Story Kanban board as
story
Launch MVP and conduct
qualitative and quantitative
analysis
Achieved PMF?
Can improve it if continuing to use
the same business model?
Conduct transition of the
organization to scale
Yes
Consider to pivot
No
Yes
No
Flow chart to decide the next step.
Yes
No
Add it on issue Kanban board as
feature
Takes amounts of
features/issues?
Yes
No
Takes an amounts of
features/issues?
Verify UX/function improvement of
existing features
Provide enough values with only
existing features?
Verify product iteration added
features
Copyright 2017 Masayuki Tadokoro All rights reserved
Add it on issue Kanban board as
feature
Add it on Story Kanban board as
story
52. Without a PMF, the value of the product is not propagated to
customers by word of mouth and customers won't be
aggressive about using it. Any media would never applaud it
and selling cycles rotate slowly.
However, with a PMF, it's very clear.
Customers come to us to buy products as we produce.
Adding services, the number of usage increases and we
might hire more sales and customer support staff and
journalists will contact me.
- Marc Andreessen
Copyright 2017 Masayuki Tadokoro All rights reserved
53. Check form of standards of
PMF achievement.
・ Maintain high retention rates?
・ Is the process from customer acquisition to
increasing revenue regeneratable?
・ Would the business model generated increase the
revenue according to the LTV>CPA model?
・ Is it built in accordance with lean canvas?
Copyright 2017 Masayuki Tadokoro All rights reserved
54. Shawn Elise
If 40% of users answer that ‘it is painful for
me if the product is removed from the
market', the product will consecutively
gain customers in the market.
Copyright 2017 Masayuki Tadokoro All rights reserved
“Traction”
56. Product-Market Fit
Deliver the MVP
to the first
customer
Clarify what to
learn with an
MVP
Figure out things to
learn using the
MVP
Chose MVP type
and build
Learn by delivering
to an evangelist customer
Conduct Qualitative analysis by
interviews with customers and
Quantitative analysis by
innovation metrics
Examine the UX
with Hooked model
Improve UX to
Enhance
the customer
engagement
Iterate product to
bring more value
to customers
Iterate product/ launch
it
Conduct
Quantitative &
Qualitative Analysis
Build MVP
Repeat the
process until
the PMF has
been achieved
Create
Highly adaptive teamPivot the business model
Pivot the
Business Model
Copyright 2017 Masayuki Tadokoro All rights reserved
57. No
Yes
Add it on Story Kanban board as
story
Launch MVP and conduct
qualitative and quantitative
analysis
Achieved PMF?
Can improve it if continuing to use
the same business model?
Conduct transition of the
organization to scale
Yes
Consider to pivot
No
Yes
No
Flow chart to decide the next step.
Yes
No
Add it on issue Kanban board as
feature
Takes amounts of
features/issues?
Yes
No
Takes an amounts of
features/issues?
Verify UX/function improvement of
existing features
Provide enough values with only
existing features?
Verify product iteration added
features
Copyright 2017 Masayuki Tadokoro All rights reserved
Add it on issue Kanban board as
feature
Add it on Story Kanban board as
story
Refer it in
deciding
next step
58. Business Model 1 Business Model 2 Business Model 3
Time
MVP Version2 Version4Version3 Version6Version5
Improve
Pivot Pivot
Render pivoting if
you realize the business model
can’t achieve PMF Improve Improve Improve Improve Improve Improve
59. No
Yes
Add it on Story Kanban board as
story
Launch MVP and conduct
qualitative and quantitative
analysis
Achieved PMF?
Can improve it if continuing to use
the same business model?
Conduct transition of the
organization to scale
Yes
Consider to pivot
No
Yes
No
Flow chart to decide the next step.
Yes
No
Add it on issue Kanban board as
feature
Takes amounts of
features/issues?
Yes
No
Takes an amounts of
features/issues?
Verify UX/function improvement of
existing features
Provide enough values with only
existing features?
Verify product iteration added
features
Copyright 2017 Masayuki Tadokoro All rights reserved
Add it on issue Kanban board as
feature
Add it on Story Kanban board as
story
Render pivoting if you realize
the business model can’t
achieve PMF
60. Business Model 1 Business Model 2 Business Model 3
Time
MVP Version2 Version4Version3 Version6Version5
Impr
ove
Impr
ove
Impr
ove
Impr
ove
Impr
ove
Impr
ove
Impr
ove
Imp
rov
e
Imp
rov
e
Imp
rov
e
Deploy Deploy Deploy Deploy Deploy Deploy Deploy Deploy Deploy
Conduct deployment of
UX/function improvement
continually
61. No
Yes
Add it on Story Kanban board as
story
Launch MVP and conduct
qualitative and quantitative
analysis
Achieved PMF?
Can improve it if continuing to use
the same business model?
Conduct transition of the
organization to scale
Yes
Consider to pivot
No
Yes
No
Flow chart to decide the next step.
Yes
No
Add it on issue Kanban board as
feature
Takes amounts of
features/issues?
Yes
No
Takes an amounts of
features/issues?
Verify UX/function improvement of
existing features
Provide enough values with only
existing features?
Verify product iteration added
features
Copyright 2017 Masayuki Tadokoro All rights reserved
Add it on issue Kanban board as
feature
Add it on Story Kanban board as
story
Conduct deployment of
UX/function improvement
continually
62. Business Model 1 Business Model 2 Business Model 3
Time
MVP Version2 Version4Version3 Version6Version5
Impr
ove
Impr
ove
Impr
ove
Impr
ove
Impr
ove
Impr
ove
Impr
ove
Imp
rov
e
Imp
rov
e
Imp
rov
e
Version2 Version4Version3 Version5
Iteration Iteration Iteration Iteration Iteration
Rotate product iteration
based on what you learned
63. No
Yes
Add it on Story Kanban board as
story
Launch MVP and conduct
qualitative and quantitative
analysis
Achieved PMF?
Can improve it if continuing to use
the same business model?
Conduct transition of the
organization to scale
Yes
Consider to pivot
No
Yes
No
Flow chart to decide the next step.
Yes
No
Add it on issue Kanban board as
feature
Takes amounts of
features/issues?
Yes
No
Takes an amounts of
features/issues?
Verify UX/function improvement of
existing features
Provide enough values with only
existing features?
Verify product iteration added
features
Copyright 2017 Masayuki Tadokoro All rights reserved
Add it on issue Kanban board as
feature
Add it on Story Kanban board as
story
Yes
Rotate product iteration
based on what you learned
64. Product-Market Fit
Deliver the MVP
to the first
customer
Clarify what to
learn with an
MVP
Figure out things to
learn using the
MVP
Chose MVP type
and build
Learn by delivering to an
evangelist customer
Conduct Qualitative analysis by
interviews with customers and
Quantitative analysis by
innovation metrics
Examine the UX
with Hooked model
Improve UX to
Enhance
the customer
engagement
Iterate product to
bring more value
to customers
Iterate product/ launch
it
Conduct
Quantitative &
Qualitative Analysis
Build MVP
Repeat the
process until
the PMF has
been achieved
Create
Highly adaptive teamPivot the business model
Pivot the
Business Model
Copyright 2017 Masayuki Tadokoro All rights reserved
65. Verify Product (MVP) Iteration
Is the existing product (MVP)
providing enough value?
Copyright 2017 Masayuki Tadokoro All rights reserved
66. No
Yes
Add it on Story Kanban board as
story
Launch MVP and conduct
qualitative and quantitative
analysis
Achieved PMF?
Can improve it if continuing to use
the same business model?
Conduct transition of the
organization to scale
Yes
Consider to pivot
No
Yes
No
Flow chart to decide the next step.
Yes
No
Add it on issue Kanban board as
feature
Takes amounts of
features/issues?
Yes
No
Takes an amounts of
features/issues?
Verify UX/function improvement of
existing features
Provide enough values with only
existing features?
Verify product iteration added
features
Copyright 2017 Masayuki Tadokoro All rights reserved
Add it on issue Kanban board as
feature
Add it on Story Kanban board as
story
67. Can you provide enough value with
only existing product features?
68. It's better to brush up existing features than
to add new features.
Newly added features will be useless for users
in many cases.
Researchers say 20% to 80% of all features of a product are
used
Basically, be negative to adding features.
*Adding new features will increase hidden costs ( test,
adjustment, complexity, less attention ).
Copyright 2017 Masayuki Tadokoro All rights reserved
Copyright 2017 Masayuki Tadokoro All rights reserved
69. 13%
7%
16%
19%
45%
Not used
Used anytime
Used often
used
sometimes
Used
occasionally
Frequently used features are 20% of
the whole.
Copyright 2017 Masayuki Tadokoro All rights reserved
73. A case where you consider
adding new features.
Copyright 2017 Masayuki Tadokoro All rights reserved
74. Consider an issue hypothesis to
verify in the next iteration.
Copyright 2017 Masayuki Tadokoro All rights reserved
75. No
Yes
Add it on Story Kanban board as
story
Launch MVP and conduct
qualitative and quantitative
analysis
Achieved PMF?
Can improve it if continuing to use
the same business model?
Conduct transition of the
organization to scale
Yes
Consider to pivot
No
Yes
No
Flow chart to decide the next step.
Yes
No
Add it on issue Kanban board as
feature
Takes amounts of
features/issues?
Yes
No
Takes an amounts of
features/issues?
Verify UX/function improvement of
existing features
Provide enough values with only
existing features?
Verify product iteration added
features
Copyright 2017 Masayuki Tadokoro All rights reserved
Add it on issue Kanban board as
feature
Add it on Story Kanban board as
story
Add new
feature!
76. What you want to learn
Can customers gain wifi data volume and utilize it?
Verified story to learn
A user activates the app and signs up.
He/she watches add and utilize gained data
volume
What KPIs do you measure to verify?
What you build Cost & time to build
Quantitative result Qualitative result
Utilization is tiresome but customers use the
solution, feeling its value.
What you learned
Customers feel its value and use it but there is less motivation to stick to the product because of its
fewer amounts of gained data volume.
What you want to learn from the next sprint
Increasing data volume customers can add one time will help customers feel less tiresome in using the
product
See whether customer retention rates are improved by it.
Iteration Canvas
Copyright 2017 Masayuki Tadokoro All rights reserved
Activation: Sign up rates, rates of completion of the sequence
Retention: Rates of re-logging in in 3 days after registration
Revenue: Average # of ad views per a day
20 ppl per a day/2 weeks
Activation: 30%
Retention:25%
Revenue: 4 times in a day
What you want to learn by
adding new features
77. Issue hypothesis:
Customers currently can't gain wifi data
volume little by little.
The solution makes customers frustrated.
Copyright 2017 Masayuki Tadokoro All rights reserved
78. Iteration canvas of MVP Iteration canvas Version 2
Copyright 2017 Masayuki Tadokoro All rights reserved
Move the issue hypothesis you
want to learn/verify in the next
sprint on the new iteration
canvas.
79. What you want to learn
Increasing data volume customers can add one time will help customers feel
less tiresome in using the product
See whether customer retention rates are improved by it.
Verified story to learn What KPIs do you measure to verify?
What you build Cost & time to build
Quantitative result Qualitative result
What you learned
What you want to learn from the next sprint
Iteration Canvas
Copyright 2017 Masayuki Tadokoro All rights reserved
80. Copyright 2017 Masayuki Tadokoro All rights reserved
Verify new possible stories (add features).
Possible story 1: Users are able to watch ad movies
in the 3 consecutive times (currently, users need to
return to the top page each time they watch an ad )
Possible story 2: Users add more data volume by
watching an ad (currently 10MB to 15MB).
Possible story 3: Users add 30MB (3 times of by watching
an ad) by answering questionnaires.
Copyright 2017 Masayuki Tadokoro All rights reserved
81. No
Yes
Add it on Story Kanban board as
story
Launch MVP and conduct
qualitative and quantitative
analysis
Achieved PMF?
Can improve it if continuing to use
the same business model?
Conduct transition of the
organization to scale
Yes
Consider to pivot
No
Yes
No
Flow chart to decide the next step.
Yes
No
Add it on issue Kanban board as
feature
Takes amounts of
features/issues?
Yes
No
Takes an amounts of
features/issues?
Verify UX/function improvement of
existing features
Provide enough values with only
existing features?
Verify product iteration added
features
Copyright 2017 Masayuki Tadokoro All rights reserved
Add it on issue Kanban board as
feature
Add it on Story Kanban board as
story
Put possible stories on
Story Kanban board
82. Story Kanban Board
Backlog In Progress Completed Validation
Backlog story
Extract stories that
you want to learn in
the iteration
Build
Build
completed
launch
Quantitative
analysis (split
test, cohort
analysis)
Qualitative
analysis/introspection
by customer interview
MVP
story
Story
1-1
Story
1-2
Copyright 2017 Masayuki Tadokoro All rights reserved
Story
2-2
Story
2-3
Story
2-1
Put potential
stories in the next
iteration
83. Story Kanban Board
Backlog In Progress Completed Validation
Backlog story
Extract stories that
you want to learn in
the iteration
Build
Build
completed
launch
Quantitative
analysis (split
test, cohort
analysis)
Qualitative
analysis/introspection
by customer interview
MVP
story
Story
1-1
Story
1-2
Copyright 2017 Masayuki Tadokoro All rights reserved
Story
2-2
Story
2-3
Story
Version 2
Extract story
Version2
84. What you want to learn
Increasing data volume customers can add one time will help customers feel
less tiresome in using the product
See whether customer retention rates are improved by it.
Verified story to learn What KPIs do you measure to verify?
What you build Cost & time to build
Quantitative result Qualitative result
What you learned
What you want to learn from the next sprint
Iteration Canvas
Copyright 2017 Masayuki Tadokoro All rights reserved
Users add 30MB (3 times of by watching an ad)
by answering questionnaires
Run the same cycle
again
85. Story Kanban Board
Backlog In Progress Completed Validation
Backlog story
Extract stories that
you want to learn in
the iteration
Build
Build
completed
launch
Quantitative
analysis (split
test, cohort
analysis)
Qualitative
analysis/introspection
by customer interview
MVP
story
Story
1-1
Story
1-2
Copyright 2017 Masayuki Tadokoro All rights reserved
Story
2-2
Story
2-3
Story
Version 2
Begin building
86. Story of Anywhere online
Version 2 Usage
confirmation
Confirm
Questio
nnaires
Confirm
screen
Ad watch Confirm
screen
Added story
Copyright 2017 Masayuki Tadokoro All rights reserved
Open screen Menu screen
A story to be verified:
A user activates the app and signs up.
He/she watches add and utilize gained data
volume
87. What you want to learn
Increasing data volume customers can add one time will help customers feel
less tiresome in using the product
See whether customer retention rates are improved by it.
Verified story to learn What KPIs do you measure to verify?
What you build Cost & time to build
Quantitative result Qualitative result
What you learned
What you want to learn from the next sprint
Iteration Canvas
Copyright 2017 Masayuki Tadokoro All rights reserved
Users add 30MB (3 times of by watching an ad)
by answering questionnaires
Activation: Sign up rates, rates of completion of
the sequence
Retention: Rates of re-logging
in 3 days after registration
Revenue: Average # of ad views per a day
5 ppl per a day/1 weeks
Wash out implement
image and estimated
production costs of
KPI/KGI
88. Story Kanban Board
Backlog In Progress Completed Validation
Backlog storyaa
Extract stories that
you want to learn in
the iteration
Build
Build
completed
launch
Quantitative
analysis (split
test, cohort
analysis)
Qualitative
analysis/introspection
by customer interview
MVP
story
Story
1-1
Story
1-2
Copyright 2017 Masayuki Tadokoro All rights reserved
Story
2-2
Story
2-3
Story
Version 2
Build, launch and conduct
quantitative verification
89. KGI
Acquisition Activation Retention Revenue
KPI Number of UU
accessing to the
sign up screen
Leaving rates
in the menu
screen
Percentage of
users starting
watching ads
Ad watch
completion
rates
Rates of
starting
using the
app
Factorize KGIs in each stage to actionable
KPIs
Leaving
rates in the
log in screen
Leaving
rates in the
menu
screen
Average number of ad
views per user
Sign up rates
Factorization
Copyright 2017 Masayuki Tadokoro All rights reserved
Log in rates in 3days
after registering
Average number of ad
views per user
Percentage of users
who completed the
sequence
Sign up rates
Number of UU
accessing to the
sign up screen
First
Sign up
screen
Menu
screen
Start
watching
ad
Complete
watching
Wifi
usage
confirmation
Start
using wifi
Re-log in
screen
Menu
screen
Start
watching
ad
Complete
watching
90. Activation rates
People who
registered:
More than 90%
Activation rates
People who f
inished the sequence:
35%
Retention rates
People who re-logged in
after 3 days: 25%
Revenue
Times of viewing ads and
answering questionnaires:
6 times
Launch Anytime online Ver 2 and measured results.
Questio
nnaires
Confirm
screen
Questio
nnaires
Confirm
screen
Copyright 2017 Masayuki Tadokoro All rights reserved
First
Sign up
screen
Menu
screen
Start
watching
ad
Complete
watching
Wifi usage
confirmation
Start
using wifi
Re-log in
screen
Menu
screen
Start
watching
ad
Complete
watching
91. Make a cohort analysis report
based on split testing.
Copyright 2017 Masayuki Tadokoro All rights reserved
92. Split test:
A scientific experiment where
they allocate object people
into groups randomly and compare and measure actions
of each group (Control group A and variation group B).
*When grouping them, change more than 1 condition and measure it
Copyright 2017 Masayuki Tadokoro All rights reserved
93. Cohort analysis:
It visualizes whether your actions/
initiatives have been making a positive
impact on your business.
Then you can
discuss with the team what you should
do next.
Copyright 2017 Masayuki Tadokoro All rights reserved
94. Measurement
result of MVP
(actual number)
Measurement
result of MVP
(rates)
Measurement
result of Version 2
(actual number)
Measurement
result of Version 2
(rates)
Objective
rates
Acquisition
(visitors)
332 ppl 100% 325 ppl 100% 100%
Activation
(registration) 305 ppl 90% 291 ppl 90% 95%
Activation
(Sequence
completed)
93 ppl 30% 102 ppl 35% 80%
Retention
(revist in 3days) 23 ppl 25% 25 ppl 25% 80%
Revenue
Number of ad views per user
in a day)
4.2 times 4.2 times 6.1 times 6.1 times 10 times
MVPと
バージョン2の
比較を行う
Compare MVP and its
Ver. 2
Copyright 2017 Masayuki Tadokoro All rights reserved
95. Introducing split test/cohort report makes
short term work annoying somehow.
However, implementing a system that
values your work quantitatively will
help you shorten the time in the
long term.
Copyright 2017 Masayuki Tadokoro All rights reserved
96. What you want to learn
Increasing data volume customers can add one time will help customers feel
less tiresome in using the product
See whether customer retention rates are improved by it.
Verified story to learn What KPIs do you measure to verify?
What you build Cost & time to build
Quantitative result Qualitative result
What you learned
What you want to learn from the next sprint
Iteration Canvas
Copyright 2017 Masayuki Tadokoro All rights reserved
5 ppl per a day/1 weeks
Users add 30MB (3 times of by watching an ad)
by answering questionnaires
Activation: Sign up rates, rates of completion of
the sequence
Retention: Rates of re-logging
in 3 days after registration
Revenue: Average # of ad views per a day
Write down quantitative
results
97. Interview with customers who
have actually used version 2.
Copyright 2017 Masayuki Tadokoro All rights reserved
98. What you want to learn
Increasing data volume customers can add one time will help customers feel
less tiresome in using the product
See whether customer retention rates are improved by it.
Verified story to learn What KPIs do you measure to verify?
What you build Cost & time to build
Quantitative result Qualitative result
What you learned
What you want to learn from the next sprint
Iteration Canvas
Copyright 2017 Masayuki Tadokoro All rights reserved
5 ppl per a day/1 weeks
Write down quantitative
reusults
Users add 30MB (3 times of by watching an ad)
by answering questionnaires
Activation: Sign up rates, rates of completion of
the sequence
Retention: Rates of re-logging
in 3 days after registration
Revenue: Average # of ad views per a day
99. What you want to learn
Increasing data volume customers can add one time will help customers feel
less tiresome in using the product
See whether customer retention rates are improved by it.
Verified story to learn What KPIs do you measure to verify?
What you build Cost & time to build
Quantitative result Qualitative result
What you learned
What you want to learn from the next sprint
Iteration Canvas
Copyright 2017 Masayuki Tadokoro All rights reserved
5 ppl per a day/1 weeks
Write down what you
learned
Users add 30MB (3 times of by watching an ad)
by answering questionnaires
Activation: Sign up rates, rates of completion of
the sequence
Retention: Rates of re-logging
in 3 days after registration
Revenue: Average # of ad views per a day
100. The batch size of iteration tends to be bigger as
time passes
As the batch proceeds to next phase, addition, redos
and delays occurs.
To make the overheads smaller, the butch will get
bigger naturally.
Copyright 2017 Masayuki Tadokoro All rights reserved
Important Points
101. Story Kanban Board
Backlog In Progress Completed Validation
Backlog story
Extract stories that
you want to learn in
the iteration
Build
Build
completed
launch
Quantitative
analysis (split
test, cohort
analysis)
Qualitative
analysis/introspection
by customer interview
MVP
story
Story
1-1
Story
1-2
Copyright 2017 Masayuki Tadokoro All rights reserved
Story
2-2
Story
2-3
Story
To shorten time, renew 2
stories (features) to
make them one and
examine it
Story Story Story
Story
Story
Story
(Up to 3)(Up to 3) (Up to 3) (Up to 3) (Up to 3)
102. Story Kanban Board
Backlog In Progress Completed Validation
Backlog story
Extract stories that
you want to learn in
the iteration
Build
Build
completed
launch
Quantitative
analysis (split
test, cohort
analysis)
Qualitative
analysis/introspection
by customer interview
MVP
story
Story
1-1
Story
1-2
Copyright 2017 Masayuki Tadokoro All rights reserved
Story
2-2
Story
2-3
StoryStory Story Story
Story
Story
(Up to 3)(Up to 3) (Up to 3) (Up to 3) (Up to 3)
Make the batch size
bigger and verify at
one time
103. Backlog In Progress Completed Validation
Backlog story
Extract stories that
you want to learn in
the iteration
Build
Build
completed
launch
Quantitative
analysis (split
test, cohort
analysis)
Qualitative
analysis/introspection
by customer interview
MVP
story
Story
1-1
Story
1-2
Copyright 2017 Masayuki Tadokoro All rights reserved
Story
2-2
Story
2-3
StoryStory Story Story
Story
Story
(Up to 3)(Up to 3) (Up to 3) (Up to 3) (Up to 3)
Not sure which story
(function) is more
effective…
Story Kanban Board
104. Fall into the death spiral of the
big batch.
Making a batch size bigger will
keep you from achieving the objective
of learning something in each
iteration.
Copyright 2017 Masayuki Tadokoro All rights reserved
105. In case you add
a simple feature
that has minimal
production costs,
add it to the issue board.
Copyright 2017 Masayuki Tadokoro All rights reserved
106. No
Yes
Add it on Story Kanban board as
story
Launch MVP and conduct
qualitative and quantitative
analysis
Achieved PMF?
Can improve it if continuing to use
the same business model?
Conduct transition of the
organization to scale
Yes
Consider to pivot
No
Yes
No
Flow chart to decide the next step.
Yes
No
Add it on issue Kanban board as
feature
Takes amounts of
features/issues?
Yes
No
Takes an amounts of
features/issues?
Verify UX/function improvement of
existing features
Provide enough values with only
existing features?
Verify product iteration added
features
Copyright 2017 Masayuki Tadokoro All rights reserved
Add it on issue Kanban board as
feature
Add it on Story Kanban board as
story
Add it to the issue
Kanban board
if it has minimal
production costs
107. As a timing of implement is coming
up, adopt a just-in-time method where
you produce a needed feature when
it's needed based on an analysis to
verify which items are needed.
Copyright 2017 Masayuki Tadokoro All rights reserved
108. ・You can analyze based on the latest and
relevant information (enforcing you to answer the
reason why you are doing it now)
・Developers can learn more from customers
・It may keep you from reworking
・You may verify qualities of each batch earlier
using split test/cohort analysis
The effect of the just-in-time
method
Copyright 2017 Masayuki Tadokoro All rights reserved
109. Product
Market-Fit
Risk
(= Risk not to
create needed
products)
Time
PMF risk is rising
without customer
Feedback
Reduce the risk that you could produce unneeded
products for which you have consumed a lot of
resources/time.
PMF risk is limited with
customer feedback
High
Low
Copyright 2017 Masayuki Tadokoro All rights reserved
110. How to rhythmically manage the
just-in-time method.
・Utilize issue Kanban board
・Conduct daily stand up
・Proceed development based on customer
interview
・Develop avoiding documentation
*Nevertheless, document customer insights to capitalize/share in company
Copyright 2017 Masayuki Tadokoro All rights reserved
111. Backlog In process Completion Validation
Backlog story Building
Build completed
launch
Quantitative analysis
(split test, cohort analysis)
Features
Features
Features
Put features
in order of priority
Copyright 2017 Masayuki Tadokoro All rights reserved
Utilize the Issue Kanban Board
112. Backlog In process Completion Validation
Backlog story Building
Build completed
launch
Quantitative analysis
(split test, cohort analysis)
Features
Features
Features
Copyright 2017 Masayuki Tadokoro All rights reserved
Set a limitation
similar to the Story
Kanban board
(Up to 3) (Up to 3) (Up to 3) (Up to 3)
Utilize the Issue Kanban Board
113. Backlog In process Completion Validation
Backlog story Building
Build completed
launch
Quantitative analysis
(split test, cohort analysis)
Features
Features
Features
Copyright 2017 Masayuki Tadokoro All rights reserved
(Up to 3) (Up to 3) (Up to 3) (Up to 3)
Feature
Feature
Feature Feature
Feature
Feature
In case there is a bottle-neck,
use resources and increase
throughput
Utilize the Issue Kanban Board
114. Share the development process in
daily stand up meetings.
3 things to share in a daily stand up:
①What you did yesterday
②What you will do today
③Troubles that may keep your team from
working efficiently
Copyright 2017 Masayuki Tadokoro All rights reserved
115. Stop documentation in developing.
You may try to work in accordance with the
document.
Copyright 2017 Masayuki Tadokoro All rights reserved
116. Qualitative feedback
from customers
Team
Quantitative data
Accumulate it for
company learning
Learning
accumulation
Copyright 2017 Masayuki Tadokoro All rights reserved
Document customer insights to share in your team.
You can not set perfect KPIs at beginning
You want to continuously verify whether you are measuring appropriate KPI in each stage for your business
Throw vanity metrics away
Page view: Number of visitors to the page; it is vanity metrics if you do not set ad-sense or banner ads.
Number of visitors: It is vanity metrics since you do not know what types of people are coming how frequently.
Unique visitors: It is vanity metrics since you do not know why users came and what they have done
It is meaningless if it is just a number.
Key point is how many people will take an action when you post something on social media stream
Visit duration: You have to measure which duration for specific pages. If customers are staying FAQ or support pages for long time, it indicates that usability is bad
Sign-up number: It is meaningless if you can not verify open rate of e-mals
With vanity metrics, you will be reassured, but it won’t provide you clear guidance what you should do next
Set up one-on-one interview in order to get feedback.
Avoid one-to-N interview, since you cannot hear open opinions
P163 start-up owener’s manual
Need to be revised
The procedure of the KJ method.
1 Collect interview data.
2 Categorize the data into small units.
3 Organize these units into groups.
4 For each group, add an appropriate label.
5 Put into a logical structure so that you can explain logically.
6 Specify what the true cause of the issue is.
インタビューの分析:KJ法を行う
ユーザビリティエンジニアリング 1099