The document discusses key principles of lean analytics and startup methodologies presented at the Lean Startup Conference in December 2015, emphasizing the importance of iteration and product/market fit. It highlights how analytics help startups measure progress towards business goals, perform effective testing, and adapt to changes in the market. Additional topics include the significance of performance metrics, the attention economy, and the necessity for deploying good metrics to drive actionable insights.

1. Lean Analytics
Lean Startup Conference
December, 2015
@acroll

2. Some housekeeping.

3. Kevin Costner is a lousy entrepreneur.
Don’t sell what you can make. Make what you can sell.

4. The core of Lean
is iteration.

5. Most startups don’t know what they’ll
be when they grow up.
Hotmail
was a
database
company
Flickr
was going to
be an MMO
Twitter
was a
podcasting
company
Autodesk
made
desktop
automation
Paypal
first built for
Palmpilots
Freshbooks
was invoicing
for a web
design firm
Wikipedia
was to be
written by
experts only
Mitel
was a
lawnmower
company

6. Waterfall, agile, and lean
(Why the old ways don’t work.)

7. Waterfall approach
You know the problem and the solution.

9. Known set of
requirements
Known ways to satisfy
them
Spec Build Test Launch

10. Agile methodologies
Know the problem, find the solution

12. Known set of
requirements
Unclear how to satisfy
them
Problem Build Test Viable? Launch
statement
Sprints
Adjust
Unknown set of

13. Lean approach
First, know that you don’t know.

14. Product/market
hypothesis Trial startup
Possible problem
space
Product/
market
hypothesis
Trial startup
Product/
market
hypothesis
Trial startup
Trial startup
Product/market
hypothesis
You are
here
PIVOT

15. Why now?
First: High rate of change of digital
technologies & channels.

16. Arbitron and radio data

17. Times a song in “heavy
rotation” is played daily
30
15
0
6 26
2007 2012

18. For modern media this means
cycle time shock.
Circulation, annually Clicks, instantaneously
Letters to the editor, weekly Hashtags, always

19. Why now?
Second: It’s no longer about whether
you can build it—it’s about whether
anyone will care.

20. The Attention Economy
“What information consumes
is rather obvious: it consumes
the attention of its recipients.
Hence a wealth of information
creates a poverty of attention, and a
need to allocate that attention efficiently
among the overabundance of
information sources that might
consume it.”
(Computers, Communications and the Public Interest, pages 40-41,
Herbert Simon Martin Greenberger, ed., The Johns Hopkins Press, 1971.)

21. Lit motors tests the risky part

22. Unfortunately,
we’re all liars.

23. Everyone’s idea is
the best right?
People love
this part!
(but that’s not always
a good thing)
No data, no
learning.
This is where
things fall apart.

24. Analytics can help.

25. Analytics is the measurement of
movement towards your business
goals.

26. In a startup, the purpose of analytics is
to iterate to product/market fit
before the money runs out.

27. I have two kids.
At least one of them is a girl.

28. What are the chances
the other is a boy?

29. BB BG
GB GG

30. 2 of 3 (66%) are boys.
GB GG BG

31. Some fundamentals.
Analytics, performance, aggregation, and the right metrics

32. Fundamental:
Web analytics and the long funnel.
(This is not a web analytics workshop.)

33. A simplistic view of web analytics
ATTENTION ENGAGEMENT CONVERSION
NEW
VISITORS
GROWTH
LOSS
BOUNCE
RATE
CONVERSION
RATE
x TIME
GOAL
VALUE
ON
SITE
PAGES
PER
VISIT
NUMBER
OF VISITS
SEARCHES
TWEETS
MENTIONS
ADS SEEN

34. Visits
Shopping cart
Payment options
Conversions

35. Visits
Shopping cart
Payment options
Conversions
KPIs
Bounce
Conversion

36. Unpaid search Community mentions
Visits
Shopping cart
Payment options
Conversions
Email campaign Banner ad
•Google PageRank
•Sessions-to-clicks
ratio
•Cost of ads
(CPM)
•Clickthrough
rate
? •Open rate
•Opt-out rate

37. Repurposing
(spread to other
communities)
Amplification
(virality and
message spread)
Seed (starting
community)
Reach
(impressions)
Visits
Shopping cart
Payment options
Conversions

38. Viral message spread
Reach
(impressions)
Visits
Shopping cart
Payment options
Conversions
Emphasis on getting
viral ratio above 1
(Retweeting, Fan, Email
forward, Reddit upvote,
other loops)

39. Megablogger proponents
Seed (starting
community)
Reach
(impressions)
Visits
Shopping cart
Payment options
Conversions
Emphasis on
convincing highly-followed,
highly acted-upon
seed group to
spread the word.

40. A call to action
Reach
(impressions)
Visits
Shopping cart
Payment options
Conversions
Emphasis on
maximizing
impression-to-click
ratio within
the community

41. Long funnel: Beers for Canada
700,000s
1,642
150
RT
32
7
10
15
x $100
x $20
x $7
Seed ratio: 20
35,000:1
Followers
Visitors:
0.23%
Conversions:
1.95%
2 Repurposing
Amplification: 2.9%
Revenues:
Average: $39.54
Median: $20
Total: $1,005
2,000

42. Fundamental:
You should care about performance.

43. Downtime costs
Amazon offline ($1M/h)
Amazon loses nearly $1M/hour if down (NYT, 2008)
1 hour of network downtime costs $42,000 (Gartner, 2003)
Network downtime ($42K/h)
22h outage at eBay cost $2M ($90,909/h) (Internetnews, 1999)
eBay offline ($90K/h)
Financial company down ($100K/h)
53.2% of finance companies lose over $100,000/hour (nextslm.org)
Let’s say $50K/h if you’re serious.

44. Availability % Downtime/year Loss @$50K/h
90% 36.5 days $43,800,000
95% 18.25 days $21,900,000
98% 7.30 days $8,760,000
99% 3.65 days $4,380,000
99.5% 1.83 days $2,196,000
99.8% 17.52 hours $876,000
99.9% 8.76 hours $438,000
99.95% 4.38 hours $219,000
99.99% 52.6 minutes $43,833
99.999% 5.26 minutes $4,383
99.9999% 31.5 seconds $438
Less than 1h/
year
Less than a
minute/year

45. You really don’t want web users to call
you.
US$12
US$10
US$7
US$5
US$2
US$0
$5.50
$3.00
$0.24 $0.45
Web self-service IVR Email Live phone
Low Average High
BiT Group White Paper: “Web Self-Service Lowers Call Center Costs and
Improves Customer Service”
Cost estimates

46. The login page Function
will have a total latency Metric
of under 4 seconds Target
with a cached browser copy User situation
from any US branch office Testing point
95% of the time Percentile
weekdays, 8AM ET to 6M PST Time window
by synth test at 5m intervals Collection type

47. 10%
visitors
7.5%
of 5%
Percent 2.5%
0%
0-2s 2-4s 4-6s 6-8s 8s +
Average page load time across visit that commented on a post

48. Fundamental:
What makes a good metric?

49. A good metric is:
Understandable
If you’re busy
explaining the
data, you won’t
be busy acting
on it.
Comparative
Comparison is
context.
A ratio or rate
The only way to
measure
change and roll
up the tension
between two
metrics (MPH)
Behavior
changing
What will you
do differently
based on the
results you
collect?

50. The
simplest
rule
If a metric won’t change how
you behave, it’s a
bad
metric.
h"p://www.flickr.com/photos/circasassy/7858155676/

51. Metrics help you know yourself.
Acquisition
Hybrid
Loyalty
You are
just like
70%
of retailers
20%
of retailers
10%
of retailers
Customers that
buy >1x in 90d
Your customers
will buy from you
Once
2-2.5
per year
>2.5
per year
Then you are
in this mode
1-15%
15-30%
>30%
Focus on
Low acquisition
cost, high checkout
Increasing return
rates, market share
Loyalty, selection,
inventory size
(Thanks to Kevin Hillstrom for this.)

52. Qualitative
Unstructured, anecdotal,
revealing, hard to
aggregate, often too
positive & reassuring.
Warm and fuzzy.
Quantitative
Numbers and stats.
Hard facts, less insight,
easier to analyze; often
sour and disappointing.
Cold and hard.

53. Exploratory
Speculative. Tries to find
unexpected or
interesting insights.
Source of unfair
advantages.
Cool.
Reporting
Predictable. Keeps you
abreast of the normal,
day-to-day operations.
Can be managed by
exception.
Necessary.

54. Rumsfeld on Analytics
Things we
know
don’t
know
(Or rather, Avinash Kaushik channeling Rumsfeld)
we know Are facts which may be wrong and
should be checked against data.
we don’t
know
Are questions we can answer by
reporting, which we should baseline
& automate.
we know
Are intuition which we should
quantify and teach to improve
effectiveness, efficiency.
we don’t
know
Are exploration which is where
unfair advantage and interesting
epiphanies live.

55. Slicing and dicing data
Feb Mar Apr May
5,000
Active users
0
Jan
Cohort:
Comparison of
similar groups
along a timeline.
(this is the April cohort)
A/B test:
Changing one thing
(i.e. color) and
measuring the
result (i.e. revenue.)
Multivariate
analysis
Changing several
things at once to
see which correlates
with a result.
☀☁☀☁
Segment:
Cross-sectional
comparison of all
people divided by
some attribute (age,
gender, etc.)
☀
☁

56. Which of these two companies
is doing better?

57. January February March April May
Is this company Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50
growing or stagnating?
Cohort 1 2 3 4 5
January $5 $3 $2 $1 $0.5
February $6 $4 $2 $1
March $7 $6 $5
April $8 $7
May $9
How about
this one?

58. Cohort 1 2 3 4 5
January $5 $3 $2 $1 $0.5
February $6 $4 $2 $1
March $7 $6 $5
April $8 $7
May $9
Averages $7 $5 $3 $1 $0.5
Look at the
same data
in cohorts

59. Lagging
Historical. Shows you
how you’re doing;
reports the news.
Example: sales.
Explaining the
past.
Leading
Forward-looking.
Number today that
predicts tomorrow;
reports the news.
Example: pipeline.
Predicting the
future.

60. Some examples
A Facebook user reaching 7 friends within 10 days of signing up
(Chamath Palihapitiya)
If someone comes back to Zynga a day after signing up for a game,
they’ll probably become an engaged, paying user (Nabeel Hyatt)
A Dropbox user who puts at least one file in one folder on one device
(ChenLi Wang)
Twitter user following a certain number of people, and a certain
percentage of those people following the user back (Josh Elman)
A LinkedIn user getting to X connections in Y days (Elliot Schmukler)
(From the 2012 Growth Hacking conference. http://growthhackersconference.com/)

61. Which means it’s time to talk
about correlation.

62. 10000
1000
100
10
1
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Ice cream consumption Drownings

63. Correlated
Two variables that are
related (but may be
dependent on
something else.)
Ice cream &
drowning.
Causal
An independent variable
that directly impacts a
dependent one.
Summertime &
drowning.

64. A leading, causal metric
is a superpower.
h"p://www.flickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/

65. Why is Nigerian spam so badly
written?

66. Experienced scammers expect a “strike rate” of 1 or 2 replies per 1,000 messages
emailed; they expect to land 2 or 3 “Mugu” (fools) each week.
One scammer boasted “When you get a reply it’s 70% sure you’ll get the money”
“By sending an email that repels all but the most gullible,” says [Microsoft Researcher
Corman] Herley, “the scammer gets the most promising marks to self-select, and tilts
the true to false positive ratio in his favor.”
This would be horribly
inefficient since
humans are involved.
Good language (10% conversion)
Not-gullible (.07% conversion)
Aunshul Rege of Rutgers University, USA in 2009
1000 emails
Bad language (0.1% conversion)
1-2 responses
Gullible (70% conversion)
1 fool and their money, parted.
1000 emails
100 responses
1 fool and their money, parted.

67. Turns out the word “Nigeria” is the best
way to identify promising prospects.

68. Nigerian spammers
really understand their target market.
They see past vanity metrics.

69. Fundamental:
Be careful rolling things up.

70. http://upload.wikimedia.org/wikipedia/commons/0/0e/Count-von-count.jpg

71. 0 10 20 30 40 50 60 70 80 90
Age
20
Count
0
Average age = 10

72. 200
0 2 4 6 8 10 12 14 16 18
Page load time (in seconds)
# of requests
0
20
Average latency = 5s
95th percentile latency = 19s

73. KISS

74. “It can scarcely be denied
that the supreme goal of all theory is
to make the irreducible basic elements
as simple and as few as possible without
having to surrender the adequate
representation of a single datum of
experience.”
http://media.photobucket.com/image/einstein/derekabril/einstein_010.png

75. “As simple as possible,
but no simpler.”
*(FYI, this is irony.)

76. 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Login
Checkout
Invite

77. 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Login
Checkout
Invite
Average 4s
Average 6s
Average 9s

78. 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Login
Checkout
Invite
Average
95%
4s
8s
Average
95%
6s
10s
Average
95%
9s
12s

79. 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Login
Checkout
Invite
Average
95%
Mode
4s
8s
2s
Average
95%
Mode
6s
10s
5s
Average
95%
Mode
9s
12s
1s

80. Login
Checkout
Invite
Aggregate?
Average
95%
Mode
4s
8s
2s
Average
95%
Mode
6s
10s
5s
Average
95%
Mode
9s
12s
1s
Average
95%
Mode
6s
12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 5s

81. 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Login:
<=4s
740 260

82. 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Total samples 1000
Below threshold 740
Login:
<=4s Percent below
target threshold 74%
740 260

83. 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Login:
<=4s
Checkout:
<=5s
Invite:
<=8s
Total samples 1000
Below threshold 740
Percent below
target threshold 74%
Total samples 1000
Below threshold 370
Percent below
target threshold 37%
Total samples 1000
Below threshold 610
Percent below
target threshold 61%
740 260
370 630
610 390

84. 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Login:
<=4s
Checkout:
<=5s
Invite:
<=8s
Aggregate?
Total samples 1000
Below threshold 740
Percent below
target threshold 74%
Total samples 1000
Below threshold 370
Percent below
target threshold 37%
Total samples 1000
Below threshold 610
Percent below
target threshold 61%
740 260
370 630
610 390
Total samples 3000
Below threshold 1720
Percent below
target threshold 57%

85. 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Login:
<=4s
Checkout:
<=5s
Invite:
<=8s
Total samples 1000
Below threshold 740
Percent below
target threshold 74%
Total samples 400
Below threshold 148
Percent below
target threshold 37%
Total samples 600
Below threshold 366
Percent below
target threshold 61%
740 260
370 630
610 390
Total samples 2000
Below threshold 1254
Percent below
target threshold 63%
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

86. 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Weight 1
Weight 5
Weight 2
Login:
<=4s
Checkout:
<=5s
Invite:
<=8s
Total samples 1000
Below threshold 740
Total samples 400
Below threshold 148
Total samples 600
Below threshold 366
740 260
370 630
610 390

87. 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
Total requests
inside target
Login page 740/1000
Checkout page 148/400
Invite process 366/600

88. Total requests
inside target
Login page 740/1000
Checkout page 148/400
Invite process 366/600
Weight
1
52
Weighted
740/1000
740/2000
732/1200
2212/4200
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

89. Total requests
inside target
Login page 740/1000
Checkout page 148/400
Invite process 366/600
Weight
1
52
Weighted
740/1000
740/2000
732/1200
Total score 2212/4200
53%
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s
1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s

90. The Lean Analytics framework.

91. Eric’s three engines of growth
Virality
Make people
invite friends.
How many they
tell, how fast they
tell them.
Price
Spend money to
get customers.
Customers are
worth more than
they cost.
Stickiness
Keep people
coming back.
Approach
Get customers
faster than you
lose them.
Math that
matters

92. Dave’s Pirate Metrics
AARRR Acquisition
How do your users become aware of you?
SEO, SEM, widgets, email, PR, campaigns, blogs ...
Activation
Do drive-by visitors subscribe, use, etc?
Features, design, tone, compensation, affirmation ...
Retention
Does a one-time user become engaged?
Notifications, alerts, reminders, emails, updates...
Revenue
Do you make money from user activity?
Transactions, clicks, subscriptions, DLC, analytics...
Referral
Do users promote your product?
Email, widgets, campaigns, likes, RTs, affiliates...

93. Gate
Stage
EMPATHY I’ve found a real, poorly-met need that a
reachable market faces.
STICKINESS I’ve figured out how to solve the problem in a
way they will keep using and pay for.
VIRALITY I’ve found ways to get them to tell their friends,
either intrinsically or through incentives.
REVENUE The users and features fuel growth organically
and artificially.
SCALE I’ve found a sustainable, scalable business with
the right margins in a healthy ecosystem.
The five stages

94. Empathy stage:
Localmind hacks Twitter
Needed to find out if a core assumption—strangers answering
questions—was valid.
Ran Twitter experiment instead of writing code
Asked senders of geolocated Tweets from Times Square random
questions; counted response rate
Conclusion: high enough to proceed

95. LikeBright’s mechanical turk
Used Mechanical Turk, Google Voice to speak w/
100 single women; paid $2. The interviews
lasted typically around 10-15 minutes.
Simple interview script with open-ended
questions, since he was digging into the problem
validation stage of his startup.
Founder Nick Soman: “I was amazed at the
feedback I got. We were able to speak with one
hundred single women that met our criteria in
four hours on one evening.”
Went back to TechStars and got accepted.
LikeBright’s website is now live with a 50%
female user base, and recently raised a round of
funding.
“Since that first foray into interviewing customers,
I’ve probably spoken with over a thousand
people through Mechanical Turk,”

96. How to avoid leading the witness
Avoid biased wording, preconceptions, or
a giveaway appearance. Word your
surveys carefully to be neutral.
Get them to purchase. Ask them to pay. Demand real
introductions. Or ask them “how many of your friends would say
Ask “why” several times. Leave lingering, uncomfortable pauses
in the conversation and let them fill them.
Don’t tip your hand
Make the question real
Keep digging
Look for other clues Have a colleague make notes of when they react, or of their body
language.

97. Stickiness stage:
qidiq streamlines invites
Survey owner adds recipient to group
Survey owner asks question
Recipient reads survey question
Recipient responds to question
Recipient sees survey results
(Later, if needed…)
Recipient visits site; no password!
Recipient does password recovery
One-time link sent to email
Recipient creates password
Recipient can edit profile, etc.
Survey owner adds recipient to group
Survey owner asks question
Recipient gets invite
Recipient installs mobile app
Recipient creates account, profile
Recipient can edit profile, etc.
Recipient reads survey question
Recipient responds to question
Recipient sees survey results
10-25% RESPONSE RATE
70-90% RESPONSE RATE

98. 1200
1000
800
600
400
200
0
January February
1 2 3 4 5 6 7 8 9
Days since last engagement
25000
20000
15000
10000
5000
0
Disengaged
(>10 days)
Number of users
A better approach to engagement
This is a
good thing.

99. Virality stage: Timehop focuses
on content sharing
Focused on percent of daily active users that share their content
Aiming for 20-30% of DAU sharing
“All that matters now is virality. Everything else—be
it press, publicity stunts or something else—is like
pushing a rock up a mountain: it will never scale.
But being viral will.”
- Jonathan Wegener, co-founder

100. ------------------------------------------------------
Get your free private email at http://www.hotmail.com
------------------------------------------------------

102. v ≠ 1, pt = δp0 (1 – vt+1) / (1 – v) + p0
http://robert.zubek.net/blog/2008/01/30/viral-coefficient-calculation/
Viral coefficient

103. Or simpler
x - > 1
Users Viral
coefficient
Churn &
abandonment

104. How to calculate it
First calculate the invitation rate,
which is the number of invites
sent divided by the number of
users you have.
Then calculate the acceptance
rate, which is the number of
signups or enrollments divided by
the number of invites.
Then multiply the two together.
Your 2,000 customers have sent
out 5,000 invitations during their
lifetime on your site.
Your invitation rate is 2.5.
For every ten invitations received,
one gets clicked.
Your acceptance rate is 0.1.
Multiply the two, and you have
your viral coefficient: 0.25. Every
customer you add will add an
addition 25% of a customer.

105. Revenue stage: Backupify’s
Customer Acquisition Payback
Initially focused on site visitors
Then focused on trials
Then switched to signups
Today, MRR
In early 2010, CAC was $243 and ARPU was only $39
Pivoted to target business users
CLV-to-CAC today is 5-6x
Now they track Customer Acquisition Payback
Target is less than 12 months

106. Scale stage:
Incremental order cost
Marginal costs
Fixed costs

107. Six business model archetypes.
E-commerce SaaS Mobile Media
app
User-gen
content
2-sided
market
The business you’re in

108. (Which means eye
charts like these.)
Customer Acquisition Cost
paid direct search wom inherent
virality
VISITOR
Freemium/trial offer
Enrollment
User
Disengaged User
Freemium
churn
Cancel
Engaged User
Free user
disengagement
Reactivate
Trial abandonment
Cancel
rate
Invite Others
Upselling
rate Upselling
Paying Customer
Reactivation
rate
Paid
conversion
FORMER USERS
User Lifetime Value
Reactivate
Capacity Limit
Support data
FORMER CUSTOMERS
Customer Lifetime Value
Viral coefficient
Viral rate
Resolution
Account Cancelled Billing Info Exp.
Paid Churn Rate
Tiering
Trial Over Disengaged Dissatisfied

109. Model + Stage = One Metric That Matters.
The business you’re in
E-Com SaaS Mobile 2-Sided Media UCG
One Metric
That Matters.
Empathy
Stickiness
Virality
Revenue
Scale
The stage you’re at

110. Really? Just one?

111. Yes, one.

112. In a startup, focus is hard to achieve.

113. Having only one metric
addresses this problem.

114. www.theeastsiderla.com

115. Moz cuts down on metrics
SaaS-based SEO toolkit in the scale stage. Focused on net adds.
Was a marketing campaign successful?
Were customer complaints lowered?
Was a product upgrade valuable?
Net adds up:
Can we acquire more valuable customers?
What product features can increase engagement?
Can we improve customer support?
Net adds flat:
Are the new customers not the right segment?
Did a marketing campaign fail?
Did a product upgrade fail somehow?
Is customer support falling apart?
Net adds down:

116. Metrics are like squeeze toys.
http://www.flickr.com/photos/connortarter/4791605202/

117. Empathy
Stickiness
Virality
Revenue
Scale
E-commerce
Mobile
app
User-gen
content
SaaS Media
2-sided
market
Interviews; qualitative results; quantitative scoring; surveys
Loyalty,
conversion
CAC, shares,
reactivation
(Money from transactions)
Transaction,
CLV
Affiliates,
white-label
Engagement,
churn
Inherent
virality, CAC
(Money from active users)
Upselling,
CAC, CLV
API, magic #,
mktplace
Content,
spam
Invites,
sharing
(Money from ad clicks)
Ads,
donations
Analytics,
user data
Inventory,
listings
SEM, sharing
Transactions,
commission
Other
verticals
Downloads,
churn, virality
WoM, app
ratings, CAC
CLV,
ARPDAU
Spinoffs,
publishers
Traffic, visits,
returns
Content
virality, SEM
CPE, affiliate
%, eyeballs
Syndication,
licenses

118. Better: bit.ly/BigLeanTable

119. Drawing some lines in the sand.

120. A company loses a quarter of its
customers every year.
Is this good or bad?

121. Not knowing what normal is
makes you do stupid things.

122. Baseline:
5-7% growth a week
“A good growth rate during YC
is 5-7% a week,” he says. “If
you can hit 10% a week you're
doing exceptionally well. If you
can only manage 1%, it's a sign
you haven't yet figured out
what you're doing.” At revenue
stage, measure growth in
revenue. Before that, measure
growth in active users.
Paul Graham, Y Combinator
• Are there enough people who really care
enough to sustain a 5% growth rate?
• Don’t strive for a 5% growth at the expense
of really understanding your customers
and building a meaningful solution
• Once you’re a pre-revenue startup at or
near product/market fit, you should have
5% growth of active users each week
• Once you’re generating revenues, they
should grow at 5% a week

123. Baseline:
10% visitor engagement/day
30% of users/month use web or mobile app
10% of users/day use web or mobile app
1%of users/day use it concurrently
Fred Wilson’s social ratios

124. Baseline:
2-5% monthly churn
• The best SaaS get 1.5% - 3% a month. They have multiple Ph.D’s
on the job.
• Get below a 5% monthly churn rate before you know you’ve got a
business that’s ready to grow (Mark MacLeod) and around 2%
before you really step on the gas (David Skok)
• Last-ditch appeals and reactivation can have a big impact.
Facebook’s “don’t leave” reduces attrition by 7%.

125. Baseline:
Calculating customer lifetime
25%
5%
monthly churn
monthly churn
100/25=4
100/5=20
The average
The average
customer lasts
customer lasts
4 months
20 months
2%
monthly churn
100/2=50
The average
customer lasts
50 months

126. Baseline:
CAC under 1/3 of CLV
• CLV is wrong. CAC Is probably wrong, too.
• Time kills all plans: It’ll take a long time to find
out whether your churn and revenue projections
are right
• Cashflow: You’re basically “loaning” the
customer money between acquisition and CLV.
• It keeps you honest: Limiting yourself to a
CAC of only a third of your CLV will forces you
to verify costs sooner.
Lifetime of 20 mo.
$30/mo. per
customer
$600 CLV
1/3 spend
$200 CAC
Now segment
those users!

127. Who is worth more?
A Lifetime:
B Lifetime:
Today
$200
$200
Roberto Medri, Etsy
Visits

128. Baseline:
35% of mobile users engaged by day 90
Day one
Day 30
Day 60
100%
54%
43%
Day 90
35%
October, 2012 study of 200,000 apps by Flurry
In recent years, third-month engagement has increased
from 25% to 35%, but frequency of use has dropped from
6.7 uses a week to 3.7 a week.
Smartphone Tablet
Uses per week 12.9 times 9.5 times
Duration of use 4.1m 8.2m

129. (Varies widely by
product category)

130. Etsy
• Online store for creative types, founded 2005
• $525M Gross Merchandise Sales in 2011, with
19,000,000 members and 800,000 active
shops offering 15,000,000 items for sale
• 1.4B pageviews per month ~2M iPhone app
downloads
• Thin revenues: Etsy makes only $0.20 or 3.5%
margin
• Heavy focus on Customer Lifetime Value (buyer
and seller)
• Actually residual lifetime value; they take this
pretty seriously.

131. Etsy
• The best customers to target are
• Recent high-profile customers
• Old-time best customers about to
churn or just churned
• Tiered campaigns
• Bronze/silver customers: reinforcement,
nudges
• Gold customers: premium services
• Platinum customers: recognition
• What they watch:
• Growth of individual product categories
• Time to first sale by a user
• Average order value
• Percentage of visits that convert to a
sale
• Percentage of return buyers
• Distinct sellers within a product
category
• Time-to-first-sale and average order
value by product category
Roberto Medri, Etsy

132. DuProprio/Comfree
• Large for-sale-by-owner marketplace
• Founded in 1997, 17,000 properties and 5M visits a month
• $900 per listing, plus value-added tools & services
• Leading goal is to create subscriptions
• Launched seller-side logins; then client accounts
• Rule of thumb: 1,000 visits equals 1 subscription
• Three business objectives:
• Convince sellers to list their property on the site;
• Convince buyers to register for property match notifications;
• Sell the properties.
KPI evolution
Static traffic
Visitor to listing ratio
List-to-sold ratio
Click-throughs, search
results

133. YPG
• Large directory publishing & local marketing w/420K customers,
2,500 employees, and $1,2B/y in revenue
• Focus on public API for listings (1.5M geo-coded listings for
location apps)
• Initially slow to embrace API, but in 2013 have tripled investment
• Lets the company find a partner or developer and have a
functional prototype in hours, testing in days, and launching in
weeks.
KPI evolution
Soft: Signups, SDK,
downloads
App usage, deals
signed
API calls generated
API-generated revenue

134. The Lean Analytics cycle

135. Pick a KPI Draw a line
Draw a new line
Pivot or
give up
Try again
Success!
Did we move the
needle?
Measure
the results
Design a test
Make changes
in production
Find a potential
improvement
With data:
find a
commonality
Without data:
make a good
guess
Hypothesis

136. Do AirBnB hosts
get more business
if their property is
professionally
photographed?

137. Gut instinct (hypothesis)
Professional photography helps AirBnB’s business
Candidate solution (MVP)
20 field photographers posing as employees
Measure the results
Compare photographed listings to a control group
Make a decision
Launch photography as a new feature for all hosts

138. 5,000 shoots per month
by February 2012

139. Hang on a second.

140. SRSLY?
Gut instinct (hypothesis)
Professional photography helps AirBnB’s business

141. Pick a KPI Draw a line
Draw a new line
Pivot or
give up
Try again
Success!
Did we move the
needle?
Measure
the results
Design a test
Make changes
in production
Find a potential
improvement
With data:
find a
commonality
Without data:
make a good
guess
Hypothesis

142. “Gee, those
houses that do
well look really
nice.”
Maybe it’s the
camera.
With data:
find a commonality
“Computer: What
do all the
highly rented
houses have in
common?”
Camera model.
Without data: make a
good guess

143. Circle of Moms: Not enough engagement
• Too few people were
actually using the
product
• Less than 20% of any
circles had any activity
after their initial creation
• A few million monthly
uniques from 10M
registered users, but no
sustained traction
• They found moms were far more engaged
• Their messages to one another were on average 50% longer
• They were 115% more likely to attach a picture to a post they wrote
• They were 110% more likely to engage in a threaded (i.e. deep)
conversation
• Circle owners’ friends were 50% more likely to engage with the circle
• They were 75% more likely to click on Facebook notifications
• They were 180% more likely to click on Facebook news feed items
• They were 60% more likely to accept invitations to the app
• Pivoted to the new market, including a name change
• By late 2009, 4.5M users and strong engagement
• Sold to Sugar, inc. in early 2012

144. Landing page design A/B testing
Cohort analysis General analytics
URL shortening
Funnel analytics
Influencer Marketing
Publisher analytics
SaaS analytics
Gaming analytics
User analytics Spying on users
User interaction Customer User segmentation satisfaction KPI dashboards

145. Consider a media company

146. Valuable, hard
Paid revenue
sources
Word of mouth;
public support
Why clickbait is
on its way out
Simple to count
Icky, easy
What do you
want visitors to
do?
The changing face of
engagement
Ignore
Back away
Bounce
One-time
Lurk
Stay silent
Hoard
Criticise
Take
Abandon
Cancel
Upgrade
Renew
Subscribe
Create
Endorse
Share
Respond
Interact
Explore
Stay
See
Click Downgrade

147. The tools media can use
Editorial decisions
Pagerank/reputation
Followers/subscribers
Topic chosen
Format (quiz, story, etc.)
Tone (controversy, etc.)
Headline, imagery
Timing, platform
Long-term, sustainable
Short-term, transient

148. Modern media’s
new gauntlet
Click
See
Stay
Explore
Interact
Respond
Share
Endorse
Create
Subscribe
Renew
Upgrade
Editorial decisions
Pagerank/reputation
Followers/subscribers
Topic chosen
Format (quiz, story, etc.)
Tone (controversy, etc.)
Headline, imagery
Timing, platform
From here
(cats and
royalty)
To here
(an informed
electorate and
citizen approval)

149. Some non-tech examples.

150. I lied. Everyone is a tech company.

151. Cost of experiments: down. Cost of attention: way up.
http://www.flickr.com/photos/puuikibeach/4789015423 http://www.flickr.com/photos/elcapitanbsc/3936927326

152. Let’s pick on restaurants
for a while.

153. Empathy: find the need
Before opening, the owner first learns about the diners in her
area, their desires, what foods aren’t available, and trends in
eating.
Key metrics: Popular items;
frequent questions; before/after
dining patterns.
Reference: Emerging need.

154. Stickiness: confirm the need is met.
She develops a menu and tests it out with consumers,
making frequent adjustments until tables are full and patrons
return regularly. She’s giving things away, asking diners what
they think. Variance and uncertain inventory make costs high.
Key metrics: Customer loyalty;
recommendations; referrals;
endorsements; inventory turnover.
Reference: Business idea.

155. Virality: will it spread?
She starts loyalty programs to bring frequent diners back, or
to encourage people to share with their friends. She engages
on Yelp and Foursquare.
Key metrics: Customer loyalty;
recommendations; referrals;
endorsements.
Reference: Business positioning

156. Revenue: prove the business viability
With virality kicked off, she works on margins—fewer free
meals, tighter controls on costs, more standardization. She
focuses on the price of acquiring new customers.
Key metrics: Acquisition cost,
revenue per cover, capacity,
turnover.
Reference: Business model.

157. Scale: prove it’s a market
Knowing she can run a profitable business, she funnels
revenues into marketing and promotion. She reaches out to
food reviewers, travel magazines, and radio stations. She
launches a second restaurant, or a franchise.
Key metrics: Franchise health;
repeatability; problems escalated;
variance; franchise revenues.
Reference: Business plan.

158. A line in the sand
Labor costs
Gross revenue
30%
24%
20%
Too costly?
Just right
Understaffed?
=

159. A leading indicator
50 reservations
at 5PM
250 covers
that night
(Varies by
restaurant.
McDonalds
≠ Fat Duck.)
http://www.flickr.com/photos/mysticcountry/3567440970 http://www.flickr.com/photos/avlxyz/4889656453

160. http://www.flickr.com/photos/southbeachcars/6892880699
Restaurant MVP

161. Is tip amount a leading indicator of long-term
revenue?

162. Why does every table get the same
menu?

163. Is tipping even a good idea?
Customers like
Servers like tipping
tipping because it puts
because it means their
them in the driver’s
talent is rewarded. As a
seat. As a diner, you
great server, you get
control your
paid more than your
experience, using the
peers, because you are
power of your tip to
a better worker.
make sure your server
works hard for you.
Owners like tipping
because it means they
don’t have to pay for
managers to closely
supervise their servers.
With customers using
tips to enforce good
service, owners can be
confident that servers
will do their best work.
Is this true? How would you know?
Jay Porter founded the Linkery, San Diego's leading farm-to-table restaurant.

164. The truth about tips
Customers don’t vary tips much according to service.
Tipped servers are rewarded for maximizing the number of guests
they serve, even though that degrades service quality.
Servers learn to profile guests, focusing on stereotypes while giving
women, ethnic minorities, the elderly and those from foreign
countries bad experiences
When a server is punished, the server can keep that information to
himself. The message never makes it to a manager, and the problem
is never addressed.

165. The truth about tips
Sharp increase in business over the first two months of the
new system:
Servers’ total pay rose to about $22/hour
Most of the cooks started making about $12-14
depending on experience
The dishwashers about $10
http://jayporter.com/dispatches/observations-from-a-tipless-restaurant-part-1-overview/

166. Is purple ink better?
http://tippingresearch.com/uploads/managing_tips.pdf

168. Stalking
customers
is pretty
easy.
http://targetmycustomers.appspot.com http://tippingresearch.com/uploads/managing_tips.pdf

169. Growth hacking
(is a word you should hate but will hear a lot about.)

170. Growth hacking, demystified.
Find
correlation
Test
causality
Optimize the
causal factor
Pick a metric
to change

171. http://blog.justgiving.com/nine-reasons-why-social-and-mobile-are-the-future-of-fundraising/
Is social action a leading
indicator of donation?

172. http://blog.justgiving.com/nine-reasons-why-social-and-mobile-are-the-future-of-fundraising/
Is mobile use?

173. Guerrilla
marketing
Data-driven
learning
GROWTH
HACKING
Subversiveness

174. The growth hack
•Growth hacking is simply what marketing should have been
doing, but it fell in love with Don Draper and opinions along the
way
•Optimize a factor you think is correlated with growth

175. AirBnB and Craigslist

179. What if you’re an Intrapreneur?

180. Business model vs.
company stage
Early stage Big/incumbent
Company size/age
B2B
Target
market
B2C
More formal decisions
Less WoM
Slower cycle time
More legacy constraints
It is way too easy to
mix these up.
Intrapreneurs

181. When you’re a startup
your goal is to find a sustainable,
repeatable business model.
When you’re a big company
your goal is to perpetuate one.

182. Intrapreneur:
Someone working to produce
disruptive change in an organization
that has already found a sustainable,
repeatable business model.

183. In a startup, the purpose of analytics is
to iterate to product/market fit
before the money runs out.

184. In a big company,
analytics replaces opinion with fact.

185. (Before we get into Lean Analytics, 2 key lessons.)
Lesson one:
Companies die because they fail to
move to new business models.

186. Cost per MB
$1
$10
$100
$1000
Clay Christensen, The Innovator’s Dilemma
Time
14”
Mainframe
8”
Minicomputer
5.25”
Desktop
3.5”
Notebook

187. Technologies
outstrip what
the market
needs, driven
by feedback
from the
“best”
current
customer.
$1
$10
$100
$1000
Clay Christensen, The Innovator’s Dilemma
8” 5.25”
Time
High end
customer
Low end
customer

188. The new
market has
different criteria
for success,
which are
uninteresting to
incumbents.
$1
$10
$100
$1000
Clay Christensen, The Innovator’s Dilemma
Time
Storage
capacity
Portability

189. Sometimes
this has
unintended
consequences
$1
$10
$100
$1000
Clay Christensen, The Innovator’s Dilemma
Smaller disc size means less vibration
impact, leading to greater density,
increasing storage capacity
Time

190. Three kinds of innovation
Sustain/core
(optimizing for more of the same)
Innovate/adjacent
(introduce nearby product,
market, or method)
Disrupt/transformative
(Fundamentally changing
the business model)
Improve along
current metrics...
...or alter
the rate of
improvement
Switch to a new
value model
Change the business
model entirely

191. Lesson two:
The difference between a rogue agent
and a special operative is permission.

192. It’s different when you’re big.

193. If big firms
can’t innovate,
it’s this guy’s
fault.

194. When product and market are known,
companies compete on how they do
things.

195. To get the incremental cost to zero,
companies competed on scale.
(Literally, an economy of scale)

196. Scale comes from process, IP, org
chart, capitalization.
All of these assume the future will be
like the past, only more so.

197. If a startup is an organization designed
to search for a sustainable, repeatable
business model, then an established
company is an organization designed
to perpetuate one.

198. Technology has radically changed the
incremental cost of businesses.

199. Software is eating the world.
http://www.flickr.com/photos/ebolasmallpox/3733059220/

200. An economic order quantity
of one.
Crafted Mass-produced
Automated Digital
Quantity Few Many Some One
Cost High Low Medium Free
Lead time Small Large Medium None
Self-service Medium None Some Lots
Customization High None Some Lots
This is why
software is
eating the
world.
• Cloud computing
• Social media
• 3D printing
• Per-customer
analysis
• Mobile tracking
• Etc...

201. Sustainable competitive advantage allows for
inertia and power to build up along the lines of
an existing business model, which will soon die.
Instead, seek transient competitive
advantage.
Rita Gunther McGrath, The End of Competitive Advantage

202. Scale is now a liability. Compete on
cycle time.

203. CW&T make a pen

204. http://www.flickr.com/photos/art_es_anna/288880795/

206. Optimizing the probable means
discounting the possible.

208. This isn’t about a lack of resources.

209. http://www.flickr.com/photos/maladjusted/5207565912

210. Blockbuster had a lot going for it.

211. Plenty of inventory, of course.
But that matters less than...

212. ...market intelligence,
customers, existing
payment approval,
and customer
history.

213. The problem was framing:
Blockbuster thought it was in the video
store management business. Netflix
realized it was in the entertainment
delivery business.

214. YOU ARE
HERE

215. LOCAL
MAXIMUM
YOU ARE
HERE
OPTIMIZATION
OF CURRENT
METRICS

216. YOU ARE
HERE
GLOBAL
INNOVATION MAXIMUM
WITH NEW
RULES

217. YOU ARE
HERE
SHORT-TERM INVESTORS
HATE GOING DOWNHILL

218. First mover advantage happens
long before the market emerges.
• $1B invested in Nook
• $475M operating loss
in April 2013
• CEO gone

219. Constraints slow things down
vs.

220. Capital cycles don’t fit the short, iterative
nature of startup uncertainty
12 month budgeting cycle; annual plan. Future based on past.
Agile, scrum, lean iterations. Today’s model. No evidence about the future.
Project
Project
Project
Project
Project
Project
Project
Project
Project
Project
Project
Project
(Requires budget insulation)

221. Everything to lose:
Why big companies need innovation.

223. F500 Life
Expectancy
(http://csinvesting.org/2012/01/06/fortune-500-extinction/)
75
years 15
years
1950 ... 2010
Growth by entering
a new business 95 % fail
Corporate
Strategy Board
99 % fail
Clay
Christensen

224. mikemace.com
The slow
death of
a market
leader.
“Time to enter
the mainstream.
Cut prices.”
“Let’s cut
prices to
accelerate
our growth.”
“We may miss the quarter.
Let’s do a price promotion.”
“That wasn’t
supposed to
happen. We’ll
have to lay
some people
off.
Revenue over time
This is what most
managers track.
Note that sales keep
rising (making you
feel safe) until you
run off the edge of
the cliff.
The adoption curve
Here’s where you
actually are, but you
don’t know it
because you can’t
draw the curve until
after the market
saturates.
Early
adopters
Late
adopters
Gross margin
percent
Declining profit
per unit (gross
margin) is actually
your best signal of
trouble.

225. In other words, if your job is change you
have your work cut out for you.

226. A dose of pragmatism

227. Many models for enterprise innovation
Core Adjacent Transformative
Do the same
Nearby product,
Start something
thing better.
market, or method.
entirely new.
Regional
optimizations.
Innovation, go-to-market
strategies.
Reinvent the
business model.
• Get there faster
• Smaller batches
• Solution, then testing
• Increased accountability
• Customer development
• Test similar cases
• Parallel deployment
• Analytics & cycle time
• Fail fast
• Skunkworks/R&D
• Focus on the search
• Ignore the current
model & margins

228. Another way to look at it
Core Adjacent Transformative
Know the problem
(customers tell you it)
Know the solution
(customers/regulations/
norms dictate it.)
Know the problem
(market analysis)
Don’t know the solution
(non-obvious innovation
confers competitive
advantage.)
Don’t know the problem
(just an emerging need/
change)
Don’t know the solution.
Waterfall:
Execution
matters
Agile/scrum:
Iteration
matters
Lean Startup:
Discovery
matters

229. The Three Horizons
Core Adjacent Transformative
Those core businesses most
readily identified with the company
name and those that provide the
greatest profits and cash flow.
Maximize remaining value.
Emerging opportunities, including
rising entrepreneurial ventures
likely to generate substantial
profits in the future but that could
require considerable investment.
Ideas for profitable growth down the
road—for instance, small ventures
such as research projects, pilot
programs, or minority stakes in new
businesses.
Horizon 1 improves the
current business operations
in the next 12 months.
Horizon 2 extends the
business into new products,
markets, or methods in the
next 3 years.
Horizon 3 changes the
industry you’re in and your
value network in the next 6
years.
http://www.mckinsey.com/insights/strategy/enduring_ideas_the_three_horizons_of_growth

230. Lean applies. Startup may not.
Core Adjacent Transformative
Lean methodologies.
Lean
startup

231. Experiment with product, market,
and method.

232. Method
(new “how”) 3 kinds of
innovation
Product
(new “what”)
Market
(new “who”)

233. Product
(new “what”)
Market
(new “who”)
Method
(new “how”)
Startup
Distribution
innovation
Market
diversification
Core
More things to more people for more money
more often more efficiently. (Zyman)
Innovative
Change what you sell, or who
you sell to, or how you sell it.
Transformative
Fundamentally change business and value
proposition. Rebrand & cannibalize.
Channel
expansion Disruptive
Create what wasn’t possible, based on
massive societal or technical change.

234. Engine as a service

235. Engine as a service
http://www.nasa.gov/images/content/365835main_airplane_noise_qtd2_3024x2016.jpg

236. “Efficiency is tied to
analytics. We’ll still look
for new materials, or for
the physics of devices,
but the analytics ... is
what’s really untapped.”

237. Business
optimization
(five mores)
Current
state
Product,
market,
method
innovation
Business
model
innovation
You can convince
executives of this
because some of it
is familiar.
This terrifies them
because it eats the
current business.
A three-maxima model
of enterprise innovation

238. Improvement Adjacency Remodeling
Do the same,
Explore what’s
only better.
nearby quickly
Try out new
business models
Lean approaches apply, but the metrics vary widely.
Sustain/
core
Innovate/
adjacent
Disrupt/
transformative

239. Sustaining Adjacent Disruptive
Next year’s car Electric car,
same dealer
On-demand, app-based
car service

240. Sustaining
innovation
is about
more of
the same.
(says Sergio Zyman)
More things
To more people
For more money
More often
Inventory increase
Gifting, wish lists
Highly viral offering
Low incremental order costs
Maximum shopping cart
Price skimming/tiering
Loyal customer base that returns
Demand prediction, notification
More efficiently Supply chain optimization
Per-transaction cost reduction

241. Blizzard extends the
lifespan of WOW
Early
adopters
Rapid
growth
Market
saturation
The infamous S-curve
(Product lifecycle, Bass diffusion curve, etc.)

242. Blizzard extends the
lifespan of WOW

243. Blizzard extends the
lifespan of WOW
Fixing this: sustaining growth with novelty
Product & market innovation
(“New & improved!”)

244. Blizzard extends the
lifespan of WOW
WOW
Wrath of
the Lich King
Burning
Crusade
Mists of
Cataclysm Pandaria

245. Most of your
innovation will
be adjacent or
sustaining.
Question marks!
(low market share,
high growth rate)
May be the next big thing.
Consumes investment, but
will require money to
increase market share.
Stars!
(high growth rate,
high market share)
What everyone wants. As
market invariably stops
growing, should become
cash cows.
Dogs!
(low market share,
low growth rate)
Barely breaks even, may
be a distraction from better
opportunities. Sell off or
shut down.
Pivot to
increase growth
rate through
disruption
Cash cows!
(high market share,
low growth rate)
Boring sources of cash, to
be milked but not worth
additional investment.
Growth rate
Pivot to
increase
market
share
through
virality,
attention
Market share
Pivot to
redefine problem/
solution through
empathy
Milk with
revenue
optimization as
growth slows
If you don’t like
this, go launch
a startup.

246. Software, experimentation, and
iterative cycles of learning help you
get to the local maximum better and
faster. That’s a good thing.
But it’s not the only thing.

247. Adjacent innovation is about changing
one part of the model in a way that
alters the value network.

248. Amazon Web Services and the
server value network
Server computing
• Density
• GHz
• Heat
• MIPS
Cloud computing
• Instances
• Objects
• Spinup time
• Scaleout
Capex, financing,
TCO, ROI
Opex, demand, time
to result
CIO, enterprise IT CTO, coder, app owner,
line of business, startup
Value
criteria
Money
Buyer

249. Adjacent product
to the same
market in the
same way

250. Selling the same product to an
adjacent market in the same
way.
Of P&G’s 38 brands, only 19 were sold in Asia as of 2011
Market expansion is seldom selling the same thing to new people. In
Asia, P&G needed to
Align pricing with novelty (prestige, mass-tige, over-the-counter)
Change consumer expectations (moving from dilutes to
concentrates)
Adjust positioning and ingredients such as white fungus, ginseng,
and the parasitic cordyceps

251. Selling the same product to the same
market in a new way.
The biggest innovation in
logistics of the 20th century.
http://www.flickr.com/photos/photohome_uk/1494590209

252. Changing the method of
C2C classifieds
A blend of who, what, how
Classified C2C sales (same
“what”)
Strictly for Japanese women
(targeted “how”)
New how (phone is capture,
display, payment, transaction)
Did 100 interviews w/target
users before launch
Key insight: Japanese women
sell their entire wardrobe
twice in their lives
5,000 and 10,000 sales in first
month
10% commission fee
Average price of items is
pretty low, at around 2,000 to
3,000 yen (or $22 to $34)
Not an auction: seller decides
price
Mobile-only model
Phone is payment, storefront,
and even a way for sellers to
build their catalog
http://www.sffashtech.com/2012/10/10/a-free-market-fashion-app-exclusively-for-women-japan/

253. Selling the same product to the
same market in a new way.

254. (At this point, observant Intrapreneurs
should be asking, should P&G be in
the house cleaning business?
And that would be transformative.)

255. Transformative innovation is about
taking a leap, changing more than one
dimension simultaneously in search of
a new business model.

256. If sustaining, incremental innovation
produces linear growth, then
disruptive, transformative innovation
produces exponential growth.

257. Transformative isolation:
Skunkworks

258. Transformative
incubation:
Metlife Infinity

259. Transformative incubation:
Taser evidence.com
Significant market
850K full-time law enforcement officers in the
US; 700K state/local; 525K patrol officers
130M incident reports/y. 70M new incidents;
200K involve use of force
Only 31% of local police agencies keep
computer files on use-of-force incidents
Strong product benefits
Exonerates the officer 96% of the time.
47% percent increase in charges and
summons (2007)
Patrol officers spend 15-25% of their time
writing incident reports, recorded evidence
reduces this by 22%, meaning 50m more on
patrol
Challenges
New business model
Pricing unclear
SaaS offering
Compliance and governance
Unions, regulation, chain of evidence
Changing the current model (radio is
everything)

260. When it’s you vs. the world.
(A bagful of tricks from agitators in companies of all sizes.)

261. The Lean Analytics lifecycle of an Intrapreneur
Beforehand Get buy-in Political fallout
Empathy Find problems; don’t test demand.
Skip the business case, do analytics
Entitled, aggrieved
customers
Stickiness Know your real minimum based on
expectations, regulations
Hidden “must haves”,
feature creep
Virality Build inherent virality in from the
start; attention is the new currency
Luddites who don’t
understand sharing
Revenue Consider the ecosystem, channels,
and established agreements
Channel conflict,
resistance, contracts
Scale Hand the baton to others gracefully Hating what happens
to your baby

262. The Zero Overhead principle
A central theme to this new wave of innovation is the
application of core product tenets from the consumer space
to the enterprise.
In particular, a universal lesson that I keep sharing with all
entrepreneurs building for the enterprise is the Zero
Overhead Principle: no feature may add training costs to
the user.
DJ Patil

263. The job of an intrapreneur is to
identify an adjacent market, product,
or method that conforms to
organizational filters.
It is not to improve the current
product, market, or method.

264. Also: a pariah.
Successful innovators share certain attributes.
Bad listener: Wilfully ignore feedback from your best customers.
Cannibal: If successful, destroying existing revenue streams.
Job killer: Automation & lower margins are your favorite tools.
Security risk: Advocate of transparency, open data, communities.
Narcissist: Worry constantly about how you’ll get attention.
Slum lord: Sell to those with less money, deviants, and weirdos.

265. The six habits of highly unrealistic leaders
Bad leaders:
Filtered information
Selective hearing
Wishful thinking
Fear
Emotional overinvestment,
Unrealistic expectations from
capital markets
Good Intrapreneurs:
Access to the real information
Go where the data takes you
Set aside your assumptions
Embrace uncertainty
Surgical detachment
Have high standards with low
expectations
Confronting Reality (Crown Business), Larry Bossidy and Ram Charan

266. Know what
kind of
innovation
you’re
after.
New
Current
Market development:
Sell existing products
to new markets,
segments, uses.
Export & license.
Startup:
New products for new
markets. New rules,
business units,
organizational
structure. Innovation.
Current New
Market
Product
Penetrate:
Increase revenues,
market share, product
quality, brand
differentiation.
Marketing.
Product
development:
Invent new products
for your market. R&D,
enhancements.
Acquisition.
Based on H. Igor Ansoff’s matrix
Increased risk of political fallout (and great success!)

267. Use outliers and missed searches to
hunt for good ideas & adjacencies
1/8 men have an incontinence issue. 1/3
women do.
When search results show a significant
number of men searching, this suggests the
adjacent (male) market is underserved.
(Multi-billion-dollar hygiene product company)

268. Frame it like a study
Product creation is almost
accidental.
Unlike a VC or startup, when
the initiative fails the
organization still learns.
http://www.flickr.com/photos/creative_tools/8544475139

269. When in doubt, collect data
From tackling the FTA rate to
visualizing the criminal justice
supply chain.

270. Use data to create a taste for
data
Sitting on Billions of rows of
transactional data
David Boyle ran 1M online surveys
Once the value was obvious to
management, got license to dig.

271. Smart Badge
4” e-ink display with
name and specialty.
Badge scans barcode and
gets specs; checks
inventory; enters data on a
touch screen.
Data Exhaust
Today: Workers see their own
productivity.
Coming soon: comparing
yourself to 400,000 other
employees.
Ultimately: Learning what
(and who) works well.
Tesco connects
its workforce

272. Don’t just collect data,
chase it.

273. Understand hidden
constraints
That pencil story is a myth.
Graphite is conductive and
explosive. The Minimum
Viable Product is Viable for a
reason.

274. Know what has to
be built in-house
SAP integration
Employment law

275. http://www.flickr.com/photos/bootbearwdc/1243690099/
Think subversively.

276. Everything’s an excuse to experiment

277. Find other ways
to collect data;
everything is an
experiment.

278. Run it as a consulting business first.
(Just don’t get addicted to it. Your goal is to
learn and overcome integration challenges and
find the 20% of features that 80% of the market
will pay for.)

279. Convince your boss she asked for this
Draw a line
in the sand Pick a KPI
Draw a new line
Pivot or
give up
Try again
Success!
Did we move
the needle?
Measure
the results
Design a test
Make changes
in production
Find a
potential
improvement
With data:
find a
commonality
Without
data: make a
good guess
Hypothesis

280. Focus on the desired behavior, not just
the information.
26% increase in towel
re-use with an appeal
to social norms; 33%
increase when tied to
the specific room.
http://www.psychologytoday.com/blog/yes/
200808/changing-minds-and-changing-towels
The effectiveness of energy
conservation “nudges” depends on
an individual’s political ideology ...
Conservatives who learn that their
consumption is less than their
neighbors’ “boomerang” whereas
liberals reduce their consumption.
Energy Conservation “Nudges” and Environmentalist
Ideology: Evidence from a Randomized Residential Electricity
Field Experiment - Costa & Kahn 2011

281. Slaughter a sacred cow:
Prove a long-held assumption is
wrong and you’ve got people’s
attention.
Know what you’ll do with it ahead of
time.

282. Take baby steps.

283. Netflix

284. Tesla
http://www.hdwallpapersinn.com/wp-content/uploads/2012/12/600-tesla.jpg

285. Twitter’s 140-character
limit isn’t arbitrary. It’s
constrained by the size
http://i.i.cbsi.com/cnwk.1d/i/tim/2011/11/18/
sms_screen_twitter_activity_stream_270x405.png

286. Figure out how to translate it back to a
simple model that fits the company’s
existing value model.
If your company dies, this is why.

287. Intrapreneurs often have to use proxies
Stage Startup metrics Intrapreneur metrics
Empathy
Customers interviewed (needs &
solutions), assumptions quantified,
TAM, monetization possibility
Non-customers interviewed;
assumptions quantified, constraints
identified, TAM, disruption potential
Stickiness Churn, engagement Support tickets, integration time, call
center data, delays
Virality Viral coefficient, viral cycle time Net Promoter Score, referrals, case
study willingness
Revenue Attention, engagement
Billable activity; signed LOIs; pilot
programs; after-development
profitability
Scale Automation Contribution, training costs, licensing

288. When you have support.
(What companies like P&G, Cognizant, GE, and Motorola do with a
formal innovation program.)

289. Do you really have permission?
What resources do you have?
Staff, budget, unfettered access to customers?
What scope of change can you make?
Pricing, product, channel, branding?

290. 2011 MIT study of 179 large publicly traded firms
Companies that use data-driven
analytics instead of intuition have
5%-6% higher productivity and
profits than competitors.
Brynjolfsson, Erik, Lorin Hitt, and Heekyung Kim. "Strength in Numbers: How Does Data-Driven
Decisionmaking Affect Firm Performance?." Available at SSRN 1819486 (2011).

291. The fundamental shift
Ask
question
Define
schema
Collect
data
Answer
question
Refine
problem
Collect
data
Ask
question
Emergent
schema
Explore
data
Answer
question
“Collect first; ask questions later.”

292. What kind of mandate do you have?

293. Step one: Develop a portfolio approach.

294. Innovation portfolios at big companies
Core Adjacent Transformative
70% 20% 10%
Investment
10% 20% 70%
Return

295. Organizations’ structures emerge as a way
to optimize the current business model.
Most innovations will come not from
product or market, but from method—
business model innovation.
Innovation groups must exercise
organizational amnesia at the outset.
1.
2.
3.

296. Tomorrow’s company: Running parallel businesses
Innovation Sustaining/core Adjacent Transformative/
disruptive
Core action Optimizing/
improving Experimenting Searching/
inventing
Focus on Known metrics Risk removal Assumption
validation
Which will live Within current
business unit
Incubated, then
integrated
As new/separate
entities
Problem is Known Known Unknown
Solution is Known Unknown Unknown

297. Step one: Frame your problems.

298. (See also: Christensen’s Jobs to be done.)

299. Step two: Define your gates and filters.
These may lead to myopia.
They are also your unfair advantages.

300. The 3 stages of the
Emerging Business Office
Generation makes increasing
investments in companies
4 pillars: Capturing,
connecting, deciding, and
acting on data
Market sizing
3 horizons & timeframes
Science: 3-5y
Development: 1-2y
Preparation: <12m
“A startup could take
a year to talk to as
many customers as we
Exploration proves out both
the tech and business model
do in a week.”
Challenge the assumptions.
“Am I hitting my milestones?”
and “Are my assumptions
still valid?”
Rapid prototyping. Focus on
risk and uncertainty: which
things don’t we know how to
do?
Adoption means customer
buying in
External partner; validate
across multiple constituents
to ensure it’s a scalable
business model.
Use customer base as an
advantage. C-level
conversations almost
immediately.
6-8 Generate projects
What’s the value proposition?
Why are we going to make money?
Why Motorola?
4 Explore projects
Number of dangling assumptions
Rate at which it’s growing/sinking
Very deep with small sample size
6-8 Adopt projects
Casting a wider net
Go-to-market metrics
Funnel size and market segments

301. Step three: Secure funding, resources, and
executive backing.

302. Reinvesting
For every $100 they cut, Metlife reinvests $66 in new projects.

303. Step four: Generate new ideas.

304. Find non-obvious adjacencies
LIGH
T
ELECTRICAL
GENERATOR
SOFTWARE TO
CUT DOWN
TREES BETTER
MRI MACHINE
POWER GRID
PLANE ENGINE
REQUIRES
TRAIN ENGINE WIND TURBINE
NEEDS
AN
WHICH
FEEDS A
HAS A
TURBINE
LIKE A
TURNED
AROUND
BECOMES A
SPINS &
VIBRATES
LIKE AN
AND
LOOKS
LIKE A

305. Build an ecosystem
Canada’s largest directory
publishing and local
marketing services company
1.5M listings from 420K
SMB & national customers
Revenues >$1.2B
2,500 employees
Created third-party listing API
Took 8-10 mo (2009-10) to
get approval
API payoff happened 2y later
Yahoo replaced Canadian
digital properties search
with the YellowAPI
Improved SEO, Comscore
Functional prototype in
hours, testing in days, and
launching in weeks.
Faster time to partnerships
Budgets tripled in 2013
KPI evolution
Soft: Signups,
SDK, downloads
App usage,
deals signed
API calls
generated
API-generated
revenue

306. Three sources of innovation
Top-down: Areas where business heads see market trends but white spaces in our offerings. Maybe we
can fill this white space.
Bottoms-up
160,000 associates worldwide use an app called Spark; if viable we put it in front of the EBA
leadership meeting
Sparktank meeting—should we put $100-$150K to go and find a first customer.
Outside-in
M&A easier when there is an EBA structure exists because it specializes in integration with the
existing organizaton we bring them into the EBA and help them match
Innovation/investment team backs a few people who have a good idea and can use the
infrastructure, channel, etc.

307. Five common models for
transformative innovation
All employ different
models at different times.
Acquisition
Collaboration
Isolation
Incubation
Integration
Buy promising startups
Crowdsource, work with
universities, suppliers, etc.
Create a separate group
with different conditions
Internal startup ecosystem;
LoB are “investors”
The LoB does innovation
internally

308. Step five: Test by doing: Experimentation
beats projection.

309. Focus on
the model,
not the
plan
Demand
People per day on sidewalk
Percent that buy a glass
Daily customers
Revenue
Price per cup
Cost of Goods
Cost per cup
Profit per cup
Daily profit
Amt
200
10%
20
$5
$1
$4
$80
Growth
4%
5%
-2%
Wk 1
204
15%
31 43 56
$5 $5 $5
.98
Wk 2
216
20%
.96
Wk 3
225
25%
.94
$156
30.6
$216
41.5
$281
52.8
4.02 4.04 4.06
125 175 228

310. A business plan is just what happens
when you drag the business model to
the right.

311. Designing an experiment
Problem, solution, and result hypothesis
Test strategy (PoC, survey, interviews, kickstarter, prototype, A/B, etc.)
Cohort & segment to be tested
Metric or assumption being tested
Timebox or total for test
Action you’ll take if you pass or fail

312. Step six: Know what happens afterwards

313. Qualcomm’s initial
innovation model
http://blogs.berkeley.edu/2013/01/28/
designing-a-corporate-entrepreneurship-
program-a-qualcomm-
case-study-part-1-of-2/
Hypothesis Experiment Implement
Idea generation
and selection
Boot camp Idea
advancement
Ideas
Existing models
New
models
Open
innovation
Tech
feasibility
Biz
feasibility
Boot
camp
decision Implement
End user/partner
desirability
Action
s
Option
value
Strategic
value
Exit
value
Company crowd storm Small team designs &
decision
conducts experiments
Company extracts value

314. Qualcomm’s updated model
Criteria
Fully open to all
employees
Ideas implemented
by existing
business/R&D units
Efficient way to
bubble-up best
ideas (and their
champions) to the
timely attention of
top execs
Hypothesis Experiment Implement
Idea generation
and selection
Boot camp
(3 mo, part time)
CEO
open call
Innovator
challenges
Idea mgmt.
system
Discover Network
Accelerate
Pitch to exec team
Self-forming
teams
Filters
Contextual education
Mentorship
Micro-seed funding
Program staff support
BU sponsor home
Employee
team
BU
Sponsor
Program team
Value extraction
Future option value
Strategic value
Exit value

315. Qualcomm’s innovation model:
What was missing
Hypothesis Experiment Implement
POC
Idea generation
and selection
Boot
camp
Idea
advancement
Ideas
Existing models
New
models
Open
innovation
Tech
feasibility
Biz
sustain-ability
Boot
camp
decision Implement
End user/partner
desirability
Action
s
Option
value
Strategic
value
Exit
value
Company crowd storm Small team designs &
decision
conducts experiments
Company extracts value
POC
decision
Unclear what
happened to
founders
Needed a
middle PoC
decision
Sustainability,
not feasibility

316. Step seven: Rinse, repeat.

317. The Lean Analytics lifecycle of an Intrapreneur
Beforehand Get buy-in Political fallout
Empathy Find problems; don’t test demand.
Skip the business case, do analytics
Entitled, aggrieved
customers
Stickiness Know your real minimum based on
expectations, regulations
Hidden “must haves”,
feature creep
Virality Build inherent virality in from the
start; attention is the new currency
Luddites who don’t
understand sharing
Revenue Consider the ecosystem, channels,
and established agreements
Channel conflict,
resistance, contracts
Scale Hand the baton to others gracefully Hating what happens
to your baby

318. Metrics for innovation portfolios.

319. Core metrics
Metrics that matter
• Return on investment
• Total cost of ownership
• Trouble tickets/issues
• Training time
• Comparing to others
Business plan.
Assume it will work.
But the market will change by the
time you’ve built it.
Example: Online parking
tickets

320. Adjacent metrics
Metrics that matter
• Questions answered
• Virality & word of mouth
• Early adopter stickiness
• Regulation
• Total addressable market
Business model.
Assume it will fail.
Your ultimate use case won’t be
what you think it is today.
Example: Mr. Clean
Magic Eraser

321. Transformative metrics
Metrics that matter
• People I’ve talked to
• Prototype creation speed
• Assumptions validated
• Problems uncovered
• Technical feasibility
• Hidden constraints
Business idea.
Assume it is possible.
You hope it will have the
consequences you want but
aren’t sure how.
Example: Headcam
recordings of all officers

322. The Emerging Business
Accelerator
Three Horizons model
Horizon one is traditional services such
as app dev (SAP, Oracle)
Horizon two are offerings that aren’t
quite as big and mainstream, not used
by everyone, but have good traction.
Smaller revenue contribution (IT
infrastructure, vertically focused BPO).
Also includes some strategy/tactics
Horizon three is about identifying ideas
that are worth investing in, allocating
investment, and incubating them
through our own practices.
Projects “graduate” into a lower horizon
20 ventures today, in 1 of 3 dimensions
Innovation along 3 dimensions
New markets: Either traditional
offering in a new place i.e. Latin
America, Can’t simply do labor
arbitrage. Can also be a new
vertical such as government.
New technologies: Social, mobile,
analytics, cloud, Internet of
Things.
New delivery models (“products”):
Platforms, recurring revenues,
building products to enable
vertical business processes.
Explore-to-graduate criteria
Explore phase are looking for a
first customer
Early startup are looking for early-stage
customers.
Late-stage startup are trying to
show they can deliver for multiple
customers; have a business
model
Growth phase has true P&L.
They’re past the cashflow
breakeven.
We’re saying, “we know enough
about price they will pay and how
many people will want, and what it
costs, so we know margin.” If they
can deliver against this we
graduate.

323. Portfolio metrics
“Do we need to
meet more often?”
is a metric.”
Number in the pipeline; is it growing?
How many are crazy vs. real business ideas?
Ideas funded; ideas that were a waste; ideas needing iteration
# of meetings, qualitative feedback, pivots
Have we convinced someone to sign on for something?
Number of proposals issued; pipeline
Satisfied delivery, on budget; trouble tickets; delays; escalation; referenceability
Profitable independent of costs like development?
Ideas
Quality
Funding
Exploration
Solution fit
Demand
Product fit
Profitability
Cashflow B/E 0% overall profitability, beginning to repay initial investment
Graduation Making money overall

324. Key points to clarify in an
innovation program
Hypothesis Experimentation Implementation
• Articulate problems
• Define the right filters
• Get ideas from many
sources
• Confirm funding (money,
people, customer access)
• Agree on analytical
framework
• Balance market, product,
& method adjacencies
• Prioritize riskiest
assumptions
• Time-box assessment
stages
• Test technology, demand,
and business feasibility
• MVP, prototype, pilot, or
science as appropriate for
type of innovation
• Temporary incubator
• Find a home or building
one
• Keep innovators involved
• Merge metrics with
existing business KPIs
• Synchronize innovation
cycles with enterprise
cycles (budget, etc.)
Portfolio metrics; Gates and KPIs for each stage; mix of core, adjacent, and
disruptive innovation.

325. Goals, constraints, context
Sourcing Filtering
Core Integration
Adjacent
Disruptive
(how to decide?)
Adoption by
existing line of
business.
Independence
Creation of a new
line of business.
Cross-pollinate
to current
managers
Evaluation of the innovation program itself
Socializin
g
Test/
validate
w/current
customers
Grow
as a
distinct
business
Top-down
Bottom-up
Outside-in
R&D M&A

326. Some tools and traps

327. Build a message map.
1. Understand the stages a buyer goes through 2. Create benefits; mitigate objections 3. Target the message to the stage the audience is at

328. Everyone in the world
A. I need a car
I should buy
B. a car
It should be
C. a hybrid
I should buy
D. a Honda Civic

329. Everyone in the world
A. I need a car
People who want to drive
I should buy
B. a car
Prospective car buyers
It should be
C. a hybrid
People looking for a hybrid
I should buy
D. a Honda Civic
Honda Civic Hybrid owners

330. “Isn’t it time you got out of the
city?” campaign showing how cars
make nature accessible & ridiculing
urban hipsters.
Ads showing how cars are needed
any time (pregnancy, errands, urgent
business) and how a car is a
“personal assistant.”
Urgency (“every time you drive a
non-hybrid car you kill the planet a
little”) and testimonials from buyers
who’ve saved money.
Honda branding ads and model-specific
promotions.
Follow-up satisfaction campaign to
encourage buyers to tell their friends
Everyone in the world
A. I need a car
People who want to drive
“I need a vehicle to get
around, be productive, and
enjoy my life.”
I should buy
B. a car
Prospective car buyers
“I want to own a car because it’s
convenient; it’s a personal
relationship; I don’t trust others.”
It should be
C. a hybrid
People looking for a hybrid
“I want to save money and fuel. I
also care about the environment
and want to be seen as ‘green’.”
I should buy
D. a Honda Civic
Honda Civic Hybrid owners

331. Everyone in the world
People who want to drive
“I need a vehicle to get
around, be productive, and
enjoy my life.”
Prospective car buyers
“I want to own a car because it’s
convenient; it’s a personal
relationship; I don’t trust others.”
People looking for a hybrid
“I want to save money and fuel. I
also care about the environment
and want to be seen as ‘green’.”
Honda Civic Hybrid owners
Those who don’t need cars
• I’m too young to drive
• I’m too old to drive
• I can walk or take public
transit
Car users who won’t buy
• It’s too expensive for me
• I will use a shared car service
• It’ll get stolen
Those who won’t buy hybrids
• Hybrids are gutless
• Batteries are toxic & explosive
• In the end it costs more than
it saves
I will buy another brand
• I buy domestic
• I’ve always driven a VW
• Toyotas are reliable
• I want something prestigious
A. I need a car
I should buy
B. a car
It should be
C. a hybrid
I should buy
D. a Honda Civic

332. Everyone in the world
People who want to drive
“I need a vehicle to get
around, be productive, and
enjoy my life.”
Prospective car buyers
“I want to own a car because it’s
convenient; it’s a personal
relationship; I don’t trust others.”
People looking for a hybrid
“I want to save money and fuel. I
also care about the environment
and want to be seen as ‘green’.”
Honda Civic Hybrid owners
Those who don’t need cars
• I’m too young to drive
• I’m too old to drive
• I can walk or take public
transit
Car users who won’t buy
• It’s too expensive for me
• I will use a shared car service
• It’ll get stolen
Those who won’t buy hybrids
• Hybrids are gutless
• Batteries are toxic & explosive
• In the end it costs more than
it saves
I will buy another brand
• I buy domestic
• I’ve always driven a VW
• Toyotas are reliable
• I want something prestigious
Sponsor a driving school
“Give the gift of driving”
campaign for grandparents.
PR on dangers of commuting,
pedestrian deaths
Financing, cashback
Sell to carshares;
underscore their limitations
Theft warranty, tracking
services, high-end locks
Independent tests,
standard metrics (0-60 in X)
Lab research, studies
ROI calculator;
replacement programs
Prove Honda hires US workers
“Time to leave Germany” ads
Spontaneous accel. stories
Premium brand (Acura)
A. I need a car
I should buy
B. a car
It should be
C. a hybrid
I should buy
D. a Honda Civic

333. “Isn’t it time you got out of the
city?” campaign showing how cars
make nature accessible & ridiculing
urban hipsters.
Ads showing how cars are needed
any time (pregnancy, errands, urgent
business) and how a car is a
“personal assistant.”
Urgency (“every time you drive a
non-hybrid car you kill the planet a
little”) and testimonials from buyers
who’ve saved money.
Honda branding ads and model-specific
promotions.
Follow-up satisfaction campaign to
encourage buyers to tell their friends
Everyone in the world
People who want to drive
“I need a vehicle to get
around, be productive, and
enjoy my life.”
Prospective car buyers
“I want to own a car because it’s
convenient; it’s a personal
relationship; I don’t trust others.”
People looking for a hybrid
“I want to save money and fuel. I
also care about the environment
and want to be seen as ‘green’.”
Honda Civic Hybrid owners
Those who don’t need cars
• I’m too young to drive
• I’m too old to drive
• I can walk or take public
transit
Car users who won’t buy
• It’s too expensive for me
• I will use a shared car service
• It’ll get stolen
Those who won’t buy hybrids
• Hybrids are gutless
• Batteries are toxic & explosive
• In the end it costs more than
it saves
I will buy another brand
• I buy domestic
• I’ve always driven a VW
• Toyotas are reliable
• I want something prestigious
Sponsor a driving school
“Give the gift of driving”
campaign for grandparents.
PR on dangers of commuting,
pedestrian deaths
Financing, cashback
Sell to carshares;
underscore their limitations
Theft warranty, tracking
services, high-end locks
Independent tests,
standard metrics (0-60 in X)
Lab research, studies
ROI calculator;
replacement programs
Prove Honda hires US workers
“Time to leave Germany” ads
Spontaneous accel. stories
Premium brand (Acura)
A. I need a car
I should buy
B. a car
It should be
C. a hybrid
I should buy
D. a Honda Civic

334. The good news:
The harsh light of data changes everything.

335. Big
Lots of information, in
flight and at rest.
Fast Reliable
Storage and retrieval
in short timeframes.
High availability in
replication, consistency,
and recoverability
(Pick
any two)
Big Data’s
iron triangle

336. Some tools and traps

337. The three threes
Three
assumptions
What big bets are you making?
•“People will answer questions”
•“Organizers are frustrated with how to run conferences”
•“We'll make money from parents”
•“Amazon is reliable enough for our users.”
Three actions
to take
What are you doing to make these assumptions happen (or
identify they’re wrong and change course?)
•Product enhancements
•Marketing strategies
Three experiments
to run
•Feature tests
•Continuous deployment
•A/B testing
•Customer survey

338. The three threes
Three
assumptions
Three actions
to take
Three experiments
to run
Monthly
Weekly
Daily
Board, investors,
founders
Executive team
Employees
Strategy
Tactics
Execution

339. The three threes
Three
assumptions
Three actions
to take
Three experiments
to run
Get more
people
Increase
answer %
Test better
questions
Change
the UI
Test
timings
Question
s from
Many people will
answer questions

340. The problem-solution canvas
CURRENT STATUS
The Goal is to Learn
• List key metrics you’re
tracking, where they’re at, and
compare with last few weeks
• How are things trending?
LAST WEEK’S LESSONS LEARNED
AND ACCOMPLISHMENTS)
• What did you learn last week?
• What was accomplished?
• On track: YES / NO?

341. The problem-solution canvas
HYPOTHESIZED SOLUTIONS
• List possible solutions that you’ll start
working on next week. Rank them.
• Why do you believe each solution will
help you solve or complete solve the
problem?
METRICS / PROOF + GOALS
Problem #1 (put name here)
• Metrics you’ll use to measure whether
or not the solutions are doing what you
hoped (solving the problem)
• List proof (qualitative) you’ll use as well
• Define goals for the metric
HYPOTHESIZED SOLUTIONS
• List possible solutions that you’ll start
working on next week. Rank them.
• Why do you believe each solution will
METRICS / PROOF + GOALS
• Metrics you’ll use to measure whether
or not the solutions are doing what you
hoped (solving the problem)
Problem #2 (put name here)

342. “The most important figures that one
needs for management are unknown
or unknowable, but successful
management must nevertheless take
account of them.”
Lloyd S. Nelson

343. Pic by Twodolla on Flickr. http://www.flickr.com/photos/twodolla/3168857844

344. ARCHIMEDES
HAD TAKEN
BATHS BEFORE.

345. Once, a leader convinced others
in the absence of data.

346. Now, a leader knows
what questions to ask.

347. Ben Yoskovitz
byosko@gmail.com
@byosko
Alistair Croll
acroll@gmail.com
@acroll

348. Workshop: Systems diagrams
and NCARB’s business

351. The mobile app!
customer lifecycle!
Ratings
Reviews
Search
Leaderboards
Purchases
App store!
App sales
Downloads
Installs
Play
Disengagement
Reactivation
Uninstallation
Disengagement
Account"
creation
Virality
Downloads,"
Gross revenue
ARPU
Activation
Churn, CLV
In-app"
purchases
Legitimate
Incentivized
Fraudulent
Ratings!

355. Traction graphs
Your business model
The stage you’re at
Your one metric
... change often if
you’re doing it right.
So how do you track
that over time?

356. Traction graphs
Jan Feb Mar Apr May Jun
Signups
Conversion
Churn
per day
rate
rate
Viral
coefficient
This axis changes for
each metric

357. Traction graphs
Jan Feb Mar Apr May Jun
Signups
Conversion
Churn
per day
rate
rate
Viral
coefficient
0%

358. Use vanity to get to
meaningful metrics
Your goal is to produce
outcomes
If the outcomes require
action, and vanity motivates
actors, use it
But show how the vanity
metric is a leading indicator of
the real one
Web traffic
Activation
x
Revenue
Cart
Size
Conversion
rate

359. Thinking Backwards:
The Solution/Problem approach
Mitigation &
execution Act & measure
Rob Van Haastrecht & Martin Scheepbouwer
Identify a clear,
known goal
Get on same
page with
relevant facts
Agree on goal
KPIs
Outline
possible
solutions
Proposed
solutions
List assumptions
(causes, actions,
costs, risks)
Agree on how
to test/analyze
them
Answer/test
them (MVP,
etc.)
See where
uncertainty
exists
Validation
& testing
Estimate ability
to mitigate
risks (SWOT)
Choose next
best action
(CxO)
Staff team
based on goal
audacity
results

360. Key points
Intrapreneurship is about adjacent or transformative innovation
Sustaining innovation focuses on the Five Mores, within the
current product, market, method, and business model.
Adjacent innovation may come from a new product, market, or
method, but the same business model
Disruptive innovation has different customers, KPIs, and models
The difference between a rogue agent and a special operative is
permission
Portfolios need sourcing, filters, metrics, and socializing
Balancing isolation and integration, R&D and M&A is contentious

361. Ben Yoskovitz
byosko@gmail.com
@byosko
Alistair Croll
acroll@gmail.com
@acroll