1. Lean Analytics
Use data to build a
better business faster.
www.leananalyticsbook.com
@byosko | @acroll
@leananalytics

2. Who you are, who we are, what
we’ll cover

3. Case studies to explore
E- 2-sided Mobile User-gen
SaaS Media
commerce market app content
Empathy
Stickiness
Virality
Revenue
Scale
Plus...

6. All attendees Hashtag
12 clicks
28 responses 3 responses

7. You’re all liars

9. What you asked for

10. Most startups don’t know what they’ll be
when they grow up.
Freshbooks Mitel
was invoicing Wikipedia was a
Paypal
for a web was to be lawnmower
first built for company
design firm written by
Palmpilots experts only
Flickr
Hotmail Twitter Autodesk
was going to
was a was a made desktop
be an MMO
database podcasting automation
company company

11. Kevin Costner is a lousy entrepreneur.
Don’t sell what you can make.
Make what you can sell.

12. Analytics keeps you honest.

13. Analytics is the measurement
of movement towards your
business goals.
http://www.flickr.com/photos/itsgreg/446061432/

14. Analytics, TL;DR
ATTENTION ACTIVATION RETENTION REVENUE
CONVERSION
NEW
RATE
VISITORS
GROWTH PAGES x
SEARCHES PER
VISIT RETURNS GOAL VALUE
TWEETS NUMBER
CONTENT
MENTIONS OF VISITS
TIME ETC.
ADS SEEN ON WORD OF
LOSS SITE MOUTH
BOUNCE
RATE REFERRAL

15. In a startup, the purpose of
analytics is to iterate to a
product/market fit before
the money runs out.

16. Good metrics, bad metrics

17. What makes a good metric?
• It’s comparable to another time period, group, competitor, etc.
• It’s understandable in a way the target audience will understand
• It’s a ratio or rate
• Which means it’s easier to act on (acquisition cost per customer)
• It allows you to represent the tension between two things (ads shown versus
bounce rate, for example)
• It’s targeted to the right audience (Internal business, developers, marketers,
investors, media)
• It changes the way you behave
• “Accounting” metrics make your predictions more accurate
• “Experimental” metrics make your future behavior more effective

18. Think about a car
• You know 60MPH is twice as fast as 30MPH
• In a country, speed limits and mileage are well understood
• Kilometers are conveniently decimal; miles map to hours
• Miles travelled is good; miles per hour is better; accelerating or decelerating
changes your gas pedal
• You can measure “MPH divided by speeding tickets” as a metric of “driving fast
without losing my license”

19. Vanity metrics
A metric from the early, foolish days of the Web. If you have a site with many objects on
Hits it, this will be a big number. Count people instead.
Only slightly better than hits, since it counts the number of times someone requests a
Page views page. Unless you’re displaying ad inventory you should count people instead.
Is this one person visiting a hundred times, or are a hundred people visiting once? Fail.
Visits
The only thing this shows you is how many people saw your home page. It tells you
Unique visitors nothing about what they did, why they stuck around, or if they left.
Counting followers rather than actions is a bad idea. Once you know how many
Followers/friends/likes followers will do your bidding when asked, you’ve got something.
Time on site, or pages Poor substitute for actual engagement or activity. If customers spend a lot of time on
per visit your support or complaints pages, that could be a bad thing.
Until you know how many will open your mails (and act on what’s inside them) this isn’t
Emails collected useful. Test some of them and see.
While it sometimes affects your place in app stores and rankings, downloads alone
Number of downloads don’t lead to lifetime value. Measure activations, account creations, or something else.

20. 5 essential dimensions in analytics
Qualitative Quantitative
Unstructured, anecdotal, revealing, hard to Numbers and stats; hard facts but less insight.
aggregate.
Vanity Actionable
Make you feel good, but don’t change how you’ll Change your behavior by helping you pick a
act. course of action.
Exploratory Reporting
Speculative, trying to find unexpected or interesting Predictable, keeping you abreast of normal,
insights. managerial operations.
Leading Lagging
Number today that shows metric tomorrow— Historical metric that shows how you’re doing—
makes the news reports the news
Correlated Causal
Two variables that change in similar ways , perhaps An independent factor that directly impacts a
because they’re linked to something else dependent one

21. Donald Rumsfeld on analytics
Are facts which may be wrong and
we know should be checked against data.
know
we don’t Are questions we can answer by
reporting, which we should baseline
know & automate.
Things we
Are intuition which we should
we know quantify and teach to improve
don’t effectiveness, efficiency.
know
we don’t Are exploration which is where
unfair advantage and interesting
know epiphanies live.
(Or rather, Avinash Kaushik channeling Rumsfeld)

22. Segments, cohorts, A/B, and multivariates
Cohort:
Comparison of
similar groups
along a timeline.
Segment: A/B test: ☀ Multivariate
Cross-sectional ☀ Changing one analysis
comparison of all thing (i.e. color)
☁ Changing several
people divided by and measuring ☀ things at once to
some attribute
☁ the result (i.e. see which correlates
☁
(age, gender, etc.) revenue.) with a result.

23. Why use cohorts? Here’s an example.
Averages January February March April May
hide the
pattern Rev/customer CA$5.00 CA$4.50 CA$4.33 CA$4.25 CA$4.50
Cohort 1 2 3 4 5
Cohorts
show the January CA$5 CA$3 CA$2 CA$1 CA$0.5
revenue
drop as February CA$6 CA$4 CA$2 CA$1
customers
age March CA$7 CA$6 CA$5
April CA$8 CA$7
May CA$9
Averages CA$7 CA$5 CA$3 CA$1 CA$0.5

24. A few words on causality
http://www.flickr.com/photos/roryfinneren/65729247

25. 50
37.5
25
12.5
0
1 2 3 4 5 6 7 8 9 10
Seat rentals
http://www.rvca.com/anp/wp-content/plugins/wp-o-matic/cache/57226_07+proof+1a+hb+beach+day.jpg

26. http://www.imdb.com/media/rm3768753408/tt0073195

27. http://www.flickr.com/photos/kapungo/2287237966

28. 10000
1000
100
10
1
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Ice cream consumption Drownings

29. http://www.flickr.com/photos/25159787@N07/3766111564

30. http://www.flickr.com/photos/wheressteve/3284532080

31. http://www.flickr.com/photos/wtlphotos/1086968783

32. 10000
1000
100
10
1
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Ice cream consumption Drownings Temperature

33. http://www.flickr.com/photos/stuttermonkey/57096884

34. http://www.flickr.com/photos/germanuncut77/3785152581

35. http://www.flickr.com/photos/fasteddie42/2421039207

36. Mostly, you’re easily distracted
Startup founder

37. One metric to rule them all
After all this, you should focus on one metric.

38. The right information
in the right place
just changes your life.
Stewart Brand

39. The right information
in the right place
at the right time
just changes your life.
Stewart Brand

40. Two dimensions show you
the One Metric That Matters
E- 2-sided Mobile User-gen
SaaS Media
commerce market app content
Business model
Empathy
Stickiness
Stage
Virality
Revenue
Scale

41. An example you might not
have expected.
• Stage: Revenue
• Model: Retailer
• Solare is an Italian fine-dining restaurant under new management. The new team
is trying to identify the key metrics and leading indicators

42. Solare watches the numbers
• One Metric That Matters: Gross Revenue to Labor Cost
• Under 30% is good
• Below 24% is great
• Lower than 20% and you may be under-staffing, leading to dissatisfied
customers
• A leading indicator to optimize: Total covers is 5x reservations at 5PM
• If you have 50 reservations at 5, you’ll have 250 covers that night.
• This ratio varies by restaurant.
• If the reservations cause the covers, then focusing on more reservations will
grow the business. How would they test this?

43. The 5 stages of Lean Analytics

44. Dave McClure’s Pirate metrics
How do your users become aware of you?
Acquisition
AARRR
SEO, SEM, widgets, email, PR, campaigns, blogs ...
Do drive-by visitors subscribe, use, etc?
Activation Features, design, tone, compensation, affirmation ...
Does a one-time user become engaged?
Retention Notifications, alerts, reminders, emails, updates...
Do you make money from user activity?
Revenue Transactions, clicks, subscriptions, DLC, analytics...
Do users promote your product?
Referral Email, widgets, campaigns, likes, RTs, affiliates...

45. Eric Ries’
Three engines
Stickiness Virality Price
Approach Keep people Make people Spend revenue
coming back invite friends getting customers
Math that Get customers How many they Customers are
matters faster than you tell, how fast worth more than
lose them they tell them they cost to get

46. Long Funnel
• Inject signal at the top
• Measure results at the bottom
• Includes multi-channel interactions

47. Social visitors flow report

48. Multi-channel funnels overview

49. Ash Maurya’s
Lean canvas
Lean Canvas box Some relevant metrics
Respondents who have this need; respondents who are aware of having
Problem the need.
Respondents who try the MVP; engagement; churn; most-used/least-
Solution used features; people willing to pay.
Feedback scores; independent ratings; sentiment analysis; customer-
Unique Value Proposition worded descriptions; surveys; search and competitive analysis.
How easy it is to find groups of prospects; unique keyword segments;
Customer Segments targeted funnel traffic from a particular source.
Leads/cust per channel; viral coefficient and cycle; net promoter score;
Channels affiliate margins; open, click-through rate; PageRank; message reach.
Respondents’ understanding of the USP; patents; brand equity; barriers
Unfair Advantage to entry; number of new entrants; exclusivity of relationships.
Lifetime customer value; Average revenue per user; Conversion rate;
Revenue Streams Shopping cart size; Click-through rate.
Fixed costs; cost of customer acquisition; cost of servicing the nth
Cost Structure customer; support costs; keyword costs.

50. Sean Ellis’
Startup growth pyramid
Step on the gas in
Scale new markets,
growth products, channels.
Find a defensible
Stack the odds unfair advantage
and tweak it.
Decide what you
Product/market fit sell to whom, then
prove it.

51. McClure! Sean Ellis
Maurya Lean Lean Lean Analytics
Pirate Growth
Canvas! Startup!
Metrics! Pyramid! phase!
Problem Acquisition!
Problem
validation
(of testers,
prospects, Empathy
Customer Solution
etc.)
segments
validation
Product/
market fit
Unique value MVP building
proposition
Activation
MVP Stickiness
Startup lifecycle stage!
Solution
iteration, Retention
sticky engine
(Natural)
channels
Organic
Stacking the
growth, viral Referral
Virality
Growth rate
engine
odds
Revenue
stream
Monetization,
price engine
Revenue
Revenue
Cost
structure
Scale
(Formal) growth
channels
Inorganic Attention!
growth, (at scale, of Scale
Unfair beyond Lean
customers)
advantage

52. Pause #1
What stage are you at?
What model are you in?
If you could have only one number what would it be?

53. What business are you in?

54. Business model flipbook
Revenue model: How you take money from someone
Product type: What you give them in return Together, these
Delivery model: How you get it to them make up a
Acquisition channel: How they learn about you business model
Selling tactic: How you convince them to buy

55. Paid advertising Banner on Informationweek.com
Search Engine Mgmt. High pagerank for ELC in kid’s toys
Acquisition Social media outreach Active on Twitter i.e. Kissmetrics
channel
How the visitor, Inherent virality Inviting team member to Asana
customer, or user finds
out about the startup. Artificial virality Rewarding Dropbox user for others’ signups
Affiliate marketing Sharing a % of sales with a referring blogger
Public relations Speaker submission to SXSW
App/ecosystem mkt. Placement in the Android market
Simple purchase Buying a PC on Dell.com
What the startup does Discounts & incentives Black Friday discount, loss leader, free ship
Selling
tactic
to convince the visitor Free trial Time-limited trial such as fitbit Premium
or user to become a Freemium Free tier, relying on upgrades, like Evernote
paying customer.
Pay for privacy Free account content is public, like Slideshare
Free-to-play Monetize in-app purchases, like Airmech
One-time transaction Single purchase from Fab
How the startup Recurring subscription Monthly charge from Freshbooks
Revenue
model
extracts money from its Consumption charges Compute cycles from Rackspace
visitors, users, or
Advertising clicks PPC revenue on CNET.com
customers.
Re-sale of user data Twitter’s firehose license
Donation Wikipedia’s annual campaign
Software Oracle’s accounting suite
What the startup does Platform Amazon’s EC2 cloud
in return. May be a
Product
Merchandising Thinkgeek’s retail store
type
product or service; may
be hardware or User-generated content Facebook’s status update
software; may be a Marketplace AirBnB’s list of house rentals
mixture. Media/content CNN’s news page
Service A hairstylist
Delivery
Hosted service Salesforce.com’s CRM
model
How the product gets Digital delivery Valve purchase of desktop game
to the customer. Physical delivery Knife shipped from Sur La Table

56. Business Flipbook
Dropbox example
aspect page(s)
Acquisition Inherent virality. Sharing files with others.
channel Artificial virality. Free storage when others sign up.
Selling Limited-capacity accounts are free;
Freemium.
tactic subscribe when you need more.
Revenue Recurring $99/year, monthly fees, enterprise
model subscription. tiers.
Product Storage-as-a-service with APIs,
Platform.
type collaboration, synchronization tools.
Delivery Hosted service. Cloud storage, web interface.
model Digital delivery. Desktop client software.

57. E-commerce
TL;DR:
• Are you focused on loyalty or acquisition?
• Pricing matters more than you think
• Don’t overlook logistics, delays, and ratings
• Old “average conversion rates”

58. Loyalty or acquisition?
How many of
your customers Then you are in Your customers You are just
Focus on
buy a second this mode will buy from you like
time in 90 days?
Low CAC,
1-15% Acquisition Once 70% high
of retailers checkout
15-30% Hybrid 2-2.5 20% Increasing
per year of retailers returns
Loyalty,
>30% Loyalty >2.5 10% inventory
per year of retailers expansion
(Thanks to Kevin Hilstrom for this.)

59. Pricing has a huge impact
http://hbr.org/1992/09/managing-price-gaining-profit/ar/8

60. Don’t forget the real world
Shipping time, stock availability,
logistics, ratings, and other factors
have a real impact on most e-
commerce companies.

61. E-commerce model:
WineExpress increases
revenues
• Stage: Revenue
• Model: E-commerce
• Exclusive wine shop partner of the Wine Enthusiast catalog and website.
• “Wine of the day” page is highly trafficked, needed optimization

63. WineExpress: before and after
• Tested 3 design variations, primarily focused on layout
• One version was a clear winner
• Conversion went up for a heavily discounted shipping promotion
• Real key was a 41% increase in revenue per visitor

64. Software-as-a-Service
TL;DR:
• Eventually, focus on Customer Acquisition Payback
• Engagement varies by intended use of the app (i.e. CRM versus travel booking)
• Credit cards up front have a huge effect
• Freemium is a sales tactic, not a business model
• Churn, acquisition cost, and lifetime value
• Subscriptions may be a bad thing (per-transaction pricing is an option too.)

65. Some sample churn calculations
• Active users have logged in at least once in the month after signing up
• New users are growing at 20% a month
• 30% use the service at least once (in the month after signing up)
• 2% convert into paid customers
• Example churn calculation for February
• Lost / Starting with (Paying Users) * 100
• 26 / 1035 * 100 = 2.5%
• If 2.5% of customers churn every month, it means that the average customer
stays around for 40 months (100 / 2.5).
• This is how you can start to calculate the Lifetime Value of a customer (40
months * Cost of the Service.)

66. SaaS model:
Backupify’s customer
lifecycle
• Stage: Scale
• Model: SaaS
• Leading backup provider for cloud based data.
• The company was founded in 2008 by Robert May and Vik Chadha
• Has gone on to raise $19.5M in several rounds of financing.

67. Shifting to Customer Acquisition
Payback as a key metric
• Initially focused on site visitors
• Then focused on trials
• Then switched to signups
• Today, MRR
• In early 2010, CAC was $243 and ARPU was only $39
• Pivoted to target business users
• CLV-to-CAC today is 5-6x
• Now they track Customer Acquisition Payback
• Target is less than 12 months

68. Mobile
TL;DR:
• App stores make or break you
• In-app revenue is the only way to make a living

69. Not all churn is equal
“Track churn at 1 day, 1 week and 1 month, because users
leave at different times for different reasons. After one day it
could be you have a lousy tutorial or just aren’t hooking
users. After a week it could be that your game isn’t ‘deep
enough,’ and after a month it could be poor update
planning.”
Keith Katz, co-founder of Execution Labs
and former VP of Monetization for OpenFeint
(Knowing when users churn gives you an indication of why they’re churning and
what you can try in order to keep them longer.)

70. Mobile app model:
Localmind hacks Twitter
• Stage: Empathy
• Model: UGC/mobile
• Real-time question and answer platform tied to locations.
• Needed to find out if a core behavior—answering questions about a place—
happened enough to make the business real

71. Localmind hacks Twitter
• Before writing a line of code, Localmind was concerned that people would never
answer questions.
• This was their biggest risk: if questions went unanswered users would have a
terrible experience and stop using Localmind.
• Ran an experiment on Twitter
• Tracked geolocated tweets in Times Square
• Sent @ messages to people who had just tweeted, asking questions about
the area: how busy is it; is the subway running on time; is something open;
etc.
• The response rate to their tweeted questions was very high.
• Good enough proxy to de-risk the solution, and convince the team and
investors that it was worth building Localmind.

72. Media
TL;DR:
• Advertising is a complicated, many-faced beast

73. Paywalls and baselines
• Paywalls might work
if done right. Jury’s
still out.
• A June, 2012 study by the Advertising Research Foundation conducted across a
half-million ad impressions showed that blank ads bearing no information had a
click-through rate of roughly 0.08%—comparable to that of some paid
campaigns.

74. Media model:
Just For Laughs and
Youtube
• Stage: Revenue
• Model: Media
• Gags program launched in 2000 shows visual pranks with no words, licensed
widely through traditional TV channels. Recently, the organization learned about
its popularity on YouTube and decided to invest time in a channel.

75. Just for Laughs Gags
• Had been a YouTube partner since 2009, but intended to build its own site
• Noticed some unauthorized activity online, so chose to bulk upload 2,000 2-
minute prank clips.
• Experimented with overlay, pre- and post-roll ads. Short format meant intro video
clips drove people away.
• 10-15 second intro resulted in a 30% drop-off.
• Tested longer versus short-form videos
• In 24h, both long- and short-form clips got 30-40K views.
• Long-form clip had 5x that of a short clip—but a long clip had 12 short ones,
so overall the long one pays less.
• Long-form video has a longer viewing tail.
• Long-form has intro, so there’s a 40% audience drop-off halfway into an
episode, versus a 15% drop-off halfway into a single short video.

76. Just for Laughs Gags
• Big growth in UGC since the focus on YouTube
• Could have done a DCMA takedown but decided to monetize instead
• Less lucrative than on-channel content, but volume makes up for it
• 100,000 user-generated videos generate 40-50% of total monthly views
• 2h mash-ups can get millions of views
• Today, JFL tries to learn from and emulate what users show them is working

77. User-generated content
TL;DR:
• Content virality and user virality
• Trying to move users up the engagement funnel
• More time on fraud than you expect
• Notifications and email are the real user interface
• Passive content creation is on the horizon

78. UGC engagement funnels

79. Notifications are the real UI
“… notifications become the primary way I use the
phone and the apps. I rarely open twitter directly. I see that
I have ‘10 new @mentions’ and I click on the notification
and go to twitter @mention tab. I see that I have ‘20 new
checkins’ and I click on the notification and go to the
foursquare friends tab…”
- Fred Wilson
• Allows use of many more engagement apps on my phone without them being on
the main page
• Have as many communications apps as I want. Don’t need to use the apps, just
the notification inbox
• Notification inbox is the new home screen

80. UGC model:
Reddit goes from links to
community
• Stage: Virality
• Model: UGC
• A graduate of the first YCombinator class, reddit was acquired by Conde Nast
but left largely to its own devices. Thanks to a vibrant community and some
good guidance by its founders, it’s a traffic powerhouse.

81. Reddit goes from links to community
• Product evolution
• Started as a simple link-sharing site with voting
• Then added the ability to comment, with votes on comments
• Then created the ability to make “self-posts” rather than only comment on off-
site traffic
• Now self-posts are more than half of all posts
• Revenue from ads and “reddit gold”
• Started as a joke, but turned into a revenue source
• One person paid $1000 for a month; some paid $0.01. Avg. around $4.
• Paying users get early access to features, since they’re an engaged beta
• Spam is a big issue
• Human flagging isn’t good enough
• 50% of the company’s development time is devoted to beating spammers
• Having a 7y “training set” helps them develop and test better algorithms

82. 2-sided market
TL;DR:
• Focus on the money side
• Breaking the chicken-and-egg

83. Plenty of examples
• Real estate listing services
• Crowdfunders like Indiegogo and Kickstarter
• Charities like Donors Choose
• Seller markets like eBay, Craigslist, and Etsy
• App stores
• Dating sites
• Excess inventory travel like Hotwire and Priceline
• They all:
• Include a shared inventory model
• Have two stakeholders—buyers and sellers; creators and supporters;
prospective partners; or hotels and travellers
• Make money when the two stakeholders come together
• Differentiate based on a particular set of search parameters or
qualifications (apartments that have been vetted; seller ratings.)
• Need an inventory to get started

84. Chickens and eggs
Seed the buyer side: demand Seed the seller side: artificial
creation inventory
• When Uber launched in Seattle, it • Amazon started selling books; then
paid drivers $30 an hour to drive broadened its offering; then launched
passengers around a marketplace for goods from many
• Only switched to a commission other suppliers.
model once they had enough • They created inventory; then
demand to make it worthwhile for the demand; and then a two-sided
drivers. marketplace.
• Knew when to switch by: • Knew when to switch by:
Measuring how much drivers would Measuring loyal buyers; identifying
be making on a commission basis, unmet search needs and pent-up
as well as the inventory and the time demand.
it took a driver to pick up a customer.

85. 2-sided market model:
AirBnB and photography
• Stage: Revenue
• Model: 2-sided marketplace
• Rental-by-owner marketplace that allows property owners to list and market
their houses. Offers a variety of related services as well.

86. AirBnB tests a hypothesis
• The hypothesis: “Hosts with professional photography will get more business.
And hosts will sign up for professional photography as a service.”
• Built a concierge MVP
• Found that professionally photographed listings got 2-3x more bookings than the
market average.
• In mid-to-late 2011, AirBnB had 20 photographers in the field taking pictures for
hosts.

87. NIGHTS BOOKED
10 million
8 million
6 million
20 photographers
4 million
2 million
2008 2009 2010 2011 2012

88. Pause #2
Which models would you like to see more about?

89. What stage are you at?

90. Lifecycle stage
Discover a known need within a sizeable market
Identify a solution reachable people will pay for
Develop and validate a viable product to sell profitably
Create a sustainable business model
Attract and appease investors
Find a successful exit

91. Where is the risk?
Real need? Key: Empathy
Right solution?
Key: Stickiness
Key: Growth rate
Good product?
Key: Virality
Sustainable biz?
Key: Revenue
Healthy market?
Successful exit? Key: Scale

92. Lean Analytics “Gate” needed! Key metrics for
this stage! Rationale!
stage! to move forward!
I’ve found a real, poorly-met Qualitative responses
need a reachable market Identifying a real problem and a real solution is
Meetings held
Empathy
faces.
The cheapest thing to do (since it’s just the
Scored qualitative
price of coffee.) It also addresses the riskiest
I’ve figured out how to solve Surveys, bulk feedback
question—will anyone care? It comes first.
the problem in a way they
will adopt and pay for.
Signup rates
Will the dogs eat the dog food? Make your
Activation on invite/beta
Stickiness
mistakes with a small, friendly audience you
MVP adoption
can love and nurture before throwing the
I’ve built the right product/
features/functionality that Retention, churn
unwashed masses at it.
keeps users around.
Message open rates
Inherent sharing
Sharing helps grow, but also verifies that what
Virality
you’ve made is good. Word of mouth is
Growth rate
Viral coefficient/cycle time
endorsement. And virality is a “force
The users and features fuel
growth organically and Word of Mouth sharing
multiplier” for paid customer acquisition.
artificially.
Key goal conversion rates
Will people open their pocketbooks? And can
Customer lifetime value
you charge them enough to fund your
Revenue
I’ve found a sustainable, Customer acquisition cost
ongoing operations, plus your artificial
scalable business with the Margins
acquisition of users?
right margins in a healthy
ecosystem.
Access to employees
Access to capital
You need channels to amortize the cost of
Scale
Competition
sales and distribution. You need an
I can achieve a successful ecosystem to cross the “hole in the middle”
exit for the right terms.
Visibility/SEO/SEM
from niche player to big company
Channel health
Exit!

93. Empathy
You need to get inside your target market’s head. You need to know you’re solving
a problem people care about. This is really risky, so getting out of the building
makes a lot of sense. Few people would dispute this.

94. Signs you’ve found a problem worth tackling
Good signs Bad signs
• They want to pay you right away • They’re distracted
• They’re actively trying to (or have • Their shoulders are slumped or
tried to) solve the problem in they’re slouching in their chair (poor
question body language)
• They talk a lot and ask a lot of • They talk a lot but it’s not about the
questions demonstrating a passion problem or the issues at-hand
for the problem (they’re rambling)
• They lean forward and are animated
(positive body language)

95. How to avoid leading the witness
Avoid biased wording, preconceptions,
Don’t tip your hand or a giveaway appearance. Word your
surveys carefully to be neutral.
Get them to purchase. Ask them to pay. Demand real introductions. Or ask them “how
Make the question real many of your friends would say X” to avoid self-effacement
Ask “why” several times. Leave lingering, uncomfortable pauses in the conversation
Keep digging and let them fill them.
Have a colleague make notes of when they react, or of their body language.
Look for other clues

96. Creating an answers-at-scale campaign
• Know why you’re doing a survey in the first place
Ask what existing Ask how customers Ask what kind of Test which tagline
brands come to try to find a product money people or unique value
mind in an industry or service spend on a proposition
problem resonates best with
customers
Market alongside Help you plan Choose the
them? Address marketing Shape your pricing winning one, or
competitors? campaigns and strategy just take that as
Choose partners? choice of media advice

97. Creating an answers-at-scale campaign
• Know why you’re doing a survey in the first place
• Design the survey Demographic Quantifiable Qualitative,
segmentation answers to your open-ended
questions research problem feedback

98. Creating an answers-at-scale campaign
• Know why you’re doing a survey in the first place
• Design the survey
• Test it (you’ll always have mistakes)

99. Creating an answers-at-scale campaign
• Know why you’re doing a survey in the first place
• Design the survey
• Test it (you’ll always have mistakes)
• Send it out
Via your NW To a paid list As an ad campaign
Beware of Give the solution or
Beware of Audience plea Name the problem
respondent bias, unique value
spamminess, low
misrepresentation
open rates
of the larger market “Are you a single “Can’t sleep? We’re “Our accounting
mom? Take this brief trying to fix that, and software automatically
survey and help us want your input.”) finds tax breaks. Help
address a big us plan the product
challenge.” roadmap.”

100. Creating an answers-at-scale campaign
• Know why you’re doing a survey in the first place
• Design the survey
• Test it (you’ll always have mistakes)
• Send it out Were you able to capture the attention of the
market? Did they click on your ads and links?
Which ones worked best?
• Collect the results
Are you on the right track? What decisions can
you now make with the data you’ve collected? By
• Analyze the data segment!
Will people try out your solution/product? How
many of your respondents were willing to be
contacted? How many agreed to join a forum
or a beta? How many asked for access in their
open-ended responses?

101. Empathy stage:
LikeBright’s mechanical
turk
• Stage: Empathy
• Model: 2-sided marketplace
• Early stage startup in the dating space that joined TechStars Seattle in 2011
• Initially rejected, saying, “We don’t think you understand your customer well
enough.”

102. Talking to a hundred single women in a
day
• Used Mechanical Turk and Google Voice to speak with 100 single women; paid
$2. The interviews lasted typically around 10-15 minutes.
• Simple interview script with open-ended questions, since he was digging into the
problem validation stage of his startup.
• Founder Nick Soman: “I was amazed at the feedback I got. We were able to
speak with one hundred single women that met our criteria in four hours on
one evening.”
• Went back to TechStars and got accepted.
• LikeBright’s website is now live with a 50% female user base, and recently raised
a round of funding.
• “Since that first foray into interviewing customers, I’ve probably spoken with
over a thousand people through Mechanical Turk,”

103. How to score problem interviews
Teach the controversy

104. How to score problem interviews
• Did the interviewee successfully rank the problems you presented?
• Are they actively trying to solve the problems (or have they done so in the past)?
• Were they engaged and focused throughout the interview?
• Did they agree to a follow-up meeting or interview (to present your solution)?
• Did they offer to refer others to you for interviews?
• Did they offer to pay you immediately for the solution?

106. Stickiness
This comes from a good product. This is the same as the Eric Ries’ Stickiness
engine of growth. You need to find out if you can build a solution to the problem
you’ve discovered. There’s no point promoting something awful if your visitors will
bounce right off it in disgust. Companies like Color that attempted to scale
prematurely, without having proven stickiness, haven’t fared well.

107. 1995 Hits
1997 Visits
1999 Visitors
2002 Conversions Who did you add? Where from? Why?
2010 Engagement What did they do? How did it benefit?
Who did you lose? Why did they leave?

110. What time on page
can tell you about goals & behaviors
• People spend ~1m on a page
when they’re engaged with it
• If a page has high traffic and
low engagement, why are
people leaving:
• Did they come expecting
something else?
• Is the layout working?
• Is it simply a page that
isn’t designed to keep
users for long?
• Show off your good stuff.
If a page has a high
engaged time but few
visitors, promote it.
(Thanks to Chartbeat for the analysis.)

111. More interesting when broken
down by business model
(Thanks to Chartbeat for the analysis.)

112. Days since last visit
1200 25000
Number of users
January February
1000
20000
800
15000
600
10000
400
5000
200
0 0
1 2 3 4 5 6 7 8 9 Disengaged
Days since last engagement
(>10 days)

113. Watch out for the dumb stuff
There’s a large number of ways to break an MVP

116. How to stay focused on the right stuff

119. Virality
Once you’ve got a product or service that’s sticky, it’s time to use inherent virality—
word-of-mouth that’s tied into the use of the product. That way, you’ll test out your
acquisition and onboarding processes on visitors who are motivated to try you,
because you have an implied endorsement from an existing user. You can also
think of virality as a force multiplier for paid promotion—so you want to maximize
that multiplier before you start spending money on customer acquisition through
inorganic methods like advertising.

120. 3 kinds of virality
•
Inherent virality is built into the product, and happens as a function of use.
•
Artificial virality is forced, and often built into a reward system.
•
Word of mouth virality is simply conversations generated by satisfied users.

121. Plagues for fun and profit

122. Viral coefficient
------------------------------------------------------
Get your free private email at http://www.hotmail.com
------------------------------------------------------

124. Viral coefficient
v ≠ 1, pt = δp0 (1 – vt+1) / (1 – v) + p0
http://robert.zubek.net/blog/2008/01/30/viral-coefficient-calculation/

125. Or simpler
x Viral - Churn & >
Users
coefficient abandonment 1

126. How to calculate it
• First calculate the invitation rate, which is the number of invites sent divided by
the number of users you have.
• Then calculate the acceptance rate, which is the number of signups or
enrollments divided by the number of invites.
• Then multiply the two together.
• Consider, for example
• Your 2,000 customers have sent out 5,000 invitations during their lifetime on
your site.
• Your invitation rate is 2.5.
• For every ten invitations received, one gets clicked.
• Your acceptance rate is 0.1.
• Multiply the two, and you have your viral coefficient: 0.25. Every customer you
add will add an addition 25% of a customer.

127. Don’t forget cycle time
How long until they invite someone?
• After 20 days with a cycle time of two days, you will have 20,470 users.
• If you half that cycle time to one day, you would have over 20 million users!

129. Virality stage:
Circle of Moms finds an
engaged market
• Stage: Stickiness
• Model: UGC
• Launched as Circle of Friends in 2007, it was a way for small groups to interact
atop Facebook’s platform; but when engagement wasn’t good enough, the
founders decided to dig deeper.

130. The problem: Not enough engagement
• Too few people were actually using the product
• Less than 20% of any circles had any activity after their initial creation
• A few million monthly uniques from 10M registered users, but no sustained
traction

131. What Circle of Moms found
• They found moms were far more engaged
• Their messages to one another were on average 50% longer
• They were 115% more likely to attach a picture to a post they wrote
• They were 110% more likely to engage in a threaded (i.e. deep) conversation
• Circle owners’ friends were 50% more likely to engage with the circle
• They were 75% more likely to click on Facebook notifications
• They were 180% more likely to click on Facebook news feed items
• They were 60% more likely to accept invitations to the app
• Pivoted to the new market, including a name change
• By late 2009, 4.5M users and strong engagement
• Sold to Sugar, inc. in early 2012

132. Revenue
You’ll want to monetize things at this point. That doesn’t mean you haven’t already
been charging—for many businesses, even the first customer has to pay. It just
means that earlier on, you’re less focused on revenue than on growth. You’re
giving away free trials, free drinks, or free copies. Now you’re focused on
maximizing and optimizing revenue. This phase is closely tied to the Lean Startup’s
Price engine of growth.

133. Sergio Zyman’s many “mores”
• If you’re dependent on physical, per-transaction costs (like direct sales, or
shipping products to a buyer, or signing up merchants) then more efficiently
will figure prominently on either the supply or demand side of your business
model.
• If you’ve found a high viral coefficient, then more people makes sense, because
you’ve got a strong force multiplier added to every dollar you pour into customer
acquisition.
• If you’ve got a loyal, returning set of customers who buy from you every time,
then more often makes sense, and you’re going to emphasize getting them to
come back more frequently.
• If you’ve got a one-time, big-ticket transaction, then more money will help a lot,
because you’ve only got one chance to extract revenue from the customer and
need to leave as little money as possible on the table.
• If you’re a subscription model, and you’re fighting churn, then upselling
customers to higher-capacity packages with broader features to additional
subscribers within their organization is your best way of growing existing

134. What’s a customer worth?
BAD FOR YOU GOOD FOR YOU
Visitor User Customer
Spam/fraud
Support/resources
Ad impressions
Usage data
Content virality
Invite virality
Inherent virality
Voting/flagging
Victim/critical mass
Revenue

136. Market/product fit
Most people’s first instinct when things aren’t going incredibly well is to build more
features. Instead, try pivoting into a new market.
• Assume the product isn’t the problem, it’s the target customer.
• It may be easier to change markets than products.

137. Scale
With revenues coming in, it’s time to move from growing your business to growing
your market. You need to acquire more customers from new markets. You can
invest in channels and distribution to help grow your user base, since direct
interaction with individual customers is less critical—you’re close to product/
market fit and you’re analyzing things quantitatively. This phase is closely tied to
the acquisition cost optimization side of the Lean Startup’s Price engine of growth.

138. The hole in the middle
Differentiation Efficiency
Apple Costco
Here be
dragons
Your local
sustainable gluten-free
cupcake shop
Niche

139. Growth hacking
Finding your way up and to the right.
Human approaches, mathematical approaches

140. The leading indicator
• A Facebook user reaching 7 friends within 10 days of signing up (Chamath
Palihapitiya)
• If someone comes back to Zynga a day after signing up for a game, they’ll
probably become an engaged, paying user (Nabeel Hyatt)
• A Dropbox user who puts at least one file in one folder on one device (ChenLi
Wang)
• Twitter user following a certain number of people, and a certain percentage of
those people following the user back (Josh Elman)
• A LinkedIn user getting to X connections in Y days (Elliot Schmukler)
• These kinds of criteria also make great segments to analyze.
(from the 2012 Growth Hacking conference)

141. But wait: correlated or causal?
Correlation lets you Causality lets you
predict the future change the future
“I will have 420 “If I can make more
engaged users and first-time visitors stay
75 paying customers on for 17 minutes I
next month.” will increase sales in
90 days.”
Optimize the
Find correlation Test causality
causal factor

142. The growth hack
• Growth hacking is simply what marketing should have been doing, but it fell in
love with Don Draper and opinions along the way
• At its most basic: Optimize a factor you think is correlated with growth

143. An example from Reddit
Logged-in users All users
Days since
last visit
Visits Pageviews Pg/visit Visits Pageviews Pg/visit
0 127,797,781 1.925B 15.06 242,650,914 3.478B 14.33
1 5,816,594 87,339,766 15.02 13,021,131 187,992,129 14.44
2 1,997,585 27,970,618 14 4,958,931 69,268,831 13.97
Causality: When we make
Correlation: If you look at 13.88
3 955,029 13,257,404
changes that get someone
2,620,037 34,047,741 13
>15 pages today you’ll
to look at 20,644,331
15 pages 12.32
they’ll
come back tomorrow. 14.23
4 625,976 8,905,483 1,675,476
come back.
5 355,643 4,256,639 11.97 1,206,731 14,162,572 11.74

144. Pause #3
What stage are you at?
What is your gating metric?

145. Asymptotes
What to do when there’s no line in the sand

146. How to draw the line yourself
• Sample company trying to drive enrollment
• At first, out of over 1,200 visitors, only 4 (0.3%) sign up.
• By the end of the month, the site is converting 8.2% of its 1,462 visitors.
• Should the company keep optimizing?

147. How to draw the line yourself
• By plotting a trendline we see that in the current situation the line is around 9%.
• To really move the line will require a revolutionary, not evolutionary, change.

148. Finding your
One Metric That Matters

149. Your OMTM must fit
Your basic The stage your
business model startup is at
(monetization) (lifecycle)
• E-Commerce • Empathy
• UGC • Stickiness
• Media • Virality
• SaaS • Revenue
• Mobile App • Scale
• 2-sided market

150. What’s your OMTM?
E- 2-sided Mobile User-gen
SaaS Media
commerce market app content
Empathy Interviews; qualitative results; quantitative scoring; surveys
Loyalty, Inventory, Engagement, Downloads, Content, Traffic, visits,
Stickiness conversion listings churn churn, virality spam returns
CAC, shares, Inherent WoM, app Invites, Content
Virality reactivation
SEM, sharing
virality, CAC ratings, CAC sharing virality, SEM
(Money from transactions) (Money from active users) (Money from ad clicks)
Transaction, Transactions, Upselling, CLV, Ads, CPE, affiliate
Revenue CLV commission CAC, CLV ARPDAU donations %, eyeballs
Affiliates, Other API, magic Spinoffs, Analytics, Syndication,
Scale white-label verticals #, mktplace publishers user data licenses

151. Choose only one metric.

152. It will soon change.

153. In a startup,
focus is hard to achieve.

154. Having only one metric
addresses this problem.

155. Metrics are like
squeeze toys.
http://www.flickr.com/photos/connortarter/4791605202/

156. Pause #4

157. B2B: Selling to the enterprise

158. Lean Analytics lifecycle
for an enterprise-focused startup
Stage Do this Fear this
Consulting to test ideas and Lock-in, IP
Empathy bootstrap the business control, overfitting
Standardization and integration; Ability to
Stickiness shift from custom to generic integrate; support
Word of mouth, references, case Bad vibes;
Virality studies exclusivity
Growing direct sales, professional Pipeline, revenue
Revenue services, support recognition, comp
Channels, analysts, ecosystems, Crossing the
Scale APIs, vertically targeted products chasm; Gorillas

159. Intrapreneur

160. Intrapreneur example:
P&G changes the mop
instead of the soap
• Stage: Empathy
• Model: Retail/consumer packaged goods
• P&G is constantly looking for better soaps. But innovation was slowing.
Frustrated, they hired a design team to help them.

161. P&G changes the mop
instead of the soap
• Heavy internal investment in R&D, but limited results
• Brought in an outside agency (Continuum) to help
• The team watched people as they mopped, recording and iterating their
research approach
• Watched someone pick up spilled coffee. Rather than mopping, the person
swept up with a broom, then wiped with a cloth
• Realized the mop, not the liquid, mattered
• Studied the makeup of floor dirt; realized much of it is dust
• Swiffer is a $500M innovation in a stalled industry

162. The Lean Analytics lifecycle
for an Intrapreneur
Stage Do this Fear this
Get buy-in Political fallout
Beforehand
Find problems; don’t test demand. Entitled, aggrieved
Empathy Skip the business case, do analytics customers
Know your real minimum based on Hidden “must haves”,
Stickiness expectations, regulations feature creep
Build inherent virality in from the Luddites who don’t
Virality start; attention is the new currency understand sharing
Consider the ecosystem, channels, Channel conflict,
Revenue and established agreements resistance, contracts
Hand the baton to others gracefully Hating what happens
Scale to your baby

163. In the end:
Ask good questions
The OMTM should answer the most important question.

164. Once, a leader convinced others in
the absence of data.

165. Now, a leader knows what
questions to ask.

166. Ben Yoskovitz
byosko@gmail.com
@byosko
Alistair Croll
acroll@gmail.com
@acroll