(Lean) Startup Metrics & Analytics
Nicola Junior Vitto - @njvitto @blomming September 26, 2012 - Better Software - Firenze, Italy

Entrepreneurship ...Entrepreneurs recognize they are
going to have to follow their own internal compass and embrace the uncertainty as part of the journey. - Steve Blank - more on: http://steveblank.com/2009/06/29/agile-opportunism-entrepreneurial-dna/

Get out of the building
(by Steve Blank) Your business assumptions can be wrong Go out and speak with your customers

Vanity Metrics

Vanity Metrics - Timelines There
will always be something up and to the right

Vanity Metrics - Google Analytics
Forget it!

Vanity Metrics - Google Analytics
...and this could become like a drug...killing you!

Vanity Metrics - TechCrunch &
Co “mDialog raises 5 Million from Blackberry partners” “GroupMe is now sending one Million texts every day” “Google Plus hits 10 Million users in 2 weeks”

Lean Metrics

What’s a startup? A startup
is first of all an experiment, it is a human institution designed to deliver a new product or service under conditions of extreme uncertainty At its heart, a startup is a catalyst that transforms ideas into product. (From Lessons Learned blog By Eric Ries)

What’s a Lean Startup? Lean
startup is a rigorous process for iterating from Plan A to a plan that works. (by Ash Maurya)

What Metrics Matter? Example: Facebook
on early days 150k users, little revenue Many superior competitors What would indicate their future success?

What Metrics Matter? Example: Facebook
on early days 75% of users visit one or more times per day Within one month of launching on a new campus, can acquire 90% of the students

What Metrics Matter? Good Metrics
(Three A’s) Are Actionable Can be Audited Are Accessible

Actionable Metric An actionable metric
is one that ties specific and repeatable actions to observed results (by Ash Maurya)

A/B Testing

A/B Testing - New Blomming
home VS Define your Goals

A/B Testing - Define your
goals A/B Testing Goals One single important Goal: % Users that sign-up for the service (as a seller) Secondary Goal: % Buyers that buy something after landing on the homepage

Multivariate Testing

Multivariate Testing at Google “Yes,
it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see whic one performs better. I has a recent debate over wheter a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that.” (by Doug Bowman on Design at Google)

Multivariate Testing at Google

Multivariate Testing at Google

Multivariate Testing at Google Google
Bing

Multivariate Testing at Google “That
blue was worth at least $80M” (by Paul Ray - Product Manager at Bing)

Funnels

David Cancel’s Funnel The Problem:
98% of people who visit your site, leave never to return

David Cancel’s Funnel 2% “The
average conversion rate in the United States of America wheter you’re selling elephants or iPods” (by Avinash Kaushik - Analytics Evangelist, Google)

David Cancel’s Funnel

David Cancel’s Funnel The Solution:
To find, engage and convert more of your website visitors into customers

Online Marketing
is shifting

Online Marketing Paradigm Shift by
David Cancel

Online Marketing Paradigm Shift by
David Cancel

Online Marketing Paradigm Shift by
David Cancel

Online Marketing Paradigm Shift Internet
Marketing 3.0 (different goal) by David Cancel

Lifecycle Marketing
by David Cancel

Optimize

Optimize for learning Optimize for
learning by creating a data-driven marketing framework

Optimize for learning Analytics &
A/B Testing are also not exactly the answer

Optimize for learning Use quantitative
& qualitative data to test *your* assumptions

Optimize for learning The point
of testing is to prove or disprove assumptions quickly

Optimize for learning Pay attention
to “overtesting” using Multivariate testing because it generally takes too long

The Local Maxima Problem

Frameworks

AARRR

AARRR Dave McClure’s Startup Metrics
for Pirates is a good framework for thinking about macro metrics

AARRR - Model

AARRR - Conversions Example

AARRR - Different orders (example)
1. Activation, Retention (happiness) 2. Revenue (sustainability) 3. Acquisition, Referral (virality) (this is the Ash Maurya or 37signals order: “Charge from day one”)

Acquisition

Tracking Marketing Channels Google Adwords
Banners Social Media Partnerships PR Blogging ...anything that links to your site!

Tracking Marketing Channels Use unique
urls (tracking parameters) on every url you create/give-out/pay for Use bit.ly Use HTTP referrers to group PR

Measuring Cost of Visits Cost
per click (site visits) Cost of campaign / Traffic generated (Social media) Cost of effort / Traffic generated

Measuring Cost of Acquisition Cost
of one visit / conversion rate = Cost of an Acquisition Acquisition could be paid for acquisition of free sign-up ...unfortunately is not so simple

Marketing Channels Per channel track:
Scale Cost Conversion Optimise and scale channels with best Cost of Acquisition (CoA) and potential to Scale

Acquisition Cost of Acquisition <
Customer Lifetime Profit Then scale acquisition at a similar or lower rate A/B test acquisition channels to optimise

Acquisition including Referral If it
costs €3 to acquire a customer through advertising But a customer brought in 2 other customers through referral The COA of that channel is €1

Activation

Activation
Sure, they signed-up!!!

Activation Sure, they signed-up!!! BUT
did the like it???

Activation at Facebook (i) which
data points predict whether a user will stay? (ii) if they stay, which data points predict how active they’ll be after three months?

Activation at Facebook (i) which
data points predict whether a user will stay? having more than one session as a new user and entering basic profile information. (ii) if they stay, which data points predict how active they’ll be after three months? how often a user was reached out by others, frequency of third party application use, and how available a user was on the site.

Activation at Facebook “At the
time, I heard many people criticize Facebook’s early investors, claiming that Facebook had ‘no business model’ and only modest revenues relative to the valuation offered by its investors...” “...Facebook was different, because it employed a different Engine of Growth. It paid nothing for customer acquisition, and its high engagement meant that it was accumulating massive amount of customer attention every day.” (Eric Ries - The Lean Startup)

Activation Metrics Activation metrics often
drive other metrics in Referral, Retention, Revenue

Retention

Returning Users Tracking unregistered users
that return often (useful for content sites with advertising) Tracking registered users that return and take some sort of action.

Cohort Analysis

Twitter Cohorts (example) by Fred
Wilson’s blog (more on: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html)

Twitter Cohorts (example) by Fred
Wilson’s blog (more on: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html)

Referral

Referral Mechanism Social Media Sharing
Invite a friend type mechanism App Reviews Word of Mouth

Viral Coefficient The average number
of customers that each customer refers

Viral Coefficient - Example Each
company customer invites 5 other customers via email 20% click on that email link 50% of those users convert into customers

Viral Coefficient - Example Viral
Coefficient = Referrals x click through x conversion 5 x 0.2 x 0.5 = 0.5

Viral Coefficient A viral co-efficient
greater than 1 means that every customer gets more than one other customer on average

Viral Coefficient A viral co-efficient
greater than 1 means that every customer gets more than one other customer on average Therefore your product will grow virally

Viral Coefficient Measure automatically for
all your built in referral mechanism Give customers unique invite URLs and track conversions

Viral Cycle Time Viral Cycle
Time = the average time taken for a referral to turn into a customer With the viral coefficient and viral cycle time you can determine your viral growth

Net Promoter Score Net Promoter
Score: Number of people who would recommend your product to a friend A single question survey

Net Promoter Score “How likely
is it you would recommend my product to a friend or colleague?” 0 1 2 3 4 5 6 7 8 9 10 NPS = % Promoters - % Detractors

Net Promoter Score Survey a
small percentage of your customers by email weekly Record who you sent to so you don’t repeat the same people every week

Net Promoter Score NPS is
not a direct measurement that translates to key metrics Can act as a leading indicator of other metrics

Revenue

Making Money Key for any
business is to find out how much profit they make for every customer and scaling the number of customers

Lifetime Customer Value How much
money you make for every customer you acquire For some this a single transaction For subscription businesses it depends on retention

Cancellation Rate Cancellation rate =
is the percentage number of customers who cancel in any given month compared to total (paying) customers [Cancellation rate] = [product utility] + [service quality] + [acceptable price]

Lifetime Customer Value (LTV) LTV
= monthly revenue x n° months in lifetime N° months = 1 / cancellation rate LTV = monthly revenue / cancellation rate

Lifetime Customer Value (LTV) For
example: If €20 a month and cancellation rate is 10% a month LTV = €200

Lifetime Customer Value (LTV) Often
not that simple. Short term cancellation rates are much higher than long term Use Cohort Analysis to determine

Lifetime Customer Value (LTV) r
= short-term cancellation rate (e.g. 0,15) p = long-term cancellation rate (e.g. 0.03) s = number of months in the “short-term” age group (e.g. 3) (1-r)^s x (s + 1/p) = expected months

Some Tools

Event based tracking

Event based tracking

Event based tracking

Customer lifecycle
management

Customer lifecycle management

Customer lifecycle management

Tips

OODA Loop

Innovation Accounting “To improve entrepreneurial
outcomes, and to hold entrepreneurs accountable, we need to focus on the boring stuff: how to measure progress, how to setup milestones, how to prioritize work. This requires a new kind of accounting, specific to startups” (Eric Ries - The Lean Startup)

Innovation Accounting Concentrate on macro
metrics Everything you do should attempt to change a metric Try to use as few tools as possible

One Page Dashboard

Customer Development: pivoting

(Lean) Product Development Problem: unknown
Solution: unknown

Scalable (lean) startup

SCRUM
Do you know SCRUM?

Kanban Board I know: this
isn’t an example of an exciting and well done Sprint :)

(Lean) Product Management Well, trying
to connect the dots: what we learnt so far is how the Product Owner can define and prioritize stories at every Sprint in a Startup that wants to be called, you know, Lean! :)

Learning Boards - Lean Stack
We will talk about the Lean Stack of Ash Maurya or other possible “lean mapping” frameworks in a future presentation

Focus focus focus

Credits Stuart Eccles - Lean
Startup Metrics Joshua Porter - Metrics-Driven Design David Cancel - Data-Driven Marketing Lean Startup Machine - How to measure the metrics that determine real progress (and the others specified directly in the slides when picking some parts of books or blogs)

The Lean Startup

Running Lean

Business Models

An alternative?
Lifestyle business

Thanks! Blomming.com Company.Blomming.com Magazine.Blomming.com Help.Blomming.com
Twitter.com/Blomming Facebook.com/Blomming nicola@blomming.com On FB & Twitter as: njvitto