How to use metrics in a startup that is yet before it's product market fit.

1. Metrics for
Early-Stage Startups
#scb13 – @andreasklinger

2. @andreasklinger
“Startup Founder”
“Product Guy”
#scb13 – @andreasklinger

3. @andreasklinger
“Startup Founder”
“Product Guy”
What we will cover
- Why early stage metrics are different.
- Applicable methods & Lessons Learned.
(this is an excerpt of 2h workshop - but with prettier slides ;) )
#scb13 – @andreasklinger

4. The Main Problem with Metrics in Early Stage:
- Product not ready or even wrong.
- Little to no useable data.
- Data points contradict each other.
- External Traffic can easily mess up our insights.
- What is actionable?
- Are we on the “right” track?
#scb13 – @andreasklinger

5. Startup Founders.
#scb13 – @andreasklinger

6. Some startups have
ideas for a new product.
Looking for customers
to buy (or at least use) it.
Customers don’t buy.
“early stage”
#scb13 – @andreasklinger

7. Product/Market Fit
traction
time
With early stage
I do not mean “X Years”
I mean before product/market fit.
#scb13 – @andreasklinger

8. Product/market fit
Being in a good market
with a product that can satisfy
that market.
~ Marc Andreessen
#scb13 – @andreasklinger

9. Product/market fit
Being in a good market
with a product that can satisfy
that market.
~ Marc Andreessen
= People want your stuff.
#scb13 – @andreasklinger

10. Product/Market Fit
traction
time
#scb13 – @andreasklinger

11. Product/Market Fit
traction
time
Discovery Validation Efficiency Scale
Steve Blank - Customer Development
#scb13 – @andreasklinger

12. Product/Market Fit
traction
time
Discovery Validation Efficiency Scale
Find a product
the market wants.
#scb13 – @andreasklinger

13. Product/Market Fit
traction
time
Discovery Validation Efficiency Scale
Find a product Optimise the product
the market wants. for the market.
#scb13 – @andreasklinger

14. Product/Market Fit
traction
time
Discovery Validation Efficiency Scale
Find a product Optimise the product
the market wants. for the market.
People in search Most clones
for new product start here.
start here. #scb13 – @andreasklinger

15. Product/Market Fit
traction
time
Discovery Validation Efficiency Scale
Product & Customer Scale Marketing
Development & Operations
#scb13 – @andreasklinger

16. Product/Market Fit
traction
time
Discovery Validation Efficiency Scale
Startups have phases
but they overlap.
#scb13 – @andreasklinger

17. Product/Market Fit
traction
time
Discovery Validation Efficiency Scale
83% of all startups are in here.
#scb13 – @andreasklinger

18. Product/Market Fit
traction
time
Discovery Validation Efficiency Scale
83% of all startups are in here. Most stuff we learn about
web analytics is meant for this part
#scb13 – @andreasklinger

19. Startups drown in
non actionable datapoints.

20. What does this mean for my product?
Are we on the right track?
Meant for channel (referral)
optimization.

21. Use of Metrics in Early Stage
#scb13 – @andreasklinger

22. Use of Metrics in Early Stage
Focus on People
- Not Hits, Pageviews, Visits, Events
Validation of customer feedback
- saying vs doing
- eg. did they really use the app?
- does the app do what they need it to?
Validation of internal opinions
- believing vs knowing
- eg. “Our users need/are/do/try…”
Doublecheck + Falsify
#scb13 – @andreasklinger

23. Segment Users into Cohorts
Cohorts = Groups of people that share attributes.
#scb13 – @andreasklinger

24. Segment Users into Cohorts
#scb13 – @andreasklinger

25. Apply a framework: AARRR
#scb13 – @andreasklinger

26. Acquisition
Visit / Signup / etc
Activation
Use of core feature
Retention
Come + use again
Referral
Invite + Signup
Revenue
$$$ Earned
(c) Dave McClure

27. Example: Photoapp
Cohorts based on registration week
WK acquisition activation retention referral revenue
twice a
Photoapp registration first photo share …
month
1 400 62,5% 25% 10%
2 875 65% 23% 9%
3 350 64% 26% 4%
… … … … …

28. Acquisition
Visit / Signup / etc
Activation
Use of core feature
Which Metrics to focus on?
Retention
Come + use again
Referral
Invite + Signup
Revenue
$$$ Earned
(c) Dave McClure

29. Acquisition
Visit / Signup / etc
Activation
Use of core feature
Short Answer:
Retention
Focus on Retention Come + use again
Referral
Invite + Signup
Revenue
$$$ Earned
(c) Dave McClure

30. Acquisition
Visit / Signup / etc
Activation
Use of core feature
Long answer - It depends on two things:
Retention
Come + use again
Phase of company Referral
Invite + Signup
Type of Product (esp. Engine of Growth)
Revenue
$$$ Earned
(c) Dave McClure

31. Long answer - It depends on two things:
Acquisition
Visit / Signup / etc
Activation
Use of core feature
Retention
Come + use again
Referral
Invite + Signup
Revenue
$$$ Earned
Source: Lean Analytics Book - highly recommend

32. Acquisition
Visit / Signup / etc
Activation
Use of core feature
Short Answer:
Retention
Focus on Retention Come + use again
Referral
Invite + Signup
Revenue
$$$ Earned
(c) Dave McClure

33. Because
Retention = f(user_happiness)

34. Because
Retention = f(user_happiness)
Crashpadder’s Happiness Index
e.g. Weighted sum over core activities by hosts.
Cohorts by cities and time.
= Health/Happiness Dashboard

35. AARRR misses something
Acquisition
And Happiness is not everything
Activation
Retention
Referral
Revenue
(c) Dave McClure

36. CUSTOMER INTENT
Acquisition
Activation
Retention
Referral
Revenue
FULFILMENT OF CUSTOMER INTENT
(c) Dave McClure

37. Metrics are horrible way to understand customer intent
(c) Dave McClure

38. Metrics are horrible way to understand customer intent
Customer Intent = His “Job to be done”
Products are bought because
they solve a “job to be done”.
Learn about Jobs to be done Framework
Watch: http://bit.ly/cc-jtbd
(c) Dave McClure

39. Startups are obsessed by their solution
And ignore the customers job/problem
Market
Job/
Problem
Our
Solution
#scb13 – @andreasklinger

40. Metrics are horrible way to understand customer intent
(c) Dave McClure

41. Metrics are horrible way to understand customer intent
Great Way: Customer Interviews
But: We bias our people,
when we ask them.
Even if we try not to.
Reason: we believe our
own bullshit.
Watch: www.hackertalks.io
(c) Dave McClure

42. Metrics are horrible way to understand customer intent
OK Way: Smoke Tests
If interviews suggest
a new feature but you are
Download Mobile Client
unsure about critical mass
(e.g. due to sample bias).
Create Smoke Tests
measure Click Conversion/
Signups
Not for verification but falsification
(c) Dave McClure

43. CUSTOMER INTENT (JOB)
Acquisition
Customer
Interviews
Activation
Retention
Referral
Customer
Interviews
& Metrics
Revenue
FULFILMENT OF CUSTOMER INTENT
(c) Dave McClure

44. Dig deeper - Good product centric KPIs:
Framework: AARRR
#scb13 – @andreasklinger

45. Dig deeper - Good product centric KPIs:
Linked to assumptions of your product (validation/falsify)
Rate or Ratio (0.X or %)
Framework: AARRR
Comparable (To your history (or a/b). Forget the market)
Explainable (If you don’t get it it means nothing)
#scb13 – @andreasklinger

46. “Industry Standards”
Framework: AARRR Use industry averages as reality check.
Not as benchmark.
- Usually very hard to get.
- Everyone defines stuff different.
- You might end up with another business
model anyway.
- Compare yourself vs your history data.
#scb13 – @andreasklinger

47. Example Mobile App: Pusher2000
Trainer2peer pressure sport app (prelaunch “beta”).
Rev channel: Trainers pay monthly fee.
Two sided => Segment AARRR for both sides (trainer/user)
Marketplace => Value = Transactions / Supplier
Social Software => DAU/MAU to see if activated users stay active
Chicken/Egg => You need a few very happy chickens for loads of eggs.
Week/Week retention to see if public launch makes sense
Framework: AARRR
Optimize retention: Interviews with Users that left
Measure Trainer Happiness Score
Activated User: More than two training sessions
Pushups / User / Week to see if the core assumption (People will do
more pushups) is valid
#scb13 – @andreasklinger

48. Dig Deeper - Dataschmutz
A layer of dirt obfuscating
your useable data.
Usually “wrong intent”.
Usually our fault.
(~ sample noise we created ourselves)
#scb13 – @andreasklinger

49. Dataschmutz
A layer of dirt obfuscating
your useable data.
e.g. Traffic Spikes of wrong
customer segment.
(have wrong intent)
#scb13 – @andreasklinger

50. Dataschmutz
Exam
MySugr
is praised as
“beautiful app”
example.…
=> Downloads
=> Problem:
Not all are diabetic
They focus on people
who activated.

51. How to minimize the impact of Dataschmutz
Base your KPIs on wavebreakers.
WK visitors acquisition activation retention referral revenue
twice a
Birchbox visit registration first photo share …
month
1 6000 66% / 4000 62,5% 25% 10%
2 25000 35% / 8750 65% 23% 9%
3 5000 70% / 3500 64% 26% 4%

52. Dataschmutz
Competitions create artificial incentive
Competition Created
“Dataschmutz”
Competitions (before P/M Fit) “Would you use my app and might
are nothing but Teflon Marketing win 1.000.000 USD?”
* Users had huge extra incentive.
People come.can hurt your numbers.
* Marketing People leave.
* While we decided on how to
relaunch we had dirty numbers.
#scb13 – @andreasklinger

53. Dig Deeper - Metrics need to hurt
#scb13 – @andreasklinger

54. Dig Deeper - Metrics need to hurt
If you are not ashamed about the KPIs in
your dashboard than something is wrong.
Either you do not drill deep enough.
Or you focus on the wrong KPIs.
#scb13 – @andreasklinger

55. Dig Deeper - Metrics need to hurt
Example: Garmz/LOOKK
Great Numbers:
90% activation (activation = vote)
But they only voted for friends
instead of actually using the platform.
We drilled (not far) deeper:
Activation = Vote for 2 different designers. Boom. Pain.
#scb13 – @andreasklinger

56. User activation.
Some users are happy (power users)
Some come never again.
What differs them? It’s their activities in their first 30 days.
How we think about Churn is wrong.
#scb13 – @andreasklinger

57. Example Twitter
How often did activated users
use twitter in the first month:
7 times
What did they do?
Follow 20 people, followed
back by 10
Churn:
If they don’t keep them 7 times
in the first 30 days.
They will lose them forever.
It doesn’t matter when a user
remembers to unsubscribe
#scb13 – @andreasklinger

58. Example Twitter
Example Twitter:
How did they get more people
to follow 30people within
7visits in the first 30 days?
Ran assumptions, created
features and ran experiments!
Watch: http://www.youtube.com/watch?v=L2snRPbhsF0
#scb13 – @andreasklinger

59. Checkout Intercom.io
Customer segmenting and messaging done right.
#scb13 – @andreasklinger

60. Summary
#scb13 – @andreasklinger

61. Summary
- Use Metrics for Product and Customer Development.
- Use Cohorts.
- Use AARRR.
- Figure Customer Intent through non-biasing interviews.
- Understand your type of product and it’s core drivers
- Find KPIs that mean something to your specific product.
- Avoid Telfonmarketing (eg Campaigns pre-product).
- Filter Dataschmutz
- Metrics need to hurt
- Focus on the first 30 days of customer activation.
TL;DR: Use metrics to validate/doublecheck.
Use those insights when designing for/speaking to your customers.
#scb13 – @andreasklinger

62. Read on
Startup metrics for Pirates by Dave McClure
http://www.slideshare.net/dmc500hats/startup-metrics-for-pirates-long-version
Actionable Metrics by Ash Mauyra
http://www.ashmaurya.com/2010/07/3-rules-to-actionable-metrics/
Data Science Secrets by DJ Patil - LeWeb London 2012
http://www.youtube.com/watch?v=L2snRPbhsF0
Twitter sign up process
http://www.lukew.com/ff/entry.asp?1128
Lean startup metrics - @stueccles
http://www.slideshare.net/stueccles/lean-startup-metrics
Cohorts in Google Analytics - @serenestudios
http://danhilltech.tumblr.com/post/12509218078/startups-hacking-a-cohort-analysis-with-google
Rob Fitzpatrick’s Collection of best Custdev Videos - @robfitz
http://www.hackertalks.io
Lean Analytics Book
http://leananalyticsbook.com/introducing-lean-analytics/
Actionable Metrics - @lfittl
http://www.slideshare.net/lfittl/actionable-metrics-lean-startup-meetup-berlin
App Engagement Matrix - Flurry
http://blog.flurry.com/bid/90743/App-Engagement-The-Matrix-Reloaded
My Blog
http://www.klinger.io
#scb13 – @andreasklinger

63. Thank you
@andreasklinger #SCB13
Slides: http://slideshare.net/andreasklinger
All pictures: http://flickr.com/commons
#scb13 – @andreasklinger