Lies, Damned Lies &
Vanity Metrics
Bernard Leong
A Simple Guide to Feedback, Analytics &
Iteration in Product Development
1
“There are three kinds of lies:
lies, damned lies & statistics.”
- Mark Twain
2
A good theoretical physicist is one who makes
all the mistakes as quickly as possible.
entrepreneur
3
4
Feedback Generation
• How do we collect and generate user feedback?
• Qualitative: Surveys, Talk to customers, Create
chan...
• Qualitative - Break it into the following buckets:
• Spam/Troll & Insults - Ignore
• “Blue Sky” or “N=1” comments in the...
Avoid Vanity Metrics
• Number of hits.
• Number of page views.
• Number of visits.
• Number of unique visitors.
• Number o...
Framework for User Growth
8
Element Function Relevant Metrics
Acquisition
Generate attention through
various means
Traffic, Mentions, Cost per Clicks,
S...
How to identify key metrics
Acquisition
Activation
Retention
Revenue
Referral
How do users discover you?
Do users have a f...
Constructing a Funnel
Internet Marketing - Google Ads
200K impressions = 20K click thrus
(10%)
User Signups
20K click thru...
Source: Lean Analytics
Usability Test
12
Figuring out the customer lifecycle
and working out the metrics which
you want to measure.
Case Study: SaaS
services
13
References
• Alistair Croll & Benjamin Yoskovitz, “Lean
Analytics”
• Jon Yongfook, “21 Actionable Growth Hacking
Tactics”,...
Upcoming SlideShare
Loading in...5
×

Jfdi 2013-product-development-metrics-130509031212-phpapp01

273

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
273
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Jfdi 2013-product-development-metrics-130509031212-phpapp01

  1. 1. Lies, Damned Lies & Vanity Metrics Bernard Leong A Simple Guide to Feedback, Analytics & Iteration in Product Development 1
  2. 2. “There are three kinds of lies: lies, damned lies & statistics.” - Mark Twain 2
  3. 3. A good theoretical physicist is one who makes all the mistakes as quickly as possible. entrepreneur 3
  4. 4. 4
  5. 5. Feedback Generation • How do we collect and generate user feedback? • Qualitative: Surveys, Talk to customers, Create channels via social media (twitter/facebook). • Quantitative: Google Analytics, Mixpanel metrics. • User interface & experience testing: User education and perception of how product works. 5
  6. 6. • Qualitative - Break it into the following buckets: • Spam/Troll & Insults - Ignore • “Blue Sky” or “N=1” comments in the form of “you should build this or that” - Think about it but don’t act • “N>50” comments on a feature or users demanding for a feature - Ascertain whether people really want it and then act. Feedback Maintenance Obviously, you need to find metrics to do that ... 6
  7. 7. Avoid Vanity Metrics • Number of hits. • Number of page views. • Number of visits. • Number of unique visitors. • Number of followers/friends/likes • Time on site/number of pages. • Emails collected • Number of downloads 7
  8. 8. Framework for User Growth 8
  9. 9. Element Function Relevant Metrics Acquisition Generate attention through various means Traffic, Mentions, Cost per Clicks, Search Results, Cost of acquisition, Open Rate Activation Turn the resulting drive-by visitors into users who are enrolled Enrollments, signups, completed onboarding process, used the service at least once, subscriptions Retention Convince users to return repeatedly or exhibit stickiness behavior Engagement, time since last visit, daily & monthly active use, churns. Revenue Business outcomes depending on business model Customer lifetime value, conversion rate, shopping cart size & click thru revenue Referral Viral or word of mouth invitations to other users Invites sent, viral coefficient, viral cycle time. 9
  10. 10. How to identify key metrics Acquisition Activation Retention Revenue Referral How do users discover you? Do users have a frictionless or seamless first experience? Do users return? Do users want to pay you money? Do users spread the word? 10
  11. 11. Constructing a Funnel Internet Marketing - Google Ads 200K impressions = 20K click thrus (10%) User Signups 20K click thrus = 2K signups (10%) User create activity 2K signups = 200 users return with 5 actions (10%) User pay for premium service = 20 users Users send invites to other users = 2 users How do users discover you? Do users have a frictionless or seamless first experience? Do users return? Do users want to pay you money? Do users spread the word? 11
  12. 12. Source: Lean Analytics Usability Test 12
  13. 13. Figuring out the customer lifecycle and working out the metrics which you want to measure. Case Study: SaaS services 13
  14. 14. References • Alistair Croll & Benjamin Yoskovitz, “Lean Analytics” • Jon Yongfook, “21 Actionable Growth Hacking Tactics”, http://yongfook.com/actionable-growth- hacking-tactics.html • Ash Maurya: 3 Rules to actionable metrics in a lean startup: http://www.ashmaurya.com/ 2010/07/3-rules-to-actionable-metrics/ 14
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×