Lean Product Analytics by Dan Olsen

  • 3,135 views
Uploaded on

Talk on how to use analytics to optimize your product that I gave at Yelp on February 5, 2014.

Talk on how to use analytics to optimize your product that I gave at Yelp on February 5, 2014.

More in: Business , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,135
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
201
Comments
0
Likes
17

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Lean Product Analytics Dan Olsen Olsen Solutions Feb 5, 2014
  • 2. My  Background   n  Educa7on   n  n  n  n  n  20  years  of  Product  Management  Experience   n  n  n  n  n    BS,  Electrical  Engineering,  Northwestern   MS,  Industrial  Engineering,  Virginia  Tech   MBA,  Stanford   Web  development  and  UI  design   Managed  submarine  design  for  5  years   5  years  at  Intuit,  led  Quicken  Product  Management   Led  Product  Management  at  Friendster   CEO  &  Cofounder  of  YourVersion,  “Pandora  for  your  news”   Consultant:  Box,  YouSendIt,  Chartboost,  One  Medical   Will  post  slides  at  hUp://slideshare.net/dan_o     Copyright  ©  2014  Olsen  Solu7ons  
  • 3. What  does  “Lean”  mean?   n  Lean  Startup   n  Achieving  product-­‐market  fit   n  Tes7ng  hypotheses  &  learning   n  Valida7ng  MVP  with  users   n  Improving  &  itera7ng  your   product  quickly   n  Minimizing  waste  =   using  resources  effec7vely   Copyright  ©  2014  Olsen  Solu7ons  
  • 4. What’s  the  Formula  for   Product-­‐Market  Fit?   n  A  product  that:   n  Meets  customers’  needs   n  Is  beUer  than  other  alterna7ves   n  Is  easy  to  use   n  Has  a  good  value/price   Copyright  ©  2014  Olsen  Solu7ons  
  • 5. Dan’s  Model  for  the  Causality   Underlying  Product-­‐Market  Fit   Customer   has  needs   Target   Customer   Customer   Needs   You  design  &   build  product   to  meet  needs   Customer   decides  how   well  product   meets  needs   (sa7sfac7on)   Product   Copyright  ©  2014  Olsen  Solu7ons  
  • 6. What  are  Customers  Reac7ng  To   When  They  Use  Your  Product?   Feature  Set   UX  Design   Messaging     Copyright  ©  2014  Olsen  Solu7ons  
  • 7. Valida7ng  New  vs.  Exis7ng  Products   New  Product   Qualita7ve  interviews   Oprah   Exis0ng  Product   Quan7ta7ve  data   Spock  
  • 8. +*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+ How to be a Lean Product Ninja slideshare.net/dan_o/ +*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+
  • 9. Copyright  ©  2014  Olsen  Solu7ons  
  • 10. Lean  Product  Analy7cs  Process   Iden7fy  What   Your    Metrics  Are   Iden7fy  highest   ROI  idea   Measure  Metrics   Baseline  Values   Evaluate  Metrics   Upside  Poten7al   Global   Level   Select   Top  Metric   Brainstorm   Ideas  to   Improve  Metric   Metric   Level   Learn   &  Iterate   Design  and   Implement   Analyze  How   the  Metric   Changes   Copyright  ©  2014  Olsen  Solu7ons  
  • 11. Valida7ng  Product-­‐Market  Fit:  Surveys   n  Net  Promoter  Score   Key  follow-­‐up  ques7ons:   •  Why  did  you  give  the  score  you  did?   •  What  do  we  need  to  do  to  improve?   Copyright  ©  2014  Olsen  Solu7ons  
  • 12. Qualita7ve  Compliments  Quan7ta7ve   Quant What? Qual Why? Copyright  ©  2014  Olsen  Solu7ons  
  • 13. Valida7ng  Product-­‐Market  Fit:  Surveys   n  Survey.io  /  Qualaroo.com   n  “How  would  you  feel  if  you  could  no  longer  use  Product  X?”   n  Very  disappointed   n  Somewhat  disappointed   n  Not  disappointed   n  General  guideline:    40%  or  more  “very  disappointed”  =   product-­‐market  fit   Copyright  ©  2014  Olsen  Solu7ons  
  • 14. Product-­‐Market  Fit:   Actual  User  Behavior  Trumps  Opinions   n  Asking  a  user  ques7ons  in  an  interview  or  survey   n  n  n  n  Observing  behavior   n  n  n  Valuable,  but…   They’re  telling  you  what  they  think  they  would  do   Measurement  bias  (because  you’re  with  them)   See  what  users  actually  do   Without  you  there   Behavioral  metrics  for  Product-­‐Market  Fit:   n  n  n  n  n  Prospects  sign  up  =  High  conversion  rate   They  keep  using  it  =  High  reten7on  rate   They  use  it  omen  =  High  frequency  of  use   They’re  deeply  engaged  with  it  =  Long  session  7mes   They  pay  for  it  =  Revenue  per  customer   Copyright  ©  2014  Olsen  Solu7ons  
  • 15. Valuable  to  Have  a  Holis7c   Analy7cs  Framework   Dave  McClure’s  “Startup  Metrics  for  Pirates”   A A R R R Focus  on  right  metric  at  right  7me  
  • 16. Using  Analy7cs  for  Op7miza7on   n  In  addi7on  to  Product-­‐Market  Fit,  you   can  apply  the  Lean  Product  Analy7cs   Process  to  op7mize:   n  Your  Business  Results   n  Your  User  Experience   Copyright  ©  2014  Olsen  Solu7ons  
  • 17. Define  the  Equa7on  of  your  Business   “Peeling  the  Onion”   Adver7sing  Business  Model:     Profit  =  Revenue  -­‐  Cost         nique  Visitors    x    Ad  Revenue  per  Visitor   U            mpressions/Visitor    x    Effec7ve  CPM  /  1000   I             isits/Visitor    x    Pageviews/Visit    x    Impressions/PV   V       ew  Visitors  +  Returning  Visitors   N      nvited  Visitors  +  Uninvited  Visitors   I        of  Users  Sending  Invites    x    Invites  Sent/User    x    Invite  Conversion  Rate   # Copyright  ©  2014  Olsen  Solu7ons  
  • 18. Equa7on  of  your  Business:   Subscrip7on  Business  Model   Profit  =  Revenue  -­‐  Cost              Paying  Users    x    Revenue  per  Paying  User                 N       ew  Paying  Users    +    Repeat  Paying  Users                           rial  Users    x    C  onv  Rate    Previous  Paying  Users    x    (  1  –  Cancella7on  Rate  )   T                          (  SEO  Visitors  +  SEM  Visitors  +  Viral  Visitors  )    x    Trial  Conversion  Rate         Copyright  ©  2014  Olsen  Solu7ons  
  • 19. How  to  Track  Your  Metrics   n  Track  each  metric  as  daily  7me  series     Date   Unique   Visitors   Page   views   Ad   New  User   Revenue   Sign-­‐ups   4/24/08   10,100   29,600   25   490   4/25/08   10,500   27,100   24   …   480   …   n  Create  ra7os  from  primary  metrics:    X  /  Y   n  Example:  How  good  is  your  registra7on  page?   n  Okay: n  BeUer:    #  of  registered  users  per  day    registra7on  conversion  rate  =    #  registered  users  /  #  uniques  to  reg  page   Copyright  ©  2014  Olsen  Solu7ons  
  • 20. Registra7on  Page  Conversion  Rate   Daily Signup Page Yield Rate vs. Registration Page Conversionvs. Time Time New Registered Users divided by Unique Visitors to Signup Page Daily Signup Page Yield Registration Page Conversion Rate 100% 90% 80% 70% 60% 50% 40% Started requiring registration 30% 20% 10% Changed messaging Added questions to signup page 0% 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1 0 Copyright  ©  2014  Olsen  Solu7ons  
  • 21. View  Each  Business  Metric  as  a  Gauge   Current   Value   Minimum   Possible   Value   Maximum   Possible   Value   Copyright  ©  2014  Olsen  Solu7ons  
  • 22. Return  (Value  Created)   Priori7zing  Product  Ideas  by  ROI   4   ? Idea  D   3   Idea  A   Idea  B   2   Idea  C    1   Idea  F    1   2    3   4   Investment  (developer-­‐weeks)   Copyright  ©  2014  Olsen  Solu7ons  
  • 23. Iden7fying  the  “Cri7cal  Few”  Metrics   n  What  is  the  upside  poten7al  of  each  metric?   How  many  resources  will  it  take  to  “move  the  needle”?   n  n  n  Developer-­‐days,  7me,  money   How  much  will  the  needle  move?  Revenue  impact?   Which  metrics  have  highest  ROI  opportuni7es?   Metric  B   Bad  ROI   Return   Return   Metric  A   Good  ROI   Investment   Metric  C   Great  ROI   Return   n  Investment   Investment   Copyright  ©  2014  Olsen  Solu7ons  
  • 24. Case  Study  from  Intuit   q  q  Improving  UX   Improving  Business  Results   -­‐>  Sign-­‐up  Conversion  Rate   Copyright  ©  2014  Olsen  Solu7ons  
  • 25. Case  Study:  Account  Signup  Process  Redesign   Abandonment Rate (7 Day Moving Average) Steps 1-2 80% 60% 50% 40% 30% 20% 10% 1/20/03 1/13/03 1/6/03 12/30/02 12/23/02 12/16/02 12/9/02 12/2/02 11/25/02 11/18/02 11/11/02 11/4/02 10/28/02 10/21/02 10/14/02 0% 10/7/02 Abandonment Rate (7 Day Moving Average) 70% Copyright  ©  2014  Olsen  Solu7ons  
  • 26. Analyzed  Drop-­‐Off  at  Each  Major  Sec7on   %  of  Users   100% 80% 100% Focus  on   biggest   drop   62.3% 60% 58.8% 50.9% 40% 34.4% 32.7% 20% 0% Sign  in  /   Account  Type   Cash  vs.   Registra7on   Margin   5  Partner   Pages   3  Partner   Pages   Copyright  ©  2014  Olsen  Solu7ons  
  • 27. Analysis  of  Sign  In/Registra7on  Flow   Open   Account   44%   Register     Registra7on   Process   55%   (24%  of  Total)   45%  drop  off   (20%  of  total)   56%   83%   (46%  of  Total)   Sign  in     Forget   Password   17%  drop  off   (10%  of  total)   64%   of  Total   Account   36%  overall   Selec7on   drop  off  for   this  step   30%   (14%  of  Total)   70%   (32%  of  Total)   Change   Password   80%   (26%  of  Total)   20%  drop  off   (6%  of  total)   Copyright  ©  2014  Olsen  Solu7ons  
  • 28. Redesigned  User  Flow  Improved   Registra7on  Conversion  Rate   Abandonment Rate (7 Day Moving Average) Steps 1-2 80% 60% 50% 40% 37% improvement in conversion rate Released New Design 30% 20% 10% 1/20/03 1/13/03 1/6/03 12/30/02 12/23/02 12/16/02 12/9/02 12/2/02 11/25/02 11/18/02 11/11/02 11/4/02 10/28/02 10/21/02 10/14/02 0% 10/7/02 Abandonment Rate (7 Day Moving Average) 70% Copyright  ©  2014  Olsen  Solu7ons  
  • 29. Case  Study  from  Friendster   q  Improving  Business  Results   -­‐>  Viral  New  User  Growth   Copyright  ©  2014  Olsen  Solu7ons  
  • 30. Case  Study:   Op7mizing  Friendster’s  Viral  Loop   % of users sending = 15% invites Active Users % of users who are active Invites per sender = 2.3 Invite Prospective Users Invite click-through rate Click Registration Process Fail Succeed Don’t Click Conversion = 85% rate Users •   Mul7plied  together,  these  metrics  determine  your  viral  ra7o   •   Which  metric  has  highest  ROI  opportunity?   Copyright  ©  2014  Olsen  Solu7ons  
  • 31. The  Upside  Poten7al  of  a  Metric   ?   100%   100%   85%   15%   0   Registra7on   Process  Yield   Max  possible   improvement   0.15  /  0.85  =  18%   2.3   0   %  of  users  sending   invita7ons   0.85  /  0.15  =  570%   0   Avg  #  of  invites   sent  per  sender   ?  /  2.3  =  ?%   Copyright  ©  2014  Olsen  Solu7ons  
  • 32. Okay,  so  how  can  we  improve  the  metric?   How  do  we  increase  the  average  number  of   invites  being  sent  out  per  sender?   n  For  each  idea:   n  n  What’s  the  expected  benefit?  (how  much  will  it   improve  the  metric?)   n  What’s  the  expected  cost?  (how  many  engineer-­‐ days  will  it  take?)   n  You  want  to  iden7fy  highest  ROI  idea   Copyright  ©  2014  Olsen  Solu7ons  
  • 33. Amer  Launching  Address  Book  Importer…   Copyright  ©  2014  Olsen  Solu7ons  
  • 34. Amer  Launching  Address  Book  Importer…   Copyright  ©  2014  Olsen  Solu7ons  
  • 35. Amer  Launching  Address  Book  Importer…   Copyright  ©  2014  Olsen  Solu7ons  
  • 36. If  you  could  only  track  1  metric  to   measure  your  Product-­‐Market  Fit:   Which  metric  would  it  be?   Copyright  ©  2014  Olsen  Solu7ons  
  • 37. Reten7on  Rate   n  Reten7on  rate  tracks  what  %  of  your   customers  are  s7ll  ac7ve  over  7me   ~80% never use app again Curve eventually flattens out Copyright  ©  2014  Olsen  Solu7ons  
  • 38. Cohort  Analysis   Copyright  ©  2014  Olsen  Solu7ons  
  • 39. Cohort  Analysis:  Data   Copyright  ©  2014  Olsen  Solu7ons  
  • 40. Improving  Reten7on  Rate  Over  Time=   Increasing  Product-­‐Market  Fit   David  Skok,  Matrix  Partners   hUp://www.forentrepreneurs.com/saas-­‐metrics-­‐2/  
  • 41. Alternate  Ways  to  Track  Reten7on   n  Having  lots  of  cohort  curves  is  hard  to  read   n  Would  be  great  to  have  a  7me  series  metric   =  one  metric  we  can  track  over  7me   n  %  Users  Retained  who  signed  up  X  days  ago   n  Can  use  single  or  mul7ple  X  (30  &  90  days)   n  Another  metric:  Returning  users   n  Good  summary  metric:  #  of  users  “locking  in”   n  Gives  a  sense  of  scale  (not  a  %)   n  Recommend  7-­‐day  average  (can  do  others  too)   Copyright  ©  2014  Olsen  Solu7ons  
  • 42. Profitability,  anyone?   Two  key  metrics:   •  Customer  Life7me  Value  (LTV)   •  Customer  Acquisi7on  Cost  (CAC)     You  want:   LTV  –  CAC  >  0  
  • 43. Profitability,  anyone?  
  • 44. Profitability,  anyone?   Two  key  metrics:   •  Customer  Life7me  Value  (LTV)   •  Customer  Acquisi7on  Cost  (CAC)     You  want:   LTV  –  CAC  >  0  
  • 45. Life7me  Value  (LTV)   n  Life7me  value  of  a  customer  =  how  much  value   your  average  customer  will  generate   n  LTV  =  ARPU  x  Avg  Customer  Life7me  x  Gross  Margin   ARPU  (Avg  Revenue  /  User)  =  Total  Revenue  /  #  of  Users   n  Average  Customer  Life7me   n  n  How  long  your  average  customer  generates  revenue   n  Equals  1  /  churn  rate    (5%  monthly  churn  =  avg  life  20  months)   n  Gross  Margin:  the  %  of  revenues  lem  over  amer   subtrac7ng  the  cost  of  providing  the  product/service   Note:  for  simplicity,  this  LTV  equa7on  ignores  the  “cost  of  capital”   Copyright  ©  2014  Olsen  Solu7ons  
  • 46. Customer  Acquisi7on  Cost  (CAC)   n  CAC  is  the  average  cost  for  you  to  obtain  a   revenue-­‐genera7ng  customer   n  So  it  takes  into  account  both  your  cost  of   acquiring  a  prospect  and  your  conversion   rate  for  conver7ng  prospects  to  revenue-­‐ genera7ng  customers   n  CAC=Cost  per  Acquisi7on  /  Conversion  Rate   Copyright  ©  2014  Olsen  Solu7ons  
  • 47. What  You’d  Like  to  See  Over  Time   n  n  LTV  increasing  as  you  improve  your  value   proposi7on,  customer  reten7on,  &  pricing   CAC  decreasing  as  you  op7mize  your  marke7ng:   segments,  channels,  messaging   Copyright  ©  2014  Olsen  Solu7ons  
  • 48. Ra7o  of  LTV  to  CAC:   Real  data  from  HubSpot   Copyright  ©  2014  Olsen  Solu7ons  
  • 49. Lean  Product  Analy7cs  Process   Iden7fy  What   Your    Metrics  Are   Iden7fy  highest   ROI  idea   Measure  Metrics   Baseline  Values   Evaluate  Metrics   Upside  Poten7al   Global   Level   Select   Top  Metric   Brainstorm   Ideas  to   Improve  Metric   Metric   Level   Learn   &  Iterate   Design  and   Implement   Analyze  How   the  Metric   Changes   Copyright  ©  2014  Olsen  Solu7ons  
  • 50. Questions? olsensolutions.com linkedin.com/in/danolsen98 @danolsen Copyright  ©  2014  Olsen  Solu7ons