SkillShare: Think Like a PM

754 views

Published on

My SkillShare deck for the class Think Like a PM: Using data to drive feature decisions.

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
754
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

SkillShare: Think Like a PM

  1. 1. Your  data  is  telling  you  something.     $1B   Who  cares?   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  2. 2. Listen  to  it.   $1B   Kevin  Systrom  realizes  customers  only  use   their  product  for  one  thing:  photos.  Burbn   dies,  Instagram  is  born.   Who  cares?   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  3. 3. Agenda   •  Background   •  IdenEfying  opportuniEes   •  Using  metrics  to  prioriEze   •  TesEng  hypothesis  with  experiments   •  Running  “post-­‐mortem”  analysis   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  4. 4. Goals  of  “Think  Like  a  PM”   •  Introduce  the  idea  of  data  driven  PM’ing   –  Focus  on  an  example  using  user  data   •  Review  the  “end-­‐to-­‐end”  process  of  a  data   driven  feature   –  Use  Foursquare  as  an  illustraEve  example   •  Provide  you  with  another  tool  for  approaching   product  development     SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  5. 5. How  do  PMs  decide  what  features  to  build?   •  Data   •  Talking  to  customers   •  Vision  about  the  future  of  the  product   •  Beliefs     •  Wild-­‐ass  guesses   •  Looking  at  the  compeEEon   •  DirecEon  from  managers  /  execs   •  They  don’t  (indecision  strikes!)   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  6. 6. What  types  of  data  do  PMs  focus  on?   •  Market  data   –  “CompeEtors  that  have  focused  on  Z  approach  have  out-­‐ performed  and  we  should  consider  that…”   •  Anecdotal  data     –  Eg,  “When  we  talk  to  customers,  they  always  complain   about  Y  taking  too  long…”   •  User  data   –  We  know  25%  of  users  take  X  acEon  in  the  game…”   –  Some  famous  examples:  Instagram’s  pivot,  Facebook’s   localizaEon  efforts,  Zynga’s  dominance  of  FB  channels   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  7. 7. Agenda   •  Background   •  Iden4fying  opportuni4es   •  Using  metrics  to  prioriEze   •  TesEng  hypothesis  with  experiments   •  Running  “post-­‐mortem”  analysis   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  8. 8. Why  study  Foursquare?   •  Everyone  can  use  it  (it’s  free)   •  People  are  familiar  with  it  (25M  users)   •  It’s  an  evolving  product  –  you  can  observe  the   Foursquare  team  making  changes  to  the  product   •  Clear  defined  user  flows  &  acEons  to  study   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  9. 9. Foursquare  top  level  metrics:  the  “Vanity”   metrics   •  2B  “check-­‐ins”   •  25M  registered  users   •  7.2M+  daily  acEve  users  (DAU)   •  20%  of  searches  result  in  a  check-­‐in   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  NOTE:  Stats  from  TechCrunch,  “Foursquare  looks  into  a  4th  round”,  Nov.  2,  2012  
  10. 10. Two  things  to  remember  when  working  with   data   What  is  this?   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  NOTE:  Stats  from  TechCrunch,  “Foursquare  looks  into  a  4th  round”,  Nov.  2,  2012  
  11. 11. Start  with  the  full  picture,  peel  back  layers  of  the  onion   Zoom  out  so  you  can     And  then  you  can  work  on     see  the  whole  picture…   peeling  back  the  layers…   It’s  a  bridge!   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  12. 12. Key  steps  to  idenEfying  opportuniEes   1)  Define  a  clear,  measurable  goal   –  Eg,  “We  want  to  increase  Foursquare  check-­‐ins  /  day”   2)  Define  the  relevant  data  set   –  Eg  “What  drives  daily  check-­‐ins?”   3)  Determine  the  status  quo   –  Eg,  “What  does  the  current  data  show  about  daily  check-­‐ ins?”   4)  IdenEfy  opportuniEes  to  improve  the  goal   –  Eg,  “What  are  the  inflecEon  points?”   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  13. 13. Defining  a  clear,  measurable  goal   Foursquare  derives  value  from   loca4on  data   •  Check-­‐ins  are  a  criEcal  piece   (eg  build  the  database  of   locaEon  data)   •  They  have  viral  value  (eg   “Kenton  checked  in  here…)   •  Check-­‐in  rates  indicate  the   health  of  the  app  /  user   base  (eg,  Check-­‐ins  /  day  is  a   good  indicator  of  user   acEvity)   •  Result:  Check-­‐ins  could  be  a   great  piece  of  data  to   understand  beCer   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  14. 14. Key  steps  to  idenEfying  opportuniEes   1)  Define  a  clear,  measurable  goal   –  Eg,  “We  want  to  increase  Foursquare  check-­‐ins  /  day”   2)  Define  the  relevant  data  set   –  Eg  “What  drives  daily  check-­‐ins?”   3)  Determine  the  status  quo   –  Eg,  “What  does  the  current  data  show  about  daily  check-­‐ ins?”   4)  IdenEfy  opportuniEes  to  improve  the  goal   –  Eg,  “What  are  the  inflecEon  points?”   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  15. 15. What’s  the  anatomy  of  a  check-­‐in?  iPhone  home  screen   Foursquare  home   Loca4on  picker   Check  in  details  •  How  many   •  How  many  users   •  How  many  users   •  How  many  users   users?   reach  it  daily?   reach  it  daily?   reach  it  daily?  •  How  many   •  How  many   •  How  many   •  How  many  share  on   decide  to  login   decide  to  click  to   decide  to  select   social  media?  On   on  any  given   iniEate  a  check   an  actual   twiper?  On   day?   in?   locaEon?   facebook?   •  How  many  include  a   photo?   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  16. 16. Key  steps  to  idenEfying  opportuniEes   1)  Define  a  clear,  measurable  goal   –  Eg,  “We  want  to  increase  Foursquare  check-­‐ins  /  day”   2)  Define  the  relevant  data  set   –  Eg  “What  drives  daily  check-­‐ins?”   3)  Determine  the  status  quo   –  Eg,  “What  does  the  current  data  show  about  daily  check-­‐ ins?”   4)  IdenEfy  opportuniEes  to  improve  the  goal   –  Eg,  “What  are  the  inflecEon  points?”   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  17. 17. Foursquare  data:  the  top  level  funnel  of  user   acEvity   Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous   Total  registered  users   25,000,000   Daily  acEve  users   7,200,000   28.8%   Click  “Check-­‐in”   1,800,000   25%   Select  locaEon   900,000   50%   Complete  check-­‐in   630,000   70%   Social  Media  sharing   189,000   30%   Share  photo   126,000   20%   No  meta  data   315,000   50%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  NOTE:  Total  registered  users,  DAU  stats  from  TechCrunch,  “Foursquare  looks  into  a  4th  round”,  Nov.  2,  2012.  All  other  numbers  are  SWAG  at  Foursquare  core  funnel  
  18. 18. Key  steps  to  idenEfying  opportuniEes   1)  Define  a  clear,  measurable  goal   –  Eg,  “We  want  to  increase  Foursquare  check-­‐ins  /  day”   2)  Define  the  relevant  data  set   –  Eg  “What  drives  daily  check-­‐ins?”   3)  Determine  the  status  quo   –  Eg,  “What  does  the  current  data  show  about  daily  check-­‐ ins?”   4)  IdenEfy  opportuniEes  to  improve  the  goal   –  Eg,  “What  are  the  inflecEon  points?”   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  19. 19. IdenEfy  opportuniEes  by  understanding  what  the  data  suggests  about  user  behavior   •  QuesEons  to  consider:   –  What’s  going  on  at  the  top  of  the  funnel?   –  At  the  bopom  of  the  funnel?   –  Which  acEons  are  we  most  concerned  with?   –  Where  do  we  “lose”  the  most  users?   –  What’s  working  well?  Why?   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  20. 20. Opportunity  #1:  Increase  daily  logins   Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous   Total  registered  users   25,000,000   Daily  acEve  users   7,200,000   28.8%   Click  “Check-­‐in”   1,800,000   25%   1   Select  locaEon   900,000   50%   Only  ~29%  of  the  user  base  logs  into  the  app  each   Complete  check-­‐in   630,000   day.  One  opportunity  would  be  to  apract  more   70%   users  to  the  app  each  day.  This  would  “widen  the   Social  Media  sharing   189,000  f  the  funnel”   top  o 30%   Share  photo   126,000   20%   No  meta  data   315,000   50%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  NOTE:  Total  registered  users,  DAU  stats  from  TechCrunch,  “Foursquare  looks  into  a  4th  round”,  Nov.  2,  2012.  All  other  numbers  are  SWAG  at  Foursquare  core  funnel  
  21. 21. Opportunity  #2:  Increase  the  daily  check-­‐ins   Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous   Total  registered  users   25,000,000   Daily  acEve  users   7,200,000   28.8%   Click  “Check-­‐in”   1,800,000   25%   Select  locaEon   900,000   50%   2   Complete  check-­‐in   630,000   70%   Only  ~25%  of  the  user  base  starts  the  “check-­‐in”   Social  Media  sharing   process  each  189,000   is  opportunity  to  increase   day.  There   30%   the  number  of  “check-­‐ins”  simply  by  gewng  the   Share  photo   126,000   20%   apenEon  of  our  logged  in  users   No  meta  data   315,000   50%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  NOTE:  Total  registered  users,  DAU  stats  from  TechCrunch,  “Foursquare  looks  into  a  4th  round”,  Nov.  2,  2012.  All  other  numbers  are  SWAG  at  Foursquare  core  funnel  
  22. 22. Opportunity  #3:  Increase  the  %  of  users   selecEng  locaEon   Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous   Total  registered  users   25,000,000   Daily  acEve  users   7,200,000   28.8%   Click  “Check-­‐in”   1,800,000   25%   Select  locaEon   900,000   50%   Complete  check-­‐in   630,000   70%   3   Social  Media  sharing   189,000   30%   Only  ~50%  of  the  users  that  start  a  “check-­‐in”   Share  photo   126,000   20%   actually  select  their  locaEon.  There  is  room  to   opEmize  this  step  of  the  funnel  and  minimize  the   No  meta  data   315,000   drop-­‐off   50%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  NOTE:  Total  registered  users,  DAU  stats  from  TechCrunch,  “Foursquare  looks  into  a  4th  round”,  Nov.  2,  2012.  All  other  numbers  are  SWAG  at  Foursquare  core  funnel  
  23. 23. Opportunity  #4:  Increase  the  number  of  users   compleEng  the  final  check-­‐in  step   Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous   Total  registered  4  sers   u 25,000,000   Daily  acEve  users   7,200,000   28.8%   We  lose  another  30%  of  users  on  the  final  step  of   Click  “Check-­‐in”   the  “check-­‐in.”  Is  there  anyway  to  prevent  that?   1,800,000   25%   Select  locaEon   900,000   50%   Complete  check-­‐in   630,000   70%   Social  Media  sharing   189,000   30%   Share  photo   126,000   20%   No  meta  data   315,000   50%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  NOTE:  Total  registered  users,  DAU  stats  from  TechCrunch,  “Foursquare  looks  into  a  4th  round”,  Nov.  2,  2012.  All  other  numbers  are  SWAG  at  Foursquare  core  funnel  
  24. 24. Summary:  4  key  steps  to  idenEfying  product  opportuniEes  with  data   Key  things  to  remember:   1)  Define  a  clear,  measurable  goal:  “Increasing   check-­‐ins”   2)  Collect  the  relevant  data  set  &  assemble  it   3)  Determine  the  status  quo   4)  IdenEfy  opportuniEes  to  improve  the  goal   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  25. 25. Dennis  says:  “I’ve  just  realized  that  …  ”   …  for  every  photo  that  gets  shared  on   Twiper  via  Foursquare,  we  acquire  2   new  users.  If  we  could  double  the   amount  of  photos  shared,  we’d  double   our  user  base.  How  many  more  photos   can  we  get  users  sharing  on  Twiper?”   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  26. 26. Which  of  the  4  opportuniEes  does  Dennis  want  to  take  advantage  of?   Eeeny  …  meeny  …  miny  …  moe  ….   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  27. 27. Opportunity  #5:  Increase  the  top  of  the   funnel  by  increasing  the  bopom!   Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous   Total  registered  users   25,000,000   Daily  acEve  users   7,200,000   28.8%   Click  “Check-­‐in”   5   1,800,000   25%   Select  locaEon   900,000   50%   Dennis’  insight:  If  we  increase  those  sharing  photos,   We  lose  another  30%  of  users  on  the  final  step  of   we  will  get  more  users  which  will  increase  the  top  of   Complete  check-­‐in   the  “check-­‐in.”  Is  there  anyway  t70%   630,000   o  prevent  that?   the  funnel   Social  Media  sharing   189,000   30%   Share  photo   126,000   20%   No  meta  data   315,000   50%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  NOTE:  Total  registered  users,  DAU  stats  from  TechCrunch,  “Foursquare  looks  into  a  4th  round”,  Nov.  2,  2012.  All  other  numbers  are  SWAG  at  Foursquare  core  funnel  
  28. 28. Agenda   •  Background   •  IdenEfying  opportuniEes   •  Using  metrics  to  priori4ze   •  TesEng  hypothesis  with  experiments   •  Running  “post-­‐mortem”  analysis   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  29. 29. Given  Dennis’  goals  of  increasing  photo  shares,  we  need  to  beper  understand  that  data   •  QuesEons  to  consider   –  What  does  the  photo  sharing  funnel  look  like?   –  What  drives  photo  sharing?   –  How  do  photos  get  shared  today?   –  How  can  we  encourage/discourage  that  behavior   to  achieve  our  goals?   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  30. 30. Zoom  in  on  the  social  media  and  photo  sharing  aspect  of  the  funnel  Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous  Compete  check-­‐ins   630,000  Social  media  shared   189,000   30%  Shared  to  Twiper   37,800   20%  Shared  to  Twiper  w/  photo   34,020   90%  Shared  to  FB   151,200   80%  Shared  to  FB  w/  photo   15,120   10%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  31. 31. #1:  Increase  the  %  of  users  who  share  a  photo  ayer  they’ve  decided  to  tweet  Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous  Compete  check-­‐ins   1   630,000   If  we  increase  the  %  of  users  who  share  a  photo  Social  media  shared   when  they  tweet,  w189,000   that  do  to  our   hat  would   30%   numbers?  Shared  to  Twiper   37,800   20%  Shared  to  Twiper  w/  photo   34,020   90%  Shared  to  FB   151,200   80%  Shared  to  FB  w/  photo   15,120   10%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  32. 32. By  increasing  Twiper  sharing,  gain  10%+  photo  shares  Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous  Compete  check-­‐ins   630,000  Social  media  shared   189,000   30%  Shared  to  Twiper   37,800   20%  Shared  to  Twiper  w/  photo   37,800  (+10%)   100%  Shared  to  FB   151,200   80%  Shared  to  FB  w/  photo   15,120   10%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  33. 33. #2:  Increase  the  %  of  people  sharing  via  social  media  channels  Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous  Compete  check-­‐ins   630,000  Social  media  shared   189,000   30%  Shared  to  Twiper   37,800   20%   2  Shared  to  Twiper  w/  photo   90%   What  happens  if  we  increase  the  %  of  people  Shared  to  FB   151,200   sharing  via  social  media  from  30%  to  50%?   80%  Shared  to  FB  w/  photo   15,120   10%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  34. 34. By  increasing  %  of  people  sharing  via  social,  gain  66.6%+  more  photo  shares!  Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous  Compete  check-­‐ins   630,000  Social  media  shared   315,000   50%  Shared  to  Twiper   63,000   20%  Shared  to  Twiper  w/  photo   56,700  (+66.6%)   90%  Shared  to  FB   252,000   80%  Shared  to  FB  w/  photo   25,200   10%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  35. 35. #3:  Increase  the  %  of  users  sharing  via  Twiper  vs.  Facebook  Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous  Compete  check-­‐ins   630,000  Social  media  shared   189,000   30%  Shared  to  Twiper   37,800   20%  Shared  to  Twiper  w/  photo   34,020   90%   3  Shared  to  FB   151,200   80%   What  happens  if  we  increase  the  %  of  users  who  Shared  to  FB  w/  photo   15,120   share  via  Twiper  from  20%  to  50%?   10%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  36. 36. By  increasing  mix  of  social  shares  to  Twiper,  gain  125%+  photo  –  holy  cow!!  Funnel  Step   Users  hiGng  that  step   %  proceeding  from  previous  Compete  check-­‐ins   630,000  Social  media  shared   189,000   30%  Shared  to  Twiper   94,500   50%  Shared  to  Twiper  w/  photo   85,050  (+125%)   90%  Shared  to  FB   94,500   50%  Shared  to  FB  w/  photo   9,450   10%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  37. 37. If  you  were  forced  to  only  make  1  change,  which  would  it  be?   •  Increase  the  %  of  users  who  share  a  photo  when  TweeEng  their   check-­‐in   –  Expected  impact:  +10%  increase  in  Tweets  w/  photo   •  Increase  the  %  of  users  who  decide  to  share  his/her  check-­‐in  on   social  media   –  Expected  impact:  +66%  increase  in  Tweets  w/  photo   •  Increase  %  of  users  who  share  his/her  check-­‐in  on  Twiper  vs.   Facebook   –  Expected  impact:  +125%  increase  in  Tweets  w/  photo   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  38. 38. If  you  were  forced  to  only  make  1  change,  which  would  it  be?   •  Increase  the  %  of  users  who  share  a  photo  when  TweeEng  their   check-­‐in   –  Expected  impact:  +10%  increase  in  Tweets  w/  photo   •  Increase  the  %  of  users  who  decide  to  share  his/her  check-­‐in  on   social  media   –  Expected  impact:  +66%  increase  in  Tweets  w/  photo   •  Increase  %  of  users  who  share  his/her  check-­‐in  on  Twiper  vs.   Facebook   –  Expected  impact:  +125%  increase  in  Tweets  w/  photo   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  39. 39. How  could  you  increase  %  of  users  sharing  via  Twiper  vs.  Facebook?   Op4ons  to  increase  %  of  TwiCer   shares   •  Remove  FB  as  an  opEon   •  Make  Twiper  “Opt-­‐out”   •  Provide  incenEve  to  “Tweet”  (eg,   “Extra  Foursquare  points”   •  Make  it  mandatory  for  any  user  w/   a  linked  Twiper  account   •  Move  it  “up”  in  the  funnel   •  Move  it  “down”  in  the  funnel  and   make  it  “opt-­‐out”   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  40. 40. How  could  you  increase  %  of  users  sharing  via  Twiper  vs.  Facebook?   Op4ons  to  increase  %  of  TwiCer   shares   •  Remove  FB  as  an  opEon   •  Make  Twiper  “Opt-­‐out”   •  Provide  incenEve  to  “Tweet”  (eg,   “Extra  Foursquare  points”   •  Make  it  mandatory  for  any  user  w/   a  linked  Twiper  account   •  Move  it  “up”  in  the  funnel   •  Move  it  “down”  in  the  funnel  and   make  it  “opt-­‐out”   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  41. 41. AddiEonal  consideraEons  when  prioriEzing  •  What  if  we  did  mul4ple  features  together?   –  Sure!  That  could  increase  the  expected  impacts  even  further   –  NOTE:  Must  be  careful  w/  experiment  design  here  so  results  aren’t  muddled  •  What  is  the  maximum  %  of  social  media  shares  that  TwiCer  could  get?   –  Data  needed:  What  %  of  users  have  linked  Twiper  accounts?  •  What  if  20%  is  the  maximum  share  percentage  (because  only  20%  of  users  have   TwiCer  linked)   –  You  need  to  apack  a  different  part  of  the  funnel   –  Build  a  feature  that  encourages  users  to  link  Twiper  accounts  •  But  there  must  be  more!?   –  Could  be  even  *more*  aggressive  by  puwng  social  media  and  photo  sharing  higher   in  the  funnel   –  Or  could  make  social  media  sharing  “opt-­‐out”  vs.  “opt-­‐in”   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  42. 42. Agenda   •  Background   •  IdenEfying  opportuniEes   •  Using  metrics  to  prioriEze   •  Tes4ng  hypothesis  with  experiments   •  Running  “post-­‐mortem”  analysis   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  43. 43. A  good  experiment  begins  with  a  clear  hypothesis   •  Our  hypothesis:   –  We  can  increase  the  %  of  users  sharing  to  Twiper   vs.  Facebook  to  50%  by  making  Twiper  “opt-­‐out”   –  This  will,  in  turn,  drive  the  number  of  Tweeted   photos  up  125%+   –  For  every  addiEonal  Tweeted  photo,  Foursquare   will  gain  2  new  users  /  day   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  44. 44. The  goal:  Prove  the  criEcal  aspects  of  our  hypothesis   •  CriEcal  aspects:   –  Get  50%  of  social  media  sharers  to  use  Twiper   –  Drive  up  Tweeted  photos  +125%   –  Acquire  2  new  users  for  each  addiEonal  photo   •  To  prove:   –  Run  a  controlled  A/B  test   –  Setup  a  test  where  50%  of  users  get  status  quo  flow   –  The  other  50%  get  the  new  Twiper  “opt-­‐out”  flow   –  Make  sure  you  have  staEsEcally  significant  sample   sizes  (eg  here  were  using  50%,  ~300K  check-­‐ins)   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  45. 45. Agenda   •  Background   •  IdenEfying  opportuniEes   •  Using  metrics  to  prioriEze   •  TesEng  hypothesis  with  experiments   •  Running  “post-­‐mortem”  analysis   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  46. 46. Key  steps  to  assembling  the  post-­‐mortem  analysis   1)  Collect  &  assemble  data  from  test  vs.  control   –  Eg,  “What  is  the  core  data  from  the  experiment”   2)  Compare  test  results  vs.  expected  results   –  Eg  “What  exceeded  or  missed  expectaEons?”   3)  What  are  the  next  steps   –  Eg,  “Should  we  invest  more  Eme/effort?  If  so,  on   what?  What  will  be  the  impact?”     SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  47. 47. Test  shows  40%  sharing  on  Twiper,  resulEng   in  +78%  in  tweeted  photos   Funnel  Step   Control  (50%  of   Test  (50%  of  users)   users)   Compete  check-­‐ins   315,000   315,000   Social  media  shared   94,500   30%   94,500   30%   Shared  to  Twiper   18,900   20%   37,800   40%   Shared  to  Twiper  w/   17,000   90%   30,240  (+78%)   80%   photo   Shared  to  FB   75,600   80%   56,700   60%   Shared  to  FB  w/  photo   7,560   10%   5,670   10%   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  NOTE:  Stats  from  TechCrunch,  “Foursquare  looks  into  a  4th  round”,  Nov.  2,  2012  
  48. 48. Key  steps  to  assembling  the  post-­‐mortem  analysis   1)  Collect  &  assemble  data  from  test  vs.  control   –  Eg,  “What  is  the  core  data  from  the  experiment”   2)  Compare  test  results  vs.  expected  results   –  Eg  “What  exceeded  or  missed  expectaEons?”   3)  What  are  the  next  steps   –  Eg,  “Should  we  invest  more  Eme/effort?  If  so,  on   what?  What  will  be  the  impact?”     SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  49. 49. How  does  this  compare  to  expectaEons?  Why  did  this  happen?  Funnel  Step   Expecta4ons   %  proceeding   Test  (50%  of  users)   %  proceeding   Delta    Compete  check-­‐ins   315,000   315,000  Social  media  shared   94,500   30%   94,500   30%   ~  Shared  to  Twiper   47,250   50%   37,800   40%   -­‐10%  Shared  to  Twiper  w/   42,525  (+125%)   90%   30,240  (+78%)   80%   -­‐10%  photo  Shared  to  FB   47,250   50%   56,700   60%   +10%  Shared  to  FB  w/  photo   4,725   10%   5,670   10%   ~   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  50. 50. Key  steps  to  assembling  the  post-­‐mortem  analysis   1)  Collect  &  assemble  data  from  test  vs.  control   –  Eg,  “What  is  the  core  data  from  the  experiment”   2)  Compare  test  results  vs.  expected  results   –  Eg  “What  exceeded  or  missed  expectaEons?”   3)  What  are  take  aways  &  next  steps   –  Eg,  “Should  we  invest  more  Eme/effort?  If  so,  on   what?  What  will  be  the  impact?”     SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  51. 51. Key  quesEons  &  take-­‐aways  Key  ques4on   Result   Why?   Next  steps    Did  we  get  50%  of  users   No.  We  got  40%   •  Maybe  hit  a  natural  limit  (%  of   •  Determine  natural  limit  to  share  on  Twiper?   users  w/  Twiper  accounts)   •  Consider  encouraging   account  linking  Did  we  get  +125%   No.  We  got  78%   •  Photo  sharing  %  dropped  to   •  Can  we  increase  photo  increase  in  photo   80%   sharing  %?  sharing?   •  We  only  got  40%  sharing  via   Twiper  (vs.  expected  50%)  What’s  the  upside  ley?   12,525  photo   •  If  we  can  tweak  to  hit  goals  of   •  What  %  of  that  upside  is   shares  /  day   50%  and  90%   *truly*  achievable  given   our  results?  Was  the  test  a  success?   Yes!   •  Proved  that  tweaking  Twiper   •  Evaluate  above  opEons,   opEon  can  drive  photo  shares   determine  prioriEes  &   repeat!   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  52. 52. Conclusions   •  OrganizaEon  is  key   –  Start  with  the  big  picture,  peel  back  the  layers   •  Define  clear  goals,  hypothesis     –  You  won’t  know  if  your  tests  or  features  worked  if  you   don’t  pre-­‐define  a  good  goal  and  hypothesis   •  Data  driven  PM’ing  is  applicable  to  all  aspects     –  We  focused  on  internal  data  but  you  could  use  it  on   market  data,  with  surveys,  with  organizaEonal  issues,   almost  anything…   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  
  53. 53. Thanks  &  final  notes   •  Slides  will  be  sent  out   •  Contact  info:   –  @kivestu   –  kivestu@gmail.com   –  kentonkivestu.com  (thoughts  on  product   development,  mobile)   SkillShare:  Think  Like  a  PM,  Kenton  Kivestu  

×