Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Keeping you watching "house of cards" by Brian Holt

437 views

Published on

What do your users want? Do you have a clue? Track your data and increase engagement - lessons by Netflix. Don't be short-sighted, try new experiences!

Published in: Entertainment & Humor
  • Be the first to comment

Keeping you watching "house of cards" by Brian Holt

  1. 1. K E E P I N G Y O U WAT C H I N G H O U S E O F C A R D S B R I A N H O LT @ H O LT B T
  2. 2. b i t . l y / t e c h s u m m i t p r 1 6
  3. 3. W H AT D O Y O U R U S E R S WA N T ?
  4. 4. Y O U H AV E N O * * * * I N G C L U E .
  5. 5. N E I T H E R D O T H E Y.
  6. 6. S O W H Y C A N ’ T I T R A C K M Y D ATA B E F O R E A N D A F T E R I A D D F E AT U R E S ?
  7. 7. B O O K S A L E S O N M Y W E B S I T E 0 17.5 35 52.5 70 1 2 3 4 5 6 7 8 launched new feature
  8. 8. W H AT A C T U A L LY H A P P E N E D launched new feature Oprah put my book in her book club 0 15 30 45 60 1 2 3 4 5 6 7 8 A B A B A B A B A B A B A B A B
  9. 9. W H AT A C T U A L LY H A P P E N E D 0 17.5 35 52.5 70 1 2 3 4 5 6 7 8 A B A B A B A B A B A B A B A B started my a/b test Oprah put my book in her book club
  10. 10. vs
  11. 11. 2 0 % L I F T I N V I E W I N G
  12. 12. W H Y ?
  13. 13. W E H AV E O U R T H E O R I E S B U T O F T E N I T ’ S T O U G H T O E S TA B L I S H C A U S A L I T Y.
  14. 14. A / B T E S T I N G I S N O T A S O L U T I O N .
  15. 15. Y O U R N E W A / B T E S T
  16. 16. R E D C E L L 💩
  17. 17. 💩 B L U E C E L L 💩 💩 💩💩 💩💩💩 💩
  18. 18. G R E E N C E L L 💰
  19. 19. I F Y O U R E F I N E A 💩 E X P E R I E N C E V I A A / B T E S T I N G , Y O U E N D U P W I T H A R E F I N E D 💩 E X P E R I E N C E
  20. 20. T RY E N T I R E LY N E W E X P E R I E N C E S T O O 💰
  21. 21. A / B T E S T S • DO • Verify hypothesis • Indicate positive directions to go in • Identify areas of consumer sensitivity • Identify the key metrics you’re tracking (revenue versus sign-ups) • DON’T • Create your hypothesis after you have run the test • Introduce your bias • Assume that A/B testing will give you answers
  22. 22. D O N ’ T B E S H O R T- S I G H T E D • Some tests may indicate positive short-term results but hide long-term wins. • e.g. SEO win versus monetization win
  23. 23. A D V I C E ? • Simpler is almost always better • Faster • Less options and less choices • Less friction • Make choices you know most users will make; allow other users the escape hatch*
  24. 24. A / B T E S T I N G H A R D PA R T S • Be willing to throw work out • Be willing to do ideas and features you don’t like • Realize your consumers will tell you one thing but act another way • Be willing to ignore the indication of the results
  25. 25. K E E P I N G Y O U WAT C H I N G H O U S E O F C A R D S ? • We A/B test everything at Netflix. Every button, every layout, every algorithm, every color. It’s all A/B tested to be the least friction possible. • That keeps the interface intuitive and out of the way between you Frank Underwood’s hostile Washington DC takeover.
  26. 26. H O W D O I A / B T E S T ? • Open Source • PlanOut (Facebook) • SixPack (SeatGeek) • Abba (Stripe) • Proctor (Indeed)
  27. 27. H O W D O I A / B T E S T ? • Commercial • Optimizely • Google Analytics Experiments • KISSmetrics
  28. 28. B R I A N H O LT @ H O LT B T

×