Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Leading the product 2017 - Susan Teschner - Speaker Slides

246 views

Published on

Title: Experimenting with Experimentation - Pro tips for doing it well.
Presented at Leading The Product 2017 - Sydney

Published in: Education
  • Be the first to comment

  • Be the first to like this

Leading the product 2017 - Susan Teschner - Speaker Slides

  1. 1. Leading The Product 2017 Speaker Slides Sydney, Australia Susan Teschner 99designs For more information go to www.leadingtheproduct.com
  2. 2. © 99designs, Inc.© 99designs, Inc. Experimenting with ExperimentationPro tips for doing it well by Susan Teschner
  3. 3. © 99designs, Inc. 3 Who am I? some days I’m not sure
  4. 4. 4 design Retail catalogues & ad networks Accounting software VoIP Shipping for eCommerce Classifieds 99 iNC MYOB Delta 3 Australia Post Fairfax SEEK
  5. 5. © 99designs, Inc. 5 Manage products. Manage people managing products. by Evilltimmby LanaD.
  6. 6. 6 World’s largest marketplace for graphic design We’re very busy — 450,000happy customers We’re famous but with hidden talents — 36 Design categories by Vladson We’re home grown and a bit grown up — 2008 Founded in Melbourne 115 Fabulous people 3 Offices (BER, OAK, MEL) $200mpaid to designers
  7. 7. © 99designs, Inc. 7 Experimentation?
  8. 8. © 99designs, Inc. 8 “doing a thing to discover something unknown or to test a principle” — poorly paraphrased from dictionary.com
  9. 9. 9 The universe of experiments — by Gorcha by huhi
  10. 10. © 99designs, Inc. 10 So this is an agile thing?
  11. 11. © 99designs, Inc. 11 Where are they now? by kazoe
  12. 12. © 99designs, Inc. 12 How does it help?
  13. 13. © 99designs, Inc. 13 8 lessons
  14. 14. © 99designs, Inc. DO pay attention — 1
  15. 15. © 99designs, Inc. 15 I don’t have all the ideas — by Thedaydreamer
  16. 16. © 99designs, Inc. 16 How 99designs was born… — by xeniatm
  17. 17. © 99designs, Inc. 17 Why we created 1:1 Projects — by ktoons
  18. 18. © 99designs, Inc. DON’T underestimate the importance of experiment design — 2
  19. 19. © 99designs, Inc.Commercial in confidence 19 Start off on the right foot — ▪ clearly stated hypothesis ▪ definition of success ▪ specific metrics by ktoons
  20. 20. © 99designs, Inc. 20 Beware the dragon of statistical significance — by ilustreishon
  21. 21. © 99designs, Inc. 21 Here’s how — https://unbounce.com/ab-test-duration-calculator/ Sample Size & Test Duration Calculator Current conversion rate (%) Desired lift (%) Number of variations Average daily visitors To achieve valid test results*, you need: A total sample of (users): Test duration of (days**): 10800 5 10 20 3 2000
  22. 22. © 99designs, Inc. Sample Size & Test Duration Calculator Current conversion rate (%) Desired lift (%) Number of variations Average daily visitors To achieve valid test results*, you need: A total sample of (users): Test duration of (days**): 10800 54 10 20 3 200 Sample Size & Test Duration Calculator Current conversion rate (%) Desired lift (%) Number of variations Average daily visitors To achieve valid test results*, you need: A total sample of (users): Test duration of (days**): 10800 5 10 20 3 2000 22https://unbounce.com/ab-test-duration-calculator/ Here’s how —
  23. 23. © 99designs, Inc. 23 This was a bit too long to wait — How long will your experiment take? How many eligible visits do you have per day? What percentage of visits will see the change? What is your current conversion rate? How will you change conversion? How confident do you want to be of this? How likely should you be to detect the change? 1,400 50% 2% 5% 95% 90% You should run this experiment for 1 year, 7 months and 28 days.
  24. 24. © 99designs, Inc. DO choose the right tools — 3
  25. 25. © 99designs, Inc. 25
  26. 26. 26 other data sets Performance UI / UX Resourcing Maturity $ Things to think about —
  27. 27. © 99designs, Inc. 27 So, who’s your favorite? — by glasshopperart
  28. 28. © 99designs, Inc. DON’T spend too much — 4
  29. 29. © 99designs, Inc. 29 Beware the gold plating — by MattDyckStudios https://www.quora.com/How-is-pretotyping-different-from-prototyping
  30. 30. © 99designs, Inc. 30 Baby steps towards a Gift Shop i1 i2
  31. 31. © 99designs, Inc. …but DO spend enough — 5
  32. 32. © 99designs, Inc.© 99designs, Inc. 32 “We increased sales by 37% and all we had to do was change the color of a button! by Pierre d’Arterie
  33. 33. © 99designs, Inc. 33 This was never going to wow.
  34. 34. © 99designs, Inc. 34 by Hildebrand
  35. 35. © 99designs, Inc. 1 Testing tools — 2 Variations — 3 Opportunity cost — by RaidenD
  36. 36. © 99designs, Inc. DON’T ignore the results — 6
  37. 37. © 99designs, Inc. 37 If it seems too good to be true… — by Frankie Cadillac
  38. 38. © 99designs, Inc. 38 Share the results. Publicly. — by Sam!
  39. 39. © 99designs, Inc. DO make decisions & follow through — 7
  40. 40. © 99designs, Inc. 40 R.I.P. — by RVST
  41. 41. © 99designs, Inc. 41 … or put your money where your experiment is —
  42. 42. © 99designs, Inc. 42 … or take your chances with the people who look after it — by Thedaydreamer
  43. 43. © 99designs, Inc. 43 A story about Social Sharing by clouds940
  44. 44. © 99designs, Inc. 44 by clouds940 A story about Social Sharing
  45. 45. © 99designs, Inc. 45 A story about Social Sharing
  46. 46. © 99designs, Inc. 46 A story about Social Sharing by miljanristic by FRITO-DESIGNstudio by Joedsign
  47. 47. © 99designs, Inc. 47 by clouds940 A story about Social Sharing
  48. 48. © 99designs, Inc. 48 WALK AWAY extremeuncertainty.com
  49. 49. © 99designs, Inc. Until one day… — Ashish Yep, I’ve also flagged this feature with @Miranda Burford as something we should critically evaluate as I’m concerned the benefits have not outweighed the costs. Susan Good call @Miranda Burford. And it didn’t go unnoticed that we’re the only ones who ticked off our To-Do’s. Miranda Just to circle back on this, we reviewed Social Sharing and made the decision to retire it. If you’re interested in how we came to that decision, you can check out the slide deck.
  50. 50. © 99designs, Inc. DO learn from how you’re learning — 8
  51. 51. © 99designs, Inc. 51 How’s it going? by stormyfuego
  52. 52. © 99designs, Inc. 2017-05-17 2017-07-05 2017-05-12 2017-04-18 Start End Team 2017-04-29 Growth Activation Growth -2% click through -11% reduction of click through to Other design category 50% 2017-07-17 -10% clicks 0% conversion impact Growth Acquisition /categories Email forward in brief Improved search on categories Spruced up cross sell Experiment Page Conversion goal /work Click to product pages Click repeat purchase Visual brief Increase email capture Minimum / Aimed improvement 5% 65% 33% Observed improvement 52 At a glance —
  53. 53. © 99designs, Inc. 53 Facing the music? — by jestyr37
  54. 54. © 99designs, Inc. 54 Treat it like an experiment — by dante banuelos
  55. 55. © 99designs, Inc. Bottom line? by Mad pepper
  56. 56. © 99designs, Inc. 56 Thank you by Thedaydreamer

×