Product Testing and Refinement

1,227 views

Published on

This presentation was given to Techgirlz annual entrepreneurial summer camp for middle school girls. This presentation was part of a 2-hour workshop on product design, testing, and refinement.

Published in: Business
  • Be the first to comment

  • Be the first to like this

Product Testing and Refinement

  1. 1. Product Testing and Refinement Chris Cera, CTO @ Arcweb July 9, 2014
  2. 2. What is the name of this cookie?
  3. 3. Samoas or Caramel deLites?
  4. 4. What should we name our product?
  5. 5. The Scientific Method
  6. 6. The Scientific Method 1. Ask a question 2. Do background research 3. Construct a hypothesis 4. Test your hypothesis 5. Analyze your data and draw a conclusion 6. Report the results
  7. 7. “Guess then test”
  8. 8. Validate your hypothesis
  9. 9. Talk to prospective customers
  10. 10. Currency
  11. 11. Currency
  12. 12. Currency
  13. 13. Currency
  14. 14. Currency
  15. 15. What should we name our product?
  16. 16. Hypotheses 1. “Samoas” 2. “Caramel deLites” 3. “Caramel DeLites” 4. … 5. … 6. ... 7. … 8. … 9. … 10. …
  17. 17. How to test our hypotheses?
  18. 18. What should we name our product?
  19. 19. Book Funnel Analysis • English speakers on earth – 100% • Read a business book – 45% • Discover the product ad – 30% • Click the product ad – 15% • Click the “Add to cart” button – 10% • Start the checkout process – 8% • Enter their credit card information – 7% • Enter their shipping information – 6% • Buy the book – 5%
  20. 20. Optimize
  21. 21. Hypotheses 1. “The 4-Hour work week” 2. “How to work less” 3. “How to get rich quick” 4. … 5. … 6. ... 7. … 8. … 9. … 10. …
  22. 22. Example Advertisement
  23. 23. Budget Challenge: $??? • Given – We have 10 names to test – We will test each one on 10 different people – It costs $1 for each test (i.e. to buy an ad) • How much money do we need?
  24. 24. Budget Challenge: $100
  25. 25. The 4-Hour Workweek (of course)
  26. 26. How else could this have been tested?
  27. 27. A/B Testing Examples
  28. 28. A/B Testing Examples
  29. 29. How will users want to sign in?
  30. 30. Sign In Funnel Analysis • English speakers on earth – 100% • Discover the product – 50% • Download the product – 25% • Launch the product – 20% • Create a username – 10% • Sign into their account – 9% • Launch the product monthly – 5% • Launch the product weekly – 3% • Launch the product daily – 2% • Launch the product hourly – 1%
  31. 31. Sign In Page: Example
  32. 32. Sign In Page: Borrowing
  33. 33. Sign In Page: Borrowing
  34. 34. Sign In Page: Borrowing 3x
  35. 35. Sample Test Results • Twitter converts 2% of people • Facebook converts 10% of people
  36. 36. A/B Testing Execution • Given 3,000 people • A/B test for the first 300 (10% of them) – 6 different campaigns with 50 tests each • Then test final 2,700 people with best campaign. • Results – Worst campaign (Twitter) converted 2% – Best campaign (Facebook) converted 10% – Difference between 54 and 270 people. – For a $50 product, that is $10,800!
  37. 37. Budget Challenge: Implementation • Implement Twitter login: $10 • Implement Facebook login: $15 • Implement LinkedIn login: $20 • But we only have $10!?! What do we do?
  38. 38. Group Work
  39. 39. Group Work • Assignment [50 minutes] – List 1-2 of your funnels, then “guess and test” – Identify a step to optimize – List your hypothesis and test(s) – Test them on other group members, collect currency • Group Presentations [30 minutes] – Steps in the funnel(s) – Optimization step(s) – List your hypothesis and test(s) – Summarize the results
  40. 40. Thank you • Chris Cera, CTO • cera@arcweb.co • 215.435.2289

×