Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Designing & Measuring
Experiments
@GoAgileCamp
#GoAgileCamp
Tristan.Kromer@TriKro.com
WWW
@TriKro
Tristan Kromer
i@mNotaNinja.com
GrasshopperHerder.com
Real Experiment
Create a social news reader that delivers
inspiring articles to you based on your likes
and follows.
You’ll know if it’s s...
Common Mistake #1:
Not having a Learning Goal
Real Experiment
Learning goal:
Do customers want our VP?
Hypothesis:
50% of people we interview will commit to
paying $20 a month for the ...
Common Mistake #2:
Assuming the hypothesis
is a hypothesis
Research
• Who is our customer?
• What are their pains?
• What job needs to be done?
• Is our customer segment too
broad?
...
Real Experiment
Learning goal:
Which value prop is better? Faster
download time or obfuscated IP address.
Hypothesis:
Using VP of Faster d...
14
15
16
17
18
Version A Version B
Common Mistake #3:
Measuring, misreading, and
miscalculating the metric
14
15
16
17
18
Version A Version B
14
15
16
17
18
Version A Version B
14
15
16
17
18
Version A Version B
Defining Good
Experiments
!
Experiment Herder Template* by TriKro LLC!
(For S.M.A.R.T Experiments: Smart Measurable Achievable Relevant Timely)
Lear...
Common Mistake #1:
Not having a learning goal
Common Mistake #2:
Assuming the hypothesis
is a hypothesis
Common Mistake #3:
Measuring, misreading, and
miscalculating the metric
Common Mistake #4:
Failure to define failure
Common Mistake #5:
Actually…
people get this one right
Common Mistake #5:
Too much time
Common Mistake #6:
Regretfully not doing retros
Common Mistake #7:
Ignoring results
Common Mistake #8:
Not learning from learning
Common Mistake #9:
Ignoring warning signs
Experiment/Research Templates*
1. Learning Goal
2. Hypothesis/Assumptions
3. Metric
4. Fail Condition
5. Plan
6. Time Box
...
The
Real Startup Book
• 5 second tests
• Comprehension
• Conjoint Analysis
• Data mining / market
research
• Surveys* (closed)
• Smoke tests
• V...
For more info and updates:
http://trikro.com/downloads/playbook
Tristan.Kromer@TriKro.com
WWW
@TriKro
Tristan Kromer
i@mNotaNinja.com
GrasshopperHerder.com
This presentation by TriKro LLC is licensed under a
Creative Commons Attribution-NonCommercial-ShareAlike
4.0 Internationa...
AgileCamp Silicon Valley 2015: Experiment Design
AgileCamp Silicon Valley 2015: Experiment Design
AgileCamp Silicon Valley 2015: Experiment Design
AgileCamp Silicon Valley 2015: Experiment Design
AgileCamp Silicon Valley 2015: Experiment Design
AgileCamp Silicon Valley 2015: Experiment Design
AgileCamp Silicon Valley 2015: Experiment Design
AgileCamp Silicon Valley 2015: Experiment Design
AgileCamp Silicon Valley 2015: Experiment Design
AgileCamp Silicon Valley 2015: Experiment Design
AgileCamp Silicon Valley 2015: Experiment Design
AgileCamp Silicon Valley 2015: Experiment Design
Upcoming SlideShare
Loading in …5
×

AgileCamp Silicon Valley 2015: Experiment Design

AgileCamp Silicon Valley 2015: Experiment Design by Tristan Kromer, Triko

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

AgileCamp Silicon Valley 2015: Experiment Design

  1. 1. Designing & Measuring Experiments @GoAgileCamp #GoAgileCamp
  2. 2. Tristan.Kromer@TriKro.com WWW
  3. 3. @TriKro Tristan Kromer i@mNotaNinja.com GrasshopperHerder.com
  4. 4. Real Experiment
  5. 5. Create a social news reader that delivers inspiring articles to you based on your likes and follows. You’ll know if it’s successful based on daily usage and the viral coefficient. Constraints You have no funding but two great developers. Vision
  6. 6. Common Mistake #1: Not having a Learning Goal
  7. 7. Real Experiment
  8. 8. Learning goal: Do customers want our VP? Hypothesis: 50% of people we interview will commit to paying $20 a month for the product. Plan: Pitch it, see who says yes!
  9. 9. Common Mistake #2: Assuming the hypothesis is a hypothesis
  10. 10. Research • Who is our customer? • What are their pains? • What job needs to be done? • Is our customer segment too broad? • How do we find them? • How can we solve this problem? • What form should this take? • How important is the design? • What’s the quickest hack? • What’s the minimum feature set? • How should we prioritize? Generate Ideas & Identify Assumptions
 Experiments • How much will they pay? • How do we convince them to buy? • How much will it cost to sell? • Can we scale marketing? • Is this solution working? • Are people using it? • Which solution is better? • How should we optimize this? • What do people like / dislike? • Why do they do that? Test Hypotheses
  11. 11. Real Experiment
  12. 12. Learning goal: Which value prop is better? Faster download time or obfuscated IP address. Hypothesis: Using VP of Faster download time instead of obfuscated IP address will increase conversion rate on home page above 15% (our current benchmark). Plan: Use optimizely to change VP on home page for 2000 visits. Time box: 7 days Result: 16%, Win!
  13. 13. 14 15 16 17 18 Version A Version B
  14. 14. Common Mistake #3: Measuring, misreading, and miscalculating the metric
  15. 15. 14 15 16 17 18 Version A Version B
  16. 16. 14 15 16 17 18 Version A Version B
  17. 17. 14 15 16 17 18 Version A Version B
  18. 18. Defining Good Experiments
  19. 19. ! Experiment Herder Template* by TriKro LLC! (For S.M.A.R.T Experiments: Smart Measurable Achievable Relevant Timely) Learning Goal! !What do we need to learn?! What is our riskiest assumption?! What is our one priority? Hypothesis! !Is it falsifiable?! Is it specific?! Is it causal? e.g. If X then Y! Is it Relevant to the learning goal? Metric! !Qualitative or quantitative?! Is it actionable?! Is it Measurable? Fail Condition! !!!!If this happens,! our hypothesis is clearly false! wtf,s!! !!!!If this happens, stop!! Experiment is broken, retro! Experiment! !How will you collect the data?! Is it Specific?! Is it Achievable?! !Link to any supporting documents Time Box! !Is the experiment Timely?! Can we get data faster?! Would less data be sufficient?! Is the retro in our calendar? Start End Retro Result! !What happened?! What data did we collect?! Anything unexpected? Next Step! !Pivot or persevere?! Another experiment for this goal?! Do we need to clean up? Designed by: TriKro LLC - Lean Coaching and Innovation Ecosystem Design! The Smart Experiment Template by TriKro LLC is licensed under! a Creative Commons Attribution-ShareAlike 4.0 International License Designed for: Designed by: * Templates are bad,! make your own!
  20. 20. Common Mistake #1: Not having a learning goal
  21. 21. Common Mistake #2: Assuming the hypothesis is a hypothesis
  22. 22. Common Mistake #3: Measuring, misreading, and miscalculating the metric
  23. 23. Common Mistake #4: Failure to define failure
  24. 24. Common Mistake #5: Actually… people get this one right
  25. 25. Common Mistake #5: Too much time
  26. 26. Common Mistake #6: Regretfully not doing retros
  27. 27. Common Mistake #7: Ignoring results
  28. 28. Common Mistake #8: Not learning from learning
  29. 29. Common Mistake #9: Ignoring warning signs
  30. 30. Experiment/Research Templates* 1. Learning Goal 2. Hypothesis/Assumptions 3. Metric 4. Fail Condition 5. Plan 6. Time Box 7. Results 8. Next Step 9. wtf,s!
  31. 31. The Real Startup Book
  32. 32. • 5 second tests • Comprehension • Conjoint Analysis • Data mining / market research • Surveys* (closed) • Smoke tests • Video • Landing page • Customer Discovery Interviews • Contextual inquiry / ethnography • Data mining • Focus groups* • Surveys* (open ended) • Solution interview • Contextual inquiry / ethnography • Demo pitch • Concierge test / Consulting • Competitor Usability • Picnic in the Graveyard • Paper prototypes
  33. 33. For more info and updates: http://trikro.com/downloads/playbook
  34. 34. Tristan.Kromer@TriKro.com WWW
  35. 35. @TriKro Tristan Kromer i@mNotaNinja.com GrasshopperHerder.com
  36. 36. This presentation by TriKro LLC is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License excepting those images where other copyright holders are noted. Based on a work at http://GrasshopperHerder.com.

×