Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hypothesise like you Mean it!

54 views

Published on

This session - developed and delivered in collaboration with Lisa Long - was an introduction to the essential components of an effective hypothesis and meaningful metrics. We wanted to give the attendees a working knowledge of what makes good AND bad hypotheses and metrics, as well as simple tools to help attendees design & iterate with confidence.

The session was aimed at people who are unfamiliar with well-structured hypotheses for product design & testing purposes, or who just want to refresh and hone their testing and iteration skills.

The goal was to give participants a working understanding of why good hypotheses and metrics are important for good iterative design, and to walk them through a simple set of guidelines, canvases and techniques for establishing both.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Hypothesise like you Mean it!

  1. 1. Hypothesize like you mean it The Essentials of Metrics, Hypotheses, and Iteration
  2. 2. Greetings :)
  3. 3. Chris Massey Product lead @ Mind the Product chris@mindtheproduct.com @camassey
  4. 4. Lisa Long Founder @ Before You Code lisa@beforeyoucode.com @beforeyoucode
  5. 5. Learning outcomes
  6. 6. Metrics & Hypotheses & Prioritisation & Iteration ● What makes a good metric ● How to design an effective hypothesis ● The essentials of Prioritisation ● Experimental Iteration
  7. 7. Let’s go!
  8. 8. What is a Metric? Meaningful metrics (data+context) drive targeted, effective actions that create tangible value
  9. 9. Good metric / Bad metric Does your metric suggest success without being directly tied to the outcome you care about? Is your metric a placeholder for something that is hard to measure?
  10. 10. What is a hypothesis? A statement of the specific (& testable) impact you believe a proposed change will have.
  11. 11. The purpose of a hypothesis To structure thinking To reveal assumptions To direct resources To avoid waste To help you prioritise
  12. 12. Null Hypotheses Assume your worldview wrong !
  13. 13. High vs Low-level Hypotheses Understand your chain of value Make sure you can isolate your variables / metrics Pick one
  14. 14. Designers & Dragons
  15. 15. Goals HIGH-LEVEL MISSION, VISION OR OBJECTIVES DESIGN PARAMETERS Metrics Framing Context Features SPECIFIC, MEASURABLE, CONTEXTUALISED ACTION- ORIENTED ENVIRONMENT, AUDIENCE, TIMEFRAMES ... HIGH-LEVEL FEATURE CONCEPTS, SPECIFIC IMPLEMENTATIONS ...
  16. 16. Goals ● Tourist attraction ● More awareness of Cambridge history / significance DESIGN PARAMETERS Metrics Framing Context Features ● Tour company inclusion ● Pre-test / Post-test of local history ● Traffic-flow assessment ● Social media buzz ENVIRONMENT, AUDIENCE, TIMEFRAMES ... HIGH-LEVEL FEATURE CONCEPTS, SPECIFIC IMPLEMENTATIONS ...
  17. 17. Instant Art Generator ™ 1. Kinetic 2. Projection 3. Light-based 4. Physical interaction 5. Multiple pieces 6. Historical 7. Phone interaction 8. Large scale 9. Persistent changes 10. Consistent start state 11. All-weather 12. University collaboration
  18. 18. Go design
  19. 19. Activity: Design a civic art installation 15 mins Goals Brainstorm a potential design for an informative civic art installation Activity ● Use google to get inspiration for art installations that use your constraints ● Consider what historically significant or helpful facts about Cambridge could be represented in a public installation ● Come up with as many ways as possible to represent those facts ● Converge on a set of design “features” for your installation Deliverable An informative civic art installation, described in “features”, potentially supported by rough sketches
  20. 20. What’s your riskiest assumption?
  21. 21. To the hypothesis canvas!
  22. 22. We believe that ... For ... Will lead to ... Because... SOME FEATURE OR CHANGE AN ENVIRONMENT OR CONTEXT SOME SPECIFIC, MEASURABLE CHANGE YOUR RATIONALE HYPOTHESIS CANVAS
  23. 23. Go Hypothesise
  24. 24. Goals Form a testable hypothesis around one aspect of your art concept Activity ● Pick the metric you want to effect ● Pick your riskiest assumption / feature ● Select your experimental context ● Think through your experimental setup ● Articulate your experimental rationale Deliverable A well-formed, testable hypothesis, allowing you remove uncertainty / validate an assumption around one “feature” of your design product Activity: Form a hypothesis 10 mins
  25. 25. How’d it go?
  26. 26. Designers & Diagnosis & Dragons
  27. 27. Some Role-play Heads = Failure Tails = Ambiguous
  28. 28. What impacts experimental outcomes? ● Bias ● Confounding factors ● Chance! ● True effect
  29. 29. Common mistakes ● Not everything deserves a hypothesis ● When to abandon (sunk cost) ● Check sample size & source ○ (67% conversion on 3 actions out of 10 million users) ● Are you asking the right question? ● Is every test succeeding / failing? ○ (check for bias or poor systematic design) ● Bad data foundations ● Technical challenges
  30. 30. What happened?
  31. 31. Activity: Diagnose errors in your experiment 10 mins Goals Brainstorm a wide range of possible ways your experiment might have yielded an ambiguous or unsuccessful result Activity ● Revisit & reassess your underlying assumptions ● List any ways your assumptions could be wrong ○ Don’t underestimate human stupidity ● Step through your experiment setup ● List the possible errors in that setup Deliverable A list of possible incorrect assumptions and experimental errors
  32. 32. NOW what’s your riskiest assumption?
  33. 33. Let’s talk about prioritisation ● Go back to your metrics ● What are your assumptions? ● What are you uncertain about? ● What are the risks? ● What are the costs to test?
  34. 34. Uncertainty Risk TRACK TEST IGNORE MITIGATE Low High High
  35. 35. Risk Cost TEST ASSESS ASSESS IGNORE Low High High
  36. 36. What’s the next Hypothesis?
  37. 37. We believe that ... For ... Will lead to ... Because... SOME FEATURE OR CHANGE AN ENVIRONMENT OR CONTEXT SOME SPECIFIC, MEASURABLE CHANGE YOUR RATIONALE HYPOTHESIS CANVAS
  38. 38. Goals Create a better-informed, testable hypothesis around one aspect of your art concept Activity ● Consider what you’ve learned thus far about your context ● Pick the metric you want to effect ● Pick your riskiest assumption / feature ● Select your experimental context ● Think through your experimental setup ● Articulate your experimental rationale Deliverable A well-formed, better-informed, testable hypothesis, allowing you remove uncertainty / validate an assumption around one “feature” of your design product Activity: Form a new hypothesis 5 mins
  39. 39. What questions do you have?
  40. 40. What did we learn?
  41. 41. Metrics & Hypotheses & Prioritisation & Iteration ● What makes a good metric ● How to design an effective hypothesis ● The essentials of Prioritisation ● Experimental Iteration
  42. 42. Further Reading http://experimentationhub.com/hypothesis-kit.html +
  43. 43. Thanks! Chris Massey & Lisa Long @camassey | @beforeyoucode chris@mindtheproduct.com lisa@beforeyoucode.com
  44. 44. Bonus round
  45. 45. How did it go this time?
  46. 46. Some more role-play Rock = Success Paper = Failure Scissors = Ambiguous
  47. 47. Presentation to Design Council … in one week … to secure more £££
  48. 48. What do you do?
  49. 49. How to present experimental findings Context, Context, Context Your sources Your sample size Your assumptions Your experiment design Your findings

×