We designed a hypothetical civic art installation for Cambridge with interactive features representing local history. We formed an initial hypothesis that adding a phone interaction feature would increase awareness of the installation. However, upon diagnosing errors in our experimental design, we realized our assumptions were flawed and risks were high. We then created a new, better-informed hypothesis to test a different feature with the goal of removing uncertainty and validating our assumptions through an improved experimental approach.

1. Hypothesize like you
mean it
The Essentials of Metrics, Hypotheses, and Iteration

2. Greetings :)

3. Chris Massey
Product lead @ Mind the Product
chris@mindtheproduct.com
@camassey

4. Lisa Long
Founder @ Before You Code
lisa@beforeyoucode.com
@beforeyoucode

5. Learning outcomes

6. Metrics &
Hypotheses &
Prioritisation &
Iteration
● What makes a good metric
● How to design an effective
hypothesis
● The essentials of
Prioritisation
● Experimental Iteration

7. Let’s go!

8. What is a Metric? Meaningful metrics
(data+context) drive targeted,
effective actions that create
tangible value

9. Good metric /
Bad metric
Does your metric suggest
success without being directly
tied to the outcome you care
about?
Is your metric a placeholder for
something that is hard to
measure?

10. What is a
hypothesis?
A statement of the
specific (& testable)
impact you believe a
proposed change will
have.

11. The purpose of a
hypothesis
To structure thinking
To reveal assumptions
To direct resources
To avoid waste
To help you prioritise

12. Null Hypotheses Assume your worldview wrong
!

13. High vs Low-level
Hypotheses
Understand your chain of value
Make sure you can isolate your
variables / metrics
Pick one

14. Designers & Dragons

15. Goals
HIGH-LEVEL MISSION,
VISION OR OBJECTIVES
DESIGN PARAMETERS
Metrics
Framing Context
Features
SPECIFIC, MEASURABLE,
CONTEXTUALISED ACTION-
ORIENTED
ENVIRONMENT, AUDIENCE,
TIMEFRAMES ...
HIGH-LEVEL FEATURE
CONCEPTS, SPECIFIC
IMPLEMENTATIONS ...

16. Goals
● Tourist attraction
● More awareness of
Cambridge history /
significance
DESIGN PARAMETERS
Metrics
Framing Context
Features
● Tour company inclusion
● Pre-test / Post-test of
local history
● Traffic-flow assessment
● Social media buzz
ENVIRONMENT, AUDIENCE,
TIMEFRAMES ...
HIGH-LEVEL FEATURE
CONCEPTS, SPECIFIC
IMPLEMENTATIONS ...

17. Instant Art Generator ™
1. Kinetic
2. Projection
3. Light-based
4. Physical interaction
5. Multiple pieces
6. Historical
7. Phone interaction
8. Large scale
9. Persistent changes
10. Consistent start state
11. All-weather
12. University collaboration

18. Go design

19. Activity: Design a civic art installation
15 mins
Goals
Brainstorm a potential design for an informative civic art installation
Activity
● Use google to get inspiration for art installations that use your
constraints
● Consider what historically significant or helpful facts about
Cambridge could be represented in a public installation
● Come up with as many ways as possible to represent those facts
● Converge on a set of design “features” for your installation
Deliverable
An informative civic art installation, described in “features”, potentially
supported by rough sketches

20. What’s your riskiest
assumption?

21. To the hypothesis canvas!

22. We believe that ... For ...
Will lead to ... Because...
SOME FEATURE OR CHANGE AN ENVIRONMENT OR CONTEXT
SOME SPECIFIC,
MEASURABLE CHANGE
YOUR RATIONALE
HYPOTHESIS CANVAS

23. Go
Hypothesise

24. Goals
Form a testable hypothesis around one aspect of your art concept
Activity
● Pick the metric you want to effect
● Pick your riskiest assumption / feature
● Select your experimental context
● Think through your experimental setup
● Articulate your experimental rationale
Deliverable
A well-formed, testable hypothesis, allowing you remove uncertainty /
validate an assumption around one “feature” of your design product
Activity: Form a hypothesis
10 mins

25. How’d it go?

26. Designers & Diagnosis
& Dragons

27. Some Role-play Heads = Failure
Tails = Ambiguous

28. What impacts
experimental
outcomes?
● Bias
● Confounding factors
● Chance!
● True effect

29. Common mistakes
● Not everything deserves a hypothesis
● When to abandon (sunk cost)
● Check sample size & source
○ (67% conversion on 3 actions out of 10 million users)
● Are you asking the right question?
● Is every test succeeding / failing?
○ (check for bias or poor systematic design)
● Bad data foundations
● Technical challenges

30. What
happened?

31. Activity: Diagnose errors in your experiment
10 mins
Goals
Brainstorm a wide range of possible ways your experiment might have
yielded an ambiguous or unsuccessful result
Activity
● Revisit & reassess your underlying assumptions
● List any ways your assumptions could be wrong
○ Don’t underestimate human stupidity
● Step through your experiment setup
● List the possible errors in that setup
Deliverable
A list of possible incorrect assumptions and experimental errors

32. NOW what’s your riskiest
assumption?

33. Let’s talk about
prioritisation
● Go back to your metrics
● What are your assumptions?
● What are you uncertain about?
● What are the risks?
● What are the costs to test?

34. Uncertainty
Risk
TRACK TEST
IGNORE MITIGATE
Low High
High

35. Risk
Cost
TEST ASSESS
ASSESS IGNORE
Low High
High

36. What’s the next Hypothesis?

37. We believe that ... For ...
Will lead to ... Because...
SOME FEATURE OR CHANGE AN ENVIRONMENT OR CONTEXT
SOME SPECIFIC,
MEASURABLE CHANGE
YOUR RATIONALE
HYPOTHESIS CANVAS

38. Goals
Create a better-informed, testable hypothesis around one aspect of your art
concept
Activity
● Consider what you’ve learned thus far about your context
● Pick the metric you want to effect
● Pick your riskiest assumption / feature
● Select your experimental context
● Think through your experimental setup
● Articulate your experimental rationale
Deliverable
A well-formed, better-informed, testable hypothesis, allowing you remove
uncertainty / validate an assumption around one “feature” of your design product
Activity: Form a new hypothesis
5 mins

39. What
questions do
you have?

40. What did we learn?

41. Metrics &
Hypotheses &
Prioritisation &
Iteration
● What makes a good metric
● How to design an effective
hypothesis
● The essentials of
Prioritisation
● Experimental Iteration

42. Further Reading
http://experimentationhub.com/hypothesis-kit.html
+

43. Thanks! Chris Massey & Lisa Long
@camassey | @beforeyoucode
chris@mindtheproduct.com
lisa@beforeyoucode.com

44. Bonus round

45. How did it go this time?

46. Some more role-play Rock = Success
Paper = Failure
Scissors = Ambiguous

47. Presentation to Design Council
… in one week
… to secure more £££

48. What do you do?

49. How to present
experimental
findings
Context, Context, Context
Your sources
Your sample size
Your assumptions
Your experiment design
Your findings