The document discusses the importance of experimentation as a key driver for innovation and growth in businesses. It outlines a structured approach to conducting digital experiments, which involves creating hypotheses based on data, testing them, and analyzing results to improve decision-making. The author highlights common misconceptions around experimentation and emphasizes the need for sufficient data volume and collaboration among teams for successful outcomes.

1. Innovation & Growth
With Experimentation
March 2023
By Haley Carpenter | Founder

2. User research &
experimentation
Consulting, execution, & training
mychirpy.com
haley@mychirpy.com
512-200-3510
https://www.linkedin.com/in/haley-carpenter-490/
Haley Carpenter
Founder
Past: CXL/Speero, Optimizely, Hanapin (now Brainlabs)

3. Every business is looking for
innovation and growth.
That’s obvious.

4. Innovate
“make changes in something established, especially by
introducing new methods, ideas, or products”
Grow
“become larger or greater over a period of time;
increase”
Google Definitions

5. How does your team decide on
what changes to make?

6. We make educated
guesses.
Common answer.

7. “We think…”
“We feel…”
“We believe…”

8. “We think…”
“We feel…”
“We believe…”

9. How does your team determine
increases?

10. We’re not measuring anything…
we don’t know what to choose.
Common answer.

11. Our data is broken.
Common answer.

12. Maybe you’re getting lucky.
That’s not strategic winning.

13. Experimentation can be a
primary driver of both
innovation and growth.
That’s maybe not so obvious.

14. Experimentation…
● Eliminates
guesswork
● Validates decisions
● Minimizes bias
● Mitigates risk

15. Experimentation will lead you
to strategic winning.

16. Experimentation you say?
What is that?

17. “Digital experimentation is similar, if not identical,
to the scientific method.”
“Businesses attempt to answer a question by
establishing a hypothesis, testing the hypothesis
through experimentation and analyzing the
results.”
“…experimentation is ‘the reliable process of
delivering winning digital experiences without
guesswork or risk…”
Contentful Link

18. Image Source
50% of traffic 50% of traffic

19. How do we start an
experiment?

20. Precursor:
Make sure you have
enough data volume.

21. Pre-test calculations…
one of the most important
things you can do.

22. WATCH THIS

23. Step 1:
Get a data point →
Create a hypothesis

24. What’s a Data Point?
Where Do I Get One?

25. What’s a Data Point?
Where Do I Get One?
RESEARCH

26. Source

27. Research Experiments
Source

28. Source: Speero
Link

29. Research will lead you to
strategic winning, too .

30. Good starting point: analytics
(e.g., Google Analytics, Amplitude, Heap, Adobe Analytics, Mixpanel, etc.)

31. Example data point:
The dropout rate from the
cart to the checkout is 92%.

32. Ideal: data points from 2+
methodologies

33. Example data points:
The dropout rate from the cart to the
checkout is 92%.
Users don’t know about the free return
policy or money-back guarantee
policy.

35. Step 2:
Get a data point →
Create a hypothesis

36. “We know X. If we do A,
then B will happen because of C.”
Hypothesis Format

37. “We know X. If we do A,
then B will happen because of C.”
Research & data Change something
Increase result

38. We know the dropout rate from the cart to the
checkout is 92% and that users don’t know about the
free return policy or the money-back guarantee policy.
If we do A, then B will happen because of C.
Hypothesis Format

39. We know the dropout rate from the cart to the
checkout is 92% and that users don’t know about the
free return policy or the money-back guarantee
policy. If we add content about the free return policy
and money-back guarantee policy on the cart page,
then B will happen because of C.
Hypothesis Format

40. We know the dropout rate from the cart to the
checkout is 92% and that users don’t know about the
free return policy or the money-back guarantee
policy. If we add content about the free return policy
and money-back guarantee policy on the cart page,
then the dropout rate from the cart to the checkout
will decrease and transactions will increase because
of C.
Hypothesis Format

41. We know the dropout rate from the cart to the
checkout is 92% and that users don’t know about the
free return policy or the money-back guarantee
policy. If we add content about the free return policy
and money-back guarantee policy on the cart page,
then the dropout rate from the cart to the checkout
will decrease and transactions will increase because
friction will decrease due to users feeling there is less
risk to place an order.
Hypothesis Format

42. We know the dropout rate from the cart to the
checkout is 92% and that users don’t know about the
free return policy or the money-back guarantee
policy. If we add content about the free return policy
and money-back guarantee policy on the cart page,
then the dropout rate from the cart to the checkout
will decrease and transactions will increase because
friction will decrease due to users feeling there is less
risk to place an order.
Hypothesis

43. “We know X. If we do A,
then B will happen because of C.”
Guessing with little to no
accurate data involved
Change something
Unclear metrics and
broken data are common
????????
?

44. Step 3:
Get a data point →
Create a hypothesis →
Create a variation design

45. Design
Image Source

46. Dev > QA
Launch > Monitor
Conclude > Analyze
Step 4+:

47. Result: +61%
Action: Implement the variation
Insight for what’s next: Should we
make the policies more visible
elsewhere, too?
RUN ANOTHER TEST!

48. Source

49. “We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
“We know X. If we do A, then B will happen because of C.”
Test 1
Test 2
Test 3
Test 4
Test 5
Test 6
Test 7
Test 8
Test 9
Test 10
Test 11
Test 12
Test 13
Test 14
Test 15
Test 16
Test 17
Test 18
Test 19
Test 20
Test 21
Test 22
Test 23
Test 24
Test 25
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight
Insight

53. Wins
+
Learnings

54. Exp
Leaders
Product
Marketing
Other teams

55. A few final topics

56. Prioritization
Article Link

57. Roadmapping

58. Common Misconceptions
● We’ll hack it together and be okay
● You just need a testing tool and an
idea
● You don’t need developers
● Every business can test
● It’s too hard
● It’s easy

62. Recap
● You need to experiment if you have enough data volume.
(Check by running pre-test calculations.)
● Starting an experiment
Step 1: Get a data point to inform your test idea.
Step 2: Write a hypothesis. (e.g., If…then…because)
Step 3+: Roadmap & prioritize your initiatives.
● The more teams involved in experimentation the better.
● If you’re not experimenting and doing user research,
your competitors are ahead of you.