This document outlines an approach for continuous experimentation and improvement. It recommends starting with data to identify high traffic, low conversion areas to improve. Form hypotheses about potential solutions and test them through small experiments. Document learnings to establish best practices and avoid regressions. The goal is to continuously run small tests to drive learning and improvements.
18. Simple 4-step experimentation process
Data
Figure out what to improve
Hypothesis
Make an educated guess
Experiment
Test your guess
Act
Decide whatās next
30. Increase Conversions
By A/B Testing
Do a Happy Dance
But Watch it Closely
Get More Traļ¬c
So You Can Start A/B Testing
Get More Traļ¬c
So You Can Do a Happy Dance
High Traļ¬c
Low Traļ¬c
High ConversionsLow Conversions
How To Figure Out What To Improve
31. What percentage of your
customers or users have
adopted each feature?
Intercom on Product Management
When youāve already got usage, analyze it.
32. Where do you have the most
visitors, users, customers,
emails, or notiļ¬cations?
More volume helps you test faster.
33. ą¹ Website Traļ¬c / App Installs
ą¹ Welcome Emails
ą¹ Web / Mobile Onboarding
ą¹ Email Digests
ą¹ Triggered Notiļ¬cations
ą¹ Cart / Funnel Abandonment
ą¹ Dormant / Churned users
Common areas with large volume and high impact potential
47. Use simple equations to assess potential impact of each idea
DAN MCKINLEY HTTP://KISS.LY/DANETSY
visits
Ā x
Ā conversion
Ā rate
Ā x
Ā order
Ā value
Ā x
Ā li4
Ā =
Ā ?
users
Ā x
Ā order
Ā value
Ā x
Ā resurrec7on
Ā rate
Ā =
Ā ?
ESTIMATED
ESTIMATED
MODELED
MODELED
48. Assess the potential impact,
difficulty and priority of
each test idea you have.
Prioritize your tests
60. 1. Your starting hypothesis.
2. Dates for when you ran the test.
3. Screenshots of control and all variations used during the test.
4. Expected change in conversion.
5. Probability of change.
6. Screenshot of the raw data used in your analysis.
7. A screenshot to the A/B Test Report in KISSmetrics.
8. The decision you've made from the test.
9. What you learned from the test.
Document every A/B test and create a playbook
61. What learnings do we want to
carry over from the past?
Helps you avoid unknowingly
decreasing your conversions.
62. ā Keep the footer links
ā Autofocus on the ļ¬rst form ļ¬eld
ā Do not include social proof
ā Link to the features page
ā Use āLog inā copy on the bu on
Create experiment checklists based on learnings