This document outlines the framework for creating a data-driven experiment from start to finish. It includes identifying the problem being solved, forming a hypothesis based on qualitative and quantitative data, selecting target audiences and test locations, determining the primary and secondary metrics to track, and establishing the duration and proposed change for the experiment. The next steps would be to prioritize the experiment against other ideas and determine the effort required to design, build, and test the changes.

1. Experiment Idea Framework
Create a Data-Driven Experiment
Idea from Start to Finish
Qualitative data pool
What customer feedback, user research, or other qualitative input do we have?
Hypothesis
Write out what you expect to happen based on the change
you’re experimenting with a clear, data-informed rationale.
Secondary metrics
What secondary metrics will we
track as part of the experiment?
Next Steps
• Prioritize this experiment against other ideas we have.
• Determine the level of effort to design, build, and QA
the experiment.
• Communicate plans for the experiment.
Targeting
Which audiences will be a
part of the experiment?
Duration
What is the longest the experiment
will need to run, based on our
statistical significance calculations?
Quantitative data pool
Which analytics data do we have to support our decision of what to experiment?
Top metric
Top metric you’re using to measure success:
revenue, NPS, or something else.
Test location
Pick a specific page, flow, or experience
that could affect the top metric.
Proposed change
Describe a change you’d like
to experiment.
OPTIMIZELY EXPERIMENTATION TOOLKIT
What problem are we trying
to solve?
Describe a high-friction experience, an
underperforming landing page, or another
strategic issue you’re working on.