How Long Should My Test Run???
• The data rate (number of conversions per
• Size of improvements found (Percentage
• Size of your test (Number of alternative
• The confidence in your answer ( How sure you
need to be )
• The data rate decribes how quickly you collect
data during your test.
• The volume of traffic for landing page
optimization tests is best measured in the
number of conversion actions per day ( and
not the number of unique visitors).
• You can decrease the size of the test. This can
be done by decreasing the granularity of your
Size of Improvements Found
• If you have managed to uncover a clearly
superior version of your landing page, the performance improvement would quickly become
• often, an initial round of changes will fix some of
the obvious problems and improve performance
• Since you do not know the size of the possible
improvements ahead of tme, the length of the
time required for the tuning test may vary
Size of your Test
• The size of your test can be measured by the size
of the search space that you are considering.
• The search space is the whole universe of the
alternative designs possible in your test.
• A simple head to head test has a search space
size of 2 ( the originl, and the alternative landing
page version that you are testing.)
• If you are testing multiple elements on the page,
you need to multiply together the number of
alternative versions for each one.
• When the setting for one variable in your test
positively or negatively influences the setting of
another variable is variable interaction.
• If they have no effect then they are said to be
• In a positive interaction, 2 variables create a
• In a negative interaction, 2 variables undercut
each other and cancel out some of the individual