The document summarizes an A/B test of different versions of a fantasy football landing page. The LONG version provided extensive information without mentioning price, while the PRICE version prominently displayed the price with minimal other information. Statistical analysis found that LONG had a higher rate of league creation, while PRICE had a higher rate of paid league creation. Eye tracking and user testing found mixed preferences among versions. The study recommends testing combinations of LONG and PRICE approaches in future iterations.

1. Fantasy Football Commissioner Split Test
CBS Interactive Research and Analytics
September, 2008

2. Purpose
 Evaluate effectiveness of four different versions of fantasy football landing page
 LONG: Print ad style presentation: Graphically intensive while providing extensive product information without
messaging price. Bright blue call to action button “Start a Free Trial Now” at the bottom of the text.
 PRICE: Simple presentation with bold call out of price: Minimal graphics and minimal product information
provided with a text link call to action. Price is messaged loudly in the header graphic.
 REG: Control page: This is the page that would have been used without testing (nearly the same as v2007).
 SIMPLE: Simple presentation: Same page as PRICE – minimal graphics, minimal product information, price is not
mentioned.
LONG PRICE REG SIMPLE
Confidential – do not distribute 2

3. Research Methodology
 Randomly present viewers with one of the four versions
when they get to the fantasy football landing page
 Results showed each version was exposed to about 25% of total
unique viewers each week
 Evaluate effectiveness of different versions on three metrics:
1. % of viewers who created leagues (secondary)
 The higher the %, the better
2. % of viewers who created and paid for leagues (primary)
 The higher the %, the better
3. Average page views per viewer
 This metrics is ambiguous: More average page views could mean more
interest, but that could also mean the page is more confusing so that
viewers need to come back to review. It will not be reviewed
Confidential – do not distribute 3

4. Key findings
 It is obvious that LONG had higher % of created leagues while
PRICE enjoyed higher % of paid leagues this season.
 Yet the % differences are not very large.
 For future seasons, what can be inferred?
 LONG is very likely (99.99%) to generate higher % of created leagues
 PRICE has a high probability (about 90%) to create higher % of paid
leagues than SIMPLE and REG.
 Although PRICE has higher % of total paid leagues than LONG (0.07%),
the chance that the difference is not significantly different from zero is
very high, at 44%.
 Recommendations:
 Use either LONG or PRICE for fantasy football landing page.
 Try to understand why LONG generate higher % of created leagues but
maybe lower % of paid leagues than PRICE.
 Test variations/combinations of LONG/PRICE with next product cycle.
Confidential – do not distribute 4

5. LONG had higher % of created leagues consistently
6.44%
Total 5.64%
5.56%
5.66%
5.01%
7/27/2008 4.07% LONG
4.29%
4.35%
PRICE
5.18%
8/3/2008 4.19%
4.39% REG
4.47%
SIMPLE
7.18%
8/10/2008 6.29%
6.40% LON
6.35%
PRI
8.29%
8/17/2008 7.39% REG
6.71%
6.80% SIM
7.42%
8/24/2008 6.40%
6.84%
6.32%
6.16%
8/31/2008 5.68%
5.44%
5.76%
3.81%
9/7/2008 3.79%
3.15%
3.68%
0% 1% 2% 3% 4% 5% 6% 7% 8% 9%
Confidential – do not distribute 5

6. Most stat tests showed LONG had higher % of created leagues
Date Page UV % Created Lg Page UV % Created Lg % Diff Prob of Indiff
27-Jul-08 LON 10,284 5.01% PRI 10,263 4.07% 0.94% 0.13%
27-Jul-08 LON 10,284 5.01% REG 9,948 4.29% 0.72% 1.57%
27-Jul-08 LON 10,284 5.01% SIM 10,572 4.35% 0.66% 2.47%
3-Aug-08 LON 12,471 5.18% PRI 12,414 4.19% 0.99% 0.02%
3-Aug-08 LON 12,471 5.18% REG 11,664 4.39% 0.79% 0.41%
3-Aug-08 LON 12,471 5.18% SIM 12,612 4.47% 0.71% 0.89%
10-Aug-08 LON 12,522 7.18% PRI 12,369 6.29% 0.89% 0.51%
10-Aug-08 LON 12,522 7.18% REG 12,207 6.40% 0.78% 1.46%
10-Aug-08 LON 12,522 7.18% SIM 12,804 6.35% 0.83% 0.85%
17-Aug-08 LON 10,815 8.29% PRI 10,401 7.39% 0.90% 1.48%
17-Aug-08 LON 10,815 8.29% REG 10,221 6.71% 1.58% 0.00%
17-Aug-08 LON 10,815 8.29% SIM 10,653 6.80% 1.49% 0.00%
24-Aug-08 LON 12,882 7.42% PRI 12,867 6.40% 1.02% 0.12%
24-Aug-08 LON 12,882 7.42% REG 12,639 6.84% 0.58% 7.32%
24-Aug-08 LON 12,882 7.42% SIM 13,038 6.32% 1.10% 0.05%
31-Aug-08 LON 12,588 6.16% PRI 12,435 5.68% 0.48% 10.83%
31-Aug-08 LON 12,588 6.16% REG 12,252 5.44% 0.72% 1.51%
31-Aug-08 LON 12,588 6.16% SIM 12,723 5.76% 0.40% 18.39%
7-Sep-08 LON 12,666 3.81% PRI 12,738 3.79% 0.02% 95.46%
7-Sep-08 LON 12,666 3.81% REG 12,567 3.15% 0.66% 0.46%
7-Sep-08 LON 12,666 3.81% SIM 12,105 3.68% 0.13% 61.61%
Total LON 80,313 6.44% PRI 79,803 5.64% 0.80% 0.00%
Total LON 80,313 6.44% REG 77,874 5.56% 0.88% 0.00%
Total LON 80,313 6.44% SIM 80,604 5.66% 0.78% 0.00%
Confidential – do not distribute 6

7. PRICE produced higher % of overall paid leagues
3.34%
Total 3.41%
3.26%
3.25%
2.72% LONG
7/27/2008 2.58%
2.47%
2.44%
PRICE
2.76%
8/3/2008 2.80% REG
2.71%
2.70%
SIMPLE
3.89%
8/10/2008 4.03%
4.04% LON
3.83%
PRI
4.38%
8/17/2008 4.53% REG
3.94%
3.98% SIM
4.02%
8/24/2008 3.90%
4.04%
3.83%
2.96%
8/31/2008 3.26%
3.20%
3.13%
1.64%
9/7/2008 1.82%
1.44%
1.71%
0% 1% 1% 2% 2% 3% 3% 4% 4% 5% 5%
Confidential – do not distribute 7

8. Statistically, PRICE had higher % of total paid leagues than REG and SIMPLE
Date Page UV % Paid Lg Page UV % Paid Lg % Diff Prob of Indiff
27-Jul-08 PRI 10,263 2.58% LON 10,284 2.72% -0.14% 53.06%
27-Jul-08 PRI 10,263 2.58% REG 9,948 2.47% 0.11% 62.09%
27-Jul-08 PRI 10,263 2.58% SIM 10,572 2.44% 0.14% 51.34%
3-Aug-08 PRI 12,414 2.80% LON 12,471 2.76% 0.04% 82.95%
3-Aug-08 PRI 12,414 2.80% REG 11,664 2.71% 0.09% 65.59%
3-Aug-08 PRI 12,414 2.80% SIM 12,612 2.70% 0.10% 63.05%
10-Aug-08 PRI 12,369 4.03% LON 12,522 3.89% 0.14% 57.92%
10-Aug-08 PRI 12,369 4.03% REG 12,207 4.04% -0.01% 96.04%
10-Aug-08 PRI 12,369 4.03% SIM 12,804 3.83% 0.20% 43.44%
17-Aug-08 PRI 10,401 4.53% LON 10,815 4.38% 0.15% 60.73%
17-Aug-08 PRI 10,401 4.53% REG 10,221 3.94% 0.59% 3.69%
17-Aug-08 PRI 10,401 4.53% SIM 10,653 3.98% 0.55% 4.87%
24-Aug-08 PRI 12,867 3.90% LON 12,882 4.02% -0.12% 62.26%
24-Aug-08 PRI 12,867 3.90% REG 12,639 4.04% -0.14% 56.26%
24-Aug-08 PRI 12,867 3.90% SIM 13,038 3.83% 0.07% 78.13%
31-Aug-08 PRI 12,435 3.26% LON 12,588 2.96% 0.30% 15.81%
31-Aug-08 PRI 12,435 3.26% REG 12,252 3.20% 0.06% 77.11%
31-Aug-08 PRI 12,435 3.26% SIM 12,723 3.13% 0.13% 53.74%
7-Sep-08 PRI 12,738 1.82% LON 12,666 1.64% 0.18% 27.39%
7-Sep-08 PRI 12,738 1.82% REG 12,567 1.44% 0.38% 1.68%
7-Sep-08 PRI 12,738 1.82% SIM 12,105 1.71% 0.11% 50.58%
Total PRI 79,803 3.41% LON 80,313 3.34% 0.07% 43.66%
Total PRI 79,803 3.41% REG 77,874 3.26% 0.15% 10.51%
Total PRI 79,803 3.41% SIM 80,604 3.25% 0.16% 7.11%
Confidential – do not distribute 8

9. User Testing
 The Fantasy Football Commissioner landing pages were shown to (7)
participants during a usability test in August, 2008.
 An Eye Tracking device (Tobii) was used to record where participants
looked on the page during a 30 second review period.
 After reviewing the pages, participants were asked to give their reactions
and rank order the landing pages to determine first, second, third, and
fourth preference.
Confidential – do not distribute 9

10. Eye Tracking Data Longer eye fixations Shorter eye fixations
LONG PRICE REG SIMPLE
 Eye tracking data can be tricky to collect and interpret.
 With 7 participants, generalization is limited.
 However, it is worth nothing that the blue buttons on the right rail (the area user’s
are expected to click to sign-up/create league) were not visually attended to in
this study. Note: Is it possible that this area of the screen suffers from Ad
avoidance due to the prominence of ads in this area on websites in general?
Confidential – do not distribute 10

11. User Testing – Rank Order Preference
LONG PRICE REG SIMPLE
 Over half of the participants (4) ranked REG as their first preference - a
somewhat unexpected result and counter to the split test data.
 (2) participants ranked LONG as their first preference.
 Expectations within the CBS team anticipated a preference for shorter versions
(PRICE and SIMPLE); however, participants may feel that full disclosure is not
provided by the shorter versions.
 Knowing the price of a product bears importance as seen in the split test.
Confidential – do not distribute 11

12. Next Steps
 Testing for Fantasy Baseball 2009
 Hybrid Commissioner page
 Right rail navigation
 Continue testing for Fantasy Football 2009
 Refine Commissioner test
 Free Fantasy Football
 Fantasy Football Premium Games
 Incorporate additional methodologies
 Usability assessments
Confidential – do not distribute 12