FFC Landing Page Optimization ProjectVersion: 2.0                                           May 29, 2008
FFC Landing Page Optimization ProjectDocument InformationTitle:                           FFC Landing Page Optimization pr...
FFC Landing Page Optimization ProjectTable of Contents1.     Project Overview ...............................................
FFC Landing Page Optimization Project1.   Project Overview      The Fantasy Sports industry has become increasingly compet...
FFC Landing Page Optimization Project      It should also be noted that this in-house testing solution is in its infancy a...
FFC Landing Page Optimization Project4.   Methodology      The basic methodology for the FFC Landing Page Optimization pro...
FFC Landing Page Optimization Project        As currently conceived, a few design changes will be introduced on the landin...
FFC Landing Page Optimization Project4.4 FFC Landing Page Exposure     In theory, there is no set rule for setting the tra...
FFC Landing Page Optimization Project5.   Primary Metric for Declaring a Winner      The primary metric proposed to declar...
FFC Landing Page Optimization Project               weeks). To be sure your test is showing a statistically meaningful imp...
FFC Landing Page Optimization Project10. Iterative A/B Split Testing        Given the methodology outlined here, an iterat...
FFC Landing Page Optimization ProjectCampbell, J. (2008, January 21). With A/B Testing - You Dont Think, You Know. Retriev...
FFC Landing Page Optimization ProjectAppendix I: Fantasy Acquisition Landing Page                         Fantasy Acquisit...
FFC Landing Page Optimization ProjectAppendix II: FFC Landing Page - Unregistered User or Registered but Not Signed In    ...
FFC Landing Page Optimization ProjectAppendix III: FFC Landing Page – Registered User, Signed In                          ...
Upcoming SlideShare
Loading in...5
×

CBS Sports landing page optimization methodology names removed

867

Published on

Published in: Sports, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
867
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

CBS Sports landing page optimization methodology names removed

  1. 1. FFC Landing Page Optimization ProjectVersion: 2.0 May 29, 2008
  2. 2. FFC Landing Page Optimization ProjectDocument InformationTitle: FFC Landing Page Optimization projectAuthor: Robert SherronCo-Author: removedStatus: Draft In Review FinalTarget Start Date: July 15, 2008 approximatelyTarget End Date: September 16, 2008 approximatelyChange HistoryVersion Distribution Date Description of Changes1.0 5/15/2008 Initial draft document.2.0 5/30/2008 Updates made as per 5/28 conference call.Project Team Members Name Group removed Research & Audience Analytics removed Research removed Business Analytics removed Technology, CBSSports.com removed Marketing Robert Sherron User Experience/Usability removed Sales Strategy & Audience Analytics TBD Quality Assurance Page 2
  3. 3. FFC Landing Page Optimization ProjectTable of Contents1. Project Overview .................................................................................................................................. 42. Challenge/Opportunity ......................................................................................................................... 43. Project Goals and Objectives ................................................................................................................ 5 3.1 FFC Objectives ............................................................................................................................... 5 3.2 IT/Infrastructure Objectives .......................................................................................................... 54. Methodology ........................................................................................................................................ 6 4.1 Basic A/B Split Test........................................................................................................................ 6 4.2 Modified A/B Split Test for FFC ..................................................................................................... 6 4.2.1 Testing Benefits ................................................................................................................. 7 4.2.2 Testing Limitations ............................................................................................................ 7 4.3 FFC Landing Page Alternatives ...................................................................................................... 7 4.4 FFC Landing Page Exposure........................................................................................................... 85. Primary Metric for Declaring a Winner................................................................................................. 9 5.1 Additional Metrics for Analysis ..................................................................................................... 96. Measurement System ........................................................................................................................... 97. Sample Size and Traffic Volume............................................................................................................ 98. Test Duration ...................................................................................................................................... 109. Analysis and Reporting ....................................................................................................................... 1010. Iterative A/B Split Testing ................................................................................................................... 1111. Complimentary Testing Strategies...................................................................................................... 1112. Open Issues and Considerations ......................................................................................................... 1113. Useful Information and References .................................................................................................... 11Appendix I: Fantasy Acquisition Landing Page ........................................................................................... 13Appendix II: FFC Landing Page - Unregistered User or Registered but Not Signed In ............................... 14Appendix III: FFC Landing Page – Registered User, Signed In .................................................................... 15 Page 3
  4. 4. FFC Landing Page Optimization Project1. Project Overview The Fantasy Sports industry has become increasingly competitive – the landscape includes organizations of all sizes – from large media operations (ESPN, YAHOO!, Fox Sports) and the league itself (NFL) with millions of unique users down to small operations with user counts in the thousands. The flagship of CBSSports.com is Fantasy Football Commissioner (FFC) - a league manager product that allows individuals to organize their league using robust customization capabilities and features. FFC is the most profitable of all fantasy football product offerings and is the primary focus of marketing efforts. FFC is subscription based and must be re-purchased each year. Annual league (customer) retention surpasses 90%; nonetheless, what can be done to promote new league creation (new customers)? The project here-in, will explore landing page optimization and testing strategies for FFC to better understand and promote customer conversion in 2008.2. Challenge/Opportunity As customer acquisition becomes more difficult each year due to outside competition in the Fantasy arena, growth opportunities have become increasingly important for CBSSports.com. Despite 90% FFC league retention annually for existing customers, new growth must also be fostered to sustain a leading industry position. To this end, CBSSports.com recognizes the importance of landing page design and its impact on attracting new customers and converting them to subscribers. In 2003, CBSSports.com (then CBSSportsline) collaborated with Optimost (now owned by Interwoven) to examine landing page optimization via Mulitvariable testing. As a result of this effort, modifications were made to landing pages based on feedback and data collected. Since that time, this type of testing has not been conducted on Fantasy product pages. After some open dialog between the Fantasy FFC, Research, and Usability groups, we believe there is an opportunity to modify and improve the current FFC landing page and grow the customer base. Given the effort and expense to work with an outside vendor at this time, an in- house (CBS Interactive) testing solution is being formulated to conduct landing page optimization testing. The challenge will be to understand the conversion elements for the current FFC landing page and create alternative designs to discover which design elements are the keys to customer conversion: What page elements (e.g., copy text, graphics, buttons, etc.), and their placement on a landing page promote this process? Page 4
  5. 5. FFC Landing Page Optimization Project It should also be noted that this in-house testing solution is in its infancy and will therefore be an effort of trial and error, discovery, and ultimately success!3. Project Goals and Objectives The goal of this project will be to formulate and implement a testing strategy to discover design elements that attract new customers and promote conversion - signup for the FFC product. Ultimately we want to provide an optimized landing page. It will also serve as a proving ground for an in-house landing page optimization process which currently is not in practice within CBS Interactive.3.1 FFC Objectives The goals specific to the FFC product are as follows. a. Determine the landing page variables to manipulate and then create alternative FFC landing pages (colors, graphics, copy, element placement, etc.). b. Determine metrics to capture on each landing page (e.g., time spent on a landing page and the number of new league creations for each landing page alternative). c. Determine how to aggregate and report metrics. d. Test, analyze, and determine which landing page designs (and the specific elements on the page) promote increased conversion rates. e. Quantify landing page optimization results (e.g., [n%] conversion rate)3.2 IT/Infrastructure Objectives The goals specific to IT/Infrastructure are as follows. a. Implement A/B Split Testing infrastructure and methodology b. Determine how to measure each metric Page 5
  6. 6. FFC Landing Page Optimization Project4. Methodology The basic methodology for the FFC Landing Page Optimization project is to identify design elements that we believe can be modified to promote increased conversion (new league creation). These design elements are then incorporated into alternative landing pages. The current landing page (baseline) and the alternative landing pages are then shown to web site visitors (live) such that each landing page is exposed fairly and uniformly for a specified duration. The conversion rates for each landing page are calculated and examined to determine the winner – that is…the landing page design that promotes the best conversion rate (FFC league creations). Figure 4-1: Generic Conversion Illustration4.1 Basic A/B Split Test The simplest A/B Split test involves creating two designs (A and B) of a landing page. Version A is the control – usually the existing landing page. Version B incorporates a single change so that the effect of that change can be tracked and translated into a conversion rate. Landing page traffic is diverted (split) to each version equally over the same time period for a specified testing duration. At the end of the test duration, the conversion rates for A and B are compared to one another and evaluated to determine which version performed best.4.2 Modified A/B Split Test for FFC The specific method planned for the FFC Landing Page Optimization project will be a modified A/B Split Test. This method incorporates more than one design alternative (A/B/C/D), “A” being the control and compares conversion rates between the landing pages. Page 6
  7. 7. FFC Landing Page Optimization Project As currently conceived, a few design changes will be introduced on the landing pages and their impact on conversion will be examined. Customer exposure to designs that are deemed risky can be diverted to a small percentage of site visitors, monitored, and removed if performing poorly in order to limit negative brand impact. Note: Some of the literature on A/B testing offers conflicting advice. Some experts vigorously suggest that only one small design change be introduced and compared to a baseline page while others recommend radical and multiple design departures from the baseline page. The trade off is the precision to which you can infer the benefits of a specific design change – with one change you can definitively measure its impact on conversion whereas with multiple radical changes you are less certain of which design element(s) accounts for the conversion impact. On the other hand, radical changes could expose a design alternative that converts quite well but would otherwise not be tested in a conservative testing scheme.4.2.1 Testing Benefits a. Measures actual behavior b. Measures small performance differences with a high confidence interval given large sample sizes c. Prescriptive tool to resolve competing design choices4.2.2 Testing Limitations a. Reasons for why one design performed better than another may not be discernable b. When users decided not to convert (signup) we don’t know why – this knowledge can be just as meaningful for design improvements4.3 FFC Landing Page Alternatives Further discussion is warranted to brainstorm on all of the design elements that could be manipulated on the FFC landing page. The following is a preliminary list of potential design elements that could be manipulated: a. Headline color and copy b. Call to action buttons , links, and graphics c. Graphics d. Color (images, background, etc.) e. Configuration/Placement of page elements (e.g., login field, button, promotion, etc.) f. Header vs No Header g. Variations in product description, page copy, etc. Page 7
  8. 8. FFC Landing Page Optimization Project4.4 FFC Landing Page Exposure In theory, there is no set rule for setting the traffic ratio diverted to landing pages in order to test conversion. For our purposes, 10% of the traffic will be sent to the Baseline group which will see the current FFC landing page design. The remaining 90% of the traffic is then equally and randomly diverted to a Test group consisting of three alternative landing pages and the current FFC landing page. Each page in the Test group receives approximately 22.5% of the traffic from this slice. The current landing page is repeated in both groups to confirm the baseline conversion rate. The following illustration outlines the proposed landing page traffic diversion for landing page designs. Divert 10% of Current FCC 10% Baseline Group traffic to Landing Page A Test Group Current FCC ≈22.5 Landing Page A % Alternative ≈22.5 Landing Page B Divert 90% of % traffic equally and randomly to Alternative ≈22.5 Landing Page C % Alternative ≈22.5 Landing Page D % Figure 4-2: Traffic Ratio for FFC Landing Pages A conservative approach would be to divert traffic in a 90/10 ratio (90% Baseline/10% Test). Depending on how radical or risky the landing page changes are, you can control exposure to limit negative brand impact. Given that the project team is confident that the design alternatives will not have a negative brand impact, the traffic ratio has been set to 10/90. This ratio will expose each design to a high traffic volume and help to ensure a proper statistical sample size. Higher sample sizes will promote confidence in the results. Once a visitor is diverted to a specific landing page, that visitor should see the same landing page on all subsequent visits. Common practice is to use a cookie to track the computer used to access the page. Page 8
  9. 9. FFC Landing Page Optimization Project5. Primary Metric for Declaring a Winner The primary metric proposed to declare a winner will be the landing page design with highest conversion rate (%) for new FFC league creations. # of New League Creations - Paid Conversion Rate % = x 100 # of Landing Page Views (A/B/C/D)5.1 Additional Metrics for Analysis The following additional metrics should be collected and analyzed to gain insight into landing page design elements and their influence on the conversion process: a. Unique visitors – records conversion rate b. Time of day conversions – examine peak conversion time or pattern c. Day of week conversions – examine peak conversion time or pattern d. Length of time spent (seconds) on each landing page – is there something about the page that is confusing or not optimally written or is the page engaging?6. Measurement System Omniture will be used to collect the page view and unique visitor data. Business Objects will be used to collect the data for paid leagues created for each design.7. Sample Size and Traffic Volume The sample size needed for significance for this project will be determined by the statistic used in the analysis. Given the high volume of traffic expected during the test duration, we should have a large enough sample size to validate the results and determine if there are significant differences. The following website may be helpful: Sample Size Calculator For thought, here is a quote from Bryan Eisenberg of FutureNow: “… you need to decide your sample size and set up the criteria for success. To decide your ultimate sample size, run a "null" test with your A/B test. The null test is really just an A/A test, where you are running the control against itself to determine where the convergence of results matches up (typically within 0.05 percent of each other, but thats up to you). When the tests converge, youll have an idea of the volume of traffic you need to test. You decide your criteria for success based on a certain number of conversions or sales, or you can measure results over course of time (two Page 9
  10. 10. FFC Landing Page Optimization Project weeks). To be sure your test is showing a statistically meaningful impact on the variables, you have to know if youve demonstrated enough of a difference (delta) between the tests to declare a clear winner. As a rule of thumb, you should have at least a three times larger result (e.g., if A is 5, B should be 15).” Here is another website that may be helpful: Split Test Calculator and Decision Tool The following figures show traffic volume for the 2007 season: Figure 7-1: 2007 Page Views for FFC Month Average Daily Page Views June 2,163 2007 July 2,679 August 6,850 Figure 7-2: 2007 Page Views for FFC8. Test Duration As planned, the testing effort will start on or about July 15th, 2008 and end on September 12th, 2008.9. Analysis and Reporting TBD. Page 10
  11. 11. FFC Landing Page Optimization Project10. Iterative A/B Split Testing Given the methodology outlined here, an iterative testing strategy could yield a better understanding of landing page optimization than one single test. Nonetheless, for this project a single test is planned. If a true winner is declared at the end of testing, discussion will be given to tweaking that design with iterative, small changes to refine it in the future.11. Complimentary Testing Strategies As a compliment to the knowledge gained in this A/B Split test, other user testing methods can help quantify and qualify user behavior on the FFC landing page. The following additional user testing methods are being considered: a. Eye Tracking: The Interactive lab in Las Vegas, NV has a Tobii Eye Tracker that could be used to collect data on what people see on each landing page and where they are looking on the page (and where they are not). b. Usability Test: Real-world users can be recruited and asked to find, navigate to, and interact with a landing page for the FFC product. Throughout the test the participant is asked to provide qualitative feedback.12. Open Issues and Considerations 1. Request Q/A person(s). 2. Determine which variables will be manipulated on alternative landing pages and when that Creative will be available (add to document when ready). 3. Confirm that all required data can be collected and transformed into a data file that can be imported into SPSS (Statistical Package for the Social Sciences). 4. Determine the sample size and confidence intervals for this test.13. Useful Information and ReferencesAlt, B., Usborne, N., & McGlaughlin, F. (2005 , August 15). A/B Split Testing. Retrieved from MarketingExperiments Journal: http://www.marketingexperiments.com/improving-website-conversion/ab-split-testing.htmlBuerba, S., & Chiu, L. (2008, March 6). Let Your Customers Decide: 5 Principles of Using A/B andMultivariate Testing. Retrieved from omniture.com:http://www.omniture.com/offer/164?fc=true&v_id=337856 Page 11
  12. 12. FFC Landing Page Optimization ProjectCampbell, J. (2008, January 21). With A/B Testing - You Dont Think, You Know. Retrieved fromFindResolution: http://www.findresolution.com/2008/01/with-ab-testing-you-dont-think-you-know.htmlEisenberg, B. (2004, May 7). A/B Testing for the Mathematically Disinclined. Retrieved from The ClickZNetwork: http://www.clickz.com/showPage.html?page=3349901Eisenberg, B. (2005, April 29). How to Improve A/B Testing. Retrieved from The ClickZ Network:http://www.clickz.com/showPage.html?page=3500811Eisenberg, B. (2007, February 16). We Tried That Already. Retrieved from The ClickZ Network:http://www.clickz.com/showPage.html?page=3624994Fortin, M. (Unknown). Splitting Hairs Over Split-Testing? Retrieved from michelfortin.com:http://www.michelfortin.com/splitting-hairs-over-split-testing/Fortin, M. (Unkown). Taguchi Explained… For Dummies. Retrieved from michelfortin.com:http://www.michelfortin.com/taguchi-explained-for-dummies/Meyers, P. J. (2007). Measuring Usability: The Basics. Retrieved from usereffect:http://www.usereffect.com/measuring-usability-basics.pdfNielsen, J. (2005, August 15). Putting A/B Testing in Its Place. Retrieved from useit.com:http://www.useit.com/alertbox/20050815.htmlQuarto-vonTivada, J. (2006). AB Testing Too Little, Too Soon? Retrieved from Future Now:http://www.futurenowinc.com/abtesting.pdfUnknown. (2008, April 15). Improving Conversions by Split Testing. Retrieved from Internet MarketingUniversity: http://www.internetmarketinguniversity.com/blog/?p=168Wilson, R. F. (2008). How to Develop a Landing Page. Retrieved fromhttp://www.wilsonweb.com/ebooks/landing.htm Page 12
  13. 13. FFC Landing Page Optimization ProjectAppendix I: Fantasy Acquisition Landing Page Fantasy Acquisition Landing Page Page 13
  14. 14. FFC Landing Page Optimization ProjectAppendix II: FFC Landing Page - Unregistered User or Registered but Not Signed In Page 14
  15. 15. FFC Landing Page Optimization ProjectAppendix III: FFC Landing Page – Registered User, Signed In Page 15

×