Upcoming SlideShare
×

# Boost Website Performance

765 views
692 views

Published on

Increasing conversions multivariate testing to maximize efficiency of lead generation campaigns

Boost Website Performance with Statistical Design of Experiment

Web page elements
Body text
Bullet list, charts, graphs, videos, pictures, callouts, or similar
Promotion
Registration Form, call-to-action
Site search
Footer
User reviews

We identified elements and the level of attention
We still don’t know which ones have bigger impact on conversion
What happens if we change one or more of them?
How do we optimize these for boosting performance?

Solution:

Statistical Controlled Experiment where variations are present
Determine the best performing combination of variations
Run a validation test on the recommendation to confirm

Controlled Experiment in 1747 in England
Testing cure for scurvy by James Lind, a surgeon

Fisher started with application to agriculture – 1918-1940
Factorial Design and ANOVA

Industrial Era: 1950-1970
Application in Chemical and process industries
Second Industrial Era: 1970-1990
Wider application in Quality Control to most industries

How do we apply this to boost website performance?

Select elements that are important or attract more attention
Elements should be as much independent as possible
Variations are created for each element
2/3 apart from the existing (CONTROL)
Develop web pages containing combination of variations
ALL possible variations or a SMALLER subset
These web pages are run along with Control
Traffic volume for creating statistically significant result is directed to these pages
Performance of each web page is measured
Statistical analysis yields which combination of variation is expected to yield the highest results
Develop a final page with recommended combination
Run this in parallel with CONTROL to validate results

Factors in choosing elements, variations and design
Independence, traffic volume, traffic sources, time, cost
Experiment Design: Full Factorial (all possible combination) or Partial (selected)

Example:
A Full Factorial Design with 3 elements with 3 levels of variations each will need 3^3 = 27 web pages to be tested

Traditional A/B Testing will also require 27 pages

Partial Factorial Design (say in Taguchi Design) will need a L9 array with 9 web pages only
Run duration should be enough to
Generate statistically significant results
Cover cyclical and important variations in environment

2 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
765
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
1
0
Likes
2
Embeds 0
No embeds

No notes for slide

### Boost Website Performance

1. 1. WELCOME Boost Website Performance withStatistical Design of Experiment 1
2. 2. Website Optimization Develop designs based on experience  Dated? Slow! Inputs from focus groups  Limited and expensive Optimization based on web analytics  Big step forward  Not granular enough-still making assumptions? What next?  What are elements that go on my web page  How do they interact/affect user behavior  Do these elements have the same impact on users If not which ones lead the pack  What happens if we change one or more of these? Confidential | The next-generation online customer acquisition engine
3. 3. Example of elements Web page elements  Header  Headline  Sub-headline  Body text  Bullet list, charts, graphs, videos, pictures, callouts, or similar  Promotion  Registration Form, call-to-action  Navigation  Site search  Links on the page  Footer  User reviews Confidential | The next-generation online customer acquisition engine
4. 4. Web pages are collection of elements: that work together! Location of search barLeft side Model selectornavigation Display pricing Add to cart Display shipping and price Image controls & guarantee video © 2010 Confiden1tial | think innovation
5. 5. Use Heat Maps To identify elements that are getting visitors attention on web pages
6. 6. In Page Analytics: More quantitative Identifies elements AND level of attention from visitors
7. 7. Web page elements: What Next? We identified elements and the level of attention We still don’t know which ones have bigger impact on conversion What happens if we change one or more of them? How do we optimize these for boosting performance?Solution: Statistical Controlled Experiment where variations are present Determine the best performing combination of variations Run a validation test on the recommendation to confirm Confidential | The next-generation online customer acquisition engine
8. 8. Statistical Controlled Experiment: Reaching back in time Controlled Experiment in 1747 in England  Testing cure for scurvy by James Lind, a surgeon Fisher started with application to agriculture – 1918-1940  Factorial Design and ANOVA Industrial Era: 1950-1970  Application in Chemical and process industries Second Industrial Era: 1970-1990  Wider application in Quality Control to most industries How do we apply this to boost website performance? Confidential | The next-generation online customer acquisition engine
9. 9. High Level Design of Experiment (DOE) Process Select elements that are important or attract more attention  Elements should be as much independent as possible Variations are created for each element  2/3 apart from the existing (CONTROL) Develop web pages containing combination of variations  ALL possible variations or a SMALLER subset These web pages are run along with Control  Traffic volume for creating statistically significant result is directed to these pages Performance of each web page is measured  Statistical analysis yields which combination of variation is expected to yield the highest results Develop a final page with recommended combination  Run this in parallel with CONTROL to validate results Confidential | The next-generation online customer acquisition engine
10. 10. Selecting the right Design Factors in choosing elements, variations and design  Independence, traffic volume, traffic sources, time, cost  Experiment Design: Full Factorial (all possible combination) or Partial (selected)Example:A Full Factorial Design with 3 elements with 3 levels of variations each will need 3^3 = 27 web pages to be tested Traditional A/B Testing will also require 27 pages Partial Factorial Design (say in Taguchi Design) will need a L9 array with 9 web pages only Run duration should be enough to  Generate statistically significant results  Cover cyclical and important variations in environment Confidential | The next-generation online customer acquisition engine
11. 11. Case Study of an E-Commerce portal Objectives  To increase the Shopping Cart creation Rate Google Analytics data revealed that the Product Details page were dropping visitors Planned DOE on the Product Details Page  Selected 7 elements with 2 variations each  Partial Factorial Design lead to L8 array (8 pages to test) Combinations were run sequentially  Each combination ran for 6 days taking 10%-15% of traffic  The page ran in parallel with CONTROL
12. 12. Possible List of variables identified1. Add to Cart placement2. Displaying price information3. Members star ratings4. Image and video controls5. Related Products6. Left hand navigation7. Member Actions – Email to friends, Add to wish list, Price Alerts8. Product search barWe finally selected the first 7 variables with one variation eachPartial Factorial design based on Taguchi required 8 pages © 2010 Confiden1tial | think innovation
13. 13. Example of Page Element Layout Link to Slide 6 TV1 Link to Slide 10 TV4 TV6Link to Slide 6 Link to Slide 9 TV2 13
14. 14. Experiments Run Experiment Run ID Variable Combinations Name Experiment-1 Test Run-2 C-TV1,C-TV2,C-TV3,C+1-TV4,C+1-TV5,C+1-TV6,C+1-TV7 Experiment-2 Test Run-3 C-TV1,C+1-TV2,C+1-TV3,C-TV4,C-TV5,C+1-TV6,C+1-TV7 Experiment-3 Test Run-4 C-TV1,C+1-TV2,C+1-TV3,C+1-TV4,C+1-TV5,C-TV6,C-TV7 Experiment-4 Test Run-5 C+1-TV1,C-TV2,C+1-TV3,C-TV4,C+1-TV5,C-TV6,C+1-TV7 Experiment-5 Test Run-6 C+1-TV1,C-TV2,C+1-TV3,C+1-TV4,C-TV5,C+1-TV6,C-TV7 Experiment-6 Test Run-7 C+1-TV1,C+1-TV2,C-TV3,C-TV4,C+1-TV5,C+1-TV6,C-TV7 Experiment-7 Test Run-8 C+1-TV1,C+1-TV2,C-TV3,C+1-TV4,C-TV5,C-TV6,C+1-TV7 Taguchi New Test C+1-TV1,C-TV2,C+1-TV3,C+1-TV4,C+1-TV5,C+1-TV6,C+1-TV7Recommendation 14
15. 15. Experiment-5: C+1-TV1,C-TV2,C+1-TV3,C+1-TV4,C-TV5,C+1-TV6,C-TV7Affected VariablesTV3: Members star ratings onthe Product details pageTV1(position change): - Add to Cart link - Price information display - Free shipping and lowestprice guaranteeTV4: View image and video linkTV6: Left side navigation links
16. 16. Experiment-4: C+1-TV1,C-TV2,C+1-TV3,C-TV4,C+1-TV5,C-TV6,C+1-TV7Affected VariablesTV3: Members star ratings onthe Product details pageTV1(position change): - Add to Cart link - Price information display - Free shipping and lowestprice guaranteeTV5: Related products – displaylogicTV7: Member control Links
17. 17. 1. Significance Test Analysis The Significant Test shows that Experiment 5 and Experiment 4 are ahead of the others EXPExp Number Mean. EXP St. Dev. Control Control Z- P-value # of Runs SC Cr. SC Cr. Mean SC Cr. St. Dev. score (accept H0) Rank 1 6 5.06% 0.69% 4.60% 0.80% 1.41 0.0700 4 2 6 4.83% 0.17% 4.68% 0.61% 0.60 0.5600 7 3 6 4.89% 0.42% 4.70% 0.36% 1.29 0.2000 6 4 6 5.27% 0.84% 4.67% 0.24% 6.12 0.0002 2 5 6 5.54% 0.98% 4.67% 0.38% 5.61 0.0002 1 6 6 4.90% 0.22% 4.48% 0.44% 2.34 0.0200 3 7 6 4.79% 0.65% 4.49% 0.45% 1.63 0.1000 5 A higher z-score means that the data is farther away from the population mean (CONTROL) 17
18. 18. 2. Taguchi Design AnalysisThe Taguchi design analyzed the variables, and for each variable itcompared Control(1) vs. its Variant(2) and calculated Signal to Noiseratio for each experimentMean – The mean gives the average SC creation rateSN Ratio – Signal-to-Noise ratio is a measure of how predictable is the SC creation rate. A high SN ratiomeans that the Standard Deviation for that given EXP is proportionally smaller than the mean. A lowerSN ratio means that data for the SC creation rate is relatively unpredictable.Experiment 5 and Experiment 4 have the highest and the second highest SN ratio,meaning that their SC creation rate is higher, robust and more predictable. 18
19. 19. 2. Taguchi Design AnalysisImpact of elements & variations: Choice 1 (Control) vs. Choice 2(Variant) for each of the 7 variablesNote that changes in variable 1 pushed the shopping cart creation rate the most whilechanges in the second variable caused the rate to drop 19
20. 20. 2. Recommendations Based on Taguchi Design AnalysisTaguchi recommended variable combination for the most performantShopping Cart creation rate as follows Variable Recommendation TV_1 2 TV_2 1 TV_3 2 TV_4 2 TV_5 2 TV_6 2 TV_7 2 NOTE: Taguchi however will not predict how much better than CONTROL the above hypothesized template will perform. We recommend creating a design based on this recommendation and running it for validation. 20
21. 21. Results SC Creation Rate moved from 4.48% to 5.02% after validation Number of carts created increased from 500 to 560 per day Average Cart value moved from \$155 to \$169 Revenue increased by 170K a month
22. 22. Things to keep in mind Variables to be kept as independent from each other as possible  Avoid confounding Keep environmental variables as much constant as possible  Avoid introducing new traffic sources Select more variables if you have traffic  Keep the number of variation of each limited to 2-3 Traffic should be randomly selected  Load balancer should not introduce any bias  Should not mix traffic between different variations and CONTROL
23. 23. Regalix Online MarketingWhy partner with RegalixIndustry Best Practices & FrameworkRegalix has developed social media framework and best “The ComplianceOnline case study offers insight into how technologypractices leveraging experience from Fortune 1,000 as well marketers can make a vision for a viable online community — thatas venture backed customers across multiple industry embraces customers, partners, and prospects alike — become real.”verticals including Hi-tech., Retail, Manufacturing, Education, – Laura Ramos, B2B Analyst, Forrester ResearchClean Tech. and Financial Services.Full-service Execution “Regalix has social mediaRegalix provides strategy, program management, creative, marketing experience that hastechnology, and operations services. This reduces the needfor engaging and managing multiple service providers. been leveraged on behalf of blue- “Citibank views Regalix as a chip Fortune 100 firms. For this trusted strategic partner. They reason, I think that you’ll get far have consistently deliveredSuperior Client-service more bang for your buck out of innovative solutions ofRegalix’s client services team consists of industry experts and leveraging them as your turn-key exceptional quality over the lastpractitioners that have created significant and successful marketing team than you would couple of years of ourmarketing programs. out of a single FTE. engagement. Their reporting dashboards are Regalix has proven experience in also first-rate. Super stream lined supporting an enterprise of our – super time efficient.” size and diversity, while respecting the stringent quality - Alyssa Rapp, CEO Bottlenotes standards that we have set.” 23
24. 24. Regalix Online MarketingAbout UsFull-Service: Digital Marketing & Technology ServicesCompany; Strategy, Creative, Campaigns, Technology,Communities Collateral & ThoughtTalent: Leadership, Advisory, 175+ Team LeadershipCustomers: Fortune 500 and Venture-Backed ContentFocus: B2B, B2C, C2CVerticals: Retail, Hi-Tech., Finance, HealthcareGlobal: HQ in Silicon Valley, 4 Offices Website, Community Portals,Background: Built on 8+ years of research & Social iPhone, iPad MediaIndustry Recognition: applications Digital Strategy Customer & Lead Prospect Generation Nurturing & Sales 24
25. 25. Companies that trust us Retail & Financial Film & Professional Healthcare, Education Hi-Tech. & Clean-Tech.,Consumer Services Entertainment Services Bio-Tech. Software Bio-FuelProducts UCLA ICICI BANK 25
26. 26. Thank You 26
27. 27. Landing Page Optimization Overview Goal: Increase Lead Conversion Rate (CR)  Use Regalix’s ROITM Intelligent Design of Experiment (DOE)  Create a best page with higher CR for the same online traffic  Create a continuous optimization program for Citibank Activity 1: Plan for DOE  Break the landing pages into individual elements  Create variations on each element for testing  Create DOE test and identify the number of pages/element combinations necessary for testing  Create landing pages with the necessary combinations Activity 2: Conduct DOE  Set-up landing pages on ROI  Run DOE and collect statistically significant data Activity 3: Analyze data and build best page Activity 4: Continuous improvement Confidential | The next-generation online customer acquisition engine
28. 28. Activity 3: Analyze data and build best page Feed data into statistical system and use Regalix’s proprietary algorithm to calculate the following:  The tested page elements that influence the conversion rate  The specific variations of the elements that worked best  The combination of elements that could provide a significant higher conversion rate Final landing page is built using the best element and their variation New Page is deployed  Data is collected to establish higher CR  Optionally to isolate any market dynamics that is likely to influence the CR: The New Page will be run simultaneously with the Older Page to get a direct measure of improvement New page is ready for wider rollout Confidential | The next-generation online customer acquisition engine