The document outlines a methodology for web optimization that focuses on testing content to enhance user experience and drive business value. It emphasizes the importance of data analysis, segmentation, and personalized content delivery to improve conversion rates and ROI. Key strategies include A/B testing, multivariate testing, and a culture of continuous measurement and learning to meet audience needs effectively.

2. TEST AND LEARN
How experimenting with content
improves the user experience and
drives business value
OCTOBER 2012

3. MARKETING
SCIENCES
ANALYTICS MONETIZATION OPTIMIZATION

4. METHODOLOGY
OUR METHODOLOGY IS
DESIGNED TO ANSWER QUESTIONS
RATHER THAN BUILD REPORTS
Where should I invest my marketing dollars?
How does my digital media relate to offline purchase?
What is the value of my customers?
How can I increase conversion?
Who should I target?
Who are my most valuable customers?
What is the impact of my social media strategy?

5. ANALYTICS
Goals and KPI definition
Technical and organizational support
for measurement
Holistic view of what is working
and what is not
Create and use an ROI-driven system
to get smarter

6. MONETIZATION
ASSIGNING MONETARY VALUES TO
DIFFERENT SITE BEHAVIORS TO
UNDERSTAND AND QUANTIFY THE
OVERALL VALUE OF THE WEB
CHANNEL.
BENEFITS
1. Measure total website value and track value over time
2. Understand how website actions relate to future profit
3. Evaluate return on marketing spend
4. Create a common language within the organization by
using a common value for stakeholders to report on
5. Project prioritization

7. COMMON TRAITS OF OUR MOST SUCCESSFUL CLIENTS
Digital Channel Goals are imperative
Analyze performance based on overall business goals
Understand data holistically, not in silos and not just digital
Focus on opportunities and recommendations
Monetize all key digital behaviors
Prioritize opportunities
Measure & quantify relationships between brands and people
Take action with an optimization program
Avoid average experiences

9. OPTIMIZATION
op ti mize [op-tuh-mahyz]
Verb
1. To make as effective, or useful as possible.
Website Optimization delivers relevancy to customers,
ROI to clients
Improve site performance (business objectives)
Bounce
Engage more users Rate
Increase sales and profits
KPIs
Improve lead generation and opt-ins
Validate assumptions
Discover what content resonates with target audience(s)

10. Right Right Right
Optimisation
Message Audience Time
Copy First time visitors Evening Performance
Imagery Return visitors Weekend improvement
Call-to-Action Paid Search traffic End of Month

11. TESTING DIFFERENT DESIGN AND CONTENT VERSIONS TO DETERMINE WHICH IS
MOST EFFECTIVE
c
o
n
EXISTING v
CONTENT e B
r
s
i A
o
n
s
ALTERNATE
Days
CONTENT
Visitors randomly split Each group is shown Results are analyzed &
into groups a unique version winner determined

12. TECHNIQUES
A/B/n Testing Single element testing
Audience size
Multivariate Testing (MVT) Multiple element testing
Complexity
Behavioral Targeting Serve content by visitor segment
Predictive Modeling People who did this are likely to do that
Personalization Serve content unique to visitor
Optimise messaging across
Multi Channel fixed web, mobile, tablet, call centre, etc

13. EVOLUTIONARY PROCESS
A/B/n and
Multivariate
Testing
Guessing,
Behavioral Segmentation,
Before and
Targeting Personalization,
After Testing
etc.
Multi-Channel

15. ONE SIZE DOES NOT FIT ALL
THE PERFORMANCE OF THE GENERAL POPULATION
LIKELY DOES NOT REPRESENT THE BEHAVIOUR
OF KEY ONLINE BUSINESS SEGMENTS
Making single answer decisions based on one bucket of visitors may miss
opportunities for large incremental benefits
Optimising content for key segments transforms a good website to a
great online experience

16. WE BELIEVE
SUCCESS COMES FROM UNDERSTANDING
AND MEETING THE NEEDS OF THE AUDIENCE
We use data to understand customer journeys, motivations, and what messages are
working and what are not
We translate this data into insights to create personalised and targeted conversations
across touch points
We can demonstrate how establishing a culture of measurement and testing has a
revolutionary impact on marketing strategies and performance

17. Environment
Variables
Site Behavior
Variables
Referrer
Variables
Temporal
Variables
Offline
Variables

18. Environment
Variables
IP address +
Country of origin +
Time zone +
Device type +
Operating system +
Browser type +
Screen resolution +

19. Site Behavior
Variables
Customer / Prospect +
New/return visitor +
Previous visit patterns +
Previous Product interests +
Searches +
Previous online purchases +
Previous Campaign exposure +
Previous Campaign responses +

20. Temporal
Variables
Time of day +
Day of week +
Recency +
Frequency +

21. ANSWERING BUSINESS QUESTIONS THROUGH TESTING
Enhance visitor HOW DOES HOMEPAGE CONTENT IMPACT BOUNCE RATE AND
engagement and SITE ENGAGEMENT?
traffic flow
Improve WHAT PAGE CONTENT ENGAGES VISITORS BEST?
messaging
Increase Form HOW MANY MORE LEADS COULD WE GENERATE WITH SIMPLER
Completion
FORMS?
Provide relevant / HOW CAN WE REDUCE BOUNCE AND INCREASE VALUE OF PAID TRAFFIC?
personalized
messages HOW CAN WE BETTER ENGAGE VISITORS FROM SOCIAL NETWORKS?
Drive purchasing
WHEN AND WHERE SHOULD CROSS-SELLS BE INTRODUCED?
TO WHOM AND WHEN SHOULD DISCOUNTS AND INCENTIVES BE OFFERED?

22. WEB
OPTIMIZATION
METHODOLOGY

23. FOUNDATION
Deepen engagement
Boost conversions
TEST Increase value of media spend
RESULTS
Teaches what resonates/performs best
with target audience
TEST DESIGN AND
VARIANT CREATION
IDENTIFY AND PRIORITIZE
TESTING OPPORTUNITIES
ESTABLISH SITE GOALS AND SUCCESS METRICS
(KEY PERFORMANCE INDICATORS)

24. WEB OPTIMISATION PROCESS
1. ASSESSMENT
Define Objectives
Primary goals of site
Goals of individual sections of site
2. TEST STRATEGY
1 2
Develop hypothesis
Define Success metrics
Determine test methodology Assessment Test Strategy
Create metrics plan
Calculate test duration
RESULTS
3. EXECUTION
Create alternates
Tool set-up Analysis Execution
Launch tests
3 4
4. RESULTS ANALYSIS
Analyze performance by variant
Analyze performance by segment
Recommendations/Next steps
Plan follow up tests

25. TESTING
AND TARGETING
CASE STUDIES

26. A/B TEST EXAMPLE: DELL iNSPIRON NOTEBOOK LANDING PAGE
Control
OBJECTIVE
Increase click thru rate
ACTION TAKEN
Removed GOOD, BETTER, BEST design
elements
Reduced copy to call out only 3 features
Changed background from blue to white
RESULTS Winner
25% CTR lift
61% increase in revenue

27. A/B TEST EXAMPLE: SIMPLIFIED TEST DRIVE FORM
Control Existing Page Alternate
3
1
Reformatted fields to make form appear
2 shorter/consider shading
Reduced the prominence of the reset button
Added relevant image
Results: 129% form completion increase

28. A/B TEST EXAMPLE: FORD OF RUSSIA HOMEPAGE ROTATION
OBJECTIVE: Increase engagement and buying activities
Control Winner
RESULTS:
8% reduction in bounce rate
13% increase in buying activities

29. TEST EXAMPLE: NOKIA ACCESSORIES CROSS SELL
OBJECTIVE: Improve merchandising of accessories to increase conversion rate and RPV
Product Page Cart Page
RESULTS:
Removal of Accessories on Brand product pages delivered 60% lift in RPV
Adding Accessories to Cart page increased purchase conversion rate by 5%

30. MULTIVARIATE TEST EXAMPLE: HELIO OFFER PAGE
Control
OBJECTIVE
Reduce landing page
11 abandon rate
3
2 ACTION TAKEN
4 Redesigned call to action to
better inform customers
480 total variants
RESULTS
Winner 57% higher CTR
130% sales conversion rate
increase

31. RIGHT PANEL OPTIONS
A (Control) D (Name/Prices on phones)
B (Big Sexy Image) E (Fa-La-La-La-La!)
C (Lead with Offer)

32. RIGHT PANEL OPTIONS
A (Control) B (More text + price) C (more text, no price)
D (no right image or button)

33. BUTTON OPTIONS
LEFT PANEL RIGHT PANEL
A (Control)
A (Control)
B (Choose Device)
B (Red Button)
C (Green Choose Device)
C (Learn More)
D (Bold Choose Device)
D (No Right Button)
E (Green Buy Now)
F (Bold Buy Now)

34. TARGETING EXAMPLE: DELL SMB RETARGETING
Control
OBJECTIVE:
Increase purchase conversion rate and RPV
ACTION TAKEN
Targeted visitors who previously self-identified
themselves as laptop shoppers with laptop content
AREAS TESTED:
Content & layout in center of page
Products promoted
A/B/n test (9 variants)
RESULTS:
24% lift in revenue per visit Winner
19% lift in purchase conversion rate
Significant annual monetized lift

35. EMAIL OPTIMISATION: YAHOO GAMES
OFFER TEST:
Discount Games Popular Games
Discounted games vs Popular games
RESULTS:
The popular games version significantly
outperformed the discounted titles:
18% higher open rate
91% higher click through rate
107% more orders
300% higher revenue
35

36. CONTENT CHANGES ARE JUST THE BEGINNING

37. THE QUESTION IS NOT CAN WE SERVE DIFFERENT CONTENT
Copy? Layout? Offer?
Imagery? CTA?
New Visitors? Repeat Purchasers?
Paid Search Traffic?

38. PROGRAM NOT PROJECT

39. WEB
OPTIMIZATION
TEST VALUATION/PRIORITIZATION

40. WHAT WILL I GET IF I RUN THIS TEST?
In order to prioritize we need an estimated impact of the test.

41. HOW TO CHOOSE?
5%? 10%? 20%?
What is the existing conversion rate?
How epic is the scale of the test variables?
How consistent is the lift relative to other tests?
Would you spend the time on the test at 5%?
Would you think 20% unrealistic?

42. WHAT WILL I GET IF I RUN THIS TEST?
TWO GOALS:
Show opportunity
Show range
Upside Summary
Project Check out page redesign
Goal Increase percent conversion rate on final page of purchase funnel
Monthly Visits 45,000
Current Conversion 5.00%
Project Cost $45,000
Monthly 12-Month
Conversion Change in Total Units Incremental Incremental Incremental
rate Coversion Sold Units Sold Profit Profit 12-Month ROI
5.00% 2,250 - - -
5.05% 0.05% 2,273 23 $2,813 $33,750 (0.25)
5.10% 0.10% 2,295 45 $5,625 $67,500 0.50
5.15% 0.15% 2,318 68 $8,438 $101,250 1.25
5.20% 0.20% 2,340 90 $11,250 $135,000 2.00
5.25% 0.25% 2,363 113 $14,063 $168,750 2.75
5.30% 0.30% 2,385 135 $16,875 $202,500 3.50
5.35% 0.35% 2,408 158 $19,688 $236,250 4.25
5.40% 0.40% 2,430 180 $22,500 $270,000 5.00
5.45% 0.45% 2,453 203 $25,313 $303,750 5.75
5.50% 0.50% 2,475 225 $28,125 $337,500 6.50
5.55% 0.55% 2,498 248 $30,937 $371,250 7.25
5.60% 0.60% 2,520 270 $33,750 $405,000 8.00
5.65% 0.65% 2,543 293 $36,562 $438,750 8.75
5.70% 0.70% 2,565 315 $39,375 $472,500 9.50
5.75% 0.75% 2,588 338 $42,188 $506,250 10.25
5.80% 0.80% 2,610 360 $45,000 $540,000 11.00
5.85% 0.85% 2,633 383 $47,813 $573,750 11.75
5.90% 0.90% 2,655 405 $50,625 $607,500 12.50
Expected Range of Values Expected Case

43. PLANNING & PRIORITIZATION
Follow the goals
Follow the revenue
Test Plan/Roadmap

44. PLANNING & PRIORITIZATION
In addition to ROI/lift
Consider implementation risks:
Ease of technical solution
CONSIDER EXECUTE
Time/production costs
Measurability of results
RETURN ON INVESTMENT (ROI)
AVOID CONSIDER
EASE OF IMPLEMENTATION

45. WEB
OPTIMIZATION
TEST OR REDESIGN

46. Plan Test Analyze Plan Test Analyze
Plan Test Analyze Plan Test Analyze
Plan Test Analyze Plan Test Analyze
Plan Test Analyze Plan Test Analyze

47. OPTIMIZATION OR REDESIGN
Fix Replace

48. OPTIMIZATION OR REDESIGN
THE THEORY OF THE LOCAL MAXIMUM:
It is as effective as its ever going to be on its current structural foundation.
Even if you make 100 tweaks you can only get so much improvement.

49. ASSUMING A REDESIGN FIRST IS EXPENSIVE
LOOK FOR STAGNANT OR DECLINING MEASUREMENT OVER TIME.
USER RESEARCH
RADICAL MODULAR OPTIMIZATION FAILS TO MOVE THE NEEDLE.
CIRCUMSTANTIAL EVIDENCE FROM COMPETITORS.

50. WEB
OPTIMIZATION
MOVING BEYOND A ONE-SIZE FITS ALL EXPERIENCE
PROVIDES OPPORUNITIES FOR CREATIVE

51. ARE YOUR BEST IDEAS BEING DISCARDED?

52. 90% 10%
EXISTING ALTERNATE
CONTENT CONTENT

54. IS IT REALISTIC TO ANTICIPATE THE PREFERENCES OF HUNDREDS
OF THOUSANDS OF INDIVIDUALS?

55. START BY ANALYZING RESULTS BY SEGMENT

56. BUILDING A CULTURE OF TESTING AND MEASUREMENT
GOALS FIRST, TESTS SECOND
FOSTER AN OBSESSION WITH OUTCOMES
PREPARE TO BE SURPRISED WHEN YOUR CUSTOMERS CHOOSE THEIR
FAVOURITES
EXPERIMENTATION HAS A REVOLUTIONARY IMPACT ON MARKETING
STRATEGIES
SEGMENTED TESTS ARE SMARTER TESTS
MAKE IT A PROGRAM, NOT A PROJECT
DEMONSTRATE HOW INSIGHTS ARE BEING INCORPORATED THROUGHOUT

57. WEB
OPTIMIZATION
OPEN QUESTION/DISCUSSIONS?

58. THANK
YOU
PETER ERNST
Associate Director of Optimization
Possible
peter.ernst@possible.com