Web Analytics: A Measurement System of Website Performance for Optimization Dave Rogers Former Senior Web Analytics Researcher @ PayPal

Who is Dave Rogers? Almost a decade in research and internet 1/3 Researcher, 1/3 Analyst, 1/3 Psychologist Education: MA Stats/Research; MBA Industry experience in academia, agency work, e-commerce, finance and social networking Was @ PayPal for over a year Source: ‘mutt’ idea stolen from Angel Morales; LightsOut Marketing answer: a mutt

Almost a decade in research and internet

1/3 Researcher, 1/3 Analyst, 1/3 Psychologist

Education: MA Stats/Research; MBA

Industry experience in academia, agency work, e-commerce, finance and social networking

Was @ PayPal for over a year

Discussion Balkans & the Internet Web Optimization Ecosystem Website Optimization Web Analytics Resources Q&A

Balkans & the Internet

Web Optimization Ecosystem

Website Optimization

Web Analytics Resources

Q&A

Balkans & the Internet Great time to get started! Tools are all figured out (consolidations) Many agencies now (available experts!) Compete on analytics (hunt vs. get hunted) -4 -2 0 -3 ? ? The Balkans

Great time to get started!

Tools are all figured out (consolidations)

Many agencies now (available experts!)

Compete on analytics (hunt vs. get hunted)

Web Optimization Ecosystem

Web Site Optimization Ecosystem Source: Web Analytics Demystified (www.webanalyticsdemystified.com) WA Web Analytics VOC Voice of the Consumer TESTING Conversion Optimization CEM Customer Experience Management

Web Analytics The WHAT: Visits / Visitors Bounce/ Exit rates Marketing tasks Acquisition SEO and SEM Affiliate Networks Comparison Shopping Retention Email Affiliate Networks Comparison Shopping SEM SEO WA VOC TESTING CEM

The WHAT:

Visits / Visitors

Bounce/ Exit rates

Marketing tasks

Acquisition

SEO and SEM

Affiliate Networks

Comparison Shopping

Retention

Email

Affiliate Networks

Comparison Shopping

Voice of the Consumer The WHY: Measure customer satisfaction continuously Identify what’s driving customer satisfaction and what will have the greatest impact on behavior and financial performance WA VOC TESTING CEM

The WHY:

Measure customer satisfaction continuously

Identify what’s driving customer satisfaction and what will have the greatest impact on behavior and financial performance

Customer Experience Management REPLAYS: Tealeaf enables you to see your customers' actual online experiences, analyze their motivations and ultimately, gain insight as to why abandonment or other site actions occur WA VOC TESTING CEM

REPLAYS:

Tealeaf enables you to see your customers' actual online experiences, analyze their motivations and ultimately, gain insight as to why abandonment or other site actions occur

Testing OPTIMIZING: Typical consumer engagement model from Marketing team perspective: Now, who owns the conversion task? Acquisition Marketing Retention Marketing WA VOC TESTING CEM Acquisition Marketing Conversion Optimization Retention Marketing

OPTIMIZING:

Website Optimization

What is MV Testing? Framework in 8 Steps Case Study Advanced Testing Techniques Ok… now how do I start? Bonus: Elements to Start Testing Today! Optimization Discussion

What is MV Testing?

Framework in 8 Steps

Case Study

Advanced Testing Techniques

Ok… now how do I start?

Bonus: Elements to Start Testing Today!

What is MV Testing? … Why? What is average conversion rate?

What is MV Testing? Framework in 8 Steps Case Study Advanced Testing Techniques Ok… now how do I start? Bonus: Elements to Start Testing Today!

What is MV Testing?

Framework in 8 Steps

Case Study

Advanced Testing Techniques

Ok… now how do I start?

Bonus: Elements to Start Testing Today!

Website Testing Framework Split traffic All Traffic Version A Version B Technology

Split traffic

Website Testing Framework Split traffic Display content www.books.com www.books.com Version A (Original, Control) Version B (Test)

Split traffic

Display content

Website Testing Framework Split traffic Display content Log which version

Split traffic

Display content

Log which version

Website Testing Framework Split traffic Display content Log which version Log success event

Split traffic

Display content

Log which version

Log success event

Website Testing Framework Split traffic Display content Log which version Log success event Measure results Conversion Rate = # of success events # of ‘starts’ / version

Split traffic

Display content

Log which version

Log success event

Measure results

Website Testing Framework Split traffic Display content Log which version Log success event Measure results Use stats!

Split traffic

Display content

Log which version

Log success event

Measure results

Use stats!

Website Testing Framework Split traffic Display content Log which version Log success event Measure results Use stats! Decide winner

Split traffic

Display content

Log which version

Log success event

Measure results

Use stats!

Decide winner

Website Testing Framework Split traffic Display content Log which version Log success event Measure results Use stats! Decide winner Push winner live Rinse & Repeat !

Split traffic

Display content

Log which version

Log success event

Measure results

Use stats!

Decide winner

Push winner live

What is MV Testing? Framework in 8 Steps Case Study Advanced Testing Techniques Ok… now how do I start? Bonus: Elements to Start Testing Today!

What is MV Testing?

Framework in 8 Steps

Case Study

Advanced Testing Techniques

Ok… now how do I start?

Bonus: Elements to Start Testing Today!

Test Plan : Improve the efficiency of the product detail page Success Metric : Percent of product page views that result in shopping cart “puts” The test consisted of 8 elements, or variables, that can take on 18 different values. The resulting search space contains 576 unique combinations of variable values (‘recipes’). ‘ Small changes often have huge impact ’

Test Plan : Improve the efficiency of the product detail page

Success Metric : Percent of product page views that result in shopping cart “puts”

The test consisted of 8 elements, or variables, that can take on 18 different values. The resulting search space contains 576 unique combinations of variable values (‘recipes’).

8 Elements Page Order Fold Line

Page Order

8 Elements Page Order Title

Page Order

Title

8 Elements Page Order Title Price Presentation

Page Order

Title

Price Presentation

8 Elements Page Order Title Price Presentation GiftBox Presentation

Page Order

Title

Price Presentation

GiftBox Presentation

8 Elements Page Order Title Price Presentation GiftBox Presentation Continue Shopping

Page Order

Title

Price Presentation

GiftBox Presentation

Continue Shopping

8 Elements Page Order Title Price Presentation GiftBox Presentation Continue Shopping Button Text

Page Order

Title

Price Presentation

GiftBox Presentation

Continue Shopping

Button Text

8 Elements Page Order Title Price Presentation GiftBox Presentation Continue Shopping Button Text Button Shape

Page Order

Title

Price Presentation

GiftBox Presentation

Continue Shopping

Button Text

Button Shape

8 Elements Page Order Title Price Presentation GiftBox Presentation Continue Shopping Button Text Button Shape Button Color

Page Order

Title

Price Presentation

GiftBox Presentation

Continue Shopping

Button Text

Button Shape

Button Color

Results of the Multivariate Test The purpose of the test was to identify which of the 576 unique versions (‘recipes’) of the product-detail page was the best (MV), and to determine how much better it was than the original. After the data was collected and analyzed, a new and improved version of the product-detail page was identified and compared head-to-head against the original (essentially an A/B test). $5.25 M Incremental Annual Revenue The new and improved page performed 4.4% better than the original. (with 99% confidence)

The purpose of the test was to identify which of the 576 unique versions (‘recipes’) of the product-detail page was the best (MV), and to determine how much better it was than the original.

After the data was collected and analyzed, a new and improved version of the product-detail page was identified and compared head-to-head against the original (essentially an A/B test).

What is MV Testing? Framework in 8 Steps Case Study Advanced Testing Techniques Ok… now how do I start? Bonus: Elements to Start Testing Today!

What is MV Testing?

Framework in 8 Steps

Case Study

Advanced Testing Techniques

Ok… now how do I start?

Bonus: Elements to Start Testing Today!

Testing outside of Web Pages Landing Pages Banners Person v. product Different value propositions Emails Product v. brand Category sale v. sale across categories Discount v. free shipping Time-of-day / day-of-week Targeted! Triggered!

Landing Pages

Banners

Person v. product

Different value propositions

Emails

Product v. brand

Category sale v. sale across categories

Discount v. free shipping

Time-of-day / day-of-week

Behavior Targeting & ‘Personas’ If like this, then show ___. Database construction Test & develop business rules Segmentation & predictive analytics Personas Represent different user types and their behavior, attitude, ‘world view’ User-centered design “ Conversion: What About the Other 98 Percent?” Predictive Modeling!

If like this, then show ___.

Database construction

Test & develop business rules

Segmentation & predictive analytics

Personas

Represent different user types and their behavior, attitude, ‘world view’

User-centered design

“ Conversion: What About the Other 98 Percent?”

‘ Manufacturing’ Testing Structure Build a reporting face Alert when sample size reached Automated decision making Output to Dev Team Develop the long-term ‘Program’ Share learnings throughout org Business unit ‘on the hook’ as driver of continually increased conversions

Build a reporting face

Alert when sample size reached

Automated decision making

Output to Dev Team

Develop the long-term ‘Program’

Share learnings throughout org

Business unit ‘on the hook’ as driver of continually increased conversions

Testing Opportunities Demographics Psych/Social Web Analytics/ Website Behavior Attitudinal Data ? Customer Support Calls & Outcome Transaction Data (R,F,M) Databases Speaking to One Another

What is MV Testing? Framework in 8 Steps Case Study Advanced Testing Techniques Ok… now how do I start? Bonus: Elements to Start Testing Today!

What is MV Testing?

Framework in 8 Steps

Case Study

Advanced Testing Techniques

Ok… now how do I start?

Bonus: Elements to Start Testing Today!

Build Yourself or Find a Partner? BUILD Long-term cost savings Back-end advanced testing structure Can build custom monitoring and notification system Data easily integrated into web analytics PARTNER Realize continuous optimization earlier Some tools include Professional Services Can ‘learn’ from the experts

BUILD

Long-term cost savings

Back-end advanced testing structure

Can build custom monitoring and notification system

Data easily integrated into web analytics

PARTNER

Realize continuous optimization earlier

Some tools include Professional Services

Can ‘learn’ from the experts

“ Which pages do I test first?” Use your web analytics tool - think ‘funnel’ All pages in the check-out process are important Where do visitors leave/bounce? What are your best selling products/categories? Know your audience understand needs and create experience for user guest or member/account? first-time buyer or frequent buyer? Landing Pages? Product Pages? Home Page?

Use your web analytics tool - think ‘funnel’

All pages in the check-out process are important

Where do visitors leave/bounce?

What are your best selling products/categories?

Know your audience

understand needs and create experience for user

guest or member/account?

first-time buyer or frequent buyer?

MV Testing is ‘doable’ See above- up on the right Think: incremental and continuous Many times, a small win builds psychological momentum at the organization for continual testing Wrap-Up: Key Considerations ‘ Small changes often have huge impact ’

MV Testing is ‘doable’

See above- up on the right

Think: incremental and continuous

Many times, a small win builds psychological momentum at the organization for continual testing

‘ Small changes often have huge impact ’ Asked to many web analytics professionals at the 2008 e-Metrics summit:

What is MV Testing? Framework in 8 Steps Case Study Advanced Testing Techniques Ok… now how do I start? Bonus: Elements to Start Testing Today!

What is MV Testing?

Framework in 8 Steps

Case Study

Advanced Testing Techniques

Ok… now how do I start?

Bonus: Elements to Start Testing Today!

What are different versions of the value proposition? Have they been tested before? Can they be seasonal? Try various formatting techniques Font, u , size, bold , color , centered Use of exclamation point! or? # of words: Many v. less Consider language colloquial; education level Testing Element: Headlines ‘ Small changes often have huge impact ’

What are different versions of the value proposition?

Have they been tested before?

Can they be seasonal?

Try various formatting techniques

Font, u , size, bold , color , centered

Use of exclamation point! or?

# of words: Many v. less

Consider language

colloquial; education level

Sale % off $ off Buy1Get1 Discount for future purchase Coupons Can pilot test campaigns! Testing Element: Offers … but what kind of shipping? ‘ Small changes often have huge impact ’ Free shipping is king!!!

Sale

% off

$ off

Buy1Get1

Discount for future purchase

Coupons

Can pilot test campaigns!

Value proposition emotional v. logical Word choice Add to Bag v. Add to Cart; Shop Now v. Order Now Color Shape Size Placement Testing Element: Call-to-Action ‘ Small changes often have huge impact ’

Value proposition

emotional v. logical

Word choice

Add to Bag v. Add to Cart; Shop Now v. Order Now

Color

Shape

Size

Placement

% image to text to white space Use very similar images in the flow Size Placement Contrast to site look & feel Personalize! Images of people are different Which direction are they facing? Where are the eyes focusing? Testing Element: Image ‘ Small changes often have huge impact ’

% image to text to white space

Use very similar images in the flow

Size

Placement

Contrast to site look & feel

Personalize!

Images of people are different

Which direction are they facing?

Where are the eyes focusing?

Common Mistake: Clutter Too much text Competing for attention Overstimulation Rest Hype-free zone ‘ Less is more’ ‘ Small changes often have huge impact ’

Too much text

Competing for attention

Overstimulation

Rest

Hype-free zone

‘ Less is more’

Web Analytics Resources

Web Analytics Resources BLOG ROLL kaushik.net/avinash/ blog.webanalyticsdemystified.com/weblog/ semphonic.blogs.com/ grokdotcom.com/ blogs.omniture.com/ juiceanalytics.com/weblog/ webanalysis.blogspot.com/ EVENTS & TRAINING webanalyticsassociation.org eMetrics Summit Web Analytics Wednesdays Web Analytics Forum Yahoo Group UBC Web Analytics X Change Conference: Gary Angel Predictive Analytics: Eric Siegel WEB ANALYTICS BOOKS OPTIMIZATION BOOKS

BLOG ROLL

kaushik.net/avinash/

blog.webanalyticsdemystified.com/weblog/

semphonic.blogs.com/

grokdotcom.com/

blogs.omniture.com/

juiceanalytics.com/weblog/

webanalysis.blogspot.com/

EVENTS & TRAINING

webanalyticsassociation.org

eMetrics Summit

Web Analytics Wednesdays

Web Analytics Forum Yahoo Group

UBC Web Analytics

X Change Conference: Gary Angel

Predictive Analytics: Eric Siegel

Q&A David Rogers [email_address] LinkedIn.com/in/DRAnalytics Twitter.com/DRAnalytics