Microsoft Case Study: 40% Conversion Rate Increase of SEM Traffic using Multivariate Testing Michael Schultz Director Product Management & Marketing, Microsoft Office Unmanaged Services Frans Keylard Director of Optimization, Widemile

Agenda A/B & Multivariate Testing Overview Microsoft Office Live Small Business Case Study Business Overview Case Study Details Top 10 Page Optimization Tips

A/B & Multivariate Testing Overview

Microsoft Office Live Small Business Case Study

Business Overview

Case Study Details

Top 10 Page Optimization Tips

Conversion Optimization * Source: eMarketer – April 2009 Investment recommendation: 5% to 15% of acquisition budget Testing and segmentation can improve retention performance Automated, data-driven action that increases revenue by optimizing a user’s online experience. $37 Billion annual US spend by 2013. $23 Billion spent in 2008.* Customer Sales Funnel Email marketing Loyalty Programs Newsletters Personalization Coupons /Surveys Widemile Focus Key web pages Sales funnels Landing pages Registration pages Search marketing Display advertising Affiliate marketing Contextual ads Ad targeting Conversion Acquisition Retention

Email marketing

Loyalty Programs

Newsletters

Personalization

Coupons /Surveys

Key web pages

Sales funnels

Landing pages

Registration pages

Search marketing

Display advertising

Affiliate marketing

Contextual ads

Ad targeting

Testing Options – Approach Overview Ideal use: > Validates traffic and conversion rates Limitation : > Simple measurement; no optimization Basic Analytics A/B Split Testing Ideal use: > Best for testing two or more radically different pages <i.e. template testing> Limitation : > No insight into why one page is better, just which won among traffic tested Multivariate Testing (MVT ) Ideal use: > Test different content, simultaneously determining most effective combination > Segmentation can be applied for more granular optimization in follow-on tests Limitation : > Requires sufficient traffic & conversions

MVT Testing Options - Full & Fractional Factorial Fractional Factorial Full Factorial

Requirements for Success Expertise User Interface Proven Architecture Reporting Interface Analyst Creative Web Developer Technology

2009 – Year for Conversion Optimization

Microsoft Office Live Small Business Audience: Small businesses, 1-10 employees (real estate agent, local bakery, video store, restaurant) Non-profits/Organizations (community groups, soccer clubs, PTSAs) Benefits: Free website and email Free services to view, store and share documents online

Audience:

Small businesses, 1-10 employees (real estate agent, local bakery, video store, restaurant)

Non-profits/Organizations (community groups, soccer clubs, PTSAs)

Benefits:

Free website and email

Free services to view, store and share documents online

Goals Summary Goals: Improve sign up conversions Better understand audience Identify highest impact page factors Testing Strategy: Leverage proven landing page template Execute multivariate test Fractional factorial approach to minimize time

Goals:

Improve sign up conversions

Better understand audience

Identify highest impact page factors

Testing Strategy:

Leverage proven landing page template

Execute multivariate test

Fractional factorial approach to minimize time

Microsoft Case Study Getting started… Define page to be tested Analyze outside factors In-bound traffic Traffic volume UX - Funnel Review page & conversion goals Original web page

Getting started…

Define page to be tested

Analyze outside factors

In-bound traffic

Traffic volume

UX - Funnel

Review page & conversion goals

Testing Considerations What is the best hero shot? Does the headline grab the customer’s attention? Do these resource links belong here? Does the button maximize conversions? Does this copy help or hurt?

Indentifying Factors to Test 1 2 3 4 5

Defining a Test Matrix No. Factor (Variable) Levels (versions) 1 Hero Shot 4 2 Title 4 3 Description 4 4 Call-To-Action Button 4 5 Resource Links 2

Audience Participation – Best CTA Button Level 2 Level 3 Level 4 Level 1

Efficiency Through Technology 1 of 16 2 of 16 3 of 16 4 of 16 16

And the Winner is… 40% Improvement

Which Factors Made the Biggest Difference Hero Shot Title Description Call-To-Action Resource Link Location High Influence Medium Influence Low Influence

High Influence Factor - Key Learning Factor/level insights “ Get Started Free” won The free aspect of the service is very important to get people to try it out More task focused terminology like “Set up my business!” or “Build My Online Business” lost “ Sign Up Free!” which is the typical text found on so many CTA buttons also lost out. Future testing around the positioning of the FREE service

Factor/level insights

“ Get Started Free” won

The free aspect of the service is very important to get people to try it out

More task focused terminology like “Set up my business!” or “Build My Online Business” lost

“ Sign Up Free!” which is the typical text found on so many CTA buttons also lost out.

Future testing around the positioning of the FREE service

Case Study Summary Microsoft Office Live Small Business’ testing and optimizing investment: Delivered a 40% lift in conversion Reduced customer acquisition cost Maximized SEM performance Testing and optimizing content is a fast and effective way to maximize the performance of your SEM spend.

Microsoft Office Live Small Business’ testing and optimizing investment:

Delivered a 40% lift in conversion

Reduced customer acquisition cost

Maximized SEM performance

Testing and optimizing content is a fast and effective

way to maximize the performance of your SEM spend.

10 Optimization Tips 1. Offer Clarity Make sure the offer is clear, relevant, and compelling Keep your pages simple, clear, and appealing 2. Path to Conversion Make sure the path to conversion is simple and straightforward Keep conversion paths efficient, focused, uncluttered 3. Performance Prospects will not tolerate slow pages or links Google penalizes slow page loads 4. Test Design and User Experience Avoid confusing landing page designs and difficult to read text Make transitions seamless and keep the consistency high (both messaging and look & feel) 5. Seasonality Seasonally optimize your pages and PPC campaign Leverage historical SEM in planning current and future campaigns

1. Offer Clarity

Make sure the offer is clear, relevant, and compelling

Keep your pages simple, clear, and appealing

2. Path to Conversion

Make sure the path to conversion is simple and straightforward

Keep conversion paths efficient, focused, uncluttered

3. Performance

Prospects will not tolerate slow pages or links

Google penalizes slow page loads

4. Test Design and User Experience

Avoid confusing landing page designs and difficult to read text

Make transitions seamless and keep the consistency high (both messaging and look & feel)

5. Seasonality

Seasonally optimize your pages and PPC campaign

Leverage historical SEM in planning current and future campaigns

10 Optimization Tips 6. Target Audience Optimize your content based on audience insight Capture contact information early to better inform your marketing 7. Segment Audience Optimize for distinct audiences to convert Segment your traffic and test to see if unique landing pages would perform better 8. Offers and Goals Create and test offers that have a high perceived value but a low cost to fulfill Force your page to have only one clear goal and link ads to landing pages 9. Consistency Build consistency and relevancy between PPC campaigns and where you send your traffic Reinforce the call to action/offer on order and lead forms 10. Competition Look at what your competition is doing, but also what they are not doing Differentiation can often help, so test new approaches, offers, and incentives

6. Target Audience

Optimize your content based on audience insight

Capture contact information early to better inform your marketing

7. Segment Audience

Optimize for distinct audiences to convert

Segment your traffic and test to see if unique landing pages would perform better

8. Offers and Goals

Create and test offers that have a high perceived value but a low cost to fulfill

Force your page to have only one clear goal and link ads to landing pages

9. Consistency

Build consistency and relevancy between PPC campaigns and where you send your traffic

Reinforce the call to action/offer on order and lead forms

10. Competition

Look at what your competition is doing, but also what they are not doing

Differentiation can often help, so test new approaches, offers, and incentives

4 Additional Tips for Marketers 1. Drink the Kool-Aid and start now! 2. Think of multivariate testing as a “fixed” marketing expense and allocate budget appropriately 3. Ensure collaboration between Creative, Optimization Testing and SEM 4. Make sure Optimization vendor learns the business and understands the audience

1. Drink the Kool-Aid and start now!

2. Think of multivariate testing as a “fixed” marketing expense and allocate budget appropriately

3. Ensure collaboration between Creative, Optimization Testing and SEM

4. Make sure Optimization vendor learns the business and understands the audience

Questions? Presenters Frans Keylard – [email_address] Michael Schultz – [email_address] Content Information www.widemile.com www.officelive.com

Presenters

Frans Keylard – [email_address]

Michael Schultz – [email_address]

Content Information

www.widemile.com

www.officelive.com