Widemile and Microsoft Multivariate Testing Case Study

Microsoft Case Study: 40% Conversion Rate Increase of SEM Traffic using Multivariate Testing Michael Schultz Director Product Management & Marketing, Microsoft Office Unmanaged Services Frans Keylard Director of Optimization, Widemile

Agenda
A/B & Multivariate Testing Overview
Microsoft Office Live Small Business Case Study
Business Overview
Case Study Details
Top 10 Page Optimization Tips

Conversion Optimization * Source: eMarketer – April 2009 Investment recommendation: 5% to 15% of acquisition budget Testing and segmentation can improve retention performance Automated, data-driven action that increases revenue by optimizing a user’s online experience. $37 Billion annual US spend by 2013. $23 Billion spent in 2008.* Customer Sales Funnel
Email marketing
Loyalty Programs
Newsletters
Personalization
Coupons /Surveys
Widemile Focus
Key web pages
Sales funnels
Landing pages
Registration pages
Search marketing
Display advertising
Affiliate marketing
Contextual ads
Ad targeting
Conversion Acquisition Retention

Testing Options – Approach Overview Ideal use: > Validates traffic and conversion rates Limitation : > Simple measurement; no optimization Basic Analytics A/B Split Testing Ideal use: > Best for testing two or more radically different pages <i.e. template testing> Limitation : > No insight into why one page is better, just which won among traffic tested Multivariate Testing (MVT ) Ideal use: > Test different content, simultaneously determining most effective combination > Segmentation can be applied for more granular optimization in follow-on tests Limitation : > Requires sufficient traffic & conversions

MVT Testing Options - Full & Fractional Factorial Fractional Factorial Full Factorial

Requirements for Success Expertise User Interface Proven Architecture Reporting Interface Analyst Creative Web Developer Technology

2009 – Year for Conversion Optimization

Microsoft Office Live Small Business
Audience:
Small businesses, 1-10 employees (real estate agent, local bakery, video store, restaurant)
Non-profits/Organizations (community groups, soccer clubs, PTSAs)
Benefits:
Free website and email
Free services to view, store and share documents online

Goals Summary
Goals:
Improve sign up conversions
Better understand audience
Identify highest impact page factors
Testing Strategy:
Leverage proven landing page template
Execute multivariate test
Fractional factorial approach to minimize time

Microsoft Case Study
Getting started…
Define page to be tested
Analyze outside factors
In-bound traffic
Traffic volume
UX - Funnel
Review page & conversion goals
Original web page

Testing Considerations What is the best hero shot? Does the headline grab the customer’s attention? Do these resource links belong here? Does the button maximize conversions? Does this copy help or hurt?

Indentifying Factors to Test 1 2 3 4 5

Defining a Test Matrix No. Factor (Variable) Levels (versions) 1 Hero Shot 4 2 Title 4 3 Description 4 4 Call-To-Action Button 4 5 Resource Links 2

Audience Participation – Best CTA Button Level 2 Level 3 Level 4 Level 1

Efficiency Through Technology 1 of 16 2 of 16 3 of 16 4 of 16 16

And the Winner is… 40% Improvement

Which Factors Made the Biggest Difference Hero Shot Title Description Call-To-Action Resource Link Location High Influence Medium Influence Low Influence

High Influence Factor - Key Learning
Factor/level insights
“ Get Started Free” won
The free aspect of the service is very important to get people to try it out
More task focused terminology like “Set up my business!” or “Build My Online Business” lost
“ Sign Up Free!” which is the typical text found on so many CTA buttons also lost out.
Future testing around the positioning of the FREE service

Case Study Summary
Microsoft Office Live Small Business’ testing and optimizing investment:
Delivered a 40% lift in conversion
Reduced customer acquisition cost
Maximized SEM performance
Testing and optimizing content is a fast and effective
way to maximize the performance of your SEM spend.

10 Optimization Tips
1. Offer Clarity
Make sure the offer is clear, relevant, and compelling
Keep your pages simple, clear, and appealing
2. Path to Conversion
Make sure the path to conversion is simple and straightforward
Keep conversion paths efficient, focused, uncluttered
3. Performance
Prospects will not tolerate slow pages or links
Google penalizes slow page loads
4. Test Design and User Experience
Avoid confusing landing page designs and difficult to read text
Make transitions seamless and keep the consistency high (both messaging and look & feel)
5. Seasonality
Seasonally optimize your pages and PPC campaign
Leverage historical SEM in planning current and future campaigns

10 Optimization Tips
6. Target Audience
Optimize your content based on audience insight
Capture contact information early to better inform your marketing
7. Segment Audience
Optimize for distinct audiences to convert
Segment your traffic and test to see if unique landing pages would perform better
8. Offers and Goals
Create and test offers that have a high perceived value but a low cost to fulfill
Force your page to have only one clear goal and link ads to landing pages
9. Consistency
Build consistency and relevancy between PPC campaigns and where you send your traffic
Reinforce the call to action/offer on order and lead forms
10. Competition
Look at what your competition is doing, but also what they are not doing
Differentiation can often help, so test new approaches, offers, and incentives

4 Additional Tips for Marketers
1. Drink the Kool-Aid and start now!
2. Think of multivariate testing as a “fixed” marketing expense and allocate budget appropriately
3. Ensure collaboration between Creative, Optimization Testing and SEM
4. Make sure Optimization vendor learns the business and understands the audience

Questions?
Presenters
Frans Keylard – [email_address]
Michael Schultz – [email_address]
Content Information
www.widemile.com
www.officelive.com

Widemile and Microsoft Multivariate Testing Case Study

by Widemile on Jun 18, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 22

Statistics

Widemile and Microsoft Multivariate Testing Case Study — Presentation Transcript

http://www.slideshare.net	21
http://a0.twimg.com	1