Your SlideShare is downloading. ×
0
#EpicFail!
A/B Test Results vs. UX Best Practices
Susan Rice & Kirk Doggett
Global UX Center of Excellence
UPA Boston Conf...
Overview
• Introduction
• Examples
– Pop Up Windows
– Progress Indicators
– Homepages
– Buttons
– Website Purchase Flow
– ...
Who We Are
Vistaprint provides high-impact personalized
products and electronic services that empower
micro businesses and...
―Test Before You Invest‖
We have dozens of multivariate
―split run‖ site tests live at any
given time during our 3 week
re...
UX & A/B Testing
• UX designs the UI and
helps inform the test plan.
• Test runs and is analyzed.
• Layer the quantitative...
POP UP WINDOWS
6UPA Boston Conference9/12/2013
What the experts say…
• Avoid popup windows: [1]
– Extra windows keep users from getting to your site content, and even if...
What the experts say…
• You’re going to lose my business if your web site
experience is annoying.
• According to prelimina...
In 2007 we added a pop-up…
Control Test
UPA Boston Conference9/12/2013 9
Results of adding the pop-up
• Significant increase in conversion rate across
most countries and across most products, fro...
Today:
In 2008, Nielsen declares the lightbox as the
interaction design technique of the year [1]
[1] Jakob Nielsen, 10 Be...
PROGRESS INDICATORS
12UPA Boston Conference9/12/2013
What the experts say…
• Show the steps of the process [1]
– Recommended for checkout
• Guideline: Provide feedback to let ...
We tried removing the progress indicator
• Removing progress bar from Options improves
conversion
Test
Control
Results:
• ...
We tried adding a progress indicator…
• Adding a progress bar to the design pages hurts
conversion:
15
Test condition sess...
Progress Bar visual design…
Control
Test
Visual redesign hurts
conversion.
Results:
• CR is down 0.9% for
New Customers
16...
HOMEPAGES
17UPA Boston Conference9/12/2013
What the experts say…
• “Clearly designate one page per site as the official
homepage.” [1]
• For a decade, one of the pri...
Two Home pages?
• It worked for HomeRuns in 2001:
19
First Time Home Returning Home
UPA Boston Conference9/12/2013
A decade later…
• For returning customers, conversion rate is significantly higher for the
Returning Home page than from t...
BUTTONS
UPA Boston Conference 219/12/2013
What the experts say…
• ―Ensure that a pushbutton‘s label clearly indicates its
action.‖ [1]
• ―The text on the button sho...
E-Commerce Examples
Staples
Amazon
UPA Boston Conference 239/12/2013
Cart Buttons
UPA Boston Conference 24
Significant decrease in CTR to ShippingSignificant increase of CTR to Shipping
Contr...
E-Commerce Examples
Amazon
UPA Boston Conference 259/12/2013
Checkout Buttons
UPA Boston Conference 26
Could simpler buttons actually decrease
cognitive load and prevent overthinking?...
Product Page Buttons
• We tested rollover states, more prominent call-to-actions
and larger targets.
UPA Boston Conference...
WEBSITE PURCHASE FLOW
UPA Boston Conference 289/12/2013
What the experts say…
• Offer ―sufficient information about your products to answer users‘ pre-
purchase questions.‖ [1]
•...
Design / Flow
UPA Boston Conference 30
Get started Choose Design Template
9/12/2013
Design / Flow
UPA Boston Conference 31
Preview
Template
Personalize! Checkout Process
9/12/2013
That seems to work well. But what if…?
UPA Boston Conference 32
Significant increases to Website CR for
new customers.
• D...
PEOPLE DON’T READ
UPA Boston Conference 339/12/2013
Good ol‘ Gary Larson…
UPA Boston Conference 349/12/2013
What the experts say…
• ―We've known since our first studies of how users read on the
Web that they typically don't read v...
Product Page
UPA Boston Conference 36
Control
Test 2
Significant increase in
paper stock upgrades
9/12/2013
Relevant copy ...
More Product Page Testing
UPA Boston Conference 37
Control Test
Marginal increase in GM/order
Significant shift in take ra...
Sign In Button
UPA Boston Conference 38
Significantly hurt
overall CR by 1.25%
Sign in page exit rate
increased significan...
Summary
• Best practices are a great starting point but they‘re
general.
• A/B testing can help you zero-in on specific ch...
Summary
• There‘s a healthy tension between Standardization
and Innovation.
• This isn‘t the end state. We‘ll continue to ...
Summary
• Consistency is one of the most powerful usability
principles…
• A foolish consistency is the hobgoblin of little...
QUESTIONS?
UPA Boston Conference 429/12/2013
Thanks!
Special thanks to Dave Yuknat for his fantastic editing
expertise… and for gathering supporting examples for the
p...
Upcoming SlideShare
Loading in...5
×

#EpicFail! A/B Test Results vs. UX Best Practices (Boston UPA 2012)

623

Published on

#EpicFail! A/B Test Results vs. UX Best Practices

Published in: Technology, Design
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
623
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
4
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • The pop-up re-presents the same selections on the pageBrowse designs byBusiness or IndustryStylesThe pop-up is automatic – it is unsolicited by the user
  • Illustrates the principle that people will keep clicking as long as they feel they are making progress toward their goal, and each step is quick & easy.
  • The progress bar shows where you are in the process, and what steps come next.This progress indicator was tested on our Japanese website, and the translation into English is not quite right.This result supports what we learned from the removing the progress bar.
  • We tried a new visual design to give more prominence to the progress bar, and to indicate forward motion with the chevrons.
  • We created an alternate Home page for our returning customers.The content we show is based on clickstream data from the First Time Home page
  • Larger target and visual hierarchy worked… verb w/ security messaging didn’t.Winner for all locales except Japan. In Japan, this test actually hurt conversion rate with the hypothesis that it was due to the loss of the Clear Cart feature, so we rolled it out to all locales except Japan. More work to understand the impact in that locale.
  • Testing helps you determine how far to take UX best practices and how to balance your desire to improve KPI’s with a possibly more elegant design.
  • Design coolness comes at a priceHover effects, but not a clear CTACTA, but no hover effect
  • Going from 5 steps to 2 steps
  • We created a “Post Order Configuration” flow that was available immediately after purchase.
  • But wait… you’re probably going to say that this research is based on sites with a lot of content like news sites, etc… and you’re right. So what does this mean to an ecommerce site with little text overall…? That’s what we’re trying to figure out.
  • Instead, the right question is, “Will the user’s current knowledge help them understand how to use what I’m designing?” Current knowledge is the knowledge the user has when they approach the design. It’s the sum of all their previous experiences with relevant products and designs.
  • Instead, the right question is, “Will the user’s current knowledge help them understand how to use what I’m designing?” Current knowledge is the knowledge the user has when they approach the design. It’s the sum of all their previous experiences with relevant products and designs.
  • Transcript of "#EpicFail! A/B Test Results vs. UX Best Practices (Boston UPA 2012)"

    1. 1. #EpicFail! A/B Test Results vs. UX Best Practices Susan Rice & Kirk Doggett Global UX Center of Excellence UPA Boston Conference 19/12/2013
    2. 2. Overview • Introduction • Examples – Pop Up Windows – Progress Indicators – Homepages – Buttons – Website Purchase Flow – People Don‘t Read • Summary • Questions? UPA Boston Conference 29/12/2013
    3. 3. Who We Are Vistaprint provides high-impact personalized products and electronic services that empower micro businesses and consumers at an affordable price. Company Facts • 25 localized websites • 3,700 employees • 3 state of art manufacturing facilities • 13 offices worldwide UPA Boston Conference 39/12/2013
    4. 4. ―Test Before You Invest‖ We have dozens of multivariate ―split run‖ site tests live at any given time during our 3 week release cycles. • Scalable: Most site changes are measured before rollout • Low Risk: Multiple tests can be run simultaneously with a sample of the population • Accurate: Test sample is representative of user base 4 Random Sample Price Control Price Test UI Control 5% 5% UI Test 5% 5% User Population UPA Boston Conference9/12/2013
    5. 5. UX & A/B Testing • UX designs the UI and helps inform the test plan. • Test runs and is analyzed. • Layer the quantitative with the qualitative to create a holistic picture to inform future design decisions. UPA Boston Conference 5 But what happens when A/B testing results challenge tried and true usability principles and best practices? 9/12/2013
    6. 6. POP UP WINDOWS 6UPA Boston Conference9/12/2013
    7. 7. What the experts say… • Avoid popup windows: [1] – Extra windows keep users from getting to your site content, and even if such windows contain valuable information, users are likely to dismiss them immediately as ads. – The other drawback to popup windows is that they go away — once the user gets rid of them, they're gone, so users often can't find the information again even if they want to. • Do not have unsolicited windows or graphics ―pop up‖ to users.[2] – Users have commented that unsolicited windows or graphics that ‗pop up‘ are annoying and distracting when they are focusing on completing their original activity. • Selective attention is very powerful, and Web users have learned to stop paying attention to any ads that get in the way of their goal-driven navigation. [3] – “pop-up purges mean that users close pop-up windoids before they have even fully rendered; sometimes with great viciousness‖ [1] Jakob Nielsen, excerpt from 113 Design Guidelines for Homepage Usability, 2001 [2] US Government, Research-Based Web Design & Usability Guidelines, 2006 [3] Jakob Nielsen, Top 10 Mistakes in Web Design, updated 2011 7UPA Boston Conference9/12/2013
    8. 8. What the experts say… • You’re going to lose my business if your web site experience is annoying. • According to preliminary information from the survey… major peeves about commercial Web sites were: [1] – Pop-up advertising (34.9 percent) – Registration log-on pages (16.7 percent) – Software installation (15.7 percent) – Slow-loading pages (9.1 percent) [1] John P. Mello Jr. http://www.ecommercetimes.com/story/44966.html, July, 2005 8UPA Boston Conference9/12/2013
    9. 9. In 2007 we added a pop-up… Control Test UPA Boston Conference9/12/2013 9
    10. 10. Results of adding the pop-up • Significant increase in conversion rate across most countries and across most products, from 5% to 11%. • More business cards were created Most Popular Most Popular Beauty/Massage/Spa Beauty/Massage/Spa Construction & Trade Construction & Trade Flowers & Candles Flowers & Candles Music & Entertainment Music & Entertainment 0 5000 10000 15000 20000 25000 3700 3701 Sum of Documents Created Feature Value Id Category Name Control Experiment Tested in August, 2007 10UPA Boston Conference9/12/2013
    11. 11. Today: In 2008, Nielsen declares the lightbox as the interaction design technique of the year [1] [1] Jakob Nielsen, 10 Best Application UIs, Alertbox, 8/12/2008 UPA Boston Conference9/12/2013 11
    12. 12. PROGRESS INDICATORS 12UPA Boston Conference9/12/2013
    13. 13. What the experts say… • Show the steps of the process [1] – Recommended for checkout • Guideline: Provide feedback to let users know where they are in the Web site. [2] – Feedback provides users with the information they need to understand where they are within the Web site, and for proceeding to the next activity. • In this example, note the right-pointing arrows. Nielsen discusses his changeover from colons to arrows. The rationale for the arrows instead of colons or other indicators was because ―the arrows seem to be slightly easier to understand as an indication of moving deeper and deeper into the site.‖ [3] [1] Nielsen Norman Group, E-Commerce User Experience, Vol. 4: Shopping Cart, Checkout and Registration, 2011 [2] US Government, Research-Based Web Design & Usability Guidelines, 2006 [3] Beth A. Martin, Breadcrumb Navigation, November 2006 13UPA Boston Conference9/12/2013
    14. 14. We tried removing the progress indicator • Removing progress bar from Options improves conversion Test Control Results: • We observed a significant increase of +0.47% of the conversion rate overall. • The test significantly increased the number of sessions that make it to the Cart. 14UPA Boston Conference9/12/2013
    15. 15. We tried adding a progress indicator… • Adding a progress bar to the design pages hurts conversion: 15 Test condition sessions CR CR Change Control 50175 7.33% Display Nav 49362 7.06% -3.69% UPA Boston Conference9/12/2013
    16. 16. Progress Bar visual design… Control Test Visual redesign hurts conversion. Results: • CR is down 0.9% for New Customers 16UPA Boston Conference9/12/2013
    17. 17. HOMEPAGES 17UPA Boston Conference9/12/2013
    18. 18. What the experts say… • “Clearly designate one page per site as the official homepage.” [1] • For a decade, one of the primary homepage usability guidelines has been to designate a single page as the one and only official homepage for any given website. Users are confused when several pages are referred to as ―home.‖ [2] • [3] [1] Jakob Nielsen, 113 Design Guidelines for Homepage Usability, 2001 [2] Jakob Nielsen, Overloaded vs. Generic Commands, 12/19/2011 [3] US Government, Research-Based Web Design & Usability Guidelines, 2006 18UPA Boston Conference9/12/2013
    19. 19. Two Home pages? • It worked for HomeRuns in 2001: 19 First Time Home Returning Home UPA Boston Conference9/12/2013
    20. 20. A decade later… • For returning customers, conversion rate is significantly higher for the Returning Home page than from the First Time Visitor home page. First Time Home Returning Home 20UPA Boston Conference9/12/2013
    21. 21. BUTTONS UPA Boston Conference 219/12/2013
    22. 22. What the experts say… • ―Ensure that a pushbutton‘s label clearly indicates its action.‖ [1] • ―The text on the button should begin with a verb. [2] • ―Trigger words are the words and phrases that trigger a user into clicking. They contain the essential elements to provide the motivation to continue with the site.‖ [3] • Fitts’s Law: ―The time required to move to a target is a function of target size and distance to the target.‖ [4] UPA Boston Conference 22 [1] US Government, Research-Based Web Design & Usability Guidelines, 2006 [2] UX Booth, Good Call-to-Action Buttons, 2009 [3] Jared Spool, The Right Trigger Words, 2004 [4] Universal Principles of Design, Revised and Updated: 125 Ways to Enhance Usability, Influence Perception, Increase Appeal, Make Better Design Decisions, and Teach through Design, 2010 9/12/2013
    23. 23. E-Commerce Examples Staples Amazon UPA Boston Conference 239/12/2013
    24. 24. Cart Buttons UPA Boston Conference 24 Significant decrease in CTR to ShippingSignificant increase of CTR to Shipping Control Test 1 Test 2 • We tested larger target areas with improved visual hierarchy and verbs with security info like Staples. Increased cognitive friction? 9/12/2013
    25. 25. E-Commerce Examples Amazon UPA Boston Conference 259/12/2013
    26. 26. Checkout Buttons UPA Boston Conference 26 Could simpler buttons actually decrease cognitive load and prevent overthinking? • We tested distinctive trigger words on our checkout buttons and similar language to Amazon. CR decreased significantly in FR CTR decreased significantly from shipping to confirmation in top 5 locales Shut off after 9 days in testing Control Test 1 9/12/2013
    27. 27. Product Page Buttons • We tested rollover states, more prominent call-to-actions and larger targets. UPA Boston Conference 27 Control Test 1 Significant increase in CR of 1.5% was seen. Law of diminishing returns? Test 2 Test 2: No impact. 9/12/2013
    28. 28. WEBSITE PURCHASE FLOW UPA Boston Conference 289/12/2013
    29. 29. What the experts say… • Offer ―sufficient information about your products to answer users‘ pre- purchase questions.‖ [1] • ―Helping customers see what they want, when they want it, and in the way they want to see it, is substantially preferable to forcing them through a hub-and-spoke, back-and-forth path.‖ [2] • ―The number of clicks isn't what is important to users, but whether or not they're successful at finding what they're seeking.‖ [3] And internally we‘ve learned… • When a customer becomes invested in one of our designs, the more likely they‘ll convert. • Multiple screens/clicks are OK if customers know they‘re making progress. UPA Boston Conference 29 [1] NN/g, E-Commerce User Experience [2] Bryan Eisenberg et. al., Call to Action: Secret Formulas to Improve Online Results, 2005 [3] Jared Spool, Testing the Three-Click Rule, 2003 9/12/2013
    30. 30. Design / Flow UPA Boston Conference 30 Get started Choose Design Template 9/12/2013
    31. 31. Design / Flow UPA Boston Conference 31 Preview Template Personalize! Checkout Process 9/12/2013
    32. 32. That seems to work well. But what if…? UPA Boston Conference 32 Significant increases to Website CR for new customers. • DTI CR increased 9.58% • Acquisition campaigns increased 5.18% Less friction? 9/12/2013
    33. 33. PEOPLE DON’T READ UPA Boston Conference 339/12/2013
    34. 34. Good ol‘ Gary Larson… UPA Boston Conference 349/12/2013
    35. 35. What the experts say… • ―We've known since our first studies of how users read on the Web that they typically don't read very much.‖ [1] • ―Satisficing describes the situation where people settle with a solution to a problem that is "good enough".‖ [2] • “Users don‘t make optimal choices…Neither do they scan web-page in a linear fashion, going sequentially from one site section to another one.‖ [3] UPA Boston Conference 35 [1] Jakob Nielsen, How Little Do Users Read?, 2008 [2] Interaction-Design.org, Satisficing, 2004 [3] Smashing Magazine, 10 Principles of Effective Web Design, 2008 9/12/2013
    36. 36. Product Page UPA Boston Conference 36 Control Test 2 Significant increase in paper stock upgrades 9/12/2013 Relevant copy is helpful for decision making.
    37. 37. More Product Page Testing UPA Boston Conference 37 Control Test Marginal increase in GM/order Significant shift in take rate of 100 lb. premium and recycled paper and a decrease in Matte and Glossy. So copy is making a difference… 9/12/2013
    38. 38. Sign In Button UPA Boston Conference 38 Significantly hurt overall CR by 1.25% Sign in page exit rate increased significantly. Shut off after 7 days of testing Control Test Even changing 2 letters can have an impact. 9/12/2013
    39. 39. Summary • Best practices are a great starting point but they‘re general. • A/B testing can help you zero-in on specific changes that can help you make changes that benefit your site and your customers. • Context and relevancy matters. • Even small changes can make a significant difference in site performance. • Nothing is straightforward and simple. UPA Boston Conference 399/12/2013
    40. 40. Summary • There‘s a healthy tension between Standardization and Innovation. • This isn‘t the end state. We‘ll continue to optimize and learn from our customers. • It‘s all about balance… and learning more about your customers through various avenues… • What the experts say: It depends… [1] UPA Boston Conference 40 [1] Jared Spool, User Interface Engineering 9/12/2013
    41. 41. Summary • Consistency is one of the most powerful usability principles… • A foolish consistency is the hobgoblin of little minds… - Ralph Waldo Emerson • A/B testing helps us learn when context is more important than consistency. UPA Boston Conference 419/12/2013
    42. 42. QUESTIONS? UPA Boston Conference 429/12/2013
    43. 43. Thanks! Special thanks to Dave Yuknat for his fantastic editing expertise… and for gathering supporting examples for the presentation. Susan Rice leads global UX design and Kirk Doggett leads global UX research at Vistaprint. And just a plug… we’re hiring! UPA Boston Conference 439/12/2013
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×