Find the Balance Between Qualitative and Quantitative Web Site Usability Measures John Lovett Senior Analyst Jupiter Research [email_address] Jeff Schueler President Usability Sciences [email_address]

Usability Sciences Overview A leading provider of qualitative usability testing research since 1988 We were doing usability before it was cool A leading provider of quantitative user experience research since 2000 Our focus is the measurement and improvement of the user experience and effectiveness of web sites Patented technology WebIQ ® Our unique combination of qualitative and quantitative capabilities and methodologies sets us apart from our competitors We provide our research services across the country and around the world

A leading provider of qualitative usability testing research since 1988

We were doing usability before it was cool

A leading provider of quantitative user experience research since 2000

Our focus is the measurement and improvement of the user experience and effectiveness of web sites

Patented technology WebIQ ®

Our unique combination of qualitative and quantitative capabilities and methodologies sets us apart from our competitors

We provide our research services across the country and around the world

Balancing Qualitative and Quantitative Web Site Usability Measures John Lovett May 21, 2008

Usability is Persistently a Top Challenge for Web Site Operators Question: Indicate your top 3 challenges in operating your company’s primary, externally-facing Web site? JupiterResearch Web site Decision-Makers Survey 2004 Improving Usability of the Site (# 1 answer) JupiterResearch/ERI Executive Survey, 12/04, n=235 (US only) 2005 Improving Usability of the Site (#2 answer) JupiterResearch/ERI Executive Survey, 12/05, n=251 (US only) 2006 Improving Usability of the Site (#1 answer) JupiterResearch/ERI Executive Survey, 12/06, n=250 (US only)

Usability Challenges are Universal Percentage of Web Decision Makers Question: Please indicate your top 3 challenges in operating your company’s primary, externally-facing Web site? (Please select up to three.) Source: JupiterResearch/ERI Executive Survey, 12/06, n=250 (US only)

Understanding Behavior, Satisfaction and Success Requires Different Tools Usability, Behavior, Intent, Satisfaction, Success

Technologies Counterbalance Expertise and Substantiate Qualitative Assumptions Percentage of Web Decision Makers Question: Which of the following tactics do you currently use to measure website usability? (Please select all that apply.) Source: JupiterResearch/ERI Executive Survey (12/06), n=250 (US only) Qualitative Usability Resources Quantitative Usability Measures

Polling Question: Which of the following tools do you currently use to measure website usability? (Please select all that apply.) Web analytics Satisfaction Surveys Multivariate or A/B testing Exit surveys In-house expertise Focus groups Lab testing E xternal usability consultants Eye Tracking None of the above

Which of the following tools do you currently use to measure website usability? (Please select all that apply.)

Web analytics

Satisfaction Surveys

Multivariate or A/B testing

Exit surveys

In-house expertise

Focus groups

Lab testing

E xternal usability consultants

Eye Tracking

None of the above

Highly Effective Tactics Require a Mix of Qualitative & Quantitative Methods Percentage of Web Decision Makers Question: Which of the following tactics have been most effective for improving the usability for your website in the last 12 months? (Please select up to 3.) Source: JupiterResearch/ERI Executive Survey (12/06), n=250 (US only) Qualitative Measures

JupiterResearch’s Usability Framework Balances Methods Source: JupiterResearch (5/08) Automated (A/B) testing framework to test hypotheses while expediting time to market Traffic and performance analyses to mine clickstream and monitor for performance impact Voice of Customer Capture to tie feedback to clickstream and split-path testing Traditional usability principles to ground framework in established “best practices”

Traditional Usability Principles Develop a usability process Understand, Prioritize, Implement, Optimize Cultivate internally – Fortify externally Adhere to best practices Resist throwing technology at the problem

Develop a usability process

Understand, Prioritize, Implement, Optimize

Cultivate internally – Fortify externally

Adhere to best practices

Resist throwing technology at the problem

Traffic & Performance Analyses Ensure clickstream visibility Find the holes in your conversion funnel Capture the What of onsite behavior Seek to understand intent Delivery reliability and consistency Usability exceeds design

Ensure clickstream visibility

Find the holes in your conversion funnel

Capture the What of onsite behavior

Seek to understand intent

Delivery reliability and consistency

Usability exceeds design

Voice of Customer Capture Quantify Satisfaction Extend further to gauge Success Manage unsolicited feedback Amass social awareness Provide a vehicle for feedback Perform damage control

Quantify Satisfaction

Extend further to gauge Success

Manage unsolicited feedback

Amass social awareness

Provide a vehicle for feedback

Perform damage control

Increased Visibility into Attitudes & Behavior Drives Satisfaction Customer Satisfaction Usability Measurement Technologies (CEM, VOC, Analytics) Lack of visibility into customer interactions Capture of implicit and explicit customer experience data Source: JupiterResearch (11/07)

Automated Testing Framework Deploy a Control Test Alternative Hypotheses Use Testing as Segmentation Tool Be Cognizant of Demo/Psycho-graphics Address Improvement not Problems

Deploy a Control

Test Alternative Hypotheses

Use Testing as Segmentation Tool

Be Cognizant of Demo/Psycho-graphics

Address Improvement not Problems

A Balanced Approach Provides a Holistic Perspective Source: JupiterResearch 5/08 Qualitative Quantitative Paper Designs Focus groups Participatory design Card Sorting Diary studies Eye Tracking Performance Analytics A/B or MVT Satisfaction Indexes Online Testing

Source: JupiterResearch 5/08 Problem Identification Clickstream Activity Customer Feedback Behavioral Analysis Focus Groups /Labs Visual & Iterative Design Satisfaction Scoring Performance Monitoring (Gomez, Keynote) Web Analytics (Coremetrics, Omniture) VOC technologies (ForeSee, iPerceptions) Design & Development (BitGroup, Allurent) Agencies & Usability Specialists (Usability Sciences, Razorfish) Usability Technologies (Morae, Eye Tracking) CEM technologies (Coradiant, Tealeaf) Usability Measures Usability Solutions

Competitive Benchmarking Survey existing site visitors Persona Development jcp.com Scorecard Scorecard of end-to-end experience vs. competitors Site Survey Collect survey and behavioral data to best understand the customer experience Customer feedback Town hall meetings Call monitoring Ideas for Understanding the Shopper Understanding the jcp.com Shopper

Competitive Benchmarking

Survey existing site visitors

Persona Development

jcp.com Scorecard

Scorecard of end-to-end experience vs. competitors

Site Survey

Collect survey and behavioral data to best understand the customer experience

Customer feedback

Town hall meetings

Call monitoring

Design Walk-Through Rapid Iterative Testing Information architecture A/B testing in design Ideas for Understanding the Shopper Understanding the jcp.com Shopper User needs Use cases and workflows Design concepts and wireframes Prototyping Information architecture Usability Panel Usability Testing Benchmark Usability study Existing vs. new functionality Competitive Benchmark Analysis

Design Walk-Through

Rapid Iterative Testing

Information architecture

A/B testing in design

User needs

Use cases and workflows

Design concepts and wireframes

Prototyping

Information architecture

Usability Panel

Usability Testing

Benchmark Usability study

Existing vs. new functionality

Competitive Benchmark Analysis

Usability testing A/B Testing in production Ideas for Understanding the Shopper Understanding the jcp.com Shopper A/B and Multivariate Testing A/B testing - test a factor, such as a graphic, button, or image, against one or more variations to see which is most persuasive. Multivariate testing - enables testing many changes simultaneously. Evaluating the impact of combinations of factors and variations often reveals significant interaction effects that can have a dramatic impact on your conversion goals.

Usability testing

A/B Testing in production

A/B and Multivariate Testing

A/B testing - test a factor, such as a graphic, button, or image, against one or more variations to see which is most persuasive.

Multivariate testing - enables testing many changes simultaneously.

Evaluating the impact of combinations of factors and variations often reveals significant interaction effects that can have a dramatic impact on your conversion goals.

Survey existing site visitors Usability test incremental enhancements Ideas for Understanding the Shopper Understanding the jcp.com Shopper Continue site surveys Continuous measurement of the customer experience Detail Clickstream Analysis Post implementation of major releases

Survey existing site visitors

Usability test incremental enhancements

Continue site surveys

Continuous measurement of the customer experience

Detail Clickstream Analysis

Post implementation of major releases

Questions?? John Lovett Senior Analyst Jupiter Research [email_address] Jeff Schueler President Usability Sciences [email_address]