UserZoom Webinar: How to Conduct Web Customer Experience Benchmarking

February 2010

Welcome everyone!
Our speakers for today’s webinar are:
Alfonso de la Nuez, Founder & CMO at UZ
Kim Oslob, Sr. Research & Product Strategy Director at UZ
Ania Rodriguez, Principal at Key Lime Interactive
Copyright 2010. Key Lime Interactive and UserZoom. All Rights Reserved.

Five things we’d like to talk to you about today
What do we mean by UX Benchmarking? Why is it important?
How to conduct UX Benchmarking studies
Results you must have
Examples
UserZoom and Key Lime Interative at a glance
Copyright 2010. Key Lime Interactive and UserZoom. All Rights Reserved.

What is UX Benchmarking & why is it important?
Definition of Benchmarking:
The process of comparing the business processes and performance metrics including cost, cycle time, productivity, or quality to another that is widely considered to be an industry standard benchmark or best practice.
It provides a snapshot of the performance of your business and helps you understand where you are in relation to a particular standard.

Definition of UX:
I t’s about how a person feels about using a system. It highlights the experiential, affective, meaningful and valuable aspects of HCI and product ownership, but it also covers a person’s perceptions of the practical aspects such as utility, ease of use and efficiency of the system.
User experience is subjective in nature, as it is about an individual’s feelings and thoughts about the system. User experience is dynamic , because it changes over time as the circumstances change.

What is UX Benchmarking & why is it important? Ok… So how do we define UX Benchmarking ?

Common UX research challenges What is UX Benchmarking & why is it important?
Difficulties obtaining objective, statistically significant, actionable data
Obtaining idea adoption from stakeholders
Geographic/demographic limitations
Research is costly and time-consuming

Some may wonder: Is UX Benchmarking… What is UX Benchmarking & why is it important? ?

Not really. What is UX Benchmarking & why is it important?
Here’s how we’d define UX Benchmarking:
The process of comparing a website’s performance, specifically in relation to UX metrics, including among others effectiveness, efficiency, overall satisfaction and quality perception , to another that is widely considered to be an industry standard benchmark or best practice
UX benchmarking provides a snapshot of the performance of your website and helps you understand where you are in relation to a particular standard.

Why is it now more important than ever before? What is UX Benchmarking & why is it important? It’s all about competitiveness, productivity, & efficiency

‘ You can’t manage what you can’t measure’ Attributed to Peter Drucker, Mgt Guru What is UX Benchmarking & why is it important? The question is: How are you actually measuring UX and defining your benchmark?

What is UX Benchmarking & why is it more important than ever?
Examples

10 must-dos with UX Benchmarking
Define measurable & comparable UX-related KPIs (key performance indicators)
Select competitors to benchmark against
Define target audience
Identify common site activities/tasks (between-subjects design)
Conduct research using scenario-based testing
Gather both quantitative & qualitative data
Use the quantitative data to test for statistically significant differences between means or proportions
Use qualitative data to support your findings and understand the why
Research at least twice a year to measure design changes and trends in expectations
Use the right research solution/tool

How to conduct UX Benchmarking studies Graph from Bill Albert, Bentley University

Our solution: UZ Self-Serve Edition
On-demand software application that empowers businesses to manage & conduct online UX research
Feature-rich, versatile* software solution, designed for task-based surveys (or unmoderated remote usability testing )
Allows researchers to manage the whole study, including: create script, launch, collect data and analyze results
*Also good for online surveys, card-sorting and voice of customer surveys

Once you have - defined measurable & comparable UX-related KPIs (key performance indicators) - selected competitors to benchmark against - defined target audience
Create a study script
Launch the study, collect data, monitor progress
Analyze results
Let’s walk through the process 1 2 3 … the 3-step process is fairly simple

How UserZoom works
Creating a study script using the UZ Editor
1
Creating a study is EASY
Test structure built in
Preview options
Advanced scripting
Ask questions depending on users’ online behavior
Integrated user recruitment capabilities
Pre-testing available

Feature-rich, advanced scripting capabilities:
High-level of flexibility
Advanced task validation criteria (by question, by page interception, by time...)
Advanced questionnaires tool (all type of questions: Choice, scales, open...)
Insert multimedia (images, videos, flash, applets, sounds...)
Manage advanced logic/branching & conditions
Behavioral questionnaires (by interception)
Randomization: tasks, answers...
Piping capabilities
Mandatory questions
Time-out options
Sample & data quality control
Multi-lingual studies support (including Asian languages).
… and more
How UserZoom works
1

How UserZoom works
1 Partners of
Choose the user recruitment yourself
Panel
Intercept
Private mailing list

Launch the study, collect data, monitor progress
How UserZoom works 2
Hundreds of users can be tested
Participation in the natural context…
… from geographically spread locations
No human moderation needed
Our browser bar connects users with our secure servers

Examples

Analyze data real-time using UZ Analytics
Results to look for 3 Effectiveness + efficiency ratios

Analyze data using UZ Analytics
How UserZoom works 3 Qualitative feedback collected. Easy visualization through ‘ text clouds ’

Analyze data using UZ Analytics
How UserZoom works 3

Examples of industry benchmark studies

Example 1: Healthcase industry study (USA)

Example: Healthcase industry study (USA)

Study Design
Introduction:
Based on what you know, how negative or positive is your image of PROVIDER?
Which of the following brand attributes, if any, do you associate with PROVIDER?
Post-task:
Self-Reported Success
Ease of Use
Satisfaction with amount of time to complete task
What did you learn that you previously did not know?
What did you like?
What did you dislike?
Problems or frustrations while completing task(s)
Wrap-up:
Overall satisfaction
Likelihood to purchase
Likelihood to recommend : (Net Promoter Score )
Post-task brand image
Post-study brand attributes
Security and Privacy Likes/Dislikes with site
Introduction Screener N=200 Aetna n-=50 BCBS Wrap-up Task 1 Task 2 Kaiser Permanente United Task1: Task 2 Task 1 Task 2 Task 1 Task 2 n-=50 n-=50 n-=50

Brand Perception: Pre/Post All sites had a positive lift in brand perception. +2% +20% +22% +14%

Brand Pillar Metrics: Pre vs. Post Differences United showed a 26% increase on the ‘helpful’ attribute and maintained “professional” while other sites had a decrease on these two attributes. All sites showed a sharp increase in the “rude” attribute.

Overall Satisfaction Panelists were most satisfied with Blue Cross Blue Shield and Aetna’s site when researching and getting rates for Medicare Supplemental Insurance. Differences of 17% or greater are significant at 90% CI

Net Promoter Score (NPS) When asked if they would recommend PROVIDER to a colleague or friend, all sites had relatively low NEGATIVE NPS with Aetna and United Healthcare being the lowest. Detractors Promoters NPS % of Promoters (9s and 10s) Detractors (0 through 6) = - 0 1 2 3 4 5 6 7 8 9 10 NPS Aetna 12% 8% 4% 2% 10% 12% 14% 8% 18% 8% 4% -50% BCBS 8% 4% 6% 10% 2% 8% 14% 10% 4% 10% 24% -18% KP 18% 6% 8% 4% 4% 6% 6% 16% 8% 8% 16% -28% United 10% 4% 6% 6% 4% 22% 16% 8% 10% 8% 6% -54%

Task 1: Learn & Explore Aetna BCBS Kaiser Permanente United The success rate for the “learn and explore” task, although the frustration rate was above 40% for all sites showing room for improvement.

Usability Metrics: Learn & Explore Blue Cross Blue Shield had the highest mean usability scores for task 1.

Blue Cross Blue Shield Liked Best: Learn & Explore Ease of navigation: “ I liked the ease of finding information directly relating to the Blue Shield’s Medicare Supplement insurance.” “ it was all user friendly....I think even people who don't use web sites a lot would be able to navigate” Comparison charts: “ clearly divided options for each plan” “ the graphic showing the differences between the supplemental plans”

Top Frustrations: Learn & Explore About half of the panelists on all sites had some sort of frustration with learning about Medicare Supplemental Insurance online. Opportunities

Example 2: Used cars websites industry study (Spain)

Example: Used cars websites industry study (Spain) USERS SAMPLE
50 users in their natural context of use (home).
Users have medium/high level of internet experience
Over 18 years old
Users use or have used Internet as a means to finding information about how to buy a second-hand car
TESTED WEBS
Supermotor.com
Autocity.com
Autoscout24.es
Coches.net
Segundamano.es
METHODOLOGY
Remote testing using UserZoom technology with real end users.
‘ Task-based analysis’: users must try to solve real tasks
Field work dates: November 2008
Place: Online, users in Spain

Example: Used cars websites industry study (Spain) Tasks Description 1. Free search of a car model Users search freely on the Internet for any car model Note: 50 users completed this task 2. Guided search for an specific car model on a selected website The task requests users to search for an specific car model, on each of the second-hand cars web sites. Note: 10 users per website evaluated

How do users search for a second-hand car on the Internet? Example: Used cars websites industry study (Spain) % I have used search engines 61 I have surfed just by directly inserting the Website name of the page 21 I used both methods 19

Example: Used cars websites industry study (Spain) WEBS Effectiveness Efficiency Time Clicks Autocity.com 90% 3:31' 21 Autoscout24.es 80% 2:08' 29 Coches.net 90% 4:00' 57 Segundamano.es 90% 2:22' 24 Supermotor.com 60% 3:55' 45

Example: Used cars websites industry study (Spain) Easiest Website to search for a car Autocity.com Autoscout24.es Coches.net Segundamano.es Supermotor.com Media It is easy to start the car search 5.9 6.6 6.9 6 5.5 6,2 The searcher fields are easy to fill up and understand 6 6.6 7 6.3 6.2 6,4 The searcher has the requiered options to do a search 5.3 6.6 6.8 5.5 5 5,9 Important information as price, car age, milage, model, is available 6 6.6 6.8 6.2 4.7 6,1

Task analisys: Main problems Example: Used cars websites industry study (Spain) Autocity.com Autoscout24.es Coches.net Segundamano.es Supermotor.com Average I have the required information to make a buying decission 5.2 5.1 6.7 5.3 5 5,5 Webpage shows car features, extras and accessories 5.3 5.7 6.5 5.2 4.8 5,5 I can clearly see the final price and included concepts 6 6.1 5.4 5.9 5.8 5,8 Photographs help to have a clear idea of the car. 6 6.2 6.6 5.9 4.2 5,8 I have all the information to contact the seller 5.5 6.1 6.4 5.5 5.6 5,8 Too much advertising quantity 4.9 5.4 6.1 5.5 4.2 5,2

Example 3: Continuous Benchmarking (Marketing + Usability)

Online scorecard
Continuous benchmarking (Marketing + Usability )

Continuous benchmarking (Marketing + Usability )

Continuous benchmarking (Marketing + Usability)

Examples

UserZoom at a glance
Leading online user experience research software company
Developes a proprietary on-demand software application for online research
Built for & by UX & marketing professionals
Offers a cost-effective, time-saving methodology
Has 9 years of experience in UX research & consulting
Has offices in Sunnyvale (CA), London (UK) and Barcelona (Spain)

Key Lime Interactive at a glance
Experienced market research and usability professionals
Over 10 years of consulting experience testing Internet 500 websites and providing actionable results
Most consultants have Masters in Human-Computer Interaction, Psychology, Industry Engineering or related field
Cross-Industry Expertise
Ability to scorecard against competition
Ability to suggest cross-industry recommendations to improve ease of use
Independent Third Party
Impartial with no-hidden agenda
Ability to work with business team and developers to “get everyone onboard”
Technology-agnostic
Can work with any solution or recommend the best solution to meet your specific needs
Expert Reviews

What makes us proud?

Questions?

Would you like a demo?
Contact us and we’ll offer you a chance to see and try out our technology
Do you have any questions? We’re sure you have many questions. Don’t hesitate to contact us and we’ll be glad to answer them.
USA Office
440 N. Wolfe Rd. - Sunnyvale, CA 94085 (USA) · [t] +1 (408) 524 7445
UK Office
AylesburyHouse, 17-18 AylesburyStreet, London EC1R 0DB(UK) · [t] + 44 (0)20 7193 2171
Spain Offices
Av. Diagonal 419, 3º 2ª - 08008 Barcelona - T. +34 93 414 75 54 - F. +34 93 368 42 95
Gran Vía 62, 7º Izda. - 28013 Madrid - T. +34 91 542 62 52 - F. +34 91 298 08 92
Contact us at [email_address] Members of

http://www.scoop.it	182
http://sprmario.posterous.com	35
http://www.slideshare.net	11
http://sprmario.tumblr.com	4
http://posterous.com	1
http://webcache.googleusercontent.com	1

UserZoom Webinar: How to Conduct Web Customer Experience Benchmarking

by UserZoom on Feb 26, 2010

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 234

Statistics

UserZoom Webinar: How to Conduct Web Customer Experience Benchmarking Presentation Transcript