• Save
How to Conduct UX Benchmarking
 

How to Conduct UX Benchmarking

on

  • 10,232 views

 

Statistics

Views

Total Views
10,232
Views on SlideShare
9,886
Embed Views
346

Actions

Likes
22
Downloads
0
Comments
0

3 Embeds 346

http://info.userzoom.com 309
http://storify.com 34
https://twitter.com 3

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
  • In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
  • In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
  • In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.

How to Conduct UX Benchmarking How to Conduct UX Benchmarking Presentation Transcript

  • Webinar:How to Conduct UX BenchmarkingMay 2nd, 2012
  • Intros Alfonso de la Nuez Kim Oslob Co-Founder & Co-CEO Sr. Client Services Director Twitter Hashtag #uzwebinar
  • Overview What we mean by UX Benchmarking and why it’s important? How to conduct UX Benchmarking studies, using remote unmoderated usability testing (aka task-based surveys)? What are some of the UX metrics you should look for? How to measure the quality of a website’s user experience across various competitors? A case study: American Airlines vs Delta Airlines Websites
  • What is Benchmarking?Definition of Benchmarking: The process of comparing the business processes and performance metrics including cost, cycle time, productivity, or quality to another that is widely considered to be an industry standard benchmark or best practice. It provides a snapshot of the performance of your business and helps you understand where you are in relation to a particular standard.
  • What is UX?Definition of UX: It’s about how a person feels about using a system. It highlights the experiential, affective, meaningful and valuable aspects of HCI and product ownership, but it also covers a person’s perceptions of the practical aspects such as utility, ease of use and efficiency of the system. User experience is subjective in nature, as it is about an individual’s feelings and thoughts about the system. User experience is dynamic, because it changes over time as the circumstances change.
  • Ok… So what is UX Benchmarking?
  • Is UX Benchmarking… Not really
  • What is UX Benchmarking? The process of comparing a website’s (or any digital product) performance, specifically in relation to UX metrics, including among others effectiveness, efficiency, overall satisfaction and quality perception, to another that is widely considered to be an industry standard benchmark or best practice UX Benchmarking provides a snapshot of the performance of your product and helps you understand where you are in relation to a particular standard.
  • The question we’d like to address today is: How do you conduct UX Benchmarking?
  • In order to Benchmark, first you need to measure
  • Why is UX Benchmarking important? Measuring the quality of UX is more important than ever Consumers have gained a lot of power in recent years More competition, more maturity in the online marketplace You can’t really manage what you can’t measure. UX Pros must deliver more measured value. Mgmt Teams LOVE metrics… UX Benchmarking allows you to MEASURE performance in two ways: 1. Baseline studies: Measure UX over time, compare evolution of your own product and/or service 2. Against competitors within your industry
  • Common UX research & design challenges Difficulties obtaining objective, statistically significant, actionable data Obtaining idea adoption from stakeholders Geographic/demographic limitations UX research is costly and time-consuming UX Benchmarking is therefore quite challenging This is where large-scale, remote unmoderated usability testing really helps
  • 3 pillars of UX Benchmarking 1. Plan & Goals UX Benchmarking 2. UX research 3. Recruiting participants methods & tools
  • 4 must-dos with UX Benchmarking1. Plan ahead:  Define measurable & comparable UX-related KPIs (will show them in the case study ahead)  Select competitors to benchmark against  Define target audience  Identify common site activities/tasks (recommended between-subjects design)2. Conduct research using task-based testing (see methods and tools later on)3. Gather both quantitative & qualitative data  Use the quantitative data to test for statistically significant differences between means or proportions  Use qualitative data to support your findings and understand the why4. Research at least twice a year (or even quarterly) to measure design changes and trends in expectations
  • Define measurable & comparable UX-related KPIs  Jeff Sauro, Measuringusability.com
  • Define measurable & comparable UX-related KPIs  Jeff Sauro, Measuringusability.com
  • UX research methods & toolsWe recommend the following combination:• Remote Unmoderated Usability Testing (or task-based survey tools)• Remote Unmoderated Usability Testing Video Tools
  • What about recruiting participants?  3 choices to recruit participants: 1. Panel vendor (SSI, Research Now, etc) 2. Private Mailing List 3. Intercept visitors
  • Case study:Benchmark Study to compare AA.com vs Delta.com
  • Study Background • Competitive benchmark of American Airlines vs. Delta Study Air Lines websites • Using task-based surveys, or Remote Unmoderated Usability Testing • Date: April 24 – April 25, 2012 • 100 participants (50 per site) Panel • Must fly for business or leisure more than twice a year • Must use websites to make travel arrangements • Panel source: Survey Sampling International (SSI) Tasks • Go through the entire process of booking a flight • Locate the total price for checking in two bagswww.userzoom.com
  • Task-based Surveys (aka unmoderated large-scale remote usability testing) Go through the entire process of booking a flight Go through the entire process of booking a flight • Large sample of participants take the study simultaneously • Natural context • International/domestic testing • No moderation needed • Usability + Behavior + Feedbackwww.userzoom.com
  • Study Design American Airlines Task 1 Task 2 Screener Pre-task questions Follow-up q’s Delta Air Lines Task 1 Task 2www.userzoom.com
  • Study Design Pre-task brand attributes: Which of the following words do Pre-task you associate with the AIRLINE brand? Go through the entire process of booking a flight Tasks Locate the total price for checking in two bags Self-reported success How satisfied were you with the process of booking a flight? Ease of Use Post-tasks Ease of navigation Clear where to start Satisfaction with amount of time to complete task Overall rating Post-task brand attributes Wrap-up What did you like the most about the AIRLINE website What did you like the least about the AIRLINE website Likelihood to recommend (Net Promoter Score )www.userzoom.com
  • Participant Profile Age range Gender Under 18 14% 18-20 8% 31% 21-30 45% Male 31-40 55% Female 41-50 27% 20% 51-60 60+ How often do you fly yearly (for business or leisure)? I dont fly regularly 10+ times Participants were 5-10 times required to book their travel online in 2-4 times order to qualify 1 time 0% 20% 40% 60% 80%www.userzoom.com
  • Brand Perception % Pre Post Change Pre Post % Change Friendly 41% 14% -27% 42% 10% -32% Customer 27% 27% 0% 26% 40% +14% Focused Safe 57% 27% -30% 44% 24% -20% Good 41% 31% -10% 36% 22% -14% valuewww.userzoom.com
  • Task 1: Finding a round-trip flightYour Task: Find a round-trip flight departing May 1st and returning May 6th that you would consider booking on Delta AirLines/American Airlines website. You are travelling from San Francisco, CA to New York, NY. After that, please go throughthe entire process of booking a flight until you get to the "Purchase" button. Delta Air Lines American Airlines Task Success Task Success Median Success Time Median Success Time 3:43 3:56On both studies the majority of participants self-reported being able to successfully complete thebooking process. There was not a significant difference in success rate or completion time (90%Confidence Level).www.userzoom.com
  • Task 1: Finding a round-trip flightPost-Task QuestionnaireTop Box Scores*How satisfied were you with the processof booking the flight? 50% 55%Booking the flight was easy 64% 67%It was easy to navigate the website 64% 65%It was clear how to start booking theflight 62% 82%I am satisfied with the amount of time ittook to complete the task 64% 80%Participants that completed the American Airlines task reported being clearer on where to start andmore satisfied with the time it took to complete booking (90% Confidence Level). All percentagesrepresent top box scores.*Top box scores show the percentage of participants that agreed strongly with the statements (selected 6-7 on a 1-7 scale)www.userzoom.com
  • Task 2: Find the price for checking 2 bagsYour Task: Find the total price for checking in two bags for your flight from San Francisco to New York.You purchased a standard economy class ticket. Delta Air Lines American Airlines Task Success Task Success Median Success Time Median Success Time 1:48 2:00On both studies the majority of participants were not able to successfully identify the price ofchecking two bags. There was not a significant difference in success rate or completion time (90%Confidence Level).www.userzoom.com
  • Task 2: Heatmaps - First ClickOn both Delta and American Airlines participants used similar strategies when searching for baggagechecking fees. The most commonly used strategy was to start by clicking the flight booking formfields/buttons (28% Delta “To Airport” field and 24% American Airlines “Find Flights” button).Participants that first located and clicked on a “Baggage” had higher success rates (70% Delta and 60%American Airlines).www.userzoom.com
  • Task 2: Find the price for checking 2 bags Post-Task Questionnaire Top Box Scores*How satisfied were you with the processof searching for the total price for 30% 20%checking in two bags? Finding the total price was easy 32% 24%It was easy to navigate the website 38% 36%It was clear how to start searching forthe total price 32% 32%I am satisfied with the amount of time ittook to complete the task 34% 34%Participants that completed the Delta task reported being more satisfied with the process and ease ofthe task. The differences are not significant (90% Confidence Level). All percentages represent topbox scores.*Top box scores show the percentage of participants that agreed strongly with the statements (selected 6-7 on a 1-7 scale) www.userzoom.com
  • Final Questionnaire: What did you like the most about AA/Delta?Representative Quotes: Representative Quotes:“very easy to navigate” “the prices”“easy to find flights” “easy to navigate”“The modern look of it”Several of the Delta participants commented that they liked the appearance of the Delta website.There were multiple American Airlines participants that responded liking American Airline’s pricing.www.userzoom.com
  • Final Questionnaire: What did you like the least about AA/Delta?Representative Quotes: Representative Quotes:“the pricing” “finding information regarding“Not easy to find bag fees” baggage pricing” “Prices” “finding things other than straight flight booking”For both websites participants commented that the pricing was high and that it was difficult to locatethe bag fees.www.userzoom.com
  • Final Questionnaire: Net Promoter Score NPS = % of Promoters (9s and 10s) - Detractors (0 through 6) 0 1 2 3 4 5 6 7 8 9 10 NPS 0% 4% 4% 4% 6% 18% 6% 10% 22% 6% 20% -16% Delta Air Lines 4% 6% 4% 0% 4% 10% 12% 18% 14% 12% 18% -10% American Airlines Detractors Promoters When asked if they would recommend the Delta/AA website to a family member, friend, or colleague both sites had negative NPS scores.www.userzoom.com
  • Some takeaways…1. UX Benchmarking, while challenging, is not mission impossible2. Given the right plan, methods and tools, results are really valuable3. UX Pros must deliver more measured value4. There are plenty of UX metrics to be measured, including:  Task success  Time on task  NPS  Reasons for problems, abandonment  The WHYs  Etc.
  • Q&A
  • Thanks so much for your time! Check out our upcoming news and events at www.userzoom.comSunnyvale (USA) Barcelona (Spain) Cheshire (UK)440 N. Wolfe Rd. Sunnyvale, Av. Diagonal 419 3º 2ª 08008 Booths Park. Chelford Road, Knutsford,CA 94085 Barcelona Phone: +34 93 414 Cheshire WA16 8GSPhone: +1 (408) 524 7445 7554 Fax: +34 93 209 8380 Phone: + 44 (0)1565 759890Contact: Alfonso de la Nuez Contact: Xavier Mestres Cell: + 44 (0)7900 472 920 Contact: Arthur Moan