You can't manage what you can't measure, so... How do you actually measure user experience? In this webinar we covered what, why, and how to conduct website user experience & usability benchmarking. We discussed how to effectively measure the quality of a website's user experience across various competitors, within one industry, across time, using an online quantitative research methodology commonly referred to as "unmoderated remote usability testing."

1. February 2010

6. What is UX Benchmarking & why is it important? Ok… So how do we define UX Benchmarking ?

8. Some may wonder: Is UX Benchmarking… What is UX Benchmarking & why is it important? ?

10. Why is it now more important than ever before? What is UX Benchmarking & why is it important? It’s all about competitiveness, productivity, & efficiency

11. ‘ You can’t manage what you can’t measure’ Attributed to Peter Drucker, Mgt Guru What is UX Benchmarking & why is it important? The question is: How are you actually measuring UX and defining your benchmark?

13. How to conduct UX Benchmarking studies

15. How to conduct UX Benchmarking studies Graph from Bill Albert, Bentley University

27. Example 1: Healthcase industry study (USA)

28. Example: Healthcase industry study (USA)

30. Brand Perception: Pre/Post All sites had a positive lift in brand perception. +2% +20% +22% +14%

31. Brand Pillar Metrics: Pre vs. Post Differences United showed a 26% increase on the ‘helpful’ attribute and maintained “professional” while other sites had a decrease on these two attributes. All sites showed a sharp increase in the “rude” attribute.

32. Overall Satisfaction Panelists were most satisfied with Blue Cross Blue Shield and Aetna’s site when researching and getting rates for Medicare Supplemental Insurance. Differences of 17% or greater are significant at 90% CI

33. Net Promoter Score (NPS) When asked if they would recommend PROVIDER to a colleague or friend, all sites had relatively low NEGATIVE NPS with Aetna and United Healthcare being the lowest. Detractors Promoters NPS % of Promoters (9s and 10s) Detractors (0 through 6) = - 0 1 2 3 4 5 6 7 8 9 10 NPS Aetna 12% 8% 4% 2% 10% 12% 14% 8% 18% 8% 4% -50% BCBS 8% 4% 6% 10% 2% 8% 14% 10% 4% 10% 24% -18% KP 18% 6% 8% 4% 4% 6% 6% 16% 8% 8% 16% -28% United 10% 4% 6% 6% 4% 22% 16% 8% 10% 8% 6% -54%

34. Task 1: Learn & Explore Aetna BCBS Kaiser Permanente United The success rate for the “learn and explore” task, although the frustration rate was above 40% for all sites showing room for improvement.

35. Usability Metrics: Learn & Explore Blue Cross Blue Shield had the highest mean usability scores for task 1.

36. Blue Cross Blue Shield Liked Best: Learn & Explore Ease of navigation: “ I liked the ease of finding information directly relating to the Blue Shield’s Medicare Supplement insurance.” “ it was all user friendly....I think even people who don't use web sites a lot would be able to navigate” Comparison charts: “ clearly divided options for each plan” “ the graphic showing the differences between the supplemental plans”

37. Top Frustrations: Learn & Explore About half of the panelists on all sites had some sort of frustration with learning about Medicare Supplemental Insurance online. Opportunities

38. Example 2: Used cars websites industry study (Spain)

40. Example: Used cars websites industry study (Spain) Tasks Description 1. Free search of a car model Users search freely on the Internet for any car model Note: 50 users completed this task 2. Guided search for an specific car model on a selected website The task requests users to search for an specific car model, on each of the second-hand cars web sites. Note: 10 users per website evaluated

41. How do users search for a second-hand car on the Internet? Example: Used cars websites industry study (Spain) % I have used search engines 61 I have surfed just by directly inserting the Website name of the page 21 I used both methods 19

42. Example: Used cars websites industry study (Spain) WEBS Effectiveness Efficiency Time Clicks Autocity.com 90% 3:31' 21 Autoscout24.es 80% 2:08' 29 Coches.net 90% 4:00' 57 Segundamano.es 90% 2:22' 24 Supermotor.com 60% 3:55' 45

43. Example: Used cars websites industry study (Spain) Easiest Website to search for a car Autocity.com Autoscout24.es Coches.net Segundamano.es Supermotor.com Media It is easy to start the car search 5.9 6.6 6.9 6 5.5 6,2 The searcher fields are easy to fill up and understand 6 6.6 7 6.3 6.2 6,4 The searcher has the requiered options to do a search 5.3 6.6 6.8 5.5 5 5,9 Important information as price, car age, milage, model, is available 6 6.6 6.8 6.2 4.7 6,1

44. Task analisys: Main problems Example: Used cars websites industry study (Spain) Autocity.com Autoscout24.es Coches.net Segundamano.es Supermotor.com Average I have the required information to make a buying decission 5.2 5.1 6.7 5.3 5 5,5 Webpage shows car features, extras and accessories 5.3 5.7 6.5 5.2 4.8 5,5 I can clearly see the final price and included concepts 6 6.1 5.4 5.9 5.8 5,8 Photographs help to have a clear idea of the car. 6 6.2 6.6 5.9 4.2 5,8 I have all the information to contact the seller 5.5 6.1 6.4 5.5 5.6 5,8 Too much advertising quantity 4.9 5.4 6.1 5.5 4.2 5,2

45. Example 3: Continuous Benchmarking (Marketing + Usability)

47. Continuous benchmarking (Marketing + Usability )

48. Continuous benchmarking (Marketing + Usability )

49. Continuous benchmarking (Marketing + Usability )

50. Continuous benchmarking (Marketing + Usability)

54. What makes us proud?

55. Questions?