This document outlines a framework for benchmarking the user experience of websites. It discusses the need to measure user experience in order to manage and improve it. A proposed method involves setting goals, understanding user journeys and contexts, selecting relevant competitors, and choosing user experience factors and metrics to score sites on. Examples are given for benchmarking a government site aiming to reduce phone calls and an airline app wanting more mobile bookings. Key aspects include having a structured expert evaluation, comparing sites to norms and standards, and balancing objective metrics with qualitative nuance. Repeated benchmarking and adjusting goals and standards over time are also discussed.
13. 1. Where we are now (benchmarking)
2. Where we want to get to (what does success
look like in terms of UX)
3. How do we know if we got there (breaking
success down into meaningful metrics
related to UX)
4. Monitor changes and improvements
13
Measurement plan
Measure
17. 17
UX Score
What works for us:
• Goes beyond just usability and looks at other
useful attributes of the user experience
• Can be compared and tracked over time
What doesn’t work as well for us:
• Reduction to a single metric
• One size fits all approach
20. 20
Google HEART Framework
What works for us:
• Focus on using categories that suit goals
• Process of mapping goals > signals > metrics
What doesn’t work as well for us:
• Implicit reliance on large data sets
• How can we improve?
22. 22
In our heads…
Specific
requests
from clients
An
objective,
repeatable
process
Why aren’t more
people booking
flights on
mobile?
How can we make
sure we’re offering
a best-in-class retail
experience?
23. 23
Our method
A structured expert evaluation that assesses how
well websites meet (or exceed) the norms and
standards within an industry.
A snapshot of the state of UX amongst competitors.
32. 32
Identify key user journey
a. What is the user journey that is most relevant to
meeting your goal?
b. What are the discrete steps in that journey?
36. 36
Select competitors
a. Consider the market (direct competitors)
b. Consider the goal (aspirations)
c. Research the state of user experience in the
industry (mobile, interfaces)
39. 39
Our method
A structured expert evaluation that assesses how
well websites meet (or exceed) the norms and
standards within an industry.
40. 40
Choosing metrics: step 1
Select UX factors
These are the key steps in the user journey plus
relevant UX elements and heuristics.
Good news: you have already identified the key
steps in the user journey!
43. 43
Choose factors
a. Steps in the user journey
b. Relevant UX elements and heuristics
Which factors are most relevant to the site and to the
goals?
44. 44
1. First impressions
2. Find products (or information)
3. Select products
4. Configure products
5. Get help and support
6. Checkout
7. Post-order communications
8. Delivery
9. Returns
Steps in the user journey
46. 46
Usability heuristics (Nielsen)
1. Visibility of system status
2. Match between system and real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognise, diagnose and recover from errors
10. Help and documentation
www.nngroup.com/articles/ten-usability-heuristics/
47. 47
1. Aesthetics
2. Anticipation
3. Autonomy
4. Colour
5. Consistency
6. Defaults
7. Discoverability
8. Efficiency of the User
9. Explorable Interfaces
10. Fitts' Law
11. Human-Interface Objects
12. Latency Reduction
13. Learnability
14. Metaphors
15. Protect Users' Work
16. Readability
17. Simplicity
18. Track State
19. Visible Interfaces
Principles of interaction design (Tognazzini)
www.asktog.com/atc/principles-of-interaction-design/
48. 48
Choosing factors
Which UX factors are most relevant to achieving your
goals?
What might you want to measure on
• A government site, where the goal is to reduce the
number of incoming phone calls?
• An airline app, where the goal is to increase mobile
bookings?
49. 49
Choosing factors
Government site:
• Find information
• Checkout*
• Navigation and menus
• Search
• Readability
• Match between system and real world
• Aesthetic and minimalist design
• Learnability
53. 53
Choosing metrics: step 2
Select UX factors
These are the key steps in the user journey plus
relevant UX elements and heuristics.
Select metrics
We score each UX factor based on the presence or
absence of our selected metrics
54. 54
UX Factor + Metrics
Checkout
Ease of passenger
details
Range of payment
options
Ease of payment Trust / security
55. 55
Our method
A structured expert evaluation that assesses how
well websites meet (or exceed) the norms and
standards within an industry.
Harmful -1
Absent 0
Present 1
Excellent 2
59. 59
Parting questions
What is the appropriate balance between 1/0 ratings and 2/-1
ratings? Objectivity vs. nuance…
What if you want to engage in repeated benchmarking?
What if the norms and standards change over time? What if
the goals change?
How can you build this process into a measurement plan?
First off – old business advice – if you cant measure it you can manage I t- TRUE? NOT SO TRUE? WHY? Qualitative – ethnographical – often data hard to categorise… still very important and helpful. Not true… but Story re pedometer…. Original studies from standform on pedometers and how they encourage people to take more phsycical activity – found just by tracking it – make visable – 27% physical activity. At very least – measure it, MORE LIKELY TO ACT on it .
Data ts about helping you make decisions…Knowledge is power…
More than knowledge… learning and turning that data into insights and knowledge to improve product and services and innovate
ALSO about making that accessible to everyone in the team – knowledge sharing – team clear idea where they are and where they want to go to.
FROM UX’ser pont of view…. Doing some research about training programmes and what situation they are facing or more about FRUSTRATIONS they are facing…
Important if we want a seat at that TABLE (what table – does not exist?.
Elevates UX work doing from ux teams into management teams – speak their language. Becomes part of the conversation.
ALL agreed its good, so what is the problem?
What are YOU doing to benchmark or measure the user experience? What is working well/not so well.
The problem is not Data… Data Data everywhere. What
Hear a lot terminology about goals, objectives
What does it mean to you in your particular sitautian as UX Manager, communication manager etc.
Sure, great data on analytics, conversions etc. BUT what about USER Experience?..
Where and HOW does UX fit in?
And what do UX KPI or goals look like?
A lot of good will out there.. Some great statements. Hear a lot of these things… but what does that actually mean?
Is good UX a good goal? Yes, certainly is – BUT NOT IN ITSELF – we need to break down what that means to each organisation and vary depending on situation
HOW do we do that? How i
SIR SETH
CAUTIONARY NOTE – Measurement not an END in itself – My twitter example…
Focus on goals/outcomes and how UX can support us in achieving those goals..
Plays that Jared speaking about How UX strategy fits it in with overall strategy. NThink about overall goals and objectives of the organiational.
Measurement plan
Subdivision of that strategy.. Measurement strategy – what does success look like
That’s what we are here today, talk about this part of measurement plan
This is Eugene - A simple 10-question survey, the UX Score goes beyond traditional usability measurements – tradditoanl SUS – system usability scores
and evaluates usability, usefulness, and aesthetics. I
Quite a bit of research what factors or items or used in user experience
We can deal with small sample sizes – instead of relying on large scale surveys or analytics, we use a structured expert evaluation approach.
We clarify goals in consultation with clients, and make sure that the user journeys and metrics we look at are relevant to those goals.
And the bulk of this tutorial will be teaching an approach for selecting metrics to benchmark and assess.
We can deal with small sample sizes – instead of relying on large scale surveys or analytics, we use a structured expert evaluation approach.
We clarify goals in consultation with clients, and make sure that the user journeys and metrics we look at are relevant to those goals.
And the bulk of this tutorial will be teaching an approach for selecting metrics to benchmark and assess.
Our method is not a one size fits all approach, but rather a bespoke service that relies upon understanding the market and being prepared to conduct a structured expert evaluation and competitor analysis.
It is not easy. But UX benchmarking shouldn’t be – like writing a survey, if its too easy, you’re probably doing it wrong.
Need to understand what the competition is doing.
This is an important step. Take the time to have a look at the full user journey on key competitors’ sites. By understanding what the common practices are, you will have a better understanding of what your customers’ expectations are likely to be.
Take our airline example. If all of your competitors allow travellers to sort flight options by price, and you don’t, then there is a good chance you will meet with some resistance from your customers. These options will become the metrics you will use later, so taking note of what the norms are now will save you time later.
As we’ve discussed before, our approach to metrics is not one size fits all. There are very popular and extremely useful lists of UX heuristics out there, and you can and should feel free to draw from those. But our experience suggests that the aspects of the user journey that are most important to measure may be broader.
The factors you choose will be different on different sites.
My name is Abi Reynolds- colleauge (who will get into detail framework) User Vision – UX agency – work with public and private, cross industry… different countries.. (reason I say that) we have been noticing much more interest in benmarking and measuring User experience.
What that means and how do to go about it a meaningful useful way