Downloaded 27 times
Uploaded byUser Vision
Safety in numbers: A framework for benchmarking the user experience
This document outlines a framework for benchmarking the user experience of websites. It discusses the need to measure user experience in order to manage and improve it. A proposed method involves setting goals, understanding user journeys and contexts, selecting relevant competitors, and choosing user experience factors and metrics to score sites on. Examples are given for benchmarking a government site aiming to reduce phone calls and an airline app wanting more mobile bookings. Key aspects include having a structured expert evaluation, comparing sites to norms and standards, and balancing objective metrics with qualitative nuance. Repeated benchmarking and adjusting goals and standards over time are also discussed.
More Related Content
Similar to Safety in numbers: A framework for benchmarking the user experience
![Number_Guessing_Game_Dsbsbssbzboc[1].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/numberguessinggamedoc1-251206215042-a076fc05-thumbnail.jpg?width=640&height=640&fit=bounds)
Safety in numbers: A framework for benchmarking the user experience
- 1. Safety in Numbers:A framework for benchmarking the user experience Jessica Cameron & Abi Reynolds UX Scotland Thursday, 8 June 2017 1
- 2. If you can’tmeasure it You can’t manage it 2
- 3. Arron Walter, UXMatters Wisdom Knowledge Information Data 3
- 4. I want othersto see the value of what we are doing. I want to see my results make a difference. I want to see my results acted upon. UX folk 4
- 5. If you don’tknow the value of what you are doing, there is no value. 5 Gerry McGovern www.customercarewords.com
- 6. Q. How dowe go about benchmarking or measuring the user experience? 6
- 7. 7 Open GI –August 2015 7
- 8.
- 9.
- 10. More customer focused Abetter user experience, easy and simple to use Customer first Put the customer at the core Increase satisfaction 10
- 11.
- 12. Start with strategy 12 VisionLandscape Roadmap Measure
- 13. 1. Where weare now (benchmarking) 2. Where we want to get to (what does success look like in terms of UX) 3. How do we know if we got there (breaking success down into meaningful metrics related to UX) 4. Monitor changes and improvements 13 Measurement plan Measure
- 14.
- 15.
- 16.
- 17. 17 UX Score What worksfor us: • Goes beyond just usability and looks at other useful attributes of the user experience • Can be compared and tracked over time What doesn’t work as well for us: • Reduction to a single metric • One size fits all approach
- 18.
- 19.
- 20. 20 Google HEART Framework Whatworks for us: • Focus on using categories that suit goals • Process of mapping goals > signals > metrics What doesn’t work as well for us: • Implicit reliance on large data sets • How can we improve?
- 21.
- 22. 22 In our heads… Specific requests fromclients An objective, repeatable process Why aren’t more people booking flights on mobile? How can we make sure we’re offering a best-in-class retail experience?
- 23. 23 Our method A structuredexpert evaluation that assesses how well websites meet (or exceed) the norms and standards within an industry. A snapshot of the state of UX amongst competitors.
- 24.
- 25.
- 26. 26 How can UXbenchmarks help improve… …an airline app?
- 27.
- 28.
- 29. 29 Goal To increase thepercentage of users completing bookings on the mobile app
- 30.
- 31.
- 32. 32 Identify key userjourney a. What is the user journey that is most relevant to meeting your goal? b. What are the discrete steps in that journey?
- 33. 33 User journey Book aflight
- 34. 34 Key steps inthe user journey FIND SELECT CHECKOUT
- 35.
- 36. 36 Select competitors a. Considerthe market (direct competitors) b. Consider the goal (aspirations) c. Research the state of user experience in the industry (mobile, interfaces)
- 37. 37 Competitors Mix of directcompetitors, market leaders and innovative new entrants
- 38.
- 39. 39 Our method A structuredexpert evaluation that assesses how well websites meet (or exceed) the norms and standards within an industry.
- 40. 40 Choosing metrics: step1 Select UX factors These are the key steps in the user journey plus relevant UX elements and heuristics. Good news: you have already identified the key steps in the user journey!
- 41. 41 Key steps inthe user journey FIND SELECT CHECKOUT
- 42. 42 UX Factors Which onesare most appropriate?
- 43. 43 Choose factors a. Stepsin the user journey b. Relevant UX elements and heuristics Which factors are most relevant to the site and to the goals?
- 44. 44 1. First impressions 2.Find products (or information) 3. Select products 4. Configure products 5. Get help and support 6. Checkout 7. Post-order communications 8. Delivery 9. Returns Steps in the user journey
- 45. 45 1. Navigation /menus 2. Search 3. Clarity 4. Readability 5. Functionality 6. Visual appeal 7. Confidence and trust 8. Speed UX elements
- 46. 46 Usability heuristics (Nielsen) 1.Visibility of system status 2. Match between system and real world 3. User control and freedom 4. Consistency and standards 5. Error prevention 6. Recognition rather than recall 7. Flexibility and efficiency of use 8. Aesthetic and minimalist design 9. Help users recognise, diagnose and recover from errors 10. Help and documentation www.nngroup.com/articles/ten-usability-heuristics/
- 47. 47 1. Aesthetics 2. Anticipation 3.Autonomy 4. Colour 5. Consistency 6. Defaults 7. Discoverability 8. Efficiency of the User 9. Explorable Interfaces 10. Fitts' Law 11. Human-Interface Objects 12. Latency Reduction 13. Learnability 14. Metaphors 15. Protect Users' Work 16. Readability 17. Simplicity 18. Track State 19. Visible Interfaces Principles of interaction design (Tognazzini) www.asktog.com/atc/principles-of-interaction-design/
- 48. 48 Choosing factors Which UXfactors are most relevant to achieving your goals? What might you want to measure on • A government site, where the goal is to reduce the number of incoming phone calls? • An airline app, where the goal is to increase mobile bookings?
- 49. 49 Choosing factors Government site: •Find information • Checkout* • Navigation and menus • Search • Readability • Match between system and real world • Aesthetic and minimalist design • Learnability
- 50. 50 Choosing factors Airline app: •Select products (flights) • Checkout • Clarity • Visual appeal – aesthetics • Speed • Error prevention • Defaults • Learnability
- 51. 51 UX Factors Flight selection,checkout, clarity, visual appeal, speed
- 52.
- 53. 53 Choosing metrics: step2 Select UX factors These are the key steps in the user journey plus relevant UX elements and heuristics. Select metrics We score each UX factor based on the presence or absence of our selected metrics
- 54. 54 UX Factor +Metrics Checkout Ease of passenger details Range of payment options Ease of payment Trust / security
- 55. 55 Our method A structuredexpert evaluation that assesses how well websites meet (or exceed) the norms and standards within an industry. Harmful -1 Absent 0 Present 1 Excellent 2
- 56.
- 57. 57 CLIENT COMP 1COMP 2 COMP 3 COMP 4 COMP 5 flight selection good poor good good fail good navigation -1 0 -1 1 1 0 changes/error recovery 1 0 0 1 -1 1 comparing options - price 1 1 1 -1 1 1 comparing options - schedule 2 -1 1 0 -1 0 comparing options - sorting 0 -1 2 1 -1 1 clarity good good fail good poor poor distinct steps 1 2 -1 -1 1 0 progress indicator 2 0 1 1 0 0 menus 1 1 -1 1 0 -1 clear icons -1 -1 -1 1 1 -1 size/readability 1 0 0 2 0 1 visual appeal good excellent good good poor excellent design 1 2 1 1 0 1 branding 1 1 1 1 0 2 emotional appeal 0 2 2 1 0 2 consistency 2 0 1 -1 1 1 checkout poor fail* good poor good poor ease of passenger details 0 1 1 0 1 0 range of payment options 2 2 -1 1 0 0 ease of payment 0 -1 1 1 1 1 trust/security 1 0 1 -1 1 -1 speed good good poor poor good good number of steps 1 1 0 1 1 1 page load time 0 1 -1 -1 1 1
- 58.
- 59. 59 Parting questions What isthe appropriate balance between 1/0 ratings and 2/-1 ratings? Objectivity vs. nuance… What if you want to engage in repeated benchmarking? What if the norms and standards change over time? What if the goals change? How can you build this process into a measurement plan?
- 60.
- 61.
Editor's Notes
- #3 First off – old business advice – if you cant measure it you can manage I t- TRUE? NOT SO TRUE? WHY? Qualitative – ethnographical – often data hard to categorise… still very important and helpful. Not true… but Story re pedometer…. Original studies from standform on pedometers and how they encourage people to take more phsycical activity – found just by tracking it – make visable – 27% physical activity. At very least – measure it, MORE LIKELY TO ACT on it . Data ts about helping you make decisions…Knowledge is power…
- #4 More than knowledge… learning and turning that data into insights and knowledge to improve product and services and innovate ALSO about making that accessible to everyone in the team – knowledge sharing – team clear idea where they are and where they want to go to.
- #5 FROM UX’ser pont of view…. Doing some research about training programmes and what situation they are facing or more about FRUSTRATIONS they are facing…
- #6 Important if we want a seat at that TABLE (what table – does not exist?. Elevates UX work doing from ux teams into management teams – speak their language. Becomes part of the conversation. ALL agreed its good, so what is the problem?
- #7 What are YOU doing to benchmark or measure the user experience? What is working well/not so well.
- #8 The problem is not Data… Data Data everywhere. What
- #9 Hear a lot terminology about goals, objectives What does it mean to you in your particular sitautian as UX Manager, communication manager etc. Sure, great data on analytics, conversions etc. BUT what about USER Experience?..
- #10 Where and HOW does UX fit in? And what do UX KPI or goals look like?
- #11 A lot of good will out there.. Some great statements. Hear a lot of these things… but what does that actually mean? Is good UX a good goal? Yes, certainly is – BUT NOT IN ITSELF – we need to break down what that means to each organisation and vary depending on situation HOW do we do that? How i
- #12 SIR SETH CAUTIONARY NOTE – Measurement not an END in itself – My twitter example… Focus on goals/outcomes and how UX can support us in achieving those goals..
- #13 Plays that Jared speaking about How UX strategy fits it in with overall strategy. NThink about overall goals and objectives of the organiational. Measurement plan
- #14 Subdivision of that strategy.. Measurement strategy – what does success look like That’s what we are here today, talk about this part of measurement plan
- #17 This is Eugene - A simple 10-question survey, the UX Score goes beyond traditional usability measurements – tradditoanl SUS – system usability scores and evaluates usability, usefulness, and aesthetics. I
- #18 Quite a bit of research what factors or items or used in user experience
- #23 We can deal with small sample sizes – instead of relying on large scale surveys or analytics, we use a structured expert evaluation approach. We clarify goals in consultation with clients, and make sure that the user journeys and metrics we look at are relevant to those goals. And the bulk of this tutorial will be teaching an approach for selecting metrics to benchmark and assess.
- #24 We can deal with small sample sizes – instead of relying on large scale surveys or analytics, we use a structured expert evaluation approach. We clarify goals in consultation with clients, and make sure that the user journeys and metrics we look at are relevant to those goals. And the bulk of this tutorial will be teaching an approach for selecting metrics to benchmark and assess.
- #25 Our method is not a one size fits all approach, but rather a bespoke service that relies upon understanding the market and being prepared to conduct a structured expert evaluation and competitor analysis. It is not easy. But UX benchmarking shouldn’t be – like writing a survey, if its too easy, you’re probably doing it wrong.
- #36 Need to understand what the competition is doing. This is an important step. Take the time to have a look at the full user journey on key competitors’ sites. By understanding what the common practices are, you will have a better understanding of what your customers’ expectations are likely to be. Take our airline example. If all of your competitors allow travellers to sort flight options by price, and you don’t, then there is a good chance you will meet with some resistance from your customers. These options will become the metrics you will use later, so taking note of what the norms are now will save you time later.
- #43 As we’ve discussed before, our approach to metrics is not one size fits all. There are very popular and extremely useful lists of UX heuristics out there, and you can and should feel free to draw from those. But our experience suggests that the aspects of the user journey that are most important to measure may be broader.
- #49 The factors you choose will be different on different sites.
- #62 My name is Abi Reynolds- colleauge (who will get into detail framework) User Vision – UX agency – work with public and private, cross industry… different countries.. (reason I say that) we have been noticing much more interest in benmarking and measuring User experience. What that means and how do to go about it a meaningful useful way