Hi & welcome My name is … with … Today we’ll be talking about usability testing – what it is and how to do it If that’s not what you were expecting or are interested in, you may want to check out another session Also, if you attended this session last year, there are not a lot of differences, so I would suggest checking out another session as well First, I’d like to get a show of hands Raise your hand if you have experience with usability testing OK, let’s get started
Here’s the agenda for the session We’ll end with a summary and Q&A
This is the definition of usability from the international standard ISO 9241-11 Users – anyone who interacts with the system to accomplish a goal System – could be software, hardware, website, or any combination of these Goals – are what usability is ultimately measured against – can the users do what they want to do? Usability has 3 components: effectiveness, efficiency, and satisfaction Efficiency: How quickly can users accomplish their goals? Effectiveness: How accurately and completely can users accomplish their goals? Satisfaction: How satisfied are users with their experience with the system?
Process – structured – not free form – not just “what do you think of this” Evaluation – not just testing, but understanding Representative users – people who are either using the system or could possibly do so in the future
To illustrate what all this means in practice, I’d like to show you a short video of a usability test of a job site I’d like you to pay attention to The role of the user doing the test in the video The role of the person running the test
Things to notice User in the video is completing a specific task – filling out a job application form She is encouraged to talk about what she is doing and how she is reacting to the system The person running the test (the facilitator) is acknowledging and asking clarifying questions
Goals: What the users want to do with the system Tasks: How they would accomplish this Example: Alarm clock Goal: Get up in time for an 8AM conference call Task: Set the alarm for 6AM Goals are important to the user, but can’t be tested Tasks can!
Usability testing can be done - During the design process After the design process is complete After is good, but during is better, because the results can be incorporated in the design
Two kinds of usability testing: quantitative and qualitative - Quantitative: focus on metrics – task completion, error rate, satisfaction Qualitative: focus on behaviour – reactions, insights, think aloud comments
How many is a few? Opinions differ! Jakob Nielsen: 3-5 users sufficient for qualitative testing; 10-20 for quantitative testing depending on desired confidence in results Who are representative users: People whose goals and level of expertise with the system match those of the target audience This usually doesn’t include the designers or developers of the system!
What you need to do What you need to keep in mind
The facilitator is also sometimes referred to as the moderator
First, you want to determine what you want to get out of the test Understand users’ pain points Compare alternative designs Get metrics like completion times and success rates The answer will help you determine how to structure the test and how many people to test with
Usability tests are not complicated to run, but they do require a lot of planning in advance This includes -Finding and scheduling participants Developing appropriate tasks Creating test instructions – you need to make sure that these are clear, concise, and simple Hardware and software setup
The day before the test, try going through it yourself You can also run a “pilot test” with another person involved in the project Make sure that - You or your pilot tester can do the test in the time allotted - All the functions you will be testing are working and accessible from your test machine
Remind them to think out loud. Tell me what you see on this screen/page. Tell me what you’re going to do next. Tell me what you expect to happen when you click this button.
During the test, avoid hinting to participants what they should do next If they say “I’m not sure what to do next”, ask “What do you think you should do next?” Similarly, avoid influencing participants’ impressions of the system Instead, ask open ended questions like “What do you like/dislike about this aspect of the system?” You’re looking for specifics, not necessarily because they are important, but because they help you understand what the user is reacting to
As a facilitator, during the test you need to be patient, reassuring, and diplomatic It’s easy for test participants to blame themselves and feel bad when they run into problems Make it clear to participants that You’re testing the system, not them You know that they are not stupid
As an observer, you are not involved in the actual running of the usability test All you have to do is listen, pay close attention to the user’s interaction with the system, and take detailed notes Remember to be quiet and avoid distracting users!
As an observer, here are the types of things you are looking for: Does the user get the purpose of screens/features? Can they find their way around the interface? Does the terminology make sense to them? “ Head slappers” – sudden insights because of what the participant does or doesn’t do “ Shocks” – behaviour that challenges your assumptions Passion – what does the user love or hate about the system – important to note this and bring this back to the design team
Don’t pay too much attention to opinions users express during the test These are unreliable – people may exaggerate them because they think that’s what you’re looking for Instead, pay attention to what they do, and how they explain what they do
As an observer, you may feel disappointed or even angry if the users’ behaviour doesn’t meet your expectations You may also be tempted to jump to conclusions after the first participant Don’t focus on this – focus on keeping an open mind to what’s going on during the test Any questions so far?
Does anyone not have a card? Does anyone not have a group of 3? If you don’t, feel free to join another group as an observer Let me know if your group doesn’t have a pen
Take about 5 minutes to do the task. Facilitator: Once your participant is done, ask them to answer the post-test question Observer: Write down the number of tasks completed and the participant’s answer.
Let’s go around the room. For each group, I’m looking for: Which brand of watch did you test? How many of the 3 tasks did your participant complete? Your participant’s answer to the post-test question
Recall that usability has 3 components: effectiveness, efficiency, and satisfaction. Task completion rate is a measure of effectiveness. Subjective questions like the one you asked your participant help measure satisfaction. So we can calculate effectiveness and satisfaction ratings for each brand.
Any observations about your experience in your role?
Now I’d like to talk briefly about some best practices for interpreting test results
Review your notes with the project team as soon as possible after a round of tests Focus on the important things you learned – these tend to be fairly obvious Identify the problems first – you can figure out how to solve them later
Kayak problems – where the user encounters an issue, but quickly gets back on track Like someone who falls off a kayak, but gets back on immediately Ignore these in favour of issues that users couldn’t deal with easily
Resist the impulse to solve problems found in usability testing by adding explanations and instructions - Even if the participants suggest that this is the way to solve a problem, this may not necessarily be a good idea - Often, the real solution is to remove distractions and confusing elements
Address the “low-hanging fruit” first: issues where Both the problem and the solution are obvious, or Effort required to implement is minimal and the impact is highly visible
Thank you - I’ll be happy to take any questions Thanks again, and don’t forget to visit the Innovation Lab, where you can participate in usability tests and focus groups run by the Open Text UXD team! If you’re specifically interested in user adoption, there’s a focus group on that topic tomorrow at noon. Come by the Innovation Lab to sign up.