Slides used in workshop session A on "" at the IWMW 2007 event held at the University of York on 16-18 July 2007. See http://www.ukoln.ac.uk/web-focus/events/workshops/webmaster-2007/sessions/tonkin/

1. A centre of expertise in digital information management
www.ukoln.ac.uk
UKOLN is supported by:
Usability testing for the
WWW
Emma Tonkin
UKOLN
www.bath.ac.uk

2. A centre of expertise in digital information management
www.ukoln.ac.uk
Introduction
• UKOLN, the University of Bath
• Why this session?

3. A centre of expertise in digital information management
www.ukoln.ac.uk
Why do projects fail?
Project Impaired Factors % of the Responses
1. Incomplete Requirements 13.1%
2. Lack of User Involvement 12.4%
3. Lack of Resources 10.6%
4. Unrealistic Expectations 9.9%
5. Lack of Executive Support 9.3%
6. Changing Requirements & Specifications 8.7%
7. Lack of Planning 8.1%
8. Didn't Need It Any Longer 7.5%
9. Lack of IT Management 6.2%
10. Technology Illiteracy 4.3%
11. Other 9.9%

4. A centre of expertise in digital information management
www.ukoln.ac.uk
Introducing usability
• Definition: the measure of a product’s
potential to accomplish the goals of a user
• How easy a user interface is to understand
and use
• Ability of a system to be used [easily?
Efficiently? Quickly?]
• The people who use the project can
accomplish their tasks quickly and easily

5. A centre of expertise in digital information management
www.ukoln.ac.uk
Assumptions
• There are several dimensions to
usability
– Focus on users
– ‘People use products to be productive’
– Users are busy people trying to
accomplish tasks quickly
– Users decide when a product is easy to
use
• (Adapted from Redish & Dumas, A Practical Guide to User Testing)

6. A centre of expertise in digital information management
www.ukoln.ac.uk
However…
• Are users always busy? Does this imply that
usability is only present in the workplace?!
• Effectiveness vs. efficiency vs. satisfaction
• Do users know when a product is ready?
• Do all users agree about usability?
• Is usability actually measurable?
• Is there one statistic that == ‘% usability’?

7. A centre of expertise in digital information management
www.ukoln.ac.uk
Elements of usability
• Nielsen refers to five elements or
components of usability:
– Learnability
– Efficiency
– Memorability
– Errors
– Satisfaction
– Usability Engineering, 1993, p.26
• These may not be of equal importance in all
cases.

8. A centre of expertise in digital information management
www.ukoln.ac.uk
In other words…
• Usability depends on context
– What does the user want to do?
– Who is the user?
• Related to:
– Internationalisation; cultural, social issues
– Task analysis; working out what the user
wants to do (what the goal is) and how
he/she would expect to accomplish it

9. A centre of expertise in digital information management
www.ukoln.ac.uk
Science vs craft
• Formal approaches:
– Research-driven
– ‘hard science’
– Laboratory-based
• Informal approaches:
– Naturalistic, qualitative observations
– Informal setting

10. A centre of expertise in digital information management
www.ukoln.ac.uk
User model vs user testing
• Either we apply our understanding of
the way users act, and test the
interface that way
• Or we simply observe users...

11. A centre of expertise in digital information management
www.ukoln.ac.uk

12. A centre of expertise in digital information management
www.ukoln.ac.uk
A note about rule-based
testing/validation
• ‘Should be’ vs ‘is’ – model
vs reality
• Great handwriting does not
guarantee a compellingly
readable result

13. A centre of expertise in digital information management
www.ukoln.ac.uk
Scenario-based user testing
• Based around tasks
• Simple scenarios (‘hypothetical
stories’/’abstract-level test cases’):
– For a company web page, locating and
using contact details
– Registration and login to a wiki
• Process: provide a task and ask the
user to complete it
– It is important to test the right tasks!

14. A centre of expertise in digital information management
www.ukoln.ac.uk
Cognitive walkthrough
• Works something like this:
Task: Climb mountain and
find the highest peak

15. A centre of expertise in digital information management
www.ukoln.ac.uk
Required for CW
• A description of the interface
• A task scenario
• Assumptions: What knowledge does
the user already have?
• Functionality: What actions will
accomplish the task with this
interface?

16. A centre of expertise in digital information management
www.ukoln.ac.uk
Method:
• Look at each step that is required to
accomplish the task:
– Will the user try this step?
– Will the user notice that this action (control,
button, switch) is available?
– Will the user associate this action with the effect
that they are hoping for?
– If this action is performed, does it appear that
progress is being made?
• Can you 'tell a success story' for each step? If not,
there is a usability problem.

17. A centre of expertise in digital information management
www.ukoln.ac.uk
Recording your test:
• Create a diary format:
– Trying to achieve whatever:
• Looking for something that does whatever
• Found a button marked foo
• But clicking on foo took me to unrelated-
looking screen blah
• Like the mountain-climbing line, you
can go back and try another trajectory
– document this in a similar way.

18. A centre of expertise in digital information management
www.ukoln.ac.uk
Developing appropriate task
scenarios
• Probably the hardest thing about any
usability testing
• On the one hand, you are not required
to support very improbable scenarios.
• On the other hand, developing and
supporting probable scenarios is key
to a user-centred development
process.

19. A centre of expertise in digital information management
www.ukoln.ac.uk
Trying out a CW
• Who's got a mobile phone?
• In groups:
– Work out a couple of tasks.
– Working from the perspective of a user
with an appropriate level of knowledge
(you will have to define what that means!),
try the tasks. Document the result.

20. A centre of expertise in digital information management
www.ukoln.ac.uk
User testing (with real users!)
• The popular example is heuristic
evaluation.
• Heuristics are rules of thumb.
• Heuristic evaluation requires about six
people and a large amount of coffee.
• Provide them with a list of the ten (or
twelve, or...) heuristics, and ask them
to examine each page ('screen') for
problems, according to the heuristics.

21. A centre of expertise in digital information management
www.ukoln.ac.uk
Ten heuristics
• Visibility of system status: Does the system give timely & appropriate
feedback?
• Match between system and the real world: Is it speaking the users’ language?
• User control and freedom: How hard is it to undo unwanted actions?
• Consistency and standards: Does it follow conventions and expectations?
• Error prevention: Are potential errors recognised before becoming a problem?
• Recognition rather than recall: Does the system rely on the users’ memory?
• Aesthetic & minimalist design: Are dialogs cluttered with information?
• Help users recognise, diagnose & recover from errors: Are error messages
useful?
• Help and documentation: Is there online help? Is it useful?

22. A centre of expertise in digital information management
www.ukoln.ac.uk
Evaluating the results
• Again, a diary form can be helpful:
'Screen 1 violates heuristic 10
because...'
• Merge these notes.
• List by frequency order to see most
obvious bugs
• List by heuristic to see severity for your
purposes

23. A centre of expertise in digital information management
www.ukoln.ac.uk
Applying the results
Bug fixes
Feature requests
Major objections
Misnamed element
Confusing colours
It would be much easier if…
…‘this textbox autocompleted’
…’the system remembered my
email preferences’
I don’t like [type of application]
I prefer [totally different type of
application]
…(Oh)
Strange interaction flow
‘Low-
hanging
fruit’?

24. A centre of expertise in digital information management
www.ukoln.ac.uk
Testing layouts via greeked
text
• Wasn't going to talk about this, but it's
turned out to be useful
• Early stage of web site design often involves
developing layouts/templates
• Because no real content exists yet, these
may be hard to test using the above
methods
• However, a layout should communicate
something about page function. Does it?

25. A centre of expertise in digital information management
www.ukoln.ac.uk
Preparing a template
• Get greeked text from the Lorem
Ipsum generator:
– http://lorem-ipsum.perbang.dk/
• Place it into template. Do not leave a
single readable word!
• Make yourself a list of elements that
should be visible on the page
• Find/bribe about six test subjects

26. A centre of expertise in digital information management
www.ukoln.ac.uk
Example list:
• Main page content
• Page title
• Person responsible for page
• Navigation elements
• Last updated date
• Logo
• How to get back to the front page?
• News items

27. A centre of expertise in digital information management
www.ukoln.ac.uk
Testing process
• One user at a time:
• On each layout 'greeked', ask the user to
identify each element or group of elements. If
they can't find it, invite them to mark where they
think it ought to be.
• Asking the user to 'think aloud' can be helpful
• Also, ask the user to give a mark (out of ten, or
from -3 to +3, or whatever...) on 'subjective
appeal'
• Note: Randomising order reduces systematic
error

28. A centre of expertise in digital information management
www.ukoln.ac.uk
Coming up with a preamble
• This is a strange thing to ask someone to do.
Do not be surprised if you get some funny
looks.
• Come up with a short, reassuring introduction to
the test. Useful items to include:
– Introducing the software (purpose, not detail)
– Your participation will help us...
– Remember, we are testing the software, not
your performance...
– Please think out loud...
– This style of test helps us to...

29. A centre of expertise in digital information management
www.ukoln.ac.uk
Examining the results
• Build a table:
• As the layout is improved, the number
of misidentified elements should
reduce