The document discusses different methods for evaluating user interface designs, including expert evaluation techniques like heuristic evaluation and cognitive walkthroughs. It also covers user testing, which is considered more reliable than expert evaluation alone. Formative evaluation involves testing prototypes during development to identify issues, while summative evaluation assesses the final product. Both qualitative and quantitative methods are important to identify usability problems from the user's perspective.

2.  Nowadays users expect much more than just a
usable system; they also look for a pleasing
and engaging experience. This means it is even
more important to carry out an evaluation.
 Imagine you have designed an app for teenagers to
share music, gossip, and photos. You have
prototyped your first design and implemented the
core functionality. How would you find out
whether it would appeal to them and if they will
use it? You could need to evaluate it – but how?

3.  In our human-centred approach to design, we
evaluate designs right from the sketches and then
as the project progresses we keep improving our
prototypes and evaluate those.

5.  Basically two types evaluation:
◦ Formative Evaluation
◦ Summative Evaluation

6.  Formative evaluations involve evaluating a
product or service during development. The goal
is to make improvements in the design prior to
release. This means identifying or diagnosing the
problems, making and implementing
recommendations, and then evaluating again.
Formative usability is always done before the
design has been finalized. In fact, the earlier the
formative evaluation, the more impact the
usability evaluations will have on the design.

7.  Here are a few key questions you will be able
answer with a formative approach:
• What are the most significant usability issues
preventing users from accomplishing their goals or
resulting in inefficiencies?
• What aspects of the product work well for the
users? What do users find frustrating?
• What are the most common errors or mistakes
users are making?

8.  Here are a few key questions you will be able
answer with a formative approach:
• What are the most significant usability issues
preventing users from accomplishing their goals or
resulting in inefficiencies?
• What aspects of the product work well for the
users? What do users find frustrating?
• What are the most common errors or mistakes
users are making?

9. • Are improvements being made from one design
iteration to the next?
• What usability issues can you expect to remain
after the product is launched?

10.  Summative usability is about evaluating the dish
after it comes out of the oven. The usability
specialist running a summative test is like a food
critic who evaluates a few sample dishes at a
restaurant or perhaps compares the same meal in
multiple restaurants. The goal of summative
usability is to evaluate how well a product or piece
of functionality meets its objectives. Summative
testing can also be about comparing several
products to each other.

11.  Summative usability evaluations answer these
questions:
• What is the overall usability of our product?
• How does our product compare against the
competition?
• Have we made improvements from one product
release to the next?
. Summative testing can also be about comparing
several products to each other.

12.  Two main types
◦ Expert Evaluation
 Heuristic Evaluation
 Cognitive Walkthroughs
◦ User Testing

13.  A simple, relatively quick and effective method of
evaluation is to get an interaction design, or
usability, expert to look at the system and try
using it
 this is no substitute for getting real people to
use your design
 expert evaluation is effective, particularly early in
the design process.

14. Most usability engineering methods will
contribute substantially to the usability of an
interface …
…if they are actually used.

15.  What is it?

16.  What is it?
A discount usability engineering method
- Easy (can be taught in ½ day seminar)
- Fast (about a day for most evaluations)
- Cheap

17.  How does it work?

18.  How does it work?
◦ Evaluators use a checklist of basic usability
heuristics
◦ Evaluators go through an interface twice
 1st pass get a feel for the flow and general scope
 2nd pass refer to checklist of usability heuristics
and focus on individual elements
◦ The findings of evaluators are combined and
assessed

19. Nielsen: 10 Usability Heuristics
(based on extensive empirical testing)
•Visibility of system status
• Match between system and
the real world
• User control and freedom
Consistency
•Error prevention
• Recognition not recall
• Flexibility and efficiency
• Aesthetic and minimalist
design
• Help users diagnose and
recover from errors
• Help and documentation

20.  One expert won’t do
 Need 3 - 5 evaluators
 Exact number
needed depends on
cost-benefit analysis

21.  Debriefing session
◦ Evaluators rate the severity of all problems
identified
◦ Use a 0 – 4, absolute scale
 0 I don’t agree that this is a prob at all
 1 Cosmetic prob only
 2 Minor prob – low priority
 3 Major prob – high priority
 4 Usability catastrophe – imperative to fix

22.  How does H.E. differ from User Testing?

23.  How does H.E. differ from User Testing?
◦ Evaluators have checklists
◦ Evaluators are not the target users
◦ Evaluators decide on their own how they want to
proceed
◦ Observer can answer evaluators’ questions about
the domain or give hints for using the interface
◦ Evaluators say what they didn’t like and why;
observer doesn’t interpret evaluators’ actions

24.  What are the shortcomings of H.E.?

25.  What are the shortcomings of H.E.?
◦ Identifies usability problems without indicating
how they are to be fixed.
 “Ideas for appropriate redesigns have to appear
magically in the heads of designers on the
basis of their sheer creative powers.”

27.  Red is used both for help messages and for error
messages (consistency, match real world)
 “There is a problem with your order”, but no
explanation or suggestions for resolution (error
reporting)

28.  No “Continue shopping" button (user control &
freedom)
 Recalculate is very close to Clear Cart (error
prevention)
 “Check Out” button doesn’t look like other
buttons (consistency, both internal & external)
 Must recall and type in cart title to load
(recognition not recall, error prevention,
efficiency)

29.  Expert Evaluation
◦ Heuristic Evaluation
◦ Cognitive Walkthroughs
 User Testing

31.  The cognitive walkthrough was originally
designed as a tool to evaluate walk-up-and-use
systems like postal kiosks, automated teller
machines (ATMs), and interactive exhibits in
museums where users would have little or no
training. However, the cognitive walkthrough has
been employed successfully with more complex
systems like CAD software and software
development tools to understand the first
experience of new users.

32.  The cognitive walkthrough is a usability
evaluation method in which one or more
evaluators work through a series of tasks and ask a
set of questions from the perspective of the user.
 Inputs:
◦ Prototype
◦ Task
◦ Sequence of actions to do the task in the
prototype
◦ User analysis

33.  For each action the evaluation asks the following
questions:
 What is the user goal and why?
Is the action obviously available?
Does the action or label match the goal?
 Is there good feedback?

36.  Expert evaluation is useful but there is no
comparison to using the actual users for
evaluation.
 There are many different types of user evaluations
for example
◦ Formative Evaluation
◦ Field Study
◦ Controlled Experiments

37.  Usability testing involves collecting of data using a
combination of methods in a controlled setting,
for example, experiments that follow basic
experimental design, observation, interviews, and
questionnaires. Often, usability testing is
conducted in labs, although increasingly
interviews and other forms of data collection are
being done remotely via phone and digital
communication (for instance, through Skype or
Zoom) or in natural settings.

38.  The primary goal of usability testing is to
determine whether an interface is usable by the
intended user population to carry out the tasks for
which it was designed. This involves investigating
how typical users perform on typical tasks. By
typical, we mean the users for whom the system
is designed (for example, teenagers, adults, and so
on) and the activities that it is designed for
them to be able to do (such as, purchasing the
latest fashions).

39.  As users perform the tasks, they may be recorded
on video. Their interactions with the software may
also be recorded, usually by logging software. User
satisfaction questionnaires and interviews can
also be used to elicit users’ opinions about how
they liked the experience of using the system.