The document discusses heuristic evaluation for usability, critiquing the reliance on Nielsen's principles and proposing a new set of 15 principles paired with 60 questions for a more detailed assessment. It emphasizes the need for combining various recommendations to enhance precision in usability evaluation and introduces a new rating system based on problem severity. The findings from evaluations suggest that the new approach is more comprehensive, offering a better framework for analyzing user interfaces.

1. Usability Evaluation with Heuristics,
Beyond Nielsen’s List
Dr. Toni Granollers

2. BUT
The problem
• Heuristic Ealuation (EH)
– Effective, UCD based and widely used
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 2

3. BUT
– Many people believes that Nielsen’s list is “the
list” and “the method”
– It is impossible to assess any single principle
without the help of specific questions for each
principle
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 3
ignoring that these principles are broad rules
of thumb and not specific usability guidelines
J. Nielsen, 1995 “10 Usability Heuristics for User Interface Design“

4. BUT
• If the evaluator wants to be more precise
he/she must combine different
recommendation sources to review the
specific application domain.
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 4
• Time consuming
• Conflicts
• Different scoring systems

5. BUT
• The severity of a usability problem is a combination of
three factors:
– frequency + impact + persistence
– 0 to 4 rating scale
• 0 = not a problem at all
• 1 = Cosmetic problem only
• 2 = Minor usability problem
• 3 = Major usability problem
• 4 = Usability catastrophe
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 5
Nielsen himself declares that his
proposal is a good measure but “It is
difficult to get good
severity estimates from
the evaluators during a
heuristic evaluation
session”
An idea also shared by UX professionals and
researchers such as J. Sauro who argues that it
is usually difficult to distinguish
easily between too many rating
levels

6. Proposal
Integration
• New list:
• 15 principles
• 60 questions
Rating method
Experimentation
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 6

7. combination of
Usability Heuristics for User
Interface Design by J. Nielsen
First Principles of Interaction
Design by B. Tognazzini
• Revised & Expanded, Mar’14
• The following principles are
fundamental to the design and
implementation of effective
interfaces, whether for
traditional GUI environments,
the web, mobile devices,
wearables, or Internet-
connected smart devices
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 7
Integration

8. Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 8
Integration

9. 15 principles
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 9
Integration

10. 60 questions
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 10
Integration

11. • Every principle has a set of interrogative
questions written in a way that is favourable
to usability
• The evaluator has to answer every question
with one of these 4 options (simplification):
• “Yes”
• “No”
• “Neither”
• “Not applicable”
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 11
Rating method

12. Answer Coments
Do link labels have the same names as their destinations? Neither
Do the same actions always have the same results? Yes
Do the icons have the same meaning everywhere?
Not applicable-It
is not a problem
Is the information displayed consistently on every page? No
Are the colours of the links standard? If not, are they suitable for its use? Yes
Do navigation elements follow the standards? (Buttons, check box, ...) Yes
4 - Consistency and standards
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 12
1
1
1
0
-
0,5
3,5
Rating method

13. Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 13
Rating method
1- Visibility and system state 5
2 - Connection between the system and the real world,
metaphor usage and human objects
3
3 - User control and freedom 3
4 - Consistency and standards 5
5 - Recognition rather than memory, learning and anticipation 5
6 - Flexibility and efficiency of use 5
7 - Help users recognize, diagnose and recover from errors 3
8 - Preventing errors 2
9 - Aesthetic and minimalist design 4
10 - Help and documentation 0
11 - Save the state and protect the work 2
12 - Color and readability 4
13 - Autonomy 2
14 - Defaults 0
15 - Latency reduction 1
Porcentaje de usabilidad
"Usability" percentage
77,2%Usability Percentage
global
evaluation
point of view
Quantitative
value

14. Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 14
http://www.eps.udl.cat
7 evaluators
• 3 >10y experience
• 2 > 5y experience
• 1 PhD candidate
• 1 Student supervised
Experimentation

15. Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 15
Eval1 Eval2 Eval3 Eval4 Eval5 Eval6 Eval7
1- Visibility and system state 5 5 4 4,5 4 5 4
2 - Connection between the system and the real world,
metaphor usage and human objects
3 2 4 4 3,5 4 2
3 - User control and freedom 3 2 3 2 2 1 1,5
4 - Consistency and standards 5 4 5 5,5 4 4 3,5
5 - Recognition rather than memory, learning and anticipation 5 4 4 5 5 5 4
6 - Flexibility and efficiency of use 5 3 3 6 3 5 3
7 - Help users recognize, diagnose and recover from errors 3 3 0 0 2 2 4
8 - Preventing errors 2 2 2 0 2 2 1
9 - Aesthetic and minimalist design 4 3 4 3 4 4 1
10 - Help and documentation 0 0 0 0 0,5 0 0
11 - Save the state and protect the work 2 0 0 1 1 0 0
12 - Color and readability 4 3 2,5 4 4 2 2
13 - Autonomy 2 2,5 2 3 3 3 2
14 - Defaults 0 0 0 2 1 0 2
15 - Latency reduction 1 1 0 0 1 0 0
Porcentaje de usabilidad
"Usability" percentage
77,2% 69,0% 83,8% 87,0% 72,7% 86,0% 66,7%
Experimentation

16. Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 16
MEAN
77,5%
Experimentation

17. Conclusions
• the new list is certainly an improvement
• the questions cover all the aspects to carry
out a proper UI analysis.
– Certainly, they can be enhanced, but it is a
reasoned and well-balanced list, with an
appropriate number of questions for each
principle
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 17
unanimous consensus among the evaluators:
this new list is a bit larger than Nielsen’s, but it is much
more complete

18. Conclusions
• good decision :
– questions in “question form”
– described with the same usability direction:
affirmative answer always representing good
usability
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 18
This may seem unimportant,
even trivial, but it is not usual
to find it in the previous works

19. Conclusions
• The Usability Percentage
– must be taken as an orientation of the usability
level.
– beyond its value, we can compare the results of
every single principle.
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 19
Eval1 Eval2 Eval3 Eval4 Eval5 Eval6 Eval7
1- Visibility and system state 5 5 4 4,5 4 5 4
2 - Connection between the system and the real world,
metaphor usage and human objects
3 2 4 4 3,5 4 2
3 - User control and freedom 3 2 3 2 2 1 1,5
4 - Consistency and standards 5 4 5 5,5 4 4 3,5
5 - Recognition rather than memory, learning and anticipation 5 4 4 5 5 5 4
6 - Flexibility and efficiency of use 5 3 3 6 3 5 3
7 - Help users recognize, diagnose and recover from errors 3 3 0 0 2 2 4
8 - Preventing errors 2 2 2 0 2 2 1
9 - Aesthetic and minimalist design 4 3 4 3 4 4 1
10 - Help and documentation 0 0 0 0 0,5 0 0
11 - Save the state and protect the work 2 0 0 1 1 0 0
12 - Color and readability 4 3 2,5 4 4 2 2
13 - Autonomy 2 2,5 2 3 3 3 2
14 - Defaults 0 0 0 2 1 0 2
15 - Latency reduction 1 1 0 0 1 0 0

20. Future work
• Consider to change used severity rate:
– Yes - High [value: 3]
– Yes - Medium[value: 2]
– Yes - Low [value: 1]
– Not a problem [value: 0]
– Not aplicable [value: eliminated]
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 20

21. Future work
• To devolop a online tool for managing
evaluations
• Do more experimentations:
– larger evaluations
– diferent contexts
• Develop the “principle by principle”
comparision
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 21

22. Thanks for
your
attencion !!!
Toni Granollers
Polytechnic School
University of Lleida (UdL)
C/ Jaume II, 69
25001 – Lleida (Catalonia)
(+34) 973 702750
antoni.granollers@udl.cat
@DCU_MPIUA
slideshare.net/DCU_MPIUA


@
Usability Evaluation with Heuristics, Beyond Nielsen’s List [Toni Granollers - 2018] 22