This document presents methods for evaluating the usability and sociability of systems, including usability testing, eye tracking, and standardized evaluation. It summarizes a case study applying these methods to evaluate an existing SEEKDA system and develop a revised prototype. Usability testing identified critical issues and design recommendations. A prototype was created and evaluated through an online participatory design workshop with users. The document concludes by describing how explorations and user walkthroughs can be used.

1. Hands-on experiences with
usability/sociability design
www.insemtives.eu 1
Roberta Cuel, University of Trento, IT; Markus
Rohde, University of Siegen, DE and
Germán Toro del Valle, Telefonica I+D, ES
ISWC 2010

2. Overview
1/30/2015 www.insemtives.eu 2
• Presentation of methods
– Usability/Sociability Testing
– Evaluation methodology
• Evaluation SEEKDA case study
– Old system
– Evaluation
– Revision/Prototype
– Online PD workshop
• Exploration/User walkthrough

3. PRESENTATION OF METHODS
Presentation of methods
Evaluation SEEKDA case study
Exploration/User walkthrough
1/30/2015 www.insemtives.eu 3

4. Usability/sociability testing
1/30/2015 www.insemtives.eu 4

5. Usability/sociability Testing (2)
1/30/2015 www.insemtives.eu 5
Eye tracking - calibration
Lab test design
Eye tracks -
results

6. Usability-design methods
12
Effectivity
Efficiency
Satisfaction
Suitability for the task
Efforts to fulfill the
task
Positive attitute
towards using the
applicationInterview
Usability Tests
Post Interview
Usability-Potential =
Relation between what a certain
application can and what the
application should can (user
expectation)
SUMI Questionnaire =
Software Usability Measurement
Inventory
1/30/2015 www.insemtives.eu 6
Usability means that certain users in a certain
context and a certain amount of time can fulfill
certain tasks effectively, efficently, and satisfying
(ISO 9241-11)

7. Standardized method (DATech)
13
Usability tests with thinking aloud and
participant observation; eye tracking &
biometric measurements
Identification of critical incidents)
Normative Evaluation with
ISO 9241-10
Design Recommendations
Validation of relevant usage scenarios;
target group(s)
Evaluation of Usability-Potential by Interviews
1/30/2015 www.insemtives.eu 7

8. Usability report: screenshot
131/30/2015 www.insemtives.eu 8
Expectation Conformity
Self Descriptiveness
Expectation Conformity

9. Usability report: video
131/30/2015 www.insemtives.eu 9
Keyboard on a copy station Touchable display

10. Usability report: PGP
131/30/2015 www.insemtives.eu 10

11. Usability report: PGP (2)
131/30/2015 www.insemtives.eu 11

12. 131/30/2015 www.insemtives.eu 12
Critical Incident Normative Evaluation ISO 9241-10 Design Recommendation
User looks at the result list of a
service search. Below the service title
and a provider information there can
be found a short description with the
grey addition “Community
Description”. User clicks on this grey
addition and nothing happens. User
had expected that he would click on
an active link and that he would get
additional descriptions/information
as a result.
Expectation Conformity
All dialogue elements should be
designed complient and consistent to
support the users‘ expectations.
DR 1: The grey annotation
“Community Description” is
associated with the users’
expectation of active links to click on.
A different design of this information
could help to avoid this conflict with
user expectations.
User clicks on the “Comments”
button and finds something looking
like a discussion thread (tree
structure).
Self Descriptiveness
All dialogue elements should be
designed in a self-descriptive way.
DR 2: The interaction opportunities
for end users should be introduced
and described somewhere.
Usability report: CI and DR

13. Usability report: Statistics
131/30/2015 www.insemtives.eu 13
1% Errors
22% Self Descriptiveness
45% Expectation Conformity
6% Error Tolerance
5% Controllability
21% Suitability for the task

14. PRESENTATION OF METHODS
Presentation of methods
Evaluation SEEKDA case study
Exploration/User walkthrough
1/30/2015 www.insemtives.eu 14

15. Usability evaluation SEEKDA
131/30/2015 www.insemtives.eu 15

16. Usability evaluation SEEKDA (2)
131/30/2015 www.insemtives.eu 16
> REPORT <
Requirements Specification:

17. Prototype
131/30/2015 www.insemtives.eu 17

18. Online Participatory Design
131/30/2015 www.insemtives.eu 18

19. Online Participatory Design (2)
131/30/2015 www.insemtives.eu 19

20. Online PD Dashboard
131/30/2015 www.insemtives.eu 20

21. Online PD Dashboard (2)
131/30/2015 www.insemtives.eu 21

22. PRESENTATION OF METHODS
Presentation of methods
Evaluation SEEKDA case study
Exploration/User walkthrough
1/30/2015 www.insemtives.eu 22