Presentation title: Heuristic usability evaluation of user interfaces for a semi-autonomous vineyard robot sprayer Presenter: George Adamides

1. Heuristic usability evaluation of
user interfaces for a semi-
autonomous vineyard robot
sprayer
George Adamides
Open University of Cyprus

2. What is and why do we careabout usability in a
human-robot interaction system?
• A user interface that supports human-robot interaction
(HRI) needs to meet specific non-functional
requirements, such as reliability, efficiency and usability.
• Usability refers to whether a system can be used with
effectiveness, efficiency, and satisfaction with which
users achieve specified goals in a particular context of
use.
• A usability issue is anything that can affect the user
experience in a negative way.
• What makes a robotic interface effective is no different
than what makes anything else usable, be it a door
handle or a piece of software.
• “Cut” and “Paste” or “Undo” does not always work!

3. HRI usability evaluation
• Heuristic evaluation is a ‘discount usability engineering’
method for evaluating user interfaces to find their
usability problems.
• “discount” because a small number of evaluators, usually
3 to 7, suffice to reliably evaluate the usability of a user
interface against a list of heuristics (the usability
principles).

4. Which usability principles we used?
Adamidesetal(2015)
1. Platform architecture and scalability: “Provide the flexibility
to iterate robotic and computing technological
developments in the UI of the HRI system.”
2. Error prevention and recovery: “Provide information and
alerts to avoid and recover from user errors.”
3. Visual design: “Provide an aesthetic, clear, and simple design
of the UI with the relevant information necessary.”
4. Information presentation: “Provide the necessary
information, in the right context, moment, and modality.”
5. Robot state awareness: “The knowledge that the robot has
about its own systems’ situation and the information it gives
to the operator about its health status and mode of
operation.”

5. 6. Interaction effectiveness and efficiency: “Provide efficient
and effective interactions between human and robot.”
7. Robot environment/surroundings awareness: “Provide
spatial information about the robot’s surroundings and the
environment where it is operating.”
8. Cognitive factors: “Use mental models and metaphors to
lower the cognitive load.”
Source: G. Adamides, G. Christou, C. Katsanos, M. Xenos, and T.
Hadzilacos, "Usability Guidelines for the Design of Robot
Teleoperation: A Taxonomy," IEEE Transactions on Human-
Machine Systems, vol. 45, pp. 256-262, 2015.
Which usability principles we used?
Adamidesetal(2015)

6. Semi-Autonomous Vineyard Robot Sprayer
Sprayer nozzle
End-effector camera
Peripheral camera
Sprayer tank
Main pan-tilt zoom camera
Summit XL robot platform

7. 3 systems under evaluation
SAARS V0
• On-screen controls for robot movement and camera movement
• Presentation of camera views
• Elements for displaying sensor information (visual and auditory feedback) for
distance from the robot sides and battery level.
• Can use the entire screen and support interaction though either the keyboard
or the mouse.

8. 3 systems under evaluation
SAARS V1• Similar to V0
• Plus functionality for target pointing.
• SAARSv1 supports both manual (user points to targets) and automated target
specification through a pattern recognition algorithm.

9. 3 systems under evaluation
SAARS V2
• Similar to V1
• Added feedback from a laser scanner.

10. HRI heuristic evaluation
• Four usability experts conducted a heuristic usability
evaluation on three user interfaces.
• The evaluators were situated at the Hellenic Open
University Software Quality Assessment laboratory at
Patra, Greece and remotely controlled (over HTTP) the
robot, which was located in Cyprus at the Open
University of Cyprus premises. An appropriate simulation
environment was created, including various paths and
targets.

11. HRI heuristic evaluation results –
SAARS V0
• 13 usability issues were identified.
• Most (77%) of these usability issues were related to
violations of the following four heuristics: a) 23% were
violations of heuristic 4 (Information presentation), b)
23% were violations of heuristic 5 (Robot state
awareness), c) 15% were violations of heuristic 6
(Interaction effectiveness and efficiency) and d) 15%
were violations of heuristic 8 (Cognitive factors).
• All in all, the system is at a satisfactory level of usability.

12. HRI heuristic evaluation results –
SAARS V1
• 10 usability issues were identified.
• Most (80%) of these usability issues were related to
violations of the following four heuristics: a) 20% were
violations of heuristic 4 (Information presentation), b)
20% were violations of heuristic 5 (Robot state
awareness), c) 20% were violations of heuristic 6
(Interaction effectiveness and efficiency) and d) 20%
were violations of heuristic 8 (Cognitive factors).
• These findings tend to provide support that the system is
at a good level of usability.

13. HRI heuristic evaluation results –
SAARS V2
• 3 usability issues were identified.
• These issues were related to violations of the following
three heuristics: a) one violation of heuristic 3 (Visual
design), b) one violation of heuristic 6 (Interaction
effectiveness and efficiency), and c) one violation of
heuristic 7 (Robot environment/surroundings
awareness).
• All in all, the system is at a very good level of usability.

14. Conclusion – Recommendations
• These findings provide evidence that the final version of the system provides
satisfactory services to its typical users.
• This can be attributed to the iterative design, development and evaluation
process followed.
• These advantages, combined with the increased usability of the SAARv2 (final)
system, may result in high adoption from its end users.
• However, there is always room for improvement. The expert evaluators argued
that a next version of the system could benefit from:
• a) an embedded representation of the robot’s body in the user interface
displaying sensors’ information and robot’s direction in relation to the
active camera views (heuristic 7),
• b) embedded help explaining functionality and controls (heuristic 8), e.g.
simplify and explain algorithmic settings for automated target
identification, embed tooltips and/or labels on the buttons related to user-
defined targets,

15. Conclusion – Recommendations
• c) mechanisms for error prevention in target identification and
spraying (heuristic 2), e.g. confirmation message for the “erase-
all-targets” action,
• d) additional information that is important for the task (heuristic
4) e.g. remaining level of spraying liquid,
• e) improvements in the visual design of the user interface
(heuristic 3), e.g. visual clarification for currently active control,
larger text labels to increase readability.
Latest news: A new robot with added functionality is currently
being built with a robotic arm. New user interface is also designed
based on the above recommendations.

16. THANK YOU FOR YOUR ATTENTION!
George Adamides
Open University of Cyprus – http://www.ouc.ac.cy
Email: george.adamides@st.ouc.ac.cy
Agricultural Research Institute – http://www.ari.gov.cy
Email: gadamides@ari.gov.cy