The document summarizes a heuristic evaluation of the Inbenta chatbot. Four evaluators tested the chatbot to find the address of Inbenta's Mexico office. They found violations in expectations setting, ability to maintain discussion themes, resilience to failure, interaction enjoyment, personality, and other heuristics. The chatbot was not able to fully help users discover the Mexico office address or answer most questions. It did not provide natural conversations and could only be used for general inquiries. Redesigning the chatbot with more information and addressing violated conversational patterns was suggested.
2. ● Inbenta chatbot: https://www.inbenta.com/en/
● 4 evaluators
Task description
You have an interview with Inbenta in a few days and you want to use Inbenta’s
chatbot to find out the address of Inbenta’s Mexico office
2/17
Evaluation process
Evaluator Di Pierro Impellizzeri Afridi Mosca
Time needed (s) 45 51 63 53
3. 3/17
Violated Heuristic
Expectation Setting
Gravity
0 1 3 4
2
The chatbot says that
it will help us and list
some things it can do
but does not specify
whether it can
perform other tasks
apart from those.
Disambiguate the
sentence specifying
if it can/cannot
perform also other
tasks
Solution
Description
4. 4/17
Violated Heuristic
Expectation Setting
Gravity
0 1 3 4
2
Solution
The chatbot
provides some
common questions
but it does not
understand simple
sentences like “What
are you able to do?”
It must tell the user
what are its
capabilities in
advance, instead of
offering a set of
questions which do
not clarify its
purposes
Description
5. 5/17
Violated Heuristic
Ability to maintain a
themed discussion
Gravity
0 1 3 4
2
The chatbot is not
able to answer to
questions correlated
with the previous
one
Make the chatbot at
least able to link the
current question
with the previous
one
Solution
Description
6. 6/17
Violated Heuristic
Resilience to failure
Gravity
0 1 3 4
2
Inbenta chatbot
doesn’t understand
query that is wrongly
typed or off topic
Eliminate
error-prone
conditions and
present users with a
confirmation option
before they commit
to the action.
Description Solution
7. 7/17
Violated Heuristic
Interaction enjoyment
Gravity
0 1 3 4
2
The chatbot is not
enjoyable at all, in
most cases it replies
asking for repeating
Enrich the chatbot
functionalities in
order to avoid these
frustrating situations
Description Solution
8. 8/17
Violated Heuristic
Personality
Gravity
0 1 3 4
2
The chatbot is not
able to understand
some greetings like
"Hi I am
Michelangelo"
Make the chatbot
understand the
different natural
ways people use to
start a conversation
Description Solution
9. 9/17
Violated Heuristic
Meet the neurodiverse
needs
Gravity
0 1 3 4
2
It is possible to
interact with Inbenta
chatbot using only
the keyboard
It should handle
user’s query given in
any manner like
voice recognition
Description Solution
10. 10/17
Violated Heuristic
Recognition and
facilitation of users’ goal
and intent
Gravity
0 1 3 4
2
The chatbot cannot
give me information
about the address of
Inbenta's Mexico
Office but it simply
provides a link in
which are listed all
the offices
Enrich the chatbot
in order to handle
specific questions
which are only
written in the
website
Description Solution
11. 11/17
Violated Heuristic
Flexibility and
Communication effort
Gravity
0 1 3 4
2
If the user adopts a
young language (i.e.
using acronyms), the
chatbot doesn’t
understand and asks
to repeat
Provide knowledge
about different
conversation styles
Description Solution
12. 12/17
Violated Heuristic
Recognition and
facilitation of users’
goal and intent
Gravity
0 1 3 4
2
The chatbot can
answer only to very
and generic questions
regarding products,
customers, offices
and technologies
Enrich the chatbot
in order to handle at
least some more
specific questions
Description Solution
13. 13/17
Violated Heuristic
Maximum of quantity
Gravity
0 1 3 4
2
If the user wants to
know more about
the products or what
is Inbenta, the
chatbot gives a long
answer, by opening
an additional
window
Allow the chatbot to
give more specific
and concise answers
Description Solution
16. Conversational UX patterns
Some conversational UX patterns are not respected.
● Opening (user) is never possible as the conversation starts always from the agent.
● Capabilities is not respected since the chatbot cannot answer to the question regarding
what it can do.
● Disengaging is not possible.
16/17
17. Conclusions
Inbenta chatbot is not able to fully help the user in discovering the Mexican Inbenta’s office
and it is not able to answer to most of questions. Moreover, it does not comply the user’s
expectations since it cannot provide a natural conversation. It can be used only in case we
want to explore some general information like products, customers and technology.
Our suggestion is to redesign it giving it much more information rather than providing
external links and respect the three conversational UX pattern that are violated.
17/17
